Re: SD card, exfat, file mtime wrong
Hi, thank you for this work :-) I have just done a quick test with this driver. It seems to work as expected. I have read directly from the SD-card with old and new photo-files. time of exifdata and mtime are still matching (tested with different timezone offset +1.00 and +2.00 in the files) It would be nice if more people can test this in different timezones. I will use this driver as a default now. (exfat-fuse 1.3.0+git20220115-1) with exfat-utils 1.3.0-2 If some problem comes up, i will let you know. Just an important info for other users: I have black listed the kernel exfat driver in order to use only the exfat-fuse driver as described here: https://wiki.debian.org/KernelModuleBlacklisting Also, i have made a symbolic link to the exfat-fuse mount helper program ;-) best regards Reiner Am 15.01.22 um 22:44 schrieb Sven Hoexter: On Tue, Dec 14, 2021 at 10:32:08AM +0100, Reiner wrote: the timestamp problematic is still not solved in the upstream linux kernel. As a workaround you could use the exfat-fuse implementation, but it must be a newer version then the latest stable in debian. There is no newer stable release in upstream either. Must be build from git! Hi, I'm a bit late but I took this thread as motivation to upload a snapshot of the exfat fuse driver to experimental. The package should also build on bullseye if that is a target platform for you. I still plan to drop the package asap from the Debian archive, because I believe the in kernel driver is the way to move on, but for now there might be a niche case for the fuse driver. If you can give it a try please let me know if it works for you. Sven
Re: SD card, exfat, file mtime wrong
On Tue, Dec 14, 2021 at 10:32:08AM +0100, Reiner wrote: > the timestamp problematic is still not solved in the upstream linux kernel. > > As a workaround you could use the exfat-fuse implementation, but it must be > a newer version then the latest stable in debian. > > There is no newer stable release in upstream either. Must be build from git! Hi, I'm a bit late but I took this thread as motivation to upload a snapshot of the exfat fuse driver to experimental. The package should also build on bullseye if that is a target platform for you. I still plan to drop the package asap from the Debian archive, because I believe the in kernel driver is the way to move on, but for now there might be a niche case for the fuse driver. If you can give it a try please let me know if it works for you. Sven
Re: SD card, exfat, file mtime wrong
On 12/14/21 1:32 AM, Reiner wrote: Hi all, the timestamp problematic is still not solved in the upstream linux kernel. As a workaround you could use the exfat-fuse implementation, but it must be a newer version then the latest stable in debian. There is no newer stable release in upstream either. Must be build from git! I have exfat-fuse build with the patch below and it works fine. See this: https://github.com/relan/exfat/commit/75f28b558a1a86b45308a3e7d0af222624bc2c61 and this: https://github.com/relan/exfat/pull/119 regards Reiner Stallknecht Is there a Linux bug report? Do you have a URL? Thank you for identifying the exfat-fuse and patch work-around. :-) David
Re: SD card, exfat, file mtime wrong
Hi all, the timestamp problematic is still not solved in the upstream linux kernel. As a workaround you could use the exfat-fuse implementation, but it must be a newer version then the latest stable in debian. There is no newer stable release in upstream either. Must be build from git! I have exfat-fuse build with the patch below and it works fine. See this: https://github.com/relan/exfat/commit/75f28b558a1a86b45308a3e7d0af222624bc2c61 and this: https://github.com/relan/exfat/pull/119 regards Reiner Stallknecht
Re: SD card, exfat, file mtime wrong
On Wed 08 Dec 2021 at 16:37:25 (+1100), David wrote: > On Wed, 8 Dec 2021 at 16:16, David Christensen wrote: > > On 12/7/21 7:28 PM, David wrote: > > > On Wed, 8 Dec 2021 at 14:07, David Christensen wrote: > > >> On 12/6/21 6:37 PM, David Christensen wrote: > > > > I might also check this info > > >https://unix.stackexchange.com/a/640906 > > > that I mentioned before, just to see if it was relevant. I didn't > > > look closely. > > > It looks like vfat only. > > I wonder did you note this part? ... > """ > Seems like the problem resides in the Linux kernel itself, as the > timezone may (and usually do) differ between the kernel and the > userspace. time.c file in the kernel/time in the Linux kernel source > tree holds (and exports) the struct timezone sys_tz, which is then > used in fs/fat/misc.c in the FAT time <-> UNIX time conventions. The > tz_minuteswest field of this struct is used to show the difference > between the current timezone and the UTC, and it is taken into > consideration if the tz=UTC option is not passed to the mount.vfat > command. > """ > Which I do not read as necessarily being "vfat only". > > Because perhaps you are likewise not passing that option when > mounting exfat, maybe because it does not even have that > option, per my other links. > > In your situation, I would run the test code given there to > see if it is relevant, and if there is any difference with the > older Debian versions that do work. It's probably worth reading the Release Notes for bullseye, §2.2.8, and making sure that you're using the appropriate utilities. That section indicates the main change that has occurred. If the camera is just a camera, my advice would still be to use UTC. It's astonishing how complicated using anything else can become, as /everything/ that touches the filesystem and the image and its thumbnails etc etc has to get it absolutely right. Cheers, David.
Re: SD card, exfat, file mtime wrong
On Wed, 8 Dec 2021 at 16:16, David Christensen wrote: > On 12/7/21 7:28 PM, David wrote: > > On Wed, 8 Dec 2021 at 14:07, David Christensen > > wrote: > >> On 12/6/21 6:37 PM, David Christensen wrote: > > I might also check this info > >https://unix.stackexchange.com/a/640906 > > that I mentioned before, just to see if it was relevant. I didn't > > look closely. > It looks like vfat only. I wonder did you note this part? ... """ Seems like the problem resides in the Linux kernel itself, as the timezone may (and usually do) differ between the kernel and the userspace. time.c file in the kernel/time in the Linux kernel source tree holds (and exports) the struct timezone sys_tz, which is then used in fs/fat/misc.c in the FAT time <-> UNIX time conventions. The tz_minuteswest field of this struct is used to show the difference between the current timezone and the UTC, and it is taken into consideration if the tz=UTC option is not passed to the mount.vfat command. """ Which I do not read as necessarily being "vfat only". Because perhaps you are likewise not passing that option when mounting exfat, maybe because it does not even have that option, per my other links. In your situation, I would run the test code given there to see if it is relevant, and if there is any difference with the older Debian versions that do work.
Re: SD card, exfat, file mtime wrong
On 12/7/21 7:28 PM, David wrote: On Wed, 8 Dec 2021 at 14:07, David Christensen wrote: On 12/6/21 6:37 PM, David Christensen wrote: I verified the clock in the camera before taking the picture. It was set to local time. Debian 9 displays the correct mtime when I mount the SDXC card. Debian 10 displays the correct mtime when I mount the SDXC card. Debian 11 displays the wrong mtime when I mount the SDXC card. How do I fix or work around this feature, bug, or whatever in Debian 11? What package do I file a bug report against? Based on the above regression I would suggest to file a bug with the above information in Debian against the mount package. https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1001306 I might also check this info https://unix.stackexchange.com/a/640906 that I mentioned before, just to see if it was relevant. I didn't look closely. It looks like vfat only. 2021-12-07 19:03:13 dpchrist@laalaa ~ $ dpkg-query -S /usr/bin/mount dpkg-query: no path found matching pattern /usr/bin/mount You've unfortunately hit a bug/breakage that is currently the subject of some controversy in Debian. See here, or search for "merge(d) user usr": https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1000715#10 Yuck. David
Re: SD card, exfat, file mtime wrong
On Wed, 8 Dec 2021 at 14:07, David Christensen wrote: > On 12/6/21 6:37 PM, David Christensen wrote: > I verified the clock in the camera before taking the picture. It was > set to local time. > Debian 9 displays the correct mtime when I mount the SDXC card. > Debian 10 displays the correct mtime when I mount the SDXC card. > Debian 11 displays the wrong mtime when I mount the SDXC card. > How do I fix or work around this feature, bug, or whatever in Debian 11? > What package do I file a bug report against? Based on the above regression I would suggest to file a bug with the above information in Debian against the mount package. I might also check this info https://unix.stackexchange.com/a/640906 that I mentioned before, just to see if it was relevant. I didn't look closely. > 2021-12-07 19:03:13 dpchrist@laalaa ~ > $ dpkg-query -S /usr/bin/mount > dpkg-query: no path found matching pattern /usr/bin/mount You've unfortunately hit a bug/breakage that is currently the subject of some controversy in Debian. See here, or search for "merge(d) user usr": https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1000715#10
Re: SD card, exfat, file mtime wrong
On 12/6/21 6:37 PM, David Christensen wrote: debian-user: I recently installed debian-live-11.1.0-amd64-xfce+nonfree.iso on a Dell Latitude E6520: 2021-12-06 17:52:45 dpchrist@laalaa ~ $ cat /etc/debian_version ; uname -a 11.1 Linux laalaa 5.10.0-9-amd64 #1 SMP Debian 5.10.70-1 (2021-09-30) x86_64 GNU/Linux I am in California, USA. My timezone is Pacific Standard Time (PST). UTC is 8 hours ahead. tzselect(1) calls my timezone "America/Los_Angeles". When I insert a SDXC card formatted with exfat into a USB adapter, plug the adapter into the laptop, and double-click on the icon that appears on the desktop, the SD card filesystem is mounted: 2021-12-06 17:52:53 dpchrist@laalaa ~ $ mount | grep media /dev/sdc1 on /media/dpchrist/**redacted** type exfat (rw,nosuid,nodev,relatime,uid=**redacted**,gid=**redacted**,fmask=0022,dmask=0022,iocharset=utf8,errors=remount-ro,uhelper=udisks2) When I list the files on the SD card, the mtime is wrong. For example, here is a photograph that I took about ten minutes ago: 2021-12-06 18:13:02 dpchrist@laalaa /media/dpchrist/**redacted**/DCIM/**redacted** $ ll IMG_6947.JPG -rwxr-xr-x 1 dpchrist dpchrist 3988022 2021-12-06 10:02:57 IMG_6947.JPG It appears that mount(8) and/or the exfat driver consider the mtime to be in UTC, and are applying my timezone adjustment of -08:00 hours (?). I would like Debian 11 to display the mtime without any modification (e.g. correct local time). Suggestions? David On 12/6/21 7:39 PM, David wrote: > In idle curiosity I spent a few seconds searching and will share what > I found in case it might be useful. > > Apparently vfat has mount options to work around this, maybe > exfat does? > > https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=998189 > https://www.kernel.org/doc/html/latest/filesystems/vfat.html > > And some details about exfat, and kernel timezone awareness. > > https://www.sans.org/blog/exfat-file-system-time-zone-concerns/ > https://unix.stackexchange.com/questions/640555/vfat-linux-invalid-file-timestamps-shown-after-a-reboot > > Hope this helps, I have nothing further to add. On 12/6/21 9:07 PM, David Wright wrote: > I would first start with the camera. What do the internal timestamps > say about when you took the photograph. (I find it useful to take a > photograph of a clock, and archive it.) Then look at the filesystem > timestamp (which you've done). And of course, there's the filename, > sometimes a sequence number, sometimes a timedate (mangled, so it > usually sorts in a bizarre order). They're all different. > > The camera in my mobile telephone writes precise UTC timestamps in > the file. I never see the filesystem as I can only bluetooth them out. > However, the filenames are local minute-accurate timestamps. > > A SIM-less mobile (same manufacturer, different brand) behaves > entirely differently, using local time for both. That mobile/must/ > display local time as it's my bedside/alarm clock. (It's odd: the > time is not precise, yet it knows when to switch from standard > to DST. I haven't worked out how it does that.) > > I keep the clock in my real camera set to UTC at all times, avoiding > any problems (when mounted with options below). But you'd never > want to routinely tell the time by working through its menus. > > Next comes the filesystem, exFAT, which I've not yet had to use. > The wiki page says it uses local time, but has a time zone marker > in offset relative to UTC (starting with Vista SP2). So that means > there are probably newer devices out there that write it, and older > ones which don't. So you'd need to check the behaviour writing onto > both FAT and exFAT filesystems. And really you need to repeat your > check in winter/summer. Does your device know the real local time > or is it just counting time since you last set it. (Trust microsoft > to screw things up yet again.) > > Finally, what does linux do, and are there mount options for exFAT. > My SD cards are always mounted with: > vfat rw,errors=remount-ro,utf8,tz=UTC,shortname=mixed,user,noauto,fmask=137,dmask=027 > regardless of which camera it was used in. When I copy the files > off the card to a "proper" filesystem, I run a program that > observes the camera's fingerprint, and makes appropriate adjustments > to the filenames, and both internal and filesystem timestamps. > (Eg, two of our phones add a/b/… suffixes to pictures taken within > the same minute, yet the seconds can be read from the internal > timestamps even if the timezone is wrong.) > > Whether there's another adjustment for people running RTC=local, > I have no idea. Ditto for anyone who lets Windows loose on the task. > > Cheers, > David. On 12/6/21 11:50 PM, Dirk Neumann wrote: > Most cameras write a timestamp also into the exif data. If you look at > such an image, you will find something like this "2021:01:18 08:46:28" I verified the clock in the camera before taking the picture. It was set to local time.
Re: SD card, exfat, file mtime wrong
On Mon, 6 Dec 2021 23:07:36 -0600 David Wright wrote: > > I would first start with the camera. What do the internal timestamps > say about when you took the photograph. (I find it useful to take a > photograph of a clock, and archive it.) Then look at the filesystem > timestamp (which you've done). And of course, there's the filename, > sometimes a sequence number, sometimes a timedate (mangled, so it > usually sorts in a bizarre order). They're all different. Most cameras write a timestamp also into the exif data. If you look at such an image, you will find something like this "2021:01:18 08:46:28"
Re: SD card, exfat, file mtime wrong
On Mon 06 Dec 2021 at 18:37:37 (-0800), David Christensen wrote: > I recently installed debian-live-11.1.0-amd64-xfce+nonfree.iso on a > Dell Latitude E6520: > > 2021-12-06 17:52:45 dpchrist@laalaa ~ > $ cat /etc/debian_version ; uname -a > 11.1 > Linux laalaa 5.10.0-9-amd64 #1 SMP Debian 5.10.70-1 (2021-09-30) > x86_64 GNU/Linux > > I am in California, USA. My timezone is Pacific Standard Time (PST). > UTC is 8 hours ahead. tzselect(1) calls my timezone > "America/Los_Angeles". > > When I insert a SDXC card formatted with exfat into a USB adapter, > plug the adapter into the laptop, and double-click on the icon that > appears on the desktop, the SD card filesystem is mounted: > > 2021-12-06 17:52:53 dpchrist@laalaa ~ > $ mount | grep media > /dev/sdc1 on /media/dpchrist/**redacted** type exfat > (rw,nosuid,nodev,relatime,uid=**redacted**,gid=**redacted**,fmask=0022,dmask=0022,iocharset=utf8,errors=remount-ro,uhelper=udisks2) > > When I list the files on the SD card, the mtime is wrong. For > example, here is a photograph that I took about ten minutes ago: > > 2021-12-06 18:13:02 dpchrist@laalaa > /media/dpchrist/**redacted**/DCIM/**redacted** > $ ll IMG_6947.JPG > -rwxr-xr-x 1 dpchrist dpchrist 3988022 2021-12-06 10:02:57 IMG_6947.JPG > > It appears that mount(8) and/or the exfat driver consider the mtime to > be in UTC, and are applying my timezone adjustment of -08:00 hours > (?). > > I would like Debian 11 to display the mtime without any modification > (e.g. correct local time). > > Suggestions? I would first start with the camera. What do the internal timestamps say about when you took the photograph. (I find it useful to take a photograph of a clock, and archive it.) Then look at the filesystem timestamp (which you've done). And of course, there's the filename, sometimes a sequence number, sometimes a timedate (mangled, so it usually sorts in a bizarre order). They're all different. The camera in my mobile telephone writes precise UTC timestamps in the file. I never see the filesystem as I can only bluetooth them out. However, the filenames are local minute-accurate timestamps. A SIM-less mobile (same manufacturer, different brand) behaves entirely differently, using local time for both. That mobile /must/ display local time as it's my bedside/alarm clock. (It's odd: the time is not precise, yet it knows when to switch from standard to DST. I haven't worked out how it does that.) I keep the clock in my real camera set to UTC at all times, avoiding any problems (when mounted with options below). But you'd never want to routinely tell the time by working through its menus. Next comes the filesystem, exFAT, which I've not yet had to use. The wiki page says it uses local time, but has a time zone marker in offset relative to UTC (starting with Vista SP2). So that means there are probably newer devices out there that write it, and older ones which don't. So you'd need to check the behaviour writing onto both FAT and exFAT filesystems. And really you need to repeat your check in winter/summer. Does your device know the real local time or is it just counting time since you last set it. (Trust microsoft to screw things up yet again.) Finally, what does linux do, and are there mount options for exFAT. My SD cards are always mounted with: vfat rw,errors=remount-ro,utf8,tz=UTC,shortname=mixed,user,noauto,fmask=137,dmask=027 regardless of which camera it was used in. When I copy the files off the card to a "proper" filesystem, I run a program that observes the camera's fingerprint, and makes appropriate adjustments to the filenames, and both internal and filesystem timestamps. (Eg, two of our phones add a/b/… suffixes to pictures taken within the same minute, yet the seconds can be read from the internal timestamps even if the timezone is wrong.) Whether there's another adjustment for people running RTC=local, I have no idea. Ditto for anyone who lets Windows loose on the task. Cheers, David.
Re: SD card, exfat, file mtime wrong
On Tue, 7 Dec 2021 at 13:38, David Christensen wrote: > It appears that mount(8) and/or the exfat driver consider the mtime to > be in UTC, and are applying my timezone adjustment of -08:00 hours (?). In idle curiosity I spent a few seconds searching and will share what I found in case it might be useful. Apparently vfat has mount options to work around this, maybe exfat does? https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=998189 https://www.kernel.org/doc/html/latest/filesystems/vfat.html And some details about exfat, and kernel timezone awareness. https://www.sans.org/blog/exfat-file-system-time-zone-concerns/ https://unix.stackexchange.com/questions/640555/vfat-linux-invalid-file-timestamps-shown-after-a-reboot Hope this helps, I have nothing further to add.
SD card, exfat, file mtime wrong
debian-user: I recently installed debian-live-11.1.0-amd64-xfce+nonfree.iso on a Dell Latitude E6520: 2021-12-06 17:52:45 dpchrist@laalaa ~ $ cat /etc/debian_version ; uname -a 11.1 Linux laalaa 5.10.0-9-amd64 #1 SMP Debian 5.10.70-1 (2021-09-30) x86_64 GNU/Linux I am in California, USA. My timezone is Pacific Standard Time (PST). UTC is 8 hours ahead. tzselect(1) calls my timezone "America/Los_Angeles". When I insert a SDXC card formatted with exfat into a USB adapter, plug the adapter into the laptop, and double-click on the icon that appears on the desktop, the SD card filesystem is mounted: 2021-12-06 17:52:53 dpchrist@laalaa ~ $ mount | grep media /dev/sdc1 on /media/dpchrist/**redacted** type exfat (rw,nosuid,nodev,relatime,uid=**redacted**,gid=**redacted**,fmask=0022,dmask=0022,iocharset=utf8,errors=remount-ro,uhelper=udisks2) When I list the files on the SD card, the mtime is wrong. For example, here is a photograph that I took about ten minutes ago: 2021-12-06 18:13:02 dpchrist@laalaa /media/dpchrist/**redacted**/DCIM/**redacted** $ ll IMG_6947.JPG -rwxr-xr-x 1 dpchrist dpchrist 3988022 2021-12-06 10:02:57 IMG_6947.JPG It appears that mount(8) and/or the exfat driver consider the mtime to be in UTC, and are applying my timezone adjustment of -08:00 hours (?). I would like Debian 11 to display the mtime without any modification (e.g. correct local time). Suggestions? David
