Hi,
Chris Roehrig wrote on 8/5/20 12:07 AM:
Hi Jean-Pierre,
That one worked! I also tried it with 1200 and 12000 renamed files,
both with no errors.
Great news !
Thanks for your info about ldd. There was indeed a
/x86_64-linux-gnu/libntfs-3g.so.833.0.0 (which I renamed just to be sure), but I see that
the test version is statically linked ("not a dynamic executable").
It should not be statically linked. I however remember having
seen a situation where executables are marked as shared objects
so they cannot be displayed by ldd.
If you get a libntfs-3g.so.* somewhere, then it is ok.
I also just discovered that ntfsresize -fiv does a filesystem consistency
check. Do you think it is good enough to trust to detect any issues, e.g.
after a power-failure? (This backup drive is likely only ever to be used with
Linux and rsync.)
The check done by ntfsresize is limited : it mostly makes sure
that the space allocated to files and directories is the same as
recorded in the global allocation table.
If you want to run chkdsk from Linux, you can use runwin,
from
https://jp-andre.pagesperso-orange.fr/runwin.zip
(however not checked with the very latest chkdsk for Win10)
Jean-Pierre
Thanks,
-- Chris
PS. The use case for this drive is a daily Linux backup but also a
"grab-and-go" backup for fleeing a forest-fire where we will likely have to
rely on borrowed computers for a couple weeks. NTFS is the only Linux-writable
filesystem I found that can preserve POSIX ownerships and permissions and is readable
without any additional software on Mac/PC/Linux. Thank you for your dedicated work on
ntfs-3g!
On Tue Aug 4 2020, at 5:33 AM, Jean-Pierre André <jean-pierre.an...@wanadoo.fr>
wrote:
Hi,
Chris Roehrig wrote on 8/4/20 2:46 AM:
Hi Jean-Pierre,
I tried it, but it made no difference: baddir still gives exactly the
same CHKDSK error.
Well, I found out my previous tries do not fix all cases,
as the behavior depends on the order of the renamings.
Please find attached a third try (discard the previous
variants before applying this one).
I'm not 100% sure if I'm using the patched version: To mount the partition
I'm running the patched ntfs-3g directly from its build directory:
./ntfs-3g_ntfsprogs-2017.3.23AR.5/src/ntfs-3g /dev/sdb1 /mnt
Will that be completely stand-alone and isolated from Ubuntu's built-in ntfs-3g
or could it e.g. pull in an old AR.3 shared lib or something? I couldn't find
any existing ntfs-3g stuff in /lib or /usr/lib. I'm not familiar with how
FUSE stuff works.
This a fifteen year old bug in a file unchanged for ten years,
so the patch applies to any non-archaic version.
The bug has been left unnoticed because it only occurs in very
specific conditions, and it does not harm. It is also probably
self-healing under some condition.
You have apparently correctly applied the recipe for
testing without installing (and without disturbing your
installed configuration). However, if you do "make install"
the generated code will not be put according to the rules
for your distribution.
You will have to declare the target directories as arguments
to configure, something like (not tested, depends on your
distribution).
./configure --bindir=/bin --sbindir=/sbin --libdir=/lib \
--mandir=/usr/share/man
You can determine where libntfs-3g is installed by executing
ldd $(which ntfs-3g)
Jean-Pierre
Thanks,
-- Chris
On Mon Aug 3 2020, at 12:16 AM, Jean-Pierre André
<jean-pierre.an...@wanadoo.fr> wrote:
Hi,
Please find attached a hopefully better variant of the fix.
Jean-Pierre
Jean-Pierre André wrote on 8/1/20 12:35 PM:
Hi,
Thank you for your report including a test pointing out
the issue.
Attached is a patch expected to fix it.
Please test and report.
Jean-Pierre
Chris Roehrig wrote on 7/30/20 1:01 AM:
I'm trying to get my Linux-based NTFS backup drive to pass a CHKDSK and came
upon this curious situation where CHKDSK finds errors.
It seems to be some issue with how ntfs-3g modifies a directory index when
renaming many files.
The CHKDSK error always seems to be of the form:
Stage 2: Examining file name linkage ...
The first free byte, 0xc0, and bytes available, 0x150, for root index $I30 in
file 0x40 are not equal.
I've attached a python script (mkbaddir.py) that creates two (apparently)
identical directories, one of which reliably causes this CHKDSK error; the
other doesn't.
How to demonstrate:
- Format an NTFS partition or thumbdrive using Windows or mkfs.ntfs.
- Mount the partition on a Linux system.
I used Mint 20 with ntfs-3g 2017.3.23AR.3 integrated FUSE 28 and
python 3.8.2.
- Chdir to the new NTFS partition and run the script:
/tmp/mkbaddir.py # creates 'baddir' in current dir.
/tmp/mkbaddir.py -G # creates 'gooddir' in current dir.
diff -r baddir gooddir # no difference
du -sB1 baddir gooddir # same size (128K)
- Boot into Windows (10 v1903) and run (from a terminal) chkdsk X: (where
X: is the NTFS drive).
- This will say:
"Errors found. CHKDSK cannot continue in read-only mode."
- Delete baddir (I used cygwin's rm -rf), and run chkdsk X: again.
- This will now have no errors.
My guess at what's happening:
The script creates a directory of 410 empty files and then renames them with
slightly larger names, which as I understand leaves a bunch of unused nodes in
the b-tree. The -G option just renames the 410 known files; without the -G
option, it uses os.walk() to traverse the directory which I'm guessing leaves
the b-tree in a slightly different state with even more unused nodes.
The 410 was chosen by trial-and-error so that some internal threshhold is just
exceeded by the baddir but not by the gooddir. With more than 410 (using the
-c option; say -c 500), both baddir and gooddir will cause CHKDSK errors.
If I run the script on Windows/cygwin (Python 3.6.9) to create the folders, it
does not give any CHKDSK errors even with many more files.
So there seems to be some issue with how ntfs-3g modifies the b-tree when
renaming many files that is causing CHKDSK to complain.
I encountered this issue when trying to get my Linux-based NTFS backup drive to
consistently pass a CHKDSK. I use a script to first rename POSIX names to
valid windows names, replacing '?' with '@@3F', etc so I can reverse the
renaming afterwards. I have some website mirror folders with many files of the
form:
details.asp?id=xxxxx&key=val
which gave rise to this issue. (In the mkbaddir script I use only
alphanumeric names to be clear this is not an illegal char issue).
<shorten_index-v2.patch>
<shorten_index-v3.patch>
_______________________________________________
ntfs-3g-devel mailing list
ntfs-3g-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ntfs-3g-devel
