Dear Jean-Pierre, Thank you! My tests so far seemed good, I can read all the files on the file-system. I am moving the plug-in for a production run.
If it works I will make a little blog post and credit you there. Thank you again! Lets hope it stays working. Kind regards, Jelle On 04/04/17 17:52, Jean-Pierre André wrote: > Hi again, > > Please retry with : > http://jp-andre.pagesperso-orange.fr/dedup124-beta.zip > > I had to take the number of index entries from another > record. Up to now, both locations showed the same value, > and I do not know the specific meaning of each. The change > should fix your latest case, and it does not break my own > tests. > > Thank you again for your cooperation. > > Jean-Pierre > > Jelle de Jong wrote: >> Dear Jean-Pierre, >> >> Hereby the full stream directory: /mnt/sr7-sdb2/System\ Volume\ >> Information/Dedup/ChunkStore/\{0DECAE8D-71D2-4BDE-8798-530201C72D8D\}.ddp/Stream/ >> >> >> >> # f66ede437b46789891bcddc2ab424cfe stream.data.full.dir.tar.gz >> https://powermail.nu/nextcloud/index.php/s/JfGOU6O9isWLtMl >> >> I did not see another 00190000.*.ccc >> >> Kind regards, >> >> Jelle de Jong >> >> On 04/04/17 13:00, Jean-Pierre André wrote: >>> Jelle de Jong wrote: >>>> Dear Jean-Pierre, >>>> >>>> I can now read the "algemeen.txt" file, thank you! >>>> >>>> However I still got an stream error on an other file. >>> >>> You have a very interesting configuration for testing >>> corner situations. I would not be able to get such a >>> rich testbed without your cooperation. Unfortunately >>> this leads to a lot of communication.... >>> >>>> http://paste.debian.net/plainh/cb081fa9 >>>> >>>> I did not get how you calculated that inode 1545290 was in cluster >>>> 30817428, but assuming this is still the case here is the information: >>> >>> Normally, this calculation is not needed, just do >>> an ntfsinfo, for example : >>> ntfsinfo -fvi 1545290 /dev/... >>> >>> I asked you because you showed an inconsistency, and >>> ntfsinfo could not display the contents (the inode >>> had probably been freed). It is now apparently too >>> late to catch the inconsistency. >>> >>> The explanation for the computation is : >>> - divide 1545290 by 4, which gives 386322 or 0x5e513 >>> - lookup for a run in mft which contains vcn 0x5e513 >>> we find vcn 0x578ac lcn 0x1d5d02d length 0xc813 >>> then lcn is 0x1d5d02d + (0x5e513 - 0x578ac) = 0x1d63c94 >>> and 0x1d63c94 is 30817428 >>> >>>> >>>> http://paste.debian.net/plainh/535a9df8 >>>> >>>> Thank you very much! >>>> >>>> If it is a quick fix I can still test it today, almost there. >>> >>> Probably not a quick fix. The reparse data points at offset >>> 90622928, which is beyond the end of 00190000.00010002.ccc >>> whose size is 87789056. Moreover the identification of the >>> user file (D8A64E6D...) is not present in 00190000.00010002.ccc >>> >>> What comes to mind is that 00190000.00010002.ccc was being >>> updated and a newer one should be used. >>> >>> Is there another 00190000.*.ccc in the same Stream directory ? >>> >>> Jean-Pierre >>> >>>> Kind regards, >>>> >>>> Jelle de Jong >>>> >>>> On 03/04/17 11:44, Jean-Pierre André wrote: >>>>> Hi, >>>>> >>>>> Please try >>>>> http://jp-andre.pagesperso-orange.fr/dedup123-beta.zip >>>>> >>>>> Regards >>>>> >>>>> Jean-Pierre >>>>> >>>>> Jean-Pierre André wrote: >>>>>> Hi, >>>>>> >>>>>> I see what is happening here : this file "algemeen.txt" >>>>>> is recorded the traditional way (like mine), not the >>>>>> way most of your files are recorded. So I have to improve >>>>>> the selection of modes. >>>>>> >>>>>> I will upload a fix next week. >>>>>> >>>>>> Was this file created recently ? Maybe, when a file is >>>>>> updated, it is recorded differently from when it was created >>>>>> initially. >>>>>> >>>>>> Regards >>>>>> >>>>>> Jean-Pierre >>>>>> >>>>>> Jelle de Jong wrote: >>>>>>> Beste Jean-Pierre, >>>>>>> >>>>>>> #getfattr -h -e hex -n system.ntfs_reparse_data ... >>>>>>> #dd if=/dev/mapper/.. skip=30817428 bs=4096 count=1 | od -t x1 >>>>>>> #and the other commands outputs: >>>>>>> https://powermail.nu/nextcloud/index.php/s/HwzpBqaIzOyZedg >>>>>>> >>>>>>> # stream.data.full.00190000.00010002.gz: >>>>>>> https://powermail.nu/nextcloud/index.php/s/KKlByC8qEPKZFrC >>>>>>> >>>>>>> Thank you in advance, >>>>>>> >>>>>>> Kind regards, >>>>>>> >>>>>>> Jelle de Jong >>>>>>> >>>>>>> On 28/03/17 16:32, Jean-Pierre André wrote: >>>>>>>> Jelle de Jong wrote: >>>>>>>>> Hi Jean-Pierre, >>>>>>>>> >>>>>>>>> When trying to read algemeen.txt I get a reproducible "Bad stream >>>>>>>>> for >>>>>>>>> offset" (I got a few more files with the same issue) >>>>>>>> >>>>>>>> So, please post the reparse tag for this file algemeen.txt : >>>>>>>> getfattr -h -e hex -n system.ntfs_reparse_data >>>>>>>> somepath/algemeen.txt >>>>>>>> >>>>>>>> Also please post the contents of file 1545290. I do not >>>>>>>> know its name, but it must be in the Stream directory, >>>>>>>> with a suffix .ccc. To get its name do : >>>>>>>> ls -li *as*usual*/Stream/*.ccc | grep 1545290 >>>>>>>> >>>>>>>>> >>>>>>>>> # ntfsinfo -fvi 0 >>>>>>>>> /dev/mapper/lvm1--vol-sr7--disk2--snapshot--copy2 >>>>>>>>> http://paste.debian.net/plainh/486b1a01 >>>>>>>> >>>>>>>> So inode 1582022 must be in cluster 30826611 and >>>>>>>> inode 1545290 in cluster 30817428. >>>>>>>> Please post the outputs of : >>>>>>>> dd if=/dev/mapper/lvm1--vol* skip=30826611 bs=4096 count=1 | od >>>>>>>> -t x1 >>>>>>>> dd if=/dev/mapper/lvm1--vol* skip=30817428 bs=4096 count=1 | od >>>>>>>> -t x1 >>>>>>>> >>>>>>>> The sequencing error which had been detected may have >>>>>>>> disappeared with no evidence left, but it is worth >>>>>>>> trying. >>>>>>>> >>>>>>>>> I feel we may be almost there (support for data deduplication), >>>>>>>>> but >>>>>>>>> still hitting a few unreadable files... >>>>>>>> >>>>>>>> I hope so, this is an uneasy situation for me, as I >>>>>>>> cannot reproduce the errors. Thanks for you cooperation >>>>>>>> to the bug hunt. >>>>>>>> >>>>>>>> Jean-Pierre >>>>>>>> >>>>>>>>> >>>>>>>>> Thanks you again! >>>>>>>>> >>>>>>>>> Kind regards, >>>>>>>>> >>>>>>>>> Jelle de Jong >>>>>>>>> >>>>>>>>> >>>>>>>>> On 23/03/17 22:03, Jean-Pierre André wrote: >>>>>>>>>> Hi, >>>>>>>>>> >>>>>>>>>> Jelle de Jong wrote: >>>>>>>>>>> Hi Jean-Pierre, >>>>>>>>>>> >>>>>>>>>>> Thank you again! >>>>>>>>>>> >>>>>>>>>>> Could you take a look at the following output, I got some "Bad >>>>>>>>>>> stream >>>>>>>>>>> for offset" messages: http://paste.debian.net/plainh/c145e066 >>>>>>>>>>> >>>>>>>>>>> Follow-up on the previous mail >>>>>>>>>>> >>>>>>>>>>> I was not able to get any node information, and have not been >>>>>>>>>>> able to >>>>>>>>>>> reproduce the NTFS messages. >>>>>>>>>>> >>>>>>>>>>> ntfsinfo -fvi 1582022 >>>>>>>>>>> /dev/mapper/lvm1--vol-sr7--disk2--snapshot--copy2 >>>>>>>>>>> Error loading node: No such file or directory >>>>>>>>>> >>>>>>>>>> Before digging into the "Bad stream for offset" messages, >>>>>>>>>> I would like to get details about those inconsistencies >>>>>>>>>> before they are wiped out (hoping it is not too late). >>>>>>>>>> Maybe the issues are related, maybe they are not. >>>>>>>>>> >>>>>>>>>> The ntfsinfo error means either that the faulty inodes >>>>>>>>>> have been freed or they are extents to other inodes. >>>>>>>>>> >>>>>>>>>> I will have to do it in two steps : first determine where >>>>>>>>>> they are located, then extract their contents. >>>>>>>>>> >>>>>>>>>> To get the mapping of inodes, please post the output of >>>>>>>>>> ntfsinfo -fvi 0 /dev/mapper/lvm1--vol-sr7* >>>>>>>>>> >>>>>>>>>> Jean-Pierre >>>>>>>>>> >>>>>>>>>>> >>>>>>>>>>> Kind regards, >>>>>>>>>>> >>>>>>>>>>> Jelle de Jong >>>>>>>>>>> >>>>>>>>>>> On 21/03/17 12:01, Jean-Pierre André wrote: >>>>>>>>>>>> Hi, >>>>>>>>>>>> >>>>>>>>>>>> Jelle de Jong wrote: >>>>>>>>>>>>> Dear Jean-Pierre, >>>>>>>>>>>>> >>>>>>>>>>>>> I am still testing the dedup plugin, I have issues with >>>>>>>>>>>>> rdiff-backup >>>>>>>>>>>>> that the checksum of the source and destination are not the >>>>>>>>>>>>> same, >>>>>>>>>>>>> but >>>>>>>>>>>>> this can be an issue with rdiff-backup. >>>>>>>>>>>>> >>>>>>>>>>>>> However I did get these messages, but I do not know if they >>>>>>>>>>>>> are >>>>>>>>>>>>> related >>>>>>>>>>>>> to the dedup plugin? >>>>>>>>>>>>> >>>>>>>>>>>>> Mar 21 06:57:37 backup ntfs-3g[12620]: Record 1582022 has >>>>>>>>>>>>> wrong >>>>>>>>>>>>> SeqNo >>>>>>>>>>>>> (159 <> 156) >>>>>>>>>>>>> Mar 21 06:57:37 backup ntfs-3g[12620]: Could not decode the >>>>>>>>>>>>> type of >>>>>>>>>>>>> inode 1582022 >>>>>>>>>>>>> Mar 21 06:57:37 backup ntfs-3g[12620]: Record 1545290 has >>>>>>>>>>>>> wrong >>>>>>>>>>>>> SeqNo >>>>>>>>>>>>> (296 <> 295) >>>>>>>>>>>>> Mar 21 06:57:37 backup ntfs-3g[12620]: Could not decode the >>>>>>>>>>>>> type of >>>>>>>>>>>>> inode 1545290 >>>>>>>>>>>> >>>>>>>>>>>> These errors are thrown by ntfs-3g proper, not by the >>>>>>>>>>>> deduplication plugin. >>>>>>>>>>>> >>>>>>>>>>>> Inconsistencies have been found in the mentioned files, >>>>>>>>>>>> and they are unlikely to be created by the plugin (which >>>>>>>>>>>> does not have updating support), ... but one never knows. >>>>>>>>>>>> Owing to the inconsistencies, ntfs-3g cannot tell if the >>>>>>>>>>>> said files are plain files, directories, sockets, etc. >>>>>>>>>>>> >>>>>>>>>>>> You have to run chkdsk to fix these inconsistencies, but >>>>>>>>>>>> this can lead to losing data, so before doing that, the >>>>>>>>>>>> parameters of these files should be analyzed to assess >>>>>>>>>>>> the possible damage. >>>>>>>>>>>> >>>>>>>>>>>> Please post the outputs of : >>>>>>>>>>>> ntfsinfo -fvi 1582022 /dev/partition >>>>>>>>>>>> ntfsinfo -fvi 1545290 /dev/partition >>>>>>>>>>>> >>>>>>>>>>>> This must be done as root, replacing /dev/partition by >>>>>>>>>>>> the actual partition path (something like /dev/sdc2) >>>>>>>>>>>> >>>>>>>>>>>> Jean-Pierre >>>>>>>>>>>> >>>>>>>>>>>>> >>>>>>>>>>>>> Kind regards, >>>>>>>>>>>>> >>>>>>>>>>>>> Jelle de Jong >>>>>>>>>>> >>>>>>>>>> >>>>>>>>>> >>>>>>>>> >>>>>>>> >>>>>>>> >>>>>>> >>>>>> >>>>>> >>>>> >>>>> >>>> >>> >>> >> > > ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, Slashdot.org! http://sdm.link/slashdot _______________________________________________ ntfs-3g-devel mailing list ntfs-3g-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ntfs-3g-devel
