Hi Jean-Pierre, Thank you!
The new plug-in seems to work for now, I am moving it into testing phase with-in our production back-up scripts. Will you release the source code eventually, would like to write a blog post about how to add the support. What do you think the changes are of the plug-in stop working again? We do not have an automatic test running to verify the back-ups at this moment _yet_, so if the plug-in stops working, incremental file-based back-ups with empty files will slowly get in the back-ups this way :| Again thank you for all your help so far! Kind regards, Jelle de Jong On 08/02/17 15:59, Jean-Pierre André wrote: > Hi, > > Can you please make a try with : > http://jp-andre.pagesperso-orange.fr/dedup120-beta.zip > > This is experimental and based on assumptions which have > to be clarified, but it should work in your environment. > > Regards > > Jean-Pierre > > Jelle de Jong wrote: >> Dear Jean-Piere, >> >> The stream.data.full.gz file: >> https://powermail.nu/owncloud/index.php/s/6Y0WXs7WQclpOBM >> >> # attempt to compile searchseq.c: >> http://paste.debian.net/plainh/49252e03/ >> >> # grep and other info how I generated the stream.data.full.gz file: >> http://paste.debian.net/plainh/34a9c9ec >> >> Again thank you for your help! >> >> Kind regards, >> >> Jelle de Jong >> >> On 05/02/17 17:44, Jean-Pierre André wrote: >>> Hi, >>> >>> Jelle de Jong wrote: >>>> Hi Jean-Piere, >>>> >>>> The requested stream.data.gz file: >>>> https://powermail.nu/nextcloud/index.php/s/QmbQLnrLZneIScT >>> >>> Well, this is unexpected : the data does not match. >>> I assume it has been relocated, but I have no idea >>> where it could be. This situation does not occur in >>> my test partition, so I need your help. >>> >>> Basically, I have to find the following hexadecimal >>> digest somewhere : C1F41B5197F9B31AFE5D65585CA4F8F8 >>> As the files are big, I have to rely on you doing >>> some investigation. >>> >>> I first assume this is to be found in the expected >>> file (00190000.00010000.ccc), and you may first check >>> whether "grep" (or "strings") find the printable >>> sub-sequence "]eX\" in it : >>> grep '\]eX\\' /mnt/sr7-sdb2/..etc../00190000.00010000.ccc >>> >>> If so, use the attached program searchseq to find out >>> precisely where the sequence is located : >>> ./findseq C1F41B5197F9B31AFE5D65585CA4F8F8 /mnt/sr7-sdb2/..etc.. >>> (I have included the source code, you may compile it if >>> you are uneasy executing foreign code). >>> >>> If it finds it, take the decimal location, divide by 512 >>> and post three records around this location. Assuming >>> you get 123456789, dividing by 512 yields 241126, so you >>> post three records from 241125 (so 241126 and nearby). >>> Then play the dd command with needed adaptations and make >>> the output available : >>> dd if=FILE bs=512 skip=START count=3 | gzip > stream.data.gz >>> >>> Thank you for your help and good luck ! >>> >>> Jean-Pierre >>> >>> >>>> root@backup:~# ls -hali >>>> /mnt/sr7-sdb2/System*/Dedup/ChunkStore/{0DECAE8D*/Stream | grep 2801748 >>>> 2801748 -rwxrwxrwx 1 root root 87M Jan 28 07:27 00190000.00010000.ccc >>>> root@backup:~# ls -hali >>>> /mnt/sr7-sdb2/System*/Dedup/ChunkStore/{0DECAE8D*/Stream/00190000.00010000.ccc >>>> >>>> >>>> >>>> >>>> 2801748 -rwxrwxrwx 1 root root 87M Jan 28 07:27 /mnt/sr7-sdb2/System >>>> Volume >>>> Information/Dedup/ChunkStore/{0DECAE8D-71D2-4BDE-8798-530201C72D8D}.ddp/Stream/00190000.00010000.ccc >>>> >>>> >>>> >>>> >>>> root@backup:~# FILE="/mnt/sr7-sdb2/System Volume >>>> Information/Dedup/ChunkStore/{0DECAE8D-71D2-4BDE-8798-530201C72D8D}.ddp/Stream/00190000.00010000.ccc" >>>> >>>> >>>> >>>> >>>> root@backup:~# dd if="$FILE" bs=512 skip=133396 count=2 | gzip > >>>> stream.data.gz >>>> >>>> Thank you! >>>> >>>> Kind regards, >>>> >>>> Jelle de Jong >>>> >>>> On 04/02/17 21:52, Jean-Pierre André wrote: >>>>> Hi again, >>>>> >>>>> A consistency check has failed, there is some unexpected >>>>> data in your Stream file. Can you post an excerpt : >>>>> >>>>> dd if=FILE bs=512 skip=133396 count=2 | gzip > stream.data.gz >>>>> (FILE being the one with name 00190000.00010000.ccc whose >>>>> inode number is 2801748). >>>>> >>>>> Regards >>>>> >>>>> Jean-Pierre >>>>> >>>>> Jelle de Jong wrote: >>>>>> Hi Jean-Pierre, >>>>>> >>>>>> Thank you! >>>>>> >>>>>> The requested information: http://paste.debian.net/hidden/bfcfff7c/ >>>>>> >>>>>> The behaviour is with with plain mount: >>>>>> >>>>>> mount -o ro -t ntfs-3g >>>>>> /dev/mapper/lvm1--vol-sr7--disk2--snapshot--copy2 /mnt/sr7-sdb2 >>>>>> >>>>>> I figured out how to load your awesome plugin into the guestmount >>>>>> system, so if mount works I should have the rest working as well. >>>>>> >>>>>> Thank you in advance, >>>>>> >>>>>> Kind regards, >>>>>> >>>>>> Jelle de Jong >>>>>> >>>>>> On 04/02/17 14:37, Jean-Pierre André wrote: >>>>>>> Hi, >>>>>>> >>>>>>> Jelle de Jong wrote: >>>>>>>> Dear Jean-Pierre, >>>>>>>> >>>>>>>> I could read the files for maybe one day, I tried to get it into >>>>>>>> production but then it didn't work anymore, going back to my base >>>>>>>> test >>>>>>>> mounting the volume without guestmount, so with just mount I get >>>>>>>> the >>>>>>>> following: >>>>>>> >>>>>>> Do you mean that the behavior with guestmount is >>>>>>> different from the one guestmount ? >>>>>>> >>>>>>> Also, are you not able to read any file any more, or >>>>>>> are there files which you can read (such as the one >>>>>>> you could read previously) ? >>>>>>> >>>>>>> (more below) >>>>>>> >>>>>>>> # with stream list: >>>>>>>> http://paste.debian.net/hidden/15d83c84/ >>>>>>>> >>>>>>>> root@backup:~# grep ntfs-3g /var/log/syslog >>>>>>>> Feb 4 11:44:10 backup ntfs-3g[30845]: Version 2016.2.22AR.1 >>>>>>>> integrated >>>>>>>> FUSE 28 >>>>>>>> Feb 4 11:44:10 backup ntfs-3g[30845]: Mounted >>>>>>>> /dev/mapper/lvm1--vol-sr7--disk2--snapshot--copy2 (Read-Only, label >>>>>>>> "DATA", NTFS 3.1) >>>>>>>> Feb 4 11:44:10 backup ntfs-3g[30845]: Cmdline options: ro >>>>>>>> Feb 4 11:44:10 backup ntfs-3g[30845]: Mount options: >>>>>>>> ro,allow_other,nonempty,relatime,fsname=/dev/mapper/lvm1--vol-sr7--disk2--snapshot--copy2,blkdev,blksize=4096 >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> >>>>>>>> Feb 4 11:44:10 backup ntfs-3g[30845]: Ownership and permissions >>>>>>>> disabled, configuration type 7 >>>>>>>> Feb 4 11:47:10 backup ntfs-3g[30845]: Bad stream at offset 0x0 for >>>>>>>> file >>>>>>>> 2801748 >>>>>>>> Feb 4 11:47:10 backup ntfs-3g[30845]: Bad stream at offset 0x20000 >>>>>>>> for >>>>>>>> file 2801748 >>>>>>>> Feb 4 11:47:12 backup ntfs-3g[30845]: Bad stream at offset 0x0 for >>>>>>>> file >>>>>>>> 2801748 >>>>>>>> Feb 4 11:47:12 backup ntfs-3g[30845]: Bad stream at offset 0x20000 >>>>>>>> for >>>>>>>> file 2801748 >>>>>>> >>>>>>> At first glance, this is related to the modification >>>>>>> made a few weeks ago. Probably a wrong file was used, >>>>>>> but I have no idea why. >>>>>>> >>>>>>> To debug this, I will need some data. To begin with, >>>>>>> I need the reparse data of an unreadable file. >>>>>>> You can get the reparse data by : >>>>>>> (replace NAME by actual name) >>>>>>> >>>>>>> getfattr -h -e hex -n system.ntfs_reparse_data NAME >>>>>>> >>>>>>> I also need to know which one is the file 2801748. >>>>>>> This should be in the directory which you have already >>>>>>> listed (/mnt/sr7-sdb2/System*/ ... /Stream), just add >>>>>>> -i to the "ls" options ("ls -hali /mnt/sr7-sdb2 ...") >>>>>>> to display the inode numbers. >>>>>>> >>>>>>>> >>>>>>>> I am at Fosdem 2017 this weekend, if you are there let me know. >>>>>>> >>>>>>> No, I am not. >>>>>>> >>>>>>>> Could you help me out and tell me what data you would like to have >>>>>>>> from me. >>>>>>> >>>>>>> I need to go through three steps, and analyze the data >>>>>>> to get to the next step. The first step is the reparse >>>>>>> data, as mentioned above. This is difficult to automate, >>>>>>> as bugs tend to occur in unanticipated cases (I obviously >>>>>>> have no specification, so I have no information about >>>>>>> cases I never met myself). >>>>>>> >>>>>>> Note : please, only mount as read-only until the issue >>>>>>> is solved. >>>>>>> >>>>>>> Regards >>>>>>> >>>>>>> Jean-Pierre >>>>>>> >>>>>>> >>>>>>>> Kind regards, >>>>>>>> >>>>>>>> Jelle de Jong >>>>>>>> >>>>>>>> >>>>>>>> On 18/01/17 21:52, Jean-Pierre André wrote: >>>>>>>>> Hi again, >>>>>>>>> >>>>>>>>> Jelle de Jong wrote: >>>>>>>>>> Dear Jean-Pierre, >>>>>>>>>> >>>>>>>>>> Thank you! I can read the files! >>>>>>>>> >>>>>>>>> Great ! >>>>>>>>> >>>>>>> >>>>>>> >>>>>> >>>>> >>>>> >>>> >>> >> > > ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, SlashDot.org! http://sdm.link/slashdot _______________________________________________ ntfs-3g-devel mailing list ntfs-3g-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ntfs-3g-devel
