Hi again,
Please retry with :
http://jp-andre.pagesperso-orange.fr/dedup124-beta.zip
I had to take the number of index entries from another
record. Up to now, both locations showed the same value,
and I do not know the specific meaning of each. The change
should fix your latest case, and it does not break my own
tests.
Thank you again for your cooperation.
Jean-Pierre
Jelle de Jong wrote:
> Dear Jean-Pierre,
>
> Hereby the full stream directory: /mnt/sr7-sdb2/System\ Volume\
> Information/Dedup/ChunkStore/\{0DECAE8D-71D2-4BDE-8798-530201C72D8D\}.ddp/Stream/
>
>
> # f66ede437b46789891bcddc2ab424cfe stream.data.full.dir.tar.gz
> https://powermail.nu/nextcloud/index.php/s/JfGOU6O9isWLtMl
>
> I did not see another 00190000.*.ccc
>
> Kind regards,
>
> Jelle de Jong
>
> On 04/04/17 13:00, Jean-Pierre André wrote:
>> Jelle de Jong wrote:
>>> Dear Jean-Pierre,
>>>
>>> I can now read the "algemeen.txt" file, thank you!
>>>
>>> However I still got an stream error on an other file.
>>
>> You have a very interesting configuration for testing
>> corner situations. I would not be able to get such a
>> rich testbed without your cooperation. Unfortunately
>> this leads to a lot of communication....
>>
>>> http://paste.debian.net/plainh/cb081fa9
>>>
>>> I did not get how you calculated that inode 1545290 was in cluster
>>> 30817428, but assuming this is still the case here is the information:
>>
>> Normally, this calculation is not needed, just do
>> an ntfsinfo, for example :
>> ntfsinfo -fvi 1545290 /dev/...
>>
>> I asked you because you showed an inconsistency, and
>> ntfsinfo could not display the contents (the inode
>> had probably been freed). It is now apparently too
>> late to catch the inconsistency.
>>
>> The explanation for the computation is :
>> - divide 1545290 by 4, which gives 386322 or 0x5e513
>> - lookup for a run in mft which contains vcn 0x5e513
>> we find vcn 0x578ac lcn 0x1d5d02d length 0xc813
>> then lcn is 0x1d5d02d + (0x5e513 - 0x578ac) = 0x1d63c94
>> and 0x1d63c94 is 30817428
>>
>>>
>>> http://paste.debian.net/plainh/535a9df8
>>>
>>> Thank you very much!
>>>
>>> If it is a quick fix I can still test it today, almost there.
>>
>> Probably not a quick fix. The reparse data points at offset
>> 90622928, which is beyond the end of 00190000.00010002.ccc
>> whose size is 87789056. Moreover the identification of the
>> user file (D8A64E6D...) is not present in 00190000.00010002.ccc
>>
>> What comes to mind is that 00190000.00010002.ccc was being
>> updated and a newer one should be used.
>>
>> Is there another 00190000.*.ccc in the same Stream directory ?
>>
>> Jean-Pierre
>>
>>> Kind regards,
>>>
>>> Jelle de Jong
>>>
>>> On 03/04/17 11:44, Jean-Pierre André wrote:
>>>> Hi,
>>>>
>>>> Please try
>>>> http://jp-andre.pagesperso-orange.fr/dedup123-beta.zip
>>>>
>>>> Regards
>>>>
>>>> Jean-Pierre
>>>>
>>>> Jean-Pierre André wrote:
>>>>> Hi,
>>>>>
>>>>> I see what is happening here : this file "algemeen.txt"
>>>>> is recorded the traditional way (like mine), not the
>>>>> way most of your files are recorded. So I have to improve
>>>>> the selection of modes.
>>>>>
>>>>> I will upload a fix next week.
>>>>>
>>>>> Was this file created recently ? Maybe, when a file is
>>>>> updated, it is recorded differently from when it was created
>>>>> initially.
>>>>>
>>>>> Regards
>>>>>
>>>>> Jean-Pierre
>>>>>
>>>>> Jelle de Jong wrote:
>>>>>> Beste Jean-Pierre,
>>>>>>
>>>>>> #getfattr -h -e hex -n system.ntfs_reparse_data ...
>>>>>> #dd if=/dev/mapper/.. skip=30817428 bs=4096 count=1 | od -t x1
>>>>>> #and the other commands outputs:
>>>>>> https://powermail.nu/nextcloud/index.php/s/HwzpBqaIzOyZedg
>>>>>>
>>>>>> # stream.data.full.00190000.00010002.gz:
>>>>>> https://powermail.nu/nextcloud/index.php/s/KKlByC8qEPKZFrC
>>>>>>
>>>>>> Thank you in advance,
>>>>>>
>>>>>> Kind regards,
>>>>>>
>>>>>> Jelle de Jong
>>>>>>
>>>>>> On 28/03/17 16:32, Jean-Pierre André wrote:
>>>>>>> Jelle de Jong wrote:
>>>>>>>> Hi Jean-Pierre,
>>>>>>>>
>>>>>>>> When trying to read algemeen.txt I get a reproducible "Bad stream
>>>>>>>> for
>>>>>>>> offset" (I got a few more files with the same issue)
>>>>>>>
>>>>>>> So, please post the reparse tag for this file algemeen.txt :
>>>>>>> getfattr -h -e hex -n system.ntfs_reparse_data somepath/algemeen.txt
>>>>>>>
>>>>>>> Also please post the contents of file 1545290. I do not
>>>>>>> know its name, but it must be in the Stream directory,
>>>>>>> with a suffix .ccc. To get its name do :
>>>>>>> ls -li *as*usual*/Stream/*.ccc | grep 1545290
>>>>>>>
>>>>>>>>
>>>>>>>> # ntfsinfo -fvi 0 /dev/mapper/lvm1--vol-sr7--disk2--snapshot--copy2
>>>>>>>> http://paste.debian.net/plainh/486b1a01
>>>>>>>
>>>>>>> So inode 1582022 must be in cluster 30826611 and
>>>>>>> inode 1545290 in cluster 30817428.
>>>>>>> Please post the outputs of :
>>>>>>> dd if=/dev/mapper/lvm1--vol* skip=30826611 bs=4096 count=1 | od
>>>>>>> -t x1
>>>>>>> dd if=/dev/mapper/lvm1--vol* skip=30817428 bs=4096 count=1 | od
>>>>>>> -t x1
>>>>>>>
>>>>>>> The sequencing error which had been detected may have
>>>>>>> disappeared with no evidence left, but it is worth
>>>>>>> trying.
>>>>>>>
>>>>>>>> I feel we may be almost there (support for data deduplication), but
>>>>>>>> still hitting a few unreadable files...
>>>>>>>
>>>>>>> I hope so, this is an uneasy situation for me, as I
>>>>>>> cannot reproduce the errors. Thanks for you cooperation
>>>>>>> to the bug hunt.
>>>>>>>
>>>>>>> Jean-Pierre
>>>>>>>
>>>>>>>>
>>>>>>>> Thanks you again!
>>>>>>>>
>>>>>>>> Kind regards,
>>>>>>>>
>>>>>>>> Jelle de Jong
>>>>>>>>
>>>>>>>>
>>>>>>>> On 23/03/17 22:03, Jean-Pierre André wrote:
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> Jelle de Jong wrote:
>>>>>>>>>> Hi Jean-Pierre,
>>>>>>>>>>
>>>>>>>>>> Thank you again!
>>>>>>>>>>
>>>>>>>>>> Could you take a look at the following output, I got some "Bad
>>>>>>>>>> stream
>>>>>>>>>> for offset" messages: http://paste.debian.net/plainh/c145e066
>>>>>>>>>>
>>>>>>>>>> Follow-up on the previous mail
>>>>>>>>>>
>>>>>>>>>> I was not able to get any node information, and have not been
>>>>>>>>>> able to
>>>>>>>>>> reproduce the NTFS messages.
>>>>>>>>>>
>>>>>>>>>> ntfsinfo -fvi 1582022
>>>>>>>>>> /dev/mapper/lvm1--vol-sr7--disk2--snapshot--copy2
>>>>>>>>>> Error loading node: No such file or directory
>>>>>>>>>
>>>>>>>>> Before digging into the "Bad stream for offset" messages,
>>>>>>>>> I would like to get details about those inconsistencies
>>>>>>>>> before they are wiped out (hoping it is not too late).
>>>>>>>>> Maybe the issues are related, maybe they are not.
>>>>>>>>>
>>>>>>>>> The ntfsinfo error means either that the faulty inodes
>>>>>>>>> have been freed or they are extents to other inodes.
>>>>>>>>>
>>>>>>>>> I will have to do it in two steps : first determine where
>>>>>>>>> they are located, then extract their contents.
>>>>>>>>>
>>>>>>>>> To get the mapping of inodes, please post the output of
>>>>>>>>> ntfsinfo -fvi 0 /dev/mapper/lvm1--vol-sr7*
>>>>>>>>>
>>>>>>>>> Jean-Pierre
>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> Kind regards,
>>>>>>>>>>
>>>>>>>>>> Jelle de Jong
>>>>>>>>>>
>>>>>>>>>> On 21/03/17 12:01, Jean-Pierre André wrote:
>>>>>>>>>>> Hi,
>>>>>>>>>>>
>>>>>>>>>>> Jelle de Jong wrote:
>>>>>>>>>>>> Dear Jean-Pierre,
>>>>>>>>>>>>
>>>>>>>>>>>> I am still testing the dedup plugin, I have issues with
>>>>>>>>>>>> rdiff-backup
>>>>>>>>>>>> that the checksum of the source and destination are not the
>>>>>>>>>>>> same,
>>>>>>>>>>>> but
>>>>>>>>>>>> this can be an issue with rdiff-backup.
>>>>>>>>>>>>
>>>>>>>>>>>> However I did get these messages, but I do not know if they are
>>>>>>>>>>>> related
>>>>>>>>>>>> to the dedup plugin?
>>>>>>>>>>>>
>>>>>>>>>>>> Mar 21 06:57:37 backup ntfs-3g[12620]: Record 1582022 has wrong
>>>>>>>>>>>> SeqNo
>>>>>>>>>>>> (159 <> 156)
>>>>>>>>>>>> Mar 21 06:57:37 backup ntfs-3g[12620]: Could not decode the
>>>>>>>>>>>> type of
>>>>>>>>>>>> inode 1582022
>>>>>>>>>>>> Mar 21 06:57:37 backup ntfs-3g[12620]: Record 1545290 has wrong
>>>>>>>>>>>> SeqNo
>>>>>>>>>>>> (296 <> 295)
>>>>>>>>>>>> Mar 21 06:57:37 backup ntfs-3g[12620]: Could not decode the
>>>>>>>>>>>> type of
>>>>>>>>>>>> inode 1545290
>>>>>>>>>>>
>>>>>>>>>>> These errors are thrown by ntfs-3g proper, not by the
>>>>>>>>>>> deduplication plugin.
>>>>>>>>>>>
>>>>>>>>>>> Inconsistencies have been found in the mentioned files,
>>>>>>>>>>> and they are unlikely to be created by the plugin (which
>>>>>>>>>>> does not have updating support), ... but one never knows.
>>>>>>>>>>> Owing to the inconsistencies, ntfs-3g cannot tell if the
>>>>>>>>>>> said files are plain files, directories, sockets, etc.
>>>>>>>>>>>
>>>>>>>>>>> You have to run chkdsk to fix these inconsistencies, but
>>>>>>>>>>> this can lead to losing data, so before doing that, the
>>>>>>>>>>> parameters of these files should be analyzed to assess
>>>>>>>>>>> the possible damage.
>>>>>>>>>>>
>>>>>>>>>>> Please post the outputs of :
>>>>>>>>>>> ntfsinfo -fvi 1582022 /dev/partition
>>>>>>>>>>> ntfsinfo -fvi 1545290 /dev/partition
>>>>>>>>>>>
>>>>>>>>>>> This must be done as root, replacing /dev/partition by
>>>>>>>>>>> the actual partition path (something like /dev/sdc2)
>>>>>>>>>>>
>>>>>>>>>>> Jean-Pierre
>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> Kind regards,
>>>>>>>>>>>>
>>>>>>>>>>>> Jelle de Jong
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>
>>
>>
>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
ntfs-3g-devel mailing list
ntfs-3g-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ntfs-3g-devel
