Dear Jean-Pierre,
Hereby the full stream directory: /mnt/sr7-sdb2/System\ Volume\
Information/Dedup/ChunkStore/\{0DECAE8D-71D2-4BDE-8798-530201C72D8D\}.ddp/Stream/
# f66ede437b46789891bcddc2ab424cfe stream.data.full.dir.tar.gz
https://powermail.nu/nextcloud/index.php/s/JfGOU6O9isWLtMl
I did not see another 00190000.*.ccc
Kind regards,
Jelle de Jong
On 04/04/17 13:00, Jean-Pierre André wrote:
> Jelle de Jong wrote:
>> Dear Jean-Pierre,
>>
>> I can now read the "algemeen.txt" file, thank you!
>>
>> However I still got an stream error on an other file.
>
> You have a very interesting configuration for testing
> corner situations. I would not be able to get such a
> rich testbed without your cooperation. Unfortunately
> this leads to a lot of communication....
>
>> http://paste.debian.net/plainh/cb081fa9
>>
>> I did not get how you calculated that inode 1545290 was in cluster
>> 30817428, but assuming this is still the case here is the information:
>
> Normally, this calculation is not needed, just do
> an ntfsinfo, for example :
> ntfsinfo -fvi 1545290 /dev/...
>
> I asked you because you showed an inconsistency, and
> ntfsinfo could not display the contents (the inode
> had probably been freed). It is now apparently too
> late to catch the inconsistency.
>
> The explanation for the computation is :
> - divide 1545290 by 4, which gives 386322 or 0x5e513
> - lookup for a run in mft which contains vcn 0x5e513
> we find vcn 0x578ac lcn 0x1d5d02d length 0xc813
> then lcn is 0x1d5d02d + (0x5e513 - 0x578ac) = 0x1d63c94
> and 0x1d63c94 is 30817428
>
>>
>> http://paste.debian.net/plainh/535a9df8
>>
>> Thank you very much!
>>
>> If it is a quick fix I can still test it today, almost there.
>
> Probably not a quick fix. The reparse data points at offset
> 90622928, which is beyond the end of 00190000.00010002.ccc
> whose size is 87789056. Moreover the identification of the
> user file (D8A64E6D...) is not present in 00190000.00010002.ccc
>
> What comes to mind is that 00190000.00010002.ccc was being
> updated and a newer one should be used.
>
> Is there another 00190000.*.ccc in the same Stream directory ?
>
> Jean-Pierre
>
>> Kind regards,
>>
>> Jelle de Jong
>>
>> On 03/04/17 11:44, Jean-Pierre André wrote:
>>> Hi,
>>>
>>> Please try
>>> http://jp-andre.pagesperso-orange.fr/dedup123-beta.zip
>>>
>>> Regards
>>>
>>> Jean-Pierre
>>>
>>> Jean-Pierre André wrote:
>>>> Hi,
>>>>
>>>> I see what is happening here : this file "algemeen.txt"
>>>> is recorded the traditional way (like mine), not the
>>>> way most of your files are recorded. So I have to improve
>>>> the selection of modes.
>>>>
>>>> I will upload a fix next week.
>>>>
>>>> Was this file created recently ? Maybe, when a file is
>>>> updated, it is recorded differently from when it was created
>>>> initially.
>>>>
>>>> Regards
>>>>
>>>> Jean-Pierre
>>>>
>>>> Jelle de Jong wrote:
>>>>> Beste Jean-Pierre,
>>>>>
>>>>> #getfattr -h -e hex -n system.ntfs_reparse_data ...
>>>>> #dd if=/dev/mapper/.. skip=30817428 bs=4096 count=1 | od -t x1
>>>>> #and the other commands outputs:
>>>>> https://powermail.nu/nextcloud/index.php/s/HwzpBqaIzOyZedg
>>>>>
>>>>> # stream.data.full.00190000.00010002.gz:
>>>>> https://powermail.nu/nextcloud/index.php/s/KKlByC8qEPKZFrC
>>>>>
>>>>> Thank you in advance,
>>>>>
>>>>> Kind regards,
>>>>>
>>>>> Jelle de Jong
>>>>>
>>>>> On 28/03/17 16:32, Jean-Pierre André wrote:
>>>>>> Jelle de Jong wrote:
>>>>>>> Hi Jean-Pierre,
>>>>>>>
>>>>>>> When trying to read algemeen.txt I get a reproducible "Bad stream
>>>>>>> for
>>>>>>> offset" (I got a few more files with the same issue)
>>>>>>
>>>>>> So, please post the reparse tag for this file algemeen.txt :
>>>>>> getfattr -h -e hex -n system.ntfs_reparse_data somepath/algemeen.txt
>>>>>>
>>>>>> Also please post the contents of file 1545290. I do not
>>>>>> know its name, but it must be in the Stream directory,
>>>>>> with a suffix .ccc. To get its name do :
>>>>>> ls -li *as*usual*/Stream/*.ccc | grep 1545290
>>>>>>
>>>>>>>
>>>>>>> # ntfsinfo -fvi 0 /dev/mapper/lvm1--vol-sr7--disk2--snapshot--copy2
>>>>>>> http://paste.debian.net/plainh/486b1a01
>>>>>>
>>>>>> So inode 1582022 must be in cluster 30826611 and
>>>>>> inode 1545290 in cluster 30817428.
>>>>>> Please post the outputs of :
>>>>>> dd if=/dev/mapper/lvm1--vol* skip=30826611 bs=4096 count=1 | od -t x1
>>>>>> dd if=/dev/mapper/lvm1--vol* skip=30817428 bs=4096 count=1 | od -t x1
>>>>>>
>>>>>> The sequencing error which had been detected may have
>>>>>> disappeared with no evidence left, but it is worth
>>>>>> trying.
>>>>>>
>>>>>>> I feel we may be almost there (support for data deduplication), but
>>>>>>> still hitting a few unreadable files...
>>>>>>
>>>>>> I hope so, this is an uneasy situation for me, as I
>>>>>> cannot reproduce the errors. Thanks for you cooperation
>>>>>> to the bug hunt.
>>>>>>
>>>>>> Jean-Pierre
>>>>>>
>>>>>>>
>>>>>>> Thanks you again!
>>>>>>>
>>>>>>> Kind regards,
>>>>>>>
>>>>>>> Jelle de Jong
>>>>>>>
>>>>>>>
>>>>>>> On 23/03/17 22:03, Jean-Pierre André wrote:
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> Jelle de Jong wrote:
>>>>>>>>> Hi Jean-Pierre,
>>>>>>>>>
>>>>>>>>> Thank you again!
>>>>>>>>>
>>>>>>>>> Could you take a look at the following output, I got some "Bad
>>>>>>>>> stream
>>>>>>>>> for offset" messages: http://paste.debian.net/plainh/c145e066
>>>>>>>>>
>>>>>>>>> Follow-up on the previous mail
>>>>>>>>>
>>>>>>>>> I was not able to get any node information, and have not been
>>>>>>>>> able to
>>>>>>>>> reproduce the NTFS messages.
>>>>>>>>>
>>>>>>>>> ntfsinfo -fvi 1582022
>>>>>>>>> /dev/mapper/lvm1--vol-sr7--disk2--snapshot--copy2
>>>>>>>>> Error loading node: No such file or directory
>>>>>>>>
>>>>>>>> Before digging into the "Bad stream for offset" messages,
>>>>>>>> I would like to get details about those inconsistencies
>>>>>>>> before they are wiped out (hoping it is not too late).
>>>>>>>> Maybe the issues are related, maybe they are not.
>>>>>>>>
>>>>>>>> The ntfsinfo error means either that the faulty inodes
>>>>>>>> have been freed or they are extents to other inodes.
>>>>>>>>
>>>>>>>> I will have to do it in two steps : first determine where
>>>>>>>> they are located, then extract their contents.
>>>>>>>>
>>>>>>>> To get the mapping of inodes, please post the output of
>>>>>>>> ntfsinfo -fvi 0 /dev/mapper/lvm1--vol-sr7*
>>>>>>>>
>>>>>>>> Jean-Pierre
>>>>>>>>
>>>>>>>>>
>>>>>>>>> Kind regards,
>>>>>>>>>
>>>>>>>>> Jelle de Jong
>>>>>>>>>
>>>>>>>>> On 21/03/17 12:01, Jean-Pierre André wrote:
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> Jelle de Jong wrote:
>>>>>>>>>>> Dear Jean-Pierre,
>>>>>>>>>>>
>>>>>>>>>>> I am still testing the dedup plugin, I have issues with
>>>>>>>>>>> rdiff-backup
>>>>>>>>>>> that the checksum of the source and destination are not the
>>>>>>>>>>> same,
>>>>>>>>>>> but
>>>>>>>>>>> this can be an issue with rdiff-backup.
>>>>>>>>>>>
>>>>>>>>>>> However I did get these messages, but I do not know if they are
>>>>>>>>>>> related
>>>>>>>>>>> to the dedup plugin?
>>>>>>>>>>>
>>>>>>>>>>> Mar 21 06:57:37 backup ntfs-3g[12620]: Record 1582022 has wrong
>>>>>>>>>>> SeqNo
>>>>>>>>>>> (159 <> 156)
>>>>>>>>>>> Mar 21 06:57:37 backup ntfs-3g[12620]: Could not decode the
>>>>>>>>>>> type of
>>>>>>>>>>> inode 1582022
>>>>>>>>>>> Mar 21 06:57:37 backup ntfs-3g[12620]: Record 1545290 has wrong
>>>>>>>>>>> SeqNo
>>>>>>>>>>> (296 <> 295)
>>>>>>>>>>> Mar 21 06:57:37 backup ntfs-3g[12620]: Could not decode the
>>>>>>>>>>> type of
>>>>>>>>>>> inode 1545290
>>>>>>>>>>
>>>>>>>>>> These errors are thrown by ntfs-3g proper, not by the
>>>>>>>>>> deduplication plugin.
>>>>>>>>>>
>>>>>>>>>> Inconsistencies have been found in the mentioned files,
>>>>>>>>>> and they are unlikely to be created by the plugin (which
>>>>>>>>>> does not have updating support), ... but one never knows.
>>>>>>>>>> Owing to the inconsistencies, ntfs-3g cannot tell if the
>>>>>>>>>> said files are plain files, directories, sockets, etc.
>>>>>>>>>>
>>>>>>>>>> You have to run chkdsk to fix these inconsistencies, but
>>>>>>>>>> this can lead to losing data, so before doing that, the
>>>>>>>>>> parameters of these files should be analyzed to assess
>>>>>>>>>> the possible damage.
>>>>>>>>>>
>>>>>>>>>> Please post the outputs of :
>>>>>>>>>> ntfsinfo -fvi 1582022 /dev/partition
>>>>>>>>>> ntfsinfo -fvi 1545290 /dev/partition
>>>>>>>>>>
>>>>>>>>>> This must be done as root, replacing /dev/partition by
>>>>>>>>>> the actual partition path (something like /dev/sdc2)
>>>>>>>>>>
>>>>>>>>>> Jean-Pierre
>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> Kind regards,
>>>>>>>>>>>
>>>>>>>>>>> Jelle de Jong
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>
>
>
------------------------------------------------------------------------------
Check out the vibrant tech community on one of the world's most
engaging tech sites, Slashdot.org! http://sdm.link/slashdot
_______________________________________________
ntfs-3g-devel mailing list
ntfs-3g-devel@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/ntfs-3g-devel
