Hi, Jelle de Jong wrote: > Dear Jean-Pierre, > > I thought version 1.2.1 of the plug-in was working so I took it further > into production, but during backups with rdiff-backup and guestmount it > created a 100% cpu load in qemu process that stayed there for days until > I killed them, I tested this twice. So I went back to a xpart/mount -t > ntfs command and found more "Bad stream for offset" and found that the > /sbin/mount.ntfs-3g command was running at 100% cpu load and hanged there.
Too bad. > I have added the whole Stream directory here: (1.1GB) > https://powermail.nu/nextcloud/index.php/s/vbq85qZ2wcVYxrG > > Separate stream file: stream.data.full.000c0000.00020001.gz > https://powermail.nu/nextcloud/index.php/s/QinV51XE4jrAH7a > > All the commands I used: > http://paste.debian.net/plainh/c0ea5950 > > I do not know how to get the reparse tags of all the files, maybe you > can help me how to get all the information you need. Just use option -R on the base directory : getfattr -e hex -n system.ntfs_reparse_data -R base-dir Notes : 1) files with no reparse tags (those which are not deduplicated) will throw an error 2) this will output the file names, which you might not want to disclose. Fortunately I do not need them for now. So you may append to the above command : 2> /dev/null | grep ntfs_reparse_data | gz > reparse-tags.gz With that, I will be able to build a configuration similar to yours... apart from the files themselves. Regards Jean-Pierre > > Thank you for your help! > > Kind regards, > > Jelle de Jong > > On 14/02/17 15:55, Jean-Pierre André wrote: >> Hi, >> >> Jelle de Jong wrote: >>> Hi Jean-Pierre, >>> >>> If we have to switch to Windows 2012 and thereby having an environment >>> similar to yours then we can switch to an other Windows version. >> >> I do not have any Windows Server, and my analysis >> and tests are based on an unofficial deduplication >> package which was adapted to Windows 10 Pro. >> >> A few months ago, following a bug report, I had to >> make changes for Windows Server 2012 which uses an >> older data format, and my only experience about this >> format is related to this report. So switching to >> Windows 2012 is not guaranteed to make debugging easier. >> >>> We are running out of disk space here so if switching Windows versions >>> makes the process of having data deduplication working easer then me >>> know. >> >> I have not yet analyzed your latest report, but it >> would probably be useful I build a full copy of >> non-user data from your partition : >> - the reparse tags of all your files, >> - all the "*.ccc" files in the Stream directory >> >> Do not do it now, I must first dig into the data you >> posted. >> >> Regards >> >> Jean-Pierre >> >> >>> Kind regards, >>> >>> Jelle de Jong >>> >>> On 09/02/17 13:46, Jelle de Jong wrote: >>>> Hi Jean-Pierre, >>>> >>>> In case you are wondering: >>>> >>>> I am using data deduplication in Windows 2016 for my test environment >>>> iso: >>>> SW_DVD9_Win_Svr_STD_Core_and_DataCtr_Core_2016_64Bit_English_-2_MLF_X21-22843.ISO >>>> >>>> >>>> >>>> Kind regards, >>>> >>>> Jelle de Jong >>>> >>>> On 09/02/17 11:41, Jean-Pierre André wrote: >>>>> Hi, >>>>> >>>>> Jelle de Jong wrote: >>>>>> Hi Jean-Pierre, >>>>>> >>>>>> Thank you! >>>>>> >>>>>> The new plug-in seems to work for now, I am moving it into testing >>>>>> phase >>>>>> with-in our production back-up scripts. >>>>> >>>>> Please wait a few hours, I have found a bug which >>>>> I have fixed. I am currently inserting your data >>>>> into my test base in order to rerun all my tests. >>>>> >>>>>> Will you release the source code eventually, would like to write a >>>>>> blog >>>>>> post about how to add the support. >>>>> >>>>> What exactly do you mean ? If it is about how to >>>>> collect the data in a unsupported condition, it is >>>>> difficult, because unsupported generally means >>>>> unknown territory... >>>>> >>>>>> What do you think the changes are of the plug-in stop working again? >>>>> >>>>> (assuming a typo changes -> chances) >>>>> Your files were in a condition not met before : data >>>>> has been relocated according to a logic I do not fully >>>>> understand. Maybe this is an intermediate step in the >>>>> process of updating the files, anyway this can happen. >>>>> >>>>> The situation I am facing is that I have a single >>>>> example from which it is difficult to derive the rules. >>>>> So yes, the plugin may stop working again. >>>>> >>>>> Note : there are strict consistency checks in the plugin, >>>>> so it is unlikely you read invalid data. Moreover if >>>>> you only mount read-only you cannot damage the deduplicated >>>>> partition. >>>>> >>>>>> We do not have an automatic test running to verify the back-ups at >>>>>> this >>>>>> moment _yet_, so if the plug-in stops working, incremental file-based >>>>>> back-ups with empty files will slowly get in the back-ups this way :| >>>>> >>>>> Usually a deduplicated partition is only used for backups, >>>>> and reading from backups is only for recovering former >>>>> versions of files (on demand). >>>>> >>>>> If you access deduplicated files with no human control, >>>>> you have to insert your own checks in the process. I >>>>> would at least check whether the size of the recovered >>>>> file is the same as the deduplicated one (also grep for >>>>> messages in the syslog). >>>>> >>>>> Regards >>>>> >>>>> Jean-Pierre >>>>> >>>>>> Again thank you for all your help so far! >>>>>> >>>>>> Kind regards, >>>>>> >>>>>> Jelle de Jong >>>>>> >>>>>> >>>>>> On 08/02/17 15:59, Jean-Pierre André wrote: >>>>>>> Hi, >>>>>>> >>>>>>> Can you please make a try with : >>>>>>> http://jp-andre.pagesperso-orange.fr/dedup120-beta.zip >>>>>>> >>>>>>> This is experimental and based on assumptions which have >>>>>>> to be clarified, but it should work in your environment. >>>>>>> >>>>>>> Regards >>>>>>> >>>>>>> Jean-Pierre ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, SlashDot.org! http://sdm.link/slashdot _______________________________________________ ntfs-3g-devel mailing list ntfs-3g-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ntfs-3g-devel
