Hi Jean-Pierre, Thank you!
The reparse-tags.gz file: https://powermail.nu/nextcloud/index.php/s/fS6Y6bpzoMgPiZ0 Generated by running: getfattr -e hex -n system.ntfs_reparse_data -R /mnt/sr7-sdb2/ 2> /dev/null | grep ntfs_reparse_data | gzip > /root/reparse-tags.gz Kind regards, Jelle de Jong On 23/02/17 12:07, Jean-Pierre André wrote: > Hi, > > Jelle de Jong wrote: >> Dear Jean-Pierre, >> >> I thought version 1.2.1 of the plug-in was working so I took it further >> into production, but during backups with rdiff-backup and guestmount it >> created a 100% cpu load in qemu process that stayed there for days until >> I killed them, I tested this twice. So I went back to a xpart/mount -t >> ntfs command and found more "Bad stream for offset" and found that the >> /sbin/mount.ntfs-3g command was running at 100% cpu load and hanged >> there. > > Too bad. > >> I have added the whole Stream directory here: (1.1GB) >> https://powermail.nu/nextcloud/index.php/s/vbq85qZ2wcVYxrG >> >> Separate stream file: stream.data.full.000c0000.00020001.gz >> https://powermail.nu/nextcloud/index.php/s/QinV51XE4jrAH7a >> >> All the commands I used: >> http://paste.debian.net/plainh/c0ea5950 >> >> I do not know how to get the reparse tags of all the files, maybe you >> can help me how to get all the information you need. > > Just use option -R on the base directory : > > getfattr -e hex -n system.ntfs_reparse_data -R base-dir > > Notes : > 1) files with no reparse tags (those which are not deduplicated) > will throw an error > 2) this will output the file names, which you might not want > to disclose. Fortunately I do not need them for now. > > So you may append to the above command : > > 2> /dev/null | grep ntfs_reparse_data | gz > reparse-tags.gz > > With that, I will be able to build a configuration similar > to yours... apart from the files themselves. > > Regards > > Jean-Pierre > >> >> Thank you for your help! >> >> Kind regards, >> >> Jelle de Jong >> >> On 14/02/17 15:55, Jean-Pierre André wrote: >>> Hi, >>> >>> Jelle de Jong wrote: >>>> Hi Jean-Pierre, >>>> >>>> If we have to switch to Windows 2012 and thereby having an environment >>>> similar to yours then we can switch to an other Windows version. >>> >>> I do not have any Windows Server, and my analysis >>> and tests are based on an unofficial deduplication >>> package which was adapted to Windows 10 Pro. >>> >>> A few months ago, following a bug report, I had to >>> make changes for Windows Server 2012 which uses an >>> older data format, and my only experience about this >>> format is related to this report. So switching to >>> Windows 2012 is not guaranteed to make debugging easier. >>> >>>> We are running out of disk space here so if switching Windows versions >>>> makes the process of having data deduplication working easer then me >>>> know. >>> >>> I have not yet analyzed your latest report, but it >>> would probably be useful I build a full copy of >>> non-user data from your partition : >>> - the reparse tags of all your files, >>> - all the "*.ccc" files in the Stream directory >>> >>> Do not do it now, I must first dig into the data you >>> posted. >>> >>> Regards >>> >>> Jean-Pierre >>> >>> >>>> Kind regards, >>>> >>>> Jelle de Jong >>>> >>>> On 09/02/17 13:46, Jelle de Jong wrote: >>>>> Hi Jean-Pierre, >>>>> >>>>> In case you are wondering: >>>>> >>>>> I am using data deduplication in Windows 2016 for my test environment >>>>> iso: >>>>> SW_DVD9_Win_Svr_STD_Core_and_DataCtr_Core_2016_64Bit_English_-2_MLF_X21-22843.ISO >>>>> >>>>> >>>>> >>>>> >>>>> Kind regards, >>>>> >>>>> Jelle de Jong >>>>> >>>>> On 09/02/17 11:41, Jean-Pierre André wrote: >>>>>> Hi, >>>>>> >>>>>> Jelle de Jong wrote: >>>>>>> Hi Jean-Pierre, >>>>>>> >>>>>>> Thank you! >>>>>>> >>>>>>> The new plug-in seems to work for now, I am moving it into testing >>>>>>> phase >>>>>>> with-in our production back-up scripts. >>>>>> >>>>>> Please wait a few hours, I have found a bug which >>>>>> I have fixed. I am currently inserting your data >>>>>> into my test base in order to rerun all my tests. >>>>>> >>>>>>> Will you release the source code eventually, would like to write a >>>>>>> blog >>>>>>> post about how to add the support. >>>>>> >>>>>> What exactly do you mean ? If it is about how to >>>>>> collect the data in a unsupported condition, it is >>>>>> difficult, because unsupported generally means >>>>>> unknown territory... >>>>>> >>>>>>> What do you think the changes are of the plug-in stop working again? >>>>>> >>>>>> (assuming a typo changes -> chances) >>>>>> Your files were in a condition not met before : data >>>>>> has been relocated according to a logic I do not fully >>>>>> understand. Maybe this is an intermediate step in the >>>>>> process of updating the files, anyway this can happen. >>>>>> >>>>>> The situation I am facing is that I have a single >>>>>> example from which it is difficult to derive the rules. >>>>>> So yes, the plugin may stop working again. >>>>>> >>>>>> Note : there are strict consistency checks in the plugin, >>>>>> so it is unlikely you read invalid data. Moreover if >>>>>> you only mount read-only you cannot damage the deduplicated >>>>>> partition. >>>>>> >>>>>>> We do not have an automatic test running to verify the back-ups at >>>>>>> this >>>>>>> moment _yet_, so if the plug-in stops working, incremental >>>>>>> file-based >>>>>>> back-ups with empty files will slowly get in the back-ups this >>>>>>> way :| >>>>>> >>>>>> Usually a deduplicated partition is only used for backups, >>>>>> and reading from backups is only for recovering former >>>>>> versions of files (on demand). >>>>>> >>>>>> If you access deduplicated files with no human control, >>>>>> you have to insert your own checks in the process. I >>>>>> would at least check whether the size of the recovered >>>>>> file is the same as the deduplicated one (also grep for >>>>>> messages in the syslog). >>>>>> >>>>>> Regards >>>>>> >>>>>> Jean-Pierre >>>>>> >>>>>>> Again thank you for all your help so far! >>>>>>> >>>>>>> Kind regards, >>>>>>> >>>>>>> Jelle de Jong >>>>>>> >>>>>>> >>>>>>> On 08/02/17 15:59, Jean-Pierre André wrote: >>>>>>>> Hi, >>>>>>>> >>>>>>>> Can you please make a try with : >>>>>>>> http://jp-andre.pagesperso-orange.fr/dedup120-beta.zip >>>>>>>> >>>>>>>> This is experimental and based on assumptions which have >>>>>>>> to be clarified, but it should work in your environment. >>>>>>>> >>>>>>>> Regards >>>>>>>> >>>>>>>> Jean-Pierre > > ------------------------------------------------------------------------------ Check out the vibrant tech community on one of the world's most engaging tech sites, SlashDot.org! http://sdm.link/slashdot _______________________________________________ ntfs-3g-devel mailing list ntfs-3g-devel@lists.sourceforge.net https://lists.sourceforge.net/lists/listinfo/ntfs-3g-devel
