Re: [BackupPC-users] add rsync -A -X and get 'Got fatal error during xfer (fileListReceive failed)'
Holger, thanks for your reply. I'll will be running some troubleshooting commands with the rsync -A and -X options to see what happens. I have not taken the time to read up on File::RsyncP, yet. Plan to do that this weekend. You insight is a big help. Thank you. --- ATTENTION: Donelson Trophy will be closed July 3rd for Independence Day. - Bob Wooden of Donelson Trophy 615.885.2846 (main) www.donelsontrophy.com [5] Everyone deserves an award!! On 2015-06-26 08:44, Holger Parplies wrote: Hi, Robert Wooden wrote on 2015-06-26 11:45:05 + [Re: [BackupPC-users] add rsync -A -X and get 'Got fatal error during xfer (fileListReceive failed)']: I also posted this on the rsync mailing list . . . more soon. and also here on the list. Sorry for not replying yet. This was: - Original Message - From: Bob of Donelson Trophy b...@donelsontrophy.net Sent: Thursday, June 25, 2015 6:56:52 AM Subject: [BackupPC-users] add rsync -A -X and get 'Got fatal error during xfer (fileListReceive failed)' I am trying to add acl file permissions and attr to rsync in RsyncArgs and RsyncRestoreArgs and have discovered that their addition is not that easy. When backups are run the log files contains: Got fatal error during xfer (fileListReceive failed). Removing the -A and the -X allows the backup to proceed without errors. Is there another 'switch' that needs to be added? Anyone familiar with this addition? Simple answer: it won't work. BackupPC (version 3) uses the Perl module File::RsyncP on the server side to do its part of the communication. It does this for the simple reason that it stores file data and metadata differently than the native file system does: file data may be optionally compressed and is pooled to save large amounts of storage space. File metadata needs to be stored independently because it may differ across instances of one pooled file. Adding arbitrary new metadata won't just magically work. Someone would need to change the format of BackupPC attrib files to allow storing the new (meta-)data as well as implement extracting the data from what the remote system sends. This means the rsync XferMethod would need to handle multiple combinations of -A and -X switches (including both not set). All other XferMethods might need to be adjusted, regardless of whether they are even capable of transferring this information. All of that could be done, but I don't see anyone doing it any time soon, if at all. I believe some people use a DumpPreUserCmd (or DumpPreShareCmd) to dump ACLs and extended attributes to a file which is then included in the backup. Restoring a subset of the backup is probably not easy in this case, though, and might require some manual work. I'm not sure whether BackupPC v4 supports ACLs and extended attributes. Hope that helps. Regards, Holger -- Monitor 25 network devices or servers for free with OpManager! OpManager is web-based network management software that monitors network devices and physical virtual servers, alerts via email sms for fault. Monitor 25 devices for free with no restriction. Download now http://ad.doubleclick.net/ddm/clk/292181274;119417398;o [1] ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List: https://lists.sourceforge.net/lists/listinfo/backuppc-users [2] Wiki: http://backuppc.wiki.sourceforge.net [3] Project: http://backuppc.sourceforge.net/ [4] Links: -- [1] http://ad.doubleclick.net/ddm/clk/292181274;119417398;o [2] https://lists.sourceforge.net/lists/listinfo/backuppc-users [3] http://backuppc.wiki.sourceforge.net [4] http://backuppc.sourceforge.net/ [5] http://www.donelsontrophy.com -- Monitor 25 network devices or servers for free with OpManager! OpManager is web-based network management software that monitors network devices and physical virtual servers, alerts via email sms for fault. Monitor 25 devices for free with no restriction. Download now http://ad.doubleclick.net/ddm/clk/292181274;119417398;o___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
[BackupPC-users] Tip for backing up an ntfs-3g partition on linux with BackupPC 4.0 alpha 3
Hi all, SUMMARY: To backup ntfs-3g shares with rsync_bpc, override the variable RsyncArgs (and RsyncRestoreArgs), and delete these parameters: --links --hardlinks If not, rsync will keep sending all files over and over again. This means that if you have a host with ntfs-3g partition and other partitions for which you want to keep hard links, you must create an host alias in backup pc for backing up the ntfs-3g shares. -- GORY DETAILS: I had some trouble configuring BackupPC 4.0 alpha 3 to work correctly when backing up ntfs-3g partitions on linux. So, for short, I have a few machines that dual boot Linux / Windows, but run mostly under Linux, and which I back up with BPC 4.0a3. To backup the ntfs partition, I just mount them via ntfs-3g, then configure backuppc to backup these partitions via rsync (using rsync daemon on host). But the default bpc configuration does not work: at each backup, rsync (so rsync_bpc in fact) will systematically send again all files on the ntfs-3g partitions, to finally report that all files are identical. The oddity is that standard rsync does not exhibit this behavior. By experimenting with rsync_bpc, and then looking at the parameters, the problem was in fact obvious: by default, rsync runs with parameters --links and --hard-links. This does not work with rsync_bpc on ntfs-3g. Why it does work with rsync is just a mystery for me yet. To reproduce the problem, here what I did: --- TEST SETUP --- - Content of /etc/fstab UUID=445E1B975E1B813A /c ntfs ro,defaults,nls=utf8,umask=007,gid=46 0 0 - Create a test share in /etc/rsyncd.conf: max connections = 2 log file = /var/log/rsync.log max verbosity = 3 transfer logging = yes log format = %o %i %h [%a] %m (%u) %f %l timeout = 18000 # 5.2GB/hr, so 20GB files requires ~4hr read only = yes list = yes uid = root gid = root auth users = backuppc secrets file = /etc/rsyncd.secrets [test1] comment = Test share 1 path = /c/Boot - In a separate window, monitor the rsyncd log file: tail -f /var/log/rsync.log - Create rsync password file: sudo -v; sudo sed -r 's/^.*://' /etc/rsyncd.secrets | sudo tee /tmp/rsyncd.secrets /dev/null; sudo chmod 600 /tmp/rsyncd.secrets -- Test case one - PASS - rsync + ntfs-3g -- - Initialize: sudo rm -rf /tmp/rsynctest; sudo mkdir /tmp/rsynctest - Repeat the following step, and observe that files are only txfed the first time: sudo rsync --super --recursive --protect-args --numeric-ids --perms --owner --group -D --times \ --links --hard-links --delete --partial --log-format='log: %o %i %B %8U, %8G %9l %f%L' --stats \ --modify-window=5 --one-file-system --password-file=/tmp/rsyncd.secrets \ --checksum backuppc@griffin::test1 /tmp/rsynctest -- Test case two - FAIL - rsync_bpc + ntfs-3g -- - Initialize: sudo rm -rf /tmp/rsynctest; sudo mkdir -p /tmp/rsynctest/cpool; i=0 - Repeat the following step, and observe that files are always txfed: cd /tmp/rsynctest; i=$((i+1)); echo backup $i; sudo cp -ar pc/griffin/$((i-1)) pc/griffin/$i; \ sudo rsync_bpc --bpc-top-dir /tmp/rsynctest --bpc-host-name griffin --bpc-share-name test1 --bpc-bkup-num $i \ --bpc-bkup-comp 3 --bpc-bkup-prevnum $((i-1)) --bpc-bkup-prevcomp 3 --bpc-log-level 3 --super --recursive \ --protect-args --numeric-ids --perms --owner --group -D --times --links --hard-links --delete --partial \ --log-format='log: %o %i %B %8U,%8G %9l %f%L' --stats --modify-window=5 --one-file-system \ --password-file=/tmp/rsyncd.secrets backuppc@griffin::test1 / Increase --bpc-log-level (say set it to 9) to get more debug output. -- Test case three - PASS - rsync_bpc + ntfs-3g, fixed -- - Initialize: sudo rm -rf /tmp/rsynctest; sudo mkdir -p /tmp/rsynctest/cpool; i=0 - Repeat the following step, and observe that files are only txfed the first time when removing parameters --links --hard-links: cd /tmp/rsynctest; i=$((i+1)); echo backup $i; sudo cp -ar pc/griffin/$((i-1)) pc/griffin/$i; \ sudo rsync_bpc --bpc-top-dir /tmp/rsynctest --bpc-host-name griffin --bpc-share-name test1 --bpc-bkup-num $i \ --bpc-bkup-comp 3 --bpc-bkup-prevnum $((i-1)) --bpc-bkup-prevcomp 3 --bpc-log-level 3 --super --recursive \ --protect-args --numeric-ids --perms --owner --group -D --times --delete --partial \ --log-format='log: %o %i %B %8U,%8G %9l %f%L' --stats --modify-window=5 --one-file-system \ --password-file=/tmp/rsyncd.secrets backuppc@griffin::test1 / -- Michaël -- Monitor 25 network devices or servers for free
Re: [BackupPC-users] add rsync -A -X and get 'Got fatal error during xfer (fileListReceive failed)'
Hi, Robert Wooden wrote on 2015-06-26 11:45:05 + [Re: [BackupPC-users] add rsync -A -X and get 'Got fatal error during xfer (fileListReceive failed)']: I also posted this on the rsync mailing list . . . more soon. and also here on the list. Sorry for not replying yet. This was: - Original Message - From: Bob of Donelson Trophy b...@donelsontrophy.net Sent: Thursday, June 25, 2015 6:56:52 AM Subject: [BackupPC-users] add rsync -A -X and get 'Got fatal error during xfer (fileListReceive failed)' I am trying to add acl file permissions and attr to rsync in RsyncArgs and RsyncRestoreArgs and have discovered that their addition is not that easy. When backups are run the log files contains: Got fatal error during xfer (fileListReceive failed). Removing the -A and the -X allows the backup to proceed without errors. Is there another 'switch' that needs to be added? Anyone familiar with this addition? Simple answer: it won't work. BackupPC (version 3) uses the Perl module File::RsyncP on the server side to do its part of the communication. It does this for the simple reason that it stores file data and metadata differently than the native file system does: file data may be optionally compressed and is pooled to save large amounts of storage space. File metadata needs to be stored independently because it may differ across instances of one pooled file. Adding arbitrary new metadata won't just magically work. Someone would need to change the format of BackupPC attrib files to allow storing the new (meta-)data as well as implement extracting the data from what the remote system sends. This means the rsync XferMethod would need to handle multiple combinations of -A and -X switches (including both not set). All other XferMethods might need to be adjusted, regardless of whether they are even capable of transferring this information. All of that could be done, but I don't see anyone doing it any time soon, if at all. I believe some people use a DumpPreUserCmd (or DumpPreShareCmd) to dump ACLs and extended attributes to a file which is then included in the backup. Restoring a subset of the backup is probably not easy in this case, though, and might require some manual work. I'm not sure whether BackupPC v4 supports ACLs and extended attributes. Hope that helps. Regards, Holger -- Monitor 25 network devices or servers for free with OpManager! OpManager is web-based network management software that monitors network devices and physical virtual servers, alerts via email sms for fault. Monitor 25 devices for free with no restriction. Download now http://ad.doubleclick.net/ddm/clk/292181274;119417398;o ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
Re: [BackupPC-users] add rsync -A -X and get 'Got fatal error during xfer (fileListReceive failed)'
I also posted this on the rsync mailing list . . . more soon. - Original Message - From: Bob of Donelson Trophy b...@donelsontrophy.net To: backuppc-users list backuppc-users@lists.sourceforge.net Sent: Thursday, June 25, 2015 6:56:52 AM Subject: [BackupPC-users] add rsync -A -X and get 'Got fatal error during xfer (fileListReceive failed)' I am trying to add acl file permissions and attr to rsync in RsyncArgs and RsyncRestoreArgs and have discovered that their addition is not that easy. When backups are run the log files contains: Got fatal error during xfer (fileListReceive failed). Removing the -A and the -X allows the backup to proceed without errors. Is there another 'switch' that needs to be added? Anyone familiar with this addition? -- Bob Wooden of Donelson Trophy 615.885.2846 (main) www.donelsontrophy.com Everyone deserves an award!! -- Monitor 25 network devices or servers for free with OpManager! OpManager is web-based network management software that monitors network devices and physical virtual servers, alerts via email sms for fault. Monitor 25 devices for free with no restriction. Download now http://ad.doubleclick.net/ddm/clk/292181274;119417398;o ___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List: https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki: http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/ -- Monitor 25 network devices or servers for free with OpManager! OpManager is web-based network management software that monitors network devices and physical virtual servers, alerts via email sms for fault. Monitor 25 devices for free with no restriction. Download now http://ad.doubleclick.net/ddm/clk/292181274;119417398;o___ BackupPC-users mailing list BackupPC-users@lists.sourceforge.net List:https://lists.sourceforge.net/lists/listinfo/backuppc-users Wiki:http://backuppc.wiki.sourceforge.net Project: http://backuppc.sourceforge.net/
