Re: [Bug 7120] Variable bandwidth limit .. bwlimit
samba-b...@samba.org wrote: --snipp-- It seems that pv is waiting for data from rsync, and rsync is waiting for data too (stuck in select()) and not closing the input to pv. So it's a deadlock. Same happens when you substitute pv with something else (like dd). It seems that those commands just don't behave like rsync expects them to. --- Would a use of "stdbuf" (coreutils) help? It allows one to change the input and/or output buffering of the tools to from full buffered to line-buffered to unbuffered for tools normally connected via a pipe. Haven't found a workaround short of killing everything: export RSYNC_RSH="sh -c 'pv -qL10k | ssh \"\$@\" | (pv -qL11k; kill \$\$)' ssh" kill is not a solution I'd be happy with. But I haven't found another. --- Maybe a suspend/continue would be more gentle than killing things? -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [Bug 12732] New: hard links can cause rsync to block or to silently skip files
just subscribed for rsync-qa from bugzilla via rsync wrote: Hard link handling seems to be broken when using "rsync -aH --compare-dest". I found two possible scenarios: 1) rsync completes without error message and exit code 0, although some files are missing from the backup 2) rsync blocks and must be interrupted/killed Further information === This problem exists at least for rsync versions 3.1.0, 3.1.1, and 3.1.2 for different Linux varieties using various file systems: https://lists.samba.org/archive/rsync/2015-April/030092.html --- I ran rsync 3.1.1 for over a year to help generate snapshots. I can't say if it copied all the files or not, as it was backing up a large "/home" partition, BUT, it never hung. It did take 45min to a few hours to do the compare, but it was comparing a large amount of data (>750G) w/a snapshot (another 750G) to dump diffs to a third, and my /home partion has a *very* large number of hard links. So I know that hardlinks are handled 'fine' on comparing 'xfs' to 'xfs'. Latest test on openSUSE 42.2 (x86_64) on ext4 + on nfs with Ah... I'd suspect nfs... Why are you using nfs? rsync was designed to compare against local file systems. You should try running rsync directly from the nfs-host machine to the client and bypassing NFS. I.e. -- you need to bypass NFS, since local->local with hardlinks works. Just checked my /home partition. find shows 9295431 names (of any type), but du shows (using du --inodes) shows 4407458 inodes. That means over half of the filenames are hard linked. While my home partition takes up 60% more space now, even cutting those counts in half would still a large number of hard links -- and rsync didn't crash doing an rsync of the partition to an empty one, but first comparing to a previous snapshot (the empty partition ended up with differences between the main partition & the snapshot. I'd remove NFS... -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Why are system-namespaces not copied?
On 9/18/2018 7:44 AM, Frank Steiner via rsync wrote: Hi, the man page states For systems that support extended-attribute namespaces, a copy being done by a super-user copies all namespaces except system.*. That's the reason why NFAv4 ACLs are not copied as they are in the system.nfs4_acl (or system.nfs4acl) namespace. Why are those namespaces excluded? Not being able rsync ACLs von NFSv4 is a major drawback now that NFsv4 becomes standard oder v3 and ACLs are getting more widely used. Because they are storing them in the security (sometimes also called system) section and not the 'root' section (at least on XFS). The linux kernel disallows you reading ex-attrs with the Security label. I don't particularly like it for the same reasons you don't. It takes patching a linux kernel to enable them being copied. I've done it but more as proof of theory. Problem comes in when you restore attribute to a secure namespace. Are those attrs really secure when you take them "off the system". If not, you could modify them, then if they are copied to a target, you could use modified attrs to give yourself root capabilities. So...have to solve that before it can be safely allowed. NeverTheLess, it's still a potential hole to allow copying of such security attrs. Unless you want to change the way security attrs are stored to use 4k-long signing strings to ensure non-tampering, I don't see how you can do it...and doing that would be adding 4k to each attribute...UG! -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [Bug 13582] New: rsync filters containing multiple adjacent slashes aren't reduced to just one slash before matching
On 8/19/2018 10:11 PM, just subscribed for rsync-qa from bugzilla via rsync wrote: The following test script shows that attempting to exclude the file /sourcedir/a/file2 by using //sourcedir//a//file2 in the excluded files list, will silently not exclude it because of all those adjacent slashes not being reduced into just one /. This is a bad example, because the leading '//' cannot be removed without potentially changing the file's location. It's in POSIX that exactly 2 slashes should not be reduced to '1' if it is at the beginning of the path. The ones in the middle -- yes, but even if they were fixed, the two in front might not match a single -- because some OS's use // to introduce a network-located system (in cygwin on windows //remotesystem/will automatically try remotesystem). Can your exclude use a regular expression?, can you say: '/?sourcedir/*a/*file2' in the exclude patterns? (assuming a POSIX RE (not a file wildcard)). -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
[Bug 5124] Parallelize the rsync run using multiple threads and/or connections
On 10/11/2018 10:51 AM, just subscribed for rsync-qa from bugzilla via rsync wrote: https://bugzilla.samba.org/show_bug.cgi?id=5124 --- Comment #7 from Luiz Angelo Daros de Luca --- I also vote for this feature. Using multiple connections, rsync can use multiples internet connections at the same time. FWIW, one of the big changes that went into SMB 3 for Win10 was adding the ability to do file transfers using more than one connection. CIFS (and windows) have traditionally been limited to 1 connection that everything was multiplexed over. However, CIFS in write/reads from a client to a linux server can easily get over 600MB/s writes, and ~275MB/s on reads. The reason it doesn't get more, is the cpu's start maxing out with processing interrupts and packets. I don't see rsync maxing out in cpu even doing a local->local copy, but I haven't done benchmarks on the newer versions of rsync, either. That said, I don't think the slow down is such that it would greatly benefit by multiple connections. My local disk can do read/writes to disk at around 1GB/s (for constant read/write). I'd be more convinced that parallel connections would benefit if there was any benchmarking done to find out where slowdowns are happening, but that's just my 2cents. :-) -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: rsync xattr support doesn't play nice with selinux
On 8/22/2018 2:09 PM, Shaya Potter via rsync wrote: If one is rsyncing a machine without selinux (therefore no security.selinux xattr on each file), to a system that has selinux (even in permissive mode), rsync doesn't play nice. basically selinux seems to make it appear that every file has security.selinux xattr on each file (I think this is virtually if there's no physical attribute, as if one disables selinux, the attribute disappears). --- normally you can't see root or security attributes as a normal user. on a non-security aware OS. rsync sees that on the temp file it created there is an xattr which is not on the source file and therefore tries to remove it, ... Ick. I thought there was going to be a list of attrs for utils that copy attrs to ignore? I guess you don't have an rsync that does that (if it has been done yet). SE linux has to label things when they get written to disk -- it's a mandatory action that a program can only "ignore", but not stop. FWIW many tests in perl that check unix mode bits fail on modern disks with ACL's. Of course they don't want to fix perl, as it might break some older program. It be nice if there was way to tell rsync to ignore some xattrs that might be automatically created on the destination while still allowing xattr syncing. --- I may be mistaken, but I thought it had been discussed and planned at one point (?). sigh. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: [Bug 13645] New: Improve efficiency when resuming transfer of large files
If you are doing a local<-> local transfer, you are wasting time with checksums. You'll get faster performance with "--whole-file". Why do you stop it at night when you could 'unlimit' the transfer speed? Seems like when you aren't there would be best time to copy everything. Doing checksums will cause a noticeable impact to local-file transfers. On 10/5/2018 10:34 AM, just subscribed for rsync-qa from bugzilla via rsync wrote: https://bugzilla.samba.org/show_bug.cgi?id=13645 When transferring large files over a slow network, ... The command used is: rsync -av --inplace --bwlimit=400 hostname::module /dest When restarting the transfer, a lot of time is "wasted" while first the local system is reading the partially transferred file and sends the checksums to the remote, ... Of course these optimizations (at least #2) may actually decrease performance when the transfer is local (not over slow network) and the disk read rate is negatively affected by reading at two different places in parallel. So #2 should only be attempted when the transfer is over a network. --- Or might decrease performance on a fast network. Not sure what you mean by 'slow' 10Mb? 100Mb -- not sure w/o measuring if it is faster or slower to do checksums, but I know at 1000Mb and 10Gb, checksums are prohibitively expensive. NOTE: you also might look at the protocol you use to do network transfers. I.e. use rsync over a locally mounted disk to a locally mounted network share, and make the network share a samba one. That way you will get parallelism automatically -- the file transfer cpu-time will happen inside of samba, while the local file gathering will happen in rsync. I regularly got ~ 119MB R/W over 1000Mb ethernet. BTW, Any place I use a power-of-2 unit like 'B' (Byte), I use the power-of-two base (1024) prefix, but if I use a singular unit like 'b' (bit), then I use decimal prefixes. Doing otherwise makes things hard to calculate and can introduce calculation inaccuracies. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
hardlinking missing files from src to a dest: didn't work way I thought it would.
Have a directory with a bunch rpms in it, mostly x86_64. Have another directory with a bunch, mostly 'noarch'. Some of the noarch files are already in the x86_64 dir and don't want to overwrite them. They are on the same physical disk, so really, just want the new 'noarch' files hardlinked into the destination. sitting in the noarch dir, I tried: rsync -auv --ignore-existing \ --link-dest=/tumbleweed/. . /tumbleweed/. I'm not "too" surprised since technically I asked for it to synchronize them, then link them into the same dir, but thought it would at least say something or create the link, but neither happened. I really didn't want to copy them -- I'd really prefer the link, so how do I have it only create a hard link from the source files to target DIR that don't already exist in the target? I know I can do it with a shell script, but I thought rsync might be faster...then again, if I count figuring out how to do it...not so sure How can I get rsync to do this? Thanks... -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Question about rsync -uav dir1/. dir2/.: possib to link?
I noticed in looking at download dirs for a project, that another mirror had "crept-in" for usage (where different mirrors are stored under mirror-URL names). To copy over the diffs, normally I'd do: rsync -uav dir1/. dir2/. (where dir1="the new mirror that I'd switched to by accident, and dir2=the original dir). The files were "smallish" so I just copied them, BUT I wass wondering if there was an option similar to using 'cp' for a dircopy, but instead of cp -a dr1 dr2 using: cp -al dr1 dr2 to just hard-link over files from "dir1" to "dir2" (both are on the same file system). I looked at (and tried) --link-dest=DIR (hardlink to files in DIR when unchanged), but either I had the syntax wrong, or didn't understand it as it didn't seem to do what I wanted: cp'ing the new files in dir1 into the orig dir). Does rsync have an option to just "copy" over the new files via a hardlink? Tnx! -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: How to manage root<-->root rsync keeping permissions?
On 2021/08/03 07:09, Chris Green via rsync wrote: I already have an rsync daemon server running elsewhere, I can add this requirement to that I think. Thank you. It seems to me, a safer bet would be to generate an ssh-cert that allows a passwdless login from your sys to the remote. Then "export RSYNC_RSH=ssh" on your source before running rsync (as root). I don't use an rsyncd on the remote. Try it in some sub-dir first. Don't cross fs boundaries, so like I use flags (for xfs->xfs) like: rsync -auvxHAXOW --del /usr/local/fonts/ remotesys:/usr/local/fonts/ pathnames are finicky. While this pair works: aa/dir/ (->) bb/dir/ and I think this one does: aa/dir bb/ there are more that aren't reliable but may work occasionally (like work 1st time, but not 2nd...). Some examples: aa/dir/ bb/dir aa/dir/. bb/dir/. aa/dir bb aa/dir/ bb/ then do your rsync as normal run rsync as root to the remote as normal. Passwordless ssh logins are used where remote root and remote-passworded logins are forbidden, since with a strong key, there is no password to crack. Since you may not want remote login directly to root, you might prohibit use of passwords for root (forcing use of a secure key). There can be many caveats, so try on smaller, backed up fs's first... If you have room, transfer to a tmpdir then move into place. Good luck... -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: How to manage root<-->root rsync keeping permissions?
On 2021/08/07 03:44, Chris Green via rsync wrote: L A Walsh via rsync wrote: It seems to me, a safer bet would be to generate an ssh-cert that allows a passwdless login from your sys to the remote. The trouble with that is that it leaves a big security hole. If you only do backups at 1am (or whenever), why would your backup machine enable ssh outside of the range 12:59 - 01:01? If (for example) I leave my laptop turned on somewhere, or someone wanders into my study where my desktop machine is they have instant, passwordless access to the remote backup machine. If your desktop machine is that open to casual wanderers, perhaps you should enable a passwd locked screen saver activating after a few minutes? I keep my home computer unlocked all the time as well, but I don't have walk-through visitors that might mess with it. My desktop computer essentially has root access FROM the windows desktop (my normal user is a domain admin, and can alter permissions or make changes to any file on my server. In my case I regard my desktop+ server as a "split system", with the Winbox being my desktop, and the Linbox being the "backend" of my computer. The Winbox doesn't normally have direct access to the network and all of my "content" files /docs/ progs residing on my linbox. The Linbox handles backups, network access, a proxy for the winbox, incoming+outgoing email (dovecot+sendmail), etc. The linbox does daily security scans and computer maintenance tasks that I don't trust to letting Windows do it as the linbox provide better feedback. Additionally my linbox has has direct access to any file on my desktop as well, thought indirectly in that my linbox acts as a samba domain server for the desktop (thus providing single-signon for my home machines based on the linbox). Its slightly moot, in my case to worry about someone on my desktop being able to access content on my linbox, since all of the "content" files (docs dir, music, video -- all personal files on desktop) actually reside on my server where they are backed up daily via xfs_backup. They are connected via a dedicated, direct 10Gb ethernet that gives 200-400MB/s(M=2**20 bytes) nominal speed up to 600MB. I try very hard to make my backups secure from attack so that if my desktop or laptop is compromised somehow the (remote) backups are still secure. --- Excellent! In my case, my laptop/desktop (used to be a laptop) is thoroughly entwined with the server such that one has trouble functioning without the other. In your case, though, I was thinking of a backup process that would only be used when my laptop was on a secure network (like @ home). If there is risk to your laptop while @ home, hopefully it has a short-timeout that bounces it to the screen saver that requires a password to unlock?t The backup system that runs the rsync daemon has its rsync configured with 'refuse options = delete' --- Ahh...I thought you were actually trying to keep them in sync. Maybe you might think about using an actual backup prog like tar. In my case, the Users/groups are the same. Tar handles ext attrs and acls and can keep track of backing files up that have actually changed rather than relying on time/date stamps. so not only does someone with access to my desktop/laptop need to know the rsyncd username and password but they also cannot delete my existing backups. It runs incremental backups so nothing is ever overwritten either. BTW, incremental backups aren't really the same as 'update' backups, they keep track of the state of the file system (including files no longer there) so you can restore your desktop to a specific day before some unwanted updated was introduced and kept by an update-only backup system. For example. My home partition: home-210501-0-0438.dump home-210512-1-0431.dump home-210523-1-0430.dump home-210601-0-0437.dump home-210603-2-0431.dump home-210612-1-0433.dump ... home-210729-6-0430.dump home-210730-9-0430.dump home-210731-8-0430.dump home-210801-0-0438.dump home-210803-2-0430.dump home-210804-5-0430.dump home-210805-4-0430.dump home-210806-7-0430.dump home-210807-6-0430.dump Can be restored to any of the dates with a script: Display_Only=1 full_restore home restore 210716 restore home-210701-0-0442.dump to /home/cache/restore restore home-210712-1-0430.dump to /home/cache/restore restore home-210714-2-0430.dump to /home/cache/restore restore home-210716-4-0430.dump to /home/cache/restore For several months I provided a few back-weeks of 'Restore previous versions' that did checkpoints 4x/day. Constructed it using rsync, but it really was too much work for too little feature. Anyway, I'm aware of various security considerations and it seems like the best single thing would be a fast-timout screen saver that would require a password to stop (in addition to the root-ssh login)... Hope
Re: How to manage root<-->root rsync keeping permissions?
On 2021/08/07 08:45, Chris Green via rsync wrote: Because cron/anacron isn't perfect and the machine being backed up nay not be turned on all the time so the time that it tries to backup is most definitely not fixed accurately! My *backups* of important data are incremental backups done once a day for every machine. I also do hourly incremental backups on my desktop machine but that is more for protecting myself against myself than for protecting against intruders or hardware failure. Yeah, that's why I had the 'previous versions thing working. I hope to get that working again at some point a bit more efficiently. I know I need the protection against myself too! The original point of this thread is about something closer to synchronising my (small, Raspberry Pi) DNS server so that if it fails I can get a DNS server back up and running as quickly as possible. Get a few small computers like your pi, and duplicate them. swap a new one in if there's a problem. Or boot from a DVD -- installs everything on boot, and then download variable info from your backup server using knock-knock...* so not only does someone with access to my desktop/laptop need to know the rsyncd username and password but they also cannot delete my existing backups. It runs incremental backups so nothing is ever overwritten either. BTW, incremental backups aren't really the same as 'update' backups, they keep track of the state of the file system (including files no longer there) so you can restore your desktop to a specific day before some unwanted updated was introduced and kept by an update-only backup system. Yes, exactly, or more to the point (in my case anyway) I can restore a specific file to a few hours ago after I've scrambled it in some disastrous way! :-) you too eh, what power we have! ;-) A pretty cool way to get your laptop "let in" to the backup server. Have a random sequence of port open attemps Choose a capital port, a small..oh wait, that's letters...anyway, have a prog that detects the probes. If it gets the right sequence of 10, 20, 60 probes, (whatever), then it opens up the ssh->backup port for 5 minutes or until your laptop connects, (whichever is shorter). If you didn't get in within 5 minutes, prolly need a faster computer. Be sure to make your OPIE check a range of of unused passwords in case you get out of sync. Have the probe-pattern be a 1-time use pattern and generate a few hundred of them for each computer in advance. now you have One-time use passwords just to turn on your secure backup. If someone breaks that, close up shop and move to baja calif and retire! -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
