Re: Syncing from two sources to one target and deleting files not in either source
On 21 Jul 2020, at 09:29, Joe via rsync wrote: > use ls or find to create a list of files on server 1 followed by a list > of files on server 2 > run it through sort -u to eliminate duplicates > use ls or find to make a list of files on ser This is basically what I am doing, though it is a lot more complicated than that. But as I said in an off-list reply to someone else, this is a quite expensive process that has to run separately and must be run when rsnc is not running. And it's fragile. Wayne's suggestion of a protected file that lists the files on server 1 when connecting to server 2 and deletes files on server 3 seems like it's a lot more workable as it is cheap to get a list of files and everything runs in the one process. The list of files is not super large, (thousands, not hundreds of thousands) but they are generally large files, so that's the direction I'll investigate. -- Everything you say is so boring, I replace it with dubstep. -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Syncing from two sources to one target and deleting files not in either source
Given three servers where Server 1 has several GB of files and Server 2 has several GB of files and Server 3 has all the files from both Server 1 and Server 2 in a single directory, how can I remove files on Server 3 that do not exist on either Server 1 or Server 2? For example, it's say Server1:/home/user1 and Server2:home/user1 have rsync jobs that sync to Server3:/backups/users/user1 and most of the files exist on both server1 and server2, then server3 contains the newest version of each file, but it also contains files that do not exist on either of Server1 nd Server2. This means that if Server1 needs to restore /home/user1/folder it will not only get all the files that should be in folder on Server2, but also all the files that USED to be in folder on either server1 or server2. I suspect the answer to this is "this isn't a task for rsync" and if so, what might it be a task for? -- "Are you pondering what I'm pondering?" "Wuh, I think so, Brain, but how will we get three pink flamingos into one pair of Capri pants?" -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Rsync with three folders
OK I have the following setup: Server A: a remote server which runs rsync to a local machine to get new files Local: A local machine Local A: A folder on the local machine with new files for Server A yto get Local B: An empty folder on an external drive connected to Local Now, normally Local B is not involved, but currently there is a LOT of data in Local A (multiple TB) that needs to be transferred to Server A, so what I want to do is take the list of files and instead of Server A grabbing them, somehow send them back to the local machine so they can instead be transferred to Local B and I can then sneaker net the external drive over to the server and cut the transfer time by about 1/50th. I can, of course, do this manually with no issue by just generating the list of files, copying it to local, and then ping this files from Local A to Local B. Caveats, I cannot rsync TO server A, only FROM (that is to say, I can run an rsync command on Server A that connect to anyone, but I cannot run an rsync anywhere that connects to Server A). rsync -avn > file xfer file to Process file list in file to cp files to external Just checking if there is something else I could/shuld do or something I am forgetting about. -- "Are you pondering what I'm pondering?" "I think so, Brain, but why does a forklift have to be so big if all it does is lift forks?" -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Exclude-from file format?
I have seen two main styles in examples for using rsync with exclude-from. The first is simply a list of filename, one per line. The second is a list of filenames, one per line, prefixed with either a - or a + to indicate exclude/include. Which is correct? Also, none show the correct style with a filname containing spaces, though I assumed quoting is OK Exclude1.txt: - filename1 - “file name 1” Exclude.txt: filename1 “file name 1” Which of these will exclude both filename1 and “file name 1” anywhere they appear in the source? -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Allow "--in-place" as an alternative option name for "--inplace"
On Jun 26, 2019, at 2:53 PM, Jan-Benedict Glaw via rsync wrote: > > As I commonly spell --inplace as --in-place, I'd like to suggest this simple > patch: If we’re going to do that add --remove-source and --delete-source as aliases for whatever the command is I have to look up every single time. I mean, really, delete-source is the OBVIOUS flag as it matches with many other flags... -- Hey kids, shake it loose together the spotlight's hitting something That's been known to change the weather we'll kill the fatted calf tonight So stick around you're gonna hear electric music: Solid walls of sound -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
rsnapshot list
Does anyone know of a list for snapshot? I was working on a website today and When I went to check in my snapshots, despite the snapshots running every houir and My working in the directory of most of the last two days, all the files in the backup were from more than three years in the past. In case there is a simple issue someone know about, a quick summary: The last three hourly backups: 8 drwxr-xr-x 3 root wheel 512 Jun 21 18:46 hour.2 8 drwxr-xr-x 3 root wheel 512 Jun 21 19:46 hour.1 8 drwxr-xr-x 3 root wheel 512 Jun 21 20:45 hour.0 Live folder: ls -lstr | tail -3 8 -rw-r--r-- 1 root wheel 3866 Jun 21 19:52 p3.php 8 -rw-r--r-- 1 root wheel 2866 Jun 21 20:17 p4.php 56 -rw-r--r-- 1 root wheel 28626 Jun 21 20:28 photos.php backup from hour.2: ls -lstr | tail -3 8 drwxr-xr-x 3 www wheel 1536 Mar 19 2015 images 8 drwxr-xr-x 2 www wheel 1024 Mar 27 2016 candc 8 drwxr-xr-x 2 www wheel512 Aug 19 2016 cgi-local cgi-local was deleted entirely yesterday and none of the files that were edited or created yesterday or today are there. -- Stone circles were common enough everywhere in the mountains. Druids built them as weather computers, and since it was always cheaper to build a new 33-Megalith circle than to upgrade an old slow one, there were generally plenty of ancient ones around --Lords and Ladies -- Please use reply-all for most replies to avoid omitting the mailing list. To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Error with paths with ()'s
On Aug 17, 2015, at 5:03 PM, Wayne Davison way...@samba.org wrote:
The use of {} on the receiving (remote) side is superfluous -- just
specifying a destination dir (your .../Drive5/ path) is enough for rsync to
use the same name as the source file on the destination.
That is the case if {} contains simply a file name, but is not the case when {}
contains a relative path as well.
/path/to/root/path/to/file
cd /path/to/root
find . -type f -exec rsync {} /path/to/dest/{}
= /path/to/dest/path/to/file
find . -type f -exec rsync {} /path/to/dest/
= /path/to/dest/file
For those instances where you want/need to specify a remote filename, see the
--protect-args (-s) option (which you can make the default via export
RSYNC_PROTECT_ARGS=1, and which will eventually become the default in the
future). Finally, the suggestion to use --files-from=- is a good one, which
fixes all quoting issues even on the local side.
10.0.0.11:/Drive5/\{}\”
Works.
--
Any man who says he can see through women is really missing a lot. -
Groucho Marx
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Error with paths with ()'s
On Aug 17, 2015, at 3:06 PM, Joe jose...@main.nc.us wrote:
First, let me state the obvious. It looks like your code is executing rsync
in a bash one liner once for each file that find returns. That's not cool!
And it's almost definitely not what you wanted to do.
It is perfect;y acceptable and definitely ‘cool’. This gives me stats on each
file, which is what I want.
Among other things, that means that bash is seeing all those embedded blanks
and the parentheses in your file names and getting upset because blanks
delimit arguments and parentheses are used for a number of syntactically
meaningful things.
That does not explain why bash is perfectly happy with the command line if it
is a locally mounted disk (the same disk, in fact). Despite the error, this is
not a bash problem.
So, at a minimum, you need to escape/quote *both* of your file references {}
- not just the one.
Find’s {} is escaped, just doesn’t seem to be escaped properly via ssh/rysnc.
As I said, this works:
find . -type f -atime -1 -exec rsync -aP {} /Drive5/{}
This fails if there are ()’s in the file name.
find . -type f -atime -1 -exec rsync -aP {} 10.0.0.11:/Drive5/{}
But the real issue is that you should probably let find put all the resulting
file names into a file or pipe and send that to rsync once using something
like
--files-from=FILE read list of source-file names from FILE
where you should be able to use - as the file name so it uses the output of
the find command as input to rsync.
Hmm. Maybe. I’ll play with that.
--
Why live in the world when you can live in your head?
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Re: Error with paths with ()'s
On Aug 17, 2015, at 4:05 AM, Kevin Korb k...@sanitarium.net wrote:
If you want find to generate your list use --files-from:
find . -type f -atime -1 -print0 | rsync -aP --files-from=- --from0 .
10.0.0.11:/Volumes/Drive5/
This works:
find . -type f -atime -1 -exec rsync -aP {} /Drive5/{}
This fails if there are ()’s in the file name.
find . -type f -atime -1 -exec rsync -aP {} 10.0.0.11:/Drive5/{}
--
I WILL NOT ENCOURAGE OTHERS TO FLY Bart chalkboard Ep. 7F03
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
Error with paths with ()'s
I was trying to process a bunch of folders to sync them to another drive and
ran across an error I haven’t seen before. Normally I do this sync via a
mounted file system, but this time I tried to do it over ssh:
find . -type f -atime -1 -exec rsync -aP {} 10.0.0.11:/Volumes/Drive5/{} \;
bash: -c: line 0: syntax error near unexpected token `('
bash: -c: line 0: `rsync --server -logDtpre.iLsfx --log-format=X --partial .
/Volumes/Drive5/./Taxes (2012)/W2 (2012).pdf'
rsync: connection unexpectedly closed (0 bytes received so far) [sender]
rsync error: error in rsync protocol data stream (code 12) at io.c(226)
[sender=3.1.1]
If I mount Drive5 and run the same command, it works fine.
Put /Volumes/Drive5/{}” in quotes doesn’t help.
Ideas?
--
Why can't you be in a good mood? How hard is it to decide to be in a
good mood and be in a good mood once in a while?
--
Please use reply-all for most replies to avoid omitting the mailing list.
To unsubscribe or change options: https://lists.samba.org/mailman/listinfo/rsync
Before posting, read: http://www.catb.org/~esr/faqs/smart-questions.html
