Re: [Dorset] Exclude hidden folder in rsync
On 20/08/17 21:58, Ralph Corderoy wrote:
Hi Tim,
I want to compress my rsynced home folder, I want the video folder in
the final compressed file but I don't want tar to try and compress it
as it compresses the rest of the home folder, does that make sense?
Yes, but it can't be done. tar(1) creates a tar file, and that's a
simple catenation of a bit of metadata about the next file, and then the
file's bytes, padded out to a nice round multiple of bytes for each
file. Traditionally, you'd have either piped tar into a compression
program, or produced foo.tar and then compressed it with a separate
command.
These days, tar has grown options for compressing, but they just do the
pipe for you. Here's tar processing the `j' option and kicking off a
bzip2(1) as the destination for write(2)s.
$ strace -fe execve tar cjf /dev/null foo
execve("/usr/bin/tar", ["tar", "cjf", "/dev/null", "foo"], 0x7ffe017683a8
/* 66 vars */) = 0
strace: Process 3162 attached
strace: Process 3163 attached
[pid 3163] execve("/bin/sh", ["/bin/sh", "-c", "bzip2"], 0x7ffe0deb68c0
/* 66 vars */) = 0
[pid 3163] execve("/usr/bin/bzip2", ["bzip2"], 0x73a950 /* 66 vars */) = 0
[pid 3163] +++ exited with 0 +++
[pid 3162] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED,
si_pid=3163, si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
[pid 3162] +++ exited with 0 +++
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=3162,
si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
+++ exited with 0 +++
$
And the compression programs just see a stream of bytes, they've no idea
that it's a tar file, and that you'd like the video directory to be
skipped.
That said, you could let the compressor attempt to do it anyway. It
will burn some CPU, and may grow slightly in size. Or, exclude it from
the compressed tar file and produce a second, video, uncompressed tar
file.
What do you do with these tar files then? Is rsync and tar the right
combination? (I use tar for backups myself so I'm not against it per
se.)
Cheers, Ralph.
Thanks for the input guys, I think I will leave the video folder in the tar.
Rsync and tar just form a simple little backup for my home folder. It
allows me to keep several copies of my home folder just encase something
goes wrong.
Tim
--
Next meeting: Bournemouth, Tuesday, 2017-09-05 20:00
Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/
New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING
Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Exclude hidden folder in rsync
> > I want to compress my rsynced home folder, I want the video > > folder in the final compressed file but I don't want tar to > > try and compress it as it compresses the rest of the home > > folder, does that make sense? > > Yes, but it can't be done. Unless you are prepared to accept two tar files as output. (One compressed, one uncompressed, with contents to be reunited upon untarring.) Patrick -- Next meeting: Bournemouth, Tuesday, 2017-09-05 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Exclude hidden folder in rsync
Hi Tim,
> I want to compress my rsynced home folder, I want the video folder in
> the final compressed file but I don't want tar to try and compress it
> as it compresses the rest of the home folder, does that make sense?
Yes, but it can't be done. tar(1) creates a tar file, and that's a
simple catenation of a bit of metadata about the next file, and then the
file's bytes, padded out to a nice round multiple of bytes for each
file. Traditionally, you'd have either piped tar into a compression
program, or produced foo.tar and then compressed it with a separate
command.
These days, tar has grown options for compressing, but they just do the
pipe for you. Here's tar processing the `j' option and kicking off a
bzip2(1) as the destination for write(2)s.
$ strace -fe execve tar cjf /dev/null foo
execve("/usr/bin/tar", ["tar", "cjf", "/dev/null", "foo"], 0x7ffe017683a8
/* 66 vars */) = 0
strace: Process 3162 attached
strace: Process 3163 attached
[pid 3163] execve("/bin/sh", ["/bin/sh", "-c", "bzip2"], 0x7ffe0deb68c0 /*
66 vars */) = 0
[pid 3163] execve("/usr/bin/bzip2", ["bzip2"], 0x73a950 /* 66 vars */) = 0
[pid 3163] +++ exited with 0 +++
[pid 3162] --- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=3163,
si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
[pid 3162] +++ exited with 0 +++
--- SIGCHLD {si_signo=SIGCHLD, si_code=CLD_EXITED, si_pid=3162,
si_uid=1000, si_status=0, si_utime=0, si_stime=0} ---
+++ exited with 0 +++
$
And the compression programs just see a stream of bytes, they've no idea
that it's a tar file, and that you'd like the video directory to be
skipped.
That said, you could let the compressor attempt to do it anyway. It
will burn some CPU, and may grow slightly in size. Or, exclude it from
the compressed tar file and produce a second, video, uncompressed tar
file.
What do you do with these tar files then? Is rsync and tar the right
combination? (I use tar for backups myself so I'm not against it per
se.)
Cheers, Ralph.
--
Next meeting: Bournemouth, Tuesday, 2017-09-05 20:00
Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/
New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING
Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Exclude hidden folder in rsync
On 20/08/17 16:12, Tim wrote: On 20/08/17 11:16, Ralph Corderoy wrote: Hi Tim, what is the correct phrase to stop rsync from copying the cache folder? Hard to say what you're doing wrong without an example. It works for me. $ find src | sort src src/bar src/bar/file1 src/.cache src/.cache/file1 src/file1 src/foo src/foo/.cache src/foo/.cache/file1 src/foo/file1 $ $ rsync -aciHAXS --exclude /.cache/ src/ dest cd+ ./ >f+ file1 cd+ bar/ >f+ bar/file1 cd+ foo/ >f+ foo/file1 cd+ foo/.cache/ >f+ foo/.cache/file1 $ $ >src/foo/file2 $ rsync -aciHAXS --exclude /.cache/ src/ dest .d..t.. foo/ >f+ foo/file2 $ $ find -name .cache | sorV ./dest/foo/.cache ./src/.cache ./src/foo/.cache $ Cheers, Ralph. Hi Ralph Thanks, from what you posted I managed to get it working, it is basically my home folder I am backing up but I am trying to safe space on my disk by simply syncing those folders in my home folder that I will want if I lost my home folder on my main disk. I don't really need .thumbnails and .cache which were two of the biggest space takers I did not need. To move this on slightly, once rsync has run I want compress the whole home folders (it copies it to another folder then compresses for storage), it is currently around 21gb but half of that is a video folder that I have which is 10gb in size, how do I exclude a folder from tar? I can see where you can choose files and patters *.ogg but the videos are various formats so it would be simpler to exclude the whole folder? is it as simple as --exclude=videofolder? Not sure I am reading the man page correctly Tim After reading something online I am going to clarify what I am trying to do, I want to compress my rsynced home folder, I want the video folder in the final compressed file but I don't want tar to try and compress it as it compresses the rest of the home folder, does that make sense? Tim -- Next meeting: Bournemouth, Tuesday, 2017-09-05 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Exclude hidden folder in rsync
On 20/08/17 11:16, Ralph Corderoy wrote: Hi Tim, what is the correct phrase to stop rsync from copying the cache folder? Hard to say what you're doing wrong without an example. It works for me. $ find src | sort src src/bar src/bar/file1 src/.cache src/.cache/file1 src/file1 src/foo src/foo/.cache src/foo/.cache/file1 src/foo/file1 $ $ rsync -aciHAXS --exclude /.cache/ src/ dest cd+ ./ >f+ file1 cd+ bar/ >f+ bar/file1 cd+ foo/ >f+ foo/file1 cd+ foo/.cache/ >f+ foo/.cache/file1 $ $ >src/foo/file2 $ rsync -aciHAXS --exclude /.cache/ src/ dest .d..t.. foo/ >f+ foo/file2 $ $ find -name .cache | sorV ./dest/foo/.cache ./src/.cache ./src/foo/.cache $ Cheers, Ralph. Hi Ralph Thanks, from what you posted I managed to get it working, it is basically my home folder I am backing up but I am trying to safe space on my disk by simply syncing those folders in my home folder that I will want if I lost my home folder on my main disk. I don't really need .thumbnails and .cache which were two of the biggest space takers I did not need. To move this on slightly, once rsync has run I want compress the whole home folders (it copies it to another folder then compresses for storage), it is currently around 21gb but half of that is a video folder that I have which is 10gb in size, how do I exclude a folder from tar? I can see where you can choose files and patters *.ogg but the videos are various formats so it would be simpler to exclude the whole folder? is it as simple as --exclude=videofolder? Not sure I am reading the man page correctly Tim -- Next meeting: Bournemouth, Tuesday, 2017-09-05 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
Re: [Dorset] Exclude hidden folder in rsync
Hi Tim, > what is the correct phrase to > stop rsync from copying the cache folder? Hard to say what you're doing wrong without an example. It works for me. $ find src | sort src src/bar src/bar/file1 src/.cache src/.cache/file1 src/file1 src/foo src/foo/.cache src/foo/.cache/file1 src/foo/file1 $ $ rsync -aciHAXS --exclude /.cache/ src/ dest cd+ ./ >f+ file1 cd+ bar/ >f+ bar/file1 cd+ foo/ >f+ foo/file1 cd+ foo/.cache/ >f+ foo/.cache/file1 $ $ >src/foo/file2 $ rsync -aciHAXS --exclude /.cache/ src/ dest .d..t.. foo/ >f+ foo/file2 $ $ find -name .cache | sorV ./dest/foo/.cache ./src/.cache ./src/foo/.cache $ Cheers, Ralph. -- Next meeting: Bournemouth, Tuesday, 2017-09-05 20:00 Meets, Mailing list, IRC, LinkedIn, ... http://dorset.lug.org.uk/ New thread: mailto:dorset@mailman.lug.org.uk / CHECK IF YOU'RE REPLYING Reporting bugs well: http://goo.gl/4Xue / TO THE LIST OR THE AUTHOR
