Re: [gentoo-user] tar over ssh
Uwe Thiem wrote: On 25 January 2006 06:46, Tom Smith wrote: I read something some time ago that suggested if you transfer a compressed file over a compressed SFTP connection, for example, that it would take longer to transfer the data versus if only the data or the connection was compressed. The reason, as I recall, had to do with compressing already compressed data--this apparently created some overhead on the connection. What is a compressed sftp connection? Uwe In ssh, sftp, scp you can use the -C option to enable compression. This simply (de)compresses, on the fly, the data being transferred between the client and the server computer. It's generally suggested for use over slow connections (such as dial-ups) but I've experienced performance improvements in X11 forwarding, and some data transfers, when using it. -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] tar over ssh
Trenton Adams wrote: Not possible on a windows machine. :P Wrong. Alexander Skwar -- printk(KERN_WARNING Warning: defective CD-ROM (volume sequence number). Enabling \cruft\ mount option.\n); linux-2.2.16/fs/isofs/inode.c -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] tar over ssh
I read something some time ago that suggested if you transfer a compressed file over a compressed SFTP connection, for example, that it would take longer to transfer the data versus if only the data or the connection was compressed. The reason, as I recall, had to do with compressing already compressed data--this apparently created some overhead on the connection. Did you look at this situation in your tests? If so, what were the results? No, I see absolutely no reason to use sftp. Just scp. By default, compression is off on that, unless you've modified your ssh_conf. But I have heard the same thing about doubly-compressing things. -- gentoo-user@gentoo.org mailing list -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] tar over ssh
It brings me great joy to know that my original post has spawned such madness. :-) John Jolet wrote: I read something some time ago that suggested if you transfer a compressed file over a compressed SFTP connection, for example, that it would take longer to transfer the data versus if only the data or the connection was compressed. The reason, as I recall, had to do with compressing already compressed data--this apparently created some overhead on the connection. Did you look at this situation in your tests? If so, what were the results? No, I see absolutely no reason to use sftp. Just scp. By default, compression is off on that, unless you've modified your ssh_conf. But I have heard the same thing about doubly-compressing things. -- gentoo-user@gentoo.org mailing list -- Han Solo: Traveling through hyperspace ain't like dusting crops, boy. -- gentoo-user@gentoo.org mailing list
[gentoo-user] tar over ssh
Hey guys. I've got this big fat backup server with no space left on the hard drive to store a tar file. I'd like to pipe a tar through ssh, but not sure what the command would be. Something to the effect of: # cat /var/backup | ssh backup.homelan.com 'tar data.info.gz' So that, the data is actually being sent over ssh, and then archived on the destination machine. Help! :-) -- Luke Skywalker: What a piece of junk! Han Solo: She'll make point five past lightspeed. She may not look like much but she's got it where it counts, kid. I've made a lot of special modifications myself. -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] tar over ssh
On Jan 24, 2006, at 10:57 AM, Jeff wrote: Hey guys. I've got this big fat backup server with no space left on the hard drive to store a tar file. I'd like to pipe a tar through ssh, but not sure what the command would be. Something to the effect of: # cat /var/backup | ssh backup.homelan.com 'tar data.info.gz' So that, the data is actually being sent over ssh, and then archived on the destination machine. you have a tar file you want transferred, or a directory called backup? if it's a directory, i'd (from the target machine) ssh sourcemachine tar /var/backup/* - data.info.gz see the very long discussion we had on this list about this a few months ago. Help! :-) -- Luke Skywalker: What a piece of junk! Han Solo: She'll make point five past lightspeed. She may not look like much but she's got it where it counts, kid. I've made a lot of special modifications myself. -- gentoo-user@gentoo.org mailing list -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] tar over ssh
Jeff wrote: Hey guys. I've got this big fat backup server with no space left on the hard drive to store a tar file. I'd like to pipe a tar through ssh, but not sure what the command would be. Something to the effect of: # cat /var/backup | ssh backup.homelan.com 'tar data.info.gz' So that, the data is actually being sent over ssh, and then archived on the destination machine. Help! :-) Not possible. What you want is more along the lines of AFS, NFS, SMB, or the like. If your backup files already exist, you can use SCP or SFTP to copy those to a different server. -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] tar over ssh
Jeff wrote: Hey guys. I've got this big fat backup server with no space left on the hard drive to store a tar file. I'd like to pipe a tar through ssh, but not sure what the command would be. Something to the effect of: # cat /var/backup | ssh backup.homelan.com 'tar data.info.gz' So that, the data is actually being sent over ssh, and then archived on the destination machine. tar -zcf - /var/backup | ssh backup.homelan.com ( cat data.info.gz ) something similar, probably is possible to avoid the use of cat bat don't came in mind at the moment -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] tar over ssh
Not possible. Wrong, we are unix sysadmins, the thing that more nearly resemble that is : I've not time to do that ;-) -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] tar over ssh
On Jan 24, 2006, at 11:20 AM, Tom Smith wrote: Jeff wrote: Hey guys. I've got this big fat backup server with no space left on the hard drive to store a tar file. I'd like to pipe a tar through ssh, but not sure what the command would be. Something to the effect of: # cat /var/backup | ssh backup.homelan.com 'tar data.info.gz' So that, the data is actually being sent over ssh, and then archived on the destination machine. Help! :-) Not possible. What you want is more along the lines of AFS, NFS, SMB, or the like. WRONG. I do it all the time. If your backup files already exist, you can use SCP or SFTP to copy those to a different server. -- gentoo-user@gentoo.org mailing list -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] tar over ssh
John Jolet wrote: On Jan 24, 2006, at 11:20 AM, Tom Smith wrote: Jeff wrote: Hey guys. I've got this big fat backup server with no space left on the hard drive to store a tar file. I'd like to pipe a tar through ssh, but not sure what the command would be. Something to the effect of: # cat /var/backup | ssh backup.homelan.com 'tar data.info.gz' So that, the data is actually being sent over ssh, and then archived on the destination machine. Help! :-) Not possible. What you want is more along the lines of AFS, NFS, SMB, or the like. WRONG. I do it all the time. Ok,. my bad. (Open mouth, insert foot. :-D ) Being a *nix junkie, I tend to do some things old school--that is, there are specific tools that are (dare I say) more specialized to such a task. You know... SSH = Secure SHell, SCP = Secure CoPy, SFTP = Secure FTP... So I had never really looked into using the ssh program for copying files between servers--it's always been more of a telnet-like application for me. -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] tar over ssh
Tom you big jerk. LOL! jk man... Well, I would call it, piping through a tunnel? Tunneling through a pipe? The concept seems very *nix-like to me. *shrug* Thanks to all... wish me luck! Tom Smith wrote: John Jolet wrote: On Jan 24, 2006, at 11:20 AM, Tom Smith wrote: Jeff wrote: Hey guys. I've got this big fat backup server with no space left on the hard drive to store a tar file. I'd like to pipe a tar through ssh, but not sure what the command would be. Something to the effect of: # cat /var/backup | ssh backup.homelan.com 'tar data.info.gz' So that, the data is actually being sent over ssh, and then archived on the destination machine. Help! :-) Not possible. What you want is more along the lines of AFS, NFS, SMB, or the like. WRONG. I do it all the time. Ok,. my bad. (Open mouth, insert foot. :-D ) Being a *nix junkie, I tend to do some things old school--that is, there are specific tools that are (dare I say) more specialized to such a task. You know... SSH = Secure SHell, SCP = Secure CoPy, SFTP = Secure FTP... So I had never really looked into using the ssh program for copying files between servers--it's always been more of a telnet-like application for me. -- Darth Vader: I find your lack of faith disturbing. -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] tar over ssh
On Jan 24, 2006, at 11:46 AM, Tom Smith wrote: John Jolet wrote: On Jan 24, 2006, at 11:20 AM, Tom Smith wrote: Jeff wrote: Hey guys. I've got this big fat backup server with no space left on the hard drive to store a tar file. I'd like to pipe a tar through ssh, but not sure what the command would be. Something to the effect of: # cat /var/backup | ssh backup.homelan.com 'tar data.info.gz' So that, the data is actually being sent over ssh, and then archived on the destination machine. Help! :-) Not possible. What you want is more along the lines of AFS, NFS, SMB, or the like. WRONG. I do it all the time. Ok,. my bad. (Open mouth, insert foot. :-D ) Being a *nix junkie, I tend to do some things old school--that is, there are specific tools that are (dare I say) more specialized to such a task. You know... SSH = Secure SHell, SCP = Secure CoPy, SFTP = Secure FTP... So I had never really looked into using the ssh program for copying files between servers--it's always been more of a telnet-like application for me. hmmm, old school, eh? I was doing that tar trick about 10 or 11 years ago. you HAVE to do that if you have no room to complete the tar file on the source, THEN transfer it. it's quicker than scping a lot of files and then tarring them up on the destination, especially if THAT doesn't have room for both the source files and the tar. I did extensive performance testing for database migrations about 5 years ago and what I said was by far the most efficient, timewise (though, I didn't simply redirect to a file, but dd of=filename) -- gentoo-user@gentoo.org mailing list -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] tar over ssh
On Tue, 24 Jan 2006 10:20:10 -0700, Tom Smith wrote: Not possible. Why, is he using Windows? I don't know how != not possible. -- Neil Bothwick Voting Democrat or Republican is like choosing a cabin in the Titanic. signature.asc Description: PGP signature
Re: [gentoo-user] tar over ssh
John Jolet wrote: On Jan 24, 2006, at 11:46 AM, Tom Smith wrote: John Jolet wrote: On Jan 24, 2006, at 11:20 AM, Tom Smith wrote: Jeff wrote: Hey guys. I've got this big fat backup server with no space left on the hard drive to store a tar file. I'd like to pipe a tar through ssh, but not sure what the command would be. Something to the effect of: # cat /var/backup | ssh backup.homelan.com 'tar data.info.gz' So that, the data is actually being sent over ssh, and then archived on the destination machine. Help! :-) Not possible. What you want is more along the lines of AFS, NFS, SMB, or the like. WRONG. I do it all the time. Ok,. my bad. (Open mouth, insert foot. :-D ) Being a *nix junkie, I tend to do some things old school--that is, there are specific tools that are (dare I say) more specialized to such a task. You know... SSH = Secure SHell, SCP = Secure CoPy, SFTP = Secure FTP... So I had never really looked into using the ssh program for copying files between servers--it's always been more of a telnet-like application for me. hmmm, old school, eh? I was doing that tar trick about 10 or 11 years ago. you HAVE to do that if you have no room to complete the tar file on the source, THEN transfer it. it's quicker than scping a lot of files and then tarring them up on the destination, especially if THAT doesn't have room for both the source files and the tar. I did extensive performance testing for database migrations about 5 years ago and what I said was by far the most efficient, timewise (though, I didn't simply redirect to a file, but dd of=filename) Well, perhaps old school has different meanings to different people. :-) I was referring to the UNIX tools philosophy in which each program has a very specific use, similar to qmail (the original, unmodified qmail, that is). And this is usually the direction I take when looking for tools to accomplish some task. But I suppose this philosophy doesn't really apply quite as much nowadays. I must say, though, I've always managed to anticipate the storage needs of my servers so running low on or (even worse) running out of disk space has never been a problem. So I've never had to research such tricks to get things to work within those types of constraints. Call me quirky, but that's part of being a sysadmin... Yes? ;-) -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] tar over ssh
Well, perhaps old school has different meanings to different people. :-) I was referring to the UNIX tools philosophy in which each program has a very specific use, similar to qmail (the original, unmodified qmail, that is). And this is usually the direction I take when looking for tools to accomplish some task. But I suppose this philosophy doesn't really apply quite as much nowadays. I must say, though, I've always managed to anticipate the storage needs of my servers so running low on or (even worse) running out of disk space has never been a problem. So I've never had to research such tricks to get things to work within those types of constraints. Call me quirky, but that's part of being a sysadmin... Yes? ;-) -- as do I, but how often do you get to start with no servers at all? I think there's this one sysadmin running around setting up servers badly, and we all get hired in after him to clean upi've got one db server with a 3-gig root partition and a 5-gig tmp partition, and all the programs aren't using the tmp partition, using the /tmp directory. and the root filesystem is 90% full. Ga! gentoo-user@gentoo.org mailing list -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] tar over ssh
This example that Francesco illustrates seems to work pretty well. I guess my main concern was with tar - would it be able to handle a filesystem this large? Myself, I haven't seen or heard any scary stories thus far. Anyone shed light on tar limitations? Thanks for all the colorful replies. :-) -Jeff Francesco Riosa wrote: Jeff wrote: Hey guys. I've got this big fat backup server with no space left on the hard drive to store a tar file. I'd like to pipe a tar through ssh, but not sure what the command would be. Something to the effect of: # cat /var/backup | ssh backup.homelan.com 'tar data.info.gz' So that, the data is actually being sent over ssh, and then archived on the destination machine. tar -zcf - /var/backup | ssh backup.homelan.com ( cat data.info.gz ) something similar, probably is possible to avoid the use of cat bat don't came in mind at the moment -- Officer: We've analyzed their attack, sir, and there is a danger. Should I have your ship standing by? Governor Tarkin: Evacuate? In our moment of triumph? I think you overestimate their chances. -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] tar over ssh
DUH ME! Open mouth, insert face... Ok, what I *meant* to say from post #1, is, the filesystem I'm tarballing is quite large - 25g. The tar command should be able to digest this, yes? Should I be worried? Thanks again all. Jeff wrote: This example that Francesco illustrates seems to work pretty well. I guess my main concern was with tar - would it be able to handle a filesystem this large? Myself, I haven't seen or heard any scary stories thus far. Anyone shed light on tar limitations? Thanks for all the colorful replies. :-) -Jeff Francesco Riosa wrote: Jeff wrote: Hey guys. I've got this big fat backup server with no space left on the hard drive to store a tar file. I'd like to pipe a tar through ssh, but not sure what the command would be. Something to the effect of: # cat /var/backup | ssh backup.homelan.com 'tar data.info.gz' So that, the data is actually being sent over ssh, and then archived on the destination machine. tar -zcf - /var/backup | ssh backup.homelan.com ( cat data.info.gz ) something similar, probably is possible to avoid the use of cat bat don't came in mind at the moment -- Officer: We've analyzed their attack, sir, and there is a danger. Should I have your ship standing by? Governor Tarkin: Evacuate? In our moment of triumph? I think you overestimate their chances. -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] tar over ssh
On Jan 24, 2006, at 2:22 PM, Jeff wrote: This example that Francesco illustrates seems to work pretty well. I guess my main concern was with tar - would it be able to handle a filesystem this large? Myself, I haven't seen or heard any scary stories thus far. Anyone shed light on tar limitations? all of tar's limitations will have to do with the output file. typically, you'll run into problems at 2 gigs on some old kernels, or *nix variants. tar used to have a limit of like 8 gigs or so, assuming the underlying kernel/filesystem would allow itbut I haven't tried to push that limit in quite a while. For instance, 4 gigs is where you crash if writing to a fat32 partition. Thanks for all the colorful replies. :-) -Jeff Francesco Riosa wrote: Jeff wrote: Hey guys. I've got this big fat backup server with no space left on the hard drive to store a tar file. I'd like to pipe a tar through ssh, but not sure what the command would be. Something to the effect of: # cat /var/backup | ssh backup.homelan.com 'tar data.info.gz' So that, the data is actually being sent over ssh, and then archived on the destination machine. tar -zcf - /var/backup | ssh backup.homelan.com ( cat data.info.gz ) something similar, probably is possible to avoid the use of cat bat don't came in mind at the moment -- Officer: We've analyzed their attack, sir, and there is a danger. Should I have your ship standing by? Governor Tarkin: Evacuate? In our moment of triumph? I think you overestimate their chances. -- gentoo-user@gentoo.org mailing list -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] tar over ssh
On Tuesday 24 January 2006 21:40, Jeff wrote: DUH ME! Open mouth, insert face... Ok, what I *meant* to say from post #1, is, the filesystem I'm tarballing is quite large - 25g. The tar command should be able to digest this, yes? Should I be worried? Last week i back'ed up a machine with 4 80G disks as RAID5 with the method mentioned before. tar-size on the desination machine was about 120GB compressed (yes, one File:). Both filesystems are reiserfs. Restore again wih the same method, only the other way around, to 4 250GB-disks. No problems. Tip: check your destination tar file with tar -tzf ... or tar -tjf ... before you delete the source. Compression is a good check that no data has been changed during transfer of the data. earny/ -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] tar over ssh
Not possible on a windows machine. :P -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] tar over ssh
Ernst Herzberg wrote:
On Tuesday 24 January 2006 21:40, Jeff wrote:
DUH ME! Open mouth, insert face...
Ok, what I *meant* to say from post #1, is, the filesystem I'm
tarballing is quite large - 25g. The tar command should be able to
digest this, yes? Should I be worried?
Last week i back'ed up a machine with 4 80G disks as RAID5 with the method
mentioned before. tar-size on the desination machine was about 120GB
compressed (yes, one File:). Both filesystems are reiserfs.
Restore again wih the same method, only the other way around, to 4
250GB-disks. No problems.
Tip: check your destination tar file with tar -tzf ... or tar -tjf ... before
you delete the source. Compression is a good check that no data has been
changed during transfer of the data.
earny/
Hey, I think I might have something useful to add here... :-D
To keep my tar file sizes more manageable, any directories containing
large directories have script blocks within the backup script that
create a tar file for each directory in that directory. (Whoa! Did I
just say that!?) Here's the block of code that, for example, handles my
/home directories:
code
dt=`date +%G%m%d-%H%M%S`
find /home/ -type s /tmp/home-sockets.tmp
for x in `ls -lA /home/ | awk '{print $9}'`
do
tar cpPj -X /tmp/home-sockets.tmp -f
/var/backups/home-$x-$dt.tbz2 /home/$x
done
rm -f /tmp/home-sockets.tmp
/code
This creates a separate tar file for each directory in /home. The $dt
var isn't required, of course... I just use it to time stamp all of my
backup files as it makes it easier to track them.
Regardless of whether or not the kernel or file system can support the
huge tar files others have referred to, I prefer to always make things
as manageable and modular as possible. The smaller the files, the
easier (quicker, really) they are to work with.
Just food for thought...
--
gentoo-user@gentoo.org mailing list
Re: [gentoo-user] tar over ssh
On Jan 24, 2006, at 5:25 PM, Iain Buchanan wrote: On Tue, 2006-01-24 at 17:23 +, Francesco Riosa wrote: Jeff wrote: Hey guys. I've got this big fat backup server with no space left on the hard drive to store a tar file. I'd like to pipe a tar through ssh, but not sure what the command would be. [snip] So that, the data is actually being sent over ssh, and then archived on the destination machine. tar -zcf - /var/backup | ssh backup.homelan.com ( cat data.info.gz ) something similar, probably is possible to avoid the use of cat bat don't came in mind at the moment wow, I am quite blown away - not only at the fact that I just tried the above command (and it worked of course) but also that I've found a new way of doing something that I never thought of before. This is one to remember! (now, if only I'd backed up yesterday, before I did an accidental `rm *` instead of `rm *~`) good thing none of US has ever done that...as root from the / on a running production serverin the middle of month-end -- Iain Buchanan iain at netspace dot net dot au It's not? Are you saying that you SHOULD allow people (other than William Wallace) to shoot lightning bolts from their arse? -- Seth Galbraith -- gentoo-user@gentoo.org mailing list -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] tar over ssh
On Tuesday 24 January 2006 03:45 pm, John Jolet wrote: (now, if only I'd backed up yesterday, before I did an accidental `rm *` instead of `rm *~`) good thing none of US has ever done that...as root from the / on a running production serverin the middle of month-end -- Yes... But it's that -R option that really kills you... OO!! Ctrl-C! Ctrl-C! Ctrl-C! -- Eric Bliss systems design and integration, CreativeCow.Net -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] tar over ssh
On Tue, 2006-01-24 at 17:23 +, Francesco Riosa wrote: Jeff wrote: Hey guys. I've got this big fat backup server with no space left on the hard drive to store a tar file. I'd like to pipe a tar through ssh, but not sure what the command would be. Something to the effect of: # cat /var/backup | ssh backup.homelan.com 'tar data.info.gz' So that, the data is actually being sent over ssh, and then archived on the destination machine. tar -zcf - /var/backup | ssh backup.homelan.com ( cat data.info.gz ) There's another way. This assumes your originating server's CPU is slow/precious and you have a 16 way node on a backup server (HAHA!!) tar cf - /var/backup | ssh backup.homelan.com gzip -c filename.tar.gz But you transfer the stream uncompressed, so more bits get transferred. -- Ow Mun Heng Gentoo/Linux on DELL D600 1.4Ghz 1.5GB RAM 98% Microsoft(tm) Free!! Neuromancer 11:10:24 up 11:00, 6 users, load average: 1.08, 1.04, 1.15 -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] tar over ssh
On Jan 24, 2006, at 9:10 PM, Ow Mun Heng wrote: On Tue, 2006-01-24 at 17:23 +, Francesco Riosa wrote: Jeff wrote: Hey guys. I've got this big fat backup server with no space left on the hard drive to store a tar file. I'd like to pipe a tar through ssh, but not sure what the command would be. Something to the effect of: # cat /var/backup | ssh backup.homelan.com 'tar data.info.gz' So that, the data is actually being sent over ssh, and then archived on the destination machine. tar -zcf - /var/backup | ssh backup.homelan.com ( cat data.info.gz ) There's another way. This assumes your originating server's CPU is slow/precious and you have a 16 way node on a backup server (HAHA!!) tar cf - /var/backup | ssh backup.homelan.com gzip -c filename.tar.gz But you transfer the stream uncompressed, so more bits get transferred. you're kidding, right? Unless you've got a PII on the originating end and are using gigabit ethernet between the two nodes, compressing the data before transmission will almost always be faster. I tested this scenerio extensively about 3 years ago, using linux, aix, and solaris hosts. In no case was transferring uncompressed data faster than compressing (at least to some degree) the data on the originating server. And frankly, no matter what you do...wouldn't you hope ALL the bits get transferred? :) -- Ow Mun Heng Gentoo/Linux on DELL D600 1.4Ghz 1.5GB RAM 98% Microsoft(tm) Free!! Neuromancer 11:10:24 up 11:00, 6 users, load average: 1.08, 1.04, 1.15 -- gentoo-user@gentoo.org mailing list -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] tar over ssh
On Tue, 2006-01-24 at 21:19 -0600, John Jolet wrote: On Jan 24, 2006, at 9:10 PM, Ow Mun Heng wrote: There's another way. This assumes your originating server's CPU is slow/precious and you have a 16 way node on a backup server (HAHA!!) tar cf - /var/backup | ssh backup.homelan.com gzip -c filename.tar.gz But you transfer the stream uncompressed, so more bits get transferred. you're kidding, right? Not really. I've not tried it out yet.. but it's one more option to throw in the mix. Laptop - 1.4Ghz P-M Server - 300Mhz laptop - Server Local Compression Real0m53.414s Remote COmpression (server COmpress) real1m53.721s Server- Laptop Local Compression real1m10.745s Remote Comression (Laptop Compress) real1m54.132s Unless you've got a PII on the originating end and are using gigabit ethernet between the two nodes, compressing the data before transmission will almost always be faster. This is done on a 10MBit/s Lan, so the bottleneck is on the LAN. Caveat-Emptor :-) In no case was transferring uncompressed data faster than compressing (at least to some degree) the data on the originating server. And frankly, no matter what you do...wouldn't you hope ALL the bits get transferred? :) But of course :-) 1 Bad bit and the whole archive gets screwed. -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] tar over ssh
John Jolet wrote: On Jan 24, 2006, at 9:10 PM, Ow Mun Heng wrote: On Tue, 2006-01-24 at 17:23 +, Francesco Riosa wrote: Jeff wrote: Hey guys. I've got this big fat backup server with no space left on the hard drive to store a tar file. I'd like to pipe a tar through ssh, but not sure what the command would be. Something to the effect of: # cat /var/backup | ssh backup.homelan.com 'tar data.info.gz' So that, the data is actually being sent over ssh, and then archived on the destination machine. tar -zcf - /var/backup | ssh backup.homelan.com ( cat data.info.gz ) There's another way. This assumes your originating server's CPU is slow/precious and you have a 16 way node on a backup server (HAHA!!) tar cf - /var/backup | ssh backup.homelan.com gzip -c filename.tar.gz But you transfer the stream uncompressed, so more bits get transferred. you're kidding, right? Unless you've got a PII on the originating end and are using gigabit ethernet between the two nodes, compressing the data before transmission will almost always be faster. I tested this scenerio extensively about 3 years ago, using linux, aix, and solaris hosts. In no case was transferring uncompressed data faster than compressing (at least to some degree) the data on the originating server. And frankly, no matter what you do...wouldn't you hope ALL the bits get transferred? :) I read something some time ago that suggested if you transfer a compressed file over a compressed SFTP connection, for example, that it would take longer to transfer the data versus if only the data or the connection was compressed. The reason, as I recall, had to do with compressing already compressed data--this apparently created some overhead on the connection. Did you look at this situation in your tests? If so, what were the results? -- gentoo-user@gentoo.org mailing list
Re: [gentoo-user] tar over ssh
On 25 January 2006 06:46, Tom Smith wrote: I read something some time ago that suggested if you transfer a compressed file over a compressed SFTP connection, for example, that it would take longer to transfer the data versus if only the data or the connection was compressed. The reason, as I recall, had to do with compressing already compressed data--this apparently created some overhead on the connection. What is a compressed sftp connection? Uwe -- Unix is sexy: who | grep -i blonde | date cd ~; unzip; touch; strip; finger mount; gasp; yes; uptime; umount sleep -- gentoo-user@gentoo.org mailing list
