> We use jsch as the library to download files over sftp in our system
> Input
> for the sftp module is a folder on a remote host, this folders often
> contains alot of subfolders with many small files, total amout of data
> can
> be a couple of 100gb. Now I think I have just a very standard way of
> downloading each file for all the subfolders on the host. So I was
> wondering
> if there was any tricks I could to speed up things? E.g. some sort of
> batch
> download of a directory. After the file have been downloaded we add it
> to a
> tar archive. I guess I could use 'scp -r ', will there be any
> implications
> with respect to performance etc?
Hi,
I am following the suggestions given in this reply to use
tar -c -f -z folder to speed up the download transfers.
I tried it in my program, it is said just send the output to std output
and later piped and save it in a file.
Is there any example to show how to do that. I would really appreciate that.
Thanks.
-------------------------------------------------------------------------------------------------------------------------
scp is not really fast, as it needs a roundtrip between each file, even
if using `-r`.
In theory, sftp can be faster by requesting multiple files in parallel,
but I'm not sure that/how the JSch implementation can do this.
If you have a shell/exec access to the remote system (not only sftp), I
think the fastest would be to create the tar file there and transfer it
at once.
You don't even need to save it as a file and transfer via sftp, simply
output the tar to the standard output, which then can be piped to a file
on the client side.
Your command (in an exec channel) would be:
tar -c -f- folder
Maybe you want to gzip it immediately, then use -z. If you don't, you
might think about enabling transport-level compression with JSch (see
the examples).
------------------------------------------------------------------------------
uberSVN's rich system and user administration capabilities and model
configuration take the hassle out of deploying and managing Subversion and
the tools developers use with it. Learn more about uberSVN and get a free
download at: http://p.sf.net/sfu/wandisco-dev2dev
_______________________________________________
JSch-users mailing list
JSch-users@lists.sourceforge.net
https://lists.sourceforge.net/lists/listinfo/jsch-users
