Arn, There is a way to transfer a directory as a single tar-stream, like this:
1. tar up source directory prior to transfer 2. transfer the tar-stream 3. untar the archive on the destination without manual taring/untaring on the client and the server. We implemented this for a community that uses GridFTP heavily for transfers of 42GB sized directories containing 130.000 rather small files in a nested directory structure. It works very reliable this way. The only downside I know is that you cannot use any of the advanced features of GridFTP then, like parallelism: The tar-stream transfers became unreliable. To do this you must enable the popen driver in GridFTP. I recommend the latest server from 5.0.0 plus a GridFTP patch. For the taring on the client-side you can use globus-url-copy using certain flags. We built on top of the jglobus Java API to get it running for Java clients. I could provide more details and instructions if you are interested in this approach. Martin Arn wrote: > Hi, > > We've set up GridFTP (4.2.1) on several nodes across our WAN (2 sites) > using the quickstart documentation. > > We are not seeing any issues while transferring large files but when > we do a batch transfer (globus-url-copy) with lots of small files > (LOSF) then we have problems. > The debug/verbose output is the following : > ---------------------------------------------------- > > error transferring: > globus_ftp_client: the server responded with an error > 500 500-Command failed. : globus_l_gfs_file_open failed. > 500-globus_xio: Unable to open file /path/to/data/losf/small0aAEq8QsYSCJ > 500-globus_xio: System error in open: Permission denied > 500-globus_xio: A system call failed: Permission denied > 500 End. > > error: There was an error with one or more transfers. > --------------------------------------------------- > > Note, that this error is intermittent as the same transfer works sometimes. > > We would appreciate some advice info on what could be the problem and > also how to investigate further. > > Thanks > Arn
