> Arn wrote: >> >> We've set up GridFTP (4.2.1) on several nodes across our WAN (2 sites) >> using the quickstart documentation. >> >> We are not seeing any issues while transferring large files but when >> we do a batch transfer (globus-url-copy) with lots of small files >> (LOSF) then we have problems. >> The debug/verbose output is the following : >> ---------------------------------------------------- >> >> error transferring: >> globus_ftp_client: the server responded with an error >> 500 500-Command failed. : globus_l_gfs_file_open failed. >> 500-globus_xio: Unable to open file /path/to/data/losf/small0aAEq8QsYSCJ >> 500-globus_xio: System error in open: Permission denied >> 500-globus_xio: A system call failed: Permission denied >> 500 End. >> >> error: There was an error with one or more transfers. >> --------------------------------------------------- >> >> Note, that this error is intermittent as the same transfer works sometimes. >> >> We would appreciate some advice info on what could be the problem and >> also how to investigate further. >>
On Tue, Feb 23, 2010 at 7:04 AM, Martin Feller <[email protected]> wrote: > There is a way to transfer a directory as a single tar-stream, like this: > > 1. tar up source directory prior to transfer > 2. transfer the tar-stream > 3. untar the archive on the destination > > without manual taring/untaring on the client and the server. > > We implemented this for a community that uses GridFTP heavily for transfers > of 42GB sized directories containing 130.000 rather small files in a nested > directory structure. > It works very reliable this way. > The only downside I know is that you cannot use any of the advanced > features of GridFTP then, like parallelism: The tar-stream transfers > became unreliable. > > To do this you must enable the popen driver in GridFTP. > I recommend the latest server from 5.0.0 plus a GridFTP patch. > > For the taring on the client-side you can use globus-url-copy using certain > flags. We built on top of the jglobus Java API to get it running for Java > clients. > > I could provide more details and instructions if you are interested in this > approach. > Martin thanks. It looks like a reasonable solution. I will check with my project lead if we can use your suggestion, but we do tend to be wary of using non-standard patches in our production environments. Also, we do need to use parallelism but I suppose we can think of a way to turn it on/off depending on the situation. Or maybe we can specify -pp 1 (1 stream) if a LOSF situation is encountered. In any case, do send me the instructions on the method you suggested. Thanks Arn
