Raj, I am trying to use guc to transfer files to an server that uses the VFS layer to mount an HPSS system. The VFS layer provides checksumming for data stored in HPSS. If writes are done in in order, then the checksum will be calculated on the fly as data come in. If the data are out of order, the checksumming stops and will be recalculated when the file is closed. While there might be some write-caching going on in the disk controllers, this is still inefficient.
I can't seem to get any decent sustained performance without using threads. While I am not expecting a single thread to write at 1 GB/s, a couple hundred MB/s per large file would be nice. On my 10Gbe link, I can't get the sustained performance I would like without multiple threads. Craig On Mon, Sep 24, 2012 at 10:09 AM, Raj Kettimuthu <[email protected]> wrote: > Hi Craig, > Do you need in order writing with parallel streams? HDFS DSI for GridFTP > writes blocks in order even with parallel streams. Can you provide details on > the context? > > Thanks, > Raj > > On Sep 24, 2012, at 10:53 AM, Raj Kettimuthu wrote: > >> Hi Craig, >> The blocks will be written in the order received (could be and most likely >> out of order to disk). >> >> Raj >> >> On Sep 24, 2012, at 10:32 AM, Craig Tierney wrote: >> >>> When I write files using parallel threads (-p), does the >>> globus-gridftp-server guarantee that the blocks are still written in >>> order, or are the written in the order received (not in order on >>> disk)? >>> >>> Thanks, >>> Craig >> >
