On 04/27/2012 06:47 PM, Pádraig Brady wrote: > On 04/27/2012 06:15 PM, Bruce Korb wrote: >> Thank you for the suggestions. I did finish my particular task with rsync, >> but I'd still like something that finishes in less than 10 hours. > > Just to be clear, you started with cp, went home, resumed with rsync. > So rsync is not faster here, as there is no remote rsync server. > >> It is longer >> than my work day. Please note: my only access is with a CIFS mount. >> *NO* shell access. Thus, anything depending upon remote commands >> doesn't work. This is not the only Windows-centric shop I've seen. > > Sure, I was just comparing all options. > >> 2012/4/27 Pádraig Brady <[email protected]>: >>>> I can either add this to cp, or roll my own toy. What say you? >>> >>> Yes, cp is bad for high latency links as it has a >>> 32KiB buffer which it serially reads to and writes from. >> >> The throughput is going to be dependent upon how much data CIFS >> sends without an ACK. e.g. 1500 bytes would mean 20 round trips >> to fill the 30KB buffer. > > If it is 1 large file you could optimize a bit like: > > dd bs=10MB if=/cifs/file of=/local/file > > However as you say, if the cifs protocol is not > efficiently streaming over the high latency link > then there's not much we can do. > > Note you could test your multiple readers idea simply like: > > ( > dd bs=32KiB skip=0000 seek=0000 count=1000 conv=notrunc & > dd bs=32KiB skip=1000 seek=1000 count=1000 conv=notrunc & > dd bs=32KiB skip=2000 seek=2000 count=1000 conv=notrunc > ) > /local/file < /cifs/file > > A simple wrapper could be used to start the correct number of dd processes. >
Actually each dd process would have to maintain the file offsets independently something like: > file.out ( dd if=file.in of=file.out bs=32KiB skip=000 seek=000 count=100 conv=notrunc & dd if=file.in of=file.out bs=32KiB skip=100 seek=100 count=100 conv=notrunc & dd if=file.in of=file.out bs=32KiB skip=200 seek=200 conv=notrunc wait ) Could you link to pcopy for reference? cheers, Pádraig.
