I know this is an old thread, but I wanted to relate the final scheme I came up
with for optimal uploading of DVD files on windows,
in case anyone else has to deal with this issue:
1) issue Powershell command from Java to recursively list all files in a root
directory, and copy each file to dev null, as follows:
String cmd = "powershell.exe \"Get-Childitem -Recurse " +
rootDirectory +
" | where {!$_.PsIsContainer} | %{ Get-Content
$_.FullName > $null; echo $_.FullName;} \" ";
2) Notice the echo command at the end; now that the file is in the windows file
cache, I can read this echoed name in java and issue an asynchronous zero copy
put on the file via
httpasyncclient (this is now very fast, since the file is in the cache)
So, this method pipelines the reads and puts, so that put time can be ignored,
and time to upload is just the time to read the files from DVD.
Cheers,
Aaron
-----Original Message-----
From: Boxer, Aaron
Sent: Friday, January 17, 2014 3:26 PM
To: HttpClient User Discussion
Subject: RE: ZeroCopyPut mystery
On Fri, 2014-01-17 at 16:28 +0000, Boxer, Aaron wrote:
> I made a simple test: I wrote a few lines of Java using File channel and
> transferTo(...) method to read files from my DVD and write to the hard drive.
> With this simple test, the transferTo(...) method gets about 4 MB /S
> transfer rate to disk.
>
> Then, I put some timing code into the LengthDelimitedEncoder.transfer(...)
> method, and ran my application, which copies files from DVD
> to a socket. And the transferTo(...) from file to socket was getting about
> 0.7 MB /s . This is a 6 X degradation in performance.
> And, if you recall, if the files are already in the OS file cache, I get
> about 100 MB/S, so the socket is not slowing me down.
>
I wanted to share the solution I came up with that is pretty optimal, in my
opinion:
1) sequentially read files from DVD, from FileChannel into Direct ByteBuffer -
once read, these files are now in the Windows OS file cache
2) after every 50 files read (or certain amount of MB), I send these files out
over the asynch client( good transfer rate; asynchronous execution)
I do this until all files are read.
The total cost to me is just the cost of reading the DVD, which I can't really
avoid. So, this is pretty optimal.
This e-mail may contain confidential and/or privileged information for the sole
use of the intended recipient.
Any review or distribution by anyone other than the person for whom it was
originally intended is strictly prohibited.
If you have received this e-mail in error, please contact the sender and delete
all copies.
Opinions, conclusions or other information contained in this e-mail may not be
that of the organization.
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
This e-mail may contain confidential and/or privileged information for the sole
use of the intended recipient.
Any review or distribution by anyone other than the person for whom it was
originally intended is strictly prohibited.
If you have received this e-mail in error, please contact the sender and delete
all copies.
Opinions, conclusions or other information contained in this e-mail may not be
that of the organization.
