Could the performance increase when downloading artefacts parallel? Jan
>-----Ursprüngliche Nachricht----- >Von: Xavier Hanin [mailto:[EMAIL PROTECTED] >Gesendet: Freitag, 22. Juni 2007 11:28 >An: [email protected] >Betreff: Re: ivy:retrieve performance > >On 6/21/07, testn <[EMAIL PROTECTED]> wrote: >> >> >> Hi Xavier, > > >Hi, but this question is not only for me, Ivy is a community >project where >the community takes decisions. > >Any plan to put it in alpha2? It doesn't look like it requires >a lot of work >> if we just want to change the buffer size. nio may be a long shot. > > >I'm +1 to increase the buffer to 64kB. I'm +1 to open an issue >for this, and >another one to later further improve this with NIO. What do >others think? > >Xavier > >Thanks >> >> >> Xavier Hanin wrote: >> > >> > Great, thanks a lot for these data! >> > >> > Xavier >> > >> > On 6/14/07, Buck, Robert <[EMAIL PROTECTED]> wrote: >> >> >> >> What the performance test did is: >> >> >> >> 1. increase by powers of 2 a file size variable >> >> >> >> 2. for each file size, create/open a new file >> >> >> >> 3. write specified number of bytes to file >> >> >> >> 4. close file >> >> >> >> Tests were performed on two systems: >> >> >> >> 1. IBM ThinkPad T42 (slow) laptop, stock hardware, 1.5GB RAM >> >> >> >> 2. Dell Dual CPU, 5150, 64-bit Red Hat 4; (2) SAS 15k >146GB disks in >> >> RAID 1 configuration; 16 GB RAM; this is production grade >hardware. >> >> >> >> Both sytems produced similar profiles (shape), however, on the >> >> production hardware with Linux (ext2, I believe) the numbers were >> >> substantially greater (makes sense when you consider the >fast disks I >> >> have; at 3GB/sec rates, these SAS drives scream). >> >> >> >> In both cases buffers were configured for both Old IO and >New IO. With >> >> NIO direct buffers were used. In both cases, I determined >the ideal >> >> buffer sizes based upon experimentation to be around 32kb - 64 kb. >> Going >> >> to buffer sizes less than, or greater than, this range >reduced overall >> >> throughput. >> >> >> >> -Bob >> >> >> >> > -----Original Message----- >> >> > From: Xavier Hanin [mailto:[EMAIL PROTECTED] >> >> > Sent: Thursday, June 14, 2007 8:38 AM >> >> > To: [email protected] >> >> > Subject: Re: ivy:retrieve performance >> >> > >> >> > On 6/14/07, Buck, Robert <[EMAIL PROTECTED]> wrote: >> >> > > >> >> > > Folks, >> >> > > >> >> > > We benchmarked a number of JDK IO API's for an >internal project. To >> >> > > neutralize any questions regarding NIO vs Old IO, please >> >> > take a look >> >> > > at the attached diagram. These rates will be largely >> >> > identical on both >> >> > > Linux and Windows. Blue line NIO, red line old io. >> >> > >> >> > >> >> > I've uploaded the image here: >> >> > http://www.flickr.com/photos/[EMAIL PROTECTED]/548163404/ >> >> > >> >> > Could you give a little bit more details about how you did >> >> > the tests: jvm used, size of buffer used for old IO, ... >> >> > >> >> > According to your tests it seems that NIO should be preferred >> >> > in any case, it wasn't what some users in the javalobby >> >> > thread seemed to say. So I wonder what makes the difference. >> >> > >> >> > Xavier >> >> > >> >> > If you do not get the attached JPEG file, let me know. >I can send it >> >> > > directly to you if you so request. >> >> > > >> >> > > Cheers, >> >> > > >> >> > > Bob >> >> > > >> >> > > > -----Original Message----- >> >> > > > From: Xavier Hanin [mailto:[EMAIL PROTECTED] >> >> > > > Sent: Thursday, June 14, 2007 6:22 AM >> >> > > > To: [email protected] >> >> > > > Subject: Re: ivy:retrieve performance >> >> > > > >> >> > > > On 6/14/07, Gilles Scokart <[EMAIL PROTECTED]> wrote: >> >> > > > > >> >> > > > > > 2. The buffer in FileUtils.java is too small. >It's set at >> >> > > > 8192. It >> >> > > > > > seems to >> >> > > > > > > be much better for me to set it much larger. >This is due to >> >> > > > > > > the fact >> >> > > > > > that >> >> > > > > > > it >> >> > > > > > > needs to read and write simultaneously. The bigger the >> >> > > > buffer is, >> >> > > > > > > the smaller number of time, HD header has to move. For >> >> > > > me, 65536 >> >> > > > > > > seems to perform much better but I haven't tried >> >> > other numbers. >> >> > > > > > >> >> > > > > > >> >> > > > > > I'd like to get more feedback on this. One use case is >> >> > > > not the other. >> >> > > > > This >> >> > > > > > size has been borrowed from Ant copy mechanism. Maybe >> >> > > > what we could >> >> > > > > > do >> >> > > > > is >> >> > > > > > make this configurable, so that one could adapt to its >> >> > > > needs. Or try >> >> > > > > > to guess a good size depending on the size (when it's >> >> > possible >> >> > > > > > to get an >> >> > > > > idea >> >> > > > > > of the size before copying). >> >> > > > > > >> >> > > > > > Xavier >> >> > > > > > >> >> > > > > >> >> > > > > >> >> > > > > Couldn't we use the nio for that? (See >> >> > > > > http://www.javalobby.org/java/forums/t17036.html) >> >> > > > >> >> > > > >> >> > > > According to comments 10 and 11 NIO have bad performance >> >> > for large >> >> > > > files on linux, and input stream with byte buffer is >> >> > pretty close to >> >> > > > NIO for small files (see comment 13 conclusion). So >I'm not sure >> >> > > > switching to NIO would indeed help a lot. According to >> >> > the tests in >> >> > > > the thread you pointed using a 64kB buffer seems to be a >> >> > good choice >> >> > > > (which confirms testn tests), at least for large >files. OTOH the >> >> > > > last conclusion (comment 17) is different. >> >> > > > So I don't really know what to think about that. We >> >> > should make some >> >> > > > tests on several platforms and jvms to draw conclusion >> >> > ourself, but >> >> > > > it takes time. >> >> > > > >> >> > > > >> >> > > > Xavier >> >> > > > >> >> > > > Gilles >> >> > > > > >> >> > > > > >> >> > > > > >> >> > > > > >> >> > > > >> >> > > > >> >> > > > -- >> >> > > > Xavier Hanin - Independent Java Consultant Manage your >> >> > dependencies >> >> > > > with Ivy! >> >> > > > http://incubator.apache.org/ivy/ >> >> > > > >> >> > > >> >> > >> >> > >> >> > >> >> > -- >> >> > Xavier Hanin - Independent Java Consultant Manage your >> >> > dependencies with Ivy! >> >> > http://incubator.apache.org/ivy/ >> >> > >> >> >> > >> > >> > >> > -- >> > Xavier Hanin - Independent Java Consultant >> > Manage your dependencies with Ivy! >> > http://incubator.apache.org/ivy/ >> > >> > >> >> -- >> View this message in context: >> >http://www.nabble.com/ivy%3Aretrieve-performance-tf3907253.html >#a11230918 >> Sent from the ivy-user mailing list archive at Nabble.com. >> >> > > >-- >Xavier Hanin - Independent Java Consultant >Manage your dependencies with Ivy! >http://incubator.apache.org/ivy/ >
