Hello Venkateswara, Edward, Thank you for the comments on how to speed up AFM prefetch with small files. We run 4.2.2-3 and the AFM mode is RO and we have just a single gateway, i.e. no parallel reads for large files. We will try to increase the value of afmNumFlushThreads. It wasn’t clear to me that these threads do read from home, too - at least for prefetch. First I will try a plain NFS mount and see how parallel reads of many small files scale the throughput. Next I will try AFM prefetch. I don’t do nice benchmarking, just watching dstat output. We prefetch 100’000 files in one bunch, so there is ample time to observe.
The basic issue is that we get just about 45MB/s for sequential read of many 1000 files with 1MB per file on the home cluster. I.e. we read one file at a time before we switch to the next. This is no surprise. Each read takes about 20ms to complete, so at max we get 50 reads of 1MB per second. We’ve seen this on classical raid storage and on DSS/ESS systems. It’s likely just the physics of spinning disks and the fact that we do one read at a time and don’t allow any parallelism. We wait for one or two I/Os to single disks to complete before we continue With larger files prefetch jumps in and fires many reads in parallel … To get 1’000MB/s I need to do 1’000 read/s and need to have ~20 reads in progress in parallel all the time … we’ll see how close we get to 1’000MB/s with ‘many small files’. Kind regards, Heiner -- Paul Scherrer Institut Science IT Heiner Billich WHGA 106 CH 5232 Villigen PSI 056 310 36 02 https://www.psi.ch _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
