Hi Aaron, to how many different files do these tiny I/O requests go? Mind that the write aggregates the I/O over a limited time (5 secs or so) and ***per file***. It is for that matter a large difference to write small chunks all to one file or to a large number of individual files . to fill a 1 MiB buffer you need about 13100 chunks of 80Bytes ***per file*** within those 5 secs.
Mit freundlichen Grüßen / Kind regards Dr. Uwe Falke IT Specialist High Performance Computing Services / Integrated Technology Services / Data Center Services ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Rathausstr. 7 09111 Chemnitz Phone: +49 371 6978 2165 Mobile: +49 175 575 2877 E-Mail: [email protected] ------------------------------------------------------------------------------------------------------------------------------------------- IBM Deutschland Business & Technology Services GmbH / Geschäftsführung: Thomas Wolter, Sven Schooß Sitz der Gesellschaft: Ehningen / Registergericht: Amtsgericht Stuttgart, HRB 17122 From: "Knister, Aaron S. (GSFC-606.2)[COMPUTER SCIENCE CORP]" <[email protected]> To: gpfsug main discussion list <[email protected]> Date: 10/04/2018 18:09 Subject: [gpfsug-discuss] Confusing I/O Behavior Sent by: [email protected] I hate admitting this but I?ve found something that?s got me stumped. We have a user running an MPI job on the system. Each rank opens up several output files to which it writes ASCII debug information. The net result across several hundred ranks is an absolute smattering of teeny tiny I/o requests to te underlying disks which they don?t appreciate. Performance plummets. The I/o requests are 30 to 80 bytes in size. What I don?t understand is why these write requests aren?t getting batched up into larger write requests to the underlying disks. If I do something like ?df if=/dev/zero of=foo bs=8k? on a node I see that the nasty unaligned 8k io requests are batched up into nice 1M I/o requests before they hit the NSD. As best I can tell the application isn?t doing any fsync?s and isn?t doing direct io to these files. Can anyone explain why seemingly very similar io workloads appear to result in well formed NSD I/O in one case and awful I/o in another? Thanks! -Stumped _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
