Could you talk about upcoming work to address excessive prefetch when reading small fractions of many large files? Some bioinformatics workloads have a client node reading relatively small regions of multiple 50GB+ files. We've seen this trigger excessive prefetch bandwidth (especially on 16MB block filesystem). Investigation shows that much of the prefetched data is never read, but cache gets full, evicts blocks, then more prefetch happens. We can avoid this by turning prefetch off, but that reduces speed of other workloads that read full files sequentially. Turning prefetch on and off based on job won't work well for our users.
We've heard this would be addressed in gpfs 5.1 at the earliest and have provided an example workload to devs. They've done some great analysis and determined the problem is worse on large (16M) block filesystems (which are now the recommended and default on new ess filesystems with sub-block allocation enabled). Best, Chris On 10/29/20, 5:49 PM, "[email protected] on behalf of Kristy Kallback-Rose" <[email protected] on behalf of [email protected]> wrote: Hi all, The Spectrum Scale User Group will be hosting two 90 minute sessions at SC20 this year and we hope you can join us. The first one is: "Storage for AI" and will be held Monday, Nov. 16th, from 11:00-12:30 EST and the second one is "What's new in Spectrum Scale 5.1?" and will be held Wednesday, Nov. 18th from 11:00-12:30 EST. Please see the calendar at https://urldefense.com/v3/__https://www.spectrumscaleug.org/eventslist/2020-11/__;!!C6sPl7C9qQ!G0wT65UH3HoMnjBM6_ZAVfZwWwJz5SoLE5gpB_LM0N8SNSU3TXItF31dfxG_8Pow$ and register by clicking on a session on the calendar and then the "Please register here to join the session" link. Best, Kristy Kristy Kallback-Rose Senior HPC Storage Systems Analyst National Energy Research Scientific Computing Center Lawrence Berkeley National Laboratory _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org https://urldefense.com/v3/__http://gpfsug.org/mailman/listinfo/gpfsug-discuss__;!!C6sPl7C9qQ!G0wT65UH3HoMnjBM6_ZAVfZwWwJz5SoLE5gpB_LM0N8SNSU3TXItF31df0lybvoA$ ________________________________ This message is for the recipient’s use only, and may contain confidential, privileged or protected information. Any unauthorized use or dissemination of this communication is prohibited. If you received this message in error, please immediately notify the sender and destroy all copies of this message. The recipient should check this email and any attachments for the presence of viruses, as we accept no liability for any damage caused by any virus transmitted by this email. _______________________________________________ gpfsug-discuss mailing list gpfsug-discuss at spectrumscale.org http://gpfsug.org/mailman/listinfo/gpfsug-discuss
