Hi, Chris, I did have an MR https://urldefense.us/v3/__https://gitlab.com/petsc/petsc/-/merge_requests/7651__;!!G_uCfscf7eWS!fWNsAOkuZRiMn1TuiZ0HasNdskk5heIHlt3O4unVNFd3mnPlFFPISeieHQ_DFsrasG1dwtpASUuFiR6eUOugJNvoDVDy$ to improve mpistream. I should rework it after Barry's !6903. See my inlined comments to your questions
On Fri, Oct 17, 2025 at 3:37 AM Klaij, Christiaan via petsc-users < [email protected]> wrote: > Attached is a petsc streams result kindly provided by a hardware > vendor for a single compute node, dual socket, with two AMD epyc > 9355 processors. Each processor has 32 cores, 12 DDR5 memory > channels and mem BW around 600 GB/s. > > * It is not immediately clear which line corresponds to which > y-axis. Could future versions of petsc please color the axis > label with the matching line color? > definitely > > > * Why would the achieved bandwidth be roughly 0.9 x 1e6 MB/s = > 900 GB/s and not closer to 1200 GB/s? > I recall it is actually not simple to get the theoretical max bandwidth. One has to use special SIMD instructions, compiler flags and streaming stores etc. > > > * The speed-up seems to be 12 out of 64, provided multiples of 8 > cores are used. As expected given 12 memory channels? > Maybe not, otherwise the speedup should be 24 as you have 24 channels. > > * Does the zig-zag pattern indicate a pinning problem, or is it > unavoidable given the 8 core building block of these type of > processors? > I checked and found "make mpistream" uses --map-by core. I think we should use --map-by socket or --map-by l3cache. > > Chris > dr. ir. Christiaan Klaij | senior researcher > Research & Development | CFD Development > T +31 317 49 33 44 <+31%20317%2049%2033%2044> | > https://urldefense.us/v3/__http://www.marin.nl__;!!G_uCfscf7eWS!fWNsAOkuZRiMn1TuiZ0HasNdskk5heIHlt3O4unVNFd3mnPlFFPISeieHQ_DFsrasG1dwtpASUuFiR6eUOugJD65Q0mk$ > > <https://urldefense.us/v3/__https://www.marin.nl/__;!!G_uCfscf7eWS!fqSBpN3Ld5fjzXGShGI09uJke12M-5LukEHe-y-gw0Bw9msZeH7wNiId6DZxQpluR_RUWpuoQWUD2HSsBfp_z4A$> > [image: Facebook] > <https://urldefense.us/v3/__https://www.facebook.com/marin.wageningen__;!!G_uCfscf7eWS!fqSBpN3Ld5fjzXGShGI09uJke12M-5LukEHe-y-gw0Bw9msZeH7wNiId6DZxQpluR_RUWpuoQWUD2HSsCH7BGfA$> > [image: LinkedIn] > <https://urldefense.us/v3/__https://www.linkedin.com/company/marin__;!!G_uCfscf7eWS!fqSBpN3Ld5fjzXGShGI09uJke12M-5LukEHe-y-gw0Bw9msZeH7wNiId6DZxQpluR_RUWpuoQWUD2HSsDAV2fAI$> > [image: YouTube] > <https://urldefense.us/v3/__https://www.youtube.com/marinmultimedia__;!!G_uCfscf7eWS!fqSBpN3Ld5fjzXGShGI09uJke12M-5LukEHe-y-gw0Bw9msZeH7wNiId6DZxQpluR_RUWpuoQWUD2HSsEyu_yEs$> >
