Backing up what everyone else says. DO you know how the 24 core jobs are being distributed between sockets and cores.
Also as a very naive diagnostic, just run 'top' or even better 'htop' while the jobs are running. I KNOW this isnt a great diagnostic for parallel programs, but sometimes you can notice patterns. Select the option in top to show you the cores which are running. Also keep an eye on the memory assignments at the top - you are not going into swap? Are any processes going into D sttes for extended amounts of time? ________________________________________ From: Beowulf [[email protected]] on behalf of Peter St. John [[email protected]] Sent: 29 February 2016 00:50 To: Jon Tegner Cc: Beowulf List Subject: Re: [Beowulf] Scaling issues on Xeon E5-2680 I come from the math side, not the electronics side, so this may be an ill-posed question, but could you try running the job with 12 cores on just one node, then with 6 cores on each of two nodes? I'm thinking, the 24 core version may get assigned to more nodes than your 12 core, and it's communication between nodes that slows it down, e.g. for a job that doesn't require so much memory as inter-process communication. Peter On Sun, Feb 28, 2016 at 11:01 AM, Jon Tegner <[email protected]<mailto:[email protected]>> wrote: Thanks! we are using openmpi-1.10.2, and I believe bind-to core and socket are on by default here. Openmpi is built using ./configure \ --build=x86_64-redhat-linux-gnu \ --host=x86_64-redhat-linux-gnu \ --disable-dependency-tracking \ --prefix=/remote/soft/OpenMPI/1.10.2 \ --disable-mpi-profile \ --enable-shared \ --with-tm \ --with-sge \ --with-verbs gcc is used throughout. Code should be using RDMA. I have verified that infiniband performance (using openmpi-1.10.2) is what is to be expected (through mpitests-osu_bw and mpitest_osu_latency). Hugepagesize haven't been touched ("grep Hugepagesize /proc/meminfo" gives 2048 kB). Is this something worth changing? numastat gives node0 node1 numa_hit 6288694 1193663 numa_miss 0 0 numa_foreign 0 0 interleave_hit 66539 66287 local_node 6288301 1126078 other_node 393 67585 on one of the nodes - but these numbers seems to be fairly representative for the nodes. However, haven't used numastat before, and maybe it should be used simultaneously as the job is running (which it wasn't now)? Thanks! /jon On 02/28/2016 04:30 PM, [email protected]<mailto:[email protected]> wrote: Do you have CPU affinity enabled? Is this omp + MPI? Which compilers and flags. Give more details about the software stack Original Message From: Jon Tegner Sent: Sunday, February 28, 2016 22:28 To: [email protected]<mailto:[email protected]> Subject: [Beowulf] Scaling issues on Xeon E5-2680 Hi, have issues with performance on E5-2680. Each of the nodes have 2 of these 12 core CPUs on SuperMicro SuperServer 1028R-WMR (i.e., 24 cores on each node). For one of our applications (CFD/OpenFOAM) we have noticed that the calculation runs faster using 12 cores on 4 nodes compared to when using 24 cores on 4 nodes. In our environment we also have older AMD hardware (nodes with 4 CPUs with 12 cores each), and here we don't see these strange scaling issues. System is CentOS-7, and communication is over FDR Infiniband. BIOS is recently updated, and hyperthreading is disabled. Feel a bit lost here, and any hints on how to proceed with this are greatly appreciated! Thanks, /jon _______________________________________________ Beowulf mailing list, [email protected]<mailto:[email protected]> sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf _______________________________________________ Beowulf mailing list, [email protected]<mailto:[email protected]> sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf ##################################################################################### Scanned by MailMarshal - M86 Security's comprehensive email content security solution. ##################################################################################### Any views or opinions presented in this email are solely those of the author and do not necessarily represent those of the company. Employees of XMA Ltd are expressly required not to make defamatory statements and not to infringe or authorise any infringement of copyright or any other legal right by email communications. Any such communication is contrary to company policy and outside the scope of the employment of the individual concerned. The company will not accept any liability in respect of such communication, and the employee responsible will be personally liable for any damages or other liability arising. XMA Limited is registered in England and Wales (registered no. 2051703). Registered Office: Wilford Industrial Estate, Ruddington Lane, Wilford, Nottingham, NG11 7EP _______________________________________________ Beowulf mailing list, [email protected] sponsored by Penguin Computing To change your subscription (digest mode or unsubscribe) visit http://www.beowulf.org/mailman/listinfo/beowulf
