Agree with Gil (w.r.t DIMM slots, channels etc) but may be not on the 'dd' script. Here's why. Some BIOS's have 'memory interleaving' option turned OFF. And if the OS-wide memory policy is non-interleaving too then case unless your application explicitly binds memory to a remote socket you cannot interleave memory. Or you would need to use numa tools (to set mem policy etc) while launching your application.
Bandwidth or latency monitoring is also dependent on the workload you are running. If the workload a) runs atomics b) is running on Node-0 only but memory across both : then the snoop-responses are going take longer because socket-1 might be in power-saving states (uncore frequency etc). So ideally you would need a 'snoozer' thread on the remote socket(Node-1) which would prevent the socket from entering one of the 'C' (or whatever) states (or you can disable hardware power-saving modes - but you may need to line up all the options because the kernel may have power-saving options too). If you use the industry standard tools like 'stream' (as others mentioned) etc they will do all of this for you (via dummy/snoozer threads and so on). If you want to write this all by yourself then you should know the numa apis (http://man7.org/linux/man-pages/man3/numa.3.html), numa tools (numactl, taskset). For latency measurements you should also disable pre-fetching else it will give you super awesome numbers. Note: In future, if you use a device-driver that does the allocation for your app then you should make sure the driver knows about numa allocation too and aligns everything for you. I found that out the hard way back in 2010 and realized that the linux-kernel had no guidance for pcie drivers(back then ... its ok now). I fixed it locally at the time. Hope this helps. Chetan Loke On Monday, January 15, 2018 at 11:19:53 AM UTC-5, Kevin Bowling wrote: > > lmbench works well http://www.bitmover.com/lmbench/man_lmbench.html, > and Larry seems happy to answer questions on building/using it. > > Unless you've explicitly built an application to work with NUMA, or > are able to run two copies of an application pinned to each domain, > you really only will get about 1 package worth of BW, and latecny is a > bigger deal (which lmbench can also measure in cooperation with > numactl) > > Regards, > > On Sun, Jan 14, 2018 at 11:44 AM, Peter Veentjer <[email protected] > <javascript:>> wrote: > > I'm working on some very simple aggregations on huge chunks of offheap > > memory (500GB+) for a hackaton. This is done using a very simple stride; > > every iteration the address increases with 20 bytes. So the prefetcher > > should not have any problems with it. > > > > According to my calculations I'm currently processing 35 GB/s. However > I'm > > not sure if I'm close to the maximum bandwidth of this machine. Specs: > > 2133 MHz, 24x HP 32GiB 4Rx4 PC4-2133P > > 2x Intel(R) Xeon(R) CPU E5-2687W v3, 3.10GHz, 10 cores per socket > > > > What is the best tool to determine the maximum bandwidth of a machine > > running Linux (RHEL 7) > > > > -- > > You received this message because you are subscribed to the Google > Groups > > "mechanical-sympathy" group. > > To unsubscribe from this group and stop receiving emails from it, send > an > > email to [email protected] <javascript:>. > > > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "mechanical-sympathy" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
