On 2021-03-05 15:40, Eric Kuhnke wrote:
For comparison purposes, I'm curious about the difference in wattage
a) Your R640 at 420W running DPDK
b) The same R640 hardware temporarily booted from a Ubuntu server live
USB, in which some common CPU stress and memory disk/IO benchmarks are
being run to intentionally load the system to 100% to characterize its
absolute maximum AC load wattage.
We've got a few more hosts waiting to be deployed that are configured
almost identically. I'll see what we can do.
I'm guessing those tests would pull slightly more power than the vEdge
hosts, just because there's not much disk IO that happens on a
networking VM. These hosts have four SSDs for local storage.
What's the delta between the 420W and absolute maximum load the server
is capable of pulling on the 208VAC side?
Server PS maximum input wattage is 900W. Present draw of 2.0A @ 208V is
~420W, so 420/900 = 46.67%
One possible factor is whether ESXI is configured to pass the pci-e
devices directly through to the guest VM, or if there is any
abstraction in between. For non-ESXI stuff, in the world of Xen or KVM
there's many different ways that a guest domU can access a dom0's
network devices, some of which can have impact on overall steady-state
wattage consumed by the system.
The 420W server has its interfaces routed through the ESXI kernel.
We're moving quickly to SR-IOV on new servers.
If the greatest possible efficiency is desired for a number of 1U
things, one thing to look at would be something similar to the open
compute platform single centralized AC to DC power units, and servers
that don't each have their own discrete 110-240VAC single or dual power
supplies. In terms of cubic meters of air moved per hour vs wattage,
the fans found in 1U servers are really quite inefficient. As a
randomly chosen example of 12VDC 40mm (1U server height) fan:
If you have a single 12.0VDC fan that's a maximum load of 1.52A, that's
a possible load of up to 18.24W for just *one* 40mm height fan. And
your typical high speed dual socket 1U server may have up to eight or
ten of those, in the typical front to back wind tunnel configuration.
Normally fans won't be running at full speed, so each one won't be a
18W load, but more like 10-12W per fan is totally normal. Plus two at
least two more fans in both hot swap power supplies. Under heavy load I
would not be surprised at all to say that 80W to 90W of your R640's
total 420W load is ventilation.
Which is of course dependent on the environmentals. Fan speeds on our
two servers are 25% for the 260W vs. 29% for 420W, so not much
difference. Inlet temp on both is ~17C.
I checked out another R640 heavily loaded with vEdge VMs, and it's
pulling similar power, 415W, but the fan speed is at 45%, because inlet
temp is 22C.
The TDP for the Xeon 6152 is 140W, which seems middle-of-the-road. From
the quick survey I did of Dell's configurator, the R640 can take CPUs up
to 205W. So we have headroom in terms of cooling.
In a situation where you're running out of power before you run out of
rack space, look at some 1.5U and 2U high chassist that use 60mm height
fans, which are much more efficient in ratio of air moved per time
period vs watts.
Or ask the colo to turn the A/C lower ;) (that moves the power problem
elsewhere, I know)