Hi,
It turns out that it was a combination of two things - turning off
hyperthreading in BIOS and using a faster disk.
I found a post from the author of lbzip2 which seems to describe what might be
happening in this case, but reference was made to a user using an i5 mobile CPU:
########################################################################
"bzip2 author here. I strongly suspect that you see what you see because your
Intel core i5 is probably only dual core PLUS hyper-threaded, not real quad-core.
Meaning, you have two instances of the L2 per-core cache, not four, and each two
hyperthreads share an L2 cache.
Since the bzip2 compression/decompression is very cache sensitive (see "man
bzip2"), the scaling factor will be determined mostly by how many OS-threads can
dispose over a dedicated cache each. In your case this number is probably 2.
Since you run two threads per core, those contend for the shared L2 cache, basically each messing
with the other (flushing / invalidating the shared cache for the other). This contention shows up
as double CPU time, because "waiting for cache" (or "waiting for main memory")
is accounted for as CPU time.
Hyperthreading is not useful but detrimental for lbzip2; so you should export LBZIP2="-n
2". You should not run more worker threads per core than: core-dedicated-cache-size
divided by 8MB."
########################################################################
Running the compression again on the same file from an SSD with hyperthreading
turned off, I was able to fully saturate all of the cores using lbzip2. None of
this seemed obvious at first, but it rectified the situation. The biggest
difference came from turning off hyperthreading (idle CPU - 20% vs the previous
90%) and then running from an SSD with hyperthreading turned off (idle CPU =
0%).
Previously, the compression was run from a single HDD, not an SSD. Concerning
the compression test using the same HDD under FreeBSD, well I don't know why it
was able to saturate the CPU. Perhaps it has something to do with ZFS's
aggressive caching. Turning that off and re-running the test would likely
answer the question. Pixz performed similarly when the above two modifications
were made.
On 12/11/2016 02:00 AM, Jasse Jansson wrote:
Have you tried to disable hypertreads in the BIOS ???
It's a long shot, I know, but it might help.
On 2016-12-10 22:14, PeerCorps Trust Fund wrote:
Hi,
On both systems HAMMER was used. One small correction concerning the 2c/2t
machine, both compression programs did effectively utilize that CPU which had
an idle % of 0.0. It is the bigger machine, 16c/32t where the CPU isn't
effectively maxed out. I'll continue to try and investigate why and report back
if I find anything.
On 12/10/2016 10:26 PM, Justin Sherrill wrote:
On the two DragonFly systems, was it Hammer or UFS? I would be
surprised if that made a difference, but it might?
On Sat, Dec 10, 2016 at 6:19 AM, PeerCorps Trust Fund
<[email protected]> wrote:
Hi,
I've observed that parallel compression tools such as pixz and lbzip2 do not
make use of all of the available CPU under Dragonfly. On other OSes, it
does.
When testing on a 50 gb file, using top I've observed that CPU idle
percentages consistently hover around the 90% range for pixz and ~70% for
lbzip2. These values under FreeBSD and Linux are typically ~0.0% idle until
compression is complete. Correspondingly, compression takes significantly
longer under Dragonfly, so the CPU is really being under utilized in this
case as opposed to erroneous reporting by top.
This was tested on two systems, one 16c/32t and a 2c/2t system on a recent
master DragonFly v4.7.0.973.g8d7da-DEVELOPMENT #2: Wed Dec 7 11:44:04 EET
2016.
Has anyone else possibly observed this?
--
Mike
