On 08/09/2016 01:41 PM, Michael Mol wrote:
On Tuesday, August 09, 2016 01:23:57 PM james wrote:
On 08/09/2016 09:17 AM, Michael Mol wrote:
On Tuesday, August 09, 2016 09:13:31 AM james wrote:
On 08/09/2016 07:42 AM, Michael Mol wrote:
On Monday, August 08, 2016 10:45:09 PM Alan McKinnon wrote:
On 08/08/2016 19:20, Michael Mol wrote:
On Monday, August 08, 2016 06:52:15 PM Alan McKinnon wrote:
On 08/08/2016 17:02, Michael Mol wrote:
I use Zabbix extensively at work, and have the Zabbix agent on my
workstation reporting back various supported metrics. There's a great
deal you can use (and--my favorite--abuse) Zabbix for, especially once
you understand how it thinks.
Congradualtions! Of the net-analyzer crowd, you've manage to find one I
have not spent time with........
Oh, man, are you in for a treat. I recently had a conversation with a guy I
happened to sit next to while traveling about how, were I in his position, I'd
improve his cash crop and hydroponics operations (he periodically tests soil
and sunlight properties) continually using a combination of cheap, custom
probes and SBCs, feeding the data into Zabbix for monitoring and trend
analysis / prediction. Zabbix will do time-series graphing and analysis of
arbitrary input data; it may have been designed for watching interface
counters, but there's no reason it need be limited to that...
Not sure of your tendencies, but yea, I tend to be more hardware and EE
oriented, than CS. Yep, I spent too many years with time-sequenced data
(turds) to not be totally excited about what we can now do with
clusters, analog (16 bit+) IO and enough processors and memory to keep
a simulation going and in RT(color). You sure know how to instigate an
itch.....
Besides, as I transcend retirement, I'm looking for greener pastures
and methodologies to enhance da(tm) dream state ......
(thx)
Any specific kernel tweaks?
Most of my tweaks for KDE revolved around tuning mysqld itself. But for
sysctls improving workstation responsiveness as it relates to memory
interactions with I/O, these are my go-tos:
vm.dirty_background_bytes = 1048576
vm.dirty_bytes = 10485760
vm.swappiness = 0
Mine are::
cat dirty_bytes
0
cat dirty_background_bytes
0
So, that means you have vm.dirty_bytes_ratio and vm.dirty_background_ratio
set, instead. I forget what those default to, but I think
dirty_bacgkround_ratio defaults to something like 10, which means *10%* of
your memory may get used for buffering disk I/O before it starts writing data
to disk. dirty_bytes_ratio will necessarily be higher, which means that if
you're performing seriously write-intensive activities on a system with 32GiB
of RAM, you may find yourself with a system that will halt until it finishes
flushing 3+GiB of data to disk.
cat swappiness
60
Yeah, you want that set to lower than that.
vm.dirty_background_bytes ensures that any data (i.e. from mmap or
fwrite, not from swapping) waiting to be written to disk *starts*
getting written to disk once you've got at least the configured amount
(1MB) of data waiting. (If you've got a disk controller with
battery-backed or flash-backed write cache, you might consider
increasing this to some significant fraction of your write cache. I.e.
if you've got a 1GB FBWC with 768MB of that dedicated to write cache,
you might set this to 512MB or so. Depending on your workload. I/O
tuning is for those of us who enjoy the dark arts.)
vm.dirty_bytes says that once you've got the configured amount (10MB) of
data waiting to be disk, then no more asynchronous I/O is permitted
until you have no more data waiting; all outstanding writes must be
finished first. (My rule of thumb is to have this between 2-10 times the
value of vm.dirty_background_bytes. Though I'm really trying to avoid it
being high enough that it could take more than 50ms to transfer to disk;
that way, any stalls that do happen are almost imperceptible.)
You want vm.dirty_background_bytes to be high enough that your hardware
doesn't spend its time powered on if it doesn't have to be, and so that
your hardware can transfer data in large, efficient, streamable chunks.
You want vm.dirty_bytes enough higher than your first number so that
your hardware has enough time to spin up and transfer data before you
put the hammer down and say, "all right, nobody else gets to queue
writes until all the waiting data has reached disk."
You want vm.dirty_bytes *low* enough that when you *do* have to put that
hammer down, it doesn't interfere with your perceptions of a responsive
system. (And in a server context, you want it low enough that things
can't time out--or be pushed into timing out--waiting for it. Call your
user attention a matter of timing out expecting things to respond to
you, and the same principle applies...)
Now, vm.swappiness? That's a weighting factor for how quickly the kernel
should try moving memory to swap to be able to speedily respond to new
allocations. Me, I prefer the kernel to not preemptively move
lesser-used data to swap, because that's going to be a few hundred
megabytes worth of data all associated with one application, and it'll
be a real drag when I switch back to the application I haven't used for
half an hour. So I set vm.swappiness to 0, to tell the kernel to only
move data to swap if it has no other alternative while trying to satisfy
a new memory allocation request.
OK, OK, OK. I need to read a bit about these. Any references or docs or
is the result of parsing out what is the least painful for a
workstation? I do not run any heavy databases on my workstation; they
are only there to hack on them. I test db centric stuff on domain
servers, sometimes with limited resources. I run lxde and I'm moving to
lxqt for workstations and humanoid (terminal) IO.
https://www.kernel.org/doc/Documentation/sysctl/vm.txt
https://lonesysadmin.net/2013/12/22/better-linux-disk-caching-performance-vm-dirty_ratio/
Excellent docs, thx.
Do you set these differently for servers?
On my servers, I keep these values similar, because I'd rather have a little
bit lower throughput than risk a catastrophic cascade failure stemming from an
I/O stall.
Nodes in a cluster?
Same story.
The exception is my storage cluster, which has dirty_bytes much higher, as
it's very solidly battery backed, so I can use its oodles of memory as a write
cache, giving its kernel time to reorder writes and flush data to disk
efficiently, and letting clients very rapidly return from write requests.
Are these TSdB (time series data) by chance?
OK, so have your systematically experimented with these parameter
settings, collected and correlated the data, domain (needs) specific ?
As unikernels collide with my work on building up minimized and
optimized linux clusters, my pathway forward is to use several small
clusters, where the codes/frameworks can be changed, even the
tweaked-tuned kernels and DFS and note the performance differences for
very specific domain solutions. My examples are quite similar to that
aforementioned flight sim above, but the ordinary and uncommon
workloads of regular admin (dev/ops) work is only a different domain.
Ideas on automating the exploration of these settings
(scripts/traces/keystores) are keenly of interest to me, just so you know.
I use OpenRC, just so you know. I also have a motherboard with IOMMU
that is currently has questionable settings in the kernel config file. I
cannot find consensus if/how IOMMU that affects IO with the Sata HD
devices versus mm mapped peripherals.... in the context of 4.x kernel
options. I'm trying very hard here to avoid a deep dive on these issues,
so trendy strategies are most welcome, as workstation and cluster node
optimizations are all I'm really working on atm.
Honestly, I'd suggest you deep dive. An image once, with clarity, will last
you a lot longer than ongoing fuzzy and trendy images from people whose
hardware and workflow is likely to be different from yours.
The settings I provided should be absolutely fine for most use cases. Only
exception would be mobile devices with spinning rust, but those are getting
rarer and rarer...
I did a quick test with games-arcade/xgalaga. It's an old, quirky game
with sporadic lag variations. On a workstation with 32G ram and (8) 4GHz
64bit cores, very lightly loaded, there is no reason for in game lag.
Your previous settings made it much better and quicker the vast majority
of the time; but not optimal (always responsive). Experiences tell me if
I can tweak a system so that that game stays responsive whilst the
application(s) mix is concurrently running then the quick
test+parameter settings is reasonably well behaved. So thats becomes a
baseline for further automated tests and fine tuning for a system under
study.
Perhaps Zabbix +TSdB can get me further down the pathway. Time
sequenced and analyzed data is over kill for this (xgalaga) test, but
those coalesced test-vectors will be most useful for me as I seek a
gentoo centric pathway for low latency clusters (on bare metal).
TIA,
James
