On 08/10/2016 10:20 AM, Michael Mol wrote:
On Wednesday, August 10, 2016 10:13:29 AM james wrote:
On 08/10/2016 07:45 AM, Michael Mol wrote:
On Tuesday, August 09, 2016 05:22:22 PM james wrote:
I did a quick test with games-arcade/xgalaga. It's an old, quirky game
with sporadic lag variations. On a workstation with 32G ram and (8) 4GHz
64bit cores, very lightly loaded, there is no reason for in game lag.
Your previous settings made it much better and quicker the vast majority
of the time; but not optimal (always responsive). Experiences tell me if
I can tweak a system so that that game stays responsive whilst the
application(s) mix is concurrently running then the quick
test+parameter settings is reasonably well behaved. So thats becomes a
baseline for further automated tests and fine tuning for a system under
study.
What kind of storage are you running on? What filesystem? If you're still
hitting swap, are you using a swap file or a swap partition?
The system I mostly referenced, rarely hits swap in days of uptime. It's
the keyboard latency, while playing the game, that I try to tune away,
while other codes are running. I try very hard to keep codes from
swapping out, cause ultimately I'm most interested in clusters that keep
everything running (in memory). AkA ultimate utilization of Apache-Spark
and other "in-memory" techniques.
Gotcha. dirty_bytes and dirty_background_bytes won't apply to anything that
doesn't call mmap() with a file backing or perform some other file I/O. If
you're not doing those things, they should have little to no impact.
Background needed:: I'm one of those (idealists?) that deeply believes
the holy grail of computing will soon emerge (nice pun huh). That is
that clusters, local clusters will run all workloads that multicore
systems currently do. So a bunch of old crap can become a beautiful
computational system, whilst I sit back and sip exotic beverages and
enjoy my day; video training to go to the gym and dominate the young
studs on the court.... New hardware (aka new computers and cosmetic
surgery) will do the rest.
So an incredible variety of memory, storage and file systems will
ultimately need to be tested. I try to stay simple and focused (believe
it or not). Initially the thought is to run a primitive desktop, like
lxde or lxqt and use those under utilized resources as
node-computational contributors, whist still remaining responsive at the
keyboard (xgalaga is a quick and dirty test for this). So, you now have
a wonderful cover story is the boss catches you noodling around with
swords and sorcery to, you can tell'm you looking for subtle latency
issues...... The game speeds up and slows down, with zero swapping, due
to my I suspect mostly as VM issues and MM issues.
An 8 core never goes above 0.2 on the load and only rarely saturates one
core, for a transient instance. Even if xgalaga is a single thread game,
it does not explain this transient keyboard lag. I'm open to other forms
of quick at-the-keyboard graphical tests as a quick and dirty
measurement of overall system attentiveness to pending addtional
input/workload demands. After that is happy, with a given set of running
codes (test-vectors) I can get a very quick feedback of performance this
way.
For deeper studies, I like trace-cmd/Ftrace/KernelShark, but those are
like zabbix on utilization and analytical studies. I use xgalaga as a
quick and dirty; but am surely open to new codes for that sort of quick
and easy feedback.
Ideal values for dirty_bytes and dirty_background_bytes will depend heavily on
the nature of your underlying storage. Dozens of other things might be tweaked
depending on what filesystem you're using. Which is why I was asking about
those things.
A myriad of combinations exist. So picking some common combinations,
will allow for others to test my work, when it is package up for sharing
and testing. For me eventually automating a collection of 'test vectors'
is what's important, not the first few test-vectors themselvs. Then the
pathway forward for other collections of running processes can become
yet another collection of 'test vectors'. No limit on these collectives.
Eventually a customer will step forward and define the collective of
'test vectors', so I do hope to work with/for one of the more
progressive vendors, eventually, in these efforts. Certainly sharing the
work, openly, is far more important to me. For now, I start with things
I like, know and have some familiarity with; no magic on these choices.
Combined codes running simultaneously never hits the HD (no swappiness)
but still there is keyboard lag.
Where are you measuring this lag? How much lag are we talking about?
Remember, I'm an EE and complex fluids computational kind of guy, so I
have no problem drudging down the sparse or full matrix types of
mentally inebriating adventuresome calculations, like computational
chemistry. But, since this approach is not yet ready for those sorts of
things, I keep things simple; for now. What I want, is an automated
installation semantic, where folks can download images and run them on
their small clusters) on a weekly basis and keep solving the same
test-vector collectives over and over. Tweaks and ideas are in the newly
released images, a group of gentoo-users test things out. But
an automated, quick and simple gentoo system, flies against what most
folks believe in this community (dammit, I have to respect, so I work on
my one scripts I have lifted from others) {wink wink; nudge nudge}.
As you already know....
Not that it is actually affecting the
running codes to any appreciable degree, but it is a test I run so that
the cluster nodes will benefit from still being (low latency) quickly
attentive to interactions with the cluster master processes, regardless
of workloads on the nodes. Sure its not totally accurate, but so far
this semantical approach, is pretty darn close. It's not part of this
conversation (on VM etc) but ultimately getting this right solves one of
the biggest problems for building any cluster; that is workload
invocation, shedding and management to optimize resource utilization,
regardless of the orchestration(s) used to manage the nodes. Swapping to
disc is verbotim, in my (ultimate) goals and target scenarios.
No worries, you have given me enough info and ideas to move forward with
testing and tuning. I'm going to evolve these into more precisely
controlled and monitored experiments, noting exact hardware differences;
that should complete the tuning of the Memory Management tasks, within
acceptable confine . Then automate it for later checking on cluster
test runs with various hardware setups. Eventually these test will be
extended to a variety of memory and storage hardware, once the
techniques are automated. No worries, I now have enough ideas and
details (thanks to you) to move forward.
You've got me curious, now you're going to go run off and play with your
thought problems and not share! Tease!
Dude, I share too much. If you had not gone of vacation (from
gentoo-user) you'd know this. Since I am way too mentally handicapped to
do all of this on my own, (and too old and wise to even try) I routinely
seek guidance and help. I read quite a lot, to remind me of the mistakes
from previous distributed parallel computational attempts; and that
reading also saddens me a bit to see so many malformed cluster ideas. Oh
well, failure is the most important lesson technical folks learn. Most
often ideas just bounces off the wall right back at me, but I have
learned to duck (most of the time). YOU and anyone else are most welcome
to join my efforts; we all shall benefit from robust, local clusters, as
masters of gentoo (or poezer of gentoo, just like me). <end philosophy>
So while we are at it, scripts or stage-4 images that can be rapidly
booted up on a given small hardware cluster, are keen to my approach.
Memory management, is probably the most challenging aspect of building
and robustly (efficient resource utilization) managing these clusters
or outsourced clusters (clouds in vendor speak). I Use the same cluster
setup, to test a myriad of different problem-solution sets on the
identical hardware, but only change the software, including file
systems: both DFS (cephfs/orangefs/openAFS/Beefs) and the local fs (xfs,
ext4,) and well as hybrids like btrfs and special file systsems like
bcache. On top of Openstack, Hadoop, Mesos, old Beowulf (with a fast DFS
replacing NFS) and others.
Once domain specific problems are moved to a cluster and that solution
set is near-optimal, after robustly testing many codes, in a CI fashion
outlined above, it becomes a stage-4 canned solution for somebody to run
on their hardware. If they need more hardware resouces, within a
specific interval, THEN outsource those resource needs to the Cloud
Vendors. Expecting a cloud vendor to be a champion of your Domain
Specific need, is a roadmap to chapter 11 or 13, for that corporation.
I suspect that once AWS and Google and MS and IBM learn what the NSA
already knows, there will be a feeding frenzy on aquisitions of old
technology companies. That's ultimately where the action is in clusters.
All of this 'smoke and mirrors' marketing centric on social networks is
just that; smoke and mirros. Why do I say this? Simple; there already is
enough processing power to solve those problems and needs with current
Snoracle style solutions and the by the bloated on wall-street.
Now HPC, dude, that's the sweet edge of clustering. There are numerous
gargantuan issues in that sphere and a few, like DESHAW are getting RICH
off of clusters. He, a single Stanford professor, mastered computational
chemistry, and locked his expertise into ASIC chips.
Now he is conquering wallstreet. Domain Specific solutions are where the
action is in clusters. It not that there's not money in the social
networking spheres, those are locked up by the 'cost barrier to entry'
semantics. OK, I digress. But the important thing is local clusters,
taht can be rapidly build and torn down and reconfigured, with a few
simple keystrokes, are the future of clusters. A given small to mid
sized company better learn how to build their own clusters, or they be
in the welfare line, like several other billion folks are.
CoreOS and unikernels are really quite similar to my approach to
clusters. A variety of Problem-solutions sets (aka test vectors) on
identical hardware will light the pathway for Domain Specific cluster
solutions. Mine will be a node cluster on amd64, for now.
So, I'm not sitting on some Stanford level of skills or knowledge base
(think amplabs). I have decades of experiences in mostly unfulfilled
promises for ubiquitous distributed processing, and only narrowly (very
tightly) focuses success stories. Still, I am a believer in that the
current crop of linux clusters will become an Utopia computation engine
system that works from the most modest of needs like mundane admin
taskloads to the most demanding, time-sequence RT simulations of some of
the grand challenges in computational dynamics and similar areas.
But, after several years of research, I mostly see kids trying the same
crap we tried decades ago, with a new 'fancy-pants' programming
language:: (hence the prediction that the current cluster kids are being
manipulated by the VC firms and deep pocketed folks toward certain
failure), whilst they pay off their debts. Same story, different overlord.
I am conflicted as to whether this is intentional or just a repeat of
tards leading the blind and innocent off the cliff. That is most of the
vendor centric cluster (marketers call these clouds), developing new
codes are clueless. That said, surely those corps with large collections
of existing software can migrate those critical codes to the cloud and
only offer new versions of that software, with a (cloud centric)
internet-needed license. Think Azure/MS, IBM etc etc. But that sort of
position, will just allow competitors to eat away larger chunks of their
market share. (But I really don't care about his part of the Cloud
illusion. I'm a hard core hardware type who already knows that the
future of clusters is mostly local, with local control. The cloud will
become a secondary or tertiary market for cpu cycles and garbage
collection (think social networking databases). Sure folks will put
their websites on commercial clouds, but that is already just a natural
evolution of Co-location of server and not some breakthrough is technology.
Down this pathway, the developments in the latest version of Clang, gcc,
etc etc, and EEs making the resources of the GPU (including DDR5+) into
a transparent computational resource for the routine compilers. rDMA is
going to change (everything). Ram will finally not be the bottleneck, as
FPGA and GPU resources can be configured, dynamically, as either highly
specialize processors or highly specialized memory (look at CAMs, or
Content Addressible Memory for a teaser). Router vendors have been
making billions of dollars by adding CAMs to otherwise mundane
processing systsems.
No more of those ancient (intel) parallel compilers and shit like
that.... Plus and avalance of re-configurable memory types; mostly
transparent to folks that use "emerge" for custom compiling. Then there
is a hardened kernel. Few in the cluster world even know such things
exist; more sadly why they are necessary and when they are necessary.
Keep puffing on that buntu hoka pipe, brah_heim.....
The flip side to this is that a lot of Vendors think that bloated linux
operating systems, on top of non-tuned, non-stripped insecure linux
kernels is going to be commercially viable. If you build your house on
turds, when it starts to rain, there is a funky smell in the air, before
it washes away. Bloated buntu, debian or RHEL are turds and are not
going to work compared to stripped, minimal linux systems. That's where
Docker, just "bitch-slapped" their competition by moving to subsume
Apline linux.....
Your postings and clarity on VM, has helped me focus, immensely. It is
the current need in my work. Have I shared enough for you, today?
Any other questions, or ideas are most welcome, publically or privately.
I could be wrong about all of this, but, my fourth generational stab at
ubiquitous (distributed || parallel) processing experiences tell me I'm
not wrong but have the right idea. I do lack current skills in so many
areas, that my work is impeded.
Without the gentoo community, I could not posses such visions of
future-present greatness; nor share it with others.
Perhaps Zabbix +TSdB can get me further down the pathway. Time
sequenced and analyzed data is over kill for this (xgalaga) test, but
those coalesced test-vectors will be most useful for me as I seek a
gentoo centric pathway for low latency clusters (on bare metal).
If you're looking to avoid Zabbix interfering with your performance,
you'll
want the Zabbix server and web interface on a machine separate from the
machines you're trying to optimize.
agreed.
Thanks Mike,
James
np
Clusters will end up on people's wrist watches, in the trunks of their
autos and at their homes:: So they control their computational needs and
security, sooner rather than later. I think the next president will
mandate the opening of the OS to many vendors and open source for Cell
phones, Apps and such. The current monopolies are excessively more
powerful than the old 'robber barrons' and that fact is well recognized
by lost of deeper thinkers. It's braned under globalization, but, it's
demise is just under the horizon, imho.
True, ubiquitous clusters will be a result of hard work on compilers
that take sequential problems and break them down into pieces and
reassemble them into a form that can leverage parallel techniques. gcc 5
and 6 and Clang are moving, rapidly in this direction. GPU vendors
understand the importance of SIMD and MIMD processing for 'systolic'
algorithms and such approaches to massive distributed processing. AMD
(Radeon) understands that this power can most effectively be used, if it
is cheap and open sourced. Nvidia, no so quick to follow (or lead) down
this open source path, imho. Intel purchasing a FPGA company and
licensing GPU technologies from many others, tells me the hardware
vendors are preparing for a revolution. A direct sales channel to the
commoners will be their greatest path to rediculous profitability. Why?
Simple, the smaller the core (competitive team) that exists, the more
excessive processing resources that will be purchased and purchased
closer to the retail price.
When hardware vendors partner with a few sofware companies, the margins
on hardware get squeezed. Besides the hackers of the work, are finding
any critical barriers to codes and publishing it so all have fair access
to the latest codes (one way or another). The NSA and such entities are
not going to stop this, because all of this software espionage,
justifies governments taxing the snot out of citizens to fight those
evil hackers. It's a far superior business model for DoD
types like intel and google, than the cold ware ever though about being.
The average tax-payer is too stupid to realize social network, with an
Onior approach, is just feeding data-sets via google, linkedin, facebook
etc, directly to the NSA and other Nation State actors. WE get jobs
and pay taxes. They set the rules and manage the data.
Problem is, eventually, the commoners will have sufficent clusters,
solar panels water wells or sources and green house and tell da_main
to stick his taxes on imports. Fine that works, then everybody gets
a 3D printer and we, the commoners are self sufficient.
The simple fact is that is a great business model for EVERYONE,
including the elites, so what are we waiting on? A stupid old man like
me? Naw, not at Gentoo, buntu, sure, RHEL definately, but not gentoo,
brah. WE are the solution to everything!
</>
James
