Re: [gentoo-user] Re: gigabyte mobo latency
Am 20.10.2014 um 01:34 schrieb Rich Freeman: On Sun, Oct 19, 2014 at 5:45 PM, Volker Armin Hemmann volkerar...@googlemail.com wrote: Am 19.10.2014 um 20:56 schrieb Rich Freeman: On Sun, Oct 19, 2014 at 12:41 PM, James wirel...@tampabay.rr.com wrote: As far as docs go - what specifically is unclear? Systemd is rapidly evolving so things do get out of date, but for the most part stuff like this can be found in man systemd.exec and such. Lennart's series of blog posts on system administration using systemd is a very good place to start. -- Rich and you are fine with that? That the package that becomes Pid1 is such an unstable, ever involving mess that not even the docs are keeping up with it? If I wasn't fine with it I'd hardly be running Gentoo. :) You just described half of the distro, though it isn't changing as rapidly as it used to. Most of the issues are in the init scripts, which technically aren't really part of systemd. It is fairly immature everywhere. If you want a really stable experience then you should probably stick with RHEL or Debian Stable. :) You just gave me another reason not to use systemd. Sounds like you didn't really need one. It is a free country - you're no more required to run systemd than any Gentoo developer is required to fix openrc init scripts or systemd units... -- Rich . you make that sound like a threat. But that does not work with me ;) I have dealt with Slackware in the past. Had no probs back then, why should I be scared now?
Re: [gentoo-user] Re: gigabyte mobo latency
James wrote: Volker Armin Hemmann volkerarmin at googlemail.com writes: I am sure there are some plugins to enable js on a per-page basis for firefox. If konqueror can do it... Short term I'll uses this. Long term, I need to learn more about cgroups and tuning for my clustering ambitions. thx, James If I understand the issue correctly, I have used adblock to block all sorts of things that annoy me. That includes scripts. Dale :-) :-)
Re: [gentoo-user] Re: gigabyte mobo latency
On 19/10/14 04:15, James wrote: thegeezer thegeezer at thegeezer.net writes: there is a little more here http://gentoo-en.vfose.ru/wiki/Improve_responsiveness_with_cgroups which will allow you to script creating a cgroup with the processID of an interactive shell, that you can start from to help save hunting down all the threads spawned by chrome. you can then do fun stuff with echo $$ /sys/fs/cgroup/cpu/high_priority/tasks Yea this is cool. But when it's a cluster, with thousands of processes cgroups are hierarchical, so for example if you start a bash script which is in cgroup cpu/high_prio which then starts your processes, all called programs go into the same cgroup which makes it a bit simpler. also openrc will start your services in the correct cgroup too this seem to be limited by the manual parsing and CLI actions that are necessary for large/busy environments. (We shall see). hopefully this will give you a bit more control over all of that though Gmane mandates that the previous lines be culled. That said; you have given me much to think about, test and refine. In /sys/fs/cgroup/cpu I have: cgroup.clone_children cgroup.procs cpu.shares release_agent cgroup.event_control cgroup.sane_behavior notify_on_release tasks So I'll have to research creating and priotizing dirs like high_priority I certainly appreciate your lucid and direct explanations. Let me play with this a bit and I'll post back when I munge things up. Are there any graphical tools for adjusting and managing cgroups? i thought that htop did this but i was wrong.. it only shows which cgroup processes are in. that would be a killer feature though. Surely when I apply this to the myriad of things running on my mesos+spark cluster I'm going to need a well thoughout tool for cgroup management, especially for non-local systems. other distros have apps such as cgclassify which provides some shortcut to managing cgroups -- creation / and moving process in and out you can also have a nohup process that does ps -eLf to search for process you want to classify and move them into the appropriate cgroup for default cgroups you can also use inotify a quick search shows http://libcg.sourceforge.net/ which daemonises this process. all this is a bit hack'n'slash though i appreciate, so if anyone else knows of suitable tools i'd also be interested to hear of them particularly on memory resources organization and allocations as spark is an in_memory environment that seems sensitive to OOM issues of all sorts. thx, James
Re: [gentoo-user] Re: gigabyte mobo latency
James wrote: thegeezer thegeezer at thegeezer.net writes: especially for non-local systems. other distros have apps such as cgclassify which provides some shortcut to managing cgroups -- creation / and moving process in and out Ok. So, if you or anyone else knows of or runs across a robust gui managment interface for cgroups, please post to this list or drop me some email. Porting something that is established is my preferred modis_operandi. SNIP thx, James I'm not sure about robust but I did find this little bread crumb. https://lists.fedorahosted.org/pipermail/cg-manager-developers/2011-July/02.html It has a few links and it could lead to a gold mine or some fake gold. Dale :-) :-)
Re: [gentoo-user] Re: gigabyte mobo latency
On Sun, Oct 19, 2014 at 12:41 PM, James wirel...@tampabay.rr.com wrote: Even if we all end up migrating to systemd (which from plentiful complaints from many very bright folks about the net and the lack of a clean, useful documentation on systemd, it's likely to be a decade before systemd stablizes and folks produce sufficiently useful documentation and examples) cgroups does illuminate how things should work in a complex environment so it still is worth it's weight in gold, before one attempts to master the (systemd) beast. So, I realize there are many strong opinions regarding systemd, but this comes across a bit like, one should be well-accustomed to building and operating a Linux-From-Scratch installation before one attempts to master the (Ubuntu) beast. Sure, all that auto-magic stuff does add complexity, but it does so with the goal of standardizing and automating this so that you can use the system without having to worry about all the details. If you are running a systemd service you can set the various cgroup controls like IO and CPU class/priority in the unit and it will take care of managing the cgroup for you. Certainly learning the nuts and bolts of how it all works is worthwhile - I wouldn't be running Gentoo if I felt otherwise. However, you really don't have to know how to build your own service manager to use one. As far as docs go - what specifically is unclear? Systemd is rapidly evolving so things do get out of date, but for the most part stuff like this can be found in man systemd.exec and such. Lennart's series of blog posts on system administration using systemd is a very good place to start. -- Rich
Re: [gentoo-user] Re: gigabyte mobo latency
On 19/10/2014 22:40, James wrote: You and the other systemd (herd/project) dudes are wonderful. Right now, I just like openrc/cgroups/assembler and stories from other old_farts. You young whipper_snappers should be very glad us old farts still hack and hang out like we do. Kids might look at him and say Wow. Older folks just murmur under their breath that this snot_nosed_kid should have been bitch_slapped by that idiot Linus. He failure to reign in that looser cannot be white_washed by anyone; so let's just let this go... The more I read about the entire affair the more pissed I get. Relax James, it's all good and the wheel turns. I'm an old fart too, the first computer I ever owned was a Sinclair Mk 14 and I built it by hand from a kit. I've met 3 people who even know what it is :-) Anyway, all the lessons of the past are never truly forgotten, and most of us do know how to pick the right tool for the job. These young whipper-snappers think that cloud is the bestest and most awesomest thing ever, folks over 35 know that this is the THIRD time we've gone through this evolution (had stuff in a data centre and now farming it out to a cloud). In 5 years from now it will all consolidate and we'll be pulling the cloud back into data centres. It will look different, but the principle will be the same. Like splitting the CPU GPU, I lose count of the number of times we've done this too (think math co-processors and mainframe front-end controllers and DMA). And so it is with systemd, right now it is the buzz word because it's declarative, easy to maintain and simplifies keeping control of complex systems with gobs and gobs of RAM (RAM is cheap, dev and sysadmin time is very expensive). Embedded: ah, that's another story. RAM is very expensive there and in short supply. If it hasn't already been done, someone will write an init system for constrained embedded devices that has no need of systemd, and it will be better than SysVInit Systemd will never take over the world - as soon as it causes too much pain for someone who doesn't need it, they will find a way to get along without it. And those who do need/want it can still have it. This is just how things work, and you've seen it all play out before :-) -- Alan McKinnon alan.mckin...@gmail.com
Re: [gentoo-user] Re: gigabyte mobo latency
On Sun, Oct 19, 2014 at 4:40 PM, James wirel...@tampabay.rr.com wrote: Rich Freeman rich0 at gentoo.org writes: Rich, embedded is my background. I'm more of an EE over the years. so YES, to me it is very important to understand hardware and the firmwares that allow all of the OO-gui stuffage that exists (and is wonderfull). Again, I wasn't bashing anybody for having a desire to understand cgroups. I just didn't want to suggest that it was necessary to have this knowledge to actually use systemd. Of course, if you're in an unusual niche (embedded, clusters, etc) then systemd is going to be less mature, and since you're blazing a trail the additional knowledge certainly wouldn't hurt. And of course you don't have to use systemd to benefit from cgroups. You have an idealized view of what is going on in the cluster code spaces. (systemd issue abound aplenty). Folks are just loading binaries on top of binaries and look for salvation via config files. You certainly have more firsthand perspective there than I do. Systemd has plenty of issues in any configuration - it is very new, and VERY susceptible to race conditions with its dependencies. When you run it on a box that provides core services (dhcp, dns, nfs, etc) as well as consuming those same services you can run into a lot of unspecified dependencies. For example, systemd tends to assume that if the network is up, DNS is up, but that doesn't work so well for your DNS server. None of these are insurmountable design issues, they just point to the immaturity of the project. But, like many here that aspect of systemd doesn't really scare me off much. If the systemd devs and perveyors feel pressure to make systemd a superior technology; what's wrong with that? I like the promise of systemd; I *hate* the way it has been jammed down on everyone. Well, certainly on Gentoo it hasn't been jammed on anybody per-se. About the closest to that in Gentoo has been the fact that they took over udev maintenance, but that is an upstream issue (and nobody can prevent anybody from forking it/etc). Within Gentoo there was some controversy when some package maintainers wanted to prevent others from adding systemd units to their packages, and from that standpoint there is a policy that forces maintainers to stay out of the way if somebody else wants to do the work to adapt a package for systemd (or for openrc for that matter). Most other distros only support a single init, and for whatever reason they've decided to switch. That is really their business, but it is hard to say that anybody is forcing anybody to do anything. Gnome is obviously a factor, but again nobody HAS to use Gnome, and it is a bit like complaining about having to have Java installed in order to play minecraft or use some features in openoffice (granted, Java isn't /quite/ as invasive though sometimes it feels that way). I think once Lennart moves on to something else (as many have pondered) systemd will become more attractive. In my youth, I did not understand 'good manners' and lennart epitomizes some episodes in my tempetuous youth I regret. Welcome to the FOSS community, where principle and idealism trumps relationships every day. In the end we tolerate contributors because they contribute. We could always choose to not use their stuff, but for the most part we're too lazy to re-implement everything. With FOSS people only have power over you if you give it to them, but not giving it to them comes with a price. Older folks just murmur under their breath that this snot_nosed_kid should have been bitch_slapped by that idiot Linus. He failure to reign in that looser cannot be white_washed by anyone; so let's just let this go... Just what would you have Linus do? Other than criticize publicly, there isn't much Linus can do about systemd, since he maintains the kernel and systemd isn't part of the kernel. Linus has about as much power to change systemd as anybody does. Again, with FOSS YOU choose who gets the power, since you choose what software you use. If you run xfce then there is nothing the gnome developers can do to touch you. Now, if all the xfce maintainers get bored and quit you're stuck, but that is true of anything in FOSS. You get what you pay for. :) -- Rich
Re: [gentoo-user] Re: gigabyte mobo latency
Am 19.10.2014 um 20:56 schrieb Rich Freeman: On Sun, Oct 19, 2014 at 12:41 PM, James wirel...@tampabay.rr.com wrote: As far as docs go - what specifically is unclear? Systemd is rapidly evolving so things do get out of date, but for the most part stuff like this can be found in man systemd.exec and such. Lennart's series of blog posts on system administration using systemd is a very good place to start. -- Rich and you are fine with that? That the package that becomes Pid1 is such an unstable, ever involving mess that not even the docs are keeping up with it? You just gave me another reason not to use systemd.
Re: [gentoo-user] Re: gigabyte mobo latency
On Sun, Oct 19, 2014 at 5:45 PM, Volker Armin Hemmann volkerar...@googlemail.com wrote: Am 19.10.2014 um 20:56 schrieb Rich Freeman: On Sun, Oct 19, 2014 at 12:41 PM, James wirel...@tampabay.rr.com wrote: As far as docs go - what specifically is unclear? Systemd is rapidly evolving so things do get out of date, but for the most part stuff like this can be found in man systemd.exec and such. Lennart's series of blog posts on system administration using systemd is a very good place to start. -- Rich and you are fine with that? That the package that becomes Pid1 is such an unstable, ever involving mess that not even the docs are keeping up with it? If I wasn't fine with it I'd hardly be running Gentoo. :) You just described half of the distro, though it isn't changing as rapidly as it used to. Most of the issues are in the init scripts, which technically aren't really part of systemd. It is fairly immature everywhere. If you want a really stable experience then you should probably stick with RHEL or Debian Stable. :) You just gave me another reason not to use systemd. Sounds like you didn't really need one. It is a free country - you're no more required to run systemd than any Gentoo developer is required to fix openrc init scripts or systemd units... -- Rich
Re: [gentoo-user] Re: gigabyte mobo latency
James wrote: Dale rdalek1967 at gmail.com writes: https://lists.fedorahosted.org/pipermail/cg-manager- developers/2011-July/02.html At the very least it has educational value. I'll have to test this and some other codes Thx, James Hi James, I did some more digging later on and it seems it may have died. I found a thread where it seems devs were of the opinion that normal users shouldn't manage this to much so a tool like this was not needed. At least that was the impression I got but the thread was a bit dated so who knows what else may have changed. They may have slapped a new name on it and it may be very much alive, somewhere out there. If it is tho, I couldn't find it, yet. I been following this thread just to see how this cgroup stuff works. I'm just curious. I to was hoping for a GUI tool to manage this sort of thing. Heck, I might would even set it up and play with it a bit. I run multiple Firefox profiles here and this could help with a few issues. If you do find something, please share what you find. I am interested for sure even if I don't post much. Thanks. Dale :-) :-)
Re: [gentoo-user] Re: gigabyte mobo latency
141018 James wrote: Volker Armin Hemmann volkerarmin at googlemail.com writes: Some websites load run MB of javascript which bogs down everything. Nothing you can do about that except turning of js. Well, that was it on the performance (typing latency). I had to diable javascript, terminate + save and restart Seamonkey. Obviously, there are too many things I need that use javascript, so I re-enabled javascript and terminated (a second time, with a save) the seamonkey app and start it again (now again with javascript enabled). Perhaps you could use Lynx for those items which don't need JS reserve Firefox for those which do. -- ,, SUPPORT ___//___, Philip Webb ELECTRIC /] [] [] [] [] []| Cities Centre, University of Toronto TRANSIT`-O--O---' purslowatchassdotutorontodotca
Re: [gentoo-user] Re: gigabyte mobo latency
Am 18.10.2014 um 21:27 schrieb Philip Webb: 141018 James wrote: Volker Armin Hemmann volkerarmin at googlemail.com writes: Some websites load run MB of javascript which bogs down everything. Nothing you can do about that except turning of js. Well, that was it on the performance (typing latency). I had to diable javascript, terminate + save and restart Seamonkey. Obviously, there are too many things I need that use javascript, so I re-enabled javascript and terminated (a second time, with a save) the seamonkey app and start it again (now again with javascript enabled). Perhaps you could use Lynx for those items which don't need JS reserve Firefox for those which do. I am sure there are some plugins to enable js on a per-page basis for firefox. If konqueror can do it...
Re: [gentoo-user] Re: gigabyte mobo latency
On Sat, 18 Oct 2014 22:24:43 +0200, Volker Armin Hemmann wrote: I am sure there are some plugins to enable js on a per-page basis for firefox. If konqueror can do it... There are several for Chromium, I think the favourite for FF is noscript. -- Neil Bothwick Top Oxymorons Number 17: Clearly misunderstood signature.asc Description: PGP signature
Re: [gentoo-user] Re: gigabyte mobo latency
On 18/10/14 22:51, James wrote: thegeezer thegeezer at thegeezer.net writes: So. Is there a make.conf setting or elsewhere to make the terminal session response times, in the browsers (seamonkey, firefox) faster? the typing latency in the browser windows). ideas? two things you might like to look into: 1. cgroups (including freezer) to help isolate your browsers and also 2. look at atop instead of htop as this includes disk io 2. The system rarely uses 8 G of the 32 G available, so disk IO is not the problem. No heavy writes. It was the java scripts 1. Ahhh! tell me more. I found these links quickly: https://www.kernel.org/doc/Documentation/cgroups/freezer-subsystem.txt http://wiki.gentoo.org/wiki/LXC#Freezer_Support I'm not sure if you've read any of my clustering_frustration posts over the last month or so, but cgroups is at the heart of clustering now. It seems many of the systemd based cluster solutions are having all sorts of OOM, OOM-killer etc etc issues. So any and all good information, examples and docs related to cgroups is of keen interests to me. My efforts to build up a mesos/spark cluster, center around openrc and therefore direct management of resources via cgroups. The freezer is exactly what I'm looking for. Maybe I also need to read up on lxc? What are the best ways to dynamically manage via cgroups? A gui? A static config file? a CLI tool? curiously, James the thing with cgroups is that you can choose to create a hierarchy of what _you_ want to have as your priority unfortunately you need to tell the machine what it is you want, it can't really guess granularly iuc e.g. your favourite terminal / ide etc you want high prio and your favourite file mangler to be low prio ( assuming you want compiling to take precedence over bit munging to usb etc) there is however cgroup support in htop, and i thought that you could adjust cgroup in stead of nice but a quick google is showing that i dreamed it as a nice feature; that would be super slick as you can easily adjust all parts of program demand -- io / memory / cpu using openRC you can start services i.e. apache to have a certain priority, and ssh to have another http://wiki.gentoo.org/wiki/OpenRC/CGroups http://qnikst.github.io/posts/2013-02-20-openrc-cgroup.html the reason i suggest freezer is that you can more easily pause or CTRL-Z something that would otherwise be in a GUI and maybe not respond to SIGSTP on my laptop :- # mount | grep freez freezer on /sys/fs/cgroup/freezer type cgroup (rw,nosuid,nodev,noexec,relatime,freezer) # cd /sys/fs/cgroup/freezer/ call the folder something meaningful # mkdir investigate # cd investigate you can use the following, i just did echo $$ for local bash pid... make sure to get all threads especially something like chrome spawns many children # ps -eLf | grep mybadapp note the single actually concatenates # echo $PID tasks to remove you actually have to move the process into a different cgroup i.e. # echo $PID /sys/fs/cgroup/freezer/tasks ok so once you have all your tasks in there just make sure you are in /sys/fs/cgroup/freezer/investigate # echo FROZEN freezer.state to thaw # echo THAWED freezer.state there is a little more here http://gentoo-en.vfose.ru/wiki/Improve_responsiveness_with_cgroups which will allow you to script creating a cgroup with the processID of an interactive shell, that you can start from to help save hunting down all the threads spawned by chrome. you can then do fun stuff with echo $$ /sys/fs/cgroup/cpu/high_priority/tasks but for your original point of maybe it's not an issue with something like IO it could still be a very high number disk reads -- low actual thoughput but the demand on the io system was high, i.e. 6zillion reads hopefully this will give you a bit more control over all of that though
