26 ноября 2015 г. 9:04:26 CET, Gabriele Bulfon <gbul...@sonicle.com> пишет: > Is this a per-zone command, or output is the same on any zone? > This is what I get in the smaller www zone: > kthr memory page disk faults > cpu r b w swap free re mf pi po fr de sr s0 s1 s2 s3 in sy > cs us sy id 1 0 0 6531500 798908 1504 6818 0 0 0 0 0 0 0 0 181 4646 > 465913 68813 13 6 81 0 0 0 6531412 796496 785 1778 0 0 0 0 0 0 24 0 > 9 6061 309567 21991 9 2 89 2 0 0 6531372 792056 639 5799 0 0 0 0 0 0 > 0 11 10890 371104 120952 15 6 79 1 0 0 6534576 824876 407 4644 0 0 0 > 0 0 0 0 77 5366 193555 10202 12 2 86 1 0 0 6555168 838108 151 > 1071 0 0 0 0 0 0 0 0 2 4107 8759 3522 4 1 95 0 0 0 6562804 > 841320 160 2223 0 0 0 0 0 0 0 0 405 6257 12961 9741 6 3 90 2 0 0 > 6570848 841524 1005 2068 0 0 0 0 0 0 17 0 9 4610 43174 6909 8 2 90 > 0 0 6578884 843892 720 1977 0 0 0 0 0 0 0 0 1 6613 35529 14808 > 3 1 96 0 0 0 6578876 842680 926 1558 0 0 0 0 0 0 0 0 21 7391 63111 > 19592 5 2 93 0 0 0 6575672 836400 1376 5808 0 0 0 0 0 0 0 0 4 6102 > 91652 20029 8 2 89 0 0 0 6574764 834404 1098 2068 0 0 0 0 0 0 0 0 > 193 9347 162067 39057 10 4 86 1 0 0 6572020 826300 1953 7228 0 0 0 0 0 > 14 0 31 10959 156369 49396 11 6 84 0 0 0 6571000 813060 1342 5191 0 > 0 0 0 0 0 0 2 5087 391732 59501 9 3 88 0 0 0 6563644 802900 1176 > 3594 0 0 0 0 0 0 0 0 3 4183 538480 16301 12 2 86 3 0 0 6561048 > 795112 1787 2832 0 0 0 0 0 0 0 0 1 8068 493355 66482 14 4 83 1 0 0 > 6563072 793696 1239 1712 0 0 0 0 0 0 0 0 265 7595 229455 16196 8 3 > 89 0 0 0 6553832 780652 1471 4747 0 0 0 0 0 0 15 0 2 4445 216211 > 5103 9 1 90 1 0 0 6545468 768120 2343 6878 0 0 0 0 0 0 0 0 22 5014 > 159315 13014 17 2 80 > and iostat in the other cloud zone: > extended device statistics r/s w/s kr/s kw/s wait actv wsvc_t > asvc_t %w %b device 22.0 27.0 337.5 931.1 0.0 0.2 0.0 > 1 0 21 sd3 16.0 27.0 241.0 930.1 0.0 0.2 0.0 5.0 0 > 17 sd4 17.0 27.0 338.5 931.1 0.0 0.2 0.0 5.2 0 18 sd5 > 26.0 212.0 921.6 0.0 0.2 0.0 4.4 0 15 sd6 17.0 > 216.5 934.6 0.0 0.2 0.0 4.1 0 14 sd7 > Sonicle S.r.l. > Gabriele Bulfon > Tel +39 028246016 - Fax +39 028243880 > via Fra Cristoforo, 14/D - 20142 - Milano - Italy > http://www.sonicle.com > ---------------------------------------------------------------------------------- > Da: Jim Klimov > disc...@lists.illumos.org Marion Hakanson > gbul...@sonicle.com > Data: 26 novembre 2015 0.56.23 CET > Oggetto: Re: [discuss] deadlocks > 25 ?????? 2015 ?. 23:32:13 CET, Marion Hakanson > ?????: > Hi Gabriele, > The "prstat -Z" numbers by themselves do not tell you if paging is > going on. > All you can really glean from it is that all values in the "MEMORY" > percent > column add up to less than 50% of the overall system memory. > Look at "vmstat 1" output, the "pi" and "po" columns in particular, > while it prints out one line per second over time. You should be > able to do this from within each zone as well as in the global zone. > That will tell you if active paging/swapping is going on. > You could also use "iostat -xn 1" and see if the drives used by your > swap space are busy, to look for clues. > Regards, > Marion > Date: Wed, 25 Nov 2015 23:00:55 +0100 > From: Gabriele Bulfon > Reply-To: > To: Marion Hakanson > , > Subject: Re: [discuss] deadlocks > Thanks Marion! > this is what I was thinking, look at my global zone prstat -Z : > ZONEID NPROC SWAP RSS MEMORY TIME CPU ZONE > 7 855 8697M 6877M 28% 16:03:15 2.6% cloudserver > 8 292 2978M 1766M 7.2% 0:20:09 0.4% www.sonicle.com > 92 1129M 477M 1.9% 110:18:26 0.1% global > 1 23 41M 31M 0.1% 11:55:15 0.0% asterisk > 5 26 524M 460M 1.9% 11:23:06 0.0% pkgserver > 3 434 2212M 1231M 5.0% 39:04:23 0.0% encoserver > 2 138 1402M 774M 3.2% 4:04:16 0.0% demo.sonicle.com > cloudserver and www.sonicle.com are the said zones. > I'm not sure what the SWAP column is saying, is cloudserver using all > that swap? > There's no clue about swaps inside the zones, swap -l says: > sonicle@cloudserver:~$ swap -l > swapfile dev swaplo blocks free > /dev/swap - 8 25149432 24050136 > but maybe it's not real... > ---------------------------------------------------------------------------------- > Da: Marion Hakanson > disc...@lists.illumos.org > Data: 25 novembre 2015 20.06.54 CET > Oggetto: Re: [discuss] deadlocks > Hi Gabriele, > The behavior you describe could be caused by saturation of any of > the resources on the system, not only CPU/load. If all of RAM were > used up, for example, a new SSH session would pause until processes > were swapped/paged out enough to give room for a new SSH to run > in memory. Similar results could happen if network or disk resources > were saturated. > Have a look at the USE method: > http://www.brendangregg.com/usemethod.html > On illumos/Solaris-based systems, you can use these commands to start > with: > prstat 1 (for CPU) > vmstat 1 (for memory) > iostat -xn 1 (for disk) > dladm show-link -s -i 1 > (for network) > Also check /var/adm/messages for errors being logged at or near the > time of the issue. > Regards, > Marion > Date: Wed, 25 Nov 2015 19:39:38 +0100 > From: Gabriele Bulfon > To: > Subject: [discuss] deadlocks > Hi, > I'm looking for help to find a solution to strange slow downs on a > long living XStream/illumos server. > This server runs 5-6 zones, on intel 8 cores, 24GB ram, separate boot > on sata mirror rpool, and data on sas raidz pool. > Two of these zones run essentially the same software: apache, tomcat, > cyrus, postfix, amavis, postgres > Apache front ends http to tomcat, running our collab webapps, working > all the day on postfix smtp, cyrus imap and postgres db. > 1st zone is our own dev machine, running 4-5 users actually on all > the stack. > 2nd zone is our customers machine, running around 1000 users on all > the stack, separated into about 10 cyrus domains > and their separated 10 instances of both webapps and databases. > Recently, it happens from time to time (1-2 times a week) that > everything starts to slow down. > Stopping one or the other zone's tomcat/apache gets everything back: > somtimes it's ours, sometimes it's the cloud. > Ok, at first sight one would say: your web app has problems. > But....then why do I have hard times connecting via ssh to the zones > during this situations? Login takes minutes, > password to shell another lots of minutes, but prstat/top don't show > any cpu high usage on global zone, nor inside the zones. > Then I stop one tomcat (sometimes one, sometimes the other), and > verything gets free. > Imap processes during these times are around 1000 in one machine, > around 100 on the other. > Then they abruptly gets down, obvioiusly the web app closes > connections. > So my question is....how can I dig this problem? > I would think that if the webapp is the problem, iniside java/tomcat, > I should not experience problem during ssh. > Any possible limits on socket? Any other idea? > Thanks.... > Gabriele > > Also do not forget that in Solaris, swap does not necessarily mean > allocated memory - it can quite be just reserved. VMs tend to do that - > requiring enough swap to be available if it needs to be used for a VM's > RAM to get swapped out. Usually it is never really used. > Try to leave a terminal (maybe in a vnc console) with 'vmstat 1' and > 'iostat -Xnz 1' so you'd see if the system is swapping durkng the > slowdown. > Also maybe walking the fragmented memory is a time-consuming task. > Revise your JVM settings (GC etc.) to rule that out - or maybe even > solve the issue by cleaning up often enough that single ops are not > fatally slow. > HTH, Jim > -- > Typos courtesy of K-9 Mail on my Samsung Android >
Interesting question. I'd wager that you better look at this from both the GZ and impacted local zones. At least some visibility would be limited if you configured e.g. cpu pools, memory/swap limits, etc. Which reminds me - I had similar slowdowns when localzones on one host were configured with resource capping, and then their servers grew in appetite, and it became the server's primary job to enforce those obsolete limits. Lots of swapping with large free RAM (in the GZ at least) etc. Jim -- Typos courtesy of K-9 Mail on my Samsung Android ------------------------------------------- illumos-discuss Archives: https://www.listbox.com/member/archive/182180/=now RSS Feed: https://www.listbox.com/member/archive/rss/182180/21175430-2e6923be Modify Your Subscription: https://www.listbox.com/member/?member_id=21175430&id_secret=21175430-6a77cda4 Powered by Listbox: http://www.listbox.com
