Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-28 Thread Aubrey Li
It looks like I was posting on the wrong mailing list. I thought this mailing list includes developers. The experiment I did is not for commercial purpose. The purpose of comparison is to find the optimization opportunity of the entire software stack on both linux and solaris. As for this zfs root

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-27 Thread Hung-Sheng Tsao (Lao Tsao 老曹) Ph.D.
hi you did not answer the question, what is the RAM of the server? how many socket and core etc what is the block size of zfs? what is the cache ram of your san array? what is the block size/strip size of your raid in san array? raid 5 or what? what is your test program and how (from what

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-27 Thread Phil Harman
One of the glories of Solaris is that it is so very observable. Then there are the many excellent blog posts, wiki entries, and books - some or which are authored by contributors to this very thread - explaining how Solaris works. But these virtues are also a snare to some, and it is not uncomm

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-26 Thread Aubrey Li
On Tue, Mar 27, 2012 at 1:15 AM, Jim Klimov wrote: > Well, as a further attempt down this road, is it possible for you to rule > out > ZFS from swapping - i.e. if RAM amounts permit, disable the swap at all > (swap -d /dev/zvol/dsk/rpool/swap) or relocate it to dedicated slices of > same or better

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-26 Thread Jim Klimov
> > As a random guess, try pointing PHP tmp directory to > > /var/tmp (backed by zfs) and see if any behaviors change? > > > > Good luck, > > //Jim > > > > Thanks for your suggestions. Actually the default PHP tmp directory > was /var/tmp, and I changed "/var/tmp" to "/tmp". This reduced zfs > roo

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-26 Thread Richard Elling
I see nothing unusual in the lockstat data. I think you're barking up the wrong tree. -- richard On Mar 25, 2012, at 10:51 PM, Aubrey Li wrote: > On Mon, Mar 26, 2012 at 1:19 PM, Richard Elling > wrote: >> Apologies to the ZFSers, this thread really belongs elsewhere. >> >> Let me explain belo

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-26 Thread Aubrey Li
On Mon, Mar 26, 2012 at 8:24 PM, Jim Mauro wrote: > > You care about #2 and #3 because you are fixated on a ZFS root > lock contention problem, and not open to a broader discussion > about what your real problem actually is. I am not saying there is > not lock contention, and I am not saying there

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-26 Thread Aubrey Li
On Mon, Mar 26, 2012 at 7:28 PM, Jim Klimov wrote: > 2012-03-26 14:27, Aubrey Li wrote: >> >> The php temporary folder is set to /tmp, which is tmpfs. >> > > By the way, how much RAM does the box have available? > "tmpfs" in Solaris is backed by "virtual memory". > It is like a RAM disk, although

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-26 Thread Jim Mauro
You care about #2 and #3 because you are fixated on a ZFS root lock contention problem, and not open to a broader discussion about what your real problem actually is. I am not saying there is not lock contention, and I am not saying there is - I'll look at the data later carefully later when I hav

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-26 Thread Jim Klimov
2012-03-26 14:27, Aubrey Li wrote: The php temporary folder is set to /tmp, which is tmpfs. By the way, how much RAM does the box have available? "tmpfs" in Solaris is backed by "virtual memory". It is like a RAM disk, although maybe slower than ramdisk FS as seen in livecd, as long as there i

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-26 Thread Aubrey Li
On Mon, Mar 26, 2012 at 4:33 PM, wrote: > >>I'm migrating a webserver(apache+php) from RHEL to solaris. During the >>stress testing comparison, I found under the same session number of client >>request, CPU% is ~70% on RHEL while CPU% is full on solaris. > > Which version of Solaris is this? Thi

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-25 Thread Aubrey Li
On Mon, Mar 26, 2012 at 1:19 PM, Richard Elling wrote: > Apologies to the ZFSers, this thread really belongs elsewhere. > > Let me explain below: > > Root documentation path of apache is in zfs, you see > it at No.3 at the above dtrace report. > > > The sort is in reverse order. The large number y

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-25 Thread Fajar A. Nugraha
On Mon, Mar 26, 2012 at 12:19 PM, Richard Elling wrote: > Apologies to the ZFSers, this thread really belongs elsewhere. Some of the info in it is informative for other zfs users as well though :) > Here is the output, I changed to "tick-5sec" and "trunc(@, 5)". > > No.2 and No.3 is what I care

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-25 Thread Richard Elling
Apologies to the ZFSers, this thread really belongs elsewhere. On Mar 25, 2012, at 10:11 PM, Aubrey Li wrote: > On Mon, Mar 26, 2012 at 11:34 AM, Richard Elling > wrote: >> On Mar 25, 2012, at 6:51 PM, Aubrey Li wrote: >>> On Mon, Mar 26, 2012 at 4:18 AM, Jim Mauro wrote: If you're chasing

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-25 Thread Aubrey Li
On Mon, Mar 26, 2012 at 11:34 AM, Richard Elling wrote: > On Mar 25, 2012, at 6:51 PM, Aubrey Li wrote: >> On Mon, Mar 26, 2012 at 4:18 AM, Jim Mauro wrote: >>> If you're chasing CPU utilization, specifically %sys (time in the kernel), >>> I would start with a time-based kernel profile. >>> >>> #

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-25 Thread Richard Elling
On Mar 25, 2012, at 6:51 PM, Aubrey Li wrote: > On Mon, Mar 26, 2012 at 4:18 AM, Jim Mauro wrote: >> If you're chasing CPU utilization, specifically %sys (time in the kernel), >> I would start with a time-based kernel profile. >> >> #dtrace -n 'profile-997hz /arg0/ { @[stack()] = count(); } tick-

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-25 Thread Aubrey Li
On Mon, Mar 26, 2012 at 4:18 AM, Jim Mauro wrote: > If you're chasing CPU utilization, specifically %sys (time in the kernel), > I would start with a time-based kernel profile. > > #dtrace -n 'profile-997hz /arg0/ { @[stack()] = count(); } tick-60sec { > trunc(@, 20); printa(@0; }' > > I would be

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-25 Thread Aubrey Li
On Mon, Mar 26, 2012 at 3:22 AM, Fajar A. Nugraha wrote: >> >> I have ever not seen any issues until I did a comparison with Linux. > > So basically you're comparing linux + ext3/4 performance with solaris > + zfs, on the same hardware? That's not really fair, is it? > If your load is I/O-intensiv

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-25 Thread Jim Mauro
If you're chasing CPU utilization, specifically %sys (time in the kernel), I would start with a time-based kernel profile. #dtrace -n 'profile-997hz /arg0/ { @[stack()] = count(); } tick-60sec { trunc(@, 20); printa(@0; }' I would be curious to see where the CPU cycles are being consumed first,

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-25 Thread Fajar A. Nugraha
On Mon, Mar 26, 2012 at 2:13 AM, Aubrey Li wrote: >> The problem is, every zfs vnode access need the **same zfs root** >> lock. When the number of >> httpd processes and the corresponding kernel threads becomes large, >> this root lock contention >> becomes horrible. This situation does not occurs

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-25 Thread Aubrey Li
On Mon, Mar 26, 2012 at 2:58 AM, Richard Elling wrote: > On Mar 25, 2012, at 10:25 AM, Aubrey Li wrote: > > On Mon, Mar 26, 2012 at 12:48 AM, Richard Elling > wrote: > > This is the wrong forum for general purpose performance tuning. So I won't > > continue this much farther.  Notice the huge num

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-25 Thread Aubrey Li
On Mon, Mar 26, 2012 at 2:10 AM, zfs user wrote: > On 3/25/12 10:25 AM, Aubrey Li wrote: >> >> On Mon, Mar 26, 2012 at 12:48 AM, Richard Elling >>  wrote: >>> >>> This is the wrong forum for general purpose performance tuning. So I >>> won't >>> continue this much farther.  Notice the huge number

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-25 Thread Richard Elling
On Mar 25, 2012, at 10:25 AM, Aubrey Li wrote: > On Mon, Mar 26, 2012 at 12:48 AM, Richard Elling > wrote: >> This is the wrong forum for general purpose performance tuning. So I won't >> continue this much farther. Notice the huge number of icsw, that is a >> bigger >> symptom than locks. >> -

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-25 Thread zfs user
On 3/25/12 10:25 AM, Aubrey Li wrote: On Mon, Mar 26, 2012 at 12:48 AM, Richard Elling wrote: This is the wrong forum for general purpose performance tuning. So I won't continue this much farther. Notice the huge number of icsw, that is a bigger symptom than locks. -- richard thanks anywa

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-25 Thread Aubrey Li
On Mon, Mar 26, 2012 at 12:48 AM, Richard Elling wrote: > This is the wrong forum for general purpose performance tuning. So I won't > continue this much farther.  Notice the huge number of icsw, that is a > bigger > symptom than locks. >  -- richard thanks anyway, lock must be a problem. the sce

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-25 Thread Richard Elling
This is the wrong forum for general purpose performance tuning. So I won't continue this much farther. Notice the huge number of icsw, that is a bigger symptom than locks. -- richard On Mar 25, 2012, at 6:24 AM, Aubrey Li wrote: > SET minf mjf xcal intr ithr csw icsw migr smtx srw syscl us

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-25 Thread Aubrey Li
On Sun, Mar 25, 2012 at 3:55 PM, Richard Elling wrote: > On Mar 24, 2012, at 10:29 PM, Aubrey Li wrote: > > Hi, > > I'm migrating a webserver(apache+php) from RHEL to solaris. During the > stress testing comparison, I found under the same session number of client > request, CPU% is ~70% on RHEL wh

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-25 Thread Richard Elling
On Mar 24, 2012, at 10:29 PM, Aubrey Li wrote: > Hi, > > I'm migrating a webserver(apache+php) from RHEL to solaris. During the > stress testing comparison, I found under the same session number of client > request, CPU% is ~70% on RHEL while CPU% is full on solaris. > > After some investigation