Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-28 Thread Aubrey Li
It looks like I was posting on the wrong mailing list. I thought this mailing list includes developers. The experiment I did is not for commercial purpose. The purpose of comparison is to find the optimization opportunity of the entire software stack on both linux and solaris. As for this zfs

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-27 Thread Phil Harman
One of the glories of Solaris is that it is so very observable. Then there are the many excellent blog posts, wiki entries, and books - some or which are authored by contributors to this very thread - explaining how Solaris works. But these virtues are also a snare to some, and it is not

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-27 Thread Hung-Sheng Tsao (Lao Tsao 老曹) Ph.D.
hi you did not answer the question, what is the RAM of the server? how many socket and core etc what is the block size of zfs? what is the cache ram of your san array? what is the block size/strip size of your raid in san array? raid 5 or what? what is your test program and how (from what

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-26 Thread Aubrey Li
On Mon, Mar 26, 2012 at 4:33 PM, casper@oracle.com wrote: I'm migrating a webserver(apache+php) from RHEL to solaris. During the stress testing comparison, I found under the same session number of client request, CPU% is ~70% on RHEL while CPU% is full on solaris. Which version of Solaris

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-26 Thread Jim Klimov
2012-03-26 14:27, Aubrey Li wrote: The php temporary folder is set to /tmp, which is tmpfs. By the way, how much RAM does the box have available? tmpfs in Solaris is backed by virtual memory. It is like a RAM disk, although maybe slower than ramdisk FS as seen in livecd, as long as there is

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-26 Thread Jim Mauro
You care about #2 and #3 because you are fixated on a ZFS root lock contention problem, and not open to a broader discussion about what your real problem actually is. I am not saying there is not lock contention, and I am not saying there is - I'll look at the data later carefully later when I

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-26 Thread Aubrey Li
On Mon, Mar 26, 2012 at 7:28 PM, Jim Klimov jimkli...@cos.ru wrote: 2012-03-26 14:27, Aubrey Li wrote: The php temporary folder is set to /tmp, which is tmpfs. By the way, how much RAM does the box have available? tmpfs in Solaris is backed by virtual memory. It is like a RAM disk,

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-26 Thread Aubrey Li
On Mon, Mar 26, 2012 at 8:24 PM, Jim Mauro james.ma...@oracle.com wrote: You care about #2 and #3 because you are fixated on a ZFS root lock contention problem, and not open to a broader discussion about what your real problem actually is. I am not saying there is not lock contention, and I

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-26 Thread Richard Elling
I see nothing unusual in the lockstat data. I think you're barking up the wrong tree. -- richard On Mar 25, 2012, at 10:51 PM, Aubrey Li wrote: On Mon, Mar 26, 2012 at 1:19 PM, Richard Elling richard.ell...@richardelling.com wrote: Apologies to the ZFSers, this thread really belongs

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-26 Thread Jim Klimov
As a random guess, try pointing PHP tmp directory to /var/tmp (backed by zfs) and see if any behaviors change? Good luck, //Jim Thanks for your suggestions. Actually the default PHP tmp directory was /var/tmp, and I changed /var/tmp to /tmp. This reduced zfs root lock contention

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-26 Thread Aubrey Li
On Tue, Mar 27, 2012 at 1:15 AM, Jim Klimov j...@cos.ru wrote: Well, as a further attempt down this road, is it possible for you to rule out ZFS from swapping - i.e. if RAM amounts permit, disable the swap at all (swap -d /dev/zvol/dsk/rpool/swap) or relocate it to dedicated slices of same or

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-25 Thread Richard Elling
On Mar 24, 2012, at 10:29 PM, Aubrey Li wrote: Hi, I'm migrating a webserver(apache+php) from RHEL to solaris. During the stress testing comparison, I found under the same session number of client request, CPU% is ~70% on RHEL while CPU% is full on solaris. After some investigation, zfs

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-25 Thread Aubrey Li
On Sun, Mar 25, 2012 at 3:55 PM, Richard Elling richard.ell...@richardelling.com wrote: On Mar 24, 2012, at 10:29 PM, Aubrey Li wrote: Hi, I'm migrating a webserver(apache+php) from RHEL to solaris. During the stress testing comparison, I found under the same session number of client

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-25 Thread Richard Elling
This is the wrong forum for general purpose performance tuning. So I won't continue this much farther. Notice the huge number of icsw, that is a bigger symptom than locks. -- richard On Mar 25, 2012, at 6:24 AM, Aubrey Li wrote: SET minf mjf xcal intr ithr csw icsw migr smtx srw syscl

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-25 Thread Aubrey Li
On Mon, Mar 26, 2012 at 12:48 AM, Richard Elling richard.ell...@richardelling.com wrote: This is the wrong forum for general purpose performance tuning. So I won't continue this much farther.  Notice the huge number of icsw, that is a bigger symptom than locks.  -- richard thanks anyway,

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-25 Thread zfs user
On 3/25/12 10:25 AM, Aubrey Li wrote: On Mon, Mar 26, 2012 at 12:48 AM, Richard Elling richard.ell...@richardelling.com wrote: This is the wrong forum for general purpose performance tuning. So I won't continue this much farther. Notice the huge number of icsw, that is a bigger symptom than

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-25 Thread Richard Elling
On Mar 25, 2012, at 10:25 AM, Aubrey Li wrote: On Mon, Mar 26, 2012 at 12:48 AM, Richard Elling richard.ell...@richardelling.com wrote: This is the wrong forum for general purpose performance tuning. So I won't continue this much farther. Notice the huge number of icsw, that is a bigger

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-25 Thread Aubrey Li
On Mon, Mar 26, 2012 at 2:10 AM, zfs user zf...@itsbeen.sent.com wrote: On 3/25/12 10:25 AM, Aubrey Li wrote: On Mon, Mar 26, 2012 at 12:48 AM, Richard Elling richard.ell...@richardelling.com  wrote: This is the wrong forum for general purpose performance tuning. So I won't continue this

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-25 Thread Aubrey Li
On Mon, Mar 26, 2012 at 2:58 AM, Richard Elling richard.ell...@richardelling.com wrote: On Mar 25, 2012, at 10:25 AM, Aubrey Li wrote: On Mon, Mar 26, 2012 at 12:48 AM, Richard Elling richard.ell...@richardelling.com wrote: This is the wrong forum for general purpose performance tuning. So I

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-25 Thread Fajar A. Nugraha
On Mon, Mar 26, 2012 at 2:13 AM, Aubrey Li aubrey...@gmail.com wrote: The problem is, every zfs vnode access need the **same zfs root** lock. When the number of httpd processes and the corresponding kernel threads becomes large, this root lock contention becomes horrible. This situation does

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-25 Thread Jim Mauro
If you're chasing CPU utilization, specifically %sys (time in the kernel), I would start with a time-based kernel profile. #dtrace -n 'profile-997hz /arg0/ { @[stack()] = count(); } tick-60sec { trunc(@, 20); printa(@0; }' I would be curious to see where the CPU cycles are being consumed first,

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-25 Thread Aubrey Li
On Mon, Mar 26, 2012 at 3:22 AM, Fajar A. Nugraha w...@fajar.net wrote: I have ever not seen any issues until I did a comparison with Linux. So basically you're comparing linux + ext3/4 performance with solaris + zfs, on the same hardware? That's not really fair, is it? If your load is

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-25 Thread Aubrey Li
On Mon, Mar 26, 2012 at 4:18 AM, Jim Mauro james.ma...@oracle.com wrote: If you're chasing CPU utilization, specifically %sys (time in the kernel), I would start with a time-based kernel profile. #dtrace -n 'profile-997hz /arg0/ { @[stack()] = count(); } tick-60sec { trunc(@, 20); printa(@0;

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-25 Thread Richard Elling
On Mar 25, 2012, at 6:51 PM, Aubrey Li wrote: On Mon, Mar 26, 2012 at 4:18 AM, Jim Mauro james.ma...@oracle.com wrote: If you're chasing CPU utilization, specifically %sys (time in the kernel), I would start with a time-based kernel profile. #dtrace -n 'profile-997hz /arg0/ { @[stack()] =

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-25 Thread Aubrey Li
On Mon, Mar 26, 2012 at 11:34 AM, Richard Elling richard.ell...@gmail.com wrote: On Mar 25, 2012, at 6:51 PM, Aubrey Li wrote: On Mon, Mar 26, 2012 at 4:18 AM, Jim Mauro james.ma...@oracle.com wrote: If you're chasing CPU utilization, specifically %sys (time in the kernel), I would start with

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-25 Thread Richard Elling
Apologies to the ZFSers, this thread really belongs elsewhere. On Mar 25, 2012, at 10:11 PM, Aubrey Li wrote: On Mon, Mar 26, 2012 at 11:34 AM, Richard Elling richard.ell...@gmail.com wrote: On Mar 25, 2012, at 6:51 PM, Aubrey Li wrote: On Mon, Mar 26, 2012 at 4:18 AM, Jim Mauro

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-25 Thread Fajar A. Nugraha
On Mon, Mar 26, 2012 at 12:19 PM, Richard Elling richard.ell...@richardelling.com wrote: Apologies to the ZFSers, this thread really belongs elsewhere. Some of the info in it is informative for other zfs users as well though :) Here is the output, I changed to tick-5sec and trunc(@, 5). No.2

Re: [zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-25 Thread Aubrey Li
On Mon, Mar 26, 2012 at 1:19 PM, Richard Elling richard.ell...@richardelling.com wrote: Apologies to the ZFSers, this thread really belongs elsewhere. Let me explain below: Root documentation path of apache is in zfs, you see it at No.3 at the above dtrace report. The sort is in reverse

[zfs-discuss] webserver zfs root lock contention under heavy load

2012-03-24 Thread Aubrey Li
Hi, I'm migrating a webserver(apache+php) from RHEL to solaris. During the stress testing comparison, I found under the same session number of client request, CPU% is ~70% on RHEL while CPU% is full on solaris. After some investigation, zfs root lock emerges as a major doubtful point. Firstly,