Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-09-15 Thread Dale Ghent
On Sep 15, 2009, at 5:21 PM, Richard Elling wrote: On Sep 15, 2009, at 1:03 PM, Dale Ghent wrote: On Sep 10, 2009, at 3:12 PM, Rich Morris wrote: On 07/28/09 17:13, Rich Morris wrote: On Mon, Jul 20, 2009 at 7:52 PM, Bob Friesenhahn wrote: Sun has opened internal CR 6859997. It is now

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-09-15 Thread Richard Elling
Reference below... On Sep 15, 2009, at 2:38 PM, Dale Ghent wrote: On Sep 15, 2009, at 5:21 PM, Richard Elling wrote: On Sep 15, 2009, at 1:03 PM, Dale Ghent wrote: On Sep 10, 2009, at 3:12 PM, Rich Morris wrote: On 07/28/09 17:13, Rich Morris wrote: On Mon, Jul 20, 2009 at 7:52 PM, Bob

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-09-15 Thread Bob Friesenhahn
On Tue, 15 Sep 2009, Dale Ghent wrote: Question though... why is bug fix that can be a watershed for performance be held back for so long? s10u9 won't be available for at least 6 months from now, and with a huge environment, I try hard not to live off of IDRs. As someone who currently

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-09-15 Thread Dale Ghent
On Sep 15, 2009, at 6:28 PM, Bob Friesenhahn wrote: On Tue, 15 Sep 2009, Dale Ghent wrote: Question though... why is bug fix that can be a watershed for performance be held back for so long? s10u9 won't be available for at least 6 months from now, and with a huge environment, I try hard

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-09-15 Thread Bob Friesenhahn
On Tue, 15 Sep 2009, Dale Ghent wrote: As someone who currently faces kernel panics with recent U7+ kernel patches (on AMD64 and SPARC) related to PCI bus upset, I expect that Sun will take the time to make sure that the implementation is as good as it can be and is thoroughly tested before

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-09-13 Thread Christian Kendi
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 is already a diff for the source available? El Sep 11, 2009, a las 4:02 PM, Rich Morris escribió: On 09/10/09 16:22, en...@businessgrade.com wrote: Quoting Bob Friesenhahn bfrie...@simple.dallas.tx.us: On Thu, 10 Sep 2009, Rich Morris wrote:

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-09-11 Thread Rich Morris
On 09/10/09 16:22, en...@businessgrade.com wrote: Quoting Bob Friesenhahn bfrie...@simple.dallas.tx.us: On Thu, 10 Sep 2009, Rich Morris wrote: On 07/28/09 17:13, Rich Morris wrote: On Mon, Jul 20, 2009 at 7:52 PM, Bob Friesenhahn wrote: Sun has opened internal CR 6859997. It is now in

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-09-10 Thread Rich Morris
On 07/28/09 17:13, Rich Morris wrote: On Mon, Jul 20, 2009 at 7:52 PM, Bob Friesenhahn wrote: Sun has opened internal CR 6859997. It is now in Dispatched state at High priority. CR 6859997 has recently been fixed in Nevada. This fix will also be in Solaris 10 Update 9. This fix speeds

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-09-10 Thread Bob Friesenhahn
On Thu, 10 Sep 2009, Rich Morris wrote: On 07/28/09 17:13, Rich Morris wrote: On Mon, Jul 20, 2009 at 7:52 PM, Bob Friesenhahn wrote: Sun has opened internal CR 6859997. It is now in Dispatched state at High priority. CR 6859997 has recently been fixed in Nevada. This fix will also be in

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-09-10 Thread Rich Morris
On 09/10/09 16:17, Bob Friesenhahn wrote: On Thu, 10 Sep 2009, Rich Morris wrote: On 07/28/09 17:13, Rich Morris wrote: On Mon, Jul 20, 2009 at 7:52 PM, Bob Friesenhahn wrote: Sun has opened internal CR 6859997. It is now in Dispatched state at High priority. CR 6859997 has recently been

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-09-10 Thread Bob Friesenhahn
On Thu, 10 Sep 2009, Rich Morris wrote: Excellent. What level of read improvement are you seeing? Is the prefetch rate improved, or does the fix simply avoid losing the prefetch? This fix avoids using a prefetch stream when it is no longer valid. BTW, ZFS prefetch appears to work well

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-28 Thread Rich Morris
On Mon, Jul 20, 2009 at 7:52 PM, Bob Friesenhahn wrote: Sun has opened internal CR 6859997. It is now in Dispatched state at High priority. CR 6859997 has been accepted and is actively being worked on. The following info has been added to that CR: This is a problem with the ZFS file

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-28 Thread Bob Friesenhahn
On Tue, 28 Jul 2009, Rich Morris wrote: 6412053 is a related CR which mentions that the zfetch code may not be issuing I/O at a sufficient pace. This behavior is also seen on a Thumper running the test script in CR 6859997 since, even when prefetch is ramping up as expected, less than half

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-28 Thread Bob Friesenhahn
On Tue, 28 Jul 2009, Rich Morris wrote: The fix for this problem may be more feedback between the ARC and the zfetch code. Or it may make sense to restart the prefetch stream after some time has passed or perhaps whenever there's a miss on a block that was expected to have already been

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-22 Thread Brad Diggs
Have you considered running your script with ZFS pre-fetching disabled altogether to see if the results are consistent between runs? Brad Brad Diggs Senior Directory Architect Virtualization Architect xVM Technology Lead Sun Microsystems, Inc. Phone x52957/+1 972-992-0002 Mail

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-22 Thread Bob Friesenhahn
On Wed, 22 Jul 2009, Roch wrote: HI Bob did you consider running the 2 runs with echo zfs_prefetch_disable/W0t1 | mdb -kw and see if performance is constant between the 2 runs (and low). That would help clear the cause a bit. Sorry, I'd do it for you but since you have the setup etc...

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-20 Thread Marion Hakanson
bfrie...@simple.dallas.tx.us said: No. I am suggesting that all Solaris 10 (and probably OpenSolaris systems) currently have a software-imposed read bottleneck which places a limit on how well systems will perform on this simple sequential read benchmark. After a certain point (which is

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-20 Thread Bob Friesenhahn
On Mon, 20 Jul 2009, Marion Hakanson wrote: Bob, have you tried changing your benchmark to be multithreaded? It occurs to me that maybe a single cpio invocation is another bottleneck. I've definitely experienced the case where a single bonnie++ process was not enough to max out the storage

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-15 Thread Joerg Schilling
Richard Elling richard.ell...@gmail.com wrote: I think a picture is emerging that if you have enough RAM, the ARC is working very well. Which means that the ARC management is suspect. I propose the hypothesis that ARC misses are not prefetched. The first time through, prefetching works.

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-15 Thread Ross
Yes, that makes sense. For the first run, the pool has only just been mounted, so the ARC will be empty, with plenty of space for prefetching. On the second run however, the ARC is already full of the data that we just read, and I'm guessing that the prefetch code is less aggressive when there

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-15 Thread My D. Truong
It would be good to see results from a few OpenSolaris users running a recent 64-bit kernel, and with fast storage to see if this is an OpenSolaris issue as well. Bob, Here's an example of an OpenSolaris machine, 2008.11 upgraded to the 117 devel release. X4540, 32GB RAM. The file

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-15 Thread Bob Friesenhahn
On Wed, 15 Jul 2009, Ross wrote: Yes, that makes sense. For the first run, the pool has only just been mounted, so the ARC will be empty, with plenty of space for prefetching. I don't think that this hypothesis is quite correct. If you use 'zpool iostat' to monitor the read rate while

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-15 Thread Bob Friesenhahn
On Wed, 15 Jul 2009, My D. Truong wrote: Here's an example of an OpenSolaris machine, 2008.11 upgraded to the 117 devel release. X4540, 32GB RAM. The file count was bumped up to 9000 to be a little over double the RAM. Your timings show a 3.1X hit so it appears that the OpenSolaris

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-15 Thread Richard Elling
Bob Friesenhahn wrote: On Wed, 15 Jul 2009, Ross wrote: Yes, that makes sense. For the first run, the pool has only just been mounted, so the ARC will be empty, with plenty of space for prefetching. I don't think that this hypothesis is quite correct. If you use 'zpool iostat' to

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-15 Thread Bob Friesenhahn
On Wed, 15 Jul 2009, Richard Elling wrote: Unfortunately, zpool iostat doesn't really tell you anything about performance. All it shows is bandwidth. Latency is what you need to understand performance, so use iostat. You are still thinking about this as if it was a hardware-related problem

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-15 Thread Richard Elling
Bob Friesenhahn wrote: On Wed, 15 Jul 2009, Richard Elling wrote: Unfortunately, zpool iostat doesn't really tell you anything about performance. All it shows is bandwidth. Latency is what you need to understand performance, so use iostat. You are still thinking about this as if it was a

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-15 Thread Bob Friesenhahn
On Wed, 15 Jul 2009, Richard Elling wrote: heh. What you would be looking for is evidence of prefetching. If there is a lot of prefetching, the actv will tend to be high and latencies relatively low. If there is no prefetching, actv will be low and latencies may be higher. This also

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-15 Thread Ross
Aaah, ok, I think I understand now. Thanks Richard. I'll grab the updated test and have a look at the ARC ghost results when I get back to work tomorrow. -- This message posted from opensolaris.org ___ zfs-discuss mailing list

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-14 Thread Ross
Ok, build 117 does seem a lot better. The second run is slower, but not by such a huge margin. This was the end of the 98GB test: Creating data file set (12000 files of 8192000 bytes) under /rpool/zfscachetest ... Done! zfs unmount rpool/zfscachetest zfs mount rpool/zfscachetest Doing initial

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-14 Thread Ross
Jorgen, Am I right in thinking the numbers here don't quite work. 48M blocks is just 9,000 files isn't it, not 93,000? I'm asking because I had to repeat a test earlier - I edited the script with vi, but when I ran it, it was still using the old parameters. I ignored it as a one off, but

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-14 Thread Jorgen Lundman
I have no idea. I downloaded the script from Bob without modifications and ran it specifying only the name of our pool. Should I have changed something to run the test? We have two kinds of x4500/x4540, those with Sol 10 10/08, and 2 running svn117 for ZFS quotas. Worth trying on both?

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-14 Thread Ross
Aaah, nevermind, it looks like there's just a rogue 9 appeared in your output. It was just a standard run of 3,000 files. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-14 Thread Jorgen Lundman
Ah yes, my apologies! I haven't quite worked out why OsX VNC server can't handle keyboard mappings. I have to copy'paste @ even. As I pasted the output into my mail over VNC, it would have destroyed the (not very) unusual characters. Ross wrote: Aaah, nevermind, it looks like there's just

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-14 Thread Kurt Schreiner
On Tue, Jul 14, 2009 at 08:54:36AM +0200, Ross wrote: Ok, build 117 does seem a lot better. The second run is slower, but not by such a huge margin. Hm, I can't support this: SunOS fred 5.11 snv_117 sun4u sparc SUNW,Sun-Fire-V440 The system has 16GB of Ram, pool is mirrored over two

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-14 Thread Jorgen Lundman
I also ran this on my future RAID/NAS. Intel Atom 330 (D945GCLF2) dual core 1.6ghz, on a single HDD pool. svn_114, 64 bit, 2GB RAM. bash-3.23 ./zfs-cache-test.ksh zboot zfs create zboot/zfscachetest creating data file set (3000 files of 8192000 bytes) under /zboot/zfscachetest ... done1

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-14 Thread Ross
For what it's worth, I just repeated that test. The timings are suspiciously similar. This is very definitely a reproducible bug: zfs unmount rc-pool/zfscachetest zfs mount rc-pool/zfscachetest Doing initial (unmount/mount) 'cpio -o /dev/null' 48000247 blocks real4m45.69s user

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-14 Thread Bob Friesenhahn
Ross, Please refresh your test script from the source. The current script tells cpio to use 128k blocks and mentions the proper command in its progress message. I have now updated it to display useful information about the system being tested, and to dump the pool configuration. It is

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-14 Thread Bob Friesenhahn
On Tue, 14 Jul 2009, Jorgen Lundman wrote: I have no idea. I downloaded the script from Bob without modifications and ran it specifying only the name of our pool. Should I have changed something to run the test? If your system has quite a lot of memory, the number of files should be

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-14 Thread lists+zfs
On Tue, Jul 14, 2009 at 11:09:32AM -0500, Bob Friesenhahn wrote: On Tue, 14 Jul 2009, Jorgen Lundman wrote: I have no idea. I downloaded the script from Bob without modifications and ran it specifying only the name of our pool. Should I have changed something to run the test? If your

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-14 Thread Ross
Hi Bob, My guess is something like it's single threaded, with each file dealt with in order and requests being serviced by just one or two disks at a time. With that being the case, an x4500 is essentially just running off 7200 rpm SATA drives, which really is nothing special. A quick

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-14 Thread Angelo Rajadurai
Just FYI. I ran a slightly different version of the test. I used SSD (for log cache)! 3 x 32GB SSDs. 2 mirrored for log and one for cache. The systems is a 4150 with 12 GB of RAM. Here are the results $ pfexec ./zfs-cache-test.ksh sdpool System Configuration: System architecture: i386

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-14 Thread Bob Friesenhahn
On Tue, 14 Jul 2009, Ross wrote: My guess is something like it's single threaded, with each file dealt with in order and requests being serviced by just one or two disks at a time. With that being the case, an x4500 is essentially just running off 7200 rpm SATA drives, which really is

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-14 Thread Gaëtan Lehmann
Le 14 juil. 09 à 18:09, Bob Friesenhahn a écrit : On Tue, 14 Jul 2009, Jorgen Lundman wrote: I have no idea. I downloaded the script from Bob without modifications and ran it specifying only the name of our pool. Should I have changed something to run the test? If your system has quite

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-14 Thread Halldor Runar Haflidason
On Tue Jul 14, 2009 at 11:09:32AM -0500, Bob Friesenhahn wrote: On Tue, 14 Jul 2009, Jorgen Lundman wrote: I have no idea. I downloaded the script from Bob without modifications and ran it specifying only the name of our pool. Should I have changed something to run the test? If your

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-14 Thread Richard Elling
Bob Friesenhahn wrote: On Tue, 14 Jul 2009, Ross wrote: My guess is something like it's single threaded, with each file dealt with in order and requests being serviced by just one or two disks at a time. With that being the case, an x4500 is essentially just running off 7200 rpm SATA

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-14 Thread Bob Friesenhahn
On Tue, 14 Jul 2009, Richard Elling wrote: That is because file prefetch is dynamic. benr wrote a good blog on the subject and includes a DTrace script to monitor DMU prefetches. http://www.cuddletech.com/blog/pivot/entry.php?id=1040 Apparently not dynamic enough. The provided DTrace script

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-14 Thread Jakov Sosic
Hi! Do you think that this issues will be seen on a ZVOL-s that are exported as iSCSI tragets? -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-14 Thread Jorgen Lundman
3 servers contained within. Both x4500 and x4540 are setup the way Sun shipped to us. With minor changes (nfsservers=1024 etc). I was a little disappointed that they were identical in speed on round one, but the x4540 looked better part 2. Which I suspect is probably just OS version?

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-14 Thread Scott Lawson
I added a second Lun identical in size as a mirror and reran test. Results are more in line with yours now. ./zfs-cache-test.ksh test1 System Configuration: Sun Microsystems sun4u Sun SPARC Enterprise M3000 Server System architecture: sparc System release level: 5.10 Generic_139555-08 CPU

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-14 Thread Bob Friesenhahn
On Wed, 15 Jul 2009, Jorgen Lundman wrote: Doing initial (unmount/mount) 'cpio -C 131072 -o /dev/null' 48000256 blocks real3m1.58s user0m1.92s sys 0m56.67s Doing initial (unmount/mount) 'cpio -C 131072 -o /dev/null' 48000256 blocks real3m5.51s user0m1.70s sys

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-14 Thread Jorgen Lundman
You have some mighty pools there. Something I find quite interesting is that those who have mighty pools generally obtain about the same data rate regardless of their relative degree of excessive might. This causes me to believe that the Solaris kernel is throttling the read rate so that

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-14 Thread Bob Friesenhahn
On Wed, 15 Jul 2009, Scott Lawson wrote: NAME STATE READ WRITE CKSUM test1 ONLINE 0 0 0 mirror ONLINE 0 0 0

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-14 Thread Bob Friesenhahn
On Tue, 14 Jul 2009, Ross wrote: Hi Bob, My guess is something like it's single threaded, with each file dealt with in order and requests being serviced by just one or two disks at a time. With that being the case, an x4500 is essentially just running off 7200 rpm SATA drives, which really

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-14 Thread Bob Friesenhahn
On Wed, 15 Jul 2009, Jorgen Lundman wrote: You have some mighty pools there. Something I find quite interesting is that those who have mighty pools generally obtain about the same data rate regardless of their relative degree of excessive might. This causes me to believe that the Solaris

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-14 Thread Scott Lawson
Bob Friesenhahn wrote: On Wed, 15 Jul 2009, Scott Lawson wrote: NAME STATE READ WRITE CKSUM test1 ONLINE 0 0 0 mirror ONLINE 0 0

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-14 Thread Scott Lawson
This system has 32 GB of RAM so I will probbaly need to increase the data set size. [r...@x tmp]# ./zfs-cache-test.ksh nbupool System Configuration: Sun Microsystems sun4v SPARC Enterprise T5220 System architecture: sparc System release level: 5.10 Generic_141414-02 CPU ISA list:

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-14 Thread Richard Elling
I think a picture is emerging that if you have enough RAM, the ARC is working very well. Which means that the ARC management is suspect. I propose the hypothesis that ARC misses are not prefetched. The first time through, prefetching works. For the second pass, ARC misses are not prefetched,

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-13 Thread Gaëtan Lehmann
Hi, Here is the result on a Dell Precision T5500 with 24 GB of RAM and two HD in a mirror (SATA, 7200 rpm, NCQ). [glehm...@marvin2 tmp]$ uname -a SunOS marvin2 5.11 snv_117 i86pc i386 i86pc Solaris [glehm...@marvin2 tmp]$ pfexec ./zfs-cache-test.ksh zfs create rpool/zfscachetest Creating

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-13 Thread Alexander Skwar
Bob, On Sun, Jul 12, 2009 at 23:38, Bob Friesenhahnbfrie...@simple.dallas.tx.us wrote: There has been no forward progress on the ZFS read performance issue for a week now.  A 4X reduction in file read performance due to having read the file before is terrible, and of course the situation is

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-13 Thread Ross
Hey Bob, Here are my results on a Dual 2.2Ghz Opteron, 8GB of RAM and 16 SATA disks connected via a Supermicro AOC-SAT2-MV8 (albeit with one dead drive). Looks like a 5x slowdown to me: Doing initial (unmount/mount) 'cpio -o /dev/null' 48000247 blocks real4m46.45s user0m10.29s sys

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-13 Thread Daniel Rock
Hi, Solaris 10U7, patched to the latest released patches two weeks ago. Four ST31000340NS attached to two SI3132 SATA controller, RAIDZ1. Selfmade system with 2GB RAM and an x86 (chipid 0x0 AuthenticAMD family 15 model 35 step 2 clock 2210 MHz) AMD Athlon(tm) 64 X2 Dual Core

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-13 Thread Jorgen Lundman
x4540 running svn117 # ./zfs-cache-test.ksh zpool1 zfs create zpool1/zfscachetest creating data file set 93000 files of 8192000 bytes0 under /zpool1/zfscachetest ... done1 zfs unmount zpool1/zfscachetest zfs mount zpool1/zfscachetest doing initial (unmount/mount) 'cpio -o . /dev/null'

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-13 Thread Alexander Skwar
Here's a more useful output, with having set the number of files to 6000, so that it has a dataset which is larger than the amount of RAM. --($ ~)-- time sudo ksh zfs-cache-test.ksh zfs create rpool/zfscachetest Creating data file set (6000 files of 8192000 bytes) under /rpool/zfscachetest ...

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-13 Thread Bob Friesenhahn
On Mon, 13 Jul 2009, Alexander Skwar wrote: This is a M4000 mit 32 GB RAM and two HDs in a mirror. I think that you should edit the script to increase the file count since your RAM size is big enough to cache most of the data. Bob -- Bob Friesenhahn bfrie...@simple.dallas.tx.us,

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-13 Thread Bob Friesenhahn
On Mon, 13 Jul 2009, Alexander Skwar wrote: Still on S10 U7 Sparc M4000. So I'm now inline with the other results - the 2nd run is WAY slower. 4x as slow. It would be good to see results from a few OpenSolaris users running a recent 64-bit kernel, and with fast storage to see if this is an

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-13 Thread Ross
Interesting, I repeated the test on a few other machines running newer builds. First impressions are good: snv_114, virtual machine, 1GB RAM, 30GB disk - 16% slowdown. (Only 9GB free so I ran an 8GB test) Doing initial (unmount/mount) 'cpio -o /dev/null' 1683 blocks real3m4.85s user

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-13 Thread Brad Diggs
You might want to have a look at my blog on filesystem cache tuning... It will probably help you to avoid memory contention between the ARC and your apps. http://www.thezonemanager.com/2009/03/filesystem-cache-optimization.html Brad Brad Diggs Senior Directory Architect

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-13 Thread Bob Friesenhahn
On Mon, 13 Jul 2009, Brad Diggs wrote: You might want to have a look at my blog on filesystem cache tuning... It will probably help you to avoid memory contention between the ARC and your apps. http://www.thezonemanager.com/2009/03/filesystem-cache-optimization.html Your post makes it

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-13 Thread sean walmsley
Sun X4500 (thumper) with 16Gb of memory running Solaris 10 U6 with patches current to the end of Feb 2009. Current ARC size is ~6Gb. ZFS filesystem created in a ~3.2 Tb pool consisting of 7 sets of mirrored 500Gb SATA drives. I used 4000 8Mb files for a total of 32Gb. run 1: ~140M/s average

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-13 Thread Mike Gerdts
On Mon, Jul 13, 2009 at 9:34 AM, Bob Friesenhahnbfrie...@simple.dallas.tx.us wrote: On Mon, 13 Jul 2009, Alexander Skwar wrote: Still on S10 U7 Sparc M4000. So I'm now inline with the other results - the 2nd run is WAY slower. 4x as slow. It would be good to see results from a few

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-13 Thread Bob Friesenhahn
On Mon, 13 Jul 2009, Mike Gerdts wrote: FWIW, I hit another bug if I turn off primarycache. http://defect.opensolaris.org/bz/show_bug.cgi?id=10004 This causes really abysmal performance - but equally so for repeat runs! It is quite facinating seeing the huge difference in I/O performance

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-13 Thread Joerg Schilling
Bob Friesenhahn bfrie...@simple.dallas.tx.us wrote: On Mon, 13 Jul 2009, Mike Gerdts wrote: FWIW, I hit another bug if I turn off primarycache. http://defect.opensolaris.org/bz/show_bug.cgi?id=10004 This causes really abysmal performance - but equally so for repeat runs! It is

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-13 Thread Jim Mauro
Bob - Have you filed a bug on this issue? I am not up to speed on this thread, so I can not comment on whether or not there is a bug here, but you seem to have a test case and supporting data. Filing a bug will get the attention of ZFS engineering. Thanks, /jim Bob Friesenhahn wrote: On Mon,

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-13 Thread Bob Friesenhahn
On Mon, 13 Jul 2009, Joerg Schilling wrote: cpio reads/writes in 8192 byte chunks from the filesystem. Yes, I was just reading the cpio manual page and see that. I think that re-reading the 128K zfs block 16 times to satisfy each request for 8192 bytes explains the 16X performance loss

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-13 Thread Mike Gerdts
On Mon, Jul 13, 2009 at 3:16 PM, Joerg Schillingjoerg.schill...@fokus.fraunhofer.de wrote: Bob Friesenhahn bfrie...@simple.dallas.tx.us wrote: On Mon, 13 Jul 2009, Mike Gerdts wrote: FWIW, I hit another bug if I turn off primarycache.

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-13 Thread Mike Gerdts
On Mon, Jul 13, 2009 at 3:23 PM, Bob Friesenhahnbfrie...@simple.dallas.tx.us wrote: On Mon, 13 Jul 2009, Joerg Schilling wrote: cpio reads/writes in 8192 byte chunks from the filesystem. Yes, I was just reading the cpio manual page and see that.  I think that re-reading the 128K zfs block 16

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-13 Thread Ross Walker
On Jul 13, 2009, at 2:54 PM, Bob Friesenhahn bfrie...@simple.dallas.tx.us wrote: On Mon, 13 Jul 2009, Brad Diggs wrote: You might want to have a look at my blog on filesystem cache tuning... It will probably help you to avoid memory contention between the ARC and your apps.

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-13 Thread Bob Friesenhahn
On Mon, 13 Jul 2009, Mike Gerdts wrote: Using cpio's -C option seems to not change the behavior for this bug, but I did see a performance difference with the case where I hadn't modified the zfs caching behavior. That is, the performance of the tmpfs backed vdisk more than doubled with cpio -o

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-13 Thread Bob Friesenhahn
On Mon, 13 Jul 2009, Ross Walker wrote: Have you tried limiting the ARC so it doesn't squash the page cache? Yes, the ARC is limited to 10GB, leaving another 10GB for the OS and applications. Resource limits are not the problem. There is a ton of memory and CPU to go around. Current

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-13 Thread Mark Shellenbaum
Bob Friesenhahn wrote: There has been no forward progress on the ZFS read performance issue for a week now. A 4X reduction in file read performance due to having read the file before is terrible, and of course the situation is considerably worse if the file was previously mmapped as well.

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-13 Thread Joerg Schilling
Bob Friesenhahn bfrie...@simple.dallas.tx.us wrote: On Mon, 13 Jul 2009, Joerg Schilling wrote: cpio reads/writes in 8192 byte chunks from the filesystem. Yes, I was just reading the cpio manual page and see that. I think that re-reading the 128K zfs block 16 times to satisfy each

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-13 Thread Joerg Schilling
Mike Gerdts mger...@gmail.com wrote: Using cpio's -C option seems to not change the behavior for this bug, but I did see a performance difference with the case where I hadn't modified the zfs caching behavior. That is, the performance of the tmpfs backed vdisk more than doubled with cpio -o

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-13 Thread Joerg Schilling
Bob Friesenhahn bfrie...@simple.dallas.tx.us wrote: On Mon, 13 Jul 2009, Mike Gerdts wrote: Using cpio's -C option seems to not change the behavior for this bug, but I did see a performance difference with the case where I hadn't modified the zfs caching behavior. That is, the

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-13 Thread Bob Friesenhahn
On Mon, 13 Jul 2009, Jim Mauro wrote: Bob - Have you filed a bug on this issue? I am not up to speed on this thread, so I can not comment on whether or not there is a bug here, but you seem to have a test case and supporting data. Filing a bug will get the attention of ZFS engineering. No,

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-13 Thread Mike Gerdts
On Mon, Jul 13, 2009 at 4:41 PM, Bob Friesenhahnbfrie...@simple.dallas.tx.us wrote: On Mon, 13 Jul 2009, Jim Mauro wrote: Bob - Have you filed a bug on this issue? I am not up to speed on this thread, so I can not comment on whether or not there is a bug here, but you seem to have a test case

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-13 Thread Bob Friesenhahn
On Mon, 13 Jul 2009, Mark Shellenbaum wrote: I've opened the following bug to track this issue: 6859997 zfs caching performance problem We need to track down if/when this problem was introduced or if it has always been there. I think that it has always been there as long as I have been

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-13 Thread Bob Friesenhahn
On Mon, 13 Jul 2009, Joerg Schilling wrote: If you continue to use cpio and the cpio archive format, you force copying a lot of data as the cpio archive format does use odd header sizes and starts new files unaligned directly after the archive header. Note that the output of cpio is sent to

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-13 Thread Randy Jones
Bob: Sun v490, 4x1.35 processors, 32GB ram, Solaris 10u7 working with a raidz1 zpool made up of 6x146 sas drives on a j4200. Results of your running your script: # zfs-cache-test.ksh pool2 zfs create pool2/zfscachetest Creating data file set (6000 files of 8192000 bytes) under

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-12 Thread Bob Friesenhahn
There has been no forward progress on the ZFS read performance issue for a week now. A 4X reduction in file read performance due to having read the file before is terrible, and of course the situation is considerably worse if the file was previously mmapped as well. Many of us have sent a

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-12 Thread Scott Lawson
Bob, Output of my run for you. System is a M3000 with 16 GB RAM and 1 zpool called test1 which is contained on a raid 1 volume on a 6140 with 7.50.13.10 firmware on the RAID controllers. RAid 1 is made up of two 146GB 15K FC disks. This machine is brand new with a clean install of S10 05/09.

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-09 Thread William Bauer
I have a much more generic question regarding this thread. I have a sun T5120 (T2 quad core, 1.4GHz) with two 10K RPM SAS drives in a mirrored pool running Solaris 10 u7. The disk performance seems horrible. I have the same apps running on a Sun X2100M2 (dual core 1.8GHz AMD) also running

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-09 Thread William Bauer
I don't swear. The word it bleeped was not a bad word -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-07 Thread Lejun Zhu
If cpu seems to be idle, the tool latencytop probably can give you some clue. It's developed for OpenSolaris but Solaris 10 should work too (with glib 2.14 installed). You can get a copy of v0.1 at http://opensolaris.org/os/project/latencytop/ To use latencytop, open a terminal and start

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-07 Thread James Andrewartha
Joerg Schilling wrote: I would be interested to see a open(2) flag that tells the system that I will read a file that I opened exactly once in native oder. This could tell the system to do read ahead and to later mark the pages as immediately reusable. This would make star even faster than

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-07 Thread Joerg Schilling
James Andrewartha jam...@daa.com.au wrote: Joerg Schilling wrote: I would be interested to see a open(2) flag that tells the system that I will read a file that I opened exactly once in native oder. This could tell the system to do read ahead and to later mark the pages as immediately

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-07 Thread Gary Mills
On Mon, Jul 06, 2009 at 04:54:16PM +0100, Andrew Gabriel wrote: Andre van Eyssen wrote: On Mon, 6 Jul 2009, Gary Mills wrote: As for a business case, we just had an extended and catastrophic performance degradation that was the result of two ZFS bugs. If we have another one like that, our

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-07 Thread Bob Friesenhahn
On Tue, 7 Jul 2009, Joerg Schilling wrote: posix_fadvise seems to be _very_ new for Solaris and even though I am frequently reading/writing the POSIX standards mailing list, I was not aware of it. From my tests with star, I cannot see a significant performance increase but it may have a 3%

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-07 Thread Joerg Schilling
Bob Friesenhahn bfrie...@simple.dallas.tx.us wrote: On Tue, 7 Jul 2009, Joerg Schilling wrote: posix_fadvise seems to be _very_ new for Solaris and even though I am frequently reading/writing the POSIX standards mailing list, I was not aware of it. From my tests with star, I

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-07 Thread Bob Friesenhahn
On Tue, 7 Jul 2009, Joerg Schilling wrote: Based on the prior discussions of using mmap() with ZFS and the way ZFS likes to work, my guess is that POSIX_FADV_NOREUSE does nothing at all and POSIX_FADV_DONTNEED probably does not work either. These are pretty straightforward to implement with

Re: [zfs-discuss] Why is Solaris 10 ZFS performance so terrible?

2009-07-06 Thread Boyd Adamson
Phil Harman phil.har...@sun.com writes: Gary Mills wrote: The Solaris implementation of mmap(2) is functionally correct, but the wait for a 64 bit address space rather moved the attention of performance tuning elsewhere. I must admit I was surprised to see so much code out there that still

  1   2   >