Re: [zfs-discuss] Terrible ZFS performance on a Dell 1850 w/ PERC 4e/Si (Sol10U6)
I ran into this issue on my Dell 1850s also. You'll need to go into BIOS, change the PERC controller mode from RAID to SCSI and reboot. Then when the system comes back up it will give a warning about switching RAID controller modes and data loss (Have Backups)!. Then if you boot from DVD/Jumpstart you should see 2 disks and just do a ZFS 2 disk mirror for rpool. Hope this helps... - Robert Loper -- Forwarded message -- From: Andrei Ghimus ghi...@gmail.com To: zfs-discuss@opensolaris.org Date: Mon, 30 Aug 2010 11:05:27 PDT Subject: Re: [zfs-discuss] Terrible ZFS performance on a Dell 1850 w/ PERC 4e/Si (Sol10U6) I have the same problem you do, ZFS performance under Solaris 10 u8 is horrible. When you say passthrough mode, do you mean non-RAID configuration? And if so, could you tell me how you configured it? The best I can manage is to configure each physical drive as a RAID 0 array then export that as a logical drive. All tips/suggestions are appreciated. ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Terrible ZFS performance on a Dell 1850 w/ PERC 4e/Si (Sol10U6)
I have the same problem you do, ZFS performance under Solaris 10 u8 is horrible. When you say passthrough mode, do you mean non-RAID configuration? And if so, could you tell me how you configured it? The best I can manage is to configure each physical drive as a RAID 0 array then export that as a logical drive. All tips/suggestions are appreciated. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Terrible ZFS performance on a Dell 1850 w/ PERC 4e/Si (Sol10U6)
Is this minutes:seconds.millisecs ? if so, you're looking at 3-4MB/s .. I would say something is wrong. Ack, you're right. I was concentrating so much on the WTFOMG problem that I completely missed the WTF problem. In other news, with the Poweredge put into SCSI mode instead of RAID mode (which is in the system setup, NOT the Megaraid setup, which is why I missed it before) and addressing the disks directly, I can do that same 512M write in 9 seconds. I'm going to rebuild the box with ZFS boot/root and see how it behaves. I'm also going to ask the Linux admins and see if they're seeing problems. It's possible that nobody's noticed. (I tend to run the high I/O tasked machines, the rest are typically CPU-bound...) -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Terrible ZFS performance on a Dell 1850 w/ PERC 4e/Si (Sol10U6)
Before you do a dd test try first to do: echo zfs_vdev_max_pending/W0t1 | mdb -kw I did actually try this about a month ago when I first made an attempt at figuring this out. Changing the pending values did make some small difference, but even the best was far, far short of acceptable performance. In other news, switching to zfs boot+root, in a ZFS-mirrored configuration, in passthrough mode, has made a huge difference. Writing the 512M file took 15 minutes before... I'm down to twelve seconds. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Terrible ZFS performance on a Dell 1850 w/ PERC 4e/Si (Sol10U6)
I've got a mail machine here that I built using ZFS boot/root. It's been having some major I/O performance problems, which I posted once before... but that post seems to have disappeared. Now I've managed to obtain another identical machine, and I've built it in the same way as the original. Running Solaris 10 U6, I've got it fully patched as of 2009/10/06. It's using a mirrored disk via the PERC (LSI Megaraid) controller. The main problem seems to be ZFS. If I do the following on a UFS filesystem: # /usr/bin/time dd if=/dev/zero of=whee.bin bs=1024000 count=x ... then I get real times of the following: x time 128 35. 4 256 1:01.8 512 2:19.8 It's all very linear and fairly decent. However, if I then destroy that filesystem and recreate it using ZFS (no special options or kernel variables set) performance degrades substantially. With the same dd, I get: x time 128 3:45.3 256 6:52.7 512 15:40.4 So basically a 6.5x loss across the board. I realize that a simple 'dd' is an extremely weak test, but real-world use on these machines shows similar problems... long delays logging in, and running a command that isn't cached can take 20-30 seconds (even something as simple as 'psrinfo -vp'). Ironically, the machine works just fine for simple email, because the files are small and very transient and thus can exist quite easily just in memory. But more complex things, like a local copy of our mailmaps, cripples the machine. I'm about to rebuild the machine with the RAID controller in passthrough mode, and I'll see what that accomplishes. Most of the machines here are Linux and use the hardware RAID1, so I was/am hesitant to break standard that way. Does anyone have any experience or suggestions for trying to make ZFS boot+root work fine on this machine? -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Terrible ZFS performance on a Dell 1850 w/ PERC 4e/Si (Sol10U6)
On 09 October, 2009 - Brandon Hume sent me these 2,0K bytes:
I've got a mail machine here that I built using ZFS boot/root. It's
been having some major I/O performance problems, which I posted once
before... but that post seems to have disappeared.
Now I've managed to obtain another identical machine, and I've built
it in the same way as the original. Running Solaris 10 U6, I've got
it fully patched as of 2009/10/06. It's using a mirrored disk via the
PERC (LSI Megaraid) controller.
The main problem seems to be ZFS. If I do the following on a UFS filesystem:
# /usr/bin/time dd if=/dev/zero of=whee.bin bs=1024000 count=x
... then I get real times of the following:
x time
128 35. 4
256 1:01.8
512 2:19.8
Is this minutes:seconds.millisecs ? if so, you're looking at 3-4MB/s ..
I would say something is wrong.
It's all very linear and fairly decent.
Decent?!
However, if I then destroy that filesystem and recreate it using ZFS
(no special options or kernel variables set) performance degrades
substantially. With the same dd, I get:
x time
128 3:45.3
256 6:52.7
512 15:40.4
0.5MB/s .. that's floppy speed :P
So basically a 6.5x loss across the board. I realize that a simple
'dd' is an extremely weak test, but real-world use on these machines
shows similar problems... long delays logging in, and running a
command that isn't cached can take 20-30 seconds (even something as
simple as 'psrinfo -vp').
Ironically, the machine works just fine for simple email, because the
files are small and very transient and thus can exist quite easily
just in memory. But more complex things, like a local copy of our
mailmaps, cripples the machine.
.. because something is messed up, and for some reason ZFS seems to feel
worse than UFS..
I'm about to rebuild the machine with the RAID controller in
passthrough mode, and I'll see what that accomplishes. Most of the
machines here are Linux and use the hardware RAID1, so I was/am
hesitant to break standard that way. Does anyone have any
experience or suggestions for trying to make ZFS boot+root work fine
on this machine?
Check for instance 'iostat -xnzmp 1' while doing this and see if any
disk is behaving badly, high service times etc.. Even your speedy
3-4MB/s is nowhere close to what you should be getting, unless you've
connected a bunch of floppy drives to your PERC..
/Tomas
--
Tomas Ögren, st...@acc.umu.se, http://www.acc.umu.se/~stric/
|- Student at Computing Science, University of Umeå
`- Sysadmin at {cs,acc}.umu.se
___
zfs-discuss mailing list
zfs-discuss@opensolaris.org
http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Terrible ZFS performance on a Dell 1850 w/ PERC 4e/Si (Sol10U6)
I think the raid card is a re-branded LSI SCSI raid. I have LSI 21320-4x and having same problem with ZFS. Do you have BBU on the card? You may want to disable cache flush and zil and see how it works. I tried passthrough and basically the result is same. I gave up on tuning this card with ZFS, replaced the raid card with plain 21320 SCSI hba and everything works well since. I used ZFS mirror for OS, raidz for data pool. 2 days ago, one disk in my rpool mirror was dead and I was able to easily replace it. A few weeks ago, when I ran scrub, zfs found checksum error on the now dead disk and was able to fix it. I don't buy the idea on hardware raid with ZFS anymore, let ZFS handle it for you. -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Terrible ZFS performance on a Dell 1850 w/ PERC 4e/Si (Sol10U6)
Before you do a dd test try first to do: echo zfs_vdev_max_pending/W0t1 | mdb -kw and let us know if it helped or not. iostat -xnz 1 output while you are doing dd would also help. -- Robert Milkowski http://milek.blogspot.com ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
