[zfs-discuss] Terrible performance when setting zfs_arc_max snv_98
Hi there. I just got a new Adaptec RAID 51645 controller in because the old (other type) was malfunctioning. It is paired with 16 Seagate 15k5 disks, of which two are used with hardware RAID 1 for OpenSolaris snv_98, and the rest is configured as striped mirrors as a zpool. I created a zfs filesystem on this pool with a blocksize of 8K. This server has 64GB of memory and will be running postgreSQL, so we need to cut down ARC memory usage. But before I do this I tested the zfs performance using iometer (it was a bit tricky getting it to compile but it's running). So far so good. Figures look very promissing, with stagering random read and write figures! There are just a few problems: every few seconds, disk LED's stop working for a few seconds, except one disk at a time. When this cycle is finished, it looks normal again. This seems to be the flushing of the NVRAM cache. Should be solved by disabling the flush...So far so good... But I also need the memory for PostgreSQL to work, so I added: set zfs:zfs_arc_max=8589934592 to /etc/system and rebooted. Now redid my test, with terrible results.. sequential read is about 120 MB/sec. One disk should be able to handle that, 14 disks should do more than a GB/sec, and in my previous benchmark without the arc_max setting, they actually make these figures...(even when not reading from ARC cache). Random figures are not much better than this. So something is clearly wrong here... Can anyone comment? -- This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Some basic questions about getting the best performance for database usage
Let ZFS deal with the redundancy part. I'm not counting redundancy offered by traditional RAID as you can see by just posts in this forums that - 1. It doesn't work. 2. It bites when you least expect it to. 3. You can do nothing but resort to tapes and LOT of aspirin when you get bitten. Thanks, that's exactly what I was asking about. This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Some basic questions about getting the best performance for database usage
Why not go to 128-256 GBytes of RAM? It isn't that expensive and would significantly help give you a big performance boost ;-) Would be nice, but it not that much inexpensive since we'd have to move up a class in server choise, and besides the extra memory cost, also brings some more money with it. The database transaction log should be relatively small, so I would look for two LUNs (disks), mirrored. Similarly, the ZIL should be relatively small -- two LUNs (disks), mirrored. You will want ZFS to manage the redundancy here, so think about mirroring at the ZFS level. The actual size needed will be based on the transaction load which causes writes. For ZIL sizing, we like to see something like 20 seconds worth of write workload. In most cases, this will fit into the write cache of a decent array, so you may not have to burn an actual pair of disks in the backing store. But since I don't now the array your using, it will be difficult to be specific. Oka, so if the array cache is large enough, there is no actual need for a seperate ZIL disk. Another consideration could be the use of SSD's for all of the stuff. You'll only need a few of these to have by far beter IO performance than the 16 SAS disks could ever do. Also, you'd probably not need a ZIL disk, nor a disk for the transaction log. It will cost about the same, but will probably give better performance This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
[zfs-discuss] Some basic questions about getting the best performance for database usage
I'm new so opensolaris and very new to ZFS. In the past we have always used linux for our database backends. So now we are looking for a new database server to give us a big performance boost, and also the possibility for scalability. Our current database consists mainly of a huge table containing about 230 million records and a few (relatively) smaller tables (something like 13 million records ans less). The main table is growing with about 800k records every day, and the prognosis is that this number will increase significantly in the near future. All of this is currently held in a Postgresql database with the largest tables divided into segments to speed up performance. This all runs on a linux machine with 4 GB of RAM and 4 10K SCSI disks in HW raid 10. The complete database is about 70 Gb in size, and growing every day. We will soon need hew hardware, and are also reviewing software needs. Besides a lot more RAM (16 or 32GB), the new machine will also get a much lager disk array. We don't need the size, but we do need the IO it can generate. And what we also need is it beeing able to scale. When needs grow, it should be possible to add more disks to be able to handle the extra IO. And that is exactly where ZFS comes in, at least as far as I read. The question is: how can we maximize IO by using the best possible combination of hardware and ZFS RAID? I'll probably be having 16 Seagate 15K5 SAS disks, 150 GB each. Two in HW raid1 for the OS, two in HW raid 1 or 10 for the transaction log. The OS does not need to be on ZFS, but could be. So that leaves 10 or 12 disks to configure for the database. The question is how to divide them to get the best IO performance by mixing the best of both worlds. For what I read, mirroring and striping should get me better performance than raidz of RAID5. But I guess you might give me some pointer on how to distribute the disk. My biggest question is what I should leave to the HW raid, and what to ZFS? This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Some basic questions about getting the best performance for database usage
Christiaan, As ZFS tuning has already been suggested, remember: a) Never tune unless you need to. b) Never tune unless you have an untuned benchmark set of figures to compare against after the system has been tuned - especially in ZFS-land which, whilst it may not be quite there, is designed to ultimately be self-tuning. Putting stuff hard into /etc/system might be counter-productive to performance in the future (although hopefully by that time, it will be blithely ignored). c) Never tune more than one parameter at one go. d) Understand as fully as possible the wider ramifications of any tuning that you undertake. If this is all master of the bleedin' obvious to you, then please accept my humble apologies - it is often not the case... Regards... Sean. This is not about tuning, but about choosing the right configuration from the start. I can surely buy the stuff and spend day's testing all kinds of scenario's to come to the best possible configuration. That all great, but I'd like to start these test with a bit of decent background so I know what to expect. So right now, I'm not babling about some ZFS tuning setting, but about the advantages and disadvantages of using ZFS, hardware raid, or a combination of the two. This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
Re: [zfs-discuss] Some basic questions about getting the best performance for database usage
Another thing: what about a seperate disk (or disk set) for the ZIL? Would it be worth sacrificing two SAS disks for two SSD disks in raid 1 handling the ZIL? This message posted from opensolaris.org ___ zfs-discuss mailing list zfs-discuss@opensolaris.org http://mail.opensolaris.org/mailman/listinfo/zfs-discuss
