Re: [PERFORM] Testing FusionIO
Greg, Did you ever contact them and get your hands on one? We eventually did see long SSD rebuild times on server crash as well. But data came back uncorrupted per my blog post. This is a good case for Slony Slaves. Anyone in a high TX low downtime environment would have already engineered around needing to wait for rebuild/recover times anyway. So it's not a deal killer in my view. -kg On Mar 8, 2010, at 12:50 PM, Greg Smith wrote: Ben Chobot wrote: We've enjoyed our FusionIO drives very much. They can do 100k iops without breaking a sweat. Just make sure you shut them down cleanly - it can up to 30 minutes per card to recover from a crash/plug pull test. Yeah...I got into an argument with Kenny Gorman over my concerns with how they were handling durability issues on his blog, the reading I did about them never left me satisfied Fusion was being completely straight with everyone about this area: http://www.kennygorman.com/wordpress/?p=398 If it takes 30 minutes to recover, but it does recover, I guess that's better than I feared was the case with them. Thanks for reporting the plug pull tests--I don't trust any report from anyone about new storage hardware that doesn't include that little detail as part of the testing. You're just asking to have your data get lost without that basic due diligence, and I'm sure not going to even buy eval hardware from a vendor that appears evasive about it. There's a reason I don't personally own any SSD hardware yet. -- Greg Smith 2ndQuadrant US Baltimore, MD PostgreSQL Training, Services and Support g...@2ndquadrant.com www.2ndQuadrant.us -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] SSD + RAID
I found a bit of time to play with this. I started up a test with 20 concurrent processes all inserting into the same table and committing after each insert. The db was achieving about 5000 inserts per second, and I kept it running for about 10 minutes. The host was doing about 5MB/s of Physical I/O to the Fusion IO drive. I set checkpoint segments very small (10). I observed the following message in the log: checkpoints are occurring too frequently (16 seconds apart). Then I pulled the cord. On reboot I noticed that Fusion IO replayed it's log, then the filesystem (vxfs) did the same. Then I started up the DB and observed the it perform auto-recovery: Nov 18 14:33:53 frutestdb002 postgres[5667]: [6-1] 2009-11-18 14:33:53 PSTLOG: database system was not properly shut down; automatic recovery in progress Nov 18 14:33:53 frutestdb002 postgres[5667]: [7-1] 2009-11-18 14:33:53 PSTLOG: redo starts at 2A/55F9D478 Nov 18 14:33:54 frutestdb002 postgres[5667]: [8-1] 2009-11-18 14:33:54 PSTLOG: record with zero length at 2A/56692F38 Nov 18 14:33:54 frutestdb002 postgres[5667]: [9-1] 2009-11-18 14:33:54 PSTLOG: redo done at 2A/56692F08 Nov 18 14:33:54 frutestdb002 postgres[5667]: [10-1] 2009-11-18 14:33:54 PSTLOG: database system is ready Thanks Kenny On Nov 13, 2009, at 1:35 PM, Kenny Gorman wrote: The FusionIO products are a little different. They are card based vs trying to emulate a traditional disk. In terms of volatility, they have an on-board capacitor that allows power to be supplied until all writes drain. They do not have a cache in front of them like a disk-type SSD might. I don't sell these things, I am just a fan. I verified all this with the Fusion IO techs before I replied. Perhaps older versions didn't have this functionality? I am not sure. I have already done some cold power off tests w/o problems, but I could up the workload a bit and retest. I will do a couple of 'pull the cable' tests on monday or tuesday and report back how it goes. Re the performance #'s... Here is my post: http://www.kennygorman.com/wordpress/?p=398 -kg In order for a drive to work reliably for database use such as for PostgreSQL, it cannot have a volatile write cache. You either need a write cache with a battery backup (and a UPS doesn't count), or to turn the cache off. The SSD performance figures you've been looking at are with the drive's write cache turned on, which means they're completely fictitious and exaggerated upwards for your purposes. In the real world, that will result in database corruption after a crash one day. No one on the drive benchmarking side of the industry seems to have picked up on this, so you can't use any of those figures. I'm not even sure right now whether drives like Intel's will even meet their lifetime expectations if they aren't allowed to use their internal volatile write cache. Here's two links you should read and then reconsider your whole design: http://www.mysqlperformanceblog.com/2009/03/02/ssd-xfs-lvm-fsync-write-cache-barrier-and-lost-transactions/ http://petereisentraut.blogspot.com/2009/07/solid-state-drive-benchmarks-and-write.html I can't even imagine how bad the situation would be if you decide to wander down the use a bunch of really cheap SSD drives path; these things are barely usable for databases with Intel's hardware. The needs of people who want to throw SSD in a laptop and those of the enterprise database market are really different, and if you believe doom forecasting like the comments at http://blogs.sun.com/BestPerf/entry/oracle_peoplesoft_payroll_sun_sparc that gap is widening, not shrinking. -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] SSD + RAID
The FusionIO products are a little different. They are card based vs trying to emulate a traditional disk. In terms of volatility, they have an on-board capacitor that allows power to be supplied until all writes drain. They do not have a cache in front of them like a disk-type SSD might. I don't sell these things, I am just a fan. I verified all this with the Fusion IO techs before I replied. Perhaps older versions didn't have this functionality? I am not sure. I have already done some cold power off tests w/o problems, but I could up the workload a bit and retest. I will do a couple of 'pull the cable' tests on monday or tuesday and report back how it goes. Re the performance #'s... Here is my post: http://www.kennygorman.com/wordpress/?p=398 -kg In order for a drive to work reliably for database use such as for PostgreSQL, it cannot have a volatile write cache. You either need a write cache with a battery backup (and a UPS doesn't count), or to turn the cache off. The SSD performance figures you've been looking at are with the drive's write cache turned on, which means they're completely fictitious and exaggerated upwards for your purposes. In the real world, that will result in database corruption after a crash one day. No one on the drive benchmarking side of the industry seems to have picked up on this, so you can't use any of those figures. I'm not even sure right now whether drives like Intel's will even meet their lifetime expectations if they aren't allowed to use their internal volatile write cache. Here's two links you should read and then reconsider your whole design: http://www.mysqlperformanceblog.com/2009/03/02/ssd-xfs-lvm-fsync-write-cache-barrier-and-lost-transactions/ http://petereisentraut.blogspot.com/2009/07/solid-state-drive-benchmarks-and-write.html I can't even imagine how bad the situation would be if you decide to wander down the use a bunch of really cheap SSD drives path; these things are barely usable for databases with Intel's hardware. The needs of people who want to throw SSD in a laptop and those of the enterprise database market are really different, and if you believe doom forecasting like the comments at http://blogs.sun.com/BestPerf/entry/oracle_peoplesoft_payroll_sun_sparc that gap is widening, not shrinking.
Re: [PERFORM] Putting tables or indexes in SSD or RAM: avoiding double caching?
Well for one thing on the IODrive. Be sure to use a FS that supports direct IO so you don't cache it on the FS level and thus take room an object not on SSD could use. We use vxfs with mincache=direct as our filesystem for just this reason. Also, there is an IO drive tuning manual that discusses the same. It's a good read if you don't already have it. I do not know of a way to partition the PG cache other than make it small and use the FS controls to force direct IO. 2.5 cents.. -kg -Original Message- From: pgsql-performance-ow...@postgresql.org on behalf of Shaul Dar Sent: Mon 5/25/2009 6:51 AM To: pgsql-performance@postgresql.org Subject: [PERFORM] Putting tables or indexes in SSD or RAM: avoiding double caching? Hi, I have sen many posts on using SSDs, and iodrive http://www.fusionio.comin particular, to accelerate the performance of Postgresql (or other DBMS) -- e.g. this discussionhttp://groups.google.co.il/group/pgsql.performance/browse_thread/thread/1d6d7434246afd97?pli=1. I have also seen the suggestion to use RAM for the same purpose by creating a tablespace on a RAM mount point.http://magazine.redhat.com/2007/12/12/tip-from-an-rhce-memory-storage-on-postgresql/Granted these make most sense when the whole database cannot fit into main memory, or if we want to avoid cold DB response times (i.e waiting for the DB to warm up as stuff gets cached in memory). My question is this: if we use either SSD or RAM tablespaces, I would imagine postgresql will be oblevient to this and would still cache the tablespace elemenst that are on SSD or RAM into memory - right? Is there a way to avoid that, i.e. to tell postgress NOT to cache tablespaces, or some other granularity of the DB? Thanks, -- Shaul *Dr. Shaul Dar* Email: i...@shauldar.com Web: www.shauldar.com
Re: [PERFORM] I have a fusion IO drive available for testing
We have a box outfitted with two of them we are testing. We are testing with VxFS and ext3, but not quite ready to publish. I will reply when we are done. -Original Message- From: pgsql-performance-ow...@postgresql.org on behalf of Dave Cramer Sent: Thu 3/26/2009 5:47 AM To: pgsql-performance@postgresql.org Subject: [PERFORM] I have a fusion IO drive available for testing So far using dd I am seeing around 264MB/s on ext3, 335MB/s on ext2 write speed. So the question becomes what is the best filesystem for this drive? Anyone want me to run anything on it ? Dave
Re: [PERFORM] [PERFORMANCE] Buying hardware
The technique Jeff is speaking of below is exactly how we do it, except we use file-system snapshots vs rsync. The problem is how slow log application is when recovering since it's a single process, and very slow at that. -kg On Jan 26, 2009, at 11:58 AM, Jeff wrote: On Jan 26, 2009, at 2:42 PM, David Rees wrote: Lots of people have databases much, much, bigger - I'd hate to imagine have to restore from backup from one of those monsters. If you use PITR + rsync you can create a binary snapshot of the db, so restore time is simply how long it takes to untar / whatever it into place. Our backup script basically does: archive backup directory pg_start_backup rsync pg_stop_backup voila. I have 2 full copies of the db. You could even expand it a bit and after the rsync friends have it fire up the instance and run pg_dump against it for a pg_restore compatible dump just in case. It takes a long time to restore a 300GB db, even if you cheat and parallelify some of it. 8.4 may get a pg_restore that can load in parallel - which will help somewhat. -- Jeff Trout j...@jefftrout.com http://www.stuarthamm.net/ http://www.dellsmartexitin.com/ -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org ) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
