[PERFORM] Re: getting the most of out multi-core systems for repeated complex SELECT statements
On 02/03/2011 10:57 AM, gnuo...@rcn.com wrote: For PG to prosper in the future, it has to embrace the multi-core/processor/SSD machine at the query level As the person who brought up the original concern, I'll add that multi-core at the query level really isn't important for us. Most of our PostgreSQL usage is through a web application which fairly automatically takes advantage of multiple cores, because there are several parallel connections. A smaller but important piece of what we do is run this cron script needs to run hundreds of thousands of variations of the same complex SELECT as fast it can. What honestly would have helped most is not technical at all-- it would have been some documentation on how to take advantage of multiple cores for this case. It looks like it's going to be trivial-- Divide up the data with a modulo, and run multiple parallel cron scripts that each processes a slice of the data. A benchmark showed that this approach sped up our processing 3x when splitting the application 4 ways across 4 processors. (I think we failed to achieve a 4x improvement because the server was already busy handling some other tasks). Part of our case is likely fairly common *today*: many servers are multi-core now, but people don't necessarily understand how to take advantage of that if it doesn't happen automatically. Mark -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
Re: [PERFORM] Re: getting the most of out multi-core systems for repeated complex SELECT statements
On Fri, Feb 4, 2011 at 2:18 PM, Mark Stosberg m...@summersault.com wrote: It looks like it's going to be trivial-- Divide up the data with a modulo, and run multiple parallel cron scripts that each processes a slice of the data. A benchmark showed that this approach sped up our processing 3x when splitting the application 4 ways across 4 processors. (I think we failed to achieve a 4x improvement because the server was already busy handling some other tasks). I once had about 2 months of machine work ahead of me for one server. Luckily it was easy to break up into chunks and run it on all the workstations at night in the office, and we were done in 1 week. pgsql was the data store for it, and it was just like what you're talking about, break it into chunks, spread it around. -- Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-performance
