[PERFORM] Slow BLOBs restoring

2010-12-07 Thread Vlad Arkhipov
I have encountered a problem while restoring the database. There is a table that contains XML data (BLOB), ~ 3 000 000 records, ~ 5.5Gb of data. pg_restore has been running for a week without any considerable progress. There are plenty of lines like these in the log: pg_restore: processing

Re: [PERFORM] Performance under contention

2010-12-07 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: I wonder if it would be possible to have a very short critical section where we grab the partition lock, acquire the heavyweight lock, and release the partition lock; and then only as a second step record (in the form of a PROCLOCK) the fact that we

Re: [PERFORM] Performance under contention

2010-12-07 Thread Dave Crooke
Hi Tom I suspect I may be missing something here, but I think it's a pretty universal truism that cache lines are aligned to power-of-2 memory addresses, so it would suffice to ensure during setup that the lower order n bits of the object address are all zeros for each critical object; if the

[PERFORM] Compared MS SQL 2000 to Postgresql 9.0 on Windows

2010-12-07 Thread Tom Polak
We are in the process of deciding on how to proceed on a database upgrade. We currently have MS SQL 2000 running on Windows 2003 (on my test server). I was shocked at the cost for MS SQL 2008 R2 for a new server (2 CPU license). I started comparing DB’s and came across postgresql. It seemed to

Re: [PERFORM] Performance under contention

2010-12-07 Thread Ivan Voras
On 7 December 2010 18:37, Robert Haas robertmh...@gmail.com wrote: On Mon, Dec 6, 2010 at 9:59 PM, Jignesh Shah jks...@gmail.com wrote: That's exactly what I concluded when I was doing the sysbench simple read-only test. I had also tried with different lock partitions and it did not help since

Re: [PERFORM] Performance under contention

2010-12-07 Thread Robert Haas
On Tue, Dec 7, 2010 at 12:50 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: I wonder if it would be possible to have a very short critical section where we grab the partition lock, acquire the heavyweight lock, and release the partition lock; and then only as

Re: [PERFORM] Performance under contention

2010-12-07 Thread Robert Haas
On Tue, Dec 7, 2010 at 1:08 PM, Ivan Voras ivo...@freebsd.org wrote: On 7 December 2010 18:37, Robert Haas robertmh...@gmail.com wrote: On Mon, Dec 6, 2010 at 9:59 PM, Jignesh Shah jks...@gmail.com wrote: That's exactly what I concluded when I was doing the sysbench simple read-only test. I

Re: [PERFORM] Compared MS SQL 2000 to Postgresql 9.0 on Windows

2010-12-07 Thread Kevin Grittner
Tom Polak t...@rockfordarearealtors.org wrote: the best I could get it to was 7.5 seconds. select name,address,city,state,statename,stateid,other from pgtemp1 left join pgtemp2 on state=stateid We'd need a lot more information. Please read this and post again:

Re: [PERFORM] Compared MS SQL 2000 to Postgresql 9.0 on Windows

2010-12-07 Thread Andy Colson
On 12/7/2010 11:34 AM, Tom Polak wrote: We are in the process of deciding on how to proceed on a database upgrade. We currently have MS SQL 2000 running on Windows 2003 (on my test server). I was shocked at the cost for MS SQL 2008 R2 for a new server (2 CPU license). I started comparing DB’s

Re: [PERFORM] Compared MS SQL 2000 to Postgresql 9.0 on Windows

2010-12-07 Thread Andres Freund
On Tuesday 07 December 2010 18:34:25 Tom Polak wrote: Then I did the same test via Postgresql and it took 8.85 seconds! I tried it again as I thought I did something wrong. I did a few tweaks such as increasing the shared buffers. Still the best I could get it to was 7.5 seconds. This is

Re: [PERFORM] Performance under contention

2010-12-07 Thread Ivan Voras
On 7 December 2010 19:10, Robert Haas robertmh...@gmail.com wrote: I'm not very familiar with PostgreSQL code but if we're brainstorming... if you're only trying to protect against a small number of expensive operations (like DROP, etc.) that don't really happen often, wouldn't an atomic

Re: [PERFORM] Compared MS SQL 2000 to Postgresql 9.0 on Windows

2010-12-07 Thread Craig James
On 12/7/10 9:34 AM, Tom Polak wrote: We are in the process of deciding on how to proceed on a database upgrade. We currently have MS SQL 2000 running on Windows 2003 (on my test server). I was shocked at the cost for MS SQL 2008 R2 for a new server (2 CPU license). I started comparing DB’s

Re: [PERFORM] Compared MS SQL 2000 to Postgresql 9.0 on Windows

2010-12-07 Thread Andy Colson
On 12/7/2010 1:22 PM, Justin Pitts wrote: Also, as a fair warning: mssql doesn't really care about transactions, but PG really does. Make sure all your code is properly starting and commiting transactions. -Andy I do not understand that statement. Can you explain it a bit better? In mssql

Re: [PERFORM] Compared MS SQL 2000 to Postgresql 9.0 on Windows

2010-12-07 Thread Richard Broersma
On Tue, Dec 7, 2010 at 11:43 AM, Andy Colson a...@squeakycode.net wrote: In PG the first statement you fire off (like an insert into for example) will start a transaction.  If you dont commit before you disconnect that transaction will be rolled back.  Even worse, if your program does not

Re: [PERFORM] Compared MS SQL 2000 to Postgresql 9.0 on Windows

2010-12-07 Thread Kenneth Marshall
On Tue, Dec 07, 2010 at 11:56:51AM -0800, Richard Broersma wrote: On Tue, Dec 7, 2010 at 11:43 AM, Andy Colson a...@squeakycode.net wrote: In PG the first statement you fire off (like an insert into for example) will start a transaction. ?If you dont commit before you disconnect that

Re: [PERFORM] Compared MS SQL 2000 to Postgresql 9.0 on Windows

2010-12-07 Thread Gary Doades
On 07/12/2010 7:43 PM, Andy Colson wrote: On 12/7/2010 1:22 PM, Justin Pitts wrote: Also, as a fair warning: mssql doesn't really care about transactions, but PG really does. Make sure all your code is properly starting and commiting transactions. -Andy I do not understand that

Re: [PERFORM] Compared MS SQL 2000 to Postgresql 9.0 on Windows

2010-12-07 Thread Andy Colson
On 12/7/2010 2:10 PM, Kenneth Marshall wrote: On Tue, Dec 07, 2010 at 11:56:51AM -0800, Richard Broersma wrote: On Tue, Dec 7, 2010 at 11:43 AM, Andy Colsona...@squeakycode.net wrote: In PG the first statement you fire off (like an insert into for example) will start a transaction. ?If you

Re: [PERFORM] Compared MS SQL 2000 to Postgresql 9.0 on Windows

2010-12-07 Thread Tom Polak
What I was really after was a quick comparison between the two. I did not create anything special, just the two tables. One table SQL generated the records for me. I did not tweak anything after installing either system. There was a primary key on the ID field of both tables, no indexes though

Re: [PERFORM] Performance under contention

2010-12-07 Thread Віталій Тимчишин
2010/12/7 Robert Haas robertmh...@gmail.com On Tue, Dec 7, 2010 at 1:08 PM, Ivan Voras ivo...@freebsd.org wrote: I'm not very familiar with PostgreSQL code but if we're brainstorming... if you're only trying to protect against a small number of expensive operations (like DROP, etc.) that

Re: [PERFORM] Compared MS SQL 2000 to Postgresql 9.0 on Windows

2010-12-07 Thread Craig James
On 12/7/10 1:29 PM, Tom Polak wrote: What I was really after was a quick comparison between the two. I did not create anything special, just the two tables. One table SQL generated the records for me. I did not tweak anything after installing either system. That's not a valid test.

Re: [PERFORM] Compared MS SQL 2000 to Postgresql 9.0 on Windows

2010-12-07 Thread Gary Doades
On 07/12/2010 9:29 PM, Tom Polak wrote: From EXPLAIN ANALYZE I can see the query ran much faster. Nested Loop Left Join (cost=0.00..138.04 rows=1001 width=1298) (actual time=0.036..4.679 rows=1001 loops=1) Join Filter: (pgtemp1.state = pgtemp2.stateid) - Seq Scan on pgtemp1

Re: [PERFORM] Update problem on large table

2010-12-07 Thread Josh Kupershmidt
On Mon, Dec 6, 2010 at 4:31 PM, felix crucialfe...@gmail.com wrote: thanks for the replies !, but actually I did figure out how to kill it but pb_cancel_backend didn't work.  here's some notes: this has been hung for 5 days: ns      |   32681 | nssql   | IDLE in transaction | f       |

Re: [PERFORM] Compared MS SQL 2000 to Postgresql 9.0 on Windows

2010-12-07 Thread Kevin Grittner
Tom Polak t...@rockfordarearealtors.org wrote: I did not tweak anything after installing either system. PostgreSQL is set up with defaults such that it will start up and run on the most ancient an underpowered system people are likely to have lying around. It is expected that people will

Re: [PERFORM] Performance under contention

2010-12-07 Thread Ivan Voras
2010/12/7 Віталій Тимчишин tiv...@gmail.com: 2010/12/7 Robert Haas robertmh...@gmail.com On Tue, Dec 7, 2010 at 1:08 PM, Ivan Voras ivo...@freebsd.org wrote: I'm not very familiar with PostgreSQL code but if we're brainstorming... if you're only trying to protect against a small number

Re: [PERFORM] Performance under contention

2010-12-07 Thread Robert Haas
2010/12/7 Віталій Тимчишин tiv...@gmail.com: As far as I can see from the source, there is a lot of code executed under the partition lock protection, like two hash searches (and possibly allocations). Yeah, that was my concern, too, though Tom seems skeptical (perhaps rightly). And I'm not

Re: [PERFORM] Performance under contention

2010-12-07 Thread Robert Haas
2010/12/7 Robert Haas robertmh...@gmail.com: 2010/12/7 Віталій Тимчишин tiv...@gmail.com: As far as I can see from the source, there is a lot of code executed under the partition lock protection, like two hash searches (and possibly allocations). Yeah, that was my concern, too, though Tom