Re: [HACKERS] Uh, I change my mind about commit_delay + commit_siblings (sort of)

2012-05-30 Thread Simon Riggs
On 29 May 2012 17:58, Robert Haas robertmh...@gmail.com wrote: On Tue, May 29, 2012 at 12:47 PM, Peter Geoghegan pe...@2ndquadrant.com wrote: Why do you think that doing this for all XLogFlush() callsites might be problematic? Well, consider the one in the background writer, for example.  

Re: [HACKERS] hash index concurrency

2012-05-30 Thread Simon Riggs
On 30 May 2012 04:54, Robert Haas robertmh...@gmail.com wrote: This was a hobby horse of mine a couple of years ago, but I never got much traction.  The main question I have is, what do we even want hash indexes to be?  NBTree is very good, has been extensively optimized, and extensively

Re: [HACKERS] Bug in new buffering GiST build code

2012-05-30 Thread Heikki Linnakangas
On 28.05.2012 00:46, Alexander Korotkov wrote: On Sat, May 26, 2012 at 12:33 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Attached is a patch to replace the path stacks with a hash table. With this patch, the index build time in my test case dropped from 59 minutes to

Re: [HACKERS] Bug in new buffering GiST build code

2012-05-30 Thread Alexander Korotkov
On Wed, May 30, 2012 at 1:01 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: I also spotted and fixed another little oversight: the temporary file didn't get deleted after the index build. I've one note not directly related to buffering build. While I debugging buffering

[HACKERS] How do I get the name of the relation on which FDW has been called?

2012-05-30 Thread Atri Sharma
SELECT * FROM table1; If,for above command,fdw1 is invoked,how do I get the name/Oid of the table(i.e. table1) in fdw1? Atri -- Regards, Atri l'apprenant -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

Re: [HACKERS] slow dropping of tables, DropRelFileNodeBuffers, tas

2012-05-30 Thread Heikki Linnakangas
On 30.05.2012 03:40, Sergey Koposov wrote: I was running some tests on PG9.2beta where I'm creating and dropping large number of tables (~ 2). And I noticed that table dropping was extremely slow -- e.g. like half a second per table. ... I also stopped PG with gdb a few times and it was

Re: [HACKERS] How do I get the name of the relation on which FDW has been called?

2012-05-30 Thread Shigeru HANADA
Hi Atri, (2012/05/30 19:49), Atri Sharma wrote: SELECT * FROM table1; If,for above command,fdw1 is invoked,how do I get the name/Oid of the table(i.e. table1) in fdw1? For 9.1 and 9.0, you can get foreign table's oid as the first parameter of PlanForeignScan function. For 9.2, you can get

Re: [HACKERS] How do I get the name of the relation on which FDW has been called?

2012-05-30 Thread Atri Sharma
On Wed, May 30, 2012 at 5:15 PM, Shigeru HANADA shigeru.han...@gmail.com wrote: Hi Atri, (2012/05/30 19:49), Atri Sharma wrote: SELECT * FROM table1; If,for above command,fdw1 is invoked,how do I get the name/Oid of the table(i.e. table1) in fdw1? For 9.1 and 9.0, you can get foreign

Re: [HACKERS] Uh, I change my mind about commit_delay + commit_siblings (sort of)

2012-05-30 Thread Robert Haas
On Wed, May 30, 2012 at 4:36 AM, Simon Riggs si...@2ndquadrant.com wrote: When I read this the first time, I was in full agreement. On closer inspection neither point is valid, though both points were worth considering. Well, consider the one in the background writer, for example.  That's

Re: [HACKERS] slow dropping of tables, DropRelFileNodeBuffers, tas

2012-05-30 Thread Robert Haas
On Wed, May 30, 2012 at 7:10 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: So we drop the buffers for each relation fork separately, which means that we scan the buffer pool four times. Relation forks in 8.4 introduced that issue, and 9.1 made it worse by adding another fork

Re: [HACKERS] WalSndWakeup() and synchronous_commit=off

2012-05-30 Thread Andres Freund
On Tuesday, May 29, 2012 08:42:43 PM Andres Freund wrote: Patch attached. Imo this patch should be backported to 9.1, 9.0 doesn't use latches and does not do explicit wakeup of the sender so its not applicable there. I can prepare a patch for 9.1 if people agree, there has been some amount of

Re: [HACKERS] Uh, I change my mind about commit_delay + commit_siblings (sort of)

2012-05-30 Thread Peter Geoghegan
On 30 May 2012 13:24, Robert Haas robertmh...@gmail.com wrote: Most of those actually do look like reasonable places to try to get grouped flushing behavior, but: 1. It seems wrong to do it in xact_redo_commit_internal().  It won't matter if commit_siblings0 since there won't be any other

[HACKERS] too low NAPTIME_PER_CYCLE /too many wakeups in walreceiver.c

2012-05-30 Thread Andres Freund
Hi, Currently the walreceiver wakeups NAPTIME_PER_CYCLE=100 miliseconds in idle state. This is rather frequent. I don't really see any reason to do so. A nice fix would be to latchify that with WaitLatchOrSocket + a SetLatch in the signal handler for shutdown but that seems to be too invasive

Re: [HACKERS] hash index concurrency

2012-05-30 Thread Merlin Moncure
On Wed, May 30, 2012 at 3:49 AM, Simon Riggs si...@2ndquadrant.com wrote: On 30 May 2012 04:54, Robert Haas robertmh...@gmail.com wrote: This was a hobby horse of mine a couple of years ago, but I never got much traction.  The main question I have is, what do we even want hash indexes to be?  

Re: [HACKERS] Function call hierarchy/path since getting the buffer until access its data

2012-05-30 Thread Waldecir Faria
Thank you for the reply Robert. I think I am getting the idea about reading buffers but I am confused about the writing part, can you give me a function name where it does some write operations like creating a table or inserting a tuple for me read as a example. Best Regards , -Waldecir

Re: [HACKERS] How could we make it simple to access the log as a table?

2012-05-30 Thread Marc Mamin
The CSV format is both rich and machine-parseable (good start!) but it takes an unreasonable amount of work to make it usefully queryable. We need something that looks more like a big red button. Hello, The Pg logs consist of a rich soup with many different information kind. It's

Re: [HACKERS] Uh, I change my mind about commit_delay + commit_siblings (sort of)

2012-05-30 Thread Simon Riggs
On 30 May 2012 13:24, Robert Haas robertmh...@gmail.com wrote: OK, but there are a lot of places where we call XLogFlush(), and it's far from obvious that it's a win to do this in all of those cases.  At least, nobody's done that analysis.  XLogFlush is called from: - WriteTruncateXlogRec(),

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Merlin Moncure
On Sun, May 27, 2012 at 1:45 PM, Sergey Koposov kopo...@ast.cam.ac.uk wrote: Hi, I did another test using the same data and the same code, which I've provided before and the performance of the single thread seems to be degrading quadratically with the number of threads. Here are the

Re: [HACKERS] Function call hierarchy/path since getting the buffer until access its data

2012-05-30 Thread Robert Haas
On Wed, May 30, 2012 at 9:37 AM, Waldecir Faria fighter2...@hotmail.com wrote: Thank you for the reply Robert. I think I am getting the idea about reading buffers but I am confused about the writing part, can you give me a function name where it does some write operations like creating a table

Re: [HACKERS] slow dropping of tables, DropRelFileNodeBuffers, tas

2012-05-30 Thread Jeff Janes
On Wed, May 30, 2012 at 4:10 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 30.05.2012 03:40, Sergey Koposov wrote: I was running some tests on PG9.2beta where I'm creating and dropping large number of tables (~ 2). And I noticed that table dropping was extremely

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Sergey Koposov
Here is the actual explain analyze of the query on the smaller dataset which I have been using for the recent testing. test=# explain analyze create table _tmp0 as select * from ( select *, (select healpixid from idt_match as m where m.transitid=o.transitid) as x from

[HACKERS] remembering locks and dynahash.c

2012-05-30 Thread Jeff Janes
Currently the resource owner does not remember what locks it holds. When a resource owner wants to release its locks or reassign them to its parent, it just digs through the backends entire LockMethodLocalHash table. When that table is very large, but the current owner owns only a small fraction

[HACKERS] Not quite a security hole: CREATE LANGUAGE for non-superusers

2012-05-30 Thread Tom Lane
We allow non-superuser database owners to execute CREATE LANGUAGE for a trusted language (one marked as tmpldbacreate in pg_pltemplate). Currently, the C-language support functions for the language end up owned by that non-superuser. This is on the hairy edge of being a security hole, since

Re: [HACKERS] Re: [COMMITTERS] pgsql: Send new protocol keepalive messages to standby servers.

2012-05-30 Thread Fujii Masao
On Wed, May 30, 2012 at 4:34 AM, Simon Riggs si...@2ndquadrant.com wrote: On 24 May 2012 21:11, Robert Haas robertmh...@gmail.com wrote: On Thu, May 24, 2012 at 2:52 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Wed, May 23, 2012 at 2:28 PM, Fujii Masao

Re: [HACKERS] Uh, I change my mind about commit_delay + commit_siblings (sort of)

2012-05-30 Thread Peter Geoghegan
On 30 May 2012 15:25, Simon Riggs si...@2ndquadrant.com wrote: 1. It seems wrong to do it in xact_redo_commit_internal().  It won't matter if commit_siblings0 since there won't be any other backends with transaction IDs anyway, but if commit_siblings==0 then we'll sleep for no possible

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Merlin Moncure
On Wed, May 30, 2012 at 10:42 AM, Sergey Koposov kopo...@ast.cam.ac.uk wrote: Here is the actual explain analyze of the query on the smaller dataset which I have been using for the recent testing. test=# explain analyze create table _tmp0 as select * from  ( select *,        (select

Re: [HACKERS] too low NAPTIME_PER_CYCLE /too many wakeups in walreceiver.c

2012-05-30 Thread Fujii Masao
On Wed, May 30, 2012 at 9:57 PM, Andres Freund and...@2ndquadrant.com wrote: Hi, Currently the walreceiver wakeups NAPTIME_PER_CYCLE=100 miliseconds in idle state. This is rather frequent. I don't really see any reason to do so. A nice fix would be to latchify that with WaitLatchOrSocket + a

Re: [HACKERS] slow dropping of tables, DropRelFileNodeBuffers, tas

2012-05-30 Thread Ants Aasma
On Wed, May 30, 2012 at 2:10 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: Also, I wonder if DropRelFileNodeBuffers() could scan the pool without grabbing the spinlocks on every buffer? It could do an unlocked test first, and only grab the spinlock on buffers that need to be

[HACKERS] Figuring out shared buffer pressure

2012-05-30 Thread Bruce Momjian
As part of a blog, I started looking at how a user could measure the pressure on shared buffers, e.g. how much are they being used, recycled, etc. They way you normally do it on older operating systems is to see how many buffers on the free list (about to be reused) are reclaimed as needed ---

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Sergey Koposov
On Wed, 30 May 2012, Merlin Moncure wrote: 1. Can we see an explain analyze during a 'bogged' case? Here is the one to one comparison of the 'bogged' ** QUERY PLAN

Re: [HACKERS] FailedAssertion(!(PrivateRefCount[i] == 0), File: bufmgr.c, Line: 1741

2012-05-30 Thread Heikki Linnakangas
On 26.05.2012 12:21, Erik Rijkers wrote: But when that if-block is added the client crashes after a while (sometimes almost immediately; it never survives longer then 20 minutes): 2012-05-26 10:44:22.617 CEST 10274 ERROR: could not fsync file base/21268/32807: No such file or directory

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Robert Haas
On Wed, May 30, 2012 at 12:58 PM, Sergey Koposov kopo...@ast.cam.ac.uk wrote: Here is the one to one comparison of the 'bogged' **  QUERY PLAN

Re: [HACKERS] Figuring out shared buffer pressure

2012-05-30 Thread Jeff Janes
On Wed, May 30, 2012 at 9:56 AM, Bruce Momjian br...@momjian.us wrote: As part of a blog, I started looking at how a user could measure the pressure on shared buffers, e.g. how much are they being used, recycled, etc. They way you normally do it on older operating systems is to see how many

Re: [HACKERS] FailedAssertion(!(PrivateRefCount[i] == 0), File: bufmgr.c, Line: 1741

2012-05-30 Thread Robert Haas
On Wed, May 30, 2012 at 1:07 PM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: There's something wrong with the way AccessExclusiveLocks work on a standby. I did begin; truncate foo; -- leave the xact open in the master, and waited until the xlog records are shipped to the

Re: [HACKERS] Figuring out shared buffer pressure

2012-05-30 Thread Bruce Momjian
On Wed, May 30, 2012 at 10:38:10AM -0700, Jeff Janes wrote: On Wed, May 30, 2012 at 9:56 AM, Bruce Momjian br...@momjian.us wrote: As part of a blog, I started looking at how a user could measure the pressure on shared buffers, e.g. how much are they being used, recycled, etc. They way

Re: [HACKERS] Figuring out shared buffer pressure

2012-05-30 Thread Jeff Janes
On Wed, May 30, 2012 at 10:57 AM, Bruce Momjian br...@momjian.us wrote: On Wed, May 30, 2012 at 10:38:10AM -0700, Jeff Janes wrote: On Wed, May 30, 2012 at 9:56 AM, Bruce Momjian br...@momjian.us wrote: As part of a blog, I started looking at how a user could measure the pressure on shared

Re: [HACKERS] Figuring out shared buffer pressure

2012-05-30 Thread Bruce Momjian
On Wed, May 30, 2012 at 11:06:45AM -0700, Jeff Janes wrote: On Wed, May 30, 2012 at 10:57 AM, Bruce Momjian br...@momjian.us wrote: On Wed, May 30, 2012 at 10:38:10AM -0700, Jeff Janes wrote: On Wed, May 30, 2012 at 9:56 AM, Bruce Momjian br...@momjian.us wrote: As part of a blog, I

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Merlin Moncure
On Wed, May 30, 2012 at 12:11 PM, Robert Haas robertmh...@gmail.com wrote: On Wed, May 30, 2012 at 12:58 PM, Sergey Koposov kopo...@ast.cam.ac.uk wrote: Here is the one to one comparison of the 'bogged' **  QUERY PLAN

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Sergey Koposov
On Wed, 30 May 2012, Merlin Moncure wrote: Hm, why aren't we getting a IOS? Just for kicks (assuming this is test data), can we drop the index on just transitid, leaving the index on transitid, healpixid?Is enable_indexonlyscan on? Has idt_match been vacuumed? What kind of plan do you

Re: [HACKERS] Figuring out shared buffer pressure

2012-05-30 Thread Jeff Janes
On Wed, May 30, 2012 at 11:23 AM, Bruce Momjian br...@momjian.us wrote: On Wed, May 30, 2012 at 11:06:45AM -0700, Jeff Janes wrote: On Wed, May 30, 2012 at 10:57 AM, Bruce Momjian br...@momjian.us wrote: On Wed, May 30, 2012 at 10:38:10AM -0700, Jeff Janes wrote: Isn't that what the

Re: [HACKERS] FailedAssertion(!(PrivateRefCount[i] == 0), File: bufmgr.c, Line: 1741

2012-05-30 Thread Robert Haas
On Wed, May 30, 2012 at 1:47 PM, Robert Haas robertmh...@gmail.com wrote: The process holding the AccessExclusiveLock is the startup process. It's holding the lock on behalf of the transaction in the master. But something's wrong, and the AccessExclusiveLock doesn't stop a regular backend from

[HACKERS] pg_restore logging inconsistency

2012-05-30 Thread Josh Kupershmidt
Hi all, Bosco Rama recently complained[1] about not seeing a message printed by pg_restore for each LO to be restored. The culprit seems to be the different level passed to ahlog() for this status message: pg_backup_archiver.c: ahlog(AH, 2, restoring large object with OID %u\n, oid);

[HACKERS] GiST subsplit question

2012-05-30 Thread Jeff Davis
http://archives.postgresql.org/message-id/CAEsn3ybKcnFno_tGQDJ=afhir2xd9ka1fqt5cqxxyu3wz_h...@mail.gmail.com I was trying to answer that question on -general, and I found it a little more challenging than I expected. There was a previous discussion here:

Re: [HACKERS] [RFC] Interface of Row Level Security

2012-05-30 Thread Kohei KaiGai
2012/5/29 Robert Haas robertmh...@gmail.com: On Fri, May 25, 2012 at 5:08 PM, Kohei KaiGai kai...@kaigai.gr.jp wrote: I think it is a good idea not to apply RLS when current user has superuser privilege from perspective of security model consistency, but it is inconsistent to check privileges

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Merlin Moncure
On Wed, May 30, 2012 at 1:45 PM, Sergey Koposov kopo...@ast.cam.ac.uk wrote: On Wed, 30 May 2012, Merlin Moncure wrote: Hm, why aren't we getting a IOS?  Just for kicks (assuming this is test data), can we drop the index on just transitid, leaving the index on transitid, healpixid?    Is

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Sergey Koposov
On Wed, 30 May 2012, Merlin Moncure wrote: hurk -- ISTM that since IOS is masikng the heap lookups, there must be contention on the index itself? Does this working set fit in shared memory? If so, what happens when you do a database restart and repeat the IOS test? The dataset fits well in

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Jeff Janes
On Wed, May 30, 2012 at 11:45 AM, Sergey Koposov kopo...@ast.cam.ac.uk wrote: On Wed, 30 May 2012, Merlin Moncure wrote: Hm, why aren't we getting a IOS?  Just for kicks (assuming this is test data), can we drop the index on just transitid, leaving the index on transitid, healpixid?    Is

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Merlin Moncure
On Wed, May 30, 2012 at 3:15 PM, Jeff Janes jeff.ja...@gmail.com wrote: On Wed, May 30, 2012 at 11:45 AM, Sergey Koposov kopo...@ast.cam.ac.uk wrote: On Wed, 30 May 2012, Merlin Moncure wrote: Hm, why aren't we getting a IOS?  Just for kicks (assuming this is test data), can we drop the

Re: [HACKERS] FailedAssertion(!(PrivateRefCount[i] == 0), File: bufmgr.c, Line: 1741

2012-05-30 Thread Robert Haas
On Wed, May 30, 2012 at 2:52 PM, Robert Haas robertmh...@gmail.com wrote: On Wed, May 30, 2012 at 1:47 PM, Robert Haas robertmh...@gmail.com wrote: The process holding the AccessExclusiveLock is the startup process. It's holding the lock on behalf of the transaction in the master. But

Re: [HACKERS] Advisory locks seem rather broken

2012-05-30 Thread Robert Haas
On Fri, May 4, 2012 at 9:17 AM, Robert Haas robertmh...@gmail.com wrote: On Thu, May 3, 2012 at 5:18 PM, Tom Lane t...@sss.pgh.pa.us wrote: ... btw, it appears to me that the fast path patch has broken things rather badly in LockReleaseAll.  AFAICS it's not honoring either the lockmethodid

Re: [HACKERS] Fake async rep target

2012-05-30 Thread james
Well, I was assuming that there was some intelligence in the receiver that could effectively parse this for the application; are you suggesting that is effectively binary deltas to apply to raw pages? Certainly, Sybase rep server works by creating function calls or SQL updates (depending on

Re: [HACKERS] Fake async rep target

2012-05-30 Thread james
Well, I was assuming that there was some intelligence in the receiver that could effectively parse this for the application; are you suggesting that is effectively binary deltas to apply to raw pages? Certainly, Sybase rep server works by creating function calls or SQL updates (depending on

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Sergey Koposov
On Wed, 30 May 2012, Merlin Moncure wrote: How big is idt_match?  What if you drop all indexes on idt_match, encouraging all the backends to do hash joins against it, which occur in local memory and so don't have contention? You just missed his post -- it's only 3G. can you run your 'small'

Re: [HACKERS] GiST subsplit question

2012-05-30 Thread Alexander Korotkov
On Wed, May 30, 2012 at 11:21 PM, Jeff Davis pg...@j-davis.com wrote: I looked for the follow-up commit to support subsplit in the contrib modules, figuring that would answer some questions, but I couldn't find it. The part that's confusing me is that the commit message says: pickSplit

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Florian Pflug
On May30, 2012, at 22:07 , Sergey Koposov wrote: If I restart the db the timings do not change significantly. There is always some variation which I don't really understand, e.g. the parallel runs sometimes take 18s, or 25 seconds, or 30 seconds per thread. So there is something else

Re: [HACKERS] Fake async rep target

2012-05-30 Thread Florian Pflug
On May30, 2012, at 22:28 , james wrote: Well, I was assuming that there was some intelligence in the receiver that could effectively parse this for the application; are you suggesting that is effectively binary deltas to apply to raw pages? In parts. The log that is streamed to replication

[HACKERS] Early hint bit setting

2012-05-30 Thread Ants Aasma
I was thinking about what is the earliest time where we could set hint bits. This would be just after the commit has been made visible. When the transaction completes and commit confirmation is sent to the client the backend will usually go to sleep waiting on the network socket waiting for

Re: [HACKERS] Figuring out shared buffer pressure

2012-05-30 Thread Bruce Momjian
On Wed, May 30, 2012 at 11:51:23AM -0700, Jeff Janes wrote: On Wed, May 30, 2012 at 11:23 AM, Bruce Momjian br...@momjian.us wrote: On Wed, May 30, 2012 at 11:06:45AM -0700, Jeff Janes wrote: On Wed, May 30, 2012 at 10:57 AM, Bruce Momjian br...@momjian.us wrote: On Wed, May 30, 2012 at

Re: [HACKERS] FailedAssertion(!(PrivateRefCount[i] == 0), File: bufmgr.c, Line: 1741

2012-05-30 Thread Erik Rijkers
On Wed, May 30, 2012 22:25, Robert Haas wrote: On Wed, May 30, 2012 at 2:52 PM, Robert Haas robertmh...@gmail.com wrote: On Wed, May 30, 2012 at 1:47 PM, Robert Haas robertmh...@gmail.com wrote: The process holding the AccessExclusiveLock is the startup process. It's holding the lock on behalf

Re: [HACKERS] Early hint bit setting

2012-05-30 Thread Merlin Moncure
On Wed, May 30, 2012 at 4:42 PM, Ants Aasma a...@cybertec.at wrote: I was thinking about what is the earliest time where we could set hint bits. This would be just after the commit has been made visible. When the transaction completes and commit confirmation is sent to the client the backend

[HACKERS] We're not lax enough about maximum time zone offset from UTC

2012-05-30 Thread Tom Lane
Currently, our datetime input code thinks that any UTC offset of more than 14:59:59 either way from Greenwich must be a mistake. However, after seeing Patric Bechtel's recent bug report, I went trolling in the Olson timezone files to see what are the largest offsets used there. I found three

[HACKERS] patch: avoid heavyweight locking on hash metapage

2012-05-30 Thread Robert Haas
I developed the attached patch to avoid taking a heavyweight lock on the metapage of a hash index. Instead, an exclusive buffer content lock is viewed as sufficient permission to modify the metapage, and a shared buffer content lock is used when such modifications need to be prevented. For the

Re: [HACKERS] Early hint bit setting

2012-05-30 Thread Ants Aasma
On Thu, May 31, 2012 at 1:01 AM, Merlin Moncure mmonc...@gmail.com wrote: I think this is a really neat idea, and could solve a lot of problems.  Since you don't have to do any clog checks (you know when you commit) -- i think it's a win all around -- so much so that it might be worth seeing

Re: [HACKERS] We're not lax enough about maximum time zone offset from UTC

2012-05-30 Thread David E. Wheeler
On May 30, 2012, at 3:10 PM, Tom Lane wrote: However, as pointed out by Patric, if you dump and restore an old timestamptz value in one of these zones, it will fail to restore because of the sanity check. I think therefore that we'd better enlarge the allowed range to 15:59:59 either way.

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Sergey Koposov
On Wed, 30 May 2012, Florian Pflug wrote: I wonder if the huge variance could be caused by non-uniform synchronization costs across different cores. That's not all that unlikely, because at least some cache levels (L2 and/or L3, I think) are usually shared between all cores on a single die.

Re: [HACKERS] WalSndWakeup() and synchronous_commit=off

2012-05-30 Thread Fujii Masao
On Wed, May 30, 2012 at 9:46 PM, Andres Freund and...@2ndquadrant.com wrote: On Tuesday, May 29, 2012 08:42:43 PM Andres Freund wrote: Patch attached. Imo this patch should be backported to 9.1, 9.0 doesn't use latches and does not do explicit wakeup of the sender so its not applicable there.

Re: [HACKERS] We're not lax enough about maximum time zone offset from UTC

2012-05-30 Thread Tom Lane
David E. Wheeler da...@justatheory.com writes: On May 30, 2012, at 3:10 PM, Tom Lane wrote: However, as pointed out by Patric, if you dump and restore an old timestamptz value in one of these zones, it will fail to restore because of the sanity check. I think therefore that we'd better

Re: [HACKERS] Not quite a security hole: CREATE LANGUAGE for non-superusers

2012-05-30 Thread Noah Misch
On Wed, May 30, 2012 at 12:02:06PM -0400, Tom Lane wrote: One thing the owner *can* do is use ALTER FUNCTION to change secondary properties of the function, such as strictness, volatility, SECURITY DEFINER, etc. So far as I can see, none of these properties are examined for a PL support

Re: [HACKERS] WalSndWakeup() and synchronous_commit=off

2012-05-30 Thread Andres Freund
On Thursday, May 31, 2012 01:33:33 AM Fujii Masao wrote: On Wed, May 30, 2012 at 9:46 PM, Andres Freund and...@2ndquadrant.com wrote: On Tuesday, May 29, 2012 08:42:43 PM Andres Freund wrote: Patch attached. Imo this patch should be backported to 9.1, 9.0 doesn't use latches and does

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Florian Pflug
On May31, 2012, at 01:16 , Sergey Koposov wrote: On Wed, 30 May 2012, Florian Pflug wrote: I wonder if the huge variance could be caused by non-uniform synchronization costs across different cores. That's not all that unlikely, because at least some cache levels (L2 and/or L3, I think) are

Re: [HACKERS] Figuring out shared buffer pressure

2012-05-30 Thread Jeff Janes
On Wed, May 30, 2012 at 2:55 PM, Bruce Momjian br...@momjian.us wrote: On Wed, May 30, 2012 at 11:51:23AM -0700, Jeff Janes wrote: On Wed, May 30, 2012 at 11:23 AM, Bruce Momjian br...@momjian.us wrote: On Wed, May 30, 2012 at 11:06:45AM -0700, Jeff Janes wrote: On Wed, May 30, 2012 at 10:57

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-05-30 Thread Tatsuo Ishii
Yeah, Jeff's experiments indicated that the remaining bottleneck is lock management in the server. What I fixed so far on the pg_dump side should be enough to let partial dumps run at reasonable speed even if the whole database contains many tables. But if psql is taking AccessShareLock on

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Sergey Koposov
On Thu, 31 May 2012, Florian Pflug wrote: Wait, so performance *increased* by spreading the backends out over as many dies as possible, not by using as few as possible? That'd be exactly the opposite of what I'd have expected. (I'm assuming that cores on one die have ascending ids on linux.

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Jeff Janes
On Wed, May 30, 2012 at 4:16 PM, Sergey Koposov kopo...@ast.cam.ac.uk wrote: But the question now is whether there is a *PG* problem here or not, or is it Intel's or Linux's problem ? Because still the slowdown was caused by locking. If there wouldn't be locking there wouldn't be any problems

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Stephen Frost
Sergey, all, * Sergey Koposov (kopo...@ast.cam.ac.uk) wrote: I did a specific test with just 6 threads (== number of cores per cpu) and ran it on a single phys cpu, it took ~ 12 seconds for each thread, and when I tried to spread it across 4 cpus it took 7-9 seconds per thread. But all these

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Stephen Frost
* Sergey Koposov (kopo...@ast.cam.ac.uk) wrote: I did a specific test with just 6 threads (== number of cores per cpu) and ran it on a single phys cpu, it took ~ 12 seconds for each thread, and when I tried to spread it across 4 cpus it took 7-9 seconds per thread. But all these numbers are

Re: [HACKERS] Not quite a security hole: CREATE LANGUAGE for non-superusers

2012-05-30 Thread Tom Lane
Noah Misch n...@leadboat.com writes: I wondered about ALTER FUNCTION SET guc = '...' and tried to test it: CREATE FUNCTION f(out ret text) RETURNS text LANGUAGE plpgsql AS 'BEGIN ret := current_setting(''work_mem''); END'; ALTER FUNCTION plpgsql_call_handler() SET

Re: [HACKERS] Uppercase tab completion keywords in psql?

2012-05-30 Thread Devrim GÜNDÜZ
Hi, On Mon, 2012-05-07 at 13:22 -0400, Robert Haas wrote: On Sat, May 5, 2012 at 9:03 AM, Bruce Momjian br...@momjian.us wrote: On Fri, May 04, 2012 at 08:46:28PM +0300, Peter Eisentraut wrote: On tor, 2012-05-03 at 15:47 -0400, Bruce Momjian wrote: Peter, where are we on this? I

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Sergey Koposov
On Wed, 30 May 2012, Jeff Janes wrote: But the question now is whether there is a *PG* problem here or not, or is it Intel's or Linux's problem ? Because still the slowdown was caused by locking. If there wouldn't be locking there wouldn't be any problems (as demonstrated a while ago by just

Re: [HACKERS] FailedAssertion(!(PrivateRefCount[i] == 0), File: bufmgr.c, Line: 1741

2012-05-30 Thread Robert Haas
On Wed, May 30, 2012 at 6:00 PM, Erik Rijkers e...@xs4all.nl wrote: (I double-checked that I got your latest commit in) I'm afraid it's not yet resolved; the sync-slave still crashes almost immediately: master logfile says: 2012-05-30 23:30:07.846 CEST 3918 LOG:  standby wal_receiver_01 is

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Robert Haas
On Wed, May 30, 2012 at 9:10 PM, Sergey Koposov kopo...@ast.cam.ac.uk wrote: I understand the need of significant locking when there concurrent writes, but not when there only reads. But I'm not a RDBMS expert, so that's maybe that's misunderstanding on my side. If we knew in advance that no

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Florian Pflug
On May31, 2012, at 02:26 , Sergey Koposov wrote: On Thu, 31 May 2012, Florian Pflug wrote: Wait, so performance *increased* by spreading the backends out over as many dies as possible, not by using as few as possible? That'd be exactly the opposite of what I'd have expected. (I'm assuming

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Stephen Frost
Robert, * Robert Haas (robertmh...@gmail.com) wrote: On Wed, May 30, 2012 at 9:10 PM, Sergey Koposov kopo...@ast.cam.ac.uk wrote: I understand the need of significant locking when there concurrent writes, but not when there only reads. But I'm not a RDBMS expert, so that's maybe that's

Re: [HACKERS] Uppercase tab completion keywords in psql?

2012-05-30 Thread Bruce Momjian
On Thu, May 31, 2012 at 04:06:50AM +0300, Devrim Gunduz wrote: Hi, On Mon, 2012-05-07 at 13:22 -0400, Robert Haas wrote: On Sat, May 5, 2012 at 9:03 AM, Bruce Momjian br...@momjian.us wrote: On Fri, May 04, 2012 at 08:46:28PM +0300, Peter Eisentraut wrote: On tor, 2012-05-03 at 15:47

Re: [HACKERS] 9.2beta1, parallel queries, ReleasePredicateLocks, CheckForSerializableConflictIn in the oprofile

2012-05-30 Thread Jeff Janes
On Wed, May 30, 2012 at 7:00 PM, Stephen Frost sfr...@snowman.net wrote: Robert, * Robert Haas (robertmh...@gmail.com) wrote: On Wed, May 30, 2012 at 9:10 PM, Sergey Koposov kopo...@ast.cam.ac.uk wrote: I understand the need of significant locking when there concurrent writes, but not

Re: [HACKERS] Synchronized scans versus relcache reinitialization

2012-05-30 Thread Jeff Davis
On Sat, 2012-05-26 at 15:14 -0400, Tom Lane wrote: 3. Having now spent a good deal of time poking at this, I think that the syncscan logic is in need of more tuning, and I am wondering whether we should even have it turned on by default. It appears to be totally useless for

Re: [HACKERS] Synchronized scans versus relcache reinitialization

2012-05-30 Thread Tom Lane
Jeff Davis pg...@j-davis.com writes: On Sat, 2012-05-26 at 15:14 -0400, Tom Lane wrote: 3. Having now spent a good deal of time poking at this, I think that the syncscan logic is in need of more tuning, and I am wondering whether we should even have it turned on by default. It appears to be

Re: [HACKERS] temporal support patch

2012-05-30 Thread Jeff Davis
On Wed, 2012-05-16 at 23:14 +0200, Miroslav Šimulčík wrote: Hi all, as a part of my master's thesis I have created temporal support patch for PostgreSQL. It enables the creation of special temporal tables with entries versioning. Modifying operations (UPDATE, DELETE, TRUNCATE) on these

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-05-30 Thread Tom Lane
Tatsuo Ishii is...@postgresql.org writes: Ok, I modified the part of pg_dump where tremendous number of LOCK TABLE are issued. I replace them with single LOCK TABLE with multiple tables. With 100k tables LOCK statements took 13 minutes in total, now it only takes 3 seconds. Comments? Shall I

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-05-30 Thread Stephen Frost
* Tom Lane (t...@sss.pgh.pa.us) wrote: Tatsuo Ishii is...@postgresql.org writes: Shall I commit to master and all supported branches? I'm not excited by this patch. It dodges the O(N^2) lock behavior for the initial phase of acquiring the locks, but it does nothing for the lock-related

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-05-30 Thread Tom Lane
Stephen Frost sfr...@snowman.net writes: The current situation where the client-to-server latency accounts for multiple minutes of time is just ridiculous, however, so I feel we need some form of this patch, even if the server side is magically made much faster. The constant back-and-forth

Re: [HACKERS] WIP: 2d-mapping based indexing for ranges

2012-05-30 Thread Jeff Davis
On Mon, 2012-05-28 at 23:42 +0400, Alexander Korotkov wrote: Hackers, attached patch implements another approach to indexing of ranges: mapping lower and upper bounds as 2d-coordinates and using same indexing approaches as for 2d points. Patch provides range_ops2 operator class which

Re: [HACKERS] FailedAssertion(!(PrivateRefCount[i] == 0), File: bufmgr.c, Line: 1741

2012-05-30 Thread Erik Rijkers
On Thu, May 31, 2012 03:30, Robert Haas wrote: On Wed, May 30, 2012 at 6:00 PM, Erik Rijkers e...@xs4all.nl wrote: directory 2012-05-30 23:40:57.909 CEST 3909 CONTEXT:  writing block 5152 of relation base/21268/26569        xlog redo multi-insert (init): rel 1663/21268/26581; blk 3852; 35

Re: [HACKERS] [PERFORM] pg_dump and thousands of schemas

2012-05-30 Thread Tatsuo Ishii
I'm not excited by this patch. It dodges the O(N^2) lock behavior for the initial phase of acquiring the locks, but it does nothing for the lock-related slowdown occurring in all pg_dump's subsequent commands. I think we really need to get in the server-side fix that Jeff Janes is working

Re: [HACKERS] Bug in new buffering GiST build code

2012-05-30 Thread Tom Lane
Alexander Korotkov aekorot...@gmail.com writes: I've one note not directly related to buffering build. While I debugging buffering GiST index build, backend was frequently crashed. After recovery partially built index file was remain. Do we have some tool to detect such dead files? If not,