Re: [HACKERS] Page replacement algorithm in buffer cache

2013-03-22 Thread Atri Sharma
Sent from my iPad On 22-Mar-2013, at 11:28, Amit Kapila amit.kap...@huawei.com wrote: On Friday, March 22, 2013 10:22 AM Atri Sharma wrote: Hello all, Sorry if this is a naive question. I was going through Greg Smith's slides on buffer

Re: [HACKERS] Let's invent a function to report lock-wait-blocking PIDs

2013-03-22 Thread Heikki Linnakangas
On 21.03.2013 05:36, Tom Lane wrote: Simon Riggssi...@2ndquadrant.com writes: On 20 March 2013 18:02, Tom Lanet...@sss.pgh.pa.us wrote: The API that comes to mind is (name subject to bikeshedding) pg_blocking_pids(pid int) returns int[] Useful. Can we also have an SRF rather than an

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-03-22 Thread Amit Kapila
On Friday, March 22, 2013 12:00 PM Atri Sharma wrote: Sent from my iPad On 22-Mar-2013, at 11:28, Amit Kapila amit.kap...@huawei.com wrote: On Friday, March 22, 2013 10:22 AM Atri Sharma wrote: Hello all, Sorry if this is a naive question. I was going through Greg Smith's

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-03-22 Thread Atri Sharma
I think that if the initialization of USAGE_COUNT starts at the maximum allowed value instead of one, we can have a better solution to this problem. So what is your idea, if you start at maximum, what we will do for further accesses to it? I havent chalked out a detailed plan yet, but I

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-03-22 Thread Amit Kapila
On Friday, March 22, 2013 4:16 PM Atri Sharma wrote: I think that if the initialization of USAGE_COUNT starts at the maximum allowed value instead of one, we can have a better solution to this problem. So what is your idea, if you start at maximum, what we will do for further

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-03-22 Thread Atri Sharma
What would you do if the only young page has usage count zero during second sweep. UmmThe same approach we take when there is no page with usage count zero in a sweep in the current algorithm? I don't think introducing another factor along with usage count would do any much help.

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-03-22 Thread Amit Kapila
On Friday, March 22, 2013 4:36 PM Atri Sharma wrote: What would you do if the only young page has usage count zero during second sweep. UmmThe same approach we take when there is no page with usage count zero in a sweep in the current algorithm? It would give more priority to young

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-03-22 Thread Atri Sharma
On Fri, Mar 22, 2013 at 4:53 PM, Amit Kapila amit.kap...@huawei.com wrote: On Friday, March 22, 2013 4:36 PM Atri Sharma wrote: What would you do if the only young page has usage count zero during second sweep. UmmThe same approach we take when there is no page with usage count zero

Re: [HACKERS] Should commit_delay be PGC_SIGHUP?

2013-03-22 Thread Simon Riggs
On 22 March 2013 02:14, Tom Lane t...@sss.pgh.pa.us wrote: Simon Riggs si...@2ndquadrant.com writes: Only one setting will be best for the whole cluster, so neither the user nor the DBA gains if a user sets this to a different value than the one that has been determined to be optimal. Since

Re: [HACKERS] Should commit_delay be PGC_SIGHUP?

2013-03-22 Thread Robert Haas
On Fri, Mar 22, 2013 at 8:06 AM, Simon Riggs si...@2ndquadrant.com wrote: Hmm. If a malicious user could hurt performance for other sessions with a bad setting of commit_delay, then USERSET is clearly a bad idea. But it still seems like it could be SUSET rather than SIGHUP. Agreed; everybody

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-03-22 Thread Ants Aasma
On Mar 22, 2013 12:46 PM, Atri Sharma atri.j...@gmail.com wrote: This is the one I think would work out best, add an age factor as to the time duration which an entry has spent in the cache along with its usage count. You might want to check out the LIRS cache replacement algorithm [1]. That

Re: [HACKERS] Should commit_delay be PGC_SIGHUP?

2013-03-22 Thread Peter Geoghegan
On Fri, Mar 22, 2013 at 12:42 PM, Robert Haas robertmh...@gmail.com wrote: This is fine with me, too, and I agree that it's warranted... but your commit message supposes that this behavior is new in 9.3, and I think it dates to 9.2. No, it doesn't. It just missed the deadline for 9.2. I'm

Re: [HACKERS] JSON Function Bike Shedding

2013-03-22 Thread Merlin Moncure
On Mon, Mar 18, 2013 at 3:12 PM, Tom Lane t...@sss.pgh.pa.us wrote: Andrew Dunstan and...@dunslane.net writes: I've been sitting here for a while mulling none too happily over the debate on the names for the proposed JSON extraction functions. I haven't really been happy with any of the

Re: [HACKERS] JSON Function Bike Shedding

2013-03-22 Thread Andrew Dunstan
On 03/22/2013 09:29 AM, Merlin Moncure wrote: On Mon, Mar 18, 2013 at 3:12 PM, Tom Lane t...@sss.pgh.pa.us wrote: Andrew Dunstan and...@dunslane.net writes: I've been sitting here for a while mulling none too happily over the debate on the names for the proposed JSON extraction functions. I

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-03-22 Thread Atri Sharma
However, I think the main issue isn't finding new algorithms that are better in some specific circumstances. The hard part is figuring out whether their performance is better in general. My idea was to create a patch to capture page pinning traffic from PostgreSQL (maybe stream out into a

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-03-22 Thread Tom Lane
Ants Aasma a...@cybertec.at writes: You might want to check out the LIRS cache replacement algorithm [1]. That algorithm tries to estimate least frequently used instead of least recently used. Mysql uses it for their buffer replacement policy. There is also a clock sweep based approximation

Re: [HACKERS] Proposal for Allow postgresql.conf values to be changed via SQL [review]

2013-03-22 Thread Alvaro Herrera
Amit Kapila escribió: On Friday, March 22, 2013 8:57 AM Alvaro Herrera wrote: Amit Kapila escribió: I think adding new syntax change is little scary for me, not for the matter of implementation but for building consensus on syntax. I cannot but agree on that point. Sorry, I don't

Re: [HACKERS] Let's invent a function to report lock-wait-blocking PIDs

2013-03-22 Thread Tom Lane
Heikki Linnakangas hlinnakan...@vmware.com writes: On 21.03.2013 05:36, Tom Lane wrote: The API that comes to mind is (name subject to bikeshedding) pg_blocking_pids(pid int) returns int[] How about inverting the function into: pg_pid_blocked_by(pid int) returns int It would take as

Re: [HACKERS] Strange Windows problem, lock_timeout test request

2013-03-22 Thread Robert Haas
On Thu, Mar 21, 2013 at 8:16 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: On Mon, Mar 18, 2013 at 10:09 AM, Tom Lane t...@sss.pgh.pa.us wrote: Because it's wrong. Removing volatile means that the compiler is permitted to optimize away stores (and fetches!)

Re: [HACKERS] Enabling Checksums

2013-03-22 Thread Ants Aasma
On Fri, Mar 22, 2013 at 3:04 AM, Jeff Davis pg...@j-davis.com wrote: I've been following your analysis and testing, and it looks like there are still at least three viable approaches: 1. Some variant of Fletcher 2. Some variant of CRC32 3. Some SIMD-based checksum Each of those has some

Re: [HACKERS] JSON Function Bike Shedding

2013-03-22 Thread Merlin Moncure
On Fri, Mar 22, 2013 at 8:58 AM, Andrew Dunstan and...@dunslane.net wrote: On 03/22/2013 09:29 AM, Merlin Moncure wrote: On Mon, Mar 18, 2013 at 3:12 PM, Tom Lane t...@sss.pgh.pa.us wrote: Andrew Dunstan and...@dunslane.net writes: I've been sitting here for a while mulling none too

[HACKERS] Default connection parameters for postgres_fdw and dblink

2013-03-22 Thread Tom Lane
It struck me while looking at the regression test arrangements for postgres_fdw that as things are set up, the default username for outgoing connections is going to be that of the operating system user running the postmaster. dblink is the same way. Now, this might not be the world's worst

Re: [HACKERS] Strange Windows problem, lock_timeout test request

2013-03-22 Thread Greg Stark
On Fri, Mar 22, 2013 at 2:29 PM, Robert Haas robertmh...@gmail.com wrote: The barrier essentially divides up the code into chunks and requires that those chunks be optimized independently by the compiler without knowledge of what earlier or later chunks are doing While all this sounds

Re: [HACKERS] Enabling Checksums

2013-03-22 Thread Jeff Davis
On Fri, 2013-03-22 at 17:09 +0200, Ants Aasma wrote: For performance the K8 results gave me confidence that we have a reasonably good overview what the performance is like for the class of CPU's that PostgreSQL is likely to run on. I don't think there is anything left to optimize there, all

Re: [HACKERS] Enabling Checksums

2013-03-22 Thread Jeff Davis
On Fri, 2013-03-22 at 17:09 +0200, Ants Aasma wrote: So to conclude, the 3 approaches: One other question: assuming that the algorithms use the full 16-bit space, is there a good way to avoid zero without skewing the result? Can we do something like un-finalize (after we figure out that it's

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-03-22 Thread Greg Stark
On Fri, Mar 22, 2013 at 2:02 PM, Tom Lane t...@sss.pgh.pa.us wrote: And we definitely looked at ARC We didn't just look at it. At least one release used it. Then patent issues were raised (and I think the implementation had some contention problems). -- greg -- Sent via pgsql-hackers

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-03-22 Thread Atri Sharma
On Fri, Mar 22, 2013 at 11:36 PM, Greg Stark st...@mit.edu wrote: On Fri, Mar 22, 2013 at 2:02 PM, Tom Lane t...@sss.pgh.pa.us wrote: And we definitely looked at ARC We didn't just look at it. At least one release used it. Then patent issues were raised (and I think the implementation had

Re: [HACKERS] Enabling Checksums

2013-03-22 Thread Ants Aasma
On Fri, Mar 22, 2013 at 7:35 PM, Jeff Davis pg...@j-davis.com wrote: On Fri, 2013-03-22 at 17:09 +0200, Ants Aasma wrote: For performance the K8 results gave me confidence that we have a reasonably good overview what the performance is like for the class of CPU's that PostgreSQL is likely to

Re: [HACKERS] Materialized view assertion failure in HEAD

2013-03-22 Thread Kevin Grittner
Kevin Grittner kgri...@ymail.com wrote: Kevin Grittner kgri...@ymail.com wrote: Robert Haas robertmh...@gmail.com wrote: It seems to me that the right place to fix this is in interpretOidsOption(), by returning false rather than default_with_oids whenever the relation is a materialized view.

[HACKERS] dump, restore, dump yields differences

2013-03-22 Thread Kevin Grittner
As part of testing the matview issues around dumping, I ran `make installcheck-world`, ran pg_dump to dump the regression database, loaded it into a new database, dumped the newly restored database, and compared the output files from the two pg_dump runs.  There were a few somewhat surprising

Re: [HACKERS] dump, restore, dump yields differences

2013-03-22 Thread Tom Lane
Kevin Grittner kgri...@ymail.com writes: As part of testing the matview issues around dumping, I ran `make installcheck-world`, ran pg_dump to dump the regression database, loaded it into a new database, dumped the newly restored database, and compared the output files from the two pg_dump

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-03-22 Thread Merlin Moncure
On Fri, Mar 22, 2013 at 1:13 PM, Atri Sharma atri.j...@gmail.com wrote: On Fri, Mar 22, 2013 at 11:36 PM, Greg Stark st...@mit.edu wrote: On Fri, Mar 22, 2013 at 2:02 PM, Tom Lane t...@sss.pgh.pa.us wrote: And we definitely looked at ARC We didn't just look at it. At least one release used

Re: [HACKERS] Default connection parameters for postgres_fdw and dblink

2013-03-22 Thread Jeff Davis
On Fri, 2013-03-22 at 12:19 -0400, Tom Lane wrote: Is there a better way to handle all this? It may be too late to rethink dblink's behavior anyhow, but perhaps it's not too late to change postgres_fdw. I think though that once we let 9.3 out the door, it *will* be too late to make any major

Re: postgres_fdw vs data formatting GUCs (was Re: [HACKERS] [v9.3] writable foreign tables)

2013-03-22 Thread Tom Lane
Daniel Farina dan...@heroku.com writes: This contains some edits to comments that referred to the obsolete and bogus TupleDesc scanning. No mechanical alterations. Applied with some substantial revisions. I didn't like where you'd put the apply/restore calls, for one thing --- we need to wait

Re: [HACKERS] Materialized view assertion failure in HEAD

2013-03-22 Thread Tom Lane
Kevin Grittner kgri...@ymail.com writes: In working up a patch for this approach, I see that if CREATE FOREIGN TABLE is executed with default_with_oids set to true, it adds an oid column which appears to be always zero in my tests so far (although maybe other FDWs support it?).  Do we want to

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-03-22 Thread Tom Lane
Merlin Moncure mmonc...@gmail.com writes: On Fri, Mar 22, 2013 at 1:13 PM, Atri Sharma atri.j...@gmail.com wrote: What is the general thinking? Is it time to start testing again and thinking about improvements to the current algorithm? well, what problem are you trying to solve exactly? the

Re: [HACKERS] dump, restore, dump yields differences

2013-03-22 Thread Greg Stark
On Fri, Mar 22, 2013 at 6:49 PM, Tom Lane t...@sss.pgh.pa.us wrote: Those are expected. You can trace the ALTER TABLE history of those tables if you want to see why they're so odd, but basically there are inheritance situations where it's hard to avoid this. Incidentally it would still be

Re: [HACKERS] Default connection parameters for postgres_fdw and dblink

2013-03-22 Thread Tom Lane
Jeff Davis pg...@j-davis.com writes: On Fri, 2013-03-22 at 12:19 -0400, Tom Lane wrote: Is there a better way to handle all this? It may be too late to rethink dblink's behavior anyhow, but perhaps it's not too late to change postgres_fdw. I think though that once we let 9.3 out the door, it

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-03-22 Thread Merlin Moncure
On Fri, Mar 22, 2013 at 2:52 PM, Tom Lane t...@sss.pgh.pa.us wrote: Merlin Moncure mmonc...@gmail.com writes: On Fri, Mar 22, 2013 at 1:13 PM, Atri Sharma atri.j...@gmail.com wrote: What is the general thinking? Is it time to start testing again and thinking about improvements to the current

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-03-22 Thread Tom Lane
Merlin Moncure mmonc...@gmail.com writes: I think there is some very low hanging optimization fruit in the clock sweep loop. first and foremost, I see no good reason why when scanning pages we have to spin and wait on a buffer in order to pedantically adjust usage_count. some simple

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-03-22 Thread Merlin Moncure
On Fri, Mar 22, 2013 at 3:16 PM, Tom Lane t...@sss.pgh.pa.us wrote: Merlin Moncure mmonc...@gmail.com writes: I think there is some very low hanging optimization fruit in the clock sweep loop. first and foremost, I see no good reason why when scanning pages we have to spin and wait on a

Re: postgres_fdw vs data formatting GUCs (was Re: [HACKERS] [v9.3] writable foreign tables)

2013-03-22 Thread Daniel Farina
On Fri, Mar 22, 2013 at 12:29 PM, Tom Lane t...@sss.pgh.pa.us wrote: Daniel Farina dan...@heroku.com writes: This contains some edits to comments that referred to the obsolete and bogus TupleDesc scanning. No mechanical alterations. Applied with some substantial revisions. I didn't like

[HACKERS] Cube extension improvement, GSoC

2013-03-22 Thread Stas Kelvich
Hello, some time ago I started working on the data search system (about 100-200M of records) with queries consisted of several diapason and equality conditions, e.g.: WHERE dim1 BETWEEN 128 AND 137 AND WHERE dim2 BETWEEN 4815 AND 162342 AND WHERE dim3 = 42 ORDER BY dim1 ASC There are

[HACKERS] Re: DROP OWNED BY fails to drop privileges granted by non-owners (was Re: [GENERAL] Bug, Feature, or what else?)

2013-03-22 Thread Alvaro Herrera
Tom Lane escribió: I believe the problem is that DROP OWNED for privileges is implemented by calling REVOKE. As noted upthread, when a superuser does REVOKE, it's executed as though the object owner did the REVOKE, so only privileges granted directly by the object owner go away. In this

Re: [HACKERS] SDP query optimizer

2013-03-22 Thread Josh Berkus
Adriano, I have developed a new query optimizer for PostgreSQL and I would like to share it with the community. The optimizer's name is Sampling and Dynamic Programming (SDP). I put it into a plugin developed some years ago, named LJQO: Woah! Way cool. As a warning, we're in the closing

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-03-22 Thread Ants Aasma
On Fri, Mar 22, 2013 at 10:22 PM, Merlin Moncure mmonc...@gmail.com wrote: well if you do a non-locking test first you could at least avoid some cases (and, if you get the answer wrong, so what?) by jumping to the next buffer immediately. if the non locking test comes good, only then do you

Re: [HACKERS] SDP query optimizer

2013-03-22 Thread Ants Aasma
On Sat, Mar 23, 2013 at 1:35 AM, Adriano Lange alange0...@gmail.com wrote: I have developed a new query optimizer for PostgreSQL and I would like to share it with the community. The optimizer's name is Sampling and Dynamic Programming (SDP). I put it into a plugin developed some years ago,

Re: [HACKERS] Proposal for Allow postgresql.conf values to be changed via SQL [review]

2013-03-22 Thread Amit Kapila
On Friday, March 22, 2013 7:33 PM Alvaro Herrera wrote: Amit Kapila escribió: On Friday, March 22, 2013 8:57 AM Alvaro Herrera wrote: Amit Kapila escribió: I think adding new syntax change is little scary for me, not for the matter of implementation but for building consensus on

Re: [HACKERS] SIGHUP not received by custom bgworkers if postmaster is notified

2013-03-22 Thread Michael Paquier
Hi all, Please find attached a simple example of bgworker that logs a message each time a SIGTERM or SIGHUP signal is received by it: - hello signal: processed SIGHUP when SIGHUP is handled by my example - hello signal: processed SIGTERM when SIGTERM is handled by my example With the current

Re: [HACKERS] Enabling Checksums

2013-03-22 Thread Craig Ringer
On 03/23/2013 02:00 AM, Jeff Davis wrote: On Fri, 2013-03-22 at 17:09 +0200, Ants Aasma wrote: So to conclude, the 3 approaches: One other question: assuming that the algorithms use the full 16-bit space, is there a good way to avoid zero without skewing the result? Can we do something like

Re: [HACKERS] Page replacement algorithm in buffer cache

2013-03-22 Thread Atri Sharma
Moreover, if the buffer happens to miss a decrement due to a data race, there's a good chance that the buffer is heavily used and wouldn't need to be evicted soon anyway. (if you arrange it to be a read-test-inc/dec-store operation then you will never go out of bounds) However, clocksweep

[HACKERS] pg_dump/restore syntax checking bug?

2013-03-22 Thread Joshua D. Drake
Hello, In testing some pg_restore functionality I found the following: postgres@jd-laptop:~$ pg_dump -U postgres -Fc -s --file=foo.sqlc postgres@jd-laptop:~$ dropdb test; postgres@jd-laptop:~$ createdb test; postgres@jd-laptop:~$ pg_restore -d test -P 'by()' foo.sqlc postgres@jd-laptop:~$ psql

Re: [HACKERS] pg_dump/restore syntax checking bug?

2013-03-22 Thread Josh Kupershmidt
On Fri, Mar 22, 2013 at 9:35 PM, Joshua D. Drake j...@commandprompt.com wrote: postgres@jd-laptop:~$ pg_restore -d test -P 'by(),hello()' foo.sqlc Note, the pg_restore doc makes no mention of trying to squeeze multiple function prototypes in a single argument you've done here, or of using