Re: [HACKERS] PostgreSQL in Windows console and Ctrl-C

2014-04-14 Thread Christian Ullrich
* From: Amit Kapila On Sun, Apr 13, 2014 at 5:59 PM, Christian Ullrich ch...@chrullrich.net wrote: There are some possible solutions: - pg_ctl could set an environment variable (unless it has to be compatible with postmasters from different versions, and it does not, does it?).

Re: Race condition between PREPARE TRANSACTION and COMMIT PREPARED (was Re: [HACKERS] Problem with txid_snapshot_in/out() functionality)

2014-04-14 Thread Heikki Linnakangas
On 04/13/2014 11:39 PM, Heikki Linnakangas wrote: However, I just noticed that there's a race condition between PREPARE TRANSACTION and COMMIT/ROLLBACK PREPARED. PostPrepare_Locks runs after the prepared transaction is already marked as fully prepared. That means that by the time we get to

Re: [HACKERS] Problem with txid_snapshot_in/out() functionality

2014-04-14 Thread Marko Kreen
On Sun, Apr 13, 2014 at 05:46:20PM -0400, Jan Wieck wrote: On 04/13/14 14:22, Jan Wieck wrote: On 04/13/14 08:27, Marko Kreen wrote: I think you need to do SET_VARSIZE also here. Alternative is to move SET_VARSIZE after sort_snapshot(). And it seems the drop-double-txid logic should be

Re: [HACKERS] Adding unsigned 256 bit integers

2014-04-14 Thread Olivier Lalonde
Thanks for helping me out everyone. I ended up simply using the numeric type (I didn't realize it could support such large numbers) and writing the hex-to-numeric conversion functions in my application code. On 11 April 2014 12:27, Leon Smith leon.p.sm...@gmail.com wrote: pgmp is also worth

[HACKERS] Minor improvements in create_foreign_table.sgml

2014-04-14 Thread Etsuro Fujita
Attached is a small patch to improve create_foreign_table.sgml. Thanks, Best regards, Etsuro Fujita diff --git a/doc/src/sgml/ref/create_foreign_table.sgml b/doc/src/sgml/ref/create_foreign_table.sgml index 06a7087..4a8cf38 100644 --- a/doc/src/sgml/ref/create_foreign_table.sgml +++

Re: [HACKERS] Problem with txid_snapshot_in/out() functionality

2014-04-14 Thread Heikki Linnakangas
On 04/12/2014 05:03 PM, Andres Freund wrote: On 2014-04-12 09:47:24 -0400, Tom Lane wrote: Heikki Linnakangas hlinnakan...@vmware.com writes: On 04/12/2014 12:07 AM, Jan Wieck wrote: the Slony team has been getting seldom reports of a problem with the txid_snapshot data type. The symptom is

Re: [HACKERS] Problem with txid_snapshot_in/out() functionality

2014-04-14 Thread Andres Freund
On 2014-04-14 12:15:30 +0300, Heikki Linnakangas wrote: Hmm. There's a field in GlobalTransactionData called locking_xid, which is used to mark the XID of the transaction that's currently operating on the prepared transaction. At prepare, that ensures that the transaction cannot be committed

Re: Custom Scan APIs (Re: [HACKERS] Custom Plan node)

2014-04-14 Thread Simon Riggs
On 24 March 2014 10:25, Kouhei Kaigai kai...@ak.jp.nec.com wrote: Brief summary of the current approach that has been revised from my original submission through the discussion on pgsql-hackers: The plannode was renamed to CustomPlan, instead of CustomScan, because it dropped all the

[HACKERS] Re: Window function optimisation, allow pushdowns of items matching PARTITION BY clauses

2014-04-14 Thread David Rowley
On 14 April 2014 02:50, Thomas Mayer thomas.ma...@student.kit.edu wrote: Hello David, thanks for your work. The results look promising. Thanks What I'm missing is a test case with multiple fields in the partition by clauses: I've modified the patch and added some regression tests that

Re: [HACKERS] Window function optimisation, allow pushdowns of items matching PARTITION BY clauses

2014-04-14 Thread David Rowley
On 14 April 2014 03:31, Tom Lane t...@sss.pgh.pa.us wrote: David Rowley dgrow...@gmail.com writes: On this thread http://www.postgresql.org/message-id/52c6f712.6040...@student.kit.eduthere was some discussion around allowing push downs of quals that happen to be in every window clause of

Re: [HACKERS] Patch: add psql tab completion for event triggers

2014-04-14 Thread Robert Haas
On Wed, Apr 9, 2014 at 8:58 PM, Ian Barwick i...@2ndquadrant.com wrote: Apologies again, that was ill-thought out. Revised patch attached with only the additions related to event triggers, and the small fix for ALTER TRIGGER mentioned above which ensures RENAME TO is applied only when ALTER

Re: [HACKERS] four minor proposals for 9.5

2014-04-14 Thread Robert Haas
On Tue, Apr 8, 2014 at 12:34 PM, Gregory Smith gregsmithpg...@gmail.com wrote: On 4/6/14 2:46 PM, Pavel Stehule wrote: Proposed options are interesting for enterprise using, when you have a some more smart tools for log entry processing, and when you need a complex view about performance of

Re: [HACKERS] Including replication slot data in base backups

2014-04-14 Thread Fujii Masao
On Thu, Apr 10, 2014 at 12:36 AM, Robert Haas robertmh...@gmail.com wrote: On Tue, Apr 8, 2014 at 3:08 AM, Michael Paquier michael.paqu...@gmail.com wrote: On Tue, Apr 8, 2014 at 1:18 AM, Robert Haas robertmh...@gmail.com wrote: Not sure if this is exactly the right way to do it, but I agree

Re: [HACKERS] four minor proposals for 9.5

2014-04-14 Thread Pavel Stehule
2014-04-14 14:57 GMT+02:00 Robert Haas robertmh...@gmail.com: On Tue, Apr 8, 2014 at 12:34 PM, Gregory Smith gregsmithpg...@gmail.com wrote: On 4/6/14 2:46 PM, Pavel Stehule wrote: Proposed options are interesting for enterprise using, when you have a some more smart tools for log entry

Re: [HACKERS] Patch: add psql tab completion for event triggers

2014-04-14 Thread Michael Paquier
On Mon, Apr 14, 2014 at 9:46 PM, Robert Haas robertmh...@gmail.com wrote: On Wed, Apr 9, 2014 at 8:58 PM, Ian Barwick i...@2ndquadrant.com wrote: Apologies again, that was ill-thought out. Revised patch attached with only the additions related to event triggers, and the small fix for ALTER

Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

2014-04-14 Thread Fujii Masao
On Tue, Apr 1, 2014 at 1:41 AM, Robert Haas robertmh...@gmail.com wrote: On Thu, Mar 20, 2014 at 1:12 PM, Jesper Krogh jes...@krogh.cc wrote: On 15/03/14 20:27, Heikki Linnakangas wrote: That said, I didn't expect the difference to be quite that big when you're appending to the end of the

Re: [HACKERS] Autonomous Transaction (WIP)

2014-04-14 Thread Simon Riggs
On 7 April 2014 05:06, Rajeev rastogi rajeev.rast...@huawei.com wrote: *Autonomous Transaction Storage:* As for main transaction, structure PGXACT is used to store main transactions, which are created in shared memory of size: (Number of process)*sizeof(struct

Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

2014-04-14 Thread Tom Lane
Fujii Masao masao.fu...@gmail.com writes: On Tue, Apr 1, 2014 at 1:41 AM, Robert Haas robertmh...@gmail.com wrote: Should we try to install some hack around fastupdate for 9.4? I fear the divergence between reasonable values of work_mem and reasonable sizes for that list is only going to

Re: [HACKERS] Signaling of waiting for a cleanup lock?

2014-04-14 Thread Simon Riggs
On 13 April 2014 16:44, Andres Freund and...@2ndquadrant.com wrote: On 2014-04-12 17:40:34 -0400, Robert Haas wrote: On Fri, Apr 11, 2014 at 10:28 AM, Andres Freund and...@2ndquadrant.com wrote: VACUUM sometimes waits synchronously for a cleanup lock on a heap page. Sometimes for a long

Re: [HACKERS] Signaling of waiting for a cleanup lock?

2014-04-14 Thread Andres Freund
On 2014-04-14 15:45:45 +0100, Simon Riggs wrote: On 13 April 2014 16:44, Andres Freund and...@2ndquadrant.com wrote: On 2014-04-12 17:40:34 -0400, Robert Haas wrote: On Fri, Apr 11, 2014 at 10:28 AM, Andres Freund and...@2ndquadrant.com wrote: VACUUM sometimes waits synchronously for a

Re: [HACKERS] Minor improvements in alter_table.sgml

2014-04-14 Thread Robert Haas
On Fri, Apr 11, 2014 at 5:00 AM, Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote: Attached is an updated version of the patch. I applied the first two hunks of this, which seem like clear oversights; and also the bit fixing the constraint_name language. I think the other changes deserve to be

Re: [HACKERS] four minor proposals for 9.5

2014-04-14 Thread Tom Lane
Pavel Stehule pavel.steh...@gmail.com writes: 2014-04-14 14:57 GMT+02:00 Robert Haas robertmh...@gmail.com: I agree. I don't think the idea of pushing this into the log_line_prefix stuff as a one-off is a very good one. Sure, we could wedge it in there, but we've got an existing precedent

Re: [HACKERS] Signaling of waiting for a cleanup lock?

2014-04-14 Thread Robert Haas
On Mon, Apr 14, 2014 at 10:50 AM, Andres Freund and...@2ndquadrant.com wrote: On 2014-04-14 15:45:45 +0100, Simon Riggs wrote: On 13 April 2014 16:44, Andres Freund and...@2ndquadrant.com wrote: On 2014-04-12 17:40:34 -0400, Robert Haas wrote: On Fri, Apr 11, 2014 at 10:28 AM, Andres Freund

Re: [HACKERS] Signaling of waiting for a cleanup lock?

2014-04-14 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: On 2014-04-14 15:45:45 +0100, Simon Riggs wrote: On 13 April 2014 16:44, Andres Freund and...@2ndquadrant.com wrote: What I am not sure about is how... It's trivial to set pg_stat_activity.waiting = true, but without a corresponding description

[HACKERS] server vs foreign server inconsistency

2014-04-14 Thread Jaime Casanova
Hi, A few days ago i was wondering why we use CREATE/DROP SERVER but then when we want to GRANT/REVOKE we need to use FOREIGN SERVER. of course options are: 1) modify both to accept both forms 2) modify one of them to accept both forms and use that way in all our examples in docs 3) do nothing

Re: [HACKERS] Signaling of waiting for a cleanup lock?

2014-04-14 Thread Andres Freund
On 2014-04-14 11:30:02 -0400, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On 2014-04-14 15:45:45 +0100, Simon Riggs wrote: On 13 April 2014 16:44, Andres Freund and...@2ndquadrant.com wrote: What I am not sure about is how... It's trivial to set pg_stat_activity.waiting =

Re: [HACKERS] server vs foreign server inconsistency

2014-04-14 Thread Tom Lane
Jaime Casanova ja...@2ndquadrant.com writes: A few days ago i was wondering why we use CREATE/DROP SERVER but then when we want to GRANT/REVOKE we need to use FOREIGN SERVER. Because the SQL standard says so. regards, tom lane -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Signaling of waiting for a cleanup lock?

2014-04-14 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: On 2014-04-14 11:30:02 -0400, Tom Lane wrote: I wonder whether we should not try to fix this by making the process wait on a heavyweight lock, if it has to wait. That would also get us out of the rather grotty business of using a special-purpose

Re: [HACKERS] Signaling of waiting for a cleanup lock?

2014-04-14 Thread Robert Haas
On Mon, Apr 14, 2014 at 12:02 PM, Tom Lane t...@sss.pgh.pa.us wrote: Andres Freund and...@2ndquadrant.com writes: On 2014-04-14 11:30:02 -0400, Tom Lane wrote: I wonder whether we should not try to fix this by making the process wait on a heavyweight lock, if it has to wait. That would also

Re: [HACKERS] Signaling of waiting for a cleanup lock?

2014-04-14 Thread Andres Freund
On 2014-04-14 12:02:22 -0400, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On 2014-04-14 11:30:02 -0400, Tom Lane wrote: I wonder whether we should not try to fix this by making the process wait on a heavyweight lock, if it has to wait. That would also get us out of the

Re: [HACKERS] Dynamic Shared Memory stuff

2014-04-14 Thread Robert Haas
On Sat, Apr 12, 2014 at 1:32 AM, Amit Kapila amit.kapil...@gmail.com wrote: On Wed, Apr 9, 2014 at 9:20 PM, Robert Haas robertmh...@gmail.com wrote: On Wed, Apr 9, 2014 at 7:41 AM, Amit Kapila amit.kapil...@gmail.com wrote: I am just not sure whether it is okay to rearrange the code and call

Re: [HACKERS] Idea for aggregates

2014-04-14 Thread Simon Riggs
On 5 April 2014 04:18, Tom Lane t...@sss.pgh.pa.us wrote: Greg Stark st...@mit.edu writes: Well in many cases stype will just be internal for many of them. That doesn't mean they're the same. Hm, I suppose it might if they have the same sfunc. This is actually where I started but we

Re: [HACKERS] [PATCH] Negative Transition Aggregate Functions (WIP)

2014-04-14 Thread Robert Haas
On Sun, Apr 13, 2014 at 2:23 PM, Tom Lane t...@sss.pgh.pa.us wrote: * I also left out the table documenting which aggregates have this optimization. That's not the kind of thing we ordinarily document, and it seems inevitable to me that such a table would be noteworthy mostly for

Re: [HACKERS] Minor improvements in create_foreign_table.sgml

2014-04-14 Thread Robert Haas
On Mon, Apr 14, 2014 at 5:03 AM, Etsuro Fujita fujita.ets...@lab.ntt.co.jp wrote: Attached is a small patch to improve create_foreign_table.sgml. OK, committed. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list

Re: Race condition between PREPARE TRANSACTION and COMMIT PREPARED (was Re: [HACKERS] Problem with txid_snapshot_in/out() functionality)

2014-04-14 Thread Tom Lane
Heikki Linnakangas hlinnakan...@vmware.com writes: I think we'll need to transfer of the locks earlier, before the transaction is marked as fully prepared. I'll take a closer look at this tomorrow. Here's a patch to do that. It's very straightforward, I just moved the calls to transfer

Re: [HACKERS] Signaling of waiting for a cleanup lock?

2014-04-14 Thread Andres Freund
On 2014-04-14 12:21:09 -0400, Robert Haas wrote: AFAICS, the big advantage of something like this is that we'd get proper deadlock detection, and that's not a trivial point. Hm. Is this actually something we need? I am not aware of deadlock prone scenarios involving buffer pins during normal

Re: [HACKERS] Including replication slot data in base backups

2014-04-14 Thread Robert Haas
On Mon, Apr 14, 2014 at 9:26 AM, Fujii Masao masao.fu...@gmail.com wrote: This makes me think that it's safer to just remove replication slot files at the beginning of the recovery when both backup_label and recovery.conf exist. Well, we could do that, but that would preempt anyone who *does*

Re: [HACKERS] Signaling of waiting for a cleanup lock?

2014-04-14 Thread Alvaro Herrera
Tom Lane wrote: In an ideal world, when we needed to wait for a cleanup lock, we'd cause the lock manager to set up pre-granted sharable page locks for all the processes currently holding buffer pins, and then wait for an exclusive page lock. The current hack of signaling when you're the

Re: [HACKERS] Signaling of waiting for a cleanup lock?

2014-04-14 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: On 2014-04-14 12:21:09 -0400, Robert Haas wrote: AFAICS, the big advantage of something like this is that we'd get proper deadlock detection, and that's not a trivial point. Hm. Is this actually something we need? I am not aware of deadlock prone

Re: [HACKERS] Signaling of waiting for a cleanup lock?

2014-04-14 Thread Andres Freund
On 2014-04-14 13:06:21 -0400, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On 2014-04-14 12:21:09 -0400, Robert Haas wrote: AFAICS, the big advantage of something like this is that we'd get proper deadlock detection, and that's not a trivial point. Hm. Is this actually

Re: Race condition between PREPARE TRANSACTION and COMMIT PREPARED (was Re: [HACKERS] Problem with txid_snapshot_in/out() functionality)

2014-04-14 Thread Andres Freund
On 2014-04-14 12:51:02 -0400, Tom Lane wrote: The whole thing feels like we are solving the wrong problem, anyway. IIUC, the complaint arises because we are allowing COMMIT PREPARED to occur before the source transaction has reported successful prepare to its client. Surely that does not need

Re: Race condition between PREPARE TRANSACTION and COMMIT PREPARED (was Re: [HACKERS] Problem with txid_snapshot_in/out() functionality)

2014-04-14 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: I wonder if the most natural way to express this wouldn't be to have a heavyweight lock for every 2pc xact 'slot'. ResourceOwnerRelease(RESOURCE_RELEASE_LOCKS) should be scheduled correctly to make error handling for this work. That seems like not

Re: Race condition between PREPARE TRANSACTION and COMMIT PREPARED (was Re: [HACKERS] Problem with txid_snapshot_in/out() functionality)

2014-04-14 Thread Andres Freund
On 2014-04-14 13:47:35 -0400, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: I wonder if the most natural way to express this wouldn't be to have a heavyweight lock for every 2pc xact 'slot'. ResourceOwnerRelease(RESOURCE_RELEASE_LOCKS) should be scheduled correctly to make

Re: [HACKERS] Signaling of waiting for a cleanup lock?

2014-04-14 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: On 2014-04-14 13:06:21 -0400, Tom Lane wrote: In particular I'm not sold on the use-case for being able to tell that a process is waiting without being able to tell what it's waiting for. I can figure that much out already. You can? How? It could

Re: [HACKERS] Signaling of waiting for a cleanup lock?

2014-04-14 Thread Robert Haas
On Mon, Apr 14, 2014 at 1:26 PM, Andres Freund and...@2ndquadrant.com wrote: On 2014-04-14 13:06:21 -0400, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On 2014-04-14 12:21:09 -0400, Robert Haas wrote: AFAICS, the big advantage of something like this is that we'd get proper

Re: Race condition between PREPARE TRANSACTION and COMMIT PREPARED (was Re: [HACKERS] Problem with txid_snapshot_in/out() functionality)

2014-04-14 Thread Heikki Linnakangas
On 04/14/2014 07:51 PM, Tom Lane wrote: I'd prefer to leave the prepare sequence alone and instead find a way to reject COMMIT PREPARED until after the source transaction is safely clear of the race conditions. The upthread idea of looking at vxid instead of xid might help, except that I see we

Re: [HACKERS] Create function prototype as part of PG_FUNCTION_INFO_V1

2014-04-14 Thread Peter Eisentraut
On 4/4/14, 10:07 AM, Andres Freund wrote: If somebody previously tried to do the correct thing and attached PGDLLEXPORT to their own *function* prototoype, it would cause problems now. What is the difference (on affected platforms) between Datum funcname(PG_FUNCTION_ARGS); and writing

Re: [HACKERS] Patch to add support of IF NOT EXISTS to others CREATE statements

2014-04-14 Thread Fabrízio de Royes Mello
On Tue, Apr 1, 2014 at 2:46 PM, Robert Haas robertmh...@gmail.com wrote: Where this is a bit more interesting is in the case of sequences, where resetting the sequence to zero may cause further inserts into an existing table to fail. Yeah. Sequences do have contained data, which makes

Re: [HACKERS] Create function prototype as part of PG_FUNCTION_INFO_V1

2014-04-14 Thread Tom Lane
Peter Eisentraut pete...@gmx.net writes: What is the difference (on affected platforms) between Datum funcname(PG_FUNCTION_ARGS); and writing (effectively) PGDLLEXPORT Datum funcname(PG_FUNCTION_ARGS); Datum funcname(PG_FUNCTION_ARGS); or for that matter Datum

[HACKERS] JSONB in-place updates?

2014-04-14 Thread kelas
Are there any plans to add in-place at-depth update operator for JSONB type, e.g.: UPDATE test SET attrs-'anwser' = 42 where attrs-'answer' = 41

Re: [HACKERS] JSONB in-place updates?

2014-04-14 Thread Josh Berkus
On 04/14/2014 09:27 AM, kelas wrote: Are there any plans to add in-place at-depth update operator for JSONB type, e.g.: UPDATE test SET attrs-'anwser' = 42 where attrs-'answer' = 41 Plans, yes. But not until 9.5, or maybe as an extension. -- Josh Berkus PostgreSQL Experts Inc.

Re: Custom Scan APIs (Re: [HACKERS] Custom Plan node)

2014-04-14 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: [ assorted comments about custom-scan patch, but particularly ] * The prune hook makes me feel very uneasy. It seems weirdly specific implementation detail, made stranger by the otherwise lack of data maintenance API calls. Calling that for every

[HACKERS] Excessive WAL generation and related performance issue

2014-04-14 Thread Joe Conway
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 I have run into a situation where bulk loading a table with fairly narrow rows and two indexes causes WAL to be generated at about 20:1 or higher ratio to the actual heap data (table plus indexes). There are 560 million loaded rows which ultimately

Re: [HACKERS] Excessive WAL generation and related performance issue

2014-04-14 Thread Andres Freund
Hi, On 2014-04-14 14:33:03 -0700, Joe Conway wrote: checkpoint_segments = 96 checkpoint_timeout = 10min I realize there are many things that can be done to improve my specific scenario, e.g. drop indexes before loading, change various configs, etc. My purpose for this post is to ask if it

Re: [HACKERS] Excessive WAL generation and related performance issue

2014-04-14 Thread Jim Nasby
On 4/14/14, 4:50 PM, Andres Freund wrote: Hi, On 2014-04-14 14:33:03 -0700, Joe Conway wrote: checkpoint_segments = 96 checkpoint_timeout = 10min I realize there are many things that can be done to improve my specific scenario, e.g. drop indexes before loading, change various configs, etc.

Re: [HACKERS] Signaling of waiting for a cleanup lock?

2014-04-14 Thread Jim Nasby
On 4/14/14, 12:06 PM, Tom Lane wrote: One concrete reason not to do the proposed trivial hack is that the lock readout views are asynchronous. Right now, if someone sees a process that claims to be waiting but they don't see any entry in pg_locks, they know they saw inconsistent state. If we

Re: [HACKERS] Signaling of waiting for a cleanup lock?

2014-04-14 Thread Jim Nasby
On 4/14/14, 12:44 PM, Tom Lane wrote: Andres Freundand...@2ndquadrant.com writes: On 2014-04-14 13:06:21 -0400, Tom Lane wrote: In particular I'm not sold on the use-case for being able to tell that a process is waiting without being able to tell what it's waiting for. I can figure that much

Re: [HACKERS] Excessive WAL generation and related performance issue

2014-04-14 Thread Joe Conway
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/14/2014 03:17 PM, Jim Nasby wrote: On 4/14/14, 4:50 PM, Andres Freund wrote: On 2014-04-14 14:33:03 -0700, Joe Conway wrote: I realize there are many things that can be done to improve my specific scenario, e.g. drop indexes before loading,

Re: [HACKERS] PostgreSQL in Windows console and Ctrl-C

2014-04-14 Thread Bruce Momjian
On Mon, Apr 14, 2014 at 09:34:14AM +0530, Amit Kapila wrote: The problem can be solved this way, but the only question here is whether it is acceptable for users to have a new console window for server. Can others also please share their opinion if this fix (start server in new console)

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-14 Thread Jim Nasby
On 4/14/14, 12:11 PM, Peter Geoghegan wrote: I have some theories about the PostgreSQL buffer manager/clock sweep. To motivate the reader to get through the material presented here, I present up-front a benchmark of a proof-of-concept patch of mine:

Re: [HACKERS] Excessive WAL generation and related performance issue

2014-04-14 Thread Jim Nasby
On 4/14/14, 5:51 PM, Joe Conway wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/14/2014 03:17 PM, Jim Nasby wrote: On 4/14/14, 4:50 PM, Andres Freund wrote: On 2014-04-14 14:33:03 -0700, Joe Conway wrote: I realize there are many things that can be done to improve my specific

Re: [HACKERS] Including replication slot data in base backups

2014-04-14 Thread Michael Paquier
On Tue, Apr 15, 2014 at 1:55 AM, Robert Haas robertmh...@gmail.com wrote: On Mon, Apr 14, 2014 at 9:26 AM, Fujii Masao masao.fu...@gmail.com wrote: This makes me think that it's safer to just remove replication slot files at the beginning of the recovery when both backup_label and recovery.conf

Re: [HACKERS] Excessive WAL generation and related performance issue

2014-04-14 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: On 2014-04-14 14:33:03 -0700, Joe Conway wrote: checkpoint_segments = 96 checkpoint_timeout = 10min I bet you'll see noticeably - while still not great - better performance by setting checkpoint_timeout to an hour (with a corresponding increase

Re: [HACKERS] Excessive WAL generation and related performance issue

2014-04-14 Thread Joe Conway
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/14/2014 04:17 PM, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On 2014-04-14 14:33:03 -0700, Joe Conway wrote: checkpoint_segments = 96 checkpoint_timeout = 10min I bet you'll see noticeably - while still not great - better

Re: [HACKERS] Excessive WAL generation and related performance issue

2014-04-14 Thread Andres Freund
On 2014-04-14 16:22:48 -0700, Joe Conway wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/14/2014 04:17 PM, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On 2014-04-14 14:33:03 -0700, Joe Conway wrote: checkpoint_segments = 96 checkpoint_timeout = 10min I

Re: [HACKERS] Excessive WAL generation and related performance issue

2014-04-14 Thread Joe Conway
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/14/2014 04:25 PM, Andres Freund wrote: On 2014-04-14 16:22:48 -0700, Joe Conway wrote: That'll help performance, but lets say I generally keep WAL files for PITR and don't turn that off before starting -- shouldn't I be very surprised to

Re: Custom Scan APIs (Re: [HACKERS] Custom Plan node)

2014-04-14 Thread Kouhei Kaigai
On 24 March 2014 10:25, Kouhei Kaigai kai...@ak.jp.nec.com wrote: Brief summary of the current approach that has been revised from my original submission through the discussion on pgsql-hackers: The plannode was renamed to CustomPlan, instead of CustomScan, because it dropped all the

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-14 Thread Bruce Momjian
On Mon, Apr 14, 2014 at 10:11:53AM -0700, Peter Geoghegan wrote: Has anyone thought about this in the last few years? I know that Tom examined the LRU-K paper back in 2000 [5], but was discouraged by some kind of contention or CPU overhead (although he did say he intended to revisit the

Re: [HACKERS] Excessive WAL generation and related performance issue

2014-04-14 Thread Stephen Frost
* Joe Conway (m...@joeconway.com) wrote: That's the thing. I'm sure there is tuning and other things to improve this particular case, but creating over 20 times as much WAL as real data seems like pathological behavior to me. Setting things up such that you are updating a single value on each

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-14 Thread Stephen Frost
* Jim Nasby (j...@nasby.net) wrote: I think it's important to mention that OS implementations (at least all I know of) have multiple page pools, each of which has it's own clock. IIRC one of the arguments for us supporting a count1 was we could get the benefits of multiple page pools

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-14 Thread Peter Geoghegan
On Mon, Apr 14, 2014 at 5:30 PM, Bruce Momjian br...@momjian.us wrote: I am glad you are looking at this. You are right that it requires a huge amount of testing, but clearly our code needs improvement in this area. Thanks. Does anyone recall the original justification for the recommendation

Re: [HACKERS] [GENERAL] CLOB BLOB limitations in PostgreSQL

2014-04-14 Thread Bruce Momjian
On Mon, Apr 14, 2014 at 02:01:19PM +0200, Ivan Voras wrote: On 11/04/2014 16:45, Jack.O'sulli...@tessella.com wrote: With point two, does this mean that any table with a bytea datatype is limited to 4 billion rows (which would seem in conflict with the unlimited rows shown by

Re: Custom Scan APIs (Re: [HACKERS] Custom Plan node)

2014-04-14 Thread Kouhei Kaigai
Simon Riggs si...@2ndquadrant.com writes: [ assorted comments about custom-scan patch, but particularly ] * The prune hook makes me feel very uneasy. It seems weirdly specific implementation detail, made stranger by the otherwise lack of data maintenance API calls. Calling that for

[HACKERS] PostgreSQL hang on FreeBSD,with CFLAGS='-O2 -pthread' workaround

2014-04-14 Thread Jov
hi~ pg hackers, I find some problems when use pg on FreeBSD.On FreeBSD,If installed extension which pthread lib is used,for example plv8,pljava,imcs etc,when query touch these extenstions,the PG backend will hang. there is a solution,which configure postgresql with CFLAGS='-O2 -pthread' and

Re: [HACKERS] Excessive WAL generation and related performance issue

2014-04-14 Thread Joe Conway
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 On 04/14/2014 05:40 PM, Stephen Frost wrote: This sounds like a great example of the unlogged table - logged table use-case and makes me wonder if we could provide an optimization similar to the existing CREATE TABLE + COPY under wal_level =

[HACKERS] assertion failure 9.3.4

2014-04-14 Thread Andrew Dunstan
With a client's code I have just managed to produce the following assertion failure on 9.3.4: 2014-04-15 01:02:46 GMT [19854] 76299: LOG: execute unnamed: select * from asp_ins_event_task_log( job_id:=1, event_id:=3164, task_name:='EventUtcComputeTask', task_status_code:='VALID'

Re: [HACKERS] [GENERAL] CLOB BLOB limitations in PostgreSQL

2014-04-14 Thread David G Johnston
Bruce Momjian wrote On Mon, Apr 14, 2014 at 02:01:19PM +0200, Ivan Voras wrote: On 11/04/2014 16:45, Jack.O' Sullivan@ wrote: With point two, does this mean that any table with a bytea datatype is limited to 4 billion rows (which would seem in conflict with the unlimited rows shown

Re: [HACKERS] [GENERAL] CLOB BLOB limitations in PostgreSQL

2014-04-14 Thread Tom Lane
Bruce Momjian br...@momjian.us writes: Uh, I had not thought of this before but I think we need oids for toast storage, which would explain this wiki text: https://wiki.postgresql.org/wiki/BinaryFilesInDB Storing binary data using bytea or text data types Minus

Re: [HACKERS] assertion failure 9.3.4

2014-04-14 Thread Andrew Dunstan
On 04/14/2014 09:28 PM, Andrew Dunstan wrote: With a client's code I have just managed to produce the following assertion failure on 9.3.4: 2014-04-15 01:02:46 GMT [19854] 76299: LOG: execute unnamed: select * from asp_ins_event_task_log( job_id:=1, event_id:=3164,

Re: [HACKERS] PostgreSQL hang on FreeBSD,with CFLAGS='-O2 -pthread' workaround

2014-04-14 Thread Tom Lane
Jov am...@amutu.com writes: I find some problems when use pg on FreeBSD.On FreeBSD,If installed extension which pthread lib is used,for example plv8,pljava,imcs etc,when query touch these extenstions,the PG backend will hang. there is a solution,which configure postgresql with CFLAGS='-O2

Re: [HACKERS] tests for client programs

2014-04-14 Thread Peter Eisentraut
On 4/4/14, 10:44 AM, Andres Freund wrote: I personally would very much like to get this patch commited. It doesn't have much risk in destabilizing stuff, rather the contrary. Peter, what's you opinion about the current state? I opine it's committed. ;-) -- Sent via pgsql-hackers mailing

Re: [HACKERS] assertion failure 9.3.4

2014-04-14 Thread Alvaro Herrera
Andrew Dunstan wrote: and here the stack trace: #0 0x00361ba36285 in __GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 #1 0x00361ba37b9b in __GI_abort () at abort.c:91 #2 0x0075c157 in ExceptionalCondition (conditionName=optimized out,

Re: [HACKERS] API change advice: Passing plan invalidation info from the rewriter into the planner?

2014-04-14 Thread Stephen Frost
Craig, Tom, all, I've been through the RLS code over the past couple of days which I pulled from Craig's repo and have a bunch of minor updates. In general, the patch seems pretty reasonable- except for the issues discussed below. Quite a bit of this patch is tied up in plan invalidation and

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-14 Thread Bruce Momjian
On Mon, Apr 14, 2014 at 05:45:56PM -0700, Peter Geoghegan wrote: On Mon, Apr 14, 2014 at 5:30 PM, Bruce Momjian br...@momjian.us wrote: I am glad you are looking at this. You are right that it requires a huge amount of testing, but clearly our code needs improvement in this area.

Re: [HACKERS] API change advice: Passing plan invalidation info from the rewriter into the planner?

2014-04-14 Thread Tom Lane
Stephen Frost sfr...@snowman.net writes: I've uploaded the latest patch, rebased against master, with my changes to here: http://snowman.net/~sfrost/rls_ringerc_sf.patch.gz as I don't believe it'd clear the mailing list (it's 29k). Please actually post it, for the archives' sake. 29k is far

Re: [HACKERS] [COMMITTERS] pgsql: Add TAP tests for client programs

2014-04-14 Thread Tom Lane
Peter Eisentraut pete...@gmx.net writes: Add TAP tests for client programs I assume the buildfarm would need to be taught about this? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

Re: [HACKERS] API change advice: Passing plan invalidation info from the rewriter into the planner?

2014-04-14 Thread Stephen Frost
* Tom Lane (t...@sss.pgh.pa.us) wrote: Stephen Frost sfr...@snowman.net writes: I've uploaded the latest patch, rebased against master, with my changes to here: http://snowman.net/~sfrost/rls_ringerc_sf.patch.gz as I don't believe it'd clear the mailing list (it's 29k). Please actually

Re: [HACKERS] assertion failure 9.3.4

2014-04-14 Thread Andrew Dunstan
On 04/14/2014 10:02 PM, Alvaro Herrera wrote: Andrew Dunstan wrote: and here the stack trace: #0 0x00361ba36285 in __GI_raise (sig=6) at ../nptl/sysdeps/unix/sysv/linux/raise.c:64 #1 0x00361ba37b9b in __GI_abort () at abort.c:91 #2 0x0075c157 in

Re: [HACKERS] JSONB in-place updates?

2014-04-14 Thread Michael Paquier
On Tue, Apr 15, 2014 at 5:16 AM, Josh Berkus j...@agliodbs.com wrote: On 04/14/2014 09:27 AM, kelas wrote: Are there any plans to add in-place at-depth update operator for JSONB type, e.g.: UPDATE test SET attrs-'anwser' = 42 where attrs-'answer' = 41 Plans, yes. But not until 9.5, or

Re: [HACKERS] [COMMITTERS] pgsql: Add TAP tests for client programs

2014-04-14 Thread Andrew Dunstan
On 04/14/2014 10:17 PM, Tom Lane wrote: Peter Eisentraut pete...@gmx.net writes: Add TAP tests for client programs I assume the buildfarm would need to be taught about this? Yes. It probably won't be a huge change, but it will need a bit of code. cheers andrew

[HACKERS] Fwd: [BUGS] Debug strategy for musl Postgres?

2014-04-14 Thread John Mudd
On Mon, Apr 14, 2014 at 2:06 PM, Stefan Kaltenbrunner ste...@kaltenbrunner.cc wrote: On 04/13/2014 10:19 PM, John Mudd wrote: On Sun, Apr 13, 2014 at 12:04 PM, Euler Taveira eu...@timbira.com.br mailto:eu...@timbira.com.br wrote: On 13-04-2014 00:40, John Mudd wrote: I

Re: [HACKERS] [COMMITTERS] pgsql: Add TAP tests for client programs

2014-04-14 Thread Andrew Dunstan
On 04/14/2014 10:30 PM, Andrew Dunstan wrote: On 04/14/2014 10:17 PM, Tom Lane wrote: Peter Eisentraut pete...@gmx.net writes: Add TAP tests for client programs I assume the buildfarm would need to be taught about this? Yes. It probably won't be a huge change, but it will need a bit of

Re: [HACKERS] PostgreSQL in Windows console and Ctrl-C

2014-04-14 Thread Amit Kapila
On Mon, Apr 14, 2014 at 11:46 AM, Christian Ullrich ch...@chrullrich.net wrote: * From: Amit Kapila Do you mean to say use some existing environment variable? Introducing an environment variable to solve this issue or infact using some existing environ variable doesn't seem to be the best way

Re: [HACKERS] PostgreSQL in Windows console and Ctrl-C

2014-04-14 Thread Amit Kapila
On Tue, Apr 15, 2014 at 4:21 AM, Bruce Momjian br...@momjian.us wrote: On Mon, Apr 14, 2014 at 09:34:14AM +0530, Amit Kapila wrote: The problem can be solved this way, but the only question here is whether it is acceptable for users to have a new console window for server. Can others also

Re: [HACKERS] Dynamic Shared Memory stuff

2014-04-14 Thread Amit Kapila
On Mon, Apr 14, 2014 at 10:03 PM, Robert Haas robertmh...@gmail.com wrote: On Sat, Apr 12, 2014 at 1:32 AM, Amit Kapila amit.kapil...@gmail.com wrote: I have checked that other place in code also check handle to decide if API has failed. Refer function PGSharedMemoryIsInUse(). So I think fix

Re: [HACKERS] Clock sweep not caching enough B-Tree leaf pages?

2014-04-14 Thread Jim Nasby
On 4/14/14, 7:43 PM, Stephen Frost wrote: * Jim Nasby (j...@nasby.net) wrote: I think it's important to mention that OS implementations (at least all I know of) have multiple page pools, each of which has it's own clock. IIRC one of the arguments for us supporting a count1 was we could get

Re: [HACKERS] integrate pg_upgrade analyze_new_cluster.sh into vacuumdb

2014-04-14 Thread Peter Eisentraut
Committed, with your suggestions. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers