Re: [HACKERS] Reducing ClogControlLock contention

2015-06-30 Thread Simon Riggs
On 30 June 2015 at 08:22, Simon Riggs si...@2ndquadrant.com wrote: This contention is masked by contention elsewhere, e.g. ProcArrayLock, so the need for testing here should come once other patches ahead of this are in. Let me explain more clearly. Andres' patch to cache snapshots and

Re: [HACKERS] [PROPOSAL] VACUUM Progress Checker.

2015-06-30 Thread Thomas Munro
On Tue, Jun 30, 2015 at 7:37 PM, Rahila Syed rahilasye...@gmail.com wrote: Hello Hackers, Following is a proposal for feature to calculate VACUUM progress. Use Case : Measuring progress of long running VACUUMs to help DBAs make informed decision whether to continue running VACUUM or abort

Re: [HACKERS] [PROPOSAL] VACUUM Progress Checker.

2015-06-30 Thread dinesh kumar
On Tue, Jun 30, 2015 at 1:07 PM, Rahila Syed rahilasye...@gmail.com wrote: Hello Hackers, Following is a proposal for feature to calculate VACUUM progress. Use Case : Measuring progress of long running VACUUMs to help DBAs make informed decision whether to continue running VACUUM or abort

Re: [HACKERS] Reducing ClogControlLock contention

2015-06-30 Thread Amit Kapila
On Tue, Jun 30, 2015 at 12:52 PM, Simon Riggs si...@2ndquadrant.com wrote: On 30 June 2015 at 08:13, Michael Paquier michael.paqu...@gmail.com wrote: Could it be possible to see some performance numbers? For example with a simple pgbench script doing a bunch of tiny transactions, with many

Re: [HACKERS] [PROPOSAL] VACUUM Progress Checker.

2015-06-30 Thread Pavel Stehule
Hi 2015-06-30 9:37 GMT+02:00 Rahila Syed rahilasye...@gmail.com: Hello Hackers, Following is a proposal for feature to calculate VACUUM progress. interesting idea - I like to see it integrated to core. Use Case : Measuring progress of long running VACUUMs to help DBAs make informed

Re: [HACKERS] [PROPOSAL] VACUUM Progress Checker.

2015-06-30 Thread Simon Riggs
On 30 June 2015 at 08:52, Pavel Stehule pavel.steh...@gmail.com wrote: I though about the possibilities of progress visualization - and one possibility is one or two special column in pg_stat_activity table - this info can be interesting for VACUUM started by autovacuum too. Yes, I suggest

Re: [HACKERS] Reducing ClogControlLock contention

2015-06-30 Thread Simon Riggs
On 30 June 2015 at 08:13, Michael Paquier michael.paqu...@gmail.com wrote: On Tue, Jun 30, 2015 at 4:02 PM, Simon Riggs si...@2ndquadrant.com wrote: ClogControlLock contention is high at commit time. This appears to be due to the fact that ClogControlLock is acquired in Exclusive mode

[HACKERS] Missing return value checks in jsonfuncs.c

2015-06-30 Thread Michael Paquier
Hi all, Coverity is pointing out that a couple of return value checks are missing for JsonbIteratorNext in jsonfuncs.c. Patch is attached to append (void) to them, as far as I am guessing we want skip the value iterated on. Regards, -- Michael diff --git a/src/backend/utils/adt/jsonfuncs.c

Re: [HACKERS] Memory leak with XLogFileCopy since de768844 (WAL file with .partial)

2015-06-30 Thread Fujii Masao
On Tue, Jun 9, 2015 at 10:09 AM, Michael Paquier michael.paqu...@gmail.com wrote: On Tue, Jun 9, 2015 at 6:25 AM, Heikki Linnakangas wrote: I'm still not sure if I should've just reverted that refactoring, to make XLogFileCopy() look the same in master and back-branches, which makes

Re: [HACKERS] Bug in bttext_abbrev_convert()

2015-06-30 Thread Peter Geoghegan
On Tue, Jun 30, 2015 at 5:25 PM, Jim Nasby jim.na...@bluetreble.com wrote: Isn't this the kind of thing Coverty's supposed to find? I don't know, but in general I'm not very excited about static analysis tools. The best things that they have going for them is that they're available, and don't

[HACKERS] WAL-related tools and .paritial WAL file

2015-06-30 Thread Fujii Masao
Hi, WAL-related tools, i.e., pg_archivecleanup, pg_resetxlog and pg_xlogdump don't seem to properly handle .paritial WAL file. I think that we should fix at least pg_archivecleanup, otherwise, in the system using pg_archivecleanup to clean up old archived files, the archived .paritial WAL file

Re: [HACKERS] pg_basebackup and replication slots

2015-06-30 Thread Peter Eisentraut
On 5/21/15 8:42 AM, Peter Eisentraut wrote: I wonder why pg_basebackup doesn't have any support for replication slots. When relying on replication slots to hang on to WAL data, there is a gap between when pg_basebackup finishes and streaming replication is started where WAL data could be

Re: [HACKERS] Memory leak with XLogFileCopy since de768844 (WAL file with .partial)

2015-06-30 Thread Michael Paquier
On Wed, Jul 1, 2015 at 10:58 AM, Fujii Masao wrote: On Tue, Jun 9, 2015 at 10:09 AM, Michael Paquier wrote: That's a valid concern. What about the attached then? I think that it is still good to keep upto to copy only data up to the switch point at recovery exit. InstallXLogFileSegment()

Re: [HACKERS] Bug in bttext_abbrev_convert()

2015-06-30 Thread Jim Nasby
On 6/29/15 6:47 PM, Peter Geoghegan wrote: As we all know, the state of automated testing is pretty lamentable. This is the kind of thing that we could catch more easily in the future if better infrastructure were in place. I caught this by eyeballing bttext_abbrev_convert() with slightly

Re: [HACKERS] Dereferenced pointer in tablesample.c

2015-06-30 Thread Petr Jelinek
On 2015-06-30 09:10, Michael Paquier wrote: Hi all, (Petr in CC) Coverity is complaining about the following pointer dereference in tablesample_init@tablesample.c: + ExprState *argstate = ExecInitExpr(argexpr, (PlanState *) scanstate); + + if (argstate == NULL) +

[HACKERS] Missing checks on return value of timestamp2tm in datetime.c

2015-06-30 Thread Michael Paquier
Hi all, timestamp2tm is called close to 40 times in the backend source code, returning -1 in case of failure. However, there are two places in datetime.c where we do not check for its return value: GetCurrentDateTime and GetCurrentTimeUsec. This does not really matter much in practice as the

[HACKERS] Unneeded NULL-pointer check in FreeSpaceMapTruncateRel

2015-06-30 Thread Michael Paquier
Hi all, In the category of nitpicky-code-style-issues, FreeSpaceMapTruncateRel is doing a NULL-pointer check for something that has been dereferenced on all the code paths leading to this check. (Yes, that's not interesting for common humans, Coverity sees things based on correctness). Regards,

Re: [HACKERS] Refactor to split nodeAgg.c?

2015-06-30 Thread David Rowley
On 30 June 2015 at 14:33, Jeff Davis pg...@j-davis.com wrote: I was going to rebase my HashAgg patch, and got some conflicts related to the grouping sets patch. I could probably sort them out, but I think that may be the tipping point where we want to break up nodeAgg.c into nodeSortedAgg.c

[HACKERS] 9.5 branch splitoff

2015-06-30 Thread Tom Lane
Barring objections, I'll create the REL9_5_STABLE branch and stamp HEAD as 9.6devel sometime this afternoon, maybe around 1800 UTC. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

Re: [HACKERS] RFC: replace pg_stat_activity.waiting with something more descriptive

2015-06-30 Thread Amit Kapila
On Fri, Jun 26, 2015 at 6:26 PM, Robert Haas robertmh...@gmail.com wrote: On Thu, Jun 25, 2015 at 11:57 PM, Amit Kapila amit.kapil...@gmail.com wrote: 3. Add new view 'pg_stat_wait_event' with following info: pid - process id of this backend waiting - true for any form of wait, false

Re: [HACKERS] Refactor to split nodeAgg.c?

2015-06-30 Thread Andres Freund
Hi, On 2015-06-29 19:33:58 -0700, Jeff Davis wrote: I was going to rebase my HashAgg patch, and got some conflicts related to the grouping sets patch. I could probably sort them out, but I think that may be the tipping point where we want to break up nodeAgg.c into nodeSortedAgg.c and

Re: [HACKERS] Solaris testers wanted for strxfrm() behavior

2015-06-30 Thread Tom Lane
Noah Misch n...@leadboat.com writes: On Sun, Jun 28, 2015 at 07:00:14PM -0400, Tom Lane wrote: Another idea would be to make a test during postmaster start to see if this bug exists, and fail if so. I'm generally on board with the thought that we don't need to work on systems with such a bad

Re: [HACKERS] Bug in bttext_abbrev_convert()

2015-06-30 Thread Michael Paquier
On Wed, Jul 1, 2015 at 9:36 AM, Peter Geoghegan p...@heroku.com wrote: On Tue, Jun 30, 2015 at 5:25 PM, Jim Nasby jim.na...@bluetreble.com wrote: Isn't this the kind of thing Coverty's supposed to find? I don't know, but in general I'm not very excited about static analysis tools. The best

Re: [HACKERS] Bug in bttext_abbrev_convert()

2015-06-30 Thread Peter Geoghegan
On Tue, Jun 30, 2015 at 9:39 PM, Michael Paquier michael.paqu...@gmail.com wrote: There is no real testing of sorting in the regression tests. It would be nice to have a way of generating a large and varied selection of sort operations programmatically, to catch this kind of thing. pg_regress

Re: [HACKERS] Bug in bttext_abbrev_convert()

2015-06-30 Thread Peter Geoghegan
On Tue, Jun 30, 2015 at 10:25 PM, Peter Geoghegan p...@heroku.com wrote: It would be nice to always have a html report from gcov always available on the internet. That would be something useful to automate, IMV. Watching that regress over time might provide useful insight, but I only use gcov

Re: [HACKERS] Bug in bttext_abbrev_convert()

2015-06-30 Thread Peter Geoghegan
On Tue, Jun 30, 2015 at 10:35 PM, Peter Geoghegan p...@heroku.com wrote: The regression tests have zero coverage for this tuplesort_performsort() btspool2 case. That's a fairly common case to have no coverage for, and that took me all of 5 minutes to find. BTW, I looked here because I added a

Re: [HACKERS] Parallel Seq Scan

2015-06-30 Thread Amit Kapila
On Tue, Jun 30, 2015 at 4:00 AM, Jeff Davis pg...@j-davis.com wrote: [Jumping in without catching up on entire thread. No problem. Please let me know if these questions have already been covered.] 1. Can you change the name to something like ParallelHeapScan? Parallel Sequential is a

Re: [HACKERS] WAL-related tools and .paritial WAL file

2015-06-30 Thread Amit Kapila
On Wed, Jul 1, 2015 at 9:09 AM, Fujii Masao masao.fu...@gmail.com wrote: Also regarding pg_xlogdump, we can just document, for example, please get rid of .paritial suffix from the WAL file name if you want to dump it by pg_xlogdump. Can't we skip such files in pg_xlogdump? With Regards,

Re: [HACKERS] Streaming replication for psycopg2

2015-06-30 Thread Shulgin, Oleksandr
On Thu, Jun 4, 2015 at 5:49 PM, Shulgin, Oleksandr oleksandr.shul...@zalando.de wrote: On Tue, Jun 2, 2015 at 2:23 PM, Shulgin, Oleksandr oleksandr.shul...@zalando.de wrote: Hello, I've submitted a patch to psycopg2 to support streaming replication protocol (COPY_BOTH):

Re: [HACKERS] thread_test's sched_yield requires -lrt on solaris

2015-06-30 Thread Alvaro Herrera
Oskari Saarenmaa wrote: I configured the dingo and binturong Solaris 10 animals to build 9.3 some time ago but apparently they always failed the configure phase. Turns out this is caused by thread_test's usage of sched_yield which is in librt on Solaris but which is not pulled in by anything

Re: [HACKERS] LWLock deadlock and gdb advice

2015-06-30 Thread Jeff Janes
On Mon, Jun 29, 2015 at 11:28 PM, Jeff Janes jeff.ja...@gmail.com wrote: On Mon, Jun 29, 2015 at 5:55 PM, Peter Geoghegan p...@heroku.com wrote: On Mon, Jun 29, 2015 at 5:37 PM, Jeff Janes jeff.ja...@gmail.com wrote: Is there a way to use gdb to figure out who holds the lock they are

Re: [HACKERS] pg_restore -t should match views, matviews, and foreign tables

2015-06-30 Thread Pavel Stehule
Hi I am sending a review of this trivial patch. 1.This patch enables the possibility to restore only selected view, mat. view, foreign table or sequence. Currently the option -t works with tables only. All other relation like objects are quietly ignored. With this patch, the check on type is

Re: [HACKERS] Refactoring speculative insertion with unique indexes a little

2015-06-30 Thread Heikki Linnakangas
On 06/11/2015 02:19 AM, Peter Geoghegan wrote: Currently, speculative insertion (the INSERT ... ON CONFLICT DO UPDATE executor/storage infrastructure) uses checkUnique == UNIQUE_CHECK_PARTIAL for unique indexes, which is a constant originally only used by deferred unique constraints. It occurred

Re: [HACKERS] LWLock deadlock and gdb advice

2015-06-30 Thread Heikki Linnakangas
On 06/30/2015 07:37 PM, Alvaro Herrera wrote: Jeff Janes wrote: I've gotten the LWLock deadlock again. User backend 24841 holds the WALInsertLocks 7 and is blocked attempting to acquire 6 . So it seems to be violating the lock ordering rules (although I don't see that rule spelled out in

Re: [HACKERS] LWLock deadlock and gdb advice

2015-06-30 Thread Alvaro Herrera
Jeff Janes wrote: I've gotten the LWLock deadlock again. User backend 24841 holds the WALInsertLocks 7 and is blocked attempting to acquire 6 . So it seems to be violating the lock ordering rules (although I don't see that rule spelled out in xlog.c) Hmm, interesting -- pg_stat_statement

Re: [HACKERS] Dereferenced pointer in tablesample.c

2015-06-30 Thread Tom Lane
Petr Jelinek p...@2ndquadrant.com writes: On 2015-06-30 09:10, Michael Paquier wrote: If the expression argstate is NULL when calling ExecInitExpr(), argstate is going to be NULL and dereferenced afterwards, see execQual.c for more details. Hence I think that the patch attached should be

Re: [HACKERS] pg_rewind failure by file deletion in source server

2015-06-30 Thread Heikki Linnakangas
On 06/29/2015 09:44 AM, Michael Paquier wrote: On Mon, Jun 29, 2015 at 4:55 AM, Heikki Linnakangas wrote: But we'll still need to handle the pg_xlog symlink case somehow. Perhaps it would be enough to special-case pg_xlog for now. Well, sure, pg_rewind does not copy the soft links either way.

Re: [HACKERS] LWLock deadlock and gdb advice

2015-06-30 Thread Heikki Linnakangas
On 06/30/2015 07:05 PM, Jeff Janes wrote: On Mon, Jun 29, 2015 at 11:28 PM, Jeff Janes jeff.ja...@gmail.com wrote: On Mon, Jun 29, 2015 at 5:55 PM, Peter Geoghegan p...@heroku.com wrote: On Mon, Jun 29, 2015 at 5:37 PM, Jeff Janes jeff.ja...@gmail.com wrote: Is there a way to use gdb to

Re: [HACKERS] 9.5 release notes

2015-06-30 Thread Andres Freund
I've gone through the release notes and added comments referencing commits as discussed earlier. Additionally I've improved and added a bunch of items. Further stuff that came up while looking: * 2014-09-25 [b0d81ad] Heikki..: Add -D option to specify data directory to pg_c.. new options,

Re: [HACKERS] LWLock deadlock and gdb advice

2015-06-30 Thread Andres Freund
On 2015-06-30 21:08:53 +0300, Heikki Linnakangas wrote: /* * XXX: We can significantly optimize this on platforms with 64bit * atomics. */ value = *valptr; if

[HACKERS] Mention column name in error messages

2015-06-30 Thread Franck Verrot
Hi all, As far as I know, there is currently no way to find which column is triggering an error on an INSERT or ALTER COLUMN statement. Example: # create table foo(bar varchar(4), baz varchar(2)); CREATE TABLE # insert into foo values ('foo!', 'ok'); INSERT 0 1 # insert into

Re: [HACKERS] LWLock deadlock and gdb advice

2015-06-30 Thread Heikki Linnakangas
On 06/30/2015 10:09 PM, Andres Freund wrote: On 2015-06-30 21:08:53 +0300, Heikki Linnakangas wrote: /* * XXX: We can significantly optimize this on platforms with 64bit * atomics. */

Re: [HACKERS] LWLock deadlock and gdb advice

2015-06-30 Thread Andres Freund
On 2015-06-30 22:19:02 +0300, Heikki Linnakangas wrote: Hm. Right. A recheck of the value after the queuing should be sufficient to fix? That's how we deal with the exact same scenarios for the normal lock state, so that shouldn't be very hard to add. Yeah. It's probably more efficient to

Re: [HACKERS] pg_trgm version 1.2

2015-06-30 Thread Jeff Janes
On Tue, Jun 30, 2015 at 2:46 AM, Alexander Korotkov a.korot...@postgrespro.ru wrote: On Sun, Jun 28, 2015 at 1:17 AM, Jeff Janes jeff.ja...@gmail.com wrote: This patch implements version 1.2 of contrib module pg_trgm. This supports the triconsistent function, introduced in version 9.4 of

[HACKERS] Dereferenced pointer in tablesample.c

2015-06-30 Thread Michael Paquier
Hi all, (Petr in CC) Coverity is complaining about the following pointer dereference in tablesample_init@tablesample.c: + ExprState *argstate = ExecInitExpr(argexpr, (PlanState *) scanstate); + + if (argstate == NULL) + { +

Re: [HACKERS] Reduce ProcArrayLock contention

2015-06-30 Thread Amit Kapila
On Tue, Jun 30, 2015 at 11:53 AM, Simon Riggs si...@2ndquadrant.com wrote: On 30 June 2015 at 04:21, Amit Kapila amit.kapil...@gmail.com wrote: Now, I would like to briefly explain how allow-one-waker idea has helped to improve the patch as not every body here was present in that

Re: [HACKERS] drop/truncate table sucks for large values of shared buffers

2015-06-30 Thread Amit Kapila
On Tue, Jun 30, 2015 at 11:00 AM, Simon Riggs si...@2ndquadrant.com wrote: On 30 June 2015 at 05:02, Amit Kapila amit.kapil...@gmail.com wrote: On Mon, Jun 29, 2015 at 7:18 PM, Simon Riggs si...@2ndquadrant.com wrote: On 28 June 2015 at 17:17, Tom Lane t...@sss.pgh.pa.us wrote: If

[HACKERS] Reducing ClogControlLock contention

2015-06-30 Thread Simon Riggs
ClogControlLock contention is high at commit time. This appears to be due to the fact that ClogControlLock is acquired in Exclusive mode prior to marking commit, which then gets starved by backends running TransactionIdGetStatus(). Proposal for improving this is to acquire the ClogControlLock in

Re: [HACKERS] Reducing ClogControlLock contention

2015-06-30 Thread Michael Paquier
On Tue, Jun 30, 2015 at 4:02 PM, Simon Riggs si...@2ndquadrant.com wrote: ClogControlLock contention is high at commit time. This appears to be due to the fact that ClogControlLock is acquired in Exclusive mode prior to marking commit, which then gets starved by backends running

Re: [HACKERS] Reduce ProcArrayLock contention

2015-06-30 Thread Simon Riggs
On 30 June 2015 at 03:43, Robert Haas robertmh...@gmail.com wrote: On Mon, Jun 29, 2015 at 1:22 PM, Simon Riggs si...@2ndquadrant.com wrote: Yes, I know. And we all had a long conversation about how to do it without waking up the other procs. Forming a list, like we use for sync rep

Re: [HACKERS] LWLock deadlock and gdb advice

2015-06-30 Thread Jeff Janes
On Mon, Jun 29, 2015 at 5:55 PM, Peter Geoghegan p...@heroku.com wrote: On Mon, Jun 29, 2015 at 5:37 PM, Jeff Janes jeff.ja...@gmail.com wrote: Is there a way to use gdb to figure out who holds the lock they are waiting for? Have you considered building with LWLOCK_STATS defined, and

Re: [HACKERS] drop/truncate table sucks for large values of shared buffers

2015-06-30 Thread Simon Riggs
On 30 June 2015 at 07:34, Amit Kapila amit.kapil...@gmail.com wrote: On Tue, Jun 30, 2015 at 11:00 AM, Simon Riggs si...@2ndquadrant.com wrote: On 30 June 2015 at 05:02, Amit Kapila amit.kapil...@gmail.com wrote: On Mon, Jun 29, 2015 at 7:18 PM, Simon Riggs si...@2ndquadrant.com wrote:

Re: [HACKERS] Reduce ProcArrayLock contention

2015-06-30 Thread Simon Riggs
On 30 June 2015 at 04:21, Amit Kapila amit.kapil...@gmail.com wrote: Now, I would like to briefly explain how allow-one-waker idea has helped to improve the patch as not every body here was present in that Un-conference. The same idea applies for marking commits in clog, for which I have

Re: [HACKERS] Reduce ProcArrayLock contention

2015-06-30 Thread Simon Riggs
On 30 June 2015 at 07:30, Amit Kapila amit.kapil...@gmail.com wrote: Sure and I think we might want to try something similar even for XLogFlush where we use LWLockAcquireOrWait for WALWriteLock, not sure how it will workout in that case as I/O is involved, but I think it is worth trying.

Re: [HACKERS] Reduce ProcArrayLock contention

2015-06-30 Thread Peter Geoghegan
On Mon, Jun 29, 2015 at 11:14 PM, Simon Riggs si...@2ndquadrant.com wrote: What I find weird is that the discussion was so intense about LWLockAcquireOrWait that when someone presented a solution there were people that didn't notice. It makes me wonder whether large group discussions are worth

Re: [HACKERS] PANIC in GIN code

2015-06-30 Thread Heikki Linnakangas
On 06/30/2015 02:18 AM, Jeff Janes wrote: On Mon, Jun 29, 2015 at 2:08 PM, Heikki Linnakangas hlinn...@iki.fi wrote: I just pushed a fix for this, but unfortunately it didn't make it 9.5alpha1. Thanks. I think that that fixed it. It survived for over an hour this time. Thanks. I grepped

Re: [HACKERS] Solaris testers wanted for strxfrm() behavior

2015-06-30 Thread Josh Berkus
On 06/29/2015 07:53 PM, Robert Haas wrote: On Mon, Jun 29, 2015 at 6:07 PM, Josh Berkus j...@agliodbs.com wrote: On 06/29/2015 02:08 PM, Tom Lane wrote: Josh Berkus j...@agliodbs.com writes: Joyent confirms that the bug is fixed on SmartOS: The more interesting bit of information would be

Re: [HACKERS] Solaris testers wanted for strxfrm() behavior

2015-06-30 Thread Tom Lane
Josh Berkus j...@agliodbs.com writes: The question is: how many folks out there are running PostgreSQL on Solaris 10? And are they at all likely to upgrade to PostgreSQL 9.5? That's only the pertinent question if the bug exists on Solaris 10, which I don't think we know do we? Oskari

Re: [HACKERS] Mention column name in error messages

2015-06-30 Thread Tom Lane
Franck Verrot fra...@verrot.fr writes: As far as I know, there is currently no way to find which column is triggering an error on an INSERT or ALTER COLUMN statement. Example: Indeed ... Given my very restricted knowledge of PG's codebase I am not sure whether my modifications are legitimate

Re: [HACKERS] pg_trgm version 1.2

2015-06-30 Thread Tom Lane
Jeff Janes jeff.ja...@gmail.com writes: On Tue, Jun 30, 2015 at 2:46 AM, Alexander Korotkov a.korot...@postgrespro.ru wrote: pg_trgm--1.1.sql andpg_trgm--1.1--1.2.sql are useful for debug, but do you expect them in final commit? As I can see in other contribs we have only last version and