Re: [HACKERS] Parallel Seq Scan

2015-01-23 Thread Amit Kapila
On Sat, Jan 24, 2015 at 12:24 AM, Joshua D. Drake wrote: > > > On 01/23/2015 10:44 AM, Jim Nasby wrote: >> >> number of workers especially at slightly higher worker count. >> >> Those fixed chunk numbers look pretty screwy. 2, 4 and 8 workers make no >> difference, then suddenly 16 cuts times by 1

[HACKERS] Unsafe coding in ReorderBufferCommit()

2015-01-23 Thread Tom Lane
There are at least two bugs in reorderbuffer.c's ReorderBufferCommit(): 1. Although "iterstate" is modified within the PG_TRY segment and referenced within the PG_CATCH segment, it is not marked "volatile". This means that its value upon reaching the PG_CATCH segment is indeterminate. In practice

Re: [HACKERS] hung backends stuck in spinlock heavy endless loop

2015-01-23 Thread Peter Geoghegan
On Thu, Jan 22, 2015 at 1:50 PM, Merlin Moncure wrote: > Quick update: not done yet, but I'm making consistent progress, with > several false starts. (for example, I had a .conf problem with the > new dynamic shared memory setting and git merrily bisected down to the > introduction of the featur

Re: [HACKERS] Perl coding error in msvc build system?

2015-01-23 Thread Brar Piening
Am 23.01.2015 um 09:17 schrieb Abhijit Menon-Sen: At 2014-06-03 22:30:50 -0400, pete...@gmx.net wrote: I'm not sure whether the following coding actually detects any errors: Solution.pm: open(P, "cl /? 2>&1 |") || die "cl command not found"; Since nobody with a Windows system has comment

[HACKERS] Abbreviated keys for Datum tuplesort (was: Re: B-Tree support function number 3 (strxfrm() optimization))

2015-01-23 Thread Andrew Gierth
[pruning the Cc: list and starting a new thread] Here's the cleaned-up version of the patch to allow abbreviated keys when sorting a single Datum. This also removes comments that suggest that the caller of tuplesort_begin_datum should ever have to care about the abbreviated key optimization. I'll

Re: [HACKERS] REVIEW: Track TRUNCATE via pgstat

2015-01-23 Thread Alvaro Herrera
Here's v0.5. (Why did you use that weird decimal versioning scheme? You could just say "v4" and save a couple of keystrokes). This patch makes perfect sense to me now. I was ready to commit, but I checked the regression test you added and noticed that you're only reading results for the last set

Re: [HACKERS] pg_upgrade and rsync

2015-01-23 Thread Bruce Momjian
On Fri, Jan 23, 2015 at 02:34:36PM -0500, Stephen Frost wrote: > > Why? Just rsync the new data directory onto the old directory on the > > standbys. That's fine and simple. > > That still doesn't address the need to use --size-only, it would just > mean that you don't need to use -H. If anything

Re: [HACKERS] pgaudit - an auditing extension for PostgreSQL

2015-01-23 Thread Stephen Frost
* Jim Nasby (jim.na...@bluetreble.com) wrote: > On 1/23/15 12:16 PM, Stephen Frost wrote: > >Just to clarify- this concept isn't actually mine but was suggested by a > >pretty sizable PG user who has a great deal of familiarity with other > >databases. I don't mean to try and invoke the 'silent ma

Re: [HACKERS] pgaudit - an auditing extension for PostgreSQL

2015-01-23 Thread Jim Nasby
On 1/23/15 12:16 PM, Stephen Frost wrote: Just to clarify- this concept isn't actually mine but was suggested by a pretty sizable PG user who has a great deal of familiarity with other databases. I don't mean to try and invoke the 'silent majority' but rather to make sure folks don't think this

Re: [HACKERS] Perl coding error in msvc build system?

2015-01-23 Thread Andrew Dunstan
On 01/23/2015 03:00 PM, Alvaro Herrera wrote: Why is the "last;" still there? AFAICS it matters in the old coding because of the while, but I don't think there is an equivalent looping construct in the new code. You're right, I missed that. It shouldn't be there, and will produce an error l

Re: [HACKERS] Minor issues with code comments related to abbreviated keys

2015-01-23 Thread Robert Haas
On Fri, Jan 23, 2015 at 2:59 PM, Peter Geoghegan wrote: > On Thu, Jan 22, 2015 at 5:17 PM, Peter Geoghegan wrote: >> Attached patch fixes minor issues in code comments that relate to >> abbreviated keys. > > There should also be a description of hyperLogLog in the new README > file within ./src/b

Re: [HACKERS] Perl coding error in msvc build system?

2015-01-23 Thread Alvaro Herrera
Why is the "last;" still there? AFAICS it matters in the old coding because of the while, but I don't think there is an equivalent looping construct in the new code. -- Álvaro Herrerahttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] Minor issues with code comments related to abbreviated keys

2015-01-23 Thread Peter Geoghegan
On Thu, Jan 22, 2015 at 5:17 PM, Peter Geoghegan wrote: > Attached patch fixes minor issues in code comments that relate to > abbreviated keys. There should also be a description of hyperLogLog in the new README file within ./src/backend/lib/ I suggest: hyperloglog.c - a streaming cardinality e

Re: [HACKERS] pg_upgrade and rsync

2015-01-23 Thread Stephen Frost
* Andres Freund (and...@2ndquadrant.com) wrote: > On 2015-01-23 14:27:51 -0500, Stephen Frost wrote: > > * Andres Freund (and...@2ndquadrant.com) wrote: > > > On 2015-01-23 14:05:10 -0500, Stephen Frost wrote: > > > > If I follow what you're suggesting, pg_upgrade would > > > > need a new 'in-place

[HACKERS] src/port/gethostname.c sure looks like dead code

2015-01-23 Thread Tom Lane
AFAICS there are no provisions to ever build this file, on any platform, and have not been since 8.0. If we did build it, it would likely fail on a large subset of platforms, because it refers to a constant SYS_NMLN that isn't too portable. I propose we just remove it. re

Re: [HACKERS] pg_upgrade and rsync

2015-01-23 Thread Andres Freund
On 2015-01-23 14:27:51 -0500, Stephen Frost wrote: > * Andres Freund (and...@2ndquadrant.com) wrote: > > On 2015-01-23 14:05:10 -0500, Stephen Frost wrote: > > > If I follow what you're suggesting, pg_upgrade would > > > need a new 'in-place' mode that removes all of the catalog tables from > > > t

Re: [HACKERS] pg_upgrade and rsync

2015-01-23 Thread Stephen Frost
* Andres Freund (and...@2ndquadrant.com) wrote: > On 2015-01-23 14:05:10 -0500, Stephen Frost wrote: > > * Andres Freund (and...@2ndquadrant.com) wrote: > > > On 2015-01-23 13:52:54 -0500, Stephen Frost wrote: > > > > That wouldn't actually help with what Bruce is trying to do, which > > > > is to

Re: [HACKERS] hung backends stuck in spinlock heavy endless loop

2015-01-23 Thread Jeff Janes
On Thu, Jan 22, 2015 at 1:50 PM, Merlin Moncure wrote: > > So far, the 'nasty' damage seems to generally if not always follow a > checksum failure and the checksum failures are always numerically > adjacent. For example: > > [cds2 12707 2015-01-22 12:51:11.032 CST 2754]WARNING: page > verificat

Re: [HACKERS] pg_upgrade and rsync

2015-01-23 Thread Andres Freund
On 2015-01-23 14:05:10 -0500, Stephen Frost wrote: > * Andres Freund (and...@2ndquadrant.com) wrote: > > On 2015-01-23 13:52:54 -0500, Stephen Frost wrote: > > > That wouldn't actually help with what Bruce is trying to do, which > > > is to duplicate the results of the pg_upgrade from the master ov

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-23 Thread Andres Freund
On 2015-01-23 13:01:34 -0600, Jim Nasby wrote: > On 1/23/15 9:10 AM, Andres Freund wrote: > >On 2015-01-22 22:58:17 +0100, Andres Freund wrote: > >>On 2015-01-22 16:38:49 -0500, Stephen Frost wrote: > >>>I'm trying to figure out why you'd do '2' in master when in discussion > >>>of '1' you say "I a

Re: [HACKERS] pg_upgrade and rsync

2015-01-23 Thread Stephen Frost
* Andres Freund (and...@2ndquadrant.com) wrote: > On 2015-01-23 13:52:54 -0500, Stephen Frost wrote: > > That wouldn't actually help with what Bruce is trying to do, which > > is to duplicate the results of the pg_upgrade from the master over to > > the standby. > > Well, it'd pretty much obliviat

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-23 Thread Andres Freund
On 2015-01-23 12:52:03 -0600, Jim Nasby wrote: > On 1/22/15 3:18 PM, Andres Freund wrote: > >>, but to then put it's entire contents into WAL? Blech. > >Besides actually having a chance of being correct, doing so will save > >having to do two checkpoints inside movedb(). I think it's pretty likely

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-23 Thread Jim Nasby
On 1/23/15 9:10 AM, Andres Freund wrote: On 2015-01-22 22:58:17 +0100, Andres Freund wrote: On 2015-01-22 16:38:49 -0500, Stephen Frost wrote: I'm trying to figure out why you'd do '2' in master when in discussion of '1' you say "I also don't think ALTER DATABASE is even intentionally run at th

Re: [HACKERS] pg_upgrade and rsync

2015-01-23 Thread Andres Freund
On 2015-01-23 13:52:54 -0500, Stephen Frost wrote: > * Andres Freund (and...@2ndquadrant.com) wrote: > > On 2015-01-22 20:54:47 -0500, Stephen Frost wrote: > > > * Bruce Momjian (br...@momjian.us) wrote: > > > > On Fri, Jan 23, 2015 at 01:19:33AM +0100, Andres Freund wrote: > > > > > Or do you - as

Re: [HACKERS] Parallel Seq Scan

2015-01-23 Thread Joshua D. Drake
On 01/23/2015 10:44 AM, Jim Nasby wrote: number of workers especially at slightly higher worker count. Those fixed chunk numbers look pretty screwy. 2, 4 and 8 workers make no difference, then suddenly 16 cuts times by 1/2 to 1/3? Then 32 cuts time by another 1/2 to 1/3? cached? First couple

Re: [HACKERS] pg_upgrade and rsync

2015-01-23 Thread Stephen Frost
* Andres Freund (and...@2ndquadrant.com) wrote: > On 2015-01-22 20:54:47 -0500, Stephen Frost wrote: > > * Bruce Momjian (br...@momjian.us) wrote: > > > On Fri, Jan 23, 2015 at 01:19:33AM +0100, Andres Freund wrote: > > > > Or do you - as the text edited in your patch, but not the quote above - > >

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-23 Thread Jim Nasby
On 1/22/15 3:18 PM, Andres Freund wrote: , but to then put it's entire contents into WAL? Blech. Besides actually having a chance of being correct, doing so will save having to do two checkpoints inside movedb(). I think it's pretty likely that that actually saves overall IO, even including the

Re: [HACKERS] pg_upgrade and rsync

2015-01-23 Thread Andres Freund
On 2015-01-22 20:54:47 -0500, Stephen Frost wrote: > * Bruce Momjian (br...@momjian.us) wrote: > > On Fri, Jan 23, 2015 at 01:19:33AM +0100, Andres Freund wrote: > > > Or do you - as the text edited in your patch, but not the quote above - > > > mean to run pg_upgrade just on the primary and then r

Re: [HACKERS] Parallel Seq Scan

2015-01-23 Thread Jim Nasby
On 1/23/15 5:42 AM, Amit Kapila wrote: *Fixed-Chunks* *No. of workers/Time (ms)* >*0* *2* *4* *8* *16**24**32* Run-1 250536 266279 251263 234347 87930 50474 35474 Run-2 249587 230628 225648 193340 83036 35140 9100 Run-3 234963 220671

Re: [HACKERS] pg_upgrade and rsync

2015-01-23 Thread Stephen Frost
* Jim Nasby (jim.na...@bluetreble.com) wrote: > On 1/22/15 7:54 PM, Stephen Frost wrote: > >* Bruce Momjian (br...@momjian.us) wrote: > >>>On Fri, Jan 23, 2015 at 01:19:33AM +0100, Andres Freund wrote: > >Or do you - as the text edited in your patch, but not the quote above - > >mean to r

Re: [HACKERS] pg_upgrade and rsync

2015-01-23 Thread Jim Nasby
On 1/22/15 7:54 PM, Stephen Frost wrote: * Bruce Momjian (br...@momjian.us) wrote: >On Fri, Jan 23, 2015 at 01:19:33AM +0100, Andres Freund wrote: > >Or do you - as the text edited in your patch, but not the quote above - > >mean to run pg_upgrade just on the primary and then rsync? > >No, I w

Re: [HACKERS] TODO : Allow parallel cores to be used by vacuumdb [ WIP ]

2015-01-23 Thread Alvaro Herrera
Alvaro Herrera wrote: > I'm tweaking your v24 a bit more now, thanks -- main change is to make > vacuum_one_database be called only to run one analyze stage, so it never > iterates for each stage; callers must iterate calling it multiple times > in those cases. (There's only one callsite that nee

Re: [HACKERS] pgaudit - an auditing extension for PostgreSQL

2015-01-23 Thread Stephen Frost
* Jim Nasby (jim.na...@bluetreble.com) wrote: > On 1/21/15 6:50 PM, Stephen Frost wrote: > >>I'm still nervous about overloading this onto the roles system; I think it > >>will end up being very easy to accidentally break. But if others think > >>it'll work then I guess I'm just being paranoid. >

Re: [HACKERS] pgaudit - an auditing extension for PostgreSQL

2015-01-23 Thread Stephen Frost
* Robert Haas (robertmh...@gmail.com) wrote: > On Wed, Jan 21, 2015 at 7:05 PM, Jim Nasby wrote: > > I'm still nervous about overloading this onto the roles system; I think it > > will end up being very easy to accidentally break. But if others think it'll > > work then I guess I'm just being para

Re: [HACKERS] pgaudit - an auditing extension for PostgreSQL

2015-01-23 Thread Jim Nasby
On 1/21/15 6:50 PM, Stephen Frost wrote: I'm still nervous about overloading this onto the roles system; I think it will end up being very easy to accidentally break. But if others think it'll work then I guess I'm just being paranoid. Break in which way..? If you're saying "it'll be easy for

Re: [HACKERS] TODO : Allow parallel cores to be used by vacuumdb [ WIP ]

2015-01-23 Thread Alvaro Herrera
Andres Freund wrote: > On 2014-12-31 18:35:38 +0530, Amit Kapila wrote: > > + PQsetnonblocking(connSlot[0].connection, 1); > > + > > + for (i = 1; i < concurrentCons; i++) > > + { > > + connSlot[i].connection = connectDatabase(dbname, host, port, > > username, > > +

Re: [HACKERS] Windows buildfarm animals are still not happy with abbreviated keys patch

2015-01-23 Thread Robert Haas
On Thu, Jan 22, 2015 at 5:51 PM, Peter Geoghegan wrote: > That having been said, it's clearer to continue to handle each case (C > locale vs other locales) separately within the new > bttext_abbrev_convert() function, just to be consistent, but also to > avoid NUL-terminating the text strings to p

Re: [HACKERS] pgaudit - an auditing extension for PostgreSQL

2015-01-23 Thread Robert Haas
On Wed, Jan 21, 2015 at 1:42 AM, Abhijit Menon-Sen wrote: > At 2015-01-20 20:36:39 -0500, robertmh...@gmail.com wrote: >> I think this is throwing the baby out with the bathwater. Neither C >> functions nor all-or-nothing are going to be of any practical use. > > Do you see some approach that has

Re: [HACKERS] pgaudit - an auditing extension for PostgreSQL

2015-01-23 Thread Robert Haas
On Wed, Jan 21, 2015 at 7:05 PM, Jim Nasby wrote: > I'm still nervous about overloading this onto the roles system; I think it > will end up being very easy to accidentally break. But if others think it'll > work then I guess I'm just being paranoid. I agree with you. I don't hear anyone who act

Re: [HACKERS] B-Tree support function number 3 (strxfrm() optimization)

2015-01-23 Thread Robert Haas
On Wed, Jan 21, 2015 at 2:22 AM, Peter Geoghegan wrote: > You'll probably prefer the attached. This patch works by disabling > abbreviation, but only after writing out runs, with the final merge > left to go. That way, it doesn't matter when abbreviated keys are not > read back from disk (or regen

Re: [HACKERS] B-Tree support function number 3 (strxfrm() optimization)

2015-01-23 Thread Robert Haas
On Fri, Jan 23, 2015 at 2:18 AM, David Rowley wrote: > On 20 January 2015 at 17:10, Peter Geoghegan wrote: >> >> On Mon, Jan 19, 2015 at 7:47 PM, Michael Paquier >> wrote: >> >> > With your patch applied, the failure with MSVC disappeared, but there >> > is still a warning showing up: >> > (ClCo

Re: [HACKERS] TODO : Allow parallel cores to be used by vacuumdb [ WIP ]

2015-01-23 Thread Alvaro Herrera
Dilip kumar wrote: > Changes: > 1. In current patch vacuum_one_database (for table list), have the table loop > outside and analyze_stage loop inside, so it will finish > All three stage for one table first and then pick the next table. But > vacuum_one_database_parallel will do the stage loop o

Re: [HACKERS] WITH CHECK and Column-Level Privileges

2015-01-23 Thread Stephen Frost
* Alvaro Herrera (alvhe...@2ndquadrant.com) wrote: > Stephen Frost wrote: > > > > +#define GetModifiedColumns(relinfo, estate) \ > > > > + (rt_fetch((relinfo)->ri_RangeTableIndex, > > > > (estate)->es_range_table)->modifiedCols) > > > > > > I assume you are aware that this GetModifiedColumn

Re: [HACKERS] Proposal: knowing detail of config files via SQL

2015-01-23 Thread Sawada Masahiko
On Fri, Jan 23, 2015 at 4:36 AM, Jim Nasby wrote: > > > Would this view have a row for every option in a config file? IE: if you set > something in both postgresql.conf and postgresql.auto.conf, would it show up > twice? I think it should, and that there should be a way to see which > setting is a

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-23 Thread Andres Freund
On 2015-01-22 22:58:17 +0100, Andres Freund wrote: > On 2015-01-22 16:38:49 -0500, Stephen Frost wrote: > > I'm trying to figure out why you'd do '2' in master when in discussion > > of '1' you say "I also don't think ALTER DATABASE is even intentionally > > run at the time of a base backup." I ag

Re: [HACKERS] WITH CHECK and Column-Level Privileges

2015-01-23 Thread Alvaro Herrera
Stephen Frost wrote: > > > +#define GetModifiedColumns(relinfo, estate) \ > > > + (rt_fetch((relinfo)->ri_RangeTableIndex, > > > (estate)->es_range_table)->modifiedCols) > > > > I assume you are aware that this GetModifiedColumns() macro is a > > duplicate of another one found elsewhere. Howeve

Re: [HACKERS] Perl coding error in msvc build system?

2015-01-23 Thread Andrew Dunstan
On 01/23/2015 03:17 AM, Abhijit Menon-Sen wrote: At 2014-06-03 22:30:50 -0400, pete...@gmx.net wrote: I'm not sure whether the following coding actually detects any errors: Solution.pm: open(P, "cl /? 2>&1 |") || die "cl command not found"; Since nobody with a Windows system has comment

Re: [HACKERS] WITH CHECK and Column-Level Privileges

2015-01-23 Thread Stephen Frost
Alvaro, Thanks for the review. * Alvaro Herrera (alvhe...@2ndquadrant.com) wrote: > Note the first comment in this hunk was not update to talk about NULL > instead of "": Ah, good catch, will fix. > Hm, this is a bit odd. I thought you were going to return the subset of > columns that the user

Re: [HACKERS] WITH CHECK and Column-Level Privileges

2015-01-23 Thread Alvaro Herrera
Note the first comment in this hunk was not update to talk about NULL instead of "": > @@ -163,13 +168,63 @@ BuildIndexValueDescription(Relation indexRelation, > Datum *values, bool *isnull) > { > StringInfoData buf; > + Form_pg_index idx

Re: [HACKERS] Parallel Seq Scan

2015-01-23 Thread Amit Kapila
On Thu, Jan 22, 2015 at 7:23 PM, Robert Haas wrote: > > On Thu, Jan 22, 2015 at 5:57 AM, Amit Kapila wrote: > > 1. Scanning block-by-block has negative impact on performance and > > I thin it will degrade more if we increase parallel count as that can lead > > to more randomness. > > > > 2. Scann

Re: [HACKERS] TODO : Allow parallel cores to be used by vacuumdb [ WIP ]

2015-01-23 Thread Dilip kumar
On 22 January 2015 23:16, Alvaro Herrera Wrote, > Here's v23. > > There are two things that continue to bother me and I would like you, > dear patch author, to change them before committing this patch: > > 1. I don't like having vacuum_one_database() and a separate > vacuum_one_database_parallel()

Re: [HACKERS] pg_basebackup fails with long tablespace paths

2015-01-23 Thread Abhijit Menon-Sen
At 2014-12-24 08:10:46 -0500, pete...@gmx.net wrote: > > As a demo for how this might look, attached is a wildly incomplete > patch to produce GNU long-link headers. Hi Peter. In what way exactly is this patch wildly incomplete? (I ask because it's been added to the current CF). -- Abhijit --

Re: [HACKERS] Perl coding error in msvc build system?

2015-01-23 Thread Abhijit Menon-Sen
At 2014-06-03 22:30:50 -0400, pete...@gmx.net wrote: > > I'm not sure whether the following coding actually detects any errors: > > Solution.pm: > > open(P, "cl /? 2>&1 |") || die "cl command not found"; Since nobody with a Windows system has commented, I'm just writing to say that from a Pe