Re: [HACKERS] pgaudit - an auditing extension for PostgreSQL

2015-01-23 Thread Robert Haas
On Wed, Jan 21, 2015 at 1:42 AM, Abhijit Menon-Sen a...@2ndquadrant.com wrote: At 2015-01-20 20:36:39 -0500, robertmh...@gmail.com wrote: I think this is throwing the baby out with the bathwater. Neither C functions nor all-or-nothing are going to be of any practical use. Do you see some

Re: [HACKERS] TODO : Allow parallel cores to be used by vacuumdb [ WIP ]

2015-01-23 Thread Alvaro Herrera
Alvaro Herrera wrote: I'm tweaking your v24 a bit more now, thanks -- main change is to make vacuum_one_database be called only to run one analyze stage, so it never iterates for each stage; callers must iterate calling it multiple times in those cases. (There's only one callsite that needs

Re: [HACKERS] B-Tree support function number 3 (strxfrm() optimization)

2015-01-23 Thread Robert Haas
On Wed, Jan 21, 2015 at 2:22 AM, Peter Geoghegan p...@heroku.com wrote: You'll probably prefer the attached. This patch works by disabling abbreviation, but only after writing out runs, with the final merge left to go. That way, it doesn't matter when abbreviated keys are not read back from

Re: [HACKERS] Windows buildfarm animals are still not happy with abbreviated keys patch

2015-01-23 Thread Robert Haas
On Thu, Jan 22, 2015 at 5:51 PM, Peter Geoghegan p...@heroku.com wrote: That having been said, it's clearer to continue to handle each case (C locale vs other locales) separately within the new bttext_abbrev_convert() function, just to be consistent, but also to avoid NUL-terminating the text

Re: [HACKERS] TODO : Allow parallel cores to be used by vacuumdb [ WIP ]

2015-01-23 Thread Alvaro Herrera
Andres Freund wrote: On 2014-12-31 18:35:38 +0530, Amit Kapila wrote: + PQsetnonblocking(connSlot[0].connection, 1); + + for (i = 1; i concurrentCons; i++) + { + connSlot[i].connection = connectDatabase(dbname, host, port, username, +

Re: [HACKERS] pg_upgrade and rsync

2015-01-23 Thread Jim Nasby
On 1/22/15 7:54 PM, Stephen Frost wrote: * Bruce Momjian (br...@momjian.us) wrote: On Fri, Jan 23, 2015 at 01:19:33AM +0100, Andres Freund wrote: Or do you - as the text edited in your patch, but not the quote above - mean to run pg_upgrade just on the primary and then rsync? No, I was

Re: [HACKERS] pgaudit - an auditing extension for PostgreSQL

2015-01-23 Thread Robert Haas
On Wed, Jan 21, 2015 at 7:05 PM, Jim Nasby jim.na...@bluetreble.com wrote: I'm still nervous about overloading this onto the roles system; I think it will end up being very easy to accidentally break. But if others think it'll work then I guess I'm just being paranoid. I agree with you. I

Re: [HACKERS] pgaudit - an auditing extension for PostgreSQL

2015-01-23 Thread Stephen Frost
* Robert Haas (robertmh...@gmail.com) wrote: On Wed, Jan 21, 2015 at 7:05 PM, Jim Nasby jim.na...@bluetreble.com wrote: I'm still nervous about overloading this onto the roles system; I think it will end up being very easy to accidentally break. But if others think it'll work then I guess

Re: [HACKERS] B-Tree support function number 3 (strxfrm() optimization)

2015-01-23 Thread Robert Haas
On Fri, Jan 23, 2015 at 2:18 AM, David Rowley dgrowle...@gmail.com wrote: On 20 January 2015 at 17:10, Peter Geoghegan p...@heroku.com wrote: On Mon, Jan 19, 2015 at 7:47 PM, Michael Paquier michael.paqu...@gmail.com wrote: With your patch applied, the failure with MSVC disappeared, but

Re: [HACKERS] pgaudit - an auditing extension for PostgreSQL

2015-01-23 Thread Jim Nasby
On 1/21/15 6:50 PM, Stephen Frost wrote: I'm still nervous about overloading this onto the roles system; I think it will end up being very easy to accidentally break. But if others think it'll work then I guess I'm just being paranoid. Break in which way..? If you're saying it'll be easy for

Re: [HACKERS] pgaudit - an auditing extension for PostgreSQL

2015-01-23 Thread Stephen Frost
* Jim Nasby (jim.na...@bluetreble.com) wrote: On 1/21/15 6:50 PM, Stephen Frost wrote: I'm still nervous about overloading this onto the roles system; I think it will end up being very easy to accidentally break. But if others think it'll work then I guess I'm just being paranoid. Break in

Re: [HACKERS] Parallel Seq Scan

2015-01-23 Thread Joshua D. Drake
On 01/23/2015 10:44 AM, Jim Nasby wrote: number of workers especially at slightly higher worker count. Those fixed chunk numbers look pretty screwy. 2, 4 and 8 workers make no difference, then suddenly 16 cuts times by 1/2 to 1/3? Then 32 cuts time by another 1/2 to 1/3? cached? First couple

Re: [HACKERS] pg_upgrade and rsync

2015-01-23 Thread Stephen Frost
* Andres Freund (and...@2ndquadrant.com) wrote: On 2015-01-23 13:52:54 -0500, Stephen Frost wrote: That wouldn't actually help with what Bruce is trying to do, which is to duplicate the results of the pg_upgrade from the master over to the standby. Well, it'd pretty much obliviate the

Re: [HACKERS] pg_upgrade and rsync

2015-01-23 Thread Stephen Frost
* Andres Freund (and...@2ndquadrant.com) wrote: On 2015-01-23 14:27:51 -0500, Stephen Frost wrote: * Andres Freund (and...@2ndquadrant.com) wrote: On 2015-01-23 14:05:10 -0500, Stephen Frost wrote: If I follow what you're suggesting, pg_upgrade would need a new 'in-place' mode that

Re: [HACKERS] pg_upgrade and rsync

2015-01-23 Thread Stephen Frost
* Jim Nasby (jim.na...@bluetreble.com) wrote: On 1/22/15 7:54 PM, Stephen Frost wrote: * Bruce Momjian (br...@momjian.us) wrote: On Fri, Jan 23, 2015 at 01:19:33AM +0100, Andres Freund wrote: Or do you - as the text edited in your patch, but not the quote above - mean to run pg_upgrade just

Re: [HACKERS] pg_upgrade and rsync

2015-01-23 Thread Andres Freund
On 2015-01-22 20:54:47 -0500, Stephen Frost wrote: * Bruce Momjian (br...@momjian.us) wrote: On Fri, Jan 23, 2015 at 01:19:33AM +0100, Andres Freund wrote: Or do you - as the text edited in your patch, but not the quote above - mean to run pg_upgrade just on the primary and then rsync?

Re: [HACKERS] pg_upgrade and rsync

2015-01-23 Thread Andres Freund
On 2015-01-23 13:52:54 -0500, Stephen Frost wrote: * Andres Freund (and...@2ndquadrant.com) wrote: On 2015-01-22 20:54:47 -0500, Stephen Frost wrote: * Bruce Momjian (br...@momjian.us) wrote: On Fri, Jan 23, 2015 at 01:19:33AM +0100, Andres Freund wrote: Or do you - as the text

Re: [HACKERS] pg_upgrade and rsync

2015-01-23 Thread Andres Freund
On 2015-01-23 14:05:10 -0500, Stephen Frost wrote: * Andres Freund (and...@2ndquadrant.com) wrote: On 2015-01-23 13:52:54 -0500, Stephen Frost wrote: That wouldn't actually help with what Bruce is trying to do, which is to duplicate the results of the pg_upgrade from the master over to

[HACKERS] src/port/gethostname.c sure looks like dead code

2015-01-23 Thread Tom Lane
AFAICS there are no provisions to ever build this file, on any platform, and have not been since 8.0. If we did build it, it would likely fail on a large subset of platforms, because it refers to a constant SYS_NMLN that isn't too portable. I propose we just remove it.

Re: [HACKERS] pg_upgrade and rsync

2015-01-23 Thread Stephen Frost
* Andres Freund (and...@2ndquadrant.com) wrote: On 2015-01-22 20:54:47 -0500, Stephen Frost wrote: * Bruce Momjian (br...@momjian.us) wrote: On Fri, Jan 23, 2015 at 01:19:33AM +0100, Andres Freund wrote: Or do you - as the text edited in your patch, but not the quote above - mean to

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-23 Thread Andres Freund
On 2015-01-23 12:52:03 -0600, Jim Nasby wrote: On 1/22/15 3:18 PM, Andres Freund wrote: , but to then put it's entire contents into WAL? Blech. Besides actually having a chance of being correct, doing so will save having to do two checkpoints inside movedb(). I think it's pretty likely that

Re: [HACKERS] Perl coding error in msvc build system?

2015-01-23 Thread Alvaro Herrera
Why is the last; still there? AFAICS it matters in the old coding because of the while, but I don't think there is an equivalent looping construct in the new code. -- Álvaro Herrerahttp://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training Services

Re: [HACKERS] Perl coding error in msvc build system?

2015-01-23 Thread Andrew Dunstan
On 01/23/2015 03:00 PM, Alvaro Herrera wrote: Why is the last; still there? AFAICS it matters in the old coding because of the while, but I don't think there is an equivalent looping construct in the new code. You're right, I missed that. It shouldn't be there, and will produce an error

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-23 Thread Jim Nasby
On 1/23/15 9:10 AM, Andres Freund wrote: On 2015-01-22 22:58:17 +0100, Andres Freund wrote: On 2015-01-22 16:38:49 -0500, Stephen Frost wrote: I'm trying to figure out why you'd do '2' in master when in discussion of '1' you say I also don't think ALTER DATABASE is even intentionally run at

Re: [HACKERS] pg_upgrade and rsync

2015-01-23 Thread Stephen Frost
* Andres Freund (and...@2ndquadrant.com) wrote: On 2015-01-23 14:05:10 -0500, Stephen Frost wrote: * Andres Freund (and...@2ndquadrant.com) wrote: On 2015-01-23 13:52:54 -0500, Stephen Frost wrote: That wouldn't actually help with what Bruce is trying to do, which is to duplicate the

Re: [HACKERS] pgaudit - an auditing extension for PostgreSQL

2015-01-23 Thread Jim Nasby
On 1/23/15 12:16 PM, Stephen Frost wrote: Just to clarify- this concept isn't actually mine but was suggested by a pretty sizable PG user who has a great deal of familiarity with other databases. I don't mean to try and invoke the 'silent majority' but rather to make sure folks don't think this

Re: [HACKERS] Parallel Seq Scan

2015-01-23 Thread Jim Nasby
On 1/23/15 5:42 AM, Amit Kapila wrote: *Fixed-Chunks* *No. of workers/Time (ms)* *0* *2* *4* *8* *16**24**32* Run-1 250536 266279 251263 234347 87930 50474 35474 Run-2 249587 230628 225648 193340 83036 35140 9100 Run-3 234963 220671

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-23 Thread Jim Nasby
On 1/22/15 3:18 PM, Andres Freund wrote: , but to then put it's entire contents into WAL? Blech. Besides actually having a chance of being correct, doing so will save having to do two checkpoints inside movedb(). I think it's pretty likely that that actually saves overall IO, even including the

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-23 Thread Andres Freund
On 2015-01-23 13:01:34 -0600, Jim Nasby wrote: On 1/23/15 9:10 AM, Andres Freund wrote: On 2015-01-22 22:58:17 +0100, Andres Freund wrote: On 2015-01-22 16:38:49 -0500, Stephen Frost wrote: I'm trying to figure out why you'd do '2' in master when in discussion of '1' you say I also don't

Re: [HACKERS] hung backends stuck in spinlock heavy endless loop

2015-01-23 Thread Jeff Janes
On Thu, Jan 22, 2015 at 1:50 PM, Merlin Moncure mmonc...@gmail.com wrote: So far, the 'nasty' damage seems to generally if not always follow a checksum failure and the checksum failures are always numerically adjacent. For example: [cds2 12707 2015-01-22 12:51:11.032 CST 2754]WARNING:

Re: [HACKERS] pg_upgrade and rsync

2015-01-23 Thread Andres Freund
On 2015-01-23 14:27:51 -0500, Stephen Frost wrote: * Andres Freund (and...@2ndquadrant.com) wrote: On 2015-01-23 14:05:10 -0500, Stephen Frost wrote: If I follow what you're suggesting, pg_upgrade would need a new 'in-place' mode that removes all of the catalog tables from the old

Re: [HACKERS] Minor issues with code comments related to abbreviated keys

2015-01-23 Thread Peter Geoghegan
On Thu, Jan 22, 2015 at 5:17 PM, Peter Geoghegan p...@heroku.com wrote: Attached patch fixes minor issues in code comments that relate to abbreviated keys. There should also be a description of hyperLogLog in the new README file within ./src/backend/lib/ I suggest: hyperloglog.c - a streaming

Re: [HACKERS] Minor issues with code comments related to abbreviated keys

2015-01-23 Thread Robert Haas
On Fri, Jan 23, 2015 at 2:59 PM, Peter Geoghegan p...@heroku.com wrote: On Thu, Jan 22, 2015 at 5:17 PM, Peter Geoghegan p...@heroku.com wrote: Attached patch fixes minor issues in code comments that relate to abbreviated keys. There should also be a description of hyperLogLog in the new

Re: [HACKERS] pgaudit - an auditing extension for PostgreSQL

2015-01-23 Thread Stephen Frost
* Jim Nasby (jim.na...@bluetreble.com) wrote: On 1/23/15 12:16 PM, Stephen Frost wrote: Just to clarify- this concept isn't actually mine but was suggested by a pretty sizable PG user who has a great deal of familiarity with other databases. I don't mean to try and invoke the 'silent

Re: [HACKERS] Perl coding error in msvc build system?

2015-01-23 Thread Brar Piening
Am 23.01.2015 um 09:17 schrieb Abhijit Menon-Sen: At 2014-06-03 22:30:50 -0400, pete...@gmx.net wrote: I'm not sure whether the following coding actually detects any errors: Solution.pm: open(P, cl /? 21 |) || die cl command not found; Since nobody with a Windows system has commented,

Re: [HACKERS] REVIEW: Track TRUNCATE via pgstat

2015-01-23 Thread Alvaro Herrera
Here's v0.5. (Why did you use that weird decimal versioning scheme? You could just say v4 and save a couple of keystrokes). This patch makes perfect sense to me now. I was ready to commit, but I checked the regression test you added and noticed that you're only reading results for the last set

[HACKERS] Abbreviated keys for Datum tuplesort (was: Re: B-Tree support function number 3 (strxfrm() optimization))

2015-01-23 Thread Andrew Gierth
[pruning the Cc: list and starting a new thread] Here's the cleaned-up version of the patch to allow abbreviated keys when sorting a single Datum. This also removes comments that suggest that the caller of tuplesort_begin_datum should ever have to care about the abbreviated key optimization.

Re: [HACKERS] hung backends stuck in spinlock heavy endless loop

2015-01-23 Thread Peter Geoghegan
On Thu, Jan 22, 2015 at 1:50 PM, Merlin Moncure mmonc...@gmail.com wrote: Quick update: not done yet, but I'm making consistent progress, with several false starts. (for example, I had a .conf problem with the new dynamic shared memory setting and git merrily bisected down to the

Re: [HACKERS] pg_upgrade and rsync

2015-01-23 Thread Bruce Momjian
On Fri, Jan 23, 2015 at 02:34:36PM -0500, Stephen Frost wrote: Why? Just rsync the new data directory onto the old directory on the standbys. That's fine and simple. That still doesn't address the need to use --size-only, it would just mean that you don't need to use -H. If anything the

[HACKERS] Unsafe coding in ReorderBufferCommit()

2015-01-23 Thread Tom Lane
There are at least two bugs in reorderbuffer.c's ReorderBufferCommit(): 1. Although iterstate is modified within the PG_TRY segment and referenced within the PG_CATCH segment, it is not marked volatile. This means that its value upon reaching the PG_CATCH segment is indeterminate. In practice,

Re: [HACKERS] Perl coding error in msvc build system?

2015-01-23 Thread Abhijit Menon-Sen
At 2014-06-03 22:30:50 -0400, pete...@gmx.net wrote: I'm not sure whether the following coding actually detects any errors: Solution.pm: open(P, cl /? 21 |) || die cl command not found; Since nobody with a Windows system has commented, I'm just writing to say that from a Perl

Re: [HACKERS] pg_basebackup fails with long tablespace paths

2015-01-23 Thread Abhijit Menon-Sen
At 2014-12-24 08:10:46 -0500, pete...@gmx.net wrote: As a demo for how this might look, attached is a wildly incomplete patch to produce GNU long-link headers. Hi Peter. In what way exactly is this patch wildly incomplete? (I ask because it's been added to the current CF). -- Abhijit --

Re: [HACKERS] Parallel Seq Scan

2015-01-23 Thread Amit Kapila
On Thu, Jan 22, 2015 at 7:23 PM, Robert Haas robertmh...@gmail.com wrote: On Thu, Jan 22, 2015 at 5:57 AM, Amit Kapila amit.kapil...@gmail.com wrote: 1. Scanning block-by-block has negative impact on performance and I thin it will degrade more if we increase parallel count as that can lead

Re: [HACKERS] Parallel Seq Scan

2015-01-23 Thread Amit Kapila
On Sat, Jan 24, 2015 at 12:24 AM, Joshua D. Drake j...@commandprompt.com wrote: On 01/23/2015 10:44 AM, Jim Nasby wrote: number of workers especially at slightly higher worker count. Those fixed chunk numbers look pretty screwy. 2, 4 and 8 workers make no difference, then suddenly 16 cuts

Re: [HACKERS] TODO : Allow parallel cores to be used by vacuumdb [ WIP ]

2015-01-23 Thread Dilip kumar
On 22 January 2015 23:16, Alvaro Herrera Wrote, Here's v23. There are two things that continue to bother me and I would like you, dear patch author, to change them before committing this patch: 1. I don't like having vacuum_one_database() and a separate vacuum_one_database_parallel(). I

Re: [HACKERS] WITH CHECK and Column-Level Privileges

2015-01-23 Thread Stephen Frost
Alvaro, Thanks for the review. * Alvaro Herrera (alvhe...@2ndquadrant.com) wrote: Note the first comment in this hunk was not update to talk about NULL instead of : Ah, good catch, will fix. Hm, this is a bit odd. I thought you were going to return the subset of columns that the user had

Re: [HACKERS] Perl coding error in msvc build system?

2015-01-23 Thread Andrew Dunstan
On 01/23/2015 03:17 AM, Abhijit Menon-Sen wrote: At 2014-06-03 22:30:50 -0400, pete...@gmx.net wrote: I'm not sure whether the following coding actually detects any errors: Solution.pm: open(P, cl /? 21 |) || die cl command not found; Since nobody with a Windows system has commented,

Re: [HACKERS] WITH CHECK and Column-Level Privileges

2015-01-23 Thread Alvaro Herrera
Stephen Frost wrote: +#define GetModifiedColumns(relinfo, estate) \ + (rt_fetch((relinfo)-ri_RangeTableIndex, (estate)-es_range_table)-modifiedCols) I assume you are aware that this GetModifiedColumns() macro is a duplicate of another one found elsewhere. However I think this is

Re: [HACKERS] Proposal: knowing detail of config files via SQL

2015-01-23 Thread Sawada Masahiko
On Fri, Jan 23, 2015 at 4:36 AM, Jim Nasby jim.na...@bluetreble.com wrote: Would this view have a row for every option in a config file? IE: if you set something in both postgresql.conf and postgresql.auto.conf, would it show up twice? I think it should, and that there should be a way to see

Re: [HACKERS] basebackups during ALTER DATABASE ... SET TABLESPACE ... not safe?

2015-01-23 Thread Andres Freund
On 2015-01-22 22:58:17 +0100, Andres Freund wrote: On 2015-01-22 16:38:49 -0500, Stephen Frost wrote: I'm trying to figure out why you'd do '2' in master when in discussion of '1' you say I also don't think ALTER DATABASE is even intentionally run at the time of a base backup. I agree with

Re: [HACKERS] WITH CHECK and Column-Level Privileges

2015-01-23 Thread Stephen Frost
* Alvaro Herrera (alvhe...@2ndquadrant.com) wrote: Stephen Frost wrote: +#define GetModifiedColumns(relinfo, estate) \ + (rt_fetch((relinfo)-ri_RangeTableIndex, (estate)-es_range_table)-modifiedCols) I assume you are aware that this GetModifiedColumns() macro is a

Re: [HACKERS] WITH CHECK and Column-Level Privileges

2015-01-23 Thread Alvaro Herrera
Note the first comment in this hunk was not update to talk about NULL instead of : @@ -163,13 +168,63 @@ BuildIndexValueDescription(Relation indexRelation, Datum *values, bool *isnull) { StringInfoData buf; + Form_pg_index idxrec;

Re: [HACKERS] TODO : Allow parallel cores to be used by vacuumdb [ WIP ]

2015-01-23 Thread Alvaro Herrera
Dilip kumar wrote: Changes: 1. In current patch vacuum_one_database (for table list), have the table loop outside and analyze_stage loop inside, so it will finish All three stage for one table first and then pick the next table. But vacuum_one_database_parallel will do the stage loop