Re: [HACKERS] group locking: incomplete patch, just for discussion

On Wed, 2014-10-15 at 00:03 -0400, Robert Haas wrote: > For parallelism, I think we need a concept of group locking. That is, > suppose we have a user backend and N worker backends collaborating to > execute some query. For the sake of argument, let's say it's a > parallel CLUSTER, requiring a fu

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

On Tue, Nov 11, 2014 at 9:29 PM, Simon Riggs wrote: > > This plan type is widely used in reporting queries, so will hit the > mainline of BI applications and many Mat View creations. > This will allow SELECT count(*) FROM foo to go faster also. > > We'd also need to add some infrastructure to mer

[HACKERS] New storage parameter pages_per_range not mentioned in CREATE INDEX doc

Hi all, The new storage parameter pages_per_range is missing in the documentation of CREATE INDEX. The patch attached corrects that. Regards, -- Michael 20141112_brin_doc_fix.patch Description: binary/octet-stream -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make c

Re: [HACKERS] postgres_fdw behaves oddly

(2014/11/11 18:45), Etsuro Fujita wrote: (2014/11/10 20:05), Ashutosh Bapat wrote: Since two separate issues 1. using reltargetlist instead of attr_needed and 2. system columns usage in FDW are being tackled here, we should separate the patch into two one for each of the issues. Agreed. Will

Re: [HACKERS] Final Patch for GROUPING SETS

On Sat, Sep 27, 2014 at 06:37:38AM +0100, Andrew Gierth wrote: > > "Andrew" == Andrew Gierth writes: > > Andrew> I was holding off on posting a recut patch with the latest > Andrew> EXPLAIN formatting changes (which are basically cosmetic) > Andrew> until it became clear whether RLS was li

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

On Tue, Nov 11, 2014 at 6:27 PM, Andres Freund wrote: > The more important thing here is that I see little chance of this > getting in before Heikki's larger rework of the wal format gets > in. Since that'll change everything around anyay I'm unsure how much > point there is to iterate till that's

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

On Tue, Nov 11, 2014 at 3:29 AM, Simon Riggs wrote: > * only functions marked as "CONTAINS NO SQL" > We don't really know what proisparallel is, but we do know what > CONTAINS NO SQL means and can easily check for it. > Plus I already have a patch for this, slightly bitrotted. Interestingly, I ha

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

> > Given > > where we are with the infrastructure, there would be a number of > > unhandled problems, such as deadlock detection (needs group locking or > > similar), assessment of quals as to parallel-safety (needs > > proisparallel or similar), general waterproofing to make sure that > > pushing

Re: [HACKERS] What exactly is our CRC algorithm?

On Tue, Nov 11, 2014 at 6:26 AM, Abhijit Menon-Sen wrote: > At 2014-11-04 14:40:36 +0100, and...@2ndquadrant.com wrote: >> On 2014-11-04 08:21:13 -0500, Robert Haas wrote: >> > Are you going to get the slice-by-N stuff working next, to speed >> > this up? >> >> I don't plan to do anything serious

Re: [HACKERS] Minor problem with comment added by 5ea86e

On Tue, Nov 11, 2014 at 3:11 PM, Peter Geoghegan wrote: > Attached patch removes incorrect comment added by recent Sortsupport > for B-Tree/CLUSTER commit (commit 5ea86e). Committed. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-

Re: [HACKERS] Column/type dependency recording inconsistencies

On 11/11/14 21:47, Tom Lane wrote: I wrote: ... Also, I find physically removing an entry fairly ugly and possibly unsafe, since it's not clear what outer recursion levels might be assuming about the contents of the array -- and that certainly won't work for the stack case. What seems better is

Re: [HACKERS] Deciding which index to use

Christophe Pettus writes: > Where in the optimizer code does PostgreSQL decide which of several > possibly-matching partial indexes to use? It costs them all out and uses the cheapest. If the cost estimates come out exactly the same, you get an arbitrary choice (I think probably always the inde

[HACKERS] Deciding which index to use

Where in the optimizer code does PostgreSQL decide which of several possibly-matching partial indexes to use? -- -- Christophe Pettus x...@thebuild.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailp

Re: [HACKERS] pgcrypto: PGP signatures

On Sat, Nov 1, 2014 at 7:52 AM, Marko Tiikkaja wrote: > Hi, > > I discovered a problem with the lack of MDC handling in the signature info > extraction code, so I've fixed that and added a test message. v9 here. > > > > Hi Marko, I get a segfault when the length of the message is exactly 16308

Re: [HACKERS] tracking commit timestamps

Jim Nasby wrote: > On 11/11/14, 2:03 PM, Alvaro Herrera wrote: > >Jim Nasby wrote: > >>On 11/10/14, 7:40 AM, Alvaro Herrera wrote: > > > >>>Ah, right. So AFAIK we don't need to keep anything older than > >>>RecentXmin or something like that -- which is not too old. If I recall > >>>correctly Josh

Re: [HACKERS] Failback to old master

Hi, 2014-10-29 17:46 GMT+01:00 Robert Haas >: Yes, but after the restart, the slave will also rewind to the most recent restart-point to begin replay, and some of the sanity checks that recovery.conf enforces will be lost during that replay. A safe wa

Re: [HACKERS] tracking commit timestamps

On 11/11/14, 2:03 PM, Alvaro Herrera wrote: Jim Nasby wrote: On 11/10/14, 7:40 AM, Alvaro Herrera wrote: Ah, right. So AFAIK we don't need to keep anything older than RecentXmin or something like that -- which is not too old. If I recall correctly Josh Berkus was saying in a thread about pg

Re: [HACKERS] Column/type dependency recording inconsistencies

I wrote: > ... Also, I find physically removing an entry fairly ugly and > possibly unsafe, since it's not clear what outer recursion levels might be > assuming about the contents of the array -- and that certainly won't work > for the stack case. What seems better is to invent another flag bit >

[HACKERS] Minor problem with comment added by 5ea86e

Attached patch removes incorrect comment added by recent Sortsupport for B-Tree/CLUSTER commit (commit 5ea86e). -- Peter Geoghegan diff --git a/src/backend/utils/sort/sortsupport.c b/src/backend/utils/sort/sortsupport.c index 16342f3..4813f57 100644 --- a/src/backend/utils/sort/sortsupport.c +++

Re: [HACKERS] tracking commit timestamps

Jim Nasby wrote: > On 11/10/14, 7:40 AM, Alvaro Herrera wrote: > >Ah, right. So AFAIK we don't need to keep anything older than > >RecentXmin or something like that -- which is not too old. If I recall > >correctly Josh Berkus was saying in a thread about pg_multixact that it > >used about 128kB

Re: [HACKERS] tracking commit timestamps

On 11/10/14, 7:40 AM, Alvaro Herrera wrote: Robert Haas wrote: On Sun, Nov 9, 2014 at 8:41 PM, Alvaro Herrera wrote: Robert Haas wrote: I think the key question here is the time for which the data needs to be retained. 2^32 of anything is a lot, but why keep around that number of records rat

Re: [HACKERS] Column/type dependency recording inconsistencies

Petr Jelinek writes: > On 09/11/14 23:36, Petr Jelinek wrote: >> On 09/11/14 23:04, Tom Lane wrote: >>> I suspect this is actually a bug in the dependency search logic in DROP. >>> Don't have the energy to chase it right now though. >> But this logic depends on the fact that the columns that belo

Re: [HACKERS] tracking commit timestamps

On 2014-11-11 17:09:54 +, Simon Riggs wrote: > On 11 November 2014 16:19, Andres Freund wrote: > > On 2014-11-11 16:10:47 +, Simon Riggs wrote: > >> On 4 November 2014 08:23, Andres Freund wrote: > >> > >> >> 6) Shouldn't any value update of track_commit_timestamp be tracked in > >> >> XL

Re: [HACKERS] tracking commit timestamps

On 11 November 2014 16:19, Andres Freund wrote: > On 2014-11-11 16:10:47 +, Simon Riggs wrote: >> On 4 November 2014 08:23, Andres Freund wrote: >> >> >> 6) Shouldn't any value update of track_commit_timestamp be tracked in >> >> XLogReportParameters? That's thinking about making the commit t

Re: [HACKERS] Missing FIN_CRC32 calls in logical replication code

On 2014-11-03 21:58:51 +0200, Heikki Linnakangas wrote: > PS. I find the name "ReplicationSlotOnDiskDynamicSize" confusing, as it is > in fact a fixed size struct. I gather it's expected that the size of that > part might change across versions, but by that definition nothing is > constant. Well,

Re: [HACKERS] tracking commit timestamps

On 9 November 2014 16:57, Steve Singer wrote: > On 11/07/2014 07:07 PM, Petr Jelinek wrote: > > >> The list of what is useful might be long, but we can't have everything >> there as there are space constraints, and LSN is another 8 bytes and I still >> want to have some bytes for storing the "orig

Re: [HACKERS] tracking commit timestamps

On 2014-11-11 16:10:47 +, Simon Riggs wrote: > On 4 November 2014 08:23, Andres Freund wrote: > > >> 6) Shouldn't any value update of track_commit_timestamp be tracked in > >> XLogReportParameters? That's thinking about making the commit timestamp > >> available on standbys as well.. > > > >

Re: [HACKERS] tracking commit timestamps

On 4 November 2014 08:23, Andres Freund wrote: >> 6) Shouldn't any value update of track_commit_timestamp be tracked in >> XLogReportParameters? That's thinking about making the commit timestamp >> available on standbys as well.. > > Yes, it should. Agree committs should be able to run on standb

Re: [HACKERS] 9.4RC1 next week

Andres Freund writes: > Apropos back branches: I think 52eed3d426 et al wasn't reverted and we > didn't really agree on a solution? I think we agreed what we wanted to do instead, but actually doing it is on my queue and hasn't reached the front yet. In any case, 52eed3d426 is better than no fix

Re: [HACKERS] 9.4RC1 next week

On 2014-11-11 10:52:30 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2014-11-11 10:18:55 -0500, Tom Lane wrote: > >> We need to get moving if we want to have RC1 out before the holiday season > >> starts. Accordingly, the core committee has agreed that we should wrap it > >> next week (us

Re: [HACKERS] 9.4RC1 next week

Andres Freund writes: > On 2014-11-11 10:18:55 -0500, Tom Lane wrote: >> We need to get moving if we want to have RC1 out before the holiday season >> starts. Accordingly, the core committee has agreed that we should wrap it >> next week (usual timing: wrap Monday 17th for announcement Thursday 2

Re: PENDING_LIST_CLEANUP_SIZE - maximum size of GIN pending list Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

Fujii Masao writes: > On Tue, Nov 11, 2014 at 10:14 PM, Robert Haas wrote: >> Not to kibitz too much after-the-fact, but wouldn't it be better to >> give this a name that has "GIN" in it somewhere? > Maybe. gin_pending_list_cleanup_size? gin_pending_list_limit? Better name? gin_pending_list_lim

Re: [HACKERS] 9.4RC1 next week

Hi, On 2014-11-11 10:18:55 -0500, Tom Lane wrote: > We need to get moving if we want to have RC1 out before the holiday season > starts. Accordingly, the core committee has agreed that we should wrap it > next week (usual timing: wrap Monday 17th for announcement Thursday 20th). Ah cool. So ther

Re: [HACKERS] Defining dedicated macro to grab a relation's persistence

Fabrízio de Royes Mello wrote: > On Fri, Nov 7, 2014 at 11:12 AM, Andres Freund > wrote: > > I personally find the direct access actually more readable, so I'm not a > > fan of further extending the scheme. Consistency with some other common > > accessors is an argument though. > > What you mean

Re: [HACKERS] Defining dedicated macro to grab a relation's persistence

On Fri, Nov 7, 2014 at 11:12 AM, Andres Freund wrote: > > Hi, > > On 2014-11-07 22:08:33 +0900, Michael Paquier wrote: > > After looking at a patch of this commit fest using > > rd_rel->relpersistence, I got a look at how many times this expression > > was being used directly in the backend code a

[HACKERS] 9.4RC1 next week

We need to get moving if we want to have RC1 out before the holiday season starts. Accordingly, the core committee has agreed that we should wrap it next week (usual timing: wrap Monday 17th for announcement Thursday 20th). According to https://wiki.postgresql.org/wiki/PostgreSQL_9.4_Open_Items t

Re: [HACKERS] WAL format and API changes (9.5)

On Tue, Nov 11, 2014 at 2:15 PM, Heikki Linnakangas wrote:z > > On 11/11/2014 09:39 AM, Michael Paquier wrote: >> >> On Tue, Nov 11, 2014 at 4:29 AM, Heikki Linnakangas < hlinnakan...@vmware.com >>> >>> wrote: >> >> >>> Attached is a new version. It's rebased on current git master, including >>> B

Re: PENDING_LIST_CLEANUP_SIZE - maximum size of GIN pending list Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

On Tue, Nov 11, 2014 at 10:14 PM, Robert Haas wrote: > On Tue, Nov 11, 2014 at 7:11 AM, Fujii Masao wrote: >>> OK, so if there are no objections of others, I'll mark this as "Ready for >>> Committer". >> >> I just pushed this. Thanks! > > Not to kibitz too much after-the-fact, but wouldn't it be

Re: [HACKERS] plpgsql plan changes causing failure after repeated invocation

Merlin Moncure wrote: > I chased down a problem today where users were reporting sporadic > failures in the application. Turns out, the function would work > exactly 5 times and then fail; this is on 9.2. I think I understand > why this is happening and I'm skeptical it's a bug in postgres, but

[HACKERS] plpgsql plan changes causing failure after repeated invocation

I chased down a problem today where users were reporting sporadic failures in the application. Turns out, the function would work exactly 5 times and then fail; this is on 9.2. I think I understand why this is happening and I'm skeptical it's a bug in postgres, but I thought I'd socialize it. W

Re: [HACKERS] [v9.5] Custom Plan API

On Tue, Nov 11, 2014 at 12:33 AM, Amit Kapila wrote: > On Mon, Nov 10, 2014 at 6:33 PM, Robert Haas wrote: >> On Mon, Nov 10, 2014 at 6:55 AM, Amit Kapila >> wrote: >> > I thought that in general if user has the API to register the custom >> > path >> > methods, it should have some way to unregi

Re: [HACKERS] BRIN indexes - TRAP: BadArgument

On Tue, Nov 11, 2014 at 1:04 PM, Alvaro Herrera wrote: >> It might be clearer ... > > Patches welcome :-) Or perhaps there could still be a single opaque field but have two optional opclass methods "scaninit" and "rescan" which allow the op class to set or reset whichever fields inside opaque tha

Re: PENDING_LIST_CLEANUP_SIZE - maximum size of GIN pending list Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

On Tue, Nov 11, 2014 at 7:11 AM, Fujii Masao wrote: >> OK, so if there are no objections of others, I'll mark this as "Ready for >> Committer". > > I just pushed this. Thanks! Not to kibitz too much after-the-fact, but wouldn't it be better to give this a name that has "GIN" in it somewhere? --

Re: [HACKERS] pg_prewarm really needs some CHECK_FOR_INTERRUPTS

On Tue, Nov 11, 2014 at 6:17 AM, Andres Freund wrote: > pg_prewarm() currently can't be cannot be interrupted - which seems odd > given that it's intended to read large amounts of data from disk. A > rather slow process. Oops. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterpr

Re: [HACKERS] BRIN indexes - TRAP: BadArgument

Greg Stark wrote: > It might be clearer to have an opclassinfo and a scaninfo which can > store information in separate opc_opaque and scan_opaque fields with > distinct liftetimes. > > In the bloom filter case the longlived info is the (initial?) size of > the bloom filter and the number of hash

Re: [HACKERS] BRIN indexes - TRAP: BadArgument

It might be clearer to have an opclassinfo and a scaninfo which can store information in separate opc_opaque and scan_opaque fields with distinct liftetimes. In the bloom filter case the longlived info is the (initial?) size of the bloom filter and the number of hash functions. But I still haven't

Re: [HACKERS] BRIN indexes - TRAP: BadArgument

Greg Stark wrote: > On Tue, Nov 11, 2014 at 12:12 PM, Alvaro Herrera > wrote: > > As far as I understand, the scan keys don't change within any given > > scan; if they do, the rescan AM method is called, at which point we > > should reset whatever is cached about the previous scan. > > But am I g

Re: [HACKERS] Add generate_series(numeric, numeric)

On Fri, Nov 7, 2014 at 3:19 PM, Fujii Masao wrote: > On Tue, Oct 14, 2014 at 3:22 PM, Michael Paquier > wrote: >> >> >> On Tue, Oct 7, 2014 at 8:38 AM, Ali Akbar wrote: >>> >>> 2014-10-06 22:51 GMT+07:00 Marti Raudsepp : >>> >>> > the one that tests values just before numeric overflow

Re: [HACKERS] BRIN indexes - TRAP: BadArgument

On Tue, Nov 11, 2014 at 12:12 PM, Alvaro Herrera wrote: > As far as I understand, the scan keys don't change within any given > scan; if they do, the rescan AM method is called, at which point we > should reset whatever is cached about the previous scan. But am I guaranteed that rescan will throw

Re: [HACKERS] BRIN indexes - TRAP: BadArgument

Greg Stark wrote: > On Tue, Nov 11, 2014 at 2:14 AM, Alvaro Herrera > wrote: > > I'm not sure why you say opaque is not associated with the specific > > scan. Are you thinking we could reuse opaque for a future scan? I > > think we could consider that opaque *is* the place to cache things such >

Re: PENDING_LIST_CLEANUP_SIZE - maximum size of GIN pending list Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

On Tue, Nov 11, 2014 at 7:01 PM, Etsuro Fujita wrote: > (2014/11/11 2:31), Fujii Masao wrote: >> >> On Mon, Nov 10, 2014 at 4:15 PM, Etsuro Fujita >>> >>> The patch looks good to me except for the following point: > > >>> *** a/src/backend/access/gin/ginfast.c >>> --- b/src/backend/access/gin/ginf

Re: [HACKERS] pg_prewarm really needs some CHECK_FOR_INTERRUPTS

On Tue, Nov 11, 2014 at 8:17 PM, Andres Freund wrote: > pg_prewarm() currently can't be cannot be interrupted - which seems odd > given that it's intended to read large amounts of data from disk. A > rather slow process. > > Unless somebody protests I'm going to add a check to the top of each of >

Re: [HACKERS] What exactly is our CRC algorithm?

At 2014-11-04 14:40:36 +0100, and...@2ndquadrant.com wrote: > > On 2014-11-04 08:21:13 -0500, Robert Haas wrote: > > > > Are you going to get the slice-by-N stuff working next, to speed > > this up? > > I don't plan to do anything serious with it, but I've hacked up the > crc code to use the hard

[HACKERS] pg_prewarm really needs some CHECK_FOR_INTERRUPTS

Hi, pg_prewarm() currently can't be cannot be interrupted - which seems odd given that it's intended to read large amounts of data from disk. A rather slow process. Unless somebody protests I'm going to add a check to the top of each of the three loops. Greetings, Andres Freund -- Andres Fre

Re: PENDING_LIST_CLEANUP_SIZE - maximum size of GIN pending list Re: [HACKERS] HEAD seems to generate larger WAL regarding GIN index

(2014/11/11 2:31), Fujii Masao wrote: On Mon, Nov 10, 2014 at 4:15 PM, Etsuro Fujita The patch looks good to me except for the following point: *** a/src/backend/access/gin/ginfast.c --- b/src/backend/access/gin/ginfast.c *** *** 25,30 --- 25,32 #include "utils/memuti

Re: [HACKERS] postgres_fdw behaves oddly

Hi Ashutosh, Thanks for the review! (2014/11/10 20:05), Ashutosh Bapat wrote: Since two separate issues 1. using reltargetlist instead of attr_needed and 2. system columns usage in FDW are being tackled here, we should separate the patch into two one for each of the issues. Agreed. Will spli

Re: [HACKERS] Scaling shared buffer eviction

On 2014-11-11 09:29:22 +, Thom Brown wrote: > On 26 September 2014 12:40, Amit Kapila wrote: > > > On Tue, Sep 23, 2014 at 10:31 AM, Robert Haas > > wrote: > > > > > > But this gets at another point: the way we're benchmarking this right > > > now, we're really conflating the effects of thr

Re: [HACKERS] Scaling shared buffer eviction

On 26 September 2014 12:40, Amit Kapila wrote: > On Tue, Sep 23, 2014 at 10:31 AM, Robert Haas > wrote: > > > > But this gets at another point: the way we're benchmarking this right > > now, we're really conflating the effects of three different things: > > > > 1. Changing the locking regimen a

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

On 2014-11-11 17:10:01 +0900, Michael Paquier wrote: > On Mon, Nov 10, 2014 at 5:26 PM, Michael Paquier > wrote: > > I'll go through the patch once again a bit later, but feel free to comment. > Reading again the patch with a fresher mind, I am not sure if the > current approach taken is really t

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

>I think this was changed based on following, if I am not wrong. >http://www.postgresql.org/message-id/54297A45.8080904@... Yes this change is the result of the above complaint. >Attaching the compression status to XLogRecord is more >in-line with the fact that all the blocks are compressed, an

Re: [HACKERS] WAL format and API changes (9.5)

On 11/11/2014 09:39 AM, Michael Paquier wrote: On Tue, Nov 11, 2014 at 4:29 AM, Heikki Linnakangas wrote: Attached is a new version. It's rebased on current git master, including BRIN. I've also fixed the laundry list of small things you reported, as well as a bunch of bugs I uncovered during

Re: [HACKERS] using custom scan nodes to prototype parallel sequential scan

On 10 November 2014 15:57, Robert Haas wrote: > On Wed, Oct 15, 2014 at 2:55 PM, Simon Riggs wrote: >> Something usable, with severe restrictions, is actually better than we >> have now. I understand the journey this work represents, so don't be >> embarrassed by submitting things with heuristics

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

On Tue, Nov 11, 2014 at 5:10 PM, Michael Paquier wrote: > On Mon, Nov 10, 2014 at 5:26 PM, Michael Paquier > wrote: >> I'll go through the patch once again a bit later, but feel free to comment. > Reading again the patch with a fresher mind, I am not sure if the > current approach taken is really

Re: [HACKERS] BRIN indexes - TRAP: BadArgument

On Tue, Nov 11, 2014 at 2:14 AM, Alvaro Herrera wrote: > I'm not sure why you say opaque is not associated with the specific > scan. Are you thinking we could reuse opaque for a future scan? I > think we could consider that opaque *is* the place to cache things such > as the hashed value of the

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

On Mon, Nov 10, 2014 at 5:26 PM, Michael Paquier wrote: > I'll go through the patch once again a bit later, but feel free to comment. Reading again the patch with a fresher mind, I am not sure if the current approach taken is really the best one. What the patch does now is looking at the header of

Re: [HACKERS] Proposal: Log inability to lock pages during vacuum

On 2014-11-10 19:36:18 -0600, Jim Nasby wrote: > On 11/10/14, 12:56 PM, Andres Freund wrote: > >On 2014-11-10 12:37:29 -0600, Jim Nasby wrote: > >>On 11/10/14, 12:15 PM, Andres Freund wrote: > >If what we want is to quantify the extent of the issue, would it be more > >convenient to save co