Re: [HACKERS] WIP: [[Parallel] Shared] Hash

2017-03-26 Thread Andres Freund
On 2017-03-23 20:35:09 +1300, Thomas Munro wrote: > Here is a new patch series responding to feedback from Peter and Andres: + +/* Per-participant shared state. */ +typedef struct SharedTuplestoreParticipant +{ + LWLock lock; Hm. No padding (ala LWLockMinimallyPadded / LWLockPadded) - but

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

2017-03-26 Thread Peter Geoghegan
On Sun, Mar 26, 2017 at 3:41 PM, Thomas Munro wrote: >> 1. Segments are what buffile.c already calls the individual >> capped-at-1GB files that it manages. They are an implementation >> detail that is not part of buffile.c's user interface. There seems to >> be

Re: [HACKERS] patch proposal

2017-03-26 Thread Venkata B Nagothi
Hi David, On Thu, Mar 23, 2017 at 4:21 AM, David Steele wrote: > On 3/21/17 8:45 PM, Venkata B Nagothi wrote: > >> On Tue, Mar 21, 2017 at 8:46 AM, David Steele > >> Unfortunately, I don't think the first patch (recoveryStartPoint) >> will work

Re: [HACKERS] pg_get_statisticsextdef() is not quite the full shilling

2017-03-26 Thread Alvaro Herrera
David Rowley wrote: > Seems pg_get_statisticsextdef() has a couple of things wrong: > > 1. HeapTupleIsValid() called on the wrong tuple. > 2. Did not schema qualify names. Actually we can solve both 1 and the first half of 2 by just using generate_relation_name, which is less code and gives

Re: [HACKERS] [PATCH] few fts functions for jsonb

2017-03-26 Thread Dmitry Dolgov
> I'm not through looking at this. However, here are a few preliminary comments I've attached new versions of the patches with improvements related to these commentaries. diff --git a/src/backend/tsearch/to_tsany.c b/src/backend/tsearch/to_tsany.c index 6e5de8f..8f7bcfe 100644 ---

Re: [HACKERS] crashes due to setting max_parallel_workers=0

2017-03-26 Thread David Rowley
On 27 March 2017 at 10:23, Tomas Vondra wrote: > I'm not sure we need to invent a new magic value, though. Can we simply look > at force_parallel_mode, and if it's 'regress' then tread 0 differently? see standard_planner() if (force_parallel_mode !=

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

2017-03-26 Thread Thomas Munro
On Mon, Mar 27, 2017 at 12:12 PM, Peter Geoghegan wrote: > On Sun, Mar 26, 2017 at 3:41 PM, Thomas Munro > wrote: >>> 1. Segments are what buffile.c already calls the individual >>> capped-at-1GB files that it manages. They are an implementation >>>

Re: [HACKERS] [COMMITTERS] pgsql: Show more processes in pg_stat_activity.

2017-03-26 Thread Michael Paquier
On Mon, Mar 27, 2017 at 11:07 AM, Robert Haas wrote: > Show more processes in pg_stat_activity. > > Previously, auxiliary processes and background workers not connected > to a database (such as the logical replication launcher) weren't > shown. Include them, so that we can

Re: [HACKERS] comments in hash_alloc_buckets

2017-03-26 Thread Robert Haas
On Sat, Mar 25, 2017 at 3:28 AM, Ashutosh Sharma wrote: >>> While working on - [1], I realised that the following comments in >>> _hash_alloc_buckets() needs to be corrected. >>> >>> /* >>> * Initialize the freed overflow page. Just zeroing the page won't >>>

Re: [HACKERS] Problem in Parallel Bitmap Heap Scan?

2017-03-26 Thread Dilip Kumar
On Sat, Mar 25, 2017 at 2:25 PM, Thomas Munro wrote: >>> I think in this area we need more testing, reason these are not tested >>> properly because these are not the natural case for parallel bitmap. >>> I think in next few days I will test more such cases by

Re: [HACKERS] Re: [COMMITTERS] pgsql: Faster expression evaluation and targetlist projection.

2017-03-26 Thread Joe Conway
On 03/25/2017 09:52 PM, Andres Freund wrote: > On 2017-03-25 20:40:23 -0700, Andres Freund wrote: >> I blindly tried to fix these, let's hope that works. > > In a second attempt (yes, reading diffs correctly is helpful), this > resolved the selinux issue. Yeha. +1! Joe -- Crunchy Data -

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

2017-03-26 Thread Thomas Munro
On Mon, Mar 27, 2017 at 9:41 AM, Andres Freund wrote: > Hi, > > > SharedBufFile allows temporary files to be created by one backend and > then exported for read-only access by other backends, with clean-up > managed by reference counting associated with a DSM segment. This

Re: [HACKERS] New CORRESPONDING clause design

2017-03-26 Thread Pavel Stehule
Hi 2017-03-25 13:41 GMT+01:00 Surafel Temesgen : > > >> >> I took a quick look through this and noted that it fails to touch >> ruleutils.c, which means that dumping of views containing CORRESPONDING >> certainly doesn't work. >> > fixed > >> Also, the changes in

Re: [HACKERS] Logical decoding on standby

2017-03-26 Thread Craig Ringer
On 20 March 2017 at 17:33, Andres Freund wrote: > Have you checked how high the overhead of XLogReadDetermineTimeline is? > A non-local function call, especially into a different translation-unit > (no partial inlining), for every single page might end up being > noticeable.

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

2017-03-26 Thread Peter Geoghegan
On Sun, Mar 26, 2017 at 6:50 PM, Thomas Munro wrote: > Like you, I also tend to suspect that people would be more likely to > use RAID type technologies to stripe things like this for both > bandwidth and space reasons these days. Tablespaces seem to make more >

Re: [HACKERS] crashes due to setting max_parallel_workers=0

2017-03-26 Thread Tomas Vondra
On 03/25/2017 05:18 PM, Rushabh Lathia wrote: On Sat, Mar 25, 2017 at 7:01 PM, Peter Eisentraut > wrote: On 3/25/17 09:01, David Rowley wrote: > On 25 March 2017 at 23:09, Rushabh Lathia

Re: [HACKERS] exposing wait events for non-backends (was: Tracking wait event for latches)

2017-03-26 Thread Michael Paquier
On Sat, Mar 25, 2017 at 5:26 PM, Kuntal Ghosh wrote: > On Fri, Mar 24, 2017 at 9:23 PM, Robert Haas wrote: >> I think this is still not good. The places where pgstat_bestart() has >> been added are not even correct. For example, the call added

Re: [HACKERS] Time to drop old-style (V0) functions?

2017-03-26 Thread Craig Ringer
On 27 March 2017 at 10:59, Craig Ringer wrote: > On 27 March 2017 at 10:45, Craig Ringer wrote: > >> Passes "make check" and recovery tests, check-world running now. > > A couple of fixes pending. Updated. I didn't have any way to make

Re: [HACKERS] \if, \elseif, \else, \endif (was Re: PSQL commands: \quit_if, \quit_unless)

2017-03-26 Thread Corey Huinker
> > Patches do not apply cleanly. > Part 1 gets: > error: patch failed: src/test/regress/parallel_schedule:89 > error: src/test/regress/parallel_schedule: patch does not apply > > There is still the useless file, ok it is removed by part2. Could have > been just one patch... > parallel_schedule

Re: [HACKERS] Performance improvement for joins where outer side is unique

2017-03-26 Thread David Rowley
On 14 March 2017 at 16:37, David Rowley wrote: > On 14 March 2017 at 11:35, David Rowley > wrote: >> >> On 14 March 2017 at 07:50, Tom Lane wrote: >>> >>> [ getting back to this patch finally... ] >>> >>> David

Re: [HACKERS] crashes due to setting max_parallel_workers=0

2017-03-26 Thread Tomas Vondra
On 03/25/2017 02:01 PM, David Rowley wrote: > I wondered if there's anything we can do here to better test cases when no workers are able to try to ensure the parallel nodes work correctly, but the more I think about it, the more I don't see wrong with just using SET max_parallel_workers = 0;

Re: [HACKERS] Speedup twophase transactions

2017-03-26 Thread Michael Paquier
On Sun, Mar 26, 2017 at 4:50 PM, Nikhil Sontakke wrote: > I was away for a bit. I will take a look at this patch and get back to you > soon. No problem. Thanks for your time! -- Michael -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make

Re: [HACKERS] logical decoding of two-phase transactions

2017-03-26 Thread Craig Ringer
On 20 March 2017 at 21:47, Stas Kelvich wrote: > >> On 20 Mar 2017, at 16:39, Craig Ringer wrote: >> >> On 20 March 2017 at 20:57, Stas Kelvich wrote: >>> On 20 Mar 2017, at 15:17, Craig Ringer

Re: [HACKERS] Performance improvement for joins where outer side is unique

2017-03-26 Thread David Rowley
On 27 March 2017 at 09:28, David Rowley wrote: > Patch is attached which fixes up the conflict between the expression > evaluation performance patch. Seems I forgot to commit locally before creating the patch... Here's the actual patch I meant to attach earlier.

Re: [HACKERS] Valgrind failures caused by multivariate stats patch

2017-03-26 Thread Tomas Vondra
On 03/26/2017 08:47 PM, Andres Freund wrote: On 2017-03-26 20:38:52 +0200, Tomas Vondra wrote: ... Hmmm, so I have a theory about what's going on, but no matter what I do I can't trigger these valgrind failures. What switches are you using to start valgrind? I'm using this: valgrind

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

2017-03-26 Thread Thomas Munro
On Mon, Mar 27, 2017 at 11:03 AM, Thomas Munro >> Is there a risk that this ends up running afoul of filename length >> limits on some platforms? > > Hmm. I didn't think so. Do we have a project guideline on maximum > path lengths based on some kind of survey?

Re: [HACKERS] pg_stat_wal_write statistics view

2017-03-26 Thread Haribabu Kommi
On Sat, Mar 25, 2017 at 6:40 AM, Fujii Masao wrote: > On Wed, Feb 15, 2017 at 12:53 PM, Haribabu Kommi > wrote: > > > > > > On Wed, Feb 8, 2017 at 9:36 PM, Amit Kapila > wrote: > >> > >> On Tue, Feb 7, 2017 at 11:47 AM,

Re: [HACKERS] Time to drop old-style (V0) functions?

2017-03-26 Thread Craig Ringer
On 7 March 2017 at 22:50, Peter Eisentraut wrote: > I think we have consensus to go ahead with this, and the patches are > mostly mechanical, so I only have a few comments on style and one > possible bug below: > > >

Re: [HACKERS] Time to drop old-style (V0) functions?

2017-03-26 Thread Craig Ringer
On 27 March 2017 at 10:45, Craig Ringer wrote: > Passes "make check" and recovery tests, check-world running now. A couple of fixes pending. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services --

Re: [HACKERS] [PATCH] Move all am-related reloption code into src/backend/access/[am-name] and get rid of relopt_kind for custom AM

2017-03-26 Thread Nikolay Shaplov
В письме от 26 марта 2017 15:02:12 Вы написали: > Nikolay Shaplov wrote: > > If I would think about it now: we always know how many options we will > > have. So we can just pass this number to palloc and assert if somebody > > adds more options then expected... What do yo think about it. > > I

Re: [HACKERS] [PATCH] Move all am-related reloption code into src/backend/access/[am-name] and get rid of relopt_kind for custom AM

2017-03-26 Thread Alvaro Herrera
Nikolay Shaplov wrote: > If I would think about it now: we always know how many options we will have. > So we can just pass this number to palloc and assert if somebody adds more > options then expected... What do yo think about it. I think we need to preserve the ability to add custom

Re: [HACKERS] PDF build is broken

2017-03-26 Thread Peter Eisentraut
On 3/25/17 07:27, Devrim Gündüz wrote: > I can't build PDFs with latest snapshot tarball: Fixed. But I also suggest that you try out the FOP based builds, because the jadetex-based builds will probably go away soon. > >

[HACKERS] Stale comments in vacuumlazy.c

2017-03-26 Thread Pavan Deolasee
I happened to notice a stale comment at the very beginning of vacuumlazy.c. ISTM we forgot to fix that when we introduced FSM. With FSM, vacuum no longer needed to track per-page free space info. I propose attached fix. Thanks, Pavan -- Pavan Deolasee

Re: [HACKERS] [PATCH] Move all am-related reloption code into src/backend/access/[am-name] and get rid of relopt_kind for custom AM

2017-03-26 Thread Nikolay Shaplov
В письме от 17 марта 2017 14:21:26 пользователь Alvaro Herrera написал: > I gave this patch a quick skim. Thanks! > At first I was confused by the term > "catalog"; I thought it meant we stored options in a system table. But > that's not what is meant at all; instead, what we do is build these >

Re: [HACKERS] Valgrind failures caused by multivariate stats patch

2017-03-26 Thread Andres Freund
On 2017-03-26 20:38:52 +0200, Tomas Vondra wrote: > On 03/25/2017 10:10 PM, Andres Freund wrote: > ... > > ==2486== Uninitialised byte(s) found during client check request > > ==2486==at 0x1C4857: printtup (printtup.c:347) > > ==2486==by 0x401FD5: ExecutePlan (execMain.c:1681) > > ==2486==

Re: [HACKERS] [sqlsmith] Unpinning error in parallel worker

2017-03-26 Thread Thomas Munro
On Mon, Mar 27, 2017 at 4:18 AM, Andreas Seltenreich wrote: > Hi, > > today's testing with master as of d253b0f6e3 yielded two clusters that > stopped processing queries. Symptoms: > > [...] Thanks Andreas. Investigating. -- Thomas Munro http://www.enterprisedb.com --

Re: [HACKERS] WIP: [[Parallel] Shared] Hash

2017-03-26 Thread Andres Freund
Hi, SharedBufFile allows temporary files to be created by one backend and then exported for read-only access by other backends, with clean-up managed by reference counting associated with a DSM segment. This includes changes to fd.c and buffile.c to support new kinds of temporary file. diff

Re: [HACKERS] [PATCH] Move all am-related reloption code into src/backend/access/[am-name] and get rid of relopt_kind for custom AM

2017-03-26 Thread Nikolay Shaplov
В письме от 23 марта 2017 16:14:58 пользователь Fabrízio de Royes Mello написал: > On Thu, Mar 23, 2017 at 3:58 PM, Alvaro Herrera > > wrote: > > Copying Fabrízio Mello here, who spent some time trying to work on > > reloptions too. He may have something to say about

Re: [HACKERS] Valgrind failures caused by multivariate stats patch

2017-03-26 Thread Tomas Vondra
On 03/25/2017 10:10 PM, Andres Freund wrote: ... ==2486== Uninitialised byte(s) found during client check request ==2486==at 0x1C4857: printtup (printtup.c:347) ==2486==by 0x401FD5: ExecutePlan (execMain.c:1681) ==2486==by 0x3FFDED: standard_ExecutorRun (execMain.c:355) ==2486==

[HACKERS] pg_dump truncating queries in error messages

2017-03-26 Thread Peter Eisentraut
When reporting an error from a query, pg_dump truncates the reported query to 128 characters (pg_backup_db.c ExecuteSqlCommand()). Is this (still) sensible? The kind of queries that pg_dump is running nowadays, I find myself unable to debug them if they are truncated at that length. -- Peter

Re: [HACKERS] pg_dump truncating queries in error messages

2017-03-26 Thread Tom Lane
Peter Eisentraut writes: > When reporting an error from a query, pg_dump truncates the reported > query to 128 characters (pg_backup_db.c ExecuteSqlCommand()). > Is this (still) sensible? The kind of queries that pg_dump is running > nowadays, I find myself

[HACKERS] [sqlsmith] Failed assertion in _hash_kill_items/MarkBufferDirtyHint

2017-03-26 Thread Andreas Seltenreich
Hi, testing with master as of cf366e97ff, sqlsmith occasionally triggers the following assertion: TRAP: FailedAssertion("!(LWLockHeldByMe(((LWLock*) (&(bufHdr)->content_lock", File: "bufmgr.c", Line: 3397) Backtraces always look like the one below. It is reproducible on a cluster once it

Re: [HACKERS] BRIN cost estimate

2017-03-26 Thread Emre Hasegeli
> If we want to have a variable which stores the number of ranges, then > I think numRanges is better than numBlocks. I can't imagine many > people would disagree there. I renamed it with other two. > At the very least please write a comment to explain this in the code. > Right now it looks

Re: [HACKERS] delta relations in AFTER triggers

2017-03-26 Thread Thomas Munro
On Fri, Mar 24, 2017 at 1:14 PM, Thomas Munro wrote: > On Tue, Mar 14, 2017 at 7:51 AM, Kevin Grittner wrote: >> On Sun, Mar 12, 2017 at 4:08 PM, Thomas Munro >> wrote: >>> I found a new way to break it: run the

Re: [HACKERS] Speedup twophase transactions

2017-03-26 Thread Nikhil Sontakke
Thanks Michael, I was away for a bit. I will take a look at this patch and get back to you soon. Regards, Nikhils On 22 March 2017 at 07:40, Michael Paquier wrote: > On Fri, Mar 17, 2017 at 5:15 PM, Michael Paquier > wrote: > > On Fri,

Re: [HACKERS] \if, \elseif, \else, \endif (was Re: PSQL commands: \quit_if, \quit_unless)

2017-03-26 Thread Fabien COELHO
Hello Corey, v25, try 2: First file is what you were used to last time. 2nd and 3rd are changes since then based on feedback. Patches do not apply cleanly. Part 1 gets: error: patch failed: src/test/regress/parallel_schedule:89 error: src/test/regress/parallel_schedule: patch does not

Re: [HACKERS] logical replication apply to run with sync commit off by default

2017-03-26 Thread Masahiko Sawada
On Fri, Mar 24, 2017 at 11:49 PM, Petr Jelinek wrote: > On 21/03/17 22:37, Petr Jelinek wrote: >> On 21/03/17 18:54, Robert Haas wrote: >>> On Mon, Mar 20, 2017 at 7:56 PM, Petr Jelinek >>> wrote: On 18/03/17 13:31, Petr Jelinek

[HACKERS] [sqlsmith] Unpinning error in parallel worker

2017-03-26 Thread Andreas Seltenreich
Hi, today's testing with master as of d253b0f6e3 yielded two clusters that stopped processing queries. Symptoms: regression=> select application_name, state, wait_event, wait_event_type, count(1), min(pid) from pg_stat_activity group by 1,2,3,4; application_name | state |

Re: [HACKERS] [PATCH] ALTER DEFAULT PRIVILEGES with GRANT/REVOKE ON SCHEMAS

2017-03-26 Thread Matheus de Oliveira
On Thu, Mar 2, 2017 at 10:27 AM, David Steele wrote: > It looks like this patch is still waiting on an update for tab > completion in psql. Hi All, Sorry about the long delay... It was so simple to add it to tab-complete.c that is a shame I didn't do it before, very sorry

Re: [HACKERS] free space map and visibility map

2017-03-26 Thread Kyotaro HORIGUCHI
At Sat, 25 Mar 2017 19:53:47 -0700, Jeff Janes wrote in

Re: [HACKERS] [POC] A better way to expand hash indexes.

2017-03-26 Thread Amit Kapila
On Sun, Mar 26, 2017 at 11:26 AM, Mithun Cy wrote: > Thanks, Amit for the review. > On Sat, Mar 25, 2017 at 7:03 PM, Amit Kapila wrote: >> >> I think one-dimensional patch has fewer places to touch, so that looks >> better to me. However, I

Re: [HACKERS] crashes due to setting max_parallel_workers=0

2017-03-26 Thread Rushabh Lathia
On Mon, Mar 27, 2017 at 3:43 AM, Tomas Vondra wrote: > On 03/25/2017 05:18 PM, Rushabh Lathia wrote: > >> >> >> On Sat, Mar 25, 2017 at 7:01 PM, Peter Eisentraut >> > > wrote: >> >> On

Re: [HACKERS] WIP: Faster Expression Processing v4

2017-03-26 Thread Andres Freund
On 2017-03-25 20:59:27 -0700, Andres Freund wrote: > On 2017-03-25 23:51:45 -0400, Tom Lane wrote: > > Andres Freund writes: > > > On March 25, 2017 4:56:11 PM PDT, Ants Aasma wrote: > > >> I haven't had the time to research this properly, but initial

Re: [HACKERS] [sqlsmith] Unpinning error in parallel worker

2017-03-26 Thread Thomas Munro
On Mon, Mar 27, 2017 at 8:38 AM, Thomas Munro wrote: > On Mon, Mar 27, 2017 at 4:18 AM, Andreas Seltenreich > wrote: >> Hi, >> >> today's testing with master as of d253b0f6e3 yielded two clusters that >> stopped processing queries. Symptoms:

Re: [HACKERS] [COMMITTERS] pgsql: Improve access to parallel query from procedural languages.

2017-03-26 Thread Rafia Sabih
On Sun, Mar 26, 2017 at 3:34 AM, Tom Lane wrote: > I wrote: >> It doesn't seem to be a platform-specific problem: I can duplicate the >> failure here by applying same settings mandrill uses, ie build with >> -DRANDOMIZE_ALLOCATED_MEMORY and set force_parallel_mode = regress. >