[HACKERS] datetime.h defines like PM conflict with external libraries

2017-10-03 Thread Andres Freund
to conflict? Alternatively we could use #pragma push_macro() around the includes, but that'd be a new dependency. Better ideas? Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref

Re: [HACKERS] Add TOAST to system tables with ACL?

2017-10-03 Thread Andres Freund
ok. It'd be a lot easier to test if there were an easier way to force columns to be toasted. Sometimes I wonder about making TOAST_TUPLE_THRESHOLD configurable... > For pg_database, you'd have to make sure that the startup-time check of > database CONNECT privilege still works if the ACL'

Re: [HACKERS] SendRowDescriptionMessage() is slow for queries with a lot of columns

2017-10-03 Thread Andres Freund
On 2017-10-03 11:06:08 -0400, Robert Haas wrote: > On Tue, Oct 3, 2017 at 3:55 AM, Andres Freund <and...@anarazel.de> wrote: > > Attached is a revised version of this patchset. > > I don't much like the functions with "_pre" affixed to their names. > It's

Re: [HACKERS] 64-bit queryId?

2017-10-03 Thread Andres Freund
On 2017-10-03 17:06:20 +0900, Michael Paquier wrote: > On Tue, Oct 3, 2017 at 3:12 PM, Andres Freund <and...@anarazel.de> wrote: > > On 2017-10-03 03:07:09 +0300, Alexander Korotkov wrote: > >> On Tue, Oct 3, 2017 at 12:32 AM, Tom Lane <t...@sss.pgh.pa.us> wrote:

Re: [HACKERS] SendRowDescriptionMessage() is slow for queries with a lot of columns

2017-10-03 Thread Andres Freund
not done before, but it's C99 and delivers significantly more efficient code. Any arguments against? Regards, Andres >From ff8c4128a46199beab2beb09c1ad0627bbc18b94 Mon Sep 17 00:00:00 2001 From: Andres Freund <and...@anarazel.de> Date: Wed, 20 Sep 2017 13:01:22 -0700 Subject: [PATCH

Re: [HACKERS] pnstrdup considered armed and dangerous

2017-10-03 Thread Andres Freund
On 2016-10-03 14:55:24 -0700, Andres Freund wrote: > Hi, > > A colleage of me just wrote innocent looking code like > char *shardRelationName = pnstrdup(relationName, NAMEDATALEN); > which is at the moment wrong if relationName isn't preallocated to >

Re: [HACKERS] 64-bit queryId?

2017-10-03 Thread Andres Freund
On 2017-10-03 03:07:09 +0300, Alexander Korotkov wrote: > On Tue, Oct 3, 2017 at 12:32 AM, Tom Lane wrote: > > > Peter Geoghegan writes: > > > You need to change the SQL interface as well, although I'm not sure > > > exactly how. The problem is that you are now

[HACKERS] Combining expr{Type,Typmod,Collation}() into one function.

2017-10-02 Thread Andres Freund
with this, some cleanup would be needed. Greetings, Andres Freund >From 04dc797d6e99d7fe5902a31ae5fdd73950c8c48f Mon Sep 17 00:00:00 2001 From: Andres Freund <and...@anarazel.de> Date: Thu, 21 Sep 2017 12:13:38 -0700 Subject: [PATCH] WIP: Combine expr{Type,Typmod,Collation} into one

Re: [HACKERS] Horrible CREATE DATABASE Performance in High Sierra

2017-10-02 Thread Andres Freund
On 2017-10-02 19:50:51 -0400, Tom Lane wrote: > Andres Freund <and...@anarazel.de> writes: > > On 2017-10-02 18:33:17 -0400, Tom Lane wrote: > >> I'm kind of surprised that machine B doesn't show obvious tanking in this > >> test given that msync() makes it su

Re: [HACKERS] Horrible CREATE DATABASE Performance in High Sierra

2017-10-02 Thread Andres Freund
Hi, On 2017-10-02 18:33:17 -0400, Tom Lane wrote: > Andres Freund <and...@anarazel.de> writes: > > To demonstrate what I'm observing here, on linux with a fairly fast ssd: > > ... > > I tried to replicate this test as closely as I could on the Mac hardware > I ha

Re: [HACKERS] Binary search in fmgr_isbuiltin() is a bottleneck.

2017-10-02 Thread Andres Freund
On 2017-10-02 17:57:51 -0400, Tom Lane wrote: > Andres Freund <and...@anarazel.de> writes: > > Done that way. It's a bit annoying, because we've to take care to > > initialize the "unused" part of the array with a valid signalling it's > > an unused

Re: [HACKERS] Binary search in fmgr_isbuiltin() is a bottleneck.

2017-10-02 Thread Andres Freund
On 2017-09-28 19:06:27 -0400, Tom Lane wrote: > Andres Freund <and...@anarazel.de> writes: > > On 2017-09-28 18:52:28 -0400, Tom Lane wrote: > >> Uh, what? Access to fmgr_nbuiltins shouldn't be part of any critical path > >> anymore after this change. > >

Re: [HACKERS] [sqlsmith] stuck spinlock in pg_stat_get_wal_receiver after OOM

2017-10-02 Thread Andres Freund
On 2017-10-02 17:30:25 -0400, Tom Lane wrote: > Andres Freund <and...@anarazel.de> writes: > > Yes, that'd be a bad idea. It's not great to have memcpys in a critical > > section, but it's way better than pallocs. So we need to use some local > > buffers that this get

Re: [HACKERS] [sqlsmith] stuck spinlock in pg_stat_get_wal_receiver after OOM

2017-10-02 Thread Andres Freund
get copied to. This seems to have been introduced as part of b1a9bad9e74 and then 9ed551e0a4f. Authors CCed. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Horrible CREATE DATABASE Performance in High Sierra

2017-10-02 Thread Andres Freund
On 2017-10-02 15:59:05 -0400, Tom Lane wrote: > Andres Freund <and...@anarazel.de> writes: > > On 2017-10-02 15:54:43 -0400, Tom Lane wrote: > >> Should I expect there to be any difference at all? We don't enable > >> *_flush_after by default on non-Linux pl

Re: [HACKERS] Horrible CREATE DATABASE Performance in High Sierra

2017-10-02 Thread Andres Freund
On 2017-10-02 15:54:43 -0400, Tom Lane wrote: > Andres Freund <and...@anarazel.de> writes: > > On 2017-10-02 15:42:25 -0400, Tom Lane wrote: > >> I experimented with this further by seeing whether the msync() code path > >> is of any value on Sierra either. The

Re: [HACKERS] Horrible CREATE DATABASE Performance in High Sierra

2017-10-02 Thread Andres Freund
. When > and if Apple changes their kernel so that this path is actually of some > value, we can figure out how to detect whether to use it. I'm inclined to think you're right. This is a surprisingly massive regression for a new OS release - we're not the only database that uses msync... Greetings,

Re: [HACKERS] Logging idle checkpoints

2017-10-01 Thread Andres Freund
On 2017-10-02 07:43:31 +0900, Michael Paquier wrote: > On Mon, Oct 2, 2017 at 7:41 AM, Andres Freund <and...@anarazel.de> wrote: > > On 2017-10-02 07:39:18 +0900, Michael Paquier wrote: > >> On Mon, Oct 2, 2017 at 7:27 AM, Andres Freund <and...@anarazel.de> wrote:

Re: [HACKERS] Logging idle checkpoints

2017-10-01 Thread Andres Freund
On 2017-10-02 07:39:18 +0900, Michael Paquier wrote: > On Mon, Oct 2, 2017 at 7:27 AM, Andres Freund <and...@anarazel.de> wrote: > > On 2017-10-02 00:19:33 +0200, Vik Fearing wrote: > > I'd be ok with applying this now, or in 10.1 - but I do think we should > > fix

Re: [HACKERS] Logging idle checkpoints

2017-10-01 Thread Andres Freund
idle"))); > return; > } I'd be ok with applying this now, or in 10.1 - but I do think we should fix this before 11. If nobody protests I'll push later today, so we can get some bf cycles for the very remote case that this causes problems.

Re: [HACKERS] pg_ctl kill support for KILL signal was Re: [COMMITTERS] pgsql: Add test for postmaster crash restarts.

2017-10-01 Thread Andres Freund
On 2017-10-01 18:01:27 -0400, Tom Lane wrote: > Andres Freund <and...@anarazel.de> writes: > > On 2017-10-01 17:47:52 -0400, Andrew Dunstan wrote: > >> So we can leave it out of there. OTOH I'm not a huge fan of security by > >> obscurity. I guess this would

Re: [HACKERS] pg_ctl kill support for KILL signal was Re: [COMMITTERS] pgsql: Add test for postmaster crash restarts.

2017-10-01 Thread Andres Freund
On 2017-10-01 17:47:52 -0400, Andrew Dunstan wrote: > > > On 10/01/2017 04:48 PM, Andres Freund wrote: > > On 2017-10-01 16:42:44 -0400, Tom Lane wrote: > >> Andrew Dunstan <andrew.duns...@2ndquadrant.com> writes: > >>> On 09/30/2017 10:32 PM, Andr

Re: [HACKERS] pg_ctl kill support for KILL signal was Re: [COMMITTERS] pgsql: Add test for postmaster crash restarts.

2017-10-01 Thread Andres Freund
On 2017-10-01 16:42:44 -0400, Tom Lane wrote: > Andrew Dunstan <andrew.duns...@2ndquadrant.com> writes: > > On 09/30/2017 10:32 PM, Andres Freund wrote: > >> Heh. I'm inclined to take it out. We could add a --use-the-force-luke > >> type parameter, but it does

Re: [HACKERS] 64-bit queryId?

2017-09-30 Thread Andres Freund
On 2017-09-30 18:17:37 -0400, Robert Haas wrote: > On Sat, Sep 30, 2017 at 11:55 AM, Andres Freund <and...@anarazel.de> wrote: > > On 2017-09-30 11:50:08 -0400, Robert Haas wrote: > >> Well, I think that the fact that pg_stat_statements.max exists at all > >> is

Re: [HACKERS] 64-bit queryId?

2017-09-30 Thread Andres Freund
mouth. If you design a hashtable you're concerned about unnecessary collisions, even if you're aware of them. Additionally, it's not clear you always can do all that much about the collisions, without accepting undue overhead - see e.g. pg_stat_statements. Greetings, Andres Freund --

Re: [HACKERS] 64-bit queryId?

2017-09-30 Thread Andres Freund
here's a huge space concern anymore? Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] SendRowDescriptionMessage() is slow for queries with a lot of columns

2017-09-29 Thread Andres Freund
On 2017-09-29 17:56:10 -0400, Tom Lane wrote: > Andres Freund <and...@anarazel.de> writes: > > Does anybody have an opinion on whether we'll want to convert examples > > like testlibpq3.c (included in libpq.sgml) too? I'm inclined not to, > > because currently us

[HACKERS] Why are we including netinet/tcp.h so widely?

2017-09-29 Thread Andres Freund
, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] SendRowDescriptionMessage() is slow for queries with a lot of columns

2017-09-29 Thread Andres Freund
On 2017-09-28 14:23:45 +0900, Michael Paquier wrote: > On Thu, Sep 28, 2017 at 1:31 PM, Andres Freund <and...@anarazel.de> wrote: > > On September 27, 2017 9:06:49 PM PDT, Andres Freund <and...@anarazel.de> > > wrote: > >>On 2017-09-28 00:01:53 -0400, T

Re: [HACKERS] Minor codegen silliness in ExecInterpExpr()

2017-09-28 Thread Andres Freund
On 2017-09-28 20:56:57 -0700, Andres Freund wrote: > +1, I saw the same in a bleeding edge c++ version. err, gcc version. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Minor codegen silliness in ExecInterpExpr()

2017-09-28 Thread Andres Freund
FWIW, it's being specificied in C++17 ([1]), which seems to make it not unlikely to end up in C as well. > but it does suggest that this is worth fixing, and is not just an artifact > of an old compiler version. +1, I saw the same in a bleeding edge c++ version. Greetings, Andres Freund [

Re: [HACKERS] Binary search in fmgr_isbuiltin() is a bottleneck.

2017-09-28 Thread Andres Freund
On 2017-09-28 18:52:28 -0400, Tom Lane wrote: > Andres Freund <and...@anarazel.de> writes: > > I might be worse than you... But anyway, here's a patch doing > > so. Looking at profiles, it turned out that having the integer limits as > > extern variables in a diff

Re: [HACKERS] Minor codegen silliness in ExecInterpExpr()

2017-09-28 Thread Andres Freund
On 2017-09-28 18:39:03 -0400, Tom Lane wrote: > Andres Freund <and...@anarazel.de> writes: > > On 2017-09-28 16:21:34 -0400, Tom Lane wrote: > >> We could save a pointless register spill > >> and reload if there were a temporary variable in there, > > &g

Re: [HACKERS] Minor codegen silliness in ExecInterpExpr()

2017-09-28 Thread Andres Freund
fo); > + *op->resvalue = fvalue; > *op->resnull = fcinfo->isnull; > > EEO_NEXT(); > } > > and likewise in the other FUNCEXPR cases. > > This is on a rather old gcc, haven't checked on bleeding-edge versions. Mak

Re: [HACKERS] Binary search in fmgr_isbuiltin() is a bottleneck.

2017-09-28 Thread Andres Freund
ly in fmgr.c. Is this roughly what you were thinking of? Greetings, Andres Freund >From 11fc054fda92fafc8b6c1ac66f70ecf059c61a9d Mon Sep 17 00:00:00 2001 From: Andres Freund <and...@anarazel.de> Date: Tue, 26 Sep 2017 12:40:56 -0700 Subject: [PATCH] Speed up fmgr_isbuiltin() by keeping an oi

Re: [HACKERS] SendRowDescriptionMessage() is slow for queries with a lot of columns

2017-09-27 Thread Andres Freund
On September 27, 2017 9:06:49 PM PDT, Andres Freund <and...@anarazel.de> wrote: >On 2017-09-28 00:01:53 -0400, Tom Lane wrote: >> Andres Freund <and...@anarazel.de> writes: >> > Attached is an extension of the already existing pg_bswap.h that >> > a) adds

Re: [HACKERS] SendRowDescriptionMessage() is slow for queries with a lot of columns

2017-09-27 Thread Andres Freund
On 2017-09-28 00:01:53 -0400, Tom Lane wrote: > Andres Freund <and...@anarazel.de> writes: > > Attached is an extension of the already existing pg_bswap.h that > > a) adds 16 bit support > > b) moves everything to inline functions, removing multiple evaluation >

Re: [HACKERS] Binary search in fmgr_isbuiltin() is a bottleneck.

2017-09-27 Thread Andres Freund
On 2017-09-27 15:30:45 -0400, Robert Haas wrote: > On Wed, Sep 27, 2017 at 1:40 PM, Andres Freund <and...@anarazel.de> wrote: > > I don't think you can even measure the overhead of building the > > table. This is inserting ~8k rows in an accurately sized hashtable - a > &g

Re: [HACKERS] Binary search in fmgr_isbuiltin() is a bottleneck.

2017-09-27 Thread Andres Freund
ely see more of these coming up, but who knows? Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Binary search in fmgr_isbuiltin() is a bottleneck.

2017-09-27 Thread Andres Freund
On 2017-09-27 14:58:36 -0400, Tom Lane wrote: > Andres Freund <and...@anarazel.de> writes: > > Honestly before going there I'd rather just have > > an oid indexed array, computed at compile time. > > Yeah, I'd been kind of wondering about that approach too. We

Re: [HACKERS] list of credits for release notes

2017-09-27 Thread Andres Freund
Hi, On 2017-09-27 14:47:45 -0400, Peter Eisentraut wrote: > At the PGCon Developer Meeting it was agreed[0] to add a list of credits > to the release notes, including everyone who was mentioned in a commit > message. I have now completed that list. Nice! > For the following mentions I could

Re: [HACKERS] Binary search in fmgr_isbuiltin() is a bottleneck.

2017-09-27 Thread Andres Freund
On 2017-09-27 14:40:20 -0400, Tom Lane wrote: > Andres Freund <and...@anarazel.de> writes: > > On 2017-09-27 13:46:50 -0400, Tom Lane wrote: > >> The other question that ought to be answered is whether a gperf hash > >> table would be faster. > > >

Re: [HACKERS] Binary search in fmgr_isbuiltin() is a bottleneck.

2017-09-27 Thread Andres Freund
On 2017-09-27 13:28:22 -0400, Robert Haas wrote: > On Wed, Sep 27, 2017 at 1:00 PM, Andres Freund <and...@anarazel.de> wrote: > > We could relatively easily move it to be once-per-postmaster start for > > !EXEC_BACKEND builds. Constantly doing expensive binary searches is

Re: [HACKERS] SendRowDescriptionMessage() is slow for queries with a lot of columns

2017-09-27 Thread Andres Freund
Hi, On 2017-09-13 23:34:18 -0700, Andres Freund wrote: >I'm not yet super sure about the implementation. For one, I'm not >sure this shouldn't instead be stringinfo.h functions, with very very >tiny pqformat.h wrappers. But conversely I think it'd make a lot of &

Re: [HACKERS] Binary search in fmgr_isbuiltin() is a bottleneck.

2017-09-27 Thread Andres Freund
r.c - but that seems like a good chunk more work... Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] v10 pg_ctl compatibility

2017-09-26 Thread Andres Freund
a difference here (Debian sid), I see postgres get SIGTERMed either way. GDBing in and doing a setsid() in postmaster before ctrl-c'ing pg_ctl "fixes" it. Is there a reason we're not doing so after the fork in pg_ctl? Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pg

Re: [HACKERS] v10 pg_ctl compatibility

2017-09-26 Thread Andres Freund
her shutting > down >64927 2017-09-26 13:56:47.261 PDT LOG: shutting down > ... It's reproducible here. Postmaster never calls setsid() for itself, nor does pg_ctl for it, therefore ctrl-c's SIGTERM to pg_ctl also go to its children, namely postmaster. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] v10 pg_ctl compatibility

2017-09-26 Thread Andres Freund
> actually running in the data directory pg_ctl has been pointed at. Reading PG_VERSION ought to do the trick. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] v10 pg_ctl compatibility

2017-09-26 Thread Andres Freund
something, but what's the use-case for allowing it, and every couple releases have some breakage? Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Built-in plugin for logical decoding output

2017-09-25 Thread Andres Freund
: I'm doubtful that the "not a priority" take is really fair - there's a lot of priorities, and they compete for scant resources. Which means people have to argue convincingly if they want to add to that burden - just actually asking the question whether it's a good use of resources doe

Re: [HACKERS] PATCH: pgbench - option to build using ppoll() for larger connection counts

2017-09-25 Thread Andres Freund
any need of using ppoll over poll? IIRC it's a good bit more common and there's, also iirc, a number of platforms with buggy ppoll implementations. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postg

Re: [HACKERS] Built-in plugin for logical decoding output

2017-09-25 Thread Andres Freund
On 2017-09-25 13:50:29 -0400, Tom Lane wrote: > Andres Freund <and...@anarazel.de> writes: > >> On 25/09/17 19:26, Tom Lane wrote: > >>> The problem with this type of argument is that it leads directly to the > >>> conclusion that every feature users want

Re: [HACKERS] Reading backup label file for checkpoint and redo location during crash recovery

2017-09-25 Thread Andres Freund
creating a backup label file (it's up to the backup software to store > the necessary information and create the file for use during recovery). > > Please see: > > https://www.postgresql.org/docs/10/static/continuous-archiving.html > > In particular, section 25.3.3.

Re: [HACKERS] Built-in plugin for logical decoding output

2017-09-25 Thread Andres Freund
think that should mean that there's no possible output plugin that could/should be integrated into core. And I think Petr's right here: > There is already about 3 million output plugins out there so I think we > did reasonable job there. The fact that vast majority of that are > various json ones gives rea

Re: [HACKERS] Built-in plugin for logical decoding output

2017-09-25 Thread Andres Freund
On 2017-09-25 10:38:52 -0700, Joshua D. Drake wrote: > On 09/25/2017 10:32 AM, Petr Jelinek wrote: > > On 25/09/17 19:26, Tom Lane wrote: > > > Alvaro Hernandez writes: > > > > > There is already about 3 million output plugins out there so I think we > > did reasonable job

Re: [HACKERS] Built-in plugin for logical decoding output

2017-09-25 Thread Andres Freund
it a non-starter for many purposes were you care about bandwidth. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Built-in plugin for logical decoding output

2017-09-25 Thread Andres Freund
& type names (which also slows down), but... Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] What's with all the fflush(stderr) calls in pg_standby.c?

2017-09-24 Thread Andres Freund
On 2017-09-25 10:01:35 +0900, Michael Paquier wrote: > On Mon, Sep 25, 2017 at 9:45 AM, Robert Haas wrote: > > On a related note, the idea of removing pg_standby altogether has been > > proposed a few times. Including recently by me

Re: [HACKERS] Built-in plugin for logical decoding output

2017-09-22 Thread Andres Freund
Hi, On 2017-09-22 17:11:47 -0700, Gregory Brail wrote: > Also in lieu of the new snapshot mechanism for logical replication, which > might not work for us This needs context... >, we were using the transaction ID to calculate what > was committed in a client's snapshot and what they need to

Re: [HACKERS] additional contrib test suites

2017-09-22 Thread Andres Freund
> > for SHA1 via the crypto module which seems by far the better choice. > > I'm also tempted to just remove it. It uses bad/outdated security > practices and it's also not ideal as an example module. Any objections? Uhm. I'm not objecting, but I doubt people really noticed you

Re: [HACKERS] Improve catcache/syscache performance.

2017-09-22 Thread Andres Freund
Hi, On 2017-09-20 18:26:50 +0530, amul sul wrote: > Patch 0007: Thanks for looking! > 1: > 400 + /* > 401 +* XXX: might be worthwhile to only handle oid sysattr, to > reduce > 402 +* overhead - it's the most common key. > 403 +*/ > > IMHO, let fix that as well. I

Re: [HACKERS] Improve catcache/syscache performance.

2017-09-22 Thread Andres Freund
Hi, On 2017-09-13 23:12:07 -0700, Andres Freund wrote: > Attached is a patch that tries to improve sys/catcache performance, > going further than the patch referenced earlier. Here's a variant that cleans up the previous changes a bit, and adds some further improvements: Here's the main

Re: [HACKERS] Windows warnings from VS 2017

2017-09-21 Thread Andres Freund
, and still 3-4 four times ~100 bytes. And both gcc and clang optimize way the memcpy entirely when the length is a fixed short length. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: Fw: [HACKERS] HACKERS[PATCH] split ProcArrayLock into multiple parts -- follow-up

2017-09-21 Thread Andres Freund
will probably regress as well. FWIW, you started a new thread with this message, that doesn't seem helpful? Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] POC: Cache data in GetSnapshotData()

2017-09-20 Thread Andres Freund
On September 20, 2017 7:22:00 PM PDT, Robert Haas wrote: >On Wed, Sep 20, 2017 at 10:04 PM, Mithun Cy > wrote: >> My current tests show on scylla (2 socket machine with 28 CPU core) I >> do not see any improvement at all as similar to Jesper.

Re: [HACKERS] JIT compiling expressions/deform + inlining prototype v2.0

2017-09-20 Thread Andres Freund
On 2017-09-19 12:57:33 +0300, Konstantin Knizhnik wrote: > > > On 04.09.2017 23:52, Andres Freund wrote: > > > > Hi. That piece of code isn't particularly clear (and has a bug in the > > submitted version), I'm revising it. > > ... > > Yea, I've change

Re: [HACKERS] Is it time to kill support for very old servers?

2017-09-20 Thread Andres Freund
On 2017-09-18 02:53:03 -0700, Andres Freund wrote: > On 2017-09-13 23:39:21 -0400, Tom Lane wrote: > > The real problem in this area, to my mind, is that we're not testing that > > code --- either end of it --- in any systematic way. If it's broken it > > could take us qu

Re: [HACKERS] [COMMITTERS] pgsql: Make WAL segment size configurable at initdb time.

2017-09-19 Thread Andres Freund
On 2017-09-20 14:51:45 +0900, Michael Paquier wrote: > On Wed, Sep 20, 2017 at 2:04 PM, Andres Freund <and...@anarazel.de> wrote: > > Make WAL segment size configurable at initdb time. > > > > For performance reasons a larger segment size than the default 16MB > >

Re: [HACKERS] increasing the default WAL segment size

2017-09-19 Thread Andres Freund
Hi, On 2017-09-14 11:31:33 +0530, Beena Emerson wrote: > The change looks good and is working as expected. > PFA the updated patch after running pgindent. I've pushed this version. Yay! Thanks for the work Beena, everyone! The only change I made is to run the pg_upgrade tests with a 1 MB

Re: [HACKERS] Re: [COMMITTERS] pgsql: Make new crash restart test a bit more robust.

2017-09-19 Thread Andres Freund
On 2017-09-19 19:00:38 -0700, Andres Freund wrote: > Given this fact pattern, I'll allow the case without a received error > message in the recovery test. Objections? Hearing none. Pushed. While debugging this, I've also introduced a pump wrapper so that we now get: ok 4 - exactly one p

[HACKERS] Re: [COMMITTERS] pgsql: Make new crash restart test a bit more robust.

2017-09-19 Thread Andres Freund
On 2017-09-19 18:06:29 -0700, Andres Freund wrote: > On 2017-09-19 16:46:58 -0400, Tom Lane wrote: > > Have we forgotten an fflush() or something? > > After hacking a fix for my previous theory, I started adding strace into > the mix, to verify this. Takes longer to reproduce, b

Re: [HACKERS] src/test/subscription/t/002_types.pl hanging on particular environment

2017-09-19 Thread Andres Freund
__gcov_flush(), and at that moment we are killed. Then a > > subsequent run of instrumented code will find the half-written file > > and print the "Merge mismatch" message. Note that newer gcc's (7+) have a feature to avoid such corruption, by renaming the files atomically. Possibly

Re: [HACKERS] Show backtrace when tap tests fail

2017-09-19 Thread Andres Freund
On 2017-09-19 17:15:21 -0400, Tom Lane wrote: > Andres Freund <and...@anarazel.de> writes: > > On 2017-09-19 21:37:26 +0100, Dagfinn Ilmari Mannsåker wrote: > >> Devel::Confess is more thorough, so +1 on that. > > > Or just try Devel::Confess first, and if

Re: [HACKERS] Show backtrace when tap tests fail

2017-09-19 Thread Andres Freund
On 2017-09-19 21:37:26 +0100, Dagfinn Ilmari Mannsåker wrote: > Andrew Dunstan <andrew.duns...@2ndquadrant.com> writes: > > > On 09/19/2017 01:31 PM, Andres Freund wrote: > >> Hi, > >> > >> I've had a couple cases where tap tests died, and I couldn't

Re: [HACKERS] More efficient truncation of pg_stat_activity query strings

2017-09-19 Thread Andres Freund
t! I think this looks good, no further tests necessary. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Re: [COMMITTERS] pgsql: Perform only one ReadControlFile() during startup.

2017-09-19 Thread Andres Freund
On 2017-09-19 13:15:28 -0400, Tom Lane wrote: > Andres Freund <and...@anarazel.de> writes: > > On 2017-09-19 13:00:33 -0400, Robert Haas wrote: > >> You mean, in the postmaster? > > > Yes. We try to avoid touch shmem there, but it's not like we

Re: [HACKERS] Running some tests with different segment sizes

2017-09-19 Thread Andres Freund
On 2017-09-19 14:05:44 -0400, Tom Lane wrote: > Andres Freund <and...@anarazel.de> writes: > > I'm working on merging the customizable segment size patch [1]. I'd > > like to run some of the regression tests using it, to guarantee > > non-standard settings have test co

[HACKERS] Running some tests with different segment sizes

2017-09-19 Thread Andres Freund
bloating the regression time unnecessarily. I thought about starting with just changing pg_upgrade's rerun of the standard test. Then maybe change one or two tests in src/test/recovery/? Complaints, better ideas? Andres Freund [1] http://archives.postgresql.org/message-id

[HACKERS] Show backtrace when tap tests fail

2017-09-19 Thread Andres Freund
Hi, I've had a couple cases where tap tests died, and I couldn't easily see where / why. For development of a new test I found it useful to show backtraces in that case - just adding a use Carp::Always; at the start of the relevant module did the trick. I'm wondering if we shouldn't always do so

Re: [HACKERS] Re: [COMMITTERS] pgsql: Perform only one ReadControlFile() during startup.

2017-09-19 Thread Andres Freund
On 2017-09-19 13:00:33 -0400, Robert Haas wrote: > On Tue, Sep 19, 2017 at 12:51 PM, Andres Freund <and...@anarazel.de> wrote: > > That'd not be that a crazy amount of > > shared memory that'd need to be touched in shared memory, ... > > You mean, in the postmaster?

Re: [HACKERS] Re: [COMMITTERS] pgsql: Perform only one ReadControlFile() during startup.

2017-09-19 Thread Andres Freund
On 2017-09-19 12:24:00 -0400, Tom Lane wrote: > Andres Freund <and...@anarazel.de> writes: > > Unfortunately the backends themselves also react with inaccurate error > > messages to things like immediate shutdowns... > > Yeah, those signals are kind of overl

Re: [HACKERS] Re: [COMMITTERS] pgsql: Perform only one ReadControlFile() during startup.

2017-09-19 Thread Andres Freund
d stop emitting that if we got an actual message from the server. Unfortunately the backends themselves also react with inaccurate error messages to things like immediate shutdowns... Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make change

Re: [HACKERS] pgbench tap tests & minor fixes.

2017-09-18 Thread Andres Freund
gi-bin/show_log.pl?nm=bowerbird=2017-09-19%2001%3A42%3A20 I first thought it might be the new recovery tests, or the changes leading to its addition, but it's a different test and in the middle of the run. Even so, I'd have looked towards my commit, except that there's a number of previous re

Re: [HACKERS] Statement timeout behavior in extended queries

2017-09-18 Thread Andres Freund
On 2017-09-10 17:12:19 -0700, Andres Freund wrote: > On 2017-09-11 09:10:49 +0900, Tatsuo Ishii wrote: > > If you don't mind, can you please commit/push the patch? > > Ok, will do so. And, done. Thanks for patch and reminder! - Andres -- Sent via pgsql-hackers mailing list

Re: [HACKERS] valgrind vs. shared typmod registry

2017-09-18 Thread Andres Freund
Hi, On 2017-09-18 18:04:36 +1200, Thomas Munro wrote: > On Mon, Sep 18, 2017 at 5:39 PM, Thomas Munro > wrote: > > Here is a patch to fix that. > > Here's a better one (same code, corrected commit message). Pushed. For a second I was tempted to also replace the

Re: [HACKERS] Re: [COMMITTERS] pgsql: Perform only one ReadControlFile() during startup.

2017-09-18 Thread Andres Freund
On 2017-09-18 12:16:42 -0400, Robert Haas wrote: > On Mon, Sep 18, 2017 at 6:32 AM, Andres Freund <and...@anarazel.de> wrote: > > One thing that I've noticed for a while, but that I was reminded of > > again here. We very frequently allow psql to reconnect in case of crash, &g

Re: [HACKERS] src/test/subscription/t/005_encoding.pl is broken

2017-09-18 Thread Andres Freund
pply workers have caught up, does it? To my knowledge here's not really any difference between the two in logical replication. Received changes are immediately applied, there's no equivalent to a walreceiver queing up "logical wal" onto disk. So I'm not sure that theory holds. Gre

Re: [HACKERS] Is it time to kill support for very old servers?

2017-09-18 Thread Andres Freund
On September 18, 2017 4:15:31 AM PDT, Michael Paquier <michael.paqu...@gmail.com> wrote: >On Mon, Sep 18, 2017 at 8:09 PM, Andres Freund <and...@anarazel.de> >wrote: >> >> >> On September 18, 2017 4:08:21 AM PDT, Michael Paquier ><michael.paqu...@gmai

Re: [HACKERS] Is it time to kill support for very old servers?

2017-09-18 Thread Andres Freund
On September 18, 2017 4:08:21 AM PDT, Michael Paquier <michael.paqu...@gmail.com> wrote: >On Mon, Sep 18, 2017 at 7:54 PM, Andres Freund <and...@anarazel.de> >wrote: >>>It seems to me that you are looking more for a connection parameter >>>here. >>

Re: [HACKERS] Is it time to kill support for very old servers?

2017-09-18 Thread Andres Freund
On September 18, 2017 3:50:17 AM PDT, Michael Paquier <michael.paqu...@gmail.com> wrote: >On Mon, Sep 18, 2017 at 6:53 PM, Andres Freund <and...@anarazel.de> >wrote: >> On 2017-09-13 23:39:21 -0400, Tom Lane wrote: >>> The real problem in this area, to m

Re: [HACKERS] Re: [COMMITTERS] pgsql: Perform only one ReadControlFile() during startup.

2017-09-18 Thread Andres Freund
On 2017-09-17 01:07:52 -0700, Andres Freund wrote: > On 2017-09-16 13:27:05 -0700, Andres Freund wrote: > > > This does not seem like a problem that justifies a system-wide change > > > that's much more delicate than you thought. > > > > We need one more initial

Re: [HACKERS] src/test/subscription/t/002_types.pl hanging on particular environment

2017-09-18 Thread Andres Freund
shutdown. There is no core file, so I don't believe anything actually > crashed. I was about to complain about these, for entirely unrelated reasons. I think it's a bad idea - and there's a couple complains on the lists too, to emit these warnings. It's not entirely trivial to fix though :(

Re: [HACKERS] Is it time to kill support for very old servers?

2017-09-18 Thread Andres Freund
ted the above. And the protocol coverage of the v2 protocol seems small enough that a single not too large file ought to cover most if it quite easily. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://w

[HACKERS] Reporting query on crash even if completed

2017-09-18 Thread Andres Freund
gnalling and the OOM killer probably are the biggest source for backends being killed even when there's no queries currently running. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] pg_control_recovery() return value when not in recovery

2017-09-18 Thread Andres Freund
On 2017-09-18 07:24:43 +0100, Simon Riggs wrote: > On 18 September 2017 at 05:50, Andres Freund <and...@anarazel.de> wrote: > > Hi, > > > > Just noticed that we're returning the underlying values for > > pg_control_recovery() without any checks: >

[HACKERS] pg_control_recovery() return value when not in recovery

2017-09-17 Thread Andres Freund
│ ├───┤ │ f │ └───┘ (1 row) Wouldn't it be more accurate to return NULLs here? Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Automatic testing of patches in commit fest

2017-09-17 Thread Andres Freund
Hi, On 2017-09-18 14:26:53 +1200, Thomas Munro wrote: > A couple of new experimental features on commitfest.cputube.org: Yay. > 2. It'll now dump a gdb backtrace for any core files found after a > check-world failure (if you can find your way to the build log...). > Thanks to Andres for the

Re: [HACKERS] Re: [COMMITTERS] pgsql: Perform only one ReadControlFile() during startup.

2017-09-17 Thread Andres Freund
On 2017-09-16 13:27:05 -0700, Andres Freund wrote: > > This does not seem like a problem that justifies a system-wide change > > that's much more delicate than you thought. > > We need one more initialization call during crash-restart - that doesn't > seem particularly hard a

Re: [HACKERS] Re: [COMMITTERS] pgsql: Perform only one ReadControlFile() during startup.

2017-09-16 Thread Andres Freund
On 2017-09-16 15:59:01 -0400, Tom Lane wrote: > Andres Freund <and...@anarazel.de> writes: > > On 2017-09-16 14:30:21 -0400, Tom Lane wrote: > >> I wonder whether we shouldn't just revert this patch altogether. > > > The problem is that the patch that makes the s

Re: [HACKERS] Re: [COMMITTERS] pgsql: Perform only one ReadControlFile() during startup.

2017-09-16 Thread Andres Freund
On 2017-09-16 14:30:21 -0400, Tom Lane wrote: > Andres Freund <and...@anarazel.de> writes: > > Looking into it. > > I wonder whether we shouldn't just revert this patch altogether. > Certainly, extra reads of pg_control are not a noticeable performance > problem. Th

<    1   2   3   4   5   6   7   8   9   10   >