Re: [HACKERS] logical decoding of two-phase transactions

2017-02-03 Thread Andres Freund
On 2017-02-03 17:47:50 -0500, Robert Haas wrote: > On Thu, Feb 2, 2017 at 7:14 PM, Craig Ringer wrote: > > We could make reorder buffers persistent and shared between decoding > > sessions but it'd totally change the logical decoding model and create > > some other problems. It's certainly not a t

Re: [HACKERS] Time to up bgwriter_lru_maxpages?

2017-02-02 Thread Andres Freund
On 2017-02-02 14:47:53 -0500, Robert Haas wrote: > I expect that increasing the maximum value of shared_buffers beyond > what can be stored by an integer could have a noticeable distributed > performance cost for the entire system. It might be a pretty small > cost, but then again maybe not; for e

Re: [HACKERS] Time to up bgwriter_lru_maxpages?

2017-02-02 Thread Andres Freund
On 2017-02-02 11:41:44 -0800, Jim Nasby wrote: > On 2/1/17 4:28 PM, Andres Freund wrote: > > On 2016-11-28 11:40:53 -0800, Jim Nasby wrote: > > > With current limits, the most bgwriter can do (with 8k pages) is 1000 > > > pages > > > * 100 times/sec = 780MB/

Re: [HACKERS] TRAP: FailedAssertion("!(hassrf)", File: "nodeProjectSet.c", Line: 180)

2017-02-01 Thread Andres Freund
On 2017-02-01 23:27:36 -0500, Tom Lane wrote: > I wrote: > > Andres Freund writes: > >> Tom, do you have an opinion? > > > Yes, it's broken. split_pathtarget_at_srfs seems to be doing the right > > thing, but then something later is recombining the last tw

Re: [HACKERS] Time to up bgwriter_lru_maxpages?

2017-02-01 Thread Andres Freund
On 2017-02-01 20:38:58 -0500, Robert Haas wrote: > On Wed, Feb 1, 2017 at 8:35 PM, Andres Freund wrote: > > On 2017-02-01 20:30:30 -0500, Robert Haas wrote: > >> On Wed, Feb 1, 2017 at 7:28 PM, Andres Freund wrote: > >> > On 2016-11-28 11:40:53 -0800, Jim Nasby wrot

Re: [HACKERS] Time to up bgwriter_lru_maxpages?

2017-02-01 Thread Andres Freund
On 2017-02-01 20:30:30 -0500, Robert Haas wrote: > On Wed, Feb 1, 2017 at 7:28 PM, Andres Freund wrote: > > On 2016-11-28 11:40:53 -0800, Jim Nasby wrote: > >> With current limits, the most bgwriter can do (with 8k pages) is 1000 pages > >> * 100 times/sec = 780MB/s. I

Re: [HACKERS] Time to up bgwriter_lru_maxpages?

2017-02-01 Thread Andres Freund
On 2016-11-28 11:40:53 -0800, Jim Nasby wrote: > With current limits, the most bgwriter can do (with 8k pages) is 1000 pages > * 100 times/sec = 780MB/s. It's not hard to exceed that with modern > hardware. Should we increase the limit on bgwriter_lru_maxpages? FWIW, I think working on replacing b

Re: [HACKERS] Time to up bgwriter_lru_maxpages?

2017-02-01 Thread Andres Freund
On 2017-02-02 09:22:46 +0900, Michael Paquier wrote: > On Thu, Feb 2, 2017 at 9:17 AM, Jim Nasby wrote: > > Speaking of which... I have a meeting in 15 minutes to discuss moving to a > > server with 4TB of memory. With current limits shared buffers maxes at 16TB, > > which isn't all that far in th

Re: [HACKERS] TRAP: FailedAssertion("!(hassrf)", File: "nodeProjectSet.c", Line: 180)

2017-02-01 Thread Andres Freund
this case, rather than allowing this in ProjectSet - on the other hand, it works after removing that Assert(). Tom, do you have an opinion? Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [COMMITTERS] pgsql: Make psql's \set display variables in alphabetical order.

2017-02-01 Thread Andres Freund
On 2017-02-01 12:59:36 -0500, Tom Lane wrote: > David Fetter writes: > > On Wed, Feb 01, 2017 at 04:25:25PM +, Tom Lane wrote: > >> Make psql's \set display variables in alphabetical order. > > > This is a substantial usability improvement which nevertheless is hard > > to imagine changes thi

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-01-31 Thread Andres Freund
On 2017-01-31 17:21:28 -0500, Tom Lane wrote: > Andres Freund writes: > > Hm, sorry for missing this earlier. I think CatalogUpdateIndexes() is > > fairly widely used in extensions - it seems like a pretty harsh change > > to not leave some backward compatibility layer

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-01-31 Thread Andres Freund
On 2017-01-31 19:10:05 -0300, Alvaro Herrera wrote: > Andres Freund wrote: > > On 2017-01-31 14:10:01 -0300, Alvaro Herrera wrote: > > > > Hmm, I was thinking we would get rid of CatalogUpdateIndexes altogether. > > > Two of the callers are in the new routines

Re: [HACKERS] Patch: Write Amplification Reduction Method (WARM)

2017-01-31 Thread Andres Freund
On 2017-01-31 14:10:01 -0300, Alvaro Herrera wrote: > Pavan Deolasee wrote: > > > Two new APIs added. > > > > - CatalogInsertHeapAndIndex which does a simple_heap_insert followed by > > catalog updates > > - CatalogUpdateHeapAndIndex which does a simple_heap_update followed by > > catalog updates

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2017-01-30 Thread Andres Freund
Hi Heikki, On 2016-11-22 10:56:07 -0800, Peter Geoghegan wrote: > I attach V4 of amcheck. Is there any chance you could look at the concurrency related parts of amcheck? I'd like to push this to be mergeable, but that area is a bit outside of my area of expertise... Andres -- Sent via pgsql-

Re: [HACKERS] Query fails when SRFs are part of FROM clause (Commit id: 69f4b9c85f)

2017-01-30 Thread Andres Freund
On 2017-01-30 17:24:31 -0500, Tom Lane wrote: > Make it work like Agg and WindowFunc. To wit, dump the actually special > function calls (the set-returning functions) into a list that's internal > to the FunctionScan node, and then anything above those goes into scalar > expressions in the node's

Re: [HACKERS] Query fails when SRFs are part of FROM clause (Commit id: 69f4b9c85f)

2017-01-30 Thread Andres Freund
Hi, On 2017-01-30 16:55:56 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2017-01-27 17:58:04 +0530, Rushabh Lathia wrote: > >> SELECT * > >> FROM pg_constraint pc, > >> CAST(CASE WHEN pc.contype IN ('f','u','p') THEN g

Re: [HACKERS] Query fails when SRFs are part of FROM clause (Commit id: 69f4b9c85f)

2017-01-30 Thread Andres Freund
Hi, On 2017-01-27 17:58:04 +0530, Rushabh Lathia wrote: > Consider the below test; > > CREATE TABLE tab ( a int primary key); > > SELECT * > FROM pg_constraint pc, > CAST(CASE WHEN pc.contype IN ('f','u','p') THEN generate_series(1, > array_upper(pc.conkey, 1)) ELSE NULL END AS int) AS position

Re: [HACKERS] WIP: About CMake v2

2017-01-29 Thread Andres Freund
On 2017-01-27 22:20:41 -0500, Peter Eisentraut wrote: > On 1/27/17 6:11 PM, Andres Freund wrote: > > On 2017-01-27 09:09:36 -0500, Peter Eisentraut wrote: > >> My preferred scenario would be to replace the Windows build system by > >> this first, then refine i

Re: [HACKERS] [PATCH] Rename pg_switch_xlog to pg_switch_wal

2017-01-29 Thread Andres Freund
On 2017-01-28 08:47:03 +0900, Michael Paquier wrote: > On Sat, Jan 28, 2017 at 8:03 AM, Peter Eisentraut > wrote: > > On 1/26/17 2:05 PM, Robert Haas wrote: > >> I do not think it can be right to rename the directory and not > >> anything else. > > > > I think this is the root of the confusion. >

Re: [HACKERS] Create a separate test file for exercising system views

2017-01-29 Thread Andres Freund
ently have. Alternatively casting the entire row to text should allow to use count(*) trickery in some of the cases at least. > Comments/objections? Sounds like a good idea here. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) T

Re: [HACKERS] [PATCH] Rename pg_switch_xlog to pg_switch_wal

2017-01-27 Thread Andres Freund
On 2017-01-27 18:06:11 -0500, Peter Eisentraut wrote: > On 1/24/17 4:47 PM, Robert Haas wrote: > > I'm not excited about starting to change pg_clog before we finish with > > xlog -> wal. Then we just have two half-done things, IMO. But I'm > > also not the only one with a commit bit. > > I think

Re: [HACKERS] WIP: About CMake v2

2017-01-27 Thread Andres Freund
On 2017-01-27 09:09:36 -0500, Peter Eisentraut wrote: > My preferred scenario would be to replace the Windows build system by > this first, then refine it, then get rid of Autoconf. > > The ideal timeline would be to have a ready patch to commit early in a > development cycle, then get rid of the

Re: [HACKERS] [GENERAL] C++ port of Postgres

2017-01-26 Thread Andres Freund
Hi Peter, On 2016-09-30 15:24:09 -0400, Peter Eisentraut wrote: > Yeah, I have committed a few of the patches now and I'll close the CF > entry now. Thanks for your research. Are you planning to push more of these at some point? - Andres -- Sent via pgsql-hackers mailing list (pgsql-hackers@

Re: [HACKERS] safer node casting

2017-01-26 Thread Andres Freund
On 2017-01-26 20:29:06 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2016-12-31 12:08:22 -0500, Peter Eisentraut wrote: > >> This is inspired by the dynamic_cast operator in C++, but follows the > >> syntax of the well-known makeNode() macro. > > > T

Re: [HACKERS] safer node casting

2017-01-26 Thread Andres Freund
Hi, On 2016-12-31 12:08:22 -0500, Peter Eisentraut wrote: > This is inspired by the dynamic_cast operator in C++, but follows the > syntax of the well-known makeNode() macro. The analogy to dynamic_cast goes only so far, because we don't actually support inheritance. I.e. in c++ we could success

Re: [HACKERS] safer node casting

2017-01-26 Thread Andres Freund
On 2017-01-26 17:27:45 -0500, Tom Lane wrote: > Andres Freund writes: > > #if defined(USE_ASSERT_CHECKING) && defined(PG_USE_INLINE) > > is probably a better gatekeeper in the back-branches, than gcc? > > Ah, yeah, that would work --- I'd already swapped ou

Re: [HACKERS] [PATCH] Rename pg_switch_xlog to pg_switch_wal

2017-01-26 Thread Andres Freund
On 2017-01-26 19:01:54 -0500, Stephen Frost wrote: > Andres, > > * Andres Freund (and...@anarazel.de) wrote: > > I hear these complaints about postgres most frequently: 1) replication > > sucks. 2) way too slow on analytics queries. 3) existing admin tools > > suck.

Re: [HACKERS] [PATCH] Rename pg_switch_xlog to pg_switch_wal

2017-01-26 Thread Andres Freund
On 2017-01-26 15:45:15 -0700, David G. Johnston wrote: > On Thu, Jan 26, 2017 at 3:28 PM, Andres Freund wrote: > > > I *personally* don't think it's worth > > changing all this without taking more care about backward compat than > > we're apparently w

Re: [HACKERS] safer node casting

2017-01-26 Thread Andres Freund
On 2017-01-26 16:55:33 -0500, Tom Lane wrote: > Andres Freund writes: > > How about something like the attached? The first patch just contains > > castNode(), the second one a rebased version of Peter's changes (with > > some long lines broken up). > > Looks ge

Re: [HACKERS] [PATCH] Rename pg_switch_xlog to pg_switch_wal

2017-01-26 Thread Andres Freund
On 2017-01-26 16:55:37 -0500, Robert Haas wrote: > On Thu, Jan 26, 2017 at 2:37 PM, Andres Freund wrote: > > On 2017-01-26 14:28:01 -0500, Robert Haas wrote: > >> On Thu, Jan 26, 2017 at 2:24 PM, Andres Freund wrote: > >> >> Whether the voters recognized that

Re: [HACKERS] safer node casting

2017-01-26 Thread Andres Freund
On 2017-01-25 19:21:40 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2016-12-31 12:08:22 -0500, Peter Eisentraut wrote: > >> RestrictInfo *rinfo = castNode(RestrictInfo, lfirst(lc)); > > > Are you planning to add this / update this patch? Because I really

Re: [HACKERS] Allow interrupts on waiting standby

2017-01-26 Thread Andres Freund
On 2017-01-26 19:36:11 +, Simon Riggs wrote: > On 26 January 2017 at 19:20, Andres Freund wrote: > > On 2017-01-26 12:24:44 -0500, Robert Haas wrote: > >> On Thu, Jan 26, 2017 at 7:18 AM, Simon Riggs wrote: > >> > Currently a waiting standby doesn't al

Re: [HACKERS] [PATCH] Rename pg_switch_xlog to pg_switch_wal

2017-01-26 Thread Andres Freund
On 2017-01-26 14:28:01 -0500, Robert Haas wrote: > On Thu, Jan 26, 2017 at 2:24 PM, Andres Freund wrote: > >> Whether the voters recognized that fact at the time I would have to concur > >> that if we are going to change from xlog to wal we should be all-in. If > >&

Re: [HACKERS] [PATCH] Rename pg_switch_xlog to pg_switch_wal

2017-01-26 Thread Andres Freund
On 2017-01-26 12:23:24 -0700, David G. Johnston wrote: > On Thu, Jan 26, 2017 at 12:13 PM, Andres Freund wrote: > > > On 2017-01-26 14:05:43 -0500, Robert Haas wrote: > > > I completely understand that position. I have always been doubtful of > > > the value of r

Re: [HACKERS] Allow interrupts on waiting standby

2017-01-26 Thread Andres Freund
On 2017-01-26 12:24:44 -0500, Robert Haas wrote: > On Thu, Jan 26, 2017 at 7:18 AM, Simon Riggs wrote: > > Currently a waiting standby doesn't allow interrupts. > > > > Patch implements that. > > > > Barring objection, patching today with backpatches. > > "today" is a little quick, but the patch

Re: [HACKERS] increasing the default WAL segment size

2017-01-26 Thread Andres Freund
values[0] = > PointerGetDatum(cstring_to_text(NameStr(MyReplicationSlot->data.name))); That seems a bit long. I've not done like the most careful review ever, but I'm in favor of the general change (provided the byval thing is fixed obviously). Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [PATCH] Rename pg_switch_xlog to pg_switch_wal

2017-01-26 Thread Andres Freund
On 2017-01-26 14:05:43 -0500, Robert Haas wrote: > I completely understand that position. I have always been doubtful of > the value of renaming pg_xlog to pg_wal, and I'm not any more > dedicated to the idea now than I was when I committed that patch. But > there was overwhelming support for it,

Re: [HACKERS] increasing the default WAL segment size

2017-01-26 Thread Andres Freund
38,6 +238,59 @@ GetConnection(void) > } > > /* > + * Run the SHOW_WAL_SEGMENT_SIZE command to set the XLogSegSize > + */ > +bool > +SetXLogSegSize(PGconn *conn) > +{ I think this is a confusing function name, because it sounds like you're setting the SegSize remotely or such. I think making it XLogRecPtr RetrieveXLogSegSize(conn); or such would lead to better code. > diff --git a/src/bin/pg_resetxlog/pg_resetxlog.c > b/src/bin/pg_resetxlog/pg_resetxlog.c > index 963802e..4ceebdc 100644 > --- a/src/bin/pg_resetxlog/pg_resetxlog.c > +++ b/src/bin/pg_resetxlog/pg_resetxlog.c > @@ -57,6 +57,7 @@ > #include "storage/large_object.h" > #include "pg_getopt.h" > > +uint32 XLogSegSize; This seems like a bad idea - having the same local variable both in frontend and backend programs seems like a recipe for disaster. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [COMMITTERS] pgsql: Check interrupts during hot standby waits

2017-01-26 Thread Andres Freund
Hi, On 2017-01-26 19:00:34 +, Simon Riggs wrote: > Check interrupts during hot standby waits > > Branch > -- > master > > Details > --- > http://git.postgresql.org/pg/commitdiff/e8ee3d6b859a18d7f7375ceb9e04d256eb18aaec > > Modified Files > -- > src/backend/storage/ipc/stand

Re: [HACKERS] increasing the default WAL segment size

2017-01-26 Thread Andres Freund
On 2017-01-26 13:16:13 -0500, Robert Haas wrote: > > OK, I have done this refactoring effort as attached because I think > > that's really worth it. And here are the diff numbers: > > 3 files changed, 113 insertions(+), 162 deletions(-) > > That's a bit less than what I thought first because of al

Re: [HACKERS] [PATCH] Rename pg_switch_xlog to pg_switch_wal

2017-01-26 Thread Andres Freund
On 2017-01-24 16:47:29 -0500, Robert Haas wrote: > I'm happy to go change every last bit of it. I quite regret not aggressively opining against the renaming of pg_xlog to pg_wal. I think the few users deleting their data don't weigh against renaming a bunch of tools and function for some sense of

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Andres Freund
is online and working > properly during an online backup. I don't think that really changes anything. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Checksums by default?

2017-01-25 Thread Andres Freund
On 2017-01-26 09:19:28 +0900, Michael Paquier wrote: > On Thu, Jan 26, 2017 at 9:14 AM, Peter Geoghegan wrote: > > On Wed, Jan 25, 2017 at 3:30 PM, Stephen Frost wrote: > >> As it is, there are backup solutions which *do* check the checksum when > >> backing up PG. This is no longer, thankfully,

Re: [HACKERS] safer node casting

2017-01-25 Thread Andres Freund
(to avoid multiple eval scenarios) if helpful Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] pg_ls_dir & friends still have a hard-coded superuser check

2017-01-25 Thread Andres Freund
On 2017-01-25 18:04:09 -0500, Stephen Frost wrote: > Andres, > > * Andres Freund (and...@anarazel.de) wrote: > > On 2017-01-25 16:52:38 -0500, Stephen Frost wrote: > > > * Robert Haas (robertmh...@gmail.com) wrote: > > > > Preventing people from calling

Re: [HACKERS] pg_ls_dir & friends still have a hard-coded superuser check

2017-01-25 Thread Andres Freund
Hi, On 2017-01-25 16:52:38 -0500, Stephen Frost wrote: > * Robert Haas (robertmh...@gmail.com) wrote: > > Preventing people from calling functions by denying the ability to > > meaningfully GRANT EXECUTE on those functions doesn't actually stop > > them from delegating those functions to non-super

Re: [HACKERS] patch: function xmltable

2017-01-25 Thread Andres Freund
On 2017-01-25 22:51:37 +0100, Pavel Stehule wrote: > 2017-01-25 22:40 GMT+01:00 Andres Freund : > > > I afraid when I cannot to reuse a SRF infrastructure, I have to > > reimplement > > > it partially :( - mainly for usage in "ROWS FROM ()" > > > >

Re: [HACKERS] patch: function xmltable

2017-01-25 Thread Andres Freund
tSRFs for backward compatibility reasons, but that's not > > true for XMLTABLE > > > > > > > ok > > I afraid when I cannot to reuse a SRF infrastructure, I have to reimplement > it partially :( - mainly for usage in "ROWS FROM ()" Huh? Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] patch: function xmltable

2017-01-25 Thread Andres Freund
Hi, On 2017-01-25 05:45:24 +0100, Pavel Stehule wrote: > 2017-01-25 1:35 GMT+01:00 Andres Freund : > > > On 2017-01-24 21:32:56 -0300, Alvaro Herrera wrote: > > > Andres Freund wrote: > > > > Hi, > > > > > > > > On 2017-01-24 17:3

Re: [HACKERS] lseek/read/write overhead becomes visible at scale ..

2017-01-25 Thread Andres Freund
Hi, On 2017-01-25 10:16:32 +0100, Tobias Oberstein wrote: > > > Using pread instead of lseek+read halfes the syscalls. > > > > > > I really don't understand what you are fighting here .. > > > > Sure, there's some overhead. And as I said upthread, I'm much less > > against this change than Tom.

Re: [HACKERS] patch: function xmltable

2017-01-24 Thread Andres Freund
On 2017-01-24 21:32:56 -0300, Alvaro Herrera wrote: > Andres Freund wrote: > > Hi, > > > > On 2017-01-24 17:38:49 -0300, Alvaro Herrera wrote: > > > +static Datum ExecEvalTableExpr(TableExprState *tstate, ExprContext > > > *econtext, > > > +

Re: [HACKERS] patch: function xmltable

2017-01-24 Thread Andres Freund
Hi, On 2017-01-24 17:38:49 -0300, Alvaro Herrera wrote: > +static Datum ExecEvalTableExpr(TableExprState *tstate, ExprContext *econtext, > + bool *isnull); > +static Datum ExecEvalTableExprFast(TableExprState *exprstate, ExprContext > *econtext, > +

Re: [HACKERS] lseek/read/write overhead becomes visible at scale ..

2017-01-24 Thread Andres Freund
emonstrate a benefit that's clearly > above the noise level." > > I have done lots of benchmarking over the last days on a massive box, and I > can provide numbers that I think show that the impact can be significant. since you've not actually shown that the impact is abov

Re: [HACKERS] lseek/read/write overhead becomes visible at scale ..

2017-01-24 Thread Andres Freund
On 2017-01-24 15:36:13 -0300, Alvaro Herrera wrote: > Tobias Oberstein wrote: > > > I am benchmarking IOPS, and while doing so, it becomes apparent that at > > these scales it does matter _how_ IO is done. > > > > The most efficient way is libaio. I get 9.7 million/sec IOPS with low CPU > > load.

Re: [HACKERS] lseek/read/write overhead becomes visible at scale ..

2017-01-24 Thread Andres Freund
Hi, On 2017-01-24 18:57:47 +0100, Tobias Oberstein wrote: > Am 24.01.2017 um 18:41 schrieb Andres Freund: > > On 2017-01-24 18:37:14 +0100, Tobias Oberstein wrote: > > > The syscall overhead is visible in production too .. I watched PG using > > > perf > > > l

Re: [HACKERS] lseek/read/write overhead becomes visible at scale ..

2017-01-24 Thread Andres Freund
t - I'm working quite heavily on > > speeding up analytics workloads for pg. > > Here: > > https://github.com/oberstet/scratchbox/raw/master/cruncher/adr_stats/ADR-PostgreSQL-READ-Statistics.pdf > > https://github.com/oberstet/scratchbox/tree/master/cruncher/adr_stats

Re: [HACKERS] lseek/read/write overhead becomes visible at scale ..

2017-01-24 Thread Andres Freund
only 48 cores > and no HT). I'd welcome seeing profiles of that - I'm working quite heavily on speeding up analytics workloads for pg. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WIP: About CMake v2

2017-01-23 Thread Andres Freund
; are beginning to smoke as well after everything that has happened for > the logical replication changes. I'm doubtful about this being ready in time too, but it seems a might heavyhanded to make that call on your own. Including the judgement about Peter's capability to handle more. G

Re: [HACKERS] Checksums by default?

2017-01-23 Thread Andres Freund
On 2017-01-23 21:11:37 -0600, Merlin Moncure wrote: > End user data damage ought to prevented at all costs IMO. Really, really, really not. We should do a lot, but if that'd be the only priority we'd enable all the expensive as shit stuff and be so slow that there'd be no users. We'd never add ne

Re: [HACKERS] Checksums by default?

2017-01-23 Thread Andres Freund
hnically wrong arguments in this thread. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Checksums by default?

2017-01-21 Thread Andres Freund
On 2017-01-21 13:04:18 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2017-01-21 12:46:05 -0500, Stephen Frost wrote: > >> Do you run with all defaults in those environments? > > > Irrelevant - changing requires re-initdb'ing. That's unrealistic.

Re: [HACKERS] Checksums by default?

2017-01-21 Thread Andres Freund
On 2017-01-21 13:03:52 -0500, Stephen Frost wrote: > * Andres Freund (and...@anarazel.de) wrote: > > On 2017-01-21 12:46:05 -0500, Stephen Frost wrote: > > > Do you run with all defaults in those environments? > > > > Irrelevant - changing requires re-initdb&#

Re: [HACKERS] Checksums by default?

2017-01-21 Thread Andres Freund
; > more about amount of WAL produced because in 90%+ environments that I > > work with any increase in WAL amount means at least double the increase > > in network bandwidth due to replication. > > Do you run with all defaults in those environments? Irrelevant - changing requ

Re: [HACKERS] Checksums by default?

2017-01-21 Thread Andres Freund
On 2017-01-21 12:09:53 -0500, Tom Lane wrote: > Also, if we do decide to do that, there's the question of timing. > As I mentioned, one of the chief risks I see is the possibility of > false-positive checksum failures due to bugs; I think that code has seen > sufficiently little field use that we s

Re: [HACKERS] Checksums by default?

2017-01-21 Thread Andres Freund
On 2017-01-22 00:41:55 +0900, Fujii Masao wrote: > On Sun, Jan 22, 2017 at 12:18 AM, Petr Jelinek > wrote: > > On 21/01/17 11:39, Magnus Hagander wrote: > >> Is it time to enable checksums by default, and give initdb a switch to > >> turn it off instead? > > > > I'd like to see benchmark first, bo

Re: [HACKERS] Checksums by default?

2017-01-21 Thread Andres Freund
On 2017-01-21 11:39:18 +0100, Magnus Hagander wrote: > Is it time to enable checksums by default, and give initdb a switch to turn > it off instead? -1 - the WAL overhead is quite massive, and in contrast to the other GUCs recently changed you can't just switch this around. Andres -- Sent via

Re: [HACKERS] Declarative partitioning - another take

2017-01-19 Thread Andres Freund
On 2017-01-19 14:18:23 -0500, Robert Haas wrote: > Committed. One of the patches around this topic committed recently seems to cause valgrind failures like https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=skink&dt=2017-01-19%2008%3A40%3A02 : ==24969== Conditional jump or move depends on uni

[HACKERS] Failure on sittella

2017-01-19 Thread Andres Freund
I'm not quite seeing the connection. Any chance you could get a backtrace for the failure? Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Re: Clarifying "server starting" messaging in pg_ctl start without --wait

2017-01-19 Thread Andres Freund
On 2017-01-19 20:45:57 -0500, Stephen Frost wrote: > * Andres Freund (and...@anarazel.de) wrote: > > On 2017-01-19 10:06:09 -0500, Stephen Frost wrote: > > > WAL replay does do more work, generally speaking (the WAL has to be > > > read, the checksum validated on it, a

Re: [HACKERS] Re: Clarifying "server starting" messaging in pg_ctl start without --wait

2017-01-19 Thread Andres Freund
of IO in parallel, you can easily run into such issues. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Implement targetlist SRFs using ROWS FROM() (was Changed SRF in targetlist handling)

2017-01-19 Thread Andres Freund
On 2017-01-19 13:06:20 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2017-01-18 16:56:46 -0500, Tom Lane wrote: > >> Andres Freund writes: > >> I have not actually looked at 0003 at all yet. So yeah, please post > >> for review after you're

Re: [HACKERS] Implement targetlist SRFs using ROWS FROM() (was Changed SRF in targetlist handling)

2017-01-18 Thread Andres Freund
On January 18, 2017 3:59:00 PM PST, Robert Haas wrote: >On Wed, Jan 18, 2017 at 6:19 PM, Andres Freund >wrote: >>> SELECT x, CASE WHEN x > 0 THEN generate_series(1, 5) ELSE 0 END >FROM tab; >>> >>> It might seem that this should produce five repetitio

Re: [HACKERS] Implement targetlist SRFs using ROWS FROM() (was Changed SRF in targetlist handling)

2017-01-18 Thread Andres Freund
On 2017-01-18 16:27:53 -0700, David G. Johnston wrote: > ​I'd rather fail now and allow for the possibility of future implementation > of the "it might seem that..." behavior.​ That's very unlikely to happen. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes

Re: [HACKERS] Implement targetlist SRFs using ROWS FROM() (was Changed SRF in targetlist handling)

2017-01-18 Thread Andres Freund
st > are always evaluated as though they are on the inside of a nestloop join > with the rest of the FROM clause, so that the function(s) are run to > completion before the next row from the FROM clause is considered. > > So is this too ugly to live, or shall we put up with it?

Re: [HACKERS] Implement targetlist SRFs using ROWS FROM() (was Changed SRF in targetlist handling)

2017-01-18 Thread Andres Freund
On 2017-01-18 17:34:56 -0500, Tom Lane wrote: > Andres Freund writes: > > (I also noticed the previous patch should have had a catversion bump :(, > > will do after the meeting). > > Uh, why? It isn't touching any on-disk data structure. Forget what I said - I was ru

Re: [HACKERS] Implement targetlist SRFs using ROWS FROM() (was Changed SRF in targetlist handling)

2017-01-18 Thread Andres Freund
On 2017-01-18 16:56:46 -0500, Tom Lane wrote: > Andres Freund writes: > I have not actually looked at 0003 at all yet. So yeah, please post > for review after you're done rebasing. Here's a rebased and lightly massaged version. I'm vanishing in a meeting for a bit, tho

Re: [HACKERS] Implement targetlist SRFs using ROWS FROM() (was Changed SRF in targetlist handling)

2017-01-18 Thread Andres Freund
Hi, On 2017-01-18 15:24:32 -0500, Tom Lane wrote: > Andres Freund writes: > > Yea, have something lying around. Let me push it then when I get back from > > lunch? > > Sure, no sweat. Pushed. Yay! There's one sgml comment you'd added: "Furthermore, ne

Re: [HACKERS] Implement targetlist SRFs using ROWS FROM() (was Changed SRF in targetlist handling)

2017-01-18 Thread Andres Freund
Hi, On January 18, 2017 12:00:12 PM PST, Tom Lane wrote: >Andres Freund writes: >> On 2017-01-18 14:24:12 -0500, Tom Lane wrote: >>> So I think we can push this patch now and get on with the downstream >>> patches. Do you want to do the honors, or shall I? > >&

Re: [HACKERS] Implement targetlist SRFs using ROWS FROM() (was Changed SRF in targetlist handling)

2017-01-18 Thread Andres Freund
Hi, On 2017-01-18 14:24:12 -0500, Tom Lane wrote: > Andres Freund writes: > > On 2017-01-18 08:43:24 -0500, Tom Lane wrote: > >> ... except for one thing. The more I look at it, > >> the more disturbed I am by the behavioral change shown in rangefuncs.out > >&

Re: [HACKERS] Implement targetlist SRFs using ROWS FROM() (was Changed SRF in targetlist handling)

2017-01-18 Thread Andres Freund
Hi, On 2017-01-18 08:43:24 -0500, Tom Lane wrote: > I did a review pass over 0001 and 0002. I think the attached updated > version is committable Cool. > ... except for one thing. The more I look at it, > the more disturbed I am by the behavioral change shown in rangefuncs.out > --- that's the

Re: [HACKERS] Implement targetlist SRFs using ROWS FROM() (was Changed SRF in targetlist handling)

2017-01-17 Thread Andres Freund
On 2017-01-17 13:43:38 -0500, Tom Lane wrote: > Although ... looking closer at Andres' patch, the new node type *is* > channeling Result, in the sense that it might or might not have any input > plan. This probably traces to what I wrote in September: > > + * XXX Possibly-temporary hack: if th

Re: [HACKERS] Implement targetlist SRFs using ROWS FROM() (was Changed SRF in targetlist handling)

2017-01-17 Thread Andres Freund
Hi, On 2017-01-17 12:52:20 -0500, Tom Lane wrote: > Robert Haas writes: > > On Mon, Jan 16, 2017 at 2:13 PM, Tom Lane wrote: > >> "Srf" is ugly as can be, and unintelligible. SetResult might be OK. > > > The operation we're performing here, IIUC, is projection. SetResult > > lacks a verb, alth

Re: [HACKERS] check_srf_call_placement() isn't always setting p_hasTargetSRFs

2017-01-16 Thread Andres Freund
On 2017-01-16 14:34:58 -0500, Tom Lane wrote: > Will go fix these things. Thanks! Andres -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Implement targetlist SRFs using ROWS FROM() (was Changed SRF in targetlist handling)

2017-01-16 Thread Andres Freund
On 2017-01-16 16:04:34 -0300, Alvaro Herrera wrote: > Andres Freund wrote: > > On 2017-01-16 12:17:46 -0300, Alvaro Herrera wrote: > > > Hmm. I wonder if your stuff could be used as support code for > > > XMLTABLE[1]. > > > > I don't immediately see w

Re: [HACKERS] Implement targetlist SRFs using ROWS FROM() (was Changed SRF in targetlist handling)

2017-01-16 Thread Andres Freund
Hi, On 2017-01-16 14:13:18 -0500, Tom Lane wrote: > Andres Freund writes: > > That worked quite well. So we have a few questions, before I clean this > > up: > > > - For now the node is named 'Srf' both internally and in explain - not > > sure if we w

Re: [HACKERS] Implement targetlist SRFs using ROWS FROM() (was Changed SRF in targetlist handling)

2017-01-16 Thread Andres Freund
On 2017-01-16 12:17:46 -0300, Alvaro Herrera wrote: > Andres Freund wrote: > > > That worked quite well. So we have a few questions, before I clean this > > up: > > > > - For now the node is named 'Srf' both internally and in explain - not > >

Re: [HACKERS] Implement targetlist SRFs using ROWS FROM() (was Changed SRF in targetlist handling)

2017-01-16 Thread Andres Freund
On 2017-01-15 19:29:52 -0800, Andres Freund wrote: > On 2016-10-31 09:06:39 -0700, Andres Freund wrote: > > On 2016-09-14 19:28:25 -0400, Tom Lane wrote: > > > Andres Freund writes: > > > > On 2016-09-12 19:35:22 -0400, Tom Lane wrote: > > > Here's a d

[HACKERS] check_srf_call_placement() isn't always setting p_hasTargetSRFs

2017-01-16 Thread Andres Freund
tless right now. But we'd get better errormessage if we made check_srf_call_placement() error out. I wonder if there should be a seperate expression type for the INSERT ... VALUES(exactly-one-row); since that behaves quite differently. Greetings, Andres Freund [1] http://archives.postgresql.o

Re: [HACKERS] Implement targetlist SRFs using ROWS FROM() (was Changed SRF in targetlist handling)

2017-01-15 Thread Andres Freund
On 2016-10-31 09:06:39 -0700, Andres Freund wrote: > On 2016-09-14 19:28:25 -0400, Tom Lane wrote: > > Andres Freund writes: > > > On 2016-09-12 19:35:22 -0400, Tom Lane wrote: > > Here's a draft patch that is just meant to investigate what the planner > > cha

Re: [HACKERS] Protect syscache from bloating with negative cache entries

2017-01-13 Thread Andres Freund
Hi, On 2017-01-13 17:58:41 -0500, Tom Lane wrote: > But, again, the catcache isn't the only source of per-process bloat > and I'm not even sure it's the main one. A more holistic approach > might be called for. It'd be helpful if we'd find a way to make it easy to get statistics about the size

Re: [HACKERS] [PATCH] Rename pg_switch_xlog to pg_switch_wal

2017-01-12 Thread Andres Freund
On January 12, 2017 10:50:18 AM PST, Stephen Frost wrote: >* Andres Freund (and...@anarazel.de) wrote: >> On 2017-01-12 13:40:50 -0500, Stephen Frost wrote: >> > * Jim Nasby (jim.na...@bluetreble.com) wrote: >> > > The way I see it, either one person

Re: [HACKERS] [PATCH] Rename pg_switch_xlog to pg_switch_wal

2017-01-12 Thread Andres Freund
e a bit to adjust to with the new release, and that isn't news > to anyone who works with PG. By that argument we can just do arbitrary backward incompat changes. We should aspire to be better than we've been in the past, not use that past as an excuse for not even trying. Gree

Re: [HACKERS] Performance degradation in Bitmapscan (commit 75ae538bc3168bf44475240d4e0487ee2f3bb376)

2017-01-06 Thread Andres Freund
On 2017-01-06 11:01:32 -0500, Robert Haas wrote: > On Fri, Jan 6, 2017 at 10:43 AM, Andres Freund wrote: > > On 2016-12-16 09:34:31 -0800, Andres Freund wrote: > >> > To fix his issue, we need something like your 0001. Are you going to > >> > polish that up soon

Re: [HACKERS] pg_stat_activity.waiting_start

2017-01-06 Thread Andres Freund
On 2017-01-06 10:43:32 -0500, Bruce Momjian wrote: > On Thu, Jan 5, 2017 at 06:48:17PM -1000, Joel Jacobson wrote: > > On Thu, Jan 5, 2017 at 4:59 PM, Bruce Momjian wrote: > > > Agreed. No need in adding overhead for short-lived locks because the > > > milli-second values are going to be meaning

Re: [HACKERS] Performance degradation in Bitmapscan (commit 75ae538bc3168bf44475240d4e0487ee2f3bb376)

2017-01-06 Thread Andres Freund
On 2016-12-16 09:34:31 -0800, Andres Freund wrote: > > To fix his issue, we need something like your 0001. Are you going to > > polish that up soon here? > > Yes. I've two versions of a fix for this. One of them basically increases the "spread" of buckets when

Re: [HACKERS] Reporting planning time with EXPLAIN

2017-01-05 Thread Andres Freund
On 2016-12-28 10:29:48 -0500, Tom Lane wrote: > How about just saying that the existing TIMING option turns this on, > if it's specified without ANALYZE? Right now that combination draws > an error: > > regression=# explain (timing on) select 1; > ERROR: EXPLAIN option TIMING requires

Re: [HACKERS] [PATCH] Rename pg_switch_xlog to pg_switch_wal

2017-01-05 Thread Andres Freund
On 2017-01-04 09:38:42 -0500, Stephen Frost wrote: > Andres, > > * Andres Freund (and...@anarazel.de) wrote: > > On 2017-01-03 10:37:08 -0500, Stephen Frost wrote: > > > * Vladimir Rusinov (vrusi...@google.com) wrote: > > > > I think I +1 on this. >

Re: [HACKERS] [PATCH] guc-ify the formerly hard-coded MAX_SEND_SIZE to max_wal_send

2017-01-05 Thread Andres Freund
7;t the OS network stack take care of this in both cases? I mean either is too big for TCP packets (including jumbo frames). What type of OS and network is involved here? Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subsc

Re: [HACKERS] Replication/backup defaults

2017-01-05 Thread Andres Freund
On 2017-01-05 09:12:49 -0800, Andres Freund wrote: > On 2017-01-05 18:08:36 +0100, Magnus Hagander wrote: > > On Thu, Jan 5, 2017 at 6:01 PM, Andres Freund wrote: > > > > > On 2017-01-05 08:38:32 -0500, Peter Eisentraut wrote: > > > > I also suggest making th

Re: [HACKERS] Replication/backup defaults

2017-01-05 Thread Andres Freund
On 2017-01-05 18:08:36 +0100, Magnus Hagander wrote: > On Thu, Jan 5, 2017 at 6:01 PM, Andres Freund wrote: > > > On 2017-01-05 08:38:32 -0500, Peter Eisentraut wrote: > > > I also suggest making the defaults for both 20 instead of 10. That > > > leaves enough ro

<    8   9   10   11   12   13   14   15   16   17   >