Re: [HACKERS] Replication/backup defaults

2017-01-05 Thread Andres Freund
On 2017-01-05 08:38:32 -0500, Peter Eisentraut wrote: > I also suggest making the defaults for both 20 instead of 10. That > leaves enough room that almost nobody ever has to change them, whereas > 10 can be a bit tight for some not-outrageous installations (8 standbys > plus backup?). I'm

Re: [HACKERS] safer node casting

2017-01-04 Thread Andres Freund
Hi, On 2017-01-03 11:00:47 +0530, Ashutosh Bapat wrote: > On Mon, Jan 2, 2017 at 2:10 PM, Andres Freund <and...@anarazel.de> wrote: > > Hi, > > > > > > On 2016-12-31 12:08:22 -0500, Peter Eisentraut wrote: > >> There is a common coding pattern that

Re: [HACKERS] [PATCH] Rename pg_switch_xlog to pg_switch_wal

2017-01-04 Thread Andres Freund
eems like a case for that, as imo is unifying configuration), but I don't see how that justifying breaking things gratuitously. Just renaming well known functions for a minor bit of cleanliness seems not to survive a cost/benefit analysis. Greetings, Andres Freund -- Sent via pgsql-hackers maili

Re: [HACKERS] [COMMITTERS] pgsql: Update copyright for 2017

2017-01-04 Thread Andres Freund
On 2017-01-03 13:02:28 -0500, Bruce Momjian wrote: > Yeah, I was doing parallel pulls of different branches in git via shell > script, and it seems the size of this commit showed me that doesn't > work. Sorry. Shouldn't you check the results of something like this before pushing? Sorry for

Re: [HACKERS] Replication/backup defaults

2017-01-02 Thread Andres Freund
On 2017-01-02 10:31:28 +, Simon Riggs wrote: > We must listen to feedback, not just try to blast through it. Not agreeing with your priorities isn't "blasting through feedback". -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

Re: [HACKERS] Replication/backup defaults

2017-01-02 Thread Andres Freund
t to support that. +1 As evidenced by the fact that a large fraction of those optimizations are actually currently entirely broken. Without anybody noticing for years: http://archives.postgresql.org/message-id/20150702220524.GA9392%40svana.org Greetings, Andres Freund -- Sent via pgsql-hackers

Re: [HACKERS] safer node casting

2017-01-02 Thread Andres Freund
Assert(c == NULL || IsA(c, t)); return c; } #define castNode(_type_, nodeptr) ((_type_ *) castNodeImpl(nodeptr, _type_)) should work without too much trouble afaics? Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make chan

Re: [HACKERS] Broken atomics code on PPC with FreeBSD 10.3

2016-12-30 Thread Andres Freund
On December 30, 2016 4:48:22 PM GMT+01:00, Tom Lane wrote: >and got no warnings and the attached output. I'm not very good at >reading >PPC assembler, but I think what is happening in the "char" case is that >gcc is trying to emulate a byte-wide operation using a word-wide

Re: [HACKERS] Broken atomics code on PPC with FreeBSD 10.3

2016-12-30 Thread Andres Freund
On December 30, 2016 4:48:22 PM GMT+01:00, Tom Lane <t...@sss.pgh.pa.us> wrote: >Andres Freund <and...@anarazel.de> writes: >> On 2016-12-30 00:44:33 -0500, Tom Lane wrote: >>> Perhaps it could be argued that there's a FreeBSD compiler bug here, >>>

Re: [HACKERS] Broken atomics code on PPC with FreeBSD 10.3

2016-12-30 Thread Andres Freund
Hi, On 2016-12-30 00:44:33 -0500, Tom Lane wrote: > *** /usr/home/tgl/pgsql/src/test/regress/expected/lock.out Thu Dec 29 > 19:37:50 2016 > --- /usr/home/tgl/pgsql/src/test/regress/results/lock.out Fri Dec 30 > 00:31:01 2016 > *** > *** 63,70 > -- atomic ops tests

Re: [HACKERS] Duplicate node tag assignments

2016-12-28 Thread Andres Freund
Hi, On 2016-12-28 11:33:31 -0500, Tom Lane wrote: > By chance I happened to notice that the recent partition patch pushed > us over the number of available node tags between > > T_A_Expr = 900, > So I'm leaning to the second, more drastic, solution. Thoughts? Alternatively we could add

Re: [HACKERS] Potential data loss of 2PC files

2016-12-27 Thread Andres Freund
On 2016-12-27 14:09:05 +0900, Michael Paquier wrote: > On Fri, Dec 23, 2016 at 3:02 AM, Andres Freund <and...@anarazel.de> wrote: > > Not quite IIRC: that doesn't deal with file size increase. All this would > > be easier if hardlinks wouldn't exist IIUC. It's basically a

Re: [HACKERS] gettimeofday is at the end of its usefulness?

2016-12-27 Thread Andres Freund
very suboptimal choice and we ruling the dice each time > you restarted... I'm pretty strongly against doing performance measurements at startup. Both the delay and the potential for differing test results seem like pretty bad consequences. Greetings, Andres Freund -- Sent via pgsql-hac

Re: [HACKERS] pg_stat_activity.waiting_start

2016-12-25 Thread Andres Freund
On December 25, 2016 1:21:43 AM GMT+01:00, Joel Jacobson wrote: >Is it really a typical real-life scenario that processes can be >waiting extremely often for extremely short periods of time, >where the timing overhead would be significant? Yes. Consider WAL insertion,

[HACKERS] Remove lower limit on checkpoint_timeout?

2016-12-23 Thread Andres Freund
Hi, for testing - like yesterday's 6ef2eba3f - it's annoying that checkpoint_timeout has 30s minimum. I've now locally patched that to be 1s a significant number of times. While it's not a particularly good idea to set it to 1s on a production system, I don't see why we need to prevent that.

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

2016-12-22 Thread Andres Freund
Hi, On 2016-12-22 08:32:56 -0800, Andres Freund wrote: > I plan to commit this later today. Hope I got the reviewers roughly right. And pushed. Thanks for the work on this everyone. Andres -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to y

Re: [HACKERS] Potential data loss of 2PC files

2016-12-22 Thread Andres Freund
On December 22, 2016 6:44:22 PM GMT+01:00, Robert Haas <robertmh...@gmail.com> wrote: >On Thu, Dec 22, 2016 at 12:39 PM, Andres Freund <and...@anarazel.de> >wrote: >> It makes more sense of you mentally separate between filename(s) and >file contents. Havin

Re: [HACKERS] Potential data loss of 2PC files

2016-12-22 Thread Andres Freund
On December 22, 2016 5:50:38 PM GMT+01:00, Robert Haas wrote: >On Wed, Dec 21, 2016 at 8:30 PM, Michael Paquier > wrote: >> Hi all, >> >> 2PC files are created using RecreateTwoPhaseFile() in two places >currently: >> - at replay on a

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

2016-12-22 Thread Andres Freund
Hi, On 2016-12-21 13:28:54 -0800, Andres Freund wrote: > A mime-type of invalid/octet-stream? That's an, uh, odd choice. > > Working on committing this (tomorrow morning, not tonight). There's > some relatively minor things I want to change: > > - I don't like the name XLo

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

2016-12-21 Thread Andres Freund
On 2016-12-21 16:35:28 -0500, Robert Haas wrote: > On Wed, Dec 21, 2016 at 4:28 PM, Andres Freund <and...@anarazel.de> wrote: > > - Similarly I don't like the name "progress LSN" much. What does > > "progress" really mean in that". Maybe "consis

Re: [HACKERS] Fix checkpoint skip logic on idle systems by tracking LSN progress

2016-12-21 Thread Andres Freund
Hi, A mime-type of invalid/octet-stream? That's an, uh, odd choice. Working on committing this (tomorrow morning, not tonight). There's some relatively minor things I want to change: - I don't like the name XLogSetFlags() - it's completely unclear what that those flags refer to - it could

Re: [HACKERS] Parallel bitmap heap scan

2016-12-21 Thread Andres Freund
cExtended(tb->ctx, > + > sizeof(SH_ELEMENT_TYPE) * tb->size, > + > MCXT_ALLOC_HUGE | MCXT_ALLOC_ZERO); If we go there, it seems bett

Re: [HACKERS] simplehash vs. pgindent

2016-12-21 Thread Andres Freund
On 2016-12-21 00:32:22 -0500, Tom Lane wrote: > Robert Haas writes: > > In commit b30d3ea824c5ccba43d3e942704f20686e7dbab8, when Andres added > > the simplehash stuff, he also added SH_TYPE, SH_ITERATOR, and > > SH_STATUS to typedefs.list. When I subsequently updated

Re: [HACKERS] Time to drop old-style (V0) functions?

2016-12-20 Thread Andres Freund
On 2016-12-20 08:35:05 -0500, Robert Haas wrote: > On Tue, Dec 20, 2016 at 8:21 AM, Andres Freund <and...@anarazel.de> wrote: > > On 2016-12-20 08:15:07 -0500, Robert Haas wrote: > >> On Tue, Dec 20, 2016 at 3:11 AM, Andres Freund <and...@anarazel.de> wrote: > >

Re: [HACKERS] Time to drop old-style (V0) functions?

2016-12-20 Thread Andres Freund
On 2016-12-20 08:15:07 -0500, Robert Haas wrote: > On Tue, Dec 20, 2016 at 3:11 AM, Andres Freund <and...@anarazel.de> wrote: > > I think a more efficient variant would make the function signature look > > something like: > > > > Datum /* directl

Re: [HACKERS] increasing the default WAL segment size

2016-12-20 Thread Andres Freund
On 2016-12-20 08:10:29 -0500, Robert Haas wrote: > We could use the GUC assign hook to compute a mask and a shift, so > that this could be written as (CurrPos & mask_variable) == 0. That > would avoid the division instruction, though not the memory access. I suspect that'd be fine. > I hope

Re: [HACKERS] Time to drop old-style (V0) functions?

2016-12-20 Thread Andres Freund
On 2016-12-20 10:44:35 +0100, Pavel Stehule wrote: > 2016-12-20 10:28 GMT+01:00 Andres Freund <and...@anarazel.de>: > > > On 2016-12-20 01:14:10 -0800, Andres Freund wrote: > > > On 2016-12-20 09:59:43 +0100, Pavel Stehule wrote: > > > > In this ca

Re: [HACKERS] Time to drop old-style (V0) functions?

2016-12-20 Thread Andres Freund
On 2016-12-20 01:14:10 -0800, Andres Freund wrote: > On 2016-12-20 09:59:43 +0100, Pavel Stehule wrote: > > In this case some benchmark can be very important (and interesting). I am > > not sure if faster function execution has significant benefit on Vulcano > > like exec

Re: [HACKERS] Time to drop old-style (V0) functions?

2016-12-20 Thread Andres Freund
On 2016-12-20 09:59:43 +0100, Pavel Stehule wrote: > In this case some benchmark can be very important (and interesting). I am > not sure if faster function execution has significant benefit on Vulcano > like executor. It's fairly to see function calls as significant overhead. In fact, I moved

Re: [HACKERS] increasing the default WAL segment size

2016-12-20 Thread Andres Freund
egSize would be defined as 1 << 24). Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Time to drop old-style (V0) functions?

2016-12-20 Thread Andres Freund
On 2016-12-19 15:25:42 -0500, Robert Haas wrote: > On Mon, Dec 19, 2016 at 3:13 PM, Peter Eisentraut > wrote: > > On 12/9/16 7:52 AM, Robert Haas wrote: > >> It's kind of ironic, at least IMHO, that the DirectionFunctionCall > >> macros are anything but direct.

Re: [HACKERS] Retire src/backend/port/dynloader/linux.c ?

2016-12-19 Thread Andres Freund
On 2016-12-18 22:19:36 -0500, Tom Lane wrote: > Andres Freund <and...@anarazel.de> writes: > > Shouldn't we just remove that code? > > What for? I every now and then end up looking at it for a few minutes, and wonder what the hell dld is, just to see that it's old stuff. &g

[HACKERS] Retire src/backend/port/dynloader/linux.c ?

2016-12-18 Thread Andres Freund
Hi, I don't think PG works on any linux without dlopen(). And DLD (what's used in the dlopen replacement) hasn't been maintained in a while. See https://www.gnu.org/software/dld/ Shouldn't we just remove that code? Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql

Re: [HACKERS] Speedup twophase transactions

2016-12-17 Thread Andres Freund
On 2016-12-17 15:30:08 -0800, David Fetter wrote: > On Sat, Dec 17, 2016 at 05:54:04PM -0500, Bruce Momjian wrote: > > On Sun, Dec 18, 2016 at 07:41:50AM +0900, Michael Paquier wrote: > > > On Sun, Dec 18, 2016 at 6:42 AM, Bruce Momjian wrote: > > > > Uh, did you mean to

Re: [HACKERS] Creating a DSA area to provide work space for parallel execution

2016-12-16 Thread Andres Freund
On 2016-12-16 12:33:11 -0500, Robert Haas wrote: > On Fri, Dec 16, 2016 at 12:32 PM, Robert Haas <robertmh...@gmail.com> wrote: > > On Fri, Dec 16, 2016 at 12:28 PM, Andres Freund <and...@anarazel.de> wrote: > >> On 2016-12-16 11:41:49 -0500, Robert Haas wrote: >

Re: [HACKERS] Creating a DSA area to provide work space for parallel execution

2016-12-16 Thread Andres Freund
On 2016-12-16 12:32:49 -0500, Robert Haas wrote: > On Fri, Dec 16, 2016 at 12:28 PM, Andres Freund <and...@anarazel.de> wrote: > > On 2016-12-16 11:41:49 -0500, Robert Haas wrote: > >> On Wed, Dec 14, 2016 at 3:25 PM, Robert Haas <robertmh...@gmail.com> wrote: >

Re: [HACKERS] Performance degradation in Bitmapscan (commit 75ae538bc3168bf44475240d4e0487ee2f3bb376)

2016-12-16 Thread Andres Freund
On 2016-12-16 10:12:42 -0500, Robert Haas wrote: > On Wed, Dec 14, 2016 at 11:20 PM, Robert Haas wrote: > > I've got no problem with that at all, but I want to unbreak things > > more or less immediately and then you/we can further improve it later. > > Committed Thanks.

Re: [HACKERS] Creating a DSA area to provide work space for parallel execution

2016-12-16 Thread Andres Freund
On 2016-12-16 11:41:49 -0500, Robert Haas wrote: > On Wed, Dec 14, 2016 at 3:25 PM, Robert Haas wrote: > > Thoughts? > > Hearing no objections, I've gone ahead and committed this. If that > makes somebody really unhappy I can revert it, but I am betting that > the real

Re: [HACKERS] Hang in pldebugger after git commit : 98a64d0

2016-12-14 Thread Andres Freund
ive to seing the problem :( Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Performance degradation in Bitmapscan (commit 75ae538bc3168bf44475240d4e0487ee2f3bb376)

2016-12-14 Thread Andres Freund
been over a month since these problems were reported, and > it is not good that the tree has been in a broken state for that > entire time. Yea, sorry for that :(. Unfortunately the JIT stuff currently occupies a large portion of my brain. I really hope to come up with a new version of the patch t

Re: [HACKERS] Logical Replication WIP

2016-12-13 Thread Andres Freund
On 2016-12-13 06:55:31 +0100, Petr Jelinek wrote: > >> This is a quadratic algorithm - that could bite us... Not sure if we > >> need to care. If we want to fix it, one approach owuld be to use > >> RangeVarGetRelid() instead, and then do a qsort/deduplicate before > >> actually opening the

Re: [HACKERS] Logical Replication WIP

2016-12-13 Thread Andres Freund
r patches are. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Hang in pldebugger after git commit : 98a64d0

2016-12-13 Thread Andres Freund
On 2016-12-12 16:46:38 +0900, Michael Paquier wrote: > OK, I think that I have spotted an issue after additional read of the > code. When a WSA event is used for read/write activity on a socket, > the same WSA event gets reused again and again. That's fine for > performance reasons It's actually

Re: [HACKERS] WIP: Faster Expression Processing and Tuple Deforming (including JIT)

2016-12-12 Thread Andres Freund
Hi, On 2016-12-12 18:11:13 -0800, CK Tan wrote: > Andres, > > dev (no jiting): > > Time: 30343.532 ms > > > dev (jiting): > > SET jit_tuple_deforming = on; > > SET jit_expressions = true; > > > > Time: 24439.803 ms > > FYI, ~20% improvement for TPCH Q1 is consistent with what we find when we >

Re: [HACKERS] Logical Replication WIP

2016-12-12 Thread Andres Freund
On 2016-12-10 08:48:55 +0100, Petr Jelinek wrote: > diff --git a/src/backend/catalog/pg_publication.c > b/src/backend/catalog/pg_publication.c > new file mode 100644 > index 000..e3560b7 > --- /dev/null > +++ b/src/backend/catalog/pg_publication.c > + > +Datum

Re: [HACKERS] Logical Replication WIP

2016-12-12 Thread Andres Freund
HJi, On 2016-12-12 09:18:48 -0500, Peter Eisentraut wrote: > On 12/8/16 4:10 PM, Petr Jelinek wrote: > > On 08/12/16 20:16, Peter Eisentraut wrote: > >> On 12/6/16 11:58 AM, Peter Eisentraut wrote: > >>> On 12/5/16 6:24 PM, Petr Jelinek wrote: > I think that the removal of changes to

Re: [HACKERS] snapbuild woes

2016-12-12 Thread Andres Freund
On 2016-12-12 23:27:30 +0100, Petr Jelinek wrote: > On 12/12/16 22:42, Andres Freund wrote: > > Hi, > > > > On 2016-12-10 23:10:19 +0100, Petr Jelinek wrote: > >> Hi, > >> First one is outright bug, which has to do with how we track running > >> t

Re: [HACKERS] snapbuild woes

2016-12-12 Thread Andres Freund
Hi, On 2016-12-10 23:10:19 +0100, Petr Jelinek wrote: > Hi, > First one is outright bug, which has to do with how we track running > transactions. What snapbuild basically does while doing initial snapshot > is read the xl_running_xacts record, store the list of running txes and > then wait until

Re: [HACKERS] exposing wait events for non-backends (was: Tracking wait event for latches)

2016-12-12 Thread Andres Freund
On 2016-12-12 13:26:32 -0300, Alvaro Herrera wrote: > Robert Haas wrote: > > > So, one of the problems in this patch as committed is that for any > > process that doesn't show up in pg_stat_activity, there's no way to > > see the wait event information. That sucks. I think there are > >

Re: [HACKERS] Macro customizable hashtable / bitmapscan & aggregation perf

2016-12-09 Thread Andres Freund
Hi, On 2016-12-09 15:21:36 -0800, Mark Dilger wrote: > Andres, > > Your patch, below, appears to have been applied to master in commit > 5dfc198146b49ce7ecc8a1fc9d5e171fb75f6ba5. It makes mention of a > function, tuplehash_start_iterate, in a macro, but the function is not > defined or

Re: [HACKERS] jsonb problematic operators

2016-12-09 Thread Andres Freund
On 2016-12-09 12:17:32 -0500, Robert Haas wrote: > As Geoff says, you don't have to use the operators; you could use the > equivalent functions instead. Every operator just gets turned into a > function call internally, so this is always possible. Well, except that only operators support

Re: [HACKERS] Time to drop old-style (V0) functions?

2016-12-08 Thread Andres Freund
On 2016-12-08 18:03:04 -0500, Tom Lane wrote: > Andres Freund <and...@anarazel.de> writes: > > On 2016-12-08 17:38:38 -0500, Tom Lane wrote: > >> The habit of zero-initializing Datums has got exactly nothing to do with > >> V0 functions; it's about ensurin

Re: [HACKERS] Time to drop old-style (V0) functions?

2016-12-08 Thread Andres Freund
On 2016-12-08 14:53:58 -0800, Andres Freund wrote: > On 2016-12-08 17:38:38 -0500, Tom Lane wrote: > > The habit of zero-initializing Datums has got exactly nothing to do with > > V0 functions; it's about ensuring consistent results and avoiding > > heisenbugs from use of uni

Re: [HACKERS] Time to drop old-style (V0) functions?

2016-12-08 Thread Andres Freund
On 2016-12-08 17:38:38 -0500, Tom Lane wrote: > Andres Freund <and...@anarazel.de> writes: > > I'm wondering if it's not time for $subject: > > - V0 causes confusion / weird crashes when PG_FUNCTION_INFO_V1 was > > forgotten > > - They have us keep weird hacks ar

[HACKERS] Time to drop old-style (V0) functions?

2016-12-08 Thread Andres Freund
if the corresponding isnull marker is set. - they allow to call arbitrary functions pretty easily I don't see any reason to keep them around. If seriously doubt anybody is using them seriously in anything but error. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers

Re: [HACKERS] Declarative partitioning - another take

2016-12-07 Thread Andres Freund
On 2016-12-07 13:20:04 -0500, Robert Haas wrote: > On Wed, Dec 7, 2016 at 11:53 AM, Erik Rijkers wrote: > >> My bad. The fix I sent last night for one of the cache flush issues > >> wasn't quite right. The attached seems to fix it. > > Yes, fixed here too. Thanks. > > Thanks

Re: [HACKERS] Back-patch use of unnamed POSIX semaphores for Linux?

2016-12-07 Thread Andres Freund
On 2016-12-06 23:54:43 -0500, Tom Lane wrote: > You're attacking a straw man. I didn't propose changing our behavior > anywhere but Linux. AFAIK, on that platform unnamed POSIX semaphores > are futexes, which have been a stable feature since 2003 according to >

Re: [HACKERS] Back-patch use of unnamed POSIX semaphores for Linux?

2016-12-07 Thread Andres Freund
's fair to users to swap a proven solution out for something that hasn't seen a lot of load. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] WIP: Faster Expression Processing and Tuple Deforming (including JIT)

2016-12-06 Thread Andres Freund
On 2016-12-06 13:27:14 -0800, Peter Geoghegan wrote: > On Mon, Dec 5, 2016 at 7:49 PM, Andres Freund <and...@anarazel.de> wrote: > > I tried to address 2) by changing the C implementation. That brings some > > measurable speedups, but it's not huge. A bigger speedup is ma

Re: [HACKERS] WIP: Faster Expression Processing and Tuple Deforming (including JIT)

2016-12-06 Thread Andres Freund
On 2016-12-06 14:35:43 -0600, Nico Williams wrote: > On Tue, Dec 06, 2016 at 12:27:51PM -0800, Andres Freund wrote: > > On 2016-12-06 14:19:21 -0600, Nico Williams wrote: > > > > I concur with your feeling that hand-rolled JIT is right out. But > > > > > >

Re: [HACKERS] WIP: Faster Expression Processing and Tuple Deforming (including JIT)

2016-12-06 Thread Andres Freund
On 2016-12-06 15:25:44 -0500, Tom Lane wrote: > Andres Freund <and...@anarazel.de> writes: > > On 2016-12-06 13:56:28 -0500, Tom Lane wrote: > >> I guess the $64 question that has to be addressed here is whether we're > >> prepared to accept LLVM as a run-time de

Re: [HACKERS] WIP: Faster Expression Processing and Tuple Deforming (including JIT)

2016-12-06 Thread Andres Freund
Hi, On 2016-12-06 14:19:21 -0600, Nico Williams wrote: > A bigger concern might be interface stability. IIRC the LLVM C/C++ > interfaces are not very stable, but bitcode is. The C API is a lot more stable than the C++ bit, that's the primary reason I ended up using it, despite the C++ docs

Re: [HACKERS] WIP: Faster Expression Processing and Tuple Deforming (including JIT)

2016-12-06 Thread Andres Freund
On 2016-12-06 15:13:21 -0500, Robert Haas wrote: > Presumably this is going to need to be something that a user can get > via yum install or apt-get install on common systems. Right. apt-get install llvm-dev (or llvm-3.9-dev or such if you want to install a specific version), does the trick

Re: [HACKERS] WIP: Faster Expression Processing and Tuple Deforming (including JIT)

2016-12-06 Thread Andres Freund
On 2016-12-06 11:10:59 -0800, Andres Freund wrote: > > * Are there any currently-interesting platforms that LLVM doesn't work > > for? (I'm worried about RISC-V as much as legacy systems.) > > LLVM itself I don't think is a problem, it seems to target a wide range > of pla

Re: [HACKERS] WIP: Faster Expression Processing and Tuple Deforming (including JIT)

2016-12-06 Thread Andres Freund
On 2016-12-06 14:04:09 -0500, Robert Haas wrote: > I've heard at least one and maybe several PGCon presentations about > people JITing tuple deformation and getting big speedups, and I'd like > to finally hear one from somebody who intends to integrate that into > PostgreSQL. I certainly want to.

Re: [HACKERS] WIP: Faster Expression Processing and Tuple Deforming (including JIT)

2016-12-06 Thread Andres Freund
Hi, On 2016-12-06 13:56:28 -0500, Tom Lane wrote: > Andres Freund <and...@anarazel.de> writes: > > I'm posting a quite massive series of WIP patches here, to get some > > feedback. > > I guess the $64 question that has to be addressed here is whether we're > pr

Re: [HACKERS] Logical Replication WIP

2016-12-05 Thread Andres Freund
On 2016-12-02 12:37:49 -0500, Peter Eisentraut wrote: > On 11/20/16 1:02 PM, Petr Jelinek wrote: > > 0001: > > This is the reworked approach to temporary slots that I sent earlier. > > Andres, you had expressed an interest in this. Will you be able to > review it soon? Yep. Needed to get that

[HACKERS] WIP: Faster Expression Processing and Tuple Deforming (including JIT)

2016-12-05 Thread Andres Freund
Hi Everyone, TL;DR: Making things faster. Architectural evalation. as some of you might be aware I've been working on making execution of larger queries in postgresl faster. While working on "batched execution" I came to the conclusion that, while necessary, isn't currently showing a large

Re: [HACKERS] [COMMITTERS] pgsql: Add support for restrictive RLS policies

2016-12-05 Thread Andres Freund
On 2016-12-05 20:51:02 +, Stephen Frost wrote: > Add support for restrictive RLS policies > Modified Files > -- > src/backend/nodes/copyfuncs.c | 1 + > src/backend/nodes/equalfuncs.c| 1 + > src/backend/parser/gram.y | 43 +++-- >

Re: [HACKERS] Time to retire Windows XP buildfarm host?

2016-12-05 Thread Andres Freund
Hi, On 2016-12-05 10:56:00 -0500, Andrew Dunstan wrote: > Windows XP has been past end of life for quite some time. Nevertheless I > have kept my instance running with three buildfarm members: frogmouth, > currawong and brolga. Howeever, a recent commit (apparently fa2fa99, but I'm > not 100%

Re: [HACKERS] Mail thread references in commits

2016-12-01 Thread Andres Freund
On 2016-12-01 18:12:34 -0500, Tom Lane wrote: > Robert Haas writes: > > On Thu, Dec 1, 2016 at 4:33 PM, Tom Lane wrote: > >> When and if somebody tries to game that, we can do something about it, > >> but I'm not very worried. It's not like it's not

Re: [HACKERS] Mail thread references in commits

2016-12-01 Thread Andres Freund
On 2016-12-01 18:05:19 -0500, Tom Lane wrote: > ... the shortener isn't really doing anything for us. You end up with a > line longer than 80 characters with message-IDs generated by either gmail > or the bug report form, for instance these examples from recent commits: Still seems quite useful

Re: [HACKERS] Proposal: scan key push down to heap [WIP]

2016-12-01 Thread Andres Freund
ner (or parse-analysis?) to set an OpExpr flag that signals that the expression can be evaluated as a scankey, that'd be easy. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Broken SSL tests in master

2016-12-01 Thread Andres Freund
On 2016-12-01 14:43:04 -0500, Robert Haas wrote: > On Thu, Dec 1, 2016 at 2:40 PM, Andres Freund <and...@anarazel.de> wrote: > > On 2016-12-01 14:22:19 -0500, Robert Haas wrote: > >> On Thu, Nov 24, 2016 at 4:38 PM, Andreas Karlsson <andr...@proxel.se> > >>

Re: [HACKERS] Broken SSL tests in master

2016-12-01 Thread Andres Freund
On 2016-12-01 14:22:19 -0500, Robert Haas wrote: > On Thu, Nov 24, 2016 at 4:38 PM, Andreas Karlsson wrote: > > The SSL test suite (src/test/ssl) is broken in the master since commit > > 9a1d0af4ad2cbd419115b453d811c141b80d872b, which is Robert's refactoring of > > getting the

Re: [HACKERS] Improve hash-agg performance

2016-11-30 Thread Andres Freund
Hi, On 2016-11-03 04:07:21 -0700, Andres Freund wrote: > Hi, > > There's two things I found while working on faster expression > evaluation, slot deforming and batched execution. As those two issues > often seemed quite dominant cost-wise it seemed worthwhile to evaluate >

[HACKERS] pgcrypto compilation error due to stack-allocated EVP_CIPHER_CTX

2016-11-30 Thread Andres Freund
Author: Dr. Stephen Henson <st...@openssl.org> Date: 2005-12-02 13:46:39 + New functions to support opaque EVP_CIPHER_CTX handling. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription

Re: [HACKERS] move collation import to backend

2016-11-29 Thread Andres Freund
On 2016-11-29 12:16:37 -0500, Peter Eisentraut wrote: > On 11/12/16 10:38 AM, Andres Freund wrote: > >>/* > >> * Also forbid matching an any-encoding entry. This test of course is > >> not > >> * backed up by the unique index, but it's not a pr

Re: [HACKERS] Proposal: scan key push down to heap [WIP]

2016-11-28 Thread Andres Freund
On 2016-11-28 09:55:00 -0500, Robert Haas wrote: > I think we should go with this approach. I don't think it's a good > idea to have the heapam layer know about executor slots. I agree that that's not pretty. > Even though > it's a little sad to pass up an opportunity for a larger performance >

Re: [HACKERS] PATCH: two slab-like memory allocators

2016-11-27 Thread Andres Freund
On 2016-11-27 22:21:49 +0100, Petr Jelinek wrote: > On 27/11/16 21:47, Andres Freund wrote: > > Hi, > > > >>> +typedef struct SlabBlockData *SlabBlock; /* forward reference */ > >>> +typedef struct SlabChunkData *SlabChunk; > >>> >

Re: [HACKERS] PATCH: two slab-like memory allocators

2016-11-27 Thread Andres Freund
Hi, > > +typedef struct SlabBlockData *SlabBlock; /* forward reference */ > > +typedef struct SlabChunkData *SlabChunk; > > > > Can we please not continue hiding pointers behind typedefs? It's a bad > > pattern, and that it's fairly widely used isn't a good excuse to > > introduce

Re: [HACKERS] Skipping PgStat_FunctionCallUsage for many expressions

2016-11-26 Thread Andres Freund
On November 26, 2016 8:06:26 AM PST, Tom Lane <t...@sss.pgh.pa.us> wrote: >Robert Haas <robertmh...@gmail.com> writes: >> On Fri, Nov 25, 2016 at 11:12 PM, Andres Freund <and...@anarazel.de> >wrote: >>> while working on my faster expression eva

[HACKERS] Skipping PgStat_FunctionCallUsage for many expressions

2016-11-25 Thread Andres Freund
Hi, while working on my faster expression evaluation stuff I noticed that a lot of expression types that call functions don't call the necessary functions to make track_functions work. ExecEvalFunc/ExecEvalOper (via ExecMakeFunctionResultNoSets) call

Re: [HACKERS] [bugfix] commit timestamps ERROR on lookup of FrozenTransactionId

2016-11-23 Thread Andres Freund
Hi, On 2016-11-23 20:58:22 +0800, Craig Ringer wrote: > Today I ran into an issue where commit timestamp lookups were failing with > > ERROR: cannot retrieve commit timestamp for transaction 2 > > which is of course FrozenTransactionId. > > TransactionIdGetCommitTsData(...) ERRORs on

Re: [HACKERS] Performance degradation in Bitmapscan (commit 75ae538bc3168bf44475240d4e0487ee2f3bb376)

2016-11-23 Thread Andres Freund
On 2016-11-18 08:00:40 -0500, Robert Haas wrote: > On Tue, Nov 15, 2016 at 2:28 PM, Andres Freund <and...@anarazel.de> wrote: > > I've a working fix for this, and for a similar issue Robert found. I'm > > still playing around with it, but basically the fix is to make the &g

Re: [HACKERS] [RFC] Should we fix postmaster to avoid slow shutdown?

2016-11-22 Thread Andres Freund
On 2016-11-22 16:15:58 -0500, Tom Lane wrote: > Alvaro Herrera writes: > > Well, the problem is that the stats data is not on disk while the system > > is in operation, as far as I recall -- it's only in the collector's > > local memory. On shutdown we tell it to write

Re: [HACKERS] [RFC] Should we fix postmaster to avoid slow shutdown?

2016-11-22 Thread Andres Freund
uential whether we write out pgstat, because we're going to nuke it either way after an immediate shutdown? Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Logical decoding on standby

2016-11-21 Thread Andres Freund
Hi, On 2016-11-21 16:17:58 +0800, Craig Ringer wrote: > I've prepared a working initial, somewhat raw implementation for > logical decoding on physical standbys. Please attach. Otherwise in a year or two it'll be impossible to look this up. Andres -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Mail thread references in commits

2016-11-18 Thread Andres Freund
On November 18, 2016 1:06:18 PM PST, Tom Lane wrote: >Robert Haas writes: >> On Thu, Nov 17, 2016 at 10:43 PM, Joshua Drake >wrote: >>> Why not hash the URL? Something like: >>> Http://postgresopen.org/archive/743257890976432

Re: [HACKERS] WAL recycle retading based on active sync rep.

2016-11-18 Thread Andres Freund
nt period. I very strongly think not. > - Is the measure acceptable? For the worst case, a master > crashes from WAL space exhaustion. (But such large transaction > won't/shouldn't exist?) No, imo not. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hac

Re: [HACKERS] Mail thread references in commits

2016-11-17 Thread Andres Freund
On November 17, 2016 1:02:38 PM PST, Andrew Dunstan wrote: >I love seeing references to email threads in commit messages. It would >make them a lot friendlier though if a full http URL were included >instead of just a Message-ID, i.e. instead of put

Re: [HACKERS] Password identifiers, protocol aging and SCRAM protocol

2016-11-16 Thread Andres Freund
Hi, On 2016-11-16 19:29:41 -0500, Robert Haas wrote: > On Wed, Nov 16, 2016 at 6:56 PM, Michael Paquier > wrote: > > On Wed, Nov 16, 2016 at 11:24 AM, Robert Haas wrote: > >> diff --git a/contrib/pgcrypto/Makefile b/contrib/pgcrypto/Makefile >

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-11-16 Thread Andres Freund
Hi, I think the patch could use a pgindent run. On 2016-09-07 11:44:01 -0700, Peter Geoghegan wrote: > diff --git a/contrib/amcheck/amcheck--1.0.sql > b/contrib/amcheck/amcheck--1.0.sql > new file mode 100644 > index 000..ebbd6ac > --- /dev/null > +++ b/contrib/amcheck/amcheck--1.0.sql >

Re: [HACKERS] Performance degradation in Bitmapscan (commit 75ae538bc3168bf44475240d4e0487ee2f3bb376)

2016-11-15 Thread Andres Freund
till playing around with it, but basically the fix is to make the growth policy a bit more adaptive. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Re: [COMMITTERS] pgsql: Change the way that LWLocks for extensions are allocated.

2016-11-14 Thread Andres Freund
On 2016-11-14 12:14:10 -0800, Andres Freund wrote: > On 2016-11-12 11:42:12 -0500, Tom Lane wrote: > > Andres Freund <and...@anarazel.de> writes: > > > On 2016-11-12 11:30:42 -0500, Tom Lane wrote: > > >> which is a rather blatant waste of cycles. I would s

Re: [HACKERS] Re: [COMMITTERS] pgsql: Change the way that LWLocks for extensions are allocated.

2016-11-14 Thread Andres Freund
On 2016-11-12 11:42:12 -0500, Tom Lane wrote: > Andres Freund <and...@anarazel.de> writes: > > On 2016-11-12 11:30:42 -0500, Tom Lane wrote: > >> which is a rather blatant waste of cycles. I would suggest an explicit > >> do-nothing installcheck rule rather th

Re: [HACKERS] Pinning a buffer in TupleTableSlot is unnecessary

2016-11-14 Thread Andres Freund
On 2016-11-14 13:12:28 -0500, Tom Lane wrote: > Andres Freund <and...@anarazel.de> writes: > > On 2016-11-14 12:32:53 -0500, Tom Lane wrote: > >> Basically my concern is that this restriction isn't documented anywhere > >> and I'm not entirely certain it's been

Re: [HACKERS] Pinning a buffer in TupleTableSlot is unnecessary

2016-11-14 Thread Andres Freund
On 2016-11-14 12:32:53 -0500, Tom Lane wrote: > Heikki Linnakangas writes: > > On 11/14/2016 06:18 PM, Tom Lane wrote: > >> You're implicitly assuming that a scan always returns its results in the > >> same slot, and that no other slot could contain a copy of that data, but > >>

Re: [HACKERS] Pinning a buffer in TupleTableSlot is unnecessary

2016-11-14 Thread Andres Freund
On 2016-11-14 10:09:02 -0500, Robert Haas wrote: > On Sat, Nov 12, 2016 at 10:28 AM, Andres Freund <and...@anarazel.de> wrote: > > On 2016-08-30 07:38:10 -0400, Tom Lane wrote: > >> Heikki Linnakangas <hlinn...@iki.fi> writes: > >> > While profiling some q

Re: [HACKERS] ExplainOneQuery_hook ignored for EXPLAIN EXECUTE

2016-11-13 Thread Andres Freund
On 2016-11-13 17:20:05 -0500, Tom Lane wrote: > Andres Freund <and...@anarazel.de> writes: > > I don't quite know what the hook in $subject was originally designed > > for, but right now it has the problem that it's not invoked for EXPLAIN > > EXECUTE. That's because E

<    9   10   11   12   13   14   15   16   17   18   >