Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

2017-11-09 Thread Andres Freund
On 2017-11-09 16:45:07 -0800, Peter Geoghegan wrote: > On Thu, Nov 9, 2017 at 4:17 PM, Andres Freund wrote: > >> Actually, on second thought, I take that back -- I don't think that > >> REINDEXing will even finish once a HOT chain is broken by the bug. > >>

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

2017-11-09 Thread Andres Freund
- I don't think that > REINDEXing will even finish once a HOT chain is broken by the bug. > IndexBuildHeapScan() actually does quite a good job of making sure > that HOT chains are sane, which is how the enhanced amcheck notices > the bug here in practice. I think that's

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

2017-11-09 Thread Andres Freund
On 2017-11-04 06:15:00 -0700, Andres Freund wrote: > The reason for that is that I hadn't yet quite figured out how the bug I > described in the commit message (and the previously committed testcase) > would cause that. I was planning to diagnose / experiment more with this > and

Re: [HACKERS] pageinspect option to forgo buffer locking?

2017-11-09 Thread Andres Freund
On 2017-11-09 17:14:11 -0500, Tom Lane wrote: > If we do this, I'd suggest exposing it as a separate SQL function > get_raw_page_unlocked() rather than as an option to get_raw_page(). > > The reasoning is that if we ever allow these functions to be controlled > via GRANT instead of hardwired super

Re: [HACKERS] pageinspect option to forgo buffer locking?

2017-11-09 Thread Andres Freund
On 2017-11-09 12:55:30 -0500, Robert Haas wrote: > On Thu, Nov 9, 2017 at 12:49 PM, Andres Freund wrote: > > Occasionally, when debugging issues, I find it quite useful to be able > > to do a heap_page_items() on a page that's actually locked exclusively > > concurrent

[HACKERS] pageinspect option to forgo buffer locking?

2017-11-09 Thread Andres Freund
, but we already allow to pass arbitrary stuff to heap_page_items(). Since pinning wouldn't be changed, there's no danger of the page being moved out from under us. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subs

Re: [HACKERS] Hang in pldebugger after git commit : 98a64d0

2017-11-08 Thread Andres Freund
k events for the indicated socket, clear internal network event records, and reset event objects (optional)." Note that it clears the state from the socket, *not* just the handle. That behaviour of WSAEnumNetworkEvents() also seems to explains why http://git.postgresql.org/gitweb/?p=postgresql.g

Re: [HACKERS] Small improvement to compactify_tuples

2017-11-08 Thread Andres Freund
ally-ordered set of tuples. I can recall > Andres or someone moaning awhile ago about lack of locality of access in > index page searches --- maybe applying that approach while vacuuming > indexes will help? I complained about multiple related things, I'm not exactly sure what exactl

Re: [HACKERS] Parallel Hash take II

2017-11-08 Thread Andres Freund
-> Parallel Hash + -> Parallel Seq Scan on extremely_skewed s +(9 rows) + +select count(*) from simple r join extremely_skewed s using (id); + count +--- + 2 +(1 row) As written before, I think it'd be good if we extracted the number of batc

Re: [HACKERS] Horrible CREATE DATABASE Performance in High Sierra

2017-11-08 Thread Andres Freund
further, but I think this is probably >good >> enough for a first cut. > >What is the status of this? Is performance on High Sierra still bad? Didn't Tom commit a workaround? Andres -- Sent from my Android device with K-9 Mail. Please excuse my brevity. -- Sent via

Re: [HACKERS] Parallel Hash take II

2017-11-07 Thread Andres Freund
t work on this batch"? +void +ExecShutdownHashJoin(HashJoinState *node) +{ + /* +* By the time ExecEndHashJoin runs in a worker, shared memory has been +* destroyed. So this is our last chance to do any shared memory cleanup. + */ This comment doesn'

Re: [HACKERS] Parallel Hash take II

2017-11-07 Thread Andres Freund
p->chunk_expansion++; + } + chunk_pages = p->chunk_pages; + + /* The next reader will start after this chunk. */ + p->read_page += chunk_pages; + } + SpinLockRelease(&p->mutex); This looks more like the job of an lwlock rather than a spinlock. +/* + * Create the name used for our shared BufFiles. + */ +static void +make_name(char *name, SharedTuplestoreAccessor *accessor, int participant) +{ + snprintf(name, MAXPGPATH, "%s.p%d", accessor->sts->name, participant); +} Name's a bit generic. And it's still not really making ;) Going to buy some groceries and then look at the next patches. - Andres -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [POC] Faster processing at Gather node

2017-11-07 Thread Andres Freund
Hi, On 2017-11-06 10:56:43 +0530, Amit Kapila wrote: > On Sun, Nov 5, 2017 at 6:54 AM, Andres Freund wrote > > On 2017-11-05 01:05:59 +0100, Robert Haas wrote: > >> skip-gather-project-v1.patch does what it says on the tin. I still > >> don't have a test case for

Re: [HACKERS] Small improvement to compactify_tuples

2017-11-07 Thread Andres Freund
qsort onto our own implementation: #define qsort(a,b,c,d) pg_qsort(a,b,c,d) Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [POC] Faster processing at Gather node

2017-11-06 Thread Andres Freund
ism specific stuff. Whereas tpcc is oltp, right? > correct In that case, could you provide before/after profiles of the performance changing runs? Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http:

Re: [HACKERS] [POC] Faster processing at Gather node

2017-11-06 Thread Andres Freund
On November 6, 2017 7:30:49 AM PST, Jim Van Fleet wrote: >Andres Freund wrote on 11/05/2017 03:40:15 PM: > >hammerdb, in this configuration, runs a variant of tpcc Hard to believe that any of the changes here are relevant in that case - this is parallelism specific stuff. Wherea

Re: [HACKERS] Early locking option to parallel backup

2017-11-06 Thread Andres Freund
r an ASL if it has 'backup' rights or such. Or actually, just allow it as a general option to LOCK, there's plenty other operational cases where the current "fair" behaviour is really annoying, e.g. when executing operational DDL/DML and such. Greetings, Andres Freund -

Re: [HACKERS] Restricting maximum keep segments by repslots

2017-11-06 Thread Andres Freund
don't think 'distance' is a good metric - that's going to continually change. Why not store the LSN that's available and provide a function that computes this? Or just rely on the lsn - lsn operator? - Andres -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)

Re: [HACKERS] Restricting maximum keep segments by repslots

2017-11-06 Thread Andres Freund
m thinking of this mainly for logical slots. That seems more like a page read callback's job than xlogreader's. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Early locking option to parallel backup

2017-11-05 Thread Andres Freund
e on-demand lock by the dump worker from being granted. It seems to me the correct fix here would be to somehow avoid the fairness logic in the parallel dump case - although I don't quite know how to best do so. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-ha

Re: [HACKERS] [POC] Faster processing at Gather node

2017-11-05 Thread Andres Freund
patch is up a bit over the base > >Net -- the patch is about the same as current base on two socket, and >on >one socket -- consistent with your pgbench (?) findings > >As an aside, it is perhaps a worry that one socket is down over 20% >from >9.6 and over 10% from v10rc1

Re: [HACKERS] [POC] Faster processing at Gather node

2017-11-04 Thread Andres Freund
yet be that big because a good chunk of slot_getattr calls come from execTuplesMatch() which doesn't really provide enough context to do JITing (when used for hashaggs, there is more so it's JITed). Similarly gather merge's heap_compare_slots() doesn't provide such cont

[HACKERS] Display number of heap accesses for index scans

2017-11-04 Thread Andres Freund
to some degree from explain analyze. The number of index lookups that failed to return anything can be a critical performance factor in OLTP workloads. Therefore it seems like it'd be a good idea to extend the explain analyze output to include that information. Greetings, Andres Freund

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

2017-11-04 Thread Andres Freund
On 2017-11-04 06:15:00 -0700, Andres Freund wrote: > The current testcase, and I think the descriptions in the relevant > threads, all actually fail to point out the precise way the bug is > triggered. As e.g. evidenced that the freeze-the-dead case appears to > not cause any

Re: [HACKERS] [POC] Faster processing at Gather node

2017-11-04 Thread Andres Freund
m_mq_get_bytes_written(volatile shm_mq *mq, bool *detached) > -{ > - uint64 v; > - > - SpinLockAcquire(&mq->mq_mutex); > - v = mq->mq_bytes_written; > - *detached = mq->mq_detached; > - SpinLockRelease(&mq->mq_mutex); > -

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

2017-11-04 Thread Andres Freund
On 2017-11-03 12:36:59 -0700, Peter Geoghegan wrote: > Andres Freund wrote: > > Here's that patch. I've stared at this some, and Robert did too. Robert > > mentioned that the commit message might need some polish and I'm not > > 100% sure about the erro

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

2017-11-03 Thread Andres Freund
On November 4, 2017 1:22:04 AM GMT+05:30, Alvaro Herrera wrote: >Peter Geoghegan wrote: >> Andres Freund wrote: > >> > Staring at the vacuumlazy hunk I think I might have found a related >bug: >> > heap_update_tuple() just copies the old xmax to the new tupl

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

2017-11-03 Thread Andres Freund
On 2017-11-02 06:05:51 -0700, Andres Freund wrote: > Hi, > > On 2017-11-02 13:49:47 +0100, Alvaro Herrera wrote: > > Andres Freund wrote: > > > I think the problem is on the pruning, rather than the freezing side. We > > > can't freeze a tuple if it

Re: [HACKERS] ArrayLists instead of List (for some things)

2017-11-02 Thread Andres Freund
y the use of lists, but the real problem is that we're doing all that work in the first place. So I personally think most uses of lists at execution time are archetecturally wrong, therefore optimizing their implementation is just a bandaid. Where I think the arguments for a more optimized i

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

2017-11-02 Thread Andres Freund
Hi, On 2017-11-02 13:49:47 +0100, Alvaro Herrera wrote: > Andres Freund wrote: > > I think the problem is on the pruning, rather than the freezing side. We > > can't freeze a tuple if it has an alive predecessor - rather than > > weakining this, we should be fixing t

Re: [HACKERS] [COMMITTERS] pgsql: Fix freezing of a dead HOT-updated tuple

2017-11-02 Thread Andres Freund
hat I'm outlining above. There'll be some trouble for people that upgraded to an unreleased version, but I don't really see what we could do about that. I could be entirely wrong - I've been travelling for the last two weeks and my brain is somewhat more fried than usual. Regards, Andres -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Re: Anyone have experience benchmarking very high effective_io_concurrency on NVME's?

2017-10-31 Thread Andres Freund
59.45 0.005.130.00 132.0919.73 0.00 0.04 100.00 but it'd be worthwhile to see whether doing the merging ourselves allows for deeper queues. I think we really should start incorporating explicit prefetching in more places. Ordered indexscans might actually be one ca

Re: [HACKERS] Current int & float overflow checking is slow.

2017-10-30 Thread Andres Freund
Hi, On 2017-10-30 22:29:42 +0530, Robert Haas wrote: > On Mon, Oct 30, 2017 at 4:57 PM, Andres Freund wrote: > > 0001) Introduces pg_{add,sub,mul}{16,32,64}_overflow(a, b, *result) > > These use compiler intrinsics on gcc/clang. If that's not > > available,

Re: [HACKERS] Remove secondary checkpoint

2017-10-30 Thread Andres Freund
On 2017-10-30 10:10:19 -0400, Tom Lane wrote: > Robert Haas writes: > > I was mostly just thinking out loud, listing another option rather > > than advocating for it. > > FWIW, I just wanted the question to be debated and resolved properly. > > After rereading the

Re: [HACKERS] Current int & float overflow checking is slow.

2017-10-30 Thread Andres Freund
Hi, On 2017-10-24 03:39:54 -0700, Andres Freund wrote: > Largely that's due to the overflow checks. > > For integers we currently do: > > #define SAMESIGN(a,b) (((a) < 0) == ((b) < 0)) > > /* >* Overflow check. If the inputs are of different sig

Re: [HACKERS] An unlikely() experiment

2017-10-30 Thread Andres Freund
On 2017-10-30 22:39:01 +1300, David Rowley wrote: > On 30 October 2017 at 22:34, Andres Freund wrote: > > Hi, > > > > On 2015-12-20 02:49:13 +1300, David Rowley wrote: > >> Alternatively, if there was some way to mark the path as cold from within > >> the pa

Re: [HACKERS] An unlikely() experiment

2017-10-30 Thread Andres Freund
figure out a way to do this. I just was thinking about this, and it turns out that __attribute__((cold)) does what we need here. We could just mark elog_start() and errstart() as cold, and afaict that should do the trick. Thanks to Jakub from #gcc for pointing that out. Greetings, Andres Freu

Re: [HACKERS] Current int & float overflow checking is slow.

2017-10-28 Thread Andres Freund
On 2017-10-24 15:28:17 -0400, Tom Lane wrote: > Also, if I recall the old discussion properly, one concern was getting > uniform behavior across different platforms. I'm worried that if we do > what Andres suggests, we'll get behavior that is not only different but > platf

Re: [HACKERS] Current int & float overflow checking is slow.

2017-10-24 Thread Andres Freund
the case. > > Also, if I recall the old discussion properly, one concern was getting > > uniform behavior across different platforms. I'm worried that if we do > > what Andres suggests, we'll get behavior that is not only different but > > platform-specific. Now, t

Re: [HACKERS] Current int & float overflow checking is slow.

2017-10-24 Thread Andres Freund
On 2017-10-24 10:09:09 -0400, Tom Lane wrote: > Andres Freund writes: > > There's no comparable overflow handling to the above integer > > intrinsics. But I think we can still do a lot better. Two very different > > ways: > > > 1) Just give up on detecting over

Re: [HACKERS] Remove secondary checkpoint

2017-10-24 Thread Andres Freund
on master > > * removes a minor bug in fast failover > > * simplifies code > > Doesn't it also make crash recovery less robust? I think it does the contrary. The current mechanism is, in my opinion, downright dangerous: https://www.postgresql.org/message-id/20160201235854.

Re: [HACKERS] [POC] hash partitioning

2017-10-24 Thread Andres Freund
testing is not good enough or if you want me to do > more testing, thanks. This looks very good! Both the tests you did, and the results for hash_combineXX. I therefore think we can go ahead with that formulation of hash_combine64? Greetings, Andres Freund -- Sent via pgsql-hackers mailing

[HACKERS] Current int & float overflow checking is slow.

2017-10-24 Thread Andres Freund
actually seems entirely reasonable. We already don't detect them in a bunch of cases anyway. I can't quite parse the standard's language around this. 2) Use platform specific float exception handling where available. We could at backend start, and in FloatExceptionHandler(), us feenableexcept() (windows has similar) to trigger SIGFPE on float overflow. 3) Magic? Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] legitimacy of using PG_TRY , PG_CATCH , PG_END_TRY in C function

2017-10-23 Thread Andres Freund
ansaction cleanup. But when you > hit a scenario where that's not true, or when a scenario that used to > not require cleanup now does, nobody is going to consider that a PG bug. It'd probably be a good idea to expand on this in the sgml docs. This has confused quite anumber of people.

Re: [HACKERS] legitimacy of using PG_TRY , PG_CATCH , PG_END_TRY in C function

2017-10-23 Thread Andres Freund
hat in a more general way so > it can be used from background workers. Right now AFAICS most > background workers have to cope with errors with a proc_exit(1) and > getting restarted to try again. Not ideal. I agree that generalizing wouldn't be bad, but there's absolutely not

Re: [HACKERS] Fix performance degradation of contended LWLock on NUMA

2017-10-19 Thread Andres Freund
in BufferTag). > Now I understand, that delays should be stored in array indexed by > tranche. But I have no time to test this idea. And I doubt it will give > cardinally better results (ie > 5%), so I think it is better to accept > patch in this way, and then experiment with per-tranche delay. I don't think tranches have any decent predictive power here. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Fix performance degradation of contended LWLock on NUMA

2017-10-18 Thread Andres Freund
00:00:00 2001 > From: Sokolov Yura > Date: Fri, 2 Jun 2017 11:34:23 + > Subject: [PATCH 6/6] Make SpinDelayStatus a bit lighter. > > It saves couple of instruction of fast path of spin-loop, and so makes > fast path of LWLock a bit faster (and in other places where spinlock is &g

Re: [HACKERS] Fix performance degradation of contended LWLock on NUMA

2017-10-18 Thread Andres Freund
initial iterations, and some iterations after > pg_usleep, because SpinDelayStatus.spins is reset after sleep). I quite strongly think it's a good idea to change this at the same time as the other changes you're proposing here. I think it's a worthwhile thing to pursue, but th

Re: [HACKERS] [POC] Faster processing at Gather node

2017-10-18 Thread Andres Freund
ly long time by workers not making any progress? > >It might also be a good idea to use a more directed form of wakeups, > >e.g. using a per-worker latch + a wait event set, to avoid iterating > >over all workers. > > I don't follow. Well, right now w

Re: [HACKERS] [POC] Faster processing at Gather node

2017-10-17 Thread Andres Freund
Hi, On 2017-10-17 14:39:57 -0700, Andres Freund wrote: > I've spent some time looking into this, and I'm not quite convinced this > is the right approach. Let me start by describing where I see the > current performance problems around gather stemming from. One further app

Re: [HACKERS] [POC] Faster processing at Gather node

2017-10-17 Thread Andres Freund
latch + a wait event set, to avoid iterating over all workers. Unfortunately the patch's "local worker queue" concept seems, to me, like it's not quite addressing the structural issues, instead opting to address them by adding another layer of queuing. I suspect that if we&#x

Re: [HACKERS] Windows warnings from VS 2017

2017-10-17 Thread Andres Freund
is pretty > far from bleeding-edge so I think it's okay as a baseline for > what optimizations we can expect to find used in the field. > > So I can't sustain Andres' assertion that memset is actually faster > for the cases we care about, but it certainly doesn&#x

Re: [HACKERS] [POC] Faster processing at Gather node

2017-10-16 Thread Andres Freund
er with your results? Regards, Andres -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Is it time to kill support for very old servers?

2017-10-16 Thread Andres Freund
On 2017-10-16 16:59:59 -0400, Peter Eisentraut wrote: > On 9/20/17 04:32, Andres Freund wrote: > > Here's what I roughly was thinking of. I don't quite like the name, and > > the way the version is specified for libpq (basically just the "raw" > > integ

Re: [HACKERS] Aggregate FILTER option is broken in v10

2017-10-16 Thread Andres Freund
> which seems like a pretty minimal extra cost for the normal case > with no filter. Thanks, that looks like a reasonable fix. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] pg_control_recovery() return value when not in recovery

2017-10-13 Thread Andres Freund
On 2017-10-13 16:31:37 -0700, Joe Conway wrote: > On 09/17/2017 11:29 PM, Andres Freund wrote: > > On 2017-09-18 07:24:43 +0100, Simon Riggs wrote: > >> On 18 September 2017 at 05:50, Andres Freund wrote: > >> > Hi, > >> > > >> > Just

Re: [HACKERS] Disable cross products in postgres

2017-10-13 Thread Andres Freund
On 2017-10-14 03:49:57 +0530, Gourav Kumar wrote: > But then is there some way to tell Optimizer not to consider transitive > joins ? What are you actually trying to achieve here? - Andres -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes t

Re: [HACKERS] Discussion on missing optimizations

2017-10-13 Thread Andres Freund
the > planner. There were some a couple years back. > It looks like there's plenty we could do in there, just nobody seems > interested enough to go and do it, everyone who cares about > performance is too busy trying to make execution run faster. I think that's largely b

Re: [HACKERS] Improve catcache/syscache performance.

2017-10-13 Thread Andres Freund
Hi, On 2017-10-13 10:38:47 -0700, Andres Freund wrote: > On 2017-10-13 13:06:41 -0400, Robert Haas wrote: > > On Thu, Sep 14, 2017 at 2:12 AM, Andres Freund wrote: > > > This patch gives me roughly 8% speedup in a workload that consists out > > > of a fast query th

Re: [HACKERS] Improve catcache/syscache performance.

2017-10-13 Thread Andres Freund
leDesc. Per type function lookups, oids, should be fairly easy in contrast. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Improve catcache/syscache performance.

2017-10-13 Thread Andres Freund
Hi, On 2017-10-13 13:06:41 -0400, Robert Haas wrote: > On Thu, Sep 14, 2017 at 2:12 AM, Andres Freund wrote: > > This patch gives me roughly 8% speedup in a workload that consists out > > of a fast query that returns a lot of columns. If I apply a few > > other performanc

Re: [HACKERS] Continuous integration on Windows?

2017-10-12 Thread Andres Freund
ed 00:04:36.37" I'd expected building would take a lot longer than that... Neat. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] [POC] hash partitioning

2017-10-12 Thread Andres Freund
On 2017-10-12 17:27:52 -0400, Robert Haas wrote: > On Thu, Oct 12, 2017 at 4:20 PM, Andres Freund wrote: > >> In other words, it's not utterly fixed in stone --- we invented > >> --load-via-partition-root primarily to cope with circumstances that > >> could

Re: [HACKERS] show precise repos version for dev builds?

2017-10-12 Thread Andres Freund
on > each rebuild, even if not necessary, though. That'd be quite painful, consider e.g. people using LTO. If done right however, that should be avoidable to some degree. When creating the version file, only replace its contents if the contents differ - that's just a few lines of scriptin

Re: [HACKERS] [POC] hash partitioning

2017-10-12 Thread Andres Freund
On 2017-10-12 16:06:11 -0400, Robert Haas wrote: > On Thu, Oct 12, 2017 at 3:43 PM, Andres Freund wrote: > > Are we going to rely on the the combine function to stay the same > > forever after? > > If we change them, it will be a pg_upgrade compatibility break for > anyon

Re: [HACKERS] [POC] hash partitioning

2017-10-12 Thread Andres Freund
e fairly random, but they should bring in enough bit perturbation when mixing in only 32bit of hash value (i.e 0x). Are we going to rely on the the combine function to stay the same forever after? Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Re: [COMMITTERS] pgsql: Add configure infrastructure to detect support for C99's restric

2017-10-11 Thread Andres Freund
On 2017-10-11 21:59:53 -0700, Andres Freund wrote: > That fixed that problem I think. But unfortunately since then another > problem has been reported by some other animals, all with older msvc > versions afaict (thrips - vs 2012, bowerbird - vs 2012). Correction, thrips is vs 2010,

[HACKERS] Re: [COMMITTERS] pgsql: Add configure infrastructure to detect support for C99's restric

2017-10-11 Thread Andres Freund
On 2017-10-11 17:13:20 -0700, Andres Freund wrote: > Hi, > > On 2017-10-11 23:11:15 +0000, Andres Freund wrote: > > Add configure infrastructure to detect support for C99's restrict. > > > > Will be used in later commits improving performance for a few key >

Re: [HACKERS] SendRowDescriptionMessage() is slow for queries with a lot of columns

2017-10-11 Thread Andres Freund
On 2017-10-11 08:54:10 -0700, Andres Freund wrote: > On 2017-10-11 10:53:56 +0200, Alvaro Herrera wrote: > > Maybe it'd be a good idea to push 0001 with some user of restrict ahead > > of the rest, just to see how older msvc reacts. > > Can do. Not quite sure which olde

Re: [HACKERS] Windows warnings from VS 2017

2017-10-11 Thread Andres Freund
> On 09/21/2017 09:41 PM, Andres Freund wrote: > > On 2017-09-21 09:30:31 -0400, Tom Lane wrote: > >> Andrew Dunstan writes: > >>> The speed of memset is hardly going to be the dominating factor in a > >>> 'CREATE DATABASE' command, so we co

Re: [HACKERS] Re: [BUGS] BUG #14821: idle_in_transaction_session_timeout sometimes gets ignored when statement timeout is pending

2017-10-11 Thread Andres Freund
x27;t do anything when it gets called > because the query cancel holdoff counter is positive. > > Andres suggested the following re-ordering of the logic on -bugs: I've pushed this. Thanks for the report & fix! Greetings, Andres Freund -- Sent via pgsql-hackers mailing list

Re: [HACKERS] SendRowDescriptionMessage() is slow for queries with a lot of columns

2017-10-11 Thread Andres Freund
ost. We literally allocate 1kb to send a single four byte integer. Fixing the output function performance requires a fairly different type of patch imo. > If do nothing, it's unlikely we'd ever get rid of the compat function. I think that's ok. Greetings, Andres Freund

Re: [HACKERS] Slow synchronous logical replication

2017-10-11 Thread Andres Freund
s per node. There should be one. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] SendRowDescriptionMessage() is slow for queries with a lot of columns

2017-10-11 Thread Andres Freund
On 2017-10-11 10:53:56 +0200, Alvaro Herrera wrote: > Andres Freund wrote: > > Hi, > > > > On 2017-10-03 13:58:37 -0400, Robert Haas wrote: > > > On Tue, Oct 3, 2017 at 12:23 PM, Andres Freund wrote: > > > > Makes sense? > > > > > &

Re: [HACKERS] SendRowDescriptionMessage() is slow for queries with a lot of columns

2017-10-10 Thread Andres Freund
Hi, On 2017-10-03 13:58:37 -0400, Robert Haas wrote: > On Tue, Oct 3, 2017 at 12:23 PM, Andres Freund wrote: > > Makes sense? > > Yes. Here's an updated version of this patchset. Changes: - renamed pq_send$type_pre to pq_write*type - renamed pq_beginmessage_pre/pq_begi

Re: [HACKERS] Is it time to kill support for very old servers?

2017-10-10 Thread Andres Freund
well. > > > > Why? Note that we don't do elsehwere in fe-connect.c. > > Because it seems to me that the default value of the parameter should > be an empty string instead of D. Feels more consistent with the > others. I'm not following. The "D" is in

Re: [HACKERS] Is it time to kill support for very old servers?

2017-10-10 Thread Andres Freund
Hi, On 2017-10-11 10:09:34 +0900, Michael Paquier wrote: > On Wed, Oct 11, 2017 at 9:39 AM, Andres Freund wrote: > > On 2017-09-20 01:32:36 -0700, Andres Freund wrote: > >> Coverage of the relevant files is a good bit higher afterwards. Although > >> our libpq covera

Re: [HACKERS] Is it time to kill support for very old servers?

2017-10-10 Thread Andres Freund
On 2017-09-20 01:32:36 -0700, Andres Freund wrote: > On 2017-09-18 02:53:03 -0700, Andres Freund wrote: > > On 2017-09-13 23:39:21 -0400, Tom Lane wrote: > > > The real problem in this area, to my mind, is that we're not testing that > > > code --- either end of i

Re: [HACKERS] Latches API on windows

2017-10-09 Thread Andres Freund
e a pointer the the owning processes preexisting latch. Besides solving this issue, it also avoids problems with various routines already waiting on the proclatch. Andres -- Sent from my Android device with K-9 Mail. Please excuse my brevity. -- Sent via pgsql-hackers mailing list (pgsql-hack

Re: [HACKERS] Discussion on missing optimizations

2017-10-08 Thread Andres Freund
On 2017-10-08 17:11:44 -0400, Tom Lane wrote: > Andres Freund writes: > > On 2017-10-08 11:28:09 -0400, Tom Lane wrote: > >> https://commitfest.postgresql.org/15/1001/ > >> The reason that's not in v10 is we haven't been able to convince >

Re: [HACKERS] is possible cache tupledesc templates in execution plan? significant performance issue, maybe bug?

2017-10-08 Thread Andres Freund
On 2017-10-08 18:57:28 +0200, Pavel Stehule wrote: > 2017-10-08 18:44 GMT+02:00 Andres Freund : > > > Hi, > > > > On 2017-10-08 18:36:23 +0200, Pavel Stehule wrote: > > > 2. Lot of used tables are pretty wide - 60, 120, .. columns > > > > > > N

Re: [HACKERS] is possible cache tupledesc templates in execution plan? significant performance issue, maybe bug?

2017-10-08 Thread Andres Freund
physical tlist "optimization". > 2. If is not possible to reduce the number of fields of target list, is > possible to store tupledesc template to plan? We should do that, but it's not a small change. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-h

Re: [HACKERS] Discussion on missing optimizations

2017-10-08 Thread Andres Freund
7;t help in this specific case (no support for UNION, just UNION ALL), but I thought it might be interesting to reference https://medium.com/@uwdb/introducing-cosette-527898504bd6 here. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Slow synchronous logical replication

2017-10-07 Thread Andres Freund
and are there > some plans to work in this direction? This seems to be a question that is a) about a commercial project we don't know much about b) hasn't received a lot of investigation. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Discussion on missing optimizations

2017-10-06 Thread Andres Freund
luated to make it worth the O(n*log(n)) of sorting the lists. Sorting them would be beneficial for other reasons as well, e.g. it improves access patterns for SAO index scans. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Discussion on missing optimizations

2017-10-06 Thread Andres Freund
ck for this without a lot of planner overhead. 7. Provably Empty Sets 8. CHECK Constraints I think some of this should actually work with constraint exclusion turned on. 9. Unneeded Self JOIN Can't remember discussions of this. Greetings, Andres Freund -- Sent via pgsql-hackers ma

Re: [HACKERS] valgrind complains about WaitEventSetWaitBlock on HEAD (fe9ba28e)

2017-10-05 Thread Andres Freund
at very closely, but this does not look normal, > hence this post. I think this might be more a valgrind bug than anything. Note the "sigmask points to to unaddressable byte" and "Address 0x0 is not stack'd, malloc'd or (recently) free'd" bits. It's vali

Re: [HACKERS] fork()-safety, thread-safety

2017-10-05 Thread Andres Freund
On October 5, 2017 5:15:41 PM PDT, Tom Lane wrote: >Andres Freund writes: >> On 2017-10-06 07:59:40 +0800, Craig Ringer wrote: >>> The only thing that gets me excited about a threaded postgres is the >>> ability to have a PL/Java, PL/Mono etc that don't suck.

Re: [HACKERS] fork()-safety, thread-safety

2017-10-05 Thread Andres Freund
On 2017-10-06 07:59:40 +0800, Craig Ringer wrote: > The only thing that gets me excited about a threaded postgres is the > ability to have a PL/Java, PL/Mono etc that don't suck. We could do > some really cool things that just aren't practical right now. Faster parallelism with a lot less reinvent

Re: [HACKERS] fork()-safety, thread-safety

2017-10-05 Thread Andres Freund
ice if we were to make it.) Yea, that's pretty much what I'm thinking too. - Andres -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] fork()-safety, thread-safety

2017-10-05 Thread Andres Freund
ood to move more towards threads, but not at all for > > the reasons mentioned here. > > You don't think eliminating a large difference between handling of WIN32 > vs. POSIX is a good reason? I seems like you'd not really get a much reduced set of differences, just a *

Re: [HACKERS] fork()-safety, thread-safety

2017-10-05 Thread Andres Freund
ant to do is exec-or-exit >since no page copying (COW or otherwise) needs be done when using >vfork(). Not on linux, at least not as of a year or two back. I do think it'd be good to move more towards threads, but not at all for the reasons mentioned here. Greetings,

Re: [HACKERS] Binary search in fmgr_isbuiltin() is a bottleneck.

2017-10-05 Thread Andres Freund
reate an internal-language function with > CREATE FUNCTION, and having it in a separate array shouldn't hurt those > lookups. When'd that be beneficial? fmgr_builtins is pretty much only used for internal-language CREATE FUNCTIONs? In other cases oid bounds + mapping array should filter out

Re: [HACKERS] JIT compiling - v4.0

2017-10-05 Thread Andres Freund
On 2017-10-05 23:43:37 +1300, David Rowley wrote: > On 5 October 2017 at 19:57, Andres Freund wrote: > > Here's some numbers for a a TPC-H scale 5 run. Obviously the Q01 numbers > > are pretty nice in partcular. But it's also visible that the shorter > > query can

Re: [HACKERS] JIT compiling - v4.0

2017-10-04 Thread Andres Freund
On 2017-10-04 11:56:47 +0300, Ants Aasma wrote: > On Wed, Oct 4, 2017 at 9:48 AM, Andres Freund wrote: > > Here's an updated version of the patchset. There's some substantial > > changes here, but it's still very obviously very far from committable as > > a

[HACKERS] Re: [BUGS] Re: [PATCH] BUG #13416: Postgres >= 9.3 doesn't use optimized shared memory on Solaris anymore

2017-10-04 Thread Andres Freund
nto similar config limits on solaris based stuff as it does on linux etc? I think if we want to do this, we should rather go with a patch like https://www.postgresql.org/message-id/20140422121921.gd4...@awork2.anarazel.de Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql

Re: [HACKERS] Binary search in fmgr_isbuiltin() is a bottleneck.

2017-10-04 Thread Andres Freund
On 2017-10-02 15:01:36 -0700, Andres Freund wrote: > On 2017-10-02 17:57:51 -0400, Tom Lane wrote: > > Andres Freund writes: > > > Done that way. It's a bit annoying, because we've to take care to > > > initialize the "unused" part of the array with

Re: [HACKERS] JIT compiling - v4.0

2017-10-03 Thread Andres Freund
g to get any smaller, I'll make smaller adjustments solely via the git tree, rather than full reposts. Greetings, Andres Freund 0001-Rely-on-executor-utils-to-build-targetlist-for-DM.v4.patch.gz Description: application/patch-gzip 0002-WIP-Allow-tupleslots-to-have-a-fixed-tupledesc-us.v

Re: [HACKERS] [BUGS] BUG #14825: enum type: unsafe use?

2017-10-03 Thread Andres Freund
the a dshash table instead of dynahash? Similar to how we're now dealing with the shared typmod registry stuff? It should be fairly simple to now simply add a new struct Session member shared_enum_whatevs_table. Greetings, Andres Freund -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] datetime.h defines like PM conflict with external libraries

2017-10-03 Thread Andres Freund
Hi, On 2017-10-03 16:34:38 -0400, Andrew Dunstan wrote: > On 10/03/2017 03:00 PM, Andres Freund wrote: > > Hi, > > > > In my llvm jit work I'd to > > > > #undef PM > > /* include some llvm headers */ > > #define PM 1 > > > > because

  1   2   3   4   5   6   7   8   9   10   >