Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2018-08-06 Thread Tomas Vondra
On 08/03/2018 04:24 PM, Dean Rasheed wrote: On 17 July 2018 at 14:03, Tomas Vondra wrote: For equalities it's going to be hard. The only thing I can think of at the moment is checking if there are any matching buckets at all, and using that to decide whether to extrapolate the MCV

Re: Standby trying "restore_command" before local WAL

2018-08-06 Thread Tomas Vondra
it? I mean, can't we try the local WAL first, use it if it passes the CRC checks (and possibly some other checks), and only fallback to the remote WAL if it's identified as broken? regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote

Re: buildfarm: could not read block 3 in file "base/16384/2662": read only 0 of 8192 bytes

2018-08-11 Thread Tomas Vondra
On 08/11/2018 04:08 PM, Andres Freund wrote: > Hi, > > On 2018-08-11 15:40:19 +0200, Tomas Vondra wrote: >> For the record, I can actually reproduce this on 9.6 (haven't tried >> older releases, but I suspect it's there too). Instead of using the >> failing subscription

Re: logical decoding / rewrite map vs. maxAllocatedDescs

2018-08-11 Thread Tomas Vondra
On 08/11/2018 04:15 PM, Tom Lane wrote: > Tomas Vondra writes: >>>> On 08/09/2018 07:47 PM, Alvaro Herrera wrote: >>>>> Actually, it seems to me that ApplyLogicalMappingFile is just leaking >>>>> the file descriptor for no good reason. > >>

Re: Some pgq table rewrite incompatibility with logical decoding?

2018-08-09 Thread Tomas Vondra
d to subtransactions (because of triggers with exception blocks). Jeremy, are you able to reproduce the issue locally, using pgq? That would be very valuable. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: logical decoding / rewrite map vs. maxAllocatedDescs

2018-08-09 Thread Tomas Vondra
On 08/09/2018 07:47 PM, Alvaro Herrera wrote: On 2018-Aug-09, Tomas Vondra wrote: I suppose there are reasons why it's done this way, and admittedly the test that happens to trigger this is a bit extreme (essentially running pgbench concurrently with 'vacuum full pg_class' in a loop). I'm

logical decoding / rewrite map vs. maxAllocatedDescs

2018-08-09 Thread Tomas Vondra
. I guess a small local cache of file descriptors would do both things. Not sure about the error handling. Even if we get rid of the file descriptor limit issue, I guess there are other ways why this we can fail here, so we probably need to improve that too. regards -- Tomas Vondra

Re: logical decoding / rewrite map vs. maxAllocatedDescs

2018-08-14 Thread Tomas Vondra
On 08/14/2018 01:49 PM, Tomas Vondra wrote: On 08/13/2018 04:49 PM, Andres Freund wrote: Hi, On 2018-08-13 11:46:30 -0300, Alvaro Herrera wrote: On 2018-Aug-11, Tomas Vondra wrote: Hmmm, it's difficult to compare "bt full" output, but my backtraces look somewhat differen

Re: logical decoding / rewrite map vs. maxAllocatedDescs

2018-08-14 Thread Tomas Vondra
On 08/13/2018 04:49 PM, Andres Freund wrote: Hi, On 2018-08-13 11:46:30 -0300, Alvaro Herrera wrote: On 2018-Aug-11, Tomas Vondra wrote: Hmmm, it's difficult to compare "bt full" output, but my backtraces look somewhat different (and all the backtraces I'm seeing are 100% exactl

Re: [PATCH] Improve geometric types

2018-08-08 Thread Tomas Vondra
On 08/03/2018 02:39 PM, Tomas Vondra wrote: On 08/03/2018 06:40 AM, Kyotaro HORIGUCHI wrote: ... I'm not confident on replacing double to float8 partially in gist code. After the 0002 patch applied, I see most of problematic usage of double or bare arithmetic on dimentional values

Re: Standby trying "restore_command" before local WAL

2018-08-06 Thread Tomas Vondra
On 08/06/2018 06:11 PM, Stephen Frost wrote: Greetings, * Tomas Vondra (tomas.von...@2ndquadrant.com) wrote: On 08/06/2018 05:19 PM, Stephen Frost wrote: * David Steele (da...@pgmasters.net) wrote: I think for the stated scenario (known good standby that has been shutdown gracefully

Re: [PATCH] Improve geometric types

2018-08-17 Thread Tomas Vondra
, because there's now 'else return false' branch. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services diff --git a/src/backend/utils/adt/geo_ops.c b/src/backend/utils/adt/geo_ops.c index e0a9a0fa4f..97b3349ff8 10

Re: [PATCH] Improve geometric types

2018-08-17 Thread Tomas Vondra
ticed is the last few lines from line_interpt_line are >> actually unreachable, because there's now 'else return false' branch. > > Which lines do you mean exactly? I don't see any being unreachable. > Apologies, I got confused - there are no unreachable lines. -- Tomas Vondra

Re: [PATCH] Improve geometric types

2018-08-17 Thread Tomas Vondra
heck for -0 inside float{4,8}_{div,mul}(). I > dropped it on the last set of patches, so the tests were broken. I > get -0 as a result of -x * 0 both on Mac and Linux. > OK, that explains is. I won't have time to get this committed before CF 2018-09, but I'll pick it up in Sept

Re: Some pgq table rewrite incompatibility with logical decoding?

2018-08-28 Thread Tomas Vondra
tially means stepping through RelationMapFilenodeToOid or something like that). regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Some pgq table rewrite incompatibility with logical decoding?

2018-08-28 Thread Tomas Vondra
Fix "base" snapshot handling in logical decoding 0a60a291c9a5b8ecdf44cbbfecc4504e3c21ef49 Add table relcache invalidation to index builds. But it's hard to say if/which of those commits did the trick, without more information. regards -- Tomas Vondra http://www.2ndQ

Re: [PATCH] Improve geometric types

2018-08-17 Thread Tomas Vondra
d > true across the whole buildfarm. > Hmm, yeah. Based on past experience, the powerpc machines are likely to stumble on this. FWIW my understanding is that these failures actually happen in new tests, it's not an issue introduced by this patch series. regards -- Tomas Vondra

Re: patch to allow disable of WAL recycling

2018-08-26 Thread Tomas Vondra
might be caused by almost filling the disk space, or something like that. I'm rerunning this with scale 600. I'm also not sure how much can we extrapolate this to other ZFS configs (I mean, this is a ZFS on a single SSD device, while I'd generally expect ZFS on multiple devices, etc.). regar

Re: GSOC 2018 Project - A New Sorting Routine

2018-07-18 Thread Tomas Vondra
enerate .ods files? > >   > > Regards, > > Kefan > >   > > *From: *Tomas Vondra <mailto:tomas.von...@2ndquadrant.com> > *Sent: *July 18, 2018 2:05 AM > *To: *Andrey Borodin <mailto:x4...@yandex-team.ru> > *Cc: *Peter Geoghegan <mailto:p...@

Re: GSOC 2018 Project - A New Sorting Routine

2018-07-14 Thread Tomas Vondra
he scripts to also set this GUC and restarted the tests on both machines. Let's see what that does. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] plpgsql - additional extra checks

2018-07-14 Thread Tomas Vondra
e. clearly does not help in this regard. This information should be either in errdetail() is deemed useful, or left out entirely. And the errhint() should say something like: errdetail("Make sure the query returns a single row, or use LIMIT 1.") and errdetail("Make sure the query returns t

Re: patch to allow disable of WAL recycling

2018-07-17 Thread Tomas Vondra
and zfsonlinux, and different types of storage would be enough. I don't have any freebsd box available, unfortunately. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2018-07-17 Thread Tomas Vondra
1.33..13424.24 rows=9063 width=0) (actual time=15.551..26.057 > rows=3 loops=3) ... > With compound index statistic estimation is almost equal to real value: > >    ->  Bitmap Heap Scan on foo  (cost=1880.20..13411.66 rows=23310 > width=0) (actual time=38.776..61.050 row

Re: One transaction and several processes

2018-07-18 Thread Tomas Vondra
nfirm it using a two-phase commit. This is more expensive than plain transactions and may not have exactly the same behavior, but it does provide the durability guarantees etc. FWIW this question is for pgsql-general rather than pgsql-hackers. regards -- Tomas Vondra http:/

Re: [HACKERS] logical decoding of two-phase transactions

2018-07-18 Thread Tomas Vondra
On 07/17/2018 08:10 PM, Robert Haas wrote: On Mon, Jul 16, 2018 at 3:25 PM, Tomas Vondra wrote: Oh, right, I forgot the patch also adds the leader into the group, for some reason (I agree it's unclear why that would be necessary, as you pointed out later). But all this is happening while

Re: [HACKERS] logical decoding of two-phase transactions

2018-07-18 Thread Tomas Vondra
On 07/18/2018 04:56 PM, Robert Haas wrote: On Wed, Jul 18, 2018 at 10:08 AM, Tomas Vondra wrote: The problem is you don't know if a transaction does DDL sometime later, in the part that you might not have decoded yet (or perhaps concurrently with the decoding). So I don't see how you could

Re: [HACKERS] logical decoding of two-phase transactions

2018-07-16 Thread Tomas Vondra
other patches touching ProcArray and related bits? regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] logical decoding of two-phase transactions

2018-07-16 Thread Tomas Vondra
On 07/16/2018 08:09 PM, Robert Haas wrote: > On Mon, Jul 16, 2018 at 1:28 PM, Tomas Vondra > wrote: >> I'm not sure I understand. Are you suggesting the process might get killed >> or something, thanks to the CHECK_FOR_INTERRUPTS() call? > > Yes. CHECK_FOR_INTERRU

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2018-07-15 Thread Tomas Vondra
l have to try hacking on this a bit I guess. But there's one obvious question - in what order should we add the clauses? Does it matter at all, or what is the optimal order? We don't need to worry about it now, because we simply consider all clauses at once, but I guess the proposed algorithm is more se

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2018-07-16 Thread Tomas Vondra
On 07/16/2018 02:54 PM, Dean Rasheed wrote: > On 16 July 2018 at 13:23, Tomas Vondra wrote: >>>> The top-level clauses allow us to make such deductions, with deeper >>>> clauses it's much more difficult (perhaps impossible). Because for >>>> examp

Re: New GUC to sample log queries

2018-07-16 Thread Tomas Vondra
uot; or "sampling" without > "rate" will not be very clear. > 1+ to sample_rate It's what auto_explain and pgbench uses, so let's keep the naming consistent. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2018-07-15 Thread Tomas Vondra
On 07/15/2018 04:43 PM, Dean Rasheed wrote: > On 15 July 2018 at 14:29, Tomas Vondra wrote: >> It's quite unclear to me how this algorithm could reliably end up with >> hist_sel=0 (in cases where we already don't end up with that). I mean, >> if a bucket matches the cond

Re: patch to allow disable of WAL recycling

2018-07-16 Thread Tomas Vondra
aybe those are not issues, as long as the result is predictable. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Parallel queries in single transaction

2018-07-16 Thread Tomas Vondra
hinder performance, because the queries will compete for parallel workers, preventing some of them from running in parallel mode. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] logical decoding of two-phase transactions

2018-07-16 Thread Tomas Vondra
On 07/16/2018 06:15 PM, Robert Haas wrote: On Mon, Jul 16, 2018 at 11:21 AM, Tomas Vondra wrote: Overall, I think it's clear the main risk associated with this patch is the decode group code - it touches PROC entries, so a bug may cause trouble pretty easily. So I've focused on this part

Re: [HACKERS] plpgsql - additional extra checks

2018-07-24 Thread Tomas Vondra
On 07/22/2018 10:24 PM, Tomas Vondra wrote: > > > On 07/19/2018 02:50 PM, Pavel Stehule wrote: >> >> >> 2018-07-15 1:38 GMT+02:00 Tomas Vondra > <mailto:tomas.von...@2ndquadrant.com>>: >> >> Hi, >> >> I've been looking

Re: [HACKERS] plpgsql - additional extra checks

2018-07-22 Thread Tomas Vondra
On 07/19/2018 02:50 PM, Pavel Stehule wrote: > > > 2018-07-15 1:38 GMT+02:00 Tomas Vondra <mailto:tomas.von...@2ndquadrant.com>>: > > Hi, > > I've been looking at the version submitted on Thursday, planning to > commit it, but I think it needs

Re: cost_sort() improvements

2018-07-22 Thread Tomas Vondra
tion gives close result to old estimation function on trivial > examples, but ~10% more expensive, and three of regression tests aren't > passed, will look closer later. Patch doesn't include  regression test > changes. Interesting. I wonder if there's a way to address the difference at the lower end? regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: cost_sort() improvements

2018-07-22 Thread Tomas Vondra
P BY > and incremental sort patches. Right. What I think the function estimating the group size could do in case of incremental sort is producing two values - maximum size of the leading group, and maximum group size overall. The first one would be useful for startup cost, the second one for t

Re: patch to allow disable of WAL recycling

2018-07-22 Thread Tomas Vondra
em here, along with the test scripts etc. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2018-07-16 Thread Tomas Vondra
On 07/16/2018 12:16 AM, Tomas Vondra wrote: On 07/15/2018 04:43 PM, Dean Rasheed wrote: On 15 July 2018 at 14:29, Tomas Vondra wrote: It's quite unclear to me how this algorithm could reliably end up with hist_sel=0 (in cases where we already don't end up with that). I mean, if a bucket

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2018-07-16 Thread Tomas Vondra
On 07/15/2018 11:36 AM, Dean Rasheed wrote: On 13 July 2018 at 18:27, Tomas Vondra wrote: I'm not so sure. The issue is that a lot of the MCV deductions depends on whether we can answer questions like "Is there a single match?" or "If we got a match in MCV, do we need to look

Re: Parallel queries in single transaction

2018-07-16 Thread Tomas Vondra
upy two IO stacks(two tablespaces) + 12 cores instead of sequentially 6 and then again 6. Well, sure. But you could just as well open multiple connections and make the queries concurrent that way. Or change the GUC to increase the number of workers for the nightly ETL. regards -- To

Re: [HACKERS] logical decoding of two-phase transactions

2018-07-16 Thread Tomas Vondra
the duration of * the apply_change call below */ I find that rather inconsistent, and I'd say those comments are useless. I suggest to remove all the per-call comments and instead add a comment about the locking into the initial file-level comment, which already explains handling of l

Re: Optimizer misses big in 10.4 with BRIN index

2018-07-25 Thread Tomas Vondra
lause. Something like CREATE INDEX ON data_table (((data_table.data ->> 'tradeDate'::text))::numeric); And then ANALYZE the table again ... regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: GSOC 2018 Project - A New Sorting Routine

2018-07-24 Thread Tomas Vondra
the boxes for some other tests. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [Proposal] Add accumulated statistics for wait event

2018-07-24 Thread Tomas Vondra
in the samples. So if you see the backend reports a particular wait event in 75% of samples, it probably spent 75% time waiting on it. I'm not saying sampling is perfect and it certainly is less convenient than what you propose. regards -- Tomas Vondra http://www.2ndQuadrant.com

Re: How can we submit code patches that implement our (pending) patents?

2018-07-24 Thread Tomas Vondra
with a clean-room re-implementation. Clean room design addresses copyright-related issues, not patents. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [Proposal] Add accumulated statistics for wait event

2018-07-24 Thread Tomas Vondra
on it. So a simple counter is not the most useful thing. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2018-07-17 Thread Tomas Vondra
On 07/17/2018 11:09 AM, Dean Rasheed wrote: On 16 July 2018 at 21:55, Tomas Vondra wrote: ... >> So, how would the proposed algorithm work? Let's start with "a=1": sel(a=1) = 0.1508 I don't see much point in applying the two "b" clauses independently (or how

Re: Some pgq table rewrite incompatibility with logical decoding?

2018-08-31 Thread Tomas Vondra
Hi, On 08/29/2018 12:01 AM, Tomas Vondra wrote: > On 08/28/2018 07:41 PM, Jeremy Finzel wrote: >> Jeremy, are you able to reproduce the issue locally, using pgq? >> That would be very valuable. >> >> >> Tomas et al: >> >> We have

Re: patch to allow disable of WAL recycling

2018-08-31 Thread Tomas Vondra
On 08/27/2018 03:59 AM, Thomas Munro wrote: > On Mon, Aug 27, 2018 at 10:14 AM Tomas Vondra > mailto:tomas.von...@2ndquadrant.com>> wrote: >> zfs (Linux) >> --- >> On scale 200, there's pretty much no difference. > > Speculation: It could be t

Re: patch to allow disable of WAL recycling

2018-09-04 Thread Tomas Vondra
;), NULL }, _recycle, true, NULL, NULL, NULL }, So the default is actually "on" which makes the commented-out config sample rather confusing. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2018-09-09 Thread Tomas Vondra
Hi, On 09/04/2018 04:16 PM, Dean Rasheed wrote: > On 3 September 2018 at 00:17, Tomas Vondra > wrote: >> Hi, >> >> Attached is an updated version of the patch series, adopting a couple of >> improvements - both for MCV lists and histograms. >> >> >&

Re: [PATCH] Incremental sort (was: PoC: Partial sort)

2018-09-06 Thread Tomas Vondra
, which in many > applications applies to a very high percentage of indexes. > I 100% with this. I see incremental sort as a way to run queries with fewer indexes that are less query-specific, while still benefiting from them. Which means lower overhead when writing data, lower disk space u

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2018-09-10 Thread Tomas Vondra
Hi, On 07/18/2018 09:32 AM, Konstantin Knizhnik wrote: > > > On 18.07.2018 02:58, Tomas Vondra wrote: >> On 07/18/2018 12:41 AM, Konstantin Knizhnik wrote: >>> ... >>> >>> Teodor Sigaev has proposed an alternative approach for calculating >>>

Re: cost_sort() improvements

2018-07-08 Thread Tomas Vondra
al vs. external sorts (after all, it's generally foolish to argue with *him* about sorting stuff). But the costing changes discussed here are due to my nagging from the GROUP BY patch (and also the "incremental sort" patch). The internal vs. external costing seems like an independent issue. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: cost_sort() improvements

2018-07-08 Thread Tomas Vondra
dexExprCost.startup + indexExprCost.per_tuple); which is likely to cause issues. Also, why do we need this? if (sortop != InvalidOid) { Oid funcOid = get_opcode(sortop); funcCost = get_func_cost(funcOid); } Presumably if we get to costing sort, we know the pathkey can be sorted and has sort operator, no? Or am I missing something? regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: shared-memory based stats collector

2018-07-10 Thread Tomas Vondra
stats snapshots "perfectly" consistent are likely doomed to fail - the messages are sent over UDP, which does not guarantee delivery etc. So there's always some level of possible inconsistency even with "perfectly consistent" snapshots. regards -- Tomas Vondra h

Re: [HACKERS] plpgsql - additional extra checks

2018-07-09 Thread Tomas Vondra
On 07/03/2018 03:45 PM, Tomas Vondra wrote: > On 03/20/2018 01:35 PM, Tomas Vondra wrote: >> >> >> On 03/20/2018 05:36 AM, Pavel Stehule wrote: >>> >>> >>> 2018-03-19 21:47 GMT+01:00 Tomas Vondra >> <mailto:tomas.von...@2ndquadrant.com>

Re: cost_sort() improvements

2018-07-10 Thread Tomas Vondra
mate the worst case. I don't know if this should be used instead of NG(i), or if those two estimates should be combined in some way. I think estimating the largest group we need to sort should be helpful for the incremental sort patch, so I'm adding Alexander to this thread. regards -- Tomas Vo

Re: Preferring index-only-scan when the cost is equal

2018-07-11 Thread Tomas Vondra
here. Yugo-san, have you observed this behavior on larger tables? regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Binary difference in pg_internal.init after running pg_initdb multiple times

2018-07-12 Thread Tomas Vondra
u describe a scenario where this actually matters? regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: patch to allow disable of WAL recycling

2018-07-12 Thread Tomas Vondra
existing file is faster than writing a new file. That might have been true in the past, but the question is if it's still true on current filesystems. The results posted here suggest it's not true on ZFS, at least. regards -- Tomas Vondra http://www.2ndQuadrant.com Postgre

Re: Preferring index-only-scan when the cost is equal

2018-07-12 Thread Tomas Vondra
On 07/12/2018 03:44 AM, Yugo Nagata wrote: On Wed, 11 Jul 2018 14:37:46 +0200 Tomas Vondra wrote: On 07/11/2018 01:28 PM, Ashutosh Bapat wrote: I don't think we should change add_path() for this. We will unnecessarily check that condition even for the cases where we do not create index

Re: WAL prefetch

2018-07-09 Thread Tomas Vondra
s be a performance loss for nearly all hardware. That's a good point - on regular filesystems with small pages we can just skip FPW (in fact, we should treat them as prefetched), while on ZFS we need to prefetch them. We probably need to make this configurable. regards -- Tomas Von

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2018-07-13 Thread Tomas Vondra
On 07/13/2018 01:19 PM, Dean Rasheed wrote: > On 24 June 2018 at 20:45, Tomas Vondra wrote: >> Attached is a rebased version of this patch series, mostly just fixing >> the breakage caused by reworked format of initial catalog data. >> >> Aside from that, the MCV bu

Re: [HACKERS] log_destination=file

2018-01-22 Thread Tomas Vondra
to stderr, then dup the other one if the config is changed > later). > I think the hook system is a really powerful tool, but it seems a bit awkward to force people to use it to improve performance like this ... That seems like something the core should to out of the box. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Built-in connection pooling

2018-01-22 Thread Tomas Vondra
On 01/22/2018 05:05 PM, Konstantin Knizhnik wrote: > > > On 19.01.2018 20:28, Tomas Vondra wrote: >>> >>> With pgbouncer you will never be able to use prepared statements which >>> slows down simple queries almost twice (unless my patch with >>> au

Re: WIP: BRIN multi-range indexes

2018-01-23 Thread Tomas Vondra
ch series along those lines in the next few days. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Parallel Aggregates for string_agg and array_agg

2018-03-11 Thread Tomas Vondra
On 03/11/2018 07:31 AM, David Rowley wrote: > On 11 March 2018 at 12:11, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: >> On 03/05/2018 04:51 AM, David Rowley wrote: >>> On 5 March 2018 at 04:54, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: >>> C

Re: Online enabling of checksums

2018-02-28 Thread Tomas Vondra
I don't think "-o" is incorrect. Secondly, the SGML docs actually say: -o relfilenode Only validate checksums in the relation with specified relfilenode. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: RFC: Add 'taint' field to pg_control.

2018-02-28 Thread Tomas Vondra
pg_upgrade was used > > What do others think? > Isn't the uncertainty usually that you know one of these things happened on the cluster, but you don't know if that's the actual root cause? That's my experience, at least. That being said, I'm not strongly opposed to adding such field.

Re: Parallel Aggregates for string_agg and array_agg

2018-03-15 Thread Tomas Vondra
On 03/11/2018 12:10 PM, Tomas Vondra wrote: > > ... > > Yeah, seems fine to me. I wonder what else would be needed before > switching the patch to RFC. I plan to do that after a bit more > testing sometime early next week, unless someone objects. > I've done more testing

Re: [HACKERS] plpgsql - additional extra checks

2018-03-15 Thread Tomas Vondra
trict SELECT INTO queries. So let's just revert to the old wording. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [PATCH] btree_gin, add support for uuid, bool, name, bpchar and anyrange types

2018-03-15 Thread Tomas Vondra
On 03/08/2018 10:20 AM, Matheus de Oliveira wrote: > Hi all. > > Em 4 de mar de 2018 16:00, "Tomas Vondra" <tomas.von...@2ndquadrant.com > <mailto:tomas.von...@2ndquadrant.com>> escreveu: > > > 1) I personally am not that sure GIN indexes on rang

Re: [HACKERS] Lazy hash table for XidInMVCCSnapshot (helps Zipfian a bit)

2018-03-15 Thread Tomas Vondra
On 03/10/2018 03:11 AM, Yura Sokolov wrote: > 08.03.2018 03:42, Tomas Vondra пишет: >> On 03/06/2018 06:23 AM, Yura Sokolov wrote: >>> 05.03.2018 18:00, Tom Lane пишет: >>>> Tomas Vondra <tomas.von...@2ndquadrant.com> writes: >>>>> Snapshots are

Re: [HACKERS] [PATCH] Incremental sort

2018-03-15 Thread Tomas Vondra
lso gets paralellyzed. In the same way as in previous > case, disk writes/reads during external sort are overestimated, > because they actually use OS cache. I would also say that it's not > necessary wrong decision of optimizer, because doing this work in > single backend may consume less resources despite being overall > slower. > Yes, that seems like a likely explanation too. I agree those don't seem like an issue in the Incremental Sort patch, but like a more generic costing problems. Thanks for looking into the benchmark results. -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2018-03-10 Thread Tomas Vondra
On 03/10/2018 02:08 PM, Mark Dilger wrote: > >> On Mar 3, 2018, at 2:40 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> >> wrote: >> >> An updated patch version, fixing the breakage caused by fd1a421fe6 >> twiddling with pg_proc. > > Hi T

Re: Sample values for pg_stat_statements

2018-03-10 Thread Tomas Vondra
y (only showing data for user's own queries). Well, we could keep per-user samples, but that might considerably inflate the file size. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: remove pg_class.relhaspkey

2018-03-10 Thread Tomas Vondra
ally at all. So +1 to get rid of it. For the other flags we would probably need to test what impact would it have (e.g. table with no indexes, many indexes on other tables, and something calling get_relation_info a lot). But this patch proposes to remove only relhaspkey. [1] https://www.postgresql.org/message-id/CA%2BTgmoYJu24Y8uUAJ4zeQAhoYxLmFxcy%2B5Hdij9ehjoxKo3j3g%40mail.gmail.com regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: Parallel Aggregates for string_agg and array_agg

2018-03-10 Thread Tomas Vondra
On 03/05/2018 04:51 AM, David Rowley wrote: > On 5 March 2018 at 04:54, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: >> 1) There seems to be forgotten declaration of initArrayResultInternal in >> arrayfuncs.c. I suppose you've renamed it to initArrayResult

Re: [PROPOSAL] timestamp informations to pg_stat_statements

2018-03-10 Thread Tomas Vondra
y something based on time of the first/last execution would be better than the current algorithm. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2018-03-14 Thread Tomas Vondra
before we had proper cache invalidations in RemoveStatistics I think. > Another XXX comment says > + * XXX All the memory is allocated in a single chunk, so that the caller > + * can simply pfree the return value to release all of it. > > but I would say just remove the XXX and l

Re: Additional Statistics Hooks

2018-03-15 Thread Tomas Vondra
EATE STATISTICS so that it handles things like CREATE STATISTICS s ON date_trunc('day', column) FROM table As David mentioned elsewhere in this thread, this was considered before CREATE STATISTICS was introduced in PG11 and it's why the features is called 'extended' and not 'multivariate'

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-08 Thread Tomas Vondra
it's obviously silly, or do we need to implement some protection? That being said the volatility check seems reasonable to me (and i would not expect it to be a huge amount of code). regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [PROPOSAL] Shared Ispell dictionaries

2018-03-07 Thread Tomas Vondra
he result in a file, and then mmap it when needed, without the expensive preprocessing. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [PATCH] btree_gin, add support for uuid, bool, name, bpchar and anyrange types

2018-04-04 Thread Tomas Vondra
On 03/30/2018 10:51 PM, Matheus de Oliveira wrote: > Hi all. > > On Wed, Mar 21, 2018 at 1:47 PM, Tomas Vondra > <tomas.von...@2ndquadrant.com <mailto:tomas.von...@2ndquadrant.com>> wrote: > > > Do you plan to post an updated version of the patc

Re: Online enabling of checksums

2018-04-06 Thread Tomas Vondra
On 04/06/2018 08:13 PM, Andres Freund wrote: > On 2018-04-06 19:59:17 +0200, Tomas Vondra wrote: >> On 04/06/2018 07:46 PM, Andres Freund wrote: >>>> Sure. But what would that be? I can't think of anything. A process that >>>> modifies a buffer (or any other piece

Re: Online enabling of checksums

2018-04-06 Thread Tomas Vondra
On 04/06/2018 08:13 PM, Andres Freund wrote: > On 2018-04-06 19:59:17 +0200, Tomas Vondra wrote: >> On 04/06/2018 07:46 PM, Andres Freund wrote: >>>> Sure. But what would that be? I can't think of anything. A process that >>>> modifies a buffer (or any oth

Re: Parallel Aggregates for string_agg and array_agg

2018-04-05 Thread Tomas Vondra
On 04/05/2018 05:41 AM, David Rowley wrote: > Hi Tomas, > > Thanks for taking another look. > > On 5 April 2018 at 07:12, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: >> Seems fine to me, although we should handle the anyarray case too, I >> guess. Tha

Re: Parallel Aggregates for string_agg and array_agg

2018-04-05 Thread Tomas Vondra
On 04/05/2018 09:10 PM, Tels wrote: > Moin, > > On Wed, April 4, 2018 11:41 pm, David Rowley wrote: >> Hi Tomas, >> >> Thanks for taking another look. >> >> On 5 April 2018 at 07:12, Tomas Vondra <tomas.von...@2ndquadrant.com> >> wrote: >&g

Re: Online enabling of checksums

2018-04-06 Thread Tomas Vondra
ad as > throwing a barrier in there? > Perhaps the easiest thing we could do is walk shared buffers and do LockBufHdr/UnlockBufHdr, which would guarantee no session is the process of writing out a buffer with possibly stale checksum flag. Of course, it's a bit brute-force-ish, but it's not that different from the waits for running transactions and temporary tables. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: pgsql: Validate page level checksums in base backups

2018-04-10 Thread Tomas Vondra
On 04/10/2018 11:24 PM, Tomas Vondra wrote: > Hi, > > I think there's a bug in sendFile(). We do check checksums on all pages > that pass this LSN check: > > /* > * Only check pages which have not been modified since the > * start of the base backup. Ot

Re: pgsql: Validate page level checksums in base backups

2018-04-10 Thread Tomas Vondra
ure it's worth it. I've found this by fairly trivial stress testing - running pgbench and pg_basebackup in a loop. It was failing pretty reliably (~75% of runs). With the proposed change I see no further failures. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development

Re: Online enabling of checksums

2018-04-05 Thread Tomas Vondra
On 4/5/18 11:07 AM, Magnus Hagander wrote: > > > On Wed, Apr 4, 2018 at 12:11 AM, Tomas Vondra > <tomas.von...@2ndquadrant.com <mailto:tomas.von...@2ndquadrant.com>> wrote: > > ... > > It however still fails to initialize the attempts field

Re: [HACKERS] logical decoding of two-phase transactions

2018-04-05 Thread Tomas Vondra
test cases > (which AFAIK we don't plan to commit in that state) which demonstrate > concurrent rollback interlocking with the lock/unlock APIs. The first > ELOG was enough to catch the interaction. If we think these elogs > should be present in the code, then yes, I can add it else

Re: [PATCH] btree_gin, add support for uuid, bool, name, bpchar and anyrange types

2018-04-05 Thread Tomas Vondra
sts > fail: > >   CREATE EXTENSION btree_gin; > + ERROR:  could not find function "gin_extract_value_uuid" in file > "/usr/local/pgsql/lib/btree_gin.so" > > -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Su

Re: [HACKERS] logical decoding of two-phase transactions

2018-04-04 Thread Tomas Vondra
tweak the decodeLocked flag. 7) I propose minor changes to a couple of comments. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services diff --git a/src/backend/replication/logical/logical.c b/src/backend/r

Re: some last patches breaks plan cache

2018-04-04 Thread Tomas Vondra
On 04/01/2018 10:01 AM, Pavel Stehule wrote: > > > 2018-04-01 1:00 GMT+02:00 Tomas Vondra <tomas.von...@2ndquadrant.com > <mailto:tomas.von...@2ndquadrant.com>>: > > > > On 03/31/2018 08:28 PM, Tomas Vondra wrote: > > > > &g

<    1   2   3   4   5   6   7   8   9   10   >