Re: SET TRANSACTION in PL/pgSQL

2018-04-04 Thread Tomas Vondra
re like the rest of the commands. Most of the patch is boilerplate to support the grammar, and the one interesting piece exec_stmt_set seems fine to me. Barring any objections, I'll mark it as RFC tomorrow morning. regards -- Tomas Vondra http://www.2ndQuadrant.com Postgre

Re: some last patches breaks plan cache

2018-04-04 Thread Tomas Vondra
On 04/04/2018 07:54 PM, Tom Lane wrote: > Tomas Vondra <tomas.von...@2ndquadrant.com> writes: >> This should do the trick - I've failed to realize exec_stmt_call may >> exit by calling elog(ERROR) too, in which case the plan pointer was not >> reset. > >>

Re: Parallel Aggregates for string_agg and array_agg

2018-04-04 Thread Tomas Vondra
On 03/31/2018 04:42 PM, David Rowley wrote: > On 30 March 2018 at 02:55, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: >> On 03/29/2018 03:09 PM, David Rowley wrote: >>> I meant to mention earlier that I coded >>> agg_args_have_sendreceive_funcs

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

2018-04-09 Thread Tomas Vondra
gnificantly - things like thin provisioning are likely to get even more common, increasing the incidence of these issues. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

2018-04-09 Thread Tomas Vondra
On 04/09/2018 10:04 PM, Andres Freund wrote: > Hi, > > On 2018-04-09 21:54:05 +0200, Tomas Vondra wrote: >> Isn't the expectation that when a fsync call fails, the next one will >> retry writing the pages in the hope that it succeeds? > > Some people expect that, I

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

2018-04-09 Thread Tomas Vondra
teback of that page fails (and gets discarded) (4) we attempt to modify the page again, but we read the stale version (5) we modify the stale version, writing the change to WAL The standby will get the full-page, and then a WAL from the stale page version. That doesn't seem like a story with a happy end, I guess. But I might be easily missing some protection built into the WAL ... regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

2018-04-09 Thread Tomas Vondra
ase, that certainly does not count as data corruption spreading from the master to standby. -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

2018-04-09 Thread Tomas Vondra
On 04/09/2018 04:22 PM, Anthony Iliopoulos wrote: > On Mon, Apr 09, 2018 at 03:33:18PM +0200, Tomas Vondra wrote: >> >> We already have dirty_bytes and dirty_background_bytes, for example. I >> don't see why there couldn't be another limit defining how much dirty >> da

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

2018-04-09 Thread Tomas Vondra
ible to do what you suggested, and simply mark the inode as failed. In which case the next fsync can't possibly retry the writes (e.g. after freeing some space on thin-provisioned system), but we'd get reliable failure mode. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] [PATCH] Incremental sort

2018-04-07 Thread Tomas Vondra
test patch version, unfortunately, certainly not before the CF end. So I guess the ultimate review / decision is up to you ... regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] [PATCH] Incremental sort

2018-04-07 Thread Tomas Vondra
or even just might need, follow-on work. > +1 to that FWIW I'm willing to spend some time on the patch for PG12, particularly on the planner / costing part. The potential gains are too interesting. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2018-04-07 Thread Tomas Vondra
keep reviewing it, and I hope Tomas will persist, so that it has a > better chance in PG12. > Thank you for the effort and for the reviews, of course. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

2018-04-09 Thread Tomas Vondra
npower to implement it (portable, reliable, handling different types of storage, ...). One has to wonder how many applications actually use this correctly, considering PostgreSQL cares about data durability/consistency so much and yet we've been misunderstanding how it works for 20+ years. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

2018-04-09 Thread Tomas Vondra
so we'd need to understand which devices are related to PostgreSQL. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: PostgreSQL's handling of fsync() errors is unsafe and risks data loss at least on XFS

2018-04-09 Thread Tomas Vondra
ases demonstrating this issue. It kinda reminds me the wisdom that not seeing planes with bullet holes in the engine does not mean engines don't need armor [1]. [1] https://medium.com/@penguinpress/an-excerpt-from-how-not-to-be-wrong-by-jordan-ellenberg-664e708cfc3d regards -- Tomas Vondra

Re: [HACKERS] [PATCH] Incremental sort

2018-04-06 Thread Tomas Vondra
On 04/06/2018 01:43 AM, Alexander Korotkov wrote: > Hi! > > On Tue, Apr 3, 2018 at 2:10 PM, Tomas Vondra > <tomas.von...@2ndquadrant.com <mailto:tomas.von...@2ndquadrant.com>> wrote: > > I think solving this may be fairly straight-forward. Essentially, until

Re: Online enabling of checksums

2018-04-06 Thread Tomas Vondra
On 04/06/2018 07:22 PM, Andres Freund wrote: > Hi, > > On 2018-04-06 14:34:43 +0200, Tomas Vondra wrote: >>> Oh, that's not my intention either -- I just wanted to make sure I >>> was thinking about the same issue you were. > >> I agree we shouldn't rely on ch

Re: Online enabling of checksums

2018-04-06 Thread Tomas Vondra
On 04/06/2018 07:46 PM, Andres Freund wrote: > On 2018-04-06 19:40:59 +0200, Tomas Vondra wrote: >> In any case, I wouldn't call LockBufHdr/UnlockBufHdr a "side channel" >> interlock. It's a pretty direct and intentional interlock, I think. > > I

Re: very slow queries when max_parallel_workers_per_gather is higher than zero

2018-04-16 Thread Tomas Vondra
Hash  (cost=27.35..27.35 rows=7 width=4) (actual > time=0.089..0.089 rows=7 loops=3) >     Buckets: 1024  Batches: 1  Memory Usage: 9kB > > What happens when you disable sequential scans on pg10? regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: very slow queries when max_parallel_workers_per_gather is higher than zero

2018-04-16 Thread Tomas Vondra
Apologies, the reduced query was missing a where condition on id_week: SELECT count(*) FROM f_ticketupdate_aad5jtwal0ayaax AS f INNER JOIN dwh_dm_aabv5kk9rxac4lz_aaonw7nchsan2n1_aad8xhr0m_aaewg8j61iagl1z AS d ON (f.dt_event_id = d.id) WHERE ( 6171 = d."id_euweek" ) regards -- To

Re: very slow queries when max_parallel_workers_per_gather is higher than zero

2018-04-16 Thread Tomas Vondra
e issue might be that the join is misestimated - it's expected to produce ~29k rows, but produces 0. Can you check if this query has the same issue? It's just the problematic join, and it should be simpler to investigate: SELECT count(*) FROM f_ticketupdate_aad5jtwal0ayaax AS f INNER JOIN dwh_dm_aabv5kk9rxac4lz_aaonw7nchsan2n1_aad8xhr0m_aaewg8j61iagl1z AS d ON (f.dt_event_id = d.id) regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: very slow queries when max_parallel_workers_per_gather is higher than zero

2018-04-16 Thread Tomas Vondra
ipled way to fix this - if the cost difference gets a bit larger (or if you increase the number of parallel workers) it's probably going to use the parallel plan again. Obviously, PostgreSQL 9.5 doesn't have parallel queries, so it does not have a chance of making this mistake. regards -- Tom

Re: auto_explain and parallel queries issue

2018-04-16 Thread Tomas Vondra
W this is the same issue that I reported in 2016: https://www.postgresql.org/message-id/3f62f24e-51b3-175c-9222-95f25fd2a9d6%402ndquadrant.com regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2018-03-26 Thread Tomas Vondra
bugs similar to those you found. Thanks for the feedback so far! regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services 0001-multivariate-MCV-lists-20180327.patch.gz Description: application/gzip 0002-multiv

Re: Online enabling of checksums

2018-03-27 Thread Tomas Vondra
On 03/27/2018 08:56 AM, Magnus Hagander wrote: > On Mon, Mar 26, 2018 at 10:09 PM, Tomas Vondra > <tomas.von...@2ndquadrant.com <mailto:tomas.von...@2ndquadrant.com>> wrote: > > Hi, > > I see enable_data_checksums() does this: > >     if (cost_lim

Re: pg_class.reltuples of brin indexes

2018-03-27 Thread Tomas Vondra
case VACUUM/ANALYZE don't flap, it's just the initial reltuples estimate set by CREATE INDEX. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2018-03-27 Thread Tomas Vondra
On 03/27/2018 07:34 PM, Tomas Vondra wrote: > On 03/27/2018 07:03 PM, Dean Rasheed wrote: >> On 27 March 2018 at 14:58, Dean Rasheed <dean.a.rash...@gmail.com> wrote: >>> On 27 March 2018 at 01:36, Tomas Vondra <tomas.von...@2ndquadrant.com> >>> wrote:

Re: Parallel Aggregates for string_agg and array_agg

2018-03-27 Thread Tomas Vondra
at for example includes jsonb_agg, which some people already mentioned as another candidate for enabling parallelism. Not great, I guess. Tom's solution is much more focused - it recognizes that array_agg is a fairly rare case where the (actual) argument type gets stored in the trans type, and de

Re: [HACKERS] AdvanceXLInsertBuffer vs. WAL segment compressibility

2018-03-27 Thread Tomas Vondra
en > encountering unwritten pages at the end of a segment? > > I don't mean that as snark; it's just nonobvious to me and I might need > a little nudge where to look. > > -Chap > regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: PL/pgSQL nested CALL with transactions

2018-03-27 Thread Tomas Vondra
On 03/24/2018 03:14 PM, Peter Eisentraut wrote: > On 3/22/18 11:50, Tomas Vondra wrote: > >> 2) spi.c >> >> I generally find it confusing when there are negative flags, which are >> then negated whenever used. That is pretty the "no_snapshots" case, >

Re: Parallel Aggregates for string_agg and array_agg

2018-03-27 Thread Tomas Vondra
ys is that rather than giving up on this > altogether, we should shelve it till we have less stupid > planner+executor behavior for ORDER BY inside aggregates. That's > been on the to-do list for a long while, but not risen to the top > ... That would be a valid choice if there was a

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2018-03-27 Thread Tomas Vondra
On 03/27/2018 04:58 PM, Dean Rasheed wrote: > On 27 March 2018 at 01:36, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: >> BTW I think there's a bug in handling the fullmatch flag - it should not >> be passed to AND/OR subclauses the way it is, because then >> >

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2018-03-27 Thread Tomas Vondra
On 03/27/2018 07:03 PM, Dean Rasheed wrote: > On 27 March 2018 at 14:58, Dean Rasheed <dean.a.rash...@gmail.com> wrote: >> On 27 March 2018 at 01:36, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: >>> 4) handling of NOT clauses in MCV lists (and in histograms) >

Re: Parallel Aggregates for string_agg and array_agg

2018-03-28 Thread Tomas Vondra
On 03/28/2018 05:28 AM, Tom Lane wrote: > Tomas Vondra <tomas.von...@2ndquadrant.com> writes: >> On 03/27/2018 04:51 AM, David Rowley wrote: >>> Seems I didn't mean "trans types". I should have said aggregate >>> function argument types. > >>

Re: PL/pgSQL nested CALL with transactions

2018-03-28 Thread Tomas Vondra
On 03/28/2018 02:54 PM, Peter Eisentraut wrote: > On 3/27/18 20:43, Tomas Vondra wrote: >>>> 3) utility.c >>>> >>>> I find this condition rather confusing: >>>> >>>> (!(context == PROCESS_UTILITY_TOP

Re: [HACKERS] [PATCH] Incremental sort

2018-03-31 Thread Tomas Vondra
ould be done when costing the plan (i.e. in costsize.c) or creating the plan. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services >From b4c2a801aa802e71b8a822f5e1cd163463e97e26 Mon Sep 17 00:00:00 2001

Re: some last patches breaks plan cache

2018-03-31 Thread Tomas Vondra
> proc(10, r, 20)": SPI_ERROR_ARGUMENT > CONTEXT:  PL/pgSQL function testproc() line 4 at CALL > postgres=# > > second call fails Yeah. d92bc83c48bdea9888e64cf1e2edbac9693099c9 seems to have broken this :-/ regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: some last patches breaks plan cache

2018-03-31 Thread Tomas Vondra
On 03/31/2018 07:56 PM, Tomas Vondra wrote: > On 03/31/2018 07:38 PM, Pavel Stehule wrote: >> Hi >> >> CREATE OR REPLACE PROCEDURE public.proc(a integer, INOUT b integer, c >> integer) >>  LANGUAGE plpgsql >> AS $procedure$ >> begin >>   b := a

Re: some last patches breaks plan cache

2018-03-31 Thread Tomas Vondra
On 03/31/2018 08:28 PM, Tomas Vondra wrote: > > > On 03/31/2018 07:56 PM, Tomas Vondra wrote: >> On 03/31/2018 07:38 PM, Pavel Stehule wrote: >>> Hi >>> >>> CREATE OR REPLACE PROCEDURE public.proc(a integer, INOUT b integer, c >>> integer)

Re: Online enabling of checksums

2018-03-30 Thread Tomas Vondra
Hi, I've just noticed the patch does not update src/backend/storage/page/README which is in fact about checksums. Most of it remains valid, but it also mentions that currently it's an initdb-time choice. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL

Re: Online enabling of checksums

2018-03-30 Thread Tomas Vondra
way how to crash without updating the result for the launcher, so checksums may end up being enabled anyway. Not great, I guess :-( regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2018-04-01 Thread Tomas Vondra
ing clauselist_selectivity() or approximating s1 to the non-MCV part. This needs a bit of care to prevent infinite loops. Of course, with 0002 this changes slightly, because we may try using a histogram to estimate the non-MCV part. But that's just an extra step right before (2a). regards -- To

Re: Changing WAL Header to reduce contention during ReserveXLogInsertLocation()

2018-03-29 Thread Tomas Vondra
of WAL records. To defeat this, this would need to happen: a) the WAL record gets written to a different location b) the xl_curr gets corrupted in sync with (a) c) the WAL checksum gets corrupted in sync with (b) d) the record overwrites existing record (same size/boundaries) That seems

Re: [HACKERS] path toward faster partition pruning

2018-03-29 Thread Tomas Vondra
opy(pc_steps)); ^ partprune.c:936:14: note: ‘pc_steps’ was declared here List *pc_steps; ^~~~ All of PostgreSQL successfully made. Ready to install. So even if it's not a bug, we probably need to fix the code somehow. regards -- Tomas Vondra http://www.2nd

Re: [HACKERS] logical decoding of two-phase transactions

2018-03-29 Thread Tomas Vondra
leader when deciding to wait. 3) LogicalDecodeRemoveTransaction - It's not clear to me what happens when a decoding backend gets killed between LogicalLockTransaction/LogicalUnlockTransaction. Doesn't that mean LogicalDecodeRemoveTransaction will get stuck, because the proc is still in the decoding group

Re: [HACKERS] logical decoding of two-phase transactions

2018-03-29 Thread Tomas Vondra
On 03/29/2018 11:58 PM, Andres Freund wrote: > On 2018-03-29 23:52:18 +0200, Tomas Vondra wrote: >>> I have added details about this in src/backend/storage/lmgr/README as >>> suggested by you. >>> >> >> Thanks. I think the README is a good sta

Re: Changing WAL Header to reduce contention during ReserveXLogInsertLocation()

2018-03-29 Thread Tomas Vondra
On 03/29/2018 11:18 PM, Tom Lane wrote: > Tomas Vondra <tomas.von...@2ndquadrant.com> writes: >> If each WAL record has xl_curr, then we know to which position the >> record belongs (after verifying the checksum). And we do know the size >> of each WAL record, so

Re: Changing WAL Header to reduce contention during ReserveXLogInsertLocation()

2018-03-29 Thread Tomas Vondra
On 03/29/2018 08:02 PM, Robert Haas wrote: > On Thu, Mar 29, 2018 at 1:13 PM, Tomas Vondra > <tomas.von...@2ndquadrant.com> wrote: >> On 03/29/2018 06:42 PM, Tom Lane wrote: >>> The long and the short of it is that this is a very dangerous-looking >>&

Re: ALTER TABLE ADD COLUMN fast default

2018-03-29 Thread Tomas Vondra
ion > that does a full table rewrite, which often enough is impractical. > I don't quite see how moving that gets solved by moving the info into a different catalog? We would need to fetch it whenever attribute meta-data from pg_attribute are loaded. cheers -- Tomas Vondra

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2018-03-26 Thread Tomas Vondra
On 03/26/2018 12:31 PM, Dean Rasheed wrote: > On 18 March 2018 at 23:57, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: >> Attached is an updated version of the patch series, addressing issues >> pointed out by Alvaro. > > I'm just starting to look at this

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2018-03-26 Thread Tomas Vondra
On 03/26/2018 09:01 PM, Dean Rasheed wrote: > On 18 March 2018 at 23:57, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: >> Attached is an updated version of the patch series, addressing issues >> pointed out by Alvaro. > > I've jus

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2018-03-26 Thread Tomas Vondra
On 03/26/2018 06:21 PM, Dean Rasheed wrote: > On 26 March 2018 at 14:08, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: >> On 03/26/2018 12:31 PM, Dean Rasheed wrote: >>> A wider concern I have is that I think this function is trying to be >>> too clever by o

Re: [HACKERS] logical decoding of two-phase transactions

2018-04-03 Thread Tomas Vondra
memory limit. It needs to be allocated ad-hoc only when actually needed. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] logical decoding of two-phase transactions

2018-04-03 Thread Tomas Vondra
me 2PC transaction, it can't possibly proceed because the subscriber won't be able to do PREPARE TRANSACTION. So I think we need a subscription parameter to enable/disable this, defaulting to 'disabled'. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7

Re: [HACKERS] logical decoding of two-phase transactions

2018-04-03 Thread Tomas Vondra
On 04/03/2018 04:37 PM, Alvaro Herrera wrote: > Tomas Vondra wrote: > >> Yes, that is a good point actually - we need to test that replication >> between PG10 and PG11 works correctly, i.e. that the protocol version is >> correctly negotiated, and features are disabled

Re: [HACKERS] logical decoding of two-phase transactions

2018-04-03 Thread Tomas Vondra
On 04/03/2018 04:07 PM, Stas Kelvich wrote: > > >> On 3 Apr 2018, at 16:56, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: >> >> >> So I think we need a subscription parameter to enable/disable this, >> defaulting to 'disabled’. > > +1 &

Re: Online enabling of checksums

2018-04-03 Thread Tomas Vondra
On 04/03/2018 02:05 PM, Magnus Hagander wrote: > On Sun, Apr 1, 2018 at 2:04 PM, Magnus Hagander <mag...@hagander.net > <mailto:mag...@hagander.net>> wrote: > > On Sat, Mar 31, 2018 at 5:38 PM, Tomas Vondra > <tomas.von...@2ndquadrant.com <mailto:tomas.von

Re: [PROPOSAL] Shared Ispell dictionaries

2018-03-31 Thread Tomas Vondra
le, e.g. based on XID. Imagine that we note the XID of the transaction removing the dictionary, or perhaps XID of the most recent running transaction. Then we could use this to decide if all running transactions actually see the DROP, and we could remove the tombstone. regards -- Tomas Vond

Re: Online enabling of checksums

2018-03-31 Thread Tomas Vondra
Hi, On 03/31/2018 02:02 PM, Magnus Hagander wrote: > On Sat, Mar 31, 2018 at 2:08 AM, Tomas Vondra > <tomas.von...@2ndquadrant.com <mailto:tomas.von...@2ndquadrant.com>> wrote: > > ... > > (a) Should not be difficult to do, I think. We don't have relation_ope

Re: Parallel Aggregates for string_agg and array_agg

2018-03-29 Thread Tomas Vondra
On 03/29/2018 05:49 AM, David Rowley wrote: > On 29 March 2018 at 03:05, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: >> On 03/28/2018 03:54 PM, Tom Lane wrote: >>> I had in mind to look at exprType() of the argument. >> >> Right, I'm fine with that.

Re: Parallel Aggregates for string_agg and array_agg

2018-03-29 Thread Tomas Vondra
On 03/29/2018 03:09 PM, David Rowley wrote: > On 30 March 2018 at 02:00, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: >> On 03/29/2018 05:49 AM, David Rowley wrote: >>> Attached is v9, which is based on Tom's v8 but includes the new tests >>> and I thin

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2018-03-29 Thread Tomas Vondra
On 03/29/2018 02:27 AM, Dean Rasheed wrote: > On 28 March 2018 at 15:50, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: >> After thinking about this a bit more, I'm not sure if updating the info >> based on recursive calls makes sense. The fullmatch flag was supposed

Re: Function to track shmem reinit time

2018-03-29 Thread Tomas Vondra
On 03/28/2018 08:55 PM, David Steele wrote: > On 3/4/18 11:17 AM, Tomas Vondra wrote: >> >> Furthermore, the patch is yet another victim of fd1a421fe - fixing the >> pg_proc entries is trivial, but a new version is needed. >> >> I'd also like to see an exam

Re: segfault due to invalid cached plan

2018-03-28 Thread Tomas Vondra
reProcessUtility, which is part of cstore_fdw, not PostgreSQL. So I guess the right place to report the issue is https://github.com/citusdata/cstore_fdw regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2018-03-28 Thread Tomas Vondra
On 03/28/2018 04:12 PM, Dean Rasheed wrote: > On 28 March 2018 at 01:34, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: >> Attached is a patch fixing this. In the end I've decided to keep both >> branches - one handling boolean Vars and one for NOT clauses. I think >&g

Re: Allow workers to override datallowconn

2018-03-29 Thread Tomas Vondra
y ASCII-only so the encoding is not an issue, but perhaps there are other issues (Tom suggested this merely as an example). So I agree with Magnus, the extra flag seems to be perfectly fine for bgworkers, and I'd leave the more generic solution for a future patch if anyone wants to hack on it. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] [PATCH] Incremental sort

2018-03-28 Thread Tomas Vondra
On 03/28/2018 05:12 PM, Alexander Korotkov wrote: > On Wed, Mar 28, 2018 at 4:44 PM, Tomas Vondra > <tomas.von...@2ndquadrant.com <mailto:tomas.von...@2ndquadrant.com>> wrote: > > On 03/28/2018 03:28 PM, Teodor Sigaev wrote: > >> BTW, patch had con

Re: Online enabling of checksums

2018-03-26 Thread Tomas Vondra
Hi, I see enable_data_checksums() does this: if (cost_limit <= 0) ereport(ERROR, (errmsg("cost limit must be a positive value"))); Is there a reason not to allow -1 (no limit), just like for vacuum_cost? regards -- Tomas Vondra

Re: Parallel Aggregates for string_agg and array_agg

2018-03-26 Thread Tomas Vondra
- it's been like that since forever. Also, how is this different from ORDER BY clause? If a user does not specify an ORDER BY clause, I don't think we'd care very much about changes to output ordering due to plan changes, for example. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: MCV lists for highly skewed distributions

2018-03-16 Thread Tomas Vondra
timate gets slightly worse (say, by 10%) compared to current master. So I think that's nice. I'm open to running additional tests with other distributions, table sizes etc. if needed. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] [PATCH] Incremental sort

2018-03-16 Thread Tomas Vondra
On 03/16/2018 09:47 AM, Alexander Korotkov wrote: > On Fri, Mar 16, 2018 at 5:12 AM, Tomas Vondra > <tomas.von...@2ndquadrant.com <mailto:tomas.von...@2ndquadrant.com>> wrote: > > I agree those don't seem like an issue in the Incremental Sort patch, > but

Re: MCV lists for highly skewed distributions

2018-03-17 Thread Tomas Vondra
On 03/17/2018 08:32 PM, Dean Rasheed wrote: > On 17 March 2018 at 18:40, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: >> Currently, analyze_mcv_list only checks if the frequency of the >> current item is significantly higher than the non-MCV selectivity. >> My

Re: strange failure in plpgsql_control tests (on fulmar, ICC 14.0.3)

2018-03-17 Thread Tomas Vondra
that triggers the issue is from December 31. Maybe the issue was there but we were lucky not to trip on it before. Anyway, I can confirm that the fix suggested by Tom does the trick (well, at least on Fulmar, which is running icc 14.0.3). I've also disassembled exec_stmt_fori both with and without the

Re: strange failure in plpgsql_control tests (on fulmar, ICC 14.0.3)

2018-03-17 Thread Tomas Vondra
On 03/17/2018 07:20 PM, Tom Lane wrote: > Tomas Vondra <tomas.von...@2ndquadrant.com> writes: >> Not sure, but the backbranches seem to be working fine, and the commit >> that triggers the issue is from December 31. Maybe the issue was there >> but we were luc

Re: strange failure in plpgsql_control tests (on fulmar, ICC 14.0.3)

2018-03-17 Thread Tomas Vondra
gt; be safe to unblock. > OK, enabled again. It'll take a while to run through the branches. I guess it might want to notify people running affected animals, because otherwise they may stay stuck for a long time. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: strange failure in plpgsql_control tests (on fulmar, ICC 14.0.3)

2018-03-17 Thread Tomas Vondra
On 03/17/2018 03:32 PM, Tom Lane wrote: > Tomas Vondra <tomas.von...@2ndquadrant.com> writes: >> I happened to be updating our machine running our buildfarm animals, and >> I noticed something quite strange - the machine was unexpectedly running >> out of disk space,

Re: MCV lists for highly skewed distributions

2018-03-17 Thread Tomas Vondra
On 03/17/2018 07:28 PM, Dean Rasheed wrote: > On 16 March 2018 at 15:26, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: >> Actually, one question - when deciding whether to keep the item in the >> MCV list, analyze_mcv_list only compares it's frequency with an a

Re: [PROPOSAL] Shared Ispell dictionaries

2018-03-19 Thread Tomas Vondra
On 03/19/2018 02:34 AM, Andres Freund wrote: > Hi, > > On 2018-03-19 01:52:41 +0100, Tomas Vondra wrote: >> I do agree with that. We have a working well-understood dsm-based >> solution, addressing the goals initially explained in this thread. > > Well, it's also awkw

Re: [HACKERS] plpgsql - additional extra checks

2018-03-20 Thread Tomas Vondra
On 03/20/2018 05:36 AM, Pavel Stehule wrote: > > > 2018-03-19 21:47 GMT+01:00 Tomas Vondra <tomas.von...@2ndquadrant.com > <mailto:tomas.von...@2ndquadrant.com>>: > > Hi, > > I'm looking at the updated patch (plpgsql-extra-check-180316.pa

Re: [HACKERS] VACUUM and ANALYZE disagreeing on what reltuples means

2018-03-20 Thread Tomas Vondra
allers, and none of them is using the reltuples estimate for anything except for passing it to index_update_stats. Aside from the bug fixed in 0001, of course. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Servi

Re: [PROPOSAL] Shared Ispell dictionaries

2018-03-20 Thread Tomas Vondra
On 03/20/2018 02:11 PM, Arthur Zakirov wrote: > Hello, > > On Mon, Mar 19, 2018 at 08:50:46PM +0100, Tomas Vondra wrote: >> Hi Arthur, >> >> I went through the patch - just skimming through the diffs, will do more >> testing tomorrow. Here are a few

Re: PL/pgSQL nested CALL with transactions

2018-03-22 Thread Tomas Vondra
nd doesn't call SPI_keepplan or exec_simple_check_plan. Why not to keep using exec_prepare_plan and add a new parameter to skip those calls? regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] VACUUM and ANALYZE disagreeing on what reltuples means

2018-03-22 Thread Tomas Vondra
On 03/22/2018 08:51 PM, Tom Lane wrote: > I wrote: >> Tomas Vondra <tomas.von...@2ndquadrant.com> writes: >>> The 0002 part is the main part, unifying the definition of reltuples on >>> three main places: > >> On to this part ... > > I've pushe

Re: [PROPOSAL] Shared Ispell dictionaries

2018-03-19 Thread Tomas Vondra
On 03/19/2018 07:07 PM, Andres Freund wrote: > On 2018-03-19 14:52:34 +0100, Tomas Vondra wrote: >> On 03/19/2018 02:34 AM, Andres Freund wrote: >>> Hi, >>> >>> On 2018-03-19 01:52:41 +0100, Tomas Vondra wrote: >>>> I do agree with that. We have a

Re: [PROPOSAL] Shared Ispell dictionaries

2018-03-19 Thread Tomas Vondra
ts. See the attached diffs, that should apply on top of 0003 and 0004 patches. Please, treat those as mere suggestions. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services diff --git a/doc/src/sgml/config.sgml b/

Re: [HACKERS] PATCH: multivariate histograms and MCV lists

2018-03-18 Thread Tomas Vondra
it into extended_statistic.c so that it can be shared. Now it's introduced in mcv.c right away, which makes it easier to understand and reduces size of the patches. 6) I've fixed a bunch of comments, obsolete FIXMEs, etc. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL

Re: [PROPOSAL] Shared Ispell dictionaries

2018-03-18 Thread Tomas Vondra
ands (because each backend would do it's own mmap). In any case, I suggest to polish the dsm-based patch, and see if we can get that one into PG11. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] plpgsql - additional extra checks

2018-03-19 Thread Tomas Vondra
seems fine to me, and I'm tempted to mark it RFC once the docs get fixed. Stephen, any objections? regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] Lazy hash table for XidInMVCCSnapshot (helps Zipfian a bit)

2018-03-16 Thread Tomas Vondra
On 03/17/2018 12:03 AM, Yura Sokolov wrote: > 16.03.2018 04:23, Tomas Vondra пишет: >> >> ... >> >> OK, a few more comments. >> >> 1) The code in ExtendXipSizeForHash seems somewhat redundant with >> my_log2 (that is, we could just call the existing f

Re: [PROPOSAL] Shared Ispell dictionaries

2018-03-16 Thread Tomas Vondra
. FWIW I am not quite sure if the mmap() approach is better than what was implemented by the patch. I'm not sure how exactly will it behave under memory pressure (AFAIK it goes through page cache, which means random parts of dictionaries might get evicted) or how well is it supported on various platforms (say, Windows). regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: WIP: BRIN multi-range indexes

2018-03-20 Thread Tomas Vondra
On 03/04/2018 01:14 AM, Tomas Vondra wrote: > ... > > The one overflow issue I found in the patch is that the numeric > "distance" function does this: > > d = DirectFunctionCall2(numeric_sub, a2, a1); /* a2 - a1 */ > > PG_RETURN_FLOAT8(Dir

Re: Hash join in SELECT target list expression keeps consuming memory

2018-03-21 Thread Tomas Vondra
oop copying a lot of data: https://github.com/postgres/postgres/blob/master/src/bin/pg_dump/pg_dump.c#L3258 But I'm having trouble verifying that, because the query fetching the list of objects is rather expensive with this number of large objects. How long does it take for you? I wonder

Re: [HACKERS] logical decoding of two-phase transactions

2018-03-21 Thread Tomas Vondra
ions (not just 2PC). Not to mention that the GID is usually much shorter than the 200B. So I suggest to use just a simple (char *) pointer for the GID, keeping it NULL for most transactions, and switching back to plain bool flags. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

bugfifx: a minor mistake in brin_inclusion.c comment

2018-03-22 Thread Tomas Vondra
Hi, while looking at brin_inclusion.c, I've noticed that the comment about INCLUSION_UNMERGEABLE and INCLUSION_CONTAINS_EMPTY uses incorrect values (1 instead of 2). Attached is a simple fix. But perhaps it would be better to use the constants in the comment. regards -- Tomas Vondra

Re: [PATCH] btree_gin, add support for uuid, bool, name, bpchar and anyrange types

2018-03-21 Thread Tomas Vondra
, and 0002 for those extras. The committer then may apply either 0001 or 0001+0002, depending on his judgment. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

strange failure in plpgsql_control tests (on fulmar, ICC 14.0.3)

2018-03-17 Thread Tomas Vondra
the behavior change suddenly (AFAICS there were no updates to the machine just before January 1st). I wonder if this might be related ICC 14.0.3, but again - it has been like that for a very long time. Ideas? -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7

Re: [HACKERS] MERGE SQL Statement for PG11

2018-03-01 Thread Tomas Vondra
On 02/06/2018 03:40 PM, Tomas Vondra wrote: > > > I plan to go through the patch and this thread over the couple of > days, and summarize what the current status is (or my understanding > of it). That is (a) what are the missing pieces, (b) why are they > missing, (c) how we pl

Re: Hash Joins vs. Bloom Filters / take 2

2018-03-01 Thread Tomas Vondra
On 03/01/2018 11:01 PM, Andres Freund wrote: > Hi, > > On 2018-02-20 22:23:54 +0100, Tomas Vondra wrote: >> So I've decided to revive the old patch, rebase it to current master, >> and see if we can resolve the issues that killed it in 2016. > > There seems

Re: Online enabling of checksums

2018-03-03 Thread Tomas Vondra
On 03/03/2018 05:08 PM, Magnus Hagander wrote: > > > On Sat, Mar 3, 2018 at 5:06 PM, Tomas Vondra > <tomas.von...@2ndquadrant.com <mailto:tomas.von...@2ndquadrant.com>> wrote: > > On 03/03/2018 01:38 PM, Robert Haas wrote: > > On Sat, Mar 3, 20

Re: [HACKERS] user-defined numeric data types triggering ERROR: unsupported type

2018-03-03 Thread Tomas Vondra
on master, of course. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services diff --git a/src/backend/utils/adt/network.c b/src/backend/utils/adt/network.c index aac7621..34cf336 100644 --- a/src/backend/utils

<    1   2   3   4   5   6   7   8   9   10   >