Re: [HACKERS] PATCH: index-only scans with partial indexes

2015-09-30 Thread Tomas Vondra
On 09/30/2015 12:55 PM, Tomas Vondra wrote: Hello! On 09/30/2015 10:29 AM, Kyotaro HORIGUCHI wrote: By the way your comment for indexrinfos is as following, * 'indexrinfos' is a list of RestrictInfo nodes from the query's WHERE * or JOIN conditions, excluding those implied by the index

Re: [HACKERS] PATCH: index-only scans with partial indexes

2015-09-26 Thread Tomas Vondra
Hi, On 09/26/2015 01:28 PM, Tomas Vondra wrote: The patch does not change the check_index_only implementation - it still needs to check the clauses, just like in v1 of the patch. To make this re-check unnecessary, we'd have to stick the remaining clauses somewhere, so that check_index_only can

Re: [HACKERS] PATCH: use foreign keys to improve join estimates v1

2015-09-28 Thread Tomas Vondra
provement, both compared to master and the previous versions of the patch. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To m

Re: [HACKERS] PATCH: use foreign keys to improve join estimates v1

2015-09-24 Thread Tomas Vondra
ke cardinality of a cartesian product of all the base relations, and then apply selectivities for all the join quals (in a pairwise manner). So for the three tables we take card(join) = card(f) * card(a) * card(b) * sel(f,a) * sel(f,b) and we can consider the foreign keys in the pairwise sel

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-09-24 Thread Tomas Vondra
Hi, On 09/23/2015 11:25 PM, Tom Lane wrote: Tomas Vondra <tomas.von...@2ndquadrant.com> writes: On 09/11/2015 07:16 PM, Robert Haas wrote: On Fri, Sep 11, 2015 at 1:12 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: I'm arguing for fixing the existing bug, and then address

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-09-24 Thread Tomas Vondra
On 09/24/2015 01:51 PM, Robert Haas wrote: On Thu, Sep 24, 2015 at 5:50 AM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: I however quite dislike the dismissal of the possible impact. It should be the responsibility of the person introducing the change to show that no such

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-09-24 Thread Tomas Vondra
On 09/24/2015 05:09 PM, Robert Haas wrote: On Thu, Sep 24, 2015 at 9:49 AM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: So while it does not introduce behavior change in this particular case (because it fails, as you point out), it introduces a behavior change in general - it

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-09-24 Thread Tomas Vondra
On 09/24/2015 07:42 PM, Robert Haas wrote: On Thu, Sep 24, 2015 at 12:40 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: There are two machines - one with 32GB of RAM and work_mem=2GB, the other one with 256GB of RAM and work_mem=16GB. The machines are hosting about the same data

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-09-24 Thread Tomas Vondra
On 09/24/2015 07:04 PM, Tom Lane wrote: Tomas Vondra <tomas.von...@2ndquadrant.com> writes: But what about computing the number of expected batches, but always start executing assuming no batching? And only if we actually fill work_mem, we start batching and use the expected number of b

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-09-24 Thread Tomas Vondra
2) run until exhausting work_mem 3) start batching, with the initially estimated number of batches regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgs

Re: [HACKERS] multivariate statistics / patch v7

2015-09-24 Thread Tomas Vondra
or not, etc. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/p

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-09-25 Thread Tomas Vondra
On 09/25/2015 02:54 AM, Robert Haas wrote: On Thu, Sep 24, 2015 at 1:58 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: Meh, you're right - I got the math wrong. It's 1.3% in both cases. However the question still stands - why should we handle the over-estimate in on

Re: [HACKERS] PATCH: use foreign keys to improve join estimates v1

2015-09-25 Thread Tomas Vondra
Hi, On 09/25/2015 03:39 AM, David Rowley wrote: On 24 September 2015 at 23:57, Tomas Vondra <tomas.von...@2ndquadrant.com <mailto:tomas.von...@2ndquadrant.com>> wrote: 2) find_best_match_foreign_key -- I think the comment before the fu

Re: [HACKERS] silent data loss with ext4 / all current versions

2015-11-27 Thread Tomas Vondra
On 11/27/2015 02:18 PM, Michael Paquier wrote: On Fri, Nov 27, 2015 at 8:17 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: So, what's going on? The problem is that while the rename() is atomic, it's not guaranteed to be durable without an explicit fsync on the parent dir

Re: [HACKERS] silent data loss with ext4 / all current versions

2015-11-29 Thread Tomas Vondra
On 11/29/2015 03:33 PM, Tomas Vondra wrote: Hi, On 11/29/2015 02:38 PM, Craig Ringer wrote: I've had a few tries at implementing a qemu-based crashtester where it hard kills the qemu instance at a random point then starts it back up. I've tried to reproduce the issue by killing a qemu VM

Re: [HACKERS] silent data loss with ext4 / all current versions

2015-12-01 Thread Tomas Vondra
/END_CRIT_SECTION is not available there) The patch is fairly trivial and I've done some rudimentary testing, but I'm sure I haven't exercised all the modified paths. regards Tomas -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training

Re: [HACKERS] silent data loss with ext4 / all current versions

2015-12-01 Thread Tomas Vondra
On 12/01/2015 10:44 PM, Peter Eisentraut wrote: On 11/27/15 8:18 AM, Michael Paquier wrote: On Fri, Nov 27, 2015 at 8:17 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: So, what's going on? The problem is that while the rename() is atomic, it's not guaranteed to be durable w

Re: [HACKERS] PATCH: index-only scans with partial indexes

2015-12-06 Thread Tomas Vondra
over the next few days. Thanks for the help so far! -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] On-demand running query plans using auto_explain and signals

2015-12-16 Thread Tomas Vondra
n every 60 seconds. Of course, this might be configurable by two GUCs: pg_stat_statements.explain_analyze_threshold = 60 # -1 is "off" pg_stat_statements.explain_analyze_refresh = 60 FWIW I'd still prefer having "EXPLAIN ANALYZE" in core, but better this than nothin

Re: [HACKERS] pam auth - add rhost item

2015-12-15 Thread Tomas Vondra
regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] pam auth - add rhost item

2015-12-15 Thread Tomas Vondra
for whitespace issues. Please, make sure you use tabs for indentation (and not spaces). regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql

Re: [HACKERS] Cube extension kNN support

2015-12-15 Thread Tomas Vondra
. Was the old behavior removed? I don't think so - it seems to behave as before, so why to remove this information? Maybe it's not useful? But then why add the bit about optimizing storage of points? regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development

Re: [HACKERS] checkpointer continuous flushing

2015-12-16 Thread Tomas Vondra
or even info about what benchmarks were done (pgbench?). If yes, do we only want to compare 0001-ckpt-14-andres.patch against master, or do we need to test one of the previous Fabien's patches? regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support

Re: [HACKERS] Performance improvement for joins where outer side is unique

2015-12-16 Thread Tomas Vondra
Hi, On 12/16/2015 11:40 PM, David Rowley wrote: On 17 December 2015 at 05:02, Tomas Vondra <tomas.von...@2ndquadrant.com <mailto:tomas.von...@2ndquadrant.com>> wrote: 0) I know the patch does not tweak costing - any plans in this direction? Would it be possible t

Re: [HACKERS] Tsvector editing functions

2015-12-14 Thread Tomas Vondra
ome of the functions use intexterm that does not match the function name. I see two such cases - to_tsvector and setweight. Is there a reason for that? regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Serv

Re: [HACKERS] Patch: Optimize memory allocation in function 'bringetbitmap'

2015-12-14 Thread Tomas Vondra
->bt_columns[i].bv_values for example. 5) brin_deform_tuple(..) The comment before the function probably should explain the purpose of the new parameter (that it can either be NULL or an existing tuple). regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Developme

Re: [HACKERS] PATCH: add pg_current_xlog_flush_location function

2015-12-13 Thread Tomas Vondra
Hi, On 12/13/2015 06:13 AM, Amit Kapila wrote: > ... > Is there a reason why you can't use existing function GetFlushRecPtr() in xlog.c? No, not really. I think I somehow missed that function when writing the initial version of the patch. Will fix in v2 of the patch. thanks --

[HACKERS] PATCH: add pg_current_xlog_flush_location function

2015-12-12 Thread Tomas Vondra
. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 60b9a09..0d67b82 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -167

Re: [HACKERS] WIP: bloom filter in Hash Joins with batches

2015-12-17 Thread Tomas Vondra
ng sequences of the same value (hash), and all of them will either have a match in the inner relation, or not have a match. That may easily skew the counters used for disabling the bloom filter dynamically. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Deve

Re: [HACKERS] WIP: bloom filter in Hash Joins with batches

2015-12-17 Thread Tomas Vondra
Hi, On 12/17/2015 10:50 AM, Shulgin, Oleksandr wrote: On Tue, Dec 15, 2015 at 11:30 PM, Tomas Vondra <tomas.von...@2ndquadrant.com <mailto:tomas.von...@2ndquadrant.com>> wrote: Attached is a spreadsheet with results for various work_mem values, and also with a smaller data set (j

Re: [HACKERS] PATCH: postpone building buckets to the end of Hash (in HashJoin)

2015-12-17 Thread Tomas Vondra
-- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] pg_hba_lookup function to get all matching pg_hba.conf entries

2015-12-15 Thread Tomas Vondra
DEFAULT NULL, ssl_inuse BOOLEAN DEFAULT NULL) regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to

Re: [HACKERS] Cube extension kNN support

2015-12-16 Thread Tomas Vondra
be (1,1),(3,3) I don't think that's what the comment says, actually. It rather refers to code like this: result = Min(LL_COORD(c, n - 1), UR_COORD(c, n - 1)); i.e. if you specifically ask for a particular corner (ll, in this case), you'll get the proper value. regards -- Tomas Vondra

Re: [HACKERS] Proposal: custom compression methods

2015-12-16 Thread Tomas Vondra
ession method providing both interfaces at the same time. In any case, I don't think this is the main challenge the patch needs to solve at this point. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] Proposal: custom compression methods

2015-12-16 Thread Tomas Vondra
in the future, this patch does not make it any more complicated. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to

Re: [HACKERS] Proposal: custom compression methods

2015-12-16 Thread Tomas Vondra
n't). regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Performance improvement for joins where outer side is unique

2015-12-16 Thread Tomas Vondra
ss * Returns true if rel has some properties which can prove the relation * to be unique over some set of columns. * regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services --

[HACKERS] PATCH: postpone building buckets to the end of Hash (in HashJoin)

2015-12-14 Thread Tomas Vondra
/19746.1443983...@sss.pgh.pa.us regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index 12dae77..8fd9c9f 100644 --- a/src/bac

Re: [HACKERS] Let PostgreSQL's On Schedule checkpoint write buffer smooth spread cycle by tuning IsCheckpointOnSchedule?

2015-12-14 Thread Tomas Vondra
arbitrary). regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] PATCH: track last known XLOG segment in control file

2015-12-12 Thread Tomas Vondra
On 12/12/2015 11:20 PM, Andres Freund wrote: Hi, On 2015-12-12 22:14:13 +0100, Tomas Vondra wrote: this is the second improvement proposed in the thread [1] about ext4 data loss issue. It adds another field to control file, tracking the last known WAL segment. This does not eliminate

[HACKERS] PATCH: track last known XLOG segment in control file

2015-12-12 Thread Tomas Vondra
. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 71fc8ff..50f10a5 100644 --- a/src/backend/access/transam/xlog.c +

Re: [HACKERS] PATCH: track last known XLOG segment in control file

2015-12-12 Thread Tomas Vondra
On 12/12/2015 11:39 PM, Andres Freund wrote: On 2015-12-12 23:28:33 +0100, Tomas Vondra wrote: On 12/12/2015 11:20 PM, Andres Freund wrote: On 2015-12-12 22:14:13 +0100, Tomas Vondra wrote: this is the second improvement proposed in the thread [1] about ext4 data loss issue. It adds another

Re: [HACKERS] [PATCH] Equivalence Class Filters

2015-12-15 Thread Tomas Vondra
, but I guess it might be modified to track the correlations through foreign keys. Interestingly enough, the data for the paper paper were collected on PostgreSQL, but the correlation maps were implemented in an application layer on top of the database. [1] http://www.vldb.org/pvldb/2/vldb09

Re: [HACKERS] GIN data corruption bug(s) in 9.6devel

2016-01-05 Thread Tomas Vondra
Hi, On 12/23/2015 09:33 PM, Jeff Janes wrote: On Mon, Dec 21, 2015 at 11:51 AM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: On 12/21/2015 07:41 PM, Jeff Janes wrote: On Sat, Dec 19, 2015 at 3:19 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: ... So both

[HACKERS] Re: More thorough planning for OLAP queries (was: [PATCH] Equivalence Class Filters)

2015-12-30 Thread Tomas Vondra
On 12/30/2015 10:30 PM, David Rowley wrote: On 31 December 2015 at 01:24, Tomas Vondra <tomas.von...@2ndquadrant.com <mailto:tomas.von...@2ndquadrant.com>> wrote: Personally I'd like to see automatic plan caching occur in PostgreSQL. There is a bit of a problem with it in regard

Re: [HACKERS] PATCH: postpone building buckets to the end of Hash (in HashJoin)

2015-12-31 Thread Tomas Vondra
s because the patch with bloom filters actually uses HLL to size the bloom filters, so this would not be really introducing any new code. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services >From 3aaf27f722e

Re: [HACKERS] WIP: bloom filter in Hash Joins with batches

2015-12-31 Thread Tomas Vondra
if they turn inefficient. I don't think that's a major issue at this point, though, and I think it'll be easier to do based on testing the current patch. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services &

[HACKERS] PATCH: Extending the HyperLogLog API a bit

2015-12-31 Thread Tomas Vondra
memory etc. Currently that's trivial to do, but perhaps we'll make the structure more complicated in the future (albeit that might be unlikely). FWIW I've already used this in the patch marrying hash joins and bloom filters. regards -- Tomas Vondra http://www.2ndQuadrant.com

Re: [HACKERS] PATCH: Extending the HyperLogLog API a bit

2015-12-31 Thread Tomas Vondra
Meh, of course I forgot to actually attach the patch. -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services >From 8f207693faa65e65e8a1e5e894c2ad96ad1f3cea Mon Sep 17 00:00:00 2001 From: Tomas Vondra <to...@pg

Re: [HACKERS] PATCH: Extending the HyperLogLog API a bit

2016-01-09 Thread Tomas Vondra
On 01/01/2016 12:08 AM, Peter Geoghegan wrote: On Thu, Dec 31, 2015 at 12:48 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: 1) initHyperLogLogError(hyperLogLogState *cState, double error) Instead of specifying bwidth (essentially the number of bits used for addr

Re: [HACKERS] checkpointer continuous flushing

2016-01-06 Thread Tomas Vondra
. The patch no longer applies to master. Can someone rebase it? regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to

Re: [HACKERS] GIN data corruption bug(s) in 9.6devel

2015-12-21 Thread Tomas Vondra
On 12/21/2015 07:41 PM, Jeff Janes wrote: On Sat, Dec 19, 2015 at 3:19 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: ... So both patches seem to do the trick, but (2) is faster. Not sure if this is expected. (BTW all the results are without asserts enabled). Do you kno

Re: [HACKERS] Let PostgreSQL's On Schedule checkpoint write buffer smooth spread cycle by tuning IsCheckpointOnSchedule?

2015-12-23 Thread Tomas Vondra
tes at the beginning of the checkpoint (due to getting confused by FPW), and OS starts flushing that to disk because we exceed dirty_(background)_bytes, that surely interferes with reads (which is a major issue for queries). regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreS

Re: [HACKERS] Let PostgreSQL's On Schedule checkpoint write buffer smooth spread cycle by tuning IsCheckpointOnSchedule?

2015-12-23 Thread Tomas Vondra
On 12/21/2015 01:11 PM, Heikki Linnakangas wrote: On 21/12/15 13:53, Tomas Vondra wrote: On 12/21/2015 12:03 PM, Heikki Linnakangas wrote: On 17/12/15 19:07, Robert Haas wrote: If it works well empirically, does it really matter that it's arbitrary? I mean, the entire planner is full

Re: [HACKERS] GIN data corruption bug(s) in 9.6devel

2015-12-19 Thread Tomas Vondra
Hi, On 11/06/2015 02:09 AM, Tomas Vondra wrote: Hi, On 11/06/2015 01:05 AM, Jeff Janes wrote: On Thu, Nov 5, 2015 at 3:50 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: ... I can do that - I see there are three patches in the two threads: 1) gin_pending_lwlock.patch (Jeff

Re: [HACKERS] Commit fest status for 2015-11

2015-12-24 Thread Tomas Vondra
t work this time! +1 -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mail

Re: [HACKERS] WIP: bloom filter in Hash Joins with batches

2015-12-24 Thread Tomas Vondra
On 12/24/2015 02:51 PM, Simon Riggs wrote: On 17 December 2015 at 16:00, Tomas Vondra <tomas.von...@2ndquadrant.com <mailto:tomas.von...@2ndquadrant.com>> wrote: On 12/17/2015 11:44 AM, Simon Riggs wrote: My understanding is that the bloom filter would be

Re: [HACKERS] WIP: bloom filter in Hash Joins with batches

2015-12-28 Thread Tomas Vondra
On 12/28/2015 11:52 AM, David Rowley wrote: On 28 December 2015 at 23:44, Tomas Vondra <tomas.von...@2ndquadrant.com <mailto:tomas.von...@2ndquadrant.com>> wrote: On 12/28/2015 11:38 AM, David Rowley wrote: If so, then a filter with all 1 bits set should be

Re: [HACKERS] WIP: bloom filter in Hash Joins with batches

2015-12-28 Thread Tomas Vondra
On 12/28/2015 11:38 AM, David Rowley wrote: On 28 December 2015 at 23:23, Tomas Vondra <tomas.von...@2ndquadrant.com <mailto:tomas.von...@2ndquadrant.com>> wrote: On 12/28/2015 03:15 AM, David Rowley wrote: Maybe it would be better to, once the filter is built, simply

Re: [HACKERS] WIP: bloom filter in Hash Joins with batches

2015-12-28 Thread Tomas Vondra
On 12/28/2015 03:15 AM, David Rowley wrote: On 18 December 2015 at 04:34, Tomas Vondra <tomas.von...@2ndquadrant.com <mailto:tomas.von...@2ndquadrant.com>> wrote: I think ultimately we'll need to measure the false positive rate, so that we can use it to dynamically disab

Re: [HACKERS] Let PostgreSQL's On Schedule checkpoint write buffer smooth spread cycle by tuning IsCheckpointOnSchedule?

2015-12-21 Thread Tomas Vondra
Hi, On 12/21/2015 12:03 PM, Heikki Linnakangas wrote: On 17/12/15 19:07, Robert Haas wrote: On Mon, Dec 14, 2015 at 6:08 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: So we know that we should expect about (prev_wal_bytes - wal_bytes) + (prev_wal_fpw_bytes - wal_fpw

Re: [HACKERS] Let PostgreSQL's On Schedule checkpoint write buffer smooth spread cycle by tuning IsCheckpointOnSchedule?

2015-12-23 Thread Tomas Vondra
Hi, On 12/23/2015 08:22 PM, Robert Haas wrote: On Wed, Dec 23, 2015 at 2:16 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: Another point (which Jan Wieck made me think of) is that the optimal behavior here likely depends on whether xlog and data are on the same disk cont

Re: [HACKERS] PATCH: use foreign keys to improve join estimates v1

2015-12-23 Thread Tomas Vondra
t guessing that the status was incorrect, I have moved it to next CF. Yes, I still think the patch is useful. Thanks for moving it to the next commitfest and sorry for not updating the status. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support,

Re: [HACKERS] PATCH: index-only scans with partial indexes

2015-12-23 Thread Tomas Vondra
Hi, On 12/24/2015 04:05 AM, Michael Paquier wrote: On Mon, Dec 7, 2015 at 7:48 AM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: ... Otherwise the reworked patch seems fine to me, but I'll give it a bit more testing over the next few days. Thanks for the help so far! Tomas, a

Re: [HACKERS] WIP: bloom filter in Hash Joins with batches

2015-12-20 Thread Tomas Vondra
is about indexes, mine is about building temporary bloom filters when executing hash joins. FWIW, I think bloom filters should be easy to add to BRIN indexes, as another type of 'summary'. That should address most of the missing pieces in your implementation (e.g. WAL). regards -- Tomas Vondra

[HACKERS] Re: More thorough planning for OLAP queries (was: [PATCH] Equivalence Class Filters)

2015-12-30 Thread Tomas Vondra
t somewhat unpredictable - the plan will depend on whether the query was planned directly or as a prepared statement, or whether plan caching is enabled. However, the same mostly applies to solutions proposed in the other threads so far. regards -- Tomas Vondra http://www.2ndQu

[HACKERS] silent data loss with ext4 / all current versions

2015-11-27 Thread Tomas Vondra
fsync the directory - then the rename won't be replayed and will be lost). regards -- Tomas Vondra http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.

Re: [HACKERS] silent data loss with ext4 / all current versions

2015-11-27 Thread Tomas Vondra
Hi, On 11/27/2015 02:28 PM, Greg Stark wrote: On Fri, Nov 27, 2015 at 11:17 AM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: I plan to do more power failure testing soon, with more complex test scenarios. I suspect there might be other similar issues (e.g. when we rename a file

Re: [HACKERS] silent data loss with ext4 / all current versions

2015-11-29 Thread Tomas Vondra
Hi, On 11/29/2015 02:38 PM, Craig Ringer wrote: On 27 November 2015 at 21:28, Greg Stark <st...@mit.edu <mailto:st...@mit.edu>> wrote: On Fri, Nov 27, 2015 at 11:17 AM, Tomas Vondra <tomas.von...@2ndquadrant.com <mailto:tomas.von...@2ndquadrant.com>> wrote:

Re: [HACKERS] silent data loss with ext4 / all current versions

2015-11-29 Thread Tomas Vondra
On 11/29/2015 02:41 PM, Craig Ringer wrote: On 27 November 2015 at 19:17, Tomas Vondra <tomas.von...@2ndquadrant.com <mailto:tomas.von...@2ndquadrant.com>> wrote: It's also possible to mitigate this by setting wal_sync_method=fsync Are you sure? https://lwn.net/Articles/3

Re: [HACKERS] pg9.6 segfault using simple query (related to use fk for join estimates)

2016-06-06 Thread Tomas Vondra
On 06/06/2016 06:15 PM, Tom Lane wrote: Tomas Vondra <tomas.von...@2ndquadrant.com> writes: On 06/04/2016 08:15 PM, Tom Lane wrote: * Make RelationGetFKeyList cache a list of ForeignKeyOptInfo structs, not just constraint OIDs. It's insane that the relcache scans pg_constraint to c

Re: [HACKERS] pg9.6 segfault using simple query (related to use fk for join estimates)

2016-06-06 Thread Tomas Vondra
On 06/06/2016 07:40 PM, Tom Lane wrote: Tomas Vondra <tomas.von...@2ndquadrant.com> writes: On 06/06/2016 06:15 PM, Tom Lane wrote: This checks that you found a joinclause mentioning foreignrel. But foreignrel need have nothing to do with the foreign key; it could be any table in the

Re: [HACKERS] pg9.6 segfault using simple query (related to use fk for join estimates)

2016-06-06 Thread Tomas Vondra
it might seem. What if the foreign keys overlap, for example? Or what if the keys go in opposite directions (cycle). And so on ... regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hac

Re: [HACKERS] New design for FK-based join selectivity estimation

2016-06-13 Thread Tomas Vondra
happens in costsize.c, of course, but was modified to use the pre-matched info. I have my doubts about this part, and I'm sure Tom had something less complex / more efficient in mind (using the pre-matched info). regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL

Re: [HACKERS] New design for FK-based join selectivity estimation

2016-06-05 Thread Tomas Vondra
s. ISTM the proposed design has the same issue, no? What happens if the relations are partitioned? regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hac

Re: [HACKERS] pg9.6 segfault using simple query (related to use fk for join estimates)

2016-06-05 Thread Tomas Vondra
gt;relid == rightvar->varno) && (fkrel->relid == leftvar->varno) && (fkinfo->confkeys[i] == rightvar->varattno) && (fkinfo->conkeys[i] == leftvar->varattno)) {

Re: [HACKERS] Re: PATCH: Split stats file per database WAS: autovacuum stress-testing our system

2016-05-31 Thread Tomas Vondra
On 05/31/2016 06:59 PM, Tom Lane wrote: Tomas Vondra <tomas.von...@2ndquadrant.com> writes: On 05/26/2016 10:10 PM, Tom Lane wrote: I posted a patch at https://www.postgresql.org/message-id/13023.1464213...@sss.pgh.pa.us which I think is functionally equivalent to what you hav

[HACKERS] Re: pg9.6 segfault using simple query (related to use fk for join estimates)

2016-06-02 Thread Tomas Vondra
if there's a consensus it's not doomed anyway (for 9.6). regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to

Re: [HACKERS] Re: PATCH: Split stats file per database WAS: autovacuum stress-testing our system

2016-05-26 Thread Tomas Vondra
Hi, On 05/26/2016 10:10 PM, Tom Lane wrote: Tomas Vondra <tomas.von...@2ndquadrant.com> writes: Attached is a patch that should fix the coalescing, including the clock skew detection. In the end I reorganized the code a bit, moving the check at the end, after the clock skew det

Re: [HACKERS] Re: PATCH: Split stats file per database WAS: autovacuum stress-testing our system

2016-05-31 Thread Tomas Vondra
On 05/31/2016 07:24 PM, Tom Lane wrote: Tomas Vondra <tomas.von...@2ndquadrant.com> writes: On 05/31/2016 06:59 PM, Tom Lane wrote: I'm confused here --- are you speaking of having removed if (msg->cutoff_time > req->request_time) req->request_time

Re: [HACKERS] Re: PATCH: Split stats file per database WAS: autovacuum stress-testing our system

2016-05-31 Thread Tomas Vondra
Hi, On 05/26/2016 10:10 PM, Tom Lane wrote: Tomas Vondra <tomas.von...@2ndquadrant.com> writes: Attached is a patch that should fix the coalescing, including the clock skew detection. In the end I reorganized the code a bit, moving the check at the end, after the clock skew det

Re: [HACKERS] New design for FK-based join selectivity estimation

2016-06-19 Thread Tomas Vondra
Hi On 06/18/2016 06:52 PM, Tom Lane wrote: Tomas Vondra <tomas.von...@2ndquadrant.com> writes: A few more comments, about re-reading the patch more thoroughly. I wouldn't say any of those qualify as bugs, but rather as discussion about some of the design choices: 1) NULL handling

Re: [HACKERS] New design for FK-based join selectivity estimation

2016-06-17 Thread Tomas Vondra
t one, but don't risk making it worse." 3) ForeignKeyOptInfo->rinfos as a List Can we actually get a list of matching RestrictInfos for a single foreign key? I've been unable to construct such query. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Deve

Re: [HACKERS] Whether to back-patch fix for aggregate transtype width estimates

2016-06-19 Thread Tomas Vondra
using per-group memory contexts. So also unrelated to this. So while I'm a fan of improving our planning, I'd lean towards not back-patching this particular bit. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training &

Re: [HACKERS] New design for FK-based join selectivity estimation

2016-06-16 Thread Tomas Vondra
Hi, On 06/16/2016 01:00 AM, Tom Lane wrote: Tomas Vondra <tomas.von...@2ndquadrant.com> writes: Attached is a reworked patch, mostly following the new design proposal from this thread. I've whacked this around quite a bit and am now reasonably happy with it. The issue of planning spe

Re: [HACKERS] PATCH: add pg_current_xlog_flush_location function

2016-01-11 Thread Tomas Vondra
before looking at it but it seems to me that using an independent variable would make the code more robust. Why? LogwrtResult only serves as a local cache of shared values, so there should be no danger of skipping something. regards -- Tomas Vondra http://www.2ndQuadrant.com Post

Re: [HACKERS] WIP: bloom filter in Hash Joins with batches

2016-01-10 Thread Tomas Vondra
Hi, On 01/10/2016 01:08 AM, Peter Geoghegan wrote: On Sat, Jan 9, 2016 at 11:02 AM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: So, this seems to bring reasonable speedup, as long as the selectivity is below 50%, and the data set is sufficiently large. What about semijoins? Appa

Re: [HACKERS] WIP: bloom filter in Hash Joins with batches

2016-01-10 Thread Tomas Vondra
ntally verified that you get the expected probability of false positives in practice? I have seen that when the bloom filter is properly sized, and the data set is not somehow skewed, the bloom filters have about the right false positive rate. regards -- Tomas Vondra http://www.2n

Re: [HACKERS] WIP: bloom filter in Hash Joins with batches

2016-01-10 Thread Tomas Vondra
Hi, On 01/10/2016 05:11 AM, Peter Geoghegan wrote: On Sat, Jan 9, 2016 at 11:02 AM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: Which means the "dim.r" column has 100 different values (0-99) with uniform distribution. So e.g. "WHERE r < 15" matche

Re: [HACKERS] PATCH: add pg_current_xlog_flush_location function

2016-01-10 Thread Tomas Vondra
Hi, On 12/13/2015 08:38 PM, Tomas Vondra wrote: Hi, On 12/13/2015 06:13 AM, Amit Kapila wrote: > ... > Is there a reason why you can't use existing function GetFlushRecPtr() in xlog.c? No, not really. I think I somehow missed that function when writing the initial version of the

Re: [HACKERS] Combining Aggregates

2016-01-18 Thread Tomas Vondra
state (I don't see why would be the case), it's a dead end. Not so long ago we've fixed exactly this issue in array_agg(), which addressed a bunch of reported OOM issues and improved array_agg() performance quite a bit. It'd be rather crazy introducing the same problem to all aggregate functions

Re: [HACKERS] Combining Aggregates

2016-01-18 Thread Tomas Vondra
anger of OOM issues? regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.or

Re: [HACKERS] Combining Aggregates

2016-01-18 Thread Tomas Vondra
On 01/19/2016 06:04 AM, Tomas Vondra wrote: Hi, On 01/19/2016 05:00 AM, David Rowley wrote: Good question. I looked at this and found my VM was swapping like crazy. Upon investigation it appears that's because, since the patch creates a memory context per aggregated group, and in this case

Re: [HACKERS] PATCH: postpone building buckets to the end of Hash (in HashJoin)

2016-01-18 Thread Tomas Vondra
Hi, On 12/17/2015 10:28 PM, Tomas Vondra wrote: Hi, On 12/17/2015 07:20 PM, Robert Haas wrote: ... If this doesn't regress performance in the case where the number of buckets is estimated accurately to begin with, then I think this is a great idea. Can you supply some performance tests

Re: [HACKERS] silent data loss with ext4 / all current versions

2016-01-18 Thread Tomas Vondra
On 01/19/2016 07:44 AM, Michael Paquier wrote: On Wed, Dec 2, 2015 at 3:24 PM, Michael Paquier <michael.paqu...@gmail.com> wrote: On Wed, Dec 2, 2015 at 3:23 PM, Michael Paquier <michael.paqu...@gmail.com> wrote: On Wed, Dec 2, 2015 at 7:05 AM, Tomas Vondra <tomas.von...@

Re: [HACKERS] silent data loss with ext4 / all current versions

2016-01-18 Thread Tomas Vondra
On 01/19/2016 08:03 AM, Michael Paquier wrote: On Tue, Jan 19, 2016 at 3:58 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: ... Tomas, I am planning to have a look at that, because it seems to be important. In case it becomes lost on my radar, do you mind if I add it to the 2

Re: [HACKERS] Explanation for bug #13908: hash joins are badly broken

2016-02-06 Thread Tomas Vondra
On 02/06/2016 08:39 PM, Andres Freund wrote: On 2016-02-06 20:34:07 +0100, Tomas Vondra wrote: On 02/06/2016 06:47 PM, Tom Lane wrote: * It incorporates a bespoke reimplementation of palloc into hash joins. This is not a maintainable/scalable way to go about reducing memory consumption

Re: [HACKERS] Explanation for bug #13908: hash joins are badly broken

2016-02-06 Thread Tomas Vondra
On 02/06/2016 09:55 PM, Tom Lane wrote: Tomas Vondra <tomas.von...@2ndquadrant.com> writes: On 02/06/2016 06:47 PM, Tom Lane wrote: I note also that while the idea of ExecHashRemoveNextSkewBucket is to reduce memory consumed by the skew table to make it available to the main hash

Re: [HACKERS] Explanation for bug #13908: hash joins are badly broken

2016-02-06 Thread Tomas Vondra
On 02/06/2016 06:47 PM, Tom Lane wrote: Tomas Vondra <tomas.von...@2ndquadrant.com> writes: On 02/06/2016 02:34 AM, Tom Lane wrote: I have sussed what's happening in bug #13908. Basically, commit 45f6240a8fa9d355 ("Pack tuples in a hash join batch densely, to save memory&quo

Re: [HACKERS] silent data loss with ext4 / all current versions

2016-02-06 Thread Tomas Vondra
Hi, On 02/06/2016 01:16 PM, Michael Paquier wrote: On Sat, Feb 6, 2016 at 2:11 AM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: On 02/04/2016 09:59 AM, Michael Paquier wrote: On Tue, Feb 2, 2016 at 9:59 AM, Andres Freund <and...@anarazel.de> wrote: On 2016-02-02 09

<    4   5   6   7   8   9   10   11   12   13   >