Re: [HACKERS] checkpointer continuous flushing

2016-01-06 Thread Tomas Vondra
. The patch no longer applies to master. Can someone rebase it? regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to

Re: [HACKERS] GIN data corruption bug(s) in 9.6devel

2016-01-05 Thread Tomas Vondra
Hi, On 12/23/2015 09:33 PM, Jeff Janes wrote: On Mon, Dec 21, 2015 at 11:51 AM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: On 12/21/2015 07:41 PM, Jeff Janes wrote: On Sat, Dec 19, 2015 at 3:19 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: ... So both

Re: [HACKERS] PATCH: postpone building buckets to the end of Hash (in HashJoin)

2015-12-31 Thread Tomas Vondra
s because the patch with bloom filters actually uses HLL to size the bloom filters, so this would not be really introducing any new code. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services >From 3aaf27f722e

Re: [HACKERS] WIP: bloom filter in Hash Joins with batches

2015-12-31 Thread Tomas Vondra
if they turn inefficient. I don't think that's a major issue at this point, though, and I think it'll be easier to do based on testing the current patch. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services &

[HACKERS] PATCH: Extending the HyperLogLog API a bit

2015-12-31 Thread Tomas Vondra
memory etc. Currently that's trivial to do, but perhaps we'll make the structure more complicated in the future (albeit that might be unlikely). FWIW I've already used this in the patch marrying hash joins and bloom filters. regards -- Tomas Vondra http://www.2ndQuadrant.com

Re: [HACKERS] PATCH: Extending the HyperLogLog API a bit

2015-12-31 Thread Tomas Vondra
Meh, of course I forgot to actually attach the patch. -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services >From 8f207693faa65e65e8a1e5e894c2ad96ad1f3cea Mon Sep 17 00:00:00 2001 From: Tomas Vondra <to...@pg

[HACKERS] Re: More thorough planning for OLAP queries (was: [PATCH] Equivalence Class Filters)

2015-12-30 Thread Tomas Vondra
On 12/30/2015 10:30 PM, David Rowley wrote: On 31 December 2015 at 01:24, Tomas Vondra <tomas.von...@2ndquadrant.com <mailto:tomas.von...@2ndquadrant.com>> wrote: Personally I'd like to see automatic plan caching occur in PostgreSQL. There is a bit of a problem with it in regard

[HACKERS] Re: More thorough planning for OLAP queries (was: [PATCH] Equivalence Class Filters)

2015-12-30 Thread Tomas Vondra
t somewhat unpredictable - the plan will depend on whether the query was planned directly or as a prepared statement, or whether plan caching is enabled. However, the same mostly applies to solutions proposed in the other threads so far. regards -- Tomas Vondra http://www.2ndQu

Re: [HACKERS] WIP: bloom filter in Hash Joins with batches

2015-12-28 Thread Tomas Vondra
On 12/28/2015 11:52 AM, David Rowley wrote: On 28 December 2015 at 23:44, Tomas Vondra <tomas.von...@2ndquadrant.com <mailto:tomas.von...@2ndquadrant.com>> wrote: On 12/28/2015 11:38 AM, David Rowley wrote: If so, then a filter with all 1 bits set should be

Re: [HACKERS] WIP: bloom filter in Hash Joins with batches

2015-12-28 Thread Tomas Vondra
On 12/28/2015 11:38 AM, David Rowley wrote: On 28 December 2015 at 23:23, Tomas Vondra <tomas.von...@2ndquadrant.com <mailto:tomas.von...@2ndquadrant.com>> wrote: On 12/28/2015 03:15 AM, David Rowley wrote: Maybe it would be better to, once the filter is built, simply

Re: [HACKERS] WIP: bloom filter in Hash Joins with batches

2015-12-28 Thread Tomas Vondra
On 12/28/2015 03:15 AM, David Rowley wrote: On 18 December 2015 at 04:34, Tomas Vondra <tomas.von...@2ndquadrant.com <mailto:tomas.von...@2ndquadrant.com>> wrote: I think ultimately we'll need to measure the false positive rate, so that we can use it to dynamically disab

Re: [HACKERS] Commit fest status for 2015-11

2015-12-24 Thread Tomas Vondra
t work this time! +1 -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mail

Re: [HACKERS] WIP: bloom filter in Hash Joins with batches

2015-12-24 Thread Tomas Vondra
On 12/24/2015 02:51 PM, Simon Riggs wrote: On 17 December 2015 at 16:00, Tomas Vondra <tomas.von...@2ndquadrant.com <mailto:tomas.von...@2ndquadrant.com>> wrote: On 12/17/2015 11:44 AM, Simon Riggs wrote: My understanding is that the bloom filter would be

Re: [HACKERS] Let PostgreSQL's On Schedule checkpoint write buffer smooth spread cycle by tuning IsCheckpointOnSchedule?

2015-12-23 Thread Tomas Vondra
tes at the beginning of the checkpoint (due to getting confused by FPW), and OS starts flushing that to disk because we exceed dirty_(background)_bytes, that surely interferes with reads (which is a major issue for queries). regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreS

Re: [HACKERS] Let PostgreSQL's On Schedule checkpoint write buffer smooth spread cycle by tuning IsCheckpointOnSchedule?

2015-12-23 Thread Tomas Vondra
On 12/21/2015 01:11 PM, Heikki Linnakangas wrote: On 21/12/15 13:53, Tomas Vondra wrote: On 12/21/2015 12:03 PM, Heikki Linnakangas wrote: On 17/12/15 19:07, Robert Haas wrote: If it works well empirically, does it really matter that it's arbitrary? I mean, the entire planner is full

Re: [HACKERS] Let PostgreSQL's On Schedule checkpoint write buffer smooth spread cycle by tuning IsCheckpointOnSchedule?

2015-12-23 Thread Tomas Vondra
Hi, On 12/23/2015 08:22 PM, Robert Haas wrote: On Wed, Dec 23, 2015 at 2:16 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: Another point (which Jan Wieck made me think of) is that the optimal behavior here likely depends on whether xlog and data are on the same disk cont

Re: [HACKERS] PATCH: use foreign keys to improve join estimates v1

2015-12-23 Thread Tomas Vondra
t guessing that the status was incorrect, I have moved it to next CF. Yes, I still think the patch is useful. Thanks for moving it to the next commitfest and sorry for not updating the status. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support,

Re: [HACKERS] PATCH: index-only scans with partial indexes

2015-12-23 Thread Tomas Vondra
Hi, On 12/24/2015 04:05 AM, Michael Paquier wrote: On Mon, Dec 7, 2015 at 7:48 AM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: ... Otherwise the reworked patch seems fine to me, but I'll give it a bit more testing over the next few days. Thanks for the help so far! Tomas, a

Re: [HACKERS] GIN data corruption bug(s) in 9.6devel

2015-12-21 Thread Tomas Vondra
On 12/21/2015 07:41 PM, Jeff Janes wrote: On Sat, Dec 19, 2015 at 3:19 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: ... So both patches seem to do the trick, but (2) is faster. Not sure if this is expected. (BTW all the results are without asserts enabled). Do you kno

Re: [HACKERS] Let PostgreSQL's On Schedule checkpoint write buffer smooth spread cycle by tuning IsCheckpointOnSchedule?

2015-12-21 Thread Tomas Vondra
Hi, On 12/21/2015 12:03 PM, Heikki Linnakangas wrote: On 17/12/15 19:07, Robert Haas wrote: On Mon, Dec 14, 2015 at 6:08 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: So we know that we should expect about (prev_wal_bytes - wal_bytes) + (prev_wal_fpw_bytes - wal_fpw

Re: [HACKERS] WIP: bloom filter in Hash Joins with batches

2015-12-20 Thread Tomas Vondra
is about indexes, mine is about building temporary bloom filters when executing hash joins. FWIW, I think bloom filters should be easy to add to BRIN indexes, as another type of 'summary'. That should address most of the missing pieces in your implementation (e.g. WAL). regards -- Tomas Vondra

Re: [HACKERS] GIN data corruption bug(s) in 9.6devel

2015-12-19 Thread Tomas Vondra
Hi, On 11/06/2015 02:09 AM, Tomas Vondra wrote: Hi, On 11/06/2015 01:05 AM, Jeff Janes wrote: On Thu, Nov 5, 2015 at 3:50 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: ... I can do that - I see there are three patches in the two threads: 1) gin_pending_lwlock.patch (Jeff

Re: [HACKERS] WIP: bloom filter in Hash Joins with batches

2015-12-17 Thread Tomas Vondra
ng sequences of the same value (hash), and all of them will either have a match in the inner relation, or not have a match. That may easily skew the counters used for disabling the bloom filter dynamically. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Deve

Re: [HACKERS] WIP: bloom filter in Hash Joins with batches

2015-12-17 Thread Tomas Vondra
Hi, On 12/17/2015 10:50 AM, Shulgin, Oleksandr wrote: On Tue, Dec 15, 2015 at 11:30 PM, Tomas Vondra <tomas.von...@2ndquadrant.com <mailto:tomas.von...@2ndquadrant.com>> wrote: Attached is a spreadsheet with results for various work_mem values, and also with a smaller data set (j

Re: [HACKERS] PATCH: postpone building buckets to the end of Hash (in HashJoin)

2015-12-17 Thread Tomas Vondra
-- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] On-demand running query plans using auto_explain and signals

2015-12-16 Thread Tomas Vondra
n every 60 seconds. Of course, this might be configurable by two GUCs: pg_stat_statements.explain_analyze_threshold = 60 # -1 is "off" pg_stat_statements.explain_analyze_refresh = 60 FWIW I'd still prefer having "EXPLAIN ANALYZE" in core, but better this than nothin

Re: [HACKERS] checkpointer continuous flushing

2015-12-16 Thread Tomas Vondra
or even info about what benchmarks were done (pgbench?). If yes, do we only want to compare 0001-ckpt-14-andres.patch against master, or do we need to test one of the previous Fabien's patches? regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support

Re: [HACKERS] Performance improvement for joins where outer side is unique

2015-12-16 Thread Tomas Vondra
Hi, On 12/16/2015 11:40 PM, David Rowley wrote: On 17 December 2015 at 05:02, Tomas Vondra <tomas.von...@2ndquadrant.com <mailto:tomas.von...@2ndquadrant.com>> wrote: 0) I know the patch does not tweak costing - any plans in this direction? Would it be possible t

Re: [HACKERS] Cube extension kNN support

2015-12-16 Thread Tomas Vondra
be (1,1),(3,3) I don't think that's what the comment says, actually. It rather refers to code like this: result = Min(LL_COORD(c, n - 1), UR_COORD(c, n - 1)); i.e. if you specifically ask for a particular corner (ll, in this case), you'll get the proper value. regards -- Tomas Vondra

Re: [HACKERS] Proposal: custom compression methods

2015-12-16 Thread Tomas Vondra
ession method providing both interfaces at the same time. In any case, I don't think this is the main challenge the patch needs to solve at this point. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] Proposal: custom compression methods

2015-12-16 Thread Tomas Vondra
in the future, this patch does not make it any more complicated. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to

Re: [HACKERS] Proposal: custom compression methods

2015-12-16 Thread Tomas Vondra
n't). regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] Performance improvement for joins where outer side is unique

2015-12-16 Thread Tomas Vondra
ss * Returns true if rel has some properties which can prove the relation * to be unique over some set of columns. * regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services --

Re: [HACKERS] pam auth - add rhost item

2015-12-15 Thread Tomas Vondra
regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] pam auth - add rhost item

2015-12-15 Thread Tomas Vondra
for whitespace issues. Please, make sure you use tabs for indentation (and not spaces). regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql

Re: [HACKERS] Cube extension kNN support

2015-12-15 Thread Tomas Vondra
. Was the old behavior removed? I don't think so - it seems to behave as before, so why to remove this information? Maybe it's not useful? But then why add the bit about optimizing storage of points? regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development

Re: [HACKERS] pg_hba_lookup function to get all matching pg_hba.conf entries

2015-12-15 Thread Tomas Vondra
DEFAULT NULL, ssl_inuse BOOLEAN DEFAULT NULL) regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to

Re: [HACKERS] [PATCH] Equivalence Class Filters

2015-12-15 Thread Tomas Vondra
, but I guess it might be modified to track the correlations through foreign keys. Interestingly enough, the data for the paper paper were collected on PostgreSQL, but the correlation maps were implemented in an application layer on top of the database. [1] http://www.vldb.org/pvldb/2/vldb09

Re: [HACKERS] Tsvector editing functions

2015-12-14 Thread Tomas Vondra
ome of the functions use intexterm that does not match the function name. I see two such cases - to_tsvector and setweight. Is there a reason for that? regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Serv

Re: [HACKERS] Patch: Optimize memory allocation in function 'bringetbitmap'

2015-12-14 Thread Tomas Vondra
->bt_columns[i].bv_values for example. 5) brin_deform_tuple(..) The comment before the function probably should explain the purpose of the new parameter (that it can either be NULL or an existing tuple). regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Developme

[HACKERS] PATCH: postpone building buckets to the end of Hash (in HashJoin)

2015-12-14 Thread Tomas Vondra
/19746.1443983...@sss.pgh.pa.us regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services diff --git a/src/backend/commands/explain.c b/src/backend/commands/explain.c index 12dae77..8fd9c9f 100644 --- a/src/bac

Re: [HACKERS] Let PostgreSQL's On Schedule checkpoint write buffer smooth spread cycle by tuning IsCheckpointOnSchedule?

2015-12-14 Thread Tomas Vondra
arbitrary). regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] PATCH: add pg_current_xlog_flush_location function

2015-12-13 Thread Tomas Vondra
Hi, On 12/13/2015 06:13 AM, Amit Kapila wrote: > ... > Is there a reason why you can't use existing function GetFlushRecPtr() in xlog.c? No, not really. I think I somehow missed that function when writing the initial version of the patch. Will fix in v2 of the patch. thanks --

[HACKERS] PATCH: add pg_current_xlog_flush_location function

2015-12-12 Thread Tomas Vondra
. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services diff --git a/doc/src/sgml/func.sgml b/doc/src/sgml/func.sgml index 60b9a09..0d67b82 100644 --- a/doc/src/sgml/func.sgml +++ b/doc/src/sgml/func.sgml @@ -167

Re: [HACKERS] PATCH: track last known XLOG segment in control file

2015-12-12 Thread Tomas Vondra
On 12/12/2015 11:20 PM, Andres Freund wrote: Hi, On 2015-12-12 22:14:13 +0100, Tomas Vondra wrote: this is the second improvement proposed in the thread [1] about ext4 data loss issue. It adds another field to control file, tracking the last known WAL segment. This does not eliminate

[HACKERS] PATCH: track last known XLOG segment in control file

2015-12-12 Thread Tomas Vondra
. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.c index 71fc8ff..50f10a5 100644 --- a/src/backend/access/transam/xlog.c +

Re: [HACKERS] PATCH: track last known XLOG segment in control file

2015-12-12 Thread Tomas Vondra
On 12/12/2015 11:39 PM, Andres Freund wrote: On 2015-12-12 23:28:33 +0100, Tomas Vondra wrote: On 12/12/2015 11:20 PM, Andres Freund wrote: On 2015-12-12 22:14:13 +0100, Tomas Vondra wrote: this is the second improvement proposed in the thread [1] about ext4 data loss issue. It adds another

Re: [HACKERS] PATCH: index-only scans with partial indexes

2015-12-06 Thread Tomas Vondra
over the next few days. Thanks for the help so far! -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] silent data loss with ext4 / all current versions

2015-12-01 Thread Tomas Vondra
/END_CRIT_SECTION is not available there) The patch is fairly trivial and I've done some rudimentary testing, but I'm sure I haven't exercised all the modified paths. regards Tomas -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training

Re: [HACKERS] silent data loss with ext4 / all current versions

2015-12-01 Thread Tomas Vondra
On 12/01/2015 10:44 PM, Peter Eisentraut wrote: On 11/27/15 8:18 AM, Michael Paquier wrote: On Fri, Nov 27, 2015 at 8:17 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: So, what's going on? The problem is that while the rename() is atomic, it's not guaranteed to be durable w

Re: [HACKERS] silent data loss with ext4 / all current versions

2015-11-29 Thread Tomas Vondra
On 11/29/2015 03:33 PM, Tomas Vondra wrote: Hi, On 11/29/2015 02:38 PM, Craig Ringer wrote: I've had a few tries at implementing a qemu-based crashtester where it hard kills the qemu instance at a random point then starts it back up. I've tried to reproduce the issue by killing a qemu VM

Re: [HACKERS] silent data loss with ext4 / all current versions

2015-11-29 Thread Tomas Vondra
Hi, On 11/29/2015 02:38 PM, Craig Ringer wrote: On 27 November 2015 at 21:28, Greg Stark <st...@mit.edu <mailto:st...@mit.edu>> wrote: On Fri, Nov 27, 2015 at 11:17 AM, Tomas Vondra <tomas.von...@2ndquadrant.com <mailto:tomas.von...@2ndquadrant.com>> wrote:

Re: [HACKERS] silent data loss with ext4 / all current versions

2015-11-29 Thread Tomas Vondra
On 11/29/2015 02:41 PM, Craig Ringer wrote: On 27 November 2015 at 19:17, Tomas Vondra <tomas.von...@2ndquadrant.com <mailto:tomas.von...@2ndquadrant.com>> wrote: It's also possible to mitigate this by setting wal_sync_method=fsync Are you sure? https://lwn.net/Articles/3

Re: [HACKERS] silent data loss with ext4 / all current versions

2015-11-27 Thread Tomas Vondra
On 11/27/2015 02:18 PM, Michael Paquier wrote: On Fri, Nov 27, 2015 at 8:17 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: So, what's going on? The problem is that while the rename() is atomic, it's not guaranteed to be durable without an explicit fsync on the parent dir

[HACKERS] silent data loss with ext4 / all current versions

2015-11-27 Thread Tomas Vondra
fsync the directory - then the rename won't be replayed and will be lost). regards -- Tomas Vondra http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services diff --git a/src/backend/access/transam/xlog.c b/src/backend/access/transam/xlog.

Re: [HACKERS] silent data loss with ext4 / all current versions

2015-11-27 Thread Tomas Vondra
Hi, On 11/27/2015 02:28 PM, Greg Stark wrote: On Fri, Nov 27, 2015 at 11:17 AM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: I plan to do more power failure testing soon, with more complex test scenarios. I suspect there might be other similar issues (e.g. when we rename a file

Re: [HACKERS] Bitmap index scans use of filters on available columns

2015-11-06 Thread Tomas Vondra
Hi, On 11/05/2015 07:36 PM, Robert Haas wrote: On Thu, Nov 5, 2015 at 1:29 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: But then again, can we come up with a way to distinguish operators that are safe to evaluate on indexes - either automatic or manual? We alre

Re: [HACKERS] Bitmap index scans use of filters on available columns

2015-11-06 Thread Tomas Vondra
Hi, On 11/07/2015 02:18 AM, Robert Haas wrote: On Fri, Nov 6, 2015 at 7:11 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: I think LEAKPROOF is probably fine for this. How would the new thing be different? I don't think so - AFAIK "leakproof" is about not revealing i

Re: [HACKERS] Bitmap index scans use of filters on available columns

2015-11-05 Thread Tomas Vondra
Hi, On 11/05/2015 06:51 PM, Robert Haas wrote: On Wed, Nov 4, 2015 at 9:15 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: I certainly understand there are cases that require care - like the leakproof thing pointed out by Robert for example. I don't immediately see why eval

Re: [HACKERS] GIN data corruption bug(s) in 9.6devel

2015-11-05 Thread Tomas Vondra
hem? Or is (1) obsoleted by (2) for example? regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscriptio

[HACKERS] GIN data corruption bug(s) in 9.6devel

2015-11-05 Thread Tomas Vondra
4) use "run" scipt (attached) to run the load n-times on a given commit $ run.sh master 10 NOTICE: The run script is the messiest one, you'll have to edit it to fix paths etc. regards -- Tomas Vondra http://www.2n

Re: [HACKERS] GIN data corruption bug(s) in 9.6devel

2015-11-05 Thread Tomas Vondra
Hi, On 11/06/2015 01:05 AM, Jeff Janes wrote: On Thu, Nov 5, 2015 at 3:50 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: ... I can do that - I see there are three patches in the two threads: 1) gin_pending_lwlock.patch (Jeff Janes) 2) gin_pending_pagelock.patch (Jeff

Re: [HACKERS] Bitmap index scans use of filters on available columns

2015-11-04 Thread Tomas Vondra
AM with every possible operator, but that's not quite manageable I guess. But why couldn't we provide a generic infrastructure that would allow filtering "safe" expressions and validating them on an index tuple? kind regards -- Tomas Vondra http://www.2ndQuadrant.com Post

Re: [HACKERS] pgbench gaussian/exponential docs improvements

2015-10-25 Thread Tomas Vondra
rrent status. There are alternative methods for almost every non-trivial piece of code, and we generally don't mention that in user docs. Why should we mention it in this case? Why would the user care which particular PRNG was used to generate the numbers? Maybe there really is a reason

[HACKERS] pgbench gaussian/exponential docs improvements

2015-10-25 Thread Tomas Vondra
-Muller transform. Does it make sense to explicitly mention the implementation detail (Box-Muller transform) here? regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing

Re: [HACKERS] pgbench gaussian/exponential docs improvements

2015-10-25 Thread Tomas Vondra
://en.wikipedia.org/wiki/Box%E2%80%93Muller_transform No, my point was exactly the opposite - removing the mention of Box-Muller entirely, not adding more details about it. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA

Re: [HACKERS] Multi-column distinctness.

2015-10-20 Thread Tomas Vondra
On 10/20/2015 05:59 PM, Tom Lane wrote: Robert Haas <robertmh...@gmail.com> writes: On Tue, Oct 20, 2015 at 10:51 AM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: ISTM that we could use COLLECT STATISTICS instead of ADD STATISTICS, and use REMOVE STATISTICS instead of DRO

Re: [HACKERS] Multi-column distinctness.

2015-10-20 Thread Tomas Vondra
Hi, On 10/20/2015 09:11 PM, Tom Lane wrote: I wrote: Tomas Vondra <tomas.von...@2ndquadrant.com> writes: The problem is that the actual syntax is ADD [COLUMN], so we can't simply use ADD STATISTICS as that would mean a conflict in the grammar. Resolving it means either making COLU

Re: [HACKERS] Multi-column distinctness.

2015-10-20 Thread Tomas Vondra
Hi, On 10/20/2015 01:14 PM, Simon Riggs wrote: On 19 October 2015 at 20:16, Tomas Vondra <tomas.von...@2ndquadrant.com <mailto:tomas.von...@2ndquadrant.com>> wrote: Hello Kyotaro-san, On 09/11/2015 06:58 PM, Tomas Vondra wrote: > Maybe the best solutio

Re: [HACKERS] Multi-column distinctness.

2015-10-20 Thread Tomas Vondra
On 10/20/2015 11:28 PM, Simon Riggs wrote: On 20 October 2015 at 16:48, Tomas Vondra <tomas.von...@2ndquadrant.com <mailto:tomas.von...@2ndquadrant.com>> wrote: >> On 10/20/2015 09:11 PM, Tom Lane wrote: Can't we just insist that there be only one statistics declaration

Re: [HACKERS] fs issues on software raid0 (PG_VERSION does not contain valid data)

2015-10-19 Thread Tomas Vondra
Hi, On 10/18/2015 08:37 PM, Andres Freund wrote: Hi, On 2015-10-18 20:25:29 +0200, Tomas Vondra wrote: I've been doing a lot of filesystem testing / benchmarking recently, and today I've ran into a really strange issue with ext4 on two SSD devices in a RAID-0 configuration (Linux software

Re: [HACKERS] Multi-column distinctness.

2015-10-19 Thread Tomas Vondra
Hello Kyotaro-san, On 09/11/2015 06:58 PM, Tomas Vondra wrote: > Maybe the best solution is to abandon the ALTER TABLE approach entirely, and instead invent a new set of commands CREATE STATISTICS DROP STATISTICS (ALTER STATISTICS seems a bit excessive at this point). Another th

[HACKERS] fs issues on software raid0 (PG_VERSION does not contain valid data)

2015-10-18 Thread Tomas Vondra
except for "cannot happen" from Greg. 2) http://www.postgresql.org/message-id/48331f9f.9030...@demabg.com - Essentially talks about failed RAID5 array, but that does not seem to be the case here (no RAID failures here). BTW this was done on PostgreSQL 9.4.x. regards

Re: [HACKERS] PATCH: index-only scans with partial indexes

2015-10-09 Thread Tomas Vondra
effectively need a separate list of "not implied" clauses per index-baserel combination. Maybe IndexClauseSet is not the right place, but I don't see how IndexOptInfo could work. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support,

Re: [HACKERS] PATCH: index-only scans with partial indexes

2015-10-08 Thread Tomas Vondra
the index in multiple places? Imagine for example table joined to itself, where both sides use the index with different conditions. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via p

Re: [HACKERS] PATCH: index-only scans with partial indexes

2015-09-30 Thread Tomas Vondra
sed in the index itself. Anyway some description on rclauses should be added in the comment for match_clause_to_index, instead of the comments currently added *within* the function. OK, will do. kind regards -- Tomas Vondra http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Supp

Re: [HACKERS] PATCH: index-only scans with partial indexes

2015-09-30 Thread Tomas Vondra
On 09/30/2015 12:55 PM, Tomas Vondra wrote: Hello! On 09/30/2015 10:29 AM, Kyotaro HORIGUCHI wrote: By the way your comment for indexrinfos is as following, * 'indexrinfos' is a list of RestrictInfo nodes from the query's WHERE * or JOIN conditions, excluding those implied by the index

Re: [HACKERS] PATCH: index-only scans with partial indexes

2015-09-29 Thread Tomas Vondra
Hello, On 09/29/2015 12:27 PM, Kyotaro HORIGUCHI wrote: Hi, At Sat, 26 Sep 2015 18:00:33 +0200, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote in <5606c121.10...@2ndquadrant.com> Hi, On 09/26/2015 01:28 PM, Tomas Vondra wrote: The patch does not change the chec

Re: [HACKERS] PATCH: index-only scans with partial indexes

2015-09-29 Thread Tomas Vondra
On 09/29/2015 04:57 PM, Tomas Vondra wrote: Hello, On 09/29/2015 12:27 PM, Kyotaro HORIGUCHI wrote: ... cost_index() seems to need to be fixed. It would count excluded clauses in estimate. Hmm, good point. The problem is that extract_nonindex_conditions uses baserel->baserestricti

Re: [HACKERS] PATCH: use foreign keys to improve join estimates v1

2015-09-28 Thread Tomas Vondra
provement, both compared to master and the previous versions of the patch. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To m

Re: [HACKERS] PATCH: index-only scans with partial indexes

2015-09-26 Thread Tomas Vondra
Hi, On 09/18/2015 03:46 AM, Kyotaro HORIGUCHI wrote: Hello, At Thu, 17 Sep 2015 17:40:27 +0200, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote in <55fadeeb.4000...@2ndquadrant.com> Yes, this seems sane. I've been poking at this a bit too, and I came to the same plan in gen

Re: [HACKERS] PATCH: index-only scans with partial indexes

2015-09-26 Thread Tomas Vondra
Hi, On 09/26/2015 01:28 PM, Tomas Vondra wrote: The patch does not change the check_index_only implementation - it still needs to check the clauses, just like in v1 of the patch. To make this re-check unnecessary, we'd have to stick the remaining clauses somewhere, so that check_index_only can

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-09-25 Thread Tomas Vondra
On 09/25/2015 02:54 AM, Robert Haas wrote: On Thu, Sep 24, 2015 at 1:58 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: Meh, you're right - I got the math wrong. It's 1.3% in both cases. However the question still stands - why should we handle the over-estimate in on

Re: [HACKERS] PATCH: use foreign keys to improve join estimates v1

2015-09-25 Thread Tomas Vondra
Hi, On 09/25/2015 03:39 AM, David Rowley wrote: On 24 September 2015 at 23:57, Tomas Vondra <tomas.von...@2ndquadrant.com <mailto:tomas.von...@2ndquadrant.com>> wrote: 2) find_best_match_foreign_key -- I think the comment before the fu

Re: [HACKERS] PATCH: use foreign keys to improve join estimates v1

2015-09-24 Thread Tomas Vondra
ke cardinality of a cartesian product of all the base relations, and then apply selectivities for all the join quals (in a pairwise manner). So for the three tables we take card(join) = card(f) * card(a) * card(b) * sel(f,a) * sel(f,b) and we can consider the foreign keys in the pairwise sel

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-09-24 Thread Tomas Vondra
Hi, On 09/23/2015 11:25 PM, Tom Lane wrote: Tomas Vondra <tomas.von...@2ndquadrant.com> writes: On 09/11/2015 07:16 PM, Robert Haas wrote: On Fri, Sep 11, 2015 at 1:12 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: I'm arguing for fixing the existing bug, and then address

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-09-24 Thread Tomas Vondra
On 09/24/2015 01:51 PM, Robert Haas wrote: On Thu, Sep 24, 2015 at 5:50 AM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: I however quite dislike the dismissal of the possible impact. It should be the responsibility of the person introducing the change to show that no such

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-09-24 Thread Tomas Vondra
On 09/24/2015 05:09 PM, Robert Haas wrote: On Thu, Sep 24, 2015 at 9:49 AM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: So while it does not introduce behavior change in this particular case (because it fails, as you point out), it introduces a behavior change in general - it

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-09-24 Thread Tomas Vondra
On 09/24/2015 07:42 PM, Robert Haas wrote: On Thu, Sep 24, 2015 at 12:40 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: There are two machines - one with 32GB of RAM and work_mem=2GB, the other one with 256GB of RAM and work_mem=16GB. The machines are hosting about the same data

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-09-24 Thread Tomas Vondra
On 09/24/2015 07:04 PM, Tom Lane wrote: Tomas Vondra <tomas.von...@2ndquadrant.com> writes: But what about computing the number of expected batches, but always start executing assuming no batching? And only if we actually fill work_mem, we start batching and use the expected number of b

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-09-24 Thread Tomas Vondra
2) run until exhausting work_mem 3) start batching, with the initially estimated number of batches regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgs

Re: [HACKERS] multivariate statistics / patch v7

2015-09-24 Thread Tomas Vondra
or not, etc. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/p

Re: [HACKERS] PATCH: index-only scans with partial indexes

2015-09-17 Thread Tomas Vondra
much make the original patch irrelevant, because check_index_only can simply walk over the new list. regards -- Tomas Vondra http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hac

Re: [HACKERS] PATCH: index-only scans with partial indexes

2015-09-14 Thread Tomas Vondra
idx1 (first plan) it was correctly detected we don't need to evaluate the condition on the partial index, on idx2 that's not true and we'll waste time doing that. So we probably can't just tweak the costing a bit - this probably needs to be addressed when actually building the index path. regards

Re: [HACKERS] On-demand running query plans using auto_explain and signals

2015-09-14 Thread Tomas Vondra
On 09/14/2015 01:15 PM, Shulgin, Oleksandr wrote: On Mon, Sep 14, 2015 at 11:53 AM, Tomas Vondra <tomas.von...@2ndquadrant.com <mailto:tomas.von...@2ndquadrant.com>> wrote: On 09/14/2015 10:23 AM, Shulgin, Oleksandr wrote: On Sat, Sep 12, 2015 at 11:50 AM,

Re: [HACKERS] PATCH: index-only scans with partial indexes

2015-09-14 Thread Tomas Vondra
o we don't add the clause. I'm not particularly familiar with this part of the code, so I wonder where's the best place to fix this, though. regards -- Tomas Vondra http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent

Re: [HACKERS] On-demand running query plans using auto_explain and signals

2015-09-14 Thread Tomas Vondra
On 09/14/2015 10:23 AM, Shulgin, Oleksandr wrote: On Sat, Sep 12, 2015 at 11:50 AM, Tomas Vondra <tomas.von...@2ndquadrant.com <mailto:tomas.von...@2ndquadrant.com>> wrote: ... - Attempts to get plan for simple insert queries like this INSERT INTO x SEL

Re: [HACKERS] PATCH: index-only scans with partial indexes

2015-09-13 Thread Tomas Vondra
ithout Internet access for the next 15 days, so hopefully someone else can have a look when a new version is posted. If it's still open I'll have a look when I get back. Thanks for the feedback! regards -- Tomas Vondra http://www.2ndQuadrant.com/ PostgreSQL Development

Re: [HACKERS] On-demand running query plans using auto_explain and signals

2015-09-12 Thread Tomas Vondra
ols. regards -- Tomas Vondra http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpre

Re: [HACKERS] Multi-column distinctness.

2015-09-11 Thread Tomas Vondra
ble to drop just one of them. regards -- Tomas Vondra http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postg

<    1   2   3   4   5   6   7   8   9   10   >