Re: [HACKERS] nested loop semijoin estimates

2015-05-30 Thread Tomas Vondra
' AND EXISTS (SELECT 1 FROM facttable_stat_fta4 f WHERE f.berechnungsart=t.term AND einheit IS NOT NULL); This will result in bitmap index scan no matter the visibility. -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training Services

Re: [HACKERS] nested loop semijoin estimates

2015-06-02 Thread Tomas Vondra
On 06/02/15 01:47, Josh Berkus wrote: On 06/01/2015 03:22 PM, Tomas Vondra wrote: On 06/01/15 23:47, Josh Berkus wrote: On 06/01/2015 02:18 PM, Tom Lane wrote: Anybody else want to speak for or against back-patching the patch as posted? I intentionally didn't push it in before today's

Re: [HACKERS] nested loop semijoin estimates

2015-06-02 Thread Tomas Vondra
On 06/02/15 16:37, Tom Lane wrote: Tomas Vondra tomas.von...@2ndquadrant.com writes: OK, so I did the testing today - with TPC-H and TPC-DS benchmarks. The results are good, IMHO. I'm a bit disturbed by that, because AFAICS from the plans, these queries did not involve any semi or anti joins

Re: [HACKERS] proposal: condition blocks in psql

2015-06-28 Thread Tomas Vondra
with this wholeheartedly. ISTM most of this thread is about limitations of our current DO implementation, partially addressed by adding a bunch of scripting commands, specific to psql. I don't think that's the right solution. regards -- Tomas Vondra http://www.2ndQuadrant.com/ PostgreSQL

Re: [HACKERS] proposal: condition blocks in psql

2015-06-28 Thread Tomas Vondra
Hi, On 06/28/2015 02:21 PM, Pavel Stehule wrote: 2015-06-28 14:12 GMT+02:00 Tomas Vondra tomas.von...@2ndquadrant.com mailto:tomas.von...@2ndquadrant.com: This proposal is not against to DO parametrization. It is same like conditional block in C (#ifdef). There is similarity with C statements

Re: [HACKERS] proposal: condition blocks in psql

2015-06-28 Thread Tomas Vondra
Hi, On 06/28/2015 08:47 AM, Corey Huinker wrote: 5. I'm actually using psql to connect to redshift, which doesn't have DO blocks at all. I don't see this as a reason to add features to psql, unless there are other compelling reasons for the addition. -- Tomas Vondra http

Re: [HACKERS] proposal: condition blocks in psql

2015-06-28 Thread Tomas Vondra
mimicked by session variables in psql. If you really need to do the scripting outsite PL/pgSQL, there's plenty of suitable tools for that purpose IMHO. Starting with bash, or languages like Perl or Python. And I think the same about readability ... -- Tomas Vondra http://www

Re: [HACKERS] Refactoring pgbench.c

2015-06-28 Thread Tomas Vondra
submitted a patch to remove fork-emulation, which I think would really help simplify the code (maybe -10% source in pgbench.c, less #ifs, avoid double implementations or more-complex-than-necessary implementations or not-implemented features). ... and cleanup of dead code. -- Tomas Vondra

Re: [HACKERS] proposal: condition blocks in psql

2015-06-28 Thread Tomas Vondra
the SQL script ... -- Tomas Vondra http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql

Re: [HACKERS] multivariate statistics / patch v7

2015-07-04 Thread Tomas Vondra
as your code assumes. Maybe, but I don't see what assumption is invalid? I see nothing wrong with the previous query. kind regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training Services -- Sent via pgsql-hackers mailing

Re: [HACKERS] Refactoring pgbench.c

2015-06-28 Thread Tomas Vondra
project, which was my thought. I respectfully disagree. I think that for this to be true, the files need to be somehow logically related (which I'd expect to be the output of the refactoring). But maybe you have some particular refactoring and split in mind? best regards -- Tomas Vondra

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-06-11 Thread Tomas Vondra
On 06/11/15 16:54, Tom Lane wrote: Tomas Vondra tomas.von...@2ndquadrant.com writes: Interestingly, the hash code checks for INT_MAX overflows on a number of places, but does not check for this ... Yeah, and at least at one time there were checks to prevent the hash table request from

Re: [HACKERS] multivariate statistics / patch v7

2015-07-07 Thread Tomas Vondra
with multivariate stats. kind regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref

Re: [HACKERS] PATCH: use foreign keys to improve join estimates v1

2015-08-19 Thread Tomas Vondra
it once into the selectivity, or what? There's a bunch of similar cases mentioned in the comment before clauselist_join_selectivity. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training Services From

Re: [HACKERS] PATCH: use foreign keys to improve join estimates v1

2015-08-19 Thread Tomas Vondra
On 08/20/2015 03:49 AM, Tomas Vondra wrote: Then on current master, I get these estimates (showing just rows, because that's what matter): while with the patch I get this: And of course I forgot to include the plans from master, so here we go: select * from f join a on (f1 = a1 and f2 = a2

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-08-19 Thread Tomas Vondra
are features, and we're dealing with a bug fix here. kind regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-08-19 Thread Tomas Vondra
size the hashtable - that's not something we should do in a bugfix I think. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make

Re: [HACKERS] (full) Memory context dump considered harmful

2015-08-21 Thread Tomas Vondra
On 08/21/2015 08:37 PM, Tom Lane wrote: Tomas Vondra tomas.von...@2ndquadrant.com writes: I also don't think logging just subset of the stats is a lost case. Sure, we can't know which of the lines are important, but for example logging just the top-level contexts with a summary of the child

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-08-21 Thread Tomas Vondra
. I don't quite see a reason to wait for the over-estimation patch. We probably should backpatch the bugfix anyway (although it's much less likely to run into that before 9.5), and we can't really backpatch the behavior change there (as there's no hash resize). regards -- Tomas Vondra

Re: [HACKERS] PATCH: numeric timestamp in log_line_prefix

2015-08-22 Thread Tomas Vondra
those changes are quite trivial. The only annoying bit is that both '%u' and '%e' are already used, so none of the obvious choices for 'Unix Epoch' are available. So I simply took (%m+1) which is %n. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7

Re: [HACKERS] PATCH: numeric timestamp in log_line_prefix

2015-08-22 Thread Tomas Vondra
' or 't' for instance. Hmmm, I'm not entirely sure how exactly the padding is supposed to work (IIRC I've never used it), and I thought it behaved correctly. But maybe not - I think the safest thing is copy what 't' does, so I've done that in attached v3 of the patch. regards -- Tomas Vondra

Re: [HACKERS] Performance improvement for joins where outer side is unique

2015-08-23 Thread Tomas Vondra
unique)' text, so Hash Join (inner unique) ... instead of Hash Join(inner unique) but that's just nitpicking at this point. Otherwise the patch seems quite solid to me. regard -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA

Re: [HACKERS] PATCH: numeric timestamp in log_line_prefix

2015-08-23 Thread Tomas Vondra
'.' in the millisecond format. Thanks for spotting that. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription

Re: [HACKERS] statistics for array types

2015-08-20 Thread Tomas Vondra
gathered but nothing met the threshold. I'm not sure whether this is the same thing I just proposed ... regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers

Re: [HACKERS] (full) Memory context dump considered harmful

2015-08-22 Thread Tomas Vondra
On 08/22/2015 06:06 PM, Tom Lane wrote: Tomas Vondra tomas.von...@2ndquadrant.com writes: Couldn't we make it a bit smarter to handle even cases like this? For example we might first count/sum the child contexts, and then print either all child contexts (if there are only a few of them

Re: [HACKERS] PATCH: numeric timestamp in log_line_prefix

2015-08-22 Thread Tomas Vondra
. I would suggest to maybe follow Tom's %u idea and fix the implementation details wrt to comments received? Yes, I plan to update it according to those comments. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-08-20 Thread Tomas Vondra
be addressed in a separate patch. kind regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

Re: [HACKERS] (full) Memory context dump considered harmful

2015-08-20 Thread Tomas Vondra
) or log_memory_stats = (full|short|off) might work. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-08-19 Thread Tomas Vondra
Hi, On 08/20/2015 04:15 AM, Tomas Vondra wrote: Hello KaiGain-san, On 08/19/2015 03:19 PM, Kohei KaiGai wrote: Unless we have no fail-safe mechanism when planner estimated too large number of tuples than actual needs, a strange estimation will consume massive amount of RAMs. It's a bad side

Re: [HACKERS] multivariate statistics / patch v7

2015-07-30 Thread Tomas Vondra
Hi, On 07/30/2015 10:21 AM, Heikki Linnakangas wrote: On 05/25/2015 11:43 PM, Tomas Vondra wrote: There are 6 files attached, but only 0002-0006 are actually part of the multivariate statistics patch itself. All of these patches are huge. In order to review this in a reasonable amount

Re: [HACKERS] multivariate statistics / patch v7

2015-07-30 Thread Tomas Vondra
, IMHO. regards Tomas -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref

Re: [HACKERS] Remaining 'needs review' patchs in July commitfest

2015-07-28 Thread Tomas Vondra
discussion of low-level implementation details (e.g. how exactly the histograms are built, data structures etc.) - we haven't got that far yet. -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training Services -- Sent via pgsql-hackers

Re: [HACKERS] multivariate statistics / patch v7

2015-07-27 Thread Tomas Vondra
Hello Horiguchi-san, On 07/27/2015 09:04 AM, Kyotaro HORIGUCHI wrote: Hello, At Sat, 25 Jul 2015 23:09:31 +0200, Tomas Vondra tomas.von...@2ndquadrant.com wrote in 55b3fb0b.7000...@2ndquadrant.com Hi, On 07/16/2015 01:51 PM, Kyotaro HORIGUCHI wrote: Hi, I'd like to show you the modified

Re: [HACKERS] multivariate statistics / patch v7

2015-07-25 Thread Tomas Vondra
the implementation (adding support for another operator). kind regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your

Re: [HACKERS] multivariate statistics / patch v7

2015-07-14 Thread Tomas Vondra
-processing before clauselist_selectivity (I assume that's the point), we'd end up repeating a lot of the code. Or maybe not, I'm not sure. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training Services -- Sent via pgsql

Re: [HACKERS] Memory Accounting v11

2015-07-14 Thread Tomas Vondra
, finalfunc and sizefunc. That seems doable, I guess. I find the memory accounting as a way more elegant solution, though. kind regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training Services -- Sent via pgsql-hackers mailing

Re: [HACKERS] Memory Accounting v11

2015-07-15 Thread Tomas Vondra
Hi, On 07/15/2015 09:21 PM, Robert Haas wrote: On Tue, Jul 14, 2015 at 9:14 PM, Tomas Vondra tomas.von...@2ndquadrant.com wrote: Firstly, do we really have good benchmarks and measurements? I really doubt that. We do have some numbers for REINDEX, where we observed 0.5-1% regression on noisy

Re: [HACKERS] Memory Accounting v11

2015-07-15 Thread Tomas Vondra
will drop this patch and proceed with the HashAgg patch, with a heuristic for internal types. Could someone briefly explain what heuristics are we talking about? -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training Services -- Sent

Re: [HACKERS] pgbench gaussian/exponential docs improvements

2015-10-25 Thread Tomas Vondra
rrent status. There are alternative methods for almost every non-trivial piece of code, and we generally don't mention that in user docs. Why should we mention it in this case? Why would the user care which particular PRNG was used to generate the numbers? Maybe there really is a reason

[HACKERS] pgbench gaussian/exponential docs improvements

2015-10-25 Thread Tomas Vondra
-Muller transform. Does it make sense to explicitly mention the implementation detail (Box-Muller transform) here? regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing

Re: [HACKERS] pgbench gaussian/exponential docs improvements

2015-10-25 Thread Tomas Vondra
://en.wikipedia.org/wiki/Box%E2%80%93Muller_transform No, my point was exactly the opposite - removing the mention of Box-Muller entirely, not adding more details about it. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA

Re: [HACKERS] Bitmap index scans use of filters on available columns

2015-11-04 Thread Tomas Vondra
AM with every possible operator, but that's not quite manageable I guess. But why couldn't we provide a generic infrastructure that would allow filtering "safe" expressions and validating them on an index tuple? kind regards -- Tomas Vondra http://www.2ndQuadrant.com Post

Re: [HACKERS] Bitmap index scans use of filters on available columns

2015-11-06 Thread Tomas Vondra
Hi, On 11/05/2015 07:36 PM, Robert Haas wrote: On Thu, Nov 5, 2015 at 1:29 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: But then again, can we come up with a way to distinguish operators that are safe to evaluate on indexes - either automatic or manual? We alre

Re: [HACKERS] Bitmap index scans use of filters on available columns

2015-11-05 Thread Tomas Vondra
Hi, On 11/05/2015 06:51 PM, Robert Haas wrote: On Wed, Nov 4, 2015 at 9:15 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: I certainly understand there are cases that require care - like the leakproof thing pointed out by Robert for example. I don't immediately see why eval

Re: [HACKERS] Bitmap index scans use of filters on available columns

2015-11-06 Thread Tomas Vondra
Hi, On 11/07/2015 02:18 AM, Robert Haas wrote: On Fri, Nov 6, 2015 at 7:11 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: I think LEAKPROOF is probably fine for this. How would the new thing be different? I don't think so - AFAIK "leakproof" is about not revealing i

Re: [HACKERS] GIN data corruption bug(s) in 9.6devel

2015-11-05 Thread Tomas Vondra
hem? Or is (1) obsoleted by (2) for example? regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscriptio

[HACKERS] GIN data corruption bug(s) in 9.6devel

2015-11-05 Thread Tomas Vondra
4) use "run" scipt (attached) to run the load n-times on a given commit $ run.sh master 10 NOTICE: The run script is the messiest one, you'll have to edit it to fix paths etc. regards -- Tomas Vondra http://www.2n

Re: [HACKERS] GIN data corruption bug(s) in 9.6devel

2015-11-05 Thread Tomas Vondra
Hi, On 11/06/2015 01:05 AM, Jeff Janes wrote: On Thu, Nov 5, 2015 at 3:50 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: ... I can do that - I see there are three patches in the two threads: 1) gin_pending_lwlock.patch (Jeff Janes) 2) gin_pending_pagelock.patch (Jeff

Re: [HACKERS] fs issues on software raid0 (PG_VERSION does not contain valid data)

2015-10-19 Thread Tomas Vondra
Hi, On 10/18/2015 08:37 PM, Andres Freund wrote: Hi, On 2015-10-18 20:25:29 +0200, Tomas Vondra wrote: I've been doing a lot of filesystem testing / benchmarking recently, and today I've ran into a really strange issue with ext4 on two SSD devices in a RAID-0 configuration (Linux software

Re: [HACKERS] PATCH: index-only scans with partial indexes

2015-10-08 Thread Tomas Vondra
the index in multiple places? Imagine for example table joined to itself, where both sides use the index with different conditions. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via p

Re: [HACKERS] Multi-column distinctness.

2015-10-19 Thread Tomas Vondra
Hello Kyotaro-san, On 09/11/2015 06:58 PM, Tomas Vondra wrote: > Maybe the best solution is to abandon the ALTER TABLE approach entirely, and instead invent a new set of commands CREATE STATISTICS DROP STATISTICS (ALTER STATISTICS seems a bit excessive at this point). Another th

[HACKERS] fs issues on software raid0 (PG_VERSION does not contain valid data)

2015-10-18 Thread Tomas Vondra
except for "cannot happen" from Greg. 2) http://www.postgresql.org/message-id/48331f9f.9030...@demabg.com - Essentially talks about failed RAID5 array, but that does not seem to be the case here (no RAID failures here). BTW this was done on PostgreSQL 9.4.x. regards

Re: [HACKERS] Multi-column distinctness.

2015-10-20 Thread Tomas Vondra
On 10/20/2015 05:59 PM, Tom Lane wrote: Robert Haas <robertmh...@gmail.com> writes: On Tue, Oct 20, 2015 at 10:51 AM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: ISTM that we could use COLLECT STATISTICS instead of ADD STATISTICS, and use REMOVE STATISTICS instead of DRO

Re: [HACKERS] Multi-column distinctness.

2015-10-20 Thread Tomas Vondra
Hi, On 10/20/2015 09:11 PM, Tom Lane wrote: I wrote: Tomas Vondra <tomas.von...@2ndquadrant.com> writes: The problem is that the actual syntax is ADD [COLUMN], so we can't simply use ADD STATISTICS as that would mean a conflict in the grammar. Resolving it means either making COLU

Re: [HACKERS] Multi-column distinctness.

2015-10-20 Thread Tomas Vondra
Hi, On 10/20/2015 01:14 PM, Simon Riggs wrote: On 19 October 2015 at 20:16, Tomas Vondra <tomas.von...@2ndquadrant.com <mailto:tomas.von...@2ndquadrant.com>> wrote: Hello Kyotaro-san, On 09/11/2015 06:58 PM, Tomas Vondra wrote: > Maybe the best solutio

Re: [HACKERS] Multi-column distinctness.

2015-10-20 Thread Tomas Vondra
On 10/20/2015 11:28 PM, Simon Riggs wrote: On 20 October 2015 at 16:48, Tomas Vondra <tomas.von...@2ndquadrant.com <mailto:tomas.von...@2ndquadrant.com>> wrote: >> On 10/20/2015 09:11 PM, Tom Lane wrote: Can't we just insist that there be only one statistics declaration

Re: [HACKERS] PATCH: index-only scans with partial indexes

2015-10-09 Thread Tomas Vondra
effectively need a separate list of "not implied" clauses per index-baserel combination. Maybe IndexClauseSet is not the right place, but I don't see how IndexOptInfo could work. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support,

Re: [HACKERS] PATCH: index-only scans with partial indexes

2015-07-11 Thread Tomas Vondra
Hi, On 07/10/2015 10:43 PM, Tom Lane wrote: Tomas Vondra tomas.von...@2ndquadrant.com writes: currently partial indexes end up not using index only scans in most cases, because check_index_only() is overly conservative, as explained in this comment: ... I've done a bunch of tests, and I do

[HACKERS] strange plan with bitmap heap scan and multiple partial indexes

2015-07-11 Thread Tomas Vondra
t WHERE b 100; QUERY PLAN Index Scan using idx001 on t (cost=0.14..29.14 rows=1000 width=4) Any idea what's going on here? FWIW all this was on 51d0fe5d (July 23). -- Tomas Vondra

Re: [HACKERS] strange plan with bitmap heap scan and multiple partial indexes

2015-07-11 Thread Tomas Vondra
this to skip some of the expensive stuff? We should have the selectivities anyway, no? regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org

Re: [HACKERS] strange plan with bitmap heap scan and multiple partial indexes

2015-07-11 Thread Tomas Vondra
Hi, On 07/11/2015 11:40 PM, Tom Lane wrote: Tomas Vondra tomas.von...@2ndquadrant.com writes: So I think the predicate proofing is a better approach, but of course the planning cost may be an issue. But maybe we can make this cheaper by some clever tricks? For example, given two predicates

Re: [HACKERS] multivariate statistics / patch v7

2015-07-07 Thread Tomas Vondra
Hello Horiguchi-san! On 07/07/2015 09:43 PM, Tomas Vondra wrote: -- histograms ALTER TABLE t ADD STATISTICS (histogram) on (a,b,c); ANALYZE t; EXPLAIN ANALYZE select * from t where a 0.3 and b 0.3 and c 0.3; Seq Scan on t (cost=0.00..23870.00 rows=267033 width=24) (actual

Re: [HACKERS] [PROPOSAL] Effective storage of duplicates in B-tree index.

2015-08-31 Thread Tomas Vondra
e are many updates. Which brings me to the other benefit of btree indexes - they are designed for high concurrency. How much is this going to be affected by introducing the posting lists? kind regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 S

Re: [HACKERS] Scaling PostgreSQL at multicore Power8

2015-08-31 Thread Tomas Vondra
ts? Well, I could test the patch on a x86 machine with 4 sockets (64 cores), but I wonder whether it makes sense at this point, as the patch really is not correct (judging by what Andres says). Also, what pgbench scale was used for the testing? regards -- Tomas Vondra

Re: [HACKERS] Scaling PostgreSQL at multicore Power8

2015-08-31 Thread Tomas Vondra
On 08/31/2015 05:48 PM, Andres Freund wrote: On 2015-08-31 17:43:08 +0200, Tomas Vondra wrote: Well, I could test the patch on a x86 machine with 4 sockets (64 cores), but I wonder whether it makes sense at this point, as the patch really is not correct (judging by what Andres says

Re: [HACKERS] Horizontal scalability/sharding

2015-09-01 Thread Tomas Vondra
multiple copies of a each shard) may be simpler than combining sharding and standbys, but I don't see why it makes low-data-loss HA impossible. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via p

Re: [HACKERS] Horizontal scalability/sharding

2015-09-01 Thread Tomas Vondra
to run into both (all the processes also have to share all the caches, killing them). I don't know if sharding can help solving (or at least improve) these issues. And if sharding in general can, I don't know if it still holds for FDW-based solution. regards -- Tomas Vondra http

Re: [HACKERS] Horizontal scalability/sharding

2015-09-01 Thread Tomas Vondra
Hi, On 09/01/2015 08:22 PM, Andres Freund wrote: On 2015-09-01 14:11:21 -0400, Robert Haas wrote: On Tue, Sep 1, 2015 at 2:04 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: Memory bandwidth, for example. It's quite difficult to spot, because the intuition is that memory i

Re: [HACKERS] Horizontal scalability/sharding

2015-09-01 Thread Tomas Vondra
Hi, On 09/01/2015 09:19 PM, Josh Berkus wrote: On 09/01/2015 11:36 AM, Tomas Vondra wrote: We want multiple copies of shards created by the sharding system itself. Having a separate, and completely orthagonal, redundancy system to the sharding system is overly burdensome on the DBA and makes

Re: [HACKERS] Horizontal scalability/sharding

2015-09-03 Thread Tomas Vondra
bitrary group of K nodes (where K is the number of replicas for each piece of data). It's also non-trivial to do this when you have to consider racks, data centers etc. With regular slaves you can't do any of this - no matter what you do, you have to load balance the additional load only on

Re: [HACKERS] Allow a per-tablespace effective_io_concurrency setting

2015-09-02 Thread Tomas Vondra
ndon the "number of spindles" idea, and just say "number of I/O requests to prefetch". Possibly with an explanation of how to estimate it (devices * queue length). regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Re

Re: [HACKERS] September 2015 Commitfest

2015-09-03 Thread Tomas Vondra
is that several authors with, in some cases numerous or large, patches are not doing the corresponding amount of review on other patches. Let's change that! Given the size of the multivariate stats patch, I guess I'm one of those slackers, Will fix. regards -- Tomas Vondra http://www

Re: [HACKERS] PATCH: index-only scans with partial indexes

2015-09-04 Thread Tomas Vondra
too much time (which would make the additional planning time noise). However that was the idea of the benchmark, and I got no difference. regards Tomas -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services queri

Re: [HACKERS] PATCH: index-only scans with partial indexes

2015-09-05 Thread Tomas Vondra
Hi, On 09/05/2015 10:53 AM, Simon Riggs wrote: On 4 September 2015 at 22:03, Tomas Vondra <tomas.von...@2ndquadrant.com <mailto:tomas.von...@2ndquadrant.com>> wrote: A summary of 100 EXPLAIN timings looks like this: master A B C

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-09-08 Thread Tomas Vondra
t, assuming someone takes the time to measure how expensive the additional resize actually is), we'll still have to fix the repalloc(). So I still fail to see why we shouldn't apply this fix. regards -- Tomas Vondra http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support

Re: [HACKERS] Allow a per-tablespace effective_io_concurrency setting

2015-09-02 Thread Tomas Vondra
On 09/03/2015 12:23 AM, Andres Freund wrote: On 2015-09-02 14:31:35 -0700, Josh Berkus wrote: On 09/02/2015 02:25 PM, Tomas Vondra wrote: As I explained, spindles have very little to do with it - you need multiple I/O requests per device, to get the benefit. Sure, the DBAs should know how

Re: [HACKERS] Horizontal scalability/sharding

2015-09-02 Thread Tomas Vondra
e have pretty clear idea of the two use cases. I'm sure it's non-trivial to design it well for both cases, but I think the questions for FWD/sharding will be much more about abstract concepts than particular technical solutions. regards -- Tomas Vondra http://www.2ndQuadrant.co

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-09-02 Thread Tomas Vondra
ray (usually resulting in crashes due to invalid pointers). I fixed it to memset(hashtable->buckets, 0, nbuckets * sizeof(HashJoinTuple)); Fixed patch attached (marked as v2). kind regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remo

Re: [HACKERS] Allow a per-tablespace effective_io_concurrency setting

2015-09-02 Thread Tomas Vondra
to read before the end of the current device". I wasn't sure exactly what we would do with something like that but it would be better than just guessing how many I/O operations we need to issue to keep all the spindles busy. I don't really see how that would help us? regards -- Tomas

Re: [HACKERS] [PROPOSAL] Effective storage of duplicates in B-tree index.

2015-09-01 Thread Tomas Vondra
nst any compression and for any bloat. Which might be a problem for some use cases, but I assume we could add an option disabling this per-index. Probably having it "off" by default, and only enabling the compression explicitly. regards -- Tomas Vondra http://www.2ndQuadrant.

Re: [HACKERS] Multi-column distinctness.

2015-09-06 Thread Tomas Vondra
Hello, On 09/06/2015 10:24 AM, Simon Riggs wrote: On 28 August 2015 at 09:33, Kyotaro HORIGUCHI <horiguchi.kyot...@lab.ntt.co.jp <mailto:horiguchi.kyot...@lab.ntt.co.jp>> wrote: Tomas Vondra is now working on heavily-equipped multivariate statistics for OLAP usage.

Re: [HACKERS] Multi-column distinctness.

2015-09-06 Thread Tomas Vondra
execution. Tomas Vondra is now working on heavily-equipped multivariate statistics for OLAP usage. In contrast, this is a lightly implemented solution which calculates only the ratio between a rows estimated by current method and a actual row number. I think this doesn't conflict with his work except

Re: [HACKERS] Multi-column distinctness.

2015-09-06 Thread Tomas Vondra
On Sep 6, 2015 10:31, "Tomas Vondra" <tomas.von...@2ndquadrant.com> wrote: > > 5) syntax > - > The syntax might be one of the pain points if we eventually decide to commit the multivariate stats patch. I have no intention in blocking this patch for that rea

Re: [HACKERS] Commitfest remaining Needs Review items

2015-08-25 Thread Tomas Vondra
progress) Data is passed through queues (hopefully with low overhead thanks to the batching). regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-09-09 Thread Tomas Vondra
On 09/09/2015 03:55 PM, Robert Haas wrote: On Tue, Sep 8, 2015 at 5:02 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: Also, I'm not sure what other places do you have in mind (could you list some examples?) but I'd bet we limit the allocation to 1GB because of the palloc()

Re: [HACKERS] PATCH: index-only scans with partial indexes

2015-09-14 Thread Tomas Vondra
idx1 (first plan) it was correctly detected we don't need to evaluate the condition on the partial index, on idx2 that's not true and we'll waste time doing that. So we probably can't just tweak the costing a bit - this probably needs to be addressed when actually building the index path. regards

Re: [HACKERS] On-demand running query plans using auto_explain and signals

2015-09-14 Thread Tomas Vondra
On 09/14/2015 01:15 PM, Shulgin, Oleksandr wrote: On Mon, Sep 14, 2015 at 11:53 AM, Tomas Vondra <tomas.von...@2ndquadrant.com <mailto:tomas.von...@2ndquadrant.com>> wrote: On 09/14/2015 10:23 AM, Shulgin, Oleksandr wrote: On Sat, Sep 12, 2015 at 11:50 AM,

Re: [HACKERS] PATCH: index-only scans with partial indexes

2015-09-13 Thread Tomas Vondra
ithout Internet access for the next 15 days, so hopefully someone else can have a look when a new version is posted. If it's still open I'll have a look when I get back. Thanks for the feedback! regards -- Tomas Vondra http://www.2ndQuadrant.com/ PostgreSQL Development

Re: [HACKERS] Multi-column distinctness.

2015-09-11 Thread Tomas Vondra
ble to drop just one of them. regards -- Tomas Vondra http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postg

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-09-11 Thread Tomas Vondra
On 09/11/2015 07:16 PM, Robert Haas wrote: On Fri, Sep 11, 2015 at 1:12 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: I'm arguing for fixing the existing bug, and then addressing the case of over-estimation separately, with proper analysis. Well, this is part of how we're l

Re: [HACKERS] On-demand running query plans using auto_explain and signals

2015-09-12 Thread Tomas Vondra
ols. regards -- Tomas Vondra http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpre

Re: [HACKERS] PATCH: index-only scans with partial indexes

2015-09-17 Thread Tomas Vondra
much make the original patch irrelevant, because check_index_only can simply walk over the new list. regards -- Tomas Vondra http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hac

Re: [HACKERS] PATCH: index-only scans with partial indexes

2015-09-14 Thread Tomas Vondra
o we don't add the clause. I'm not particularly familiar with this part of the code, so I wonder where's the best place to fix this, though. regards -- Tomas Vondra http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent

Re: [HACKERS] Partitioned checkpointing

2015-09-11 Thread Tomas Vondra
ain. But perhaps partitioning the checkpoints is not the best approach? regards -- Tomas Vondra http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-09-11 Thread Tomas Vondra
On 09/11/2015 06:55 PM, Robert Haas wrote: On Wed, Sep 9, 2015 at 11:54 AM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: Secondly, we limit the number of buckets to INT_MAX, so about 16GB (because buckets are just pointers). No matter how awful estimate you get (or how insanely hi

Re: [HACKERS] On-demand running query plans using auto_explain and signals

2015-09-14 Thread Tomas Vondra
On 09/14/2015 10:23 AM, Shulgin, Oleksandr wrote: On Sat, Sep 12, 2015 at 11:50 AM, Tomas Vondra <tomas.von...@2ndquadrant.com <mailto:tomas.von...@2ndquadrant.com>> wrote: ... - Attempts to get plan for simple insert queries like this INSERT INTO x SEL

Re: [HACKERS] PATCH: index-only scans with partial indexes

2015-09-26 Thread Tomas Vondra
Hi, On 09/18/2015 03:46 AM, Kyotaro HORIGUCHI wrote: Hello, At Thu, 17 Sep 2015 17:40:27 +0200, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote in <55fadeeb.4000...@2ndquadrant.com> Yes, this seems sane. I've been poking at this a bit too, and I came to the same plan in gen

Re: [HACKERS] PATCH: index-only scans with partial indexes

2015-09-29 Thread Tomas Vondra
Hello, On 09/29/2015 12:27 PM, Kyotaro HORIGUCHI wrote: Hi, At Sat, 26 Sep 2015 18:00:33 +0200, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote in <5606c121.10...@2ndquadrant.com> Hi, On 09/26/2015 01:28 PM, Tomas Vondra wrote: The patch does not change the chec

Re: [HACKERS] PATCH: index-only scans with partial indexes

2015-09-29 Thread Tomas Vondra
On 09/29/2015 04:57 PM, Tomas Vondra wrote: Hello, On 09/29/2015 12:27 PM, Kyotaro HORIGUCHI wrote: ... cost_index() seems to need to be fixed. It would count excluded clauses in estimate. Hmm, good point. The problem is that extract_nonindex_conditions uses baserel->baserestricti

Re: [HACKERS] PATCH: index-only scans with partial indexes

2015-09-30 Thread Tomas Vondra
sed in the index itself. Anyway some description on rclauses should be added in the comment for match_clause_to_index, instead of the comments currently added *within* the function. OK, will do. kind regards -- Tomas Vondra http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Supp

<    3   4   5   6   7   8   9   10   11   12   >