Re: [HACKERS] DBT-3 with SF=20 got failed

2015-09-11 Thread Tomas Vondra
On 09/11/2015 07:16 PM, Robert Haas wrote: On Fri, Sep 11, 2015 at 1:12 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: I'm arguing for fixing the existing bug, and then addressing the case of over-estimation separately, with proper analysis. Well, this is part of how we're l

Re: [HACKERS] Partitioned checkpointing

2015-09-11 Thread Tomas Vondra
ain. But perhaps partitioning the checkpoints is not the best approach? regards -- Tomas Vondra http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-09-11 Thread Tomas Vondra
On 09/11/2015 06:55 PM, Robert Haas wrote: On Wed, Sep 9, 2015 at 11:54 AM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: Secondly, we limit the number of buckets to INT_MAX, so about 16GB (because buckets are just pointers). No matter how awful estimate you get (or how insanely hi

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-09-09 Thread Tomas Vondra
On 09/09/2015 03:55 PM, Robert Haas wrote: On Tue, Sep 8, 2015 at 5:02 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: Also, I'm not sure what other places do you have in mind (could you list some examples?) but I'd bet we limit the allocation to 1GB because of the palloc()

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-09-08 Thread Tomas Vondra
t, assuming someone takes the time to measure how expensive the additional resize actually is), we'll still have to fix the repalloc(). So I still fail to see why we shouldn't apply this fix. regards -- Tomas Vondra http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support

Re: [HACKERS] Multi-column distinctness.

2015-09-06 Thread Tomas Vondra
Hello, On 09/06/2015 10:24 AM, Simon Riggs wrote: On 28 August 2015 at 09:33, Kyotaro HORIGUCHI <horiguchi.kyot...@lab.ntt.co.jp <mailto:horiguchi.kyot...@lab.ntt.co.jp>> wrote: Tomas Vondra is now working on heavily-equipped multivariate statistics for OLAP usage.

Re: [HACKERS] Multi-column distinctness.

2015-09-06 Thread Tomas Vondra
execution. Tomas Vondra is now working on heavily-equipped multivariate statistics for OLAP usage. In contrast, this is a lightly implemented solution which calculates only the ratio between a rows estimated by current method and a actual row number. I think this doesn't conflict with his work except

Re: [HACKERS] Multi-column distinctness.

2015-09-06 Thread Tomas Vondra
On Sep 6, 2015 10:31, "Tomas Vondra" <tomas.von...@2ndquadrant.com> wrote: > > 5) syntax > - > The syntax might be one of the pain points if we eventually decide to commit the multivariate stats patch. I have no intention in blocking this patch for that rea

Re: [HACKERS] PATCH: index-only scans with partial indexes

2015-09-05 Thread Tomas Vondra
Hi, On 09/05/2015 10:53 AM, Simon Riggs wrote: On 4 September 2015 at 22:03, Tomas Vondra <tomas.von...@2ndquadrant.com <mailto:tomas.von...@2ndquadrant.com>> wrote: A summary of 100 EXPLAIN timings looks like this: master A B C

Re: [HACKERS] PATCH: index-only scans with partial indexes

2015-09-04 Thread Tomas Vondra
too much time (which would make the additional planning time noise). However that was the idea of the benchmark, and I got no difference. regards Tomas -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services queri

Re: [HACKERS] Horizontal scalability/sharding

2015-09-03 Thread Tomas Vondra
bitrary group of K nodes (where K is the number of replicas for each piece of data). It's also non-trivial to do this when you have to consider racks, data centers etc. With regular slaves you can't do any of this - no matter what you do, you have to load balance the additional load only on

Re: [HACKERS] September 2015 Commitfest

2015-09-03 Thread Tomas Vondra
is that several authors with, in some cases numerous or large, patches are not doing the corresponding amount of review on other patches. Let's change that! Given the size of the multivariate stats patch, I guess I'm one of those slackers, Will fix. regards -- Tomas Vondra http://www

Re: [HACKERS] Allow a per-tablespace effective_io_concurrency setting

2015-09-02 Thread Tomas Vondra
ndon the "number of spindles" idea, and just say "number of I/O requests to prefetch". Possibly with an explanation of how to estimate it (devices * queue length). regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Re

Re: [HACKERS] Allow a per-tablespace effective_io_concurrency setting

2015-09-02 Thread Tomas Vondra
On 09/03/2015 12:23 AM, Andres Freund wrote: On 2015-09-02 14:31:35 -0700, Josh Berkus wrote: On 09/02/2015 02:25 PM, Tomas Vondra wrote: As I explained, spindles have very little to do with it - you need multiple I/O requests per device, to get the benefit. Sure, the DBAs should know how

Re: [HACKERS] Horizontal scalability/sharding

2015-09-02 Thread Tomas Vondra
e have pretty clear idea of the two use cases. I'm sure it's non-trivial to design it well for both cases, but I think the questions for FWD/sharding will be much more about abstract concepts than particular technical solutions. regards -- Tomas Vondra http://www.2ndQuadrant.co

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-09-02 Thread Tomas Vondra
ray (usually resulting in crashes due to invalid pointers). I fixed it to memset(hashtable->buckets, 0, nbuckets * sizeof(HashJoinTuple)); Fixed patch attached (marked as v2). kind regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remo

Re: [HACKERS] Allow a per-tablespace effective_io_concurrency setting

2015-09-02 Thread Tomas Vondra
to read before the end of the current device". I wasn't sure exactly what we would do with something like that but it would be better than just guessing how many I/O operations we need to issue to keep all the spindles busy. I don't really see how that would help us? regards -- Tomas

Re: [HACKERS] Horizontal scalability/sharding

2015-09-01 Thread Tomas Vondra
multiple copies of a each shard) may be simpler than combining sharding and standbys, but I don't see why it makes low-data-loss HA impossible. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via p

Re: [HACKERS] Horizontal scalability/sharding

2015-09-01 Thread Tomas Vondra
to run into both (all the processes also have to share all the caches, killing them). I don't know if sharding can help solving (or at least improve) these issues. And if sharding in general can, I don't know if it still holds for FDW-based solution. regards -- Tomas Vondra http

Re: [HACKERS] Horizontal scalability/sharding

2015-09-01 Thread Tomas Vondra
Hi, On 09/01/2015 08:22 PM, Andres Freund wrote: On 2015-09-01 14:11:21 -0400, Robert Haas wrote: On Tue, Sep 1, 2015 at 2:04 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: Memory bandwidth, for example. It's quite difficult to spot, because the intuition is that memory i

Re: [HACKERS] Horizontal scalability/sharding

2015-09-01 Thread Tomas Vondra
Hi, On 09/01/2015 09:19 PM, Josh Berkus wrote: On 09/01/2015 11:36 AM, Tomas Vondra wrote: We want multiple copies of shards created by the sharding system itself. Having a separate, and completely orthagonal, redundancy system to the sharding system is overly burdensome on the DBA and makes

Re: [HACKERS] [PROPOSAL] Effective storage of duplicates in B-tree index.

2015-09-01 Thread Tomas Vondra
nst any compression and for any bloat. Which might be a problem for some use cases, but I assume we could add an option disabling this per-index. Probably having it "off" by default, and only enabling the compression explicitly. regards -- Tomas Vondra http://www.2ndQuadrant.

Re: [HACKERS] [PROPOSAL] Effective storage of duplicates in B-tree index.

2015-08-31 Thread Tomas Vondra
e are many updates. Which brings me to the other benefit of btree indexes - they are designed for high concurrency. How much is this going to be affected by introducing the posting lists? kind regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 S

Re: [HACKERS] Scaling PostgreSQL at multicore Power8

2015-08-31 Thread Tomas Vondra
ts? Well, I could test the patch on a x86 machine with 4 sockets (64 cores), but I wonder whether it makes sense at this point, as the patch really is not correct (judging by what Andres says). Also, what pgbench scale was used for the testing? regards -- Tomas Vondra

Re: [HACKERS] Scaling PostgreSQL at multicore Power8

2015-08-31 Thread Tomas Vondra
On 08/31/2015 05:48 PM, Andres Freund wrote: On 2015-08-31 17:43:08 +0200, Tomas Vondra wrote: Well, I could test the patch on a x86 machine with 4 sockets (64 cores), but I wonder whether it makes sense at this point, as the patch really is not correct (judging by what Andres says

Re: [HACKERS] Commitfest remaining Needs Review items

2015-08-25 Thread Tomas Vondra
progress) Data is passed through queues (hopefully with low overhead thanks to the batching). regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers

Re: [HACKERS] Performance improvement for joins where outer side is unique

2015-08-23 Thread Tomas Vondra
unique)' text, so Hash Join (inner unique) ... instead of Hash Join(inner unique) but that's just nitpicking at this point. Otherwise the patch seems quite solid to me. regard -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA

Re: [HACKERS] PATCH: numeric timestamp in log_line_prefix

2015-08-23 Thread Tomas Vondra
'.' in the millisecond format. Thanks for spotting that. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription

Re: [HACKERS] PATCH: numeric timestamp in log_line_prefix

2015-08-22 Thread Tomas Vondra
those changes are quite trivial. The only annoying bit is that both '%u' and '%e' are already used, so none of the obvious choices for 'Unix Epoch' are available. So I simply took (%m+1) which is %n. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7

Re: [HACKERS] PATCH: numeric timestamp in log_line_prefix

2015-08-22 Thread Tomas Vondra
' or 't' for instance. Hmmm, I'm not entirely sure how exactly the padding is supposed to work (IIRC I've never used it), and I thought it behaved correctly. But maybe not - I think the safest thing is copy what 't' does, so I've done that in attached v3 of the patch. regards -- Tomas Vondra

Re: [HACKERS] (full) Memory context dump considered harmful

2015-08-22 Thread Tomas Vondra
On 08/22/2015 06:06 PM, Tom Lane wrote: Tomas Vondra tomas.von...@2ndquadrant.com writes: Couldn't we make it a bit smarter to handle even cases like this? For example we might first count/sum the child contexts, and then print either all child contexts (if there are only a few of them

Re: [HACKERS] PATCH: numeric timestamp in log_line_prefix

2015-08-22 Thread Tomas Vondra
. I would suggest to maybe follow Tom's %u idea and fix the implementation details wrt to comments received? Yes, I plan to update it according to those comments. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training

Re: [HACKERS] (full) Memory context dump considered harmful

2015-08-21 Thread Tomas Vondra
On 08/21/2015 08:37 PM, Tom Lane wrote: Tomas Vondra tomas.von...@2ndquadrant.com writes: I also don't think logging just subset of the stats is a lost case. Sure, we can't know which of the lines are important, but for example logging just the top-level contexts with a summary of the child

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-08-21 Thread Tomas Vondra
. I don't quite see a reason to wait for the over-estimation patch. We probably should backpatch the bugfix anyway (although it's much less likely to run into that before 9.5), and we can't really backpatch the behavior change there (as there's no hash resize). regards -- Tomas Vondra

Re: [HACKERS] statistics for array types

2015-08-20 Thread Tomas Vondra
gathered but nothing met the threshold. I'm not sure whether this is the same thing I just proposed ... regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-08-20 Thread Tomas Vondra
be addressed in a separate patch. kind regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

Re: [HACKERS] (full) Memory context dump considered harmful

2015-08-20 Thread Tomas Vondra
) or log_memory_stats = (full|short|off) might work. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

Re: [HACKERS] PATCH: use foreign keys to improve join estimates v1

2015-08-19 Thread Tomas Vondra
it once into the selectivity, or what? There's a bunch of similar cases mentioned in the comment before clauselist_join_selectivity. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training Services From

Re: [HACKERS] PATCH: use foreign keys to improve join estimates v1

2015-08-19 Thread Tomas Vondra
On 08/20/2015 03:49 AM, Tomas Vondra wrote: Then on current master, I get these estimates (showing just rows, because that's what matter): while with the patch I get this: And of course I forgot to include the plans from master, so here we go: select * from f join a on (f1 = a1 and f2 = a2

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-08-19 Thread Tomas Vondra
are features, and we're dealing with a bug fix here. kind regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-08-19 Thread Tomas Vondra
size the hashtable - that's not something we should do in a bugfix I think. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-08-19 Thread Tomas Vondra
Hi, On 08/20/2015 04:15 AM, Tomas Vondra wrote: Hello KaiGain-san, On 08/19/2015 03:19 PM, Kohei KaiGai wrote: Unless we have no fail-safe mechanism when planner estimated too large number of tuples than actual needs, a strange estimation will consume massive amount of RAMs. It's a bad side

Re: [HACKERS] multivariate statistics / patch v7

2015-07-30 Thread Tomas Vondra
Hi, On 07/30/2015 10:21 AM, Heikki Linnakangas wrote: On 05/25/2015 11:43 PM, Tomas Vondra wrote: There are 6 files attached, but only 0002-0006 are actually part of the multivariate statistics patch itself. All of these patches are huge. In order to review this in a reasonable amount

Re: [HACKERS] multivariate statistics / patch v7

2015-07-30 Thread Tomas Vondra
, IMHO. regards Tomas -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref

Re: [HACKERS] Remaining 'needs review' patchs in July commitfest

2015-07-28 Thread Tomas Vondra
discussion of low-level implementation details (e.g. how exactly the histograms are built, data structures etc.) - we haven't got that far yet. -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training Services -- Sent via pgsql-hackers

Re: [HACKERS] multivariate statistics / patch v7

2015-07-27 Thread Tomas Vondra
Hello Horiguchi-san, On 07/27/2015 09:04 AM, Kyotaro HORIGUCHI wrote: Hello, At Sat, 25 Jul 2015 23:09:31 +0200, Tomas Vondra tomas.von...@2ndquadrant.com wrote in 55b3fb0b.7000...@2ndquadrant.com Hi, On 07/16/2015 01:51 PM, Kyotaro HORIGUCHI wrote: Hi, I'd like to show you the modified

Re: [HACKERS] multivariate statistics / patch v7

2015-07-25 Thread Tomas Vondra
the implementation (adding support for another operator). kind regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your

Re: [HACKERS] Memory Accounting v11

2015-07-15 Thread Tomas Vondra
Hi, On 07/15/2015 09:21 PM, Robert Haas wrote: On Tue, Jul 14, 2015 at 9:14 PM, Tomas Vondra tomas.von...@2ndquadrant.com wrote: Firstly, do we really have good benchmarks and measurements? I really doubt that. We do have some numbers for REINDEX, where we observed 0.5-1% regression on noisy

Re: [HACKERS] Memory Accounting v11

2015-07-15 Thread Tomas Vondra
will drop this patch and proceed with the HashAgg patch, with a heuristic for internal types. Could someone briefly explain what heuristics are we talking about? -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training Services -- Sent

Re: [HACKERS] multivariate statistics / patch v7

2015-07-14 Thread Tomas Vondra
-processing before clauselist_selectivity (I assume that's the point), we'd end up repeating a lot of the code. Or maybe not, I'm not sure. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training Services -- Sent via pgsql

Re: [HACKERS] Memory Accounting v11

2015-07-14 Thread Tomas Vondra
, finalfunc and sizefunc. That seems doable, I guess. I find the memory accounting as a way more elegant solution, though. kind regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training Services -- Sent via pgsql-hackers mailing

Re: [HACKERS] PATCH: index-only scans with partial indexes

2015-07-11 Thread Tomas Vondra
Hi, On 07/10/2015 10:43 PM, Tom Lane wrote: Tomas Vondra tomas.von...@2ndquadrant.com writes: currently partial indexes end up not using index only scans in most cases, because check_index_only() is overly conservative, as explained in this comment: ... I've done a bunch of tests, and I do

[HACKERS] strange plan with bitmap heap scan and multiple partial indexes

2015-07-11 Thread Tomas Vondra
t WHERE b 100; QUERY PLAN Index Scan using idx001 on t (cost=0.14..29.14 rows=1000 width=4) Any idea what's going on here? FWIW all this was on 51d0fe5d (July 23). -- Tomas Vondra

Re: [HACKERS] strange plan with bitmap heap scan and multiple partial indexes

2015-07-11 Thread Tomas Vondra
this to skip some of the expensive stuff? We should have the selectivities anyway, no? regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org

Re: [HACKERS] strange plan with bitmap heap scan and multiple partial indexes

2015-07-11 Thread Tomas Vondra
Hi, On 07/11/2015 11:40 PM, Tom Lane wrote: Tomas Vondra tomas.von...@2ndquadrant.com writes: So I think the predicate proofing is a better approach, but of course the planning cost may be an issue. But maybe we can make this cheaper by some clever tricks? For example, given two predicates

Re: [HACKERS] multivariate statistics / patch v7

2015-07-07 Thread Tomas Vondra
with multivariate stats. kind regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref

Re: [HACKERS] multivariate statistics / patch v7

2015-07-07 Thread Tomas Vondra
Hello Horiguchi-san! On 07/07/2015 09:43 PM, Tomas Vondra wrote: -- histograms ALTER TABLE t ADD STATISTICS (histogram) on (a,b,c); ANALYZE t; EXPLAIN ANALYZE select * from t where a 0.3 and b 0.3 and c 0.3; Seq Scan on t (cost=0.00..23870.00 rows=267033 width=24) (actual

Re: [HACKERS] multivariate statistics / patch v7

2015-07-04 Thread Tomas Vondra
as your code assumes. Maybe, but I don't see what assumption is invalid? I see nothing wrong with the previous query. kind regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training Services -- Sent via pgsql-hackers mailing

Re: [HACKERS] proposal: condition blocks in psql

2015-06-28 Thread Tomas Vondra
with this wholeheartedly. ISTM most of this thread is about limitations of our current DO implementation, partially addressed by adding a bunch of scripting commands, specific to psql. I don't think that's the right solution. regards -- Tomas Vondra http://www.2ndQuadrant.com/ PostgreSQL

Re: [HACKERS] proposal: condition blocks in psql

2015-06-28 Thread Tomas Vondra
Hi, On 06/28/2015 02:21 PM, Pavel Stehule wrote: 2015-06-28 14:12 GMT+02:00 Tomas Vondra tomas.von...@2ndquadrant.com mailto:tomas.von...@2ndquadrant.com: This proposal is not against to DO parametrization. It is same like conditional block in C (#ifdef). There is similarity with C statements

Re: [HACKERS] proposal: condition blocks in psql

2015-06-28 Thread Tomas Vondra
Hi, On 06/28/2015 08:47 AM, Corey Huinker wrote: 5. I'm actually using psql to connect to redshift, which doesn't have DO blocks at all. I don't see this as a reason to add features to psql, unless there are other compelling reasons for the addition. -- Tomas Vondra http

Re: [HACKERS] proposal: condition blocks in psql

2015-06-28 Thread Tomas Vondra
mimicked by session variables in psql. If you really need to do the scripting outsite PL/pgSQL, there's plenty of suitable tools for that purpose IMHO. Starting with bash, or languages like Perl or Python. And I think the same about readability ... -- Tomas Vondra http://www

Re: [HACKERS] Refactoring pgbench.c

2015-06-28 Thread Tomas Vondra
submitted a patch to remove fork-emulation, which I think would really help simplify the code (maybe -10% source in pgbench.c, less #ifs, avoid double implementations or more-complex-than-necessary implementations or not-implemented features). ... and cleanup of dead code. -- Tomas Vondra

Re: [HACKERS] proposal: condition blocks in psql

2015-06-28 Thread Tomas Vondra
the SQL script ... -- Tomas Vondra http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql

Re: [HACKERS] Refactoring pgbench.c

2015-06-28 Thread Tomas Vondra
project, which was my thought. I respectfully disagree. I think that for this to be true, the files need to be somehow logically related (which I'd expect to be the output of the refactoring). But maybe you have some particular refactoring and split in mind? best regards -- Tomas Vondra

Re: [HACKERS] pg_stat_*_columns?

2015-06-26 Thread Tomas Vondra
On 06/27/2015 12:30 AM, Jim Nasby wrote: On 6/24/15 6:41 PM, Tomas Vondra wrote: Were the stories (or the experience which lead to the stories) on 9.3 or later? Do they have a good way to reproduce it for testing purposes? The per-db split can only improve things if there actually

Re: [HACKERS] pg_stat_*_columns?

2015-06-24 Thread Tomas Vondra
is supposed to be atomic or something like that. I don't know whether addressing this filesystem feature is worth it, or if pgstat.c is the right place to do that. -- Tomas Vondra http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training Services

Re: [HACKERS] pretty bad n_distinct estimate, causing HashAgg OOM on TPC-H

2015-06-22 Thread Tomas Vondra
On 06/22/2015 07:21 AM, Jeff Janes wrote: On Sat, Jun 20, 2015 at 9:55 AM, Tomas Vondra tomas.von...@2ndquadrant.com mailto:tomas.von...@2ndquadrant.com wrote: Hi, On 06/20/2015 03:01 AM, Jeff Janes wrote: I don't think we need to really assume the density

Re: [HACKERS] pretty bad n_distinct estimate, causing HashAgg OOM on TPC-H

2015-06-22 Thread Tomas Vondra
Hi, On 06/22/2015 07:47 AM, Jeff Janes wrote: On Sat, Jun 20, 2015 at 8:28 AM, Tomas Vondra tomas.von...@2ndquadrant.com mailto:tomas.von...@2ndquadrant.com wrote: Hi Tomas, I've lobotomized the sampling a bit to really produce a random set of blocks first, and that produces way

Re: [HACKERS] pretty bad n_distinct estimate, causing HashAgg OOM on TPC-H

2015-06-20 Thread Tomas Vondra
doing the sort by an indexscan seems rather cheap, and you only need to scan the table once. regards -- Tomas Vondra http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers

Re: [HACKERS] pretty bad n_distinct estimate, causing HashAgg OOM on TPC-H

2015-06-20 Thread Tomas Vondra
On 06/20/2015 04:17 PM, Robert Haas wrote: On Wed, Jun 17, 2015 at 1:52 PM, Tomas Vondra tomas.von...@2ndquadrant.com wrote: I'm currently running some tests on a 3TB TPC-H data set, and I tripped over a pretty bad n_distinct underestimate, causing OOM in HashAgg (which somehow illustrates

Re: [HACKERS] pretty bad n_distinct estimate, causing HashAgg OOM on TPC-H

2015-06-20 Thread Tomas Vondra
a good way to serialize/deserialize the aggregate state etc. There are also various corner cases how you can end up with writing much more data than you assumed, but let's discuss that in the thread about the patch, not here. regards -- Tomas Vondra http://www

Re: [HACKERS] pretty bad n_distinct estimate, causing HashAgg OOM on TPC-H

2015-06-20 Thread Tomas Vondra
minor compared to I/O generated by the other queries etc. The current sample is already random enough not to work well with read-ahead, and it scans only a slightly lower number of blocks. And if the larger random sample results in better plans (especially plans without OOM) ... -- Tomas

Re: [HACKERS] pg_stat_*_columns?

2015-06-20 Thread Tomas Vondra
see why we couldn't partition the stats in DSM in a similar manner. kind regards -- Tomas Vondra http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make

Re: [HACKERS] pretty bad n_distinct estimate, causing HashAgg OOM on TPC-H

2015-06-19 Thread Tomas Vondra
On 06/19/2015 09:48 PM, Jeff Janes wrote: On Fri, Jun 19, 2015 at 12:27 PM, Tomas Vondra tomas.von...@2ndquadrant.com mailto:tomas.von...@2ndquadrant.com wrote: But I think you might be on to something, because I manually collected a random sample with 30k rows (by explicitly generating

Re: [HACKERS] pretty bad n_distinct estimate, causing HashAgg OOM on TPC-H

2015-06-19 Thread Tomas Vondra
On 06/19/2015 08:32 PM, Jeff Janes wrote: On Wed, Jun 17, 2015 at 10:52 AM, Tomas Vondra tomas.von...@2ndquadrant.com mailto:tomas.von...@2ndquadrant.com wrote: Hi, I'm currently running some tests on a 3TB TPC-H data set, and I tripped over a pretty bad n_distinct underestimate

[HACKERS] pretty bad n_distinct estimate, causing HashAgg OOM on TPC-H

2015-06-17 Thread Tomas Vondra
), but apparently that does not change the estimates much :-( Any ideas? -- Tomas Vondra http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription

Re: [HACKERS] PATCH: adaptive ndistinct estimator v4

2015-06-17 Thread Tomas Vondra
in an area of the code they aren't particularly interested in and then recompile their code and then juggle two different sets of binaries, they will likely just shrug it off without investigation. +1 -- Tomas Vondra http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support

Re: [HACKERS] FIX : teach expression walker about RestrictInfo

2015-06-15 Thread Tomas Vondra
On 04/29/15 18:33, Tomas Vondra wrote: OK, I do understand that. So what about pull_varnos_walker and pull_varattnos_walker - what about teaching them about RestrictInfos? Attached is a patch fixing the issue by handling RestrictInfo in pull_varnos_walker and pull_varattnos_walker

Re: [HACKERS] On columnar storage

2015-06-14 Thread Tomas Vondra
orthogonal to this, because it splits the table vertically into multiple pieces, each stored in a separate relfilenode and thus using a separate sequence of page numbers. And of course, the usual 'horizontal' partitioning has a very similar effect (separate filenodes). regards -- Tomas

Re: [HACKERS] On columnar storage

2015-06-14 Thread Tomas Vondra
had something like Alexander Korotkov's proposed pluggable WAL stuff. Probably worth mentioning we don't expect the column store API to be used only by extensions, but even by code from within the core which can use the current WAL infrastructure etc. -- Tomas Vondra http

Re: [HACKERS] Memory Accounting v11

2015-06-14 Thread Tomas Vondra
-case. That's true, but I think the plan was always to wait for the whole feature, and only then commit all the pieces. -- Tomas Vondra http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training Services -- Sent via pgsql-hackers mailing list

Re: [HACKERS] Memory Accounting v11

2015-06-14 Thread Tomas Vondra
the performance impact to be acceptable. We already have accounting solution for each of those places, so I don't think the unification alone outweighs the regression. regards -- Tomas Vondra http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Remote DBA, Training

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-06-11 Thread Tomas Vondra
. Interestingly, the hash code checks for INT_MAX overflows on a number of places, but does not check for this ... regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training Services -- Sent via pgsql-hackers mailing list

Re: [HACKERS] 9.5 release notes

2015-06-11 Thread Tomas Vondra
/release.html I am ready to make suggested adjustments, though I am traveling for conferences for the next ten days so there might a delay in my replies. I wonder whether this point: * Improve hash creation performance (Tomas Vondra, Teodor Sigaev, Tom Lane) is really about and 45f6240a, 8cce08f1

Re: [HACKERS] DBT-3 with SF=20 got failed

2015-06-11 Thread Tomas Vondra
On 06/11/15 16:54, Tom Lane wrote: Tomas Vondra tomas.von...@2ndquadrant.com writes: Interestingly, the hash code checks for INT_MAX overflows on a number of places, but does not check for this ... Yeah, and at least at one time there were checks to prevent the hash table request from

Re: [HACKERS] The Future of Aggregation

2015-06-09 Thread Tomas Vondra
well for fixed-length states, but for handling custom states in 'internal' data type, the (de)serialization seems like a must for this use case. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training Services -- Sent via

Re: [HACKERS] The Future of Aggregation

2015-06-09 Thread Tomas Vondra
On 06/09/15 16:10, Tomas Vondra wrote: Hi, On 06/09/15 12:58, David Rowley wrote: ... Items 1-4 above I believe require support of Aggregate State Combine Support - https://commitfest.postgresql.org/5/131/ which I believe will need to be modified to implement complex database types

Re: [HACKERS] The Future of Aggregation

2015-06-09 Thread Tomas Vondra
On 06/09/15 17:27, Andres Freund wrote: On 2015-06-09 17:19:33 +0200, Tomas Vondra wrote: ... and yet another use case for 'aggregate state combine' that I just remembered about is grouping sets. What GROUPING SET (ROLLUP, ...) do currently is repeatedly sorting the input, once for each

Re: [HACKERS] [Proposal] More Vacuum Statistics

2015-06-07 Thread Tomas Vondra
when this might be handy) - I just had a VACUUM that ran for 12 hours. These jobs should report the stats incrementally, not just once at the very end, because that makes it rather useless IMNSHO. -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support

Re: [HACKERS] nested loop semijoin estimates

2015-06-06 Thread Tomas Vondra
of the add_path changes. So both changes seem fine. -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http

Re: [HACKERS] nested loop semijoin estimates

2015-06-02 Thread Tomas Vondra
On 06/02/15 01:47, Josh Berkus wrote: On 06/01/2015 03:22 PM, Tomas Vondra wrote: On 06/01/15 23:47, Josh Berkus wrote: On 06/01/2015 02:18 PM, Tom Lane wrote: Anybody else want to speak for or against back-patching the patch as posted? I intentionally didn't push it in before today's

Re: [HACKERS] nested loop semijoin estimates

2015-06-02 Thread Tomas Vondra
On 06/02/15 16:37, Tom Lane wrote: Tomas Vondra tomas.von...@2ndquadrant.com writes: OK, so I did the testing today - with TPC-H and TPC-DS benchmarks. The results are good, IMHO. I'm a bit disturbed by that, because AFAICS from the plans, these queries did not involve any semi or anti joins

Re: [HACKERS] nested loop semijoin estimates

2015-06-01 Thread Tomas Vondra
Mark Wong to test this on DBT3, if that's possible for him. I'm very worried about an unanticipated regression. AFAIK Mark is busy with other stuff at the moment, but I can do the TPC-H (which DBT3 is equal to, IIRC). -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL

Re: [HACKERS] nested loop semijoin estimates

2015-05-31 Thread Tomas Vondra
On 06/01/15 00:08, Tom Lane wrote: Tomas Vondra tomas.von...@2ndquadrant.com writes: On 05/30/15 23:16, Tom Lane wrote: Attached is a draft patch for that. It fixes the problem for me: Seems to be working OK, but I still do get a Bitmap Heap Scan there (but more about that later

Re: [HACKERS] [Proposal] More Vacuum Statistics

2015-05-30 Thread Tomas Vondra
it in memory in a lot of places. IMHO stat files is exactly the right place for data like this - I can't really think about other place with less overhead / impact. That of course assumes the new fields really are useful, and I do have my doubts about usefulness of this data. regards -- Tomas

Re: [HACKERS] nested loop semijoin estimates

2015-05-30 Thread Tomas Vondra
On 05/30/15 03:52, Tomas Vondra wrote: On 05/30/15 01:20, Tomas Vondra wrote: Notice the cost - it's way lover than the previous plan (9.2 vs ~111k), yet this plan was not chosen. So either the change broke something (e.g. by violating some optimizer assumption), or maybe there's a bug

Re: [HACKERS] [Proposal] More Vacuum Statistics

2015-05-30 Thread Tomas Vondra
in the same way, without introducing any additional cache invalidations or bloat. But I do agree pg_class really is not the right place for this for the other reasons. -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training

Re: [HACKERS] nested loop semijoin estimates

2015-05-30 Thread Tomas Vondra
other point about the indexscan getting rejected too soon. That doesn't seem to be happening for me, at least not in HEAD. FWIW I can reproduce that reliably on current HEAD, with the test case I sent yesterday. I'm using default config (as produced by initdb). regards -- Tomas Vondra

Re: [HACKERS] nested loop semijoin estimates

2015-05-30 Thread Tomas Vondra
' AND EXISTS (SELECT 1 FROM facttable_stat_fta4 f WHERE f.berechnungsart=t.term AND einheit IS NOT NULL); This will result in bitmap index scan no matter the visibility. -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training Services

<    2   3   4   5   6   7   8   9   10   11   >