Re: [HACKERS] [PATCH] fix DROP OPERATOR to reset links to itself on commutator and negator

2016-03-20 Thread Tomas Vondra
op->oprnegate)) OperatorUpd(operOid, operOid == op->oprcom ? InvalidOid : op->oprcom, operOid == op->oprnegate ? InvalidOid : op->oprnegate, true); regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Develop

Re: [HACKERS] POC, WIP: OR-clause support for indexes

2016-03-19 Thread Tomas Vondra
times. ... and it does not address this at all. I really doubt a costing derived from the bitmap index scan nodes will make much sense - you essentially need to revert unknown parts of the costing to only include building the bitmap once, etc. regards -- Tomas Vondra http://www.2ndQuadra

Re: [HACKERS] Combining Aggregates

2016-03-19 Thread Tomas Vondra
e been sticking to *_combine() for these, so maybe float8_combine() and float8_regr_combine() are better names. +1 to the _combine naming convention. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -

Re: [HACKERS] FIX : teach expression walker about RestrictInfo

2016-03-19 Thread Tomas Vondra
On 03/18/2016 08:53 PM, Robert Haas wrote: On Fri, Mar 18, 2016 at 3:29 PM, Tom Lane <t...@sss.pgh.pa.us> wrote: Robert Haas <robertmh...@gmail.com> writes: On Wed, Apr 29, 2015 at 12:33 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: On 04/29/15 18:

Re: [HACKERS] checkpointer continuous flushing

2016-03-19 Thread Tomas Vondra
e under 4000 tps. You mean the 'throttled-tps' chart? Yes, that one shows that without the patches, there's a lot of intervals where the tps was much lower - presumably due to a lot of slow transactions. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL

Re: [HACKERS] oldest xmin is far in the past

2016-03-19 Thread Tomas Vondra
value After looking at the code a bit more, I see it uses some additional configuration options: * freeze_min_age * vacuum_freeze_min_age * autovacuum_freeze_max_age (we already know this one) What values are set for those? regards -- Tomas Vondra http://www.2ndQuadrant.com

Re: [HACKERS] Weighted Stats

2016-03-19 Thread Tomas Vondra
we can come up with proper definition or at least good examples demonstrating the usefulness. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hacker

Re: [HACKERS] incorrect docs for pgbench / skipped transactions

2016-03-19 Thread Tomas Vondra
think that the timestamp should be made explicit milliseconds, i.e. "123.004567" instead of "123 4567", but that is another question not relevant to this patch. Agreed. -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Trai

Re: [HACKERS] improving GROUP BY estimation

2016-03-19 Thread Tomas Vondra
On 03/13/2016 11:09 PM, Tomas Vondra wrote: Hi, On Sun, 2016-03-13 at 15:24 +, Dean Rasheed wrote: On 4 March 2016 at 13:10, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: ... >>> I think that a better formula to use would be reldistinct *= (1 - powl(1 - rel-row

Re: [HACKERS] Combining Aggregates

2016-03-19 Thread Tomas Vondra
ticularly useful argument at this point, considering we don't really know if the current format is actually a good match for that. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-

Re: [HACKERS] checkpointer continuous flushing

2016-03-19 Thread Tomas Vondra
so the differences for large values are actually way more significant than it might look. So, good work I guess! regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing li

Re: [HACKERS] checkpointer continuous flushing

2016-03-19 Thread Tomas Vondra
n't think this would make any measurable difference in practice. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make

Re: [HACKERS] Proposal: Generic WAL logical messages

2016-03-19 Thread Tomas Vondra
is simple enough, but why to break the rule when all the other methods around have at least a short one? 4) ReorderBufferChange The new struct in the 'union' would probably deserve at least a short comment explaining the purpose (just like the other structs around). regards -- Tomas Vondra

Re: [HACKERS] multivariate statistics v14

2016-03-18 Thread Tomas Vondra
a perfect match). regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql

Re: [HACKERS] oldest xmin is far in the past

2016-03-18 Thread Tomas Vondra
nge. That probably means there's an old transaction somewhere - either a regular one (check pg_stat_activity) or a prepared one (pg_prepared_xacts). The meaning of "old" depends on autovacuum_freeze_max_age - what value is set in the session running the VACUUM FREEZE? regards -- Toma

Re: [HACKERS] Re: PATCH: Split stats file per database WAS: autovacuum stress-testing our system

2016-03-18 Thread Tomas Vondra
Hi, On 03/15/2016 03:04 AM, Noah Misch wrote: On Mon, Mar 14, 2016 at 01:33:08PM +0100, Tomas Vondra wrote: On 03/14/2016 07:14 AM, Noah Misch wrote: On Mon, Mar 14, 2016 at 02:00:03AM +0100, Tomas Vondra wrote: +* XXX Maybe this should also care about the clock skew, just like

[HACKERS] incorrect docs for pgbench / skipped transactions

2016-03-18 Thread Tomas Vondra
will contain "skipped" for transactions exceeding the limit latency (it's only mentioned in the example output). So I think we should apply the attached patch (and also backpatch it to 9.5, where the --latency-limit got introduced). regards -- Tomas Vondra

Re: [HACKERS] Combining Aggregates

2016-03-15 Thread Tomas Vondra
e type of the internal state and not to the input types. An example of that is the SUM(bigint) example mentioned above. That's actually the one thing I think the patch is missing. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remo

Re: [HACKERS] Parallel Aggregate

2016-03-14 Thread Tomas Vondra
numPartialGroups = numWorkers * workerGroups It's probably better to see Dean's message from 13/3. 4) Is clauses.h the right place for PartialAggType? regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Servic

Re: [HACKERS] PATCH: use foreign keys to improve join estimates v1

2016-03-14 Thread Tomas Vondra
015, while this is definitely an issue worth fixing, I believe it needs to be done in some other way, not by foreign keys. Attached is v3 of the patch, and also three SQL scripts demonstrating the impact of the patch on simple examples. regards -- Tomas Vondra http://www.2ndQu

Re: [HACKERS] Re: PATCH: Split stats file per database WAS: autovacuum stress-testing our system

2016-03-14 Thread Tomas Vondra
wrapping algorithm is broken. I've switched back to Thunderbird, so hopefully that'll fix it. On Mon, Mar 14, 2016 at 02:00:03AM +0100, Tomas Vondra wrote: On Sun, 2016-03-13 at 18:46 -0400, Noah Misch wrote: I've not attempted to study the behavior on slow hardware. Instead, my report used stat

Re: [HACKERS] Re: PATCH: Split stats file per database WAS: autovacuum stress-testing our system

2016-03-13 Thread Tomas Vondra
On Sun, 2016-03-13 at 18:46 -0400, Noah Misch wrote: > On Thu, Mar 03, 2016 at 06:08:07AM +0100, Tomas Vondra wrote: > > > > On 02/03/2016 06:46 AM, Noah Misch wrote: > > > > > > On Mon, Feb 01, 2016 at 07:03:45PM +0100, Tomas Vondra wrote: > > > >

Re: [HACKERS] improving GROUP BY estimation

2016-03-13 Thread Tomas Vondra
Hi, On Sun, 2016-03-13 at 15:24 +, Dean Rasheed wrote: > On 4 March 2016 at 13:10, Tomas Vondra <tomas.von...@2ndquadrant.com> > wrote: > > > > The risk associated with over-estimation is that we may switch from > > HashAggregate to GroupAggregate, and

Re: [HACKERS] Refectoring of receivelog.c

2016-03-12 Thread Tomas Vondra
the fields explicitly, but gcc is nagging about it. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscrip

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-11 Thread Tomas Vondra
as much as possible, etc. Currently this is tedious because we essentially find/fix the pages one by one. It'd be very useful to list all broken pages in one go and then fix all of them. Obviously, that's about heapam checks, but perhaps it would be useful for an index too? regard -- Tomas Vondra   

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-11 Thread Tomas Vondra
ksum, but broken page. Or perhaps issues coming from outside of PostgreSQL, like a subtle change in the locales. Simulating those issues requires a tool that allows minor tweaks to the page, dd is a bit too dull for that. regards -- Tomas Vondra  http://www.2ndQuadrant.com Postgr

Re: [HACKERS] auto_explain sample rate

2016-03-11 Thread Tomas Vondra
ple_ratio) as that's what pgbench uses for the same thing. That'd be more consistent. regards -- Tomas Vondra  http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql

Re: [HACKERS] amcheck (B-Tree integrity checking tool)

2016-03-10 Thread Tomas Vondra
;: /* * A B-Tree cannot possibly have this many levels, since there must be * one block per page, and that is bound by the range of BlockNumber: */ 2) comment before bt_check_every_level() says: ... assumed to prevent any kind of physically modification ... probably should be "physical modific

Re: [HACKERS] Using quicksort for every external sort run

2016-03-10 Thread Tomas Vondra
hat kind of thing. > > Attached is a revision that significantly overhauls the memory patch, > with several smaller changes. I was thinking about running some benchmarks on this patch, but the thread is pretty huge so I want to make sure I'm not missing something and this is ind

Re: [HACKERS] POC, WIP: OR-clause support for indexes

2016-03-10 Thread Tomas Vondra
s independently. Needless to say that without the patch this works just fine. 10) Also, this already breaks some regression tests, apparently because it changes how 'width' is computed. So I think this way of building the index path from a BitmapOr path is pretty much a dead-end. regards -- To

Re: [HACKERS] [PROPOSAL] VACUUM Progress Checker.

2016-03-09 Thread Tomas Vondra
; i++) nulls[i+3] = true; Now let's say PARAM=10, which means COLS=13. The last index accessed by the loop will be i=10, which means we'll do this: nulls[13] = true; which is above bounds. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Developmen

Re: [HACKERS] multivariate statistics v14

2016-03-09 Thread Tomas Vondra
On Wed, 2016-03-09 at 18:21 +0100, Tomas Vondra wrote: > Hi, > > On Wed, 2016-03-09 at 08:45 -0800, Jeff Janes wrote: > > On Wed, Mar 9, 2016 at 7:02 AM, Tomas Vondra > > <tomas.von...@2ndquadrant.com> wrote: > > > Hi, > > > > > > thanks

Re: [HACKERS] multivariate statistics v14

2016-03-09 Thread Tomas Vondra
Hi, On Wed, 2016-03-09 at 08:45 -0800, Jeff Janes wrote: > On Wed, Mar 9, 2016 at 7:02 AM, Tomas Vondra > <tomas.von...@2ndquadrant.com> wrote: > > Hi, > > > > thanks for the feedback. Attached is v14 of the patch series, fixing > > most of the points you've ra

Re: [HACKERS] More stable query plans via more predictable column statistics

2016-03-09 Thread Tomas Vondra
On Wed, 2016-03-09 at 12:02 -0300, Alvaro Herrera wrote: > Tomas Vondra wrote: > > > FWIW while looking at the code I noticed that we skip wide varlena > > values but not cstrings. Seems a bit suspicious. > > Uh, can you actually have columns of cstring type? I don't t

Re: [HACKERS] More stable query plans via more predictable column statistics

2016-03-09 Thread Tomas Vondra
ems to be (nonnull_cnt==toowide_cnt). Why not to use this condition instead? FWIW while looking at the code I noticed that we skip wide varlena values but not cstrings. Seems a bit suspicious. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Suppor

Re: [HACKERS] More stable query plans via more predictable column statistics

2016-03-09 Thread Tomas Vondra
to use default selectivity estimates (~5% for equality expressions, IIRC), not the per-column statistics. Combined with the independence assumption for multiple AND clauses, this may easily confuse the planner. I'd expect that would affect both indexes equally, but perhaps not. Would be u

Re: [HACKERS] silent data loss with ext4 / all current versions

2016-03-08 Thread Tomas Vondra
; > Note that I've split the patch into two. One for the infrastructure, and > one for the callsites. I've repeated the power-loss testing today. With the patches applied I'm not longer able to reproduce the issue (despite trying about 10x), while without them I've hit it on the first

Re: [HACKERS] More stable query plans via more predictable column statistics

2016-03-07 Thread Tomas Vondra
upgrade them without much additional testing and while we can't promise any of the fixes won't change the plans, we kinda try to minimize such cases. That being said, I don't have a clear opinion whether to backpatch this. I think that it's clearly a bug (especially the first part dealing with NUL

Re: [HACKERS] Static code checker research worth investigating (Communications of the ACM, 03/2016, Vol. 59, No. 03, p. 99)

2016-03-05 Thread Tomas Vondra
d a few years back (2013?) along with a paper that also mentioned a few issues in PostgreSQL. And it was dealt with, see for example this thread http://www.postgresql.org/message-id/flat/20130715215950.ga4...@eldon.alvh.no-ip.org Or is this something new? regards -- Tomas Vondra

Re: [HACKERS] improving GROUP BY estimation

2016-03-04 Thread Tomas Vondra
On Thu, 2016-03-03 at 11:42 -0800, Mark Dilger wrote: > > On Mar 3, 2016, at 11:27 AM, Alexander Korotkov <a.korot...@postgrespro.ru> > > wrote: > > > > On Thu, Mar 3, 2016 at 10:16 PM, Tomas Vondra > > <tomas.von...@2ndquadrant.com> wrote: > >

Re: [HACKERS] WIP: Upper planner pathification

2016-03-04 Thread Tomas Vondra
(speedup), but the points are actually below. See the chart attached. But I do agree this is mostly at the noise level, pretty good for a first cut that intentionally does not include any hacks. It's definitely way below the benefits of this patch, +1 to applying this sooner than later. regards

Re: [HACKERS] improving GROUP BY estimation

2016-03-03 Thread Tomas Vondra
Hi, On 03/03/2016 06:40 PM, Mark Dilger wrote: On Mar 3, 2016, at 8:35 AM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: Hi, On 03/03/2016 12:53 PM, Alexander Korotkov wrote: Hi, Tomas! I've assigned to review this patch. I've checked version estimate-num-groups-v2.txt b

Re: [HACKERS] improving GROUP BY estimation

2016-03-03 Thread Tomas Vondra
this may fail in some other cases. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://w

Re: [HACKERS] Re: PATCH: Split stats file per database WAS: autovacuum stress-testing our system

2016-03-02 Thread Tomas Vondra
Hi, On 02/03/2016 06:46 AM, Noah Misch wrote: On Mon, Feb 01, 2016 at 07:03:45PM +0100, Tomas Vondra wrote: On 12/22/2015 03:49 PM, Noah Misch wrote: On Mon, Feb 18, 2013 at 06:19:12PM -0300, Alvaro Herrera wrote: I have pushed it now. Further testing, of course, is always welcome. While

Re: [HACKERS] multivariate statistics v10

2016-03-02 Thread Tomas Vondra
_parameter [= value] [, ... ] ) ] Yeah, will fix. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subs

Re: [HACKERS] pg_dump / copy bugs with "big lines" ?

2016-03-02 Thread Tomas Vondra
On 03/02/2016 04:23 PM, Tom Lane wrote: Tomas Vondra <tomas.von...@2ndquadrant.com> writes: On 03/02/2016 03:18 PM, Daniel Verite wrote: However, getting it to the client with \copy big2 to 'file' still produces the error in psql: lost synchronization with server: got message t

Re: [HACKERS] pg_dump / copy bugs with "big lines" ?

2016-03-02 Thread Tomas Vondra
are more problems to solve to go beyond 2GB text per row. My guess is this is a problem at the protocol level - the 'd' message is CopyData, and all the messages use int32 to define length. So if there's a 2GB row, it's likely to overflow. regards -- Tomas Vondra http://www.

Re: [HACKERS] The plan for FDW-based sharding

2016-03-01 Thread Tomas Vondra
-- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] improving GROUP BY estimation

2016-02-25 Thread Tomas Vondra
Hi, On 02/26/2016 04:32 AM, Mark Dilger wrote: On Feb 25, 2016, at 4:59 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: ... The culprit here is that the two columns are not independent, but are (rather strongly) correlated due to the way you've generated the data.

Re: [HACKERS] improving GROUP BY estimation

2016-02-25 Thread Tomas Vondra
s agood idea. Yes, that's why I haven't bothered with fixing the regression tests in the patch, actually. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing lis

Re: [HACKERS] GIN data corruption bug(s) in 9.6devel

2016-02-25 Thread Tomas Vondra
running other benchmarks. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql

Re: [HACKERS] PATCH: index-only scans with partial indexes

2016-02-25 Thread Tomas Vondra
Hi, On 02/25/2016 11:56 AM, Kyotaro HORIGUCHI wrote: Hello, Tomas. my cerebral cortext gets squeezed by jumping from parser to planner. LOL At Wed, 24 Feb 2016 01:13:22 +0100, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote in <56ccf5a2.5040...@2ndquadrant.com> Hi, On 12/0

Re: [HACKERS] PATCH: use foreign keys to improve join estimates v1

2016-02-24 Thread Tomas Vondra
unique constraints in other cases. Overall, I still believe the FK patch is a clear improvement of the current status - while it's true it does not improve all possible cases and there's a room for additional improvements (like handling multiple candidate FK constraints), it s

Re: [HACKERS] GIN data corruption bug(s) in 9.6devel

2016-02-24 Thread Tomas Vondra
On 02/24/2016 06:56 AM, Robert Haas wrote: On Wed, Feb 24, 2016 at 9:17 AM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: ... Are we going to anything about this? While the bug is present in 9.5 (and possibly other versions), fixing it before 9.6 gets out seems important b

Re: [HACKERS] GIN data corruption bug(s) in 9.6devel

2016-02-23 Thread Tomas Vondra
Hi, On 01/05/2016 10:38 AM, Tomas Vondra wrote: Hi, ... There shouldn't be a difference between the two approaches (although I guess there could be if one left a larger pending list than the other, as pending lists is very space inefficient), but since you included 9.5 in your test I

Re: [HACKERS] Add generate_series(date,date) and generate_series(date,date,integer)

2016-02-23 Thread Tomas Vondra
the FROM part? If yes, we could do the check only in the FROM part, which is the case that does not work (and consumes arbitrary amounts of memory). regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- S

[HACKERS] improving GROUP BY estimation

2016-02-23 Thread Tomas Vondra
of the OOM issues in Hash Aggregate. regards [1] http://math.stackexchange.com/questions/72223/finding-expected-number-of-distinct-values-selected-from-a-set-of-integers -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] PATCH: index-only scans with partial indexes

2016-02-23 Thread Tomas Vondra
Hi, On 12/06/2015 11:48 PM, Tomas Vondra wrote: /* * Frequently, there will be no partial indexes, so first check to * make sure there's something useful to do here. */ have_partial = false; foreach(lc, rel->indexlist) { IndexOptInfo *index = (IndexOptInfo *) lfi

Re: [HACKERS] More stable query plans via more predictable column statistics

2016-02-23 Thread Tomas Vondra
one. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers

Re: [HACKERS] silent data loss with ext4 / all current versions

2016-02-23 Thread Tomas Vondra
renaming files between directories (or do we), so we're OK with respect to this. But it seems like a good idea to defend against this, or at least mention that in the comments. 4) nitpicking: There are some unnecessary newlines added/removed in RemoveXlogFile, XLogArchiveForceDone. re

[HACKERS] significant semi join overestimates (with CTEs)

2016-02-21 Thread Tomas Vondra
an product, eliminating at least the most obvious over-estimates. Secondly, I think it'd be nice to somehow eliminate the sudden jumps in the estimates - I'm not quite sure why need the three different formulas in eqjoinsel_semi, so maybe we can get rid of some of them? regards --

Re: [HACKERS] Explanation for bug #13908: hash joins are badly broken

2016-02-07 Thread Tomas Vondra
On 02/06/2016 10:22 PM, Tom Lane wrote: Tomas Vondra <tomas.von...@2ndquadrant.com> writes: What about using the dense allocation even for the skew buckets, but not one context for all skew buckets but one per bucket? Then when we delete a bucket, we simply destroy the context (an

Re: [HACKERS] Explanation for bug #13908: hash joins are badly broken

2016-02-06 Thread Tomas Vondra
On 02/06/2016 08:39 PM, Andres Freund wrote: On 2016-02-06 20:34:07 +0100, Tomas Vondra wrote: On 02/06/2016 06:47 PM, Tom Lane wrote: * It incorporates a bespoke reimplementation of palloc into hash joins. This is not a maintainable/scalable way to go about reducing memory consumption

Re: [HACKERS] Explanation for bug #13908: hash joins are badly broken

2016-02-06 Thread Tomas Vondra
On 02/06/2016 09:55 PM, Tom Lane wrote: Tomas Vondra <tomas.von...@2ndquadrant.com> writes: On 02/06/2016 06:47 PM, Tom Lane wrote: I note also that while the idea of ExecHashRemoveNextSkewBucket is to reduce memory consumed by the skew table to make it available to the main hash

Re: [HACKERS] Explanation for bug #13908: hash joins are badly broken

2016-02-06 Thread Tomas Vondra
On 02/06/2016 06:47 PM, Tom Lane wrote: Tomas Vondra <tomas.von...@2ndquadrant.com> writes: On 02/06/2016 02:34 AM, Tom Lane wrote: I have sussed what's happening in bug #13908. Basically, commit 45f6240a8fa9d355 ("Pack tuples in a hash join batch densely, to save memory&quo

Re: [HACKERS] silent data loss with ext4 / all current versions

2016-02-06 Thread Tomas Vondra
Hi, On 02/06/2016 01:16 PM, Michael Paquier wrote: On Sat, Feb 6, 2016 at 2:11 AM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: On 02/04/2016 09:59 AM, Michael Paquier wrote: On Tue, Feb 2, 2016 at 9:59 AM, Andres Freund <and...@anarazel.de> wrote: On 2016-02-02 09

Re: [HACKERS] Explanation for bug #13908: hash joins are badly broken

2016-02-06 Thread Tomas Vondra
e hashjoin goodies. FWIW I'm willing to put some time into fixing this over the weekend. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services diff --git a/src/backend/executor/nodeHash.c b/src/backend/executor

Re: [HACKERS] silent data loss with ext4 / all current versions

2016-02-05 Thread Tomas Vondra
don't even notice that. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.post

Re: [HACKERS] Re: PATCH: Split stats file per database WAS: autovacuum stress-testing our system

2016-02-03 Thread Tomas Vondra
On 02/03/2016 06:46 AM, Noah Misch wrote: On Mon, Feb 01, 2016 at 07:03:45PM +0100, Tomas Vondra wrote: On 12/22/2015 03:49 PM, Noah Misch wrote: ... If the collector entered pgstat_write_statsfile() with more inquiries waiting in its socket receive buffer, it would ignore them as being too

Re: [HACKERS] Re: PATCH: Split stats file per database WAS: autovacuum stress-testing our system

2016-02-01 Thread Tomas Vondra
ecific design in mind, though. That's a nice idea, but I agree that binding the coalescing to buffer like this seems like a pretty bad idea exactly because of the starving. What might work though is if we could look at how much data is there in the buffer, process only those requests

Re: [HACKERS] PATCH: track last known XLOG segment in control file

2016-01-31 Thread Tomas Vondra
dback / discussion about the protection in general. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your

Re: [HACKERS] Template for commit messages

2016-01-28 Thread Tomas Vondra
On 01/28/2016 03:37 PM, Robert Haas wrote: On Thu, Jan 28, 2016 at 9:34 AM, Bruce Momjian <br...@momjian.us> wrote: On Thu, Jan 28, 2016 at 09:31:46AM -0500, Robert Haas wrote: On Thu, Jan 28, 2016 at 6:16 AM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: How about Us

Re: [HACKERS] Template for commit messages

2016-01-28 Thread Tomas Vondra
on that he's not prepared to endorse is not going to have much value.) I can't speak for Tom, but I'm sitting fairly close to him and I haven't heard any complains or even groaning. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA

Re: [HACKERS] Template for commit messages

2016-01-28 Thread Tomas Vondra
Dne 28. 1. 2016 11:57 napsal uživatel "Alvaro Herrera" < alvhe...@2ndquadrant.com>: > > Magnus Hagander wrote: > > > They also had tested-by, it might be an idea to include that as well? > > How about > User-Interface-Bikeshedded-By: > ? +1 > > -- > Álvaro Herrera

Re: [HACKERS] Template for commit messages

2016-01-28 Thread Tomas Vondra
by Backpatch through Any reason why not to adapt the git message conventions from kernel? https://git.wiki.kernel.org/index.php/CommitMessageConventions I'd expect there are tools already working with that format, making the life easier for us. regards -- Tomas Vondra http

Re: [HACKERS] silent data loss with ext4 / all current versions

2016-01-25 Thread Tomas Vondra
Thanks for the time and improvements! Tomas -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscrip

Re: [HACKERS] Add generate_series(date,date) and generate_series(date,date,integer)

2016-01-25 Thread Tomas Vondra
n't possibly work. Imagine for example a script that in some rare cases passes happens to pass infinity into generate_series() - in that case I'd much rather error out than wait till the end of the universe. So +1 from me to checking for infinity. regard -- Tomas Vondra

Re: [HACKERS] easy way of copying regex_t

2016-01-25 Thread Tomas Vondra
ng one segment per dictionary as needed, or something like that. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make chang

Re: [HACKERS] More stable query plans via more predictable column statistics

2016-01-23 Thread Tomas Vondra
6791 So this seems to significantly improve the ndistinct estimate (patch attached). regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services diff --git a/src/backend/commands/analyze.c b/src/backend/commands/anal

Re: [HACKERS] PoC: Partial sort

2016-01-23 Thread Tomas Vondra
e the partial sort allows to keep only the "current group" in a hash table, making it much more memory efficient / faster. What do you think? regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services

Re: [HACKERS] silent data loss with ext4 / all current versions

2016-01-22 Thread Tomas Vondra
t makes the database easier to operate (less manual interventions) and the performance impact is 0 (as those fsyncs are really rare). regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hac

Re: [HACKERS] Declarative partitioning

2016-01-22 Thread Tomas Vondra
a valuable hint how to define the catalogs and such. regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subs

Re: [HACKERS] Tsvector editing functions

2016-01-22 Thread Tomas Vondra
On 12/30/2015 06:49 PM, Stas Kelvich wrote: Hi, Tomáš! Thanks for comprehensive review. On 15 Dec 2015, at 06:07, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: 1) It's a bit difficult to judge the usefulness of the API, as I've always been a mere user of full-text search,

Re: [HACKERS] Performance improvement for joins where outer side is unique

2016-01-22 Thread Tomas Vondra
Hi, On 12/17/2015 02:17 PM, David Rowley wrote: On 17 December 2015 at 19:11, Simon Riggs <si...@2ndquadrant.com <mailto:si...@2ndquadrant.com>> wrote: On 17 December 2015 at 00:17, Tomas Vondra <tomas.von...@2ndquadrant.com <mailto:tomas.von...@2ndquadra

Re: [HACKERS] silent data loss with ext4 / all current versions

2016-01-22 Thread Tomas Vondra
On 01/23/2016 02:35 AM, Michael Paquier wrote: On Fri, Jan 22, 2016 at 9:41 PM, Greg Stark <st...@mit.edu> wrote: On Fri, Jan 22, 2016 at 8:26 AM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: On 01/22/2016 06:45 AM, Michael Paquier wrote: So, I have been playing wit

Re: [HACKERS] multivariate statistics v8

2016-01-20 Thread Tomas Vondra
On 01/20/2016 10:54 PM, Alvaro Herrera wrote: Bruce Momjian wrote: On Wed, Jan 20, 2016 at 02:20:38PM -0500, Robert Haas wrote: On Wed, Dec 23, 2015 at 2:07 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: The remaining question is how unique the statistics name should be.

Re: [HACKERS] PATCH: Extending the HyperLogLog API a bit

2016-01-19 Thread Tomas Vondra
that's exactly what merge does. So why not just fix the bug? -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscrip

Re: [HACKERS] PATCH: postpone building buckets to the end of Hash (in HashJoin)

2016-01-19 Thread Tomas Vondra
On 01/19/2016 08:34 PM, Robert Haas wrote: On Mon, Jan 18, 2016 at 10:57 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: If this doesn't regress performance in the case where the number of buckets is estimated accurately to begin with, then I think this is a great idea. Can you

Re: [HACKERS] Combining Aggregates

2016-01-18 Thread Tomas Vondra
state (I don't see why would be the case), it's a dead end. Not so long ago we've fixed exactly this issue in array_agg(), which addressed a bunch of reported OOM issues and improved array_agg() performance quite a bit. It'd be rather crazy introducing the same problem to all aggregate functions

Re: [HACKERS] Combining Aggregates

2016-01-18 Thread Tomas Vondra
anger of OOM issues? regards -- Tomas Vondra http://www.2ndQuadrant.com PostgreSQL Development, 24x7 Support, Remote DBA, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.or

Re: [HACKERS] Combining Aggregates

2016-01-18 Thread Tomas Vondra
On 01/19/2016 06:04 AM, Tomas Vondra wrote: Hi, On 01/19/2016 05:00 AM, David Rowley wrote: Good question. I looked at this and found my VM was swapping like crazy. Upon investigation it appears that's because, since the patch creates a memory context per aggregated group, and in this case

Re: [HACKERS] PATCH: postpone building buckets to the end of Hash (in HashJoin)

2016-01-18 Thread Tomas Vondra
Hi, On 12/17/2015 10:28 PM, Tomas Vondra wrote: Hi, On 12/17/2015 07:20 PM, Robert Haas wrote: ... If this doesn't regress performance in the case where the number of buckets is estimated accurately to begin with, then I think this is a great idea. Can you supply some performance tests

Re: [HACKERS] silent data loss with ext4 / all current versions

2016-01-18 Thread Tomas Vondra
On 01/19/2016 07:44 AM, Michael Paquier wrote: On Wed, Dec 2, 2015 at 3:24 PM, Michael Paquier <michael.paqu...@gmail.com> wrote: On Wed, Dec 2, 2015 at 3:23 PM, Michael Paquier <michael.paqu...@gmail.com> wrote: On Wed, Dec 2, 2015 at 7:05 AM, Tomas Vondra <tomas.von...@

Re: [HACKERS] silent data loss with ext4 / all current versions

2016-01-18 Thread Tomas Vondra
On 01/19/2016 08:03 AM, Michael Paquier wrote: On Tue, Jan 19, 2016 at 3:58 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: ... Tomas, I am planning to have a look at that, because it seems to be important. In case it becomes lost on my radar, do you mind if I add it to the 2

Re: [HACKERS] PATCH: add pg_current_xlog_flush_location function

2016-01-11 Thread Tomas Vondra
before looking at it but it seems to me that using an independent variable would make the code more robust. Why? LogwrtResult only serves as a local cache of shared values, so there should be no danger of skipping something. regards -- Tomas Vondra http://www.2ndQuadrant.com Post

Re: [HACKERS] WIP: bloom filter in Hash Joins with batches

2016-01-10 Thread Tomas Vondra
Hi, On 01/10/2016 01:08 AM, Peter Geoghegan wrote: On Sat, Jan 9, 2016 at 11:02 AM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: So, this seems to bring reasonable speedup, as long as the selectivity is below 50%, and the data set is sufficiently large. What about semijoins? Appa

Re: [HACKERS] WIP: bloom filter in Hash Joins with batches

2016-01-10 Thread Tomas Vondra
ntally verified that you get the expected probability of false positives in practice? I have seen that when the bloom filter is properly sized, and the data set is not somehow skewed, the bloom filters have about the right false positive rate. regards -- Tomas Vondra http://www.2n

Re: [HACKERS] WIP: bloom filter in Hash Joins with batches

2016-01-10 Thread Tomas Vondra
Hi, On 01/10/2016 05:11 AM, Peter Geoghegan wrote: On Sat, Jan 9, 2016 at 11:02 AM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: Which means the "dim.r" column has 100 different values (0-99) with uniform distribution. So e.g. "WHERE r < 15" matche

Re: [HACKERS] PATCH: add pg_current_xlog_flush_location function

2016-01-10 Thread Tomas Vondra
Hi, On 12/13/2015 08:38 PM, Tomas Vondra wrote: Hi, On 12/13/2015 06:13 AM, Amit Kapila wrote: > ... > Is there a reason why you can't use existing function GetFlushRecPtr() in xlog.c? No, not really. I think I somehow missed that function when writing the initial version of the

Re: [HACKERS] PATCH: Extending the HyperLogLog API a bit

2016-01-09 Thread Tomas Vondra
On 01/01/2016 12:08 AM, Peter Geoghegan wrote: On Thu, Dec 31, 2015 at 12:48 PM, Tomas Vondra <tomas.von...@2ndquadrant.com> wrote: 1) initHyperLogLogError(hyperLogLogState *cState, double error) Instead of specifying bwidth (essentially the number of bits used for addr

<    1   2   3   4   5   6   7   8   9   10   >