On 12.9.2014 22:24, Robert Haas wrote:
On Fri, Sep 12, 2014 at 3:39 PM, Tomas Vondra t...@fuzzy.cz wrote:
On 12.9.2014 18:49, Robert Haas wrote:
I'm comfortable with this version if you are, but (maybe as a
follow-on commit) I think we could make this even a bit smarter. If
inner_rel_bytes
On 12.9.2014 22:24, Robert Haas wrote:
On Fri, Sep 12, 2014 at 3:39 PM, Tomas Vondra t...@fuzzy.cz wrote:
Yes, I like those changes and I think your reasoning is correct in both
cases. It certainly makes the method shorter and more readable - I was
too stuck in the original logic, so thanks
On 12.9.2014 23:22, Robert Haas wrote:
On Fri, Sep 12, 2014 at 4:55 PM, Tomas Vondra t...@fuzzy.cz wrote:
I'm actually quite surprised that you find batching to be a
better strategy than skimping on buckets, because I would have
expect the opposite, almost categorically. Batching means having
On 11 Září 2014, 15:31, Robert Haas wrote:
On Wed, Sep 10, 2014 at 5:09 PM, Tomas Vondra t...@fuzzy.cz wrote:
OK. So here's v13 of the patch, reflecting this change.
With the exception of ExecChooseHashTableSize() and a lot of stylistic
issues along the lines of what I've already complained
On 11 Září 2014, 16:11, Tom Lane wrote:
Robert Haas robertmh...@gmail.com writes:
On Thu, Sep 11, 2014 at 9:59 AM, Tom Lane t...@sss.pgh.pa.us wrote:
Robert Haas robertmh...@gmail.com writes:
(3) It allows the number of batches to increase on the fly while the
hash join is in process.
On 11 Září 2014, 17:28, Tom Lane wrote:
Tomas Vondra t...@fuzzy.cz writes:
On 11 Z?? 2014, 16:11, Tom Lane wrote:
Ah. Well, that would mean that we need a heuristic for deciding when
to
increase the number of buckets versus the number of batches ... seems
like a difficult decision
On 10.9.2014 22:39, Heikki Linnakangas wrote:
The bad news is that the rest don't seem to moving graduating from the
Needs Review state.
ISTM that many patches
(a) in 'needs review' actually have a review, or are being thoroughly
discussed
(b) in 'waiting on author' are not really
On 11.9.2014 16:33, Tomas Vondra wrote:
On 11 Září 2014, 15:31, Robert Haas wrote:
On Wed, Sep 10, 2014 at 5:09 PM, Tomas Vondra t...@fuzzy.cz wrote:
OK. So here's v13 of the patch, reflecting this change.
[...] It does three things:
(1) It changes NTUP_PER_BUCKET to 1. Although
On 11.9.2014 21:14, Petr Jelinek wrote:
On 11/09/14 18:59, Tomas Vondra wrote:
On 10.9.2014 22:39, Heikki Linnakangas wrote:
The bad news is that the rest don't seem to moving graduating from the
Needs Review state.
ISTM that many patches
(b) in 'waiting on author' are not really waiting
On 10.9.2014 20:25, Heikki Linnakangas wrote:
On 09/10/2014 01:49 AM, Tomas Vondra wrote:
I also did a few 'minor' changes to the dense allocation patch, most
notably:
* renamed HashChunk/HashChunkData to MemoryChunk/MemoryChunkData
The original naming seemed a bit awkward.
That's too
On 10.9.2014 20:31, Robert Haas wrote:
On Wed, Sep 10, 2014 at 2:25 PM, Heikki Linnakangas
hlinnakan...@vmware.com wrote:
The dense-alloc-v5.patch looks good to me. I have committed that with minor
cleanup (more comments below). I have not looked at the second patch.
Gah. I was in the
On 10.9.2014 20:55, Heikki Linnakangas wrote:
On 09/10/2014 09:31 PM, Robert Haas wrote:
* the chunks size is 32kB (instead of 16kB), and we're using 1/4
threshold for 'oversized' items
We need the threshold to be =8kB, to trigger the special case
within AllocSet. The 1/4 rule is
On 10.9.2014 20:25, Heikki Linnakangas wrote:
On 09/10/2014 01:49 AM, Tomas Vondra wrote:
I also did a few 'minor' changes to the dense allocation patch, most
notably:
* renamed HashChunk/HashChunkData to MemoryChunk/MemoryChunkData
The original naming seemed a bit awkward.
That's too
On 10.9.2014 21:34, Robert Haas wrote:
On Wed, Sep 10, 2014 at 3:12 PM, Tomas Vondra t...@fuzzy.cz wrote:
On 10.9.2014 20:25, Heikki Linnakangas wrote:
On 09/10/2014 01:49 AM, Tomas Vondra wrote:
I also did a few 'minor' changes to the dense allocation patch, most
notably:
* renamed
On 9.9.2014 16:09, Robert Haas wrote:
On Mon, Sep 8, 2014 at 5:53 PM, Tomas Vondra t...@fuzzy.cz wrote:
So I only posted the separate patch for those who want to do a review,
and then a complete patch with both parts combined. But it sure may be
a bit confusing.
Let's do this: post each new
On 8.9.2014 22:44, Robert Haas wrote:
On Fri, Sep 5, 2014 at 3:23 PM, Tomas Vondra t...@fuzzy.cz wrote:
as Heikki mentioned in his commitfest status message, this patch
still has no reviewers :-( Is there anyone willing to pick up that,
together with the 'dense allocation' patch (as those two
On 6.9.2014 23:34, Andrew Gierth wrote:
Tomas == Tomas Vondra t...@fuzzy.cz writes:
Tomas I have significant doubts about the whole design,
Tomas though. Especially the decision not to use HashAggregate,
There is no decision not to use HashAggregate. There is simply no
support
On 7.9.2014 15:11, Andrew Gierth wrote:
Tomas == Tomas Vondra t...@fuzzy.cz writes:
It's not one sort per grouping set, it's the minimal number of
sorts needed to express the result as a union of ROLLUP
clauses. The planner code will (I believe) always find the
smallest number
On 7.9.2014 18:52, Andrew Gierth wrote:
Tomas == Tomas Vondra t...@fuzzy.cz writes:
Tomas Maybe preventing this completely (i.e. raising an ERROR with
Tomas duplicate columns in CUBE/ROLLUP/... clauses) would be
Tomas appropriate. Does the standard says anything about this?
The spec
On 31.8.2014 22:52, Andrew Gierth wrote:
Recut patches:
gsp1.patch - phase 1 code patch (full syntax, limited functionality)
gsp2.patch - phase 2 code patch (adds full functionality using the
new chained aggregate mechanism)
gsp-doc.patch - docs
Hi everyone,
as Heikki mentioned in his commitfest status message, this patch still
has no reviewers :-( Is there anyone willing to pick up that, together
with the 'dense allocation' patch (as those two are closely related)?
I'm looking in Robert's direction, as he's the one who came up with the
On 4.9.2014 00:42, Tomas Vondra wrote:
Attached are two CSV files contain both raw results (4 runs per query),
and aggregated results (average of the runs), logs with complete logs
and explain (analyze) plans of the queries for inspection.
Of course, I forgot to attach the CSV files ... here
On 20.8.2014 20:32, Robert Haas wrote:
On Sun, Aug 17, 2014 at 1:17 PM, Tomas Vondra t...@fuzzy.cz wrote:
Being able to batch inner and outer relations in a matching way is
certainly one of the reasons why hashjoin uses that particular scheme.
There are other reasons, though - for example
On 4.9.2014 01:34, Tomas Vondra wrote:
On 20.8.2014 20:32, Robert Haas wrote:
As I see it, the advantage of Jeff's approach is that it doesn't
really matter whether our estimates are accurate or not. We don't
have to decide at the beginning how many batches to do, and then
possibly end up
On 29.8.2014 16:12, Tom Lane wrote:
Jeff Davis pg...@j-davis.com writes:
I have a new approach to the patch which is to call a callback at each
block allocation and child contexts inherit the callback from their
parents. The callback could be defined to simply dereference and
increment its
On 26.8.2014 21:38, Jeff Davis wrote:
On Tue, 2014-08-26 at 12:39 +0300, Heikki Linnakangas wrote:
I think this is enough for this commitfest - we have consensus on
the design. For the next one, please address those open items, and
resubmit.
Agreed, return with feedback.
I need to get
On 27 Srpen 2014, 21:41, Merlin Moncure wrote:
On Wed, Aug 27, 2014 at 2:07 PM, Pavel Stehule pavel.steh...@gmail.com
Are there some plans to use partitioning for aggregation?
Besides min/max, what other aggregates (mean/stddev come to mind)
would you optimize and how would you determine
On 24.8.2014 18:01, Tom Lane wrote:
Tomas Vondra t...@fuzzy.cz writes:
I stopped the already running test on addax and started the test on
barnacle again. Let's see in a few days/weeks/months what is the result.
It seems to be running much faster (probably after removing the
randomization
On 24.8.2014 18:28, Tom Lane wrote:
Tomas Vondra t...@fuzzy.cz writes:
Regarding those leaks we've detected so far - is it the kind of leaks
that can happen only in testing with those specific flags, or is it
something that can happen in production too? (Assuming no one is running
On 20.8.2014 08:11, Jeff Davis wrote:
On Tue, 2014-08-19 at 12:54 +0200, Tomas Vondra wrote:
The use-case for this is tracking a chosen subtree of contexts - e.g.
aggcontext and below, so I'd expect the tracked subtrees to be relatively
shallow. Am I right?
Right.
My fear
On 20.8.2014 08:11, Jeff Davis wrote:
On Tue, 2014-08-19 at 12:54 +0200, Tomas Vondra wrote:
It would be easier to resolve the performance concern if I could
reliably get the results Robert is getting. I think I was able to
reproduce the regression with the old patch, but the results were
On 20 Srpen 2014, 14:05, Heikki Linnakangas wrote:
On 07/20/2014 07:17 PM, Tomas Vondra wrote:
On 19.7.2014 20:24, Tomas Vondra wrote:
On 13.7.2014 21:32, Tomas Vondra wrote:
The current patch only implemnents this for tuples in the main
hash table, not for skew buckets. I plan to do
Hi,
On 13.8.2014 19:17, Tomas Vondra wrote:
On 13.8.2014 17:52, Tom Lane wrote:
* I'm a bit dubious about testing -DRANDOMIZE_ALLOCATED_MEMORY in the
same build as -DCLOBBER_CACHE_RECURSIVELY, because each of these is
darned expensive and it's not clear you'd learn anything by running
them
On 19 Srpen 2014, 9:52, Jeff Davis wrote:
On Fri, 2014-08-15 at 13:53 -0400, Robert Haas wrote:
I think that's right, and I rather like your (Jeff's) approach. It's
definitely true that we could do better if we have a mechanism for
serializing and deserializing group states, but (1) I think
On 19 Srpen 2014, 10:26, Jeff Davis wrote:
On Sat, 2014-08-16 at 23:09 +0200, Tomas Vondra wrote:
But maybe the inheritance really is not necessary - maybe it would be
enough to track this per-context, and then just walk through the
contexts and collect this. Because my observation
Hi all,
while working on a prototype of memory-bounded hash aggregate (alternative
to Jeff's patch discussed in [1]), I ran into difficulties when dealing
with dynahash. So I'm asking for help ...
Some of the difficulties stem from my limited knowledge of dynahash, and
complexity in
On 19.8.2014 19:05, Robert Haas wrote:
On Sat, Aug 16, 2014 at 9:31 AM, Tomas Vondra t...@fuzzy.cz wrote:
On 12.8.2014 00:30, Tomas Vondra wrote:
On 11.8.2014 20:25, Robert Haas wrote:
It also strikes me that when there's only 1 batch, the set of bits
that map onto the batch number is zero
On 10.8.2014 23:26, Jeff Davis wrote:
This patch is requires the Memory Accounting patch, or something similar
to track memory usage.
The attached patch enables hashagg to spill to disk, which means that
hashagg will contain itself to work_mem even if the planner makes a
bad misestimate of
On 15.8.2014 19:53, Robert Haas wrote:
On Thu, Aug 14, 2014 at 2:21 PM, Jeff Davis pg...@j-davis.com wrote:
On Thu, 2014-08-14 at 12:53 -0400, Tom Lane wrote:
Oh? So if we have aggregates like array_agg whose memory footprint
increases over time, the patch completely fails to avoid bloat?
On 12.8.2014 00:30, Tomas Vondra wrote:
On 11.8.2014 20:25, Robert Haas wrote:
It also strikes me that when there's only 1 batch, the set of bits
that map onto the batch number is zero-width, and one zero-width bit
range is as good as another. In other words, if you're only planning
to do
On 10.8.2014 22:50, Jeff Davis wrote:
On Fri, 2014-08-08 at 01:16 -0700, Jeff Davis wrote:
Either way, it's better to be conservative. Attached is a version
of the patch with opt-in memory usage tracking. Child contexts
inherit the setting from their parent.
There was a problem with the
On 16.8.2014 20:00, Tom Lane wrote:
Tomas Vondra t...@fuzzy.cz writes:
I believe this should check parent-track_mem, just like
update_allocation, because this way it walks all the parent context up
to the TopMemoryContext.
TBH, I don't think this opt-in tracking business has been thought
the only way to find out is to some actual tests.
On Wed, 2014-08-13 at 12:31 +0200, Tomas Vondra wrote:
My understanding of the batching algorithm (and I may be wrong on this
one) is that once you choose the number of batches, it's pretty much
fixed. Is that the case?
It's only fixed for that one
On 14 Srpen 2014, 18:12, Tom Lane wrote:
Jeff Davis pg...@j-davis.com writes:
HashJoin only deals with tuples. With HashAgg, you have to deal with a
mix of tuples and partially-computed aggregate state values. Not
impossible, but it is a little more awkward than HashJoin.
Not sure that I
On 14 Srpen 2014, 18:02, Atri Sharma wrote:
On Thursday, August 14, 2014, Jeff Davis pg...@j-davis.com wrote:
On Thu, 2014-08-14 at 10:06 -0400, Tom Lane wrote:
If you're following the HashJoin model, then what you do is the same
thing
it does: you write the input tuple back out to the
On 14.8.2014 18:54, Jeff Davis wrote:
On Thu, 2014-08-14 at 16:17 +0200, Tomas Vondra wrote:
Either it belongs to the current batch (and either it's in the hash
table, or you add it there), or it's not - in that case write it to a
temp file.
I think the part you left out is that you need
On 14.8.2014 21:47, Tomas Vondra wrote:
On 14.8.2014 18:54, Jeff Davis wrote:
On Thu, 2014-08-14 at 16:17 +0200, Tomas Vondra wrote:
Either it belongs to the current batch (and either it's in the hash
table, or you add it there), or it's not - in that case write it to a
temp file.
I think
On 13 Srpen 2014, 7:02, Jeff Davis wrote:
On Tue, 2014-08-12 at 14:58 +0200, Tomas Vondra wrote:
CREATE AGGREGATE myaggregate (
...
SERIALIZE_FUNC = 'dump_data',
DESERIALIZE_FUNC = 'read_data',
...
);
Seems reasonable.
I don't see why it should get messy? In the end, you
On 13.8.2014 17:52, Tom Lane wrote:
Tomas Vondra t...@fuzzy.cz writes:
So after 83 days, the regression tests on barnacle completed, and it
smells like another memory leak in CacheMemoryContext, similar to those
fixed in 078b2ed on May 18.
I've pushed fixes for the issues I was able
On 13.8.2014 12:31, Tomas Vondra wrote:
On 13 Srpen 2014, 7:02, Jeff Davis wrote:
On Tue, 2014-08-12 at 14:58 +0200, Tomas Vondra wrote:
(b) bad estimate of required memory - this is common for aggregates
passing 'internal' state (planner uses some quite high defaults)
Maybe some
On 12 Srpen 2014, 7:06, Jeff Davis wrote:
On Mon, 2014-08-11 at 01:29 +0200, Tomas Vondra wrote:
On 10.8.2014 23:26, Jeff Davis wrote:
This patch is requires the Memory Accounting patch, or something
similar to track memory usage.
I think the patch you sent actually includes the accounting
Hi!
So after 83 days, the regression tests on barnacle completed, and it
smells like another memory leak in CacheMemoryContext, similar to those
fixed in 078b2ed on May 18.
Barnacle is one of those machines with -DCLOBBER_CACHE_RECURSIVELY, and
the tests were running with a snapshot from May 19,
On 11.8.2014 20:25, Robert Haas wrote:
On Sat, Aug 9, 2014 at 9:13 AM, Tomas Vondra t...@fuzzy.cz wrote:
Adding least-significant bit does not work, we need get back to adding
the most-significant one. Not sure what's the least complex way to do
that, though.
I'm thinking about computing
On 12.8.2014 02:05, Tom Lane wrote:
Evidently the OOM killer is at large on this machine.
Yes. It's a machine with only 8GB of RAM, and there are 3 VMs (LXC
containers), with 2GB of RAM each. That's not much, but while it's
mostly out of necessity, it's apparently a good way to catch leaks.
Hi,
it's 1AM here, so only a few comments after quickly reading the patch.
On 10.8.2014 23:26, Jeff Davis wrote:
This patch is requires the Memory Accounting patch, or something
similar to track memory usage.
I think the patch you sent actually includes the accounting patch. Is
that on
On 20.7.2014 18:29, Tomas Vondra wrote:
Attached v9 of the patch. Aside from a few minor fixes, the main change
is that this is assumed to be combined with the dense allocation patch.
It also rewrites the ExecHashIncreaseNumBuckets to follow the same
pattern as ExecHashIncreaseNumBatches
On 2.8.2014 22:40, Jeff Davis wrote:
Attached is a patch that explicitly tracks allocated memory (the blocks,
not the chunks) for each memory context, as well as its children.
This is a prerequisite for memory-bounded HashAgg, which I intend to
Anyway, I'm really looking forward to the
On 23 Červenec 2014, 7:36, Pavel Stehule wrote:
updated version is in attachment
OK, thanks. The new version seems OK to me.
Tomas
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
On 28.6.2014 21:29, Pavel Stehule wrote:
Hello
rebase for 9.5
test:
\pset linestyle unicode \pset border 2
\pset unicode_header_linestyle double
\l
Regards
Pavel
I did a quick review of the patch today:
* it applies cleanly to current HEAD (no failures, small offsets)
*
On 20.7.2014 00:12, Tomas Vondra wrote:
On 19.7.2014 23:07, Tomas Vondra wrote:
On 19.7.2014 20:28, Tomas Vondra wrote:
For the first case, a WARNING at the end of estimate_hash_bucketsize
says this:
WARNING: nbuckets=8388608.00 estfract=0.01
WARNING: nbuckets=65536.00 estfract
On 19.7.2014 20:24, Tomas Vondra wrote:
On 13.7.2014 21:32, Tomas Vondra wrote:
The current patch only implemnents this for tuples in the main
hash table, not for skew buckets. I plan to do that, but it will
require separate chunks for each skew bucket (so we can remove it
without messing
of buckets). It's cleaner and more consistent.
hashjoin-nbuckets-growth-v9.patch contains only this patch, so you need
to grab the hashjoin-alloc-v4.patch from a different thread and apply it
first)
hashjoin-nbuckets-growth-v9-combined.patch contains both patches combined
regards
Tomas Vondra
diff --git
On 14.7.2014 06:29, Stephen Frost wrote:
Tomas,
* Tomas Vondra (t...@fuzzy.cz) wrote:
On 6.7.2014 17:57, Stephen Frost wrote:
* Tomas Vondra (t...@fuzzy.cz) wrote:
I can't find the thread / test cases in the archives. I've found this
thread in hackers:
http://www.postgresql.org/message
On 13.7.2014 21:32, Tomas Vondra wrote:
The current patch only implemnents this for tuples in the main hash
table, not for skew buckets. I plan to do that, but it will require
separate chunks for each skew bucket (so we can remove it without
messing with all of them). The chunks for skew
On 19.7.2014 20:24, Tom Lane wrote:
Tomas Vondra t...@fuzzy.cz writes:
I've reviewed the two test cases mentioned here, and sadly there's
nothing that can be 'fixed' by this patch. The problem here lies in the
planning stage, which decides to hash the large table - we can't fix
On 19.7.2014 20:28, Tomas Vondra wrote:
On 19.7.2014 20:24, Tom Lane wrote:
Tomas Vondra t...@fuzzy.cz writes:
I've reviewed the two test cases mentioned here, and sadly there's
nothing that can be 'fixed' by this patch. The problem here lies in the
planning stage, which decides to hash
On 19.7.2014 23:07, Tomas Vondra wrote:
On 19.7.2014 20:28, Tomas Vondra wrote:
For the first case, a WARNING at the end of estimate_hash_bucketsize
says this:
WARNING: nbuckets=8388608.00 estfract=0.01
WARNING: nbuckets=65536.00 estfract=0.000267
There are 4.3M rows
On 13.7.2014 12:27, Simon Riggs wrote:
On 12 July 2014 12:43, Tomas Vondra t...@fuzzy.cz wrote:
So lets just this change done and then do more later.
There's no way back, sadly. The dense allocation turned into a
challenge. I like challenges. I have to solve it or I won't be able
to sleep
On 11.7.2014 19:25, Tomas Vondra wrote:
2) walking through the tuples sequentially
--
The other option is not to walk the tuples through buckets, but by
walking throught the chunks - we know the tuples are stored as
HashJoinTuple/MinimalTuple, so
On 3.7.2014 19:36, Tomas Vondra wrote:
On 1.7.2014 01:24, Tomas Vondra wrote:
On 30.6.2014 23:12, Tomas Vondra wrote:
Hi,
Hopefully I got it right this time. At least it seems to be working
for cases that failed before (no file leaks, proper rowcounts so
far).
Attached v7, fixing nbatch
On 12.7.2014 11:39, Simon Riggs wrote:
On 11 July 2014 18:25, Tomas Vondra t...@fuzzy.cz wrote:
Turns out getting this working properly will quite complicated.
Lets keep this patch simple then. Later research can be another patch.
Well, the dense allocation is independent
On 11 Červenec 2014, 9:27, Simon Riggs wrote:
On 9 July 2014 18:54, Tomas Vondra t...@fuzzy.cz wrote:
(1) size the buckets for NTUP_PER_BUCKET=1 (and use whatever number
of batches this requires)
If we start off by assuming NTUP_PER_BUCKET = 1, how much memory does
it save
On 10.7.2014 21:33, Tomas Vondra wrote:
On 9.7.2014 16:07, Robert Haas wrote:
On Tue, Jul 8, 2014 at 5:16 PM, Tomas Vondra t...@fuzzy.cz wrote:
Thinking about this a bit more, do we really need to build the hash
table on the first pass? Why not to do this:
(1) batching
- read the tuples
On 9.7.2014 16:07, Robert Haas wrote:
On Tue, Jul 8, 2014 at 5:16 PM, Tomas Vondra t...@fuzzy.cz wrote:
Thinking about this a bit more, do we really need to build the hash
table on the first pass? Why not to do this:
(1) batching
- read the tuples, stuff them into a simple list
On 9.7.2014 16:07, Robert Haas wrote:
On Tue, Jul 8, 2014 at 5:16 PM, Tomas Vondra t...@fuzzy.cz wrote:
Thinking about this a bit more, do we really need to build the
hash table on the first pass? Why not to do this:
(1) batching - read the tuples, stuff them into a simple list -
don't
On 8 Červenec 2014, 14:49, Robert Haas wrote:
On Wed, Jul 2, 2014 at 8:13 PM, Tomas Vondra t...@fuzzy.cz wrote:
I propose dynamic increase of the nbuckets (up to NTUP_PER_BUCKET=1)
once the table is built and there's free space in work_mem. The patch
mentioned above makes implementing
On 8 Červenec 2014, 16:16, Robert Haas wrote:
On Tue, Jul 8, 2014 at 9:35 AM, Tomas Vondra t...@fuzzy.cz wrote:
Maybe. I'm not against setting NTUP_PER_BUCKET=1, but with large outer
relations it may be way cheaper to use higher NTUP_PER_BUCKET values
instead of increasing the number
On 8.7.2014 19:00, Robert Haas wrote:
On Tue, Jul 8, 2014 at 12:06 PM, Tomas Vondra t...@fuzzy.cz wrote:
On 8 Červenec 2014, 16:16, Robert Haas wrote:
Right, I think that's clear. I'm just pointing out that you get
to decide: you can either start with a larger NTUP_PER_BUCKET and
then reduce
On 8.7.2014 21:53, Jeff Janes wrote:
On Tue, Jul 8, 2014 at 6:35 AM, Tomas Vondra t...@fuzzy.cz wrote:
Maybe. I'm not against setting NTUP_PER_BUCKET=1, but with large
outer relations it may be way cheaper to use higher NTUP_PER_BUCKET
values instead of increasing the number of batches
Hi,
Thinking about this a bit more, do we really need to build the hash
table on the first pass? Why not to do this:
(1) batching
- read the tuples, stuff them into a simple list
- don't build the hash table yet
(2) building the hash table
- we have all the tuples in a simple list,
On 6.7.2014 06:47, Stephen Frost wrote:
* Greg Stark (st...@mit.edu) wrote:
Last time was we wanted to use bloom filters in hash joins to
filter out tuples that won't match any of the future hash batches
to reduce the amount of tuples that need to be spilled to disk.
However the problem was
On 6.7.2014 17:57, Stephen Frost wrote:
Tomas,
* Tomas Vondra (t...@fuzzy.cz) wrote:
I can't find the thread / test cases in the archives. I've found this
thread in hackers:
http://www.postgresql.org/message-id/caoezvif-r-ilf966weipk5by-khzvloqpwqurpak3p5fyw-...@mail.gmail.com
Can you
On 1.7.2014 01:24, Tomas Vondra wrote:
On 30.6.2014 23:12, Tomas Vondra wrote:
Hi,
Hopefully I got it right this time. At least it seems to be working for
cases that failed before (no file leaks, proper rowcounts so far).
Attached v7, fixing nbatch/ntuples in an assert.
regards
Tomas
diff
On 3.7.2014 02:13, Tomas Vondra wrote:
Hi,
while hacking on the 'dynamic nbucket' patch, scheduled for the next CF
(https://commitfest.postgresql.org/action/patch_view?id=1494) I was
repeatedly stumbling over NTUP_PER_BUCKET. I'd like to propose a change
in how we handle it.
TL;DR
On 3.7.2014 15:42, Atri Sharma wrote:
On Tue, Jul 1, 2014 at 4:54 AM, Tomas Vondra t...@fuzzy.cz
mailto:t...@fuzzy.cz wrote:
On 30.6.2014 23:12, Tomas Vondra wrote:
Hi,
attached is v5 of the patch. The main change is that scaling the
number
of buckets
Hi Stephen,
On 3.7.2014 20:10, Stephen Frost wrote:
Tomas,
* Tomas Vondra (t...@fuzzy.cz) wrote:
However it's likely there are queries where this may not be the case,
i.e. where rebuilding the hash table is not worth it. Let me know if you
can construct such query (I wasn't).
Thanks
On 3.7.2014 20:50, Tomas Vondra wrote:
Hi Stephen,
On 3.7.2014 20:10, Stephen Frost wrote:
Tomas,
* Tomas Vondra (t...@fuzzy.cz) wrote:
However it's likely there are queries where this may not be the case,
i.e. where rebuilding the hash table is not worth it. Let me know if you
can
Hi,
attached is v5 of the patch. The main change is that scaling the number
of buckets is done only once, after the initial hash table is build. The
main advantage of this is lower price. This also allowed some cleanup of
unecessary code.
However, this new patch causes warning like this:
On 30.6.2014 23:12, Tomas Vondra wrote:
Hi,
attached is v5 of the patch. The main change is that scaling the number
of buckets is done only once, after the initial hash table is build. The
main advantage of this is lower price. This also allowed some cleanup of
unecessary code.
However
On 26.6.2014 23:48, Tomas Vondra wrote:
On 26.6.2014 20:43, Tomas Vondra wrote:
Attached is v2 of the patch, with some cleanups / minor improvements:
* there's a single FIXME, related to counting tuples in the
Meh, I couldn't resist resolving this FIXME, so attached is v3 of the
patch
Hi,
Dne 2014-06-26 14:10, Pavel Stehule napsal:
Hello all,
today I had to work with one slow query - when I checked different
scenarios I found a dependency on work_mem size - but it is atypical -
bigger work_mem increased query execution 31 minutes (600MB work mem)
and 1 minute (1MB).
The
Attached is v2 of the patch, with some cleanups / minor improvements:
* improved comments, whitespace fixed / TODOs etc.
* tracking inital # of buckets (similar to initial # of batches)
* adding info about buckets to EXPLAIN ANALYZE, similar to batches - I
didn't want to make it overly complex,
On 26.6.2014 20:43, Tomas Vondra wrote:
Attached is v2 of the patch, with some cleanups / minor improvements:
* there's a single FIXME, related to counting tuples in the
Meh, I couldn't resist resolving this FIXME, so attached is v3 of the
patch. This just adds a proper 'batch tuples' counter
On 15.5.2014 00:41, Tomas Vondra wrote:
On 13.5.2014 20:42, Tomas Vondra wrote:
On 10.5.2014 20:21, Tomas Vondra wrote:
On 9.5.2014 00:47, Tomas Vondra wrote:
And I've requested 6 more animals - two for each compiler. One set for
tests with basic CLOBBER, one set for recursive CLOBBER.
Can
On 28.5.2014 19:52, Fujii Masao wrote:
On Thu, May 29, 2014 at 12:37 AM, Peter Geoghegan p...@heroku.com wrote:
On Wed, May 28, 2014 at 7:01 AM, Fujii Masao masao.fu...@gmail.com wrote:
But pg_stat_statements file is saved under $PGDATA/global yet.
Is this intentional or just oversight?
I
Hi,
I've been running a few longer pgbench tests (~week), and I've run into
this:
transaction type: SELECT only
scaling factor: 1250
query mode: simple
number of clients: 32
number of threads: 4
duration: 605000 s
number of transactions actually processed: -1785047856
latency average: -10.846 ms
On 25.5.2014 19:05, Andres Freund wrote:
printf(number of transactions per client: %d\n, nxacts);
-printf(number of transactions actually processed: %d/%d\n,
+printf(number of transactions actually processed: %ld/%d\n,
normal_xacts,
On 25.5.2014 20:32, Tom Lane wrote:
Tomas Vondra t...@fuzzy.cz writes:
On 25.5.2014 19:05, Andres Freund wrote:
That's not right though. On windows a long (indicated by the %l) is only
4 bytes wide. Check INT64_FORMAT. That's generated by configure/platform
template files and should always
On 13.5.2014 20:58, Tom Lane wrote:
Tomas Vondra t...@fuzzy.cz writes:
Yeah, not really what we were shooting for. I've fixed this by
defining the missing locales, and indeed - magpie now fails in
plpython tests.
I saw that earlier today (tho right now the buildfarm server seems
On 14.5.2014 15:17, Andres Freund wrote:
On 2014-05-14 15:08:08 +0200, Tomas Vondra wrote:
On 14 Květen 2014, 13:51, Andres Freund wrote:
On 2014-05-13 20:42:16 +0200, Tomas Vondra wrote:
Can someone please approve the animals I've requested a few days ago?
I'm already running the clobber
901 - 1000 of 1369 matches
Mail list logo
