Re: [PERFORM] select subquery versus join subquery

2017-05-24 Thread Gunnar "Nick" Bluth
Am 05/23/2017 um 06:59 PM schrieb Jeff Janes:
> On Tue, May 23, 2017 at 4:03 AM, Gunnar "Nick" Bluth
> <gunnar.bluth.ext...@elster.de <mailto:gunnar.bluth.ext...@elster.de>>
> wrote:
8>< -
> 
> Unfortunately that always uses the index scan, even at a high cutoff
> where aggregation on the seq scan and then hash joining is more
> appropriate.  So it is very similar to view2, except that it doesn't
> return the rows from "thing" which have zero corresponding rows in
> thing_alias.
> 
> *
> Note btw. that both view1 and view2 don't return any md5 values for me,
> while view3 does!
> *
> 
> 
> Because of the way I constructed the data, using the power transform of
> the uniform random distribution, the early rows of the view (if sorted
> by thing_id) are mostly null in the md5 column, so if you only look at
> the first few screen-fulls you might not see any md5.  But your view
> does effectively an inner join rather than a left join, so your view
> gets rid of the rows with a NULL md5.  Most things don't have aliases;
> of the things that do, most have 1; and some have a several.

D'oh, of course! My bad... shouldn't have looked at the results with
LIMIT :-/

My next best guess would involve a MatView for the aggregates...
-- 
Gunnar "Nick" Bluth
DBA ELSTER

Tel:   +49 911/991-4665
Mobil: +49 172/8853339



smime.p7s
Description: S/MIME Cryptographic Signature


Re: [PERFORM] select subquery versus join subquery

2017-05-23 Thread Gunnar &quot;Nick&quot; Bluth
Am 05/22/2017 um 09:57 PM schrieb Jeff Janes:
> I need to make a view which decorates rows from a parent table with
> aggregated values from a child table.  I can think of two ways to write
> it, one which aggregates the child table to make a new query table and
> joins the parent to that, as shown in "view1" below.  Or does subselect
> in the select list to aggregate just the currently matching rows, and
> returns that value, as in "view2" below.
> 
> While these two are semantically equivalent, the planner doesn't
> understand that, and always executes them pretty much the way you would
> naively do it based on the text of the query.
> 
> But view1 is pretty slow if the WHERE clause is highly selective (like
> "WHERE cutoff<0.1") because it has to summarize the entire child
> table just to pull out a few rows.  But view2 is pretty slow if the
> entire view or most of it (like "WHERE cutoff<0.9") is being returned.
> 
> Is there some 3rd way to write the query which allows the planner to
> switch between strategies (summarize whole table vs summarize values on
> demand) depending on the known selectivity of the where clause?
> 
> In this case, the planner is getting the relative cost estimates roughly
> correct.  It is not a problem of mis-estimation.
> 
> I can always create two views, view_small and view_large, and swap
> between them based on my own knowledge of how restrictive a query is
> likely to be, but that is rather annoying.  Especially in the real-world
> situation, which is quite a bit more complex than this.
> 
> create table thing as select x as id, random() as cutoff from
> generate_series(1,200) f(x);
> 
> create table thing_alias as select
> floor(power(random()*power(200,5),0.2))::int thing_id, md5(x::text),
> random() as priority from generate_series(1,15) f(x);
> 
> create index on thing_alias (thing_id );
> 
> create index on thing (cutoff );
> 
> vacuum; analyze;
> 
> create view view1 as select id, md5,cutoff from thing left join 
>   (
>  select distinct on (thing_id) thing_id, md5 from thing_alias 
>  order by thing_id, priority desc
>   ) as foo 
>   on (thing_id=id);
> 
> create view view2 as select id, 
>   (
>  select md5 from thing_alias where thing_id=id 
> order by priority desc limit 1
>   ) as md5, 
>   cutoff from thing;
> 
> Cheers,
> 
> Jeff

Hi Jeff,

how does something like

CREATE OR REPLACE VIEW public.view3 AS
 SELECT thing.id,
foo.md5,
thing.cutoff
   FROM thing,
LATERAL ( SELECT DISTINCT ON (thing_alias.thing_id)
thing_alias.thing_id,
thing_alias.md5
   FROM thing_alias
  WHERE thing_alias.thing_id = thing.id
  ORDER BY thing_alias.thing_id, thing_alias.priority DESC) foo

work for you? At least that's always using an index scan here, as
opposed to view1, which (for me) defaults to a SeqScan on thing_alias at
a low cutoff.

*
Note btw. that both view1 and view2 don't return any md5 values for me,
while view3 does!
*

Results (ms, median of 3 runs):
cutoff<  0.1   0.9
view1:   348   1022
view2:   844   6484
view3:   842   5976

With

 LATERAL ( SELECT string_agg(thing_alias.md5, ','::text) AS md5
   FROM thing_alias
  WHERE thing_alias.thing_id = thing.id
  GROUP BY thing_alias.thing_id) foo

(which seems to make more sense ;-)

I yield 483 (0.1) and 3410 (0.9) ms (and return md5-Aggregates).

Cheers,
-- 
Gunnar "Nick" Bluth
DBA ELSTER

Tel:   +49 911/991-4665
Mobil: +49 172/8853339



smime.p7s
Description: S/MIME Cryptographic Signature


Re: [PERFORM] Cloud versus buying my own iron

2016-02-24 Thread Gunnar &quot;Nick&quot; Bluth
Am 24.02.2016 um 06:06 schrieb Craig James:
> At some point in the next year we're going to reconsider our hosting
> environment, currently consisting of several medium-sized servers (2x4
> CPUs, 48GB RAM, 12-disk RAID system with 8 in RAID 10 and 2 in RAID 1
> for WAL). We use barman to keep a hot standby and an archive.
> 
> The last time we dug into this, we were initially excited, but our
> excitement turned to disappointment when we calculated the real costs of
> hosted services, and the constraints on performance and customizability.
> 
> Due to the nature of our business, we need a system where we can install
> plug-ins to Postgres. I expect that alone will limit our choices. In
> addition to our Postgres database, we run a fairly ordinary Apache web site.
> 
> There is constant chatter in this group about buying servers vs. the
> various hosted services. Does anyone have any sort of summary comparison
> of the various solutions out there? Or is it just a matter of
> researching it myself and maybe doing some benchmarking and price
> comparisons?

For starters, did you see Josh Berkus' presentation on the topic?
  https://www.youtube.com/watch?v=WV5P2DgxPoI

I for myself would probably always go the "own iron" road, but alas!
that's just the way I feel about control. And I'm kind of a Linux
oldshot, so managing a (hosted root) server doesn't scare me off.

OTOH, I do see the advantages of having things like monitoring, backup,
HDD replacements etc. done for you. Which is essentially what you pay for.

In essence, there's obviously no silver bullet ;-)

Best regards,
-- 
Gunnar "Nick" Bluth
DBA ELSTER

Tel:   +49 911/991-4665
Mobil: +49 172/8853339


-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Estimation row error

2015-12-15 Thread Gunnar &quot;Nick&quot; Bluth
Am 15.12.2015 um 09:05 schrieb Mathieu VINCENT:
> Hello,
> 
> No one to help me to understand this bad estimation rows ?

Well,

on a rather beefy machine, I'm getting quite a different plan:
http://explain.depesz.com/s/3y5r

Which may be related to this setting:
perftest=# show default_statistics_target ;
 default_statistics_target
---
 1000
(1 Zeile)


I guess the wrong row assumption (which I get as well!) is caused by the
given correlation of t3.c1 and t3.c2 (which the planner doesn't "see").

Tomas Vondra has written a nice blog post, covering that topic as well:
http://blog.pgaddict.com/posts/common-issues-with-planner-statistics

AFAIK, 9.5 has received some improvements in that field, but I didn't
try that yet.

Best regards,

Nick

> 
> Mathieu VINCENT
> 
> 2015-12-11 12:35 GMT+01:00 Mathieu VINCENT
> <mathieu.vinc...@pmsipilot.com <mailto:mathieu.vinc...@pmsipilot.com>>:
> 
> Sorry, I forget to precise Postgresql version
> 
> 'PostgreSQL 9.4.4 on x86_64-unknown-linux-gnu, compiled by gcc (GCC)
> 4.4.7 20120313 (Red Hat 4.4.7-11), 64-bit'
> 
> 
> BR
> 
> Mathieu VINCENT
> 
>   
> 
> 
> 2015-12-11 9:53 GMT+01:00 Mathieu VINCENT
> <mathieu.vinc...@pmsipilot.com <mailto:mathieu.vinc...@pmsipilot.com>>:
> 
> Hello,
> 
> I would like to know how row estimation is calculed by explain ?
> In my execution plan, this estimation is extremely wrong (267
> instead of 198000)
> I reproduced this estimation error in this simple case :
> 
> drop table if exists t1;
> drop table if exists t2;
> drop table if exists t3;
> drop table if exists t4;
> 
> create table t1 as select generate_Series(1,30) as c1; 
> create table t2 as select generate_Series(1,400) as c1; 
> create table t3 as select generate_Series(1,20)%100 as
> c1,generate_Series(1,20) as c2;
> create table t4 as select generate_Series(1,20) as c1;
> 
> alter table t1 add PRIMARY KEY (c1);
> alter table t2 add PRIMARY KEY (c1);
> alter table t3 add PRIMARY KEY (c1,c2);
> create index on t3 (c1);
> create index on t3 (c2);
> alter table t4 add PRIMARY KEY (c1);
> 
> analyze t1;
> analyze t2;
> analyze t3;
> analyze t4;
> 
> EXPLAIN (analyze on, buffers on, verbose on)
> select 
> *
> from 
> t1 t1
> inner join t2 on t1.c1=t2.c1
> inner join t3 on t2.c1=t3.c1
> inner join t4 on t3.c2=t4.c1
> 
> Explain plan :
> http://explain.depesz.com/s/wZ3v
> 
> I think this error may be problematic because planner will
> choose nested loop instead of hash joins for ultimate join. Can
> you help me to improve this row estimation ? 
> 
>     Thank you for answering
> 
> Best Regards,
> <http://www.psih.fr/> PSIH Décisionnel en santé
> Mathieu VINCENT 
> Data Analyst
> PMSIpilot - 61 rue Sully - 69006 Lyon - France
> 
> 
> 


-- 
Gunnar "Nick" Bluth
DBA ELSTER

Tel:   +49 911/991-4665
Mobil: +49 172/8853339


-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Estimation row error

2015-12-15 Thread Gunnar &quot;Nick&quot; Bluth
Am 15.12.2015 um 10:49 schrieb Andreas Kretschmer:
> Gunnar Nick Bluth <gunnar.bluth.ext...@elster.de> wrote:
> 
>> Am 15.12.2015 um 09:05 schrieb Mathieu VINCENT:
>>> Hello,
>>>
>>> No one to help me to understand this bad estimation rows ?
>>
>> Well,
>>
>> on a rather beefy machine, I'm getting quite a different plan:
>> http://explain.depesz.com/s/3y5r
> 
> you are using 9.5, right? Got the same plan with 9.5.

Nope...:
  version


 PostgreSQL 9.3.5 on x86_64-unknown-linux-gnu, compiled by gcc
(Ubuntu/Linaro 4.6.3-1ubuntu5) 4.6.3, 64-bit

So much for those correlation improvements then ;-/


> Btw.: Hi Gunnar ;-)

Hi :)

-- 
Gunnar "Nick" Bluth
DBA ELSTER

Tel:   +49 911/991-4665
Mobil: +49 172/8853339


-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Re: Increasing work_mem and shared_buffers on Postgres 9.2 significantly slows down queries

2012-11-06 Thread Gunnar Nick Bluth

Am 05.11.2012 18:09, schrieb Jeff Janes:

On Mon, Nov 5, 2012 at 8:48 AM, Claudio Freire klaussfre...@gmail.com wrote:

On Mon, Nov 5, 2012 at 1:44 PM, Jeff Janes jeff.ja...@gmail.com wrote:

Well, I'm pretty sure that having more work_mem is a good thing (tm)
normally ;-)

In my experience when doing sorts in isolation, having more work_mem
is a bad thing, unless it enables you to remove a layer of
tape-merging.  I always blamed it on the L1/L2 etc. levels of caching.

Blame it on quicksort, which is quite cache-unfriendly.

The observation applies to heap sort.  If you can't set work_mem large
enough to do the sort in memory, then you want to set it just barely
large enough to avoid two layers of tape sorting.  Any larger than
that reduces performance rather than increasing it.  Of course that
assumes you have the luxury of knowing ahead of time exactly how large
your sort will be and can set work_mem accordingly on a case by case
basis, which is unlikely in the real world.


Perhaps PG should consider using in-memory mergesort for the bigger chunks.
I don't want to be the party pooper here, but when you have another look 
at the EXPLAINs, you'll realize that there's not a single sort involved. 
The expensive parts are HASH, HASH JOIN and HASH RIGHT JOIN (although 
the SeqScan takes longer as well, for whatever reason). In those parts, 
the difference is clearly in the # of buckets and batches. So to a 
degree, PG even does tell us that it uses a different code path (sorry, 
PG ;-)...


Greg Smith mentions an optimization wrt. Hash Joins that can become a 
pitfall. His advise is to increase the statistic targets on the hashed 
outer relation. Might be worth a try.


--
Gunnar Nick Bluth
RHCE/SCLA

Mobil   +49 172 8853339
Email: gunnar.bl...@pro-open.de
__
In 1984 mainstream users were choosing VMS over UNIX.  Ten years later
they are choosing Windows over UNIX.  What part of that message aren't you
getting? - Tom Payne



--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Re: Increasing work_mem and shared_buffers on Postgres 9.2 significantly slows down queries

2012-11-06 Thread Gunnar Nick Bluth

Am 06.11.2012 18:38, schrieb Petr Praus:


Yes, but note that this happens only in Linux. Increasing work_mem on 
my iMac increases performance (but the queries are slower under OSX 
than on virtualized Ubuntu on the same machine). Over the weekend, I 
tried the same test on my Ubuntu home machine with Ivy Bridge i5 3570K 
and it also slows down (from ~900ms with work_mem=1MB to ~1200ms with 
work_mem=96MB).


OS X is rather different from a memory access point of view, IIRC. So 
the direct comparison actually only shows how well the Linux FS cache 
works (for the temp files created with small work_mem ;-).


The i5 puzzles me a bit though...



I'm pretty sure you're hitting some subtle, memory-access-related
cornercase here.

The L2 cache of your X7350 CPUs is 2MB, could you run the tests
with, say, 1, 2, 4 and 8MB of work_mem and post the results?

I made a pgbench test with the same query and run it 25 times (5 
clients, 5 transactions each):

work_mem   speed
1MB1794ms
2MB1877ms
4MB2084ms
8MB2141ms
10MB   2124ms
12MB   3018ms
16MB   3004ms
32MB   2999ms
64MB   3015ms

It seems that there is some sort of plateau.
Two, afaics. The 1-2 change hints towards occasionally breaching your 
L2 cache, so it can probably be ignored. The actual plateaus thus seem 
to be 0-2, 2-12, = 12.
It'd be interesting to see the EXPLAIN ANALYSE outputs for these levels, 
the buckets and batches in particular. I'd reckon we'll see significant 
changes at 2-4 and 10-12MB work_mem.


So, to sum this up (and make someone more competent bite on it maybe 
;-), on your SMP, FSB, fake-multicore system all hash-related 
works that potentially switch to different implementations internally 
(but w/out telling us so) when given more work_mem are slower.
See other post... it actually does tell us (# of buckets/batches). 
However, the result is not good and could potentially be improved be 
twealing the statistic_targets of the joined tables/columns.


I wonder why noone actually understanding the implementation chipped in 
yet... Andres, Greg, Tom, whoever actually understands what's happening 
here, anyone reading this? ;-)


Cheers,

--
Gunnar Nick Bluth
RHCE/SCLA

Mobil   +49 172 8853339
Email: gunnar.bl...@pro-open.de
__
In 1984 mainstream users were choosing VMS over UNIX.  Ten years later
they are choosing Windows over UNIX.  What part of that message aren't you
getting? - Tom Payne



Re: [PERFORM] Re: Increasing work_mem and shared_buffers on Postgres 9.2 significantly slows down queries

2012-11-06 Thread Gunnar Nick Bluth

Am 06.11.2012 21:08, schrieb Petr Praus:


2MB: http://explain.depesz.com/s/ul1
4MB: http://explain.depesz.com/s/IlVu
10MB: http://explain.depesz.com/s/afx3
12MB: http://explain.depesz.com/s/i0vQ

See the change in the plan between 10MB and 12MB, directly at top level? 
That narrows the thing down quite a bit.


Though I wonder why this didn't show in the original plans...

--
Gunnar Nick Bluth
RHCE/SCLA

Mobil   +49 172 8853339
Email: gunnar.bl...@pro-open.de
__
In 1984 mainstream users were choosing VMS over UNIX.  Ten years later
they are choosing Windows over UNIX.  What part of that message aren't you
getting? - Tom Payne



--
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance


Re: [PERFORM] Re: Increasing work_mem and shared_buffers on Postgres 9.2 significantly slows down queries

2012-11-06 Thread Gunnar Nick Bluth

Am 06.11.2012 21:24, schrieb Petr Praus:
On 6 November 2012 14:17, Gunnar Nick Bluth 
gunnar.bl...@pro-open.de mailto:gunnar.bl...@pro-open.de wrote:


Am 06.11.2012 21:08, schrieb Petr Praus:


2MB: http://explain.depesz.com/s/ul1
4MB: http://explain.depesz.com/s/IlVu
10MB: http://explain.depesz.com/s/afx3
12MB: http://explain.depesz.com/s/i0vQ

See the change in the plan between 10MB and 12MB, directly at top
level? That narrows the thing down quite a bit.

Though I wonder why this didn't show in the original plans...


Yes, the 2,4 and 10 are the same, the only difference is number of 
buckets. But with 12, it makes completely different choices, it 
decides to make sequential scans and hash right joins instead of merge 
joins. And those sequential scans take a loong time. Could this be 
caused by some missing indices perhaps?


Well, you do have indices, as we can clearly see.

The original plans I posted at the start are the same as the 12MB 
plan, I'm not sure why is that, I really hope I didn't make some sort 
of mistake there.


I had been wondering why you didn't have any indices, tbth. However, the 
execution times still grow with work_mem, which is interesting 
independent of the actual plan change...




Thanks for your help by the way! :-)


Oh, no worries there... this is by far the most interesting challenge 
I've encountered in months ;-)


But I do admit that I've reached the end of the ladder now. No idea how 
you can improve your runtime yet. Probably

- using full text search on personinfo
- try different join_collapse_limit / from_collapse_limit / 
enable_hashjoin values


The most pragmatic approach is probably to just stick with work_mem = 
1MB (or less) ;-), but that may potentially bite you later.


--
Gunnar Nick Bluth
RHCE/SCLA

Mobil   +49 172 8853339
Email: gunnar.bl...@pro-open.de
__
In 1984 mainstream users were choosing VMS over UNIX.  Ten years later
they are choosing Windows over UNIX.  What part of that message aren't you
getting? - Tom Payne



Re: [PERFORM] Re: Increasing work_mem and shared_buffers on Postgres 9.2 significantly slows down queries

2012-11-04 Thread Gunnar Nick Bluth

Am 03.11.2012 18:19, schrieb Petr Praus:
On 3 November 2012 12:09, Gunnar Nick Bluth 
gunnar.bl...@pro-open.de mailto:gunnar.bl...@pro-open.de wrote:


Am 03.11.2012 16:20, schrieb Petr Praus:


Your CPUs are indeed pretty oldschool. FSB based, IIRC, not
NUMA. A process migration would be even more expensive there.



Ok, I've actually looked these up now... at the time these were
current, I was in the lucky situation to only deal with Opterons.
And actually, with these CPUs it is pretty possible that Scott
Marlowe's hint (check vm.zone_reclaim_mode) was pointing in the
right direction. Did you check that?


I did check that, it's zero. I responded to his message, but my 
messages to the mailing list are getting delayed by ~24 hours because 
somebody has to always bless them.





Yes, same behaviour. I let the shared_buffers be the default
(which is 8MB). With work_mem 1MB the query runs fast, with
96MB it runs slow (same times as before). It really seems
that the culprit is work_mem.




Well, I'm pretty sure that having more work_mem is a good thing
(tm) normally ;-)


Well, that's what I always thought too! :-)

So, to sum this up (and make someone more competent bite on it maybe 
;-), on your SMP, FSB, fake-multicore system all hash-related works 
that potentially switch to different implementations internally (but 
w/out telling us so) when given more work_mem are slower.


I'm pretty sure you're hitting some subtle, memory-access-related 
cornercase here.


The L2 cache of your X7350 CPUs is 2MB, could you run the tests with, 
say, 1, 2, 4 and 8MB of work_mem and post the results?


--
Gunnar Nick Bluth
RHCE/SCLA

Mobil   +49 172 8853339
Email: gunnar.bl...@pro-open.de
__
In 1984 mainstream users were choosing VMS over UNIX.  Ten years later
they are choosing Windows over UNIX.  What part of that message aren't you
getting? - Tom Payne



Re: [PERFORM] Re: Increasing work_mem and shared_buffers on Postgres 9.2 significantly slows down queries

2012-11-03 Thread Gunnar Nick Bluth

Am 02.11.2012 17:12, schrieb Petr Praus:

Your CPUs are indeed pretty oldschool. FSB based, IIRC, not NUMA. A 
process migration would be even more expensive there.



Might be worth to
- manually pin (with taskset) the session you test this in to a
particular CPU (once on each socket) to see if the times change


I tested this and it does not seem to have any effect (assuming I used 
taskset correctly but I think so: taskset 02 psql to pin down to CPU 
#1 and taskset 01 psql to pin to CPU #0).

Well, that pinned your _client_ to the CPUs, not the server side session ;-)
You'd have to spot for the PID of the new IDLE server process and pin 
that using taskset -p. Also, 01 and 02 are probably cores in the same 
package/socket. Try lscpu first and spot for NUMA node* lines at the 
bottom.

But anyway... let's try something else first:


- try reducing work_mem in the session you're testing in (so you
have large SHM, but small work mem)


Did this and it indicates to me that shared_buffers setting actually 
does not have an effect on this behaviour as I previously thought it 
has. It really boils down to work_mem: when I set shared_buffers to 
something large (say 4GB) and just play with work_mem the problem 
persists.
This only confirms what we've seen before. As soon as your work_mem 
permits an in-memory sort of the intermediate result set (which at that 
point in time is where? In the SHM, or in the private memory of the 
backend? I can't tell, tbth), the sort takes longer than when it's using 
a temp file.


What if you reduce the shared_buffers to your original value and only 
increase/decrease the session's work_mem? Same behaviour?


Cheers,

--
Gunnar Nick Bluth
RHCE/SCLA

Mobil   +49 172 8853339
Email: gunnar.bl...@pro-open.de
__
In 1984 mainstream users were choosing VMS over UNIX.  Ten years later
they are choosing Windows over UNIX.  What part of that message aren't you
getting? - Tom Payne



Re: [PERFORM] Re: Increasing work_mem and shared_buffers on Postgres 9.2 significantly slows down queries

2012-11-03 Thread Gunnar Nick Bluth

Am 03.11.2012 16:20, schrieb Petr Praus:


Your CPUs are indeed pretty oldschool. FSB based, IIRC, not NUMA.
A process migration would be even more expensive there.



Ok, I've actually looked these up now... at the time these were current, 
I was in the lucky situation to only deal with Opterons. And actually, 
with these CPUs it is pretty possible that Scott Marlowe's hint (check 
vm.zone_reclaim_mode) was pointing in the right direction. Did you check 
that?




Yes, same behaviour. I let the shared_buffers be the default
(which is 8MB). With work_mem 1MB the query runs fast, with 96MB
it runs slow (same times as before). It really seems that the
culprit is work_mem.




Well, I'm pretty sure that having more work_mem is a good thing (tm) 
normally ;-)


--
Gunnar Nick Bluth
RHCE/SCLA

Mobil   +49 172 8853339
Email: gunnar.bl...@pro-open.de
__
In 1984 mainstream users were choosing VMS over UNIX.  Ten years later
they are choosing Windows over UNIX.  What part of that message aren't you
getting? - Tom Payne



Re: [PERFORM] Re: Increasing work_mem and shared_buffers on Postgres 9.2 significantly slows down queries

2012-11-01 Thread Gunnar Nick Bluth

Am 01.11.2012 21:40, schrieb Marcos Ortiz:

Regards, Petr.
Tuning PostgreSQL is not just change the postgresql.conf, it includes 
more things like:

- the filesystem that you are using
- the kernel version that you using (particularly in Linux systems)
- the tuning to kernel variables
- the type of discs that you are using (SSDs are very fast, like you 
saw in your iMac system)


On 10/30/2012 02:44 PM, Petr Praus wrote:
I just found one particularly interesting fact: when I perform the 
same test on my mid-2010 iMac (OSX 10.7.5) also with Postgres 9.2.1 
and 16GB RAM, I don't experience the slow down.

Specifically:
set work_mem='1MB';
select ...; // running time is ~1800 ms
set work_mem='96MB';
select ...' // running time is ~1500 ms

When I do exactly the same query (the one from my previous post) with 
exactly the same data on the server:

I get 2100 ms with work_mem=1MB and 3200 ms with 96 MB.

Just some thoughts (interested in this, once seen a Sybase ASE come 
close to a halt when we threw a huge lot of SHM at it...).


8 cores, so probably on 2 sockets? What CPU generation?

Both explain outputs show an amount of read buffers. Did you warm the 
caches before testing?


Maybe you're hitting a NUMA issue there? If those reads come from the 
OS' cache, the scheduler might decide to move your process to a 
different core (that can access the cache better), then moves it back 
when you access the SHM segment more (the ~4GB get allocated at startup, 
so probably close to the CPU the postmaster ist running on). A 
migration to a different cacheline is very expensive.


The temp reads/writes (i.e., the OS cache for the temp files) would 
probably be allocated close to the CPU requesting the temp file.


Just groping about in the dark though... but the iMac is obviously not 
affected by this, with one socket/memory channel/cache line.


Might be worth to
- manually pin (with taskset) the session you test this in to a 
particular CPU (once on each socket) to see if the times change
- try reducing work_mem in the session you're testing in (so you have 
large SHM, but small work mem)


Cheers,

--
Gunnar Nick Bluth
RHCE/SCLA

Mobil   +49 172 8853339
Email: gunnar.bl...@pro-open.de
__
In 1984 mainstream users were choosing VMS over UNIX.  Ten years later
they are choosing Windows over UNIX.  What part of that message aren't you
getting? - Tom Payne