Re: [PERFORM] Partitions and joins lead to index lookups on all partitions

[voodooless]

Back again,

I did some tests with our test machine, having a difficult query doing some
fancy stuff ;)

I made two versions, one using partitioned data, one, using unpartitioned
data, both having the same equivalent indexes. It's using two of those big
tables, one 28GB data and 17GB index, one 25GB data and 41GB indexes (both
for the unpartitioned versions). Our test machine has 32GB of memory, short
config:

maintenance_work_mem = 1GB
checkpoint_completion_target = 0.9
effective_cache_size = 22GB
work_mem = 80MB
wal_buffers = 8MB
checkpoint_segments = 16
shared_buffers = 7680MB
max_connections = 400

At first I tested the query performance. It turned out that the
unpartitioned version was about 36 times faster, of course for the obvious
reason stated in my initial post. both are fully using the indexes they
have, and the partitioned version even has it's indexes on SSD.

Then I did some insert tests using generate_series to insert 10 rows
into one of the tables. It turns out that the unpartitioned version is again
faster, this time 30.9 vs 1.8 seconds. This is a huge difference. For the
second table, with the huge 41GB index it's 30.5 vs 5.2 seconds, still a big
difference.

Conclusion: partitioning does not benefit us, and probably others, specially
when doing lots of joins and using parameterized queries.



--
View this message in context: 
http://postgresql.1045698.n5.nabble.com/Partitions-and-joins-lead-to-index-lookups-on-all-partitions-tp5055965p5074907.html
Sent from the PostgreSQL - performance mailing list archive at Nabble.com.

-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance

Re: [PERFORM] Partitions and joins lead to index lookups on all partitions

[voodooless]

Hi Ondrej,

Your solution has occurred to me, and wil even work in some cases. But in
more advanced queries, where for example, I would need the group ID again to
do some window function magic, this will sadly not work, without again doing
a reverse lookup to the ref_table to find it again. This scheme might still
be faster though even though it would take more queries.

Im now testing some of queries against a non-paritioned version of our
dataset to see what the difference is.

I'm wondering how much the insert performance wil be impacted when
not-paritioning our data. We do have a few indexes and constriants on these
tables, but not whole lot. I'll so some measurements to see how this wil
work out.

The general dilemma would be as  follows:

What if the suggested max of 100 partions would mean that a partition table
will also not fit into memory efficiently, and/or that the access pattern is
such that because of the query planner, it needs to work it's way though all
the partitions for virtually most of the serious queries being done on the
data set.


--
View this message in context: 
http://postgresql.1045698.n5.nabble.com/Partitions-and-joins-lead-to-index-lookups-on-all-partitions-tp5055965p5058853.html
Sent from the PostgreSQL - performance mailing list archive at Nabble.com.

-- 
Sent via pgsql-performance mailing list (pgsql-performance@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-performance