RE: How to make partitioning scale better for larger numbers of partitions

2018-07-16 Thread Kato, Sho
On 2018/07/16 13:16, Tsunakawa-san wrote: >Thanks. And what does the comparison look like between the unpartitioned case >and various partition counts? What's the performance characteristics in terms >of the latency and partition count? I thought that's what you tried to >reveal first? In

Re: How to make partitioning scale better for larger numbers of partitions

2018-07-16 Thread Amit Langote
On 2018/07/17 12:14, Ashutosh Bapat wrote: > On Tue, Jul 17, 2018 at 8:31 AM, Kato, Sho wrote: >> On 2018/07/17 10:49, Amit Langote wrote: >>> Perhaps, Kato-san only intended to report that the time that planner spends >>> for a partitioned table with 1100 partitions is just too high compared to

Re: How to make partitioning scale better for larger numbers of partitions

2018-07-16 Thread Ashutosh Bapat
On Tue, Jul 17, 2018 at 8:31 AM, Kato, Sho wrote: > On 2018/07/17 10:49, Amit Langote wrote: >>Perhaps, Kato-san only intended to report that the time that planner spends >>for a partitioned table with 1100 partitions is just too high compared to the >>time it spends on a non-partitioned table.

RE: How to make partitioning scale better for larger numbers of partitions

2018-07-16 Thread Kato, Sho
On 2018/07/17 10:49, Amit Langote wrote: >Perhaps, Kato-san only intended to report that the time that planner spends >for a partitioned table with 1100 partitions is just too high compared to the >time it spends on a non-partitioned table. yes, It is included for the purposes of this

Re: How to make partitioning scale better for larger numbers of partitions

2018-07-16 Thread Amit Langote
On 2018/07/13 22:10, Ashutosh Bapat wrote: > On Fri, Jul 13, 2018 at 9:23 AM, Kato, Sho wrote: >>> I wondered if you compared to PG10 or to inheritence-partitioning (parent >>> with relkind='r' and either trigger or rule or >INSERT/UPDATE directly into >>> child) ? >> >> Thank you for your

RE: How to make partitioning scale better for larger numbers of partitions

2018-07-15 Thread Tsunakawa, Takayuki
From: Kato, Sho [mailto:kato-...@jp.fujitsu.com] > I did pgbench -M prepared and perf record. > > UPDATE latency in prepared mode is 95% shorter than in simple mode. > SELECT latency in prepared mode is 54% shorter than in simple mode. > INSERT latency in prepared mode is 8% shorter than in

Re: How to make partitioning scale better for larger numbers of partitions

2018-07-13 Thread Ashutosh Bapat
On Fri, Jul 13, 2018 at 9:23 AM, Kato, Sho wrote: >>I wondered if you compared to PG10 or to inheritence-partitioning (parent >>with relkind='r' and either trigger or rule or >INSERT/UPDATE directly into >>child) ? > > Thank you for your reply. > > I compared to PG11beta2 with non-partitioned

RE: How to make partitioning scale better for larger numbers of partitions

2018-07-13 Thread Kato, Sho
Tsunakawa-san >Kato-san, could you try pgbench -M prepared? I did pgbench -M prepared and perf record. UPDATE latency in prepared mode is 95% shorter than in simple mode. SELECT latency in prepared mode is 54% shorter than in simple mode. INSERT latency in prepared mode is 8% shorter than in

Re: How to make partitioning scale better for larger numbers of partitions

2018-07-13 Thread Amit Langote
On 2018/07/13 16:29, Kato, Sho wrote: > I also benchmark PG10. > Actually, SELECT latency on PG11beta2 + patch1 is faster than PG10. > > SELECT latency with 800 leaf partition > -- > PG10 5.62 ms > PG11 3.869 ms > > But, even PG11, SELECT statement takes

RE: How to make partitioning scale better for larger numbers of partitions

2018-07-13 Thread Tsunakawa, Takayuki
From: Amit Langote [mailto:langote_amit...@lab.ntt.co.jp] > The immediate one I think is to refactor the planner such that the new > pruning code, that we were able to utilize for SELECT in PG 11, can also > be used for UPDATE/DELETE. Refactoring needed to replace the pruning > algorithm was

Re: How to make partitioning scale better for larger numbers of partitions

2018-07-13 Thread David Rowley
On 13 July 2018 at 18:53, Tsunakawa, Takayuki wrote: > By the way, what do you think is the "ideal and should-be-feasible" goal and > the "realistic" goal we can reach in the near future (e.g. PG 12)? Say, Depends. Patched don't move that fast without review and nothing gets committed without

RE: How to make partitioning scale better for larger numbers of partitions

2018-07-13 Thread Tsunakawa, Takayuki
From: David Rowley [mailto:david.row...@2ndquadrant.com] > > David has submitted multiple patches for PG 12, one of which speeds up > pruning of UPDATE/DELETE (I couldn't find it in the current CF, though.) > What challenges are there for future versions, and which of them are being > addressed by

Re: How to make partitioning scale better for larger numbers of partitions

2018-07-13 Thread Amit Langote
On 2018/07/13 14:49, Tsunakawa, Takayuki wrote: > From: Amit Langote [mailto:langote_amit...@lab.ntt.co.jp] >> For SELECT/UPDATE/DELETE, overhead of partitioning in the planning phase >> is pretty significant and gets worse as the number of partitions grows. >> I >> had intended to fix that in PG

Re: How to make partitioning scale better for larger numbers of partitions

2018-07-13 Thread David Rowley
On 13 July 2018 at 14:58, Kato, Sho wrote: > Of course I'm sure table partitioning work well with up to a hundred > partitions as written on the postgresql document. > > But, my customer will use partitioned table with 1.1k leaf partitions. > > So, we need to improve performance. > > Any ideas?

Re: How to make partitioning scale better for larger numbers of partitions

2018-07-13 Thread David Rowley
On 13 July 2018 at 17:49, Tsunakawa, Takayuki wrote: > David has submitted multiple patches for PG 12, one of which speeds up > pruning of UPDATE/DELETE (I couldn't find it in the current CF, though.) > What challenges are there for future versions, and which of them are being > addressed by

Re: How to make partitioning scale better for larger numbers of partitions

2018-07-13 Thread Justin Pryzby
On Fri, Jul 13, 2018 at 05:49:20AM +, Tsunakawa, Takayuki wrote: > David has submitted multiple patches for PG 12, one of which speeds up > pruning of UPDATE/DELETE (I couldn't find it in the current CF, though.) > What challenges are there for future versions, and which of them are being

RE: How to make partitioning scale better for larger numbers of partitions

2018-07-12 Thread Tsunakawa, Takayuki
From: Amit Langote [mailto:langote_amit...@lab.ntt.co.jp] > For SELECT/UPDATE/DELETE, overhead of partitioning in the planning phase > is pretty significant and gets worse as the number of partitions grows. > I > had intended to fix that in PG 11, but we could only manage to get part > of > that

Re: How to make partitioning scale better for larger numbers of partitions

2018-07-12 Thread Amit Langote
Kato-san, On 2018/07/13 11:58, Kato, Sho wrote: > Hi, > > I benchmarked on a RANGE partitioned table with 1.1k leaf partitions and no > sub-partitioned tables. Thanks for sharing the results. > But, statement latencies on a partitioned table is much slower than on a > non-partitioned table.

How to make partitioning scale better for larger numbers of partitions

2018-07-12 Thread Kato, Sho
Hi, I benchmarked on a RANGE partitioned table with 1.1k leaf partitions and no sub-partitioned tables. But, statement latencies on a partitioned table is much slower than on a non-partitioned table. UPDATE latency is 210 times slower than a non-partitioned table. SELECT latency is 36 times