Re: Improving connection scalability: GetSnapshotData()

2020-03-31 Thread Andres Freund
Hi, I'm still fighting with snapshot_too_old. The feature is just badly undertested, underdocumented, and there's lots of other oddities. I've now spent about as much time on that feature than on the whole rest of the patchset. As an example for under-documented, here's a definitely non-trivial

Re: Improving connection scalability: GetSnapshotData()

2020-03-30 Thread Alexander Korotkov
On Sun, Mar 29, 2020 at 9:50 PM Andres Freund wrote: > On March 29, 2020 11:24:32 AM PDT, Alexander Korotkov > wrote: > > clearly a big win on majority > >of workloads, I think we still need to investigate different workloads > >on different hardware to ensure there is no regression. > >

Re: Improving connection scalability: GetSnapshotData()

2020-03-29 Thread Tomas Vondra
On Sun, Mar 29, 2020 at 11:50:10AM -0700, Andres Freund wrote: Hi, On March 29, 2020 11:24:32 AM PDT, Alexander Korotkov wrote: clearly a big win on majority of workloads, I think we still need to investigate different workloads on different hardware to ensure there is no regression.

Re: Improving connection scalability: GetSnapshotData()

2020-03-29 Thread Andres Freund
Hi, On March 29, 2020 11:24:32 AM PDT, Alexander Korotkov wrote: > clearly a big win on majority >of workloads, I think we still need to investigate different workloads >on different hardware to ensure there is no regression. Definitely. Which workloads are you thinking of? I can think of

Re: Improving connection scalability: GetSnapshotData()

2020-03-29 Thread Alexander Korotkov
On Sun, Mar 29, 2020 at 4:40 AM Peter Geoghegan wrote: > On Sun, Mar 1, 2020 at 12:36 AM Andres Freund wrote: > > The workload is a pgbench readonly, with pgbench -M prepared -c $conns > > -j $conns -S -n for each client count. This is on a machine with 2 > > Intel(R) Xeon(R) Platinum 8168, but

Re: Improving connection scalability: GetSnapshotData()

2020-03-28 Thread Andres Freund
Hi, On 2020-03-28 18:39:32 -0700, Peter Geoghegan wrote: > I have heard quite a few complaints about the scalability of snapshot > acquisition in Postgres. Generally from very large users that are not > well represented on the mailing lists, for a variety of reasons. The > GetSnapshotData()

Re: Improving connection scalability: GetSnapshotData()

2020-03-28 Thread Bruce Momjian
On Sat, Mar 28, 2020 at 06:39:32PM -0700, Peter Geoghegan wrote: > On Sun, Mar 1, 2020 at 12:36 AM Andres Freund wrote: > > The workload is a pgbench readonly, with pgbench -M prepared -c $conns > > -j $conns -S -n for each client count. This is on a machine with 2 > > Intel(R) Xeon(R) Platinum

Re: Improving connection scalability: GetSnapshotData()

2020-03-28 Thread Peter Geoghegan
On Sun, Mar 1, 2020 at 12:36 AM Andres Freund wrote: > The workload is a pgbench readonly, with pgbench -M prepared -c $conns > -j $conns -S -n for each client count. This is on a machine with 2 > Intel(R) Xeon(R) Platinum 8168, but virtualized. > > conns tps master tps

Re: Improving connection scalability: GetSnapshotData()

2020-03-28 Thread Thomas Munro
On Sun, Mar 29, 2020 at 1:49 PM Andres Freund wrote: > > 13. is -> are > > > > * accessed data is stored contiguously in memory in as few cache lines as > > Oh? 'data are stored' sounds wrong to me, somehow. In computer contexts it seems pretty well established that we treat "data" as an

Re: Improving connection scalability: GetSnapshotData()

2020-03-28 Thread Andres Freund
Hi, On 2020-03-17 23:59:14 +1300, David Rowley wrote: > Nice performance gains. Thanks. > On Sun, 1 Mar 2020 at 21:36, Andres Freund wrote: > 2. I don't quite understand your change in > UpdateSubscriptionRelState(). snap seems unused. Drilling down into > SearchSysCacheCopy2, in

Re: Improving connection scalability: GetSnapshotData()

2020-03-20 Thread Andres Freund
Hi, Thanks for looking! On 2020-03-20 18:23:03 +1300, Thomas Munro wrote: > On Sun, Mar 1, 2020 at 9:36 PM Andres Freund wrote: > > I'm still working on cleaning that part of the patch up, I'll post it in > > a bit. > > I looked at that part on your public pgxact-split branch. In that >

Re: Improving connection scalability: GetSnapshotData()

2020-03-19 Thread Thomas Munro
On Sun, Mar 1, 2020 at 9:36 PM Andres Freund wrote: > conns tps master tps pgxact-split > > 1 26842.49284526524.194821 > 10 246923.158682 249224.782661 > 50 695956.539704 709833.746374 > 100 1054727.043139 1903616.306028

Re: Improving connection scalability: GetSnapshotData()

2020-03-17 Thread David Rowley
Hi, Nice performance gains. On Sun, 1 Mar 2020 at 21:36, Andres Freund wrote: > The series currently consists out of: > > 0001-0005: Fixes and assert improvements that are independent of the patch, > but >are hit by the new code (see also separate thread). > > 0006: Move delayChkpt

Re: Improving connection scalability: GetSnapshotData()

2020-03-04 Thread David Steele
On 3/1/20 3:36 AM, Andres Freund wrote: I think these are pretty nice results. Indeed they are. Is the target version PG13 or PG14? It seems like a pretty big patch to go in the last commitfest for PG13. Regards, -- -David da...@pgmasters.net

Re: Improving connection scalability: GetSnapshotData()

2020-03-02 Thread Andres Freund
Hi, On 2020-03-01 00:36:01 -0800, Andres Freund wrote: > Here are some numbers for the submitted patch series. I'd to cull some > further improvements to make it more manageable, but I think the numbers > still are quite convincing. > > The workload is a pgbench readonly, with pgbench -M

Re: Improving connection scalability: GetSnapshotData()

2020-03-01 Thread Amit Kapila
On Sun, Mar 1, 2020 at 2:17 PM Andres Freund wrote: > > Hi, > > On 2020-03-01 00:36:01 -0800, Andres Freund wrote: > > conns tps mastertps pgxact-split > > > > 1 26842.49284526524.194821 > > 10 246923.158682 249224.782661 > > 50 695956.539704

Re: Improving connection scalability: GetSnapshotData()

2020-03-01 Thread Andres Freund
Hi, On 2020-03-01 00:36:01 -0800, Andres Freund wrote: > conns tps mastertps pgxact-split > > 1 26842.49284526524.194821 > 10 246923.158682 249224.782661 > 50 695956.539704 709833.746374 > 100 1054727.043139

<    1   2