Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Fri, Sep 1, 2017 at 9:17 PM, Robert Haas wrote: > On Fri, Sep 1, 2017 at 10:03 AM, Dilip Kumar wrote: >>> Sure will do so. In the meantime, I have rebased the patch. >> >> I have repeated some of the tests we have performed earlier. > Thanks for

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Fri, Sep 1, 2017 at 10:03 AM, Dilip Kumar wrote: >> Sure will do so. In the meantime, I have rebased the patch. > > I have repeated some of the tests we have performed earlier. OK, these tests seem to show that this is still working. Committed, again. Let's hope this

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Wed, Aug 30, 2017 at 12:54 PM, Amit Kapila wrote: > > That would have been better. In any case, will do the tests on some > higher end machine and will share the results. > >> Given that we've changed the approach here somewhat, I think we need >> to validate that

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Wed, Aug 30, 2017 at 2:43 AM, Robert Haas wrote: > On Tue, Jul 4, 2017 at 12:33 AM, Amit Kapila wrote: >> I have updated the patch to support wait events and moved it to upcoming CF. > > This patch doesn't apply any more, but I made it apply

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Tue, Jul 4, 2017 at 12:33 AM, Amit Kapila wrote: > I have updated the patch to support wait events and moved it to upcoming CF. This patch doesn't apply any more, but I made it apply with a hammer and then did a little benchmarking (scylla, EDB server, Intel Xeon

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Mon, Jul 3, 2017 at 6:15 PM, Amit Kapila wrote: > On Thu, Mar 23, 2017 at 1:18 PM, Ashutosh Sharma > wrote: >> >> Conclusion: >> As seen from the test results mentioned above, there is some performance >> improvement with 3 SP(s), with 5 SP(s)

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Thu, Mar 23, 2017 at 1:18 PM, Ashutosh Sharma wrote: > > *Conclusion:* > As seen from the test results mentioned above, there is some performance > improvement with 3 SP(s), with 5 SP(s) the results with patch is slightly > better than HEAD, with 7 and 10 SP(s) we do see

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Thu, Mar 9, 2017 at 5:49 PM, Robert Haas wrote: > However, I just realized that in > both this case and in the case of group XID clearing, we weren't > advertising a wait event for the PGSemaphoreLock calls that are part > of the group locking machinery. I think we

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

Hi All, I have tried to test 'group_update_clog_v11.1.patch' shared upthread by Amit on a high end machine. I have tested the patch with various savepoints in my test script. The machine details along with test scripts and the test results are shown below, Machine details: 24

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Mon, Mar 20, 2017 at 8:27 AM, Robert Haas wrote: > On Fri, Mar 17, 2017 at 2:30 AM, Amit Kapila wrote: >>> I was wondering about doing an explicit test: if the XID being >>> committed matches the one in the PGPROC, and nsubxids matches, and the

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Fri, Mar 17, 2017 at 2:30 AM, Amit Kapila wrote: >> I was wondering about doing an explicit test: if the XID being >> committed matches the one in the PGPROC, and nsubxids matches, and the >> actual list of XIDs matches, then apply the optimization. That could >>

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Sun, Mar 12, 2017 at 8:11 AM, Robert Haas wrote: > On Fri, Mar 10, 2017 at 7:39 PM, Amit Kapila wrote: >> I agree that more analysis can help us to decide if we can use subxids >> from PGPROC and if so under what conditions. Have you considered

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Fri, Mar 10, 2017 at 7:39 PM, Amit Kapila wrote: > I agree that more analysis can help us to decide if we can use subxids > from PGPROC and if so under what conditions. Have you considered the > another patch I have posted to fix the issue which is to do this >

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Sat, Mar 11, 2017 at 2:10 AM, Robert Haas wrote: > On Fri, Mar 10, 2017 at 6:25 AM, Amit Kapila wrote: >> On Fri, Mar 10, 2017 at 11:51 AM, Tom Lane wrote: >>> Amit Kapila writes: Just to let

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

Robert Haas wrote: > The smoking gun was in 009_twophase_slave.log: > > TRAP: FailedAssertion("!(nsubxids == MyPgXact->nxids)", File: > "clog.c", Line: 288) > > ...and then the node shuts down, which is why this hangs forever. > (Also... what's up with it hanging forever instead of timing out

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Fri, Mar 10, 2017 at 3:40 PM, Robert Haas wrote: > Finally, I had an unexplained hang during the TAP tests while testing > out your fix patch. I haven't been able to reproduce that so it > might've just been an artifact of something stupid I did, or of some > unrelated

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Fri, Mar 10, 2017 at 6:25 AM, Amit Kapila wrote: > On Fri, Mar 10, 2017 at 11:51 AM, Tom Lane wrote: >> Amit Kapila writes: >>> Just to let you know that I think I have figured out the reason of >>> failure. If we run the

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Fri, Mar 10, 2017 at 11:51 AM, Tom Lane wrote: > Amit Kapila writes: >> Just to let you know that I think I have figured out the reason of >> failure. If we run the regressions with attached patch, it will make >> the regression tests fail

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Fri, Mar 10, 2017 at 11:43 AM, Amit Kapila wrote: > On Fri, Mar 10, 2017 at 10:51 AM, Tom Lane wrote: >> >> Also, I see clam reported in green just now, so it's not 100% >> reproducible :-( >> > > Just to let you know that I think I have figured

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

Amit Kapila writes: > Just to let you know that I think I have figured out the reason of > failure. If we run the regressions with attached patch, it will make > the regression tests fail consistently in same way. The patch just > makes all transaction status updates to

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Fri, Mar 10, 2017 at 10:51 AM, Tom Lane wrote: > Robert Haas writes: >> On Thu, Mar 9, 2017 at 9:17 PM, Tom Lane wrote: >>> Buildfarm thinks eight wasn't enough. >>>

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

Robert Haas writes: > On Thu, Mar 9, 2017 at 9:17 PM, Tom Lane wrote: >> Buildfarm thinks eight wasn't enough. >> https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=clam=2017-03-10%2002%3A00%3A01 > At first I was confused how you knew that this

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Thu, Mar 9, 2017 at 9:17 PM, Tom Lane wrote: > Robert Haas writes: >> I think eight is enough. Committed with some cosmetic changes. > > Buildfarm thinks eight wasn't enough. > >

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Fri, Mar 10, 2017 at 7:47 AM, Tom Lane wrote: > Robert Haas writes: >> I think eight is enough. Committed with some cosmetic changes. > > Buildfarm thinks eight wasn't enough. > >

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

Robert Haas writes: > I think eight is enough. Committed with some cosmetic changes. Buildfarm thinks eight wasn't enough. https://buildfarm.postgresql.org/cgi-bin/show_log.pl?nm=clam=2017-03-10%2002%3A00%3A01 regards, tom lane -- Sent via

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Tue, Jan 31, 2017 at 11:35 PM, Michael Paquier wrote: >> Thanks for the review. > > Moved to CF 2017-03, the 8th commit fest of this patch. I think eight is enough. Committed with some cosmetic changes. I think the turning point for this somewhat-troubled patch

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Tue, Jan 17, 2017 at 9:18 PM, Amit Kapila wrote: > On Tue, Jan 17, 2017 at 11:39 AM, Dilip Kumar wrote: >> On Wed, Jan 11, 2017 at 10:55 AM, Dilip Kumar wrote: >>> I have reviewed the latest patch and I don't have any

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Tue, Jan 17, 2017 at 11:39 AM, Dilip Kumar wrote: > On Wed, Jan 11, 2017 at 10:55 AM, Dilip Kumar wrote: >> I have reviewed the latest patch and I don't have any more comments. >> So if there is no objection from other reviewers I can move it to

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Wed, Jan 11, 2017 at 10:55 AM, Dilip Kumar wrote: > I have reviewed the latest patch and I don't have any more comments. > So if there is no objection from other reviewers I can move it to > "Ready For Committer"? Seeing no objections, I have moved it to Ready For

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Sat, Dec 31, 2016 at 9:01 AM, Amit Kapila wrote: > Agreed and changed accordingly. > >> 2. It seems that we have missed one unlock in case of absorbed >> wakeups. You have initialised extraWaits with -1 and if there is one >> extra wake up then extraWaits will become 0

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Thu, Dec 29, 2016 at 10:41 AM, Dilip Kumar wrote: > > I have done one more pass of the review today. I have few comments. > > + if (nextidx != INVALID_PGPROCNO) > + { > + /* Sleep until the leader updates our XID status. */ > + for (;;) > + { > + /* acts as a read

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Fri, Dec 23, 2016 at 8:28 AM, Amit Kapila wrote: > The results look positive. Do you think we can conclude based on all > the tests you and Dilip have done, that we can move forward with this > patch (in particular group-update) or do you still want to do more >

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On 12/23/2016 03:58 AM, Amit Kapila wrote: On Thu, Dec 22, 2016 at 6:59 PM, Tomas Vondra wrote: Hi, But as discussed with Amit in Tokyo at pgconf.asia, I got access to a Power8e machine (IBM 8247-22L to be precise). It's a much smaller machine compared to the x86

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Thu, Dec 22, 2016 at 6:59 PM, Tomas Vondra wrote: > Hi, > > But as discussed with Amit in Tokyo at pgconf.asia, I got access to a > Power8e machine (IBM 8247-22L to be precise). It's a much smaller machine > compared to the x86 one, though - it only has 24 cores

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

Hi, The attached results show that: (a) master shows the same zig-zag behavior - No idea why this wasn't observed on the previous runs. (b) group_update actually seems to improve the situation, because the performance keeps stable up to 72 clients, while on master the fluctuation starts way

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Mon, Dec 5, 2016 at 1:14 PM, Amit Kapila wrote: > On Mon, Dec 5, 2016 at 6:00 AM, Haribabu Kommi > wrote: > > No, that is not true. You have quoted the wrong message, that > discussion was about WALWriteLock contention not about the patch

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Mon, Dec 5, 2016 at 6:00 AM, Haribabu Kommi wrote: > > > On Fri, Nov 4, 2016 at 8:20 PM, Amit Kapila wrote: >> >> On Thu, Nov 3, 2016 at 8:38 PM, Robert Haas wrote: >> > On Tue, Nov 1, 2016 at 11:31 PM, Tomas Vondra >>

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Fri, Nov 4, 2016 at 8:20 PM, Amit Kapila wrote: > On Thu, Nov 3, 2016 at 8:38 PM, Robert Haas wrote: > > On Tue, Nov 1, 2016 at 11:31 PM, Tomas Vondra > >> The difference is that both the fast-path locks and msgNumLock went into > >> 9.2, so

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Thu, Nov 3, 2016 at 8:38 PM, Robert Haas wrote: > On Tue, Nov 1, 2016 at 11:31 PM, Tomas Vondra >> The difference is that both the fast-path locks and msgNumLock went into >> 9.2, so that end users probably never saw that regression. But we don't know >> if that happens

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Tue, Nov 1, 2016 at 11:31 PM, Tomas Vondra wrote: > I don't think I've suggested not committing any of the clog patches (or > other patches in general) because shifting the contention somewhere else > might cause regressions. At the end of the last CF I've however

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On 11/02/2016 05:52 PM, Amit Kapila wrote: On Wed, Nov 2, 2016 at 9:01 AM, Tomas Vondra wrote: On 11/01/2016 08:13 PM, Robert Haas wrote: On Mon, Oct 31, 2016 at 5:48 PM, Tomas Vondra wrote: The one remaining thing is the

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On 11/02/2016 05:52 PM, Amit Kapila wrote: On Wed, Nov 2, 2016 at 9:01 AM, Tomas Vondra wrote: On 11/01/2016 08:13 PM, Robert Haas wrote: On Mon, Oct 31, 2016 at 5:48 PM, Tomas Vondra wrote: The one remaining thing is the

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Wed, Nov 2, 2016 at 9:01 AM, Tomas Vondra wrote: > On 11/01/2016 08:13 PM, Robert Haas wrote: >> >> On Mon, Oct 31, 2016 at 5:48 PM, Tomas Vondra >> wrote: >>> > > The one remaining thing is the strange zig-zag behavior, but that

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On 11/01/2016 08:13 PM, Robert Haas wrote: On Mon, Oct 31, 2016 at 5:48 PM, Tomas Vondra wrote: Honestly, I have no idea what to think about this ... I think a lot of the details here depend on OS scheduler behavior. For example, here's one of the first

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Mon, Oct 31, 2016 at 5:48 PM, Tomas Vondra wrote: > Honestly, I have no idea what to think about this ... I think a lot of the details here depend on OS scheduler behavior. For example, here's one of the first scalability graphs I ever did:

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On 10/31/2016 02:24 PM, Tomas Vondra wrote: On 10/31/2016 05:01 AM, Jim Nasby wrote: On 10/30/16 1:32 PM, Tomas Vondra wrote: Now, maybe this has nothing to do with PostgreSQL itself, but maybe it's some sort of CPU / OS scheduling artifact. For example, the system has 36 physical cores, 72

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On 10/31/2016 08:43 PM, Amit Kapila wrote: On Mon, Oct 31, 2016 at 7:58 PM, Tomas Vondra wrote: On 10/31/2016 02:51 PM, Amit Kapila wrote: And moreover, this setup (single device for the whole cluster) is very common, we can't just neglect it. But my main point

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Mon, Oct 31, 2016 at 7:58 PM, Tomas Vondra wrote: > On 10/31/2016 02:51 PM, Amit Kapila wrote: > And moreover, this setup (single device for the whole cluster) is very > common, we can't just neglect it. > > But my main point here really is that the trade-off in

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On 10/31/2016 02:51 PM, Amit Kapila wrote: On Mon, Oct 31, 2016 at 12:02 AM, Tomas Vondra wrote: Hi, On 10/27/2016 01:44 PM, Amit Kapila wrote: I've read that analysis, but I'm not sure I see how it explains the "zig zag" behavior. I do understand that shifting

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Mon, Oct 31, 2016 at 7:02 PM, Tomas Vondra wrote: > > The remaining benchmark with 512 clog buffers completed, and the impact > roughly matches Dilip's benchmark - that is, increasing the number of clog > buffers eliminates all positive impact of the patches

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Mon, Oct 31, 2016 at 12:02 AM, Tomas Vondra wrote: > Hi, > > On 10/27/2016 01:44 PM, Amit Kapila wrote: > > I've read that analysis, but I'm not sure I see how it explains the "zig > zag" behavior. I do understand that shifting the contention to some other >

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On 10/30/2016 07:32 PM, Tomas Vondra wrote: Hi, On 10/27/2016 01:44 PM, Amit Kapila wrote: On Thu, Oct 27, 2016 at 4:15 AM, Tomas Vondra wrote: FWIW I plan to run the same test with logged tables - if it shows similar regression, I'll be much more worried,

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On 10/31/2016 05:01 AM, Jim Nasby wrote: On 10/30/16 1:32 PM, Tomas Vondra wrote: Now, maybe this has nothing to do with PostgreSQL itself, but maybe it's some sort of CPU / OS scheduling artifact. For example, the system has 36 physical cores, 72 virtual ones (thanks to HT). I find it strange

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On 10/30/16 1:32 PM, Tomas Vondra wrote: Now, maybe this has nothing to do with PostgreSQL itself, but maybe it's some sort of CPU / OS scheduling artifact. For example, the system has 36 physical cores, 72 virtual ones (thanks to HT). I find it strange that the "good" client counts are always

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

Hi, On 10/27/2016 01:44 PM, Amit Kapila wrote: On Thu, Oct 27, 2016 at 4:15 AM, Tomas Vondra wrote: FWIW I plan to run the same test with logged tables - if it shows similar regression, I'll be much more worried, because that's a fairly typical scenario (logged

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Thu, Oct 27, 2016 at 5:14 PM, Amit Kapila wrote: >>> Thanks Tomas and Dilip for doing detailed performance tests for this >>> patch. I would like to summarise the performance testing results. >>> >>> 1. With update intensive workload, we are seeing gains from 23%~192%

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Thu, Oct 27, 2016 at 4:15 AM, Tomas Vondra wrote: > On 10/25/2016 06:10 AM, Amit Kapila wrote: >> >> On Mon, Oct 24, 2016 at 2:48 PM, Dilip Kumar >> wrote: >>> >>> On Fri, Oct 21, 2016 at 7:57 AM, Dilip Kumar >>>

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On 10/25/2016 06:10 AM, Amit Kapila wrote: On Mon, Oct 24, 2016 at 2:48 PM, Dilip Kumar wrote: On Fri, Oct 21, 2016 at 7:57 AM, Dilip Kumar wrote: On Thu, Oct 20, 2016 at 9:03 PM, Tomas Vondra wrote: In the

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Mon, Oct 24, 2016 at 2:48 PM, Dilip Kumar wrote: > On Fri, Oct 21, 2016 at 7:57 AM, Dilip Kumar wrote: >> On Thu, Oct 20, 2016 at 9:03 PM, Tomas Vondra >> wrote: >> >>> In the results you've posted on 10/12, you've

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Fri, Oct 21, 2016 at 7:57 AM, Dilip Kumar wrote: > On Thu, Oct 20, 2016 at 9:03 PM, Tomas Vondra > wrote: > >> In the results you've posted on 10/12, you've mentioned a regression with 32 >> clients, where you got 52k tps on master but only

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Fri, Oct 21, 2016 at 1:07 PM, Tomas Vondra wrote: > On 10/21/2016 08:13 AM, Amit Kapila wrote: >> >> On Fri, Oct 21, 2016 at 6:31 AM, Robert Haas >> wrote: >>> >>> On Thu, Oct 20, 2016 at 4:04 PM, Tomas Vondra >>>

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On 10/21/2016 08:13 AM, Amit Kapila wrote: On Fri, Oct 21, 2016 at 6:31 AM, Robert Haas wrote: On Thu, Oct 20, 2016 at 4:04 PM, Tomas Vondra wrote: I then started a run at 96 clients which I accidentally killed shortly before it was

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Fri, Oct 21, 2016 at 6:31 AM, Robert Haas wrote: > On Thu, Oct 20, 2016 at 4:04 PM, Tomas Vondra > wrote: >>> I then started a run at 96 clients which I accidentally killed shortly >>> before it was scheduled to finish, but the results are

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Thu, Oct 20, 2016 at 9:15 PM, Robert Haas wrote: > So here's my theory. The whole reason why Tomas is having difficulty > seeing any big effect from these patches is because he's testing on > x86. When Dilip tests on x86, he doesn't see a big effect either, >

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Thu, Oct 20, 2016 at 9:03 PM, Tomas Vondra wrote: > In the results you've posted on 10/12, you've mentioned a regression with 32 > clients, where you got 52k tps on master but only 48k tps with the patch (so > ~10% difference). I have no idea what scale was used

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Thu, Oct 20, 2016 at 4:04 PM, Tomas Vondra wrote: >> I then started a run at 96 clients which I accidentally killed shortly >> before it was scheduled to finish, but the results are not much >> different; there is no hint of the runaway CLogControlLock contention

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On 10/20/2016 07:59 PM, Robert Haas wrote: On Thu, Oct 20, 2016 at 11:45 AM, Robert Haas wrote: On Thu, Oct 20, 2016 at 3:36 AM, Dilip Kumar wrote: On Thu, Oct 13, 2016 at 12:25 AM, Robert Haas wrote: >> ... So here's

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Thu, Oct 20, 2016 at 11:45 AM, Robert Haas wrote: > On Thu, Oct 20, 2016 at 3:36 AM, Dilip Kumar wrote: >> On Thu, Oct 13, 2016 at 12:25 AM, Robert Haas wrote: >>> I agree with these conclusions. I had a chance to talk

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Thu, Oct 20, 2016 at 3:36 AM, Dilip Kumar wrote: > On Thu, Oct 13, 2016 at 12:25 AM, Robert Haas wrote: >> I agree with these conclusions. I had a chance to talk with Andres >> this morning at Postgres Vision and based on that conversation I'd >>

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On 10/20/2016 09:36 AM, Dilip Kumar wrote: On Thu, Oct 13, 2016 at 12:25 AM, Robert Haas wrote: I agree with these conclusions. I had a chance to talk with Andres this morning at Postgres Vision and based on that conversation I'd like to suggest a couple of additional

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Thu, Oct 13, 2016 at 12:25 AM, Robert Haas wrote: > I agree with these conclusions. I had a chance to talk with Andres > this morning at Postgres Vision and based on that conversation I'd > like to suggest a couple of additional tests: > > 1. Repeat this test on x86.

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Thu, Oct 13, 2016 at 7:53 AM, Tomas Vondra wrote: > On 10/12/2016 08:55 PM, Robert Haas wrote: >> On Wed, Oct 12, 2016 at 3:21 AM, Dilip Kumar wrote: >>> I think at higher client count from client count 96 onwards contention >>> on

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On 10/12/2016 08:55 PM, Robert Haas wrote: > On Wed, Oct 12, 2016 at 3:21 AM, Dilip Kumar wrote: >> I think at higher client count from client count 96 onwards contention >> on CLogControlLock is clearly visible and which is completely solved >> with group lock patch. >> >>

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Wed, Oct 12, 2016 at 3:21 AM, Dilip Kumar wrote: > I think at higher client count from client count 96 onwards contention > on CLogControlLock is clearly visible and which is completely solved > with group lock patch. > > And at lower client count 32,64 contention on

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Mon, Oct 10, 2016 at 2:17 AM, Tomas Vondra wrote: > after testing each combination (every ~9 hours). Inspired by Robert's wait > event post a few days ago, I've added wait event sampling so that we can > perform similar analysis. (Neat idea!) I have done wait

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On 10/08/2016 07:47 AM, Amit Kapila wrote: On Fri, Oct 7, 2016 at 3:02 PM, Tomas Vondra wrote: > > ... > In total, I plan to test combinations of: (a) Dilip's workload and pgbench (regular and -N) (b) logged and unlogged tables (c) scale 300 and scale 3000 (both

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Fri, Oct 7, 2016 at 3:02 PM, Tomas Vondra wrote: > > I got access to a large machine with 72/144 cores (thanks to Oleg and > Alexander from Postgres Professional), and I'm running the tests on that > machine too. > > Results from Dilip's workload (with scale 300,

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On 10/05/2016 10:03 AM, Amit Kapila wrote: On Wed, Oct 5, 2016 at 12:05 PM, Tomas Vondra wrote: Hi, After collecting a lot more results from multiple kernel versions, I can confirm that I see a significant improvement with 128 and 192 clients, roughly by 30%:

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Wed, Oct 5, 2016 at 12:05 PM, Tomas Vondra wrote: > Hi, > > After collecting a lot more results from multiple kernel versions, I can > confirm that I see a significant improvement with 128 and 192 clients, > roughly by 30%: > >64

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

Hi, After collecting a lot more results from multiple kernel versions, I can confirm that I see a significant improvement with 128 and 192 clients, roughly by 30%: 64128192 master

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Thu, Sep 29, 2016 at 8:05 PM, Robert Haas wrote: > OK, another theory: Dilip is, I believe, reinitializing for each run, > and you are not. Yes, I am reinitializing for each run. -- Regards, Dilip Kumar EnterpriseDB: http://www.enterprisedb.com -- Sent via

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Thu, Sep 29, 2016 at 10:14 AM, Tomas Vondra wrote: >> It's not impossible that the longer runs could matter - performance >> isn't necessarily stable across time during a pgbench test, and the >> longer the run the more CLOG pages it will fill. > > Sure, but I'm

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On 09/29/2016 03:47 PM, Robert Haas wrote: On Wed, Sep 28, 2016 at 9:10 PM, Tomas Vondra wrote: I feel like we must be missing something here. If Dilip is seeing huge speedups and you're seeing nothing, something is different, and we don't know what it is. Even

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Wed, Sep 28, 2016 at 9:10 PM, Tomas Vondra wrote: >> I feel like we must be missing something here. If Dilip is seeing >> huge speedups and you're seeing nothing, something is different, and >> we don't know what it is. Even if the test case is artificial, it >>

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Thu, Sep 29, 2016 at 12:56 PM, Dilip Kumar wrote: > On Thu, Sep 29, 2016 at 6:40 AM, Tomas Vondra > wrote: >> Yes, definitely - we're missing something important, I think. One difference >> is that Dilip is using longer runs, but I don't

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Thu, Sep 29, 2016 at 6:40 AM, Tomas Vondra wrote: > Yes, definitely - we're missing something important, I think. One difference > is that Dilip is using longer runs, but I don't think that's a problem (as I > demonstrated how stable the results are). > > I wonder

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On 09/29/2016 01:59 AM, Robert Haas wrote: On Wed, Sep 28, 2016 at 6:45 PM, Tomas Vondra wrote: So, is 300 too little? I don't think so, because Dilip saw some benefit from that. Or what scale factor do we think is needed to reproduce the benefit? My machine has

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Wed, Sep 28, 2016 at 6:45 PM, Tomas Vondra wrote: > So, is 300 too little? I don't think so, because Dilip saw some benefit from > that. Or what scale factor do we think is needed to reproduce the benefit? > My machine has 256GB of ram, so I can easily go up to

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On 09/28/2016 05:39 PM, Robert Haas wrote: On Tue, Sep 27, 2016 at 5:15 PM, Tomas Vondra wrote: So, I got the results from 3.10.101 (only the pgbench data), and it looks like this: 3.10.101 1 8 16 32 64128192

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Tue, Sep 27, 2016 at 5:15 PM, Tomas Vondra wrote: > So, I got the results from 3.10.101 (only the pgbench data), and it looks > like this: > > 3.10.101 1 8 16 32 64128192 >

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On 09/26/2016 08:48 PM, Tomas Vondra wrote: On 09/26/2016 07:16 PM, Tomas Vondra wrote: The averages (over the 10 runs, 5 minute each) look like this: 3.2.80 1 8 16 32 64128192

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Wed, Sep 21, 2016 at 8:47 AM, Dilip Kumar wrote: > Summary: > -- > At 32 clients no gain, I think at this workload Clog Lock is not a problem. > At 64 Clients we can see ~10% gain with simple update and ~5% with TPCB. > At 128 Clients we can see > 50% gain. >

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On 09/26/2016 07:16 PM, Tomas Vondra wrote: The averages (over the 10 runs, 5 minute each) look like this: 3.2.80 1 8 16 32 64128192 granular-locking1567 12146 26341 44188

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Fri, Sep 23, 2016 at 9:20 AM, Amit Kapila wrote: > On Fri, Sep 23, 2016 at 6:50 AM, Robert Haas wrote: >> On Thu, Sep 22, 2016 at 7:44 PM, Tomas Vondra >> wrote: >>> I don't dare to suggest rejecting the patch, but

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On 09/24/2016 06:06 AM, Amit Kapila wrote: On Fri, Sep 23, 2016 at 8:22 PM, Tomas Vondra wrote: ... >> So I'm using 16GB shared buffers (so with scale 300 everything fits into shared buffers), min_wal_size=16GB, max_wal_size=128GB, checkpoint timeout 1h etc. So

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Fri, Sep 23, 2016 at 8:22 PM, Tomas Vondra wrote: > On 09/23/2016 03:07 PM, Amit Kapila wrote: >> >> On Fri, Sep 23, 2016 at 6:16 PM, Tomas Vondra >> wrote: >>> >>> On 09/23/2016 01:44 AM, Tomas Vondra wrote: ...

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On 09/23/2016 02:59 PM, Pavan Deolasee wrote: On Fri, Sep 23, 2016 at 6:05 PM, Tomas Vondra > wrote: On 09/23/2016 05:10 AM, Amit Kapila wrote: On Fri, Sep 23, 2016 at 5:14 AM, Tomas Vondra

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On 09/23/2016 03:07 PM, Amit Kapila wrote: On Fri, Sep 23, 2016 at 6:16 PM, Tomas Vondra wrote: On 09/23/2016 01:44 AM, Tomas Vondra wrote: ... The 4.5 kernel clearly changed the results significantly: ... (c) Although it's not visible in the results, 4.5.5

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Fri, Sep 23, 2016 at 6:50 AM, Robert Haas wrote: > On Thu, Sep 22, 2016 at 7:44 PM, Tomas Vondra > wrote: >> I don't dare to suggest rejecting the patch, but I don't see how we could >> commit any of the patches at this point. So perhaps

Re: [HACKERS] Speed up Clog Access by increasing CLOG buffers

On Fri, Sep 23, 2016 at 6:29 PM, Pavan Deolasee wrote: > On Fri, Sep 23, 2016 at 6:05 PM, Tomas Vondra > wrote: >> >> On 09/23/2016 05:10 AM, Amit Kapila wrote: >>> >>> On Fri, Sep 23, 2016 at 5:14 AM, Tomas Vondra >>>

  1   2   3   >