Re: [PERFORM] Having some problems with concurrent COPY commands

2015-10-13 Thread Heikki Linnakangas
On 10/12/2015 11:14 PM, Shaun Thomas wrote: On Mon, Oct 12, 2015 at 1:28 PM, Andres Freund wrote: Any chance you could provide profiles of such a run? This is as simple as I could make it reliably. With one copy running, the thread finishes in about 1 second. With 2,

Re: [PERFORM] Having some problems with concurrent COPY commands

2015-10-13 Thread Shaun Thomas
On Tue, Oct 13, 2015 at 2:32 AM, Heikki Linnakangas wrote: > 80% of the CPU time is spent in the b-tree comparison function. In the logs, my duration per COPY command increases from about 1400ms for one process to about 3800ms when I have four running concurrently. That's really

Re: [PERFORM] Having some problems with concurrent COPY commands

2015-10-13 Thread Andres Freund
On 2015-10-13 07:14:01 -0700, Shaun Thomas wrote: > On Tue, Oct 13, 2015 at 2:32 AM, Heikki Linnakangas wrote: > > 80% of the CPU time is spent in the b-tree comparison function. > > In the logs, my duration per COPY command increases from about 1400ms > for one process to about

Re: [PERFORM] Having some problems with concurrent COPY commands

2015-10-13 Thread Shaun Thomas
On Tue, Oct 13, 2015 at 7:23 AM, Andres Freund wrote: > and send the results. Whelp, I'm an idiot. I can't account for how I did it, but I can only assume I didn't export my ports in the tests properly. I ran everything again and there's a marked difference between 9.3 and

Re: [PERFORM] Having some problems with concurrent COPY commands

2015-10-13 Thread David Rowley
On 14 October 2015 at 08:33, Shaun Thomas wrote: > On Tue, Oct 13, 2015 at 7:23 AM, Andres Freund wrote: > > and send the results. > > Whelp, I'm an idiot. I can't account for how I did it, but I can only > assume I didn't export my ports in the tests

Re: [PERFORM] Having some problems with concurrent COPY commands

2015-10-12 Thread Jeff Janes
On Mon, Oct 12, 2015 at 11:17 AM, Shaun Thomas wrote: > Hi guys, > > I've been doing some design investigation and ran into an interesting snag > I didn't expect to find on 9.4 (and earlier). I wrote a quick python script > to fork multiple simultaneous COPY commands to

Re: [PERFORM] Having some problems with concurrent COPY commands

2015-10-12 Thread Shaun Thomas
On Mon, Oct 12, 2015 at 1:28 PM, Andres Freund wrote: > Any chance > you could provide profiles of such a run? This is as simple as I could make it reliably. With one copy running, the thread finishes in about 1 second. With 2, it's 1.5s each, and with all 4, it's a little

Re: [PERFORM] Having some problems with concurrent COPY commands

2015-10-12 Thread Andres Freund
Hi, On 2015-10-12 13:17:53 -0500, Shaun Thomas wrote: > It would appear I'm running into whatever issue the xloginsert_slots patch > tried to address, but not much discussion exists afterwards. That patch is merged, it's just that the number of slots is hardcoded. You can recompile postgres with