Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On Mon, Oct 2, 2017 at 8:31 PM, Daniel Gustafsson wrote: > > On 13 Sep 2017, at 07:44, Vaishnavi Prabakaran < > vaishnaviprabaka...@gmail.com> wrote: > > > > On Wed, Sep 13, 2017 at 3:33 PM, Craig Ringer > wrote: > > > > I

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

> On 13 Sep 2017, at 07:44, Vaishnavi Prabakaran > wrote: > > On Wed, Sep 13, 2017 at 3:33 PM, Craig Ringer > wrote: > > I really do not like calling it "commit" as that conflates with a database > commit.

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On 13 September 2017 at 13:44, Vaishnavi Prabakaran < vaishnaviprabaka...@gmail.com> wrote: > Thanks for explaining. Will change this too in next version. > > Thankyou, a lot, for picking up this patch. -- Craig Ringer http://www.2ndQuadrant.com/ PostgreSQL Development,

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On Wed, Sep 13, 2017 at 3:33 PM, Craig Ringer wrote: > > I really do not like calling it "commit" as that conflates with a database > commit. > > A batch can embed multiple BEGINs and COMMITs. It's entirely possible for > an earlier part of the batch to succeed and commit,

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On 13 September 2017 at 13:06, Vaishnavi Prabakaran < vaishnaviprabaka...@gmail.com> wrote: > > > On Wed, Aug 23, 2017 at 7:40 PM, Andres Freund wrote: > >> >> >> >> > Am failing to see the benefit in allowing user to set >> > PQBatchAutoFlush(true|false) property? Is it

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On Wed, Aug 23, 2017 at 7:40 PM, Andres Freund wrote: > > > > > Am failing to see the benefit in allowing user to set > > PQBatchAutoFlush(true|false) property? Is it really needed? > > I'm inclined not to introduce that for now. If somebody comes up with a > convincing

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

Hi, On 2017-08-10 15:23:06 +1000, Vaishnavi Prabakaran wrote: > Andres Freund wrote : > > If you were to send a gigabyte of queries, it'd buffer them all up in > memory... So some > >more intelligence is going to be needed. > > Firstly, sorry for the delayed response as I got busy with other >

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

Andres Freund wrote : > If you were to send a gigabyte of queries, it'd buffer them all up in memory... So some >more intelligence is going to be needed. Firstly, sorry for the delayed response as I got busy with other activities. To buffer up the queries before flushing them to the socket, I

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

Andres Freund wrote: > > One option may be to leave that decision to the user by providing a > > PQBatchAutoFlush(true|false) property, along with a PQBatchFlush() > > function. > > What'd be the difference between PQflush() and PQbatchFlush()? I guess no difference, I was just not

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

Hi, On 2017-06-22 13:43:35 +0200, Daniel Verite wrote: > But OTOH there are certainly batch workloads where it will be preferrable > for the first query to reach the server ASAP, rather than waiting to be > coalesced with the next ones. Is that really something people expect from a batch API? I

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

Andres Freund wrote: - if (pqFlush(conn) < 0) - goto sendFailed; + if (conn->batch_status == PQBATCH_MODE_OFF) + { + /* +* Give the data a push. In nonblock mode, don't complain if we're unable +* to send it

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

Craig Ringer wrote: > The kernel will usually do some packet aggregation unless we use > TCP_NODELAY (which we don't and shouldn't) Not sure. As a point of comparison, Oracle has it as a tunable parameter (TCP.NODELAY), and they changed its default from No to Yes starting from their 10g

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On 2017-06-21 18:07:21 -0700, Andres Freund wrote: > On 2017-06-22 09:03:05 +0800, Craig Ringer wrote: > > On 22 June 2017 at 08:29, Andres Freund wrote: > > > > > I.e. we're doing tiny write send() syscalls (they should be coalesced) > > > > That's likely worth doing, but

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On 22 June 2017 at 09:07, Andres Freund wrote: > On 2017-06-22 09:03:05 +0800, Craig Ringer wrote: >> On 22 June 2017 at 08:29, Andres Freund wrote: >> >> > I.e. we're doing tiny write send() syscalls (they should be coalesced) >> >> That's likely worth

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On 2017-06-22 09:03:05 +0800, Craig Ringer wrote: > On 22 June 2017 at 08:29, Andres Freund wrote: > > > I.e. we're doing tiny write send() syscalls (they should be coalesced) > > That's likely worth doing, but can probably wait for a separate patch. I don't think so, we

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On 22 June 2017 at 08:29, Andres Freund wrote: > I.e. we're doing tiny write send() syscalls (they should be coalesced) That's likely worth doing, but can probably wait for a separate patch. The kernel will usually do some packet aggregation unless we use TCP_NODELAY (which

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On 2017-06-21 16:40:48 -0700, Andres Freund wrote: > On 2017-06-20 17:51:23 +0200, Daniel Verite wrote: > > Andres Freund wrote: > > > > > FWIW, I still think this needs a pgbench or similar example integration, > > > so we can actually properly measure the benefits. > > > > Here's an

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On 22 Jun. 2017 07:40, "Andres Freund" wrote: On 2017-06-20 17:51:23 +0200, Daniel Verite wrote: > Andres Freund wrote: > > > FWIW, I still think this needs a pgbench or similar example integration, > > so we can actually properly measure the benefits. > > Here's an

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On 2017-06-20 17:51:23 +0200, Daniel Verite wrote: > Andres Freund wrote: > > > FWIW, I still think this needs a pgbench or similar example integration, > > so we can actually properly measure the benefits. > > Here's an updated version of the patch I made during review, > adding

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

Andres Freund wrote: > FWIW, I still think this needs a pgbench or similar example integration, > so we can actually properly measure the benefits. Here's an updated version of the patch I made during review, adding \beginbatch and \endbatch to pgbench. The performance improvement

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On Tue, Jun 20, 2017 at 10:43 AM, Craig Ringer wrote: > Especially with a 6-week-old baby now Congratulations! -- Michael -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription:

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On 20 June 2017 at 06:49, Andres Freund wrote: > On 2017-04-05 15:45:26 -0700, Andres Freund wrote: >> Hi, >> >> On 2017-04-05 17:00:42 +1000, Vaishnavi Prabakaran wrote: >> > Regarding test patch, I have corrected the test suite after David Steele's >> > comments. >> > Also,

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On Tue, Jun 20, 2017 at 8:49 AM, Andres Freund wrote: > On 2017-04-05 15:45:26 -0700, Andres Freund wrote: > > Hi, > > > > On 2017-04-05 17:00:42 +1000, Vaishnavi Prabakaran wrote: > > > Regarding test patch, I have corrected the test suite after David > Steele's > > >

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On 2017-04-05 15:45:26 -0700, Andres Freund wrote: > Hi, > > On 2017-04-05 17:00:42 +1000, Vaishnavi Prabakaran wrote: > > Regarding test patch, I have corrected the test suite after David Steele's > > comments. > > Also, I would like to mention that a companion patch was submitted by David > >

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

Hi, On 2017-04-05 17:00:42 +1000, Vaishnavi Prabakaran wrote: > Regarding test patch, I have corrected the test suite after David Steele's > comments. > Also, I would like to mention that a companion patch was submitted by David > Steele up-thread. > > Attached the latest code and test patch.

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On 2017-04-04 08:57:33 +0900, Michael Paquier wrote: > On Tue, Apr 4, 2017 at 8:26 AM, Andres Freund wrote: > > On 2017-04-04 09:24:23 +1000, Vaishnavi Prabakaran wrote: > >> Just quickly, Is it not ok to consider only the code patch for this CF > >> without test patch? > > >

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On Tue, Apr 4, 2017 at 8:26 AM, Andres Freund wrote: > On 2017-04-04 09:24:23 +1000, Vaishnavi Prabakaran wrote: >> Just quickly, Is it not ok to consider only the code patch for this CF >> without test patch? > > I'd say no, it's not acceptable. This is too much new code for

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On 2017-04-04 09:24:23 +1000, Vaishnavi Prabakaran wrote: > Hi, > On Tue, Apr 4, 2017 at 7:05 AM, Andres Freund wrote: > > > On 2017-04-03 14:10:47 +1000, Vaishnavi Prabakaran wrote: > > > > The CF has been extended until April 7 but time is still growing short. > > > >

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

Hi, On Tue, Apr 4, 2017 at 7:05 AM, Andres Freund wrote: > On 2017-04-03 14:10:47 +1000, Vaishnavi Prabakaran wrote: > > > The CF has been extended until April 7 but time is still growing short. > > > Please respond with a new patch by 2017-04-04 00:00 AoE (UTC-12) or > this

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On 2017-04-03 14:10:47 +1000, Vaishnavi Prabakaran wrote: > > The CF has been extended until April 7 but time is still growing short. > > Please respond with a new patch by 2017-04-04 00:00 AoE (UTC-12) or this > > submission will be marked "Returned with Feedback". > > > > > Thanks for the

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On Sat, Apr 1, 2017 at 2:03 AM, David Steele wrote: > Hi, > > On 3/30/17 2:12 PM, Daniel Verite wrote: > >> Vaishnavi Prabakaran wrote: >> >> Hmm, With batch mode, after sending COPY command to server(and server >>> started processing the query and goes into COPY

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

Hi, On 3/30/17 2:12 PM, Daniel Verite wrote: Vaishnavi Prabakaran wrote: Hmm, With batch mode, after sending COPY command to server(and server started processing the query and goes into COPY state) , client does not immediately read the result , but it keeps sending other queries to

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

Vaishnavi Prabakaran wrote: > Hmm, With batch mode, after sending COPY command to server(and server > started processing the query and goes into COPY state) , client does not > immediately read the result , but it keeps sending other queries to the > server. By this time, server already

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On Thu, Mar 30, 2017 at 12:08 PM, Michael Paquier wrote: > On Wed, Mar 29, 2017 at 12:40 PM, Vaishnavi Prabakaran > wrote: > > Michael Paquier wrote: > >>Could you as well update src/tools/msvc/vcregress.pl, aka the routine >

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On Wed, Mar 29, 2017 at 12:40 PM, Vaishnavi Prabakaran wrote: > Michael Paquier wrote: >>Could you as well update src/tools/msvc/vcregress.pl, aka the routine >>modulescheck so as this new test is skipped. I am sure that nobody >>will scream if this test is not run

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

Michael Paquier wrote: > # Running: testlibpqbatch dbname=postgres 1 copyfailure > dispatching SELECT query failed: cannot queue commands during COPY > > COPYBUF: 5 > > Error status and message got from server due to COPY command failure > are : PGRES_FATAL_ERROR ERROR: unexpected

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On Tue, Mar 28, 2017 at 1:57 PM, Vaishnavi Prabakaran wrote: > Thanks Craig and Michael for confirming that "PQsetSingleRowMode" should be > called right after "PQbatchQueueProcess". > > Michael Paquier wrote: > >> It seems to me that >>this should also be effective

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

Thanks Craig and Michael for confirming that "PQsetSingleRowMode" should be called right after "PQbatchQueueProcess". Michael Paquier wrote: > It seems to me that >this should also be effective only during the fetching of one single >result set. When the client moves on to the next item in the

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

Vaishnavi Prabakaran wrote: > Am going to include the test which you shared in the test patch. Please let > me know if you want to cover anymore specific cases to gain confidence. One bit of functionality that does not work in batch mode and is left as is by the patch is the PQfn() fast

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

Vaishnavi Prabakaran wrote: > Please let me know if you think this is not enough but wanted to update > section 33.6 also? Yes, if the right place to call PQsetSingleRowMode() is immediately after PQbatchQueueProcess(), I think we need to update "33.6. Retrieving Query Results

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On Mon, Mar 27, 2017 at 4:42 PM, Craig Ringer wrote: > On 27 March 2017 at 15:26, Michael Paquier wrote: >> On Sat, Mar 25, 2017 at 9:50 PM, Craig Ringer wrote: >>> I'm fairly confident that I overlooked single row mode

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On 27 March 2017 at 15:26, Michael Paquier wrote: > On Sat, Mar 25, 2017 at 9:50 PM, Craig Ringer wrote: >> I'm fairly confident that I overlooked single row mode entirely in the >> original patch, though it's long enough ago that it's hard for

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On Sat, Mar 25, 2017 at 9:50 PM, Craig Ringer wrote: > I'm fairly confident that I overlooked single row mode entirely in the > original patch, though it's long enough ago that it's hard for me to > remember exactly. > > I don't really have much of an opinion on the best

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On 24 March 2017 at 23:21, David Steele wrote: > Hi Vaishnavi, > > On 3/19/17 9:32 PM, Vaishnavi Prabakaran wrote: >> >> On Fri, Mar 17, 2017 at 12:37 AM, Daniel Verite > >> Please let me know if you think this is not enough but wanted to update >>

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

Hi Vaishnavi, On 3/19/17 9:32 PM, Vaishnavi Prabakaran wrote: On Fri, Mar 17, 2017 at 12:37 AM, Daniel Verite I would also like to hear Craig's

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On Fri, Mar 17, 2017 at 12:37 AM, Daniel Verite wrote: > Vaishnavi Prabakaran wrote: > > > So, attached the alternative fix for this issue. > > Please share me your thoughts. > > I assume you prefer the alternative fix because it's simpler. > I would like add

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

Vaishnavi Prabakaran wrote: > So, attached the alternative fix for this issue. > Please share me your thoughts. I assume you prefer the alternative fix because it's simpler. > I would also like to hear Craig's opinion on it before applying this fix > to the original patch, just to make

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On Tue, Mar 14, 2017 at 5:50 PM, Vaishnavi Prabakaran < vaishnaviprabaka...@gmail.com> wrote: > > > On Tue, Mar 14, 2017 at 4:19 AM, Daniel Verite > wrote: > >> >> I mean the next iteration of the above while statement. Referring >> to the doc, that would be the "next

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On Tue, Mar 14, 2017 at 4:19 AM, Daniel Verite wrote: > > I mean the next iteration of the above while statement. Referring > to the doc, that would be the "next batch entry": > > " To get the result of the first batch entry the client must call >

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

Vaishnavi Prabakaran wrote: > > while (QbatchQueueProcess(conn)) { > >r = PQsetSingleRowMode(conn); > >if (r!=1) { > > fprintf(stderr, "PQsetSingleRowMode() failed"); > >} > >.. > Thanks for investigating the problem, and could you kindly

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On 13 March 2017 at 08:54, Vaishnavi Prabakaran wrote: > Before going with this fix, I would like you to consider the option of > asking batch processing users(via documentation) to set single-row mode > before calling PQgetResult(). > Either way we need to fix the

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On Sat, Mar 11, 2017 at 12:52 AM, Daniel Verite wrote: > Hi, > > I notice that PQsetSingleRowMode() doesn't work when getting batch results. > > The function is documented as: > " int PQsetSingleRowMode(PGconn *conn); > > This function can only be called immediately

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

Hi, I notice that PQsetSingleRowMode() doesn't work when getting batch results. The function is documented as: " int PQsetSingleRowMode(PGconn *conn); This function can only be called immediately after PQsendQuery or one of its sibling functions, before any other operation on the

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

Vaishnavi Prabakaran wrote: >if (PQbatchStatus(st->con) != PQBATCH_MODE_ON) > >But, it is better to use if (PQbatchStatus(st->con) == > PQBATCH_MODE_OFF) for this verification. Reason is there is one more state > in batch mode - PQBATCH_MODE_ABORTED. So, if the batch mode is in

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On Wed, Mar 8, 2017 at 3:52 AM, Daniel Verite wrote: > Vaishnavi Prabakaran wrote: > > > Yes, I have created a new patch entry into the commitfest 2017-03 and > > attached the latest patch with this e-mail. > > Please find attached a companion patch implementing

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On 8 March 2017 at 00:52, Daniel Verite wrote: > Vaishnavi Prabakaran wrote: > >> Yes, I have created a new patch entry into the commitfest 2017-03 and >> attached the latest patch with this e-mail. > > Please find attached a companion patch implementing the batch

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

Vaishnavi Prabakaran wrote: > Yes, I have created a new patch entry into the commitfest 2017-03 and > attached the latest patch with this e-mail. Please find attached a companion patch implementing the batch API in pgbench, exposed as \beginbatch and \endbatch meta-commands (without

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On 22 Feb. 2017 14:14, "Vaishnavi Prabakaran" wrote: Thanks for reviewing the patch. > Thanks for picking it up! I've wanted to see this process for some time, but just haven't had the bandwidth for it.

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On Fri, Feb 17, 2017 at 2:17 PM, Prabakaran, Vaishnavi wrote: > On 22 November 2016 at 18:32, Craig Ringer wrote: > I am interested in this patch and addressed various below comments from > reviewers. And, I have separated out code and test

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On 22 November 2016 at 18:32, Craig Ringer wrote: > On 22 November 2016 at 15:14, Haribabu Kommi > wrote: > > > > On Fri, Nov 18, 2016 at 7:18 PM, Craig Ringer > wrote: > >> > >> The latest is what's attached upthread and

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

Hi, On 18 November 2016 at 08:18, Craig Ringer wrote: >At this point I doubt I'll be able to >get update it again in time for v10, so anyone who wants to adopt it >is welcome. I am interested in pipeline/batch support for ECPG, and found this thread. I updated Craig's patch [1] to apply

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On 22 November 2016 at 15:14, Haribabu Kommi wrote: > > On Fri, Nov 18, 2016 at 7:18 PM, Craig Ringer wrote: >> >> The latest is what's attached upthread and what's in the git repo also >> referenced upthread. >> >> I haven't been able to update

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On Fri, Nov 18, 2016 at 7:18 PM, Craig Ringer wrote: > The latest is what's attached upthread and what's in the git repo also > referenced upthread. > > I haven't been able to update in response to more recent review due to > other development commitments. At this point I

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On 18 November 2016 at 14:04, Tsunakawa, Takayuki wrote: > Hello, Craig, > > I'm sorry to be late to review your patch. I've just been able to read the > HTML doc first. Can I get the latest .patch file for reading and running the > code? The latest is what's

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

Hello, Craig, I'm sorry to be late to review your patch. I've just been able to read the HTML doc first. Can I get the latest .patch file for reading and running the code? Here are some comments and questions. I tried to avoid the same point as other reviewers, but there may be an overlap.

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On 14 October 2016 at 22:15, Shay Rojansky wrote: > Unless I'm mistaken TCP_CORK is not necessarily going to work across all > platforms (e.g. Windows), although SO_LINGER (which is more standard) also > helps here. Yeah, true. You can also twiddle TCP_NODELAY on and off on

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

> > > Of course, this is a > > relatively minor performance issue (especially when compared to the > overall > > performance benefits provided by batching), and providing an API > distinction > > between adding a Sync and flushing the buffer may over-complicate the > API. I > > just thought I'd

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On 14 October 2016 at 18:09, Shay Rojansky wrote: >> > It has recently come to my attention that this implementation is >> > problematic >> > because it forces the batch to occur within a transaction; in other >> > words, >> > there's no option for a non-transactional batch. >> >>

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

> > > It has recently come to my attention that this implementation is > problematic > > because it forces the batch to occur within a transaction; in other > words, > > there's no option for a non-transactional batch. > > That's not strictly the case. If you explicitly BEGIN and COMMIT, > those

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On 12 October 2016 at 19:51, Shay Rojansky wrote: > Hi all. I thought I'd share some experience from Npgsql regarding > batching/pipelining - hope this isn't off-topic. Not at all. > Npgsql has supported batching for quite a while, similar to what this patch > proposes - with a

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On 10/4/16 11:54 PM, Michael Paquier wrote: + +Much like asynchronous query mode, there is no performance disadvantage to +using batching and pipelining. It somewhat increased client application +complexity and extra caution is required to prevent client/server network +

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

Hi all. I thought I'd share some experience from Npgsql regarding batching/pipelining - hope this isn't off-topic. Npgsql has supported batching for quite a while, similar to what this patch proposes - with a single Sync message is sent at the end. It has recently come to my attention that this

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On Mon, Oct 3, 2016 at 12:48 PM, Craig Ringer wrote: > On 3 October 2016 at 10:10, Michael Paquier wrote: >> On Tue, Sep 6, 2016 at 8:01 PM, Craig Ringer wrote: >>> On 6 September 2016 at 16:10, Daniel Verite

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On 04/10/16 20:15, Michael Paquier wrote: On Mon, Oct 3, 2016 at 11:52 PM, Daniel Verite wrote: Wouldn't pgbench benefit from it? It was mentioned some time ago [1], in relationship to the \into construct, how client-server latency was important enough to justify the

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On 4 Oct. 2016 15:15, "Michael Paquier" wrote: > > On Mon, Oct 3, 2016 at 11:52 PM, Daniel Verite wrote: > > Wouldn't pgbench benefit from it? > > It was mentioned some time ago [1], in relationship to the > > \into construct, how client-server

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On Mon, Oct 3, 2016 at 11:52 PM, Daniel Verite wrote: > Wouldn't pgbench benefit from it? > It was mentioned some time ago [1], in relationship to the > \into construct, how client-server latency was important enough to > justify the use of a "\;" separator between

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

Craig Ringer wrote: > I think it's mostly of interest to app authors and driver developers > and that's what it's aimed at. pg_restore could benefit a lot too. Wouldn't pgbench benefit from it? It was mentioned some time ago [1], in relationship to the \into construct, how client-server

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On 3 October 2016 at 10:10, Michael Paquier wrote: > On Tue, Sep 6, 2016 at 8:01 PM, Craig Ringer wrote: >> On 6 September 2016 at 16:10, Daniel Verite wrote: >>> Craig Ringer wrote: >>> Updated patch

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On Tue, Sep 6, 2016 at 8:01 PM, Craig Ringer wrote: > On 6 September 2016 at 16:10, Daniel Verite wrote: >> Craig Ringer wrote: >> >>> Updated patch attached. >> >> Please find attached a couple fixes for typos I've came across in >> the

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On 6 September 2016 at 16:10, Daniel Verite wrote: > Craig Ringer wrote: > >> Updated patch attached. > > Please find attached a couple fixes for typos I've came across in > the doc part. Thanks, will apply and post a rebased patch soon, or if someone picks this

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

Craig Ringer wrote: > Updated patch attached. Please find attached a couple fixes for typos I've came across in the doc part. Also it appears that PQqueriesInBatch() doesn't work as documented. It says: "Returns the number of queries still in the queue for this batch" but in fact it's

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On 23 August 2016 at 08:27, Craig Ringer wrote: > On 10 August 2016 at 14:44, Michael Paquier > wrote: > >> > I am looking a bit more seriously at this patch and assigned myself as >> a reviewer. >> > > Much appreciated. > > >> > macos

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On 10 August 2016 at 14:44, Michael Paquier wrote: > On Fri, Jun 3, 2016 at 8:51 PM, Dmitry Igrishin wrote: > >> BTW, I've publushed the HTML-ified SGML docs to > >> http://2ndquadrant.github.io/postgres/libpq-batch-mode.html as a > preview. > >

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On Fri, Jun 3, 2016 at 8:51 PM, Dmitry Igrishin wrote: >> BTW, I've publushed the HTML-ified SGML docs to >> http://2ndquadrant.github.io/postgres/libpq-batch-mode.html as a preview. > Typo detected: "Returns 1 if the batch curently being received" -- "curently". I am looking

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

2016-05-24 5:01 GMT+03:00 Craig Ringer : > > On 24 May 2016 at 00:00, Michael Paquier wrote: >> >> On Mon, May 23, 2016 at 8:50 AM, Andres Freund wrote: > > >> >> > yay^2. >> >> I'll follow this mood. Yeha. > > > > BTW, I've

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

From: pgsql-hackers-ow...@postgresql.org [mailto:pgsql-hackers-ow...@postgresql.org] On Behalf Of Craig Ringer I'll follow this mood. Yeha. BTW, I've publushed the HTML-ified SGML docs to http://2ndquadrant.github.io/postgres/libpq-batch-mode.html as a preview. Sorry for my late reply.

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On 5/23/16 4:19 AM, Craig Ringer wrote: +Batching less useful when information from one operation is required by the SB "Batching is less useful". -- Jim Nasby, Data Architect, Blue Treble Consulting, Austin TX Experts in Analytics, Data Architecture and PostgreSQL Data in Trouble? Get it

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

Craig Ringer writes: > On 24 May 2016 at 00:00, Michael Paquier wrote: >> Did you consider the use of simple_list.c instead of introducing a new >> mimic as PGcommandQueueEntry? It would be cool avoiding adding new >> list emulations on

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On 24 May 2016 at 00:00, Michael Paquier wrote: > > Did you consider the use of simple_list.c instead of introducing a new > mimic as PGcommandQueueEntry? It would be cool avoiding adding new > list emulations on frontends. > I'd have to extend simple_list to add a

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On 24 May 2016 at 00:00, Michael Paquier wrote: > On Mon, May 23, 2016 at 8:50 AM, Andres Freund wrote: > > > yay^2. > > I'll follow this mood. Yeha. BTW, I've publushed the HTML-ified SGML docs to

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On 24 May 2016 at 00:00, Michael Paquier wrote: > On Mon, May 23, 2016 at 8:50 AM, Andres Freund > wrote: > >> This should be very useful for optimising FDWs, Postgres-XC, etc. > > > > And optimizing normal clients. > > > > Not easy, but I'd be

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

On Mon, May 23, 2016 at 8:50 AM, Andres Freund wrote: > On 2016-05-23 17:19:09 +0800, Craig Ringer wrote: >> Following on from the foreign table batch inserts thread[1], here's a patch >> to add support for pipelining queries into asynchronous batches in libpq. > > Yay! >> I'm

Re: [HACKERS] PATCH: Batch/pipelining support for libpq

Hi, On 2016-05-23 17:19:09 +0800, Craig Ringer wrote: > Hi all > Following on from the foreign table batch inserts thread[1], here's a patch > to add support for pipelining queries into asynchronous batches in libpq. Yay! > I'm measuring 300x (not %) performance improvements doing batches on >

[HACKERS] PATCH: Batch/pipelining support for libpq

Hi all Following on from the foreign table batch inserts thread[1], here's a patch to add support for pipelining queries into asynchronous batches in libpq. Attached, and also available at https://github.com/2ndQuadrant/postgres/tree/dev/libpq-async-batch (subject to rebasing and force pushes).