On Thu, Aug 2, 2012 at 8:19 AM, Tom Lane t...@sss.pgh.pa.us wrote:
Marko Kreen mark...@gmail.com writes:
On Wed, Aug 1, 2012 at 6:18 PM, Tom Lane t...@sss.pgh.pa.us wrote:
So I'm working from the first set of patches in your message
20120721194907.ga28...@gmail.com.
Great!
Here's an
Marko Kreen mark...@gmail.com writes:
On Thu, Aug 2, 2012 at 8:19 AM, Tom Lane t...@sss.pgh.pa.us wrote:
In principle I suppose we could hack PQconsumeInput enough that it
didn't damage the row buffer while still meeting its API contract of
clearing the read-ready condition on the socket; but
On Fri, Aug 3, 2012 at 12:01 AM, Tom Lane t...@sss.pgh.pa.us wrote:
Marko Kreen mark...@gmail.com writes:
On Thu, Aug 2, 2012 at 8:19 AM, Tom Lane t...@sss.pgh.pa.us wrote:
In principle I suppose we could hack PQconsumeInput enough that it
didn't damage the row buffer while still meeting its
[ getting back to this now after assorted distractions ]
Marko Kreen mark...@gmail.com writes:
Just to show agreement: both PQgetRowData() and optimized PGresult
do not belong to 9.2.
OK, we're all on board with leaving those for later.
Only open questions are:
* Is there better API than
On Wed, Aug 1, 2012 at 6:18 PM, Tom Lane t...@sss.pgh.pa.us wrote:
Marko Kreen mark...@gmail.com writes:
* Is there better API than PQsetSingleRowMode()? New PQsend*
functions is my alternative.
After thinking it over, I'm really unexcited about adding new versions
of all the PQsend
Marko Kreen mark...@gmail.com writes:
On Wed, Aug 1, 2012 at 6:18 PM, Tom Lane t...@sss.pgh.pa.us wrote:
So I'm working from the first set of patches in your message
20120721194907.ga28...@gmail.com.
Great!
Here's an updated draft patch. I've not looked at the docs yet so
this doesn't
On Mon, Jul 30, 2012 at 10:26 PM, Jan Wieck janwi...@yahoo.com wrote:
On 7/30/2012 10:31 PM, Leon Smith wrote:
This is not necessarily true, on multiple levels. I mean, some of
the programs I write are highly concurrent, and this form of batching
would have almost no risk of stalling the
Hey, this thread was pointed out to me just a few days ago, but I'll start
by saying that I think this thread is on exactly the right track. I don't
like the callback API, and think that PQsetSingleRowMode should be offered
in place of it. But I do have one
On Sat, Jun 16, 2012 at 10:22 AM,
On 7/30/2012 8:11 PM, Leon Smith wrote:
One other possibility, Tom Lane fretted ever so slightly about the use
of malloc/free per row... what about instead of PQsetSingleRowMode, you
have PQsetChunkedRowMode that takes a chunkSize parameter. A chunkSize
= 0 would be equivalent to what we
On Mon, Jul 30, 2012 at 9:59 PM, Jan Wieck janwi...@yahoo.com wrote:
On 7/30/2012 8:11 PM, Leon Smith wrote:
One other possibility, Tom Lane fretted ever so slightly about the use
of malloc/free per row... what about instead of PQsetSingleRowMode, you
have PQsetChunkedRowMode that takes a
On 7/30/2012 10:31 PM, Leon Smith wrote:
This is not necessarily true, on multiple levels. I mean, some of
the programs I write are highly concurrent, and this form of batching
would have almost no risk of stalling the network buffer.And
the possible use case would be when you are
On Mon, Jul 23, 2012 at 5:07 PM, Marko Kreen mark...@gmail.com wrote:
Does PQgetRowData() break the ability to call PQgetvalue() against the
result as well as other functions like PQgetisnull()? If so, I
object.
I don't get what are you objecting to. The PQgetRowData()
is called instead of
Merlin Moncure mmonc...@gmail.com writes:
I'm arguing that *all* data getting must continue to do so through the
result object, and bypassing the result to get at data is breaking the
result abstraction in the libpq api.
That's a fair point, but the single-row mode without PQgetRowData still
On Tue, Jul 24, 2012 at 10:49 AM, Tom Lane t...@sss.pgh.pa.us wrote:
Merlin Moncure mmonc...@gmail.com writes:
I'm arguing that *all* data getting must continue to do so through the
result object, and bypassing the result to get at data is breaking the
result abstraction in the libpq api.
On Tue, Jul 24, 2012 at 6:18 PM, Merlin Moncure mmonc...@gmail.com wrote:
On Mon, Jul 23, 2012 at 5:07 PM, Marko Kreen mark...@gmail.com wrote:
Does PQgetRowData() break the ability to call PQgetvalue() against the
result as well as other functions like PQgetisnull()? If so, I
object.
I
On Tue, Jul 24, 2012 at 11:08 AM, Marko Kreen mark...@gmail.com wrote:
I'm arguing that *all* data getting must continue to do so through the
result object, and bypassing the result to get at data is breaking the
result abstraction in the libpq api. I bet you can still maintain
data access
On Tue, Jul 24, 2012 at 7:08 PM, Marko Kreen mark...@gmail.com wrote:
The PQgetRowData() is unimportant as it just exposes
the rowBuf to user and thats all.
So to get back to the more interesting PQgetRowData() discussion...
Did you notice that it's up to app to decide whether it calls
On Tue, Jul 24, 2012 at 7:25 PM, Merlin Moncure mmonc...@gmail.com wrote:
On Tue, Jul 24, 2012 at 11:08 AM, Marko Kreen mark...@gmail.com wrote:
I'm arguing that *all* data getting must continue to do so through the
result object, and bypassing the result to get at data is breaking the
result
On Tue, Jul 24, 2012 at 11:39 AM, Marko Kreen mark...@gmail.com wrote:
On Tue, Jul 24, 2012 at 7:25 PM, Merlin Moncure mmonc...@gmail.com wrote:
On Tue, Jul 24, 2012 at 11:08 AM, Marko Kreen mark...@gmail.com wrote:
I'm arguing that *all* data getting must continue to do so through the
result
On Tue, Jul 24, 2012 at 7:52 PM, Merlin Moncure mmonc...@gmail.com wrote:
But, the faster rowbuf method is a generally incompatible way of
dealing with data vs current libpq -- this is bad. If it's truly
impossible to get those benefits without bypassing result API that
then I remove my
Merlin Moncure mmonc...@gmail.com writes:
I think the dummy copy of PGresult is plausible (if by that you mean
optimizing PQgetResult when in single row mode). That would be even
better: you'd remove the need for the rowbuf mode.
I haven't spent any time looking at this, but my gut tells me
On Tue, Jul 24, 2012 at 11:57 AM, Marko Kreen mark...@gmail.com wrote:
On Tue, Jul 24, 2012 at 7:52 PM, Merlin Moncure mmonc...@gmail.com wrote:
But, the faster rowbuf method is a generally incompatible way of
dealing with data vs current libpq -- this is bad. If it's truly
impossible to get
On Tue, Jul 24, 2012 at 1:33 PM, Merlin Moncure mmonc...@gmail.com wrote:
The 'source' result (or source data that would be copied into the
destination result) would be stored in the PGconn, right? So, the idea
is that when you set up single row mode the connection generates a
template PGconn
On Tue, Jul 24, 2012 at 1:49 PM, Merlin Moncure mmonc...@gmail.com wrote:
On Tue, Jul 24, 2012 at 1:33 PM, Merlin Moncure mmonc...@gmail.com wrote:
The 'source' result (or source data that would be copied into the
destination result) would be stored in the PGconn, right? So, the idea
is that
Merlin Moncure mmonc...@gmail.com writes:
Either way, it looks like there's plausible paths to optimizing
repeated result fetch without having expose an alternate data-fetching
API and that the proposed implementation doesn't appear to be a wall
in terms of getting there. So I'm firmly in the
Hm, I think it's possible to rig the test to do dummy
copy of pgresult, thus it's possible to see what kind
of speed is possible.. Will try.
I added a new method (-x) to rowdump where it asks for row
with PQgetRowData() and then tries to emulate super-efficient
PGresult copy, then loads data
On Tue, Jul 24, 2012 at 2:37 PM, Tom Lane t...@sss.pgh.pa.us wrote:
Merlin Moncure mmonc...@gmail.com writes:
Either way, it looks like there's plausible paths to optimizing
repeated result fetch without having expose an alternate data-fetching
API and that the proposed implementation doesn't
On Tue, Jul 24, 2012 at 11:34 PM, Merlin Moncure mmonc...@gmail.com wrote:
On Tue, Jul 24, 2012 at 2:37 PM, Tom Lane t...@sss.pgh.pa.us wrote:
In particular I agree that PQsetSingleRowMode is a bit ugly, so I'm
wondering about your thoughts on providing PQgetSingleRowResult instead.
I don't
On Tue, Jul 24, 2012 at 10:37 PM, Tom Lane t...@sss.pgh.pa.us wrote:
Merlin Moncure mmonc...@gmail.com writes:
Either way, it looks like there's plausible paths to optimizing
repeated result fetch without having expose an alternate data-fetching
API and that the proposed implementation doesn't
On Tue, Jul 24, 2012 at 3:52 PM, Marko Kreen mark...@gmail.com wrote:
* Is there better API than PQsetSingleRowMode()? New PQsend*
functions is my alternative.
I would prefer to have PQsetSingleRowMode() over new flavor of PQsend.
To consolidate my argument above: since you're throwing
On Wed, Jul 25, 2012 at 12:35 AM, Merlin Moncure mmonc...@gmail.com wrote:
On Tue, Jul 24, 2012 at 3:52 PM, Marko Kreen mark...@gmail.com wrote:
* Is there better API than PQsetSingleRowMode()? New PQsend*
functions is my alternative.
I would prefer to have PQsetSingleRowMode() over new
On Tue, Jul 24, 2012 at 4:59 PM, Marko Kreen mark...@gmail.com wrote:
So if we give only PQgetResult() in 9.2, I do not see that we
are locked out from any interesting optimizations.
Well, you are locked out of having PQgetResult reuse the conn's result
since that would then introduce
On Tue, Jul 24, 2012 at 5:29 PM, Merlin Moncure mmonc...@gmail.com wrote:
so #2 seems like the lowest common
denominator (it would permanently preclude #3 and would require #4 to
introduce two new functions instead of just one). #1 of course would
bolt on to #2.
oops, got #1 and #2 backwards
On Wed, Jul 25, 2012 at 1:29 AM, Merlin Moncure mmonc...@gmail.com wrote:
On Tue, Jul 24, 2012 at 4:59 PM, Marko Kreen mark...@gmail.com wrote:
So if we give only PQgetResult() in 9.2, I do not see that we
are locked out from any interesting optimizations.
Well, you are locked out of having
On Tuesday, July 24, 2012, Marko Kreen mark...@gmail.com wrote:
On Wed, Jul 25, 2012 at 1:29 AM, Merlin Moncure mmonc...@gmail.com
wrote:
On Tue, Jul 24, 2012 at 4:59 PM, Marko Kreen mark...@gmail.com wrote:
So if we give only PQgetResult() in 9.2, I do not see that we
are locked out from any
Here is a simple test program that takes a SELECT
query, reads it and outputs a COPY-formatted stream
to standard output, to simulate some activity.
It operates on 3 modes, specified by commant-line switches:
-f Load full resultset at once - old way.
-s Single-Row mode using PQgetResult().
On Mon, Jul 23, 2012 at 2:05 PM, Marko Kreen mark...@gmail.com wrote:
Here is a simple test program that takes a SELECT
query, reads it and outputs a COPY-formatted stream
to standard output, to simulate some activity.
It operates on 3 modes, specified by commant-line switches:
-f Load
On Tue, Jul 24, 2012 at 12:27 AM, Merlin Moncure mmonc...@gmail.com wrote:
On Mon, Jul 23, 2012 at 2:05 PM, Marko Kreen mark...@gmail.com wrote:
It seems odd (but maybe ok) that you have to set the single row mode
on the connection only to have the server reset it whenever you call a
send
Here is 2 approaches how to get to state where only PQsetSingleRowMode()
is available. Both on top of REL9_2_STABLE branch.
a) Remove callback hooks, keep rowBuf, implement single-row-mode on
top of that. This was posted before, I just additionally removed
the PQgetRowData() function.
Marko Kreen mark...@gmail.com writes:
Here is 2 approaches how to get to state where only PQsetSingleRowMode()
is available. Both on top of REL9_2_STABLE branch.
a) Remove callback hooks, keep rowBuf, implement single-row-mode on
top of that. This was posted before, I just additionally
On Mon, Jul 16, 2012 at 4:11 AM, Tom Lane t...@sss.pgh.pa.us wrote:
Marko Kreen mark...@gmail.com writes:
Now, looking at the problem with some perspective, the solution
is obvious: when in single-row mode, the PQgetResult() must return
proper PGresult for that single row. And everything else
Marko Kreen mark...@gmail.com writes:
On Mon, Jul 16, 2012 at 4:11 AM, Tom Lane t...@sss.pgh.pa.us wrote:
I'm starting to look at this patch now. I think we could drop the
PQgetRowData() API: it complicates matters for little gain that I can
see. The argument for it was to avoid the cost of
On Mon, Jul 16, 2012 at 4:33 PM, Tom Lane t...@sss.pgh.pa.us wrote:
Marko Kreen mark...@gmail.com writes:
On Mon, Jul 16, 2012 at 4:11 AM, Tom Lane t...@sss.pgh.pa.us wrote:
I'm starting to look at this patch now. I think we could drop the
PQgetRowData() API: it complicates matters for little
Marko Kreen mark...@gmail.com writes:
On Mon, Jul 16, 2012 at 4:33 PM, Tom Lane t...@sss.pgh.pa.us wrote:
Mm. I still think we should drop it, because it's still a dangerous API
that's not necessary for the principal benefit of this feature.
Yes, it is a secondary feature, but it fits the
On Mon, Jul 16, 2012 at 6:47 PM, Tom Lane t...@sss.pgh.pa.us wrote:
Marko Kreen mark...@gmail.com writes:
On Mon, Jul 16, 2012 at 4:33 PM, Tom Lane t...@sss.pgh.pa.us wrote:
Mm. I still think we should drop it, because it's still a dangerous API
that's not necessary for the principal benefit
Marko Kreen mark...@gmail.com writes:
Now, looking at the problem with some perspective, the solution
is obvious: when in single-row mode, the PQgetResult() must return
proper PGresult for that single row. And everything else follows that.
Such API is implemented in attached patch:
I'm
On 16 June 2012 23:09, Tom Lane t...@sss.pgh.pa.us wrote:
Marko Kreen mark...@gmail.com writes:
Now, looking at the problem with some perspective, the solution
is obvious: when in single-row mode, the PQgetResult() must return
proper PGresult for that single row. And everything else follows
On Sat, Jun 16, 2012 at 7:58 PM, Marko Kreen mark...@gmail.com wrote:
So my preference would be to simply remove the callback API
but keep the processing and provide PQgetRowData() instead.
This is implemented in attached patch. It also
converts dblink to use single-row API.
The patch should
On Sun, Jun 17, 2012 at 2:07 PM, Simon Riggs si...@2ndquadrant.com wrote:
I prefer the description of Marko's API than the one we have now.
Adopting one API in 9.2 and another in 9.3 would be fairly bad.
Perhaps we can have both?
I see no reason the keep the (public) callback API around,
On 17 June 2012 19:37, Marko Kreen mark...@gmail.com wrote:
On Sun, Jun 17, 2012 at 2:07 PM, Simon Riggs si...@2ndquadrant.com wrote:
I prefer the description of Marko's API than the one we have now.
Adopting one API in 9.2 and another in 9.3 would be fairly bad.
Perhaps we can have both?
I
Demos:
https://github.com/markokr/libpq-rowproc-demos/blob/master/demo-onerow-sync.c
https://github.com/markokr/libpq-rowproc-demos/blob/master/demo-onerow-async.c
Few clarifications below.
On Fri, Jun 15, 2012 at 9:21 PM, Marko Kreen mark...@gmail.com wrote:
Now, looking at the problem with
Marko Kreen mark...@gmail.com writes:
Now, looking at the problem with some perspective, the solution
is obvious: when in single-row mode, the PQgetResult() must return
proper PGresult for that single row. And everything else follows that.
* PQgetRowData(): can be called instead
On Sat, Jun 16, 2012 at 6:09 PM, Tom Lane t...@sss.pgh.pa.us wrote:
I guess this raises the question of whether we ought to revert the
row-callback patch entirely and support only this approach. IMO
it is (barely) not too late to do that for 9.2, if we want to.
If we don't want to, then this
On Fri, Jun 15, 2012 at 1:21 PM, Marko Kreen mark...@gmail.com wrote:
The row-processor API is now in 9.2, but it solves only the
different-row-storage problem, but not the one-row-at-a-time
problem, as libpq is still in control until all rows are received.
This means libpq cannet still be
54 matches
Mail list logo
