One patch that fell off the truck during a turn in the November
CommitFest was "Allow substitute allocators for PGresult". Re-reading
the whole thing again, it actually turned into a rather different
submission in the middle, and I know I didn't follow that shift
correctly then. I'm replying to its thread but have changed the subject
to reflect that change. From a procedural point of view, I don't feel
right kicking this back to its author on a Friday night when the
deadline for resubmitting it would be Sunday. Instead I've refreshed
the patch myself and am adding it to the January CommitFest. The new
patch is a single file; it's easy enough to split out the dblink changes
if someone wants to work with the pieces separately.
After my meta-review I think we should get another reviewer familiar
with using dblink to look at this next. This is fundamentally a
performance patch now. Some results and benchmarking code were
submitted along with it; the other issues are moot if those aren't
reproducible. The secondary goal for a new review here is to provide
another opinion on the naming issues and abstraction concerns raised so far.
To clear out the original line of thinking, this is not a replacement
low-level storage allocator anymore. The idea of using such a mechanism
to help catch memory leaks has also been dropped.
Instead this adds and documents a new path for libpq callers to more
directly receive tuples, for both improved speed and lower memory
usage. dblink has been modified to use this new mechanism.
Benchmarking by the author suggests no significant change in libpq speed
when only that change was made, while the modified dblink using the new
mechanism was significantly faster. It jumped from 332K tuples/sec to
450K, a 35% gain, and had a lower memory footprint too. Test
methodology and those results are at
http://archives.postgresql.org/pgsql-hackers/2011-12/msg00008.php
Robert Haas did a quick code review of this already, it along with
author response mixed in are at
http://archives.postgresql.org/pgsql-hackers/2011-12/msg01149.php I see
two areas of contention there:
-There are several naming bits no one is happy with yet. Robert didn't
like some of them, but neither did Kyotaro. I don't have an opinion
myself. Is it the case that some changes to the existing code's
terminology are what's actually needed to make this all better? Or is
this just fundamentally warty and there's nothing to be done about it.
Dunno.
-There is an abstraction wrapper vs. coding convenience trade-off
centering around PGresAttValue. It sounded to me like it raised always
fun questions like "where's the right place for the line between
lipq-fe.h and libpq-int.h to be?"
dblink is pretty popular, and this is a big performance win for it. If
naming and API boundary issues are the worst problems here, this sounds
like something well worth pursuing as part of 9.2's still advancing
performance theme.
--
Greg Smith 2ndQuadrant US g...@2ndquadrant.com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com
diff --git a/contrib/dblink/dblink.c b/contrib/dblink/dblink.c
index 36a8e3e..f48fe4f 100644
*** a/contrib/dblink/dblink.c
--- b/contrib/dblink/dblink.c
*************** typedef struct remoteConn
*** 63,73 ****
bool newXactForCursor; /* Opened a transaction for a cursor */
} remoteConn;
/*
* Internal declarations
*/
static Datum dblink_record_internal(FunctionCallInfo fcinfo, bool is_async);
- static void materializeResult(FunctionCallInfo fcinfo, PGresult *res);
static remoteConn *getConnectionByName(const char *name);
static HTAB *createConnHash(void);
static void createNewConnection(const char *name, remoteConn *rconn);
--- 63,85 ----
bool newXactForCursor; /* Opened a transaction for a cursor */
} remoteConn;
+ typedef struct storeInfo
+ {
+ Tuplestorestate *tuplestore;
+ int nattrs;
+ AttInMetadata *attinmeta;
+ MemoryContext oldcontext;
+ char *attrvalbuf;
+ void **valbuf;
+ size_t *valbufsize;
+ bool error_occurred;
+ bool nummismatch;
+ } storeInfo;
+
/*
* Internal declarations
*/
static Datum dblink_record_internal(FunctionCallInfo fcinfo, bool is_async);
static remoteConn *getConnectionByName(const char *name);
static HTAB *createConnHash(void);
static void createNewConnection(const char *name, remoteConn *rconn);
*************** static char *escape_param_str(const char
*** 90,95 ****
--- 102,111 ----
static void validate_pkattnums(Relation rel,
int2vector *pkattnums_arg, int32 pknumatts_arg,
int **pkattnums, int *pknumatts);
+ static void initStoreInfo(storeInfo *sinfo, FunctionCallInfo fcinfo);
+ static void finishStoreInfo(storeInfo *sinfo);
+ static void *addTuple(PGresult *res, AddTupFunc func, int id, size_t size);
+
/* Global */
static remoteConn *pconn = NULL;
*************** dblink_fetch(PG_FUNCTION_ARGS)
*** 503,508 ****
--- 519,525 ----
char *curname = NULL;
int howmany = 0;
bool fail = true; /* default to backward compatible */
+ storeInfo storeinfo;
DBLINK_INIT;
*************** dblink_fetch(PG_FUNCTION_ARGS)
*** 559,573 ****
--- 576,605 ----
appendStringInfo(&buf, "FETCH %d FROM %s", howmany, curname);
/*
+ * Result is stored into storeinfo.tuplestore instead of
+ * res->result retuned by PQexec below
+ */
+ initStoreInfo(&storeinfo, fcinfo);
+ PQregisterTupleAdder(conn, addTuple, &storeinfo);
+
+ /*
* Try to execute the query. Note that since libpq uses malloc, the
* PGresult will be long-lived even though we are still in a short-lived
* memory context.
*/
res = PQexec(conn, buf.data);
+ finishStoreInfo(&storeinfo);
+
if (!res ||
(PQresultStatus(res) != PGRES_COMMAND_OK &&
PQresultStatus(res) != PGRES_TUPLES_OK))
{
+ /* This is only for backward compatibility */
+ if (storeinfo.nummismatch)
+ ereport(ERROR,
+ (errcode(ERRCODE_DATATYPE_MISMATCH),
+ errmsg("remote query result rowtype does not match "
+ "the specified FROM clause rowtype")));
dblink_res_error(conname, res, "could not fetch from cursor", fail);
return (Datum) 0;
}
*************** dblink_fetch(PG_FUNCTION_ARGS)
*** 580,586 ****
errmsg("cursor \"%s\" does not exist", curname)));
}
- materializeResult(fcinfo, res);
return (Datum) 0;
}
--- 612,617 ----
*************** dblink_record_internal(FunctionCallInfo
*** 640,645 ****
--- 671,677 ----
remoteConn *rconn = NULL;
bool fail = true; /* default to backward compatible */
bool freeconn = false;
+ storeInfo storeinfo;
/* check to see if caller supports us returning a tuplestore */
if (rsinfo == NULL || !IsA(rsinfo, ReturnSetInfo))
*************** dblink_record_internal(FunctionCallInfo
*** 715,878 ****
rsinfo->setResult = NULL;
rsinfo->setDesc = NULL;
/* synchronous query, or async result retrieval */
if (!is_async)
res = PQexec(conn, sql);
else
- {
res = PQgetResult(conn);
- /* NULL means we're all done with the async results */
- if (!res)
- return (Datum) 0;
- }
! /* if needed, close the connection to the database and cleanup */
! if (freeconn)
! PQfinish(conn);
! if (!res ||
! (PQresultStatus(res) != PGRES_COMMAND_OK &&
! PQresultStatus(res) != PGRES_TUPLES_OK))
{
! dblink_res_error(conname, res, "could not execute query", fail);
! return (Datum) 0;
}
- materializeResult(fcinfo, res);
return (Datum) 0;
}
- /*
- * Materialize the PGresult to return them as the function result.
- * The res will be released in this function.
- */
static void
! materializeResult(FunctionCallInfo fcinfo, PGresult *res)
{
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
! Assert(rsinfo->returnMode == SFRM_Materialize);
! PG_TRY();
{
! TupleDesc tupdesc;
! bool is_sql_cmd = false;
! int ntuples;
! int nfields;
!
! if (PQresultStatus(res) == PGRES_COMMAND_OK)
! {
! is_sql_cmd = true;
! /*
! * need a tuple descriptor representing one TEXT column to return
! * the command status string as our result tuple
! */
! tupdesc = CreateTemplateTupleDesc(1, false);
! TupleDescInitEntry(tupdesc, (AttrNumber) 1, "status",
! TEXTOID, -1, 0);
! ntuples = 1;
! nfields = 1;
! }
! else
! {
! Assert(PQresultStatus(res) == PGRES_TUPLES_OK);
! is_sql_cmd = false;
! /* get a tuple descriptor for our result type */
! switch (get_call_result_type(fcinfo, NULL, &tupdesc))
! {
! case TYPEFUNC_COMPOSITE:
! /* success */
! break;
! case TYPEFUNC_RECORD:
! /* failed to determine actual type of RECORD */
! ereport(ERROR,
! (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
! errmsg("function returning record called in context "
! "that cannot accept type record")));
! break;
! default:
! /* result type isn't composite */
! elog(ERROR, "return type must be a row type");
! break;
! }
! /* make sure we have a persistent copy of the tupdesc */
! tupdesc = CreateTupleDescCopy(tupdesc);
! ntuples = PQntuples(res);
! nfields = PQnfields(res);
}
! /*
! * check result and tuple descriptor have the same number of columns
! */
! if (nfields != tupdesc->natts)
! ereport(ERROR,
! (errcode(ERRCODE_DATATYPE_MISMATCH),
! errmsg("remote query result rowtype does not match "
! "the specified FROM clause rowtype")));
!
! if (ntuples > 0)
! {
! AttInMetadata *attinmeta;
! Tuplestorestate *tupstore;
! MemoryContext oldcontext;
! int row;
! char **values;
! attinmeta = TupleDescGetAttInMetadata(tupdesc);
! oldcontext = MemoryContextSwitchTo(
! rsinfo->econtext->ecxt_per_query_memory);
! tupstore = tuplestore_begin_heap(true, false, work_mem);
! rsinfo->setResult = tupstore;
! rsinfo->setDesc = tupdesc;
! MemoryContextSwitchTo(oldcontext);
! values = (char **) palloc(nfields * sizeof(char *));
! /* put all tuples into the tuplestore */
! for (row = 0; row < ntuples; row++)
{
! HeapTuple tuple;
! if (!is_sql_cmd)
! {
! int i;
! for (i = 0; i < nfields; i++)
! {
! if (PQgetisnull(res, row, i))
! values[i] = NULL;
! else
! values[i] = PQgetvalue(res, row, i);
! }
! }
! else
! {
! values[0] = PQcmdStatus(res);
! }
! /* build the tuple and put it into the tuplestore. */
! tuple = BuildTupleFromCStrings(attinmeta, values);
! tuplestore_puttuple(tupstore, tuple);
! }
! /* clean up and return the tuplestore */
! tuplestore_donestoring(tupstore);
! }
! PQclear(res);
}
PG_CATCH();
{
! /* be sure to release the libpq result */
! PQclear(res);
! PG_RE_THROW();
}
PG_END_TRY();
}
/*
--- 747,952 ----
rsinfo->setResult = NULL;
rsinfo->setDesc = NULL;
+
+ /*
+ * Result is stored into storeinfo.tuplestore instead of
+ * res->result retuned by PQexec/PQgetResult below
+ */
+ initStoreInfo(&storeinfo, fcinfo);
+ PQregisterTupleAdder(conn, addTuple, &storeinfo);
+
/* synchronous query, or async result retrieval */
if (!is_async)
res = PQexec(conn, sql);
else
res = PQgetResult(conn);
! finishStoreInfo(&storeinfo);
! /* NULL res from async get means we're all done with the results */
! if (res || !is_async)
{
! if (freeconn)
! PQfinish(conn);
!
! if (!res ||
! (PQresultStatus(res) != PGRES_COMMAND_OK &&
! PQresultStatus(res) != PGRES_TUPLES_OK))
! {
! /* This is only for backward compatibility */
! if (storeinfo.nummismatch)
! {
! ereport(ERROR,
! (errcode(ERRCODE_DATATYPE_MISMATCH),
! errmsg("remote query result rowtype does not match "
! "the specified FROM clause rowtype")));
! }
! dblink_res_error(conname, res, "could not execute query", fail);
! return (Datum) 0;
! }
}
return (Datum) 0;
}
static void
! initStoreInfo(storeInfo *sinfo, FunctionCallInfo fcinfo)
{
ReturnSetInfo *rsinfo = (ReturnSetInfo *) fcinfo->resultinfo;
+ TupleDesc tupdesc;
+ int i;
+
+ switch (get_call_result_type(fcinfo, NULL, &tupdesc))
+ {
+ case TYPEFUNC_COMPOSITE:
+ /* success */
+ break;
+ case TYPEFUNC_RECORD:
+ /* failed to determine actual type of RECORD */
+ ereport(ERROR,
+ (errcode(ERRCODE_FEATURE_NOT_SUPPORTED),
+ errmsg("function returning record called in context "
+ "that cannot accept type record")));
+ break;
+ default:
+ /* result type isn't composite */
+ elog(ERROR, "return type must be a row type");
+ break;
+ }
+
+ sinfo->oldcontext = MemoryContextSwitchTo(
+ rsinfo->econtext->ecxt_per_query_memory);
! /* make sure we have a persistent copy of the tupdesc */
! tupdesc = CreateTupleDescCopy(tupdesc);
! sinfo->error_occurred = FALSE;
! sinfo->nummismatch = FALSE;
! sinfo->nattrs = tupdesc->natts;
! sinfo->tuplestore = tuplestore_begin_heap(true, false, work_mem);
! sinfo->attinmeta = TupleDescGetAttInMetadata(tupdesc);
! sinfo->valbuf = (void **)malloc(sinfo->nattrs * sizeof(void *));
! sinfo->valbufsize = (size_t *)malloc(sinfo->nattrs * sizeof(size_t));
! for (i = 0 ; i < sinfo->nattrs ; i++)
{
! sinfo->valbuf[i] = NULL;
! sinfo->valbufsize[i] = 0;
! }
! /* Preallocate memory of same size with PGresAttDesc array for values. */
! sinfo->attrvalbuf = (char *) malloc(sinfo->nattrs * sizeof(PGresAttValue));
! rsinfo->setResult = sinfo->tuplestore;
! rsinfo->setDesc = tupdesc;
! }
! static void
! finishStoreInfo(storeInfo *sinfo)
! {
! int i;
! for (i = 0 ; i < sinfo->nattrs ; i++)
! {
! if (sinfo->valbuf[i])
! {
! free(sinfo->valbuf[i]);
! sinfo->valbuf[i] = NULL;
}
+ }
+ if (sinfo->attrvalbuf)
+ free(sinfo->attrvalbuf);
+ sinfo->attrvalbuf = NULL;
+ MemoryContextSwitchTo(sinfo->oldcontext);
+ }
! static void *
! addTuple(PGresult *res, AddTupFunc func, int id, size_t size)
! {
! storeInfo *sinfo = (storeInfo *)PQgetAddTupleParam(res);
! HeapTuple tuple;
! int fields = PQnfields(res);
! int i;
! PGresAttValue *attval;
! char **cstrs;
! if (sinfo->error_occurred)
! return NULL;
! switch (func)
! {
! case ADDTUP_ALLOC_TEXT:
! case ADDTUP_ALLOC_BINARY:
! if (id == -1)
! return sinfo->attrvalbuf;
! if (id < 0 || id >= sinfo->nattrs)
! return NULL;
! if (sinfo->valbufsize[id] < size)
{
! if (sinfo->valbuf[id] == NULL)
! sinfo->valbuf[id] = malloc(size);
! else
! sinfo->valbuf[id] = realloc(sinfo->valbuf[id], size);
! sinfo->valbufsize[id] = size;
! }
! return sinfo->valbuf[id];
! case ADDTUP_ADD_TUPLE:
! break; /* Go through */
! default:
! /* Ignore */
! break;
! }
! if (sinfo->nattrs != fields)
! {
! sinfo->error_occurred = TRUE;
! sinfo->nummismatch = TRUE;
! finishStoreInfo(sinfo);
! PQsetAddTupleErrMes(res,
! strdup("function returning record called in "
! "context that cannot accept type record"));
! return NULL;
! }
! /*
! * Rewrite PGresAttDesc[] to char(*)[] in-place.
! */
! Assert(sizeof(char*) <= sizeof(PGresAttValue));
! attval = (PGresAttValue *)sinfo->attrvalbuf;
! cstrs = (char **)sinfo->attrvalbuf;
! for(i = 0 ; i < fields ; i++)
! cstrs[i] = PQgetAsCstring(attval++);
! PG_TRY();
! {
! tuple = BuildTupleFromCStrings(sinfo->attinmeta, cstrs);
! tuplestore_puttuple(sinfo->tuplestore, tuple);
}
PG_CATCH();
{
! /*
! * Return the error message in the exception to the caller and
! * cancel the exception.
! */
! ErrorData *edata;
!
! sinfo->error_occurred = TRUE;
! sinfo->nummismatch = TRUE;
!
! finishStoreInfo(sinfo);
!
! edata = CopyErrorData();
! FlushErrorState();
!
! PQsetAddTupleErrMes(res, strdup(edata->message));
! return NULL;
}
PG_END_TRY();
+
+ return sinfo->attrvalbuf;
}
/*
diff --git a/doc/src/sgml/libpq.sgml b/doc/src/sgml/libpq.sgml
index 72c9384..af90952 100644
*** a/doc/src/sgml/libpq.sgml
--- b/doc/src/sgml/libpq.sgml
*************** int PQisthreadsafe();
*** 7233,7238 ****
--- 7233,7557 ----
</sect1>
+ <sect1 id="libpq-alterstorage">
+ <title>Alternative result storage</title>
+
+ <indexterm zone="libpq-alterstorage">
+ <primary>PGresult</primary>
+ <secondary>PGconn</secondary>
+ </indexterm>
+
+ <para>
+ As the standard usage, users can get the result of command
+ execution from <structname>PGresult</structname> aquired
+ with <function>PGgetResult</function>
+ from <structname>PGConn</structname>. While the memory areas for
+ the PGresult are allocated with malloc() internally within calls of
+ command execution functions such as <function>PQexec</function>
+ and <function>PQgetResult</function>. If you have difficulties to
+ handle the result records in the form of PGresult, you can instruct
+ PGconn to store them into your own storage instead of PGresult.
+ </para>
+
+ <variablelist>
+ <varlistentry id="libpq-registertupleadder">
+ <term>
+ <function>PQregisterTupleAdder</function>
+ <indexterm>
+ <primary>PQregisterTupleAdder</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ Sets a function to allocate memory for each tuple and column
+ values, and add the completed tuple into your storage.
+ <synopsis>
+ void PQregisterTupleAdder(PGconn *conn,
+ addTupleFunction func,
+ void *param);
+ </synopsis>
+ </para>
+
+ <para>
+ <variablelist>
+ <varlistentry>
+ <term><parameter>conn</parameter></term>
+ <listitem>
+ <para>
+ The connection object to set the tuple adder
+ function. PGresult created from this connection calles
+ this function to store the result tuples instead of
+ storing into its internal storage.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><parameter>func</parameter></term>
+ <listitem>
+ <para>
+ Tuple adder function to set. NULL means to use the
+ default storage.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><parameter>param</parameter></term>
+ <listitem>
+ <para>
+ A pointer to contextual parameter passed
+ to <parameter>func</parameter>.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+
+ <variablelist>
+ <varlistentry id="libpq-addtuplefunction">
+ <term>
+ <type>addTupleFunction</type>
+ <indexterm>
+ <primary>addTupleFunction</primary>
+ </indexterm>
+ </term>
+
+ <listitem>
+ <para>
+ The type for the callback function to serve memory blocks for
+ each tuple and its column values, and to add the constructed
+ tuple into your own storage.
+ <synopsis>
+ typedef enum
+ {
+ ADDTUP_ALLOC_TEXT,
+ ADDTUP_ALLOC_BINARY,
+ ADDTUP_ADD_TUPLE
+ } AddTupFunc;
+
+ void *(*addTupleFunction)(PGresult *res,
+ AddTupFunc func,
+ int id,
+ size_t size);
+ </synopsis>
+ </para>
+
+ <para>
+ Generally this function must return NULL for failure and should
+ set the error message
+ with <function>PGsetAddTupleErrMes</function> if the cause is
+ other than out of memory. This funcion must not throw any
+ exception. This function is called in the sequence following.
+
+ <itemizedlist spacing="compact">
+ <listitem>
+ <simpara>Call with <parameter>func</parameter>
+ = <firstterm>ADDTUP_ALLOC_BINARY</firstterm>
+ and <parameter>id</parameter> = -1 to request the memory
+ for tuple used as an array
+ of <type>PGresAttValue</type> </simpara>
+ </listitem>
+ <listitem>
+ <simpara>Call with <parameter>func</parameter>
+ = <firstterm>ADDTUP_ALLOC_TEXT</firstterm>
+ or <firstterm>ADDTUP_ALLOC_TEXT</firstterm>
+ and <parameter>id</parameter> is zero or positive number
+ to request the memory for each column value in current
+ tuple.</simpara>
+ </listitem>
+ <listitem>
+ <simpara>Call with <parameter>func</parameter>
+ = <firstterm>ADDTUP_ADD_TUPLE</firstterm> to request the
+ constructed tuple to store.</simpara>
+ </listitem>
+ </itemizedlist>
+ </para>
+ <para>
+ Calling <type>addTupleFunction</type>
+ with <parameter>func</parameter> =
+ <firstterm>ADDTUP_ALLOC_TEXT</firstterm> is telling to return a
+ memory block with at least <parameter>size</parameter> bytes
+ which may not be aligned to the word boundary.
+ <parameter>id</parameter> is a zero or positive number
+ distinguishes the usage of requested memory block, that is the
+ position of the column for which the memory block is used.
+ </para>
+ <para>
+ When <parameter>func</parameter>
+ = <firstterm>ADDTUP_ALLOC_BINARY</firstterm>, this function is
+ telled to return a memory block with at
+ least <parameter>size</parameter> bytes which is aligned to the
+ word boundary.
+ <parameter>id</parameter> is the identifier distinguishes the
+ usage of requested memory block. -1 means that it is used as an
+ array of <type>PGresAttValue</type> to store the tuple. Zero or
+ positive numbers have the same meanings as for
+ <firstterm>ADDTUP_ALLOC_BINARY</firstterm>.
+ </para>
+ <para>When <parameter>func</parameter>
+ = <firstterm>ADDTUP_ADD_TUPLE</firstterm>, this function is
+ telled to store the <type>PGresAttValue</type> structure
+ constructed by the caller into your storage. The pointer to the
+ tuple structure is not passed so you should memorize the
+ pointer to the memory block passed the caller on
+ <parameter>func</parameter>
+ = <parameter>ADDTUP_ALLOC_BINARY</parameter>
+ with <parameter>id</parameter> is -1. This function must return
+ any non-NULL values for success. You must properly put back the
+ memory blocks passed to the caller for this function if needed.
+ </para>
+ <variablelist>
+ <varlistentry>
+ <term><parameter>res</parameter></term>
+ <listitem>
+ <para>
+ A pointer to the <type>PGresult</type> object.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><parameter>func</parameter></term>
+ <listitem>
+ <para>
+ An <type>enum</type> value telling the function to perform.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><parameter>param</parameter></term>
+ <listitem>
+ <para>
+ A pointer to contextual parameter passed to func.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+
+ <variablelist>
+ <varlistentry id="libpq-pqgestasctring">
+ <term>
+ <function>PQgetAsCstring</function>
+ <indexterm>
+ <primary>PQgetAsCstring</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Get the value of the column pointed
+ by <parameter>attval</parameter> in the form of
+ zero-terminated C string. Returns NULL if the value is null.
+ <synopsis>
+ char *PQgetAsCstring(PGresAttValue *attval)
+ </synopsis>
+ </para>
+ <para>
+ <variablelist>
+ <varlistentry>
+ <term><parameter>attval</parameter></term>
+ <listitem>
+ <para>
+ A pointer to the <type>PGresAttValue</type> object
+ to retrieve the value.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+
+ <variablelist>
+ <varlistentry id="libpq-pqgetaddtupleparam">
+ <term>
+ <function>PQgetAddTupleParam</function>
+ <indexterm>
+ <primary>PQgetAddTupleParam</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Get the pointer passed to <function>PQregisterTupleAdder</function>
+ as <parameter>param</parameter>.
+ <synopsis>
+ void *PQgetTupleParam(PGresult *res)
+ </synopsis>
+ </para>
+ <para>
+ <variablelist>
+ <varlistentry>
+ <term><parameter>res</parameter></term>
+ <listitem>
+ <para>
+ A pointer to the <type>PGresult</type> object.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+
+ <variablelist>
+ <varlistentry id="libpq-pqsetaddtupleerrmes">
+ <term>
+ <function>PQsetAddTupleErrMes</function>
+ <indexterm>
+ <primary>PQsetAddTupleErrMes</primary>
+ </indexterm>
+ </term>
+ <listitem>
+ <para>
+ Set the message for the error occurred in <type>addTupleFunction</type>.
+ If this message is not set, the error is assumed to be out of
+ memory.
+ <synopsis>
+ void PQsetAddTupleErrMes(PGresult *res, char *mes)
+ </synopsis>
+ </para>
+ <para>
+ <variablelist>
+ <varlistentry>
+ <term><parameter>res</parameter></term>
+ <listitem>
+ <para>
+ A pointer to the <type>PGresult</type> object
+ in <type>addTupleFunction</type>.
+ </para>
+ </listitem>
+ </varlistentry>
+ <varlistentry>
+ <term><parameter>mes</parameter></term>
+ <listitem>
+ <para>
+ A pointer to the memory block containing the error
+ message, which must be allocated by alloc(). The
+ memory block will be freed with free() in the caller
+ of
+ <type>addTupleFunction</type> only if it returns NULL.
+ </para>
+ <para>
+ If <parameter>res</parameter> already has a message
+ previously set, it is freed and the given message is
+ set. Set NULL to cancel the the costom message.
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </para>
+ </listitem>
+ </varlistentry>
+ </variablelist>
+ </sect1>
+
+
<sect1 id="libpq-build">
<title>Building <application>libpq</application> Programs</title>
diff --git a/src/interfaces/libpq/exports.txt b/src/interfaces/libpq/exports.txt
index 1af8df6..a360d78 100644
*** a/src/interfaces/libpq/exports.txt
--- b/src/interfaces/libpq/exports.txt
*************** PQconnectStartParams 157
*** 160,162 ****
--- 160,166 ----
PQping 158
PQpingParams 159
PQlibVersion 160
+ PQregisterTupleAdder 161
+ PQgetAsCstring 162
+ PQgetAddTupleParam 163
+ PQsetAddTupleErrMes 164
diff --git a/src/interfaces/libpq/fe-connect.c b/src/interfaces/libpq/fe-connect.c
index d454538..15d6216 100644
*** a/src/interfaces/libpq/fe-connect.c
--- b/src/interfaces/libpq/fe-connect.c
*************** makeEmptyPGconn(void)
*** 2692,2697 ****
--- 2692,2698 ----
conn->allow_ssl_try = true;
conn->wait_ssl_try = false;
#endif
+ conn->addTupleFunc = NULL;
/*
* We try to send at least 8K at a time, which is the usual size of pipe
*************** PQregisterThreadLock(pgthreadlock_t newh
*** 5076,5078 ****
--- 5077,5086 ----
return prev;
}
+
+ void
+ PQregisterTupleAdder(PGconn *conn, addTupleFunction func, void *param)
+ {
+ conn->addTupleFunc = func;
+ conn->addTupleFuncParam = param;
+ }
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index b743566..3f774fd 100644
*** a/src/interfaces/libpq/fe-exec.c
--- b/src/interfaces/libpq/fe-exec.c
*************** char *const pgresStatus[] = {
*** 48,54 ****
static int static_client_encoding = PG_SQL_ASCII;
static bool static_std_strings = false;
-
static PGEvent *dupEvents(PGEvent *events, int count);
static bool PQsendQueryStart(PGconn *conn);
static int PQsendQueryGuts(PGconn *conn,
--- 48,53 ----
*************** static PGresult *PQexecFinish(PGconn *co
*** 66,72 ****
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
!
/* ----------------
* Space management for PGresult.
--- 65,73 ----
static int PQsendDescribe(PGconn *conn, char desc_type,
const char *desc_target);
static int check_field_number(const PGresult *res, int field_num);
! static void *pqDefaultAddTupleFunc(PGresult *res, AddTupFunc func,
! int id, size_t len);
! static void *pqAddTuple(PGresult *res, PGresAttValue *tup);
/* ----------------
* Space management for PGresult.
*************** PQmakeEmptyPGresult(PGconn *conn, ExecSt
*** 160,165 ****
--- 161,169 ----
result->curBlock = NULL;
result->curOffset = 0;
result->spaceLeft = 0;
+ result->addTupleFunc = pqDefaultAddTupleFunc;
+ result->addTupleFuncParam = NULL;
+ result->addTupleFuncErrMes = NULL;
if (conn)
{
*************** PQmakeEmptyPGresult(PGconn *conn, ExecSt
*** 194,199 ****
--- 198,209 ----
}
result->nEvents = conn->nEvents;
}
+
+ if (conn->addTupleFunc)
+ {
+ result->addTupleFunc = conn->addTupleFunc;
+ result->addTupleFuncParam = conn->addTupleFuncParam;
+ }
}
else
{
*************** PQresultAlloc(PGresult *res, size_t nByt
*** 487,492 ****
--- 497,529 ----
return pqResultAlloc(res, nBytes, TRUE);
}
+ void *
+ pqDefaultAddTupleFunc(PGresult *res, AddTupFunc func, int id, size_t len)
+ {
+ void *p;
+
+ switch (func)
+ {
+ case ADDTUP_ALLOC_TEXT:
+ return pqResultAlloc(res, len, TRUE);
+
+ case ADDTUP_ALLOC_BINARY:
+ p = pqResultAlloc(res, len, FALSE);
+
+ if (id == -1)
+ res->addTupleFuncParam = p;
+
+ return p;
+
+ case ADDTUP_ADD_TUPLE:
+ return pqAddTuple(res, res->addTupleFuncParam);
+
+ default:
+ /* Ignore */
+ break;
+ }
+ return NULL;
+ }
/*
* pqResultAlloc -
* Allocate subsidiary storage for a PGresult.
*************** pqInternalNotice(const PGNoticeHooks *ho
*** 830,838 ****
/*
* pqAddTuple
* add a row pointer to the PGresult structure, growing it if necessary
! * Returns TRUE if OK, FALSE if not enough memory to add the row
*/
! int
pqAddTuple(PGresult *res, PGresAttValue *tup)
{
if (res->ntups >= res->tupArrSize)
--- 867,875 ----
/*
* pqAddTuple
* add a row pointer to the PGresult structure, growing it if necessary
! * Returns tup if OK, NULL if not enough memory to add the row.
*/
! static void *
pqAddTuple(PGresult *res, PGresAttValue *tup)
{
if (res->ntups >= res->tupArrSize)
*************** pqAddTuple(PGresult *res, PGresAttValue
*** 858,870 ****
newTuples = (PGresAttValue **)
realloc(res->tuples, newSize * sizeof(PGresAttValue *));
if (!newTuples)
! return FALSE; /* malloc or realloc failed */
res->tupArrSize = newSize;
res->tuples = newTuples;
}
res->tuples[res->ntups] = tup;
res->ntups++;
! return TRUE;
}
/*
--- 895,907 ----
newTuples = (PGresAttValue **)
realloc(res->tuples, newSize * sizeof(PGresAttValue *));
if (!newTuples)
! return NULL; /* malloc or realloc failed */
res->tupArrSize = newSize;
res->tuples = newTuples;
}
res->tuples[res->ntups] = tup;
res->ntups++;
! return tup;
}
/*
*************** PQgetisnull(const PGresult *res, int tup
*** 2822,2827 ****
--- 2859,2901 ----
return 0;
}
+ /* PQgetAsCString
+ * returns the field as C string.
+ */
+ char *
+ PQgetAsCstring(PGresAttValue *attval)
+ {
+ return attval->len == NULL_LEN ? NULL : attval->value;
+ }
+
+ /* PQgetAddTupleParam
+ * Get the pointer to the contextual parameter from PGresult which is
+ * registered to PGconn by PQregisterTupleAdder
+ */
+ void *
+ PQgetAddTupleParam(const PGresult *res)
+ {
+ if (!res)
+ return NULL;
+ return res->addTupleFuncParam;
+ }
+
+ /* PQsetAddTupleErrMes
+ * Set the error message pass back to the caller of addTupleFunc
+ * mes must be a malloc'ed memory block and it is released by the
+ * caller of addTupleFunc if set.
+ * You can replace the previous message by alternative mes, or clear
+ * it with NULL.
+ */
+ void
+ PQsetAddTupleErrMes(PGresult *res, char *mes)
+ {
+ /* Free existing message */
+ if (res->addTupleFuncErrMes)
+ free(res->addTupleFuncErrMes);
+ res->addTupleFuncErrMes = mes;
+ }
+
/* PQnparams:
* returns the number of input parameters of a prepared statement.
*/
diff --git a/src/interfaces/libpq/fe-protocol2.c b/src/interfaces/libpq/fe-protocol2.c
index a7c3899..721c812 100644
*** a/src/interfaces/libpq/fe-protocol2.c
--- b/src/interfaces/libpq/fe-protocol2.c
*************** getAnotherTuple(PGconn *conn, bool binar
*** 733,741 ****
if (conn->curTuple == NULL)
{
conn->curTuple = (PGresAttValue *)
! pqResultAlloc(result, nfields * sizeof(PGresAttValue), TRUE);
if (conn->curTuple == NULL)
! goto outOfMemory;
MemSet(conn->curTuple, 0, nfields * sizeof(PGresAttValue));
/*
--- 733,742 ----
if (conn->curTuple == NULL)
{
conn->curTuple = (PGresAttValue *)
! result->addTupleFunc(result, ADDTUP_ALLOC_BINARY, -1,
! nfields * sizeof(PGresAttValue));
if (conn->curTuple == NULL)
! goto addTupleError;
MemSet(conn->curTuple, 0, nfields * sizeof(PGresAttValue));
/*
*************** getAnotherTuple(PGconn *conn, bool binar
*** 757,763 ****
{
bitmap = (char *) malloc(nbytes);
if (!bitmap)
! goto outOfMemory;
}
if (pqGetnchar(bitmap, nbytes, conn))
--- 758,764 ----
{
bitmap = (char *) malloc(nbytes);
if (!bitmap)
! goto addTupleError;
}
if (pqGetnchar(bitmap, nbytes, conn))
*************** getAnotherTuple(PGconn *conn, bool binar
*** 787,795 ****
vlen = 0;
if (tup[i].value == NULL)
{
! tup[i].value = (char *) pqResultAlloc(result, vlen + 1, binary);
if (tup[i].value == NULL)
! goto outOfMemory;
}
tup[i].len = vlen;
/* read in the value */
--- 788,799 ----
vlen = 0;
if (tup[i].value == NULL)
{
! AddTupFunc func =
! (binary ? ADDTUP_ALLOC_BINARY : ADDTUP_ALLOC_TEXT);
! tup[i].value =
! (char *) result->addTupleFunc(result, func, i, vlen + 1);
if (tup[i].value == NULL)
! goto addTupleError;
}
tup[i].len = vlen;
/* read in the value */
*************** getAnotherTuple(PGconn *conn, bool binar
*** 812,819 ****
}
/* Success! Store the completed tuple in the result */
! if (!pqAddTuple(result, tup))
! goto outOfMemory;
/* and reset for a new message */
conn->curTuple = NULL;
--- 816,824 ----
}
/* Success! Store the completed tuple in the result */
! if (!result->addTupleFunc(result, ADDTUP_ADD_TUPLE, 0, 0))
! goto addTupleError;
!
/* and reset for a new message */
conn->curTuple = NULL;
*************** getAnotherTuple(PGconn *conn, bool binar
*** 821,827 ****
free(bitmap);
return 0;
! outOfMemory:
/* Replace partially constructed result with an error result */
/*
--- 826,832 ----
free(bitmap);
return 0;
! addTupleError:
/* Replace partially constructed result with an error result */
/*
*************** outOfMemory:
*** 829,836 ****
* there's not enough memory to concatenate messages...
*/
pqClearAsyncResult(conn);
! printfPQExpBuffer(&conn->errorMessage,
! libpq_gettext("out of memory for query result\n"));
/*
* XXX: if PQmakeEmptyPGresult() fails, there's probably not much we can
--- 834,854 ----
* there's not enough memory to concatenate messages...
*/
pqClearAsyncResult(conn);
! resetPQExpBuffer(&conn->errorMessage);
!
! /*
! * If error message is passed from addTupleFunc, set it into
! * PGconn, assume out of memory if not.
! */
! appendPQExpBufferStr(&conn->errorMessage,
! libpq_gettext(result->addTupleFuncErrMes ?
! result->addTupleFuncErrMes :
! "out of memory for query result\n"));
! if (result->addTupleFuncErrMes)
! {
! free(result->addTupleFuncErrMes);
! result->addTupleFuncErrMes = NULL;
! }
/*
* XXX: if PQmakeEmptyPGresult() fails, there's probably not much we can
diff --git a/src/interfaces/libpq/fe-protocol3.c b/src/interfaces/libpq/fe-protocol3.c
index 892dcbc..1417d59 100644
*** a/src/interfaces/libpq/fe-protocol3.c
--- b/src/interfaces/libpq/fe-protocol3.c
*************** getAnotherTuple(PGconn *conn, int msgLen
*** 634,642 ****
if (conn->curTuple == NULL)
{
conn->curTuple = (PGresAttValue *)
! pqResultAlloc(result, nfields * sizeof(PGresAttValue), TRUE);
if (conn->curTuple == NULL)
! goto outOfMemory;
MemSet(conn->curTuple, 0, nfields * sizeof(PGresAttValue));
}
tup = conn->curTuple;
--- 634,643 ----
if (conn->curTuple == NULL)
{
conn->curTuple = (PGresAttValue *)
! result->addTupleFunc(result, ADDTUP_ALLOC_BINARY, -1,
! nfields * sizeof(PGresAttValue));
if (conn->curTuple == NULL)
! goto addTupleError;
MemSet(conn->curTuple, 0, nfields * sizeof(PGresAttValue));
}
tup = conn->curTuple;
*************** getAnotherTuple(PGconn *conn, int msgLen
*** 673,683 ****
vlen = 0;
if (tup[i].value == NULL)
{
! bool isbinary = (result->attDescs[i].format != 0);
!
! tup[i].value = (char *) pqResultAlloc(result, vlen + 1, isbinary);
if (tup[i].value == NULL)
! goto outOfMemory;
}
tup[i].len = vlen;
/* read in the value */
--- 674,685 ----
vlen = 0;
if (tup[i].value == NULL)
{
! AddTupFunc func = (result->attDescs[i].format != 0 ?
! ADDTUP_ALLOC_BINARY : ADDTUP_ALLOC_TEXT);
! tup[i].value =
! (char *) result->addTupleFunc(result, func, i, vlen + 1);
if (tup[i].value == NULL)
! goto addTupleError;
}
tup[i].len = vlen;
/* read in the value */
*************** getAnotherTuple(PGconn *conn, int msgLen
*** 689,710 ****
}
/* Success! Store the completed tuple in the result */
! if (!pqAddTuple(result, tup))
! goto outOfMemory;
/* and reset for a new message */
conn->curTuple = NULL;
return 0;
! outOfMemory:
/*
* Replace partially constructed result with an error result. First
* discard the old result to try to win back some memory.
*/
pqClearAsyncResult(conn);
! printfPQExpBuffer(&conn->errorMessage,
! libpq_gettext("out of memory for query result\n"));
pqSaveErrorResult(conn);
/* Discard the failed message by pretending we read it */
--- 691,726 ----
}
/* Success! Store the completed tuple in the result */
! if (!result->addTupleFunc(result, ADDTUP_ADD_TUPLE, 0, 0))
! goto addTupleError;
!
/* and reset for a new message */
conn->curTuple = NULL;
return 0;
! addTupleError:
/*
* Replace partially constructed result with an error result. First
* discard the old result to try to win back some memory.
*/
pqClearAsyncResult(conn);
! resetPQExpBuffer(&conn->errorMessage);
!
! /*
! * If error message is passed from addTupleFunc, set it into
! * PGconn, assume out of memory if not.
! */
! appendPQExpBufferStr(&conn->errorMessage,
! libpq_gettext(result->addTupleFuncErrMes ?
! result->addTupleFuncErrMes :
! "out of memory for query result\n"));
! if (result->addTupleFuncErrMes)
! {
! free(result->addTupleFuncErrMes);
! result->addTupleFuncErrMes = NULL;
! }
pqSaveErrorResult(conn);
/* Discard the failed message by pretending we read it */
diff --git a/src/interfaces/libpq/libpq-fe.h b/src/interfaces/libpq/libpq-fe.h
index ef26ab9..bfa6556 100644
*** a/src/interfaces/libpq/libpq-fe.h
--- b/src/interfaces/libpq/libpq-fe.h
*************** typedef enum
*** 116,121 ****
--- 116,131 ----
PQPING_NO_ATTEMPT /* connection not attempted (bad params) */
} PGPing;
+ /* AddTupFunc is one of the parameters of addTupleFunc that decides
+ * the function of the addTupleFunction. See addTupleFunction for
+ * details */
+ typedef enum
+ {
+ ADDTUP_ALLOC_TEXT, /* Returns non-aligned memory for text value */
+ ADDTUP_ALLOC_BINARY, /* Returns aligned memory for binary value */
+ ADDTUP_ADD_TUPLE /* Adds tuple data into tuple storage */
+ } AddTupFunc;
+
/* PGconn encapsulates a connection to the backend.
* The contents of this struct are not supposed to be known to applications.
*/
*************** typedef struct pgresAttDesc
*** 225,230 ****
--- 235,246 ----
int atttypmod; /* type-specific modifier info */
} PGresAttDesc;
+ typedef struct pgresAttValue
+ {
+ int len; /* length in bytes of the value */
+ char *value; /* actual value, plus terminating zero byte */
+ } PGresAttValue;
+
/* ----------------
* Exported functions of libpq
* ----------------
*************** extern PGPing PQping(const char *conninf
*** 416,421 ****
--- 432,483 ----
extern PGPing PQpingParams(const char *const * keywords,
const char *const * values, int expand_dbname);
+ /*
+ * Typedef for tuple storage function.
+ *
+ * This function pointer is used for tuple storage function in
+ * PGresult and PGconn.
+ *
+ * addTupleFunction is called for four types of function designated by
+ * the enum AddTupFunc.
+ *
+ * id is the identifier for allocated memory block. The caller sets -1
+ * for PGresAttValue array, and 0 to number of cols - 1 for each
+ * column.
+ *
+ * ADDTUP_ALLOC_TEXT requests the size bytes memory block for a text
+ * value which may not be alingned to the word boundary.
+ *
+ * ADDTUP_ALLOC_BINARY requests the size bytes memory block for a
+ * binary value which is aligned to the word boundary.
+ *
+ * ADDTUP_ADD_TUPLE requests to add tuple data into storage, and
+ * free the memory blocks allocated by this function if necessary.
+ * id and size are ignored.
+ *
+ * This function must return non-NULL value for success and must
+ * return NULL for failure and may set error message by
+ * PQsetAddTupleErrMes in malloc'ed memory. Assumed by caller as out
+ * of memory if the error message is NULL on failure. This function is
+ * assumed not to throw any exception.
+ */
+ typedef void *(*addTupleFunction)(PGresult *res, AddTupFunc func,
+ int id, size_t size);
+
+ /*
+ * Register alternative tuple storage function to PGconn.
+ *
+ * By registering this function, pg_result disables its own tuple
+ * storage and calls it to append rows one by one.
+ *
+ * func is tuple store function. See addTupleFunction.
+ *
+ * addTupFuncParam is contextual storage that can be get with
+ * PQgetAddTupleParam in func.
+ */
+ extern void PQregisterTupleAdder(PGconn *conn, addTupleFunction func,
+ void *addTupFuncParam);
+
/* Force the write buffer to be written (or at least try) */
extern int PQflush(PGconn *conn);
*************** extern char *PQcmdTuples(PGresult *res);
*** 454,459 ****
--- 516,524 ----
extern char *PQgetvalue(const PGresult *res, int tup_num, int field_num);
extern int PQgetlength(const PGresult *res, int tup_num, int field_num);
extern int PQgetisnull(const PGresult *res, int tup_num, int field_num);
+ extern char *PQgetAsCstring(PGresAttValue *attdesc);
+ extern void *PQgetAddTupleParam(const PGresult *res);
+ extern void PQsetAddTupleErrMes(PGresult *res, char *mes);
extern int PQnparams(const PGresult *res);
extern Oid PQparamtype(const PGresult *res, int param_num);
diff --git a/src/interfaces/libpq/libpq-int.h b/src/interfaces/libpq/libpq-int.h
index d967d60..01e8c3e 100644
*** a/src/interfaces/libpq/libpq-int.h
--- b/src/interfaces/libpq/libpq-int.h
*************** typedef struct pgresParamDesc
*** 134,145 ****
#define NULL_LEN (-1) /* pg_result len for NULL value */
- typedef struct pgresAttValue
- {
- int len; /* length in bytes of the value */
- char *value; /* actual value, plus terminating zero byte */
- } PGresAttValue;
-
/* Typedef for message-field list entries */
typedef struct pgMessageField
{
--- 134,139 ----
*************** struct pg_result
*** 209,214 ****
--- 203,213 ----
PGresult_data *curBlock; /* most recently allocated block */
int curOffset; /* start offset of free space in block */
int spaceLeft; /* number of free bytes remaining in block */
+
+ addTupleFunction addTupleFunc; /* Tuple storage function. See
+ * addTupleFunction for details. */
+ void *addTupleFuncParam; /* Contextual parameter for addTupleFunc */
+ char *addTupleFuncErrMes; /* Error message returned from addTupFunc */
};
/* PGAsyncStatusType defines the state of the query-execution state machine */
*************** struct pg_conn
*** 443,448 ****
--- 442,454 ----
/* Buffer for receiving various parts of messages */
PQExpBufferData workBuffer; /* expansible string */
+
+ /* Tuple store function. The two fields below is copied to newly
+ * created PGresult if addTupleFunc is not NULL. Use default
+ * function if addTupleFunc is NULL. */
+ addTupleFunction addTupleFunc; /* Tuple storage function. See
+ * addTupleFunction for details. */
+ void *addTupleFuncParam; /* Contextual parameter for addTupFunc */
};
/* PGcancel stores all data necessary to cancel a connection. A copy of this
*************** extern void
*** 507,513 ****
pqInternalNotice(const PGNoticeHooks *hooks, const char *fmt,...)
/* This lets gcc check the format string for consistency. */
__attribute__((format(PG_PRINTF_ATTRIBUTE, 2, 3)));
- extern int pqAddTuple(PGresult *res, PGresAttValue *tup);
extern void pqSaveMessageField(PGresult *res, char code,
const char *value);
extern void pqSaveParameterStatus(PGconn *conn, const char *name,
--- 513,518 ----
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
