Re: [HACKERS] check for missing tablespaces?
Andrew Dunstan wrote: I am in the middle of helping a customer recover from a situation where a tablespace was missing when a machine was rebooted and postgres restarted, Have you uncovered why the tablespace went missing? and I'm wondering if we should not have some sort of check for this on startup +1 Andrew Chernow -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] check for missing tablespaces?
Anyway, from this POV all we really need to know is that the device hosting this tablespace failed to mount when the machine was rebooted, and then postgres restarted. Good to know postgresql had nothing to do with the missing data. I wasn't sure if it was user error, config problem or hardware. From my experience, it doesn't really matter if you have a sophisticated SAN or put together an ad-hoc nas/das solution, storage likes to bite you. The only thing that helps me sleep at night is sound replication and backups (in some cases triple redundancy with aging data sets). Andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [pgsql-www] commitfest.postgresql.org
Robert Treat wrote: On Friday 03 July 2009 11:50:29 Tom Lane wrote: Robert Haas robertmh...@gmail.com writes: On Fri, Jul 3, 2009 at 10:35 AM, Tom Lanet...@sss.pgh.pa.us wrote: The current URL seems to be http://commitfest.postgresql.org/action/commitfest_view?id=2 which is both opaque as can be and not looking like it's intended to be stable over the long term. I'm not sure why you would think that it's not stable. Because it's exposing three or four details of your implementation, which you might wish to change later. I'm also not sure what you would think that it's not self-explanatory, since it looks pretty self explanatory to me. It's impossible to know that this is commitfest 2009-07. commitfest.postgresql.org/2009/07 ? That, or any similar scheme, seems easily doable with a little apache rewrite magic and some programming. See my blog urls for one such example. I believe Tom wants details removed from the URL, so future implementation changes don't either a) break bookmarks because more stuff is needed in the URL or b) don't break bookmarks but be limited to existing sutff in the URL (ie. hacky work arounds). If that's the case, your best best is to use some kind of key, like 16 random bytes displayed in hex, that looks up the data. IMHO, I don't see much gain to encoding the date into the url either. This is not a great way of telling the user when something occurred. A lookup is going to occur either way, so why not get all data at once using a single method? -- Andrew Chernow eSilo, LLC every bit counts http://www.esilo.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [pgsql-www] commitfest.postgresql.org
I *am* using some kind of key. Specifically, in integer derived from a serial column. It's just as stable as 16 random bytes displayed in hex, but a lot shorter and easier to remember, if you're the sort of person who likes to remember URLs. :-) Wasn't aware of exately what you were doing. It sounded like multiple things were in the query_string. If its already a single key, than there is no need to use a different key. And no, I don't like remebering URLs ... thus all the fuss about breaking bookmarks ;-) It's impossible to know that this is commitfest 2009-07. commitfest.postgresql.org/2009/07 ? That, or any similar scheme, seems easily doable with a little apache rewrite magic and some programming. See my blog urls for one such example. IMHO, I don't see much gain to encoding the date into the url either. This is not a great way of telling the user when something occurred. A lookup is going to occur either way, so why not get all data at once using a single method? Sorry, I'm not following this part. Using a URL to encode when something occurred was being offered as a solution to know what commitfest it is. I'm not sure where your confusion is? -- Andrew Chernow eSilo, LLC every bit counts http://www.esilo.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] New types for transparent encryption
Encrypting lots of small chunks of data with the same key is a very dangerous thing to do and it's very tricky to get right. Using an initialization vector (IV) is the way to go, recommend using CBC or CFB mode. Although, an IV is never supposed to be used more than once with the same key; that can leak hints about the plaintext. Where is the randomly generated IV stored for use during decryption? -- Andrew Chernow eSilo, LLC every bit counts http://www.esilo.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] New types for transparent encryption
Andrew Dunstan wrote: Andrew Chernow wrote: Encrypting lots of small chunks of data with the same key is a very dangerous thing to do and it's very tricky to get right. Using an initialization vector (IV) is the way to go, recommend using CBC or CFB mode. Although, an IV is never supposed to be used more than once with the same key; that can leak hints about the plaintext. Where is the randomly generated IV stored for use during decryption? Well, you can store it along with the encrypted data. The IV doesn't need to be secret, just random. I do that for one of my clients. That's correct. Duh!?! Probably the first N bytes of the cipher text. Would the IV be regenerated every time the plaintext is updated, to avoid using it twice? For instace: update t set text = 'abc' where id = 1 . ISTM that the IV for OLD.text should be thrown away. Where would the key come from? Where would it be stored? What cipher is used? -- Andrew Chernow eSilo, LLC every bit counts http://www.esilo.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [GENERAL] queriing the version of libpq
And it *could* be used in exactly the case you're outlining as long as you load the function dynamically. Detecting the presence of a function does not require a version number. If the symbol is in the library, use it. The only reason the version number would come into play is if you were attempting to detect behavior differences. -- Andrew Chernow eSilo, LLC every bit counts http://www.esilo.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Suggesting a libpq addition
to consider libpqtypes for contrib (which we don't have time for atm). ... or as a libpq sibling :) -- Andrew Chernow eSilo, LLC every bit counts http://www.esilo.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Visual Studio 2010/Windows SDK 7.1 support
On 1/15/2011 1:42 PM, Magnus Hagander wrote: On Mon, Jan 3, 2011 at 16:45, Brar Pieningb...@gmx.de wrote: On Mon, 3 Jan 2011 10:44:19 +0100, Magnus Hagandermag...@hagander.net wrote: Yeah, it looks that way - it's missing the ordering of the contrib I'll run it once for that now, and then please rebase your patch on top of that - makes it easier to review it. The rebased patch can be grabbed from http://www.piening.info/VS2010v2.patch Hi! Please make sure this goes on the commitfest page (https://commitfest.postgresql.org/action/commitfest_view?id=9), so it's not missed. I'm currently lacking an env where I can put VS2010 on it (given that the amazon cloud no longer works reasonably for windows machines, and I can't put another version of VS on the other VM I'm using right now), so it'll be a while before I can look at this. I can provide a windows box with VS2010 if you'd like. Wouldn't be until Monday or Tuesday. Any preference on windows version? Maybe Windows 7? You want 64-bit? Send a private email. -- Andrew Chernow eSilo, LLC global backup http://www.esilo.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PQdeleteTuple function in libpq
On 6/1/2011 11:43 AM, Pavel Golub wrote: Hello. I'm some kind of PQdeleteTuple function will be very usefull in libpq. Because right now after deleting some record I need refetch result set, or mark tuple as deleted and this is headache for me. IMHO, this should be handled by the application. You could track tuples removed in an int[] or copy the result set into an application defined array of C structures. I've always been under the impression that PGresult objects are immutable once delivered to the application. -- Andrew Chernow eSilo, LLC global backup http://www.esilo.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PQdeleteTuple function in libpq
On 6/2/2011 4:28 AM, Pavel Golub wrote: Hello, Andrew. You wrote: AC On 6/1/2011 11:43 AM, Pavel Golub wrote: Hello. I'm some kind of PQdeleteTuple function will be very usefull in libpq. Because right now after deleting some record I need refetch result set, or mark tuple as deleted and this is headache for me. AC IMHO, this should be handled by the application. You could track tuples AC removed in an int[] or copy the result set into an application defined AC array of C structures. I've always been under the impression that AC PGresult objects are immutable once delivered to the application. Andrew, why we have PQmakeEmptyPGresult, PQcopyResult, PQsetResultAttrs, PQsetvalue and PQresultAlloc in this case? Of course there's no big deal with their absence but let's be consistent. I'm not entirely sure what you are trying to do, but can't you use PQmakeEmptyPGresult, PQsetResultAttrs and PQsetvalue to construct a result that excludes the tuples you don't want followed by a PQclear(initial_result)? -- Andrew Chernow eSilo, LLC global backup http://www.esilo.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PQdeleteTuple function in libpq
AC IMHO, this should be handled by the application. You could track tuples AC removed in an int[] or copy the result set into an application defined AC array of C structures. I've always been under the impression that AC PGresult objects are immutable once delivered to the application. Andrew, why we have PQmakeEmptyPGresult, PQcopyResult, PQsetResultAttrs, PQsetvalue and PQresultAlloc in this case? Of course there's no big deal with their absence but let's be consistent. I'm not entirely sure what you are trying to do, but can't you use PQmakeEmptyPGresult, PQsetResultAttrs and PQsetvalue to construct a result that excludes the tuples you don't want followed by a PQclear(initial_result)? Actually the best solution would be to call PQcopyResult with all PG_COPYRES_XXX flags enabled except PG_COPYRES_TUPLES. Now call PQsetvalue for each tuple you want to add. -- Andrew Chernow eSilo, LLC global backup http://www.esilo.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] PQdeleteTuple function in libpq
On 6/2/2011 11:02 AM, Alvaro Herrera wrote: Excerpts from Andrew Chernow's message of jue jun 02 10:12:40 -0400 2011: Andrew, why we have PQmakeEmptyPGresult, PQcopyResult, PQsetResultAttrs, PQsetvalue and PQresultAlloc in this case? Of course there's no big deal with their absence but let's be consistent. I'm not entirely sure what you are trying to do, but can't you use PQmakeEmptyPGresult, PQsetResultAttrs and PQsetvalue to construct a result that excludes the tuples you don't want followed by a PQclear(initial_result)? Seems pretty wasteful if you want to delete a single tuple from a large result. I think if you desired to compact the result to free some memory after deleting a large fraction of the tuples in the result it could be useful to do that, otherwise just live with the unused holes in the storage area as suggested by Pavel. Another solution is to manually cursor through the set (like grab 1000 tuples at a time) and copy the set to your own structure. That way, the temporary double memory to perform the copy is not as big of a hit. By using your own structure, you can organize the memory in a fashion that is optimized for your requirement. -- Andrew Chernow eSilo, LLC global backup http://www.esilo.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Error in PQsetvalue
On 6/3/2011 3:03 PM, Pavel Golub wrote: Hello. Reproduced under Windows XP SP3 using Visual C++ 2008 and Delphi. If PQsetvalue is called with second parameter equals to PQntuples then memory corruption appears. But it should grow internal tuples array and populate the last item with provided data. Please see the code: At first glance (have not tested this theory), looks like pqAddTuple() doesn't zero the newly allocated tuples slots like PQsetvalue() does. PQsetvalue is depending on the unassigned tuple table slots to be NULL to detect when a tuple must be allocated. Around line 446 on fe-exec.c. I never tested this case since libpqtypes never tried to call PQsetvalue on a PGresult created by the standard libpq library. The solution I see would be to zero the new table slots within pqAddTuple. Any other ideas? -- Andrew Chernow eSilo, LLC global backup http://www.esilo.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Error in PQsetvalue
On 6/3/2011 4:06 PM, Andrew Chernow wrote: On 6/3/2011 3:03 PM, Pavel Golub wrote: Hello. Reproduced under Windows XP SP3 using Visual C++ 2008 and Delphi. If PQsetvalue is called with second parameter equals to PQntuples then memory corruption appears. But it should grow internal tuples array and populate the last item with provided data. Please see the code: At first glance (have not tested this theory), looks like pqAddTuple() doesn't zero the newly allocated tuples slots like PQsetvalue() does. PQsetvalue is depending on the unassigned tuple table slots to be NULL to detect when a tuple must be allocated. Around line 446 on fe-exec.c. I never tested this case since libpqtypes never tried to call PQsetvalue on a PGresult created by the standard libpq library. The solution I see would be to zero the new table slots within pqAddTuple. Any other ideas? Eeekks. Found an additional bug. PQsetvalue only allocates the actual tuple if the provided tup_num equals the number of tuples (append) and that slot is NULL. This is wrong. The original idea behind PQsetvalue was you can add tuples in any order and overwrite existing ones. Also, doing res-ntups++ whenever a tuple is allocated is incorrect as well; since we may first add tuple 3. In that case, if ntups is currently = 3 we need to set it to 3+1, otherwise do nothing. In other words, while adding tuples via PQsetvalue the ntups should always be equal to (max_supplied_tup_num + 1) with all unset slots being NULL. PQsetvalue(res, 3, 0, x, 1); // currently sets res-ntups to 1 PQsetvalue(res, 2, 0, x, 1); // as code is now, this returns FALSE due to bounds checking -- Andrew Chernow eSilo, LLC global backup http://www.esilo.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Error in PQsetvalue
At first glance (have not tested this theory), looks like pqAddTuple() doesn't zero the newly allocated tuples slots like PQsetvalue() does. PQsetvalue is depending on the unassigned tuple table slots to be NULL to detect when a tuple must be allocated. Around line 446 on fe-exec.c. I never tested this case since libpqtypes never tried to call PQsetvalue on a PGresult created by the standard libpq library. The solution I see would be to zero the new table slots within pqAddTuple. Any other ideas? It might not be necessary to do that. AIUI the tuple table slot guard is there essentially to let setval know if it needs to allocate tuple attributes, which always has to be done after a new tuple is created after a set. Trying to append a tuple to a libpq generated PGresult will core dump due to the pqAddTuple issue, unless the append operation forces PQsetvalue to grow the tuple table; in which case new elements are zero'd. OP attempted to append. res = PQexec(returns 2 tuples); PQsetvalue(res, PQntuples(res), ...); -- Andrew Chernow eSilo, LLC global backup http://www.esilo.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Error in PQsetvalue
On 6/3/2011 5:54 PM, Merlin Moncure wrote: On Fri, Jun 3, 2011 at 3:38 PM, Andrew Chernowa...@esilo.com wrote: Eeekks. Found an additional bug. PQsetvalue only allocates the actual tuple if the provided tup_num equals the number of tuples (append) and that slot is NULL. This is wrong. The original idea behind PQsetvalue was you can add tuples in any order and overwrite existing ones. That was by design -- you are only supposed to be able to add a tuple if you pass in exactly the insertion position (which is the same as PQntuples()). If you pass less than that, you will overwrite the value at that position. If you pass greater, you should get an error. Actually, the original idea, as I stated UT, was to allow adding tuples in any order as well as overwriting them. Obviously lost that while trying to get libpqtypes working, which only appends. This is also how the function is documented. That's why you don't have to zero out the tuple slots at all -- the insertion position is always known and you just need to make sure the tuple atts are not allocated more than one time per inserted tuple. Meaning, PQsetvalue needs to work more like pqAddTuple (and the insertion code should probably be abstracted). You need to have insertion point zero'd in either case. Look at the code. It only allocates when appending *AND* the tuple at that index is NULL. If pqAddTuple allocated the table, the tuple slots are uninitialized memory, thus it is very unlikely that the next tuple slot is NULL. It is trivial to make this work the way it was initially intended (which mimics PQgetvalue in that you can fetch values in any order, that was the goal). Are there any preferences? I plan to supply a patch that allows setting values in any order as well as overwriting unless someone speaks up. I fix the docs as well. -- Andrew Chernow eSilo, LLC global backup http://www.esilo.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Error in PQsetvalue
On 6/3/2011 7:14 PM, Merlin Moncure wrote: On Fri, Jun 3, 2011 at 6:22 PM, Andrew Chernowa...@esilo.com wrote: Actually, the original idea, as I stated UT, was to allow adding tuples in any order as well as overwriting them. Obviously lost that while trying to get libpqtypes working, which only appends. Well, that may or not be the case, but that's irrelevant. This has to be backpatched to 9.0 and we can't use a bug to sneak in a behavior change...regardless of what's done, it has to work as documented -- the current behavior isn't broken. If we want PQsetvalue to support creating tuples past the insertion point, that should be proposed for 9.2. Well, not really irrelevant since understanding the rationale behind changes is important. I actually reversed my opinion on this and was preparing a patch that simply did a memset in pqAddTuple. See below for why You need to have insertion point zero'd in either case. Look at the code. It only allocates when appending *AND* the tuple at that index is NULL. If pqAddTuple allocated the table, the tuple slots are uninitialized memory, thus it is very unlikely that the next tuple slot is NULL. I disagree -- I think the fix is a one-liner. line 446: if (tup_num == res-ntups !res-tuples[tup_num]) should just become if (tup_num == res-ntups) also the memset of the tuple slots when the slot array is expanded can be removed. (in addition, the array tuple array expansion should really be abstracted, but that isn't strictly necessary here). All true. This is a cleaner fix to something that was in fact broken ;) You want to apply it? The fact that the tuples were being zero'd plus the NULL check is evidence of the original intent; which is what confused me. I found internal libpqtypes notes related to this change, it actually had to do with producing a result with dead tuples that would cause PQgetvalue and others to crash. Thus, it seemed better to only allow creating a result that is always *valid*. -- Andrew Chernow eSilo, LLC every bit counts http://www.esilo.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Error in PQsetvalue
I disagree -- I think the fix is a one-liner. line 446:
if (tup_num == res-ntups !res-tuples[tup_num])
should just become
if (tup_num == res-ntups)
also the memset of the tuple slots when the slot array is expanded can
be removed. (in addition, the array tuple array expansion should
really be abstracted, but that isn't strictly necessary here).
All true. This is a cleaner fix to something that was in fact broken ;) You want
Attached a patch that fixes the OP's issue. PQsetvalue now uses pqAddTuple to
grow the tuple table and has removed the remnants of an older idea that caused
the bug.
--
Andrew Chernow
eSilo, LLC
every bit counts
http://www.esilo.com/
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 83c5ea3..9f013ed 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -424,28 +424,8 @@ PQsetvalue(PGresult *res, int tup_num, int field_num, char *value, int len)
if (tup_num 0 || tup_num res-ntups)
return FALSE;
- /* need to grow the tuple table? */
- if (res-ntups = res-tupArrSize)
- {
- int n = res-tupArrSize ? res-tupArrSize * 2 : 128;
- PGresAttValue **tups;
-
- if (res-tuples)
- tups = (PGresAttValue **) realloc(res-tuples, n * sizeof(PGresAttValue *));
- else
- tups = (PGresAttValue **) malloc(n * sizeof(PGresAttValue *));
-
- if (!tups)
- return FALSE;
-
- memset(tups + res-tupArrSize, 0,
- (n - res-tupArrSize) * sizeof(PGresAttValue *));
- res-tuples = tups;
- res-tupArrSize = n;
- }
-
- /* need to allocate a new tuple? */
- if (tup_num == res-ntups !res-tuples[tup_num])
+ /* need to allocate a new tuple. */
+ if (tup_num == res-ntups)
{
PGresAttValue *tup;
int i;
@@ -457,6 +437,12 @@ PQsetvalue(PGresult *res, int tup_num, int field_num, char *value, int len)
if (!tup)
return FALSE;
+ if (!pqAddTuple(res, tup))
+ {
+ free(tup);
+ return FALSE;
+ }
+
/* initialize each column to NULL */
for (i = 0; i res-numAttributes; i++)
{
@@ -464,11 +450,12 @@ PQsetvalue(PGresult *res, int tup_num, int field_num, char *value, int len)
tup[i].value = res-null_field;
}
- res-tuples[tup_num] = tup;
- res-ntups++;
+ attval = tup[tup_num][field_num];
+ }
+ else
+ {
+ attval = res-tuples[tup_num][field_num];
}
-
- attval = res-tuples[tup_num][field_num];
/* treat either NULL_LEN or NULL value pointer as a NULL field */
if (len == NULL_LEN || value == NULL)
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Error in PQsetvalue
On 6/3/2011 10:26 PM, Andrew Chernow wrote:
I disagree -- I think the fix is a one-liner. line 446:
if (tup_num == res-ntups !res-tuples[tup_num])
should just become
if (tup_num == res-ntups)
also the memset of the tuple slots when the slot array is expanded can
be removed. (in addition, the array tuple array expansion should
really be abstracted, but that isn't strictly necessary here).
All true. This is a cleaner fix to something that was in fact broken ;) You want
Attached a patch that fixes the OP's issue. PQsetvalue now uses pqAddTuple to
grow the tuple table and has removed the remnants of an older idea that caused
the bug.
Sorry, I attached the wrong patch. Here is the correct one.
--
Andrew Chernow
eSilo, LLC
every bit counts
http://www.esilo.com/
diff --git a/src/interfaces/libpq/fe-exec.c b/src/interfaces/libpq/fe-exec.c
index 83c5ea3..b4a394f 100644
--- a/src/interfaces/libpq/fe-exec.c
+++ b/src/interfaces/libpq/fe-exec.c
@@ -424,28 +424,8 @@ PQsetvalue(PGresult *res, int tup_num, int field_num, char *value, int len)
if (tup_num 0 || tup_num res-ntups)
return FALSE;
- /* need to grow the tuple table? */
- if (res-ntups = res-tupArrSize)
- {
- int n = res-tupArrSize ? res-tupArrSize * 2 : 128;
- PGresAttValue **tups;
-
- if (res-tuples)
- tups = (PGresAttValue **) realloc(res-tuples, n * sizeof(PGresAttValue *));
- else
- tups = (PGresAttValue **) malloc(n * sizeof(PGresAttValue *));
-
- if (!tups)
- return FALSE;
-
- memset(tups + res-tupArrSize, 0,
- (n - res-tupArrSize) * sizeof(PGresAttValue *));
- res-tuples = tups;
- res-tupArrSize = n;
- }
-
- /* need to allocate a new tuple? */
- if (tup_num == res-ntups !res-tuples[tup_num])
+ /* need to allocate a new tuple. */
+ if (tup_num == res-ntups)
{
PGresAttValue *tup;
int i;
@@ -457,15 +437,18 @@ PQsetvalue(PGresult *res, int tup_num, int field_num, char *value, int len)
if (!tup)
return FALSE;
+ if (!pqAddTuple(res, tup))
+ {
+ free(tup);
+ return FALSE;
+ }
+
/* initialize each column to NULL */
for (i = 0; i res-numAttributes; i++)
{
tup[i].len = NULL_LEN;
tup[i].value = res-null_field;
}
-
- res-tuples[tup_num] = tup;
- res-ntups++;
}
attval = res-tuples[tup_num][field_num];
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Error in PQsetvalue
On 6/8/2011 12:03 PM, Tom Lane wrote: Merlin Moncuremmonc...@gmail.com writes: On Wed, Jun 8, 2011 at 10:18 AM, Tom Lanet...@sss.pgh.pa.us wrote: Merlin Moncuremmonc...@gmail.com writes: I went ahead and tested andrew's second patch -- can we get this reviewed and committed? Add it to the upcoming commitfest. It's a client crashing bug in PQsetvalue that goes back to 9.0 :(. I was under the impression that this was extending PQsetvalue to let it be used in previously unsupported ways, ie, to modify a server-returned Well, it was supposed to support that but the implementation is busted (sorry) and core dumps instead. PGresult. That's a feature addition, not a bug fix. I'm not even sure it's a feature addition I approve of. I think serious consideration ought to be given to locking down returned results so PQsetvalue refuses I don't disagree at all, but I do think this is a bit more involved of a change. Several functions like PQmakeEmptyPGresult, PQsetvalue, PQcopyResult (one used by libpqtypes), the PGresult object needs a bool member and probably others things all must be aware of the distinction bwteen client and server generated results. That is a little tricky because both use PQmakeEmptyPGresult that has no argument to indicate that. Since libpqtypes only calls PQcopyResult, you could just set a flag on the result in that function that PQsetvalue uses as a guard. However, this hard codes knowledge about the libpqtypes calling pattern which is rather weak. Maybe it would be better to just apply the patch (since it also removes duplicate code from pqAddTuple in addition to fixing a crash) and update the docs to say this is an unsupported feature, don't do it. If it happens to work forever or just for a while, than so be it. -- Andrew Chernow eSilo, LLC global backup http://www.esilo.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Libpq enhancement
On 6/19/2011 11:04 AM, Jeff Shanab wrote: I am wondering If I am missing something obvious. If not, I have a suggestion for plpgsql. Stored procedures can accept rows. Libpq can receive rows (PQResult). Wouldn’t it be a great interface if PQResult was “bi-directional”? Create a result set on the client then call the database with a command. Perhaps… PQinsert(PQResult,”schema.table”); //iterate thru rows inserting PQupdate(PQResult,”schema.table”); //iterate thru rows updateing PQexec(connection,”scheme.function”,PQResult) //iterate thru rows passing row as arg to stored procedure. Have you looked into libpqtypes? It allows you to pack nested structures/arrays and pass them as query/function parameters. http://pgfoundry.org/projects/libpqtypes/ http://libpqtypes.esilo.com/ (docs) -- Andrew Chernow eSilo, LLC every bit counts http://www.esilo.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] fixing PQsetvalue()
you are creating as you iterate through. This behavior was unnecessary in terms of what libpqtypes and friends needed and may (as Tom suggested) come back to bite us at some point. As it turns out, PQsetvalue's operation on results that weren't created via PQmakeEmptyResult was totally busted because of the bug, so we have a unique opportunity to tinker with libpq here: you could argue that you +1 Exactly at this moment I am thinking about using modifiable (via PQsetvalue) PGresult instead of std::map in my C++ library for store parameters for binding to executing command. I am already designed how to implement it, and I supposed that PQsetvalue is intended to work with any PGresult and not only with those which has been created via PQmakeEmptyResult... So, I am absolutely sure, that PQsetvalue should works with any PGresult. All PGresults are created via PQmakeEmptyPGresult, including libpqtypes. Actually, libpqtypes calls PQcopyResult which calls PQmakeEmptyPGresult. It might be better to say a server result vs. a client result. Currently, PQsetvalue is broken when provided a server generated result. -- Andrew Chernow eSilo, LLC global backup http://www.esilo.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Suggesting a libpq addition
On 12/5/2010 4:22 AM, Marc Balmer wrote: I am suggesting adding a function to libpq: PGresult *PQvexec(PGconn *conn, const char *fmt, ...); It behaves similar to PQexec, but it allows for printf style varargs and does connection re-establishing if the connection fails (it can be discussed if this already to much magic, maybe remove this part). It has been carefully designed to handle memory the right way. We use this since a long time. What do you think? I think it is a wonderful idea. Check out libpqtypes. It has a PQexecf, PQexecvf, PQsendf and PQsendvf. But that is just the beginning http://libpqtypes.esilo.com http://pgfoundry.org/projects/libpqtypes/ -- Andrew Chernow eSilo, LLC every bit counts http://www.esilo.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Suggesting a libpq addition
A varargs version of PQexecParams() would be handy, though. Imagine being able to do: PQexecVParams(SELECT * FROM mytable WHERE foo = $1 AND bar = $2, foovar, barvar); instead of constructing an array for the variables. http://libpqtypes.esilo.com/man3/PQexecf.html -- Andrew Chernow eSilo, LLC every bit counts http://www.esilo.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Suggesting a libpq addition
That would be a *HUGE* piece of software compared the relatively small thing I am suggesting... Sometimes complex and large solutions are required for the simplest of ideas. I believe this is one of those cases. You can't solve the printf style PQexec properly by merely implementing a sprintf wrapper. As for escaping (or not escaping) of string arguments, that can be seen as a bug or a feature. I do not wan't automatic escaping of string arguments in all cases, e.g. I might to construct an SQL statement with dynamic parts WHERE xy or AND a = b. hypothetical example: filter = WHERE name like 'Balmer%'; if (sort == SORT_DESC) sort = ORDER BY name DESCENDING; PQvexec(conn, SELECT name, nr, id FROM address %s%s, filter, sort); So what I am aiming at right now is a PQvexec() function that basically has printf() like semantics, but adds an additional token to the format string (printf uses %s and %b to produce strings.) I am thinking of adding %S and %B, which produce strings that are escaped. This suffers from becoming cryptic over time, see Tom Lane's comments back in 2007 on this (http://archives.postgresql.org/pgsql-hackers/2007-12/msg00362.php). libpqtypes uses the human readable %schema.typename (schema is optional) to specify format specifiers. There is no learning curve or ambiguity, if you want a point than use %point, or %my_type libpqtypes allows you to register aliases (PQregisterSubClasses) so that you can map %text to %s to make it feel more like C.. -- Andrew Chernow eSilo, LLC every bit counts http://www.esilo.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Suggesting a libpq addition
On 12/6/2010 11:40 AM, Dmitriy Igrishin wrote: IMO, it would be better to implement some utility functions to make it easy to construct arrays dynamically for PQexecParams and PQexecPrepared. This seems to me more universal solution and it is useful for both -- high level libpq-libraries authors and for those who like to use libpq directly. Hmm, your idea isn't better, it is identical to what libpqtypes already does :) http://libpqtypes.esilo.com/browse_source.html?file=exec.c We wrap PQexecParams and friends. You are coding libpq. We extended much effort to provide the same result interface (PGresult), including handling composites and arrays. You getf composites and arrays as PGresults; where a composite is a single tuple multiple field result, an array is a multiple tuple single field result and composite arrays are multiple tuples and multiple fields. We've just made a more formal set of utility functions, typically called an API, in an attempt to match the coding standards of the postgresql project. There is no libpq param interface like results, so we added PGparam stuff. This allows you to pack parameters (PQputf) and than execute it. -- Andrew Chernow eSilo, LLC every bit counts http://www.esilo.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Correctly producing array literals for prepared statements
Binary mode had serious limitations, such as portability. What are the other limitations? As far as portability is concerned, we are using it on many different operating systems and architectures without issue. Even our most recent bump to 9.0.1 and 9.0.3 was flawless in regard to libpq/libpqtypes. We do need some support in libpq for constructing and deconstructing arrays (and probably for composites too, although that will be harder, I suspect). [sigh...] -- Andrew Chernow eSilo, LLC global backup http://www.esilo.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Correctly producing array literals for prepared statements
On 2/23/2011 3:06 PM, Peter Geoghegan wrote: On 23 February 2011 15:34, Merlin Moncuremmonc...@gmail.com wrote: You can send nested arrays safely. You just have to be very formal about escaping *everything* both as you get it and as it goes into the container. This is what postgres does on the backend as it sends arrays out the door in text. It might be instructive to see what the server does in terms of escaping. Note that the way this works it's not impossible to see 128+ consecutive backslashes when dealing with arrays of composites. Sounds tedious. It is tedious, which is one reason why libpqtypes went binary. There are some compelling performance reasons as well that affect both client and server. libpqtypes was originally developed to serve a very particular need and wasn't aiming to be general purpose. That came about along the way trying to solve the problem. Personally, PQexec is dead to me as well as text results from a C/C++ app. I see no advantage over libpqtypes in that context. Unless I am missing your ultimate goal, you'd probably get what you want by wrapping libpqtypes. -- Andrew Chernow eSilo, LLC global backup http://www.esilo.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Correctly producing array literals for prepared statements
It's probably fine if you can control both ends. But there is no guarantee of portability, nor does it seem likely to me there ever will be, so I don't find your assertion terribly useful. The fact that it hasn't broken for you doesn't mean it can't or won't be. All true. If you change the protocol, libpqtypes needs to be adjusted. I think that is a very fair statement. It already toggles on server version around a few changes in the past ... like the money data type. So far, since 8.1, the number of changes to the binary protocol has put me to sleep :) The other downside I see is that binary protocols are often a lot harder to debug, but maybe that's just me. Also very true. However, libpqtypes addresses this by abstracting the end user from the binary transformation or preparation. Instead, users are presented with a printf/scanf style interface. PQexecf(conn, select %int4 + %int4, 4, 4) is pretty far removed from the underlying byte swapping, parallel array setup for PQexecParams and other nastiness. But yes, the maintainer of the library must deal with protocol changes and provide backward compatibility. -- Andrew Chernow eSilo, LLC global backup http://www.esilo.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Itanium HP-UX build failure, register stack
I wish we could get some buildfarm coverage for HPUX. I've whined about this in the past, but nobody's ever made an offer to provide suitable platform(s) that I know of. cheers I've got two HP-UX 11 boxes, PA-RISC and IA64. From uname: HP-UX B.11.23 U 9000/785 HP-UX B.11.23 U ia64 They are build farm boxes (which already build postgres) so my guess is they wouldn't require much fiddling. I don't time to help with this, but am more than willing to give you all the access you need to get it going. I also have HP-UX 10.20 on a PA-RISC for the courageous. -- Andrew Chernow eSilo, LLC every bit counts http://www.esilo.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] windows consolidated cleanup
On 4/24/2011 1:29 AM, Andrew Dunstan wrote: The attached patch is intended to clean up a bunch of compiler warnings seen on Windows due to mismatches of signedness or constness, unused variables, redefined macros and a missing prototype. It doesn't clean up all the warnings by any means, but it fixes quite a few. One thing I'm a bit confused about is this type of warning: src\backend\utils\misc\guc-file.c(977): warning C4003: not enough actual parameters for macro 'GUC_yywrap' If someone can suggest a good fix That would be nice. The macro is defined as taking one argument. // guc-file.c line 354 #define GUC_yywrap(n) 1 The macro is overriding the prototype declared at line 627, which has a void argument list (assuming YY_SKIP_YYWRAP is !defined). Since all code references to this do not provide an argument, I'd say the macro is incorrect. -- Andrew Chernow eSilo, LLC every bit counts http://www.esilo.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
