Re: [HACKERS] COPY enhancements
On Mon, 5 Oct 2009, Josh Berkus wrote: I think that this was the original idea but we should probably rollback the error logging if the command has been rolled back. It might be more consistent to use the same hi_options as the copy command. Any idea what would be best? Well, if we're logging to a file, you wouldn't be *able* to roll them back. Also, presumbly, if you abort a COPY because of errors, you probably want to keep the errors around for later analysis. No? Absolutely, that's the whole point of logging to a file in the first place. What needs to happen here is that when one is aborted, you need to make sure that fact is logged, and with enough information (the pid?) to tie it to the COPY that failed. Then someone can crawl the logs to figure out what happened and what data did and didn't get loaded. Ideally you'd want to have that as database table information instead, to allow automated recovery and re-commit in the cases where the error wasn't inherent in the data but instead some other type of database failure. I know this patch is attracting more reviewers lately, is anyone tracking the general architecture of the code yet? Emmanuel's work is tough to review just because there's so many things mixed together, and there's other inputs I think should be considered at the same time while we're all testing in there (such as the COPY patch Andrew Dunstan put together). What I'd like to see is for everything to get broken more into component chunks that can get commited and provide something useful one at a time, because I doubt taskmaster Robert is going to let this one linger around with scope creep for too long before being pushed out to the next CommitFest. It would be nice to have a clear game plan that involves the obvious 3 smaller subpatches that can be commited one at a time as they're ready to focus the work on, so that something might be polished enough for this CF. And, yes, I'm suggesting work only because I now have some time to help with that if the idea seems reasonable to persue. Be glad to set up a public git repo or something to serve as an integration point for dependency merge management and testing that resists bit-rot while splitting things up functionally. -- * Greg Smith gsm...@gregsmith.com http://www.gregsmith.com Baltimore, MD -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Feature Suggestion: PL/Js
by using latest v8 engine from google, is it possible to build PL/Js just like other PL in Postgre? such as PL/PHP what should i learn if i want to build PL/Js? thanks in advance. regards, Kis GB -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Feature Suggestion: PL/Js
On Wed, 2009-10-07 at 15:00 +0700, Kiswono Prayogo wrote: by using latest v8 engine from google, is it possible to build PL/Js just like other PL in Postgre? such as PL/PHP what should i learn if i want to build PL/Js? Start here: http://developer.postgresql.org/pgdocs/postgres/plhandler.html I notice that this has not been updated for the new inline handlers, but that shouldn't stop you. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Review of SQLDA support for ECPG
On Mon, Oct 05, 2009 at 02:23:57PM +0200, Boszormenyi Zoltan wrote: Noah Misch ?rta: I will post a new patch for SQLDA and for all others that need updating. Thanks; that one does apply cleanly. The main test suite acquired no regressions, but I get failures in two tests of the ecpg test suite (make -C src/interfaces/ecpg/test check). No, this is not a real error. I have run into this as well, which is quickly solved if you execute make install before running make check under the ecpg directory. I guess you already have an installed PG tree at the same prefix as where you have pointed the new one with the SQLDA patch applied. Another solution may be to use a different prefix for the SQLDA source tree. This was exactly the problem; with an unoccupied prefix, all the tests do pass. As a side note, with patch 1* but not patch 2, test_informix entered an infinite loop with this error: ERROR: syntax error at or near c at character 15 STATEMENT: fetch forward c Do you mean that you applied all the split-up patches posted for the dynamic cursorname extension? I didn't get this error. What did you do exactly? Having started over from a clean base tree, I can no longer reproduce this. Another operator error, no doubt. All tests now pass here with 1a-1h of Dynamic cursor support for ECPG alone, with SQLDA support for ECPG also applied, and with DESCRIBE [OUTPUT] support for ECPG additionally. I will report on the sqlda patch in more detail on 2009-10-10. The one concern that's clear now is a lack of documentation update. Thank you, nm -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] COPY enhancements
On Wed, 2009-10-07 at 03:17 -0400, Greg Smith wrote: On Mon, 5 Oct 2009, Josh Berkus wrote: Also, presumbly, if you abort a COPY because of errors, you probably want to keep the errors around for later analysis. No? Absolutely, that's the whole point of logging to a file in the first place. Yes, essential. (Not read patch, so some later comments may misunderstand where we're at) What needs to happen here is that when one is aborted, you need to make sure that fact is logged, and with enough information (the pid?) to tie it to the COPY that failed. Then someone can crawl the logs to figure out what happened and what data did and didn't get loaded. Ideally you'd want to have that as database table information instead, to allow automated recovery and re-commit in the cases where the error wasn't inherent in the data but instead some other type of database failure. Adding something to the logs is the wrong place for that IMHO. If we do write a log message it should be a NOTICE not a LOG. If the user specifies an error file then it should be straightforward for them to look directly in that error file afterwards to see if there are rejects. It's typical to only create the error file if rejects exist, to allow a simple if-file-exists test after the COPY. I don't recommend having the server automatically generate an error file name because that will encourage conflicts between multiple users on production systems. If the user wants an errorfile they should specify it, that way they will know the name. It will be best to have the ability to have a specific rejection reason for each row rejected. That way we will be able to tell the difference between uniqueness violation errors, invalid date format on col7, value fails check constraint on col22 etc.. An implicit I got an error on this record isn't enough information and can lead to chaos, since you may be getting more than one error on a record and we need to be able to fix them one at a time. If we can't tell what the error message is then we might think our first valid fix actually failed and start looking for other solutions. There are security implications of writing stuff to an error file, so will the error file option still be available when we do \copy or COPY FROM STDIN, and if so how will it work? With what restrictions? -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] COPY enhancements
On Wed, Oct 7, 2009 at 3:17 AM, Greg Smith gsm...@gregsmith.com wrote: I know this patch is attracting more reviewers lately, is anyone tracking the general architecture of the code yet? Emmanuel's work is tough to review just because there's so many things mixed together, and there's other inputs I think should be considered at the same time while we're all testing in there (such as the COPY patch Andrew Dunstan put together). I hadn't realized this was an issue, but I think it's a good point: a patch that does one thing well is much more likely to get accepted than a patch that does two things well, let alone two things poorly. It's just much easier to review and verify. Or maybe the name of the patch maybe should have tipped me off: COPY enhancements vs. make COPY have feature X. What I'd like to see is for everything to get broken more into component chunks that can get commited and provide something useful one at a time, because I doubt taskmaster Robert is going to let this one linger around with scope creep for too long before being pushed out to the next CommitFest. I'm can't decide whether to feel good or bad about that appelation, so I'm going with both. But in all seriousness if this patch needs substantial reworking (which it sounds like it does) we should postpone it to the next CF; we are quickly running out of days, and it's not fair to reviewers or committers to ask for new reviews of substantially revised code with a only a week to go. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standby on git
On Fri, 2009-10-02 at 10:32 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: I will docuemnt the recommendation to set max_standby_delay = 0 if performing an archive recovery (and explain why). Hmm, not sure if that's such a good piece of advice either. It will mean waiting for queries forever, which probably isn't what you want if you're performing archive recovery. Or maybe it is? Maybe -1? I guess it depends on the situation... I've changed that to -1 now, which was what I meant. 0 is likely to be a very wrong setting during archive recovery. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] COPY enhancements
Hi all, I think there is a misunderstanding about what the current patch is about. The patch includes 2 things: - error logging in a table for bad tuples in a COPY operation (see http://wiki.postgresql.org/wiki/Error_logging_in_COPY for an example; the error message, command and so on are automatically logged) - auto-partitioning in a hierarchy of child table if the COPY targets a parent table. The patch does NOT include: - logging errors into a file (a feature we can add later on (next commit fest?)) I can put the auto-partitioning in a separate patch if that helps but this patch relies on error logging to log possible errors in the routing of tuples. I think that the way to move forward is first to have a basic error logging facility that logs into a database table (like the current patch does) and then add enhancements. I don't think we should try to do the file logging at the same time. If you prefer to postpone the auto-partitioning to the next commit fest, I can strip it from the current patch and re-submit it for the next fest (but it's just 2 isolated methods really easy to review). Emmanuel Robert Haas wrote: On Wed, Oct 7, 2009 at 3:17 AM, Greg Smith gsm...@gregsmith.com wrote: I know this patch is attracting more reviewers lately, is anyone tracking the general architecture of the code yet? Emmanuel's work is tough to review just because there's so many things mixed together, and there's other inputs I think should be considered at the same time while we're all testing in there (such as the COPY patch Andrew Dunstan put together). I hadn't realized this was an issue, but I think it's a good point: a patch that does one thing well is much more likely to get accepted than a patch that does two things well, let alone two things poorly. It's just much easier to review and verify. Or maybe the name of the patch maybe should have tipped me off: COPY enhancements vs. make COPY have feature X. What I'd like to see is for everything to get broken more into component chunks that can get commited and provide something useful one at a time, because I doubt taskmaster Robert is going to let this one linger around with scope creep for too long before being pushed out to the next CommitFest. I'm can't decide whether to feel good or bad about that appelation, so I'm going with both. But in all seriousness if this patch needs substantial reworking (which it sounds like it does) we should postpone it to the next CF; we are quickly running out of days, and it's not fair to reviewers or committers to ask for new reviews of substantially revised code with a only a week to go. ...Robert -- Emmanuel Cecchet Aster Data Systems Web: http://www.asterdata.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] COPY enhancements
Emmanuel Cecchet wrote: If you prefer to postpone the auto-partitioning to the next commit fest, I can strip it from the current patch and re-submit it for the next fest (but it's just 2 isolated methods really easy to review). I certainly think this should be separated out. In general it is not a good idea to roll distinct features together. It complicates both the reviewing process and the discussion. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] COPY enhancements
Simon Riggs si...@2ndquadrant.com writes: It will be best to have the ability to have a specific rejection reason for each row rejected. That way we will be able to tell the difference between uniqueness violation errors, invalid date format on col7, value fails check constraint on col22 etc.. In case that helps, what pgloader does is logging into two files, named after the table name (not scalable to server-side solution): table.rej --- lines it could not load, straight from source file table.rej.log --- errors as given by the server, plus pgloader comment The pgloader comment is necessary for associating each log line to the source file line, as it's operating by dichotomy, the server always report error on line 1. The idea of having two errors file could be kept though, the aim is to be able to fix the setup then COPY again the table.rej file when it happens the errors are not on the file content. Or for loading into another table, with all columns as text or bytea, then clean data from a procedure. Regards, -- dim -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standby 0.2.1
On Tue, 2009-09-22 at 12:53 +0300, Heikki Linnakangas wrote:
It looks like the standby tries to remove XID 4323 from the
known-assigned hash table, but it's not there because it was removed
and set in pg_subtrans by an XLOG_XACT_ASSIGNMENT record earlier. I
guess we should just not throw an error in that case, but is there a
way we could catch that narrow case and still keep the check in
KnownAssignedXidsRemove()? It seems like the check might help catch
other bugs, so I'd rather keep it if possible.
Fix attached.
--
Simon Riggs www.2ndQuadrant.com
diff --git a/src/backend/storage/ipc/procarray.c b/src/backend/storage/ipc/procarray.c
index f6f50be..031a6a2 100644
--- a/src/backend/storage/ipc/procarray.c
+++ b/src/backend/storage/ipc/procarray.c
@@ -2561,8 +2561,15 @@ KnownAssignedXidsRemove(TransactionId xid, bool report_error)
if (!found report_error)
{
- KnownAssignedXidsDisplay(LOG);
- elog(ERROR, cannot remove KnownAssignedXid %u, xid);
+ /*
+ * Check to see whether we have updated subtrans with this xid.
+ * If we did, its OK that it is no longer present in KnownAssignedXids
+ */
+ if (!TransactionIdIsValid(SubTransGetParent(xid)))
+ {
+ KnownAssignedXidsDisplay(LOG);
+ elog(ERROR, cannot remove KnownAssignedXid %u, xid);
+ }
}
}
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] COPY enhancements
On Wed, 2009-10-07 at 15:33 +0200, Dimitri Fontaine wrote: Simon Riggs si...@2ndquadrant.com writes: It will be best to have the ability to have a specific rejection reason for each row rejected. That way we will be able to tell the difference between uniqueness violation errors, invalid date format on col7, value fails check constraint on col22 etc.. In case that helps, what pgloader does is logging into two files, named after the table name (not scalable to server-side solution): table.rej --- lines it could not load, straight from source file table.rej.log --- errors as given by the server, plus pgloader comment The pgloader comment is necessary for associating each log line to the source file line, as it's operating by dichotomy, the server always report error on line 1. The idea of having two errors file could be kept though, the aim is to be able to fix the setup then COPY again the table.rej file when it happens the errors are not on the file content. Or for loading into another table, with all columns as text or bytea, then clean data from a procedure. I like it. -- Simon Riggs www.2ndQuadrant.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] COPY enhancements
On Wed, Oct 7, 2009 at 9:12 AM, Emmanuel Cecchet m...@asterdata.com wrote: Hi all, I think there is a misunderstanding about what the current patch is about. The patch includes 2 things: - error logging in a table for bad tuples in a COPY operation (see http://wiki.postgresql.org/wiki/Error_logging_in_COPY for an example; the error message, command and so on are automatically logged) - auto-partitioning in a hierarchy of child table if the COPY targets a parent table. The patch does NOT include: - logging errors into a file (a feature we can add later on (next commit fest?)) My failure to have read the patch is showing here, but it seems to me that error logging to a table could be problematic: if the transaction aborts, we'll lose the log. If this is in fact a problem, we should be implementing logging to a file (or stdout) FIRST. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Feature Suggestion: PL/Js
Kiswono Prayogo escribió: by using latest v8 engine from google, is it possible to build PL/Js just like other PL in Postgre? such as PL/PHP what should i learn if i want to build PL/Js? I think Josh Tolley has some slides on how we built PL/LOLCODE that could prove useful. BTW I've seen requests for PL/Js so I'm sure it'll be welcome. What license is v8 under? -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standby 0.2.1
On Wed, 2009-09-23 at 12:07 +0300, Heikki Linnakangas wrote:
we need be careful to avoid putting any extra work into the normal
recovery path. Otherwise bugs in hot standby related code can cause
crash recovery to fail.
Re-checked code and found a couple of additional places that needed
tests
if (InHotStandby)
--
Simon Riggs www.2ndQuadrant.com
diff --git a/src/backend/commands/tablespace.c b/src/backend/commands/tablespace.c
index bef0df1..b14d90a 100644
--- a/src/backend/commands/tablespace.c
+++ b/src/backend/commands/tablespace.c
@@ -1345,40 +1345,48 @@ tblspc_redo(XLogRecPtr lsn, XLogRecord *record)
{
VirtualTransactionId *temp_file_users;
- /*
- * Standby users may be currently using this tablespace for
- * for their temporary files. We only care about current
- * users because temp_tablespace parameter will just ignore
- * tablespaces that no longer exist.
- *
- * Ask everybody to cancel their queries immediately so
- * we can ensure no temp files remain and we can remove the
- * tablespace. Nuke the entire site from orbit, it's the only
- * way to be sure.
- *
- * XXX: We could work out the pids of active backends
- * using this tablespace by examining the temp filenames in the
- * directory. We would then convert the pids into VirtualXIDs
- * before attempting to cancel them.
- *
- * We don't wait for commit because drop tablespace is
- * non-transactional.
- */
- temp_file_users = GetConflictingVirtualXIDs(InvalidTransactionId,
- InvalidOid,
- false);
- ResolveRecoveryConflictWithVirtualXIDs(temp_file_users,
- drop tablespace,
- CONFLICT_MODE_ERROR_IF_NOT_IDLE,
- InvalidXLogRecPtr);
+ if (InHotStandby)
+ {
+/*
+ * Standby users may be currently using this tablespace for
+ * for their temporary files. We only care about current
+ * users because temp_tablespace parameter will just ignore
+ * tablespaces that no longer exist.
+ *
+ * Ask everybody to cancel their queries immediately so
+ * we can ensure no temp files remain and we can remove the
+ * tablespace. Nuke the entire site from orbit, it's the only
+ * way to be sure.
+ *
+ * XXX: We could work out the pids of active backends
+ * using this tablespace by examining the temp filenames in the
+ * directory. We would then convert the pids into VirtualXIDs
+ * before attempting to cancel them.
+ *
+ * We don't wait for commit because drop tablespace is
+ * non-transactional.
+ */
+temp_file_users = GetConflictingVirtualXIDs(InvalidTransactionId,
+ InvalidOid,
+ false);
+ResolveRecoveryConflictWithVirtualXIDs(temp_file_users,
+ drop tablespace,
+ CONFLICT_MODE_ERROR_IF_NOT_IDLE,
+ InvalidXLogRecPtr);
- /*
- * If we did recovery processing then hopefully the
- * backends who wrote temp files should have cleaned up and
- * exited by now. So lets recheck before we throw an error.
- * If !process_conflicts then this will just fail again.
- */
- if (!remove_tablespace_directories(xlrec-ts_id, true))
+/*
+ * If we did recovery processing then hopefully the
+ * backends who wrote temp files should have cleaned up and
+ * exited by now. So lets recheck before we throw an error.
+ * If !process_conflicts then this will just fail again.
+ */
+if (!remove_tablespace_directories(xlrec-ts_id, true))
+ ereport(ERROR,
+ (errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
+ errmsg(tablespace %u is not empty,
+ xlrec-ts_id)));
+ }
+ else
ereport(ERROR,
(errcode(ERRCODE_OBJECT_NOT_IN_PREREQUISITE_STATE),
errmsg(tablespace %u is not empty,
diff --git a/src/backend/utils/cache/inval.c b/src/backend/utils/cache/inval.c
index a8cce99..a23c7a0 100644
--- a/src/backend/utils/cache/inval.c
+++ b/src/backend/utils/cache/inval.c
@@ -1820,13 +1820,15 @@ relation_redo(XLogRecPtr lsn, XLogRecord *record)
{
xl_rel_inval *xlrec = (xl_rel_inval *) XLogRecGetData(record);
- relation_redo_inval(xlrec);
+ if (InHotStandby)
+ relation_redo_inval(xlrec);
}
else if (info == XLOG_RELATION_LOCK)
{
xl_rel_lock *xlrec = (xl_rel_lock *) XLogRecGetData(record);
- relation_redo_locks(xlrec, 1);
+ if (InHotStandby)
+ relation_redo_locks(xlrec, 1);
}
else
elog(PANIC, relation_redo: unknown op code %u, info);
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] COPY enhancements
Andrew Dunstan and...@dunslane.net writes: Emmanuel Cecchet wrote: If you prefer to postpone the auto-partitioning to the next commit fest, I can strip it from the current patch and re-submit it for the next fest (but it's just 2 isolated methods really easy to review). I certainly think this should be separated out. In general it is not a good idea to roll distinct features together. It complicates both the reviewing process and the discussion. I think though that Greg was suggesting that we need some more thought about the overall road map. Agglomerating independent features onto COPY one at a time is going to lead to a mess, unless they fit into an overall design plan. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Deadlock error in INSERT statements
Hi, We've a web-based application. We are trying to do concurrency testing, Three person doing same operation on different data. I'm facing the following deadlock error thrown by PostgreSQL: org.postgresql.util.PSQLException: ERROR: deadlock detected Detail: Process 13560 waits for ShareLock on transaction 3147316424; blocked by process 13566. Process 13566 waits for ShareLock on transaction 3147316408; blocked by process 13560. Above error is consistently thrown in INSERT statement. Scenario of the Operation: 1) The following operation is done in parallel a) User A: Doing a Send Mail for 200 email ids b) User B: Doing a Send Mail for 200 email ids c) User C: Doing a Send Mail for 200 email ids 2) For each email id an activity is recorded in table named ACTIVITY. 3) Then mail is sent through smtp server. 4) For User A, User B ACTIVITY is created and Mail is also sent successfully. 5) For User C deadlock error is thrown in INSERT INTO ACTIVITY. 6) Our code rolls back, activity is not created and mail is not sent for User C Additional Information: 1) I have PRIMARY KEY defined in ACTIVITY table. 2) There are FOREIGN KEY references in ACTIVITY table. 3) There are INDEXes in ACTIVITY table Technologies Web Server: Tomcat v6.0.10 Java v1.6.0 Servlet Database: PostgreSQL v8.2.3 Connection Management: pgpool II NOTE: I've seen deadlock errors in UPDATE statement but why it is throwing in INSERT statements. Regards, Gnanam -- View this message in context: http://www.nabble.com/Deadlock-error-in-INSERT-statements-tp25787834p25787834.html Sent from the PostgreSQL - hackers mailing list archive at Nabble.com. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] COPY enhancements
Tom Lane wrote: Andrew Dunstan and...@dunslane.net writes: Emmanuel Cecchet wrote: If you prefer to postpone the auto-partitioning to the next commit fest, I can strip it from the current patch and re-submit it for the next fest (but it's just 2 isolated methods really easy to review). I certainly think this should be separated out. In general it is not a good idea to roll distinct features together. It complicates both the reviewing process and the discussion. I think though that Greg was suggesting that we need some more thought about the overall road map. Agglomerating independent features onto COPY one at a time is going to lead to a mess, unless they fit into an overall design plan. I don't disagree with that. But even if we get a roadmap of some set of features we want to implement, rolling them all together isn't a good way to go. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Feature Suggestion: PL/Js
Alvaro Herrera wrote: Kiswono Prayogo escribió: by using latest v8 engine from google, is it possible to build PL/Js just like other PL in Postgre? such as PL/PHP what should i learn if i want to build PL/Js? I think Josh Tolley has some slides on how we built PL/LOLCODE that could prove useful. BTW I've seen requests for PL/Js so I'm sure it'll be welcome. What license is v8 under? It's a BSD license, but it's a C++ API. While it looks cool, I think SpiderMonkey is possibly a better bet. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Deadlock error in INSERT statements
On Wed, Oct 7, 2009 at 10:34 AM, Gnanam gna...@zoniac.com wrote: Hi, We've a web-based application. We are trying to do concurrency testing, Three person doing same operation on different data. I'm facing the following deadlock error thrown by PostgreSQL: org.postgresql.util.PSQLException: ERROR: deadlock detected Detail: Process 13560 waits for ShareLock on transaction 3147316424; blocked by process 13566. Process 13566 waits for ShareLock on transaction 3147316408; blocked by process 13560. Above error is consistently thrown in INSERT statement. Scenario of the Operation: 1) The following operation is done in parallel a) User A: Doing a Send Mail for 200 email ids b) User B: Doing a Send Mail for 200 email ids c) User C: Doing a Send Mail for 200 email ids 2) For each email id an activity is recorded in table named ACTIVITY. 3) Then mail is sent through smtp server. 4) For User A, User B ACTIVITY is created and Mail is also sent successfully. 5) For User C deadlock error is thrown in INSERT INTO ACTIVITY. 6) Our code rolls back, activity is not created and mail is not sent for User C Additional Information: 1) I have PRIMARY KEY defined in ACTIVITY table. 2) There are FOREIGN KEY references in ACTIVITY table. 3) There are INDEXes in ACTIVITY table Technologies Web Server: Tomcat v6.0.10 Java v1.6.0 Servlet Database: PostgreSQL v8.2.3 Connection Management: pgpool II NOTE: I've seen deadlock errors in UPDATE statement but why it is throwing in INSERT statements. Regards, Gnanam Can you provide the actual queries that you are executing here? Ideally with EXPLAIN ANALYZE output? ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Deadlock error in INSERT statements
On Wed, 2009-10-07 at 16:34 +0200, Gnanam wrote: NOTE: I've seen deadlock errors in UPDATE statement but why it is throwing in INSERT statements. It is because of the foreign key. Inserting a child row will lock the corresponding parent row, and if you insert multiple rows with different parents in the same transaction, and do that in different concurrent transactions but in different order of the parent rows, you can get a deadlock. If you keep in mind that the parent row is locked on the insert of a child row, you will figure out what's happening... BTW, I don't think the hackers list is the right one for this kind of question, better use the general list... Cheers, Csaba. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Feature Suggestion: PL/Js
On Wed, Oct 07, 2009 at 10:22:02AM -0400, Alvaro Herrera wrote: Kiswono Prayogo escribió: by using latest v8 engine from google, is it possible to build PL/Js just like other PL in Postgre? such as PL/PHP what should i learn if i want to build PL/Js? I think Josh Tolley has some slides on how we built PL/LOLCODE that could prove useful. Said slides are available here: http://www.pgcon.org/2009/schedule/events/159.en.html I hope they can be useful. -- Joshua Tolley / eggyknap End Point Corporation http://www.endpoint.com signature.asc Description: Digital signature
Re: [HACKERS] Feature Suggestion: PL/Js
Joshua Tolley escribió: On Wed, Oct 07, 2009 at 10:22:02AM -0400, Alvaro Herrera wrote: Kiswono Prayogo escribió: by using latest v8 engine from google, is it possible to build PL/Js just like other PL in Postgre? such as PL/PHP what should i learn if i want to build PL/Js? I think Josh Tolley has some slides on how we built PL/LOLCODE that could prove useful. Huh, I didn't mean we built but he built! -- Alvaro Herrerahttp://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Feature Suggestion: PL/Js
I think Josh Tolley has some slides on how we built PL/LOLCODE that could prove useful. BTW I've seen requests for PL/Js so I'm sure it'll be welcome. What license is v8 under? the new BSD License http://code.google.com/p/v8/ -- GHUM Harald Massa persuadere et programmare Harald Armin Massa Spielberger Straße 49 70435 Stuttgart 0173/9409607 no fx, no carrier pigeon - %s is too gigantic of an industry to bend to the whims of reality
Re: [HACKERS] COPY enhancements
Robert Haas wrote: On Wed, Oct 7, 2009 at 9:12 AM, Emmanuel Cecchet m...@asterdata.com wrote: Hi all, I think there is a misunderstanding about what the current patch is about. The patch includes 2 things: - error logging in a table for bad tuples in a COPY operation (see http://wiki.postgresql.org/wiki/Error_logging_in_COPY for an example; the error message, command and so on are automatically logged) - auto-partitioning in a hierarchy of child table if the COPY targets a parent table. The patch does NOT include: - logging errors into a file (a feature we can add later on (next commit fest?)) My failure to have read the patch is showing here, but it seems to me that error logging to a table could be problematic: if the transaction aborts, we'll lose the log. If this is in fact a problem, we should be implementing logging to a file (or stdout) FIRST. I don't think this is really a problem. You can always do a SELECT in the error table after the COPY operation if you want to diagnose what happened before the transaction rollbacks. I think that using a file to process the bad tuples is actually less practical than a table if you want to re-insert these tuples in the database. But anyway, the current implementation captures the tuple with all the needed information for logging and delegate the logging to a specific method. If we want to log to a file (or stdout), we can just provide another method to log the already captured info in a file. I think that the file approach is a separate feature but can be easily integrated in the current design. There is just extra work to make sure concurrent COPY commands writing to the same error file are using proper locking. Emmanuel -- Emmanuel Cecchet Aster Data Systems Web: http://www.asterdata.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] COPY enhancements
On Wed, Oct 7, 2009 at 11:39 AM, Emmanuel Cecchet m...@asterdata.com wrote: Robert Haas wrote: On Wed, Oct 7, 2009 at 9:12 AM, Emmanuel Cecchet m...@asterdata.com wrote: Hi all, I think there is a misunderstanding about what the current patch is about. The patch includes 2 things: - error logging in a table for bad tuples in a COPY operation (see http://wiki.postgresql.org/wiki/Error_logging_in_COPY for an example; the error message, command and so on are automatically logged) - auto-partitioning in a hierarchy of child table if the COPY targets a parent table. The patch does NOT include: - logging errors into a file (a feature we can add later on (next commit fest?)) My failure to have read the patch is showing here, but it seems to me that error logging to a table could be problematic: if the transaction aborts, we'll lose the log. If this is in fact a problem, we should be implementing logging to a file (or stdout) FIRST. I don't think this is really a problem. You can always do a SELECT in the error table after the COPY operation if you want to diagnose what happened before the transaction rollbacks. Uhhh if the transaction has aborted, no new commands (including SELECT) will be accepted until you roll back. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] COPY enhancements
The roadmap I would propose for the current list of enhancements to COPY is as follows: 1. new syntax for COPY options (already committed) 2. error logging in a table 3. auto-partitioning (just relies on basic error logging, so can be scheduled anytime after 2) 4. error logging in a file manu Andrew Dunstan wrote: Tom Lane wrote: Andrew Dunstan and...@dunslane.net writes: Emmanuel Cecchet wrote: If you prefer to postpone the auto-partitioning to the next commit fest, I can strip it from the current patch and re-submit it for the next fest (but it's just 2 isolated methods really easy to review). I certainly think this should be separated out. In general it is not a good idea to roll distinct features together. It complicates both the reviewing process and the discussion. I think though that Greg was suggesting that we need some more thought about the overall road map. Agglomerating independent features onto COPY one at a time is going to lead to a mess, unless they fit into an overall design plan. I don't disagree with that. But even if we get a roadmap of some set of features we want to implement, rolling them all together isn't a good way to go. cheers andrew -- Emmanuel Cecchet Aster Data Systems Web: http://www.asterdata.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] COPY enhancements
Robert Haas wrote: On Wed, Oct 7, 2009 at 11:39 AM, Emmanuel Cecchet m...@asterdata.com wrote: Robert Haas wrote: On Wed, Oct 7, 2009 at 9:12 AM, Emmanuel Cecchet m...@asterdata.com wrote: Hi all, I think there is a misunderstanding about what the current patch is about. The patch includes 2 things: - error logging in a table for bad tuples in a COPY operation (see http://wiki.postgresql.org/wiki/Error_logging_in_COPY for an example; the error message, command and so on are automatically logged) - auto-partitioning in a hierarchy of child table if the COPY targets a parent table. The patch does NOT include: - logging errors into a file (a feature we can add later on (next commit fest?)) My failure to have read the patch is showing here, but it seems to me that error logging to a table could be problematic: if the transaction aborts, we'll lose the log. If this is in fact a problem, we should be implementing logging to a file (or stdout) FIRST. I don't think this is really a problem. You can always do a SELECT in the error table after the COPY operation if you want to diagnose what happened before the transaction rollbacks. Uhhh if the transaction has aborted, no new commands (including SELECT) will be accepted until you roll back. You are suggesting then that it is the COPY command that aborts the transaction. That would only happen if you had set a limit on the number of errors that you want to accept in a COPY command (in which case you know that there is something really wrong with your input file and you don't want the server to process that file). manu -- Emmanuel Cecchet Aster Data Systems Web: http://www.asterdata.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] COPY enhancements
Greg Smith wrote: Absolutely, that's the whole point of logging to a file in the first place. What needs to happen here is that when one is aborted, you need to make sure that fact is logged, and with enough information (the pid?) to tie it to the COPY that failed. Then someone can crawl the logs to figure out what happened and what data did and didn't get loaded. Ideally you'd want to have that as database table information instead, to allow automated recovery and re-commit in the cases where the error wasn't inherent in the data but instead some other type of database failure. If one is aborted, nothing gets loaded anyway. Now if you want a separate log of the successful commands, I don't think that should apply just to COPY but to any operation (insert, update, DDL, ...) if you want to build some automatic retry mechanism. But this seems independent from COPY to me. manu -- Emmanuel Cecchet Aster Data Systems Web: http://www.asterdata.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Concurrency testing
Folks, As we move forward, we run into increasingly complex situations under the general rubric of concurrency. What test frameworks are already out there that we can use in our regression test suite? If there aren't any, how might we build one? Cheers, David. -- David Fetter da...@fetter.org http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Feature Suggestion: PL/Js
On Oct 7, 2009, at 7:55 AM, Andrew Dunstan wrote: BTW I've seen requests for PL/Js so I'm sure it'll be welcome. What license is v8 under? It's a BSD license, but it's a C++ API. While it looks cool, I think SpiderMonkey is possibly a better bet. SquirrelFish? http://webkit.org/blog/189/announcing-squirrelfish/ Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] COPY enhancements
On Wed, Oct 7, 2009 at 11:45 AM, Emmanuel Cecchet m...@asterdata.com wrote: You are suggesting then that it is the COPY command that aborts the transaction. That would only happen if you had set a limit on the number of errors that you want to accept in a COPY command (in which case you know that there is something really wrong with your input file and you don't want the server to process that file). But I still want my log even if the COPY bombs out. I want to report the first group of errors back to my user... ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Feature Suggestion: PL/Js
On Wed, Oct 7, 2009 at 5:00 AM, Kiswono Prayogo kisw...@gmail.com wrote: by using latest v8 engine from google, is it possible to build PL/Js just like other PL in Postgre? such as PL/PHP what should i learn if i want to build PL/Js? thanks in advance. regards, Kis GB You also can see this link: http://xen.samason.me.uk/~sam/repos/pljs/README -- Marcelo Costa www.marcelocosta.net - “You can't always get what want”, Doctor House in apology to Mike Jagger
Re: [HACKERS] Concurrency testing
On Wed, Oct 07, 2009 at 09:17:52AM -0700, David Fetter wrote: Folks, As we move forward, we run into increasingly complex situations under the general rubric of concurrency. What test frameworks are already out there that we can use in our regression test suite? If there aren't any, how might we build one? Apparently, this wasn't as crystal clear as I thought. Let's imagine you want to test some odd combination of transactions: T1: SET TRANSACTION ISOLATION SERIALIZABLE; T1: SELECT...FOR UPDATE...; T2: UPDATE... T3: COPY... T1: ... At the moment, we have no way to test this, although with certain Perl modules, it would be pretty trivial. I seem to recall that there were some patches to get psql to help with such things, but they didn't go in. Time to revive them? Cheers, David. -- David Fetter da...@fetter.org http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Feature Suggestion: PL/Js
On Wed, Oct 07, 2009 at 11:29:15AM -0400, Alvaro Herrera wrote: Joshua Tolley escribió: On Wed, Oct 07, 2009 at 10:22:02AM -0400, Alvaro Herrera wrote: Kiswono Prayogo escribió: by using latest v8 engine from google, is it possible to build PL/Js just like other PL in Postgre? such as PL/PHP what should i learn if i want to build PL/Js? I think Josh Tolley has some slides on how we built PL/LOLCODE that could prove useful. Huh, I didn't mean we built but he built! I didn't feel like you were stealing credit. I certainly couldn't have built it, such as it is, without support from -hackers. :) -- Joshua Tolley / eggyknap End Point Corporation http://www.endpoint.com signature.asc Description: Digital signature
Re: [HACKERS] Concurrency testing
David Fetter wrote: I seem to recall that there were some patches to get psql to help with such things, but they didn't go in. Time to revive them? Yeah, the API they implemented wasn't ideal, so there was some discussion that ended up with a specification everyone was happy with, but then nobody got around to implementing it. Feel free to whack that patch and resubmit ... See in the archives around here: http://archives.postgresql.org/message-id/8204.1207689...@sss.pgh.pa.us -- Alvaro Herrerahttp://www.CommandPrompt.com/ The PostgreSQL Company - Command Prompt, Inc. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Scaling up deferred unique checks and the after trigger queue
I've started looking at the following TODO item: Improve deferrable unique constraints for cases with many conflicts and Tom's suggestion that the rows to be checked can be stored in a bitmap, which would become lossy when the number of rows becomes large enough. There is also another TODO item: Add deferred trigger queue file to prevent the trigger queue from exhausting backend memory. I've got some prototype code which attempts to replace all the after-triggers-queue stuff with TID bitmaps (not just for deferred constraint triggers). This would solve the memory-usage problem without resorting to file storage, and makes it easier to then optimise constraint checks by doing a bulk check if the number of rows is large enough. The initial results are encouraging, but I'm still pretty new to a lot of this code, so I wanted to check that this is a sane thing to try to do. For UPDATEs, I'm storing the old tid in the bitmap and relying on its ctid pointer to retrieve the new tuple for the trigger function. AFAICS heap_update() always links the old and new tuples in this way. I'm aware that the triggers on columns patch is going to be a problem for this. I haven't looked at it in any detail, but I suspect that it won't work with a lossy queue, because the information about exactly which rows to trigger on is only known at update time. So maybe I could fall back on a tuplestore, spilling to disk in that case? Thoughts? - Dean -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Concurrency testing
On Oct 7, 2009, at 9:53 AM, David Fetter wrote: At the moment, we have no way to test this, although with certain Perl modules, it would be pretty trivial. No non-core modules necessary. Just use Test::More and file handles opened via pipes to two or more psql sessions. Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Concurrency testing
On Wed, Oct 07, 2009 at 10:28:08AM -0700, David Wheeler wrote: On Oct 7, 2009, at 9:53 AM, David Fetter wrote: At the moment, we have no way to test this, although with certain Perl modules, it would be pretty trivial. No non-core modules necessary. Just use Test::More and file handles opened via pipes to two or more psql sessions. When did Test::More become core? I believe we support back to Perl 5.6 :/ Cheers, David. -- David Fetter da...@fetter.org http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Concurrency testing
On Oct 7, 2009, at 10:38 AM, David Fetter wrote: When did Test::More become core? I believe we support back to Perl 5.6 :/ Module::CoreList says 5.006002, though I would have sworn it was added much earlier than that. You could always use Test.pm, I suppose, which has been in core since 5.00405. Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Concurrency testing
On Wed, 7 Oct 2009, Alvaro Herrera wrote: Yeah, the API they implemented wasn't ideal, so there was some discussion that ended up with a specification everyone was happy with, but then nobody got around to implementing it. I needed something like this and didn't implement those suggestions because I thought the whole idea didn't scale up enough. That's close to the right API to allow more complicated regression tests in psql itself, but I doubted that would hit real complexity level needed to find the good concurrent bugs. The stuff I've been building lately takes how many sessions at once? as an input, and the useful range on that (particularly as we wander further toward multi-core land) is much higher than you'd want to manage one at a time manually. I'd rather see a session simulator program that picks among several types of behaviors and executes them, and then you can throw as many of those as you want into the mix. You can do something like that with pgbench right now if you pass it multiple files, the tricky part is figuring out what regression output you should expect. We had a quick session on more complicated PG testing at PGCon, and it seems many of us who looked at this from multiple perspectives have settled into the idea that the current regression tests should stay as they are, and the more complicated ones are going to need to be in Perl to keep the code complexity managable but still allow more complicated tests. Another example of a complicted test is something that tests pg_dump/pg_restore, which is also hard to shoehorn into pg_regress. I personally would rather see progress made in that area than trying to hack increasingly difficult tests into the existing regression framework, particular given how changes there impact existing committer's daily workflow. The complexity level of test you need to find the juicy concurrent issues is not one you're going to want to run every time you do make check. -- * Greg Smith gsm...@gregsmith.com http://www.gregsmith.com Baltimore, MD -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Concurrency testing
Greg Smith wrote: On Wed, 7 Oct 2009, Alvaro Herrera wrote: Yeah, the API they implemented wasn't ideal, so there was some discussion that ended up with a specification everyone was happy with, but then nobody got around to implementing it. I needed something like this and didn't implement those suggestions because I thought the whole idea didn't scale up enough. That's close to the right API to allow more complicated regression tests in psql itself, but I doubted that would hit real complexity level needed to find the good concurrent bugs. I don't find this a compelling argument against concurrent psql. Sure there are things you can't do with it, but it doesn't mean it's not useful. Are we going to need further tools to find the good concurrent bugs? No doubt. -- Alvaro Herrerahttp://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Performance testing framework..
David Fetter wrote: Folks, As we move forward, we run into increasingly complex situations under the general rubric of concurrency. What test frameworks are already out there that we can use in our regression test suite? If there aren't any, how might we build one? Not entirely on-topic, but I thought I'd chime in: After seeing the results from the hungarian blogger whose name I can't find [1] and quite a bit of discussion in the german usergroup before that I wanted to see how hard it would be to actually implement a framework to automate performance test runs. I mainly used it as a reason to have a look at Moose [2], but since it was quite entertaining so far and I don't see any roadblocks I think I'll follow through. Design Targets: *) OO Design so it's easily pluggable, extensible ( testable ?) *) Standalone CLI mode as well as daemonized benchmark client (think buildfarm) *) Automate all aspects of benchmark operation, starting with a clean slate and (optionally) returning to same *) Extensible Benchmark system, currently thinking about: *) Synthetic cpu/memory/disk benchmarks (for evaluating and verifying the performance baseline of a given system, probably sysbench?) *) pgbench *) Whatever is freely available in the DBT/TPC environment *) As discussed, custom benchmarks testing specific subsystems with high concurrency/throughput, would need a proper tool I presume Currently it doesn't do much more than building branches and creating clusters, but I guess I'll have something rudimentarily usable by mid november. I haven't thought about result aggregation rendering/UI part of the whole thing so far, so if anyone has some ideas in that direction they'd be very much appreciated when the time has come. If I duplicated someones efforts already I'm sorry, I didn't want to raise anybodys hopes before I was sure that I really want to do this ;). best regards, Michael [1] http://suckit.blog.hu/2009/09/26/postgresql_history [2] http://www.iinteractive.com/moose/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Issues for named/mixed function notation patch
Pavel Stehule pavel.steh...@gmail.com writes: Sorry, I'm having trouble understanding what you're driving at here. I think we should just not allow named notation to be combined with VARIADIC, at least for a first version of this feature, either when defining a function or when calling one. Â We can consider relaxing that restriction at a later date if we can agree on what the semantics should be. This is maybe too strict. I thing, so safe version is allow variadic packed parameter with VARIADIC keyword as Jeff proposes. I'm working through this patch now, and I find myself not very satisfied on the question of variadic versus named arguments. What the latest patch actually does is: * completely ignores variadic functions when trying to match a call having any named arguments * does not throw an error for use of the VARIADIC keyword in a call together with named arguments Neither of these behaviors quite seem to me to satisfy the principle of least astonishment, and in combination they definitely do not. It seems to me that there is not anything wrong with using named arguments together with VARIADIC and getting a match to a variadic function. VARIADIC in the argument list essentially turns off the special behavior of variadic functions, and after that you might as well allow either named or positional matching. (I guess if you wanted to be really strict you'd insist that the VARIADIC keyword be attached to the specific named argument that matches the variadic parameter, but I don't mind being a bit lax there.) When VARIADIC is not specified, then I think that silently ignoring variadic functions for a named-argument call is probably reasonable. This can be argued by imagining that the function's implicit array element parameters do not have any names (the variadic array parameter might have a name, but the elements generated from it do not). Since these must be at the right end of the effective parameter list, and we only allow named arguments at the right of the call list, there is no way for the named arguments to match non-variadic named parameters and still have anything matching to the variadic array elements. Therefore a variadic function can never match such a call and ignoring it isn't surprising. Comments? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Writeable CTEs and side effects
I've made progress in implementing writeable CTEs (repo at git://git.postgresql.org/git/writeable_cte.git , branch actually_write) and I've hit a few corner-cases which have lead me to think that we should be handling DML inside CTEs a bit differently. Before I go on implementing this, I'd like to hear your input. 1) WITH t AS (UPDATE foo SET bar = bar+1 RETURNING *) SELECT * FROM t LIMIT 1; What's problematic here is that only 1 row is read from the CTE, meaning also that only one row is updated which, at least how I see it, is not what we want. The CTE should only store one row and return that after it has completely processed the UPDATE statement. 2) WITH t1 AS (UPDATE foo SET bar=bar+1 RETURNING *), t2 AS (UPDATE foo SET bar=bar+1 RETURNING *) SELECT * FROM t1 UNION ALL SELECT * FROM t2; This is probably not the most common scenario, but is still very surprising if you for some reason happen to hit it. Both of the updates actually have the same transaction ID and command ID, so the rows are first updated by t1, but when t2 is processed, it looks at the rows and thinks that it already updated them. 3) WITH t1 AS (UPDATE foo SET bar=bar+1 RETURNING *), t2 AS (UPDATE baz SET bat=bat+1 RETURNING *) VALUES (true); This isn't probably the most common situation either, but I think it's worth looking at; the user wants to update two different tables, but ignore the RETURNING data completely. On IRC, this has been requested multiple times. Even if we wouldn't agree that this feature is useful, it pretty much follows the semantics of example #1. Trying to tackle all of these at once, I've come up with this kind of execution strategy: Before starting the execution of the main plan tree, for every CTE which is a DML query, do the following: 1) Get a new CID 2a) If there are no references to the CTE (example #3), run the DML query to the end but ignore the results of the RETURNING query, or 2b) If there are references, run the DML query to the end but store either as many as rows as you need to to answer the outer query (example #1) or if we can't determine the number of rows we need (most cases, example #2) run the query and store all of its results. Then, if required, get a new CID for the main execution tree and execute it using the data we now have inside the CTEs. This way we can avoid storing useless rows in memory without unexpected behaviour and caveats. Regards, Marko Tiikkaja -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Writeable CTEs and side effects
On Wed, Oct 7, 2009 at 4:08 PM, Marko Tiikkaja marko.tiikk...@cs.helsinki.fi wrote: 1) WITH t AS (UPDATE foo SET bar = bar+1 RETURNING *) SELECT * FROM t LIMIT 1; What's problematic here is that only 1 row is read from the CTE, meaning also that only one row is updated which, at least how I see it, is not what we want. The CTE should only store one row and return that after it has completely processed the UPDATE statement. i'm talking from my ignorance... but AFAIU this should process the whole UPDATE and then form a WorkTable with the results... what the select reads from that results is another thing... -- Atentamente, Jaime Casanova Soporte y capacitación de PostgreSQL Asesoría y desarrollo de sistemas Guayaquil - Ecuador Cel. +59387171157 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Unicode UTF-8 table formatting for psql text output
On Tuesday 06 October 2009 11:35:03 Roger Leigh wrote: On Tue, Oct 06, 2009 at 10:44:27AM +0100, Roger Leigh wrote: On Mon, Oct 05, 2009 at 04:32:08PM -0400, Tom Lane wrote: Roger Leigh rle...@codelibre.net writes: On Sun, Oct 04, 2009 at 11:22:27PM +0300, Peter Eisentraut wrote: Elsewhere in the psql code, notably in mbprint.c, we make the decision on whether to apply certain Unicode-aware processing based on whether the client encoding is UTF8. The same should be done here. There is a patch somewhere in the pipeline that would automatically set the psql client encoding to whatever the locale says, but until that is done, the client encoding should be the sole setting that rules what kind of character set processing is done on the client side. OK, that makes sense to a certain extent. However, the characters used to draw the table lines are not really that related to the client encoding for data sent from the database (IMHO). Huh? The data *in* the table is going to be in the client_encoding, and psql contains no mechanisms that would translate it to something else. Surrounding it with decoration in a different encoding is just a recipe for breakage. Ah, I was under the mistaken assumption that this was iconv()ed or otherwise translated for correct display. In that case, I'll leave the patch as is (using the client encoding for table lines). I've attached an updated copy of the patch (it just removes the now unneeded langinfo.h header). This patch included a bit of code not intended for inclusion (setting of client encoding based on locale), which the attached (and hopefully final!) revision of the patch excludes. Regards, Roger I looked at psql-utf8-table-9.patch. It applies, builds and installs. `gmake check` passes and is not affected by values of LANG or LC_ALL in the environment. HTML and man documentation builds and looks good. The patch doesn't introduce new lint. The psql utility displays UTF8 line art when the client encoding is UTF8, and ASCII line art is displayed otherwise. This can be overridden with `\pset linestyle ASCII` or `\pset linestyle UTF8`. The psql line art is no longer affected by the values of LANG or LC_ALL in the environment. As far as I can tell, everything looks reasonable. Thanks, --bts -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Writeable CTEs and side effects
Jaime Casanova wrote: On Wed, Oct 7, 2009 at 4:08 PM, Marko Tiikkaja marko.tiikk...@cs.helsinki.fi wrote: 1) WITH t AS (UPDATE foo SET bar = bar+1 RETURNING *) SELECT * FROM t LIMIT 1; What's problematic here is that only 1 row is read from the CTE, meaning also that only one row is updated which, at least how I see it, is not what we want. The CTE should only store one row and return that after it has completely processed the UPDATE statement. i'm talking from my ignorance... but AFAIU this should process the whole UPDATE and then form a WorkTable with the results... what the select reads from that results is another thing... Right. This is exactly what I'm trying to do, except I think we could easily optimize this case and store only the first processed row inside the CTE. Regards, Marko Tiikkaja -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Writeable CTEs and side effects
On Wed, Oct 7, 2009 at 4:20 PM, Marko Tiikkaja marko.tiikk...@cs.helsinki.fi wrote: Jaime Casanova wrote: On Wed, Oct 7, 2009 at 4:08 PM, Marko Tiikkaja marko.tiikk...@cs.helsinki.fi wrote: 1) WITH t AS (UPDATE foo SET bar = bar+1 RETURNING *) SELECT * FROM t LIMIT 1; What's problematic here is that only 1 row is read from the CTE, meaning also that only one row is updated which, at least how I see it, is not what we want. The CTE should only store one row and return that after it has completely processed the UPDATE statement. i'm talking from my ignorance... but AFAIU this should process the whole UPDATE and then form a WorkTable with the results... what the select reads from that results is another thing... Right. This is exactly what I'm trying to do, except I think we could easily optimize this case and store only the first processed row inside the CTE. why? we don't should be thinking in optimizations in this phase -- Atentamente, Jaime Casanova Soporte y capacitación de PostgreSQL Asesoría y desarrollo de sistemas Guayaquil - Ecuador Cel. +59387171157 -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Writeable CTEs and side effects
Jaime Casanova wrote: On Wed, Oct 7, 2009 at 4:20 PM, Marko Tiikkaja marko.tiikk...@cs.helsinki.fi wrote: Right. This is exactly what I'm trying to do, except I think we could easily optimize this case and store only the first processed row inside the CTE. why? we don't should be thinking in optimizations in this phase Maybe you're right. This probably isn't a very common case any way. Regards, Marko Tiikkaja -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Issues for named/mixed function notation patch
2009/10/7 Tom Lane t...@sss.pgh.pa.us: Pavel Stehule pavel.steh...@gmail.com writes: Sorry, I'm having trouble understanding what you're driving at here. I think we should just not allow named notation to be combined with VARIADIC, at least for a first version of this feature, either when defining a function or when calling one. We can consider relaxing that restriction at a later date if we can agree on what the semantics should be. This is maybe too strict. I thing, so safe version is allow variadic packed parameter with VARIADIC keyword as Jeff proposes. I'm working through this patch now, and I find myself not very satisfied on the question of variadic versus named arguments. What the latest patch actually does is: * completely ignores variadic functions when trying to match a call having any named arguments * does not throw an error for use of the VARIADIC keyword in a call together with named arguments Neither of these behaviors quite seem to me to satisfy the principle of least astonishment, and in combination they definitely do not. It seems to me that there is not anything wrong with using named arguments together with VARIADIC and getting a match to a variadic function. VARIADIC in the argument list essentially turns off the special behavior of variadic functions, and after that you might as well allow either named or positional matching. (I guess if you wanted to be really strict you'd insist that the VARIADIC keyword be attached to the specific named argument that matches the variadic parameter, but I don't mind being a bit lax there.) When VARIADIC is not specified, then I think that silently ignoring variadic functions for a named-argument call is probably reasonable. This can be argued by imagining that the function's implicit array element parameters do not have any names (the variadic array parameter might have a name, but the elements generated from it do not). Since these must be at the right end of the effective parameter list, and we only allow named arguments at the right of the call list, there is no way for the named arguments to match non-variadic named parameters and still have anything matching to the variadic array elements. Therefore a variadic function can never match such a call and ignoring it isn't surprising. It's same as my origin ideas, much better formulated. It is ok for me. Pavel Comments? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Issues for named/mixed function notation patch
On Wed, 2009-10-07 at 16:58 -0400, Tom Lane wrote: * completely ignores variadic functions when trying to match a call having any named arguments * does not throw an error for use of the VARIADIC keyword in a call together with named arguments Neither of these behaviors quite seem to me to satisfy the principle of least astonishment, and in combination they definitely do not. I agree that the combination is wrong, and we should either allow that call notation or throw a useful error message when it's attempted. It seems to me that there is not anything wrong with using named arguments together with VARIADIC and getting a match to a variadic function. The general feeling was that we should only support the most obvious call notations so we wouldn't have backwards compatibility problems if we tried to change it later. If we allow calling a variadic function using named notation, the VARIADIC keyword is not strictly necessary, but I think we should require it. It seems strange without it. Pavel indicated that there may be some implementation difficulty in requiring the VARIADIC keyword when calling a variadic function using named notation: http://archives.postgresql.org/pgsql-hackers/2009-09/msg01792.php and that just kind of pushed the idea from maybe that's OK to probably not a good idea right now. Robert Haas weighed in here: http://archives.postgresql.org/pgsql-hackers/2009-09/msg01732.php Its fine with me to allow it, assuming there's a reasonable way to implement it, and assuming that the VARIADIC keyword is required. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Issues for named/mixed function notation patch
Jeff Davis pg...@j-davis.com writes: If we allow calling a variadic function using named notation, the VARIADIC keyword is not strictly necessary, but I think we should require it. It seems strange without it. Yeah. My first thought was to just remove the check in FuncnameGetCandidates, which would have the effect of matching with or without VARIADIC. It would be self-consistent but probably surprising. But it should just be a small tweak to match only with VARIADIC. Pavel indicated that there may be some implementation difficulty in requiring the VARIADIC keyword when calling a variadic function using named notation: I think what he was considering was the question of insisting that the VARIADIC keyword be attached to the named argument that actually matches the VARIADIC parameter. I think we could do it, but it might be a bit of a wart. I notice that right now, an unnecessary VARIADIC keyword in a regular positional call does not cause an error, it's just ignored --- so we're already being a bit lax with it. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Issues for named/mixed function notation patch
On Wed, 2009-10-07 at 23:32 +0200, Pavel Stehule wrote: It's same as my origin ideas, much better formulated. It is ok for me. You indicated that there may be some implementation difficulty if the VARIADIC keyword is required for calling using named notation: http://archives.postgresql.org/pgsql-hackers/2009-09/msg01792.php Do you think it would be reasonable to implement? Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Triggers on columns
2009/10/7 Peter Eisentraut pete...@gmx.net: On Sun, 2009-10-04 at 22:07 -0400, Tom Lane wrote: In short: while I haven't looked at the patch, I think Peter may be steering you in the wrong direction. In cases where you do have related functions, I suggest having SQL-callable V1 functions that absorb their arguments in this style, and then have them call a common subroutine that's a plain C function. Yeah, that's what he did. So forget what I said. :-) What's the current status of this patch in the commitfest process? I see the last version posted by the author was on 14 Sep and there's been some discussion since then, but no new patch version has surfaced. Are we waiting on a new version of the patch, or is there more review of the existing version yet to be done? The status of the patch on the commitfest app is still Needs Review, but I'm not sure whether that is accurate. Cheers, BJ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Issues for named/mixed function notation patch
On Wed, 2009-10-07 at 17:45 -0400, Tom Lane wrote: I think what he was considering was the question of insisting that the VARIADIC keyword be attached to the named argument that actually matches the VARIADIC parameter. I think we could do it, but it might be a bit of a wart. I notice that right now, an unnecessary VARIADIC keyword in a regular positional call does not cause an error, it's just ignored --- so we're already being a bit lax with it. From a semantic standpoint, I lean towards requiring the VARIADIC keyword consistently between named and positional notation. It seems strange to me if we have a situation where changing the call: foo(a, b, VARIADIC c) to be more explicit by using named call notation: foo(a AS x, b AS y, VARIADIC c AS z) is less correct in the sense that the VARIADIC keyword goes from required to ignored. Also, requiring VARIADIC seems to guard us better against future changes, which seemed like a concern before. I don't have a strong opinion or a specific problem with making VARIADIC optional, so it's OK with me. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Issues for named/mixed function notation patch
Jeff Davis pg...@j-davis.com writes: On Wed, 2009-10-07 at 17:45 -0400, Tom Lane wrote: I think what he was considering was the question of insisting that the VARIADIC keyword be attached to the named argument that actually matches the VARIADIC parameter. It seems strange to me if we have a situation where changing the call: foo(a, b, VARIADIC c) to be more explicit by using named call notation: foo(a AS x, b AS y, VARIADIC c AS z) is less correct in the sense that the VARIADIC keyword goes from required to ignored. No, that's not what I'm driving at. The small change that I've got in mind would require you to say VARIADIC, but it would allow the function to match both the above call and foo(a AS x, c AS z, VARIADIC b AS y) when really z is the variadic parameter in this case. I'm not sure if this would bother anyone or not. It seems impossible that a function could ever have more than one variadic parameter, so there's not really any ambiguity from maintaining the syntactic rule that the VARIADIC keyword is at the end even when the variadic argument isn't, but it might look a bit odd. What I *don't* want to do is fix this by allowing/requiring foo(a AS x, VARIADIC c AS z, b AS y) because it would be a bigger change in the grammar output structure than seems warranted. We could possibly have VARIADIC throw an error if the named argument that matches to the variadic parameter isn't the last one, but I'm not sure that that's important rather than just pedantry. People would probably tend to write it that way anyway. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Re: [COMMITTERS] pgsql: Make it possibly to specify GUC params per user and per database.
Alvaro Herrera wrote: Log Message: --- psql has gained a \drds command to display the settings. And, of course, I only noticed after the commit that it's not listed in \?. -- Alvaro Herrerahttp://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Issues for named/mixed function notation patch
2009/10/8 Tom Lane t...@sss.pgh.pa.us: Jeff Davis pg...@j-davis.com writes: On Wed, 2009-10-07 at 17:45 -0400, Tom Lane wrote: I think what he was considering was the question of insisting that the VARIADIC keyword be attached to the named argument that actually matches the VARIADIC parameter. It seems strange to me if we have a situation where changing the call: foo(a, b, VARIADIC c) to be more explicit by using named call notation: foo(a AS x, b AS y, VARIADIC c AS z) is less correct in the sense that the VARIADIC keyword goes from required to ignored. No, that's not what I'm driving at. The small change that I've got in mind would require you to say VARIADIC, but it would allow the function to match both the above call and foo(a AS x, c AS z, VARIADIC b AS y) when really z is the variadic parameter in this case. I'm not sure if this would bother anyone or not. It seems impossible that a function could ever have more than one variadic parameter, so there's not really any ambiguity from maintaining the syntactic rule that the VARIADIC keyword is at the end even when the variadic argument isn't, but it might look a bit odd. What I *don't* want to do is fix this by allowing/requiring foo(a AS x, VARIADIC c AS z, b AS y) because it would be a bigger change in the grammar output structure than seems warranted. We could possibly have VARIADIC throw an error if the named argument that matches to the variadic parameter isn't the last one, but I'm not sure that that's important rather than just pedantry. People would probably tend to write it that way anyway. It is just pedantry. Position in named notation isn't important, and there are no reason, why we VARIADIC should be last. Pavel regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Issues for named/mixed function notation patch
2009/10/7 Jeff Davis pg...@j-davis.com: On Wed, 2009-10-07 at 23:32 +0200, Pavel Stehule wrote: It's same as my origin ideas, much better formulated. It is ok for me. You indicated that there may be some implementation difficulty if the VARIADIC keyword is required for calling using named notation: http://archives.postgresql.org/pgsql-hackers/2009-09/msg01792.php Do you think it would be reasonable to implement? I thing, so this is possible. But it needs some instructions more. I would not add some unnecessary checks. It needs one cycle over parameters more (and one array). * check if last variadic parameter isn't default * check if last variadic parameter has flag VARIADIC * check if there are not any other parameter with VARIADIC flag * some correction in gram.y (procedural code), that allows VARIADIC in any position when named notation is active. Pavel Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Issues for named/mixed function notation patch
On Wed, 2009-10-07 at 18:17 -0400, Tom Lane wrote: No, that's not what I'm driving at. The small change that I've got in mind would require you to say VARIADIC, but it would allow the function to match both the above call and foo(a AS x, c AS z, VARIADIC b AS y) when really z is the variadic parameter in this case. I'm not sure if this would bother anyone or not. It seems impossible that a function could ever have more than one variadic parameter, so there's not really any ambiguity from maintaining the syntactic rule that the VARIADIC keyword is at the end even when the variadic argument isn't, but it might look a bit odd. I'm worried about allowing such strange notation. Someone might have a new idea later that conflicts with it, and then we have a backwards-compatibility problem. What I *don't* want to do is fix this by allowing/requiring foo(a AS x, VARIADIC c AS z, b AS y) because it would be a bigger change in the grammar output structure than seems warranted. If it's the right thing to do (or might be the right thing to do), someone will want to do that later, and that would be incompatible with the: foo(a AS x, c AS z, VARIADIC b AS y) notation (where z is the variadic parameter). We could possibly have VARIADIC throw an error if the named argument that matches to the variadic parameter isn't the last one, but I'm not sure that that's important rather than just pedantry. I would prefer such a restriction if it's reasonable to do. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Issues for named/mixed function notation patch
On Wed, Oct 7, 2009 at 5:45 PM, Tom Lane t...@sss.pgh.pa.us wrote: I think what he was considering was the question of insisting that the VARIADIC keyword be attached to the named argument that actually matches the VARIADIC parameter. I think we could do it, but it might be a bit of a wart. I notice that right now, an unnecessary VARIADIC keyword in a regular positional call does not cause an error, it's just ignored --- so we're already being a bit lax with it. I'd be more inclined to to tighten up the place where we're currently being lax than to treat additional situations in a similarly lax manner. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Issues for named/mixed function notation patch
Jeff Davis pg...@j-davis.com writes: We could possibly have VARIADIC throw an error if the named argument that matches to the variadic parameter isn't the last one, but I'm not sure that that's important rather than just pedantry. I would prefer such a restriction if it's reasonable to do. [ pokes around ... ] It seems to only take a few more lines, so will do. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] COPY enhancements
On Wed, 7 Oct 2009, Emmanuel Cecchet wrote: I think there is a misunderstanding about what the current patch is about...the patch does NOT include logging errors into a file (a feature we can add later on (next commit fest?)) I understand that (as one of the few people who has read the patch it seems), but as you can might guess from the feedback here that's not the way I expect your patch is actually going to be handled. Something that logs only to a table without the interface for the file output at least specified isn't the sort of thing that this group tends to go along with very well. Since as it is we haven't even nailed down how the file output is even going to look or work yet, the table logging isn't very likely to go in unless it's at least clear that the future plans there will cleanly stack on top. That's what people are alluding to mentioning the whole roadmap concept for all this. I get the impression you were hoping to get another chunk of this commited on this round. I would guess that realistically it's going to take at least a few more weeks for the rest of this to get nailed down, and that a really solid feature should be ready by the next CF instead. You should actually be proud of the progress you spurred on the COPY options stuff that's in there now, that idea has been floating around for a while now but until your patch there wasn't any momentum behind doing something about it. The problem you're facing now is that so many people have been thinking about this particular area for so long without any code to talk about that you're now stuck with all that pent up uncoded design to clear. I can put the auto-partitioning in a separate patch if that helps but this patch relies on error logging to log possible errors in the routing of tuples. Understood. I know you've gotten some other coding feedback here, and you've been very good about taking that and producing updated versions which is appreciated. I would recommend that the next time you resubmit this, if you could break the logging and partitioning patches into a pair that depend on one another, that would make life easier for everyone else (albeit probably a harder for you). At that point we should be set to have some others take over some of that tweaking, with myself, Selena, and Jeff all interested and capable of helping out here. If you prefer to postpone the auto-partitioning to the next commit fest, I can strip it from the current patch and re-submit it for the next fest (but it's just 2 isolated methods really easy to review). That was one of points I was trying to make--that would be an easier to review by itself, but as it is people interested in it can't consider it without also staring at the logging stuff. And people who are focusing on the logging bits find it distracting, so nobody is really happy with the current combined patch. -- * Greg Smith gsm...@gregsmith.com http://www.gregsmith.com Baltimore, MD -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] COPY enhancements
On Wed, 7 Oct 2009, Robert Haas wrote: On Wed, Oct 7, 2009 at 3:17 AM, Greg Smith gsm...@gregsmith.com wrote: I doubt taskmaster Robert is going to let this one linger around with scope creep for too long before being pushed out to the next CommitFest. I'm can't decide whether to feel good or bad about that appelation, so I'm going with both. That was a compliment on your project management skills. Keeping the CF work moving forward steadily is both unglamorous and extremely valuable, and I don't think anyone else even understands why you've volunteered to handle so much of it. But I know I appreciate it. -- * Greg Smith gsm...@gregsmith.com http://www.gregsmith.com Baltimore, MD -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Feature Suggestion: PL/Js
Said slides are available here: http://www.pgcon.org/2009/schedule/events/159.en.html I hope they can be useful. -- Joshua Tolley / eggyknap ok i will read it, thanks ^^ On Wed, Oct 7, 2009 at 11:18 PM, David E. Wheeler da...@kineticode.com wrote: On Oct 7, 2009, at 7:55 AM, Andrew Dunstan wrote: BTW I've seen requests for PL/Js so I'm sure it'll be welcome. What license is v8 under? It's a BSD license, but it's a C++ API. While it looks cool, I think SpiderMonkey is possibly a better bet. SquirrelFish? http://webkit.org/blog/189/announcing-squirrelfish/ Best, David i guess v8 still the best javascript interpreter http://waynepan.com/2008/09/02/v8-tracemonkey-squirrelfish-ie8-benchmarks/ nitro from apple safari would be the second best.. http://www.computerworld.com/s/article/9128638/Safari_4_rivals_Google_Chrome_in_JavaScript_race?intsrc=news_ts_head but possibly that benchmark is not really valid because it also benchmark the rendering.. On Wed, Oct 7, 2009 at 11:38 PM, Marcelo Costa marcelojsco...@gmail.com wrote: On Wed, Oct 7, 2009 at 5:00 AM, Kiswono Prayogo kisw...@gmail.com wrote: by using latest v8 engine from google, is it possible to build PL/Js just like other PL in Postgre? such as PL/PHP what should i learn if i want to build PL/Js? thanks in advance. regards, Kis GB You also can see this link: http://xen.samason.me.uk/~sam/repos/pljs/README -- Marcelo Costa www.marcelocosta.net - “You can't always get what want”, Doctor House in apology to Mike Jagger hmm.. someone has built it first, i'll try it in the moment ^^ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Concurrency testing
On Wed, 7 Oct 2009, Alvaro Herrera wrote: I don't find this a compelling argument against concurrent psql. Sure there are things you can't do with it, but it doesn't mean it's not useful. Are we going to need further tools to find the good concurrent bugs? No doubt. Don't get me wrong, I wasn't arguing against concurrent psql being useful. Certainly it is. I was just suggesting that the scale of issues it can be useful for is still pretty limited, and that accordingly I found my time better spent working on a higher-level solution that didn't need C-psql anyway. Whether C-psql is sufficient for what David had in mind I can't say. -- * Greg Smith gsm...@gregsmith.com http://www.gregsmith.com Baltimore, MD -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Feature Suggestion: PL/Js
On Thu, 2009-10-08 at 07:06 +0700, Kiswono Prayogo wrote: Said slides are available here: http://www.pgcon.org/2009/schedule/events/159.en.html I hope they can be useful. -- Joshua Tolley / eggyknap ok i will read it, thanks ^^ And video: http://www.vimeo.com/3728119 Joshua D. Drake -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 503.667.4564 Consulting, Training, Support, Custom Development, Engineering If the world pushes look it in the eye and GRR. Then push back harder. - Salamander -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Concurrency testing
On Wed, Oct 07, 2009 at 08:06:50PM -0400, Greg Smith wrote: On Wed, 7 Oct 2009, Alvaro Herrera wrote: I don't find this a compelling argument against concurrent psql. Sure there are things you can't do with it, but it doesn't mean it's not useful. Are we going to need further tools to find the good concurrent bugs? No doubt. Don't get me wrong, I wasn't arguing against concurrent psql being useful. Certainly it is. I was just suggesting that the scale of issues it can be useful for is still pretty limited, and that accordingly I found my time better spent working on a higher-level solution that didn't need C-psql anyway. Whether C-psql is sufficient for what David had in mind I can't say. The kind of stuff I have in mind is regression tests. As with coverage and Dtrace, they might not be the ones that get run by default, but right now, we have no real way in the regression tests to test concurrency issues at all, as far as I know, so it would be good to have some way to catch this stuff. Cheers, David. -- David Fetter da...@fetter.org http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Feature Suggestion: PL/Js
Kiswono Prayogo escribió: i guess v8 still the best javascript interpreter http://waynepan.com/2008/09/02/v8-tracemonkey-squirrelfish-ie8-benchmarks/ nitro from apple safari would be the second best.. http://www.computerworld.com/s/article/9128638/Safari_4_rivals_Google_Chrome_in_JavaScript_race?intsrc=news_ts_head but possibly that benchmark is not really valid because it also benchmark the rendering.. Performance is not the only useful measure. Embeddability in Postgres is very important, and it being C++ is not a good starting point. -- Alvaro Herrerahttp://www.CommandPrompt.com/ PostgreSQL Replication, Consulting, Custom Development, 24x7 support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Concurrency testing
On Wed, Oct 7, 2009 at 10:38 AM, David Fetter da...@fetter.org wrote: On Wed, Oct 07, 2009 at 10:28:08AM -0700, David Wheeler wrote: On Oct 7, 2009, at 9:53 AM, David Fetter wrote: At the moment, we have no way to test this, although with certain Perl modules, it would be pretty trivial. No non-core modules necessary. Jus use Test::More and file handles opened via pipes to two or more psql sessions. I'd much rather live without Test::More and use DBD::Pg, then have Test::More but need to open pipes to psql to talk to the database, rather than using DBI to do it. But I guess we would need to worry about whether we can make DBD::Pg work with the installation being tested, rather than finding some other install. When did Test::More become core? I believe we support back to Perl 5.6 :/ Do we need to restrict ourselves to core? Developers already need flex and bison, which aren't needed when installing from the tarball. Couldn't we also have make dev-check that has higher requirements than make check does, but does a more thorough job? Jeff -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Performance testing framework..
On Wed, 7 Oct 2009, Michael Renner wrote: I haven't thought about result aggregation rendering/UI part of the whole thing so far, so if anyone has some ideas in that direction they'd be very much appreciated when the time has come. What I did in pgbench-tools (now available at http://git.postgresql.org/ ) is just dump all the individual run results, tagged by the name of test and similar key information, into a database. That lets people aggregate however they want using SQL, and cleanly separates out the data from the UI. My current rendering approach just pulls data from there into columnar text files I feed into gnuplot, but you could easily render with pretty much anything else instead just by tweaking how you pull things from the database. All of the dbt-X tools (also in that git repo) instead use simple columnar text files (and their memory resident equivalent) to hold everything, feeding the data through the appropriate Perl libraries to compute stats and render things. Since it drives me crazy to manipulate tables of data that could go into a database in a scripting language, I never bought into that approach. But it's completely viable too for a standalone tool. I'm not aware of anyone working on the job management side of the performance farm yet, so I don't think you duplicated anybody else's work. Mark Wong (who manages the dbt stuff) and I have both been doing work on the benchmark results collection and organization side of things, but basically one test at a time. The specific piece I've been working on lately is spawning off system monitoring daemons to collect information during the test, I think I'm on my 3rd generation of trying to get a solution I'm happy with to that problem. -- * Greg Smith gsm...@gregsmith.com http://www.gregsmith.com Baltimore, MD -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Concurrency testing
On Oct 7, 2009, at 5:18 PM, Jeff Janes wrote: I'd much rather live without Test::More and use DBD::Pg, then have Test::More but need to open pipes to psql to talk to the database, rather than using DBI to do it. But I guess we would need to worry about whether we can make DBD::Pg work with the installation being tested, rather than finding some other install. The test architecture depends on Perl, but not on the DBI. I don't think that Andrew wants to add any dependencies. Therefore we'd need to use file handles. That's not to say that we couldn't write a nice little interface for it such that the implementation could later change. Do we need to restrict ourselves to core? Developers already need flex and bison, which aren't needed when installing from the tarball. Couldn't we also have make dev-check that has higher requirements than make check does, but does a more thorough job? flex and bison are not Perl modules. Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Performance testing framework..
Greg Smith wrote: On Wed, 7 Oct 2009, Michael Renner wrote: I haven't thought about result aggregation rendering/UI part of the whole thing so far, so if anyone has some ideas in that direction they'd be very much appreciated when the time has come. What I did in pgbench-tools (now available at http://git.postgresql.org/ Thanks, I'll look into it! I think the nicest solution would be something in the liking of Sun's Analytics [1] framework, especially when you've got large amounts of historic data to correlate. But this is probably a completely different beast to implement, though nice Javascript rendering libraries become commonplace these days. We'll see... I'm not aware of anyone working on the job management side of the performance farm yet, so I don't think you duplicated anybody else's work. Good to hear :) things, but basically one test at a time. The specific piece I've been working on lately is spawning off system monitoring daemons to collect information during the test, I think I'm on my 3rd generation of trying to get a solution I'm happy with to that problem. Ahh, that'd be also nice to have to see how ressource consumption differs between different runs branches. This is probably highly platform dependent though... Michael [1] http://ctistrategy.com/2008/12/17/sun-storage-7000-analytics-overview/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Concurrency testing
On Wed, Oct 7, 2009 at 8:33 PM, David E. Wheeler da...@kineticode.com wrote: Do we need to restrict ourselves to core? Developers already need flex and bison, which aren't needed when installing from the tarball. Couldn't we also have make dev-check that has higher requirements than make check does, but does a more thorough job? flex and bison are not Perl modules. True, but so what? I don't really see what's wrong with using Perl modules that are likely to be installed most places and easy to obtain where not, if it makes writing a test framework much easier. But I also think that we should not get bogged down on exactly which tools to use - it seems to me the first thing is to find someone who is willing to do the work. If someone makes an AWESOME test suite that uses a module which is a little too adventurous, we can probably find a way of adjusting it after the fact so as to remove the dependency (I fancy myself fairly good at this sort of thing, where Perl is concerned). But if we argue about tools now, we're just going to discourage anyone from taking a stab at it. Just MHO, of course. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] COPY enhancements
On Wed, Oct 7, 2009 at 7:52 PM, Greg Smith gsm...@gregsmith.com wrote: On Wed, 7 Oct 2009, Robert Haas wrote: On Wed, Oct 7, 2009 at 3:17 AM, Greg Smith gsm...@gregsmith.com wrote: I doubt taskmaster Robert is going to let this one linger around with scope creep for too long before being pushed out to the next CommitFest. I'm can't decide whether to feel good or bad about that appelation, so I'm going with both. That was a compliment on your project management skills. Keeping the CF work moving forward steadily is both unglamorous and extremely valuable, and I don't think anyone else even understands why you've volunteered to handle so much of it. But I know I appreciate it. Thanks. I'm not sure I completely understand it, either. I'm sort of hoping someone else will step up to the plate, but in the absence of such a person I kind of hate to leave it hanging. Some defective part of my brain enjoys seeing things run smoothly more than it enjoys being lazy. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] postgres 8.3.8 and Solaris 10_x86 64 bit problems?
So I compiled postgres with Solaris 10 and have problems running it. # ./pg_ctl ld.so.1: pg_ctl: fatal: relocation error: R_AMD64_32: file /usr/local/postgres64/lib/libpq.so.5: symbol (unknown): value 0xfd7fff1cf210 does not fit Killed # ldd pg_ctl libpq.so.5 =/usr/local/postgres64/lib/libpq.so.5 libm.so.2 = /usr/lib/64/libm.so.2 libxml2.so.2 = /usr/lib/64/libxml2.so.2 libz.so.1 = /usr/lib/64/libz.so.1 libreadline.so.6 = /usr/local/lib/libreadline.so.6 libcurses.so.1 =/usr/lib/64/libcurses.so.1 librt.so.1 =/usr/lib/64/librt.so.1 libsocket.so.1 =/usr/lib/64/libsocket.so.1 libc.so.1 = /usr/lib/64/libc.so.1 libpthread.so.1 = /usr/lib/64/libpthread.so.1 libnsl.so.1 = /lib/64/libnsl.so.1 libgcc_s.so.1 = /usr/sfw/lib/amd64/libgcc_s.so.1 libaio.so.1 = /lib/64/libaio.so.1 libmd.so.1 =/lib/64/libmd.so.1 libmp.so.2 =/lib/64/libmp.so.2 libscf.so.1 = /lib/64/libscf.so.1 libdoor.so.1 = /lib/64/libdoor.so.1 libuutil.so.1 = /lib/64/libuutil.so.1 libgen.so.1 = /lib/64/libgen.so.1 # file /usr/local/postgres64/lib/libpq.so.5 /usr/local/postgres64/lib/libpq.so.5: ELF 64-bit LSB dynamic lib AMD64 Version 1 [SSE CMOV], dynamically linked, not stripped What am I missing??? Here's my environment. Solaris 10 x86_64 with postgres 8.3.8 and openssl 98k using gcc version 3.4.3 (csl-sol210-3_4-branch+sol_rpath) , sunstudio12.1 and GNU Make 3.80 I've even monkied with LD_LIBRARY_PATH but getting the same issues. Seems when I don't compile in openssl everything is fine. Thanks! -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Concurrency testing
David E. Wheeler wrote: On Oct 7, 2009, at 5:18 PM, Jeff Janes wrote: I'd much rather live without Test::More and use DBD::Pg, then have Test::More but need to open pipes to psql to talk to the database, rather than using DBI to do it. But I guess we would need to worry about whether we can make DBD::Pg work with the installation being tested, rather than finding some other install. The test architecture depends on Perl, but not on the DBI. I don't think that Andrew wants to add any dependencies. Therefore we'd need to use file handles. That's not to say that we couldn't write a nice little interface for it such that the implementation could later change. Well, that's true of the buildfarm. And there are reasons I don't want to use DBI for the buildfarm, mainly to do with its intended role in simulating what a human would do by hand. What we do for the core testing framework is a different question. Nevertheless, a requirement for DBI and DBD::Pg would be a significant escalation of testing prerequisites. Test::More is comparatively modest requirement, and is fairly universal where Perl is installed. And since we'd just be using it to drive psql, we wouldn't be having to decide if a problem we saw was due to a problem in Postgres or a problem in DBD::Pg. If we want something to drive a huge number of clients, I suspect neither of these is a good way to go, and something more custom built would be required. Last time I built something to drive a huge client load (many thousands of simultaneous connections to a web app) I did it in highly threaded Java using HttpUnit from a number of separate client machines. You wouldn't believe what that managed to do to MySQL on the backend ;-) cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] postgres 8.3.8 and Solaris 10_x86 64 bit problems?
u235sentinel wrote: So I compiled postgres with Solaris 10 and have problems running it. # ./pg_ctl ld.so.1: pg_ctl: fatal: relocation error: R_AMD64_32: file /usr/local/postgres64/lib/libpq.so.5: symbol (unknown): value 0xfd7fff1cf210 does not fit Killed Maybe libpq.so wasn't built with the PIC option (gcc -fPIC, sun compiler -Kpic). -- Andrew Chernow eSilo, LLC every bit counts http://www.esilo.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] COPY enhancements
On Fri, Sep 25, 2009 at 10:01 AM, Emmanuel Cecchet m...@asterdata.com wrote: Robert, Here is the new version of the patch that applies to CVS HEAD as of this morning. Emmanuel I took a look at this patch tonight and, having now read through some of it, I have some more detailed comments. It seems quite odd to me that when COPY succeeds but there are errors, the transaction commits. The only indication that some of my data didn't end up in the table is that the output says COPY n where n is less than the total number of rows I attempted to copy. On the other hand, it would be equally bad if the transaction aborted, because then my error logging table would not get populated - I note that that's the behavior we do get if the max errors threshold is exceeded. I'm not sure what the right answer is here, but it needs some thought and discussion. I think at a minimum the caller needs some indication of the number of FAILED rows, not just the number of succesful ones. What's really bad about this is that a flag called error_logging is actually changing the behavior of the command in a way that is far more dramatic than (and doesn't actually have much to do with) error logging. It's actually making a COPY command succeed that would otherwise have failed. That's not just a logging change. I think the underlying problem is that there is more than one feature here, and they're kind of mixed together. The fact that the partitioning code needs to be separated from the error logging stuff has already been discussed, but I think it actually needs to be broken down even further. I think there are three separate features in the error logging code: A. Ability to ignore a certain number of errors (of certain types?) and still get the other tuples into the table anyway. B. Ability to return error information in a structured format rather than as an exception message. C. Ability to direct the structured error messages to a table. Each of those features deserves a separate discussion to decide whether we want it and how best to implement it. Personally, I think we should skip (C), at least as a starting point. Instead of logging to a table, I think we should consider making COPY return the tuples indicating the error as a query result, the same way EXPLAIN returns the query plan. It looks like this would eliminate the need for at least three of the new COPY options without losing much functionality. I also think that, if possible, (A) and (B) should be implemented as separate features, so that each one can be used separately or both can be used in combination. Some other notes: 1. The copy_errorlogging.out regression test fails, at least on my machine. I guess this problem was discussed previously, but perhaps you should change the setup of the test in some way so that you can generate a patch that doesn't require manual fiddling to get it the regression tests to pass. 2. The error logging doesn't seem to work in all cases. rhaas=# copy foo from '/home/rhaas/x' (error_logging, error_logging_table_name 'errlogging'); ERROR: duplicate key value violates unique constraint foo_pkey DETAIL: Key (id)=(1) already exists. CONTEXT: COPY foo, line 1: 1 Robert I assume that this fails because the code is trained to catch some specific category of errors and redirect those to the error logging table while letting everything take its natural course. I don't feel that that's a terribly useful behavior, and on the other hand, I'm not sure it's going to be possible to fix it without killing performance. 3. The option checking for the error_logging options is insufficient. For example, if ERROR_LOGGING_TABLE_NAME is specified but ERROR_LOGGING is not specified, COPY still works (and just ignores the ERROR_LOGGING_TABLE_NAME). Note that this is quite different from what HEADER does, for example. 4. Specifiying an error_logging_table_name with spaces or other funny characters in it fails. COPY foo FROM STDIN (ERROR_LOGGING, ERROR_LOGGING_TABLE_NAME 'foo bar baz'); 5. The assumption that the public schema is an appropriate default does not seem right to me. I would think that we would want to default to the first schema in the user's search path. 6. errorlogging.c is poorly named, because it has specifically to do with logging errors in COPY, rather than with error logging in general; there's also no reason for it to live in utils/misc. Also, it does not include the same headers as the other files in the same directory. 7. There's no need for the OidLinkedList structure; we already have a system for handling lists of OIDs. See pg_list.h. 8. The OID cache is interesting; have you benchmarked this to see how much it improves performance? If so, you should describe your methodology and performance results. It might be worthwhile to leave this out of the first version of the partitioning patch for simplicity and add submit as a follow-on performance patch. Do we really need a GUC to size this cache, or
Re: [HACKERS] Issues for named/mixed function notation patch
Pavel Stehule pavel.steh...@gmail.com writes: [ latest named-args patch ] Committed with a fair amount of corner-case cleanup and refactoring. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Concurrency testing
On Wed, Oct 7, 2009 at 6:01 PM, Robert Haas robertmh...@gmail.com wrote: On Wed, Oct 7, 2009 at 8:33 PM, David E. Wheeler da...@kineticode.com wrote: Do we need to restrict ourselves to core? Developers already need flex and bison, which aren't needed when installing from the tarball. Couldn't we also have make dev-check that has higher requirements than make check does, but does a more thorough job? flex and bison are not Perl modules. True, but so what? Right, my point was only that we already have different levels of requirements for different things. I don't really see what's wrong with using Perl modules that are likely to be installed most places and easy to obtain where not, if it makes writing a test framework much easier. But I also think that we should not get bogged down on exactly which tools to use - it seems to me the first thing is to find someone who is willing to do the work. If someone makes an AWESOME test suite that uses a module which is a little too adventurous, we can probably find a way of adjusting it after the fact so as to remove the dependency (I fancy myself fairly good at this sort of thing, where Perl is concerned). But if we argue about tools now, we're just going to discourage anyone from taking a stab at it. OK, but what would we be taking a stab at? Would it simply be something like make dev-check or make concurrency-check, which the build farm would then just invoke, thereby killing two birds with one stone? Or would it have to be something fancier than just that? Or am I already too lost in the details? Jeff -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Concurrency testing
On Wed, Oct 7, 2009 at 5:33 PM, David E. Wheeler da...@kineticode.com wrote: On Oct 7, 2009, at 5:18 PM, Jeff Janes wrote: I'd much rather live without Test::More and use DBD::Pg, then have Test::More but need to open pipes to psql to talk to the database, rather than using DBI to do it. But I guess we would need to worry about whether we can make DBD::Pg work with the installation being tested, rather than finding some other install. The test architecture depends on Perl, but not on the DBI. I don't think that Andrew wants to add any dependencies. Therefore we'd need to use file handles. If we are going to talk to multiple psql pipes at the same time, and not self-deadlock in the process, we would probably need to use some form of non-blocking IO. Unfortunately, I've often found it isn't as easy as one might hope to do that in a completely portable way. Maybe IPC::Run, but I think that that is non-core also. Are we interested only in simple deterministic concurrency testing: T1 does A, T2 does B, T1 does C and must block until T2 does D, at which point (and not before) T1 must respond with E. (Of course from some recent testing I've done, we would also want T1 does C and must block either until T2 does D or until X+/-Y microseconds have passed) Or would we want to beat the snot out the database in parallel in a nondeterministic fashion, and checking for self-consistency and non-deadlock? Cheers, Jeff -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] postgres 8.3.8 and Solaris 10_x86 64 bit problems?
Andrew Chernow a...@esilo.com writes: u235sentinel wrote: So I compiled postgres with Solaris 10 and have problems running it. # ./pg_ctl ld.so.1: pg_ctl: fatal: relocation error: R_AMD64_32: file /usr/local/postgres64/lib/libpq.so.5: symbol (unknown): value 0xfd7fff1cf210 does not fit Maybe libpq.so wasn't built with the PIC option (gcc -fPIC, sun compiler -Kpic). That is sort of what the error suggests, all right, but Makefile.solaris has set up CFLAGS_SL that way for a long time. Perhaps CFLAGS_SL is getting overridden from somewhere? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] postgres 8.3.8 and Solaris 10_x86 64 bit problems?
Andrew Chernow wrote: u235sentinel wrote: So I compiled postgres with Solaris 10 and have problems running it. # ./pg_ctl ld.so.1: pg_ctl: fatal: relocation error: R_AMD64_32: file /usr/local/postgres64/lib/libpq.so.5: symbol (unknown): value 0xfd7fff1cf210 does not fit Killed Maybe libpq.so wasn't built with the PIC option (gcc -fPIC, sun compiler -Kpic). I'll give that a shot. It's possible. Thanks for the hint :D -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standby 0.2.1
Simon Riggs wrote: On Tue, 2009-09-22 at 12:53 +0300, Heikki Linnakangas wrote: It looks like the standby tries to remove XID 4323 from the known-assigned hash table, but it's not there because it was removed and set in pg_subtrans by an XLOG_XACT_ASSIGNMENT record earlier. I guess we should just not throw an error in that case, but is there a way we could catch that narrow case and still keep the check in KnownAssignedXidsRemove()? It seems like the check might help catch other bugs, so I'd rather keep it if possible. Fix attached. I fixed that on Friday already, in a slightly different manner. Do you see a problem with that approach? I've made public the version I'm working on. That's the version I'm ultimately going to commit. It would be a lot more helpful if you provided these patches over that version. Otherwise I have to refactor them over that codebase, possibly introducing new bugs. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Hot Standby 0.2.1
Heikki Linnakangas wrote: I've made public the version I'm working on. That's the version I'm ultimately going to commit. It would be a lot more helpful if you provided these patches over that version. Otherwise I have to refactor them over that codebase, possibly introducing new bugs. Actually, it makes most sense if you push your changes directly to my git repository. I just granted you write access to it. Please do these changes against the version currently there, and push any fixes there directly. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Issues for named/mixed function notation patch
2009/10/8 Tom Lane t...@sss.pgh.pa.us: Pavel Stehule pavel.steh...@gmail.com writes: [ latest named-args patch ] Committed with a fair amount of corner-case cleanup and refactoring. regards, tom lane Thank you Pavel Stehule -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Concurrency testing
On Wed, Oct 7, 2009 at 10:41 PM, Jeff Janes jeff.ja...@gmail.com wrote: I don't really see what's wrong with using Perl modules that are likely to be installed most places and easy to obtain where not, if it makes writing a test framework much easier. But I also think that we should not get bogged down on exactly which tools to use - it seems to me the first thing is to find someone who is willing to do the work. If someone makes an AWESOME test suite that uses a module which is a little too adventurous, we can probably find a way of adjusting it after the fact so as to remove the dependency (I fancy myself fairly good at this sort of thing, where Perl is concerned). But if we argue about tools now, we're just going to discourage anyone from taking a stab at it. OK, but what would we be taking a stab at? Would it simply be something like make dev-check or make concurrency-check, which the build farm would then just invoke, thereby killing two birds with one stone? Or would it have to be something fancier than just that? Or am I already too lost in the details? Well, I don't know. I think the first thing would be to figure out what sort of tests we want to run. Taking a stab in the dark, I'm thinking that the point of any sort of concurrency testing must be to check for race conditions, so presumably it's the sort of thing you'd want to have an easy way to run over and over again in a loop. Possibly you would also want to test recovery by killing a backend in the middle somewhere. It may be too much to put into the core tree at all; it might be easier to maintain as a separate project. But I don't have a strong opinion on this. The subject of the thread is concurrency testing so it seems to me we ought to start by figuring out exactly what that means and how we could test it. ...Robert -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Feature Suggestion: PL/Js
Peter Eisentraut pete...@gmx.net writes: Start here: http://developer.postgresql.org/pgdocs/postgres/plhandler.html I notice that this has not been updated for the new inline handlers, but that shouldn't stop you. Actually, that chapter is so old it didn't get updated for language validators either :-(. I added some text, and changed the link in create_language.sgml so hopefully we'll notice it next time. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
