Re: [HACKERS] review: FDW API
On 07.02.2011 08:00, Shigeru HANADA wrote: Sorry for late, attached are revised version of FDW API patches which reflect Heikki's comments except removing catalog lookup via IsForeignTable(). ISTM that the point is avoiding catalog lookup during planning, but I have not found when we can set foreign table flag without catalog lookup during RelOptInfo generation. In get_relation_info(), you do the catalog lookup anyway and have the Relation object at hand. Add a flag to RelOptInfo indicating if it's a foreign table or not, and set that in get_relation_info(). -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [GENERAL] Issues with generate_series using integer boundaries
On Fri, Feb 4, 2011 at 21:32, Thom Brown t...@linux.com wrote:
The issue is that generate_series will not return if the series hits
either the upper or lower boundary during increment, or goes beyond
it. The attached patch fixes this behaviour, but should probably be
done a better way. The first 3 examples above will not return.
There are same bug in int8 and timestamp[tz] versions.
We also need fix for them.
=# SELECT x FROM generate_series(9223372036854775807::int8,
9223372036854775807::int8) AS a(x);
=# SELECT x FROM generate_series('infinity'::timestamp, 'infinity', '1
sec') AS a(x);
=# SELECT x FROM generate_series('infinity'::timestamptz, 'infinity',
'1 sec') AS a(x);
postgres=# SELECT x FROM generate_series(1, 9,-1) AS a(x);
postgres=# SELECT x FROM generate_series(1, 9,3) AS a(x);
They work as expected in 9.1dev.
--
Itagaki Takahiro
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] A different approach to extension NO USER DATA feature
Hi,
Tom Lane t...@sss.pgh.pa.us writes:
As I work through the extensions patch, the aspect of it that I like the
least is the NO USER DATA clause and related functions. I think it's
badly designed, badly implemented, and doesn't solve the problem.
I'm not loving it either, but wanting to stay general enough, then
that's all I came up with.
After a bit of thought I believe that we can fix this if we are willing
to teach pg_dump explicitly about extension configuration tables.
Ok, if we're now talking about tables only (which makes a lot of sense,
but somehow I didn't want to restrict it that way).…
SELECT pg_extension_partial_dump('my_table', 'WHERE NOT system_data');
I'm envisioning that this function would add that information to the
pg_extension entry, and then pg_dump would use it to select out a subset
of rows that it would dump as user data, even though it's not dumping
the table definition.
…then what about taking it the extra mile and have something even more
simple and easier to check and control?
From your idea, where I get is that we could force the system_data
column name and type. We're talking about only supporting tables, and a
boolean expression is all we will be able to use. Then, if the column
name and type are fixed, we only need the non-qualified relation name.
I'd expect that in the control file, so that we don't have to provide
new SQL level tools.
People that need a complex boolean expression will be able to maintain
the column from a trigger.
If we get there, then all we get to store is an oidvector column in the
catalog, the list of the table OIDs we will have to only dump the
content of those rows WHERE NOT system_data.
Or do you want to keep some generality here?
--
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Where the Quals are actually 'List'ed
Hi, I find that ExecInitExpr creates a ExprState tree (and so say the comments above the function in the source). Also, it seems to decide which function would get called when the expression is to be evaluated when ExecQual runs, by setting the function pointer, for example: bstate-xprstate.evalfunc = (ExprStateEvalFunc)ExecEvalAnd; But ExecQual goes through a List, whereas, ExecInitExpr creates a tree. So is that same tree converted to a qual list or are we adding some more information. Vaibhav (*_*) -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Foreign servers and user mappings versus the extensions patch
Robert Haas robertmh...@gmail.com writes: On Sat, Feb 5, 2011 at 5:41 PM, Tom Lane t...@sss.pgh.pa.us wrote: Currently, the extensions patch considers that foreign data wrappers, foreign servers, and user mapping objects can all be parts of extensions. This is slightly problematic for pg_dump, where somebody decided to take a shortcut and not implement user mappings using the full DumpableObject infrastructure. That could be fixed, but I'm wondering whether it's worth the trouble. I can see the point of writing an FDW as an extension but it's a lot harder to see why either foreign server or user mapping objects would ever be part of an extension. So it might just be best to remove those two object types from the set that can be managed by an extension. Comments? I agree it's probably not that useful to make a foreign server or foreign user mapping part of an extension, but I'd rather not have us fail to support it just because we can't think of a use case right now. So my vote would be to fix it. I would have though that it could allow you to distribute internally at the system level the user mappings and server details, then have a DBA install it without exposing them the password or other details. Well, I don't recall offhand what you see as a superuser in the system view, but that could be a use case. Other than that, I confess I added the support just to try at being exhaustive. Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] A different approach to extension NO USER DATA feature
On 06/02/11 18:23, Tom Lane wrote: After a bit of thought I believe that we can fix this if we are willing to teach pg_dump explicitly about extension configuration tables. The behavior we want for those is for the table schema definition to never be dumped (the table should always be created by CREATE EXTENSION), but for some subset of the table data to get dumped, excluding any system-provided rows. [snip] pg_extension_partial_dump (table_name regclass, where_condition text) Possible alternative approach? 1. Extension provides list of config tables/views/set-returning functions to be dumped via e.g. my_config_tables() 2. They get dumped, but each as a TEMP TABLE (need unique names for multiple extensions though). 3. On restore, tables are created and populated, then read_your_config(ARRAY-OF-TABLE-NAMES) is called in the extension. This separates the configuration-for-user from configuration-for-extension. It allows the extension to decide whether to load the new config or reject it. It lets you test/demonstrate multiple configurations fairly simply. The system_data column scenario can then be a default implementation of read_your_config(). -- Richard Huxton Archonet Ltd -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SSI patch version 14
On 05.02.2011 21:43, Kevin Grittner wrote: Kevin Grittner wrote: So now that I'm sure we actually do need code there, I'll add it. In working on this I noticed the apparent need to move two calls to PredicateLockTuple a little bit to keep them inside the buffer lock. Without at least a share lock on the buffer, it seems that here is a window where a read could miss the MVCC from a write and the write could fail to see the predicate lock. Please see whether this seems reasonable: http://git.postgresql.org/gitweb?p=users/kgrittn/postgres.git;a=commitdiff;h=7841a22648c3f4ae46f674d7cf4a7c2673cf9ed2 And I'll add the new test to the isolation suite. We don't need all permutations for this test, which is a good thing since it has such a long setup time. Is there an easy way to just run the one schedule of statements on three connections? Not at the moment, but we can add that.. I added the capability to specify exact permutations like: permutation rwx1 rwx2 c1 c2 See my git repository at git://git.postgresql.org/git/users/heikki/postgres.git, branch serializable. I also added a short README to explain what the isolation test suite is all about, as well as separate make check and make installcheck targets. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [GENERAL] Issues with generate_series using integer boundaries
On 7 February 2011 09:04, Itagaki Takahiro itagaki.takah...@gmail.com wrote:
On Fri, Feb 4, 2011 at 21:32, Thom Brown t...@linux.com wrote:
The issue is that generate_series will not return if the series hits
either the upper or lower boundary during increment, or goes beyond
it. The attached patch fixes this behaviour, but should probably be
done a better way. The first 3 examples above will not return.
There are same bug in int8 and timestamp[tz] versions.
We also need fix for them.
=# SELECT x FROM generate_series(9223372036854775807::int8,
9223372036854775807::int8) AS a(x);
Yes, of course, int8 functions are separate. I attach an updated
patch, although I still think there's a better way of doing this.
=# SELECT x FROM generate_series('infinity'::timestamp, 'infinity', '1
sec') AS a(x);
=# SELECT x FROM generate_series('infinity'::timestamptz, 'infinity',
'1 sec') AS a(x);
I'm not sure how this should be handled. Should there just be a check
for either kind of infinity and return an error if that's the case? I
didn't find anything wrong with using timestamp boundaries:
postgres=# SELECT x FROM generate_series('1 Jan 4713 BC
00:00:00'::timestamp, '1 Jan 4713 BC 00:00:05'::timestamp, '1 sec') AS
a(x);
x
4713-01-01 00:00:00 BC
4713-01-01 00:00:01 BC
4713-01-01 00:00:02 BC
4713-01-01 00:00:03 BC
4713-01-01 00:00:04 BC
4713-01-01 00:00:05 BC
(6 rows)
Although whether this demonstrates a true timestamp boundary, I'm not sure.
postgres=# SELECT x FROM generate_series(1, 9,-1) AS a(x);
postgres=# SELECT x FROM generate_series(1, 9,3) AS a(x);
They work as expected in 9.1dev.
Those 2 were to demonstrate that the changes don't affect existing
functionality. My previous patch proposal (v2) caused these to return
unexpected output.
--
Thom Brown
Twitter: @darkixion
IRC (freenode): dark_ixion
Registered Linux user: #516935
generate_series_fix.v4.patch
Description: Binary data
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SQL/MED - file_fdw
On Mon, Feb 7, 2011 at 16:01, Shigeru HANADA han...@metrosystems.co.jp wrote:
This patch is based on latest FDW API patches which are posted in
another thread SQL/MED FDW API, and copy_export-20110104.patch which
was posted by Itagaki-san.
I have questions about estimate_costs().
* What value does baserel-tuples have?
Foreign tables are never analyzed for now. Is the number correct?
* Your previous measurement showed it has much more startup_cost.
When you removed ReScan, it took long time but planner didn't choose
materialized plans. It might come from lower startup costs.
* Why do you use lstat() in it?
Even if the file is a symlink, we will read the linked file in the
succeeding copy. So, I think it should be stat() rather than lstat().
+estimate_costs(const char *filename, RelOptInfo *baserel,
+ double *startup_cost, double *total_cost)
+{
...
+ /* get size of the file */
+ if (lstat(filename, stat) == -1)
+ {
+ ereport(ERROR,
+ (errcode_for_file_access(),
+errmsg(could not stat file \%s\: %m, filename)));
+ }
+
+ /*
+* The way to estimate costs is almost same as cost_seqscan(), but there
+* are some differences:
+* - DISK costs are estimated from file size.
+* - CPU costs are 10x of seq scan, for overhead of parsing records.
+*/
+ pages = stat.st_size / BLCKSZ + (stat.st_size % BLCKSZ 0 ? 1 : 0);
+ run_cost += seq_page_cost * pages;
+
+ *startup_cost += baserel-baserestrictcost.startup;
+ cpu_per_tuple = cpu_tuple_cost + baserel-baserestrictcost.per_tuple;
+ run_cost += cpu_per_tuple * 10 * baserel-tuples;
+ *total_cost = *startup_cost + run_cost;
+
+ return stat.st_size;
+}
--
Itagaki Takahiro
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] DROP SCHEMA xxx CASCADE: ERROR: could not open relation with OID yyy
Hi all, I'm trying to debug an ugly error triggered from a DROP SCHEMA xxx CASCADE call inside a function. The call is the last step of the stored pl/pgsql procedure. I've verified that removing the DROP SCHEMA command from _inside_ the function body and performing it _outside_ it (right after return) everything works fine. Note that the schema that the function is trying to drop was created by a function called by the function attempting to drop it. Both function (the one which creates the schema and the one which attempts to drop it) are defined as VOLATILE. Also, I can see traces of the DROP SCHEMA CASCADE being executed, till the ERROR comes out (lots of traces for cascading objects). This is : PostgreSQL 8.4.3 on x86_64-pc-linux-gnu, compiled by GCC gcc-4.4.real (Ubuntu 4.4.3-4ubuntu5) 4.4.3, 64-bit Do you have an idea on how to further debug this ? TIA. --strk; () Free GIS Flash consultant/developer /\ http://strk.keybit.net/services.html -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SQL/MED - file_fdw
On Mon, Feb 07, 2011 at 03:39:39PM +0900, Itagaki Takahiro wrote: On Sun, Feb 6, 2011 at 09:01, Noah Misch n...@leadboat.com wrote: Most parse analysis-type bits of DoCopy() move to BeginCopy(). It would be possible to move more FROM-only or TO-only members in BeginCopy() into their BeginCopyFrom/To(), but it is just a code refactoring requires more code rearrangement. It should be done by another try even if needed. Agreed. CopyState.processed is gone, with the row count now bubbling up through return values. I removed it from CopyState because it represents number of inserted rows in COPY FROM. However, for the viewpoint of API, the numbers of lines in the input file is more suitable. To avoid the confusion, I moved it from a common member to COPY FROM-only variable. Perhaps I'm missing something. The new API does not expose a processed count at all; that count is used for the command tag of a top-level COPY. This part of the patch is just changing how we structure the code to maintain that tally, and it has no implications for observers outside copy.c. Right? For COPY TO, the relkind checks move from DoCopyTo() to BeginCopyTo(). ??I'm skeptical about this one. ??It's not required for correctness, and the relkind checks for COPY FROM remain in CopyFrom(). I think error checks should be done in the early phase, so I moved the check to BeginCopyTo(). However, relkind check in COPY FROM is needed only for COPY FROM command because the relation is the inserted target. For APIs, the relation is used as a template for input file. So, we cannot perform the check in BeginCopyFrom(). The choice of where to put them does not affect anything outside of copy.c, and placement in DoCopyTo() would make the symmetry between the TO and FROM code paths easier to follow. Not a major concern, though. file_fdw uses CopyFromErrorCallback() to give errors the proper context. ??The function uses template strings like COPY %s, line %d, where %s is the name of the relation being copied. ??Presumably file_fdw and other features using this API would wish to customize that error message prefix, and the relation name might not be apropos at all. ??How about another argument to BeginCopyFrom, specifying an error prefix to be stashed in the CopyState? I changed COPY %s, .. to relation %s, ... because the first string is the relation name anyway. We could have another prefix argument, but I think it has little information for errors. That's perhaps an improvement for file_fdw, but not for regular COPY. My comment originated with a faulty idea that file_fdw's internal use of COPY was an implementation detail that users should not need to see. Looking now, the file_fdw documentation clearly explains the tie to COPY, even referring users to the COPY documentation. I no longer see a need to hide the fact that the foreign source is a PostgreSQL COPY command. The error messages are fine as they were. Some future client of the new API may wish to hide its internal COPY use, but there's no need to design for that now. We also have many COPY in other messages, but they won't be used actually because the are messages for invalid arguments and file_fdw will have own validater function. All invalid arguments will be filtered in CREATE commands. Agreed; could not read from COPY file: %m appears to be the primary one liable to happen in practice. The greater failure with that one is that, given a query reading from multiple file_fdw tables, you can't tell which file had a problem. This issue runs broader than the patch at hand; I will start another thread to address it. Let's proceed with this patch, not changing any error messages. If other discussion concludes that the desired behavior requires an enhancement to this new API, a followup commit can implement that. ! ?? ?? ?? ?? ?? ?? /* Reset the per-tuple exprcontext */ ! ?? ?? ?? ?? ?? ?? ResetPerTupleExprContext(estate); ! ! ?? ?? ?? ?? ?? ?? /* Switch into its memory context */ ! ?? ?? ?? ?? ?? ?? MemoryContextSwitchTo(GetPerTupleMemoryContext(estate)); Shouldn't a switch to this context happen inside NextCopyFrom(), then again for the calls to heap_form_tuple/HeapTupleSetOid? ??I found previous discussion on this matter, but no conclusion. In my understanding, NextCopyFrom() should use CurrentMemoryContext provided by the caller. The file_fdw will use executor's per-tuple context for it. In a direct call to COPY FROM, all of NextCopyFrom() uses the per-tuple context of CopyState-estate. We reset that context before each call to NextCopyFrom(). This is true before (ignoring code movement) and after the patch. A file_fdw NextCopyFrom() call will use the per-tuple context of the executor performing a foreign scan. Allocations will arise primarily in type input functions. ExecEvalExpr(), used to acquire default values, will still use the per-tuple context of CopyState-estate.
[HACKERS] Error attribution in foreign scans
Suppose you create several file_fdw foreign tables, query them together, and read(2) returns EIO for one of the files: [local] postgres=# SELECT * FROM ft0, ft1, ft2; ERROR: could not read from COPY file: Input/output error The message does not show which foreign table yielded the error. We could evade the problem in this case by adding a file name to the error message in the COPY code, but that strategy doesn't translate to twitter_fdw, firebird_fdw, etc. We need a convention for presenting foreign errors that clearly attributes them to the originating foreign table. What should it be? Perhaps something as simple as having the core foreign scan code push an error context callback that does errcontext(scan of foreign table \%s\, tabname)? Disclaimer: I have only skimmed SQL/MED patches other than copy_export. Thanks, nm -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range Types
Hi, Jeff Davis pg...@j-davis.com writes: * Should pg_range reference the btree opclass or the compare function directly? I would say yes. We use the btree opclass in other similar situations. * Should subtype_cmp default to the default btree opclass's compare function? My vote is yes too. * Right now only superusers can define a range type. Should we open it up to normal users? Is there any reason to restrict who's get to use the feature? I don't see any… * Should the SQL (inlinable) function length, which relies on polymorphic -, be immutable, strict, or volatile? I would think stable: polymorphic means that the function implementing the - operator depends on the argument. I don't recall that it depends on them in a volatile way… except if you change the operator definition, which is possible to do (so not immutable). * Later we might consider whether we should include btree_gist in core, to make range types more useful with exclusion constraints out-of-the-box. This should be left for later, I'm just including this for the archives so it doesn't get lost. I would expect the extension to have something to offer here. ~=# create extension btree_gist with schema pg_catalog; CREATE EXTENSION Now you can act as if it were part of core. Including not being able to ALTER EXTENSION SET SCHEMA (cannot remove dependency on schema pg_catalog because it is a system object), but maybe the recent changes that Tom done on the usage of DEPENDENCY_INTERNAL in the extension patch will open that again. Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Problem with postgresql database connection in combination with HUAWEI data modem
On 02/03/2011 04:45 PM, Robert Haas wrote: Your data modem is probably doing something funky to your network stack, but I don't know what. Are other network services affected as well? In that case, I'd file a bug against the modem driver software. Regards Markus Wanner -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] odd output of pg_basebackup
Hi, When I ran pg_basebackup with -x, -P and -v options, I encountered the following odd output. $ pg_basebackup -D hoge -x -P -v xlog start point: 0/220 33708/17322 kB (100%) 1/1 tablespaces ( )02) 02) is a part of the previously reported message, it looks odd that the subsequent progress report doesn't reset that. We should reset the output buffer with each progress report? Regards, -- Fujii Masao NIPPON TELEGRAPH AND TELEPHONE CORPORATION NTT Open Source Software Center -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] DROP SCHEMA xxx CASCADE: ERROR: could not open relation with OID yyy
strk s...@keybit.net writes: Do you have an idea on how to further debug this ? That usually goes with providing a self-contained test case… that is a minimum script that creates the function(s) and calls them. Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Error attribution in foreign scans
On 07.02.2011 14:17, Noah Misch wrote: Suppose you create several file_fdw foreign tables, query them together, and read(2) returns EIO for one of the files: [local] postgres=# SELECT * FROM ft0, ft1, ft2; ERROR: could not read from COPY file: Input/output error The message does not show which foreign table yielded the error. We could evade the problem in this case by adding a file name to the error message in the COPY code, but that strategy doesn't translate to twitter_fdw, firebird_fdw, etc. We need a convention for presenting foreign errors that clearly attributes them to the originating foreign table. What should it be? Perhaps something as simple as having the core foreign scan code push an error context callback that does errcontext(scan of foreign table \%s\, tabname)? Yeah, an error context callback like that makes sense. In the case of the file FDW, though, just including the filename in the error message seems even better. Especially if the error is directly related to failure in reading the file. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] DROP SCHEMA xxx CASCADE: ERROR: could not open relation with OID yyy
I've handled to produce a small testcase: http://strk.keybit.net/tmp/could_not_open_relation.sql It still requires postgis (svn), but if anyone has that it might help. Will try to go on with the reduction. --strk; On Mon, Feb 07, 2011 at 12:38:08PM +0100, strk wrote: Hi all, I'm trying to debug an ugly error triggered from a DROP SCHEMA xxx CASCADE call inside a function. The call is the last step of the stored pl/pgsql procedure. I've verified that removing the DROP SCHEMA command from _inside_ the function body and performing it _outside_ it (right after return) everything works fine. Note that the schema that the function is trying to drop was created by a function called by the function attempting to drop it. Both function (the one which creates the schema and the one which attempts to drop it) are defined as VOLATILE. Also, I can see traces of the DROP SCHEMA CASCADE being executed, till the ERROR comes out (lots of traces for cascading objects). This is : PostgreSQL 8.4.3 on x86_64-pc-linux-gnu, compiled by GCC gcc-4.4.real (Ubuntu 4.4.3-4ubuntu5) 4.4.3, 64-bit Do you have an idea on how to further debug this ? TIA. --strk; () Free GIS Flash consultant/developer /\ http://strk.keybit.net/services.html -- () Free GIS Flash consultant/developer /\ http://strk.keybit.net/services.html -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] DROP SCHEMA xxx CASCADE: ERROR: could not open relation with OID yyy
I've uploaded also the script output ( CASCADE traces ) : http://strk.keybit.net/tmp/could_not_open_relation.sql http://strk.keybit.net/tmp/could_not_open_relation.log And realized that the relation oid is the one first requested for deletion. Ie: DROP TABLE XXX CASCADE; .. ERROR: could not open relation with OID XXX:regclass::oid I've found two ways to avoid the error: 1. Perform the DROP TABLE outside the transaction triggering its creation 2. Avoiding population of the table being dropped (the AddEdge call) Note that the 'edge_data' table has a foreign key to itself, and the constraint is initially deferred (may have a role here, I guess ?) next_left_edge_exists FOREIGN KEY (abs_next_left_edge) REFERENCES.edge_data(edge_id) DEFERRABLE INITIALLY DEFERRED, next_right_edge_exists FOREIGN KEY (abs_next_right_edge) REFERENCES edge_data(edge_id) DEFERRABLE INITIALLY DEFERRED --strk; () Free GIS Flash consultant/developer /\ http://strk.keybit.net/services.html -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] DROP SCHEMA xxx CASCADE: ERROR: could not open relation with OID yyy
On Mon, Feb 07, 2011 at 02:31:49PM +0100, Dimitri Fontaine wrote:
strk s...@keybit.net writes:
Do you have an idea on how to further debug this ?
That usually goes with providing a self-contained test case⦠that is a
minimum script that creates the function(s) and calls them.
I've finally completed the debugging phase and have
a minimal self-contained testcase showing the problem.
It has to do with INITIALLY DEFERRED constraints.
The testcase is attached.
--strk;
() Free GIS Flash consultant/developer
/\ http://strk.keybit.net/services.html
-- {
CREATE OR REPLACE FUNCTION makeTopo()
RETURNS void AS $$
BEGIN
CREATE SCHEMA topo;
CREATE TABLE topo.edge (
id int4 PRIMARY KEY,
ref int4,
FOREIGN KEY(ref) REFERENCES topo.edge(id)
DEFERRABLE INITIALLY DEFERRED);
-- NOTE: returning here fixes the bug
--RETURN;
INSERT INTO topo.edge VALUES (1,1);
END
$$ language 'plpgsql' VOLATILE STRICT;
-- }
DROP SCHEMA IF EXISTS topo CASCADE;
BEGIN;
SELECT makeTopo();
SELECT 'topo.edge'::regclass::oid;
-- NOTE: this fixes it!
-- SET CONSTRAINTS ALL IMMEDIATE;
-- ERROR: could not open relation with OID XXX
-- NOTE: X is 'topo.edge'::regclass::oid
DROP TABLE topo.edge CASCADE;
COMMIT;
-- NOTE: doing it here is fine
--DROP TABLE topo.edge CASCADE;
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] A different approach to extension NO USER DATA feature
On Mon, Feb 7, 2011 at 4:18 AM, Dimitri Fontaine dimi...@2ndquadrant.fr wrote: Or do you want to keep some generality here? I think it might be slightly advantageous to keep some generality, because some people might already have catalog columns that do this (but with a different name) or might have other reasons for needing an integer entry type column from which the system property can be inferred. I can't think how many times I've written a web interface that lets users edit a foo, but denies them the ability to edit any foo where bar_id = 1 (because those are the magic ones that get created some other way). -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [COMMITTERS] pgsql: remove tags.
Robert Haas wrote: On Sun, Feb 6, 2011 at 11:12 PM, Tom Lane t...@sss.pgh.pa.us wrote: Bruce Momjian br...@momjian.us writes: Bruce Momjian wrote: remove tags. Sorry, vague commit message (I forgot squash). Can I will use git ammend to improve this message? Absolutely not. How about git revert, instead? ?It's not apparent to me that these changes were improvements. I'll buy that one. [ CC to docs, committers removed. ] Well, if we want to revert, then we have to add literal to all the numbers used in our docs --- there was no logic in what we previously had. Do we want that? Here is an example line I did not change: an otherwise idle connection. A value of 0 uses the system default. Do we want that 0 to appear in a fixed-width font via literal? It is easy to do but we should decide. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sync Rep for 2011CF1
On Sun, Jan 30, 2011 at 11:44 AM, Robert Haas robertmh...@gmail.com wrote: Time's running short - do you have an updated patch? This patch hasn't been updated in more than three weeks. I assume this should now be marked Returned with Feedback, and we'll revisit synchronous replication for 9.2? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] little mistakes in HS/SR
Robert Haas wrote: On Sun, Feb 6, 2011 at 6:44 PM, Bruce Momjian br...@momjian.us wrote: Robert Haas wrote: On Thu, Jul 22, 2010 at 1:41 AM, Fujii Masao masao.fu...@gmail.com wrote: We should enclose -1 with literal tag. A quick survey of the documentation as a whole suggests that we enclose -1 with literal in a few places but more commonly we don't. I have no position on whether we should do it or not, but maybe we should try to be consistent throughout the docs? ?Or at least have a consistent rule for deciding what to do in a particular case? Excellent question. ?I went through the documentation and removed literal tags where appropriate --- there are cases where we are referencing an actual number, and there literal makes sense. ?Applied patch attached. I think the larger question is whether we should say zero for 0 and one for 1, etc. ?Prose typography suggests we should, but for technical manuals, I am not sure. ?Ideas? I am doubtful that this makes sense in general. I suspect it depends somewhat on context. We do spell out zero in a few cases where it makes sense, so it seems we need no change here. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Re: [GENERAL] Why copy ... from stdio does not return immediately when reading invalid data?
I have analyzed the PostgreSQL protocol using Wireshark (an open source packet analyzer), and I observed that the PostgreSQL backend, while doing a COPY ... FROM STDIN, reports errors as soon as possible (especially errors related to invalid data). Therefore, the late reporting of errors while doing a COPY ... FROM STDIN is not a limitation of the underlying protocol; it is a limitation (or a design choice) of the libpq library. It looks like this is a well known issue because it is listed on the todo list: http://wiki.postgresql.org/wiki/Todo#COPY And was discussed before: http://archives.postgresql.org/pgsql-hackers/2008-04/msg01169.php Do you think it is possible to change that behavior, or work around it? While reading libpq source code, I noticed the function pqParseInput3 (file fe-protocol3.c) ignores error responses while the connection is in PGASYNC_COPY_IN state. Maybe we can make a special case for the COPY FROM subprotocol and handle errors early, in order to make them available to PQgetResult? Is is feasible in a simple way or is it a bad idea? Regards, Nicolas Grilly On Wed, Feb 2, 2011 at 20:06, John R Pierce pie...@hogranch.com wrote: On 02/02/11 10:20 AM, Nicolas Grilly wrote: Is the copy protocol (aka PQputCopyData and PQputCopyEnd) designed to send gigabytes of data with just one copy ... from stdio query, and is there a way to be notified of a potential error before calling PQputCopyEnd? Or do I have to send my data in small chunks (for example batch of 1 rows), issue a PQputCopyEnd, check for errors, and continue with the next chunk? I would batch the data, maybe 1000 lines or even 100 lines at a time if these errors are at all frequent. put the errored batches in an exception list or something so you can sort them out later.
Re: [HACKERS] A different approach to extension NO USER DATA feature
Robert Haas robertmh...@gmail.com writes: On Mon, Feb 7, 2011 at 4:18 AM, Dimitri Fontaine dimi...@2ndquadrant.fr wrote: Or do you want to keep some generality here? I think it might be slightly advantageous to keep some generality, Yeah. I had also thought about hard-wiring the WHERE clause, but there's at least one big objection to that: it fails to cover cases where there's no need for a flag column because all the entries are user-provided. The catalog representation I'd been thinking of was a regclass[] array for the table names and a text[] array of the same length for the WHERE clauses. It's slightly ugly but no big deal. There are likely to be some other array columns in pg_extension before we're done, anyway --- one I'd been thinking about a bit was OIDs of modules this one depends on. The current design doesn't cope very well with modules that depend on other ones. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] A different approach to extension NO USER DATA feature
Richard Huxton d...@archonet.com writes: Possible alternative approach? 1. Extension provides list of config tables/views/set-returning functions to be dumped via e.g. my_config_tables() 2. They get dumped, but each as a TEMP TABLE (need unique names for multiple extensions though). 3. On restore, tables are created and populated, then read_your_config(ARRAY-OF-TABLE-NAMES) is called in the extension. This is kind of problematic because it requires those tables to be created before the extension is created. One difficulty with that is it would break parallel pg_restore. Another is that it's not unusual for the config tables to not be populatable at all before the extension's been loaded. For example, IIRC how the old contrib/tsearch2 config tables worked, there were regproc columns that contained references to functions created by tsearch2. It is true that the extension author may sometimes need to run some code when the user-provided data is loaded into the config table, but in the design I suggested, that could be handled by attaching triggers to the config table. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Spread checkpoint sync
2011/2/7 Greg Smith g...@2ndquadrant.com: Robert Haas wrote: With the fsync queue compaction patch applied, I think most of this is now not needed. Attached please find an attempt to isolate the portion that looks like it might still be useful. The basic idea of what remains here is to make the background writer still do its normal stuff even when it's checkpointing. In particular, with this patch applied, PG will: 1. Absorb fsync requests a lot more often during the sync phase. 2. Still try to run the cleaning scan during the sync phase. 3. Pause for 3 seconds after every fsync. Yes, the bits you extracted were the remaining useful parts from the original patch. Today was quiet here because there were sports on or something, and I added full auto-tuning magic to the attached update. I need to kick off benchmarks and report back tomorrow to see how well this does, but any additional patch here would only be code cleanup on the messy stuff I did in here (plus proper implementation of the pair of GUCs). This has finally gotten to the exact logic I've been meaning to complete as spread sync since the idea was first postponed in 8.3, with the benefit of some fsync aborption improvements along the way too The automatic timing is modeled on the existing checkpoint_completion_target concept, except with a new tunable (not yet added as a GUC) currently called CheckPointSyncTarget, set to 0.8 right now. What I think I want to do is make the existing checkpoint_completion_target now be the target for the end of the sync phase, matching its name; people who bumped it up won't necessarily even have to change anything. Then the new guc can be checkpoint_write_target, representing the target that is in there right now. Is it worth a new thread with the different IO improvements done so far or on-going and how we may add new GUC(if required !!!) with intelligence between those patches ? ( For instance, hint bit IO limit needs probably a tunable to define something similar to hint_write_completion_target and/or IO_throttling strategy, ...items which are still in gestation...) I tossed the earlier idea of counting relations to sync based on the write phase data as too inaccurate after testing, and with it for now goes checkpoint sorting. Instead, I just take a first pass over pendingOpsTable to get a total number of things to sync, which will always match the real count barring strange circumstances (like dropping a table). As for the automatically determining the interval, I take the number of syncs that have finished so far, divide by the total, and get a number between 0.0 and 1.0 that represents progress on the sync phase. I then use the same basic CheckpointWriteDelay logic that is there for spreading writes out, except with the new sync target. I realized that if we assume the checkpoint writes should have finished in CheckPointCompletionTarget worth of time or segments, we can compute a new progress metric with the formula: progress = CheckPointCompletionTarget + (1.0 - CheckPointCompletionTarget) * finished / goal; Where finished is the number of segments written out, while goal is the total. To turn this into an example, let's say the default parameters are set, we've finished the writes, and finished 1 out of 4 syncs; that much work will be considered: progress = 0.5 + (1.0 - 0.5) * 1 / 4 = 0.625 On a scale that effectively aimes to be finished sync work by 0.8. I don't use quite the same logic as the CheckpointWriteDelay though. It turns out the existing checkpoint_completion implementation doesn't always work like I thought it did, which provide some very interesting insight into why my attempts to work around checkpoint problems haven't worked as well as expected the last few years. I thought that what it did was wait until an amount of time determined by the target was reached until it did the next write. That's not quite it; what it actually does is check progress against the target, then sleep exactly one nap interval if it is is ahead of schedule. That is only the same thing if you have a lot of buffers to write relative to the amount of time involved. There's some alternative logic if you don't have bgwriter_lru_maxpages set, but in the normal situation it effectively it means that: maximum write spread time=bgwriter_delay * checkpoint dirty blocks No matter how far apart you try to spread the checkpoints. Now, typically, when people run into these checkpoint spikes in production, reducing shared_buffers improves that. But I now realize that doing so will then reduce the average number of dirty blocks participating in the checkpoint, and therefore potentially pull the spread down at the same time! Also, if you try and tune bgwriter_delay down to get better background cleaning, you're also reducing the maximum spread. Between this issue and the bad behavior when the fsync queue fills, no wonder this
Re: [HACKERS] SQL/MED - file_fdw
On Mon, 7 Feb 2011 21:00:53 +0900 Itagaki Takahiro itagaki.takah...@gmail.com wrote: On Mon, Feb 7, 2011 at 16:01, Shigeru HANADA han...@metrosystems.co.jp wrote: This patch is based on latest FDW API patches which are posted in another thread SQL/MED FDW API, and copy_export-20110104.patch which was posted by Itagaki-san. I have questions about estimate_costs(). * What value does baserel-tuples have? Foreign tables are never analyzed for now. Is the number correct? No, baserel-tuples is always 0, and baserel-pages is 0 too. In addition, width and rows are set to 100 and 1, as default values. I think ANALYZE support is needed to make PostgreSQL's standard optimization for foreign scans. At present, estimation for foreign tables would be awful. * Your previous measurement showed it has much more startup_cost. When you removed ReScan, it took long time but planner didn't choose materialized plans. It might come from lower startup costs. I tested joining file_fdw tables, accounts and branches, which are initialized with pgbench -i -s 10 and exported to csv files. At first, I tried adding random_page_cost (4.0) to startup_cost as cost to open file (though it's groundless), but materialized was not chosen. After updating pg_class.reltuples to correct value, Hash-join was choosen for same query. With disabling Hash-join, finally materialized was choosen. ISTM that choosing simple nested loop would come from wrong estimation of loop count, but not from startup cost. IMHO, supporting analyze (PG-style statistics) is necessary to make PostgreSQL to generate sane plan. * Why do you use lstat() in it? Even if the file is a symlink, we will read the linked file in the succeeding copy. So, I think it should be stat() rather than lstat(). Good catch! Fixed version is attached. Regards, -- Shigeru Hanada file_fdw-20110208.patch.gz Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Spread checkpoint sync
Cédric Villemain wrote: Is it worth a new thread with the different IO improvements done so far or on-going and how we may add new GUC(if required !!!) with intelligence between those patches ? ( For instance, hint bit IO limit needs probably a tunable to define something similar to hint_write_completion_target and/or IO_throttling strategy, ...items which are still in gestation...) Maybe, but I wouldn't bring all that up right now. Trying to wrap up the CommitFest, too distracting, etc. As a larger statement on this topic, I'm never very excited about redesigning here starting from any point other than saw a bottleneck doing x on a production system. There's a long list of such things already around waiting to be addressed, and I've never seen any good evidence of work related to hint bits being on it. Please correct me if you know of some--I suspect you do from the way you're brining this up. If we were to consider kicking off some larger work here, I would drive that by asking where the data supporting that work being necessary is at first. It's hard enough to fix a bottleneck that's staring right at you, trying to address one that's just theorized is impossible. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us PostgreSQL 9.0 High Performance: http://www.2ndQuadrant.com/books -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] limiting hint bit I/O
Robert Haas wrote: On Sat, Feb 5, 2011 at 4:31 PM, Bruce Momjian br...@momjian.us wrote: Uh, in this C comment: + ? ? ? ?* or not we want to take the time to write it. ?We allow up to 5% of + ? ? ? ?* otherwise-not-dirty pages to be written due to hint bit changes, 5% of what? ?5% of all buffers? ?5% of all hint-bit-dirty ones? ?Can you clarify this in the patch? 5% of buffers that are hint-bit-dirty but not otherwise dirty. ISTM that's exactly what the comment you just quoted says on its face, but I'm open to some other wording you want to propose. How about: otherwise-not-dirty - only-hint-bit-dirty So 95% of your hint bit modificates are discarded if the pages is not otherwise dirtied? That seems pretty radical. -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] More extension issues: ownership and search_path
There are some things that the current extensions patch leaves indeterminate during a dump and reload cycle, which strikes me as a bad thing. One is ownership. Since we don't record the identity of the user who created an extension, there's no way for pg_dump to ensure that it's recreated by the same user. I think we'll regret that in future even if you don't think it's problematic today. In particular, I foresee eventually allowing non-superusers to load extensions, so I think this is going to follow pretty much the same trajectory as procedural languages, which we originally did not track ownership for. In short, I think we'd better add an extowner column to pg_extension. Another is the search_path setting. Currently this is actually rather broken since pg_dump makes no effort at all to ensure that search_path is the same at reload time as it was when the extension was created, and thus the contained objects could easily go into the wrong schema. But even without thinking about dump/reload, it seems to me that it would be a good thing for reproducibility if CREATE EXTENSION were to forcibly set search_path before running the extension's SQL script. The level-zero version of that would be to do the equivalent of SET LOCAL search_path = @extschema@ such that the path only contains the target schema and nothing else. The trouble with this simplistic approach is that it doesn't work nicely if one extension depends on another --- you probably want the search path to include the schema(s) the required extensions got installed into. Of course inter-extension dependencies aren't going to work anyway unless pg_dump knows about them so it can make sure to load the extensions in the right order. So where I think we're going to end up is adding a clause along the line of USING list-of-extension-names to CREATE EXTENSION, storing those dependencies explicitly, and having the CREATE EXTENSION code set search_path to the target schema followed by the target schema(s) of the USING extensions. Not sure if this is worth doing immediately or can be left for 9.2. At least in the contrib modules, there are no interdependencies, and I don't know whether any exist in the wider world of existing extensions. Comments? regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [PERFORM] pgbench to the MAXINT
The update on the work to push towards a bigger pgbench is that I now
have the patch running and generating databases larger than any
previously possible scale:
$ time pgbench -i -s 25000 pgbench
...
25 tuples done.
...
real258m46.350s
user14m41.970s
sys0m21.310s
$ psql -d pgbench -c select
pg_size_pretty(pg_relation_size('pgbench_accounts'));
pg_size_pretty
313 GB
$ psql -d pgbench -c select
pg_size_pretty(pg_relation_size('pgbench_accounts_pkey'));
pg_size_pretty
52 GB
$ time psql -d pgbench -c select count(*) from pgbench_accounts
count
25
real18m48.363s
user0m0.010s
sys0m0.000s
The only thing wrong with the patch sent already needed to reach this
point was this line:
for (k = 0; k naccounts * scale; k++)
Which needed a (int64) cast for the multiplied value in the middle there.
Unfortunately the actual test itself doesn't run yet. Every line I see
when running the SELECT-only test says:
client 0 sending SELECT abalance FROM pgbench_accounts WHERE aid = 1;
So something about the updated random generation code isn't quite right
yet. Now that I have this monster built, I'm going to leave it on the
server until I can sort that out, which hopefully will finish up in the
next day or so.
--
Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us
PostgreSQL 9.0 High Performance: http://www.2ndQuadrant.com/books
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Varlena and binary
=?utf-8?q?Rados=C5=82aw_Smogura?= m...@smogura.eu writes: I'm sending small patch for textsend. It reduces unnecessary copies, and memory usage for duplication of varlena data. May you look? This code will break the day that text and bytea don't have the same internal representation, which seems likely to be soon. Barring some compelling evidence of a major performance improvement obtainable this way, I don't think we want this patch. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Where the Quals are actually 'List'ed
Vaibhav Kaushal vaibhavkaushal...@gmail.com writes: Hi, I find that ExecInitExpr creates a ExprState tree (and so say the comments above the function in the source). Also, it seems to decide which function would get called when the expression is to be evaluated when ExecQual runs, by setting the function pointer, for example: bstate-xprstate.evalfunc = (ExprStateEvalFunc)ExecEvalAnd; But ExecQual goes through a List, whereas, ExecInitExpr creates a tree. So is that same tree converted to a qual list or are we adding some more information. ExecInitExpr produces a List of state trees from a List of expr trees --- look at the last case in its switch. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] A different approach to extension NO USER DATA feature
Tom Lane t...@sss.pgh.pa.us writes: one I'd been thinking about a bit was OIDs of modules this one depends on. The current design doesn't cope very well with modules that depend on other ones. Or even at all. I guess here modules is referring to shared object libraries, right? Or are you already thinking about extension that depend on other extensions, like earthdistance depends on cube? Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sync Rep for 2011CF1
Robert Haas wrote: On Sun, Jan 30, 2011 at 11:44 AM, Robert Haas robertmh...@gmail.com wrote: Time's running short - do you have an updated patch? This patch hasn't been updated in more than three weeks. I assume this should now be marked Returned with Feedback, and we'll revisit synchronous replication for 9.2? Seems it is time for someone else to take the patch and complete it? Who can do that? -- Bruce Momjian br...@momjian.ushttp://momjian.us EnterpriseDB http://enterprisedb.com + It's impossible for everything to be true. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] A different approach to extension NO USER DATA feature
Dimitri Fontaine dimi...@2ndquadrant.fr writes: Tom Lane t...@sss.pgh.pa.us writes: one I'd been thinking about a bit was OIDs of modules this one depends on. The current design doesn't cope very well with modules that depend on other ones. Or even at all. I guess here modules is referring to shared object libraries, right? Or are you already thinking about extension that depend on other extensions, like earthdistance depends on cube? Sorry, I meant module == extension. If it's not intended that we try to support dependent extensions yet, I'd be fine with leaving that for 9.2. However, if earthdistance already has such a dependency, maybe we can't put that issue off. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sync Rep for 2011CF1
On Mon, 2011-02-07 at 09:55 -0500, Robert Haas wrote: On Sun, Jan 30, 2011 at 11:44 AM, Robert Haas robertmh...@gmail.com wrote: Time's running short - do you have an updated patch? This patch hasn't been updated in more than three weeks. I assume this should now be marked Returned with Feedback, and we'll revisit synchronous replication for 9.2? Hi Robert, I have time to complete that in next two weeks, but you are right I haven't had it in last few weeks. Cheers -- Simon Riggs http://www.2ndQuadrant.com/books/ PostgreSQL Development, 24x7 Support, Training and Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] exposing COPY API
On 02/04/2011 05:49 AM, Itagaki Takahiro wrote: Here is a demonstration to support jagged input files. It's a patch on the latest patch. The new added API is: bool NextLineCopyFrom( [IN] CopyState cstate, [OUT] char ***fields, [OUT] int *nfields, [OUT] Oid *tupleOid) It just returns separated fields in the next line. Fortunately, I need no extra code for it because it is just extracted from NextCopyFrom(). I'm willing to include the change into copy APIs, but we still have a few issues. See below. I've looked at this, and I think it will do what I want. I haven't had time to play with it, although I hope to soon. AIUI, it basically hands back the raw parsed strings to the user, who then has the responsibility of constructing Datum and Nulls arrays to form the tuple. That should be all I need. So +1 from me for including it. In fact, +10. And many thanks. I think we need a better name though. NextCopyFromRawFields maybe. On Fri, Feb 4, 2011 at 16:53, Andrew Dunstanand...@dunslane.net wrote: The problem with COPY FROM is that nobody's come up with a good syntax for allowing it as a FROM target. Doing what I want via FDW neatly gets us around that problem. But I'm quite OK with doing the hard work inside the COPY code - that's what my working prototype does in fact. I think it is not only syntax issue. I found an issue that we hard to support FORCE_NOT_NULL option for extra fields. See FIXME in the patch. It is a fundamental problem to support jagged fields. I don't think we need to worry about it. The caller will have access to the raw strings so they can handle it. In fact, I'd take out that bit of code in NextCopyLine_From, and replace it with a comment about how it's the caller's responsibility to handle. One thing I'd like is to to have file_fdw do something we can't do another way. currently it doesn't, so it's nice but uninteresting. BTW, how do you determine which field is shifted in your broken CSV file? For example, the case you find AB,CD,EF for 2 columns tables. I could provide a raw CSV reader for jagged files, but you still have to cook the returned fields into a proper tuple... I answered this previously, but in the case of a text array it won't even arise - the array will have however many fields have been read. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] More extension issues: ownership and search_path
Tom Lane t...@sss.pgh.pa.us writes: One is ownership. Since we don't record the identity of the user who created an extension, there's no way for pg_dump to ensure that it's recreated by the same user. I think we'll regret that in future even if you don't think it's problematic today. In particular, I foresee eventually allowing non-superusers to load extensions, so I think this is going to follow pretty much the same trajectory as procedural languages, which we originally did not track ownership for. In short, I think we'd better add an extowner column to pg_extension. Agreed. There's no need to have it now but we will add it at some point. So if now is when that works the best for you, I'm happy to see that happen :) Would it help that I prepare some of those modifications, as patches over the extension's patch that you started from? Another is the search_path setting. Currently this is actually rather broken since pg_dump makes no effort at all to ensure that search_path is the same at reload time as it was when the extension was created, and thus the contained objects could easily go into the wrong schema. Well there's some code to place the extension's schema at the first place in the search_path before executing the script, already. But even without thinking about dump/reload, it seems to me that it would be a good thing for reproducibility if CREATE EXTENSION were to forcibly set search_path before running the extension's SQL script. The level-zero version of that would be to do the equivalent of SET LOCAL search_path = @extschema@ such that the path only contains the target schema and nothing else. Spelled this way, I could see attaching SET to CREATE EXTENSION the same way we did for CREATE FUNCTION. I'm not too sure about what other set of GUCs would be useful to support here, but that would be a good mechanism to use I would say. The trouble with this simplistic approach is that it doesn't work nicely if one extension depends on another --- you probably want the search path to include the schema(s) the required extensions got installed into. Of course inter-extension dependencies aren't going to work anyway unless pg_dump knows about them so it can make sure to load the extensions in the right order. So where I think we're going to end up is adding a clause along the line of USING list-of-extension-names to CREATE EXTENSION, storing those dependencies explicitly, and having the CREATE EXTENSION code set search_path to the target schema followed by the target schema(s) of the USING extensions. Not sure if this is worth doing immediately or can be left for 9.2. At least in the contrib modules, there are no interdependencies, and I don't know whether any exist in the wider world of existing extensions. We have a interdependency in contrib, earthdistance depends on cube already. In skytools, PGQ is composed of several parts that are packaged as a single extension now, but whose sources are maintained in separate parts. Other than that, I think tricky scripts that depends on some objects of the extension to already be usable will be simpler to solve with splitting the extensions and adding some dependency. So while we said this is 9.2 material, if you want to tackle the whole search_path at restore time issue (I did only the creation namespace, thinking it would be enough) fully, you need dependency too. I think we should then register some core components as extensions for the sake of interdependencies here, too. \dx would then list PostgreSQL itself and its (major) version, and the installed PL would need to be there, and maybe some features too — but the way we handle bugfix only minor upgrade makes that useless for us I think. Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] A different approach to extension NO USER DATA feature
On Feb6, 2011, at 19:23 , Tom Lane wrote: After a bit of thought I believe that we can fix this if we are willing to teach pg_dump explicitly about extension configuration tables. The behavior we want for those is for the table schema definition to never be dumped (the table should always be created by CREATE EXTENSION), but for some subset of the table data to get dumped, excluding any system-provided rows. An extension that wants to make use of that ability would probably need to add a boolean column system_data or something similar to its configuration tables. Having done so, the extension's SQL script could call a function that identifies the configuration table and tells how to decide which rows to dump, along the lines of We could avoid the need for a per-row system_data flag if we required extensions to split user-editable and system-provided configuration data into different tables. For convenient access to the configuration data, the extension could let the user-editable table inherit from the system-provided one, or use a view to combine the two. We'd then only need a per-table flag which tells pg_dump to dump the data (but not the structure), so instead of a pg_extension_partial_dump (table_name regclass, where_condition text) we'd have pg_extension_userdata(table_name regclass). Apart from getting rid of the slightly ugly userdata flag-column, this would also make it easier for users to distinguish between user-editable and system-provided configuration options. best regards, Florian Pflug -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range Types - cache lookup failed for function
On Sun, 2011-02-06 at 20:10 +0100, Erik Rijkers wrote: I was trying where intrange @ integer which admittedly is not in the documentation, but does already half work, and would be really convenient to have. As it stands the construct seems to fail after ANALYZE, when there is more than 1 row: Thank you for the report! I actually did make some mention of that in the documentation, albeit brief (in the operators table, using timestamp as an example). I have a fix for it. There may still be an issue with the constructors like range(1,2), so I'll look into it a little more, but an updated patch should come soon. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] More extension issues: ownership and search_path
Dimitri Fontaine dimi...@2ndquadrant.fr writes: Tom Lane t...@sss.pgh.pa.us writes: I think we'd better add an extowner column to pg_extension. Agreed. There's no need to have it now but we will add it at some point. So if now is when that works the best for you, I'm happy to see that happen :) Would it help that I prepare some of those modifications, as patches over the extension's patch that you started from? No, I've hacked the code enough already that merging would be painful. I'll keep working on it. Another is the search_path setting. Currently this is actually rather broken since pg_dump makes no effort at all to ensure that search_path is the same at reload time as it was when the extension was created, and thus the contained objects could easily go into the wrong schema. Well there's some code to place the extension's schema at the first place in the search_path before executing the script, already. Oh, duh, I'd forgotten about the OverrideSearchPath usage. So never mind the above claim. But I still think it'd be a good idea to ensure that the search path is the same during dump/reload as it was the first time, and the current arrangement isn't going to ensure that. So while we said this is 9.2 material, if you want to tackle the whole search_path at restore time issue (I did only the creation namespace, thinking it would be enough) fully, you need dependency too. Quite aside from search path, cross-extension dependencies simply aren't going to work unless pg_dump knows about them so it can load the extensions in the right order. I had forgotten about the earthdistance case, but given that I think we probably can't put this issue off. I think we should then register some core components as extensions for the sake of interdependencies here, too. Maybe that sort of refactoring could be done in 9.2 or further out. I don't see it happening for 9.1. I'm not really sure what the value is anyway. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] A different approach to extension NO USER DATA feature
Florian Pflug f...@phlo.org writes: We could avoid the need for a per-row system_data flag if we required extensions to split user-editable and system-provided configuration data into different tables. For convenient access to the configuration data, the extension could let the user-editable table inherit from the system-provided one, or use a view to combine the two. Yeah, this is another approach that one could take instead of having per-row flags. I'm not sure that it's better, much less so much better that we should force extensions to do it that way and not the other. But it's definitely another argument against making a hard-wired assumption that there will be a flag column. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range Types
On Mon, 2011-02-07 at 13:33 +0100, Dimitri Fontaine wrote: Hi, Jeff Davis pg...@j-davis.com writes: * Should pg_range reference the btree opclass or the compare function directly? I would say yes. We use the btree opclass in other similar situations. Ok, but what should the parameter to CREATE TYPE ... AS RANGE be then? CREATE TYPE foo AS RANGE ( SUBTYPE = ... SUBTYPE_BTREE_OPERATOR_CLASS = ... ); is a little verbose. Ideas? Is there any reason to restrict who's get to use the feature? I don't see any… Mostly just paranoia. If they define a strange canonical function, maybe that would cause a problem. Then again, they would have to define that in C to cause a problem anyway. I'll leave it as superuser-only for now, and see if anyone else raises potential problems. * Should the SQL (inlinable) function length, which relies on polymorphic -, be immutable, strict, or volatile? I would think stable: polymorphic means that the function implementing the - operator depends on the argument. I don't recall that it depends on them in a volatile way… except if you change the operator definition, which is possible to do (so not immutable). I was concerned about someone defining - with a stable or volatile function as the definition. I'm not sure if that is a reasonable concern or not. * Later we might consider whether we should include btree_gist in core, to make range types more useful with exclusion constraints out-of-the-box. This should be left for later, I'm just including this for the archives so it doesn't get lost. I would expect the extension to have something to offer here. Yes. With extensions and PGXN, I would hope that installing btree_gist would not be much of a problem. However, I eventually intend to submit features like RANGE KEY, a language extension that would need something like btree_gist to work very well at all. Technically btree_gist is not required, but in practice it is necessary to use ranges and exclusion constraints together effectively. Regards, Jeff Davis -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] limiting hint bit I/O
On Mon, Feb 7, 2011 at 10:48 AM, Bruce Momjian br...@momjian.us wrote: Robert Haas wrote: On Sat, Feb 5, 2011 at 4:31 PM, Bruce Momjian br...@momjian.us wrote: Uh, in this C comment: + ? ? ? ?* or not we want to take the time to write it. ?We allow up to 5% of + ? ? ? ?* otherwise-not-dirty pages to be written due to hint bit changes, 5% of what? ?5% of all buffers? ?5% of all hint-bit-dirty ones? ?Can you clarify this in the patch? 5% of buffers that are hint-bit-dirty but not otherwise dirty. ISTM that's exactly what the comment you just quoted says on its face, but I'm open to some other wording you want to propose. How about: otherwise-not-dirty - only-hint-bit-dirty So 95% of your hint bit modificates are discarded if the pages is not otherwise dirtied? That seems pretty radical. No, it's more subtle than that, although I admit it *is* radical. There are three ways that pages can get written out to disk: 1. Checkpoints. 2. Background writer activity. 3. Backends writing out dirty buffers because there are no clean buffers available to allocate. What the latest version of the patch implements is: 1. Checkpoints no longer write only-hint-bit-dirty pages to disk. Since a checkpoint doesn't evict pages from memory, the hint bits are still there to be written out (or not) by (2) or (3), below. 2. When the background writer's cleaning scan hits an only-hint-bit-dirty page, it writes it, same as before. This definitely doesn't result in the loss of any hint bits. 3. When a backend writes out a dirty buffer itself, because there are no clean buffers available to allocate, it initially writes them. But if there are more than 100 such pages per block of 2000 allocations, it recycles any after the first 100 without writing them. In normal operation, I suspect that there will be very little impact from this change. The change described in #1 may slightly reduce the size of some checkpoints, but it's unclear that it will be enough to be material. The change described in #3 will probably also not matter, because, in a well-tuned system, the background writer should be set aggressively enough to provide a supply of clean pages, and therefore backends shouldn't be doing many writes themselves, and therefore most buffer allocations will be of already-clean pages, and the logic described in #3 will probably never kick in. Even if they are writing a lot of buffers themselves, the logic in #3 still won't kick in if many of the pages being written are actually dirty - it will only matter if the backends are writing out lots and lots of pages *solely because they are only-hint-bit-dirty*. Where I expect this to make a big difference is on sequential scans of just-loaded tables. In that case, the BufferAccessStrategy machinery will force the backend to reuse the same buffers over and over again, and all of those pages will be only-hint-bit-dirty. So the backend has to do a write for every page it allocates, and even though those writes are being absorbed by the OS cache, it's still slow. With this patch, what will happen is that the backend will write about 100 pages, then perform the next 1900 allocations without writing, then write another 100 pages, etc. So at the end of the scan, instead of having written an amount of data equal to the size of the table, we will have written 5% of that amount, and 5% of the hint bits will be on disk. Each subsequent scan will get another 5% of the hint bits on disk until after 20 scans they are all set. So the work of setting the hint bits is spread out across the first 20 table scans instead of all being done the first time through. Clearly, there's further jiggering that can be done here. But the overall goal is simply that some of our users don't seem to like it when the first scan of a newly loaded table generates a huge storm of *write* traffic. Given that the hint bits appear to be quite important from a performance perspective (see benchmark numbers upthread), we don't really have the option of just not writing them - but we can try to not to do it all at once, if we think that's an improvement, which I think is likely. Overall, I'm inclined to move this patch to the next CommitFest and forget about it for now. I don't think we're going to get enough testing of this in the next week to be really confident that it's right. I might be willing to commit with some more moderate amount of testing if we were right at the beginning of a development cycle, figuring that we'd shake out any warts as the cycle went along, but this isn't seeming like the right time for this kind of a change. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] OpenVMS - an effort which needs guidance and support.
peder...@ccsscorp.com (Bill Pedersen) writes: I look forward to hearing from people in the PostgreSQL community as well as from others interested in this effort. To a number of us, it's academically interesting, though, as we don't have VMS systems, it's not likely to be super-easy to assist in the matter. It certainly would be interesting to see how easy or difficult the port would be. I suspect that's a more interesting port than, say, Digital UNIX, these days. -- (reverse (concatenate 'string ofni.sesabatadxunil @ enworbbc)) http://linuxfinances.info/info/slony.html On the other hand, you have different fingers. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] More extension issues: ownership and search_path
Tom Lane t...@sss.pgh.pa.us writes: No, I've hacked the code enough already that merging would be painful. I'll keep working on it. I supposed so much, but got to ask :) Oh, duh, I'd forgotten about the OverrideSearchPath usage. So never mind the above claim. But I still think it'd be a good idea to ensure that the search path is the same during dump/reload as it was the first time, and the current arrangement isn't going to ensure that. Right. Something automated would be best (no user intervention), it would force extension scripts to be self-contained or to explicitly declare all their external dependencies. I'm in fact doping out entirely my previous SET idea. So while we said this is 9.2 material, if you want to tackle the whole search_path at restore time issue (I did only the creation namespace, thinking it would be enough) fully, you need dependency too. Quite aside from search path, cross-extension dependencies simply aren't going to work unless pg_dump knows about them so it can load the extensions in the right order. I had forgotten about the earthdistance case, but given that I think we probably can't put this issue off. Well in fact the ordering already works because some earthdistance objects depend on some cube objects, and pg_dump sees that in pg_depend. I think we should then register some core components as extensions for the sake of interdependencies here, too. Maybe that sort of refactoring could be done in 9.2 or further out. I don't see it happening for 9.1. I'm not really sure what the value is anyway. Imagine that you wrote a set of plpgsql and plpythonu functions that you want to maintain as an extension. You certainly want to mark that this extension depends on the procedural languages being installed, right? Also, I didn't bite this bullet, but maybe we should provide core PLs as extension. Then CREATE LANGUAGE would maybe get deprecated and only valid when used in an extension's script — or the next patch (UPGRADE) will take care of create a plpythonu extension then attaching the PL into it. Again, pg_depend already allows pg_dump to create plpythonu before it creates the extension that depends on it, though. Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Range Types
Jeff Davis pg...@j-davis.com writes: Ok, but what should the parameter to CREATE TYPE ... AS RANGE be then? CREATE TYPE foo AS RANGE ( SUBTYPE = ... SUBTYPE_BTREE_OPERATOR_CLASS = ... ); is a little verbose. Ideas? I would think CREATE TYPE foo AS RANGE (bar) USING (btree_ops); The USING clause is optional, because you generally have a default btree opclass for the datatype. Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sync Rep for 2011CF1
On Mon, Feb 7, 2011 at 11:33 AM, Simon Riggs si...@2ndquadrant.com wrote: On Mon, 2011-02-07 at 09:55 -0500, Robert Haas wrote: On Sun, Jan 30, 2011 at 11:44 AM, Robert Haas robertmh...@gmail.com wrote: Time's running short - do you have an updated patch? This patch hasn't been updated in more than three weeks. I assume this should now be marked Returned with Feedback, and we'll revisit synchronous replication for 9.2? I have time to complete that in next two weeks, but you are right I haven't had it in last few weeks. Well, the current CommitFest ends in one week, and we need to leave time for someone (Heikki, most likely) to review, so there's really only a couple of days left. Bruce's suggestion of having someone else pick it up seems like it might work. The obvious candidates are probably Heikki Linnakangas, Tom Lane, Fujii Masao, and (if you squint a little) me. I am not clear that any of those people have the necessary time available immediately, however. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sync Rep for 2011CF1
On Mon, Feb 7, 2011 at 12:28 PM, Robert Haas robertmh...@gmail.com wrote: On Mon, Feb 7, 2011 at 11:33 AM, Simon Riggs si...@2ndquadrant.com wrote: On Mon, 2011-02-07 at 09:55 -0500, Robert Haas wrote: On Sun, Jan 30, 2011 at 11:44 AM, Robert Haas robertmh...@gmail.com wrote: Time's running short - do you have an updated patch? This patch hasn't been updated in more than three weeks. I assume this should now be marked Returned with Feedback, and we'll revisit synchronous replication for 9.2? I have time to complete that in next two weeks, but you are right I haven't had it in last few weeks. Well, the current CommitFest ends in one week, and we need to leave time for someone (Heikki, most likely) to review, so there's really only a couple of days left. Bruce's suggestion of having someone else pick it up seems like it might work. The obvious candidates are probably Heikki Linnakangas, Tom Lane, Fujii Masao, and (if you squint a little) me. I am not clear that any of those people have the necessary time available immediately, however. I just spoke to my manager at EnterpriseDB and he cleared my schedule for the next two days to work on this. So I'll go hack on this now. I haven't read the patch yet so I don't know for sure how quite I'll be able to get up to speed on it, so if someone who is more familiar with this code wants to grab the baton away from me, feel free. Otherwise, I'll see what I can do with it. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SSI patch version 14
Kevin Grittner kgri...@wicourts.gov wrote: Kevin Grittner wrote: Jeff Davis wrote: What does PredicateLockTuple do that needs a share lock? Does a pin suffice? If one process is reading a tuple and another is writing it (e.g., UPDATE or DELETE) the concern is that we need to be able to guarantee that either the predicate lock appears in time for the writer to see it on the call to CheckForSerializableConflictIn or the reader sees the MVCC changes in CheckForSerializableConflictOut. It's OK if the conflict is detected both ways, but if it's missed both ways then we could have a false negative, allowing anomalies to slip through. It didn't seem to me on review that acquiring the predicate lock after releasing the shared buffer lock (and just having the pin) would be enough to ensure that a write which followed that would see the predicate lock. reader has pin and shared lock writer increments pin count reader releases shared lock writer acquires exclusive lock writer checks predicate lock and fails to see one reader adds predicate lock we have a problem Hmmm... Or do we? If both sides were careful to record what they're doing before checking for a conflict, the pin might be enough. I'll check for that. In at least one of those moves I was moving the predicate lock acquisition from after the conflict check to before, but maybe I didn't need to move it quite so far Some of these calls are placed where there is no reasonable choice but to do them while holding a buffer lock. There are some which *might* be able to be moved out, but I'm inclined to say that should be on a list of possible performance improvements in future releases. My concern is that the code paths are complicated enough to make it hard to confidently desk-check such changes, we don't yet have a good way to test whether such a change is introducing a race condition. The src/test/isolation tests are good at proving the fundamental correctness of the logic in predicate.c, and the DBT-2 stress tests Dan ran are good at flushing out race conditions within predicate.c code; but we don't have a test which is good at pointing out race conditions based on *placement of the calls* to predicate.c from other code. Until we have such tests, I think we should be cautious about risking possible hard-to-reproduce correctness violations to shave a few nanoseconds off the time a shared buffer lock is held. Particularly since we're so close to the end of the release cycle. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sync Rep for 2011CF1
Robert Haas robertmh...@gmail.com writes: ... Well, the current CommitFest ends in one week, ... Really? I thought the idea for the last CF of a development cycle was that it kept going till we'd dealt with everything. Arbitrarily rejecting stuff we haven't dealt with doesn't seem fair. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] A different approach to extension NO USER DATA feature
On Feb 7, 2011, at 8:57 AM, Tom Lane wrote: Yeah, this is another approach that one could take instead of having per-row flags. I'm not sure that it's better, much less so much better that we should force extensions to do it that way and not the other. But it's definitely another argument against making a hard-wired assumption that there will be a flag column. Would the flag column usually be invisible, like the system oid column? Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] More extension issues: ownership and search_path
Dimitri Fontaine dimi...@2ndquadrant.fr writes: Tom Lane t...@sss.pgh.pa.us writes: Quite aside from search path, cross-extension dependencies simply aren't going to work unless pg_dump knows about them so it can load the extensions in the right order. I had forgotten about the earthdistance case, but given that I think we probably can't put this issue off. Well in fact the ordering already works because some earthdistance objects depend on some cube objects, and pg_dump sees that in pg_depend. Really? I think it's more likely just luckily working because of the alphabetical-ordering default. To make it work reliably off of those dependencies, we'd need some code to pull up the dependency links from the individual objects to the module level, and we haven't got that. I'd prefer to make the module dependencies explicit anyway. Imagine that you wrote a set of plpgsql and plpythonu functions that you want to maintain as an extension. You certainly want to mark that this extension depends on the procedural languages being installed, right? Interesting point. It's all right at the moment because I tweaked pg_dump_sort.c so that procedural languages are dumped before modules. But maybe we should convert the PLs to modules. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] More extension issues: ownership and search_path
On Feb 7, 2011, at 9:20 AM, Dimitri Fontaine wrote: Also, I didn't bite this bullet, but maybe we should provide core PLs as extension. Then CREATE LANGUAGE would maybe get deprecated and only valid when used in an extension's script — or the next patch (UPGRADE) will take care of create a plpythonu extension then attaching the PL into it. I anticipate dependencies becoming a big deal. I already have ideas for extensions that depend on citext, for example (domains for time zone, email address, etc.). And yeah, some of those might depend on procedural languages. FWIW, I've been putting PL prereqs in META.json files on PGXN. pgTAP, for example, requires PL/pgSQL: http://master.pgxn.org/dist/pgTAP/pgTAP-0.25.0.json Best, David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] More extension issues: ownership and search_path
On Feb 7, 2011, at 9:51 AM, Tom Lane wrote: Interesting point. It's all right at the moment because I tweaked pg_dump_sort.c so that procedural languages are dumped before modules. But maybe we should convert the PLs to modules. s/modules/extensions/? David -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] A different approach to extension NO USER DATA feature
David E. Wheeler da...@kineticode.com writes: On Feb 7, 2011, at 8:57 AM, Tom Lane wrote: Yeah, this is another approach that one could take instead of having per-row flags. I'm not sure that it's better, much less so much better that we should force extensions to do it that way and not the other. But it's definitely another argument against making a hard-wired assumption that there will be a flag column. Would the flag column usually be invisible, like the system oid column? No, it's just a column. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] More extension issues: ownership and search_path
David E. Wheeler da...@kineticode.com writes: On Feb 7, 2011, at 9:51 AM, Tom Lane wrote: Interesting point. It's all right at the moment because I tweaked pg_dump_sort.c so that procedural languages are dumped before modules. But maybe we should convert the PLs to modules. s/modules/extensions/? Yeah, I keep saying module. Memo to self: grep the whole patch for module before committing. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sync Rep for 2011CF1
On Mon, 2011-02-07 at 12:39 -0500, Robert Haas wrote: I just spoke to my manager at EnterpriseDB and he cleared my schedule for the next two days to work on this. So I'll go hack on this now. I haven't read the patch yet so I don't know for sure how quite I'll be able to get up to speed on it, so if someone who is more familiar with this code wants to grab the baton away from me, feel free. Otherwise, I'll see what I can do with it. Presumably you have a reason for declaring war? I'm sorry for that, I really am. -- Simon Riggs http://www.2ndQuadrant.com/books/ PostgreSQL Development, 24x7 Support, Training and Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sync Rep for 2011CF1
On Mon, Feb 7, 2011 at 12:43 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: ... Well, the current CommitFest ends in one week, ... Really? I thought the idea for the last CF of a development cycle was that it kept going till we'd dealt with everything. Arbitrarily rejecting stuff we haven't dealt with doesn't seem fair. Uh, we did that with 8.4 and it was a disaster. The CommitFest lasted *five months*. We've been doing schedule-based CommitFests ever since and it's worked much better. I agree it's unfair to reject things without looking at them, and I'd like to avoid that if at all possible, but punting things because they need more work than can be done in the time available is another thing entirely. I do NOT want to still be working on the items for this CommitFest in June - that's about when I'd like to be releasing beta3. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sync Rep for 2011CF1
On Mon, Feb 7, 2011 at 12:59 PM, Simon Riggs si...@2ndquadrant.com wrote: On Mon, 2011-02-07 at 12:39 -0500, Robert Haas wrote: I just spoke to my manager at EnterpriseDB and he cleared my schedule for the next two days to work on this. So I'll go hack on this now. I haven't read the patch yet so I don't know for sure how quite I'll be able to get up to speed on it, so if someone who is more familiar with this code wants to grab the baton away from me, feel free. Otherwise, I'll see what I can do with it. Presumably you have a reason for declaring war? I'm sorry for that, I really am. What the hell are you talking about? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] More extension issues: ownership and search_path
Dimitri Fontaine dimi...@2ndquadrant.fr writes: Tom Lane t...@sss.pgh.pa.us writes: I think we'd better add an extowner column to pg_extension. Agreed. There's no need to have it now but we will add it at some point. So if now is when that works the best for you, I'm happy to see that happen :) BTW, on trying this I notice that pg_dump's default approach to ownership doesn't work because of the lack of an ALTER EXTENSION OWNER TO command. I'm going to go ahead and add extowner to the catalog anyway, because it's easy and I'm convinced we're going to want it later. But I don't feel like writing ALTER EXTENSION OWNER TO right now, so pg_dump will continue its current behavior of creating the extension as the user running the script. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] More extension issues: ownership and search_path
I wrote: ... So where I think we're going to end up is adding a clause along the line of USING list-of-extension-names to CREATE EXTENSION, storing those dependencies explicitly, and having the CREATE EXTENSION code set search_path to the target schema followed by the target schema(s) of the USING extensions. On reflection, the set of extensions that an extension depends on is obviously a property of the extension, which means it ought to be specified in the extension's control file, not in the CREATE EXTENSION command. So now I'm thinking something like requires = 'foo, bar, baz' in the file. We could even imagine autoloading such dependencies if they're not already installed, but that's a frammish for later. (One objection to autoloading is it's not clear which schema to drop the other extensions into.) regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] WIP: RangeTypes
On sön, 2011-01-30 at 14:52 -0800, Jeff Davis wrote: * naming issues: - period - tsrange ? - periodtz - tstzrange ? - intrange - int4range Have you considered a grammar approach like for arrays, so that you would write something like CREATE TABLE ... ( foo RANGE OF int ); instead of explicitly creating a range type for every scalar type in existence? I think that that might be easier to use in the common case. I guess the trick might be how to store and pass the operator class and some other parameters. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Spread checkpoint sync
Greg Smith g...@2ndquadrant.com wrote: As a larger statement on this topic, I'm never very excited about redesigning here starting from any point other than saw a bottleneck doing x on a production system. There's a long list of such things already around waiting to be addressed, and I've never seen any good evidence of work related to hint bits being on it. Please correct me if you know of some--I suspect you do from the way you're brining this up. There are occasional posts from those wondering why their read-only queries are so slow after a bulk load, and why they are doing heavy writes. (I remember when I posted about that, as a relative newbie, and I know I've seen others.) I think worst case is probably: - Bulk load data. - Analyze (but don't vacuum) the new data. - Start a workload with a lot of small, concurrent random reads. - Watch performance tank when the write cache gluts. This pattern is why we've adopted a pretty strict rule in our shop that we run VACUUM FREEZE ANALYZE between a bulk load and putting the database back into production. It's probably a bigger issue for those who can't do that. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] More extension issues: ownership and search_path
Tom Lane t...@sss.pgh.pa.us writes: On reflection, the set of extensions that an extension depends on is obviously a property of the extension, which means it ought to be specified in the extension's control file, not in the CREATE EXTENSION command. So now I'm thinking something like requires = 'foo, bar, baz' +1 And that can change at upgrade time, of course, but that's another story. Ditto for recommends and conflict dependency types, that's material for 9.2 at best. in the file. We could even imagine autoloading such dependencies if they're not already installed, but that's a frammish for later. (One objection to autoloading is it's not clear which schema to drop the other extensions into.) Well I don't see why it wouldn't be the same schema in case of auto solving dependencies, but I don't see a pressing need to have automatic dependency following at install time (we're still in the realm of dpkg, we'll get into apt-get next) :) That said, we should do something about ALTER EXTENSION SET SCHEMA and the relocatable property. I'm thinking that an extension stops being relocatable as soon as any of its reverse dependencies (all the tree) is not relocatable. Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] limiting hint bit I/O
2011/2/7 Robert Haas robertmh...@gmail.com: On Mon, Feb 7, 2011 at 10:48 AM, Bruce Momjian br...@momjian.us wrote: Robert Haas wrote: On Sat, Feb 5, 2011 at 4:31 PM, Bruce Momjian br...@momjian.us wrote: Uh, in this C comment: + ? ? ? ?* or not we want to take the time to write it. ?We allow up to 5% of + ? ? ? ?* otherwise-not-dirty pages to be written due to hint bit changes, 5% of what? ?5% of all buffers? ?5% of all hint-bit-dirty ones? ?Can you clarify this in the patch? 5% of buffers that are hint-bit-dirty but not otherwise dirty. ISTM that's exactly what the comment you just quoted says on its face, but I'm open to some other wording you want to propose. How about: otherwise-not-dirty - only-hint-bit-dirty So 95% of your hint bit modificates are discarded if the pages is not otherwise dirtied? That seems pretty radical. No, it's more subtle than that, although I admit it *is* radical. There are three ways that pages can get written out to disk: 1. Checkpoints. 2. Background writer activity. 3. Backends writing out dirty buffers because there are no clean buffers available to allocate. What the latest version of the patch implements is: 1. Checkpoints no longer write only-hint-bit-dirty pages to disk. Since a checkpoint doesn't evict pages from memory, the hint bits are still there to be written out (or not) by (2) or (3), below. 2. When the background writer's cleaning scan hits an only-hint-bit-dirty page, it writes it, same as before. This definitely doesn't result in the loss of any hint bits. 3. When a backend writes out a dirty buffer itself, because there are no clean buffers available to allocate, it initially writes them. But if there are more than 100 such pages per block of 2000 allocations, it recycles any after the first 100 without writing them. In normal operation, I suspect that there will be very little impact from this change. The change described in #1 may slightly reduce the size of some checkpoints, but it's unclear that it will be enough to be material. The change described in #3 will probably also not matter, because, in a well-tuned system, the background writer should be set aggressively enough to provide a supply of clean pages, and therefore backends shouldn't be doing many writes themselves, and therefore most buffer allocations will be of already-clean pages, and the logic described in #3 will probably never kick in. Even if they are writing a lot of buffers themselves, the logic in #3 still won't kick in if many of the pages being written are actually dirty - it will only matter if the backends are writing out lots and lots of pages *solely because they are only-hint-bit-dirty*. Where I expect this to make a big difference is on sequential scans of just-loaded tables. In that case, the BufferAccessStrategy machinery will force the backend to reuse the same buffers over and over again, and all of those pages will be only-hint-bit-dirty. So the backend has to do a write for every page it allocates, and even though those writes are being absorbed by the OS cache, it's still slow. With this patch, what will happen is that the backend will write about 100 pages, then perform the next 1900 allocations without writing, then write another 100 pages, etc. So at the end of the scan, instead of having written an amount of data equal to the size of the table, we will have written 5% of that amount, and 5% of the hint bits will be on disk. Each subsequent scan will get another 5% of the hint bits on disk until after 20 scans they are all set. So the work of setting the hint bits is spread out across the first 20 table scans instead of all being done the first time through. Clearly, there's further jiggering that can be done here. But the overall goal is simply that some of our users don't seem to like it when the first scan of a newly loaded table generates a huge storm of *write* traffic. Given that the hint bits appear to be quite important from a performance perspective (see benchmark numbers upthread), those are not real benchmarks, just quick guess to check behavior. (and I agree it looks good, but I also got inconsistent results, the patched postgresql hardly reach the same speed of the original 9.1devel even after 200 hundreds select of your testcase) we don't really have the option of just not writing them - but we can try to not to do it all at once, if we think that's an improvement, which I think is likely. Overall, I'm inclined to move this patch to the next CommitFest and forget about it for now. I don't think we're going to get enough testing of this in the next week to be really confident that it's right. I might be willing to commit with some more moderate amount of testing if we were right at the beginning of a development cycle, figuring that we'd shake out any warts as the cycle went along, but this isn't seeming like the right time
Re: [HACKERS] limiting hint bit I/O
2011/2/7 Cédric Villemain cedric.villemain.deb...@gmail.com: 2011/2/7 Robert Haas robertmh...@gmail.com: On Mon, Feb 7, 2011 at 10:48 AM, Bruce Momjian br...@momjian.us wrote: Robert Haas wrote: On Sat, Feb 5, 2011 at 4:31 PM, Bruce Momjian br...@momjian.us wrote: Uh, in this C comment: + ? ? ? ?* or not we want to take the time to write it. ?We allow up to 5% of + ? ? ? ?* otherwise-not-dirty pages to be written due to hint bit changes, 5% of what? ?5% of all buffers? ?5% of all hint-bit-dirty ones? ?Can you clarify this in the patch? 5% of buffers that are hint-bit-dirty but not otherwise dirty. ISTM that's exactly what the comment you just quoted says on its face, but I'm open to some other wording you want to propose. How about: otherwise-not-dirty - only-hint-bit-dirty So 95% of your hint bit modificates are discarded if the pages is not otherwise dirtied? That seems pretty radical. No, it's more subtle than that, although I admit it *is* radical. There are three ways that pages can get written out to disk: 1. Checkpoints. 2. Background writer activity. 3. Backends writing out dirty buffers because there are no clean buffers available to allocate. What the latest version of the patch implements is: 1. Checkpoints no longer write only-hint-bit-dirty pages to disk. Since a checkpoint doesn't evict pages from memory, the hint bits are still there to be written out (or not) by (2) or (3), below. 2. When the background writer's cleaning scan hits an only-hint-bit-dirty page, it writes it, same as before. This definitely doesn't result in the loss of any hint bits. 3. When a backend writes out a dirty buffer itself, because there are no clean buffers available to allocate, it initially writes them. But if there are more than 100 such pages per block of 2000 allocations, it recycles any after the first 100 without writing them. In normal operation, I suspect that there will be very little impact from this change. The change described in #1 may slightly reduce the size of some checkpoints, but it's unclear that it will be enough to be material. The change described in #3 will probably also not matter, because, in a well-tuned system, the background writer should be set aggressively enough to provide a supply of clean pages, and therefore backends shouldn't be doing many writes themselves, and therefore most buffer allocations will be of already-clean pages, and the logic described in #3 will probably never kick in. Even if they are writing a lot of buffers themselves, the logic in #3 still won't kick in if many of the pages being written are actually dirty - it will only matter if the backends are writing out lots and lots of pages *solely because they are only-hint-bit-dirty*. Where I expect this to make a big difference is on sequential scans of just-loaded tables. In that case, the BufferAccessStrategy machinery will force the backend to reuse the same buffers over and over again, and all of those pages will be only-hint-bit-dirty. So the backend has to do a write for every page it allocates, and even though those writes are being absorbed by the OS cache, it's still slow. With this patch, what will happen is that the backend will write about 100 pages, then perform the next 1900 allocations without writing, then write another 100 pages, etc. So at the end of the scan, instead of having written an amount of data equal to the size of the table, we will have written 5% of that amount, and 5% of the hint bits will be on disk. Each subsequent scan will get another 5% of the hint bits on disk until after 20 scans they are all set. So the work of setting the hint bits is spread out across the first 20 table scans instead of all being done the first time through. Clearly, there's further jiggering that can be done here. But the overall goal is simply that some of our users don't seem to like it when the first scan of a newly loaded table generates a huge storm of *write* traffic. Given that the hint bits appear to be quite important from a performance perspective (see benchmark numbers upthread), those are not real benchmarks, just quick guess to check behavior. (and I agree it looks good, but I also got inconsistent results, the patched postgresql hardly reach the same speed of the original 9.1devel even after 200 hundreds select of your testcase) we don't really have the option of just not writing them - but we can try to not to do it all at once, if we think that's an improvement, which I think is likely. Overall, I'm inclined to move this patch to the next CommitFest and forget about it for now. I don't think we're going to get enough testing of this in the next week to be really confident that it's right. I might be willing to commit with some more moderate amount of testing if we were right at the beginning of a development cycle, figuring that we'd shake out any warts as
Re: [HACKERS] Sync Rep for 2011CF1
Robert Haas robertmh...@gmail.com writes: done in the time available is another thing entirely. I do NOT want to still be working on the items for this CommitFest in June - that's about when I'd like to be releasing beta3. Except that's not how we work here. You want to change that with respect to the release management process and schedule (or lack thereof). Tradition and current practice say you need to reach consensus to be able to bypass compromising. Good luck with that. Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sync Rep for 2011CF1
On Mon, 2011-02-07 at 17:59 +, Simon Riggs wrote: On Mon, 2011-02-07 at 12:39 -0500, Robert Haas wrote: I just spoke to my manager at EnterpriseDB and he cleared my schedule for the next two days to work on this. So I'll go hack on this now. I haven't read the patch yet so I don't know for sure how quite I'll be able to get up to speed on it, so if someone who is more familiar with this code wants to grab the baton away from me, feel free. Otherwise, I'll see what I can do with it. Presumably you have a reason for declaring war? I'm sorry for that, I really am. Simon, My impression was that Robert had received a release from current responsibilities to help you with your patch, not that he was declaring war or some such thing. I believe we all want SyncRep to be successful. Sincerely, JD -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 509.416.6579 Consulting, Training, Support, Custom Development, Engineering http://twitter.com/cmdpromptinc | http://identi.ca/commandprompt -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] More extension issues: ownership and search_path
Dimitri Fontaine dimi...@2ndquadrant.fr writes: That said, we should do something about ALTER EXTENSION SET SCHEMA and the relocatable property. I'm thinking that an extension stops being relocatable as soon as any of its reverse dependencies (all the tree) is not relocatable. If you're worried about that, then it's questionable whether ALTER EXTENSION SET SCHEMA makes sense at all, ever. I don't see any reason to think that an extension is more fragile for this purpose than any other random SQL dependencies. Also, an extension being relocatable doesn't seem to me to guarantee that it can cope with its dependencies moving around; they're really independent properties. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sync Rep for 2011CF1
On Mon, Feb 7, 2011 at 2:09 PM, Dimitri Fontaine dimi...@2ndquadrant.fr wrote: Robert Haas robertmh...@gmail.com writes: done in the time available is another thing entirely. I do NOT want to still be working on the items for this CommitFest in June - that's about when I'd like to be releasing beta3. Except that's not how we work here. You want to change that with respect to the release management process and schedule (or lack thereof). Tradition and current practice say you need to reach consensus to be able to bypass compromising. Good luck with that. I'm not trying to bypass compromising, and I don't know what makes you think otherwise. I am trying to ensure that the CommitFest wraps up in a timely fashion, which is something we have done consistently for every CommitFest in the 9.0 and 9.1 cycles to date, including the last CommitFest of the 9.0 cycle. It is not somehow a deviation from past community practice to boot patches that can't be finished up in the time available during the CommitFest. That has been routine practice for a long time. I have worked very hard on this CommitFest, both to line up patch reviewers and to review myself. I want to make sure that every patch gets a good, thorough review before the CommitFest is over. I think there is general consensus that this is important and that we will lose contributors if we don't do it. However, I don't want to prolong the CommitFest indefinitely in the face of patches that the authors are not actively working on or can't finish in the time available, or where there is no consensus that the proposed change is what we want. I believe that this, too, is a generally accepted principle in our community, not something I just made up. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sync Rep for 2011CF1
I just spoke to my manager at EnterpriseDB and he cleared my schedule for the next two days to work on this. So I'll go hack on this now. I haven't read the patch yet so I don't know for sure how quite I'll be able to get up to speed on it, so if someone who is more familiar with this code wants to grab the baton away from me, feel free. Otherwise, I'll see what I can do with it. Presumably you have a reason for declaring war? I'm sorry for that, I really am. How is clearing out his whole schedule to help review fix the patch declaring war? You have an odd attitude towards assistance, Simon. -- -- Josh Berkus PostgreSQL Experts Inc. http://www.pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] More extension issues: ownership and search_path
On Feb 7, 2011, at 10:23 AM, Tom Lane wrote:
On reflection, the set of extensions that an extension depends on is
obviously a property of the extension, which means it ought to be
specified in the extension's control file, not in the CREATE EXTENSION
command. So now I'm thinking something like
requires = 'foo, bar, baz'
in the file.
And that takes us one step closer to PGXN's META.json file. Here's the spec:
http://pgxn.org/meta/spec.txt
It includes a prereqs section, which looks like this:
prereqs: {
runtime: {
requires: {
citext: 0,
plpgsql: 0,
PostgreSQL: 8.0.0
},
recommends: {
PostgreSQL: 8.4.0
}
}
},
Best,
David
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sync Rep for 2011CF1
On Mon, Feb 7, 2011 at 2:56 PM, Dave Page dp...@pgadmin.org wrote: On Mon, Feb 7, 2011 at 6:55 PM, Robert Haas robertmh...@gmail.com wrote: On Mon, Feb 7, 2011 at 12:43 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: ... Well, the current CommitFest ends in one week, ... Really? I thought the idea for the last CF of a development cycle was that it kept going till we'd dealt with everything. Arbitrarily rejecting stuff we haven't dealt with doesn't seem fair. Uh, we did that with 8.4 and it was a disaster. The CommitFest lasted *five months*. We've been doing schedule-based CommitFests ever since and it's worked much better. Rejecting stuff because we haven't gotten round to dealing with it in such a short period of time is a damn good way to limit the number of contributions we get. I don't believe we've agreed at any point that the last commitfest should be the same time length as the others News to me. http://wiki.postgresql.org/wiki/PostgreSQL_9.1_Development_Plan -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sync Rep for 2011CF1
On Mon, Feb 7, 2011 at 6:55 PM, Robert Haas robertmh...@gmail.com wrote: On Mon, Feb 7, 2011 at 12:43 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: ... Well, the current CommitFest ends in one week, ... Really? I thought the idea for the last CF of a development cycle was that it kept going till we'd dealt with everything. Arbitrarily rejecting stuff we haven't dealt with doesn't seem fair. Uh, we did that with 8.4 and it was a disaster. The CommitFest lasted *five months*. We've been doing schedule-based CommitFests ever since and it's worked much better. Rejecting stuff because we haven't gotten round to dealing with it in such a short period of time is a damn good way to limit the number of contributions we get. I don't believe we've agreed at any point that the last commitfest should be the same time length as the others (when we originally came up with the commitfest idea, it certainly wasn't expected), and deciding that without giving people advanced notice is a really good way to piss them off and encourage them to go work on other things. If we're going to put a time limit on this - and I think we should - we should publish a date ASAP, that gives everyone a fair chance to finish their work - say, 4 weeks. Then, if we want to make the last commitfest the same length as the others next year, we can make that decision and document those plans. -- Dave Page Blog: http://pgsnake.blogspot.com Twitter: @pgsnake EnterpriseDB UK: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SQL/MED - file_fdw
On 02/07/2011 01:39 AM, Itagaki Takahiro wrote:
file_fdw uses CopyFromErrorCallback() to give errors the proper context. The
function uses template strings like COPY %s, line %d, where %s is the name of
the relation being copied. Presumably file_fdw and other features using this
API would wish to customize that error message prefix, and the relation name
might not be apropos at all. How about another argument to BeginCopyFrom,
specifying an error prefix to be stashed in the CopyState?
I changed COPY %s, .. to relation %s, ... because the first string is
the relation name anyway. We could have another prefix argument, but I think
it has little information for errors.
We also have many COPY in other messages, but they won't be used actually
because the are messages for invalid arguments and file_fdw will have own
validater function. All invalid arguments will be filtered in CREATE commands.
These changes have broken the regression tests. The attached patches
(one for the core regression tests and one for file_fdw) fix that.
But I don't know that your change is terribly helpful. I rather like
Noah's idea better, if we need to make a change.
cheers
andrew
diff --git a/contrib/file_fdw/output/file_fdw.source b/contrib/file_fdw/output/file_fdw.source
index f8ce4ca..7021ad8 100644
--- a/contrib/file_fdw/output/file_fdw.source
+++ b/contrib/file_fdw/output/file_fdw.source
@@ -89,7 +89,7 @@ SELECT * FROM agg_csv c JOIN agg_text t ON (t.a = c.a) ORDER BY c.a;
-- error context report tests
SELECT * FROM agg_bad; -- ERROR
ERROR: invalid input syntax for type real: aaa
-CONTEXT: COPY agg_bad, line 3, column b: aaa
+CONTEXT: relation agg_bad, line 3, column b: aaa
-- misc query tests
\t on
EXPLAIN (VERBOSE, COSTS FALSE) SELECT * FROM agg_csv;
diff --git a/src/test/regress/expected/alter_table.out b/src/test/regress/expected/alter_table.out
index 3280065..eea0b09 100644
--- a/src/test/regress/expected/alter_table.out
+++ b/src/test/regress/expected/alter_table.out
@@ -1002,7 +1002,7 @@ copy test(pg.dropped.1) to stdout;
ERROR: column pg.dropped.1 of relation test does not exist
copy test from stdin;
ERROR: extra data after last expected column
-CONTEXT: COPY test, line 1: 10 11 12
+CONTEXT: relation test, line 1: 10 11 12
select * from test;
b | c
---+---
diff --git a/src/test/regress/expected/copy2.out b/src/test/regress/expected/copy2.out
index 15cbe02..d9630f8 100644
--- a/src/test/regress/expected/copy2.out
+++ b/src/test/regress/expected/copy2.out
@@ -35,17 +35,17 @@ ERROR: column d specified more than once
-- missing data: should fail
COPY x from stdin;
ERROR: invalid input syntax for integer:
-CONTEXT: COPY x, line 1, column a:
+CONTEXT: relation x, line 1, column a:
COPY x from stdin;
ERROR: missing data for column e
-CONTEXT: COPY x, line 1: 2000 230 23 23
+CONTEXT: relation x, line 1: 2000 230 23 23
COPY x from stdin;
ERROR: missing data for column e
-CONTEXT: COPY x, line 1: 2001 231 \N \N
+CONTEXT: relation x, line 1: 2001 231 \N \N
-- extra data: should fail
COPY x from stdin;
ERROR: extra data after last expected column
-CONTEXT: COPY x, line 1: 2002 232 40 50 60 70 80
+CONTEXT: relation x, line 1: 2002 232 40 50 60 70 80
-- various COPY options: delimiters, oids, NULL string
COPY x (b, c, d, e) from stdin with oids delimiter ',' null 'x';
COPY x from stdin WITH DELIMITER AS ';' NULL AS '';
diff --git a/src/test/regress/expected/domain.out b/src/test/regress/expected/domain.out
index 7d72791..daadcef 100644
--- a/src/test/regress/expected/domain.out
+++ b/src/test/regress/expected/domain.out
@@ -49,7 +49,7 @@ INSERT INTO basictest values ('88', 'haha', 'short', '123.1212');-- Truncate
-- Test copy
COPY basictest (testvarchar) FROM stdin; -- fail
ERROR: value too long for type character varying(5)
-CONTEXT: COPY basictest, line 1, column testvarchar: notsoshorttext
+CONTEXT: relation basictest, line 1, column testvarchar: notsoshorttext
COPY basictest (testvarchar) FROM stdin;
select * from basictest;
testint4 | testtext | testvarchar | testnumeric
@@ -130,7 +130,7 @@ select testint4arr[1], testchar4arr[2:2] from domarrtest;
COPY domarrtest FROM stdin;
COPY domarrtest FROM stdin; -- fail
ERROR: value too long for type character varying(4)
-CONTEXT: COPY domarrtest, line 1, column testchar4arr: {qwerty,w,e}
+CONTEXT: relation domarrtest, line 1, column testchar4arr: {qwerty,w,e}
select * from domarrtest;
testint4arr |testchar4arr
---+-
@@ -173,14 +173,14 @@ INSERT INTO nulltest values ('a', 'b', 'c', NULL, 'd'); -- Good
-- Test copy
COPY nulltest FROM stdin; --fail
ERROR: null value in column col3 violates not-null constraint
-CONTEXT: COPY nulltest, line 1: a b \N d d
+CONTEXT: relation nulltest, line 1: a b \N d d
COPY nulltest FROM stdin; --fail
ERROR: domain dcheck does not allow null values
-CONTEXT: COPY
Re: [HACKERS] More extension issues: ownership and search_path
Tom Lane t...@sss.pgh.pa.us writes: If you're worried about that, then it's questionable whether ALTER EXTENSION SET SCHEMA makes sense at all, ever. I don't see any reason to think that an extension is more fragile for this purpose than any other random SQL dependencies. Also, an extension being relocatable doesn't seem to me to guarantee that it can cope with its dependencies moving around; they're really independent properties. Well a relocatable extension certainly supports SET SCHEMA just fine, or it would not have the property. Then your conclusion is right. My comment was about what happens when things are setup the other way. We have earthdistance that depends on cube. Let's pretend that earthdistance is not relocatable. I think we should then consider (when both are installed) that cube is not relocatable, whatever its control file says. That's because not relocatable means that the install script is using the @extschema@ place holder and the fragility there is known quite high: the install script and some installed objects do depend on @extschema@. Moving the dependencies underneath it in this case looks to me more than a risk: we know we're breaking things. What you're saying (or what I'm reading at least) is that if earthdistance is relocatable, you don't have faith that it means we can actually move cube without collateral damages. Well, the author said it would cope fine, and in this case I see no reason not to believe him. Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sync Rep for 2011CF1
Robert Haas robertmh...@gmail.com wrote: Dave Page dp...@pgadmin.org wrote: Robert Haas robertmh...@gmail.com wrote: Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: ... Well, the current CommitFest ends in one week, ... Really? I thought the idea for the last CF of a development cycle was that it kept going till we'd dealt with everything. Arbitrarily rejecting stuff we haven't dealt with doesn't seem fair. Uh, we did that with 8.4 and it was a disaster. The CommitFest lasted *five months*. We've been doing schedule-based CommitFests ever since and it's worked much better. Rejecting stuff because we haven't gotten round to dealing with it in such a short period of time is a damn good way to limit the number of contributions we get. I don't believe we've agreed at any point that the last commitfest should be the same time length as the others News to me. http://wiki.postgresql.org/wiki/PostgreSQL_9.1_Development_Plan I believe that with tighter management of the process, it should be possible to reduce the average delay between someone writing a feature and that feature appearing in a production release by about two months without compromising quality. Getting hypothetical for a moment, delaying release of 50 features for two months to allow release of one feature ten months earlier is likely to frustrate a lot more people than having the train leave the station on time and putting that one feature into the next release. My impression was that Robert is trying to find a way to help get Simon's patch into this release without holding everything up for it. In my book, that's not a declaration of war; it's community spirit. -Kevin -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sync Rep for 2011CF1
On Mon, Feb 7, 2011 at 8:59 PM, Robert Haas robertmh...@gmail.com wrote: On Mon, Feb 7, 2011 at 2:56 PM, Dave Page dp...@pgadmin.org wrote: On Mon, Feb 7, 2011 at 6:55 PM, Robert Haas robertmh...@gmail.com wrote: On Mon, Feb 7, 2011 at 12:43 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: ... Well, the current CommitFest ends in one week, ... Really? I thought the idea for the last CF of a development cycle was that it kept going till we'd dealt with everything. Arbitrarily rejecting stuff we haven't dealt with doesn't seem fair. Uh, we did that with 8.4 and it was a disaster. The CommitFest lasted *five months*. We've been doing schedule-based CommitFests ever since and it's worked much better. Rejecting stuff because we haven't gotten round to dealing with it in such a short period of time is a damn good way to limit the number of contributions we get. I don't believe we've agreed at any point that the last commitfest should be the same time length as the others News to me. http://wiki.postgresql.org/wiki/PostgreSQL_9.1_Development_Plan Yes, and? It doesn't say beta 1 at the after a month of the last commitfest, which is the milestone which marks the end of development. It says alpha 4, and possibly more alphas. It's pretty clear that it is expected that development and polishing will continue past the 20th February. -- Dave Page Blog: http://pgsnake.blogspot.com Twitter: @pgsnake EnterpriseDB UK: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sync Rep for 2011CF1
Robert Haas robertmh...@gmail.com writes: I'm not trying to bypass compromising, and I don't know what makes you think otherwise. I am trying to ensure that the CommitFest wraps up Well, I'm too tired to allow myself posting such comments, sorry to have left the previous mail through. More than one commit fest saw its time frame extended for 1 or 2 weeks already, I think, all I'm saying is that this one will certainly not be an exception, and that's for the best. Be sure I appreciate the efforts you're putting into the mix! Regards, -- Dimitri Fontaine http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sync Rep for 2011CF1
On 2/7/11 11:41 AM, Robert Haas wrote: However, I don't want to prolong the CommitFest indefinitely in the face of patches that the authors are not actively working on or can't finish in the time available, or where there is no consensus that the proposed change is what we want. I believe that this, too, is a generally accepted principle in our community, not something I just made up. +1. I, for one, would vote against extending beta if Sync Rep isn't ready yet. There's plenty of other big features in 9.1, and Sync Rep will benefit from having additional development time given the number of major spec points we only cleared up a few weeks ago. I think the majority of our users would prefer a 9.1 in May to one that has Sync Rep and is delivered in September. If they had a choice. Speaking of which, time to do some reviewing ... -- -- Josh Berkus PostgreSQL Experts Inc. http://www.pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sync Rep for 2011CF1
On Mon, Feb 7, 2011 at 3:06 PM, Dave Page dp...@pgadmin.org wrote: Rejecting stuff because we haven't gotten round to dealing with it in such a short period of time is a damn good way to limit the number of contributions we get. I don't believe we've agreed at any point that the last commitfest should be the same time length as the others News to me. http://wiki.postgresql.org/wiki/PostgreSQL_9.1_Development_Plan Yes, and? It doesn't say beta 1 at the after a month of the last commitfest, which is the milestone which marks the end of development. It says alpha 4, and possibly more alphas. It's pretty clear that it is expected that development and polishing will continue past the 20th February. You're moving the bar. It DOES say that the CommitFest will end on February 15th. Now, if we want to have a discussion about changing that, let's have that discussion (perhaps on a thread where the subject has something to do with the topic), but we DID talk about this, it WAS agreed, and it's been sitting there on the wiki for something like 8 months. Obviously, there will continue to be polishing after the CommitFest is over, but that's not the same thing as saying we're going to lengthen the CommitFest itself. I think we need to step back a few paces here and talk about what we're trying to accomplish by making some change to the proposed and agreed CommitFest schedule. If there's a concern that some patches haven't been thoroughly reviewed at this point, then I think that's a fair concern, and let's talk about which ones they are and see what we can do about it. I don't believe that's the case, and it's certainly not the case for sync rep, which was submitted in an unpolished state by Simon's own admission, reviewed and discussed, then sat for three weeks without an update. So perhaps the concern is that sync rep is a make or break for this release. OK, then fine, let's talk about whether it's worth slipping the release for that feature. I have no problem with either of those conversations, and I'm happy to offer my opinions and listen to the opinions of others, and we can make some decision. I think, though, that we need to be explicit about what we're doing, and why we're doing it. I have been working hard on this CommitFest for a long time (since approximately a month before it started) at the cost of development projects I would have liked to have worked on, because I knew we were going to be overwhelmed with patches. I have helped as many people as I can with as many patches as I have been able to. I think that finishing on time (or at least as close to on time as we can manage) is important to our success as a development community, just as having good features is. We don't have to agree on what the best thing to do is, but I would certainly appreciate it if everyone could at least credit me with acting in good faith. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sync Rep for 2011CF1
On Mon, 2011-02-07 at 12:24 -0800, Josh Berkus wrote: +1. I, for one, would vote against extending beta if Sync Rep isn't ready yet. There's plenty of other big features in 9.1, and Sync Rep will benefit from having additional development time given the number of major spec points we only cleared up a few weeks ago. I think the majority of our users would prefer a 9.1 in May to one that has Sync Rep and is delivered in September. If they had a choice. +1 JD -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 509.416.6579 Consulting, Training, Support, Custom Development, Engineering http://twitter.com/cmdpromptinc | http://identi.ca/commandprompt -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sync Rep for 2011CF1
On Mon, Feb 7, 2011 at 9:25 PM, Robert Haas robertmh...@gmail.com wrote: You're moving the bar. It DOES say that the CommitFest will end on February 15th. Now, if we want to have a discussion about changing that, let's have that discussion (perhaps on a thread where the subject has something to do with the topic), but we DID talk about this, it WAS agreed, and it's been sitting there on the wiki for something like 8 months. Obviously, there will continue to be polishing after the CommitFest is over, but that's not the same thing as saying we're going to lengthen the CommitFest itself. I'm not moving the bar - I'm talking practically. Regardless of when we consider the commitfest itself over, development and commit work of new features has always continued until beta 1, and that has not changed as far as I'm aware. I think, though, that we need to be explicit about what we're doing, and why we're doing it. I have been working hard on this CommitFest for a long time (since approximately a month before it started) at the cost of development projects I would have liked to have worked on, because I knew we were going to be overwhelmed with patches. I have helped as many people as I can with as many patches as I have been able to. I think that finishing on time (or at least as close to on time as we can manage) is important to our success as a development community, just as having good features is. We don't have to agree on what the best thing to do is, but I would certainly appreciate it if everyone could at least credit me with acting in good faith. Oh, I have absolutely no doubt you're working in good faith, and personally I thank you for the hard work you've put in. I just disagree with your interpretation of the timetable. -- Dave Page Blog: http://pgsnake.blogspot.com Twitter: @pgsnake EnterpriseDB UK: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sync Rep for 2011CF1
On Mon, Feb 7, 2011 at 3:14 PM, Dimitri Fontaine dimi...@2ndquadrant.fr wrote: Robert Haas robertmh...@gmail.com writes: I'm not trying to bypass compromising, and I don't know what makes you think otherwise. I am trying to ensure that the CommitFest wraps up Well, I'm too tired to allow myself posting such comments, sorry to have left the previous mail through. Thanks, I understand. More than one commit fest saw its time frame extended for 1 or 2 weeks already, I think, all I'm saying is that this one will certainly not be an exception, and that's for the best. We've actually done really well. The last CommitFest in 9.0 wrapped up on 2/17 (two days late), and the others were mostly right on time as well. The CommitFests for 9.1 ended on: 8/15 (on time), 10/26 (9 days late, but there was no activity on the last two of those days, so say 7 days late), and 12/21 (six days late). As far as I can tell, the difference primarily has to do with who manages the CommitFests and how aggressively they follow up on patches that are dropped. The last CommitFest we have that really ran late was the final CommitFest of the 8.4 cycle, and it was that event that led me to accept Josh Berkus's invitation to be a CF manager for the first 9.0 CommitFest. Because five month CommitFests with the tree frozen are awful and sucky for everyone except the people who are getting extra time to finish their patches, and they aren't really that great for those people either. As far as I am concerned, everything from now until we've released a stable beta with no known issues is time that I can't spend doing development. So I'd like to minimize that time - not by arbitrarily throwing patches out the window - but by a combination of postponing patches that are not done and working my ass off to finish as much as possible. Be sure I appreciate the efforts you're putting into the mix! Thanks. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] PgEast 2011: 3 days until close of CFP
Hello hackers, Just FYI, the CFP for PgEast in NYC closes in three days. https://www.postgresqlconference.org/talk_types Sincerely, JD -- PostgreSQL.org Major Contributor Command Prompt, Inc: http://www.commandprompt.com/ - 509.416.6579 Consulting, Training, Support, Custom Development, Engineering http://twitter.com/cmdpromptinc | http://identi.ca/commandprompt -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sync Rep for 2011CF1
On Mon, Feb 7, 2011 at 3:34 PM, Dave Page dp...@pgadmin.org wrote: On Mon, Feb 7, 2011 at 9:25 PM, Robert Haas robertmh...@gmail.com wrote: You're moving the bar. It DOES say that the CommitFest will end on February 15th. Now, if we want to have a discussion about changing that, let's have that discussion (perhaps on a thread where the subject has something to do with the topic), but we DID talk about this, it WAS agreed, and it's been sitting there on the wiki for something like 8 months. Obviously, there will continue to be polishing after the CommitFest is over, but that's not the same thing as saying we're going to lengthen the CommitFest itself. I'm not moving the bar - I'm talking practically. Regardless of when we consider the commitfest itself over, development and commit work of new features has always continued until beta 1, and that has not changed as far as I'm aware. I don't think that's really true. Go back and read the output of 'git log REL9_0_BETA1'. It's bug fixes, rearrangements of things that were committed but turned out to be controversial, documentation work, release note editing, pgindent crap... sure, it wasn't a totally hard freeze, but it was pretty solid slush. We did a good job not letting things drag out, and FWIW I think that was a good decision. I don't remember too many people being unhappy about their patches getting punted, either. There were one or two, but generally we punted things that needed major rework or just weren't getting updated in a timely fashion, and that, combined with a lot of hard work on Tom's part among others, worked fine. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sync Rep for 2011CF1
On Mon, Feb 7, 2011 at 9:46 PM, Robert Haas robertmh...@gmail.com wrote: On Mon, Feb 7, 2011 at 3:34 PM, Dave Page dp...@pgadmin.org wrote: On Mon, Feb 7, 2011 at 9:25 PM, Robert Haas robertmh...@gmail.com wrote: You're moving the bar. It DOES say that the CommitFest will end on February 15th. Now, if we want to have a discussion about changing that, let's have that discussion (perhaps on a thread where the subject has something to do with the topic), but we DID talk about this, it WAS agreed, and it's been sitting there on the wiki for something like 8 months. Obviously, there will continue to be polishing after the CommitFest is over, but that's not the same thing as saying we're going to lengthen the CommitFest itself. I'm not moving the bar - I'm talking practically. Regardless of when we consider the commitfest itself over, development and commit work of new features has always continued until beta 1, and that has not changed as far as I'm aware. I don't think that's really true. Go back and read the output of 'git log REL9_0_BETA1'. It's bug fixes, rearrangements of things that were committed but turned out to be controversial, documentation work, release note editing, pgindent crap... sure, it wasn't a totally hard freeze, but it was pretty solid slush. We did a good job not letting things drag out, and FWIW I think that was a good decision. I don't remember too many people being unhappy about their patches getting punted, either. There were one or two, but generally we punted things that needed major rework or just weren't getting updated in a timely fashion, and that, combined with a lot of hard work on Tom's part among others, worked fine. I guess we disagree on what we consider to be development then. Just looking back to April, I see various committers whacking things around that look to me like the fine tuning and completion of earlier patches. Oh - and just so we're clear... I too want us to get the release out promptly, I'm just concerned that we don't blindside developers. -- Dave Page Blog: http://pgsnake.blogspot.com Twitter: @pgsnake EnterpriseDB UK: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Error code for terminating connection due to conflict with recovery
On Wed, 2011-02-02 at 21:21 -0500, Robert Haas wrote:
On Tue, Feb 1, 2011 at 3:17 AM, Simon Riggs si...@2ndquadrant.com wrote:
ERRCODE_DATABASE_DROPPED57P04 looks best
So I guess the only remaining issue is whether we should distinguish
the error message text, as well as the error codes. Tom was in favor
of that upthread, and I am too. Right now we have:
else if (RecoveryConflictPending RecoveryConflictRetryable)
{
pgstat_report_recovery_conflict(RecoveryConflictReason);
ereport(FATAL,
(errcode(ERRCODE_T_R_SERIALIZATION_FAILURE),
errmsg(terminating connection due to
conflict with recovery),
errdetail_recovery_conflict()));
}
else if (RecoveryConflictPending)
{
/* Currently there is only one non-retryable
recovery conflict */
Assert(RecoveryConflictReason ==
PROCSIG_RECOVERY_CONFLICT_DATABASE);
pgstat_report_recovery_conflict(RecoveryConflictReason);
ereport(FATAL,
(errcode(ERRCODE_DATABASE_DROPPED),
errmsg(terminating connection due to
conflict with recovery),
errdetail_recovery_conflict()));
}
The simplest thing to do seems to be to make the second one read
terminating connection because the database has been dropped.
The error text is already differentiated by
errdetail_recovery_conflict().
--
Simon Riggs http://www.2ndQuadrant.com/books/
PostgreSQL Development, 24x7 Support, Training and Services
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] More extension issues: ownership and search_path
Dimitri Fontaine dimi...@2ndquadrant.fr writes: Tom Lane t...@sss.pgh.pa.us writes: If you're worried about that, then it's questionable whether ALTER EXTENSION SET SCHEMA makes sense at all, ever. I don't see any reason to think that an extension is more fragile for this purpose than any other random SQL dependencies. Also, an extension being relocatable doesn't seem to me to guarantee that it can cope with its dependencies moving around; they're really independent properties. Well a relocatable extension certainly supports SET SCHEMA just fine, or it would not have the property. Then your conclusion is right. My comment was about what happens when things are setup the other way. Yes, I understood that. We have earthdistance that depends on cube. Let's pretend that earthdistance is not relocatable. I think we should then consider (when both are installed) that cube is not relocatable, whatever its control file says. That's because not relocatable means that the install script is using the @extschema@ place holder and the fragility there is known quite high: the install script and some installed objects do depend on @extschema@. But @extschema@ isn't affected by where the other modules are. The basic issue here is that we might have wired something about the referenced schemas into one of the objects in the dependent extension, for example via create function foo() ... SET search_path FROM CURRENT; I agree that this is a risk. My point is that you can do that without any extension, and you're just as much at risk. If you think that this is something we must protect against, I think ripping out ALTER EXTENSION SET SCHEMA altogether is the only real answer. I see no argument whatsoever why we should lock down extensions and only extensions against this risk. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sync Rep for 2011CF1
dp...@pgadmin.org (Dave Page) writes: On Mon, Feb 7, 2011 at 6:55 PM, Robert Haas robertmh...@gmail.com wrote: On Mon, Feb 7, 2011 at 12:43 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: ... Well, the current CommitFest ends in one week, ... Really? I thought the idea for the last CF of a development cycle was that it kept going till we'd dealt with everything. Arbitrarily rejecting stuff we haven't dealt with doesn't seem fair. Uh, we did that with 8.4 and it was a disaster. The CommitFest lasted *five months*. We've been doing schedule-based CommitFests ever since and it's worked much better. Rejecting stuff because we haven't gotten round to dealing with it in such a short period of time is a damn good way to limit the number of contributions we get. I don't believe we've agreed at any point that the last commitfest should be the same time length as the others (when we originally came up with the commitfest idea, it certainly wasn't expected), and deciding that without giving people advanced notice is a really good way to piss them off and encourage them to go work on other things. If we're going to put a time limit on this - and I think we should - we should publish a date ASAP, that gives everyone a fair chance to finish their work - say, 4 weeks. Then, if we want to make the last commitfest the same length as the others next year, we can make that decision and document those plans. There *is* a problem that there doesn't seem to be enough time to readily allow development of larger features without people getting stuck fighting with the release periods. But that's not the problem taking place here. It was documented, last May, that the final CommitFest for 9.1 was to complete 2011-02-15, and there did seem to be agreement on that. It sure looks to me like there are going to be a bunch of items that, based on the recognized policies, need to get deferred to 9.2, and the prospects for Sync Rep getting into 9.1 don't look notably good to me. Looking at things statistically, the 9.1 commitfests have had the following numbers of items: #1 - 2010-09 - 52, of which 26 were committed #2 - 2010-11 - 43, of which 23 were committed #3 - 2011-01 - 98, of which 35 have been committed, and 10 are considered ready to commit. It may appear unfair to not offer everyone a fair chance to finish their work, but it's not as if the date wasn't published Plenty Long Ago. and well-publicized. But deferring the end of the CommitFest would be Not Fair to those that *did* get their proposed changes ready for the preceding Fests. We cannot evade unfairness. It's definitely readily arguable that fairness requires that: - Items not committable by 2011-02-15 be deferred to the 2011-Next fest There are around 25 items right now that are sitting with [Waiting for Author] and [Returned with Feedback] statuses. They largely seem like pretty fair game for next fest. - Large items that weren't included in the 2010-11 fest be considered problematic to try to integrate into 9.1 There sure seem to be some large items in the 2011-01 fest, which I thought wasn't supposed to be the case. We shouldn't just impose policy for the sake of imposing policy, but I do recall Really Long CommitFests being pretty disastrous. And there's *SO* much outstanding in this particular fest that it's getting past time for doing some substantial triage so that reviewer attentions may be directed towards the items most likely to be acceptable for 9.1. I hate to think that 9.1 won't include Simon's SR material, but that may have to be. -- http://www3.sympatico.ca/cbbrowne/slony.html It's a pretty rare beginner who isn't clueless. If beginners weren't clueless, the infamous Unix learning cliff wouldn't be a problem. -- david parsons -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Varlena and binary
Just from curious may I ask in which direction this will go, and how this will affect performance of text and binary format? Actually I started to make smaller improvements, and I think about one big to encode text (when client and server encoding are different) directly to StringInfo, without intermediate buffer. Thanks in advice Radek Tom Lane t...@sss.pgh.pa.us Monday 07 February 2011 17:12:07 =?utf-8?q?Rados=C5=82aw_Smogura?= m...@smogura.eu writes: I'm sending small patch for textsend. It reduces unnecessary copies, and memory usage for duplication of varlena data. May you look? This code will break the day that text and bytea don't have the same internal representation, which seems likely to be soon. Barring some compelling evidence of a major performance improvement obtainable this way, I don't think we want this patch. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sync Rep for 2011CF1
On Mon, 2011-02-07 at 15:25 -0500, Robert Haas wrote: I would certainly appreciate it if everyone could at least credit me with acting in good faith. I think you are, if that helps. -- Simon Riggs http://www.2ndQuadrant.com/books/ PostgreSQL Development, 24x7 Support, Training and Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Spread checkpoint sync
Kevin Grittner wrote: There are occasional posts from those wondering why their read-only queries are so slow after a bulk load, and why they are doing heavy writes. (I remember when I posted about that, as a relative newbie, and I know I've seen others.) Sure; I created http://wiki.postgresql.org/wiki/Hint_Bits a while back specifically to have a resource to explain that mystery to offer people. But there's a difference between having a performance issue that people don't understand, and having a real bottleneck you can't get rid of. My experience is that people who have hint bit issues run into them as a minor side-effect of a larger vacuum issue, and that if you get that under control they're only a minor detail in comparison. Makes it hard to get too excited about optimizing them. -- Greg Smith 2ndQuadrant USg...@2ndquadrant.com Baltimore, MD PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.us PostgreSQL 9.0 High Performance: http://www.2ndQuadrant.com/books -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SSI patch version 14
On 06.02.2011 20:30, Kevin Grittner wrote: Kevin Grittner wrote: I'm working on it now, and hope to have it all settled down today. Done and pushed to the git repo. Branch serializable on: git://git.postgresql.org/git/users/kgrittn/postgres.git Heikki: I'm back to not having any outstanding work on the patch. Ok, committed. Thank you for all this! It has been a huge effort from you and Dan, and a great feature as a result. -- Heikki Linnakangas EnterpriseDB http://www.enterprisedb.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Sync Rep for 2011CF1
On Mon, 2011-02-07 at 11:50 -0800, Josh Berkus wrote: I just spoke to my manager at EnterpriseDB and he cleared my schedule for the next two days to work on this. So I'll go hack on this now. I haven't read the patch yet so I don't know for sure how quite I'll be able to get up to speed on it, so if someone who is more familiar with this code wants to grab the baton away from me, feel free. Otherwise, I'll see what I can do with it. Presumably you have a reason for declaring war? I'm sorry for that, I really am. How is clearing out his whole schedule to help review fix the patch declaring war? You have an odd attitude towards assistance, Simon. It seems likely that Robert had not read my reply where I said I had time to work on this project before posting. In that case, I withdraw my comments and apologise. -- Simon Riggs http://www.2ndQuadrant.com/books/ PostgreSQL Development, 24x7 Support, Training and Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SSI patch version 14
On Mon, Feb 7, 2011 at 23:14, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: On 06.02.2011 20:30, Kevin Grittner wrote: Kevin Grittner wrote: I'm working on it now, and hope to have it all settled down today. Done and pushed to the git repo. Branch serializable on: git://git.postgresql.org/git/users/kgrittn/postgres.git Heikki: I'm back to not having any outstanding work on the patch. Ok, committed. Thank you for all this! It has been a huge effort from you and Dan, and a great feature as a result. +1! (or more) -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/ -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
