Re: [HACKERS] Query Plan Columns

2010-11-09 Thread Dimitri Fontaine
David E. Wheeler da...@kineticode.com writes: I'm wondering if there's any reason why we couldn't have EXPLAIN do something like this itself in core: EXPLAIN (format table) SELECT * FROM bar; +1 from me here, as it happens parsing a table-like resultset is already implemented everywhere 

Re: [HACKERS] Query Plan Columns

2010-11-09 Thread David E. Wheeler
On Nov 9, 2010, at 12:12 AM, Dimitri Fontaine wrote: WITH plan AS ( EXPLAIN (format table) SELECT * FROM bar ) INSERT INTO plan_audit SELECT * FROM plan WHERE actual_total_time 12 * interval '100 ms'; Yeah, that would be nice, but my current implementation has a row for each node, and a

Re: [HACKERS] CLUSTER can change t_len

2010-11-09 Thread Itagaki Takahiro
On Tue, Nov 9, 2010 at 12:44 PM, Jeff Davis pg...@j-davis.com wrote: See case below. After the item length gets changed, then when reading the tuple later you get a t_len that includes padding. We can easily find it with pageinspect: \i pageinspect.sql create table foo(i int4); insert into foo

Re: [HACKERS] proposal: plpgsql - iteration over fields of rec or row variable

2010-11-09 Thread Dmitriy Igrishin
Hey Pavel, 2010/11/9 Pavel Stehule pavel.steh...@gmail.com a) transformation to common type + simple - it is one day job - function record_to_array, array_to_record, and fieldnames_to_array - lost of type info, hidden problems with IO cast - int a := 10.0/2.0 is a problem using

Re: [HACKERS] CLUSTER can change t_len

2010-11-09 Thread Heikki Linnakangas
On 09.11.2010 11:11, Itagaki Takahiro wrote: On Tue, Nov 9, 2010 at 12:44 PM, Jeff Davispg...@j-davis.com wrote: See case below. After the item length gets changed, then when reading the tuple later you get a t_len that includes padding. We can easily find it with pageinspect: \i

[HACKERS] security hooks on object creation

2010-11-09 Thread KaiGai Kohei
The attached patch provides plugin modules a hook just after object creation time. In typical use cases, it enables to assign default security labels on object creation by the external security providers. As Robert suggested before, it provides a generic purpose main hook. It takes an enum of

Re: [HACKERS] W3C Specs: Web SQL

2010-11-09 Thread Sam Mason
On Mon, Nov 08, 2010 at 12:55:22PM -0300, Alvaro Herrera wrote: Excerpts from Charles Pritchard's message of sáb nov 06 23:20:13 -0300 2010: Simple async sql sub-set (the spec in trouble): http://dev.w3.org/html5/webdatabase/ This is insane. This spec allows the server to run arbitrary

Re: [HACKERS] Avoid memory leaks during ANALYZE's compute_index_stats() ?

2010-11-09 Thread Jakub Ouhrabka
Hi Tom, thanks for brilliant analysis - now we know how to avoid the problem. As a side note: from the user's point of view it would be really nice to know that the error was caused by auto-ANALYZE - at least on 8.2 it's not that obvious from the server log. It was the first message with

Re: [HACKERS] W3C Specs: Web SQL

2010-11-09 Thread Andrew Dunstan
On 11/09/2010 09:59 AM, Alvaro Herrera wrote: And this makes me think that SQLite is indeed the right tool for the job here, and not PostgreSQL. If someone intrudes, it's going to be in the same process running the web browser, not in some server running under another user identity in the

Re: [HACKERS] improved parallel make support

2010-11-09 Thread Dave Page
On Sat, Nov 6, 2010 at 4:35 AM, Peter Eisentraut pete...@gmx.net wrote: On ons, 2010-11-03 at 16:34 +0200, Peter Eisentraut wrote: On tis, 2010-11-02 at 10:21 -0400, Tom Lane wrote: Do we have a handle on how many buildfarm members this will break? I suppose we don't.  One way to find out

Re: [HACKERS] CLUSTER can change t_len

2010-11-09 Thread Tom Lane
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: On 09.11.2010 11:11, Itagaki Takahiro wrote: We have a comment /* be conservative */ in the function, but I'm not sure we actually need the MAXALIGN. However, there would be almost no benefits to keep t_len in small value because

Re: [HACKERS] Hot Standby b-tree delete records review

2010-11-09 Thread Simon Riggs
On Tue, 2010-11-09 at 13:34 +0200, Heikki Linnakangas wrote: (cleaning up my inbox, and bumped into this..) On 22.04.2010 12:31, Simon Riggs wrote: On Thu, 2010-04-22 at 12:18 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: On Thu, 2010-04-22 at 11:56 +0300, Heikki Linnakangas wrote:

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Greg Stark
On Mon, Nov 8, 2010 at 5:59 PM, Aidan Van Dyk ai...@highrise.ca wrote: The problem that putting checksums in a different place solves is the page layout (binary upgrade) problem.  You're still doing to need to buffer the page as you calculate the checksum and write it out. buffering that page

Re: [HACKERS] W3C Specs: Web SQL

2010-11-09 Thread Kevin Grittner
Alvaro Herrera wrote: Excerpts from Charles Pritchard's message: I don't believe the webmaster is granted free rein: Disk quotas are enforced, data is separated per origin, hanging processes are up to the implementer, and postgres has plenty of settings for that. The day a privilege

Re: [HACKERS] security hooks on object creation

2010-11-09 Thread KaiGai Kohei
(2010/11/09 20:34), Itagaki Takahiro wrote: 2010/11/9 KaiGai Koheikai...@ak.jp.nec.com: The attached patch provides plugin modules a hook just after object creation time. In typical use cases, it enables to assign default security labels on object creation by the external security providers.

Re: [HACKERS] timestamp of the last replayed transaction

2010-11-09 Thread Fujii Masao
On Tue, Nov 9, 2010 at 1:05 AM, Robert Haas robertmh...@gmail.com wrote: On Mon, Nov 8, 2010 at 6:00 AM, Fujii Masao masao.fu...@gmail.com wrote: On Sat, Nov 6, 2010 at 9:58 AM, Robert Haas robertmh...@gmail.com wrote: This looks good, but how about adding: if (!RecoveryInProgress())    

[HACKERS] pg_ctl init doc bug

2010-11-09 Thread Euler Taveira de Oliveira
Hi, While executing the following command I got: $ pg_ctl init -D /tmp/foo -o -N 11 /home/euler/pg/bin/initdb: invalid option -- N Try initdb --help for more information. pg_ctl: database system initialization failed I tried -N 11 (postgres option) after looking at the manual but the -o option

Re: [HACKERS] How can we tell how far behind the standby is?

2010-11-09 Thread Fujii Masao
On Tue, Nov 9, 2010 at 7:57 AM, Josh Berkus j...@agliodbs.com wrote: The patch which I'm proposing is helpful for you? http://archives.postgresql.org/pgsql-hackers/2010-11/msg00167.php Depends.  Is that the timestamp on the master (when it was synced), or the timestamp on the standby (when

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Aidan Van Dyk
On Tue, Nov 9, 2010 at 8:45 AM, Greg Stark gsst...@mit.edu wrote: But buffering the page only means you've got some consistent view of the page. It doesn't mean the checksum will actually match the data in the page that gets written out. So when you read it back in the checksum may be

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Greg Stark
On Tue, Nov 9, 2010 at 2:28 PM, Aidan Van Dyk ai...@highrise.ca wrote: On Tue, Nov 9, 2010 at 8:45 AM, Greg Stark gsst...@mit.edu wrote: But buffering the page only means you've got some consistent view of the page. It doesn't mean the checksum will actually match the data in the page that

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Greg Stark
On Tue, Nov 9, 2010 at 3:25 PM, Greg Stark gsst...@mit.edu wrote: Oh, I'm mistaken. The problem was that buffering the writes was insufficient to deal with torn pages. Even if you buffer the writes if the machine crashes while only having written half the buffer out then the checksum won't

Re: [HACKERS] CLUSTER can change t_len

2010-11-09 Thread Heikki Linnakangas
On 09.11.2010 15:57, Greg Stark wrote: On Tue, Nov 9, 2010 at 10:20 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: We have a comment /* be conservative */ in the function, but I'm not sure we actually need the MAXALIGN. However, there would be almost no benefits to keep

Re: [HACKERS] CLUSTER can change t_len

2010-11-09 Thread Greg Stark
On Tue, Nov 9, 2010 at 10:20 AM, Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote: We have a comment /* be conservative */ in the function, but I'm not sure we actually need the MAXALIGN. However, there would be almost no benefits to keep t_len in small value because we often treat

[HACKERS] TODO Alter Table Rename Constraint

2010-11-09 Thread Viktor Valy
Hi Everybody! I looked up this todo, and figured out a plan, how the implementation could be written. The main challenge is to keep constraints and indexes (for unique and PK constraints) consistent. Fortunately this is already realized in the case of an index. So at ALTER INDEX RENAME the

Re: [HACKERS] CLUSTER can change t_len

2010-11-09 Thread Heikki Linnakangas
On 09.11.2010 17:14, Tom Lane wrote: Heikki Linnakangasheikki.linnakan...@enterprisedb.com writes: On 09.11.2010 11:11, Itagaki Takahiro wrote: We have a comment /* be conservative */ in the function, but I'm not sure we actually need the MAXALIGN. However, there would be almost no benefits

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Jim Nasby
On Nov 9, 2010, at 9:27 AM, Greg Stark wrote: On Tue, Nov 9, 2010 at 3:25 PM, Greg Stark gsst...@mit.edu wrote: Oh, I'm mistaken. The problem was that buffering the writes was insufficient to deal with torn pages. Even if you buffer the writes if the machine crashes while only having written

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Gurjeet Singh
On Tue, Nov 9, 2010 at 12:32 AM, Tom Lane t...@sss.pgh.pa.us wrote: There are also crosschecks that you can apply: if it's a heap page, are there any index pages with pointers to it? If it's an index page, are there downlink or sibling links to it from elsewhere in the index? A page that

[HACKERS] DROP TABLESPACE needs crash-resistance

2010-11-09 Thread Gurjeet Singh
We are facing a problem in dropping a tablespace after crash recovery. The recovery starts from the last checkpoint, but the tables that were created by a transaction in a tablespace before the checkpoint are still lying around; the transaction had not finished by the time of crash. After

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Greg Stark
On Tue, Nov 9, 2010 at 4:26 PM, Jim Nasby j...@nasby.net wrote: On Tue, Nov 9, 2010 at 3:25 PM, Greg Stark gsst...@mit.edu wrote: Oh, I'm mistaken. The problem was that buffering the writes was insufficient to deal with torn pages. Even if you buffer the writes if the machine crashes while

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Aidan Van Dyk
On Tue, Nov 9, 2010 at 11:26 AM, Jim Nasby j...@nasby.net wrote: Huh, this implies that if we did go through all the work of segregating the hint bits and could arrange that they all appear on the same 512-byte sector and if we buffered them so that we were writing the same bits we

Re: [HACKERS] proposal: plpgsql - iteration over fields of rec or row variable

2010-11-09 Thread David E. Wheeler
On Nov 9, 2010, at 1:38 AM, Dmitriy Igrishin wrote: * text[] = record_to_array(record) * table(id, key, datatype, value) = record_to_table(record) * text = record_get_field(record, text) * record = record_set_field(record, text, anyelement) ?? I personally like it. But I propose to add as

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Tom Lane
Gurjeet Singh singh.gurj...@gmail.com writes: On Tue, Nov 9, 2010 at 12:32 AM, Tom Lane t...@sss.pgh.pa.us wrote: IMO there are a lot of methods that can separate filesystem misfeasance from Postgres errors, probably with greater reliability than this hack. Doing this postmortem on a regular

Re: [HACKERS] proposal: plpgsql - iteration over fields of rec or row variable

2010-11-09 Thread Dmitriy Igrishin
Hey David, 2010/11/9 David E. Wheeler da...@kineticode.com On Nov 9, 2010, at 1:38 AM, Dmitriy Igrishin wrote: * text[] = record_to_array(record) * table(id, key, datatype, value) = record_to_table(record) * text = record_get_field(record, text) * record = record_set_field(record,

Re: [HACKERS] proposal: plpgsql - iteration over fields of rec or row variable

2010-11-09 Thread David E. Wheeler
On Nov 9, 2010, at 9:18 AM, Dmitriy Igrishin wrote: Yep, but hstore is an additional module. Although, its not a problem. Yeah, but JSON will be in core, and with luck, before long, it will have the same (or similar) capabilities. Best, David -- Sent via pgsql-hackers mailing list

Re: [HACKERS] DROP TABLESPACE needs crash-resistance

2010-11-09 Thread Tom Lane
Gurjeet Singh singh.gurj...@gmail.com writes: We are facing a problem in dropping a tablespace after crash recovery. The recovery starts from the last checkpoint, but the tables that were created by a transaction in a tablespace before the checkpoint are still lying around; the transaction

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Greg Stark
On Tue, Nov 9, 2010 at 5:06 PM, Aidan Van Dyk ai...@highrise.ca wrote: So, for getting checksums, we have to offer up a few things: 1) zero-copy writes, we need to buffer the write to get a consistent checksum (or lock the buffer tight) 2) saving hint-bits on an otherwise unchanged page.  We

Re: [HACKERS] proposal: plpgsql - iteration over fields of rec or row variable

2010-11-09 Thread Tom Lane
David E. Wheeler da...@kineticode.com writes: You realize you can pretty much do all this with hstore, right? Yeah. Anything that involves smashing all the fields to text is not really an advance over (a) hstore or (b) using plperl or one of the other weakly-typed PLs. I think there's a fairly

Re: [HACKERS] proposal: plpgsql - iteration over fields of rec or row variable

2010-11-09 Thread Pavel Stehule
2010/11/9 David E. Wheeler da...@kineticode.com: On Nov 9, 2010, at 1:38 AM, Dmitriy Igrishin wrote: * text[] = record_to_array(record) * table(id, key, datatype, value) = record_to_table(record) * text = record_get_field(record, text) * record = record_set_field(record, text, anyelement)

Re: [HACKERS] proposal: plpgsql - iteration over fields of rec or row variable

2010-11-09 Thread David E. Wheeler
On Nov 9, 2010, at 9:35 AM, Pavel Stehule wrote: You realize you can pretty much do all this with hstore, right? hstore has similar functionality, but missing a some details and add lot of other functionality - it doesn't identify type of field. Personally - it is nothing what I like - but

Re: [HACKERS] proposal: plpgsql - iteration over fields of rec or row variable

2010-11-09 Thread Pavel Stehule
2010/11/9 Tom Lane t...@sss.pgh.pa.us: David E. Wheeler da...@kineticode.com writes: You realize you can pretty much do all this with hstore, right? Yeah.  Anything that involves smashing all the fields to text is not really an advance over (a) hstore or (b) using plperl or one of the other

Re: [HACKERS] proposal: plpgsql - iteration over fields of rec or row variable

2010-11-09 Thread David E. Wheeler
On Nov 9, 2010, at 9:34 AM, Tom Lane wrote: I think there's a fairly fundamental contradiction involved here. One of the basic design attributes of plpgsql is that it's strongly typed. Sometimes that's a blessing, and sometimes it's not, but it's a fact. There really isn't a good way to

Re: [HACKERS] proposal: plpgsql - iteration over fields of rec or row variable

2010-11-09 Thread David E. Wheeler
On Nov 9, 2010, at 9:35 AM, Pavel Stehule wrote: hstore has similar functionality, but missing a some details and add lot of other functionality - it doesn't identify type of field. Personally - it is nothing what I like - but can be better than nothing. What are you going to do with the

Re: [HACKERS] proposal: plpgsql - iteration over fields of rec or row variable

2010-11-09 Thread Pavel Stehule
2010/11/9 David E. Wheeler da...@kineticode.com: On Nov 9, 2010, at 9:35 AM, Pavel Stehule wrote: You realize you can pretty much do all this with hstore, right? hstore has similar functionality, but missing a some details and add lot of other functionality - it doesn't identify type of

Re: [HACKERS] proposal: plpgsql - iteration over fields of rec or row variable

2010-11-09 Thread Pavel Stehule
2010/11/9 David E. Wheeler da...@kineticode.com: On Nov 9, 2010, at 9:34 AM, Tom Lane wrote: I think there's a fairly fundamental contradiction involved here. One of the basic design attributes of plpgsql is that it's strongly typed.  Sometimes that's a blessing, and sometimes it's not, but

Re: [HACKERS] proposal: plpgsql - iteration over fields of rec or row variable

2010-11-09 Thread Pavel Stehule
2010/11/9 David E. Wheeler da...@kineticode.com: On Nov 9, 2010, at 9:35 AM, Pavel Stehule wrote: hstore has similar functionality, but missing a some details and add lot of other functionality - it doesn't identify type of field. Personally - it is nothing what I like - but can be better

Re: [HACKERS] Hot Standby b-tree delete records review

2010-11-09 Thread Heikki Linnakangas
(cleaning up my inbox, and bumped into this..) On 22.04.2010 12:31, Simon Riggs wrote: On Thu, 2010-04-22 at 12:18 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: On Thu, 2010-04-22 at 11:56 +0300, Heikki Linnakangas wrote: If none of the removed heap tuples were present anymore, we

Re: [HACKERS] security hooks on object creation

2010-11-09 Thread Itagaki Takahiro
2010/11/9 KaiGai Kohei kai...@ak.jp.nec.com: The attached patch provides plugin modules a hook just after object creation time. In typical use cases, it enables to assign default security labels on object creation by the external security providers. It looks like DDL Trigger on other database

Re: [HACKERS] proposal: plpgsql - iteration over fields of rec or row variable

2010-11-09 Thread Pavel Stehule
What are you going to do with the type once you have it? for example, you can use it for formatting, for explicit cast, for different serialization type - like JSON - without knowledge of type, you can't to build correct JSON value. So you can write a application with knowledge of type and

Re: [HACKERS] W3C Specs: Web SQL

2010-11-09 Thread Alvaro Herrera
Excerpts from Sam Mason's message of mar nov 09 08:06:12 -0300 2010: On Mon, Nov 08, 2010 at 12:55:22PM -0300, Alvaro Herrera wrote: Excerpts from Charles Pritchard's message of sáb nov 06 23:20:13 -0300 2010: Simple async sql sub-set (the spec in trouble):

Re: [HACKERS] pg_ctl init doc bug

2010-11-09 Thread Tom Lane
Euler Taveira de Oliveira eu...@timbira.com writes: I tried -N 11 (postgres option) after looking at the manual but the -o option only accept initdb options so I think there is a documentation bug. Patch is attached. Yeah, the init option was squeezed in later with only minimal attention to

Re: [HACKERS] How can we tell how far behind the standby is?

2010-11-09 Thread Josh Berkus
It's the former. The function which I'm proposing returns the timestamp of the last replayed commit/abort log record. That timestamp is given when the commit/abort log record is generated in the master. That would be *extremely* helpful for all kinds of monitoring tools. Please

Re: [HACKERS] ALTER TYPE recursion to typed tables

2010-11-09 Thread Peter Eisentraut
Here is the patch that adds [RESTRICT|CASCADE] to ALTER TYPE ... ADD/ALTER/DROP/RENAME ATTRIBUTE, so that recurses to typed tables. diff --git a/doc/src/sgml/ref/alter_type.sgml b/doc/src/sgml/ref/alter_type.sgml index 90de2e8..04395c9 100644 --- a/doc/src/sgml/ref/alter_type.sgml +++

Re: [HACKERS] timestamp of the last replayed transaction

2010-11-09 Thread Josh Berkus
On 11/9/10 5:44 AM, Fujii Masao wrote: But, pg_last_xact_replay_timestamp is more intuitive for many people? If so, let's change the name. *None* of these names are intuitive. So let's just go for consistency. If you want an intuitive name, it would be: pg_replication_log_timestamp() --

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Robert Haas
On Tue, Nov 9, 2010 at 12:31 PM, Greg Stark gsst...@mit.edu wrote: On Tue, Nov 9, 2010 at 5:06 PM, Aidan Van Dyk ai...@highrise.ca wrote: So, for getting checksums, we have to offer up a few things: 1) zero-copy writes, we need to buffer the write to get a consistent checksum (or lock the

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Kenneth Marshall
On Tue, Nov 09, 2010 at 02:05:57PM -0500, Robert Haas wrote: On Tue, Nov 9, 2010 at 12:31 PM, Greg Stark gsst...@mit.edu wrote: On Tue, Nov 9, 2010 at 5:06 PM, Aidan Van Dyk ai...@highrise.ca wrote: So, for getting checksums, we have to offer up a few things: 1) zero-copy writes, we need to

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Alvaro Herrera
Excerpts from Robert Haas's message of mar nov 09 16:05:57 -0300 2010: And it still allows silent data corruption, because bogusly clearing a hint bit is, at the moment, harmless, but bogusly setting one is not. I really have to wonder how other products handle this. PostgreSQL isn't the

Re: [HACKERS] TODO Alter Table Rename Constraint

2010-11-09 Thread Robert Haas
On Tue, Nov 9, 2010 at 10:50 AM, Viktor Valy vili0...@gmail.com wrote: Hi Everybody! I looked up this todo, and figured out a plan, how the implementation could be written. The main challenge is to keep constraints and indexes (for unique and PK constraints) consistent. Fortunately this is

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Josh Berkus
PostgreSQL isn't the only database product that uses MVCC - not by a long shot - and the problem of detecting whether an XID is visible to the current snapshot can't be ours alone. So what do other people do about this? They either don't cache the information about whether the XID is

Re: [HACKERS] proposal: plpgsql - iteration over fields of rec or row variable

2010-11-09 Thread Merlin Moncure
On Tue, Nov 9, 2010 at 12:34 PM, Tom Lane t...@sss.pgh.pa.us wrote: David E. Wheeler da...@kineticode.com writes: You realize you can pretty much do all this with hstore, right? Yeah.  Anything that involves smashing all the fields to text is not really an advance over (a) hstore or (b) using

Re: [HACKERS] CREATE CONSTRAINT TRIGGER

2010-11-09 Thread Alvaro Herrera
Excerpts from Richard Broersma's message of vie nov 05 18:54:54 -0300 2010: On Fri, Nov 5, 2010 at 2:37 PM, Alvaro Herrera alvhe...@alvh.no-ip.org wrote: Recent developments have turned it back into non-deprecated mode; it's not going anywhere, and it needs to be fully documented. From

Re: [HACKERS] proposal: plpgsql - iteration over fields of rec or row variable

2010-11-09 Thread Alvaro Herrera
Excerpts from Merlin Moncure's message of mar nov 09 16:41:32 -0300 2010: The only exception I see is in trigger functions. If the trigger function plan is specific to the firing trigger, new and old are defined at plan time, so something like: new{TG_FIELDNAMES[1]} = 5; -- is ok (at

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Greg Stark
On Tue, Nov 9, 2010 at 7:37 PM, Josh Berkus j...@agliodbs.com wrote: Well, most of the other MVCC-in-table DBMSes simply don't deal with large, on-disk databases.  In fact, I can't think of one which does, currently; while MVCC has been popular for the New Databases, they're all focused on

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Josh Berkus
The whole point of the hint bits is that it's in the same place as the data. Yes, but the hint bits are currently causing us trouble on several features or potential features: * page-level CRC checks * eliminating vacuum freeze for cold data * index-only access * replication * this patch *

[HACKERS] multi-platform, multi-locale regression tests

2010-11-09 Thread Peter Eisentraut
I'm looking for some ideas on how to deal with the regression tests for the per-column collation feature. These are the issues: * The feature only works on some platforms (tentatively: Linux, Windows). - Possible solution: like xml test * The locale names are platform dependent, so there would

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Greg Stark
On Tue, Nov 9, 2010 at 8:12 PM, Josh Berkus j...@agliodbs.com wrote: The whole point of the hint bits is that it's in the same place as the data. Yes, but the hint bits are currently causing us trouble on several features or potential features: Then we might have to get rid of hint bits. But

Re: [HACKERS] W3C Specs: Web SQL

2010-11-09 Thread Charles Pritchard
On 11/8/2010 4:47 PM, Alvaro Herrera wrote: Excerpts from Charles Pritchard's message of lun nov 08 20:25:21 -0300 2010: On 11/8/2010 3:03 PM, Alvaro Herrera wrote: Excerpts from Kevin Grittner's message of lun nov 08 19:30:54 -0300 2010: David Fetterda...@fetter.org wrote: That's not

Re: [HACKERS] W3C Specs: Web SQL

2010-11-09 Thread Jochem van Dieten
On Tue, Nov 9, 2010 at 9:14 PM, Charles Pritchard wrote: Postgres is more stable than Sqlite for enterprise-level activity, hardened/enterprise browser distributions would choose Postgres over Sqlite for Web SQL implementations. I find that very unlikely. Web SQL is to be an upgrade from

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Aidan Van Dyk
On Tue, Nov 9, 2010 at 3:25 PM, Greg Stark gsst...@mit.edu wrote: Then we might have to get rid of hint bits. But they're hint bits for a metadata file that already exists, creating another metadata file doesn't solve anything. Is there any way to instrument the writes of dirty buffers from

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Josh Berkus
Though incidentally all of the other items you mentioned are generic problems caused by with MVCC, not hint bits. Yes, but the hint bits prevent us from implementing workarounds. -- -- Josh Berkus PostgreSQL Experts Inc.

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Robert Haas
On Tue, Nov 9, 2010 at 2:05 PM, Robert Haas robertmh...@gmail.com wrote: On Tue, Nov 9, 2010 at 12:31 PM, Greg Stark gsst...@mit.edu wrote: On Tue, Nov 9, 2010 at 5:06 PM, Aidan Van Dyk ai...@highrise.ca wrote: So, for getting checksums, we have to offer up a few things: 1) zero-copy writes,

Re: [HACKERS] multi-platform, multi-locale regression tests

2010-11-09 Thread David E. Wheeler
On Nov 9, 2010, at 12:18 PM, Peter Eisentraut wrote: One possible way out is not to include these tests in the main test set and instead require manual invocation. Better ideas? I've been talking to Nasby and Dunstan about adding a Test::More/pgTAP-based test suite to the core. It wouldn't

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Josh Berkus
On 11/9/10 1:50 PM, Robert Haas wrote: 5. It would be pretty much impossible to run with autovacuum turned off, and in fact you would likely need to make it a good deal more aggressive in the specific case of aborted transactions, to mitigate problems #1, #3, and #4. 6. This would require us

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Kevin Grittner
Josh Berkus j...@agliodbs.com wrote: 6. This would require us to be more aggressive about VACUUMing old-cold relations/page, e.g. VACUUM FREEZE. This it would make one of our worst issues for data warehousing even worse. I continue to feel that it is insane that when a table is populated

[HACKERS] Build farm server database migration complete

2010-11-09 Thread Andrew Dunstan
It took a little longer than expected, due to a slightly clagged network between the old and new servers, but the database migration is complete and the server is back up and running. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Robert Haas
On Tue, Nov 9, 2010 at 5:03 PM, Josh Berkus j...@agliodbs.com wrote: On 11/9/10 1:50 PM, Robert Haas wrote: 5. It would be pretty much impossible to run with autovacuum turned off, and in fact you would likely need to make it a good deal more aggressive in the specific case of aborted

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Robert Haas
On Tue, Nov 9, 2010 at 5:15 PM, Kevin Grittner kevin.gritt...@wicourts.gov wrote: Josh Berkus j...@agliodbs.com wrote: 6. This would require us to be more aggressive about VACUUMing old-cold relations/page, e.g. VACUUM FREEZE.  This it would make one of our worst issues for data warehousing

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Robert Haas
On Tue, Nov 9, 2010 at 3:05 PM, Greg Stark gsst...@mit.edu wrote: On Tue, Nov 9, 2010 at 7:37 PM, Josh Berkus j...@agliodbs.com wrote: Well, most of the other MVCC-in-table DBMSes simply don't deal with large, on-disk databases.  In fact, I can't think of one which does, currently; while MVCC

Re: [HACKERS] multi-platform, multi-locale regression tests

2010-11-09 Thread Cédric Villemain
2010/11/9 David E. Wheeler da...@kineticode.com: On Nov 9, 2010, at 12:18 PM, Peter Eisentraut wrote: One possible way out is not to include these tests in the main test set and instead require manual invocation. Better ideas? I've been talking to Nasby and Dunstan about adding a

Re: [HACKERS] multi-platform, multi-locale regression tests

2010-11-09 Thread David E. Wheeler
On Nov 9, 2010, at 2:42 PM, Cédric Villemain wrote: Are you thinking of a contrib module 'pgtap' that we can use to accomplish the optionnal regression tests ? Oh, if the project wants it in contrib, sure. Otherwise I'd probably just have the test stuff include it somehow. David -- Sent

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Josh Berkus
Robert, Uh, no it doesn't. It only requires you to be more aggressive about vacuuming the transactions that are in the aborted-XIDs array. It doesn't affect transaction wraparound vacuuming at all, either positively or negatively. You still have to freeze xmins before they flip from being

Re: [HACKERS] patch: Add JSON datatype to PostgreSQL (GSoC, WIP)

2010-11-09 Thread Robert Haas
On Mon, Nov 8, 2010 at 9:52 PM, Itagaki Takahiro itagaki.takah...@gmail.com wrote: Are there any activities in JSON data types for the next commitfest? I'm leaning toward the view that we shouldn't commit a JSON implementation to core (or contrib) for 9.1. We have at least three competing

Re: [HACKERS] multi-platform, multi-locale regression tests

2010-11-09 Thread Cédric Villemain
2010/11/9 David E. Wheeler da...@kineticode.com: On Nov 9, 2010, at 2:42 PM, Cédric Villemain wrote: Are you thinking of a contrib module 'pgtap' that we can use to accomplish the optionnal regression tests ? Oh, if the project wants it in contrib, sure. Otherwise I'd probably just have

Re: [HACKERS] patch: Add JSON datatype to PostgreSQL (GSoC, WIP)

2010-11-09 Thread Terry Laurenzo
Robert, I think I agree. At a minimum, I would like to see the chosen of the competing priorities live on as an outside module for use by previous versions. Even having proposed one, and soon to be two of the competing implementations, it makes me nervous to commit to one at this juncture. I'm

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Tom Lane
Josh Berkus j...@agliodbs.com writes: Though incidentally all of the other items you mentioned are generic problems caused by with MVCC, not hint bits. Yes, but the hint bits prevent us from implementing workarounds. If we got rid of hint bits, we'd need workarounds for the ensuing massive

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes: dons asbestos underpants 4. There would presumably be some finite limit on the size of the shared memory structure for aborted transactions. I don't think there'd be any reason to make it particularly small, but if you sat there and aborted

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Robert Haas
On Tue, Nov 9, 2010 at 5:45 PM, Josh Berkus j...@agliodbs.com wrote: Robert, Uh, no it doesn't.  It only requires you to be more aggressive about vacuuming the transactions that are in the aborted-XIDs array.  It doesn't affect transaction wraparound vacuuming at all, either positively or

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Gurjeet Singh
On Wed, Nov 10, 2010 at 1:15 AM, Tom Lane t...@sss.pgh.pa.us wrote: Once you know that there is, or isn't, a filesystem-level error involved, what are you going to do next? You're going to go try to debug the component you know is at fault, that's what. And that problem is still AI-complete.

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Robert Haas
On Tue, Nov 9, 2010 at 6:42 PM, Tom Lane t...@sss.pgh.pa.us wrote: Robert Haas robertmh...@gmail.com writes: dons asbestos underpants 4. There would presumably be some finite limit on the size of the shared memory structure for aborted transactions.  I don't think there'd be any reason to

Re: [HACKERS] DROP TABLESPACE needs crash-resistance

2010-11-09 Thread Gurjeet Singh
On Wed, Nov 10, 2010 at 1:24 AM, Tom Lane t...@sss.pgh.pa.us wrote: Gurjeet Singh singh.gurj...@gmail.com writes: We are facing a problem in dropping a tablespace after crash recovery. The recovery starts from the last checkpoint, but the tables that were created by a transaction in a

Re: [HACKERS] timestamp of the last replayed transaction

2010-11-09 Thread Fujii Masao
On Wed, Nov 10, 2010 at 3:28 AM, Josh Berkus j...@agliodbs.com wrote: On 11/9/10 5:44 AM, Fujii Masao wrote: But, pg_last_xact_replay_timestamp is more intuitive for many people? If so, let's change the name. *None* of these names are intuitive.  So let's just go for consistency. OK. I

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Robert Haas
On Tue, Nov 9, 2010 at 7:04 PM, Robert Haas robertmh...@gmail.com wrote: On Tue, Nov 9, 2010 at 5:45 PM, Josh Berkus j...@agliodbs.com wrote: Robert, Uh, no it doesn't.  It only requires you to be more aggressive about vacuuming the transactions that are in the aborted-XIDs array.  It

Re: [HACKERS] timestamp of the last replayed transaction

2010-11-09 Thread Robert Haas
On Tue, Nov 9, 2010 at 8:55 PM, Fujii Masao masao.fu...@gmail.com wrote: On Wed, Nov 10, 2010 at 3:28 AM, Josh Berkus j...@agliodbs.com wrote: On 11/9/10 5:44 AM, Fujii Masao wrote: But, pg_last_xact_replay_timestamp is more intuitive for many people? If so, let's change the name. *None* of

[HACKERS] We need index-only scans

2010-11-09 Thread Bruce Momjian
We last researched index-only scans, also called covering indexes, in September of 2008, but have made little progress on it since. Many have been waiting for Heikki to implement this but I talked to him and he doesn't have time. I believe it is time for the community to move forward and I

Re: [HACKERS] timestamp of the last replayed transaction

2010-11-09 Thread Fujii Masao
On Wed, Nov 10, 2010 at 12:57 PM, Robert Haas robertmh...@gmail.com wrote: On Tue, Nov 9, 2010 at 8:55 PM, Fujii Masao masao.fu...@gmail.com wrote: On Wed, Nov 10, 2010 at 3:28 AM, Josh Berkus j...@agliodbs.com wrote: On 11/9/10 5:44 AM, Fujii Masao wrote: But, pg_last_xact_replay_timestamp is

Re: [HACKERS] security hooks on object creation

2010-11-09 Thread Robert Haas
2010/11/9 KaiGai Kohei kai...@ak.jp.nec.com: The attached patch provides plugin modules a hook just after object creation time. In typical use cases, it enables to assign default security labels on object creation by the external security providers. As Robert suggested before, it provides a

Re: [HACKERS] Which file does the SELECT?

2010-11-09 Thread Robert Haas
On Mon, Nov 8, 2010 at 9:55 PM, Vaibhav Kaushal vaibhavkaushal...@gmail.com wrote: I have started with the work and am using Eclipse and it helps quite a lot. I can find the declarations quite easily. Thanks to open Source. BTW, I am encountering too many (just too many) data types as I try to

Re: [HACKERS] multi-platform, multi-locale regression tests

2010-11-09 Thread Peter Eisentraut
On tis, 2010-11-09 at 14:00 -0800, David E. Wheeler wrote: I've been talking to Nasby and Dunstan about adding a Test::More/pgTAP-based test suite to the core. It wouldn't run with the usual core suite used by developers, which would continue to use pg_regress. But they could run it if they