Re: [HACKERS] multi-platform, multi-locale regression tests

2010-11-09 Thread Peter Eisentraut
On tis, 2010-11-09 at 14:00 -0800, David E. Wheeler wrote: > I've been talking to Nasby and Dunstan about adding a Test::More/pgTAP-based > test suite to the core. It wouldn't run with the usual core suite used by > developers, which would continue to use pg_regress. But they could run it if > t

Re: [HACKERS] Which file does the SELECT?

2010-11-09 Thread Robert Haas
On Mon, Nov 8, 2010 at 9:55 PM, Vaibhav Kaushal wrote: > I have started with the work and am using Eclipse and it helps quite a lot. > I can find the declarations quite easily. Thanks to open Source. > > BTW, I am encountering too many (just too many) data types as I try to > understand the backen

Re: [HACKERS] security hooks on object creation

2010-11-09 Thread Robert Haas
2010/11/9 KaiGai Kohei : > The attached patch provides plugin modules a hook just after object > creation time. In typical use cases, it enables to assign default > security labels on object creation by the external security providers. > > As Robert suggested before, it provides a generic purpose m

Re: [HACKERS] timestamp of the last replayed transaction

2010-11-09 Thread Fujii Masao
On Wed, Nov 10, 2010 at 12:57 PM, Robert Haas wrote: > On Tue, Nov 9, 2010 at 8:55 PM, Fujii Masao wrote: >> On Wed, Nov 10, 2010 at 3:28 AM, Josh Berkus wrote: >>> On 11/9/10 5:44 AM, Fujii Masao wrote: But, pg_last_xact_replay_timestamp is more intuitive for many people? If so, let's

[HACKERS] We need index-only scans

2010-11-09 Thread Bruce Momjian
We last researched index-only scans, also called covering indexes, in September of 2008, but have made little progress on it since. Many have been waiting for Heikki to implement this but I talked to him and he doesn't have time. I believe it is time for the community to move forward and I woul

Re: [HACKERS] timestamp of the last replayed transaction

2010-11-09 Thread Robert Haas
On Tue, Nov 9, 2010 at 8:55 PM, Fujii Masao wrote: > On Wed, Nov 10, 2010 at 3:28 AM, Josh Berkus wrote: >> On 11/9/10 5:44 AM, Fujii Masao wrote: >>> But, pg_last_xact_replay_timestamp is more intuitive for many people? >>> If so, let's change >>> the name. >> >> *None* of these names are intuit

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Robert Haas
On Tue, Nov 9, 2010 at 7:04 PM, Robert Haas wrote: > On Tue, Nov 9, 2010 at 5:45 PM, Josh Berkus wrote: >> Robert, >> >>> Uh, no it doesn't.  It only requires you to be more aggressive about >>> vacuuming the transactions that are in the aborted-XIDs array.  It >>> doesn't affect transaction wrap

Re: [HACKERS] timestamp of the last replayed transaction

2010-11-09 Thread Fujii Masao
On Wed, Nov 10, 2010 at 3:28 AM, Josh Berkus wrote: > On 11/9/10 5:44 AM, Fujii Masao wrote: >> But, pg_last_xact_replay_timestamp is more intuitive for many people? >> If so, let's change >> the name. > > *None* of these names are intuitive.  So let's just go for consistency. OK. I changed the n

Re: [HACKERS] DROP TABLESPACE needs crash-resistance

2010-11-09 Thread Gurjeet Singh
On Wed, Nov 10, 2010 at 1:24 AM, Tom Lane wrote: > Gurjeet Singh writes: > > We are facing a problem in dropping a tablespace after crash recovery. > The > > recovery starts from the last checkpoint, but the tables that were > created > > by > > a transaction in a tablespace before the checkpoin

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Robert Haas
On Tue, Nov 9, 2010 at 6:42 PM, Tom Lane wrote: > Robert Haas writes: >> >> 4. There would presumably be some finite limit on the size of the >> shared memory structure for aborted transactions.  I don't think >> there'd be any reason to make it particularly small, but if you sat >> there and ab

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Gurjeet Singh
On Wed, Nov 10, 2010 at 1:15 AM, Tom Lane wrote: > Once you know that there is, or isn't, > a filesystem-level error involved, what are you going to do next? > You're going to go try to debug the component you know is at fault, > that's what. And that problem is still AI-complete. > > If we know

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Robert Haas
On Tue, Nov 9, 2010 at 5:45 PM, Josh Berkus wrote: > Robert, > >> Uh, no it doesn't.  It only requires you to be more aggressive about >> vacuuming the transactions that are in the aborted-XIDs array.  It >> doesn't affect transaction wraparound vacuuming at all, either >> positively or negatively

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Tom Lane
Robert Haas writes: > > 4. There would presumably be some finite limit on the size of the > shared memory structure for aborted transactions. I don't think > there'd be any reason to make it particularly small, but if you sat > there and aborted transactions at top speed you might eventually run

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Tom Lane
Josh Berkus writes: >> Though incidentally all of the other items you mentioned are generic >> problems caused by with MVCC, not hint bits. > Yes, but the hint bits prevent us from implementing workarounds. If we got rid of hint bits, we'd need workarounds for the ensuing massive performance los

Re: [HACKERS] patch: Add JSON datatype to PostgreSQL (GSoC, WIP)

2010-11-09 Thread Terry Laurenzo
Robert, I think I agree. At a minimum, I would like to see the "chosen" of the competing priorities live on as an outside module for use by previous versions. Even having proposed one, and soon to be two of the competing implementations, it makes me nervous to commit to one at this juncture. I'm

Re: [HACKERS] multi-platform, multi-locale regression tests

2010-11-09 Thread Cédric Villemain
2010/11/9 David E. Wheeler : > On Nov 9, 2010, at 2:42 PM, Cédric Villemain wrote: > >> Are you thinking of a contrib module 'pgtap' that we can use to >> accomplish the optionnal regression tests ? > > Oh, if the project wants it in contrib, sure. Otherwise I'd probably just > have the test stuff

Re: [HACKERS] patch: Add JSON datatype to PostgreSQL (GSoC, WIP)

2010-11-09 Thread Robert Haas
On Mon, Nov 8, 2010 at 9:52 PM, Itagaki Takahiro wrote: > Are there any activities in JSON data types for the next commitfest? I'm leaning toward the view that we shouldn't commit a JSON implementation to core (or contrib) for 9.1. We have at least three competing proposals on the table. I thou

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Josh Berkus
Robert, > Uh, no it doesn't. It only requires you to be more aggressive about > vacuuming the transactions that are in the aborted-XIDs array. It > doesn't affect transaction wraparound vacuuming at all, either > positively or negatively. You still have to freeze xmins before they > flip from b

Re: [HACKERS] multi-platform, multi-locale regression tests

2010-11-09 Thread David E. Wheeler
On Nov 9, 2010, at 2:42 PM, Cédric Villemain wrote: > Are you thinking of a contrib module 'pgtap' that we can use to > accomplish the optionnal regression tests ? Oh, if the project wants it in contrib, sure. Otherwise I'd probably just have the test stuff include it somehow. David -- Sent

Re: [HACKERS] multi-platform, multi-locale regression tests

2010-11-09 Thread Cédric Villemain
2010/11/9 David E. Wheeler : > On Nov 9, 2010, at 12:18 PM, Peter Eisentraut wrote: > >> One possible way out is not to include these tests in the main test set >> and instead require manual invocation. >> >> Better ideas? > > I've been talking to Nasby and Dunstan about adding a Test::More/pgTAP-b

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Robert Haas
On Tue, Nov 9, 2010 at 3:05 PM, Greg Stark wrote: > On Tue, Nov 9, 2010 at 7:37 PM, Josh Berkus wrote: >> Well, most of the other MVCC-in-table DBMSes simply don't deal with >> large, on-disk databases.  In fact, I can't think of one which does, >> currently; while MVCC has been popular for the N

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Robert Haas
On Tue, Nov 9, 2010 at 5:15 PM, Kevin Grittner wrote: > Josh Berkus wrote: > >> 6. This would require us to be more aggressive about VACUUMing >> old-cold relations/page, e.g. VACUUM FREEZE.  This it would make >> one of our worst issues for data warehousing even worse. > > I continue to feel tha

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Robert Haas
On Tue, Nov 9, 2010 at 5:03 PM, Josh Berkus wrote: > On 11/9/10 1:50 PM, Robert Haas wrote: >> 5. It would be pretty much impossible to run with autovacuum turned >> off, and in fact you would likely need to make it a good deal more >> aggressive in the specific case of aborted transactions, to mi

[HACKERS] Build farm server database migration complete

2010-11-09 Thread Andrew Dunstan
It took a little longer than expected, due to a slightly clagged network between the old and new servers, but the database migration is complete and the server is back up and running. cheers andrew -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Kevin Grittner
Josh Berkus wrote: > 6. This would require us to be more aggressive about VACUUMing > old-cold relations/page, e.g. VACUUM FREEZE. This it would make > one of our worst issues for data warehousing even worse. I continue to feel that it is insane that when a table is populated within the same

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Josh Berkus
On 11/9/10 1:50 PM, Robert Haas wrote: > 5. It would be pretty much impossible to run with autovacuum turned > off, and in fact you would likely need to make it a good deal more > aggressive in the specific case of aborted transactions, to mitigate > problems #1, #3, and #4. 6. This would require

Re: [HACKERS] multi-platform, multi-locale regression tests

2010-11-09 Thread David E. Wheeler
On Nov 9, 2010, at 12:18 PM, Peter Eisentraut wrote: > One possible way out is not to include these tests in the main test set > and instead require manual invocation. > > Better ideas? I've been talking to Nasby and Dunstan about adding a Test::More/pgTAP-based test suite to the core. It would

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Robert Haas
On Tue, Nov 9, 2010 at 2:05 PM, Robert Haas wrote: > On Tue, Nov 9, 2010 at 12:31 PM, Greg Stark wrote: >> On Tue, Nov 9, 2010 at 5:06 PM, Aidan Van Dyk wrote: >>> So, for getting checksums, we have to offer up a few things: >>> 1) zero-copy writes, we need to buffer the write to get a consisten

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Josh Berkus
> Though incidentally all of the other items you mentioned are generic > problems caused by with MVCC, not hint bits. Yes, but the hint bits prevent us from implementing workarounds. -- -- Josh Berkus PostgreSQL Experts Inc

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Aidan Van Dyk
On Tue, Nov 9, 2010 at 3:25 PM, Greg Stark wrote: > Then we might have to get rid of hint bits. But they're hint bits for > a metadata file that already exists, creating another metadata file > doesn't solve anything. Is there any way to instrument the writes of dirty buffers from the share memo

Re: [HACKERS] W3C Specs: Web SQL

2010-11-09 Thread Jochem van Dieten
On Tue, Nov 9, 2010 at 9:14 PM, Charles Pritchard wrote: > Postgres is more stable than Sqlite for "enterprise-level" activity, > hardened/enterprise > browser distributions would choose Postgres over Sqlite for Web SQL > implementations. I find that very unlikely. Web SQL is to be an upgrade from

Re: [HACKERS] W3C Specs: Web SQL

2010-11-09 Thread Charles Pritchard
On 11/8/2010 4:47 PM, Alvaro Herrera wrote: Excerpts from Charles Pritchard's message of lun nov 08 20:25:21 -0300 2010: On 11/8/2010 3:03 PM, Alvaro Herrera wrote: Excerpts from Kevin Grittner's message of lun nov 08 19:30:54 -0300 2010: David Fetter wrote: That's not proof against a DoS

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Greg Stark
On Tue, Nov 9, 2010 at 8:12 PM, Josh Berkus wrote: >> The whole point of the hint bits is that it's in the same place as the data. > > Yes, but the hint bits are currently causing us trouble on several > features or potential features: Then we might have to get rid of hint bits. But they're hint

[HACKERS] multi-platform, multi-locale regression tests

2010-11-09 Thread Peter Eisentraut
I'm looking for some ideas on how to deal with the regression tests for the per-column collation feature. These are the issues: * The feature only works on some platforms (tentatively: Linux, Windows). -> Possible solution: like xml test * The locale names are platform dependent, so there would

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Josh Berkus
> The whole point of the hint bits is that it's in the same place as the data. Yes, but the hint bits are currently causing us trouble on several features or potential features: * page-level CRC checks * eliminating vacuum freeze for cold data * index-only access * replication * this patch * etc

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Greg Stark
On Tue, Nov 9, 2010 at 7:37 PM, Josh Berkus wrote: > Well, most of the other MVCC-in-table DBMSes simply don't deal with > large, on-disk databases.  In fact, I can't think of one which does, > currently; while MVCC has been popular for the New Databases, they're > all focused on "in-memory" datab

Re: [HACKERS] proposal: plpgsql - iteration over fields of rec or row variable

2010-11-09 Thread Alvaro Herrera
Excerpts from Merlin Moncure's message of mar nov 09 16:41:32 -0300 2010: > The only exception I see is in trigger functions. If the trigger > function plan is specific to the firing trigger, new and old are > defined at plan time, so something like: > > new{TG_FIELDNAMES[1]} = 5; -- is ok (at

Re: [HACKERS] CREATE CONSTRAINT TRIGGER

2010-11-09 Thread Alvaro Herrera
Excerpts from Richard Broersma's message of vie nov 05 18:54:54 -0300 2010: > On Fri, Nov 5, 2010 at 2:37 PM, Alvaro Herrera > wrote: > > > Recent developments have turned it back into non-deprecated mode; it's > > not going anywhere, and it needs to be fully documented. > > From what I recall,

Re: [HACKERS] proposal: plpgsql - iteration over fields of rec or row variable

2010-11-09 Thread Merlin Moncure
On Tue, Nov 9, 2010 at 12:34 PM, Tom Lane wrote: > "David E. Wheeler" writes: >> You realize you can pretty much do all this with hstore, right? > > Yeah.  Anything that involves smashing all the fields to text is not > really an advance over (a) hstore or (b) using plperl or one of the > other w

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Josh Berkus
> PostgreSQL > isn't the only database product that uses MVCC - not by a long shot - > and the problem of detecting whether an XID is visible to the current > snapshot can't be ours alone. So what do other people do about this? > They either don't cache the information about whether the XID is >

Re: [HACKERS] TODO Alter Table Rename Constraint

2010-11-09 Thread Robert Haas
On Tue, Nov 9, 2010 at 10:50 AM, Viktor Valy wrote: > Hi Everybody! > I looked up this todo, and figured out a plan, how the implementation could > be written. > The main challenge is to keep constraints and indexes (for unique and PK > constraints) consistent. > Fortunately this is already realiz

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Alvaro Herrera
Excerpts from Robert Haas's message of mar nov 09 16:05:57 -0300 2010: > And it still allows silent data corruption, because bogusly clearing a > hint bit is, at the moment, harmless, but bogusly setting one is not. > I really have to wonder how other products handle this. PostgreSQL > isn't the

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Kenneth Marshall
On Tue, Nov 09, 2010 at 02:05:57PM -0500, Robert Haas wrote: > On Tue, Nov 9, 2010 at 12:31 PM, Greg Stark wrote: > > On Tue, Nov 9, 2010 at 5:06 PM, Aidan Van Dyk wrote: > >> So, for getting checksums, we have to offer up a few things: > >> 1) zero-copy writes, we need to buffer the write to get

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Robert Haas
On Tue, Nov 9, 2010 at 12:31 PM, Greg Stark wrote: > On Tue, Nov 9, 2010 at 5:06 PM, Aidan Van Dyk wrote: >> So, for getting checksums, we have to offer up a few things: >> 1) zero-copy writes, we need to buffer the write to get a consistent >> checksum (or lock the buffer tight) >> 2) saving hin

Re: [HACKERS] timestamp of the last replayed transaction

2010-11-09 Thread Josh Berkus
On 11/9/10 5:44 AM, Fujii Masao wrote: > But, pg_last_xact_replay_timestamp is more intuitive for many people? > If so, let's change > the name. *None* of these names are intuitive. So let's just go for consistency. If you want an intuitive name, it would be: pg_replication_log_timestamp() --

Re: [HACKERS] ALTER TYPE recursion to typed tables

2010-11-09 Thread Peter Eisentraut
Here is the patch that adds [RESTRICT|CASCADE] to ALTER TYPE ... ADD/ALTER/DROP/RENAME ATTRIBUTE, so that recurses to typed tables. diff --git a/doc/src/sgml/ref/alter_type.sgml b/doc/src/sgml/ref/alter_type.sgml index 90de2e8..04395c9 100644 --- a/doc/src/sgml/ref/alter_type.sgml +++ b/doc/src/sgm

Re: [HACKERS] How can we tell how far behind the standby is?

2010-11-09 Thread Josh Berkus
> It's the former. The function which I'm proposing returns the timestamp > of the last replayed commit/abort log record. That timestamp is given > when the commit/abort log record is generated in the master. That would be *extremely* helpful for all kinds of monitoring tools. Please complete/sub

Re: [HACKERS] pg_ctl init doc bug

2010-11-09 Thread Tom Lane
Euler Taveira de Oliveira writes: > I tried "-N 11" (postgres option) after looking at the manual but the -o > option only accept initdb options so I think there is a documentation bug. > Patch is attached. Yeah, the init option was squeezed in later with only minimal attention to documentation.

Re: [HACKERS] W3C Specs: Web SQL

2010-11-09 Thread Alvaro Herrera
Excerpts from Sam Mason's message of mar nov 09 08:06:12 -0300 2010: > On Mon, Nov 08, 2010 at 12:55:22PM -0300, Alvaro Herrera wrote: > > Excerpts from Charles Pritchard's message of sáb nov 06 23:20:13 -0300 2010: > > > > > Simple async sql sub-set (the spec in trouble): > > > http://dev.w3.org/

Re: [HACKERS] proposal: plpgsql - iteration over fields of rec or row variable

2010-11-09 Thread Pavel Stehule
>> >> What are you going to do with the type once you have it? >> > > for example, you can use it for formatting, for explicit cast, for > different serialization type - like JSON - without knowledge of type, > you can't to build correct JSON value. So you can write a application > with knowledge o

Re: [HACKERS] security hooks on object creation

2010-11-09 Thread Itagaki Takahiro
2010/11/9 KaiGai Kohei : > The attached patch provides plugin modules a hook just after object > creation time. In typical use cases, it enables to assign default > security labels on object creation by the external security providers. It looks like "DDL Trigger" on other database products. Do we

Re: [HACKERS] Hot Standby b-tree delete records review

2010-11-09 Thread Heikki Linnakangas
(cleaning up my inbox, and bumped into this..) On 22.04.2010 12:31, Simon Riggs wrote: On Thu, 2010-04-22 at 12:18 +0300, Heikki Linnakangas wrote: Simon Riggs wrote: On Thu, 2010-04-22 at 11:56 +0300, Heikki Linnakangas wrote: If none of the removed heap tuples were present anymore, we curr

Re: [HACKERS] proposal: plpgsql - iteration over fields of rec or row variable

2010-11-09 Thread Pavel Stehule
2010/11/9 David E. Wheeler : > On Nov 9, 2010, at 9:35 AM, Pavel Stehule wrote: > >> hstore has similar functionality, but missing a some details and add >> lot of other functionality - it doesn't identify type of field. >> Personally - it is nothing what I like - but can be better than >> nothing.

Re: [HACKERS] proposal: plpgsql - iteration over fields of rec or row variable

2010-11-09 Thread Pavel Stehule
2010/11/9 David E. Wheeler : > On Nov 9, 2010, at 9:34 AM, Tom Lane wrote: > >> I think there's a fairly fundamental contradiction involved here. >> One of the basic design attributes of plpgsql is that it's strongly >> typed.  Sometimes that's a blessing, and sometimes it's not, but >> it's a fact

Re: [HACKERS] proposal: plpgsql - iteration over fields of rec or row variable

2010-11-09 Thread Pavel Stehule
2010/11/9 David E. Wheeler : > On Nov 9, 2010, at 9:35 AM, Pavel Stehule wrote: > >>> You realize you can pretty much do all this with hstore, right? >> >> hstore has similar functionality, but missing a some details and add >> lot of other functionality - it doesn't identify type of field. >> Pers

Re: [HACKERS] proposal: plpgsql - iteration over fields of rec or row variable

2010-11-09 Thread David E. Wheeler
On Nov 9, 2010, at 9:35 AM, Pavel Stehule wrote: > hstore has similar functionality, but missing a some details and add > lot of other functionality - it doesn't identify type of field. > Personally - it is nothing what I like - but can be better than > nothing. What are you going to do with the

Re: [HACKERS] proposal: plpgsql - iteration over fields of rec or row variable

2010-11-09 Thread David E. Wheeler
On Nov 9, 2010, at 9:34 AM, Tom Lane wrote: > I think there's a fairly fundamental contradiction involved here. > One of the basic design attributes of plpgsql is that it's strongly > typed. Sometimes that's a blessing, and sometimes it's not, but > it's a fact. There really isn't a good way to

Re: [HACKERS] proposal: plpgsql - iteration over fields of rec or row variable

2010-11-09 Thread Pavel Stehule
2010/11/9 Tom Lane : > "David E. Wheeler" writes: >> You realize you can pretty much do all this with hstore, right? > > Yeah.  Anything that involves smashing all the fields to text is not > really an advance over (a) hstore or (b) using plperl or one of the > other weakly-typed PLs. > > I think

Re: [HACKERS] proposal: plpgsql - iteration over fields of rec or row variable

2010-11-09 Thread David E. Wheeler
On Nov 9, 2010, at 9:35 AM, Pavel Stehule wrote: >> You realize you can pretty much do all this with hstore, right? > > hstore has similar functionality, but missing a some details and add > lot of other functionality - it doesn't identify type of field. > Personally - it is nothing what I like -

Re: [HACKERS] proposal: plpgsql - iteration over fields of rec or row variable

2010-11-09 Thread Pavel Stehule
2010/11/9 David E. Wheeler : > On Nov 9, 2010, at 1:38 AM, Dmitriy Igrishin wrote: > >> * text[] = record_to_array(record) >> * table(id, key, datatype, value) = record_to_table(record) >> * text = record_get_field(record, text) >> * record = record_set_field(record, text, anyelement) >> >> ?? >> I

Re: [HACKERS] proposal: plpgsql - iteration over fields of rec or row variable

2010-11-09 Thread Tom Lane
"David E. Wheeler" writes: > You realize you can pretty much do all this with hstore, right? Yeah. Anything that involves smashing all the fields to text is not really an advance over (a) hstore or (b) using plperl or one of the other weakly-typed PLs. I think there's a fairly fundamental contr

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Greg Stark
On Tue, Nov 9, 2010 at 5:06 PM, Aidan Van Dyk wrote: > So, for getting checksums, we have to offer up a few things: > 1) zero-copy writes, we need to buffer the write to get a consistent > checksum (or lock the buffer tight) > 2) saving hint-bits on an otherwise unchanged page.  We either need to

Re: [HACKERS] DROP TABLESPACE needs crash-resistance

2010-11-09 Thread Tom Lane
Gurjeet Singh writes: > We are facing a problem in dropping a tablespace after crash recovery. The > recovery starts from the last checkpoint, but the tables that were created > by > a transaction in a tablespace before the checkpoint are still lying around; > the > transaction had not finished by

Re: [HACKERS] proposal: plpgsql - iteration over fields of rec or row variable

2010-11-09 Thread David E. Wheeler
On Nov 9, 2010, at 9:18 AM, Dmitriy Igrishin wrote: > Yep, but hstore is an additional module. Although, its not a problem. Yeah, but JSON will be in core, and with luck, before long, it will have the same (or similar) capabilities. Best, David -- Sent via pgsql-hackers mailing list (pgsql-

Re: [HACKERS] proposal: plpgsql - iteration over fields of rec or row variable

2010-11-09 Thread Dmitriy Igrishin
Hey David, 2010/11/9 David E. Wheeler > On Nov 9, 2010, at 1:38 AM, Dmitriy Igrishin wrote: > > > * text[] = record_to_array(record) > > * table(id, key, datatype, value) = record_to_table(record) > > * text = record_get_field(record, text) > > * record = record_set_field(record, text, anyelemen

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Tom Lane
Gurjeet Singh writes: > On Tue, Nov 9, 2010 at 12:32 AM, Tom Lane wrote: >> IMO there are a lot of methods that can separate filesystem misfeasance >> from Postgres errors, probably with greater reliability than this hack. > Doing this postmortem on a regular deployment and fixing the problem wo

Re: [HACKERS] proposal: plpgsql - iteration over fields of rec or row variable

2010-11-09 Thread David E. Wheeler
On Nov 9, 2010, at 1:38 AM, Dmitriy Igrishin wrote: > * text[] = record_to_array(record) > * table(id, key, datatype, value) = record_to_table(record) > * text = record_get_field(record, text) > * record = record_set_field(record, text, anyelement) > > ?? > I personally like it. But I propose to

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Aidan Van Dyk
On Tue, Nov 9, 2010 at 11:26 AM, Jim Nasby wrote: >> Huh, this implies that if we did go through all the work of >> segregating the hint bits and could arrange that they all appear on >> the same 512-byte sector and if we buffered them so that we were >> writing the same bits we checksummed then

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Greg Stark
On Tue, Nov 9, 2010 at 4:26 PM, Jim Nasby wrote: >> On Tue, Nov 9, 2010 at 3:25 PM, Greg Stark wrote: >>> Oh, I'm mistaken. The problem was that buffering the writes was >>> insufficient to deal with torn pages. Even if you buffer the writes if >>> the machine crashes while only having written ha

[HACKERS] DROP TABLESPACE needs crash-resistance

2010-11-09 Thread Gurjeet Singh
We are facing a problem in dropping a tablespace after crash recovery. The recovery starts from the last checkpoint, but the tables that were created by a transaction in a tablespace before the checkpoint are still lying around; the transaction had not finished by the time of crash. After recovery

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Gurjeet Singh
On Tue, Nov 9, 2010 at 12:32 AM, Tom Lane wrote: > There are also crosschecks that you can apply: if it's a heap page, are > there any index pages with pointers to it? If it's an index page, are > there downlink or sibling links to it from elsewhere in the index? > A page that Postgres left as z

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Jim Nasby
On Nov 9, 2010, at 9:27 AM, Greg Stark wrote: > On Tue, Nov 9, 2010 at 3:25 PM, Greg Stark wrote: >> Oh, I'm mistaken. The problem was that buffering the writes was >> insufficient to deal with torn pages. Even if you buffer the writes if >> the machine crashes while only having written half the b

Re: [HACKERS] CLUSTER can change t_len

2010-11-09 Thread Heikki Linnakangas
On 09.11.2010 17:14, Tom Lane wrote: Heikki Linnakangas writes: On 09.11.2010 11:11, Itagaki Takahiro wrote: We have a comment /* be conservative */ in the function, but I'm not sure we actually need the MAXALIGN. However, there would be almost no benefits to keep t_len in small value because

[HACKERS] TODO Alter Table Rename Constraint

2010-11-09 Thread Viktor Valy
Hi Everybody! I looked up this todo, and figured out a plan, how the implementation could be written. The main challenge is to keep constraints and indexes (for unique and PK constraints) consistent. Fortunately this is already realized in the case of an index. So at ALTER INDEX RENAME the consist

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Greg Stark
On Tue, Nov 9, 2010 at 3:25 PM, Greg Stark wrote: > Oh, I'm mistaken. The problem was that buffering the writes was > insufficient to deal with torn pages. Even if you buffer the writes if > the machine crashes while only having written half the buffer out then > the checksum won't match. If the o

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Greg Stark
On Tue, Nov 9, 2010 at 2:28 PM, Aidan Van Dyk wrote: > On Tue, Nov 9, 2010 at 8:45 AM, Greg Stark wrote: > >> But buffering the page only means you've got some consistent view of >> the page. It doesn't mean the checksum will actually match the data in >> the page that gets written out. So when y

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Aidan Van Dyk
On Tue, Nov 9, 2010 at 8:45 AM, Greg Stark wrote: > But buffering the page only means you've got some consistent view of > the page. It doesn't mean the checksum will actually match the data in > the page that gets written out. So when you read it back in the > checksum may be invalid. I was ass

Re: [HACKERS] How can we tell how far behind the standby is?

2010-11-09 Thread Fujii Masao
On Tue, Nov 9, 2010 at 7:57 AM, Josh Berkus wrote: > >> The patch which I'm proposing is helpful for you? >> http://archives.postgresql.org/pgsql-hackers/2010-11/msg00167.php > > Depends.  Is that the timestamp on the master (when it was synced), or > the timestamp on the standby (when it was repl

[HACKERS] pg_ctl init doc bug

2010-11-09 Thread Euler Taveira de Oliveira
Hi, While executing the following command I got: $ pg_ctl init -D /tmp/foo -o "-N 11" /home/euler/pg/bin/initdb: invalid option -- N Try "initdb --help" for more information. pg_ctl: database system initialization failed I tried "-N 11" (postgres option) after looking at the manual but the -o op

Re: [HACKERS] CLUSTER can change t_len

2010-11-09 Thread Heikki Linnakangas
On 09.11.2010 15:57, Greg Stark wrote: On Tue, Nov 9, 2010 at 10:20 AM, Heikki Linnakangas wrote: We have a comment /* be conservative */ in the function, but I'm not sure we actually need the MAXALIGN. However, there would be almost no benefits to keep t_len in small value because we often t

Re: [HACKERS] CLUSTER can change t_len

2010-11-09 Thread Greg Stark
On Tue, Nov 9, 2010 at 10:20 AM, Heikki Linnakangas wrote: >> >> We have a comment /* be conservative */ in the function, but I'm not sure >> we actually need the MAXALIGN. However, there would be almost no benefits >> to keep t_len in small value because we often treat memory in MAXALIGN >> unit.

Re: [HACKERS] timestamp of the last replayed transaction

2010-11-09 Thread Fujii Masao
On Tue, Nov 9, 2010 at 1:05 AM, Robert Haas wrote: > On Mon, Nov 8, 2010 at 6:00 AM, Fujii Masao wrote: >> On Sat, Nov 6, 2010 at 9:58 AM, Robert Haas wrote: >>> This looks good, but how about adding: >>> >>> if (!RecoveryInProgress()) >>>    PG_RETURN_NULL(); >>> >>> Otherwise, if we're in Hot

Re: [HACKERS] security hooks on object creation

2010-11-09 Thread KaiGai Kohei
(2010/11/09 20:34), Itagaki Takahiro wrote: 2010/11/9 KaiGai Kohei: The attached patch provides plugin modules a hook just after object creation time. In typical use cases, it enables to assign default security labels on object creation by the external security providers. It looks like "DDL Tr

Re: [HACKERS] W3C Specs: Web SQL

2010-11-09 Thread Kevin Grittner
Alvaro Herrera wrote: > Excerpts from Charles Pritchard's message: >> I don't believe the webmaster is granted free rein: >> Disk quotas are enforced, data is separated per origin, >> hanging processes are up to the implementer, and postgres has >> plenty of settings for that. > > The day a priv

Re: [HACKERS] Protecting against unexpected zero-pages: proposal

2010-11-09 Thread Greg Stark
On Mon, Nov 8, 2010 at 5:59 PM, Aidan Van Dyk wrote: > The problem that putting checksums in a different place solves is the > page layout (binary upgrade) problem.  You're still doing to need to > "buffer" the page as you calculate the checksum and write it out. > buffering that page is absolutel

Re: [HACKERS] Hot Standby b-tree delete records review

2010-11-09 Thread Simon Riggs
On Tue, 2010-11-09 at 13:34 +0200, Heikki Linnakangas wrote: > (cleaning up my inbox, and bumped into this..) > > On 22.04.2010 12:31, Simon Riggs wrote: > > On Thu, 2010-04-22 at 12:18 +0300, Heikki Linnakangas wrote: > >> Simon Riggs wrote: > >>> On Thu, 2010-04-22 at 11:56 +0300, Heikki Linnaka

Re: [HACKERS] CLUSTER can change t_len

2010-11-09 Thread Tom Lane
Heikki Linnakangas writes: > On 09.11.2010 11:11, Itagaki Takahiro wrote: >> We have a comment /* be conservative */ in the function, but I'm not sure >> we actually need the MAXALIGN. However, there would be almost no benefits >> to keep t_len in small value because we often treat memory in MAXAL

Re: [HACKERS] improved parallel make support

2010-11-09 Thread Dave Page
On Sat, Nov 6, 2010 at 4:35 AM, Peter Eisentraut wrote: > On ons, 2010-11-03 at 16:34 +0200, Peter Eisentraut wrote: >> On tis, 2010-11-02 at 10:21 -0400, Tom Lane wrote: >> > Do we have a handle on how many buildfarm members this will break? >> >> I suppose we don't.  One way to find out would be

Re: [HACKERS] W3C Specs: Web SQL

2010-11-09 Thread Andrew Dunstan
On 11/09/2010 09:59 AM, Alvaro Herrera wrote: And this makes me think that SQLite is indeed the right tool for the job here, and not PostgreSQL. If someone intrudes, it's going to be in the same process running the web browser, not in some server running under another user identity in the mach

Re: [HACKERS] Avoid memory leaks during ANALYZE's compute_index_stats() ?

2010-11-09 Thread Jakub Ouhrabka
Hi Tom, thanks for brilliant analysis - now we know how to avoid the problem. As a side note: from the user's point of view it would be really nice to know that the error was caused by auto-ANALYZE - at least on 8.2 it's not that obvious from the server log. It was the first message with give

Re: [HACKERS] W3C Specs: Web SQL

2010-11-09 Thread Sam Mason
On Mon, Nov 08, 2010 at 12:55:22PM -0300, Alvaro Herrera wrote: > Excerpts from Charles Pritchard's message of sáb nov 06 23:20:13 -0300 2010: > > > Simple async sql sub-set (the spec in trouble): > > http://dev.w3.org/html5/webdatabase/ > > This is insane. This spec allows the server to run arb

[HACKERS] security hooks on object creation

2010-11-09 Thread KaiGai Kohei
The attached patch provides plugin modules a hook just after object creation time. In typical use cases, it enables to assign default security labels on object creation by the external security providers. As Robert suggested before, it provides a generic purpose main hook. It takes an enum of Obje

Re: [HACKERS] CLUSTER can change t_len

2010-11-09 Thread Heikki Linnakangas
On 09.11.2010 11:11, Itagaki Takahiro wrote: On Tue, Nov 9, 2010 at 12:44 PM, Jeff Davis wrote: See case below. After the item length gets changed, then when reading the tuple later you get a t_len that includes padding. We can easily find it with pageinspect: \i pageinspect.sql create table

Re: [HACKERS] proposal: plpgsql - iteration over fields of rec or row variable

2010-11-09 Thread Dmitriy Igrishin
Hey Pavel, 2010/11/9 Pavel Stehule > > a) transformation to common type > > > > + simple - it is one day job - function record_to_array, > > array_to_record, and fieldnames_to_array > > - lost of type info, hidden problems with IO cast - int a := 10.0/2.0 > > is a problem > > > > using a plperl

Re: [HACKERS] CLUSTER can change t_len

2010-11-09 Thread Itagaki Takahiro
On Tue, Nov 9, 2010 at 12:44 PM, Jeff Davis wrote: > See case below. After the item length gets changed, then when reading > the tuple later you get a t_len that includes padding. We can easily find it with pageinspect: \i pageinspect.sql create table foo(i int4); insert into foo values(1); SELE

Re: [HACKERS] Query Plan Columns

2010-11-09 Thread David E. Wheeler
On Nov 9, 2010, at 12:12 AM, Dimitri Fontaine wrote: > WITH plan AS ( > EXPLAIN (format table) SELECT * FROM bar > ) > INSERT INTO plan_audit > SELECT * FROM plan > WHERE actual_total_time > 12 * interval '100 ms'; Yeah, that would be nice, but my current implementation has a row for each node,

Re: [HACKERS] Query Plan Columns

2010-11-09 Thread Dimitri Fontaine
"David E. Wheeler" writes: > I'm wondering if there's any reason why we couldn't have EXPLAIN do something > like this itself in core: > > EXPLAIN (format table) SELECT * FROM bar; +1 from me here, as it happens "parsing" a table-like resultset is already implemented everywhere :) I'd add a