[HACKERS] Reading deleted records - PageHeader v3

2010-02-05 Thread Jonathan Bond-Caron
Hi, So first I'm a pgsql hacker newbie and I've been reading up on the storage structure: http://www.postgresql.org/docs/8.2/interactive/storage-page-layout.html I'm trying to recover deleted records from a page file (postgresql 8.2) : i.e. base/dbId/20132 I am able to successfully

[HACKERS] Add on_plperl_init and on_plperlu_init to plperl UPDATE 3 [PATCH]

2010-02-05 Thread Tim Bunce
This is the third update to the fourth of the patches to be split out from the former 'plperl feature patch 1'. Changes in this patch: - Added plperl.on_plperl_init and plperl.on_plperlu_init GUCs Both are PGC_SUSET SPI functions are not available when the code is run. Errors are

[HACKERS] Failed assertion during recovery of partial WAL file

2010-02-05 Thread Martin Pihlak
Encountered the following FailedAssertion while testing database recovery (actually this would be HS) with partial WAL file: LOG: restored log file 00010003 from archive LOG: consistent recovery state reached at 0/3001EEC LOG: record with zero length at 0/3001EEC LOG: redo

[HACKERS] Confusion over Python drivers

2010-02-05 Thread Bruce Momjian
My son has brought to my attention that our current crop of Python client libraries is inadequate/confusing. I took a look myself, and asked on our IRC channel, and am now convinced this area needs attention. First, the sheer number of libraries is confusing. This is the page from the Postgres

Re: [HACKERS] Personal Copyright Notices

2010-02-05 Thread Greg Stark
On Fri, Feb 5, 2010 at 3:24 AM, Bruce Momjian br...@momjian.us wrote: Bruce Momjian wrote: The intagg copyright is on a _Makefile_:       # Makefile for integer aggregator       # Copyright (C) 2001 Digital Music Network.       # by Mark L. Woodward       # $PostgreSQL:

Re: [HACKERS] Personal Copyright Notices

2010-02-05 Thread Bruce Momjian
Greg Stark wrote: On Fri, Feb 5, 2010 at 3:24 AM, Bruce Momjian br...@momjian.us wrote: Bruce Momjian wrote: The intagg copyright is on a _Makefile_: ? ? ? # Makefile for integer aggregator ? ? ? # Copyright (C) 2001 Digital Music Network. ? ? ? # by Mark L. Woodward ? ? ? #

Re: [HACKERS] Confusion over Python drivers

2010-02-05 Thread Massa, Harald Armin
Bruce, http://wiki.postgresql.org/wiki/Python The first one listed, Psycopg, is noted as preferred libpq-based driver, but the license is GPL. Isn't that a problem for many client applications? The licence of psycopg2 is a little more complicated; the GPL in that summary just tries

Re: [HACKERS] Confusion over Python drivers

2010-02-05 Thread Bruce Momjian
Massa, Harald Armin wrote: Bruce, http://wiki.postgresql.org/wiki/Python The first one listed, Psycopg, is noted as preferred libpq-based driver, but the license is GPL. Isn't that a problem for many client applications? The licence of psycopg2 is a little more complicated;

Re: [HACKERS] Confusion over Python drivers

2010-02-05 Thread Peter Eisentraut
On fre, 2010-02-05 at 09:19 -0500, Bruce Momjian wrote: While I realize experienced people can easily navigate this confusion, I am concerned about new Postgres adopters being very confused by this and it is hurting our database adoption in general. What is really needed is for someone

Re: [HACKERS] Confusion over Python drivers

2010-02-05 Thread Tim Bunce
On Fri, Feb 05, 2010 at 09:19:26AM -0500, Bruce Momjian wrote: My son has brought to my attention that our current crop of Python client libraries is inadequate/confusing. I took a look myself, and asked on our IRC channel, and am now convinced this area needs attention.

Re: [HACKERS] Confusion over Python drivers

2010-02-05 Thread Bruce Momjian
Peter Eisentraut wrote: On fre, 2010-02-05 at 09:19 -0500, Bruce Momjian wrote: While I realize experienced people can easily navigate this confusion, I am concerned about new Postgres adopters being very confused by this and it is hurting our database adoption in general. What

Re: [HACKERS] Confusion over Python drivers

2010-02-05 Thread Bruce Momjian
Tim Bunce wrote: On Fri, Feb 05, 2010 at 09:19:26AM -0500, Bruce Momjian wrote: My son has brought to my attention that our current crop of Python client libraries is inadequate/confusing. I took a look myself, and asked on our IRC channel, and am now convinced this area needs attention.

Re: [HACKERS] [COMMITTERS] pgsql: Remove tabs in SGML.

2010-02-05 Thread Bruce Momjian
Peter Eisentraut wrote: On tis, 2010-01-26 at 10:20 -0800, David Fetter wrote: On Tue, Jan 26, 2010 at 02:21:29PM +, Bruce Momjian wrote: Log Message: --- Remove tabs in SGML. Can we see about making a commit hook for CVS that disallows \t in SGML files? The process

Re: [HACKERS] Confusion over Python drivers

2010-02-05 Thread Peter Eisentraut
On fre, 2010-02-05 at 14:45 +, Tim Bunce wrote: Does Perl have a similar mess? I don't think so. The primary database interface is DBI and as far as I can see there's only one DBI PostgreSQL driver: http://search.cpan.org/dist/DBD-Pg/ I think another difference is that the Perl DBI

Re: [HACKERS] Failed assertion during recovery of partial WAL file

2010-02-05 Thread Fujii Masao
On Fri, Feb 5, 2010 at 10:46 PM, Martin Pihlak martin.pih...@gmail.com wrote: Encountered the following FailedAssertion while testing database recovery (actually this would be HS) with partial WAL file: LOG:  restored log file 00010003 from archive LOG:  consistent recovery

Re: [HACKERS] synchronized snapshots

2010-02-05 Thread Markus Wanner
Hello Joachim, a little daughter eats lots of spare cycles - among other things. Sorry it took that long to review. On Fri, 8 Jan 2010 20:36:44 +0100, Joachim Wieland j...@mcknight.de wrote: The attached patch implements the idea of Heikki / Simon published in

Re: [HACKERS] Confusion over Python drivers

2010-02-05 Thread Bruce Momjian
Peter Eisentraut wrote: On fre, 2010-02-05 at 14:45 +, Tim Bunce wrote: Does Perl have a similar mess? I don't think so. The primary database interface is DBI and as far as I can see there's only one DBI PostgreSQL driver: http://search.cpan.org/dist/DBD-Pg/ I think another

Re: [HACKERS] MySQL-ism help patch for psql

2010-02-05 Thread Bruce Momjian
Josh Berkus wrote: My problem with this whole idea is that it seems to be very MySQL-specific. Why aren't we providing help for users migrating from Oracle, Sybase, Informix, Ingres, DB2, SQLServer and Firebird, to name but a few? And if we turn all those on by default, we'll have a

Re: [HACKERS] [CFReview] Red-Black Tree

2010-02-05 Thread Mark Cave-Ayland
Teodor Sigaev wrote: I would like to see point #2 of the following email addressed before commit. As things stand, it is not clear (at least to me) whether this is a win. Reimplementation of ginInsertRecordBA reduces difference of HEAD and HEAD+rbtree in regular case. Test suite is taken

Re: [HACKERS] Patch: psql \whoami option

2010-02-05 Thread Bruce Momjian
I have added this patch to the next commit-fest. Thanks: https://commitfest.postgresql.org/action/commitfest_view?id=6 --- David Christensen wrote: -hackers, In the spirit of small, but hopefully useful

Re: [HACKERS] remove contrib/xml2

2010-02-05 Thread Robert Haas
On Thu, Feb 4, 2010 at 10:51 PM, M Z jm80...@gmail.com wrote: I did some tests followed Robert's test cases on both postgresql 8.4.2-0ubu and 8.3.8-1, OS: Ubuntu Karmic. 1) 1st test case, it doesn't crash on 8.3.8 but crash on 8.4.2; Interesting. So, that's a regression of some kind. 2)

Re: [HACKERS] remove contrib/xml2

2010-02-05 Thread Robert Haas
On Wed, Feb 3, 2010 at 8:49 AM, Alvaro Herrera alvhe...@commandprompt.com wrote: Robert Haas escribió: On Mon, Feb 1, 2010 at 5:23 PM, Andrew Dunstan and...@dunslane.net wrote: Robert Haas wrote: (2) add a very, very large warning that this will crash if you do almost anything with it.

Re: [HACKERS] Confusion over Python drivers

2010-02-05 Thread Josh Berkus
I think another difference is that the Perl DBI interface is very rich, whereas the Python DB-API is quite minimal and almost forces people to write (incompatible) extensions. The DB-SIG at Python that ought to drive all this is also quite dead, possibly because everyone has moved on to

Re: [HACKERS] [CFReview] Red-Black Tree

2010-02-05 Thread Teodor Sigaev
That's all around 1% 0.7 patch = rbtest=# CREATE INDEX idin_rbtree_idx ON links2 USING gin (idin); CREATE INDEX Time: 1910741.352 ms rbtest=# CREATE INDEX idout_rbtree_idx ON links2 USING gin (idout); CREATE INDEX Time: 1647609.300 ms 0.11 patch == rbtest=# CREATE INDEX

Re: [HACKERS] [COMMITTERS] pgsql: Remove tabs in SGML.

2010-02-05 Thread Bruce Momjian
Alvaro Herrera wrote: Bruce Momjian wrote: Peter Eisentraut wrote: On tis, 2010-01-26 at 10:20 -0800, David Fetter wrote: On Tue, Jan 26, 2010 at 02:21:29PM +, Bruce Momjian wrote: Log Message: --- Remove tabs in SGML. Can we see about making a commit

Re: [HACKERS] [COMMITTERS] pgsql: Remove tabs in SGML.

2010-02-05 Thread Alvaro Herrera
Bruce Momjian wrote: Alvaro Herrera wrote: Maybe have the check-tabs rule as a dependency of the check, html and/or draft rules? + check-tabs: + ( ! grep ' ' $(wildcard $(srcdir)/*.sgml $(srcdir)/ref/*.sgml) ) || (echo Tabs appear in SGML files; exit 1) Uh, I

Re: [HACKERS] [COMMITTERS] pgsql: Remove tabs in SGML.

2010-02-05 Thread Alvaro Herrera
Bruce Momjian wrote: Peter Eisentraut wrote: On tis, 2010-01-26 at 10:20 -0800, David Fetter wrote: On Tue, Jan 26, 2010 at 02:21:29PM +, Bruce Momjian wrote: Log Message: --- Remove tabs in SGML. Can we see about making a commit hook for CVS that disallows \t

Re: [HACKERS] Confusion over Python drivers

2010-02-05 Thread Josh Berkus
The situation is unfortunate, but you might as well argue that too many Linux desktops or Linux distributions confuse new users and hurt adoption. These alternatives all exist for a reason, and it will be difficult to get some of them to abandon their work with the aim of reducing the

Re: [HACKERS] Confusion over Python drivers

2010-02-05 Thread Bruce Momjian
Josh Berkus wrote: The situation is unfortunate, but you might as well argue that too many Linux desktops or Linux distributions confuse new users and hurt adoption. These alternatives all exist for a reason, and it will be difficult to get some of them to abandon their work with the

Re: [HACKERS] plperl db access documentation enhancement

2010-02-05 Thread Bruce Momjian
Patch applied for 9.0. We don't normally backpatch such documentation improvements unless we receive multiple reports of confusion. --- Alexey Klyukin wrote: Hello, We were asked by Enova Financial to improve the

Re: [HACKERS] [COMMITTERS] pgsql: Remove tabs in SGML.

2010-02-05 Thread Bruce Momjian
Alvaro Herrera wrote: Bruce Momjian wrote: Alvaro Herrera wrote: Maybe have the check-tabs rule as a dependency of the check, html and/or draft rules? + check-tabs: + ( ! grep ' ' $(wildcard $(srcdir)/*.sgml $(srcdir)/ref/*.sgml) ) || (echo Tabs appear in

Re: [HACKERS] plperl db access documentation enhancement

2010-02-05 Thread Alvaro Herrera
Bruce Momjian wrote: Patch applied for 9.0. We don't normally backpatch such documentation improvements unless we receive multiple reports of confusion. I think that's a mistake in this case. The documentation wasn't confusing -- it was bogus. (Actually, the bug fixing is a smaller change

Re: [HACKERS] Confusion over Python drivers

2010-02-05 Thread Bruce Momjian
Marko Kreen wrote: Psycopg was the leader, especially in web-environments, but it has non-obvious license and with dead website it does not seem that attractive. Although it is well-maintained still. Best path forward would be to talk with Psycopg guys about license clarification/change.

Re: [HACKERS] Confusion over Python drivers

2010-02-05 Thread Greg Smith
Bruce Momjian wrote: While I realize experienced people can easily navigate this confusion... No, that's the worst part--the more you know and the deeper you dig into it, the more broken you realize the whole thing is. When one of the best drivers (in some respects) has a web page that

Re: [HACKERS] Strange heuristic in analyze.c

2010-02-05 Thread Bruce Momjian
Greg Stark wrote: So I never realized the consequences of this little heuristic in analyze.c in the handling of very low cardinality columns where we want to just capture the complete list of values in the mcv and throw away the histogram: else if (toowide_cnt == 0 nmultiple

Re: [HACKERS] Confusion over Python drivers

2010-02-05 Thread Bruce Momjian
Greg Smith wrote: Bruce Momjian wrote: While I realize experienced people can easily navigate this confusion... No, that's the worst part--the more you know and the deeper you dig into it, the more broken you realize the whole thing is. When one of the best drivers (in some respects)

Re: [HACKERS] Confusion over Python drivers

2010-02-05 Thread Jeff Davis
On Fri, 2010-02-05 at 09:38 -0500, Bruce Momjian wrote: Wow, that is super-confusing. Agreed. Standardization among licenses is useful, and I think it's important to have a driver with a license that people already understand. Regards, Jeff Davis -- Sent via pgsql-hackers mailing

[HACKERS] WIP patch for system-catalog vacuuming via a relation map

2010-02-05 Thread Tom Lane
Attached is the current state of my work on letting system catalogs be processed by new-style VACUUM FULL (a/k/a CLUSTER). I haven't done the WAL support nor worried about interlocking concurrent updates of relation map files, but it passes the regression tests and can do VACUUM FULL of every

Re: [HACKERS] WIP patch for system-catalog vacuuming via a relation map

2010-02-05 Thread Alvaro Herrera
Tom Lane wrote: Barring objections I'm going to press ahead with completing and committing this; then in a separate patch remove VACUUM FULL INPLACE. With the second patch, we will continue to support reading XVAC_MOVED_OUT and IN hint bits, but never set them, correct? -- Alvaro Herrera

Re: [HACKERS] WIP patch for system-catalog vacuuming via a relation map

2010-02-05 Thread Tom Lane
Alvaro Herrera alvhe...@commandprompt.com writes: Tom Lane wrote: Barring objections I'm going to press ahead with completing and committing this; then in a separate patch remove VACUUM FULL INPLACE. With the second patch, we will continue to support reading XVAC_MOVED_OUT and IN hint bits,

Re: [HACKERS] Confusion over Python drivers

2010-02-05 Thread Marko Kreen
On 2/5/10, Josh Berkus j...@agliodbs.com wrote: I think another difference is that the Perl DBI interface is very rich, whereas the Python DB-API is quite minimal and almost forces people to write (incompatible) extensions. The DB-SIG at Python that ought to drive all this is also

Re: [HACKERS] WIP patch for system-catalog vacuuming via a relation map

2010-02-05 Thread Josh Berkus
Barring objections I'm going to press ahead with completing and committing this; then in a separate patch remove VACUUM FULL INPLACE. Was it our determination that we could remove VFI if we eliminated the system catalogs? I'm fine with it, I just thought some people had a marginal use case

Re: [HACKERS] WIP patch for system-catalog vacuuming via a relation map

2010-02-05 Thread Tom Lane
Josh Berkus j...@agliodbs.com writes: Barring objections I'm going to press ahead with completing and committing this; then in a separate patch remove VACUUM FULL INPLACE. Was it our determination that we could remove VFI if we eliminated the system catalogs? I'm fine with it, I just thought

Re: [HACKERS] Add on_plperl_init and on_plperlu_init to plperl UPDATE 3 [PATCH]

2010-02-05 Thread Alex Hunsaker
On Fri, Feb 5, 2010 at 06:40, Tim Bunce tim.bu...@pobox.com wrote: This is the third update to the fourth of the patches to be split out from the former 'plperl feature patch 1'. Changes in this patch: - Added plperl.on_plperl_init and plperl.on_plperlu_init GUCs    Both are PGC_SUSET    

Re: [HACKERS] WIP patch for system-catalog vacuuming via a relation map

2010-02-05 Thread Bruce Momjian
Tom Lane wrote: Josh Berkus j...@agliodbs.com writes: Barring objections I'm going to press ahead with completing and committing this; then in a separate patch remove VACUUM FULL INPLACE. Was it our determination that we could remove VFI if we eliminated the system catalogs? I'm fine

Re: [HACKERS] plperl db access documentation enhancement

2010-02-05 Thread Bruce Momjian
Alvaro Herrera wrote: Bruce Momjian wrote: Patch applied for 9.0. We don't normally backpatch such documentation improvements unless we receive multiple reports of confusion. I think that's a mistake in this case. The documentation wasn't confusing -- it was bogus. (Actually, the

Re: [HACKERS] Database-Role settings behaviour and docs mismatch

2010-02-05 Thread Bruce Momjian
Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: On Mon, 2010-02-01 at 20:11 -0300, Alvaro Herrera wrote: It'd probably be worth changing the order of the ApplySetting calls so that it doesn't look suspicious. Just a comment would be enough I think Yeah. Changing the order

Re: [HACKERS] Mammoth in Core?

2010-02-05 Thread Bruce Momjian
Joshua D. Drake wrote: On Tue, 2010-01-19 at 21:55 +0100, Markus Wanner wrote: Hi, So, that's what I'd recommend the Mammoth developers to do as well: cherry-picking, sort of. Maybe that fulfills one or the other item on our wish-list (in one way or another)... I doubt we are

Re: [HACKERS] archive_timeout behavior for no activity

2010-02-05 Thread Bruce Momjian
Fujii Masao wrote: On Fri, Jan 15, 2010 at 12:50 AM, Bruce Momjian br...@momjian.us wrote: Looking at the archive_timeout documentation and CheckArchiveTimeout(), it appears we force a new xlog file and archive it even if no activity has been recorded in the xlog file. ?Is this correct?

Re: [HACKERS] archive_timeout behavior for no activity

2010-02-05 Thread Bruce Momjian
Kevin Grittner wrote: Bruce Momjian br...@momjian.us wrote: Looking at the archive_timeout documentation and CheckArchiveTimeout(), it appears we force a new xlog file and archive it even if no activity has been recorded in the xlog file. Is this correct? Should we document this or

Re: [HACKERS] Hot standby documentation

2010-02-05 Thread Bruce Momjian
Joshua Tolley wrote: -- Start of PGP signed section. Having concluded I really need to start playing with hot standby, I started looking for documentation on the subject. I found what I was looking for; I also found this page[1], which, it seems, ought to mention hot standby. Comments? [1]

Re: [HACKERS] Confusion over Python drivers

2010-02-05 Thread Josh Berkus
Imho a big problem is that it does way too much itself - i.e. it does not use things like PQExecParams but does escaping/parsing itself... Other people may think thats a good idea - I definitely do not think so. It also has issues with transaction control which cause idle transactions if the

[HACKERS] VAC FULL/CLUSTER on system catalogs is prone to deadlock

2010-02-05 Thread Tom Lane
I discovered $subject while continuing to test my patch. The problem occurs because many parts of the backend are in the habit of releasing lock on system catalogs as soon as they've updated whatever they wanted to update. This means that we can encounter INSERT_IN_PROGRESS or DELETE_IN_PROGRESS

Re: [HACKERS] VAC FULL/CLUSTER on system catalogs is prone to deadlock

2010-02-05 Thread Bruce Momjian
Tom Lane wrote: 2. In VAC FULL and CLUSTER, tell index rebuild not to wait around for INSERT_IN_PROGRESS or DELETE_IN_PROGRESS tuples. I believe that the price of this would be not re-verifying the integrity of unique indexes. Which is kind of annoying, but then again rechecking data

Re: [HACKERS] WIP patch for system-catalog vacuuming via a relation map

2010-02-05 Thread Tom Lane
Heikki Linnakangas heikki.linnakan...@enterprisedb.com writes: Tom Lane wrote: I thought the consensus was to remove it if possible. There may still be some marginal use cases, but they don't justify the work that'd be needed to make it play safely with HS; let alone fixing the other

Re: [HACKERS] Confusion over Python drivers

2010-02-05 Thread Andres Freund
On Friday 05 February 2010 21:34:53 Marko Kreen wrote: On 2/5/10, Josh Berkus j...@agliodbs.com wrote: I think another difference is that the Perl DBI interface is very rich, whereas the Python DB-API is quite minimal and almost forces people to write (incompatible) extensions. The

Re: [HACKERS] Confusion over Python drivers

2010-02-05 Thread Marko Kreen
On 2/5/10, Greg Smith g...@2ndquadrant.com wrote: The issues here have already been identified: the Perl DBI is an excellent spec, while the Python one is so weak everybody ends up needing their own extensions to it. And then portability *even among Python PostgreSQL drivers* goes out the

[HACKERS] Hot Standby and DROP DATABASE

2010-02-05 Thread Tatsuo Ishii
Hi, While testing Hot Standby, I have encountered strange behavior with DROP DATABASE command. 1) connect to test database at standby via psql 2) issue DROP DATABASE test command to primary 3) session #1 works fine 4) close session #1 5) test database dropped on standby Fromt the manual:

Re: [HACKERS] Confusion over Python drivers

2010-02-05 Thread James William Pye
On Feb 5, 2010, at 1:34 PM, Marko Kreen wrote: py-postgresql seems to be more serious, but as it's python3 only which makes it irrelevant today. Furthermore, if it did work on python2, it's *not* something that's going to appeal to mainstream users (Python heavy web frameworks) as it

Re: [HACKERS] Confusion over Python drivers

2010-02-05 Thread James William Pye
On Feb 5, 2010, at 11:34 AM, Josh Berkus wrote: For people who use Python a lot, could I have a list of the deficiencies in DBAPI? I've got my horse and lance ready. Given that SQLAlchemy isn't for everyone, of course ... it couldn't be, or Django would use it, no? Here are some to start

Re: [HACKERS] Confusion over Python drivers

2010-02-05 Thread James William Pye
On Feb 5, 2010, at 8:00 AM, Peter Eisentraut wrote: I think another difference is that the Perl DBI interface is very rich, whereas the Python DB-API is quite minimal and almost forces people to write (incompatible) extensions. Yep. The DB-SIG at Python that ought to drive all this is also

Re: [HACKERS] Re: Faster CREATE DATABASE by delaying fsync (was 8.4.1 ubuntu karmic slow createdb)

2010-02-05 Thread Greg Smith
Andres Freund wrote: On 02/03/10 14:42, Robert Haas wrote: Well, maybe we should start with a discussion of what kernel calls you're aware of on different platforms and then we could try to put an API around it. In linux there is sync_file_range. On newer Posixish systems one can emulate that

Re: [HACKERS] Reading deleted records - PageHeader v3

2010-02-05 Thread Tom Lane
Jonathan Bond-Caron jbo...@gmail.com writes: I think part of my problem is I haven't really understood what 'Then make sure you have the right alignment' means. My approach currently is: After reading HeapTupleHeaderData (23 bytes), I advance another 4 bytes (hoff) and try to read a 32 bit

Re: [HACKERS] [CFReview] Red-Black Tree

2010-02-05 Thread Robert Haas
2010/2/4 Teodor Sigaev teo...@sigaev.ru: Oleg's test (http://www.sai.msu.su/~megera/wiki/rbtree_test) are made with v0.10 which is differ from 0.11 only by comments around ginInsertRecordBA() That looks pretty good. I confess I don't fully understand why it works. If we're inserting a bunch