Re: [HACKERS] Performance Improvement by reducing WAL for Update Operation

2012-12-07 Thread Kyotaro HORIGUCHI
Hello, I looked into the patch and have some comments. From the restriction of the time for this rather big patch, please excuse that these comments are on a part of it. Others will follow in few days. heaptuple.c noncachegetattr(_with_len): - att_getlength should do strlen as worst

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Simon Riggs
On 7 December 2012 04:02, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Andrew Dunstan wrote: On 12/06/2012 09:23 PM, Bruce Momjian wrote: As soon as pg_dump stopped dumping the CREATE INDEX, pg_upgrade would stop creating creating it in the new cluster, and not transfer the index files.

Re: [HACKERS] [v9.3] writable foreign tables

2012-12-07 Thread Albe Laurenz
Kohei KaiGai wrote: The attached patch is revised version. One most difference from the previous version is, it constructed PoC features on Hanada-san's latest postgres-fdw.v5.patch. Yesh, it looks to me working fine on RDBMS backend also. Cool. Even though the filename of this patch

Re: [HACKERS] -DCLOBBER_CACHE_ALWAYS shows COPY FREEZE regression problem

2012-12-07 Thread Simon Riggs
On 7 December 2012 00:13, Tom Lane t...@sss.pgh.pa.us wrote: Andrew Dunstan and...@dunslane.net writes: On a new buildfarm member friarbird http://www.pgbuildfarm.org/cgi-bin/show_log.pl?nm=friarbirddt=2012-12-06%2020%3A55%3A31, configured with _DCLOBBER_CACHE_ALWAYS: BEGIN;

Re: [HACKERS] Setting visibility map in VACUUM's second phase

2012-12-07 Thread Pavan Deolasee
On Fri, Dec 7, 2012 at 9:21 AM, Pavan Deolasee pavan.deola...@gmail.com wrote: On Thu, Dec 6, 2012 at 11:31 PM, Tom Lane t...@sss.pgh.pa.us wrote: I think taking a second whack at setting the visibility bit is a fine idea, but let's drop all the rest of this premature optimization. Fair

Re: [HACKERS] Support for REINDEX CONCURRENTLY

2012-12-07 Thread Simon Riggs
On 7 December 2012 12:37, Michael Paquier michael.paqu...@gmail.com wrote: There are still two things that are missing in this patch, but I would like to have more feedback before moving forward: - REINDEX CONCURRENTLY needs tests in src/test/isolation Yes, it needs those - There is still a

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Alvaro Herrera
Bruce Momjian wrote: On Thu, Dec 6, 2012 at 09:23:14PM -0500, Bruce Momjian wrote: On Thu, Dec 6, 2012 at 09:10:21PM -0500, Tom Lane wrote: Bruce Momjian br...@momjian.us writes: On Thu, Dec 6, 2012 at 07:53:57PM -0500, Tom Lane wrote: Because CREATE INDEX CONCURRENTLY can't drop

Re: [HACKERS] Review: Patch to compute Max LSN of Data Pages

2012-12-07 Thread Muhammad Usama
Hi Amit On Mon, Dec 3, 2012 at 6:56 PM, Amit kapila amit.kap...@huawei.com wrote: I think we should expect provided path to be relative to current directory or may consider it to be relative to either one of Data or CWD. Because normally we expect path to be relative to CWD if some program

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Bruce Momjian
On Thu, Dec 6, 2012 at 10:27:21PM -0500, Stephen Frost wrote: * Bruce Momjian (br...@momjian.us) wrote: On Thu, Dec 6, 2012 at 09:45:11PM -0500, Stephen Frost wrote: Or preserve it as-is. I don't really like the 'make them fix it' option, as a user could run into that in the middle of

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Bruce Momjian
On Fri, Dec 7, 2012 at 09:16:56AM +, Simon Riggs wrote: On 7 December 2012 04:02, Alvaro Herrera alvhe...@2ndquadrant.com wrote: Andrew Dunstan wrote: On 12/06/2012 09:23 PM, Bruce Momjian wrote: As soon as pg_dump stopped dumping the CREATE INDEX, pg_upgrade would stop creating

Re: [HACKERS] Serious problem: media recovery fails after system or PostgreSQL crash

2012-12-07 Thread MauMau
Tomas wrote: So why don't you use an archive command that does not create such incomplete files? I mean something like this: archive_command = 'cp %p /arch/%f.tmp mv /arch/%f.tmp /arch/%f' Until the file is renamed, it's considered 'incomplete'. That is a good idea. This would surely

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Andres Freund
On 2012-12-07 09:21:58 -0500, Bruce Momjian wrote: On Thu, Dec 6, 2012 at 10:27:21PM -0500, Stephen Frost wrote: * Bruce Momjian (br...@momjian.us) wrote: On Thu, Dec 6, 2012 at 09:45:11PM -0500, Stephen Frost wrote: Or preserve it as-is. I don't really like the 'make them fix it'

Re: [HACKERS] WIP patch for hint bit i/o mitigation

2012-12-07 Thread Hari Babu
On Thu, Dec 6, 2012 at 8:52 PM, Merlin Moncure mmonc...@gmail.com wrote: Thanks for that -- that's fairly comprehensive I'd say. I'm quite interested in that benchmarking framework as well. Do you need help setting up the scripts? Presently I am testing with pgbench custom query option taking

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Bruce Momjian
On Fri, Dec 7, 2012 at 03:24:46PM +0100, Andres Freund wrote: On 2012-12-07 09:21:58 -0500, Bruce Momjian wrote: On Thu, Dec 6, 2012 at 10:27:21PM -0500, Stephen Frost wrote: * Bruce Momjian (br...@momjian.us) wrote: On Thu, Dec 6, 2012 at 09:45:11PM -0500, Stephen Frost wrote:

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Andres Freund
On 2012-12-07 09:27:39 -0500, Bruce Momjian wrote: On Fri, Dec 7, 2012 at 03:24:46PM +0100, Andres Freund wrote: On 2012-12-07 09:21:58 -0500, Bruce Momjian wrote: On Thu, Dec 6, 2012 at 10:27:21PM -0500, Stephen Frost wrote: * Bruce Momjian (br...@momjian.us) wrote: On Thu, Dec

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Bruce Momjian
On Fri, Dec 7, 2012 at 03:32:51PM +0100, Andres Freund wrote: Well, it is a CONCURRENT index creation, so locking would be minimal. I wouldn't call a ShareUpdateExclusiveLock minimal... Do we want pg_upgrade to be groveling over the lock view to look for locks? I don't think so.

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Tom Lane
Bruce Momjian br...@momjian.us writes: On Fri, Dec 7, 2012 at 03:32:51PM +0100, Andres Freund wrote: ISTM that anybody who does DDL during or after pg_upgrade --check deserves any pain. So throwing an error in both seems perfectly fine for me. I agree with Andres on this. I am just

Re: [HACKERS] Switching timeline over streaming replication

2012-12-07 Thread Heikki Linnakangas
On 06.12.2012 15:39, Amit Kapila wrote: On Thursday, December 06, 2012 12:53 AM Heikki Linnakangas wrote: On 05.12.2012 14:32, Amit Kapila wrote: On Tuesday, December 04, 2012 10:01 PM Heikki Linnakangas wrote: After some diversions to fix bugs and refactor existing code, I've committed a

Re: [HACKERS] Support for REINDEX CONCURRENTLY

2012-12-07 Thread Andres Freund
On 2012-12-07 21:37:06 +0900, Michael Paquier wrote: Hi all, Long time this thread has not been updated... Please find attached the version 3 of the patch for support of REINDEX CONCURRENTLY. The code has been realigned with master up to commit da07a1e (6th December). Here are the things

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Bruce Momjian
On Fri, Dec 7, 2012 at 10:29:22AM -0500, Tom Lane wrote: Bruce Momjian br...@momjian.us writes: On Fri, Dec 7, 2012 at 03:32:51PM +0100, Andres Freund wrote: ISTM that anybody who does DDL during or after pg_upgrade --check deserves any pain. So throwing an error in both seems

Re: [HACKERS] Commits 8de72b and 5457a1 (COPY FREEZE)

2012-12-07 Thread Simon Riggs
On 4 December 2012 20:38, Jeff Davis pg...@j-davis.com wrote: Even if it is, I would think that a sufficiently narrow case -- such as CTAS outside of a transaction block -- would be safe CTAS would be safe to do that with. CLUSTER and VACUUM FULL are already done. Outside of a transaction

Re: [HACKERS] ALTER TABLE ... NOREWRITE option

2012-12-07 Thread Robert Haas
On Thu, Dec 6, 2012 at 3:34 PM, Dimitri Fontaine dimi...@2ndquadrant.fr wrote: I think we need to be solving that problem when we implement new firing points for event trigger. The 'table_rewrite' event needs to fire at a time when the code can cope with it. That's the main difficulty in

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Tom Lane
Bruce Momjian br...@momjian.us writes: On Fri, Dec 7, 2012 at 10:29:22AM -0500, Tom Lane wrote: On balance I am coming around to support the just throw an error if there are any invalid indexes position. Adding extra complication in pg_dump and pg_upgrade to handle ignoring them doesn't seem

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Bruce Momjian
On Fri, Dec 7, 2012 at 11:46:51AM -0500, Tom Lane wrote: Bruce Momjian br...@momjian.us writes: On Fri, Dec 7, 2012 at 10:29:22AM -0500, Tom Lane wrote: On balance I am coming around to support the just throw an error if there are any invalid indexes position. Adding extra complication

Re: [HACKERS] Support for REINDEX CONCURRENTLY

2012-12-07 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes: On 7 December 2012 12:37, Michael Paquier michael.paqu...@gmail.com wrote: - There is still a problem with toast indexes. If the concurrent reindex of a toast index fails for a reason or another, pg_relation will finish with invalid toast index

Re: [HACKERS] Support for REINDEX CONCURRENTLY

2012-12-07 Thread Andres Freund
On 2012-12-07 12:01:52 -0500, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: On 7 December 2012 12:37, Michael Paquier michael.paqu...@gmail.com wrote: - There is still a problem with toast indexes. If the concurrent reindex of a toast index fails for a reason or another,

Re: [HACKERS] Fix for pg_upgrade status display

2012-12-07 Thread Bruce Momjian
On Wed, Dec 5, 2012 at 10:04:53PM -0500, Bruce Momjian wrote: Pg_upgrade displays file names during copy and database names during dump/restore. Andrew Dunstan identified three bugs: * long file names were being truncated to 60 _leading_ characters, which often do not change for long file

Re: [HACKERS] Proof of concept: auto updatable views [Review of Patch]

2012-12-07 Thread Tom Lane
[ finally getting back to this patch, after many distractions ] Dean Rasheed dean.a.rash...@gmail.com writes: On 8 November 2012 21:13, Tom Lane t...@sss.pgh.pa.us wrote: I think the most reasonable backwards-compatibility argument is that we shouldn't change the behavior if there are either

Re: [HACKERS] Support for REINDEX CONCURRENTLY

2012-12-07 Thread Simon Riggs
On 7 December 2012 17:19, Andres Freund and...@2ndquadrant.com wrote: On 2012-12-07 12:01:52 -0500, Tom Lane wrote: Simon Riggs si...@2ndquadrant.com writes: On 7 December 2012 12:37, Michael Paquier michael.paqu...@gmail.com wrote: - There is still a problem with toast indexes. If the

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Josh Berkus
Yes, I thought of not dumping it. The problem is that we don't delete the index when it fails, so I assumed we didn't want to lose the index creation information. I need to understand why we did that. Why do we have pg_dump dump the index then? Because pg_restore will recreate the index

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Andres Freund
On 2012-12-07 10:22:12 -0800, Josh Berkus wrote: Yes, I thought of not dumping it. The problem is that we don't delete the index when it fails, so I assumed we didn't want to lose the index creation information. I need to understand why we did that. Why do we have pg_dump dump the

Re: [HACKERS] Review of Row Level Security

2012-12-07 Thread Simon Riggs
On 5 December 2012 11:16, Kohei KaiGai kai...@kaigai.gr.jp wrote: * TRUNCATE works, and allows you to remove all rows of a table, even ones you can't see to run a DELETE on. Er... It was my oversight. My preference is to rewrite TRUNCATE command with DELETE statement in case when

Re: [HACKERS] Review of Row Level Security

2012-12-07 Thread Simon Riggs
On 5 December 2012 11:16, Kohei KaiGai kai...@kaigai.gr.jp wrote: Oracle defaults to putting VPD on all event types: INSERT, UPDATE, DELETE, SELECT. ISTM we should be doing the same, not just say we can add an INSERT trigger if you want. Adding a trigger just begs the question as to why we

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Josh Berkus
I wonder though if we shouldn't ignore !indislive indexes in pg_dump (and the respective bw-compat hack). Quite likely we shouldn't. However, that is why it wasn't considered a problem. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Andres Freund
On 2012-12-07 10:44:53 -0800, Josh Berkus wrote: I wonder though if we shouldn't ignore !indislive indexes in pg_dump (and the respective bw-compat hack). Obviously I wanted to ask whether we *should* ignore them in the future. Quite likely we shouldn't. However, that is why it wasn't

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: !indislive indexes are created during DROP INDEX CONCURRENTLY. Thats a different case than CREATE INDEX CONCURRENTLY. Accessing their definition is actually problematic because i can vanish while youre examing it which could cause errors both in the

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Bruce Momjian
On Fri, Dec 7, 2012 at 07:49:14PM +0100, Andres Freund wrote: On 2012-12-07 10:44:53 -0800, Josh Berkus wrote: I wonder though if we shouldn't ignore !indislive indexes in pg_dump (and the respective bw-compat hack). Obviously I wanted to ask whether we *should* ignore them in the

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Tom Lane
Bruce Momjian br...@momjian.us writes: On Fri, Dec 7, 2012 at 07:49:14PM +0100, Andres Freund wrote: !indislive indexes are created during DROP INDEX CONCURRENTLY. Thats a different case than CREATE INDEX CONCURRENTLY. Accessing their definition is actually problematic because i can vanish

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Andres Freund
On 2012-12-07 13:54:44 -0500, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: !indislive indexes are created during DROP INDEX CONCURRENTLY. Thats a different case than CREATE INDEX CONCURRENTLY. Accessing their definition is actually problematic because i can vanish while

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Andres Freund
On 2012-12-07 13:59:41 -0500, Tom Lane wrote: Bruce Momjian br...@momjian.us writes: On Fri, Dec 7, 2012 at 07:49:14PM +0100, Andres Freund wrote: !indislive indexes are created during DROP INDEX CONCURRENTLY. Thats a different case than CREATE INDEX CONCURRENTLY. Accessing their

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Bruce Momjian
On Fri, Dec 7, 2012 at 11:57:34AM -0500, Bruce Momjian wrote: On Fri, Dec 7, 2012 at 11:46:51AM -0500, Tom Lane wrote: Bruce Momjian br...@momjian.us writes: On Fri, Dec 7, 2012 at 10:29:22AM -0500, Tom Lane wrote: On balance I am coming around to support the just throw an error if

Re: [HACKERS] User control over psql error stream

2012-12-07 Thread Karl O. Pinc
On 11/21/2012 11:33:18 PM, Karl O. Pinc wrote: On 11/21/2012 01:41:56 PM, Peter Eisentraut wrote: On 11/15/12 3:53 PM, Karl O. Pinc wrote: This patch gives the end user control over psql's error stream. This allows a single psql session to use \o to send both errors and table output

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Tom Lane
Andres Freund and...@2ndquadrant.com writes: On 2012-12-07 13:59:41 -0500, Tom Lane wrote: indisvalid should be sufficient. If you try to test more than that you're going to make the code more version-specific, without actually buying much. Doesn't the check need to be at least indisvalid

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Bruce Momjian
On Fri, Dec 7, 2012 at 04:21:48PM -0500, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On 2012-12-07 13:59:41 -0500, Tom Lane wrote: indisvalid should be sufficient. If you try to test more than that you're going to make the code more version-specific, without actually

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Tom Lane
Bruce Momjian br...@momjian.us writes: On Fri, Dec 7, 2012 at 04:21:48PM -0500, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: Doesn't the check need to be at least indisvalid indisready? Given that 9.2 represents !indislive as indisvalid !indisready? Um, good point. It's

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Andres Freund
On 2012-12-07 16:30:36 -0500, Bruce Momjian wrote: On Fri, Dec 7, 2012 at 04:21:48PM -0500, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On 2012-12-07 13:59:41 -0500, Tom Lane wrote: indisvalid should be sufficient. If you try to test more than that you're going to

Re: [HACKERS] pg_upgrade problem with invalid indexes

2012-12-07 Thread Bruce Momjian
On Fri, Dec 7, 2012 at 10:38:39PM +0100, Andres Freund wrote: On 2012-12-07 16:30:36 -0500, Bruce Momjian wrote: On Fri, Dec 7, 2012 at 04:21:48PM -0500, Tom Lane wrote: Andres Freund and...@2ndquadrant.com writes: On 2012-12-07 13:59:41 -0500, Tom Lane wrote: indisvalid should be

Re: [HACKERS] ALTER TABLE ... NOREWRITE option

2012-12-07 Thread Dimitri Fontaine
Robert Haas robertmh...@gmail.com writes: is needed there, or we may need to move some things from the exec phase to the prep phase to make it all work out. I think it's totally doable, but it's not going to be a 50-line patch. If you want to work on it, please be my guest :) Regards, --

Re: [HACKERS] Commits 8de72b and 5457a1 (COPY FREEZE)

2012-12-07 Thread Jeff Davis
On Thu, 2012-12-06 at 22:34 -0500, Stephen Frost wrote: * Jeff Davis (pg...@j-davis.com) wrote: That is documented in the committed patch -- it's a trade, basically saying that you lose isolation but avoid extra writes. It seems reasonable that the user gets this behavior if specifically

Re: [HACKERS] Commits 8de72b and 5457a1 (COPY FREEZE)

2012-12-07 Thread Stephen Frost
Jeff, * Jeff Davis (pg...@j-davis.com) wrote: Most of your concerns seem to be related to freezing, and I'm mostly interested in HEAP_XMIN_COMMITTED optimizations. So I think we're talking past each other. My concern is about this patch/capability as a whole. I agree that the hint bit

Re: [HACKERS] Serious problem: media recovery fails after system or PostgreSQL crash

2012-12-07 Thread Jeff Janes
On Thu, Dec 6, 2012 at 3:52 PM, Tomas Vondra t...@fuzzy.cz wrote: Hi, On 6.12.2012 23:45, MauMau wrote: From: Tom Lane t...@sss.pgh.pa.us Well, that's unfortunate, but it's not clear that automatic recovery is possible. The only way out of it would be if an undamaged copy of the segment

Re: [HACKERS] Review: Patch to compute Max LSN of Data Pages

2012-12-07 Thread Amit kapila
Hi Muhammad, On Friday, December 07, 2012 7:43 PM Muhammad Usama wrote: Hi Amit On Mon, Dec 3, 2012 at 6:56 PM, Amit kapila amit.kap...@huawei.commailto:amit.kap...@huawei.com wrote: I think we should expect provided path to be relative to current directory or may consider it to be relative to

Re: [HACKERS] Review: Patch to compute Max LSN of Data Pages

2012-12-07 Thread Tom Lane
Amit kapila amit.kap...@huawei.com writes: On Friday, December 07, 2012 7:43 PM Muhammad Usama wrote: Although I am thinking why are you disallowing the absolute path of file. Any particular reason? The reason to disallow absolute path is that, we need to test on multiple platforms and to

Re: [HACKERS] Minor optimizations in lazy_scan_heap

2012-12-07 Thread Pavan Deolasee
On Tue, Dec 4, 2012 at 11:02 PM, Robert Haas robertmh...@gmail.com wrote: On Mon, Dec 3, 2012 at 1:23 AM, Pavan Deolasee pavan.deola...@gmail.com Comments ? Anyone thinks any/all of above is useful ? I doubt that any of these things make enough difference to be worth bothering with,