Re: [HACKERS] Freeze avoidance of very large table.

2015-04-23 Thread Sawada Masahiko
On Thu, Apr 23, 2015 at 3:24 AM, Robert Haas wrote: > On Wed, Apr 22, 2015 at 12:39 PM, Heikki Linnakangas wrote: >> The thing that made me nervous about that approach is that it made the LSN >> of each page critical information. If you somehow zeroed out the LSN, you >> could no longer tell whic

Re: [HACKERS] Freeze avoidance of very large table.

2015-04-23 Thread Heikki Linnakangas
On 04/22/2015 09:24 PM, Robert Haas wrote: I would feel safer if we added a completely new "epoch" counter to the page >header, instead of reusing LSNs. But as we all know, changing the page >format is a problem for in-place upgrade, and takes some space too. Yeah. We have a serious need to red

Re: [HACKERS] Code paths where LWLock should be released on failure

2015-04-23 Thread Andres Freund
Hi,m On 2015-04-23 13:51:57 +0900, Michael Paquier wrote: > After looking at bug #13128, I have been looking at the code around > LWLockAcquire/Release to see if there are similar issues elsewhere. > Here are my findings: Afaics all of these should actually be handled by the paths that release lo

Re: [HACKERS] PL/pgSQL, RAISE and error context

2015-04-23 Thread Marko Tiikkaja
On 4/2/15 9:37 AM, Pavel Stehule wrote: estate->err_text = stmt->elog_level == ERROR ? estate->err_text : raise_skip_msg ; Can we do this simple change? It will produce a stackinfo for exceptions and it will not to make mad developers by lot of useless content. I'm not sure everyone agrees w

Re: [HACKERS] INSERT ... ON CONFLICT IGNORE (and UPDATE) 3.0

2015-04-23 Thread Andres Freund
On 2015-04-22 15:23:16 -0700, Peter Geoghegan wrote: > On Tue, Apr 21, 2015 at 7:57 AM, Andres Freund wrote: > > * Iff we're going to have the XLOG_HEAP_AFFIRM record, I'd rather have > > that guide the logical decoding code. Seems slightly cleaner. > > I thought that you didn't think that woul

Re: [HACKERS] [BUGS] Failure to coerce unknown type to specific type

2015-04-23 Thread Kyotaro HORIGUCHI
Hello, I think this is a bug. The core of this problem is that coerce_type() fails for Var of type UNKNOWNOID. The comment for the function says that, > * The caller should already have determined that the coercion is possible; > * see can_coerce_type. But can_coerce_type() should say it's poss

Re: [HACKERS] Streaming replication and WAL archive interactions

2015-04-23 Thread Heikki Linnakangas
On 04/22/2015 11:58 PM, Robert Haas wrote: On Wed, Apr 22, 2015 at 3:34 PM, Heikki Linnakangas wrote: On 04/22/2015 10:21 PM, Robert Haas wrote: On Wed, Apr 22, 2015 at 3:01 PM, Heikki Linnakangas wrote: For example, imagine that perform point-in-time recovery to WAL position 0/1237E568, on

Re: [HACKERS] [BUGS] Failure to coerce unknown type to specific type

2015-04-23 Thread Kyotaro HORIGUCHI
Sorry, the patch had obvious bug.. -+ Int32GetDatum(inputTypeMod), ++ Int32GetDatum(targetTypeMod), regards, > Hello, I

Re: [HACKERS] Freeze avoidance of very large table.

2015-04-23 Thread Simon Riggs
On 21 April 2015 at 22:21, Robert Haas wrote: > I'm not saying those ideas don't have problems, because they do. But > I think they are worth further exploring. The main reason I gave up > on that is because Heikki was working on the XID-to-LSN mapping stuff. > That seemed like a better approa

Re: [HACKERS] [BUGS] Failure to coerce unknown type to specific type

2015-04-23 Thread Kyotaro HORIGUCHI
Now I found a comment at just where I patched, > * XXX if the typinput function is not immutable, we really ought to > * postpone evaluation of the function call until runtime. But there > * is no way to represent a typinput function call as an expression > * tree, because C-string values are

Re: [HACKERS] [BUGS] Failure to coerce unknown type to specific type

2015-04-23 Thread Kyotaro HORIGUCHI
Very sorry for the trash.. === Now I found a comment at just where I patched, > * XXX if the typinput function is not immutable, we really ought to > * postpone evaluation of the function call until runtime. But there > * is no way to represent a typinput function call as an expression > * tr

Re: [HACKERS] [BUGS] Failure to coerce unknown type to specific type

2015-04-23 Thread David G. Johnston
On Wednesday, April 22, 2015, Jeff Davis wrote: > On Wed, 2015-04-22 at 20:35 -0700, David G. Johnston wrote: > > > But the fact that column "b" has the data type "unknown" is only a > > warning - not an error. > > > I get an error: > > postgres=# SELECT ' '::text = 'a'; > ?column? > --

Re: [HACKERS] PL/pgSQL, RAISE and error context

2015-04-23 Thread Pavel Stehule
2015-04-23 9:53 GMT+02:00 Marko Tiikkaja : > On 4/2/15 9:37 AM, Pavel Stehule wrote: > >> estate->err_text = stmt->elog_level == ERROR ? estate->err_text : >>> raise_skip_msg ; >>> >> >> Can we do this simple change? It will produce a stackinfo for exceptions >> and it will not to make mad develo

Re: [HACKERS] [BUGS] Failure to coerce unknown type to specific type

2015-04-23 Thread David G. Johnston
On Thu, Apr 23, 2015 at 1:49 AM, David G. Johnston < david.g.johns...@gmail.com> wrote: > On Wednesday, April 22, 2015, Jeff Davis wrote: > >> On Wed, 2015-04-22 at 20:35 -0700, David G. Johnston wrote: >> > >> > ​My gut reaction is if you feel strongly enough to add some additional >> > document

Re: [HACKERS] [BUGS] Failure to coerce unknown type to specific type

2015-04-23 Thread David G. Johnston
On Thu, Apr 23, 2015 at 1:35 AM, Kyotaro HORIGUCHI < horiguchi.kyot...@lab.ntt.co.jp> wrote: > Very sorry for the trash.. > > === > Now I found a comment at just where I patched, > > > * XXX if the typinput function is not immutable, we really ought to > > * postpone evaluation of the function c

Re: [HACKERS] [BUGS] Failure to coerce unknown type to specific type

2015-04-23 Thread David G. Johnston
On Thu, Apr 23, 2015 at 1:07 AM, Kyotaro HORIGUCHI < horiguchi.kyot...@lab.ntt.co.jp> wrote: > Hello, I think this is a bug. > > The core of this problem is that coerce_type() fails for Var of > type UNKNOWNOID. > > The comment for the function says that, > > > * The caller should already have det

Re: [HACKERS] Auditing extension for PostgreSQL (Take 2)

2015-04-23 Thread Sawada Masahiko
On Mon, Apr 20, 2015 at 10:17 PM, David Steele wrote: > On 4/20/15 4:40 AM, Sawada Masahiko wrote: >> >> Thank you for updating the patch. >> >> One question about regarding since v7 (or later) patch is; >> What is the different between OBJECT logging and SESSION logging? > > In brief, session log

Re: [HACKERS] [BUGS] Failure to coerce unknown type to specific type

2015-04-23 Thread Kyotaro HORIGUCHI
Hello, > > Very sorry for the trash.. > > > > === > > Now I found a comment at just where I patched, > > > > > * XXX if the typinput function is not immutable, we really ought to > > > * postpone evaluation of the function call until runtime. But there > > > * is no way to represent a typinput

[HACKERS] anole - test case sha2 fails on all branches

2015-04-23 Thread Sandeep Thakkar
Hi, The test case sha2 in contrib pgcrypto module is failing with a diff on anole because the results file contains the additional lines as: -- + WARNING: detected write past chunk end in ExprContext 6021cbb0 -- Ex: The log for REL9_2_STABLE can be seen at http://buildfarm.postgresql.o

Re: [HACKERS] [BUGS] Failure to coerce unknown type to specific type

2015-04-23 Thread Kyotaro HORIGUCHI
Hello, > > Hello, I think this is a bug. > > > > The core of this problem is that coerce_type() fails for Var of > > type UNKNOWNOID. > > > > The comment for the function says that, > > > > > * The caller should already have determined that the coercion is > > possible; > > > * see can_coerce_type

Re: [HACKERS] INSERT ... ON CONFLICT IGNORE (and UPDATE) 3.0

2015-04-23 Thread Andres Freund
On 2015-04-22 16:40:07 -0700, Peter Geoghegan wrote: > On Wed, Apr 22, 2015 at 3:23 PM, Peter Geoghegan wrote: > > * We need to sort out those issues with the grammar, since that only > > really applies to the inference specification. Maybe the WHERE clause > > that the inference specification acc

[HACKERS] Split the 'Server Programming' chapter into two?

2015-04-23 Thread Andres Freund
Hi, While playing around with where exactly to put the replication origin/progress docs I once more noticed that the 'Server Programming' book is a mix of different topics. It currently contains: 35. Extending SQL 36. Triggers 37. Event Triggers 38. The Rule System 39. Procedu

Re: [HACKERS] Replication identifiers, take 4

2015-04-23 Thread Andres Freund
On 2015-03-24 22:22:29 +0100, Petr Jelinek wrote: > Perhaps we should have some Logical replication developer documentation > section and put all those three as subsections of that? So I just played around with this and it didn't find it worthwhile. Primarily because there's lots of uses of logica

Re: [HACKERS] INSERT ... ON CONFLICT IGNORE (and UPDATE) 3.0

2015-04-23 Thread Geoff Winkless
Apologies for butting in but can I (as a user) express a preference as a user against DO? Firstly, it looks horrible. And what's to stop me having "SELECT true AS do" in the where clause (as per your UPDATE objection)? Shouldn't UPDATE be a reserved keyword anyway? AIUI ANSI suggests so. http://

Re: [HACKERS] INSERT ... ON CONFLICT IGNORE (and UPDATE) 3.0

2015-04-23 Thread Petr Jelinek
On 23/04/15 14:34, Geoff Winkless wrote: Apologies for butting in but can I (as a user) express a preference as a user against DO? Firstly, it looks horrible. And what's to stop me having "SELECT true AS do" in the where clause (as per your UPDATE objection)? DO is already reserved keyword. T

Re: [HACKERS] INSERT ... ON CONFLICT IGNORE (and UPDATE) 3.0

2015-04-23 Thread Andres Freund
On April 23, 2015 3:34:07 PM GMT+03:00, Geoff Winkless wrote: >Apologies for butting in but can I (as a user) express a preference as >a >user against DO? Sure. If you propose an alternative ;) >Firstly, it looks horrible. And what's to stop me having "SELECT true >AS >do" in the where clause (

Re: [HACKERS] tablespaces inside $PGDATA considered harmful

2015-04-23 Thread Robert Haas
On Wed, Apr 22, 2015 at 10:41 PM, Bruce Momjian wrote: >> What is a real problem is that we don't block creating tablespaces >> anywhere at all, including in obviously problematic places like the >> transaction log directory: >> >> josh=# create tablespace tbl2 location '/home/josh/pg94/data/pg_xl

Re: [HACKERS] Allow SQL/plpgsql functions to accept record

2015-04-23 Thread Robert Haas
On Wed, Apr 22, 2015 at 6:12 PM, Jim Nasby wrote: > On 4/22/15 2:12 PM, Merlin Moncure wrote: >> That being said, I think json types with their associated API, given >> that they are core types, will ultimately handle these types of >> problems. That way, at least, we can avoid adding syntax and

Re: [HACKERS] Turning off HOT/Cleanup sometimes

2015-04-23 Thread Robert Haas
On Wed, Apr 22, 2015 at 5:17 PM, Bruce Momjian wrote: > On Wed, Apr 22, 2015 at 06:07:00PM -0300, Alvaro Herrera wrote: >> > Good point, but doesn't vacuum remove the need for pruning as it removes >> > all the old rows? >> >> Sure. The point, I think, is to make autovacuum runs of some sort that

Re: [HACKERS] Shouldn't CREATE TABLE LIKE copy the relhasoids property?

2015-04-23 Thread Robert Haas
On Wed, Apr 22, 2015 at 8:57 PM, Bruce Momjian wrote: > On Tue, Apr 21, 2015 at 05:36:41PM -0400, Robert Haas wrote: >> On Mon, Apr 20, 2015 at 5:41 PM, Bruce Momjian wrote: >> > On Mon, Apr 20, 2015 at 05:04:14PM -0400, Robert Haas wrote: >> >> On Mon, Apr 20, 2015 at 4:11 PM, Bruce Momjian wro

Re: [HACKERS] Allow SQL/plpgsql functions to accept record

2015-04-23 Thread Pavel Stehule
2015-04-23 15:19 GMT+02:00 Robert Haas : > On Wed, Apr 22, 2015 at 6:12 PM, Jim Nasby > wrote: > > On 4/22/15 2:12 PM, Merlin Moncure wrote: > >> That being said, I think json types with their associated API, given > >> that they are core types, will ultimately handle these types of > >> problems

Re: [HACKERS] INSERT ... ON CONFLICT IGNORE (and UPDATE) 3.0

2015-04-23 Thread Geoff Winkless
On 23 April 2015 at 13:51, Andres Freund wrote: > On April 23, 2015 3:34:07 PM GMT+03:00, Geoff Winkless < > pgsqlad...@geoff.dj> wrote: > ​ > ​>​ > And what's to stop me having "SELECT true > ​ > AS >do" in the where clause (as per your UPDATE objection)? > > A syntax error. DO is a reserved k

Re: [HACKERS] Freeze avoidance of very large table.

2015-04-23 Thread Robert Haas
On Thu, Apr 23, 2015 at 4:19 AM, Simon Riggs wrote: > We were talking about having an incremental backup map also. Which sounds a > lot like the freeze map. Yeah, possibly. I think we should try to set things up so that the backup map can be updated asynchronously by a background worker, so that

Re: [HACKERS] Freeze avoidance of very large table.

2015-04-23 Thread Robert Haas
On Wed, Apr 22, 2015 at 8:55 PM, Bruce Momjian wrote: > On Wed, Apr 22, 2015 at 06:36:23PM -0500, Jim Nasby wrote: >> On 4/22/15 6:12 PM, Bruce Momjian wrote: >> >My point is that for the life of 200M transactions, you would have the >> >overhead of an additional file per table in the file system,

Re: [HACKERS] PL/pgSQL, RAISE and error context

2015-04-23 Thread Robert Haas
On Thu, Apr 23, 2015 at 4:56 AM, Pavel Stehule wrote: > I don't see a contradiction. There is clean agreement, so ERROR level should > to show the context. NOTICE and WARNINGs doesn't need it - and there is a > backward compatibility and usability reasons don't do it. Whether notices and warnings

Re: [HACKERS] PL/pgSQL, RAISE and error context

2015-04-23 Thread Pavel Stehule
2015-04-23 15:47 GMT+02:00 Robert Haas : > On Thu, Apr 23, 2015 at 4:56 AM, Pavel Stehule > wrote: > > I don't see a contradiction. There is clean agreement, so ERROR level > should > > to show the context. NOTICE and WARNINGs doesn't need it - and there is a > > backward compatibility and usabil

Re: [HACKERS] INSERT ... ON CONFLICT IGNORE (and UPDATE) 3.0

2015-04-23 Thread Andres Freund
On 2015-04-23 14:34:02 +0100, Geoff Winkless wrote: > > A syntax error. DO is a reserved keyword. Update is just unreserved (and > > thus can be used as a column label). Ignore is unreserved with the patch > > and was unreserved before. We obviously can make both reserved, but of so > > we have to

Re: [HACKERS] Code paths where LWLock should be released on failure

2015-04-23 Thread Kevin Grittner
Michael Paquier wrote: > I have also been surprised by the inconsistencies particularly in > predicate.c or other places regarding LWLock releases. Sometimes > they are released on failure, sometimes not. Those are not needed for correctness; they are a micro-optimization for SerializableXactHas

Re: [HACKERS] anole - test case sha2 fails on all branches

2015-04-23 Thread Robert Haas
On Thu, Apr 23, 2015 at 6:13 AM, Sandeep Thakkar wrote: > The test case sha2 in contrib pgcrypto module is failing with a diff on > anole because the results file contains the additional lines as: > -- > > + WARNING: detected write past chunk end in ExprContext 6021cbb0 That's clearly a

Re: [HACKERS] Split the 'Server Programming' chapter into two?

2015-04-23 Thread Robert Haas
On Thu, Apr 23, 2015 at 8:21 AM, Andres Freund wrote: > While playing around with where exactly to put the replication > origin/progress docs I once more noticed that the 'Server Programming' > book is a mix of different topics. > > It currently contains: > 35. Extending SQL > 36. Triggers

Re: [HACKERS] PL/pgSQL, RAISE and error context

2015-04-23 Thread Robert Haas
On Thu, Apr 23, 2015 at 9:55 AM, Pavel Stehule wrote: >> On Thu, Apr 23, 2015 at 4:56 AM, Pavel Stehule >> wrote: >> > I don't see a contradiction. There is clean agreement, so ERROR level >> > should >> > to show the context. NOTICE and WARNINGs doesn't need it - and there is >> > a >> > backwar

Re: [HACKERS] Split the 'Server Programming' chapter into two?

2015-04-23 Thread Alvaro Herrera
Robert Haas wrote: > On Thu, Apr 23, 2015 at 8:21 AM, Andres Freund wrote: > > To me at least 44 - 47 don't really fit well to the rest. I think we > > either should invent a new category for them, or move them to > > 'Internals'. Maybe we could introduce 'Extending the Server' category > > for t

Re: [HACKERS] anole - test case sha2 fails on all branches

2015-04-23 Thread Alvaro Herrera
Robert Haas wrote: > There are lots of machines failing in pg_upgradeCheck, but I don't see > details of the failures in the logs. Yeah, I think the buildfarm script is failing to save the error log. Anyway AFAIR this is related to the move from contrib to src/bin; machines that have updated to b

Re: [HACKERS] anole - test case sha2 fails on all branches

2015-04-23 Thread Tom Lane
Robert Haas writes: > On Thu, Apr 23, 2015 at 6:13 AM, Sandeep Thakkar > wrote: >> The test case sha2 in contrib pgcrypto module is failing with a diff on >> anole because the results file contains the additional lines as: >> + WARNING: detected write past chunk end in ExprContext 6021cb

Re: [HACKERS] Turning off HOT/Cleanup sometimes

2015-04-23 Thread Jim Nasby
On 4/23/15 8:25 AM, Robert Haas wrote: Some users are partitioning tables just so that each partition can be autovac'd separately. That really shouldn't be required. Are they doing this for improved heap scan performance? Index scan performance? If the table wasn't partitioned, would they nee

Re: [HACKERS] Allow SQL/plpgsql functions to accept record

2015-04-23 Thread Jim Nasby
On 4/23/15 8:33 AM, Pavel Stehule wrote: I agree that that would be useful. I think the problem with an expression like rowvar.something is that PL/pgsql cannot infer the type of the result, and nothing else works without that. I doubt that it's practical to lift that restrictio

Re: [HACKERS] INSERT ... ON CONFLICT IGNORE (and UPDATE) 3.0

2015-04-23 Thread Geoff Winkless
On 23 April 2015 at 14:50, Andres Freund wrote: > > ​Maybe I'm misreading it, but isn't index_predicate meant to be inside > the > > brackets? > > > > > http://postgres-benchmarks.s3-website-us-east-1.amazonaws.com/on-conflict-docs/sql-insert.html > > That has changed since. ​Oh, helpful. :)​

Re: [HACKERS] Freeze avoidance of very large table.

2015-04-23 Thread Jim Nasby
On 4/23/15 2:42 AM, Heikki Linnakangas wrote: On 04/22/2015 09:24 PM, Robert Haas wrote: Yeah. We have a serious need to reduce the size of our on-disk format. On a TPC-C-like workload Jan Wieck recently tested, our data set was 34% larger than another database at the beginning of the test, an

Re: [HACKERS] anole - test case sha2 fails on all branches

2015-04-23 Thread Tom Lane
I wrote: > Given that anole is the only one reporting this, I'm not sure that we > should immediately blame Postgres itself. I have a vague recollection > that we've seen this symptom before and traced it to a bug in some > supporting library. Is anole using any particularly out-of-date versions

Re: [HACKERS] tablespaces inside $PGDATA considered harmful

2015-04-23 Thread Bruce Momjian
On Thu, Apr 23, 2015 at 09:13:52AM -0400, Robert Haas wrote: > On Wed, Apr 22, 2015 at 10:41 PM, Bruce Momjian wrote: > >> What is a real problem is that we don't block creating tablespaces > >> anywhere at all, including in obviously problematic places like the > >> transaction log directory: > >

Re: [HACKERS] INSERT ... ON CONFLICT IGNORE (and UPDATE) 3.0

2015-04-23 Thread Andres Freund
On 2015-04-23 15:52:40 +0100, Geoff Winkless wrote: > When I set out I was really only hoping to express a preference as a user; > on balance I would really rather not have DO IGNORE, if it were possible to > avoid, because it's really ugly, but DO UPDATE/DO NOTHING I could just > about cope with (

Re: [HACKERS] Add CINE for ALTER TABLE ... ADD COLUMN

2015-04-23 Thread Fabrízio de Royes Mello
On Wed, Apr 22, 2015 at 3:48 PM, Payal Singh wrote: > > The following review has been posted through the commitfest application: > make installcheck-world: tested, failed > Implements feature: not tested > Spec compliant: not tested > Documentation:not tested > > Seein

Re: [HACKERS] Freeze avoidance of very large table.

2015-04-23 Thread Jim Nasby
On 4/23/15 8:42 AM, Robert Haas wrote: On Thu, Apr 23, 2015 at 4:19 AM, Simon Riggs wrote: We were talking about having an incremental backup map also. Which sounds a lot like the freeze map. Yeah, possibly. I think we should try to set things up so that the backup map can be updated asynchr

Re: [HACKERS] tablespaces inside $PGDATA considered harmful

2015-04-23 Thread Andres Freund
On 2015-04-23 11:00:43 -0400, Bruce Momjian wrote: > On Thu, Apr 23, 2015 at 09:13:52AM -0400, Robert Haas wrote: > > I think this is a good thing to do, but I sure wish we could go > > further and block it completely. That may require more thought than > > we have time to put in at this stage of

Re: [HACKERS] Split the 'Server Programming' chapter into two?

2015-04-23 Thread Jim Nasby
On 4/23/15 9:23 AM, Alvaro Herrera wrote: Robert Haas wrote: On Thu, Apr 23, 2015 at 8:21 AM, Andres Freund wrote: To me at least 44 - 47 don't really fit well to the rest. I think we either should invent a new category for them, or move them to 'Internals'. Maybe we could introduce 'Extendi

Re: [HACKERS] tablespaces inside $PGDATA considered harmful

2015-04-23 Thread Jim Nasby
On 4/22/15 9:41 PM, Bruce Momjian wrote: The case this doesn't catch is referencing a symbolic link that points to the same directory. We can't make it an error so people can use pg_upgrade these setups. Couldn't we make it an ERROR unless IsBinaryUpgrade? -- Jim Nasby, Data Architect, Blue Tr

Re: [HACKERS] PL/pgSQL, RAISE and error context

2015-04-23 Thread Pavel Stehule
2015-04-23 16:12 GMT+02:00 Robert Haas : > On Thu, Apr 23, 2015 at 9:55 AM, Pavel Stehule > wrote: > >> On Thu, Apr 23, 2015 at 4:56 AM, Pavel Stehule > > >> wrote: > >> > I don't see a contradiction. There is clean agreement, so ERROR level > >> > should > >> > to show the context. NOTICE and W

Re: [HACKERS] Freeze avoidance of very large table.

2015-04-23 Thread Heikki Linnakangas
On 04/23/2015 05:52 PM, Jim Nasby wrote: On 4/23/15 2:42 AM, Heikki Linnakangas wrote: On 04/22/2015 09:24 PM, Robert Haas wrote: Yeah. We have a serious need to reduce the size of our on-disk format. On a TPC-C-like workload Jan Wieck recently tested, our data set was 34% larger than another

Re: [HACKERS] tablespaces inside $PGDATA considered harmful

2015-04-23 Thread Bruce Momjian
On Thu, Apr 23, 2015 at 05:05:14PM +0200, Andres Freund wrote: > On 2015-04-23 11:00:43 -0400, Bruce Momjian wrote: > > On Thu, Apr 23, 2015 at 09:13:52AM -0400, Robert Haas wrote: > > > I think this is a good thing to do, but I sure wish we could go > > > further and block it completely. That may

Re: [HACKERS] INSERT ... ON CONFLICT IGNORE (and UPDATE) 3.0

2015-04-23 Thread Bruce Momjian
On Thu, Apr 23, 2015 at 05:02:19PM +0200, Andres Freund wrote: > On 2015-04-23 15:52:40 +0100, Geoff Winkless wrote: > > When I set out I was really only hoping to express a preference as a user; > > on balance I would really rather not have DO IGNORE, if it were possible to > > avoid, because it's

Re: [HACKERS] Freeze avoidance of very large table.

2015-04-23 Thread Bruce Momjian
On Thu, Apr 23, 2015 at 10:42:59AM +0300, Heikki Linnakangas wrote: > On 04/22/2015 09:24 PM, Robert Haas wrote: > >>I would feel safer if we added a completely new "epoch" counter to the page > >>>header, instead of reusing LSNs. But as we all know, changing the page > >>>format is a problem for i

Re: [HACKERS] Freeze avoidance of very large table.

2015-04-23 Thread Petr Jelinek
On 23/04/15 17:24, Heikki Linnakangas wrote: On 04/23/2015 05:52 PM, Jim Nasby wrote: On 4/23/15 2:42 AM, Heikki Linnakangas wrote: On 04/22/2015 09:24 PM, Robert Haas wrote: Yeah. We have a serious need to reduce the size of our on-disk format. On a TPC-C-like workload Jan Wieck recently te

Re: [HACKERS] Freeze avoidance of very large table.

2015-04-23 Thread Bruce Momjian
On Thu, Apr 23, 2015 at 06:24:00PM +0300, Heikki Linnakangas wrote: > >I've often wondered if there was some way we could consolidate XMIN/XMAX > >from multiple tuples at the page level; that could be a big win for OLAP > >environments where most of your tuples belong to a pretty small range of > >

Re: [HACKERS] Freeze avoidance of very large table.

2015-04-23 Thread Bruce Momjian
On Thu, Apr 23, 2015 at 09:45:38AM -0400, Robert Haas wrote: > > Right. My point is that either you do X 2M times to maintain that fork > > and the overhead of the file existence, or you do one VACUUM FREEZE. I > > am saying that 2M is a large number and adding all those X's might > > exceed the

Re: [HACKERS] Freeze avoidance of very large table.

2015-04-23 Thread Heikki Linnakangas
On 04/23/2015 06:39 PM, Petr Jelinek wrote: On 23/04/15 17:24, Heikki Linnakangas wrote: On 04/23/2015 05:52 PM, Jim Nasby wrote: I've often wondered if there was some way we could consolidate XMIN/XMAX from multiple tuples at the page level; that could be a big win for OLAP environments where

Re: [HACKERS] Shouldn't CREATE TABLE LIKE copy the relhasoids property?

2015-04-23 Thread Bruce Momjian
On Thu, Apr 23, 2015 at 09:26:50AM -0400, Robert Haas wrote: > >> What I meant was - I didn't see an attachment on that message. > > > > I didn't attach it as people have told me they can just as easily see > > the patch via git, and since it was so similar, I didn't repost it. > > Should I have?

Re: [HACKERS] Freeze avoidance of very large table.

2015-04-23 Thread Heikki Linnakangas
On 04/23/2015 06:38 PM, Bruce Momjian wrote: On Thu, Apr 23, 2015 at 10:42:59AM +0300, Heikki Linnakangas wrote: On 04/22/2015 09:24 PM, Robert Haas wrote: I would feel safer if we added a completely new "epoch" counter to the page header, instead of reusing LSNs. But as we all know, changing

Re: [HACKERS] Freeze avoidance of very large table.

2015-04-23 Thread Bruce Momjian
On Thu, Apr 23, 2015 at 06:52:20PM +0300, Heikki Linnakangas wrote: > >Agreed. Are you saying we can't find a way to fit an 8-byte value into > >the existing page in a backward-compatible way? > > I'm sure we can find a way. We've discussed ways to handle page > format updates in pg_upgrade befor

Re: [HACKERS] tablespaces inside $PGDATA considered harmful

2015-04-23 Thread Andres Freund
On April 23, 2015 6:12:05 PM GMT+03:00, Jim Nasby wrote: >On 4/22/15 9:41 PM, Bruce Momjian wrote: >> The case this doesn't catch is referencing a >> symbolic link that points to the same directory. We can't make it an >> error so people can use pg_upgrade these setups. > >Couldn't we make it an

Re: [HACKERS] INSERT ... ON CONFLICT IGNORE (and UPDATE) 3.0

2015-04-23 Thread Heikki Linnakangas
On 04/20/2015 07:37 AM, Peter Geoghegan wrote: if (wco->commandType == CMD_INSERT) command = "INSERT-applicable "; else if (wco->commandType == CMD_UPDATE) command = "UPDATE-applicable

Re: [HACKERS] Freeze avoidance of very large table.

2015-04-23 Thread Petr Jelinek
On 23/04/15 17:45, Bruce Momjian wrote: On Thu, Apr 23, 2015 at 09:45:38AM -0400, Robert Haas wrote: Right. My point is that either you do X 2M times to maintain that fork and the overhead of the file existence, or you do one VACUUM FREEZE. I am saying that 2M is a large number and adding all

Re: [HACKERS] INSERT ... ON CONFLICT IGNORE (and UPDATE) 3.0

2015-04-23 Thread Peter Geoghegan
On Thu, Apr 23, 2015 at 9:02 AM, Heikki Linnakangas wrote: > That code in ExecWithCheckOptions is not translatable. See style guide: > http://www.postgresql.org/docs/devel/static/nls-programmer.html#NLS-GUIDELINES It's probably going to need to change when I rebase on top of Dean's/Stephen's work

Re: [HACKERS] Fwd: [GENERAL] 4B row limit for CLOB tables

2015-04-23 Thread Bruce Momjian
On Sun, Feb 1, 2015 at 03:54:03PM +0100, Álvaro Hernández Tortosa wrote: > > On 31/01/15 14:55, Roger Pack wrote: > >[...] > > > >Now, the reality is that GetNewOidWithIndex() is going to keep > >incrementing the global OID counter until it finds an OID that isn't in > >the toast table. That mean

Re: [HACKERS] anole - test case sha2 fails on all branches

2015-04-23 Thread Stephen Frost
Robert, On Thursday, April 23, 2015, Robert Haas wrote: > On Thu, Apr 23, 2015 at 6:13 AM, Sandeep Thakkar > > wrote: > > The test case sha2 in contrib pgcrypto module is failing with a diff on > > anole because the results file contains the additional lines as: > > -- > > > > + WARNING: detect

[HACKERS] Reducing tuple overhead

2015-04-23 Thread Andres Freund
Split into a new thread, the other one is already growing fast enough. This discussion started at http://archives.postgresql.org/message-id/55391469.5010506%40iki.fi On April 23, 2015 6:48:57 PM GMT+03:00, Heikki Linnakangas wrote: >Stop right there. You need to reserve enough space on the page

Re: [HACKERS] Freeze avoidance of very large table.

2015-04-23 Thread Jim Nasby
On 4/23/15 11:06 AM, Petr Jelinek wrote: On 23/04/15 17:45, Bruce Momjian wrote: On Thu, Apr 23, 2015 at 09:45:38AM -0400, Robert Haas wrote: Agreed, no extra file, and the same write volume as currently. It would also match pg_clog, which uses two bits per transaction --- maybe we can reuse so

Re: [HACKERS] Reducing tuple overhead

2015-04-23 Thread Jim Nasby
On 4/23/15 11:24 AM, Andres Freund wrote: I do wonder what, in realistic cases, is actually the bigger contributor to the overhead. The tuple header or the padding we liberally add in many cases... Assuming you're talking about padding between fields... Several years ago Enova paid Command Pro

Re: [HACKERS] Reducing tuple overhead

2015-04-23 Thread Petr Jelinek
On 23/04/15 18:24, Andres Freund wrote: Whether that's feasible complexity wise is debatable, but it's certainly possible. I do wonder what, in realistic cases, is actually the bigger contributor to the overhead. The tuple header or the padding we liberally add in many cases... The logical o

[HACKERS] adding more information about process(es) cpu and memory usage

2015-04-23 Thread Radovan Jablonovsky
During current encounters with amazon web services - RDS, the DBA does not have access to OS/linux shell of underlying instance. That render some postgresql monitoring technique of process CPU and memory usage, not useful. Even if the AWS provide internal tools/programming interface for monitoring,

Re: [HACKERS] Reducing tuple overhead

2015-04-23 Thread Joshua D. Drake
On 04/23/2015 09:42 AM, Jim Nasby wrote: On 4/23/15 11:24 AM, Andres Freund wrote: I do wonder what, in realistic cases, is actually the bigger contributor to the overhead. The tuple header or the padding we liberally add in many cases... Assuming you're talking about padding between fields.

Re: [HACKERS] adding more information about process(es) cpu and memory usage

2015-04-23 Thread Heikki Linnakangas
On 04/23/2015 08:00 PM, Radovan Jablonovsky wrote: During current encounters with amazon web services - RDS, the DBA does not have access to OS/linux shell of underlying instance. That render some postgresql monitoring technique of process CPU and memory usage, not useful. Even if the AWS provide

Re: [HACKERS] [BUGS] Failure to coerce unknown type to specific type

2015-04-23 Thread Jeff Davis
On Thu, Apr 23, 2015 at 1:49 AM, David G. Johnston wrote: > Reading and writing all this I'm convinced you have gotten the idea in your > mind an expectation of equivalency and consistency where there really is > little or none from an overall design perspective. And none insofar as > would merit

Re: [HACKERS] improving speed of make check-world

2015-04-23 Thread Jeff Janes
On Thu, Aug 14, 2014 at 10:45 PM, Peter Eisentraut wrote: > make check-world creates a temporary installation in every subdirectory > it runs a test in, which is stupid: it's very slow and uses a lot of > disk space. It's enough to do this once per run. That is the essence > of what I have impl

Re: [HACKERS] [BUGS] Failure to coerce unknown type to specific type

2015-04-23 Thread David G. Johnston
On Thursday, April 23, 2015, Jeff Davis wrote: > On Thu, Apr 23, 2015 at 1:49 AM, David G. Johnston > > wrote: > > Reading and writing all this I'm convinced you have gotten the idea in > your > > mind an expectation of equivalency and consistency where there really is > > little or none from an

Re: [HACKERS] Reducing tuple overhead

2015-04-23 Thread Alvaro Herrera
Thanks for posting this. Joshua D. Drake wrote: > First of all I should note that I discussed the approach mentioned above to > pgsql-hackers and got a very interesting comment from Tom Lane that adding > sorting info to Var and TargetEntry nodes was not a very good idea because > it'd break stor

[HACKERS] Moving ExecInsertIndexTuples and friends to new file

2015-04-23 Thread Heikki Linnakangas
While looking at Peter's INSERT ... ON CONFLICT patch, I started to feel that ExecInsertIndexTuples() and friends would deserve a file of their own, and not be buried in the middle of execUtils.c. I propose that we split execUtils.c into two, moving ExecOpenIndices(), ExecCloseIndices() ExecIns

[HACKERS] Improving vacuum/VM/etc

2015-04-23 Thread Jim Nasby
I mentioned this idea in the "other"[1] vacuum thread [2], but I think it got lost. Kevin Grittner pointed out that there's a potentially huge number of writes we incur over the life of a tuple [3]: (1) WAL log the insert. (2) Write the tuple. (3) Hint and rewrite the tuple. (4) WAL log the f

Re: [HACKERS] Moving ExecInsertIndexTuples and friends to new file

2015-04-23 Thread Peter Geoghegan
On Thu, Apr 23, 2015 at 12:05 PM, Heikki Linnakangas wrote: > While looking at Peter's INSERT ... ON CONFLICT patch, I started to feel > that ExecInsertIndexTuples() and friends would deserve a file of their own, > and not be buried in the middle of execUtils.c. I propose that we split > execUtils

[HACKERS] pg_dump: largeobject behavior issues (possible bug)

2015-04-23 Thread Joshua D. Drake
Hello, I have been working a problem with Andrew Gierth (sp?) in regards to pg_dump. Here is the basic breakdown: FreeBSD 10.1 PostgreSQL 9.3.6 64GB ~ memory 500GB database 228G of largeobjects (106M objects) The database dumps fine as long as we don't dump large objects. However, if we try

Re: [HACKERS] Reducing tuple overhead

2015-04-23 Thread Jim Nasby
On 4/23/15 11:45 AM, Petr Jelinek wrote: On 23/04/15 18:24, Andres Freund wrote: Whether that's feasible complexity wise is debatable, but it's certainly possible. I do wonder what, in realistic cases, is actually the bigger contributor to the overhead. The tuple header or the padding we liber

Re: [HACKERS] Turning off HOT/Cleanup sometimes

2015-04-23 Thread Robert Haas
On Thu, Apr 23, 2015 at 10:44 AM, Jim Nasby wrote: > On 4/23/15 8:25 AM, Robert Haas wrote: >> Some users are partitioning tables just so that each >> partition can be autovac'd separately. That really shouldn't be >> required. > > Are they doing this for improved heap scan performance? Index sca

Re: [HACKERS] Turning off HOT/Cleanup sometimes

2015-04-23 Thread Andres Freund
On 2015-04-23 15:40:36 -0400, Robert Haas wrote: > The issue is that you have to vacuum a table frequently enough to > avoid accumulating bloat. The frequency with which you need to vacuum > varies depending on the size of the table and how frequently it's > updated. However, a large, heavily-upd

Re: [HACKERS] INSERT ... ON CONFLICT IGNORE (and UPDATE) 3.0

2015-04-23 Thread Peter Geoghegan
On Thu, Apr 23, 2015 at 12:55 AM, Andres Freund wrote: > I think you misread my statement: I'm saying we don't need the new > argument anymore, even if we still do the super-deletion in > heap_delete(). Now that the speculative insertion will not be visible > (as in seen on a tuple they could dele

Re: [HACKERS] tablespaces inside $PGDATA considered harmful

2015-04-23 Thread Robert Haas
On Thu, Apr 23, 2015 at 11:00 AM, Bruce Momjian wrote: >> > I have developed the attached patch to warn about creating tablespaces >> > inside the data directory. The case this doesn't catch is referencing a >> > symbolic link that points to the same directory. We can't make it an >> > error so

Re: [HACKERS] [BUGS] Failure to coerce unknown type to specific type

2015-04-23 Thread Jim Nasby
On 4/23/15 5:07 AM, Kyotaro HORIGUCHI wrote: This is because parsing of UNION immediately converts constants of unknown type in the UNION's both arms to text so the top level select won't be bothered by this problem. But the problematic query doesn't have appropriate timing to do that until the f

Re: [HACKERS] tablespaces inside $PGDATA considered harmful

2015-04-23 Thread Andres Freund
On 2015-04-23 15:46:20 -0400, Robert Haas wrote: > Well, we've made backward-incompatible changes before. Not to this > specific thing, but in general. I don't think there's anything > preventing us from doing so here, except that we don't want to annoy > too many users. I think the number of us

Re: [HACKERS] INSERT ... ON CONFLICT IGNORE (and UPDATE) 3.0

2015-04-23 Thread Andres Freund
On 2015-04-23 12:45:59 -0700, Peter Geoghegan wrote: > On Thu, Apr 23, 2015 at 12:55 AM, Andres Freund wrote: > > I think you misread my statement: I'm saying we don't need the new > > argument anymore, even if we still do the super-deletion in > > heap_delete(). Now that the speculative insertion

Re: [HACKERS] pg_dump: largeobject behavior issues (possible bug)

2015-04-23 Thread Andrew Gierth
> "Joshua" == Joshua D Drake writes: Joshua> The database dumps fine as long as we don't dump large Joshua> objects. However, if we try to dump the large objects, FreeBSD Joshua> will kill pg_dump as it will consume all free memory and Joshua> swap. With Andrew's help we were able to dete

Re: [HACKERS] INSERT ... ON CONFLICT IGNORE (and UPDATE) 3.0

2015-04-23 Thread Peter Geoghegan
On Thu, Apr 23, 2015 at 12:53 PM, Andres Freund wrote: > Unconvinced. Not breaking an API has its worth. Yeah, and I acknowledge that - but it's not something that it's appropriate to encapsulate IMV. Let's just leave it to Heikki...I'd say he has the deciding vote, especially as the reviewer th

Re: [HACKERS] INSERT ... ON CONFLICT IGNORE (and UPDATE) 3.0

2015-04-23 Thread Heikki Linnakangas
On 04/23/2015 10:53 PM, Andres Freund wrote: On 2015-04-23 12:45:59 -0700, Peter Geoghegan wrote: On Thu, Apr 23, 2015 at 12:55 AM, Andres Freund wrote: I think you misread my statement: I'm saying we don't need the new argument anymore, even if we still do the super-deletion in heap_delete().

  1   2   >