Re: [HACKERS] [BUG] Checkpointer on hot standby runs without looking checkpoint_segments

2012-04-27 Thread Kyotaro HORIGUCHI
Hello,

At Wed, 25 Apr 2012 02:31:24 +0900, Fujii Masao masao.fu...@gmail.com wrote 
in CAHGQGwE1OvB=HLcahLeL5oP66sxsfsGMgwU3MqAAwZ_Vr=x...@mail.gmail.com
  If we are allowed to be tolerant of the temporary lack of
  coherence in shared memory there, the spinlock could be removed.
  But the possibility to read garbage by using XLogCtl itself to
  access standbyMode does not seem to be tolerable. What do you
  think about that?
 
 I'm not sure if we really need to worry about that for such shared variable
 that doesn't change since it's been initialized at the start of recovery.
 Anyway, if we really need to worry about that, we need to protect the
 shared variable RecoveryTargetTLI and archiveCleanupCommand with
 the spinlock because they are in the same situation as standbyMode.

From I said that the former (spinlock) could be dropped, but the
latter (read as volatile) should be needed.

From the view of maintenancibility (suspicious-proof
expression?), it may be preferable that the manner to read shared
memory be uniform whole source code if no particular reasons.

Concerning this point, I think I will do 'volatization' and do
not spinlock and put comment instead.

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

== My e-mail address has been changed since Apr. 1, 2012.

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] [BUG] Checkpointer on hot standby runs without looking checkpoint_segments

2012-04-27 Thread Kyotaro HORIGUCHI
Thank you for sugestion.

 This still makes catching up in standby mode slower, as you get
 many more restartpoints. The reason for ignoring
 checkpoint_segments during recovery was to avoid that.

I may have a misunderstanding around there, or your intention.

I understand that standby creates no WAL archive, and can not
recover from WAL archive, and both master and standby keeps WAL
segment no longer than about them for about 2 * 1h, spans two
maximum checkpoint_timeout intervals and some more.

Could you please tell me whether the above is correct?


If you meant crash recovery with the word 'recovery', there's
WALs no more than for 2+ hours, far less than days, or weeks
long.

Otherwise, if you meant archive recovery, this patch does not
change the behavior of archive recovery as far as I
intended. This patch intended to change the behavior of standby
under WAL shipping.

If it is correct and the patch works correctly, your anxiety
below should disappear, I hope. And if not correct, I *MUST*
avoid such negative impacts on the functions out of the target -
governing checkpoint progress on standby server shipping WALs
from its master.

 Maybe it's still better than what we have currently, I'm not
 sure, but at least it needs to be discussed. Would be good to
 do some performance testing of recovery with various
 checkpoint_segments and _timeout settings, with and without
 this patch.

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

== My e-mail address has been changed since Apr. 1, 2012.

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] [BUG] Checkpointer on hot standby runs without looking checkpoint_segments

2012-04-27 Thread Kyotaro HORIGUCHI
Sorry for broken message.

 From I said that the former (spinlock) could be dropped, but the
 latter (read as volatile) should be needed.

I said that the former (spinlock) could be dropped from the view
of functionarity, but the latter (read as volatile) should be
needed.
 
 From the view of maintenancibility (suspicious-proof
 expression?), it may be preferable that the manner to read shared
 memory be uniform whole source code if no particular reasons.

regards,

-- 
Kyotaro Horiguchi
NTT Open Source Software Center

== My e-mail address has been changed since Apr. 1, 2012.

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Future In-Core Replication

2012-04-27 Thread Simon Riggs
On Fri, Apr 27, 2012 at 1:26 AM, Josh Berkus j...@agliodbs.com wrote:
 Simon,

 I'm beginning to work on advanced additions to in-core replication for
 PostgreSQL.
 ...
 Those are the basic requirements that I am trying to address. There
 are a great many important details, but the core of this is probably
 what I would call logical replication, that is shipping changes to
 other nodes in a way that does not tie us to the same physical
 representation that recovery/streaming replication does now. Of
 course, non-physical replication can take many forms.

 So, I'm a bit confused.  You talk about this as additions to in-core
 replication, but then you talk about implementing logical replication,
 which would NOT be an addition to Binary replication.  Can you explain
 what you mean?

The key point is that there is a specific objective of including
additional features in-core. That places some restrictions, but also
offers some opportunities. Tight integration allows performance
improvements, as well as ease of use etc..

I'm not sure what you mean by would not be an addition to binary
replication. Yes, for reasons most elegantly explained by Robert here
[http://rhaas.blogspot.co.uk/2011/02/case-for-logical-replication.html],
physical/binary replication puts too many restrictions on us and we
cannot solve all of the problems that way. I was unaware of Robert's
post, but it sets the scene clearly.

So the future of in-core replication, IMHO, is some form of
non-physical replication. There are various options there, but
anything that goes in would reuse significant parts of the existing
replication setup that already works so well. Put that another way:
the infrastructure for the secure and efficient transport of
replication messages is already in place. Reuse is also what makes
something useful be achievable in a reasonable timescale.

What we need to consider is the form of those new non-physicalWAL
messages, how they are built on the sender and how they are handled at
the receiving end.

What I'm hoping to do is to build a basic prototype of logical
replication using WAL translation, so we can inspect it to see what
the downsides are. It's an extremely non-trivial problem and so I
expect there to be mountains to climb. There are other routes to
logical replication, with messages marshalled in a similar way to
Slony/Londiste/Bucardo/Mammoth(?). So there are options, with
measurements to be made and discussions to be had.

It will take time for people to believe this is possible and longer to
analyse and agree the options.

-- 
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Future In-Core Replication

2012-04-27 Thread Simon Riggs
On Fri, Apr 27, 2012 at 12:40 AM, Robert Haas robertmh...@gmail.com wrote:
 On Thu, Apr 26, 2012 at 6:48 PM, Josh Berkus j...@agliodbs.com wrote:
 So the idea is that you'll present briefly your intentions for 9.3 at
 the developer meeting, and then have this in-depth afterwards?  Sounds
 great.

 I really, really do not want the developer meeting to turn into a
 series of presentations.  That's what we had last year, and it was
 boring and unproductive.  Furthermore, it excludes from the
 conversation everyone who doesn't fit into a room in Ottawa.  I think
 that plans should be presented here, on pgsql-hackers, and the
 developer meeting should be reserved for discussion of issues with
 which everyone who will be there is already familiar.  If a
 presentation is required, the developer meeting is the wrong forum,
 IMHO of course.

I agree.

Obviously, the word presentation is the issue here. If one person
speaks and everybody else is silent that is not a good use of
anybody's time.

On any topic, I expect the introducer of that topic to set the scene,
establish the major questions and encourage discussion.

I would follow that form myself: I want to hear the wisdom of others
and alter my plans accordingly.

-- 
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] patch submission: truncate trailing nulls from heap rows to reduce the size of the null bitmap

2012-04-27 Thread Simon Riggs
On Fri, Apr 27, 2012 at 1:51 AM, Josh Berkus j...@agliodbs.com wrote:

 Now, the other issue I'd be worried about for this optimization is what
 happens when the nulls become non-trailing?  For example, this pattern:

 1. Out of 700 columns, columns 301+ are all Null, so we map them away.
 2. User updates column 688 to non-null
 3. Suddenly we have a MUCH larger row which will no longer fit on the page.

 If your application had a lot of that kind of update pattern, I'd be
 concerned that this would be a deoptimzation.

Currently, we have a long row before and a long row after. Jamie's
proposals would give us a short row before and a long row after.

Since we don't ever update in place, we're much more likely to fit on
the same page with this optimisation than without it. I guess we can
check that with a performance test.

(Perhaps a more obvious optimisation would be to use a compressed NULL
bitmap. That would respond better in a wider range of use cases than
just truncation of trailing zeroes.)

-- 
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


[HACKERS] A problem with dump/restore of views containing whole row references

2012-04-27 Thread Abbas Butt
Hi,

This is the version I used to run the following commands

select version();

version

 PostgreSQL 9.2devel on x86_64-unknown-linux-gnu, compiled by gcc
(Ubuntu/Linaro 4.4.4-14ubuntu5) 4.4.5, 64-bit
(1 row)

Run these commands

  CREATE TABLE price (id INT PRIMARY KEY, active BOOLEAN NOT NULL, price
NUMERIC);
  insert into price values (1,false,42), (10,false,100), (11,true,17.99);
  create view v2 as select price.*::price from price;
  select * from v2;
  price
  --
   (1,f,42)
   (10,f,100)
   (11,t,17.99)
  (3 rows)

  \d+ v2;
View public.v2
   Column | Type  | Modifiers | Storage  | Description
  +---+---+--+-
   price  | price |   | extended |
  View definition:
   SELECT price AS price
 FROM price;

Note the output from the view, also note the Type in view defination.

Now take dump of this database.

./pg_dump --file=/home/user_name/d.sql --format=p --inserts -p  test

The dump file is attached with the mail. (d.sql)

Now lets restore this dump.

./createdb test2 -p 
./psql -p  -f /home/user_name/d.sql test2
./psql test2 -p 
psql (9.2devel)
Type help for help.

test2=# select * from v2;
 price
---
42
   100
 17.99
(3 rows)

test2=# \d+ v2
   View public.v2
 Column |  Type   | Modifiers | Storage | Description
+-+---+-+-
 price  | numeric |   | main|
View definition:
 SELECT price.price
   FROM price;

In the database test2 the view was not restored correctly.
The output of the view as well as the Type in its defination is wrong.

The cause of the problem is as follows

The notation relation.* represents a whole-row reference.
While parsing a whole-row reference is transformed into a Var with varno
set to the correct range table entry,
and varattno == 0 to signal that it references the whole tuple. (For
reference see comments of function makeWholeRowVar)
While deparsing we need to take care of this case.
The attached patch provides deparsing of a whole-row reference.
A whole row reference will be deparsed either into alias.*::relation or
relation.*::relation depending on alias

--
Abbas
EnterpriseDB Corporation
The Enterprise PostgreSQL Company
diff --git a/src/backend/utils/adt/ruleutils.c b/src/backend/utils/adt/ruleutils.c
index 3beed37..272d1a5 100644
--- a/src/backend/utils/adt/ruleutils.c
+++ b/src/backend/utils/adt/ruleutils.c
@@ -4007,7 +4007,47 @@ get_variable(Var *var, int levelsup, bool showstar, deparse_context *context)
 	}
 
 	if (attnum == InvalidAttrNumber)
+	{
 		attname = NULL;
+		/*
+		 * The notation relation.* represents a whole-row reference.
+		 * While parsing a whole-row reference is transformed into
+		 * a Var with varno set to the correct range table entry,
+		 * and varattno == 0 to signal that it references the whole tuple.
+		 * For reference see comments of function makeWholeRowVar
+		 * While deparsing we need to take care of this case
+		 * This block of code is deparsing a whole-row reference.
+		 * A whole row reference will be deparsed either into alias.*::relation
+		 * or relation.*::relation depending on alias
+		 * Explicit typecasting to relation is needed because
+		 * input of anonymous composite types is not implemented
+		 */
+
+		if (rte-relid != InvalidOid  refname  var-varattno == 0 
+			var-varno = 1  var-varno = list_length(dpns-rtable))
+		{
+			char *rel_name;
+			List *schemalist;
+
+			schemalist = NULL;
+			if (schemaname)
+list_make1(schemaname);
+
+			/* This relation name is required for explicit type casting later */
+			rel_name = generate_relation_name(rte-relid, schemalist);
+
+			/* Add name space qualification if required */
+			if (schemaname)
+appendStringInfo(buf, %s., quote_identifier(schemaname));
+			appendStringInfoString(buf, quote_identifier(refname));
+			appendStringInfoString(buf, .*::);
+			appendStringInfoString(buf, quote_identifier(rel_name));
+
+			pfree(rel_name);
+
+			return attname;
+		}
+	}
 	else
 		attname = get_rte_attribute_name(rte, attnum);
 
diff --git a/src/test/regress/expected/triggers.out b/src/test/regress/expected/triggers.out
index 0e7177e..cb39881 100644
--- a/src/test/regress/expected/triggers.out
+++ b/src/test/regress/expected/triggers.out
@@ -387,9 +387,9 @@ SELECT pg_get_triggerdef(oid, false) FROM pg_trigger WHERE tgrelid = 'main_table
 (1 row)
 
 SELECT pg_get_triggerdef(oid, true) FROM pg_trigger WHERE tgrelid = 'main_table'::regclass AND tgname = 'modified_any';
-  pg_get_triggerdef   
---
- CREATE 

[HACKERS] default_transaction_isolation = serializable causes crash under Hot Standby

2012-04-27 Thread Robert Haas
When I do this:

rhaas=# set default_transaction_isolation = 'serializable';
SET
rhaas=# begin;
BEGIN
rhaas=# select 1;

Then I get this:

TRAP: FailedAssertion(!(!RecoveryInProgress()), File: predicate.c,
Line: 1637)
LOG:  server process (PID 290) was terminated by signal 6: Abort trap

The root of the problem here seems to be that we're imagining that
it's possible to prevent serializable mode from being used under HS
from within the check function for the transaction_isolation GUC - see
check_XactIsoLevel.  However, because there's a second GUC
(default_transaction_isolation) that can also be used to change the
initial setting, it doesn't work.

I'm not exactly sure what the best way to fix this is.  I assume that
this problem also exists in 9.1, but I haven't checked.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] A problem with dump/restore of views containing whole row references

2012-04-27 Thread Andrew Dunstan



On 04/27/2012 08:25 AM, Abbas Butt wrote:


The notation relation.* represents a whole-row reference.
While parsing a whole-row reference is transformed into a Var with 
varno set to the correct range table entry,
and varattno == 0 to signal that it references the whole tuple. (For 
reference see comments of function makeWholeRowVar)

While deparsing we need to take care of this case.
The attached patch provides deparsing of a whole-row reference.
A whole row reference will be deparsed either into alias.*::relation 
or relation.*::relation depending on alias




I agree there's a bug, although it's easily worked around: in the case 
of your example:


   CREATE VIEW v2 AS
SELECT p AS price FROM price p;

would do the trick.

However, is this a change we really want to make?:

   pg_get_triggerdef
   
---
   - CREATE TRIGGER modified_any BEFORE UPDATE OF a ON main_table FOR EACH ROW 
WHEN (old.* IS DISTINCT FROM new.*) EXECUTE PROCEDURE 
trigger_func('modified_any')
   +
  pg_get_triggerdef
   
+--
   + CREATE TRIGGER modified_any BEFORE UPDATE OF a ON main_table FOR EACH ROW 
WHEN (old.*::main_table IS DISTINCT FROM new.*::main_table) EXECUTE PROCEDURE 
trigger_func('modified_any')


Maybe we need to be a bit more selective about when the cast is 
supplied. It's not adding any extra disambiguation (or clarity) here.


cheers

andrew


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] default_transaction_isolation = serializable causes crash under Hot Standby

2012-04-27 Thread Kevin Grittner
Robert Haas robertmh...@gmail.com wrote:
 
 When I do this:
 
 rhaas=# set default_transaction_isolation = 'serializable';
 SET
 rhaas=# begin;
 BEGIN
 rhaas=# select 1;
 
 Then I get this:
 
 TRAP: FailedAssertion(!(!RecoveryInProgress()), File:
 predicate.c, Line: 1637)
 LOG:  server process (PID 290) was terminated by signal 6: Abort
 trap
 
 The root of the problem here seems to be that we're imagining that
 it's possible to prevent serializable mode from being used under
 HS from within the check function for the transaction_isolation
 GUC - see check_XactIsoLevel.  However, because there's a second
 GUC (default_transaction_isolation) that can also be used to
 change the initial setting, it doesn't work.
 
 I'm not exactly sure what the best way to fix this is.  I assume
 that this problem also exists in 9.1, but I haven't checked.
 
My first thought was that if we can detect that we are in HS, we
should probably throw an ERROR on an attempt to set
default_transaction_isolation = 'serializable'.  But that raises the
question about what to do with environments where people want the
master to be running with that default (set from postgresql.conf) --
fail-over could be, well, interesting.  We haven't run into this
situation yet because we don't use our production postgresql.conf on
our hot standbys -- when we make a base backup we rename
postgresql.conf to postgresql.conf.production and copy in a special
configuration file.  For those who want the same configuration on
both, and who want to use serializable transactions on the master, I
don't see a really clean solution.  Does anyone else?

We didn't want to allow SERIALIZABLE to be set while still allowing
read-only anomalies like this (assuming T3 was on the HS):
 
http://wiki.postgresql.org/wiki/SSI#Deposit_Report
 
-Kevin

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Request to add options to tools/git_changelog

2012-04-27 Thread Bruce Momjian
On Thu, Apr 26, 2012 at 03:19:04PM -0400, Bruce Momjian wrote:
 On Thu, Apr 26, 2012 at 02:05:23PM -0400, Tom Lane wrote:
  Bruce Momjian br...@momjian.us writes:
   I agree adding rarely-used options to a tool doesn't make sense, but the
   question is what percentage of the git_changelog userbase am I?
  
  50% I think.  The only thing that's really concerning me here is that
  the reverse-sort option seems likely to be bug-inducing, and I really
  don't grasp that it has real value.  But whatever.
 
 Well, newest first would show this:
 
   add feature D to feature ABC
   add feature C to feature AB
   add feature B to feature A
   add feature A
 
 More logical (oldest-first) is:
 
   add feature A
   add feature B to feature A
   add feature C to feature AB
   add feature D to feature ABC
 
 Also consider that A is usually the big, clear commit message, and B,C,D
 are just minor adjustments with more brief commits, which might require
 adjusting the release note item for feature A.  When they are in
 newest-first order, that is much harder.

Oh, one more thing.  The contributor names appended to each release note
item usually has to be listed A,B,C,D because A is usually the most
significant contribution.

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] default_transaction_isolation = serializable causes crash under Hot Standby

2012-04-27 Thread Robert Haas
On Fri, Apr 27, 2012 at 10:21 AM, Kevin Grittner
kevin.gritt...@wicourts.gov wrote:
 My first thought was that if we can detect that we are in HS, we
 should probably throw an ERROR on an attempt to set
 default_transaction_isolation = 'serializable'.

I think that would result in the server failing to start.  We could
throw a warning there and give repeatable read.

Or, maybe there's a way to throw an error when serializable mode is
used rather than when it's requested.  So in the above example SELECT
1; would say, hey, somehow I ended up in serializable mode under HS,
abort, abort!

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] default_transaction_isolation = serializable causes crash under Hot Standby

2012-04-27 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes:
 Or, maybe there's a way to throw an error when serializable mode is
 used rather than when it's requested.

Couldn't we check and throw an error at the place in transaction startup
where default_transaction_isolation is copied to the active variable?

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] default_transaction_isolation = serializable causes crash under Hot Standby

2012-04-27 Thread Robert Haas
On Fri, Apr 27, 2012 at 11:02 AM, Tom Lane t...@sss.pgh.pa.us wrote:
 Robert Haas robertmh...@gmail.com writes:
 Or, maybe there's a way to throw an error when serializable mode is
 used rather than when it's requested.

 Couldn't we check and throw an error at the place in transaction startup
 where default_transaction_isolation is copied to the active variable?

Yeah, possibly.  Although the user might still use SET TRANSACTION
ISOLATION, so it would ideally be nice to postpone throwing the error
until after that opportunity is past.  Not sure if that's feasible,
though.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] default_transaction_isolation = serializable causes crash under Hot Standby

2012-04-27 Thread Kevin Grittner
Tom Lane t...@sss.pgh.pa.us wrote:
 Robert Haas robertmh...@gmail.com writes:
 Or, maybe there's a way to throw an error when serializable mode
 is used rather than when it's requested.
 
 Couldn't we check and throw an error at the place in transaction
 startup where default_transaction_isolation is copied to the
 active variable?
 
Wouldn't that leave users stuck if the postgresql.conf set the
default to serializable?  Nobody would be able to start a
transaction, even to change the default, would they?  If that's the
case, we might as well refuse to start.
 
Robert's suggestion might be the least of the various evils.
 
-Kevin

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


[HACKERS] enable_indexonly

2012-04-27 Thread Simon Riggs
I notice that there isn't a parameter called enable_indexonly (or similar).

ISTM that such a major new feature should be controlled by a planner
method parameter, just as all the existing planner methods are.

This will help us evaluate index only scans in production, and turn
them off if they have negative impacts.

-- 
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] enable_indexonly

2012-04-27 Thread Thom Brown
On 27 April 2012 16:08, Simon Riggs si...@2ndquadrant.com wrote:
 I notice that there isn't a parameter called enable_indexonly (or similar).

 ISTM that such a major new feature should be controlled by a planner
 method parameter, just as all the existing planner methods are.

 This will help us evaluate index only scans in production, and turn
 them off if they have negative impacts.

It already exists and is called enable_indexonlyscan.

-- 
Thom

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] enable_indexonly

2012-04-27 Thread Simon Riggs
On Fri, Apr 27, 2012 at 4:41 PM, Thom Brown t...@linux.com wrote:
 On 27 April 2012 16:08, Simon Riggs si...@2ndquadrant.com wrote:
 I notice that there isn't a parameter called enable_indexonly (or similar).

 ISTM that such a major new feature should be controlled by a planner
 method parameter, just as all the existing planner methods are.

 This will help us evaluate index only scans in production, and turn
 them off if they have negative impacts.

 It already exists and is called enable_indexonlyscan.

Hmm. Forgive me, I pressed the wrong button and looked at current docs
rather than dev docs.

(Easier when they used to look different...)

-- 
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] enable_indexonly

2012-04-27 Thread Andrew Dunstan



On 04/27/2012 11:45 AM, Simon Riggs wrote:

On Fri, Apr 27, 2012 at 4:41 PM, Thom Brownt...@linux.com  wrote:

On 27 April 2012 16:08, Simon Riggssi...@2ndquadrant.com  wrote:

I notice that there isn't a parameter called enable_indexonly (or similar).

ISTM that such a major new feature should be controlled by a planner
method parameter, just as all the existing planner methods are.

This will help us evaluate index only scans in production, and turn
them off if they have negative impacts.

It already exists and is called enable_indexonlyscan.

Hmm. Forgive me, I pressed the wrong button and looked at current docs
rather than dev docs.

(Easier when they used to look different...)


Maybe we should have the stylesheet watermark the dev docs pages.

cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] A problem with dump/restore of views containing whole row references

2012-04-27 Thread Abbas Butt
On Fri, Apr 27, 2012 at 6:25 PM, Andrew Dunstan and...@dunslane.net wrote:



 On 04/27/2012 08:25 AM, Abbas Butt wrote:


 The notation relation.* represents a whole-row reference.
 While parsing a whole-row reference is transformed into a Var with varno
 set to the correct range table entry,
 and varattno == 0 to signal that it references the whole tuple. (For
 reference see comments of function makeWholeRowVar)
 While deparsing we need to take care of this case.
 The attached patch provides deparsing of a whole-row reference.
 A whole row reference will be deparsed either into alias.*::relation or
 relation.*::relation depending on alias


 I agree there's a bug, although it's easily worked around: in the case of
 your example:

   CREATE VIEW v2 AS
SELECT p AS price FROM price p;

 would do the trick.

 However, is this a change we really want to make?:

   pg_get_triggerdef
   --**--**
 --**--**
 --**-
   - CREATE TRIGGER modified_any BEFORE UPDATE OF a ON main_table FOR EACH
 ROW WHEN (old.* IS DISTINCT FROM new.*) EXECUTE PROCEDURE
 trigger_func('modified_any')
   +
pg_get_triggerdef
   +-**--**
 --**--**
 --**--**---
   + CREATE TRIGGER modified_any BEFORE UPDATE OF a ON main_table FOR EACH
 ROW WHEN (old.*::main_table IS DISTINCT FROM new.*::main_table) EXECUTE
 PROCEDURE trigger_func('modified_any')


 Maybe we need to be a bit more selective about when the cast is supplied.
 It's not adding any extra disambiguation (or clarity) here.


I ran the regression and found that my patch is causing a diff in the
trigger test case, thats why I changed the expected output of the test case
accordingly. This is a side effect of the change I did to fix the bug.



 cheers

 andrew




-- 
--
Abbas
Architect
EnterpriseDB Corporation
The Enterprise PostgreSQL Company

Phone: 92-334-5100153

Website: www.enterprisedb.com
EnterpriseDB Blog: http://blogs.enterprisedb.com/
Follow us on Twitter: http://www.twitter.com/enterprisedb

This e-mail message (and any attachment) is intended for the use of
the individual or entity to whom it is addressed. This message
contains information from EnterpriseDB Corporation that may be
privileged, confidential, or exempt from disclosure under applicable
law. If you are not the intended recipient or authorized to receive
this for the intended recipient, any use, dissemination, distribution,
retention, archiving, or copying of this communication is strictly
prohibited. If you have received this e-mail in error, please notify
the sender immediately by reply e-mail and delete this message.


Re: [HACKERS] A problem with dump/restore of views containing whole row references

2012-04-27 Thread Andrew Dunstan



On 04/27/2012 12:02 PM, Abbas Butt wrote:




However, is this a change we really want to make?:

  pg_get_triggerdef
 
---

  - CREATE TRIGGER modified_any BEFORE UPDATE OF a ON main_table
FOR EACH ROW WHEN (old.* IS DISTINCT FROM new.*) EXECUTE PROCEDURE
trigger_func('modified_any')
  +  
   pg_get_triggerdef
 
+--

  + CREATE TRIGGER modified_any BEFORE UPDATE OF a ON main_table
FOR EACH ROW WHEN (old.*::main_table IS DISTINCT FROM
new.*::main_table) EXECUTE PROCEDURE trigger_func('modified_any')


Maybe we need to be a bit more selective about when the cast is
supplied. It's not adding any extra disambiguation (or clarity) here.



I ran the regression and found that my patch is causing a diff in the 
trigger test case, thats why I changed the expected output of the test 
case accordingly. This is a side effect of the change I did to fix the 
bug.




Right, what I'm asking is whether or not we actually want that side 
effect in all cases, and specifically in this case where it's clearly 
not necessary.


cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] default_transaction_isolation = serializable causes crash under Hot Standby

2012-04-27 Thread Tom Lane
Kevin Grittner kevin.gritt...@wicourts.gov writes:
 Tom Lane t...@sss.pgh.pa.us wrote:
 Couldn't we check and throw an error at the place in transaction
 startup where default_transaction_isolation is copied to the
 active variable?
 
 Wouldn't that leave users stuck if the postgresql.conf set the
 default to serializable?  Nobody would be able to start a
 transaction, even to change the default, would they?

I was assuming BEGIN TRANSACTION LEVEL ... would still work;
if not, it's a non-starter.  I haven't looked at the code to see
if the sequence of operations is amenable to that though.

 Robert's suggestion might be the least of the various evils.

Yeah, it would definitely be nicer if BEGIN; SET TRANSACTION LEVEL
would work too.  Maybe the place to put the check is where we
establish the transaction snapshot.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] enable_indexonly

2012-04-27 Thread Simon Riggs
On Fri, Apr 27, 2012 at 4:56 PM, Andrew Dunstan and...@dunslane.net wrote:

 (Easier when they used to look different...)


 Maybe we should have the stylesheet watermark the dev docs pages.

+1

Many users have 1 version in production. This would help

-- 
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] enable_indexonly

2012-04-27 Thread Magnus Hagander
On Fri, Apr 27, 2012 at 17:56, Andrew Dunstan and...@dunslane.net wrote:


 On 04/27/2012 11:45 AM, Simon Riggs wrote:

 On Fri, Apr 27, 2012 at 4:41 PM, Thom Brownt...@linux.com  wrote:

 On 27 April 2012 16:08, Simon Riggssi...@2ndquadrant.com  wrote:

 I notice that there isn't a parameter called enable_indexonly (or
 similar).

 ISTM that such a major new feature should be controlled by a planner
 method parameter, just as all the existing planner methods are.

 This will help us evaluate index only scans in production, and turn
 them off if they have negative impacts.

 It already exists and is called enable_indexonlyscan.

 Hmm. Forgive me, I pressed the wrong button and looked at current docs
 rather than dev docs.

 (Easier when they used to look different...)


 Maybe we should have the stylesheet watermark the dev docs pages.

Are you offering to write the patch? ;)

(if someone can provide a CSS that will look good and work cross
browser, I can do the website code side of things that enables it for
the devel docs)

-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] default_transaction_isolation = serializable causes crash under Hot Standby

2012-04-27 Thread Kevin Grittner
Tom Lane t...@sss.pgh.pa.us wrote:
 
 Yeah, it would definitely be nicer if BEGIN; SET TRANSACTION LEVEL
 would work too.  Maybe the place to put the check is where we
 establish the transaction snapshot.
 
That makes sense, and doesn't seem like it would be hard, from what
I recall of that code.  I know I've fallen down on a couple other
things which I meant to look at lately, but this one should be small
enough to fit in between the brickbats the non-PostgreSQL world has
been throwing at me lately.  I'll take a shot at it sometime today,
although it may be late.
 
-Kevin

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


[HACKERS] smart shutdown at end of transaction (was: Default mode for shutdown)

2012-04-27 Thread Robert Haas
On Wed, Dec 15, 2010 at 10:11 AM, Alvaro Herrera
alvhe...@commandprompt.com wrote:
 It occurs to me that we may need a new mode, which disconnects sessions
 that are not in a transaction (or as soon as they are) but leaves
 in-progress transactions alone; this could be the new default.  Of
 course, this is much more difficult to implement than the current modes.

This idea appeared to have some support.  I'd like to suggest that we
take this a step further.  Instead of adding a fourth mode, I'd like
to suggest that we redefine smart to have the behavior described
above.  This is based on the theory that (1) people who like smart
shutdown like it because it allows currently-running transactions to
complete without error, and will find it acceptable to have idle
transactions terminated immediately and other sessions terminated
after the command completes; and (2) people who dislike smart shutdown
(such as me) dislike it primarily because a completely idle session
that someone's forgotten to close can prevent shutdown indefinitely.
Either part of this theory could be wrong, of course, although I'm
pretty sure #2 holds for me personally at the least.

Patch is attached.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


smart-shutdown-at-eoxact.patch
Description: Binary data

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] smart shutdown at end of transaction (was: Default mode for shutdown)

2012-04-27 Thread Magnus Hagander
On Fri, Apr 27, 2012 at 19:42, Robert Haas robertmh...@gmail.com wrote:
 On Wed, Dec 15, 2010 at 10:11 AM, Alvaro Herrera
 alvhe...@commandprompt.com wrote:
 It occurs to me that we may need a new mode, which disconnects sessions
 that are not in a transaction (or as soon as they are) but leaves
 in-progress transactions alone; this could be the new default.  Of
 course, this is much more difficult to implement than the current modes.

 This idea appeared to have some support.  I'd like to suggest that we
 take this a step further.  Instead of adding a fourth mode, I'd like
 to suggest that we redefine smart to have the behavior described

+1762329!

 above.  This is based on the theory that (1) people who like smart
 shutdown like it because it allows currently-running transactions to
 complete without error, and will find it acceptable to have idle
 transactions terminated immediately and other sessions terminated

Uh, I don't think it's ok to terminate an idle transaction
immediately. An idle *session* is ok, though - maybe that's what you
mean?

Because every transaction that's *doing* multiple things will be idle
for milliseconds every now and then.

-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] smart shutdown at end of transaction (was: Default mode for shutdown)

2012-04-27 Thread Robert Haas
On Fri, Apr 27, 2012 at 1:46 PM, Magnus Hagander mag...@hagander.net wrote:
 On Fri, Apr 27, 2012 at 19:42, Robert Haas robertmh...@gmail.com wrote:
 On Wed, Dec 15, 2010 at 10:11 AM, Alvaro Herrera
 alvhe...@commandprompt.com wrote:
 It occurs to me that we may need a new mode, which disconnects sessions
 that are not in a transaction (or as soon as they are) but leaves
 in-progress transactions alone; this could be the new default.  Of
 course, this is much more difficult to implement than the current modes.

 This idea appeared to have some support.  I'd like to suggest that we
 take this a step further.  Instead of adding a fourth mode, I'd like
 to suggest that we redefine smart to have the behavior described

 +1762329!

Thanks.  :-)

 above.  This is based on the theory that (1) people who like smart
 shutdown like it because it allows currently-running transactions to
 complete without error, and will find it acceptable to have idle
 transactions terminated immediately and other sessions terminated

 Uh, I don't think it's ok to terminate an idle transaction
 immediately. An idle *session* is ok, though - maybe that's what you
 mean?

Yes, exactly.  What the patch does is arrange things so that, when
smart shutdown is requested, we terminate each session as soon as it
is both (1) idle and (2) not in a transaction.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Future In-Core Replication

2012-04-27 Thread Josh Berkus

 What I'm hoping to do is to build a basic prototype of logical
 replication using WAL translation, so we can inspect it to see what
 the downsides are. 

Sounds like Mammoth.  You take a look at that?

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] A problem with dump/restore of views containing whole row references

2012-04-27 Thread Tom Lane
Andrew Dunstan and...@dunslane.net writes:
 Right, what I'm asking is whether or not we actually want that side 
 effect in all cases, and specifically in this case where it's clearly 
 not necessary.

We could dodge that case by only changing the behavior when showstar is
false; there is no need to change it otherwise.  The patch has assorted
other bugs too, in particular its schema-name treatment seems completely
wrong (hint: RelationIsVisible is not the same as TypeIsVisible, and
it's at best shaky to assume that a relation's name is the same as its
rowtype's name anyway).

More generally, it seems rather inelegant to be forcibly adding a cast
when in most cases the existing notation is not wrong.  AFAICS the
plain relname notation is only ambiguous if there is a column of the
same name as the relation.  I wonder whether we should instead address
this by not letting the parser strip the no op cast in the first
place.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] smart shutdown at end of transaction (was: Default mode for shutdown)

2012-04-27 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes:
 On Wed, Dec 15, 2010 at 10:11 AM, Alvaro Herrera
 alvhe...@commandprompt.com wrote:
 It occurs to me that we may need a new mode, which disconnects sessions
 that are not in a transaction (or as soon as they are) but leaves
 in-progress transactions alone; this could be the new default.  Of
 course, this is much more difficult to implement than the current modes.

 This idea appeared to have some support.  I'd like to suggest that we
 take this a step further.  Instead of adding a fourth mode, I'd like
 to suggest that we redefine smart to have the behavior described
 above.

No, I'm not happy with that.  Smart shutdown is defined to not affect
current sessions.  I'm fine with having a fourth mode that acts as you
suggest (and, probably, even with making it the default); but not with
taking away a behavior that people may well be relying on.

 This is based on the theory that (1) people who like smart
 shutdown like it because it allows currently-running transactions to
 complete without error,

I think they like it because it allows currently-running *sessions*
to complete without error.  You have no real basis for asserting that
relocating that goalpost won't change the game.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Future In-Core Replication

2012-04-27 Thread Simon Riggs
On Fri, Apr 27, 2012 at 6:59 PM, Josh Berkus j...@agliodbs.com wrote:

 What I'm hoping to do is to build a basic prototype of logical
 replication using WAL translation, so we can inspect it to see what
 the downsides are.

 Sounds like Mammoth.  You take a look at that?

Well, they all sound similar. My info was that Mammoth was not WAL-based.

-- 
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] A problem with dump/restore of views containing whole row references

2012-04-27 Thread Abbas Butt
On Fri, Apr 27, 2012 at 11:21 PM, Tom Lane t...@sss.pgh.pa.us wrote:

 Andrew Dunstan and...@dunslane.net writes:
  Right, what I'm asking is whether or not we actually want that side
  effect in all cases, and specifically in this case where it's clearly
  not necessary.

 We could dodge that case by only changing the behavior when showstar is
 false; there is no need to change it otherwise.  The patch has assorted
 other bugs too, in particular its schema-name treatment seems completely
 wrong (hint: RelationIsVisible is not the same as TypeIsVisible, and
 it's at best shaky to assume that a relation's name is the same as its
 rowtype's name anyway).

 More generally, it seems rather inelegant to be forcibly adding a cast
 when in most cases the existing notation is not wrong.  AFAICS the
 plain relname notation is only ambiguous if there is a column of the
 same name as the relation.  I wonder whether we should instead address
 this by not letting the parser strip the no op cast in the first
 place.


You mean that the parser should not strip the no op cast in all cases or
in the case only when the parser somehow detects a column of the same name
as the relation?



regards, tom lane


--
Abbas
EnterpriseDB Corporation
The Enterprise PostgreSQL Company


Re: [HACKERS] smart shutdown at end of transaction (was: Default mode for shutdown)

2012-04-27 Thread Simon Riggs
On Fri, Apr 27, 2012 at 7:29 PM, Tom Lane t...@sss.pgh.pa.us wrote:
 Robert Haas robertmh...@gmail.com writes:
 On Wed, Dec 15, 2010 at 10:11 AM, Alvaro Herrera
 alvhe...@commandprompt.com wrote:
 It occurs to me that we may need a new mode, which disconnects sessions
 that are not in a transaction (or as soon as they are) but leaves
 in-progress transactions alone; this could be the new default.  Of
 course, this is much more difficult to implement than the current modes.

 This idea appeared to have some support.  I'd like to suggest that we
 take this a step further.  Instead of adding a fourth mode, I'd like
 to suggest that we redefine smart to have the behavior described
 above.

 No, I'm not happy with that.  Smart shutdown is defined to not affect
 current sessions.  I'm fine with having a fourth mode that acts as you
 suggest (and, probably, even with making it the default); but not with
 taking away a behavior that people may well be relying on.

Agreed, but not sure what to call the new mode: smarter?

-- 
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] smart shutdown at end of transaction (was: Default mode for shutdown)

2012-04-27 Thread Andres Freund
Hi,

On Friday, April 27, 2012 07:42:59 PM Robert Haas wrote:
 On Wed, Dec 15, 2010 at 10:11 AM, Alvaro Herrera
 alvhe...@commandprompt.com wrote:
  It occurs to me that we may need a new mode, which disconnects sessions
  that are not in a transaction (or as soon as they are) but leaves
  in-progress transactions alone; this could be the new default.  Of
  course, this is much more difficult to implement than the current modes.
 
 This idea appeared to have some support.  I'd like to suggest that we
 take this a step further.  Instead of adding a fourth mode, I'd like
 to suggest that we redefine smart to have the behavior described
 above.  This is based on the theory that (1) people who like smart
 shutdown like it because it allows currently-running transactions to
 complete without error, and will find it acceptable to have idle
 transactions terminated immediately and other sessions terminated
 after the command completes; and (2) people who dislike smart shutdown
 (such as me) dislike it primarily because a completely idle session
 that someone's forgotten to close can prevent shutdown indefinitely.
 Either part of this theory could be wrong, of course, although I'm
 pretty sure #2 holds for me personally at the least.
I think the current smart mode is rather useful. There is quite some stuff 
that you cannot do inside a transaction - or it doesn't make sense - which 
still needs to shutdown gracefully. E.g. transaction managers.

Andres

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] smart shutdown at end of transaction (was: Default mode for shutdown)

2012-04-27 Thread Andres Freund
On Friday, April 27, 2012 08:38:10 PM Simon Riggs wrote:
 On Fri, Apr 27, 2012 at 7:29 PM, Tom Lane t...@sss.pgh.pa.us wrote:
  Robert Haas robertmh...@gmail.com writes:
  On Wed, Dec 15, 2010 at 10:11 AM, Alvaro Herrera
  
  alvhe...@commandprompt.com wrote:
  It occurs to me that we may need a new mode, which disconnects sessions
  that are not in a transaction (or as soon as they are) but leaves
  in-progress transactions alone; this could be the new default.  Of
  course, this is much more difficult to implement than the current
  modes.
  
  This idea appeared to have some support.  I'd like to suggest that we
  take this a step further.  Instead of adding a fourth mode, I'd like
  to suggest that we redefine smart to have the behavior described
  above.
  
  No, I'm not happy with that.  Smart shutdown is defined to not affect
  current sessions.  I'm fine with having a fourth mode that acts as you
  suggest (and, probably, even with making it the default); but not with
  taking away a behavior that people may well be relying on.
 
 Agreed, but not sure what to call the new mode: smarter?
graceful?

Andres

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] smart shutdown at end of transaction (was: Default mode for shutdown)

2012-04-27 Thread Tom Lane
Simon Riggs si...@2ndquadrant.com writes:
 On Fri, Apr 27, 2012 at 7:29 PM, Tom Lane t...@sss.pgh.pa.us wrote:
 No, I'm not happy with that.  Smart shutdown is defined to not affect
 current sessions.  I'm fine with having a fourth mode that acts as you
 suggest (and, probably, even with making it the default); but not with
 taking away a behavior that people may well be relying on.

 Agreed, but not sure what to call the new mode: smarter?

I'm not necessarily opposed to commandeering the name smart for the
new behavior, so that what we have to find a name for is the old smart
behavior.  How about

slow- allow existing sessions to finish (old smart)
smart   - allow existing transactions to finish (new)
fast- kill active queries
immediate - unclean shutdown

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] smart shutdown at end of transaction (was: Default mode for shutdown)

2012-04-27 Thread Magnus Hagander
On Fri, Apr 27, 2012 at 20:48, Tom Lane t...@sss.pgh.pa.us wrote:
 Simon Riggs si...@2ndquadrant.com writes:
 On Fri, Apr 27, 2012 at 7:29 PM, Tom Lane t...@sss.pgh.pa.us wrote:
 No, I'm not happy with that.  Smart shutdown is defined to not affect
 current sessions.  I'm fine with having a fourth mode that acts as you
 suggest (and, probably, even with making it the default); but not with
 taking away a behavior that people may well be relying on.

 Agreed, but not sure what to call the new mode: smarter?

 I'm not necessarily opposed to commandeering the name smart for the
 new behavior, so that what we have to find a name for is the old smart
 behavior.  How about

        slow    - allow existing sessions to finish (old smart)

How about wait instead of slow?

        smart   - allow existing transactions to finish (new)

and still default, right?

        fast    - kill active queries
        immediate - unclean shutdown



-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] smart shutdown at end of transaction (was: Default mode for shutdown)

2012-04-27 Thread Tom Lane
Magnus Hagander mag...@hagander.net writes:
 On Fri, Apr 27, 2012 at 20:48, Tom Lane t...@sss.pgh.pa.us wrote:
 I'm not necessarily opposed to commandeering the name smart for the
 new behavior, so that what we have to find a name for is the old smart
 behavior.  How about
 
slow- allow existing sessions to finish (old smart)

 How about wait instead of slow?

I kinda liked slow vs fast, but if you think that's too cute ...
(wait doesn't seem very good, though, since all these except immediate
are waiting, just for different things.)

smart   - allow existing transactions to finish (new)

 and still default, right?

Right.

fast- kill active queries
immediate - unclean shutdown

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] smart shutdown at end of transaction (was: Default mode for shutdown)

2012-04-27 Thread Robert Haas
On Fri, Apr 27, 2012 at 2:29 PM, Tom Lane t...@sss.pgh.pa.us wrote:
 This idea appeared to have some support.  I'd like to suggest that we
 take this a step further.  Instead of adding a fourth mode, I'd like
 to suggest that we redefine smart to have the behavior described
 above.

 No, I'm not happy with that.  Smart shutdown is defined to not affect
 current sessions.  I'm fine with having a fourth mode that acts as you
 suggest (and, probably, even with making it the default); but not with
 taking away a behavior that people may well be relying on.

I think there is no point at all in having a discussion about this
unless we can first agree that the overwhelming majority of people who
have commented on this issue on this list are unhappy with the current
default behavior.  If we are not going to change the default behavior,
then there is zero point in talking about this.  So I am nervous about
your use of the word probably, because I do not want to do a bunch
of work on this just to add a fourth shutdown mode without changing
the default to something that does not suck.  I would like to get some
agreement that we ARE going to change the default behavior, and then
we can argue about what exactly we're going to change it to.

 This is based on the theory that (1) people who like smart
 shutdown like it because it allows currently-running transactions to
 complete without error,

 I think they like it because it allows currently-running *sessions*
 to complete without error.  You have no real basis for asserting that
 relocating that goalpost won't change the game.

I'm not asserting that.  What I am asserting is that the vast majority
of users will consider the revised game to be more fun than the
original one.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] smart shutdown at end of transaction (was: Default mode for shutdown)

2012-04-27 Thread Robert Haas
On Fri, Apr 27, 2012 at 2:48 PM, Tom Lane t...@sss.pgh.pa.us wrote:
 I'm not necessarily opposed to commandeering the name smart for the
 new behavior, so that what we have to find a name for is the old smart
 behavior.  How about

        slow    - allow existing sessions to finish (old smart)
        smart   - allow existing transactions to finish (new)
        fast    - kill active queries
        immediate - unclean shutdown

I could live with that.  Really, I'd like to have fast just be the
default.  But the above compromise would still be a big improvement
over what we have now, assuming the new smart becomes the default.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] psql omits row count under \x auto

2012-04-27 Thread Noah Misch
On Thu, Apr 26, 2012 at 09:25:25PM +0300, Peter Eisentraut wrote:
 On m??n, 2012-04-23 at 12:30 -0400, Noah Misch wrote:
  I've been enjoying \x auto in .psqlrc, but I noticed the row count footer
  missing when it chooses ordinary output:
 
  Looks like the logic in printQuery() needs further treatment.
 
 Hmm, this looks a bit tricky, because at the time we add the footer we
 don't yet know which output format will be used.  I don't have a good
 idea how to fix that at the moment.

I fiddled with this and settled on moving the default_footer boolean setting
and the interpretation thereof down from the printQuery() level to the
printTable() level.  That permits delaying the decision until we determine
whether the final output format is indeed vertical.
*** a/src/bin/psql/command.c
--- b/src/bin/psql/command.c
***
*** 2407,2418  do_pset(const char *param, const char *value, printQueryOpt 
*popt, bool quiet)
else if (strcmp(param, footer) == 0)
{
if (value)
!   popt-default_footer = ParseVariableBool(value);
else
!   popt-default_footer = !popt-default_footer;
if (!quiet)
{
!   if (popt-default_footer)
puts(_(Default footer is on.));
else
puts(_(Default footer is off.));
--- 2407,2418 
else if (strcmp(param, footer) == 0)
{
if (value)
!   popt-topt.default_footer = ParseVariableBool(value);
else
!   popt-topt.default_footer = !popt-topt.default_footer;
if (!quiet)
{
!   if (popt-topt.default_footer)
puts(_(Default footer is on.));
else
puts(_(Default footer is off.));
*** a/src/bin/psql/describe.c
--- b/src/bin/psql/describe.c
***
*** 1130,1135  describeOneTableDetails(const char *schemaname,
--- 1130,1136 
  
retval = false;
  
+   myopt.default_footer = false;
/* This output looks confusing in expanded mode. */
myopt.expanded = false;
  
***
*** 2363,2368  describeRoles(const char *pattern, bool verbose)
--- 2364,2371 
const char  align = 'l';
char  **attr;
  
+   myopt.default_footer = false;
+ 
initPQExpBuffer(buf);
  
if (pset.sversion = 80100)
***
*** 3362,3368  describeOneTSParser(const char *oid, const char *nspname, 
const char *prsname)
sprintf(title, _(Text search parser \%s\), prsname);
myopt.title = title;
myopt.footers = NULL;
!   myopt.default_footer = false;
myopt.translate_header = true;
myopt.translate_columns = translate_columns;
  
--- 3365,3371 
sprintf(title, _(Text search parser \%s\), prsname);
myopt.title = title;
myopt.footers = NULL;
!   myopt.topt.default_footer = false;
myopt.translate_header = true;
myopt.translate_columns = translate_columns;
  
***
*** 3393,3399  describeOneTSParser(const char *oid, const char *nspname, 
const char *prsname)
sprintf(title, _(Token types for parser \%s\), prsname);
myopt.title = title;
myopt.footers = NULL;
!   myopt.default_footer = true;
myopt.translate_header = true;
myopt.translate_columns = NULL;
  
--- 3396,3402 
sprintf(title, _(Token types for parser \%s\), prsname);
myopt.title = title;
myopt.footers = NULL;
!   myopt.topt.default_footer = true;
myopt.translate_header = true;
myopt.translate_columns = NULL;
  
***
*** 3725,3731  describeOneTSConfig(const char *oid, const char *nspname, 
const char *cfgname,
myopt.nullPrint = NULL;
myopt.title = title.data;
myopt.footers = NULL;
!   myopt.default_footer = false;
myopt.translate_header = true;
  
printQuery(res, myopt, pset.queryFout, pset.logfile);
--- 3728,3734 
myopt.nullPrint = NULL;
myopt.title = title.data;
myopt.footers = NULL;
!   myopt.topt.default_footer = false;
myopt.translate_header = true;
  
printQuery(res, myopt, pset.queryFout, pset.logfile);
*** a/src/bin/psql/print.c
--- b/src/bin/psql/print.c
***
*** 44,49  static char *decimal_point;
--- 44,52 
  static char *grouping;
  static char *thousands_sep;
  
+ static char   default_footer[100];
+ static printTableFooter default_footer_cell = { default_footer, NULL };
+ 
  /* Line style control structures */
  const printTextFormat pg_asciiformat =
  {
***
*** 278,283  print_separator(struct separator sep, FILE *fout)
--- 281,314 
  }
  
  
+ /*
+  * Return 

Re: [HACKERS] smart shutdown at end of transaction (was: Default mode for shutdown)

2012-04-27 Thread Heikki Linnakangas

On 27.04.2012 21:56, Tom Lane wrote:

Magnus Hagandermag...@hagander.net  writes:

On Fri, Apr 27, 2012 at 20:48, Tom Lanet...@sss.pgh.pa.us  wrote:

I'm not necessarily opposed to commandeering the name smart for the
new behavior, so that what we have to find a name for is the old smart
behavior.  How about

slow- allow existing sessions to finish (old smart)



How about wait instead of slow?


I kinda liked slow vs fast, but if you think that's too cute ...
(wait doesn't seem very good, though, since all these except immediate
are waiting, just for different things.)


All the modes indeed wait (except for immediate), so I think it would 
make sense to define the modes in terms of *what* they wait for.


wait sessions   - allow existing sessions to finish (old smart)
wait transactions   - allow existing transactions to finish (new)
wait checkpoint - kill active queries
wait none - unclean shutdown

Hmm, the latter two are perhaps a bit confusing. So maybe:

wait_sessions   - allow existing sessions to finish (old smart)
wait_transactions   - allow existing transactions to finish (new)
fast- kill active queries
immediate - unclean shutdown

Just thinking out loud here..

--
  Heikki Linnakangas
  EnterpriseDB   http://www.enterprisedb.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] xReader, double-effort (was: Temporary tables under hot standby)

2012-04-27 Thread Aakash Goel
Sure Kevin, will get the wiki page ready asap, and reply back. Thanks.

On Thu, Apr 26, 2012 at 8:10 PM, Kevin Grittner kevin.gritt...@wicourts.gov
 wrote:

 [resending because of postgresql.org bounces on first try]

 Simon Riggs si...@2ndquadrant.com wrote:
  Kevin Grittner kevin.gritt...@wicourts.gov wrote:

  The GSoC xReader project is intended to be a major step toward
  that, by providing a way to translate the WAL stream to a series
  of notifications of logical events to clients which register with
  xReader.
 
  This is already nearly finished in prototype and will be published
  in May. Andres Freund is working on it, copied here.

 URL?

  It looks like there is significant overlap there.

 Hard for me to know without more information.  It sounds like there
 is at least some overlap.  I hope that can involve cooperation, with
 the efforts of Andres forming the basis of Aakash's GSoC effort.
 That might leave him more time to polish up the user filters.

 Aakash: It seems like we need that Wiki page rather sooner than
 later.  Can you get to that quickly?  I would think that just
 copying the text from your approved GSoC proposal would be a very
 good start.  If you need help figuring out how to embed the images
 from your proposal, let me know.

 -Kevin



Re: [HACKERS] smart shutdown at end of transaction (was: Default mode for shutdown)

2012-04-27 Thread Robert Haas
On Fri, Apr 27, 2012 at 3:00 PM, Robert Haas robertmh...@gmail.com wrote:
 On Fri, Apr 27, 2012 at 2:48 PM, Tom Lane t...@sss.pgh.pa.us wrote:
 I'm not necessarily opposed to commandeering the name smart for the
 new behavior, so that what we have to find a name for is the old smart
 behavior.  How about

        slow    - allow existing sessions to finish (old smart)
        smart   - allow existing transactions to finish (new)
        fast    - kill active queries
        immediate - unclean shutdown

 I could live with that.  Really, I'd like to have fast just be the
 default.  But the above compromise would still be a big improvement
 over what we have now, assuming the new smart becomes the default.

So right now, we have a mapping from signals to shutdown types that
looks like this:

[Current] SIGTERM - smart, SIGINT - fast, SIGQUIT - immediate

It seems we need another signal for the new mode, and the obvious
candidate is SIGUSR2.  But what shall the mapping look like?

[Choice #1] SIGUSR2 - slow, SIGTERM - smart, SIGINT - fast, SIGQUIT
- immediate
[Choice #2] SIGTERM - slow, SIGUSR2 - smart, SIGINT - fast, SIGQUIT
- immediate

In other words, should we retain the existing behavior for SIGTERM and
make SIGUSR2 have the new behavior (choice #2)?  Or shall we preserve
the invariant that SIGTERM invokes the default shutdown mode, and move
the current default behavior off into SIGUSR2 land (choice #1)?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] smart shutdown at end of transaction (was: Default mode for shutdown)

2012-04-27 Thread Kevin Grittner
Heikki Linnakangas heikki.linnakan...@enterprisedb.com wrote:
 
 Just thinking out loud here..
 
In the spirit of kicking around ideas...
 
For those writing service scripts where you want a time limit on how
long a stop can take, so that the service script doesn't prevent OS
shutdown within a bounded time, it would also be nice to add an
escalation time limit; so if there isn't a shutdown withing k
seconds at one level it goes to the next.  If the building is on
fire and you need to power down all equipment before the fire
department cuts power and starts spraying water (a situation we had
at a courthouse a year or two ago), you really don't want the OS
waiting for anything for more than a limited number of seconds
before escalating to immediate.  We do that in our sh scripts now,
by using kill and sleep instead of trusting pg_ctl, but it seems
like it would be better to have pg_ctl know how to do that.
 
maybe?:
 
  --escalate-after=seconds
 
-Kevin

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] smart shutdown at end of transaction (was: Default mode for shutdown)

2012-04-27 Thread Peter Eisentraut
On fre, 2012-04-27 at 20:39 +0200, Andres Freund wrote:
 I think the current smart mode is rather useful. There is quite some
 stuff that you cannot do inside a transaction - or it doesn't make
 sense - which still needs to shutdown gracefully. E.g. transaction
 managers.

Could you elaborate on that?  What would happen to the transaction
manager if you terminate any idle, not-in-a-transaction database backend
sessions it has established?


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] plpython crash (PG 92)

2012-04-27 Thread Peter Eisentraut
On tor, 2012-04-26 at 17:32 +0500, Asif Naeem wrote:
 PFA test case. It used simple select statement to retrieve data via
 plpython. It crashes latest pg 9.2 with the following stack trace i.e.

 Apparently it is being crashed because of invalid related pointer value of
 pfree() *header-context-methods-free_p. It is reproducible with latest
 version of python i.e. Python-2.7.3 and Python-3.2.3. Thanks.

This is because of this code:

static void
PLy_result_dealloc(PyObject *arg)
{
PLyResultObject *ob = (PLyResultObject *) arg;

Py_XDECREF(ob-nrows);
Py_XDECREF(ob-rows);
Py_XDECREF(ob-status);
if (ob-tupdesc)
{
FreeTupleDesc(ob-tupdesc);  // -- dies here
ob-tupdesc = NULL;
}

arg-ob_type-tp_free(arg);
}

I must have been confused about the tuple descriptor APIs.  ob-tupdesc
is created using CreateTupleDescCopy(), which copies the refcount of the
original tuple descriptor, thus causing a failure when the (seemingly
still referenced) tupdesc is freed.  Is this behavior correct and
useful?

The dominant coding practice appears to be to not explicitly free copied
tuple descriptors, so maybe that should be done here as well.


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] smart shutdown at end of transaction (was: Default mode for shutdown)

2012-04-27 Thread Andres Freund
On Friday, April 27, 2012 10:17:59 PM Peter Eisentraut wrote:
 On fre, 2012-04-27 at 20:39 +0200, Andres Freund wrote:
  I think the current smart mode is rather useful. There is quite some
  stuff that you cannot do inside a transaction - or it doesn't make
  sense - which still needs to shutdown gracefully. E.g. transaction
  managers.
 Could you elaborate on that?  What would happen to the transaction
 manager if you terminate any idle, not-in-a-transaction database backend
 sessions it has established?
In the few cases where I investigated it TMs don't use transactions themselves 
(which I think is correct, they don't need them), so terminating any idle 
session - which the TM would appear as, as its not using txns - would leave 
prepared transactions in a limbo state till the database is up again, instead 
of waiting till all prepared transactions are either aborted or committed. It 
may also choose to coordinate to abort all transactions, but all that is hard 
if the database shuts you out.
I actually also have co-maintained other software where some processes have an 
idle connection open on which some shutdown stuff will happen. Obviously all 
those will need to handle the case where the connection was aborted, but that 
may result in suboptimal behaviour. Requiring such processes to keep open a 
transaction doesn't seem to be a good design choice in the pg world.

Andres

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] xReader, double-effort (was: Temporary tables under hot standby)

2012-04-27 Thread Aakash Goel
All, the wiki page is now up at http://wiki.postgresql.org/wiki/XReader.

On Sat, Apr 28, 2012 at 1:19 AM, Aakash Goel aakash.b...@gmail.com wrote:

 Sure Kevin, will get the wiki page ready asap, and reply back. Thanks.


 On Thu, Apr 26, 2012 at 8:10 PM, Kevin Grittner 
 kevin.gritt...@wicourts.gov wrote:

 [resending because of postgresql.org bounces on first try]

 Simon Riggs si...@2ndquadrant.com wrote:
  Kevin Grittner kevin.gritt...@wicourts.gov wrote:

  The GSoC xReader project is intended to be a major step toward
  that, by providing a way to translate the WAL stream to a series
  of notifications of logical events to clients which register with
  xReader.
 
  This is already nearly finished in prototype and will be published
  in May. Andres Freund is working on it, copied here.

 URL?

  It looks like there is significant overlap there.

 Hard for me to know without more information.  It sounds like there
 is at least some overlap.  I hope that can involve cooperation, with
 the efforts of Andres forming the basis of Aakash's GSoC effort.
 That might leave him more time to polish up the user filters.

 Aakash: It seems like we need that Wiki page rather sooner than
 later.  Can you get to that quickly?  I would think that just
 copying the text from your approved GSoC proposal would be a very
 good start.  If you need help figuring out how to embed the images
 from your proposal, let me know.

 -Kevin





Re: [HACKERS] smart shutdown at end of transaction (was: Default mode for shutdown)

2012-04-27 Thread Merlin Moncure
On Fri, Apr 27, 2012 at 1:57 PM, Robert Haas robertmh...@gmail.com wrote:
 I think there is no point at all in having a discussion about this
 unless we can first agree that the overwhelming majority of people who
 have commented on this issue on this list are unhappy with the current
 default behavior.

count me in. the current behavior sucks.

merlin

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


[HACKERS] Re: xReader, double-effort (was: Temporary tables under hot standby)

2012-04-27 Thread Kevin Grittner
[replaced bad email address for Josh (which was my fault)] 
 
Aakash Goel aakash.b...@gmail.com wrote: 
 All, the wiki page is now up at
  http://wiki.postgresql.org/wiki/XReader.
 
Note that the approach Aakash is taking doesn't involve changes to
the backend code, it is strictly a standalone executable to which
functions as a proxy to a hot standby and to which clients like
replications systems connect.  There is a possible additional
configuration which wouldn't require a hot standby, if time permits.
I am not clear on whether 2nd Quadrant's code takes this approach
or builds it into the server.  I think we need to know that much
before we can get very far in discussion.
 
-Kevin

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] 9.2 release notes, beta time?

2012-04-27 Thread Josh Berkus

 I suspect that's a bit aggressive.  We have a couple of issues that I
 think we should fix before beta1:

Hmmm.  So are we looking at May 7? Or later?

I'd like to do some press work for this.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Request to add options to tools/git_changelog

2012-04-27 Thread Bruce Momjian
On Thu, Apr 26, 2012 at 03:19:04PM -0400, Bruce Momjian wrote:
 Also consider that A is usually the big, clear commit message, and B,C,D
 are just minor adjustments with more brief commits, which might require
 adjusting the release note item for feature A.  When they are in
 newest-first order, that is much harder.

Updated, attached patch applied.  Thanks.

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +
*** /pg/tools/git_changelog	2012-01-01 17:48:55.0 -0500
--- /rtmp/pggitlog	2012-04-27 17:06:19.0 -0400
***
*** 40,48 
  # Might want to make this parameter user-settable.
  my $timestamp_slop = 600;
  
  my $post_date = 0;
  my $since;
! Getopt::Long::GetOptions('post-date' = \$post_date,
   'since=s' = \$since) || usage();
  usage() if @ARGV;
  
--- 40,57 
  # Might want to make this parameter user-settable.
  my $timestamp_slop = 600;
  
+ my $details_after = 0;
  my $post_date = 0;
+ my $master_only = 0;
+ my $oldest_first = 0;
  my $since;
! my @output_buffer;
! my $output_line = '';
! 
! Getopt::Long::GetOptions('details-after' = \$details_after,
! 			 'master-only' = \$master_only,
! 			 'post-date' = \$post_date,
! 			 'oldest-first' = \$oldest_first,
   'since=s' = \$since) || usage();
  usage() if @ARGV;
  
***
*** 179,195 
  	last if !defined $best_branch;
  	my $winner =
  		$all_commits_by_branch{$best_branch}-[$position{$best_branch}];
! 	printf Author: %s\n, $winner-{'author'};
! 	foreach my $c (@{$winner-{'commits'}}) {
! 	printf Branch: %s, $c-{'branch'};
! 	if (defined $c-{'last_tag'}) {
! 		printf  Release: %s, $c-{'last_tag'};
! 	}
! 	printf  [%s] %s\n, substr($c-{'commit'}, 0, 9), $c-{'date'};
  	}
! 	print \n;
! 	print $winner-{'message'};
! 	print \n;
  	$winner-{'done'} = 1;
  	for my $branch (@BRANCHES) {
  		my $leader = $all_commits_by_branch{$branch}-[$position{$branch}];
--- 188,204 
  	last if !defined $best_branch;
  	my $winner =
  		$all_commits_by_branch{$best_branch}-[$position{$best_branch}];
! 
! 	# check for master-only
! 	if (! $master_only || ($winner-{'commits'}[0]-{'branch'} eq 'master' 
! 	@{$winner-{'commits'}} == 1)) {
! 		output_details($winner) if (! $details_after);
! 		output_str(%s, $winner-{'message'} . \n);
! 		output_details($winner) if ($details_after);
! 		unshift(@output_buffer, $output_line) if ($oldest_first);
! 		$output_line = '';
  	}
! 
  	$winner-{'done'} = 1;
  	for my $branch (@BRANCHES) {
  		my $leader = $all_commits_by_branch{$branch}-[$position{$branch}];
***
*** 200,205 
--- 209,216 
  	}
  }
  
+ print @output_buffer if ($oldest_first);
+ 
  sub push_commit {
  	my ($c) = @_;
  	my $ht = hash_commit($c);
***
*** 258,268 
  	return $gm - $tzoffset;
  }
  
  sub usage {
  	print STDERR EOM;
! Usage: git_changelog [--post-date/-p] [--since=SINCE]
! --post-date Show branches made after a commit occurred
! --since Print only commits dated since SINCE
  EOM
  	exit 1;
  }
--- 269,306 
  	return $gm - $tzoffset;
  }
  
+ sub output_str {
+ 	($oldest_first) ? ($output_line .= sprintf(shift, @_)) : printf(@_);
+ }
+ 
+ sub output_details {
+ 	my $item = shift;
+ 
+ 	if ($details_after) {
+ 		$item-{'author'} =~ m{^(.*?)\s*[^]*$};
+ 		# output only author name, not email address
+ 		output_str((%s)\n, $1);
+ 	} else {
+ 		output_str(Author: %s\n, $item-{'author'});
+ 	}
+ 	foreach my $c (@{$item-{'commits'}}) {
+ 	output_str(Branch: %s , $c-{'branch'}) if (! $master_only);
+ 	if (defined $c-{'last_tag'}) {
+ 		output_str(Release: %s , $c-{'last_tag'});
+ 	}
+ 	output_str([%s] %s\n, substr($c-{'commit'}, 0, 9), $c-{'date'});
+ 	}
+ 	output_str(\n);
+ }
+ 
  sub usage {
  	print STDERR EOM;
! Usage: git_changelog [--details-after/-d] [--master-only/-m] [--oldest-first/-o] [--post-date/-p] [--since=SINCE]
! --details-after Show branch and author info after the commit description
! --master-only   Show commits made exclusively to the master branch
! --oldest-first  Show oldest commits first
! --post-date Show branches made after a commit occurred
! --since Print only commits dated since SINCE
  EOM
  	exit 1;
  }

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] 9.2 release notes, beta time?

2012-04-27 Thread Bruce Momjian
On Fri, Apr 27, 2012 at 02:16:02PM -0700, Josh Berkus wrote:
 
  I suspect that's a bit aggressive.  We have a couple of issues that I
  think we should fix before beta1:
 
 Hmmm.  So are we looking at May 7? Or later?
 
 I'd like to do some press work for this.

Someone said one week as not enough, so I assumed a wrap on May 10 with
beta release on May 14.

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Re: xReader, double-effort (was: Temporary tables under hot standby)

2012-04-27 Thread Andres Freund
On Friday, April 27, 2012 11:04:04 PM Kevin Grittner wrote:
 [replaced bad email address for Josh (which was my fault)]
 
 Aakash Goel aakash.b...@gmail.com wrote:
  All, the wiki page is now up at
  
   http://wiki.postgresql.org/wiki/XReader.
 
 Note that the approach Aakash is taking doesn't involve changes to
 the backend code, it is strictly a standalone executable to which
 functions as a proxy to a hot standby and to which clients like
 replications systems connect.  There is a possible additional
 configuration which wouldn't require a hot standby, if time permits.
 I am not clear on whether 2nd Quadrant's code takes this approach
 or builds it into the server.  I think we need to know that much
 before we can get very far in discussion.
In the current, prototypal, state there is one component thats integrated into 
the server (because it needs information thats only available there). That 
component is layered ontop of a totally generic xlog reading/parsing library 
that doesn't care at all where its running. Its also used in another cluster 
to read the received (filtered) stream.
I plan to submit the XLogReader (thats what its called atm) before everything 
else, so everybody can take a look as soon as possible.

I took a *very* short glance over the current wiki description of xReader and 
from that it seems to me it would benefit from trying to make it 
architecturally more similar to the rest of pg. I also would suggest reviewing 
how the current walreceiver/sender, and their protocol, work.

Andres


-- 
 Andres Freund http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] 9.2 release notes, beta time?

2012-04-27 Thread Josh Berkus
On 4/27/12 2:29 PM, Bruce Momjian wrote:
 On Fri, Apr 27, 2012 at 02:16:02PM -0700, Josh Berkus wrote:

 I suspect that's a bit aggressive.  We have a couple of issues that I
 think we should fix before beta1:

 Hmmm.  So are we looking at May 7? Or later?

 I'd like to do some press work for this.
 
 Someone said one week as not enough, so I assumed a wrap on May 10 with
 beta release on May 14.

Doing a release on May 14 will be a challenge, thanks to people being on
airplanes to pgCon.  We might want to verify availability from the
packagers.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] 9.2 release notes, beta time?

2012-04-27 Thread Bruce Momjian
On Fri, Apr 27, 2012 at 02:43:13PM -0700, Josh Berkus wrote:
 On 4/27/12 2:29 PM, Bruce Momjian wrote:
  On Fri, Apr 27, 2012 at 02:16:02PM -0700, Josh Berkus wrote:
 
  I suspect that's a bit aggressive.  We have a couple of issues that I
  think we should fix before beta1:
 
  Hmmm.  So are we looking at May 7? Or later?
 
  I'd like to do some press work for this.
  
  Someone said one week as not enough, so I assumed a wrap on May 10 with
  beta release on May 14.
 
 Doing a release on May 14 will be a challenge, thanks to people being on
 airplanes to pgCon.  We might want to verify availability from the
 packagers.

Do we want to do the normal package Thursday, release Monday process?  I
assumed we did.

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] smart shutdown at end of transaction (was: Default mode for shutdown)

2012-04-27 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes:
 It seems we need another signal for the new mode, and the obvious
 candidate is SIGUSR2.  But what shall the mapping look like?

 [Choice #1] SIGUSR2 - slow, SIGTERM - smart, SIGINT - fast, SIGQUIT
 - immediate
 [Choice #2] SIGTERM - slow, SIGUSR2 - smart, SIGINT - fast, SIGQUIT
 - immediate

SIGTERM needs to correspond to a fairly aggressive shutdown mode,
since (at least on some systems) init will send that during the system
shutdown sequence, shortly before escalating to SIGKILL.  So I think
choice #2 is not sensible at all.

If we were willing to consider wholesale breakage of any scripts that
send these signals directly, I'd almost consider that it should be
SIGUSR2, SIGINT, SIGTERM, SIGQUIT.  But that might be more churn than
we want.  Keeping SIGTERM attached to the default/smart shutdown mode
seems like a reasonable compromise.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Re: xReader, double-effort (was: Temporary tables under hot standby)

2012-04-27 Thread Kevin Grittner
Andres Freund and...@2ndquadrant.com wrote:
 
 In the current, prototypal, state there is one component thats
 integrated into the server (because it needs information thats
 only available there).
 
The xReader design was based on the idea that it would be nice not
to cause load on the master machine, and that by proxying the WAL
stream to the HS, using synchronous replication style to write from
xReader to the HS, you could use the HS for a source for that data
with it being at exactly the right point in time to query it.
 
I'm not convinced that I would rather see the logic fixed inside the
master as opposed to being deployable on the master's machine, the
slave machine, or even on its own machine in between.
 
 That component is layered ontop of a totally generic xlog
 reading/parsing library that doesn't care at all where its
 running.
 
That's cool.
 
 Its also used in another cluster to read the received (filtered)
 stream.
 
I don't quite follow what you're saying there.
 
 I plan to submit the XLogReader (thats what its called atm)
 before everything else, so everybody can take a look as soon as
 possible.
 
Great!  That will allow more discussion and planning.
 
 I took a *very* short glance over the current wiki description of
 xReader and from that it seems to me it would benefit from trying
 to make it architecturally more similar to the rest of pg.
 
We're planning on using existing protocol to talk between pieces. 
Other than breaking it out so that it can run somewhere other than
inside the server, and allowing clients to connect to xReader to
listen to WAL events of interest, are you referring to anything
else?
 
 I also would suggest reviewing how the current walreceiver/sender,
 and their protocol, work.
 
Of course!  The first inch-stone in the GSoC project plan
basically consists of creating an executable that functions as a
walreceiver and a walsender to just pass things through from the
master to the slave.  We build from there by allowing clients to
connect (again, over existing protocol) and register for events of
interest, and then recognizing different WAL records to generate
events.  The project was just going to create a simple client to
dump the information to disk, but with the time saved by adopting
what you've already done, that might leave more time for generating
a useful client.
 
Aakash, when you get a chance, could you fill in the inch-stones
from the GSoC proposal page onto the Wiki page?  I think the
descriptions of those interim steps would help people understand
your proposal better.  Obviously, some of the particulars of tasks
and the dates may need adjustment based on the new work which is
expected to appear before you start, but what's there now would be a
helpful reference.
 
-Kevin

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] xReader, double-effort

2012-04-27 Thread Greg Smith

On 04/26/2012 12:57 PM, Robert Haas wrote:

I don't recall seeing any previous postings about the 2Q project,
either.  We should try to keep these discussions on-list.


First brought up two months ago to the list it seemed most relevant to 
at the time: 
http://archives.postgresql.org/pgsql-cluster-hackers/2012-02/msg1.php and 
the solid commentary started coming out yesterday, right near the 
approach of May as promised.  There hasn't been much posted on hackers 
about this out of respect for the extended CommitFest.


I personally suggested Aakash vent ideas out on pgsql-students (or here) 
at the beginning of the month: 
http://archives.postgresql.org/pgsql-students/2012-04/msg2.php but 
the next time I heard about the subject was the acceptance notice.


--
Greg Smith   2ndQuadrant USg...@2ndquadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Future In-Core Replication

2012-04-27 Thread Joshua D. Drake


On 04/27/2012 11:33 AM, Simon Riggs wrote:
Well, they all sound similar. My info was that Mammoth was not WAL-based. 

Mammoth was transaction log based but not WAL based.

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Re: xReader, double-effort (was: Temporary tables under hot standby)

2012-04-27 Thread Andres Freund
Hi Kevin, Hi Aakash,

On Saturday, April 28, 2012 12:18:38 AM Kevin Grittner wrote:
 Andres Freund and...@2ndquadrant.com wrote:
  In the current, prototypal, state there is one component thats
  integrated into the server (because it needs information thats
  only available there).
 The xReader design was based on the idea that it would be nice not
 to cause load on the master machine, and that by proxying the WAL
 stream to the HS, using synchronous replication style to write from
 xReader to the HS, you could use the HS for a source for that data
 with it being at exactly the right point in time to query it.
Yes, that does make sense for some workloads. I don't think its viable for 
everything though, thats why were not aiming for that ourselves atm.

 I'm not convinced that I would rather see the logic fixed inside the
 master as opposed to being deployable on the master's machine, the
 slave machine, or even on its own machine in between.
I don't think that you can do everything apart from the master. We currently 
need shared memory for coordination between the moving parts, thats why we 
have it inside the master.
It also have the advantage of being easier to setup.

  That component is layered ontop of a totally generic xlog
  reading/parsing library that doesn't care at all where its
  running.
 That's cool.

  Its also used in another cluster to read the received (filtered)
  stream.
 I don't quite follow what you're saying there.
To interpret the xlog back into something that can be used for replication you 
need to read it again. After filtering we again write valid WAL, so we can use 
the same library on the sending|filtering side and on the receiving side.
But thats actually off topic for this thread ;)


  I took a *very* short glance over the current wiki description of
  xReader and from that it seems to me it would benefit from trying
  to make it architecturally more similar to the rest of pg.
 We're planning on using existing protocol to talk between pieces.
 Other than breaking it out so that it can run somewhere other than
 inside the server, and allowing clients to connect to xReader to
 listen to WAL events of interest, are you referring to anything
 else?
It sounds like the xReader is designed to be one multiplexing process. While 
this definitely has some advantages resource-usage-wise it doesn't seem to be 
fitting the rest of the design that well. The advantages might outweigh 
everything else, but I am not sure about that.
Something like registering/deregistering also doesn't fit that well with the 
way walsender works as far as I understand it.

Greetings,

Andres
-- 
 Andres Freund http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] A problem with dump/restore of views containing whole row references

2012-04-27 Thread Tom Lane
Abbas Butt abbas.b...@enterprisedb.com writes:
 On Fri, Apr 27, 2012 at 11:21 PM, Tom Lane t...@sss.pgh.pa.us wrote:
 More generally, it seems rather inelegant to be forcibly adding a cast
 when in most cases the existing notation is not wrong.  AFAICS the
 plain relname notation is only ambiguous if there is a column of the
 same name as the relation.  I wonder whether we should instead address
 this by not letting the parser strip the no op cast in the first
 place.

 You mean that the parser should not strip the no op cast in all cases or
 in the case only when the parser somehow detects a column of the same name
 as the relation?

On reflection that's the wrong thing anyway.  While (AFAICS) one could
only initially create this type of situation by using an explicit cast
as in your example, the ambiguity could be introduced after the fact by
a rename or ALTER TABLE ADD COLUMN, which wouldn't even have to affect
the troublesome table itself --- a column matching the table's name
anywhere in the FROM clause would create the same ambiguity.  So there's
no guarantee that there ever was a cast there.

So I think that your patch is the right approach, if wrong in detail.
What we have to do is stop using the ambiguous table-name-only syntax,
and instead always print tabname.*, and then add a cast to prevent
expansion of the * if we are at top level of a SELECT targetlist.

Attached is a patch that I think does this correctly.  I renamed the
flag parameter (and flipped its sense) since it is no longer controlling
whether or not a * gets printed.  One thing I like about this is that
whole-row Vars are no longer ever special in terms of naming; looking
at the code with a fresh eye, I wonder whether we didn't have other bugs
here in cases such as where a schema qualification is needed.  Omitting
the star is just asking for trouble ...

regards, tom lane

diff --git a/src/backend/utils/adt/ruleutils.c b/src/backend/utils/adt/ruleutils.c
index 3beed37dd26d6fc51a4365b3f4086b97f71cc16c..7ad99a0ec32579760f6e10278ec8585a0b2ab855 100644
*** a/src/backend/utils/adt/ruleutils.c
--- b/src/backend/utils/adt/ruleutils.c
*** static void get_rule_orderby(List *order
*** 209,215 
  static void get_rule_windowclause(Query *query, deparse_context *context);
  static void get_rule_windowspec(WindowClause *wc, List *targetList,
  	deparse_context *context);
! static char *get_variable(Var *var, int levelsup, bool showstar,
  			 deparse_context *context);
  static RangeTblEntry *find_rte_by_refname(const char *refname,
  	deparse_context *context);
--- 209,215 
  static void get_rule_windowclause(Query *query, deparse_context *context);
  static void get_rule_windowspec(WindowClause *wc, List *targetList,
  	deparse_context *context);
! static char *get_variable(Var *var, int levelsup, bool istoplevel,
  			 deparse_context *context);
  static RangeTblEntry *find_rte_by_refname(const char *refname,
  	deparse_context *context);
*** get_target_list(List *targetList, depars
*** 3074,3084 
  		 * foo.*, which is the preferred notation in most contexts, but at
  		 * the top level of a SELECT list it's not right (the parser will
  		 * expand that notation into multiple columns, yielding behavior
! 		 * different from a whole-row Var).  We want just foo, instead.
  		 */
  		if (tle-expr  IsA(tle-expr, Var))
  		{
! 			attname = get_variable((Var *) tle-expr, 0, false, context);
  		}
  		else
  		{
--- 3074,3085 
  		 * foo.*, which is the preferred notation in most contexts, but at
  		 * the top level of a SELECT list it's not right (the parser will
  		 * expand that notation into multiple columns, yielding behavior
! 		 * different from a whole-row Var).  We need to call get_variable
! 		 * directly so that we can tell it to do the right thing.
  		 */
  		if (tle-expr  IsA(tle-expr, Var))
  		{
! 			attname = get_variable((Var *) tle-expr, 0, true, context);
  		}
  		else
  		{
*** get_utility_query_def(Query *query, depa
*** 3803,3815 
   * the Var's varlevelsup has to be interpreted with respect to a context
   * above the current one; levelsup indicates the offset.
   *
!  * If showstar is TRUE, whole-row Vars are displayed as foo.*;
!  * if FALSE, merely as foo.
   *
!  * Returns the attname of the Var, or NULL if not determinable.
   */
  static char *
! get_variable(Var *var, int levelsup, bool showstar, deparse_context *context)
  {
  	StringInfo	buf = context-buf;
  	RangeTblEntry *rte;
--- 3804,3823 
   * the Var's varlevelsup has to be interpreted with respect to a context
   * above the current one; levelsup indicates the offset.
   *
!  * If istoplevel is TRUE, the Var is at the top level of a SELECT's
!  * targetlist, which means we need special treatment of whole-row Vars.
!  * Instead of the normal tab.*, we'll print tab.*::typename, which is a
!  * dirty hack to prevent tab.* from being expanded into multiple 

Re: [HACKERS] xReader, double-effort

2012-04-27 Thread Kevin Grittner
Greg Smith g...@2ndquadrant.com wrote:
 On 04/26/2012 12:57 PM, Robert Haas wrote:
 I don't recall seeing any previous postings about the 2Q project,
 either.  We should try to keep these discussions on-list.
 
 First brought up two months ago to the list it seemed most
 relevant to at the time: 

http://archives.postgresql.org/pgsql-cluster-hackers/2012-02/msg1.php
 and the solid commentary started coming out yesterday, right near
 the approach of May as promised.  There hasn't been much posted on
 hackers about this out of respect for the extended CommitFest.
 
 I personally suggested Aakash vent ideas out on pgsql-students (or
 here) at the beginning of the month: 
 http://archives.postgresql.org/pgsql-students/2012-04/msg2.php
 but the next time I heard about the subject was the acceptance
 notice.
 
That's largely my fault.  Aakash's original proposal was naively
assuming that logical records were available in our xlog stream and
wanted to work on tools to do something useful with that data. 
There was some confusion among the various core team members,
committers, and other contributors who volunteered to review the
proposals about whether he was talking about analyzing the server
log or xlog.  We had a wide-ranging discussion on the freenode
#postgresql channel about it, and brainstormed to the current idea
-- at which point Aakash had about a week and a half to rewrite his
proposal along the lines discussed on irc.  A week after that the
acceptance notices came out.
 
If we had had any idea that 2Q was working on something along these
lines, we would have coordinated or guided him in another direction.
The post you cite from a couple months ago was way too vague to
have even an inkling that it had anything to do with providing feeds
of the logical interpretation of the WAL stream.  I don't recall
having heard the idea even mentioned before.
 
The timing in regard to the release process seems to have had an
effect on both groups here, since I suggested he read up on
walsender and walreceiver and a few other areas, put together a Wiki
page with the plan, and *then* describe the preliminary plan to the
community (a month or so before starting to code), so that details
could be hashed out in public before time was spent coding.  Had
there not been so many people heads-down trying to get the release
out, I might have advised him to post a hand-wavy description of the
effort before he got the Wiki page set up and reviewed the relevant
code.
 
As Josh points out, his first choice was to write utilities to *do*
something with the logical xlog information, and he only switched
over to working on *producing* it when he heard that no such thing
was currently available.  I think there's a lot of room to
coordinate these efforts so that there *isn't* a waste of effort as
long as communication is open enough.
 
-Kevin

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Future In-Core Replication

2012-04-27 Thread Christopher Browne
On Fri, Apr 27, 2012 at 4:11 AM, Simon Riggs si...@2ndquadrant.com wrote:
 What I'm hoping to do is to build a basic prototype of logical
 replication using WAL translation, so we can inspect it to see what
 the downsides are. It's an extremely non-trivial problem and so I
 expect there to be mountains to climb. There are other routes to
 logical replication, with messages marshalled in a similar way to
 Slony/Londiste/Bucardo/Mammoth(?). So there are options, with
 measurements to be made and discussions to be had.

I'll note that the latest version of Slony, expected to be 2.2 (which
generally seems to work, but we're stuck at the moment waiting to get
free cycles to QA it) has made a substantial change to its data
representation.

The triggers used to cook data into a sort of fractional WHERE
clause, transforming an I/U/D into a string that you'd trivially
combine with the string INSERT INTO/UPDATE/DELETE FROM to get the
logical update.  If there was need to do anything fancier, you'd be
left having to have a fractional SQL parser to split the data out by
hand.

New in 2.2 is that the log data is split out into an array of text
values which means that if someone wanted to do some transformation,
such as filtering on value, or filtering out columns, they could
modify the application-of-updates code to query for the data that they
want to fiddle with.  No parser needed.

It's doubtless worthwhile to take a peek at that to make sure it
informs your data representation appropriately.  It's important to
have data represented in a fashion that is amenable to manipulation,
and that decidedly wasn't the case pre-2.2.

I wonder if a meaningful transport mechanism might involve combining:
a) A trigger that indicates that some data needs to be captured in a
logical form (rather than the presently pretty purely physical form
of WAL)
b) Perhaps a way of capturing logical updates in WAL
c) One of the old ideas that fell through was to try to capture commit
timestamps via triggers.  Doing it directly turned out to be too
controversial to get in.  Perhaps that's something that could be
captured via some process that parses WAL.

Something seems wrong about that in that it mixes together updates of
multiple forms into WAL, physical *and* logical, and perhaps that
implies that there should be an altogether separate logical updates
log. (LUL?)  That still involves capturing updates in a duplicative
fashion, e.g. - WAL + LUL, which seems somehow wrong.  Or perhaps I'm
tilting at a windmill here.  With Slony/Londiste/Bucardo, we're
capturing LUL in some tables, meaning that it gets written both to
the tables' data files as well as WAL.  Adding a binary LUL eliminates
those table files and attendant WAL updates, thus providing some
savings.

[Insert a LULCATS joke here...]

Perhaps I've just had too much coffee...
-- 
When confronted by a difficult problem, solve it by reducing it to the
question, How would the Lone Ranger handle this?

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] plpython crash (PG 92)

2012-04-27 Thread Tom Lane
Peter Eisentraut pete...@gmx.net writes:
 I must have been confused about the tuple descriptor APIs.  ob-tupdesc
 is created using CreateTupleDescCopy(), which copies the refcount of the
 original tuple descriptor,

Um, surely not.  That would be nonsensical, and anyway a look at the
code shows it isn't doing that.

 thus causing a failure when the (seemingly still referenced) tupdesc
 is freed.  Is this behavior correct and useful?

I think there must be some mistake in your analysis.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Re: xReader, double-effort (was: Temporary tables under hot standby)

2012-04-27 Thread Kevin Grittner
Andres Freund and...@2ndquadrant.com wrote:
 
 Something like registering/deregistering also doesn't fit that
 well with the way walsender works as far as I understand it.
 
If you look at the diagrams on the xReader Wiki page, the lines
labeled XLOG stream are the ones using walsender/walreceiver.  The
green arrows represent normal connections to the database, to run
queries to retrieve metadata needed to interpret the WAL records,
and the lines labeled Listener n are expected to use the pg
protocol to connect, but won't be talking page-oriented WAL -- they
will be dealing with logical interpretation of the WAL.  The sort of
data which could be fed to a database which doesn't have the same
page images.  Like Slony et al do.
 
Perhaps, given other points you made, the library for interpreting
the WAL records could be shared, and hopefully a protocol for the
clients, although that seems a lot more muddy to me at this point. 
If we can share enough code, there may be room for both approaches
with minimal code duplication.
 
-Kevin

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] 9.2 release notes, beta time?

2012-04-27 Thread Tom Lane
Josh Berkus j...@agliodbs.com writes:
 On 4/27/12 2:29 PM, Bruce Momjian wrote:
 Someone said one week as not enough, so I assumed a wrap on May 10 with
 beta release on May 14.

 Doing a release on May 14 will be a challenge, thanks to people being on
 airplanes to pgCon.  We might want to verify availability from the
 packagers.

Packagers wouldn't be the issue; their work should be done by the
Saturday at latest.  The website crew might have a problem though.

The main gating factor here is whether Dave Page's crew can get the
Windows installers ready any sooner; given his previous comments,
I don't think that moving the wrap up a few days would change anything.
We could wrap next week if we were OK with not having Windows installers
at the announcement, but I bet you'll say that's a no-go.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] 9.2 release notes, beta time?

2012-04-27 Thread Devrim GÜNDÜZ

Hi,

On Fri, 2012-04-27 at 19:02 -0400, Tom Lane wrote:
 The website crew might have a problem though.

Magnus is flying on 13th and 14th, Dave, Stefan and Thom are flying on
14th. I'm flying on 15th -- it seems I have some cycles for updating
website, I hope. Not sure about the others.

(As a side note, RPMs *may not* be ready, because I (and Magnus) will be
at PGDay Turkey on 12th, and will be busy over the whole weekend).

Regards,
-- 
Devrim GÜNDÜZ
Principal Systems Engineer @ EnterpriseDB: http://www.enterprisedb.com
PostgreSQL Danışmanı/Consultant, Red Hat Certified Engineer
Community: devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr
http://www.gunduz.org  Twitter: http://twitter.com/devrimgunduz


signature.asc
Description: This is a digitally signed message part


Re: [HACKERS] 9.2 release notes, beta time?

2012-04-27 Thread Tom Lane
Devrim =?ISO-8859-1?Q?G=DCND=DCZ?= dev...@gunduz.org writes:
 (As a side note, RPMs *may not* be ready, because I (and Magnus) will be
 at PGDay Turkey on 12th, and will be busy over the whole weekend).

Ugh.  Maybe the whole idea of getting a beta out before PGCon is doomed.
Still, if we don't try for this schedule, we're looking at at least two
more weeks' slip, because we're surely not going to wrap during PGCon.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] 9.2 release notes, beta time?

2012-04-27 Thread Bruce Momjian
On Fri, Apr 27, 2012 at 07:19:55PM -0400, Tom Lane wrote:
 Devrim =?ISO-8859-1?Q?G=DCND=DCZ?= dev...@gunduz.org writes:
  (As a side note, RPMs *may not* be ready, because I (and Magnus) will be
  at PGDay Turkey on 12th, and will be busy over the whole weekend).
 
 Ugh.  Maybe the whole idea of getting a beta out before PGCon is doomed.
 Still, if we don't try for this schedule, we're looking at at least two
 more weeks' slip, because we're surely not going to wrap during PGCon.

Should we go for an alpha?

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] 9.2 release notes, beta time?

2012-04-27 Thread Josh Berkus

 Ugh.  Maybe the whole idea of getting a beta out before PGCon is doomed.
 Still, if we don't try for this schedule, we're looking at at least two
 more weeks' slip, because we're surely not going to wrap during PGCon.

We could do it in person!


-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] 9.2 release notes, beta time?

2012-04-27 Thread Josh Berkus

 Do we want to do the normal package Thursday, release Monday process?  I
 assumed we did.

Given that in the last 2 years, our betas have been news, I think we
need to.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] xReader, double-effort

2012-04-27 Thread Greg Smith

On 04/27/2012 06:48 PM, Kevin Grittner wrote:

That's largely my fault.


It may not have come out that way, but I was trying more to point out 
the complexity of the story rather than assign blame.  I find it hard to 
point at any obvious don't do that again target here.


--
Greg Smith   2ndQuadrant USg...@2ndquadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] 9.2 release notes, beta time?

2012-04-27 Thread Tom Lane
Bruce Momjian br...@momjian.us writes:
 On Fri, Apr 27, 2012 at 07:19:55PM -0400, Tom Lane wrote:
 Ugh.  Maybe the whole idea of getting a beta out before PGCon is doomed.
 Still, if we don't try for this schedule, we're looking at at least two
 more weeks' slip, because we're surely not going to wrap during PGCon.

 Should we go for an alpha?

How would that help?  The bottleneck is packaging, which is the same
work whatever we call it.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] 9.2 release notes, beta time?

2012-04-27 Thread Bruce Momjian
On Fri, Apr 27, 2012 at 09:10:54PM -0400, Tom Lane wrote:
 Bruce Momjian br...@momjian.us writes:
  On Fri, Apr 27, 2012 at 07:19:55PM -0400, Tom Lane wrote:
  Ugh.  Maybe the whole idea of getting a beta out before PGCon is doomed.
  Still, if we don't try for this schedule, we're looking at at least two
  more weeks' slip, because we're surely not going to wrap during PGCon.
 
  Should we go for an alpha?
 
 How would that help?  The bottleneck is packaging, which is the same
 work whatever we call it.

We release the alpha with no packaging.

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] 9.2 release notes, beta time?

2012-04-27 Thread Tom Lane
Bruce Momjian br...@momjian.us writes:
 On Fri, Apr 27, 2012 at 09:10:54PM -0400, Tom Lane wrote:
 How would that help?  The bottleneck is packaging, which is the same
 work whatever we call it.

 We release the alpha with no packaging.

Think we'd lose a lot of potential testers that way.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] 9.2 release notes, beta time?

2012-04-27 Thread David Johnston
On Apr 27, 2012, at 21:24, Tom Lane t...@sss.pgh.pa.us wrote:

 Bruce Momjian br...@momjian.us writes:
 On Fri, Apr 27, 2012 at 09:10:54PM -0400, Tom Lane wrote:
 How would that help?  The bottleneck is packaging, which is the same
 work whatever we call it.
 
 We release the alpha with no packaging.
 
 Think we'd lose a lot of potential testers that way.
 

It seems that even if you published before PGCon a reasonable number of 
possible testers would be there and/or responding to any feedback is going to 
be delayed since developers are going to be there.

I could see the goal being having a package-ready commit before PGCon but 
schedule the official release until after.  No matter what you call it those 
who want a head start can self-compile while others will just wait until the 
packagers are done.  Either way the same codebase will be in the wild (so 
probably don't want to call it alpha).

The original reason for pre-conference is so the developers can feel less bad 
talking about 9.3 features (and 9.2 post-mortem) since the beta for 9.2 will be 
completed.

David J.






-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] plpython crash (PG 92)

2012-04-27 Thread Tom Lane
Peter Eisentraut pete...@gmx.net writes:
 On tor, 2012-04-26 at 17:32 +0500, Asif Naeem wrote:
 PFA test case. It used simple select statement to retrieve data via
 plpython. It crashes latest pg 9.2 with the following stack trace i.e.

 Apparently it is being crashed because of invalid related pointer value of
 pfree() *header-context-methods-free_p. It is reproducible with latest
 version of python i.e. Python-2.7.3 and Python-3.2.3. Thanks.

 This is because of this code:

I traced through this test case.  The situation is:

(1) PLy_spi_execute_fetch_result is executed with CurrentMemoryContext
being the SPI Proc context for the current SPI invocation, so that's
where CreateTupleDescCopy creates the tupledesc that's placed into
the PLyResultObject.

(2) As we fall out of the SETOF function with the first result row,
PLy_exec_function does SPI_finish() (plpy_exec.c:146).  That causes
the SPI contexts to go away.  The PLyResultObject's tupdesc pointer
is now pointing at freed memory, and in a cassert build, that memory
has been actively wiped.

(3) The main executor calls back to plpython for the next value-per-call
result.  Control goes to the PyIter_Next call at plpy_exec.c:108, which
decides that we're done iterating (since this example only returns one
row), and evidently it tries to deallocate the PLyResultObject
immediately.  Whether that happens immediately or later, though,
you're screwed because FreeTupleDesc will be invoked on garbage.

I'm inclined to think that the best fix is for
PLy_spi_execute_fetch_result to copy the tupledesc into
TopMemoryContext, not the current context.  This is a tad scary from a
memory leakage standpoint, but I suppose that if python fails to recover
the PLyResultObject, this isn't the only memory that's going to be
leaked.

This area appears to be shy a regression test case or two, in any event.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] default_transaction_isolation = serializable causes crash under Hot Standby

2012-04-27 Thread Kevin Grittner
 Kevin Grittner  wrote:
 Tom Lane  wrote:
 
 Yeah, it would definitely be nicer if BEGIN; SET TRANSACTION LEVEL
 would work too. Maybe the place to put the check is where we
 establish the transaction snapshot.
 
 That makes sense, 
 
 I'll take a shot at it sometime today,
 
Attached.  With a default setting in postgresql.conf it handles
Robert's test case nicely, and does what Tom asks for above:

test=# set default_transaction_isolation = 'serializable';
SET
test=# begin;
BEGIN
test=# select 1;
ERROR:  can not create a serializable snapshot during recovery
test=# rollback;
ROLLBACK
test=# begin;
BEGIN
test=# set transaction isolation level repeatable read;
SET
test=# select 1;
 ?column? 
--
1
(1 row)

test=# commit;
COMMIT

But if you set it in the postgresql.conf file, it's not pretty:
 
kevin@kevin-desktop:~$ psql -p 5433 test
psql: FATAL:  can not create a serializable snapshot during recovery
 
Ideas?
 
-Kevin




hs-default-serializable-v1.diff
Description: Binary data

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers