Re: [HACKERS] [JDBC] Regarding GSoc Application

2012-04-11 Thread Hitoshi Harada
On Tue, Apr 10, 2012 at 8:41 PM, Atri Sharma atri.j...@gmail.com wrote:
 Hi All,

 I think we are back on the initial approach I proposed(hooking directly into
 the JVM and executing Java code that calls JDBC).I think the best way to do
 this is create a JVM that executes the Java code and give the control of the
 JVM to the native API.

 I agree,the only need of Pl/Java that is apparent here is the need of the
 Java internals(JDK et al).If we set them up independently,then,we can have
 the FDW wrapping JDBC directly through JNI.JNI would call pure Java
 functions to connect to the JDBC.

 I think we can proceed with this.Once we are done with the API calling Java
 functions,I think the rest of the path is easily mapped(writing Java
 functions to connect to JDBC).

 Please let me know your opinions on this.


I think Multicorn is a good example, which invokes Python from FDW
routines though it is not using PL/Python.

http://multicorn.org/

Thanks,
-- 
Hitoshi Harada

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] [JDBC] Regarding GSoc Application

2012-04-11 Thread Atri Sharma

I think Multicorn is a good example, which invokes Python from FDW
routines though it is not using PL/Python.

http://multicorn.org/



Hi Hitoshi,

Thanks for the link.

You mean,I should try to build something like Multicorn for Java?

Atri


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Last gasp

2012-04-11 Thread Magnus Hagander
On Wednesday, April 11, 2012, Robert Haas wrote:

 On Tue, Apr 10, 2012 at 10:05 PM, Peter Geoghegan 
 pe...@2ndquadrant.comjavascript:;
 wrote:
  On 11 April 2012 02:14, Robert Haas robertmh...@gmail.comjavascript:;
 wrote:
  My perception of what's going on here is dramatically different from
  yours.  I don't think there was any overflow of submissions for 9.2.
 
  That is just not true. See the attached graph (couldn't produce one
  with better resolution at short notice) - I've just eyeballed the
  graph, but it looks like an upward trend to me.

 I don't know what this is a graph of, but if you look at the number of
 patches in each of the CommitFests for the last couple of releases,
 you see this:

 9.0: 66, 40, 38, 60
 9.1: 53, 52, 43, 96
 9.2: 60, 52, 53, 104

 There might be an upward trend there, but it isn't a very steep one.
 It also seems pretty clear to me (although you are welcome to
 disagree) that even if the *number* of patches in 9.2 was higher than
 9.1, the average complexity was less, at least for the first three
 CommitFests.

 By the way, let's take a look at the major features list for 9.0 and
 9.1, as well as who wrote them.

 From the 9.0 release notes:

 - Streaming Replication (Fujii Masao - not a committer)
 - Hot Standby (Simon Riggs - committer - though he wasn't when he
 wrote the first version of this patch)
 - GRANT/REVOKE IN SCHEMA (Petr Jelinek - not a committer)
 - ALTER DEFAULT PRIVILEGES (Petr Jelinek - not a committer)
 - DO (Peter Jelinek - not a committer)
 - 64-bit Windows (Tsutomu Yamada, Magnus Hagander - not a committer
 and a committer, respectively)
 - Better window functions (Hitoshi Harada - not a committer)
 - ORDER BY for aggregates (Andrew Gierth - not a committer)
 - Deferrable unique constraints (Dean Rasheed - not a committer)
 - Exclusion constraints (Jeff Davis - not a committer)
 - RADIUS (Magnus Hagander - a committer)
 - LDAP improvements (Robert Fleming, Magnus Hagander - not a committer
 and a committer, respectively)
 - Better LISTEN/NOTIFY (Joachim Wieland - not a committer)
 - Better VACUUM FULL (Itagaki Takahiro, Tom Lane - both now
 committers, but Itagaki Takahiro became one only later)
 - pg_upgrade (Bruce Momjian - a committer)
 - join removal (Robert Haas - now a committer, but not then)
 - EXPLAIN enhancements (Robert Haas - now a committer, but not then)
 - hstore enhancments (Andrew Gierth - now a committer, but not then)


For the sake of the archives - Andrew Gierth is not a committer now either.

//Magnus



-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/


[HACKERS] [Patch] Fix little typo in a comment

2012-04-11 Thread Etsuro Fujita
This is a little patch to fix a typo in contrib/file_fdw.

 

Best regards,

Etsuro Fujita

 



file_fdw_typo_fix.patch
Description: Binary data

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Another review of URI for libpq, v7 submission

2012-04-11 Thread Alvaro Herrera

Excerpts from Alvaro Herrera's message of lun abr 09 16:41:50 -0300 2012:

 There are three minor things that need to be changed for this to be
 committable:

Committed this patch after some more editorialization; in particular the
test was rewritten so that instead of trying to connect, it uses
PQconninfoParse to figure out how the URI is parsed, which makes a
lot more sense.  Also some other changes to the accepted URI, in
particular so that username, pwd, and port are possible to be specified
when using unix-domain sockets.

Now that it is a done deal I'm sure people will start complaining how
bad the documentation change was; please keep the flames up.

-- 
Álvaro Herrera alvhe...@commandprompt.com
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Last gasp

2012-04-11 Thread Daniel Farina
On Tue, Apr 10, 2012 at 8:12 PM, Robert Haas robertmh...@gmail.com wrote:
 However exactly
 the list turns out, there is no question that non-committers have been
 quite successful in getting significant feature enhancements committed
 in each of the last three releases, and I'm pretty confident it goes
 back a lot further than that.

I agree that in practice non-committers do get a lot of great stuff
done, but the question is if *more* stuff would get done if things
were slightly different. To that end, I'd like to share my own
anecdote on why I don't attempt large projects to Postgres at this
time:

I used to work on a proprietary postgres offspring and ship quite a
bit of internals code.  A couple of people I worked with are
frequenters of this list, even. I spent nearly two years doing it,
full time, without having to (or being able to) go through a
full-blown community process from design to ship: I got a lot of
nuts-and-bolts practice, and I really enjoyed it.  Yet I don't take on
large projects in the project now, and I'm even employed in a place
where I could start doing that on-the-job. Why don't I?

One reason I don't do that is because there is an unavoidable
information asymmetry problem between myself and the committers.  When
I think of a committer and what makes me different than them, this is
what I come up with:

* More experience and expertise, both in general and with the project

* Proven intent to maintain the work submitted by others for a long
time.  In a word, stewardship or ownership

I'm grateful for both, but the latter point is one where some
mind-reading is required: what's strategically important enough that's
it is important enough to compromise on something?  What compromises
are acceptable?  That is tantamount to guessing what compromises is a
committer willing to maintain?  And that can be a pretty personal
thing and is hard to guess, and I don't think that's solvable as long
as there seems to be this handoff from the contributor to the
project.

It's hard to feel a sense of ownership -- and thus commitment -- if
one cannot simply change one's own code, especially for trivia or
churning around a future intent or purpose.  If there is a bottleneck
with the development process that is having a chilling effect on my
ability to justify larger projects, it is this.

I don't know what the most apparent way to optimize that bottleneck
is, but there's my thoughts.  I think part of the answer is more
hooks, even if they come with reduced contracts in terms of
maintenance (here this release, gone the next release), and less
required justification to commit those; consider
https://github.com/dimitri/pgextwlist, which relies on such a hook and
is the only thing that makes 9.1's extension support viable for
Heroku, yet is cohesive feeling with the rest of the system to the
point that it pretty much goes unnoticed.  That's a great property I
wish I could see more of.

Also, I am not able to spend as much time on large Postgres-projects
because some of the tooling outside the database is still the weakest
link in my chain for now, so the good news is that I've shifted my
attention to projects that are very much related but not part
postgres-proper.  The point of having more hooks is to open up more
opportunities for such tools without making the result feel incohesive
and terrible for reasons not intrinsic to the idea it is trying to add
(consider: using the statement hook in pgextwlist rather than
preloaded security-definer UDFs that would run CREATE EXTENSION for
you.  Yuck.)

-- 
fdr

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] pgsql_fdw, FDW for PostgreSQL server

2012-04-11 Thread Shigeru HANADA
Hi all,

(2012/03/07 3:39), Tom Lane wrote:
 A bigger issue with postgresql_fdw_validator is that it supposes that
 the core backend is authoritative as to what options libpq supports,
 which is bad design on its face.  It would be much more sensible for
 dblink to be asking libpq what options libpq supports, say via
 PQconndefaults().
 
 We might find that we have to leave postgresql_fdw_validator as-is
 for backwards compatibility reasons (in particular, being able to load
 existing FDW definitions) but I think we should migrate away from using
 it.

In the discussion about pgsql_fdw which was proposed for 9.2, some
issues about postgresql_fdw_validator are pointed out.

* The name postgresql_fdw_validator conflicts with the name of the FDW
for PostgreSQL which
follows the naming habit of other FDWs.
* dblink depends on postgresql_fdw_validator.
* postgresql_fdw_validator assumes that libpq supports some particular
options.

An idea to resolve these is to add dblink's own validator which doesn't
assume much about libpq, and obsolete postgresql_fdw_validator.

* Add dblink_fdw_validator to contrib/dblink, which is similar to
postgresql_fdw_validator but it assumes less about libpq.
* Add dblink_fdw as default FDW of dblink, which uses
dblink_fdw_validator, and recommend to use it.  This would prevent users
from using postgresql_fdw_validator with dblink.
* Mention that postgresql_fdw_validator might be obsolete in future
release in the document of CREATE FOREIGN DATA WRAPPER.

To make the behavior of dblink_fdw_validator similar to that of current
postgresql_fdw_validator, we need to assume that libpq supports user
option, and allow it and secret options in only USER MAPPING options,
and allow others in only SERVER options (and reject all debug options).
 IMO this is not unreasonable assumption.

Is this proposal reasonable?  Any comments and questions are welcome.

Regards,
-- 
Shigeru HANADA

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Last gasp

2012-04-11 Thread Greg Smith

On 04/10/2012 09:14 PM, Robert Haas wrote:

I wouldn't object to creating some doc-only committers.  OTOH, I would
object to anyone making non-trivial documentation enhancements without
posting their patches first and having a second person look it over,
so how much difference is there, really?


This workflow is the easy one:

-Committer suggests doc change
-No one has an opinion against it strong enough to comment, or minor 
review comments are made

-Commit change with feedback when received

That's a predictable, short process unless the change is controversial, 
in which case good feedback normally comes out of that discussion.  And 
if I feel review is needed but don't catch any volunteers, there is a 
much broader pool of people who can be hired to help with review work, 
relative to the size of the committer one.


Compare with:

-Submitter suggests doc change
-No one has a strong opinion on it, may not be picked up at all
-Submitter adds to the next CF
-Wait for review
-[Possible repost update with reviewer changes]
-Ready for committer
-Committer takes time away from code review to look at it
-Possibly another feedback/review resubmission
-Commit final versions

It's usually not this bad, but in every case it's pulling resources off 
of more valuable jobs.


I'd like to dump around 50 pages of new material into the docs as a 
start, but I don't want to take so much time away from the code oriented 
committers to chew on that much.


--
Greg Smith   2ndQuadrant USg...@2ndquadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support www.2ndQuadrant.com

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Patch: add timing of buffer I/O requests

2012-04-11 Thread k...@rice.edu
On Wed, Apr 11, 2012 at 01:53:06AM +0100, Peter Geoghegan wrote:
 On 11 April 2012 01:16, Tom Lane t...@sss.pgh.pa.us wrote:
  Peter Geoghegan pe...@2ndquadrant.com writes:
  On 11 April 2012 00:35, Robert Haas robertmh...@gmail.com wrote:
  If people need something like that, couldn't they create it by hashing
  the normalized query text with an arbitrary algorithm?
 
  That supposes that the normalised query text is perfectly stable. It
  may well not be, particularly for things like ad-hoc queries or
  queries generated by ORMs, across database clusters and over long
  periods of time -
 
  Indeed, but the hash value isn't stable either given those sorts of
  assumptions, so I'm not convinced that there's any advantage there.
 
 Isn't it? The hash captures the true meaning of the query, while
 having the database server's platform as a usually irrelevant
 artefact. Another thing that I forgot to mention is client encoding -
 it may well be fairly inconvenient to have to use the same algorithm
 to hash the query string across applications. You also have to hash
 the query string yourself again and again, which is expensive to do
 from Python or something, and is often inconvenient - differences
 beyond track_activity_query_size bytes (default:1024) are not
 recognised. Using an SQL code beautifier where a single byte varies
 now breaks everything, which developers don't expect at all (we've
 trained them not to), so in many ways you're back to the same
 limitations as classic pg_stat_statements if you attempt to aggregate
 queries over time and across machines, which is a very real use case.
 
 It's probably pretty annoying to have to get your Python app to use
 the same hash function as your Java app or whatever I, unless you want
 to use something heavyweight like a cryptographic hash function. By
 doing it within Postgres, you avoid those headaches.
 
 I'm not asking you to very loudly proclaim that it should be used like
 this - just expose it, accurately document it, and I'm quite confident
 that it will be widely used and relied upon by those that are
 reasonably well informed, and understand its limitations, which are
 really quite straightforward.
 
  What I think people would actually like to know, if they're in a
  situation where distinct query texts are getting hashed to the same
  thing, is *which* different texts got hashed to the same thing.
  But there's no good way to expose that given the pg_stat_statements
  infrastructure, and exposing the hash value doesn't help.
 
 Apart from detecting the case where we get a straightforward
 collision, I don't expect that that would be useful. The whole point
 is that the user doesn't care about the difference, and I think we've
 specified a practical, widely useful standard for when queries should
 be considered equivalent.
 -- 
 Peter Geoghegan       http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training and Services
 

By using all 64-bits of the hash that we currently calculate, instead
of the current use of 32-bits only, the collision probabilities are
very low.

Regards,
Ken

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Last gasp

2012-04-11 Thread Tom Lane
Greg Smith g...@2ndquadrant.com writes:
 On 04/10/2012 09:14 PM, Robert Haas wrote:
 I wouldn't object to creating some doc-only committers.  OTOH, I would
 object to anyone making non-trivial documentation enhancements without
 posting their patches first and having a second person look it over,
 so how much difference is there, really?

 ...
 I'd like to dump around 50 pages of new material into the docs as a 
 start, but I don't want to take so much time away from the code oriented 
 committers to chew on that much.

Well, with all due respect, that does not sound like a change that
doesn't need review.

I have not noticed people adding docs-only changes to the CFs; usually
it's more like post a patch, somebody looks it over and commits it.
I agree that this is still too much overhead for simple fixes, such
as the editorial glitches that Thom Brown is so good at finding
(and I'm about ready to vote to give him a commit bit for that work).
But a fifty-page chunk is not that, indeed it sounds like it would have
enough technical content that it might actually merit a full-scale
review.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Last gasp

2012-04-11 Thread Peter Geoghegan
On 11 April 2012 03:26, Tom Lane t...@sss.pgh.pa.us wrote:
 [ scratches head... ]  That's supposed to be total lines of code in our
 source tree?  What's the big drop in late 2009, then?

I had wondered about that myself - all I can tell you is that I used
the tool as advertised, without any adornments. This particular tool
is a bit misleading, because it counts lines of code as lines of
checked-in text, which can include things that are very verbose
without necessarily being what we really think of as code - it doesn't
make any effort to discriminate against non-code, nor does it expose
the option of doing so. I chose it at short notice simply because it
produces graphs. I do intend to take a look at this problem in more
detail, and get better statistics on changes to our codebase - it's a
tricky proposition, though.

-- 
Peter Geoghegan       http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training and Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Last gasp

2012-04-11 Thread Magnus Hagander
On Wed, Apr 11, 2012 at 16:24, Tom Lane t...@sss.pgh.pa.us wrote:
 Greg Smith g...@2ndquadrant.com writes:
 On 04/10/2012 09:14 PM, Robert Haas wrote:
 I wouldn't object to creating some doc-only committers.  OTOH, I would
 object to anyone making non-trivial documentation enhancements without
 posting their patches first and having a second person look it over,
 so how much difference is there, really?

 ...
 I'd like to dump around 50 pages of new material into the docs as a
 start, but I don't want to take so much time away from the code oriented
 committers to chew on that much.

 Well, with all due respect, that does not sound like a change that
 doesn't need review.

 I have not noticed people adding docs-only changes to the CFs; usually
 it's more like post a patch, somebody looks it over and commits it.
 I agree that this is still too much overhead for simple fixes, such
 as the editorial glitches that Thom Brown is so good at finding
 (and I'm about ready to vote to give him a commit bit for that work).
 But a fifty-page chunk is not that, indeed it sounds like it would have
 enough technical content that it might actually merit a full-scale
 review.

Since the topic is somewhat drifting here anyway.. :-)

Might it be worthwhile to allow some sort of staging repository and
actually start using the git stuff a bit more around this? E.g. we
could have a docs repo somewhere where more people have commit bits,
and then they are just regularly merged back into the main tree? Sort
of the way different people can own different subsystems in other
projects, but someone does the trusted merge?

For example, Thom (and others) could collect a number of typo fixes in
their own repo and then just ask for a merge.The advantage over just
staging multiple commits and then submitting a patch would be that
multiple people could work on it...

-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] [JDBC] Regarding GSoc Application

2012-04-11 Thread Merlin Moncure
On Tue, Apr 10, 2012 at 10:41 PM, Atri Sharma atri.j...@gmail.com wrote:
Well. maybe I spoke too soon...JNI is probably the best route.  Since
SPI is off the table, all we're really pulling in from pl/java is the
(non-trivial) proper installation of a jvm into a postgres process.
pl/java is essentially a wrapper to JNI that does postgres to jni type
conversion and builds a database driven class lookup system that
allows building the java environment inside the database.   That part
is not needed at all; since we're not going to install our routines as
pl/java installed they will be installed into a separate and more
typical class path from a jvm point of view. There are no exposed
routines in pljava.so that allow for easy manual invocation of a java
routine -- it's all pretty much hardwired to the language function
call handler system.

This begs the question about why pl/java should be used at all. Well,
as I said the jvm insertion into postgres is non-trivial so that alone
is worth something.  Maybe though we should just rip out the jvm guts
that we really need (mostly some parts of backend.c and some of the
type files) and do a completely standalone jni wrapper.  pl/java's
build system is a mess anyways so we're not losing all that much in
trying to go off on our own and do something.  OTOH, throwing a build
system on top of the growing pile of things to do is turning this into
a daunting project vs the (admittedly cobbled together) approach I was
hoping to do earlier.  Anyways, it's really Atri's call if he's
comfortable proceeding.

 Hi All,

 I think we are back on the initial approach I proposed(hooking directly into
 the JVM and executing Java code that calls JDBC).I think the best way to do
 this is create a JVM that executes the Java code and give the control of the
 JVM to the native API.

 I agree,the only need of Pl/Java that is apparent here is the need of the
 Java internals(JDK et al).If we set them up independently,then,we can have
 the FDW wrapping JDBC directly through JNI.JNI would call pure Java
 functions to connect to the JDBC.

 I think we can proceed with this.Once we are done with the API calling Java
 functions,I think the rest of the path is easily mapped(writing Java
 functions to connect to JDBC).

yeah -- it sound plausible.  I think the next step is to pull one of
the fdw projects that is working and currently builds with pgxs. Make
sure it is bsd licensed and that you retain the original copyright in
the code (feel free to add your own).   The mysql_fdw project for
example is a good candidate.  After that we need to strip out all the
mysql specific parts so that we have a basic skeleton of the fdw
wrapper that still builds.   From there we will need to build in the
jni calls as well as jvm initialization code we are going to more or
less directly copy from pl/java, but one thing at a time:  IMO,
getting the jvm installed and executing a 'hello world' jni is a good
milestone to reach before getting into all the mechanics of the remote
querying.

The jvm initialization code in pl/java we are going to use is in
src/C/pljava/backend.c.  A lot of the initialization work done there
is irrelevant to what we are trying to do but there are some very
important bits like the on_proc_exit handler that does cleanup when
the postgres process exits.  Ideally we can drop just the stuff we
need from there and get our project to compile.

merlin

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] [GENERAL] [streaming replication] 9.1.3 streaming replication bug ?

2012-04-11 Thread Michael Nolan
On 4/11/12, 乔志强 qiaozhiqi...@leadcoretech.com wrote:

 Yes, increase wal_keep_segments. Even if you set wal_keep_segments to 64,
 the amount of disk space for WAL files is only 1GB, so there is no need to
 worry so much, I think. No?

 But when a transaction larger than 1GB...

Then you may need WAL space larger than 1GB as well.  For replication to work,
it seems likely that you may need to have sufficient WAL space to
handle a row, possibly the entire transaction..  But since a single
statement can update thousands or millions of rows, do you always need
enough WAL space to hold the entire transaction?

 So in sync streaming replication, if master delete WAL before sent to the
 only standby, all transaction will fail forever,
 the master tries to avoid a PANIC error rather than termination of
 replication. but in sync replication, termination of replication is THE
 bigger PANIC error.

That's somewhat debatable.  Would I rather have a master that PANICED or
a slave that lost replication?  I would choose the latter.   A third
option, which
may not even be feasible, would be to have the master fail the
transaction if synchronous replication cannot be achieved, although
that might have negative consequences as well.

 Another question:
   Does master send WAL to standby before the transaction commit ?

That's another question for the core team, I suspect.  A related
question is what happens
if there is a rollback?
--
Mike Nolan

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] [JDBC] Regarding GSoc Application

2012-04-11 Thread Atri Sharma
Hi all,

In continuation to the discussion regarding my JDBC wrapping FDW,I have been
talking to members of the community and I have two approaches on which I
would request your suggestions and opinions:



I think we are back on the initial approach I proposed(hooking directly
into the JVM and executing Java code that calls JDBC).
I think the best way to do this is create a JVM that executes the Java code
and give the control of the JVM to the native API.

I agree,the only need of Pl/Java that is apparent here is the need of the
Java internals(JDK et al).If we set them up independently,then,we can have
the FDW wrapping JDBC directly through JNI.JNI would call pure Java
functions to connect to the JDBC.

In the above context,I think if we directly wrap JNI and call pure Java
routines(not using Pl/Java codes directly),we can build something on the
lines of Multicorn(a nice suggestion by Hitoshi) http://multicorn.org/

On the other hand,as suggested by John,we can use the Pl/Java routines that
already exist to build our FDW's API.This shall be the flow:


user application - SQL - PLjava FDW functions - pljava code - JDBC 
- foreign database

The user will have to pass security levels and permissions before being able
to get to the FDW.This would look to the user very much like dblink looks
now, and have similar security issues and mitigations.

Please let me know how to proceed further.

Atri



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] [JDBC] Regarding GSoc Application

2012-04-11 Thread Atri Sharma

On Tue, Apr 10, 2012 at 10:41 PM, Atri Sharma atri.j...@gmail.com wrote:
Well. maybe I spoke too soon...JNI is probably the best route.  Since
SPI is off the table, all we're really pulling in from pl/java is the
(non-trivial) proper installation of a jvm into a postgres process.
pl/java is essentially a wrapper to JNI that does postgres to jni type
conversion and builds a database driven class lookup system that
allows building the java environment inside the database.   That part
is not needed at all; since we're not going to install our routines as
pl/java installed they will be installed into a separate and more
typical class path from a jvm point of view. There are no exposed
routines in pljava.so that allow for easy manual invocation of a java
routine -- it's all pretty much hardwired to the language function
call handler system.

This begs the question about why pl/java should be used at all. Well,
as I said the jvm insertion into postgres is non-trivial so that alone
is worth something.  Maybe though we should just rip out the jvm guts
that we really need (mostly some parts of backend.c and some of the
type files) and do a completely standalone jni wrapper.  pl/java's
build system is a mess anyways so we're not losing all that much in
trying to go off on our own and do something.  OTOH, throwing a build
system on top of the growing pile of things to do is turning this into
a daunting project vs the (admittedly cobbled together) approach I was
hoping to do earlier.  Anyways, it's really Atri's call if he's
comfortable proceeding.

 Hi All,

 I think we are back on the initial approach I proposed(hooking directly
into
 the JVM and executing Java code that calls JDBC).I think the best way to
do
 this is create a JVM that executes the Java code and give the control of
the
 JVM to the native API.

 I agree,the only need of Pl/Java that is apparent here is the need of the
 Java internals(JDK et al).If we set them up independently,then,we can
have
 the FDW wrapping JDBC directly through JNI.JNI would call pure Java
 functions to connect to the JDBC.

 I think we can proceed with this.Once we are done with the API calling
Java
 functions,I think the rest of the path is easily mapped(writing Java
 functions to connect to JDBC).

yeah -- it sound plausible.  I think the next step is to pull one of
the fdw projects that is working and currently builds with pgxs. Make
sure it is bsd licensed and that you retain the original copyright in
the code (feel free to add your own).   The mysql_fdw project for
example is a good candidate.  After that we need to strip out all the
mysql specific parts so that we have a basic skeleton of the fdw
wrapper that still builds.   From there we will need to build in the
jni calls as well as jvm initialization code we are going to more or
less directly copy from pl/java, but one thing at a time:  IMO,
getting the jvm installed and executing a 'hello world' jni is a good
milestone to reach before getting into all the mechanics of the remote
querying.

The jvm initialization code in pl/java we are going to use is in
src/C/pljava/backend.c.  A lot of the initialization work done there
is irrelevant to what we are trying to do but there are some very
important bits like the on_proc_exit handler that does cleanup when
the postgres process exits.  Ideally we can drop just the stuff we
need from there and get our project to compile.


I agree,atm I'll work on getting JNI set up and to build a JVM using it and
getting the Hello,World running in it in PostGreSQl.

I'll keep you posted,

Atri


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Last gasp

2012-04-11 Thread Andrew Dunstan



On 04/10/2012 08:43 PM, Greg Smith wrote:

On 04/10/2012 01:28 PM, Robert Haas wrote:

The fact is that we have no shortage of committers - there are 19
people who have access to push code into our master git repository.  A
handful of those people have basically completely left the project and
their commit rights should probably be revoked on that basis; most of
them are still involved in one way or another but just not able to
devote a lot of time to reviewing other people's code.


Let's use realistic numbers here:  I count 7 people who regularly 
review and commit other people's code in a variety of areas.  There 
are also several subject matter experts who commit things in a 
relatively narrow area.  But most bigger patches are going to hit a 
bottleneck whose capacity could be measured in 3 bits.



Robert's point is fairly important. A few years ago one of the Postgres 
companies hired me for a few weeks to push along some of the stuff that 
was holding up a release at a critical time. (This was in the pre CF 
days.) Several important features made it into that release that almost 
certainly would not have otherwise. But generally that hasn't happened, 
and the time I've devoted has been my own, and thus subject to competing 
demands of business and the mythical life AFK. Sometimes that makes it 
quite hard to arrange time to spend on reviewing and committing large 
patches.






There are a significant number of companies who are willing to sponsor 
committers to open-source projects; there are almost none who will 
sponsor reviewers or contributors of any stature unless they're 
already deep into the PostgreSQL community.  That's one of the many 
reasons it's easier for a committer to attract funding for core 
PostgreSQL work, be it in the form of a full-time job or 
project-oriented funding.  The corresponding flip side to that is that 
the small number of committers is limiting the scope of funding the 
project can accumulate.






Yep. The plus side is that Postgres business seems to be growing quite 
rapidly (if our experience at PostgreSQL Experts Inc is anything to go 
by), so maybe there will be more sources of funding available. More work 
needs to be done to generate funds to support getting features reviewed 
and committed. Patches seems to be getting ever larger and more complex, 
and that makes reviewing them harder and much more time consuming.



cheers

andrew



--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Last gasp

2012-04-11 Thread Alvaro Herrera

Excerpts from Magnus Hagander's message of mié abr 11 11:35:10 -0300 2012:

 For example, Thom (and others) could collect a number of typo fixes in
 their own repo and then just ask for a merge.The advantage over just
 staging multiple commits and then submitting a patch would be that
 multiple people could work on it...

The other advantage is that committers could have that tree as another
remote in their main PG tree, and so changesets could be pulled into the
same clone and cherry-picked into the master branch.

(I'd also set up a clone for Alex Hunsaker to commit to pl/perl).

-- 
Álvaro Herrera alvhe...@commandprompt.com
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] [GENERAL] [streaming replication] 9.1.3 streaming replication bug ?

2012-04-11 Thread Kevin Grittner
Michael Nolan htf...@gmail.com wrote:
 On 4/11/12, 乔志强 qiaozhiqi...@leadcoretech.com wrote:
 
 But when a transaction larger than 1GB...
 
 Then you may need WAL space larger than 1GB as well.  For
 replication to work, it seems likely that you may need to have
 sufficient WAL space to handle a row, possibly the entire
 transaction..  But since a single statement can update thousands
 or millions of rows, do you always need enough WAL space to hold
 the entire transaction?
 
No.
 
   Does master send WAL to standby before the transaction commit ?
 
Yes.
 
 A related question is what happens if there is a rollback?
 
PostgreSQL doesn't use a rollback log; WAL files can be reclaimed as
soon as the work they represent has been persisted to the database
by a CHECKPOINT, even if it is not committed.  Because there can be
multiple versions of each row in the base table, each with its own
xmin (telling which transaction committed it) and xmax (telling
which transaction expired it) visibiliity checking can handle the
commits and rollbacks correctly.  It also uses a commit log (CLOG),
hint bits, and other structures to help resolve visibility.  It is a
complex topic, but it does work.
 
-Kevin

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Last gasp

2012-04-11 Thread Robert Haas
On Wed, Apr 11, 2012 at 10:35 AM, Magnus Hagander mag...@hagander.net wrote:
 Since the topic is somewhat drifting here anyway.. :-)

 Might it be worthwhile to allow some sort of staging repository and
 actually start using the git stuff a bit more around this? E.g. we
 could have a docs repo somewhere where more people have commit bits,
 and then they are just regularly merged back into the main tree? Sort
 of the way different people can own different subsystems in other
 projects, but someone does the trusted merge?

 For example, Thom (and others) could collect a number of typo fixes in
 their own repo and then just ask for a merge.The advantage over just
 staging multiple commits and then submitting a patch would be that
 multiple people could work on it...

If our goal is to give people more or less unfettered access to
certain areas of the tree, but not the whole thing, we should perhaps
consider just doing that directly.  There's no particular reason why
Thom Brown can't be made a committer just for docs, or why Alex
Hunsaker can't be made a committer just for PL/perl (and presumably
docs, since he'd need to update the docs if he updates the code), if
that's actually what we want to do.

Now, the advantage of a staging tree is that it gives the people who
have commit rights to the main repository the ability to decline to
merge.  The question is - what happens then, especially given that we
have so many committers already?  In Linux-land, it becomes the
subsystem maintainer's responsibility to put the tree into a state
where Linus will again become willing to merge it, or he can fire the
subsystem maintainer and pick a new one that'll do what he wants.  But
we don't work that way.  Instead, the committers as a group have the
responsibility for not breaking stuff.  So who would decide whether to
do the merge, and who would be responsible for fixing things if the
merge were refused?  As far as I can see, this basically amounts to
bundling lots of unrelated changes into one big pile and then asking
to have them all committed at once instead of one at a time, which
sounds like more work not less, unless we're just going to blindly
merge without reviewing, in which case we may as well just let people
commit to the main repository themselves.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Last gasp

2012-04-11 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes:
 On Wed, Apr 11, 2012 at 10:35 AM, Magnus Hagander mag...@hagander.net wrote:
 Might it be worthwhile to allow some sort of staging repository and
 actually start using the git stuff a bit more around this?

 ... As far as I can see, this basically amounts to
 bundling lots of unrelated changes into one big pile and then asking
 to have them all committed at once instead of one at a time, which
 sounds like more work not less, unless we're just going to blindly
 merge without reviewing, in which case we may as well just let people
 commit to the main repository themselves.

Yeah.  I'm also worried that we would either lose linear history, or
(if we squash the merge commits) lose change log history for the
individual fixes.  Neither of those sounds terribly attractive.

We've frequently had, and still have today, committers who are
understood to have limited areas of expertise and are given commit
bits on the honor system to not break what they don't know well.
I don't have any problem with continuing in that line.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Last gasp

2012-04-11 Thread Robert Haas
On Wed, Apr 11, 2012 at 11:38 AM, Tom Lane t...@sss.pgh.pa.us wrote:
 Robert Haas robertmh...@gmail.com writes:
 On Wed, Apr 11, 2012 at 10:35 AM, Magnus Hagander mag...@hagander.net 
 wrote:
 Might it be worthwhile to allow some sort of staging repository and
 actually start using the git stuff a bit more around this?

 ... As far as I can see, this basically amounts to
 bundling lots of unrelated changes into one big pile and then asking
 to have them all committed at once instead of one at a time, which
 sounds like more work not less, unless we're just going to blindly
 merge without reviewing, in which case we may as well just let people
 commit to the main repository themselves.

 Yeah.  I'm also worried that we would either lose linear history, or
 (if we squash the merge commits) lose change log history for the
 individual fixes.  Neither of those sounds terribly attractive.

 We've frequently had, and still have today, committers who are
 understood to have limited areas of expertise and are given commit
 bits on the honor system to not break what they don't know well.
 I don't have any problem with continuing in that line.

Me neither, but I don't know how far it scales.  Having certain people
who are defined as, say, doc-only committers will not only make it
clear to those people what they're expected to commit, but also clear
to everyone else who the people are who might commit any given patch
they might write.  If we just end up with 50 committers and you have
to follow pgsql-hackers to understand who knows what and which people
are even still around, it's not going to make anything easier for
anyone.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Last gasp

2012-04-11 Thread Alvaro Herrera

Excerpts from Robert Haas's message of mié abr 11 12:44:02 -0300 2012:

 Me neither, but I don't know how far it scales.  Having certain people
 who are defined as, say, doc-only committers will not only make it
 clear to those people what they're expected to commit, but also clear
 to everyone else who the people are who might commit any given patch
 they might write.  If we just end up with 50 committers and you have
 to follow pgsql-hackers to understand who knows what and which people
 are even still around, it's not going to make anything easier for
 anyone.

Since we're so keen on copying what Linux does, we could just have a
MAINTAINERS file.

-- 
Álvaro Herrera alvhe...@commandprompt.com
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] [GENERAL] [streaming replication] 9.1.3 streaming replication bug ?

2012-04-11 Thread Michael Nolan
On 4/11/12, Kevin Grittner kevin.gritt...@wicourts.gov wrote:
 Michael Nolan htf...@gmail.com wrote:
 On 4/11/12, 乔志强 qiaozhiqi...@leadcoretech.com wrote:

 But when a transaction larger than 1GB...

 Then you may need WAL space larger than 1GB as well.  For
 replication to work, it seems likely that you may need to have
 sufficient WAL space to handle a row, possibly the entire
 transaction..  But since a single statement can update thousands
 or millions of rows, do you always need enough WAL space to hold
 the entire transaction?

 No.

   Does master send WAL to standby before the transaction commit ?

 Yes.

 A related question is what happens if there is a rollback?

 PostgreSQL doesn't use a rollback log; WAL files can be reclaimed as
 soon as the work they represent has been persisted to the database
 by a CHECKPOINT, even if it is not committed.  Because there can be
 multiple versions of each row in the base table, each with its own
 xmin (telling which transaction committed it) and xmax (telling
 which transaction expired it) visibiliity checking can handle the
 commits and rollbacks correctly.  It also uses a commit log (CLOG),
 hint bits, and other structures to help resolve visibility.  It is a
 complex topic, but it does work.

Thanks, Kevin.  That does lead to a question about the problem that
started this thread, though.  How does one determine how big the WAL
space needs to be to not cause streaming replication to fail?  Or
maybe this is a bug after all?
--
Mike Nolan

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Last gasp

2012-04-11 Thread Joshua Berkus
All,

From my observation, the CF process ... in fact, all development processes 
we've had in Postgres ... have suffered from only one problem: lack of 
consensus on how the process should work.  For example, we've *never* had 
consensus around the criteria for kicking a patch out of a commitfest.  This 
lack of consensus has resulted in disorganization, ennui towards the process, 
deadline overruns, and a lot of general unhappiness.   People have stopped 
believing in the CF system because we've stopped running it.

I'm encouraged at this point that we've seen where this lack of consensus can 
lead us, maybe at this point we're willing to set aside individual differences 
of opinion on what the criteria should be (especially when it comes to the 
patches we each individually care about) in service of a smoother-running 
process.  Some suggestions:

- for the first 2 weeks of each CF, there should be a *total* moritorium on 
discussing any features not in the current CF on -hackers.
- the CF manager should have unquestioned authority to kick patches.  As in, no 
arguing.
- we should have simple rules for the CF manager for kicking patches, as in:
   * no response from author in 5 days
   * judged as needing substantial work by reviewer
   * feature needs spec discussion

However, the real criteria don't matter as much as coming up with a set of 
criteria we're all willing to obey, whatever they are.

We also need better tools for the CF, but frankly better tools is a minor issue 
and easily solved if we have a consensus which people are willing to obey.  For 
that matter, if we have a smooth and impartial process, we can do other things, 
including: training new reviewers, promoting new committers, changing the 
length of the CF cycle, or changing the PostgreSQL release cycle (yes, really). 
 While our review and commit process is completely subjective and inconsistent, 
though, we can't do any of these things.

--Josh Berkus


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] [GENERAL] [streaming replication] 9.1.3 streaming replication bug ?

2012-04-11 Thread Fujii Masao
On Wed, Apr 11, 2012 at 3:31 PM, 乔志强 qiaozhiqi...@leadcoretech.com wrote:
 So in sync streaming replication, if master delete WAL before sent to the 
 only standby, all transaction will fail forever,
 the master tries to avoid a PANIC error rather than termination of 
 replication. but in sync replication, termination of replication is THE 
 bigger PANIC error.

I see your point. When there are backends waiting for replication, the WAL files
which the standby might not have received yet must not be removed. If they are
removed, replication keeps failing forever because required WAL files don't
exist in the master, and then waiting backends will never be released unless
replication mode is changed to async. This should be avoided.

To fix this issue, we should prevent the master from deleting the WAL files
including the minimum waiting LSN or bigger ones. I'll think more and implement
the patch.

Regards,

-- 
Fujii Masao

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Last gasp

2012-04-11 Thread Peter Geoghegan
On 11 April 2012 15:35, Magnus Hagander mag...@hagander.net wrote:
 Might it be worthwhile to allow some sort of staging repository and
 actually start using the git stuff a bit more around this? E.g. we
 could have a docs repo somewhere where more people have commit bits,
 and then they are just regularly merged back into the main tree? Sort
 of the way different people can own different subsystems in other
 projects, but someone does the trusted merge?

 For example, Thom (and others) could collect a number of typo fixes in
 their own repo and then just ask for a merge.The advantage over just
 staging multiple commits and then submitting a patch would be that
 multiple people could work on it...

This is hardly a radical idea at all - it's basically how git was
really intended to be used at scale. Of course, unless some committer
is going to make it their responsibility to merge those commits say
every 3 months, there's no point in bothering. This could consolidate
the number of typo commits to boot, as they could be rebased. TBH, I
find it slightly embarrassing to have to ask a committer to fix a
minor typo, and it's hardly reasonable to expect me to save my typos
up.

Big +1 from me.

It might be the case that over time, we become comfortable with this
approach and upgrade the tree to a linux-next style tree (much like
the  -mm tree was repurposed into the progenitor of linux-next), with
a lesser (though still substantial) standard for committers to meet.
There could be an understanding that by committing to the tree, the
developer makes a representation that they are confident that the
feature is ready for prime-time, in just the same way that a commit
currently represents - don't underestimate the power of that ceremony.
Less senior contributors could have their work scrutinised by a wider
body of people that haven't necessarily taken enough of an interest in
the contributor's work to want to follow them on github or whatever -
enthusiast power users who wouldn't really consider themselves
hackers.

This partially solves the you want us to fund feature development but
you're not even a committer? problem that Greg referred to. It's also
how big projects scale - technically, there are relatively few
committers to the linux-stable tree.

This approach formalises Tom's view that I think the key point here
is that people have to expect that it's going to take more than one
round of review to land most nontrivial patches.

On 11 April 2012 16:27, Robert Haas robertmh...@gmail.com wrote:
 Now, the advantage of a staging tree is that it gives the people who
 have commit rights to the main repository the ability to decline to
 merge.  The question is - what happens then, especially given that we
 have so many committers already?  In Linux-land, it becomes the
 subsystem maintainer's responsibility to put the tree into a state
 where Linus will again become willing to merge it, or he can fire the
 subsystem maintainer and pick a new one that'll do what he wants.  But
 we don't work that way.  Instead, the committers as a group have the
 responsibility for not breaking stuff.  So who would decide whether to
 do the merge, and who would be responsible for fixing things if the
 merge were refused?

This seems like a non-issue to me. We just try and match the Linux
model. In practice I'd imagine that the roles would not really be
perfectly delineated like that, and I find it really doubtful that
they are in Linux land either. There was a time when some people were
somewhat skeptical of the git migration (at least privately), and that
was, as far as I can tell, a roaring success.
-- 
Peter Geoghegan       http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training and Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Last gasp

2012-04-11 Thread Kevin Grittner
Tom Lane t...@sss.pgh.pa.us wrote:
 
 What I'd be interested to see is number of lines changed per unit
 time; that would be a much better measure of patch rate IMHO.
 
Based on `git diff --shortstat` between tags, for the whole tree,
this is what shows up:
 
   files
git tag  changed  insertions  deletions
===  ===  ==  =
PG95-1_01
Release_1_0_2133   10373   1046
REL2_0  1079   65115  41987
REL6_1   770   70656  42995
REL6_2   919  195720 141153
REL6_3  1140  104845  34326
REL6_4_21583  171053  69832
REL6_5  1366  136665  85013
REL7_0  1991  272086 142201
REL7_1  2164  441090 212481
REL7_2  1826  314583 139238
REL7_3  2258  631434 245295
REL7_4  2018  436323 283074
REL8_0_02398  446392 258825
REL8_1_01796  235680 181193
REL8_2_02343  288467 168438
REL8_3_02355  405974 174798
REL8_4_02464  339085 462065
REL9_0_02270  292187 191852
REL9_1_03225  443037 186042
HEAD2014  189648 94
 
-Kevin

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] [GENERAL] [streaming replication] 9.1.3 streaming replication bug ?

2012-04-11 Thread Michael Nolan
On 4/11/12, Fujii Masao masao.fu...@gmail.com wrote:
 On Wed, Apr 11, 2012 at 3:31 PM, 乔志强 qiaozhiqi...@leadcoretech.com wrote:
 So in sync streaming replication, if master delete WAL before sent to the
 only standby, all transaction will fail forever,
 the master tries to avoid a PANIC error rather than termination of
 replication. but in sync replication, termination of replication is THE
 bigger PANIC error.

 I see your point. When there are backends waiting for replication, the WAL
 files
 which the standby might not have received yet must not be removed. If they
 are
 removed, replication keeps failing forever because required WAL files don't
 exist in the master, and then waiting backends will never be released unless
 replication mode is changed to async. This should be avoided.

 To fix this issue, we should prevent the master from deleting the WAL files
 including the minimum waiting LSN or bigger ones. I'll think more and
 implement
 the patch.

With asynchonous replication, does the master even know if a slave
fails because of a WAL problem?  And does/should it care?

Isn't there a separate issue with synchronous replication?  If it
fails, what's the appropriate action to take on the master?  PANICing
it seems to be a bad idea, but having transactions never complete
because they never hear back from the synchronous slave (for whatever
reason) seems bad too.
--
Mike Nolan

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Last gasp

2012-04-11 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes:
 On Wed, Apr 11, 2012 at 11:38 AM, Tom Lane t...@sss.pgh.pa.us wrote:
 We've frequently had, and still have today, committers who are
 understood to have limited areas of expertise and are given commit
 bits on the honor system to not break what they don't know well.
 I don't have any problem with continuing in that line.

 Me neither, but I don't know how far it scales.  Having certain people
 who are defined as, say, doc-only committers will not only make it
 clear to those people what they're expected to commit, but also clear
 to everyone else who the people are who might commit any given patch
 they might write.  If we just end up with 50 committers and you have
 to follow pgsql-hackers to understand who knows what and which people
 are even still around, it's not going to make anything easier for
 anyone.

When and if we have 50 committers, we can worry about that problem ;-).
But in practice people have different areas of expertise already.
There was some mumbling upthread about trying to identify early who
would be the responsible committer for any given CF submission ---
doesn't seem to me like that is much different.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Last gasp

2012-04-11 Thread Josh Kupershmidt
On Wed, Apr 11, 2012 at 8:59 AM, Peter Geoghegan pe...@2ndquadrant.com wrote:
 On 11 April 2012 15:35, Magnus Hagander mag...@hagander.net wrote:

 For example, Thom (and others) could collect a number of typo fixes in
 their own repo and then just ask for a merge.The advantage over just
 staging multiple commits and then submitting a patch would be that
 multiple people could work on it...

 This is hardly a radical idea at all - it's basically how git was
 really intended to be used at scale. Of course, unless some committer
 is going to make it their responsibility to merge those commits say
 every 3 months, there's no point in bothering. This could consolidate
 the number of typo commits to boot, as they could be rebased. TBH, I
 find it slightly embarrassing to have to ask a committer to fix a
 minor typo, and it's hardly reasonable to expect me to save my typos
 up.

 Big +1 from me.

Particularly for the docs, it'd be nice to have more committer
bandwidth available, if there's a reasonable way to do so without
causing needless merge work for existing committers. Like Peter, I
hate to bother busy committers with trivial typofixes, and sometimes I
just don't bother sending such changes in, and they get lost :-(

Maybe keeping doc/ as a 'git submodule' could work? Or, as Tom
suggests, adding a global committer who could focus on docs changes
would effectively solve the problem as well.

Josh

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] [Patch] Fix little typo in a comment

2012-04-11 Thread Tom Lane
Etsuro Fujita fujita.ets...@lab.ntt.co.jp writes:
 This is a little patch to fix a typo in contrib/file_fdw.

I think that comment is fine as-is.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] [GENERAL] [streaming replication] 9.1.3 streaming replication bug ?

2012-04-11 Thread Fujii Masao
On Thu, Apr 12, 2012 at 12:56 AM, Fujii Masao masao.fu...@gmail.com wrote:
 On Wed, Apr 11, 2012 at 3:31 PM, 乔志强 qiaozhiqi...@leadcoretech.com wrote:
 So in sync streaming replication, if master delete WAL before sent to the 
 only standby, all transaction will fail forever,
 the master tries to avoid a PANIC error rather than termination of 
 replication. but in sync replication, termination of replication is THE 
 bigger PANIC error.

 I see your point. When there are backends waiting for replication, the WAL 
 files
 which the standby might not have received yet must not be removed. If they are
 removed, replication keeps failing forever because required WAL files don't
 exist in the master, and then waiting backends will never be released unless
 replication mode is changed to async. This should be avoided.

On second thought, we can avoid the issue by just increasing
wal_keep_segments enough. Even if the issue happens and some backends
get stuck to wait for replication, we can release them by taking fresh backup
and restarting the standby from that backup. This is the basic procedure to
restart replication after replication is terminated because required WAL files
are removed from the master. So this issue might not be worth implementing
the patch for now (though I'm not against improving things in the future), but
it seems just a tuning-problem of wal_keep_segments.

Regards,

-- 
Fujii Masao

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Last gasp

2012-04-11 Thread Magnus Hagander
On Wed, Apr 11, 2012 at 18:29, Josh Kupershmidt schmi...@gmail.com wrote:
 On Wed, Apr 11, 2012 at 8:59 AM, Peter Geoghegan pe...@2ndquadrant.com 
 wrote:
 On 11 April 2012 15:35, Magnus Hagander mag...@hagander.net wrote:

 For example, Thom (and others) could collect a number of typo fixes in
 their own repo and then just ask for a merge.The advantage over just
 staging multiple commits and then submitting a patch would be that
 multiple people could work on it...

 This is hardly a radical idea at all - it's basically how git was
 really intended to be used at scale. Of course, unless some committer
 is going to make it their responsibility to merge those commits say
 every 3 months, there's no point in bothering. This could consolidate
 the number of typo commits to boot, as they could be rebased. TBH, I
 find it slightly embarrassing to have to ask a committer to fix a
 minor typo, and it's hardly reasonable to expect me to save my typos
 up.

 Big +1 from me.

 Particularly for the docs, it'd be nice to have more committer
 bandwidth available, if there's a reasonable way to do so without
 causing needless merge work for existing committers. Like Peter, I
 hate to bother busy committers with trivial typofixes, and sometimes I
 just don't bother sending such changes in, and they get lost :-(

 Maybe keeping doc/ as a 'git submodule' could work? Or, as Tom
 suggests, adding a global committer who could focus on docs changes
 would effectively solve the problem as well.

git submodule would be a really bad idea imho. Then you couldn't make
a single commit that deals with both code and docs.

-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Last gasp

2012-04-11 Thread Tom Lane
Joshua Berkus j...@agliodbs.com writes:
 From my observation, the CF process ... in fact, all development
 processes we've had in Postgres ... have suffered from only one
 problem: lack of consensus on how the process should work.  For
 example, we've *never* had consensus around the criteria for kicking
 a patch out of a commitfest.

True, but put that decision entirely in the hands of the CF manager
doesn't seem to me to be a workable solution.  Half the time we
don't even have a CF manager, AFAICT.  Now admittedly the opportunity
to wield absolute power might attract more interest in the position ;-)
but I don't think we want people who are attracted by that.

 Some suggestions:

 - for the first 2 weeks of each CF, there should be a *total* moritorium on 
 discussing any features not in the current CF on -hackers.

We've tried that in the past, and it's never been adhered to very well,
and I think it's folly to assume that we'll get much better at it.
The nature of a mailing list is that there's a lot of noise.  Even if
95% of the membership knows about and agrees with the restriction, the
other 5% will still post about non-CF stuff.

 - we should have simple rules for the CF manager for kicking patches, as in:
* no response from author in 5 days
* judged as needing substantial work by reviewer
* feature needs spec discussion

These rules still seem to me to require a lot of judgment, hence
opportunity for argument.  What's substantial work?  How big a quibble
about the spec is big enough to get a patch booted?

 However, the real criteria don't matter as much as coming up with a set of 
 criteria we're all willing to obey, whatever they are.

Ultimately, we're herding cats here.  I don't think you're going to get
the community to suddenly be willing to march in lockstep instead.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Last gasp

2012-04-11 Thread Robert Haas
On Wed, Apr 11, 2012 at 11:49 AM, Alvaro Herrera
alvhe...@commandprompt.com wrote:
 Excerpts from Robert Haas's message of mié abr 11 12:44:02 -0300 2012:
 Me neither, but I don't know how far it scales.  Having certain people
 who are defined as, say, doc-only committers will not only make it
 clear to those people what they're expected to commit, but also clear
 to everyone else who the people are who might commit any given patch
 they might write.  If we just end up with 50 committers and you have
 to follow pgsql-hackers to understand who knows what and which people
 are even still around, it's not going to make anything easier for
 anyone.

 Since we're so keen on copying what Linux does, we could just have a
 MAINTAINERS file.

I'm actually not particularly keen on copying what Linux does.  It
seems that they have a lot of the same problems that we have, only
worse.  Unless you can get the proper person to take notice of your
patch and merge it into his tree, from where it will get merged into
somebody else's tree, from where it will eventually get merged into
Linus's tree (maybe with one more tree in the path just for good
measure), your patch is just going to fall on the floor and die.  The
lseek contention stuff in Linux 3.2 was submitted in substantially
identical form years ago and didn't get merged - mostly, AFAICT, just
because nobody cared about it enough.

Our process is not perfect, but in recent years we have at least done
a fairly good job preventing things from dying of apathy, even if not
everybody agrees on which things ultimately should or should not have
gotten committed.  Small doc changes might be an exception.  I used to
apply those regularly, but I've had to fall back to making occasional
sweeps of my pgsql-docs email box and looking for unresolved issues.
Having a few more committers, and specifically people focused on
documentation, would, I think, be a a step forward.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Last gasp

2012-04-11 Thread Peter Eisentraut
On ons, 2012-04-11 at 06:04 -0400, Greg Smith wrote:
 Compare with:
 
 -Submitter suggests doc change
 -No one has a strong opinion on it, may not be picked up at all
 -Submitter adds to the next CF
 -Wait for review
 -[Possible repost update with reviewer changes]
 -Ready for committer
 -Committer takes time away from code review to look at it
 -Possibly another feedback/review resubmission
 -Commit final versions

I totally get that.

Just as a personal view, if people were to send me doc or trivial
patches in git-am format, with proper commit message, and Acked or
Signed-off etc. lines from recognized contributors, and proper
References: mail header linked to the discussion or suggestion
message, I could probably commit 20 of those in an hour.

Instead, I have to review the entire email thread for discussion, any
possible reviews or test runs, extract the patch from the email, mangle
it into proper form, apply it, think of a commit message, make sure I
register all the right people in the message, re-review the commit,
push, reply to email, optionally, log into commit fest, find the patch,
click a bunch of times, close it, done -- I think.  That takes 15
minutes per patch, and after two patches like that I'm tired.



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Last gasp

2012-04-11 Thread Joshua Berkus

 Ultimately, we're herding cats here.  I don't think you're going to
 get
 the community to suddenly be willing to march in lockstep instead.

If you, Peter, Simon, Robert, Heikki, Magnus, Peter G., Greg, Bruce and Andrew 
agreed on a calendar-driven, mostly unambiguous process and adhered to that 
process, then the one or two people who didn't follow along wouldn't matter.  
Everyone else would follow you.  The reason things are chaotic now is that our 
lead committers do not have consensus and are even inconsistent from CF to CF 
individually.

In other words: the problem is only unsolvable because *you* think it's 
unsolvable.   If you decide the problem is solvable, you already have the means 
to solve it.

--Josh Berkus

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Last gasp

2012-04-11 Thread Peter Eisentraut
On ons, 2012-04-11 at 12:48 -0400, Tom Lane wrote:
  However, the real criteria don't matter as much as coming up with a
 set of criteria we're all willing to obey, whatever they are.
 
 Ultimately, we're herding cats here.  I don't think you're going to
 get the community to suddenly be willing to march in lockstep
 instead. 

Yeah, I think what's forgotten in this discussion is that we're all
volunteers.  (Even those who have jobs related to this are still
volunteers with respect to the project.)  So people will do whatever
they want.  If you don't let them do what they want, they will do it
elsewhere.  Now it's fine to encourage a little bit of team spirit and
community values here and there, but you can't build your process on
making people do what they don't want to do.



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] patch: bytea_agg

2012-04-11 Thread Peter Eisentraut
On lör, 2012-04-07 at 10:38 -0400, Tom Lane wrote:
  Nevertheless, the problem would now be that adding string_agg(bytea)
  would effectively forbid adding string_agg(bytea, delim) in the
 future.
  So making a two-argument string_agg(bytea, bytea) now seems like the
  best solution anyway.  (This applies independently of the function
  renaming, actually.)
 
 Hm.  So are you now suggesting we should get rid of one-argument
 bytea_agg and replace it with two-argument string_agg(bytea,bytea)?
 I could support that, since we've not released bytea_agg yet.

Yes, that looks like the best solution.  Here is a patch for that.
diff --git i/doc/src/sgml/func.sgml w/doc/src/sgml/func.sgml
index ae22ee5..131384c 100644
--- i/doc/src/sgml/func.sgml
+++ w/doc/src/sgml/func.sgml
@@ -10965,24 +10965,6 @@ SELECT NULLIF(value, '(none)') ...
  row
   entry
indexterm
-primarybytea_agg/primary
-   /indexterm
-   function
- bytea_agg(replaceable class=parameterexpression/replaceable)
-   /function
-  /entry
-  entry
-   typebytea/type
-  /entry
-  entry
-   typebytea/type
-  /entry
-  entryinput values concatenated into a bytea/entry
- /row
-
- row
-  entry
-   indexterm
 primarycount/primary
/indexterm
functioncount(*)/function
@@ -11061,7 +11043,7 @@ SELECT NULLIF(value, '(none)') ...
/function
   /entry
   entry
-   typetext/type, typetext/type
+   typetext/type, typetext/type or typebytea/type, typebytea/type
   /entry
   entry
typetext/type
diff --git i/src/backend/utils/adt/varlena.c w/src/backend/utils/adt/varlena.c
index 65e9af8..a5592d5 100644
--- i/src/backend/utils/adt/varlena.c
+++ w/src/backend/utils/adt/varlena.c
@@ -397,7 +397,7 @@ byteasend(PG_FUNCTION_ARGS)
 }
 
 Datum
-bytea_agg_transfn(PG_FUNCTION_ARGS)
+bytea_string_agg_transfn(PG_FUNCTION_ARGS)
 {
 	StringInfo	state;
 
@@ -408,21 +408,28 @@ bytea_agg_transfn(PG_FUNCTION_ARGS)
 	{
 		bytea	   *value = PG_GETARG_BYTEA_PP(1);
 
+		/* On the first time through, we ignore the delimiter. */
 		if (state == NULL)
 			state = makeStringAggState(fcinfo);
+		else if (!PG_ARGISNULL(2))
+		{
+			bytea	   *delim = PG_GETARG_BYTEA_PP(2);
+
+			appendBinaryStringInfo(state, VARDATA_ANY(delim), VARSIZE_ANY_EXHDR(delim));
+		}
 
 		appendBinaryStringInfo(state, VARDATA_ANY(value), VARSIZE_ANY_EXHDR(value));
 	}
 
 	/*
-	 * The transition type for bytea_agg() is declared to be internal,
+	 * The transition type for string_agg() is declared to be internal,
 	 * which is a pass-by-value type the same size as a pointer.
 	 */
 	PG_RETURN_POINTER(state);
 }
 
 Datum
-bytea_agg_finalfn(PG_FUNCTION_ARGS)
+bytea_string_agg_finalfn(PG_FUNCTION_ARGS)
 {
 	StringInfo	state;
 
diff --git i/src/include/catalog/pg_aggregate.h w/src/include/catalog/pg_aggregate.h
index adda07c..461772c 100644
--- i/src/include/catalog/pg_aggregate.h
+++ w/src/include/catalog/pg_aggregate.h
@@ -229,7 +229,7 @@ DATA(insert ( 2335	array_agg_transfn	array_agg_finalfn		0	2281	_null_ ));
 DATA(insert ( 3538	string_agg_transfn	string_agg_finalfn		0	2281	_null_ ));
 
 /* bytea */
-DATA(insert ( 3545	bytea_agg_transfn	bytea_agg_finalfn		0	2281	_null_ ));
+DATA(insert ( 3545	bytea_string_agg_transfn	bytea_string_agg_finalfn		0	2281	_null_ ));
 
 /*
  * prototypes for functions in pg_aggregate.c
diff --git i/src/include/catalog/pg_proc.h w/src/include/catalog/pg_proc.h
index 6414b33..aa4d350 100644
--- i/src/include/catalog/pg_proc.h
+++ w/src/include/catalog/pg_proc.h
@@ -2433,11 +2433,11 @@ DATA(insert OID = 3536 (  string_agg_finalfn		PGNSP PGUID 12 1 0 0 0 f f f f f f
 DESCR(aggregate final function);
 DATA(insert OID = 3538 (  string_aggPGNSP PGUID 12 1 0 0 0 t f f f f f i 2 0 25 25 25 _null_ _null_ _null_ _null_ aggregate_dummy _null_ _null_ _null_ ));
 DESCR(concatenate aggregate input into a string);
-DATA(insert OID = 3543 (  bytea_agg_transfn		PGNSP PGUID 12 1 0 0 0 f f f f f f i 2 0 2281 2281 17 _null_ _null_ _null_ _null_ bytea_agg_transfn _null_ _null_ _null_ ));
+DATA(insert OID = 3543 (  bytea_string_agg_transfn	PGNSP PGUID 12 1 0 0 0 f f f f f f i 3 0 2281 2281 17 17 _null_ _null_ _null_ _null_ bytea_string_agg_transfn _null_ _null_ _null_ ));
 DESCR(aggregate transition function);
-DATA(insert OID = 3544 (  bytea_agg_finalfn		PGNSP PGUID 12 1 0 0 0 f f f f f f i 1 0 17 2281 _null_ _null_ _null_ _null_ bytea_agg_finalfn _null_ _null_ _null_ ));
+DATA(insert OID = 3544 (  bytea_string_agg_finalfn	PGNSP PGUID 12 1 0 0 0 f f f f f f i 1 0 17 2281 _null_ _null_ _null_ _null_ bytea_string_agg_finalfn _null_ _null_ _null_ ));
 DESCR(aggregate final function);
-DATA(insert OID = 3545 (  bytea_aggPGNSP PGUID 12 1 0 0 0 t f f f f f i 1 0 17 17 _null_ _null_ _null_ _null_ aggregate_dummy _null_ _null_ _null_ ));
+DATA(insert OID = 3545 (  string_aggPGNSP PGUID 12 1 0 0 0 t f f f f f i 2 0 17 17 17 _null_ _null_ _null_ _null_ aggregate_dummy 

Re: [HACKERS] patch: bytea_agg

2012-04-11 Thread Tom Lane
Peter Eisentraut pete...@gmx.net writes:
 On lör, 2012-04-07 at 10:38 -0400, Tom Lane wrote:
 Hm.  So are you now suggesting we should get rid of one-argument
 bytea_agg and replace it with two-argument string_agg(bytea,bytea)?
 I could support that, since we've not released bytea_agg yet.

 Yes, that looks like the best solution.  Here is a patch for that.

Looks sane in a quick once-over, except for the documentation entry.
I'm not really thrilled with text, text or bytea, bytea because it
seems easy to misparse.  Moreover, as written the entry claims that
the return type is text either way, which is wrong.  You could fix
the latter by writing same as argument data type, but I wonder
whether it'd be better to make separate table entries for the two
forms.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Last gasp

2012-04-11 Thread Robert Haas
On Wed, Apr 11, 2012 at 1:39 PM, Joshua Berkus j...@agliodbs.com wrote:
 Ultimately, we're herding cats here.  I don't think you're going to
 get
 the community to suddenly be willing to march in lockstep instead.

 If you, Peter, Simon, Robert, Heikki, Magnus, Peter G., Greg, Bruce and 
 Andrew agreed on a calendar-driven, mostly unambiguous process and adhered to 
 that process, then the one or two people who didn't follow along wouldn't 
 matter.  Everyone else would follow you.  The reason things are chaotic now 
 is that our lead committers do not have consensus and are even inconsistent 
 from CF to CF individually.

 In other words: the problem is only unsolvable because *you* think it's 
 unsolvable.   If you decide the problem is solvable, you already have the 
 means to solve it.

That's a somewhat bizarre list of people.  It both includes people who
haven't expressed many concerns about our process one way or the other
and excludes some who have.  At any rate, clearly the problem is
exactly that there isn't consensus on this.  I would generally say
that Tom, Greg Smith, and I are pretty close together on this issue,
and Peter G., Simon, and Dimitri are pretty close together on this
issue, but with a big gap in between those two groups.  I am less
clear on how everyone else feels, but I think that saying that all we
need is to get consensus among those people is to define the problem,
not the solution.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] how to create a non-inherited CHECK constraint in CREATE TABLE

2012-04-11 Thread Nikhil Sontakke
Hi,

So, I have a patch for this. This patch introduces support for

CHECK ONLY syntax while doing a CREATE TABLE as well as during the usual
ALTER TABLE command.

Example:

create table atacc7 (test int, test2 int CHECK ONLY (test0), CHECK
(test210));
create table atacc8 () inherits (atacc7);

postgres=# \d+ atacc7
Table public.atacc7
 Column |  Type   | Modifiers | Storage | Description
+-+---+-+-
 test   | integer |   | plain   |
 test2  | integer |   | plain   |
Check constraints:
atacc7_test2_check CHECK (test2  10)
atacc7_test_check CHECK ONLY (test  0)
Child tables: atacc8
Has OIDs: no

postgres=# \d+ atacc8
Table public.atacc8
 Column |  Type   | Modifiers | Storage | Description
+-+---+-+-
 test   | integer |   | plain   |
 test2  | integer |   | plain   |
Check constraints:
atacc7_test2_check CHECK (test2  10)
Inherits: atacc7
Has OIDs: no


This patch removes the support for :

ALTER TABLE ONLY constraint_rename_test ADD CONSTRAINT con2 CHECK (b  0);

and uses

ALTER TABLE constraint_rename_test ADD CONSTRAINT con2 CHECK ONLY (b  0);

Is this what we want? Or we would want the earlier support in place for
backward compatibility as well? We are actually introducing this in 9.2 so
I guess we can remove this.

This is a much cleaner implementation and we might not even need the
changes in pg_dump now because the pg_get_constraintdef can provide the
info about the ONLY part too. So some cleanup can be done if needed.

I know it's a bit late in the commitfest, but if this patch makes this
feature more complete, maybe we should consider...

Thoughts?

P.S Here's the discussion thread in its entirety for reference:
http://postgresql.1045698.n5.nabble.com/how-to-create-a-non-inherited-CHECK-constraint-in-CREATE-TABLE-td5152184.html

Regards,
Nikhils

On Thu, Feb 2, 2012 at 1:32 AM, Peter Eisentraut
pete...@gmx.net wrote:

 On ons, 2012-01-18 at 18:17 -0500, Robert Haas wrote:
  I agree with Peter that we should have we should have CHECK ONLY.
  ONLY is really a property of the constraint, not the ALTER TABLE
  command -- if it were otherwise, we wouldn't need to store it the
  system catalogs, but of course we do.  The fact that it's not a
  standard property isn't a reason not to have proper syntax for it.

 Clearly, we will eventually want to support inherited and non-inherited
 constraints of all types.  Currently, each type of constraint has an
 implicit default regarding this property:

 check - inherited
 not null - inherited
 foreign key - not inherited
 primary key - not inherited
 unique - not inherited
 exclusion - not inherited

 As discussed above, we need to have a syntax that is attached to the
 constraint, not the table operation that creates the constraint, so that
 we can also create these in CREATE TABLE.

 How should we resolve these different defaults?

 Also, in ALTER TABLE, if you want to add either an inherited or not
 inherited constraint to a parent table, you should really say ALTER
 TABLE ONLY in either case.  Because it's conceivably valid that ALTER
 TABLE foo ADD CHECK () NOINHERIT would add an independent, not inherited
 check constraint to each child table.

 So, there are all kinds of inconsistencies and backward compatibility
 problems lurking here.  We might need either a grand transition plan or
 document the heck out of these inconsistencies.





check_constraint_create_table_support.patch
Description: Binary data

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] [GENERAL] [streaming replication] 9.1.3 streaming replication bug ?

2012-04-11 Thread Robert Haas
On Wed, Apr 11, 2012 at 12:35 PM, Fujii Masao masao.fu...@gmail.com wrote:
 On Thu, Apr 12, 2012 at 12:56 AM, Fujii Masao masao.fu...@gmail.com wrote:
 On Wed, Apr 11, 2012 at 3:31 PM, 乔志强 qiaozhiqi...@leadcoretech.com wrote:
 So in sync streaming replication, if master delete WAL before sent to the 
 only standby, all transaction will fail forever,
 the master tries to avoid a PANIC error rather than termination of 
 replication. but in sync replication, termination of replication is THE 
 bigger PANIC error.

 I see your point. When there are backends waiting for replication, the WAL 
 files
 which the standby might not have received yet must not be removed. If they 
 are
 removed, replication keeps failing forever because required WAL files don't
 exist in the master, and then waiting backends will never be released unless
 replication mode is changed to async. This should be avoided.

 On second thought, we can avoid the issue by just increasing
 wal_keep_segments enough. Even if the issue happens and some backends
 get stuck to wait for replication, we can release them by taking fresh backup
 and restarting the standby from that backup. This is the basic procedure to
 restart replication after replication is terminated because required WAL files
 are removed from the master. So this issue might not be worth implementing
 the patch for now (though I'm not against improving things in the future), but
 it seems just a tuning-problem of wal_keep_segments.

We've talked about teaching the master to keep track of how far back
all of its known standbys are, and retaining WAL back to that specific
point, rather than the shotgun approach that is wal_keep_segments.
It's not exactly clear what the interface to that should look like,
though.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Last gasp

2012-04-11 Thread Tom Lane
Peter Eisentraut pete...@gmx.net writes:
 Just as a personal view, if people were to send me doc or trivial
 patches in git-am format, with proper commit message, and Acked or
 Signed-off etc. lines from recognized contributors, and proper
 References: mail header linked to the discussion or suggestion
 message, I could probably commit 20 of those in an hour.

 Instead, I have to review the entire email thread for discussion, any
 possible reviews or test runs, extract the patch from the email, mangle
 it into proper form, apply it, think of a commit message, make sure I
 register all the right people in the message, re-review the commit,
 push, reply to email, optionally, log into commit fest, find the patch,
 click a bunch of times, close it, done -- I think.  That takes 15
 minutes per patch, and after two patches like that I'm tired.

I hear you ... but, given that the source material is a mailing-list
thread, *somebody* has to do all that work to produce an acceptable
commit.  And if you're just going to commit what that somebody sends you
without further review, then you might as well give that person a commit
bit, because you're trusting them to get all this right.  So I'm not
sure how this moves us forward, other than to the obvious observation
that it'd be great if we had more qualified committers.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] how to create a non-inherited CHECK constraint in CREATE TABLE

2012-04-11 Thread Alvaro Herrera

Excerpts from Nikhil Sontakke's message of mié abr 11 15:07:45 -0300 2012:

 This patch removes the support for :
 
 ALTER TABLE ONLY constraint_rename_test ADD CONSTRAINT con2 CHECK (b  0);
 
 and uses
 
 ALTER TABLE constraint_rename_test ADD CONSTRAINT con2 CHECK ONLY (b  0);
 
 Is this what we want? Or we would want the earlier support in place for
 backward compatibility as well? We are actually introducing this in 9.2 so
 I guess we can remove this.

I'm not quite following that logic.  I don't think support for the
previous syntax should be removed -- does it cause some serious problem?

 This is a much cleaner implementation and we might not even need the
 changes in pg_dump now because the pg_get_constraintdef can provide the
 info about the ONLY part too. So some cleanup can be done if needed.
 
 I know it's a bit late in the commitfest, but if this patch makes this
 feature more complete, maybe we should consider...
 
 Thoughts?

Personally I don't think we should consider this for 9.2.

-- 
Álvaro Herrera alvhe...@commandprompt.com
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] how to create a non-inherited CHECK constraint in CREATE TABLE

2012-04-11 Thread Tom Lane
Alvaro Herrera alvhe...@commandprompt.com writes:
 Excerpts from Nikhil Sontakke's message of mié abr 11 15:07:45 -0300 2012:
 This patch removes the support for :
 
 ALTER TABLE ONLY constraint_rename_test ADD CONSTRAINT con2 CHECK (b  0);
 
 and uses
 
 ALTER TABLE constraint_rename_test ADD CONSTRAINT con2 CHECK ONLY (b  0);
 
 I know it's a bit late in the commitfest, but if this patch makes this
 feature more complete, maybe we should consider...

 Personally I don't think we should consider this for 9.2.

Well, if we're going to regret having offered the other syntax, now
would be the time to figure that out, before we ship it not after.
I would go so far as to say that if we don't accept this for 9.2
we probably shouldn't accept it at all, because two different ways
to spell the same thing isn't nice.

I don't really care for the idea that the ONLY goes in a different place
for this operation than for every other kind of ALTER TABLE, but it does
make sense if you subscribe to the quoted theory that ONLY is a property
of the constraint and not the ALTER command as such.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] how to create a non-inherited CHECK constraint in CREATE TABLE

2012-04-11 Thread Andrew Dunstan



On 04/11/2012 02:45 PM, Tom Lane wrote:

Alvaro Herreraalvhe...@commandprompt.com  writes:

Excerpts from Nikhil Sontakke's message of mié abr 11 15:07:45 -0300 2012:

This patch removes the support for :

ALTER TABLE ONLY constraint_rename_test ADD CONSTRAINT con2 CHECK (b  0);

and uses

ALTER TABLE constraint_rename_test ADD CONSTRAINT con2 CHECK ONLY (b  0);

I know it's a bit late in the commitfest, but if this patch makes this
feature more complete, maybe we should consider...

Personally I don't think we should consider this for 9.2.

Well, if we're going to regret having offered the other syntax, now
would be the time to figure that out, before we ship it not after.
I would go so far as to say that if we don't accept this for 9.2
we probably shouldn't accept it at all, because two different ways
to spell the same thing isn't nice.

I don't really care for the idea that the ONLY goes in a different place
for this operation than for every other kind of ALTER TABLE, but it does
make sense if you subscribe to the quoted theory that ONLY is a property
of the constraint and not the ALTER command as such.





I think I rather dislike it. ONLY should be followed by the name of the 
parent table whose children it causes us to exclude, IMNSHO. Moving it 
elsewhere doesn't seem to me to be a blow for clarity at all.


cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] how to create a non-inherited CHECK constraint in CREATE TABLE

2012-04-11 Thread Alvaro Herrera

Excerpts from Andrew Dunstan's message of mié abr 11 15:51:51 -0300 2012:
 
 On 04/11/2012 02:45 PM, Tom Lane wrote:
  Alvaro Herreraalvhe...@commandprompt.com  writes:
  Excerpts from Nikhil Sontakke's message of mié abr 11 15:07:45 -0300 2012:
  This patch removes the support for :
 
  ALTER TABLE ONLY constraint_rename_test ADD CONSTRAINT con2 CHECK (b  0);
 
  and uses
 
  ALTER TABLE constraint_rename_test ADD CONSTRAINT con2 CHECK ONLY (b  0);
 
  I know it's a bit late in the commitfest, but if this patch makes this
  feature more complete, maybe we should consider...
  Personally I don't think we should consider this for 9.2.
  Well, if we're going to regret having offered the other syntax, now
  would be the time to figure that out, before we ship it not after.
  I would go so far as to say that if we don't accept this for 9.2
  we probably shouldn't accept it at all, because two different ways
  to spell the same thing isn't nice.
 
  I don't really care for the idea that the ONLY goes in a different place
  for this operation than for every other kind of ALTER TABLE, but it does
  make sense if you subscribe to the quoted theory that ONLY is a property
  of the constraint and not the ALTER command as such.
 
 I think I rather dislike it. ONLY should be followed by the name of the 
 parent table whose children it causes us to exclude, IMNSHO. Moving it 
 elsewhere doesn't seem to me to be a blow for clarity at all.

If that's the only objection, maybe we could use a different keyword
then, perhaps NOINHERIT:

ALTER TABLE constraint_rename_test ADD CONSTRAINT con2 CHECK NOINHERIT (b  0);

-- 
Álvaro Herrera alvhe...@commandprompt.com
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] how to create a non-inherited CHECK constraint in CREATE TABLE

2012-04-11 Thread Andrew Dunstan



On 04/11/2012 03:06 PM, Tom Lane wrote:
I'd propose CHECK NO INHERIT, though, as (a) it seems better English 
and (b) it avoids creating any new keyword. 



I could live with that too.

cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] how to create a non-inherited CHECK constraint in CREATE TABLE

2012-04-11 Thread Robert Haas
On Wed, Apr 11, 2012 at 2:45 PM, Tom Lane t...@sss.pgh.pa.us wrote:
 Alvaro Herrera alvhe...@commandprompt.com writes:
 Excerpts from Nikhil Sontakke's message of mié abr 11 15:07:45 -0300 2012:
 This patch removes the support for :

 ALTER TABLE ONLY constraint_rename_test ADD CONSTRAINT con2 CHECK (b  0);

 and uses

 ALTER TABLE constraint_rename_test ADD CONSTRAINT con2 CHECK ONLY (b  0);

 I know it's a bit late in the commitfest, but if this patch makes this
 feature more complete, maybe we should consider...

 Personally I don't think we should consider this for 9.2.

 Well, if we're going to regret having offered the other syntax, now
 would be the time to figure that out, before we ship it not after.
 I would go so far as to say that if we don't accept this for 9.2
 we probably shouldn't accept it at all, because two different ways
 to spell the same thing isn't nice.

+1 for fixing up the syntax before 9.2 goes out the door.  I think the
original syntax was misguided to begin with.

CHECK NO INHERIT sounds fine to me; will that display ALTER TABLE ONLY
x as the one true way of doing this?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] how to create a non-inherited CHECK constraint in CREATE TABLE

2012-04-11 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes:
 +1 for fixing up the syntax before 9.2 goes out the door.  I think the
 original syntax was misguided to begin with.

Well, it was fine in isolation, but once you consider how to make CREATE
TABLE do this too, it's hard to avoid the conclusion that you need to
attach the modifier to the CHECK constraint not the ALTER TABLE command.

 CHECK NO INHERIT sounds fine to me; will that display ALTER TABLE ONLY
 x as the one true way of doing this?

s/display/displace/, I think you meant?  Yeah, that's what I understand
the proposal to be.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] man pages for contrib programs

2012-04-11 Thread Peter Eisentraut
On ons, 2012-04-04 at 21:53 +0300, Peter Eisentraut wrote:
 I think it would be useful to split this up into three sections:

 F.1. Extensions
 F.2. Client Applications
 F.3. Server Applications

 where the first looks like now and the other two contain the refentry
 pages.

 We could also consider making two separate appendixes.  Maybe that
 would result in a better table of contents.

I've played around with this a little bit to see how the tables of
contents etc. turn out.  I think the best approach is to have two
appendixes

F. Additional Supplied Extensions

with one sect1 per extension, like now, and

G. Additional Supplied Applications

with two subsections Client and Server Applications, and one refentry
per application.  That would end up looking much like the SPI chapter.


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] man pages for contrib programs

2012-04-11 Thread Thom Brown
On 11 April 2012 21:29, Peter Eisentraut pete...@gmx.net wrote:
 On ons, 2012-04-04 at 21:53 +0300, Peter Eisentraut wrote:
 I think it would be useful to split this up into three sections:

 F.1. Extensions
 F.2. Client Applications
 F.3. Server Applications

 where the first looks like now and the other two contain the refentry
 pages.

 We could also consider making two separate appendixes.  Maybe that
 would result in a better table of contents.

 I've played around with this a little bit to see how the tables of
 contents etc. turn out.  I think the best approach is to have two
 appendixes

 F. Additional Supplied Extensions

 with one sect1 per extension, like now, and

 G. Additional Supplied Applications

 with two subsections Client and Server Applications, and one refentry
 per application.  That would end up looking much like the SPI chapter.

Could you clarify what you're defining to be a client application and
a server application?  This could be confusing as we already have
sections under Reference called PostgreSQL Client Applications and
PostgreSQL Server Applications, visible in the root table of
contents.

-- 
Thom

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


[HACKERS] Columns of pg_stat_activity

2012-04-11 Thread Bruce Momjian
Since we are wacking around pg_stat_activity for 9.2, what do people
think about these column names?

 backend_start| timestamp with time zone |
 xact_start   | timestamp with time zone |
 query_start  | timestamp with time zone |

Arguably:

backend_start - session_start
query_start - statment_start

Should we make any of these changes?

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Columns of pg_stat_activity

2012-04-11 Thread Thom Brown
On 11 April 2012 21:46, Bruce Momjian br...@momjian.us wrote:
 Since we are wacking around pg_stat_activity for 9.2, what do people
 think about these column names?

         backend_start    | timestamp with time zone |
         xact_start       | timestamp with time zone |
         query_start      | timestamp with time zone |

 Arguably:

        backend_start - session_start
        query_start - statment_start

 Should we make any of these changes?

Sounds like a lot of potential breakage to solve something I don't
think is a problem.  Besides, isn't the door for 9.2 changes now
closed and bolted?

-- 
Thom

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] man pages for contrib programs

2012-04-11 Thread Peter Eisentraut
On ons, 2012-04-11 at 21:42 +0100, Thom Brown wrote:
  G. Additional Supplied Applications
 
  with two subsections Client and Server Applications, and one refentry
  per application.  That would end up looking much like the SPI chapter.
 
 Could you clarify what you're defining to be a client application and
 a server application?  This could be confusing as we already have
 sections under Reference called PostgreSQL Client Applications and
 PostgreSQL Server Applications, visible in the root table of
 contents.
 
By the same criteria as the main reference: client applications can run
anywhere and connect to a server, server applications run on the same
host as the database server.


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Columns of pg_stat_activity

2012-04-11 Thread Bruce Momjian
On Wed, Apr 11, 2012 at 09:50:43PM +0100, Thom Brown wrote:
 On 11 April 2012 21:46, Bruce Momjian br...@momjian.us wrote:
  Since we are wacking around pg_stat_activity for 9.2, what do people
  think about these column names?
 
          backend_start    | timestamp with time zone |
          xact_start       | timestamp with time zone |
          query_start      | timestamp with time zone |
 
  Arguably:
 
         backend_start - session_start
         query_start - statment_start
 
  Should we make any of these changes?
 
 Sounds like a lot of potential breakage to solve something I don't
 think is a problem.  Besides, isn't the door for 9.2 changes now
 closed and bolted?

Well, we renamed procpid - pid and I noticed these others.  Not sure if
it is a win or not, but just asking.

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] man pages for contrib programs

2012-04-11 Thread Thom Brown
On 11 April 2012 21:58, Peter Eisentraut pete...@gmx.net wrote:
 On ons, 2012-04-11 at 21:42 +0100, Thom Brown wrote:
 Could you clarify what you're defining to be a client application and
 a server application?  This could be confusing as we already have
 sections under Reference called PostgreSQL Client Applications and
 PostgreSQL Server Applications, visible in the root table of
 contents.

 By the same criteria as the main reference: client applications can run
 anywhere and connect to a server, server applications run on the same
 host as the database server.

Fair enough.

So will you be classifying things like auto_explain and auth_delay as
extensions? (i.e. things which aren't installed via CREATE
EXTENSION)

-- 
Thom

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Columns of pg_stat_activity

2012-04-11 Thread Magnus Hagander
On Wed, Apr 11, 2012 at 23:04, Bruce Momjian br...@momjian.us wrote:
 On Wed, Apr 11, 2012 at 09:50:43PM +0100, Thom Brown wrote:
 On 11 April 2012 21:46, Bruce Momjian br...@momjian.us wrote:
  Since we are wacking around pg_stat_activity for 9.2, what do people
  think about these column names?
 
          backend_start    | timestamp with time zone |
          xact_start       | timestamp with time zone |
          query_start      | timestamp with time zone |
 
  Arguably:
 
         backend_start - session_start
         query_start - statment_start
 
  Should we make any of these changes?

 Sounds like a lot of potential breakage to solve something I don't
 think is a problem.  Besides, isn't the door for 9.2 changes now
 closed and bolted?

 Well, we renamed procpid - pid and I noticed these others.  Not sure if
 it is a win or not, but just asking.

We also renamed current_query - query, but that was mainly because it
actually changed meaning.

But. Since we already whacked around procpid-pid, yes, if we're ever
going to change those, now is the time, really.

I think at least backend_start - session_start would make sense.

Not sure about the other one - what's wrong with query_start?

-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Columns of pg_stat_activity

2012-04-11 Thread Tom Lane
Bruce Momjian br...@momjian.us writes:
 On Wed, Apr 11, 2012 at 09:50:43PM +0100, Thom Brown wrote:
 On 11 April 2012 21:46, Bruce Momjian br...@momjian.us wrote:
 Arguably:
backend_start - session_start
query_start - statment_start

 Sounds like a lot of potential breakage to solve something I don't
 think is a problem.  Besides, isn't the door for 9.2 changes now
 closed and bolted?

We do still have open issues that include such proposed changes,
so I'd say that too late isn't a good argument.  However ...

 Well, we renamed procpid - pid and I noticed these others.  Not sure if
 it is a win or not, but just asking.

We were talking about renaming columns if we changed their semantics.
I don't think renaming for the sake of a slightly cleaner name will
win us any friends.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Columns of pg_stat_activity

2012-04-11 Thread Bruce Momjian
On Wed, Apr 11, 2012 at 11:11:18PM +0200, Magnus Hagander wrote:
   Should we make any of these changes?
 
  Sounds like a lot of potential breakage to solve something I don't
  think is a problem.  Besides, isn't the door for 9.2 changes now
  closed and bolted?
 
  Well, we renamed procpid - pid and I noticed these others.  Not sure if
  it is a win or not, but just asking.
 
 We also renamed current_query - query, but that was mainly because it
 actually changed meaning.
 
 But. Since we already whacked around procpid-pid, yes, if we're ever
 going to change those, now is the time, really.
 
 I think at least backend_start - session_start would make sense.
 
 Not sure about the other one - what's wrong with query_start?

We consistently use statement for commands, not queries, because
some feel query means SELECT.

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Columns of pg_stat_activity

2012-04-11 Thread Bruce Momjian
On Wed, Apr 11, 2012 at 05:14:51PM -0400, Tom Lane wrote:
 Bruce Momjian br...@momjian.us writes:
  On Wed, Apr 11, 2012 at 09:50:43PM +0100, Thom Brown wrote:
  On 11 April 2012 21:46, Bruce Momjian br...@momjian.us wrote:
  Arguably:
 backend_start - session_start
 query_start - statment_start
 
  Sounds like a lot of potential breakage to solve something I don't
  think is a problem.  Besides, isn't the door for 9.2 changes now
  closed and bolted?
 
 We do still have open issues that include such proposed changes,
 so I'd say that too late isn't a good argument.  However ...
 
  Well, we renamed procpid - pid and I noticed these others.  Not sure if
  it is a win or not, but just asking.
 
 We were talking about renaming columns if we changed their semantics.
 I don't think renaming for the sake of a slightly cleaner name will
 win us any friends.

The procpid change was for accuracy, I guess.

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Last gasp

2012-04-11 Thread Peter Eisentraut
On ons, 2012-04-11 at 14:29 -0400, Tom Lane wrote:
 I hear you ... but, given that the source material is a mailing-list
 thread, *somebody* has to do all that work to produce an acceptable
 commit.  And if you're just going to commit what that somebody sends
 you without further review, then you might as well give that person a
 commit bit, because you're trusting them to get all this right.

I'd still review it, but I'd be able to spend say 3 minutes on review
and 30 seconds on committing it, versus 3 minutes on review, 3 minutes
on research, and 8 minutes on bookkeeping.



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


[HACKERS] About bug #6579

2012-04-11 Thread Tom Lane
I've looked into this:
http://archives.postgresql.org/pgsql-bugs/2012-04/msg00058.php
and concluded that it's not very practical to fix it properly
right now.  A real fix will involve rearranging things so that
construction of the filter-condition list happens at Path creation
time, not createplan time, and that's a rather invasive change.
So I want to put it off until 9.3.

However, I did think of a simple one-line hack we could apply to mask
the worst effects of the bogus estimate, which is just to clamp the
correction factor from the indexquals to be not more than the original
cost estimate for the baserestrict quals, at line 461 in HEAD's
costsize.c:

-   cpu_per_tuple -= index_qual_cost.per_tuple;
+   cpu_per_tuple -= Min(index_qual_cost.per_tuple,
+baserel-baserestrictcost.per_tuple);

This seems safe and back-patchable.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] About bug #6579

2012-04-11 Thread Tom Lane
I wrote:
 I've looked into this:
 http://archives.postgresql.org/pgsql-bugs/2012-04/msg00058.php
 and concluded that it's not very practical to fix it properly
 right now.  A real fix will involve rearranging things so that
 construction of the filter-condition list happens at Path creation
 time, not createplan time, and that's a rather invasive change.
 So I want to put it off until 9.3.

... and on still further review, I've concluded that this isn't that
expensive to fix locally after all, at least in HEAD; and we get the
further benefit of saner costing of join cases.  (As per attached.
Basically it'll cost us one list_difference_ptr operation per IndexPath,
on what will typically be pretty short lists.  The cost_qual_eval
operation should be negligible either way, because it will be hitting
RestrictInfo nodes with already-cached costs.)

I'm still inclined to put the quick Min() hack into older branches,
though.  While this larger fix could possibly be back-patched, it might
change cost estimates by enough to destabilize plan choices.  Given
the small number of complaints about the issue to date, it doesn't seem
worth taking any risk for in released branches.

regards, tom lane

diff --git a/src/backend/optimizer/path/costsize.c b/src/backend/optimizer/path/costsize.c
index 9cae27b99eb253362fddf6ec00e6f736a5243c52..0330f01531e5bc1ae2a98c807b60fa212b50ed17 100644
*** a/src/backend/optimizer/path/costsize.c
--- b/src/backend/optimizer/path/costsize.c
*** cost_index(IndexPath *path, PlannerInfo 
*** 228,233 
--- 228,234 
  	IndexOptInfo *index = path-indexinfo;
  	RelOptInfo *baserel = index-rel;
  	bool		indexonly = (path-path.pathtype == T_IndexOnlyScan);
+ 	List	   *allclauses;
  	Cost		startup_cost = 0;
  	Cost		run_cost = 0;
  	Cost		indexStartupCost;
*** cost_index(IndexPath *path, PlannerInfo 
*** 239,244 
--- 240,246 
  spc_random_page_cost;
  	Cost		min_IO_cost,
  max_IO_cost;
+ 	QualCost	qpqual_cost;
  	Cost		cpu_per_tuple;
  	double		tuples_fetched;
  	double		pages_fetched;
*** cost_index(IndexPath *path, PlannerInfo 
*** 267,274 
  		 * Note that we force the clauses to be treated as non-join clauses
  		 * during selectivity estimation.
  		 */
- 		List	   *allclauses;
- 
  		allclauses = list_union_ptr(baserel-baserestrictinfo,
  	path-indexclauses);
  		path-path.rows = baserel-tuples *
--- 269,274 
*** cost_index(IndexPath *path, PlannerInfo 
*** 283,288 
--- 283,291 
  	}
  	else
  	{
+ 		/* allclauses should just be the rel's restriction clauses */
+ 		allclauses = baserel-baserestrictinfo;
+ 
  		/*
  		 * The number of rows is the same as the parent rel's estimate, since
  		 * this isn't a parameterized path.
*** cost_index(IndexPath *path, PlannerInfo 
*** 442,465 
  	/*
  	 * Estimate CPU costs per tuple.
  	 *
! 	 * Normally the indexquals will be removed from the list of restriction
! 	 * clauses that we have to evaluate as qpquals, so we should subtract
! 	 * their costs from baserestrictcost.  But if we are doing a join then
! 	 * some of the indexquals are join clauses and shouldn't be subtracted.
! 	 * Rather than work out exactly how much to subtract, we don't subtract
! 	 * anything.
  	 */
! 	startup_cost += baserel-baserestrictcost.startup;
! 	cpu_per_tuple = cpu_tuple_cost + baserel-baserestrictcost.per_tuple;
! 
! 	if (path-path.required_outer == NULL)
! 	{
! 		QualCost	index_qual_cost;
  
! 		cost_qual_eval(index_qual_cost, path-indexquals, root);
! 		/* any startup cost still has to be paid ... */
! 		cpu_per_tuple -= index_qual_cost.per_tuple;
! 	}
  
  	run_cost += cpu_per_tuple * tuples_fetched;
  
--- 445,467 
  	/*
  	 * Estimate CPU costs per tuple.
  	 *
! 	 * What we want here is cpu_tuple_cost plus the evaluation costs of any
! 	 * qual clauses that we have to evaluate as qpquals.  We approximate that
! 	 * list as allclauses minus any clauses appearing in indexquals (as
! 	 * before, assuming that pointer equality is enough to recognize duplicate
! 	 * RestrictInfos).  This method neglects some considerations such as
! 	 * clauses that needn't be checked because they are implied by a partial
! 	 * index's predicate.  It does not seem worth the cycles to try to factor
! 	 * those things in at this stage, even though createplan.c will take pains
! 	 * to remove such unnecessary clauses from the qpquals list if this path
! 	 * is selected for use.
  	 */
! 	cost_qual_eval(qpqual_cost,
!    list_difference_ptr(allclauses, path-indexquals),
!    root);
  
! 	startup_cost += qpqual_cost.startup;
! 	cpu_per_tuple = cpu_tuple_cost + qpqual_cost.per_tuple;
  
  	run_cost += cpu_per_tuple * tuples_fetched;
  

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] pg_upgrade improvements

2012-04-11 Thread Bruce Momjian
On Sat, Apr 07, 2012 at 01:13:23PM +0300, Peter Eisentraut wrote:
 On ons, 2012-04-04 at 19:26 -0700, Harold Giménez wrote:
  It would also be nice if the invocation of pg_ctl didn't pipe its
  output to /dev/null. I'm sure it would contain information that would
  directly point at the root cause and could've saved some debugging and
  hand waving time.
 
 This aspect has been reworked in 9.2devel.  Check it out to see if it
 works better for you.

Yes, we now spit out 5 log files if pg_upgrade fails, and tell you which
one to look at.

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] pg_upgrade improvements

2012-04-11 Thread Bruce Momjian
On Wed, Apr 04, 2012 at 07:26:58PM -0700, Harold Giménez wrote:
 Hi all,
 
 I've written a pg_upgrade wrapper for upgrading our users (heroku)
 to postgres 9.1. In the process I encountered a specific issue that
 could easily be improved. We've had this process work consistently
 for many users both internal and external, with the exception of just
 a few for whom the process fails and required manual handholding.
 
 Before it performs the upgrade, the pg_upgrade program starts the
 old cluster, does various checks, and then attempts to stop it. On
 occasion stopping the cluster fails - I've posted command output
 on a gist [1]. Manually running the pg_upgrade shortly afterwards
 succeeds. We believe stopping the cluster times out because there
 are other connections to the cluster that are established in that
 small window. There could be incoming connections for a number of
 reasons: either the user or the user's applications are reestablishing
 connections, or something like collectd on the localhost attempts to
 connect during that small window.

Well, we did address this in PG 9.2 by having pg_upgrade use a
non-default port number when starting servers, 50432.  You can do that
too in PG 9.1 by just specifying non-default port numbers when you run
pg_upgrade.  We do start the server with a special --binary-upgrade
mode, and we could do all sorts of connection limits in that mode, but
having the port number be different seemed the clearest solution.

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] pg_upgrade improvements

2012-04-11 Thread Harold Giménez
On Wed, Apr 11, 2012 at 5:40 PM, Bruce Momjian br...@momjian.us wrote:

 On Wed, Apr 04, 2012 at 07:26:58PM -0700, Harold Giménez wrote:
  There could be incoming connections for a number of
  reasons: either the user or the user's applications are reestablishing
  connections, or something like collectd on the localhost attempts to
  connect during that small window.

 Well, we did address this in PG 9.2 by having pg_upgrade use a
 non-default port number when starting servers, 50432.  You can do that
 too in PG 9.1 by just specifying non-default port numbers when you run
 pg_upgrade.  We do start the server with a special --binary-upgrade
 mode, and we could do all sorts of connection limits in that mode, but
 having the port number be different seemed the clearest solution.


The non-default port number is a good solution, better than modifying
pg_hba.
Thanks for pointing that out!

-Harold


Re: [HACKERS] Last gasp

2012-04-11 Thread Robert Haas
On Wed, Apr 11, 2012 at 5:36 PM, Peter Eisentraut pete...@gmx.net wrote:
 On ons, 2012-04-11 at 14:29 -0400, Tom Lane wrote:
 I hear you ... but, given that the source material is a mailing-list
 thread, *somebody* has to do all that work to produce an acceptable
 commit.  And if you're just going to commit what that somebody sends
 you without further review, then you might as well give that person a
 commit bit, because you're trusting them to get all this right.

 I'd still review it, but I'd be able to spend say 3 minutes on review
 and 30 seconds on committing it, versus 3 minutes on review, 3 minutes
 on research, and 8 minutes on bookkeeping.

Well, I am not averse to figuring out a better workflow, or some
better tools.   In practice, I think it's going to be hard to reduce
the time to review a trivial patch much below 5-10 minutes, which is
what it takes me now, because you've got to read the email, download
the patch, check that it doesn't break the build, review, commit, and
push, and I can't really see any of those steps going away.  But that
doesn't mean we shouldn't make the attempt, because I've got to admit
that the current workflow seems a little cumbersome to me, too.  I'm
not sure I have a better idea, though.  git remotes seem useful for
collaborating on topic branches, but I don't think they can really be
expected to save much of anything during the final commit process -
which is basically all the process there is, when the patch is
trivial.

Now what would be sort of neat is if we had a way to keep all the
versions of patch X plus author and reviewer information, links to
reviews and discussion, etc. in some sort of centralized place.  The
CommitFest app was actually designed to track a lot of this
information, but it's obviously not completely succeeding in tracking
everything that people care about - it only contains links to patches
and not patches themselves; it doesn't have any place to store
proposed commit messages; etc.  There might be room for improvement
there, although getting consensus on what improvement looks like may
not be totally straightforward, since I think Tom's ideal process for
submitting a patch starts with attaching a file to an email and many
other people I think would like to see it start with a pull request.
This is not entirely a tools issue, of course, but it's in there
somewhere.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Last gasp

2012-04-11 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes:
 On Wed, Apr 11, 2012 at 5:36 PM, Peter Eisentraut pete...@gmx.net wrote:
 I'd still review it, but I'd be able to spend say 3 minutes on review
 and 30 seconds on committing it, versus 3 minutes on review, 3 minutes
 on research, and 8 minutes on bookkeeping.

 Well, I am not averse to figuring out a better workflow, or some
 better tools.   In practice, I think it's going to be hard to reduce
 the time to review a trivial patch much below 5-10 minutes, which is
 what it takes me now, because you've got to read the email, download
 the patch, check that it doesn't break the build, review, commit, and
 push, and I can't really see any of those steps going away.  But that
 doesn't mean we shouldn't make the attempt, because I've got to admit
 that the current workflow seems a little cumbersome to me, too.  I'm
 not sure I have a better idea, though.  git remotes seem useful for
 collaborating on topic branches, but I don't think they can really be
 expected to save much of anything during the final commit process -
 which is basically all the process there is, when the patch is
 trivial.

 Now what would be sort of neat is if we had a way to keep all the
 versions of patch X plus author and reviewer information, links to
 reviews and discussion, etc. in some sort of centralized place.  The
 CommitFest app was actually designed to track a lot of this
 information, but it's obviously not completely succeeding in tracking
 everything that people care about - it only contains links to patches
 and not patches themselves; it doesn't have any place to store
 proposed commit messages; etc.  There might be room for improvement
 there, although getting consensus on what improvement looks like may
 not be totally straightforward, since I think Tom's ideal process for
 submitting a patch starts with attaching a file to an email and many
 other people I think would like to see it start with a pull request.
 This is not entirely a tools issue, of course, but it's in there
 somewhere.

It strikes me that there are two different scenarios being discussed
here, and we'd better be sure we keep them straight: small-to-trivial
patches, and complex patches.

I think that for the trivial case, what we need is less tooling not more.
Entering a patch in the CF app, updating and closing it will add a
not-small percentage to the total effort required to deal with a small
patch (as Peter already noted, and he wasn't even counting the time to
put the patch into CF initially).  The only reason to even consider
doing that is to make sure the patch doesn't get forgotten.  Perhaps
we could have some lighter-weight method of tracking such things?

At the other end of the scale, I think it's true that the CF app could
be more helpful than it is for tracking the state of complex patches.
I don't really have any concrete suggestions, other than that I've
seen far too many cases where the latest version of a patch was not
linked into the CF entry.  Somehow we've got to make that more robust.
Maybe the answer is to tie things more directly into git workflows,
though I'm not sure about details.  I am concerned about losing
traceability of submissions if all that ever shows up in the list
archives is a URL.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] how to create a non-inherited CHECK constraint in CREATE TABLE

2012-04-11 Thread Nikhil Sontakke
Hi,

Cumulative reaction to all the responses first:

Whoa! :)

I was under the impression that a majority of us felt that the current
mechanism was inadequate. Also if you go through the nabble thread, the
fact that CREATE TABLE did not support such constraints was considered to
be an annoyance. And I was enquired if/when I can provide this
functionality. Apologies though with the timing.


  +1 for fixing up the syntax before 9.2 goes out the door.  I think the
  original syntax was misguided to begin with.

 Well, it was fine in isolation, but once you consider how to make CREATE
 TABLE do this too, it's hard to avoid the conclusion that you need to
 attach the modifier to the CHECK constraint not the ALTER TABLE command.


Yeah, exactly.


  CHECK NO INHERIT sounds fine to me; will that display ALTER TABLE ONLY
  x as the one true way of doing this?

 s/display/displace/, I think you meant?  Yeah, that's what I understand
 the proposal to be.


Displace yes. It would error out if someone says

ALTER TABLE ONLY... CHECK ();

suggesting to use the ONLY with the CHECK.

This patch does this and also makes both CREATE TABLE and ALTER TABLE use
it in a uniform manner.

Regarding NO INHERIT versus ONLY, we again have had discussions on the
longish original thread quite a while back:

http://postgresql.1045698.n5.nabble.com/Check-constraints-on-partition-parents-only-tt464.html

But now if we prefer NO INHERIT, I can live with that.

Regards,
Nikhils