Re: [HACKERS] Why not install pgstattuple by default?

2011-05-19 Thread Marko Kreen
On Wed, May 18, 2011 at 3:37 PM, Magnus Hagander mag...@hagander.net wrote:
 On Wed, May 18, 2011 at 15:29, Marko Kreen mark...@gmail.com wrote:
 On Wed, May 18, 2011 at 2:57 PM, Magnus Hagander mag...@hagander.net wrote:
 On Wed, May 18, 2011 at 10:25, Greg Smith g...@2ndquadrant.com wrote:
 Some of my personal discussions of this topic have suggested that some 
 other
 popular extensions like pgcrypto and hstore get converted too.  I think
 those all fail test (3), and I'm not actually sure where pgcrypto adds any
 special dependency/distribution issues were it to be moved to the main
 database package.  If this general idea catches on, a wider discussion of
 what else should get promoted to this extensions area would be
 appropriate.  The ones I picked seemed the easiest to justify by this
 criteria set.

 pgcrypto would cause trouble for any builds *without* SSL. I don't
 think any packagers do that, but people doing manual builds would
 certainly get different results.

 What kind of trouble?  It should work fine without SSL.

 Oh, you're right - it does. But it does provide different
 functionalties? Or does it actually do exactly the same stuff, just in
 different ways?

Same stuff, assuming you use recommended algorithms
(Blowfish, AES, MD5, SHA1, SHA2)

OpenSSL brings in more speedy implementations (maybe),
and additional algorithms (ripemd160, 3des, cast5, twofish).

-- 
marko

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-18 Thread Greg Smith

Greg Smith wrote:
Attached is a second patch to move a number of extensions from 
contrib/ to src/test/.  Extensions there are built by the default 
built target, making installation of the postgresql-XX-contrib package 
unnecessary for them to be available.


That was supposed to be contrib/ to src/extension , no idea where that 
test bit came from.


--
Greg Smith   2ndQuadrant USg...@2ndquadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support  www.2ndQuadrant.us



--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-18 Thread Magnus Hagander
On Wed, May 18, 2011 at 10:25, Greg Smith g...@2ndquadrant.com wrote:
 Attached is a second patch to move a number of extensions from contrib/ to
 src/test/.  Extensions there are built by the default built target, making
 installation of the postgresql-XX-contrib package unnecessary for them to be
 available.

+1 in general on the concept :-)


 This request--making some of these additions available without the contrib
 name/package being involved--has popped up many times before, and it turys
 out to be really easy to resolve with the new extensions infrastructure.  I
 think it's even a reasonable change to consider applying now, between 9.1
 Beta 1 and Beta 2.  The documentation adjustments are the only serious bit
 left here that I've been able to find, the code changes here are all
 internal to the build process and easy.

Does this include regression tests? Or will they need some mods?

 I moved the following extensions:

 auto_explain pageinspect pg_buffercache pg_freespacemap pgrowlocks
 pg_stat_statements pgstattuple

 My criteria was picking extensions that:

 1) Don't have any special dependencies
 2) Are in contrib mainly because they don't need to be internal functions,
 not because their code quality is demo/early
 3) Tend to be installed on a production server for troubleshooting problems,
 rather than being required by development.
 4) Regularly pop up as necessary/helpful in production deployment

These seem like reasonable criteria.


 Some of my personal discussions of this topic have suggested that some other
 popular extensions like pgcrypto and hstore get converted too.  I think
 those all fail test (3), and I'm not actually sure where pgcrypto adds any
 special dependency/distribution issues were it to be moved to the main
 database package.  If this general idea catches on, a wider discussion of
 what else should get promoted to this extensions area would be
 appropriate.  The ones I picked seemed the easiest to justify by this
 criteria set.

pgcrypto would cause trouble for any builds *without* SSL. I don't
think any packagers do that, but people doing manual builds would
certainly get different results.


 Any packager who grabs the shared/postgresql/extension directory in 9.1,
 which I expect to be all of them, shouldn't need any changes to pick up this
 adjustment.  For example, pgstattuple installs these files:

 share/postgresql/extension/pgstattuple--1.0.sql
 share/postgresql/extension/pgstattuple--unpackaged--1.0.sql
 share/postgresql/extension/pgstattuple.control

 And these are the same locations they were already at.  The location of the
 source and which target built it is the change here, the result isn't any
 different.  This means that this change won't even break extensions already
 installed.

 Once the basic directory plumbing is in place, conversion of a single
 extension from contrib/ to src/test/ is, trivial.  The diff view

 I did five of them in an hour once I figured out what was needed.  Easiest
 to view the changes at
 https://github.com/greg2ndQuadrant/postgres/commits/move-contrib , the patch
 file is huge because of all the renames.
 https://github.com/greg2ndQuadrant/postgres/commit/d647091b18c4448c5a582d423f8839ef0c717e91
 show a good example of one convert, that changes pg_freespacemap.  There are
 more changes to the comments listing the name of the file than to any code.
  (Yes, I know there are some whitespace issues I introduced in the new
 Makefile, they should be fixed by a later commit in the series)

This is where the compare view rocks:

https://github.com/greg2ndQuadrant/postgres/compare/postgres:master...greg2ndQuadrant:move-contrib

-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-18 Thread Marko Kreen
On Wed, May 18, 2011 at 2:57 PM, Magnus Hagander mag...@hagander.net wrote:
 On Wed, May 18, 2011 at 10:25, Greg Smith g...@2ndquadrant.com wrote:
 Some of my personal discussions of this topic have suggested that some other
 popular extensions like pgcrypto and hstore get converted too.  I think
 those all fail test (3), and I'm not actually sure where pgcrypto adds any
 special dependency/distribution issues were it to be moved to the main
 database package.  If this general idea catches on, a wider discussion of
 what else should get promoted to this extensions area would be
 appropriate.  The ones I picked seemed the easiest to justify by this
 criteria set.

 pgcrypto would cause trouble for any builds *without* SSL. I don't
 think any packagers do that, but people doing manual builds would
 certainly get different results.

What kind of trouble?  It should work fine without SSL.

-- 
marko

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-18 Thread Magnus Hagander
On Wed, May 18, 2011 at 15:29, Marko Kreen mark...@gmail.com wrote:
 On Wed, May 18, 2011 at 2:57 PM, Magnus Hagander mag...@hagander.net wrote:
 On Wed, May 18, 2011 at 10:25, Greg Smith g...@2ndquadrant.com wrote:
 Some of my personal discussions of this topic have suggested that some other
 popular extensions like pgcrypto and hstore get converted too.  I think
 those all fail test (3), and I'm not actually sure where pgcrypto adds any
 special dependency/distribution issues were it to be moved to the main
 database package.  If this general idea catches on, a wider discussion of
 what else should get promoted to this extensions area would be
 appropriate.  The ones I picked seemed the easiest to justify by this
 criteria set.

 pgcrypto would cause trouble for any builds *without* SSL. I don't
 think any packagers do that, but people doing manual builds would
 certainly get different results.

 What kind of trouble?  It should work fine without SSL.

Oh, you're right - it does. But it does provide different
functionalties? Or does it actually do exactly the same stuff, just in
different ways?


-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-18 Thread Greg Smith

Greg Smith wrote:
Any packager who grabs the shared/postgresql/extension directory in 
9.1, which I expect to be all of them, shouldn't need any changes to 
pick up this adjustment.  For example, pgstattuple installs these files:


share/postgresql/extension/pgstattuple--1.0.sql
share/postgresql/extension/pgstattuple--unpackaged--1.0.sql
share/postgresql/extension/pgstattuple.control

And these are the same locations they were already at.


...and the bit I missed here is that there's a fourth file here:

lib/postgresql/pgstattuple.so

If you look at a 9.1 spec file, such as 
http://svn.pgrpms.org/browser/rpm/redhat/9.1/postgresql/EL-6/postgresql-9.1.spec 
, you'll find:


%files contrib
...
%{pgbaseinstdir}/lib/pgstattuple.so

Which *does* require a packager change to relocate from the 
postgresql-91-package to the main server one.  So the theory that a 
change here might happen without pushing a repackaging suggestion toward 
packagers is busted.  This does highlight that some packaging guidelines 
would be needed here to completely this work.


--
Greg Smith   2ndQuadrant USg...@2ndquadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support  www.2ndQuadrant.us



--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-13 Thread Devrim GÜNDÜZ
On Thu, 2011-05-12 at 19:37 -0400, Tom Lane wrote:
 
 
 It should be okay to move, since the -devel subpackage requires the
 main one.  Therefore there is no configuration in which pg_config
 would be present before and missing after the change. 

Thanks Tom. I can make this change in next build set.

Regards,

-- 
Devrim GÜNDÜZ
Principal Systems Engineer @ EnterpriseDB: http://www.enterprisedb.com
PostgreSQL Danışmanı/Consultant, Red Hat Certified Engineer
Community: devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr
http://www.gunduz.org  Twitter: http://twitter.com/devrimgunduz


signature.asc
Description: This is a digitally signed message part


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-12 Thread Devrim GÜNDÜZ
On Sat, 2011-05-07 at 21:47 -0400, Greg Smith wrote:
 On 05/06/2011 04:06 PM, Tom Lane wrote:
  FWIW, I did move pg_config from -devel to the main (really client)
  postgresql package in Fedora, as of 9.0.  That will ensure it's
 present
  in either client or server installations.  Eventually that packaging
  will reach RHEL ...
 
 
 We should make sure that the PGDG packages adopt that for 9.1 then, so
 it starts catching on more.  Unless Devrim changed to catch up since I
 last installed an RPM set, in that 9.0 it's still in the same place:
 
 $ rpm -qf /usr/pgsql-9.0/bin/pg_config
 postgresql90-devel-9.0.2-2PGDG.rhel5 

I'm not sure that I can move it to main package in 9.0 package set, I
need to make sure that I won't break anything. But it is pretty doable
for 9.1.

Regards,
-- 
Devrim GÜNDÜZ
Principal Systems Engineer @ EnterpriseDB: http://www.enterprisedb.com
PostgreSQL Danışmanı/Consultant, Red Hat Certified Engineer
Community: devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr
http://www.gunduz.org  Twitter: http://twitter.com/devrimgunduz


signature.asc
Description: This is a digitally signed message part


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-12 Thread Tom Lane
Devrim =?ISO-8859-1?Q?G=DCND=DCZ?= dev...@gunduz.org writes:
 I'm not sure that I can move it to main package in 9.0 package set, I
 need to make sure that I won't break anything. But it is pretty doable
 for 9.1.

It should be okay to move, since the -devel subpackage requires the main
one.  Therefore there is no configuration in which pg_config would be
present before and missing after the change.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-10 Thread Bruce Momjian
Tom Lane wrote:
 Christopher Browne cbbro...@gmail.com writes:
  But people are evidently still setting packaging policies based on how
  things were back in 7.3, even though that perhaps isn't necessary
  anymore.
 
 FWIW, once you get past the client versus server distinction, I think
 most subpackaging decisions are based on either the idea that only a
 minority of people will want this, or a desire to limit how many
 dependencies are pulled in by the main package(s).  Both of those
 concerns apply to various subsets of -contrib, which means it's going
 to be hard to persuade packagers to fold -contrib into the -server
 package altogether.  Nor would you gain their approval by trying to
 pre-empt the decision.
 
 We might get somewhere by trying to identify a small set of particularly
 popular contrib modules that don't add any extra dependencies, and then
 recommending to packagers that those ones get bundled into the main
 server package.
 
  Certainly it's not a huge amount of code; less than 2MB these days.
  - % wc `dpkg -L postgresql-contrib-9.0` | tail -1
15952   67555 1770987 total
 
 Well, to add some concrete facts rather than generalities to my own post,
 here are the sizes of the built RPMs from my last build for Fedora:
 
 -rw-r--r--. 1 tgl tgl  3839458 Apr 18 10:50 postgresql-9.0.4-1.fc13.x86_64.rpm
 -rw-r--r--. 1 tgl tgl   490788 Apr 18 10:50 
 postgresql-contrib-9.0.4-1.fc13.x86_64.rpm
 -rw-r--r--. 1 tgl tgl 27337677 Apr 18 10:51 
 postgresql-debuginfo-9.0.4-1.fc13.x86_64.rpm
 -rw-r--r--. 1 tgl tgl   961660 Apr 18 10:50 
 postgresql-devel-9.0.4-1.fc13.x86_64.rpm
 -rw-r--r--. 1 tgl tgl  7569048 Apr 18 10:50 
 postgresql-docs-9.0.4-1.fc13.x86_64.rpm
 -rw-r--r--. 1 tgl tgl   246506 Apr 18 10:50 
 postgresql-libs-9.0.4-1.fc13.x86_64.rpm
 -rw-r--r--. 1 tgl tgl64940 Apr 18 10:50 
 postgresql-plperl-9.0.4-1.fc13.x86_64.rpm
 -rw-r--r--. 1 tgl tgl65776 Apr 18 10:50 
 postgresql-plpython-9.0.4-1.fc13.x86_64.rpm
 -rw-r--r--. 1 tgl tgl45941 Apr 18 10:50 
 postgresql-pltcl-9.0.4-1.fc13.x86_64.rpm
 -rw-r--r--. 1 tgl tgl  5302117 Apr 18 10:50 
 postgresql-server-9.0.4-1.fc13.x86_64.rpm
 -rw-r--r--. 1 tgl tgl  1370509 Apr 18 10:50 
 postgresql-test-9.0.4-1.fc13.x86_64.rpm
 -rw-r--r--. 1 tgl tgl  3644113 Apr 18 10:50 
 postgresql-upgrade-9.0.4-1.fc13.x86_64.rpm

Is that last one pg_upgrade?  It seems very big.

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-10 Thread Tom Lane
Bruce Momjian br...@momjian.us writes:
 Tom Lane wrote:
 here are the sizes of the built RPMs from my last build for Fedora:
 
 -rw-r--r--. 1 tgl tgl  3839458 Apr 18 10:50 
 postgresql-9.0.4-1.fc13.x86_64.rpm
 -rw-r--r--. 1 tgl tgl   490788 Apr 18 10:50 
 postgresql-contrib-9.0.4-1.fc13.x86_64.rpm
 -rw-r--r--. 1 tgl tgl 27337677 Apr 18 10:51 
 postgresql-debuginfo-9.0.4-1.fc13.x86_64.rpm
 -rw-r--r--. 1 tgl tgl   961660 Apr 18 10:50 
 postgresql-devel-9.0.4-1.fc13.x86_64.rpm
 -rw-r--r--. 1 tgl tgl  7569048 Apr 18 10:50 
 postgresql-docs-9.0.4-1.fc13.x86_64.rpm
 -rw-r--r--. 1 tgl tgl   246506 Apr 18 10:50 
 postgresql-libs-9.0.4-1.fc13.x86_64.rpm
 -rw-r--r--. 1 tgl tgl64940 Apr 18 10:50 
 postgresql-plperl-9.0.4-1.fc13.x86_64.rpm
 -rw-r--r--. 1 tgl tgl65776 Apr 18 10:50 
 postgresql-plpython-9.0.4-1.fc13.x86_64.rpm
 -rw-r--r--. 1 tgl tgl45941 Apr 18 10:50 
 postgresql-pltcl-9.0.4-1.fc13.x86_64.rpm
 -rw-r--r--. 1 tgl tgl  5302117 Apr 18 10:50 
 postgresql-server-9.0.4-1.fc13.x86_64.rpm
 -rw-r--r--. 1 tgl tgl  1370509 Apr 18 10:50 
 postgresql-test-9.0.4-1.fc13.x86_64.rpm
 -rw-r--r--. 1 tgl tgl  3644113 Apr 18 10:50 
 postgresql-upgrade-9.0.4-1.fc13.x86_64.rpm

 Is that last one pg_upgrade?  It seems very big.

pg_upgrade plus a supporting set of 8.4 files ...

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-10 Thread Bruce Momjian
Tom Lane wrote:
 Bruce Momjian br...@momjian.us writes:
  Tom Lane wrote:
  here are the sizes of the built RPMs from my last build for Fedora:
  
  -rw-r--r--. 1 tgl tgl  3839458 Apr 18 10:50 
  postgresql-9.0.4-1.fc13.x86_64.rpm
  -rw-r--r--. 1 tgl tgl   490788 Apr 18 10:50 
  postgresql-contrib-9.0.4-1.fc13.x86_64.rpm
  -rw-r--r--. 1 tgl tgl 27337677 Apr 18 10:51 
  postgresql-debuginfo-9.0.4-1.fc13.x86_64.rpm
  -rw-r--r--. 1 tgl tgl   961660 Apr 18 10:50 
  postgresql-devel-9.0.4-1.fc13.x86_64.rpm
  -rw-r--r--. 1 tgl tgl  7569048 Apr 18 10:50 
  postgresql-docs-9.0.4-1.fc13.x86_64.rpm
  -rw-r--r--. 1 tgl tgl   246506 Apr 18 10:50 
  postgresql-libs-9.0.4-1.fc13.x86_64.rpm
  -rw-r--r--. 1 tgl tgl64940 Apr 18 10:50 
  postgresql-plperl-9.0.4-1.fc13.x86_64.rpm
  -rw-r--r--. 1 tgl tgl65776 Apr 18 10:50 
  postgresql-plpython-9.0.4-1.fc13.x86_64.rpm
  -rw-r--r--. 1 tgl tgl45941 Apr 18 10:50 
  postgresql-pltcl-9.0.4-1.fc13.x86_64.rpm
  -rw-r--r--. 1 tgl tgl  5302117 Apr 18 10:50 
  postgresql-server-9.0.4-1.fc13.x86_64.rpm
  -rw-r--r--. 1 tgl tgl  1370509 Apr 18 10:50 
  postgresql-test-9.0.4-1.fc13.x86_64.rpm
  -rw-r--r--. 1 tgl tgl  3644113 Apr 18 10:50 
  postgresql-upgrade-9.0.4-1.fc13.x86_64.rpm
 
  Is that last one pg_upgrade?  It seems very big.
 
 pg_upgrade plus a supporting set of 8.4 files ...

OK, where do I get to dance around that pg_upgrade is packaged in Fedora
thanks to you?  At PGCon?  LOL

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-09 Thread Robert Haas
On Sun, May 8, 2011 at 12:02 AM, Greg Smith g...@2ndquadrant.com wrote:
 Attached patch is a first cut at what moving one contrib module (in this
 case pg_buffercache) to a new directory structure might look like.  The idea
 is that src/extension could become a place for first-class extensions to
 live.  Those are ones community is committed to providing in core, but are
 just better implemented as extensions than in-database functions, for
 reasons that include security.  This idea has been shared by a lot of people
 for a while, only problem is that it wasn't really practical to implement
 cleanly until the extensions code hit.  I think it is now, this attempts to
 prove it.

 Since patches involving file renaming are clunky, the changes might be
 easier to see at
 https://github.com/greg2ndQuadrant/postgres/commit/507923e21e963c873a84f1b850d64e895776574f
 where I just pushed this change too.  The install step for the modules looks
 like this now:

 gsmith@grace:~/pgwork/src/move-contrib/src/extension/pg_buffercache$ make
 install
 /bin/mkdir -p '/home/gsmith/pgwork/inst/move-contrib/lib/postgresql'
 /bin/mkdir -p
 '/home/gsmith/pgwork/inst/move-contrib/share/postgresql/extension'
 /bin/sh ../../../config/install-sh -c -m 755  pg_buffercache.so
 '/home/gsmith/pgwork/inst/move-contrib/lib/postgresql/pg_buffercache.so'
 /bin/sh ../../../config/install-sh -c -m 644 ./pg_buffercache.control
 '/home/gsmith/pgwork/inst/move-contrib/share/postgresql/extension/'
 /bin/sh ../../../config/install-sh -c -m 644 ./pg_buffercache--1.0.sql
 ./pg_buffercache--unpackaged--1.0.sql
  '/home/gsmith/pgwork/inst/move-contrib/share/postgresql/extension/'
 $ psql -c create extension pg_buffercache
 CREATE EXTENSION

 The only clunky bit I wasn't really happy with is the amount of code
 duplication that comes from having a src/extension/Makefile that looks
 almost, but not quite, identical to contrib/Makefile.  The rest of the
 changes don't seem too bad to me, and even that's really only 36 lines that
 aren't touched often.  Yes, the paths are different, so backports won't
 happen without an extra step.  But the code changes required were easier
 than I was expecting, due to the general good modularity of the extensions
 infrastructure.  So long as the result ends up in
 share/postgresql/extension/ , whether they started in contrib/module or
 src/extension/module doesn't really matter to CREATE EXTENSION.  But
 having them broke out this way makes it easy for the default Makefile to
 build and install them all.  (I recognize I didn't do that last step yet
 though)

 I'll happily go covert pgstattuple and the rest of the internal diagnostics
 modules to this scheme, and do the doc cleanups, this upcoming week if it
 means I'll be able to use those things without installing all of contrib one
 day.  Ditto for proposing RPM and Debian packaging changes that match them.
  All that work will get paid back the first time I don't have to fill out a
 bunch of paperwork (again) at a customer site justifying why they need to
 install the contrib [RPM|deb] package (which has some scary stuff in it) on
 all their servers, just so I can get some bloat or buffer inspection module.

I would really like to see us try to group things by topic, and not
just by whether or not we can all agree that the extension is
important enough to be first-class (which is bound to be a bit
tendentious).  We probably can't completely avoid some bikeshedding on
that topic, but even there it strikes me that sorting by topic might
make things a bit more clear.  For example, if we could somehow group
together all the diagnostic tools, maybe something like the list
below, I think that would be a start.  Now then we might go on to
argue about which are the more useful diagnostic tools, but I think
it's easier to argue about that category than it is to argue in the
abstract about whether you'd rather have hstore or pgstattuple, to
which the answer can only be that depends.

auto_explain
oid2name
pageinspect
pg_buffercache
pg_freespacemap
pg_stat_statements
pg_test_fsync (perhaps)
pgrowlocks
pgstattuple

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-09 Thread Robert Haas
On Mon, May 9, 2011 at 1:14 PM, Greg Smith g...@2ndquadrant.com wrote:
 On 05/09/2011 10:53 AM, Robert Haas wrote:

 I would really like to see us try to group things by topic, and not
 just by whether or not we can all agree that the extension is
 important enough to be first-class (which is bound to be a bit
 tendentious).

 Having played around with the prototype, I think it doesn't actually matter
 if there's a further division below the new one I introduced.  The main
 thing I think is worth pointing out is that I only feel extensions with no
 external dependencies are worth the trouble of re-classifying here.  If it
 were worth reorganizing contrib just for the sake of categorizing it, that
 would have been done years ago.  The new thing is that extensions make it
 really easy to make some tools available in the server's extensions
 subdirectly, without actually activating them in the default install.

 Looking at your list:

 auto_explain
 oid2name
 pageinspect
 pg_buffercache
 pg_freespacemap
 pg_stat_statements
 pg_test_fsync (perhaps)
 pgrowlocks
 pgstattuple


 oid2name and pg_test_fsync would be out because those are real executables.
  I'd rather not introduce the risk/complexity of playing around with moving
 standalone utilities of such marginal value.  Whereas I think it sets an
 excellent precedent if the server is shipping with some standard add-ons,
 built using the same extension mechanism available to external code, in the
 core server package.  I'd certainly be happy to add auto_explain and
 pg_stat_statements (also extremely popular things to install for me) to that
 list.

I'm happy enough with that set of guidelines: namely, that we'd use
src/extension only for things that don't require additional
dependencies, and not for things that build standalone executables.
If we're going to move things around, I think we should take the
trouble to categorize them along the way, and your idea of inserting
one more subdirectory under src/extension for grouping seems fine to
me.

I don't think we should be too obstinate about trying to twist the arm
of packagers who (as Tom points out) will do whatever they want in
spite of us, but the current state of contrib, with all sorts of
things of varying type, complexity, and value mixed together cannot
possibly be a good thing.  Even if the effect of all this is that some
distributions end up with postgresql-server-instrumentation and
postgresql-server-datatypes packages, rather than putting everything
in postgresql-server, I still think that'd be better than having a
monolithic lump called postgresql-contrib.  Heaven only knows what
could be in there (says the sys admin)...

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-09 Thread Alvaro Herrera
Excerpts from Robert Haas's message of lun may 09 14:31:33 -0400 2011:

 I'm happy enough with that set of guidelines: namely, that we'd use
 src/extension only for things that don't require additional
 dependencies, and not for things that build standalone executables.
 If we're going to move things around, I think we should take the
 trouble to categorize them along the way, and your idea of inserting
 one more subdirectory under src/extension for grouping seems fine to
 me.

For executables we already have src/bin.  Do we really need a separate
place for, say, pg_standby or pg_upgrade?

-- 
Álvaro Herrera alvhe...@commandprompt.com
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-09 Thread Tom Lane
Alvaro Herrera alvhe...@commandprompt.com writes:
 Excerpts from Robert Haas's message of lun may 09 14:31:33 -0400 2011:
 I'm happy enough with that set of guidelines: namely, that we'd use
 src/extension only for things that don't require additional
 dependencies, and not for things that build standalone executables.
 If we're going to move things around, I think we should take the
 trouble to categorize them along the way, and your idea of inserting
 one more subdirectory under src/extension for grouping seems fine to
 me.

 For executables we already have src/bin.  Do we really need a separate
 place for, say, pg_standby or pg_upgrade?

Putting them in there implies we think they are of core-code quality.
I'm definitely *not* ready to grant that status to pg_upgrade, for
instance.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-09 Thread Dimitri Fontaine
Tom Lane t...@sss.pgh.pa.us writes:
 Sure, but that's a documentation issue, which again is not going to be
 helped by a source-tree rearrangement.

So we have several problem to solve here, and I agree that source code
rearrangement is fixing none of them.  Maybe it would ease maintaining
down the road, though, but I'll leave that choice up to you.

Which contribs are ready (safe) for production?  We could handle that in
the version numbers, having most of contrib at version 0.9.1 (say) and
some of them at version 1.0.  We could also stop distributing examples
in binary form, only ship them in source package.

Then we need to include inspection extensions into the core packaging,
but still as extensions.  That's more of a packager problem, except that
they need a clear and strong message about it.  Maybe have a new
Makefile and build those extensions as part of the server build, and
install them as part as the normal install.

Another mix of those ideas is to ship inspection extensions and ready
for production ones all into a new package postgresql-server-extensions
that the main server would depend on, and the ones that are adding more
dependencies still in contrib, where not-ready for production extensions
would not get built.

This kind of ideas would also allow to quite easily remove things from
the main server and have them as extensions, available on any install
but a CREATE EXTENSION (WITH SCHEMA pg_catalog) command away.  And still
maintained at the same quality level in the source tree.

Regards,
-- 
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-09 Thread Greg Smith

On 05/09/2011 03:31 PM, Alvaro Herrera wrote:

For executables we already have src/bin. Do we really need a separate
place for, say, pg_standby or pg_upgrade?
   


There's really no executables in contrib that I find myself regularly 
desperate for/angry at because they're not installed as an integral part 
of the server, the way I regularly find myself cursing some things that 
are now extensions.  The only contrib executable I use often is pgbench, 
and that's normally early in server life--when it's still possible to 
get things installed easily most places.  And it's something that can be 
removed when finished, in cases where people are nervous about the 
contrib package.


Situations where pg_standby or pg_upgrade suddenly pop up as emergency 
needs seem unlikely too, which is also the case with oid2name, 
pg_test_fsync, pg_archivecleanup, and vacuumlo.  I've had that happen 
with pg_archivecleanup exactly once since it appeared--running out of 
space and that was the easiest way to make the problem go away 
immediately and permanently--but since it was on an 8.4 server we had to 
download the source and build anyway.


Also, my experience is that people are not that paranoid about running 
external binaries, even though they could potentially do harm to the 
database.  Can always do a backup beforehand.  But the idea of loading a 
piece of code that lives in the server all the time freaks them out.  
The way the word contrib implies (and sometimes is meant to mean) low 
quality, while stuff that ships with the main server package does not, 
has been beaten up here for years already.  It's only a few cases where 
that's not fully justified, and the program can easily become an 
extension, that I feel are really worth changing here.  There are 49 
directories in contrib/ ; at best maybe 20% of them will ever fall into 
that category.


--
Greg Smith   2ndQuadrant USg...@2ndquadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support  www.2ndQuadrant.us



--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-09 Thread Greg Smith

On 05/09/2011 02:31 PM, Robert Haas wrote:

I don't think we should be too obstinate about trying to twist the arm
of packagers who (as Tom points out) will do whatever they want in
spite of us, but the current state of contrib, with all sorts of
things of varying type, complexity, and value mixed together cannot
possibly be a good thing.


I think the idea I'm running with for now means that packagers won't 
actually have to do anything.  I'd expect typical packaging for 9.1 to 
include share/postgresql/extension from the build results without being 
more specific.  You need to grab 3 files from there to get the plpgsql 
extension, and I can't imagine any packager listing them all by name.  
So if I dump the converted contrib extensions to put files under there, 
and remove them from the contrib build area destination, I suspect they 
will magically jump from the contrib to the extensions area of the 
server package at next package build; no packager level changes 
required.  The more I look at this, the less obtrusive of a change it 
seems to be.  Only people who will really notice are users who discover 
more in the basic server package, and of course committers with 
backporting to do.


Since packaged builds of 9.1 current with beta1 seem to be in short 
supply still, this theory looks hard to prove just yet.  I'm very 
excited that it's packaging week here however (rolls eyes), so I'll 
check it myself.  I'll incorporate the suggestions made since I posted 
that test patch and do a bigger round of this next, end to end with an 
RPM set as the output.  It sounds like everyone who has a strong opinion 
on what this change might look like has sketched a similar looking 
bikeshed.  Once a reasonable implementation is hammered out, I'd rather 
jump to the big argument between not liking change vs. the advocacy 
benefits to PostgreSQL of doing this; they are considerable in my mind.


--
Greg Smith   2ndQuadrant USg...@2ndquadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support  www.2ndQuadrant.us



--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-08 Thread Peter Eisentraut
On lör, 2011-05-07 at 17:38 -0400, Andrew Dunstan wrote:
 
 On 05/07/2011 05:26 PM, Peter Eisentraut wrote:
  On lör, 2011-05-07 at 17:16 -0400, Andrew Dunstan wrote:
  pg_config is useful quite apart from its use in building things, as was
  discussed upthread.
  Link please.
 
 
 http://archives.postgresql.org/pgsql-hackers/2011-05/msg00275.php

That thread just asserts that it might be useful, and I responded by
asking for what.


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-08 Thread Andrew Dunstan



On 05/08/2011 05:24 AM, Peter Eisentraut wrote:

On lör, 2011-05-07 at 17:38 -0400, Andrew Dunstan wrote:

On 05/07/2011 05:26 PM, Peter Eisentraut wrote:

On lör, 2011-05-07 at 17:16 -0400, Andrew Dunstan wrote:

pg_config is useful quite apart from its use in building things, as was
discussed upthread.

Link please.


http://archives.postgresql.org/pgsql-hackers/2011-05/msg00275.php

That thread just asserts that it might be useful, and I responded by
asking for what.



As I said there: to see how the libraries are configured, for example.

Just the other day I wanted to know what compilation options had been 
used for a particular installation. pg_config wasn't installed because 
the -devel package wasn't installed, and it would have saved me quite 
some time if pg_config had been available.


Another example is to find out what the installation is using for 
shares, the service directory and so on.


cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-08 Thread Peter Eisentraut
On sön, 2011-05-08 at 07:21 -0400, Andrew Dunstan wrote:
 As I said there: to see how the libraries are configured, for example.
 
 Just the other day I wanted to know what compilation options had been 
 used for a particular installation. pg_config wasn't installed because 
 the -devel package wasn't installed, and it would have saved me quite 
 some time if pg_config had been available.
 
 Another example is to find out what the installation is using for 
 shares, the service directory and so on.

Yeah, those are decent reasons.


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-08 Thread Christopher Browne
My example is of doing self-discovery to see if all needful database
components seem to be properly installed.

E.g. - the app needs pgcrypto, intarray, and a custom data type.  The
install script can consequently inform the production folk either looks
good, or, alternately, seems problematic!

Actually, I haven't coded a sample of the look for custom SPI  types
part, but it's a natural extension of what I have.

Of course, it only provides a legitimate test when run on the database
server, which isn't always how production folk want to do it, but that's
part of a different argument...


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-07 Thread Dimitri Fontaine
Christopher Browne cbbro...@gmail.com writes:
 I don't expect the extension system to help with any of this, since if
 production folk try to install minimal sets of packages, they're
 liable to consciously exclude extension support.  The improvement
 would come from drawing contrib a bit closer to core, and encouraging
 packagers (dpkg, rpm, ports) to fold contrib into base rather than
 separating it.  I'm sure that would get some pushback, though.

What I think the next step is here is to better classify contribs/ into
what we consider production ready core extension (adminpack, hstore,
ltree, pgstattuple, you name it — that's the trick), code example (spi,
some more I guess) and extensibility examples (for hooks or whatnot).

We've been talking about renaming contrib for a long time, but that will
not cut it.  Classifying it and agreeing to maintain some parts of it
the same way we maintain the core is what's asked here.  Is there a will
to go there?

If there's a will to maintain some contribs the way core is maintained
itself, we have to pick a new name for that, and to pick a list of
current contribs to move in there.  Then packagers will either include
that in the main package or have the main package depend on the new one.

Regards,
-- 
Dimitri Fontaine
http://2ndQuadrant.fr PostgreSQL : Expertise, Formation et Support

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-07 Thread Robert Haas
On Sat, May 7, 2011 at 8:54 AM, Dimitri Fontaine dimi...@2ndquadrant.fr wrote:
 We've been talking about renaming contrib for a long time, but that will
 not cut it.  Classifying it and agreeing to maintain some parts of it
 the same way we maintain the core is what's asked here.  Is there a will
 to go there?

I'm game.  I doubt it'll be a lot more maintenance; we already pretty
much patch contrib when we patch everything else.  It'll just be
easier to sort the wheat from the chaff.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-07 Thread Johann 'Myrkraverk' Oskarsson

On Fri, 06 May 2011 20:06:04 -, Tom Lane t...@sss.pgh.pa.us wrote:


Bundling pg_config into a -libs package is probably not going to happen,
at least not on Red Hat systems, because it would create multilib issues
(ie, you're supposed to be able to install 32-bit and 64-bit libraries
concurrently, but there's noplace to put a /usr/bin file without causing
a conflict).


There is no separate directory for 64bit binaries like /usr/bin and
/usr/bin/amd64 on Solaris?  And even in the absence of such a convention
I'd still like to have both 32bit and 64bit binaries of the server/client
available.

I have a feeling this topic is digressing a bit.


--
  Johann Oskarssonhttp://www.2ndquadrant.com/|[]
  PostgreSQL Development, 24x7 Support, Training and Services  --+--
 |
  Blog: http://my.opera.com/myrkraverk/blog/

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-07 Thread Tom Lane
Greg Stark gsst...@mit.edu writes:
 On Fri, May 6, 2011 at 11:32 PM, Greg Smith g...@2ndquadrant.com wrote:
 I use pgstattuple, pageinspect, pg_freespacemap, and pg_buffercache
 regularly enough that I wish they were more common.  Throw in pgrowlocks and
 you've got the whole group Robert put into the debug set.  It makes me sad
 every time I finish a utility using one of these and realize I'll have to
 include the whole make sure you have the contrib modules installed
 disclaimer in its documentation again.

 Well the lightweight way to achieve what you want is to just move
 these functions into core.

I'm completely not in favor of that.  We have spent man-years upon
man-years on making Postgres an extensible system.  If we can't actually
*use* the extension features then that was all a waste of effort.

If anything I'd rather see us looking at what parts of the current core
system could be pushed out to extensions.  The geometric types are a
pretty obvious candidate, for example.

 The only argument I see as particularly frightening on that front is
 people playing the sekurity card.

Yeah, and it's a reasonable argument.  Even if it's not a reasonable
argument, you won't win any friends from the other side of the fence
by taking away their ability to choose.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-07 Thread Peter Eisentraut
On fre, 2011-05-06 at 14:32 -0400, Greg Smith wrote:
 Given the other improvements in being able to build extensions in 9.1,
 we really should push packagers to move pg_config from the PostgreSQL 
 development package into the main one starting in that version.  I've 
 gotten bit by this plenty of times.

Do you need pg_config to install extensions?


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-07 Thread Euler Taveira de Oliveira

Em 07-05-2011 13:42, Peter Eisentraut escreveu:

Do you need pg_config to install extensions?


No. But we need it to build other extensions.


--
  Euler Taveira de Oliveira - Timbira   http://www.timbira.com.br/
  PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-07 Thread Peter Eisentraut
On lör, 2011-05-07 at 17:35 -0300, Euler Taveira de Oliveira wrote:
 Em 07-05-2011 13:42, Peter Eisentraut escreveu:
  Do you need pg_config to install extensions?
 
 No. But we need it to build other extensions.

But for that you need the -dev[el] package anyway, so there would be no
point in moving pg_config out of it.


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-07 Thread Greg Smith

On 05/07/2011 12:42 PM, Peter Eisentraut wrote:

On fre, 2011-05-06 at 14:32 -0400, Greg Smith wrote:
   

Given the other improvements in being able to build extensions in 9.1,
we really should push packagers to move pg_config from the PostgreSQL
development package into the main one starting in that version.  I've
gotten bit by this plenty of times.
 

Do you need pg_config to install extensions?
   


No, but you still need it to build them.  PGXN is a source code 
distribution method, not a binary one.  It presumes users can build 
modules they download using PGXS.  No pg_config, no working PGXS, no 
working PGXN.  For such a small binary to ripple out to that impact is bad.


The repmgr program we distribute has the same problem, so I've been 
getting first-hand reports of just how many people are likely to run 
into this recently.  You have to install postgresql-devel with RPM and 
on Debian, the very non-obvious postgresql-server-dev-$version


Anyway, didn't want to hijack this thread beyond pointing out that if 
there any package reshuffling that happens for contrib changes, it 
should check for and resolve this problem too.


--
Greg Smith   2ndQuadrant USg...@2ndquadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support  www.2ndQuadrant.us



--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-07 Thread Andrew Dunstan



On 05/07/2011 04:43 PM, Peter Eisentraut wrote:

On lör, 2011-05-07 at 17:35 -0300, Euler Taveira de Oliveira wrote:

Em 07-05-2011 13:42, Peter Eisentraut escreveu:

Do you need pg_config to install extensions?


No. But we need it to build other extensions.

But for that you need the -dev[el] package anyway, so there would be no
point in moving pg_config out of it.




pg_config is useful quite apart from its use in building things, as was 
discussed upthread.


cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-07 Thread Peter Eisentraut
On lör, 2011-05-07 at 17:16 -0400, Andrew Dunstan wrote:
 pg_config is useful quite apart from its use in building things, as was 
 discussed upthread.

Link please.


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-07 Thread Peter Eisentraut
On lör, 2011-05-07 at 17:06 -0400, Greg Smith wrote:
 The repmgr program we distribute has the same problem, so I've been 
 getting first-hand reports of just how many people are likely to run 
 into this recently.  You have to install postgresql-devel with RPM and
 on Debian, the very non-obvious postgresql-server-dev-$version 

I'm pretty sure that for installing modules that need compilation from
CPAN or PyPI, you will need the corresponding p*-dev* package installed.
Nothing new here.  I don't think you will get distributors to abandon
the concept of -dev(el) packages for this.  Just saying.


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-07 Thread Andrew Dunstan



On 05/07/2011 05:26 PM, Peter Eisentraut wrote:

On lör, 2011-05-07 at 17:16 -0400, Andrew Dunstan wrote:

pg_config is useful quite apart from its use in building things, as was
discussed upthread.

Link please.



http://archives.postgresql.org/pgsql-hackers/2011-05/msg00275.php

cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-07 Thread Greg Smith

On 05/06/2011 04:06 PM, Tom Lane wrote:

FWIW, I did move pg_config from -devel to the main (really client)
postgresql package in Fedora, as of 9.0.  That will ensure it's present
in either client or server installations.  Eventually that packaging
will reach RHEL ...
   


We should make sure that the PGDG packages adopt that for 9.1 then, so 
it starts catching on more.  Unless Devrim changed to catch up since I 
last installed an RPM set, in that 9.0 it's still in the same place:


$ rpm -qf /usr/pgsql-9.0/bin/pg_config
postgresql90-devel-9.0.2-2PGDG.rhel5

While Peter's question about whether it's really all that useful is 
reasonable, I'd at least like to get a better error message when you 
don't have everything needed to compile extensions.  I think the 
shortest path to that is making pg_config more likely to be installed, 
then to check whether the file pg_config --pgxs references exists.  
I'll see if I can turn that idea into an actual change to propose.


--
Greg Smith   2ndQuadrant USg...@2ndquadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support  www.2ndQuadrant.us



--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-07 Thread Greg Smith
Attached patch is a first cut at what moving one contrib module (in this 
case pg_buffercache) to a new directory structure might look like.  The 
idea is that src/extension could become a place for first-class 
extensions to live.  Those are ones community is committed to providing 
in core, but are just better implemented as extensions than in-database 
functions, for reasons that include security.  This idea has been shared 
by a lot of people for a while, only problem is that it wasn't really 
practical to implement cleanly until the extensions code hit.  I think 
it is now, this attempts to prove it.


Since patches involving file renaming are clunky, the changes might be 
easier to see at 
https://github.com/greg2ndQuadrant/postgres/commit/507923e21e963c873a84f1b850d64e895776574f 
where I just pushed this change too.  The install step for the modules 
looks like this now:


gsmith@grace:~/pgwork/src/move-contrib/src/extension/pg_buffercache$ 
make install

/bin/mkdir -p '/home/gsmith/pgwork/inst/move-contrib/lib/postgresql'
/bin/mkdir -p 
'/home/gsmith/pgwork/inst/move-contrib/share/postgresql/extension'
/bin/sh ../../../config/install-sh -c -m 755  pg_buffercache.so 
'/home/gsmith/pgwork/inst/move-contrib/lib/postgresql/pg_buffercache.so'
/bin/sh ../../../config/install-sh -c -m 644 ./pg_buffercache.control 
'/home/gsmith/pgwork/inst/move-contrib/share/postgresql/extension/'
/bin/sh ../../../config/install-sh -c -m 644 ./pg_buffercache--1.0.sql 
./pg_buffercache--unpackaged--1.0.sql  
'/home/gsmith/pgwork/inst/move-contrib/share/postgresql/extension/'

$ psql -c create extension pg_buffercache
CREATE EXTENSION

The only clunky bit I wasn't really happy with is the amount of code 
duplication that comes from having a src/extension/Makefile that looks 
almost, but not quite, identical to contrib/Makefile.  The rest of the 
changes don't seem too bad to me, and even that's really only 36 lines 
that aren't touched often.  Yes, the paths are different, so backports 
won't happen without an extra step.  But the code changes required were 
easier than I was expecting, due to the general good modularity of the 
extensions infrastructure.  So long as the result ends up in 
share/postgresql/extension/ , whether they started in contrib/module 
or src/extension/module doesn't really matter to CREATE EXTENSION.  
But having them broke out this way makes it easy for the default 
Makefile to build and install them all.  (I recognize I didn't do that 
last step yet though)


I'll happily go covert pgstattuple and the rest of the internal 
diagnostics modules to this scheme, and do the doc cleanups, this 
upcoming week if it means I'll be able to use those things without 
installing all of contrib one day.  Ditto for proposing RPM and Debian 
packaging changes that match them.  All that work will get paid back the 
first time I don't have to fill out a bunch of paperwork (again) at a 
customer site justifying why they need to install the contrib [RPM|deb] 
package (which has some scary stuff in it) on all their servers, just so 
I can get some bloat or buffer inspection module.


--
Greg Smith   2ndQuadrant USg...@2ndquadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support  www.2ndQuadrant.us


diff --git a/contrib/Makefile b/contrib/Makefile
index 6967767..04cf330 100644
--- a/contrib/Makefile
+++ b/contrib/Makefile
@@ -30,7 +30,6 @@ SUBDIRS = \
 		pageinspect	\
 		passwordcheck	\
 		pg_archivecleanup \
-		pg_buffercache	\
 		pg_freespacemap \
 		pg_standby	\
 		pg_stat_statements \
diff --git a/contrib/pg_buffercache/Makefile b/contrib/pg_buffercache/Makefile
deleted file mode 100644
index 323c0ac..000
--- a/contrib/pg_buffercache/Makefile
+++ /dev/null
@@ -1,18 +0,0 @@
-# contrib/pg_buffercache/Makefile
-
-MODULE_big = pg_buffercache
-OBJS = pg_buffercache_pages.o
-
-EXTENSION = pg_buffercache
-DATA = pg_buffercache--1.0.sql pg_buffercache--unpackaged--1.0.sql
-
-ifdef USE_PGXS
-PG_CONFIG = pg_config
-PGXS := $(shell $(PG_CONFIG) --pgxs)
-include $(PGXS)
-else
-subdir = contrib/pg_buffercache
-top_builddir = ../..
-include $(top_builddir)/src/Makefile.global
-include $(top_srcdir)/contrib/contrib-global.mk
-endif
diff --git a/contrib/pg_buffercache/pg_buffercache--1.0.sql b/contrib/pg_buffercache/pg_buffercache--1.0.sql
deleted file mode 100644
index 9407d21..000
--- a/contrib/pg_buffercache/pg_buffercache--1.0.sql
+++ /dev/null
@@ -1,17 +0,0 @@
-/* contrib/pg_buffercache/pg_buffercache--1.0.sql */
-
--- Register the function.
-CREATE FUNCTION pg_buffercache_pages()
-RETURNS SETOF RECORD
-AS 'MODULE_PATHNAME', 'pg_buffercache_pages'
-LANGUAGE C;
-
--- Create a view for convenient access.
-CREATE VIEW pg_buffercache AS
-	SELECT P.* FROM pg_buffercache_pages() AS P
-	(bufferid integer, relfilenode oid, reltablespace oid, reldatabase oid,
-	 relforknumber int2, relblocknumber int8, isdirty bool, usagecount int2);
-
--- Don't want these to be available to public.
-REVOKE ALL ON 

Re: [HACKERS] Why not install pgstattuple by default?

2011-05-06 Thread Magnus Hagander
On Fri, May 6, 2011 at 00:34, Josh Berkus josh.ber...@pgexperts.com wrote:
 Hackers,

 I've run into a couple of occasions lately where I really wanted
 pgstattuple on a production server in order to check table/index bloat.
  However, in the production environment at a large site installing a
 contrib module can involve a process which takes days or weeks.

That can be said for a lot of things in contrib. pg_standby in 8.4 for
example. Or adminpack. Or dblink. Or hstore. There's a mix of example
stuff and actually pretty darn useful in production stuff. I'm sure
you can find a couple of hundred emails in the archives on this very
topic.

From 9.1, it'll be a simple CREATE EXTENSION command - so much of the
problem goes away. Well. It doesn't go away, but it gets a lot more
neatly swept under the rug.

-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-06 Thread Euler Taveira de Oliveira

Em 06-05-2011 05:06, Magnus Hagander escreveu:

On Fri, May 6, 2011 at 00:34, Josh Berkusjosh.ber...@pgexperts.com  wrote:

Hackers,

I've run into a couple of occasions lately where I really wanted
pgstattuple on a production server in order to check table/index bloat.
  However, in the production environment at a large site installing a
contrib module can involve a process which takes days or weeks.



I already faced that problem too.


 From 9.1, it'll be a simple CREATE EXTENSION command - so much of the
problem goes away. Well. It doesn't go away, but it gets a lot more
neatly swept under the rug.

That's half of the history. Admin needs to install postgresql-contrib package. 
Sometimes it takes too much time to convince clients that some additional 
supplied modules are useful for them.


Now that we have extensions, why not build and package the contrib modules by 
default? 'make world' is not the answer. There is not an option for install 
all pieces of software. Let's install pg+contrib and leave only 'CREATE 
EXTENSION foo' for the admins.



--
  Euler Taveira de Oliveira - Timbira   http://www.timbira.com.br/
  PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-06 Thread Magnus Hagander
On Fri, May 6, 2011 at 18:22, Euler Taveira de Oliveira
eu...@timbira.com wrote:
 Em 06-05-2011 05:06, Magnus Hagander escreveu:

 On Fri, May 6, 2011 at 00:34, Josh Berkusjosh.ber...@pgexperts.com
  wrote:

 Hackers,

 I've run into a couple of occasions lately where I really wanted
 pgstattuple on a production server in order to check table/index bloat.
  However, in the production environment at a large site installing a
 contrib module can involve a process which takes days or weeks.

 I already faced that problem too.

  From 9.1, it'll be a simple CREATE EXTENSION command - so much of the
 problem goes away. Well. It doesn't go away, but it gets a lot more
 neatly swept under the rug.

 That's half of the history. Admin needs to install postgresql-contrib
 package. Sometimes it takes too much time to convince clients that some
 additional supplied modules are useful for them.

 Now that we have extensions, why not build and package the contrib modules
 by default? 'make world' is not the answer. There is not an option for
 install all pieces of software. Let's install pg+contrib and leave only
 'CREATE EXTENSION foo' for the admins.

That's mostly an issue to be solved by the packagers. Some contrib
modules add dependencies, but those that don't could easily be
packaged in the main server package.

-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-06 Thread Christopher Browne
On Fri, May 6, 2011 at 1:32 PM, Magnus Hagander mag...@hagander.net wrote:
 On Fri, May 6, 2011 at 18:22, Euler Taveira de Oliveira
 eu...@timbira.com wrote:
 Em 06-05-2011 05:06, Magnus Hagander escreveu:

 On Fri, May 6, 2011 at 00:34, Josh Berkusjosh.ber...@pgexperts.com
  wrote:

 Hackers,

 I've run into a couple of occasions lately where I really wanted
 pgstattuple on a production server in order to check table/index bloat.
  However, in the production environment at a large site installing a
 contrib module can involve a process which takes days or weeks.

 I already faced that problem too.

  From 9.1, it'll be a simple CREATE EXTENSION command - so much of the
 problem goes away. Well. It doesn't go away, but it gets a lot more
 neatly swept under the rug.

 That's half of the history. Admin needs to install postgresql-contrib
 package. Sometimes it takes too much time to convince clients that some
 additional supplied modules are useful for them.

 Now that we have extensions, why not build and package the contrib modules
 by default? 'make world' is not the answer. There is not an option for
 install all pieces of software. Let's install pg+contrib and leave only
 'CREATE EXTENSION foo' for the admins.

 That's mostly an issue to be solved by the packagers. Some contrib
 modules add dependencies, but those that don't could easily be
 packaged in the main server package.

It seems to me that there's something of a packaging policy question to this.

A long time ago, on a pre-buildfarm planet, far, far away, it was
pretty uncertain what contrib modules could be hoped to run on what
platform.

At Afilias, we used to have to be *really* picky, because the subset
that ran on Solaris and AIX were not even close to all of them.
pgstattuples *was* one that the DBAs always wanted, but what would
compile was alway hit-and-miss.

Once we got AIX running a buildfarm node, that led to getting *ALL* of
contrib working there, and I'm pretty sure that similar happened with
other platforms at around the same time (I'm thinking this was 7.4,
but it might have been 8.0)

Be that all as it may, there has been a sea change, where we have
moved from sporadic usability of contrib to it being *continually*
tested on *all* buildfarm platforms, which certainly adds to the
confidence level.

But people are evidently still setting packaging policies based on how
things were back in 7.3, even though that perhaps isn't necessary
anymore.

Certainly it's not a huge amount of code; less than 2MB these days.
- % wc `dpkg -L postgresql-contrib-9.0` | tail -1
  15952   67555 1770987 total

I'm getting paper cuts quite a bit these days over the differences
between what different packaging systems decide to install.  The one
*I* get notably bit on, of late, is that I have written code that
expects to have pg_config to do some degree of self-discovery, only to
find production folk complaining that they only have psql available
in their environment.

I don't expect the extension system to help with any of this, since if
production folk try to install minimal sets of packages, they're
liable to consciously exclude extension support.  The improvement
would come from drawing contrib a bit closer to core, and encouraging
packagers (dpkg, rpm, ports) to fold contrib into base rather than
separating it.  I'm sure that would get some pushback, though.
-- 
When confronted by a difficult problem, solve it by reducing it to the
question, How would the Lone Ranger handle this?

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-06 Thread Andrew Dunstan



On 05/06/2011 01:55 PM, Christopher Browne wrote:


Once we got AIX running a buildfarm node, that led to getting *ALL* of
contrib working there, and I'm pretty sure that similar happened with
other platforms at around the same time (I'm thinking this was 7.4,
but it might have been 8.0)


FYI, the buildfarm started in late 2004, near the end of the 8.0 
development cycle. It quickly led to a number of contrib fixes.


Time flies when you're having fun ...

cheers

andrew



--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-06 Thread Euler Taveira de Oliveira

Em 06-05-2011 14:55, Christopher Browne escreveu:

The improvement
would come from drawing contrib a bit closer to core, and encouraging
packagers (dpkg, rpm, ports) to fold contrib into base rather than
separating it.  I'm sure that would get some pushback, though.


I'm in favor of find out what are the popular extensions and make them into 
base; the other ones could be moved to PGXN.



--
  Euler Taveira de Oliveira - Timbira   http://www.timbira.com.br/
  PostgreSQL: Consultoria, Desenvolvimento, Suporte 24x7 e Treinamento

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-06 Thread Greg Smith

Christopher Browne wrote:

I'm getting paper cuts quite a bit these days over the differences
between what different packaging systems decide to install.  The one
*I* get notably bit on, of late, is that I have written code that
expects to have pg_config to do some degree of self-discovery, only to
find production folk complaining that they only have psql available
in their environment.
  


Given the other improvements in being able to build extensions in 9.1, 
we really should push packagers to move pg_config from the PostgreSQL 
development package into the main one starting in that version.  I've 
gotten bit by this plenty of times.


--
Greg Smith   2ndQuadrant USg...@2ndquadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support  www.2ndQuadrant.us



--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-06 Thread Christopher Browne
On Fri, May 6, 2011 at 2:32 PM, Greg Smith g...@2ndquadrant.com wrote:
 Christopher Browne wrote:

 I'm getting paper cuts quite a bit these days over the differences
 between what different packaging systems decide to install.  The one
 *I* get notably bit on, of late, is that I have written code that
 expects to have pg_config to do some degree of self-discovery, only to
 find production folk complaining that they only have psql available
 in their environment.

 Given the other improvements in being able to build extensions in 9.1, we
 really should push packagers to move pg_config from the PostgreSQL
 development package into the main one starting in that version.  I've gotten
 bit by this plenty of times.

I'm agreeable to that, in general.

If there's a server package and a client package, it likely only
fits with the server package.  On a host where only the client is
installed, they won't be able to install extensions, so it's pretty
futile to have it there.
-- 
When confronted by a difficult problem, solve it by reducing it to the
question, How would the Lone Ranger handle this?

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-06 Thread Andrew Dunstan



On 05/06/2011 03:14 PM, Christopher Browne wrote:

On Fri, May 6, 2011 at 2:32 PM, Greg Smithg...@2ndquadrant.com  wrote:

Christopher Browne wrote:

I'm getting paper cuts quite a bit these days over the differences
between what different packaging systems decide to install.  The one
*I* get notably bit on, of late, is that I have written code that
expects to have pg_config to do some degree of self-discovery, only to
find production folk complaining that they only have psql available
in their environment.

Given the other improvements in being able to build extensions in 9.1, we
really should push packagers to move pg_config from the PostgreSQL
development package into the main one starting in that version.  I've gotten
bit by this plenty of times.

I'm agreeable to that, in general.

If there's a server package and a client package, it likely only
fits with the server package.  On a host where only the client is
installed, they won't be able to install extensions, so it's pretty
futile to have it there.


I don't agree. It can be useful even there, to see how the libraries are 
configured, for example. I'd be inclined to bundle it with 
postgresql-libs or the moral equivalent.


cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-06 Thread Magnus Hagander
On Fri, May 6, 2011 at 21:19, Andrew Dunstan and...@dunslane.net wrote:


 On 05/06/2011 03:14 PM, Christopher Browne wrote:

 On Fri, May 6, 2011 at 2:32 PM, Greg Smithg...@2ndquadrant.com  wrote:

 Christopher Browne wrote:

 I'm getting paper cuts quite a bit these days over the differences
 between what different packaging systems decide to install.  The one
 *I* get notably bit on, of late, is that I have written code that
 expects to have pg_config to do some degree of self-discovery, only to
 find production folk complaining that they only have psql available
 in their environment.

 Given the other improvements in being able to build extensions in 9.1, we
 really should push packagers to move pg_config from the PostgreSQL
 development package into the main one starting in that version.  I've
 gotten
 bit by this plenty of times.

 I'm agreeable to that, in general.

 If there's a server package and a client package, it likely only
 fits with the server package.  On a host where only the client is
 installed, they won't be able to install extensions, so it's pretty
 futile to have it there.

 I don't agree. It can be useful even there, to see how the libraries are
 configured, for example. I'd be inclined to bundle it with postgresql-libs
 or the moral equivalent.

+1.

And it's not like it wastes huge amount of space...


-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-06 Thread Tom Lane
Magnus Hagander mag...@hagander.net writes:
 On Fri, May 6, 2011 at 21:19, Andrew Dunstan and...@dunslane.net wrote:
 On 05/06/2011 03:14 PM, Christopher Browne wrote:
 If there's a server package and a client package, it likely only
 fits with the server package.  On a host where only the client is
 installed, they won't be able to install extensions, so it's pretty
 futile to have it there.

 I don't agree. It can be useful even there, to see how the libraries are
 configured, for example. I'd be inclined to bundle it with postgresql-libs
 or the moral equivalent.

 +1.

Well, actually, I think packagers have generally put it into a -devel
subpackage.  If it were in either a server or client package there
would be much less of an issue.

Bundling pg_config into a -libs package is probably not going to happen,
at least not on Red Hat systems, because it would create multilib issues
(ie, you're supposed to be able to install 32-bit and 64-bit libraries
concurrently, but there's noplace to put a /usr/bin file without causing
a conflict).

FWIW, I did move pg_config from -devel to the main (really client)
postgresql package in Fedora, as of 9.0.  That will ensure it's present
in either client or server installations.  Eventually that packaging
will reach RHEL ...

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-06 Thread Andrew Dunstan



On 05/06/2011 04:06 PM, Tom Lane wrote:

Magnus Hagandermag...@hagander.net  writes:

On Fri, May 6, 2011 at 21:19, Andrew Dunstanand...@dunslane.net  wrote:

On 05/06/2011 03:14 PM, Christopher Browne wrote:

If there's a server package and a client package, it likely only
fits with the server package.  On a host where only the client is
installed, they won't be able to install extensions, so it's pretty
futile to have it there.

I don't agree. It can be useful even there, to see how the libraries are
configured, for example. I'd be inclined to bundle it with postgresql-libs
or the moral equivalent.

+1.

Well, actually, I think packagers have generally put it into a -devel
subpackage.  If it were in either a server or client package there
would be much less of an issue.

Bundling pg_config into a -libs package is probably not going to happen,
at least not on Red Hat systems, because it would create multilib issues
(ie, you're supposed to be able to install 32-bit and 64-bit libraries
concurrently, but there's noplace to put a /usr/bin file without causing
a conflict).

FWIW, I did move pg_config from -devel to the main (really client)
postgresql package in Fedora, as of 9.0.  That will ensure it's present
in either client or server installations.  Eventually that packaging
will reach RHEL ...




That's reasonable, and certainly better than having it in -devel.

cheers

andrew

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-06 Thread Tom Lane
Christopher Browne cbbro...@gmail.com writes:
 But people are evidently still setting packaging policies based on how
 things were back in 7.3, even though that perhaps isn't necessary
 anymore.

FWIW, once you get past the client versus server distinction, I think
most subpackaging decisions are based on either the idea that only a
minority of people will want this, or a desire to limit how many
dependencies are pulled in by the main package(s).  Both of those
concerns apply to various subsets of -contrib, which means it's going
to be hard to persuade packagers to fold -contrib into the -server
package altogether.  Nor would you gain their approval by trying to
pre-empt the decision.

We might get somewhere by trying to identify a small set of particularly
popular contrib modules that don't add any extra dependencies, and then
recommending to packagers that those ones get bundled into the main
server package.

 Certainly it's not a huge amount of code; less than 2MB these days.
 - % wc `dpkg -L postgresql-contrib-9.0` | tail -1
   15952   67555 1770987 total

Well, to add some concrete facts rather than generalities to my own post,
here are the sizes of the built RPMs from my last build for Fedora:

-rw-r--r--. 1 tgl tgl  3839458 Apr 18 10:50 postgresql-9.0.4-1.fc13.x86_64.rpm
-rw-r--r--. 1 tgl tgl   490788 Apr 18 10:50 
postgresql-contrib-9.0.4-1.fc13.x86_64.rpm
-rw-r--r--. 1 tgl tgl 27337677 Apr 18 10:51 
postgresql-debuginfo-9.0.4-1.fc13.x86_64.rpm
-rw-r--r--. 1 tgl tgl   961660 Apr 18 10:50 
postgresql-devel-9.0.4-1.fc13.x86_64.rpm
-rw-r--r--. 1 tgl tgl  7569048 Apr 18 10:50 
postgresql-docs-9.0.4-1.fc13.x86_64.rpm
-rw-r--r--. 1 tgl tgl   246506 Apr 18 10:50 
postgresql-libs-9.0.4-1.fc13.x86_64.rpm
-rw-r--r--. 1 tgl tgl64940 Apr 18 10:50 
postgresql-plperl-9.0.4-1.fc13.x86_64.rpm
-rw-r--r--. 1 tgl tgl65776 Apr 18 10:50 
postgresql-plpython-9.0.4-1.fc13.x86_64.rpm
-rw-r--r--. 1 tgl tgl45941 Apr 18 10:50 
postgresql-pltcl-9.0.4-1.fc13.x86_64.rpm
-rw-r--r--. 1 tgl tgl  5302117 Apr 18 10:50 
postgresql-server-9.0.4-1.fc13.x86_64.rpm
-rw-r--r--. 1 tgl tgl  1370509 Apr 18 10:50 
postgresql-test-9.0.4-1.fc13.x86_64.rpm
-rw-r--r--. 1 tgl tgl  3644113 Apr 18 10:50 
postgresql-upgrade-9.0.4-1.fc13.x86_64.rpm

The separate debuginfo package is distro policy enforced by toolchain;
I couldn't do anything about that even if I wanted to.  The separate
-libs subpackage is also hard to avoid because of distro policy about
multilib installations.  Separating devel support files (such as
headers) is also standard practice.  The other subdivisions are either
my fault or those of my predecessors.  plperl, plpython, and pltcl are
split out for dependency reasons, ie to not have the -server package
require you to install those languages and their respective ecosystems.
I think the separation of the -docs, -test, and -upgrade subpackages is
also pretty easy to defend on the grounds that they're big and not
everyone wants 'em, especially not in production.

That leaves us with these three subpackages about which there's room
for argument:

-rw-r--r--. 1 tgl tgl  3839458 Apr 18 10:50 postgresql-9.0.4-1.fc13.x86_64.rpm
-rw-r--r--. 1 tgl tgl   490788 Apr 18 10:50 
postgresql-contrib-9.0.4-1.fc13.x86_64.rpm
-rw-r--r--. 1 tgl tgl  5302117 Apr 18 10:50 
postgresql-server-9.0.4-1.fc13.x86_64.rpm

Merging -contrib into the server package would increase the size of the
latter by almost 10%, which is enough to bother people.  Also, a bit of
dependency extraction shows that -contrib has these dependencies beyond
the ones in the two main packages:

libcrypt.so.1
libossp-uuid.so.16
libxslt.so.1

That's not a particularly large list, I guess, but they're still the
sorts of dependencies that don't win any friends when it's time to get
the distro to fit on a DVD.

Bottom line is that I'd rather have a smaller postgresql-server package
that gets included in the shipping DVD than a complete one that gets
kicked off because it's too large and pulls in too many other non-core
dependencies.

So, again, some selective migration of contrib modules into the main
-server package might be doable, but the key word there is selective.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-06 Thread Josh Berkus
All,

 We might get somewhere by trying to identify a small set of particularly
 popular contrib modules that don't add any extra dependencies, and then
 recommending to packagers that those ones get bundled into the main
 server package.

Yeah, I wasn't thinking of including all of contrib.  There's a lot of
reasons not to do that.  I was asking about pgstattuple in particular,
since it's:
(a) small
(b) has no external dependancies
(c) adds no stability risk or performance overhead
(d) is usually needed on production systems when it's needed at all

It's possible that we have one or two other diagnostic utilities which
meet the above profile. pageinspect, maybe?

The reason why this is such an issue is that for big users with
high-demand production environments, installing any software, even
postgresql-devel or postgresql-contrib packages, are big major IT deals
which require weeks of advance scheduling.  As a result, diagnostic
tools from contrib tend not to be used because the problem they need to
diagnose is much more urgent than that.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-06 Thread Robert Haas
On Fri, May 6, 2011 at 5:58 PM, Josh Berkus j...@agliodbs.com wrote:
 Yeah, I wasn't thinking of including all of contrib.  There's a lot of
 reasons not to do that.

Slightly off-topic, but I really think we would benefit from trying to
divide up contrib.  Right now it's a mixture of (a) debugging and
instrumentation tools (e.g. pgstattuple, pageinspect, pgrowlocks,
pg_freespacemap, pg_buffercache), (b) server functionality that is
generally useful but considered worth including in core (e.g. hstore,
citext, pg_trgm), (c) deprecated modules that we keep around mostly
for hysterical reasons (tsearch2, xml2, intagg), and (d) examples and
regression test support (dummy_seclabel, spi, start-scripts).  I think
it would make things a lot easier for both packagers and actual users
if we separated these things into different directories, e.g.:

debugging and instrumentation tools - src/debug
server functionality - contrib
server functionality (deprecated) - contrib/deprecated
examples  regression test suport - src/test/examples

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-06 Thread Greg Smith

On 05/06/2011 05:58 PM, Josh Berkus wrote:

Yeah, I wasn't thinking of including all of contrib. There's a lot of
reasons not to do that.  I was asking about pgstattuple in particular,
since it's:
(a) small
(b) has no external dependancies
(c) adds no stability risk or performance overhead
(d) is usually needed on production systems when it's needed at all

It's possible that we have one or two other diagnostic utilities which
meet the above profile. pageinspect, maybe?
   


I use pgstattuple, pageinspect, pg_freespacemap, and pg_buffercache 
regularly enough that I wish they were more common.  Throw in pgrowlocks 
and you've got the whole group Robert put into the debug set.  It makes 
me sad every time I finish a utility using one of these and realize I'll 
have to include the whole make sure you have the contrib modules 
installed disclaimer in its documentation again.


These are the only ones I'd care about moving into a more likely place.  
The rest of the contrib modules are the sort where if you need them, you 
realize that early and get them installed.  These are different by 
virtue of their need popping up most often during emergencies.  The fact 
that I believe they all match the low impact criteria too makes it even 
easier to consider.


--
Greg Smith   2ndQuadrant USg...@2ndquadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support  www.2ndQuadrant.us



--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-06 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes:
 On Fri, May 6, 2011 at 5:58 PM, Josh Berkus j...@agliodbs.com wrote:
 Yeah, I wasn't thinking of including all of contrib.  There's a lot of
 reasons not to do that.

 Slightly off-topic, but I really think we would benefit from trying to
 divide up contrib. [ snip ]
 I think
 it would make things a lot easier for both packagers and actual users
 if we separated these things into different directories, e.g.:

 debugging and instrumentation tools - src/debug
 server functionality - contrib
 server functionality (deprecated) - contrib/deprecated
 examples  regression test suport - src/test/examples

From a packager's standpoint, that would be entirely worthless.  The
source tree's just a source tree, they don't care what lives where
within it.  I was just thinking about what it'd take to actually
repackage things for Fedora, and the main problem is here:

%files contrib
...
%{_datadir}/pgsql/contrib/
...

If you're not adept at reading RPM specfiles, what that is saying
is that everything that make install has stuck under
${prefix}/share/pgsql/contrib/ is to be included in the contrib RPM.
To selectively move some stuff to the server RPM, I'd have to replace
that one line with a file-by-file list of *everything* in share/contrib,
and then move some of those lines to the %files server section, and
then look forward to having to maintain that list in future versions.
I'm already maintaining a file-by-file list of contrib's .so's, and I
can tell you it's a PITA.

As a packager, what I'd really want to see from a division into
recommended and not-so-recommended packages is that they get installed
into different subdirectories by make install.  Then I could just
point RPM at those directories and I'd be done.

I don't know how practical this is from our development standpoint,
nor from a user's standpoint --- I doubt we want to ask people to use
different CREATE EXTENSION commands depending on the preferredness of
the extension.

A possibly workable compromise would be to provide two separate makefile
installation targets for preferred and less preferred modules.  The RPM
script could then do something like

make install-contrib-preferred
ls -R .../sharedir contrib.files.for.server-package
make install-contrib-second-class-citizens
ls -R .../sharedir all.contrib.files
... and then some magic with comm to separate out the contrib
... files not mentioned in contrib.files.for.server-package ...

Pretty grotty but it would work.  Anyway my point is that this is all
driven off the *installed* file tree.  A specfile writer doesn't know
nor want to know where make install is getting things from in the
source tree.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-06 Thread Josh Berkus
On 5/6/11 3:19 PM, Robert Haas wrote:
 Slightly off-topic, but I really think we would benefit from trying to
 divide up contrib.

I don't agree, unless by divide up you mean move several things to
extensions.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-06 Thread Josh Berkus

 These are the only ones I'd care about moving into a more likely place. 
 The rest of the contrib modules are the sort where if you need them, you
 realize that early and get them installed.  These are different by
 virtue of their need popping up most often during emergencies.  The fact
 that I believe they all match the low impact criteria too makes it even
 easier to consider.

Yes, precisely.  If I need intarray, I'm going to need it for a
development push, which is planned well in advance.  But if I need
pageinspect, it's almost certainly because an emergency has arisen.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-06 Thread Greg Stark
On Fri, May 6, 2011 at 11:32 PM, Greg Smith g...@2ndquadrant.com wrote:
 I use pgstattuple, pageinspect, pg_freespacemap, and pg_buffercache
 regularly enough that I wish they were more common.  Throw in pgrowlocks and
 you've got the whole group Robert put into the debug set.  It makes me sad
 every time I finish a utility using one of these and realize I'll have to
 include the whole make sure you have the contrib modules installed
 disclaimer in its documentation again.

Well the lightweight way to achieve what you want is to just move
these functions into core. There's a pretty good argument to be made
for debugging tools being considered an integral part of a base
system. I remember making the same argument when Sun first made the
radical move for a Unix vendor to stop shipping a working C compiler
and debugger as part of the base Solaris packages.

The only argument I see as particularly frightening on that front is
people playing the sekurity card. A naive attacker who obtains access
to the postgres account could do more damage than they might be able
to do without these modules installed. Of course an attacker with
postgres can do just about anything but it's not entirely baseless
--  we don't set up the database with modules like plsh installed by
default for example.

The only actual security issue I can think of is that the pageinspect
module would let users look at deleted records more easily. It would
be pretty tricky, but not impossible, to do that without it.

-- 
greg

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-06 Thread Robert Haas
On Fri, May 6, 2011 at 6:47 PM, Tom Lane t...@sss.pgh.pa.us wrote:
 As a packager, what I'd really want to see from a division into
 recommended and not-so-recommended packages is that they get installed
 into different subdirectories by make install.  Then I could just
 point RPM at those directories and I'd be done.

Well, that might be good, too.  But, right now, if someone pulls up
our documentation, or our source tree, they could easily be forgiven
for thinking that hstore and dummy_seclabel are comparable, and they
aren't.

 I don't know how practical this is from our development standpoint,
 nor from a user's standpoint --- I doubt we want to ask people to use
 different CREATE EXTENSION commands depending on the preferredness of
 the extension.

Certainly not.

 A possibly workable compromise would be to provide two separate makefile
 installation targets for preferred and less preferred modules.  The RPM
 script could then do something like

        make install-contrib-preferred
        ls -R .../sharedir contrib.files.for.server-package
        make install-contrib-second-class-citizens
        ls -R .../sharedir all.contrib.files
        ... and then some magic with comm to separate out the contrib
        ... files not mentioned in contrib.files.for.server-package ...

 Pretty grotty but it would work.  Anyway my point is that this is all
 driven off the *installed* file tree.  A specfile writer doesn't know
 nor want to know where make install is getting things from in the
 source tree.

This isn't any uglier than some other RPM hacks I've seen, and less
ugly than some, but you'd have a better sense of that than I do.  At
any rate, having the various categories separated in the source tree
can't possibly hurt the effort to make something like this work, and
might make it somewhat easier.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-06 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes:
 On Fri, May 6, 2011 at 6:47 PM, Tom Lane t...@sss.pgh.pa.us wrote:
 As a packager, what I'd really want to see from a division into
 recommended and not-so-recommended packages is that they get installed
 into different subdirectories by make install.

 Well, that might be good, too.  But, right now, if someone pulls up
 our documentation, or our source tree, they could easily be forgiven
 for thinking that hstore and dummy_seclabel are comparable, and they
 aren't.

Sure, but that's a documentation issue, which again is not going to be
helped by a source-tree rearrangement.

As somebody who spends a lot of time on back-patching, I'm not excited
in the least by suggestions to rearrange the source tree for marginal
cosmetic benefits, which is all that I see here.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Why not install pgstattuple by default?

2011-05-06 Thread Robert Haas
On Fri, May 6, 2011 at 9:17 PM, Tom Lane t...@sss.pgh.pa.us wrote:
 Robert Haas robertmh...@gmail.com writes:
 On Fri, May 6, 2011 at 6:47 PM, Tom Lane t...@sss.pgh.pa.us wrote:
 As a packager, what I'd really want to see from a division into
 recommended and not-so-recommended packages is that they get installed
 into different subdirectories by make install.

 Well, that might be good, too.  But, right now, if someone pulls up
 our documentation, or our source tree, they could easily be forgiven
 for thinking that hstore and dummy_seclabel are comparable, and they
 aren't.

 Sure, but that's a documentation issue, which again is not going to be
 helped by a source-tree rearrangement.

I disagree - I think it would be helpful to rearrange both things.

 As somebody who spends a lot of time on back-patching, I'm not excited
 in the least by suggestions to rearrange the source tree for marginal
 cosmetic benefits, which is all that I see here.

I understand, but we have back-patched only 32 patches that touch
contrib into REL9_0_STABLE since its creation, of which 9 were done by
you, and only 4 of those would have required adjustment under the
separation criteria I proposed.  I think, therefore, that the impact
would be bearable.  Source-code rearrangement is never going to be
completely free, but that seems like a tolerable level of annoyance.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers