Re: [HACKERS] Proof of concept: standalone backend with full FE/BE protocol

2012-09-08 Thread Albert Cervera i Areny
A Dijous, 6 de setembre de 2012 00:30:53, Josh Berkus va escriure:
  In general I think the selling point for such a feature would be no
  administrative hassles, and I believe that has to go not only for the
  end-user experience but also for the application-developer experience.
  If you have to manage checkpointing and vacuuming in the application,
  you're probably soon going to look for another database.
 
 Well, don't discount the development/testing case.  If you do agile or
 TDD (a lot of people do), you often have a workload which looks like:
 
 1) Start framework
 2) Start database
 3) Load database with test data
 4) Run tests
 5) Print results
 6) Shut down database
 
 In a case like that, you can live without checkpointing, even; the
 database is ephemeral.
 
 In other words, let's make this a feature and document it for use in
 testing, and that it's not really usable for production embedded apps yet.

+1.

Some projects such as tryton would benefit from this feature.

-- 
Albert Cervera i Areny
http://www.NaN-tic.com
Tel: +34 93 553 18 03

http://twitter.com/albertnan 
http://www.nan-tic.com/blog


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] build farm machine using make -j 8 mixed results

2012-09-08 Thread Tom Lane
Andrew Dunstan and...@dunslane.net writes:
 This seems totally stupid, but it happens when the path to the current 
 directory includes a cross-device symlink. If I cd following the link, 
 then this effect doesn't happen. Weird.

Huh.  So maybe a gmake bug, or maybe there's something wrong with our
make rules for the case that `pwd` doesn't match what gmake thinks the
current path is.  Could you specify the setup more fully?

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


[HACKERS] patch: disable bypass of expand variadic for ANY variable function, format_array function for formatting with params in array

2012-09-08 Thread Pavel Stehule
Hello

This patch disable bypassing of parameters for variadic function with
ANY type variadic parameter. Now - this functionality is just
broken. Because there are no any requests for fixing this issue, I
propose the most simply solution - just disable using this type of
variadic function when variadic variables are not expanded. Internally
it has impact to format function only. There are no possibility put
parameters in array. But this possibility can be useful. I cannot to
use overloading due ambiguous functions with any and anyarray
params. Solution can be new function format_array - that share code
with format function.

postgres=# select format_array('Hello %s, %1$s', array['World']);
format_array

 Hello World, World
(1 row)

postgres=# select format('Hello %s, %1$s', variadic array['World']);
ERROR:  function format(unknown, text[]) does not exist
LINE 1: select format('Hello %s, %1$s', variadic array['World']);
   ^
HINT:  No function matches the given name and argument types. You
might need to add explicit type casts.
postgres=# select format('Hello %s, %1$s', 'World');
   format

 Hello World, World
(1 row)

Regards

Pavel Stehule


text_format_array.diff
Description: Binary data

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] patch: disable bypass of expand variadic for ANY variable function, format_array function for formatting with params in array

2012-09-08 Thread Tom Lane
Pavel Stehule pavel.steh...@gmail.com writes:
 This patch disable bypassing of parameters for variadic function with
 ANY type variadic parameter.

This seems completely silly.  If you think it's broken now (which I
don't particularly agree with: does not do what you want in some corner
cases is not broken) then propose a fix.  Breaking it worse is not an
improvement.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] patch: disable bypass of expand variadic for ANY variable function, format_array function for formatting with params in array

2012-09-08 Thread Pavel Stehule
2012/9/8 Tom Lane t...@sss.pgh.pa.us:
 Pavel Stehule pavel.steh...@gmail.com writes:
 This patch disable bypassing of parameters for variadic function with
 ANY type variadic parameter.

 This seems completely silly.  If you think it's broken now (which I
 don't particularly agree with: does not do what you want in some corner
 cases is not broken) then propose a fix.  Breaking it worse is not an
 improvement.

it is broken

format('%s %s, 'Hello', 'World') -- is ok -- case A
format('%s %s', variadic array['Hello', 'World']) -- fails  -- case B

Now, there are no possibility detect from function if there is a A
case or B case.

probably there are three fixes:

a) enhance FunctionCallInfoData by expand_variadic field - and then
different behave should be implemented in function,

b) enhance executor to use a updated FmgrInfo for every call of
function. FmgrInfo should be updated every call because fn_nargs can
be changed every call,

c) change mechanism how variadic parameters are passed to variadic
any function. Now we use FunctionCallInfoData. We can pass only
pointer to some structure with parameters enhanced about type info.
This mechanism can be same for A case and B case. And we can share
FmgrInfo - because there will be only one real parameter of type
internal. But this change is not compatible with current design. But
is a most simple probably and decrease difference between variadic
any functions and others variadic functions.

We can inplement a new datatype any[] - and this can be flag for new
implementation and any for old implementation. So there should not
be problem with compatibility.

Regards

Pavel



 regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


[HACKERS] Question about SSI, subxacts, and aborted read-only xacts

2012-09-08 Thread Jeff Davis
This question comes about after reading the VLDB paper Serializable
Snapshot Isolation in PostgreSQL.

We release predicate locks after a transaction abort, but not after a
subtransaction abort. The paper says that the reason is:

We do not drop SIREAD locks acquired during a subtransaction if the
subtransaction is aborted (i.e. all SIREAD locks belong to the top-level
transaction). This is because data read during the subtransaction may
have been reported to the user or otherwise externalized. (section
7.3).

But that doesn't make sense to me, because that reasoning would also
apply to top-level transactions that are aborted, but we release the
SIREAD locks for those.

In other words, this introduces an inconsistency between:

  BEGIN ISOLATION LEVEL SERIALIZABLE;
  SAVEPOINT s1;
  ...
  ROLLBACK TO s1;
  COMMIT;

and:

  BEGIN ISOLATION LEVEL SERIALIZABLE;
  ...
  ROLLBACK;

I'm not suggesting this is a correctness problem: holding SIREAD locks
for longer never causes incorrect results. But it does seem a little
inconsistent.

For top-level transactions, I don't think it's possible to preserve
SIREAD locks after an abort, because we rely on aborts to alleviate
conflicts (and when using 2PC, we may need to abort a read-only
transaction to correct the situation). So it seems like users must not
rely on any answers they get from a transaction (or subtransaction)
unless it commits.

Does that make sense?

If so, I think we need a documentation update. The serializable
isolation level docs don't quite make it clear that serializability only
applies to transactions that commit. It might not be obvious to a user
that there's a difference between commit and abort for a RO transaction.
I think that, in S2PL, serializability applies even to aborted
transactions (though I haven't spent much time thinking about it), so
users accustomed to other truly-serializable implementations might be
surprised.

Regards,
Jeff Davis





-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] build farm machine using make -j 8 mixed results

2012-09-08 Thread Andrew Dunstan


On 09/08/2012 11:06 AM, Tom Lane wrote:

Andrew Dunstan and...@dunslane.net writes:

This seems totally stupid, but it happens when the path to the current
directory includes a cross-device symlink. If I cd following the link,
then this effect doesn't happen. Weird.

Huh.  So maybe a gmake bug, or maybe there's something wrong with our
make rules for the case that `pwd` doesn't match what gmake thinks the
current path is.  Could you specify the setup more fully?




Scratch that theory, that was just a transient. If anything it looks 
like it is related to system load. When almost nothing was running on 
the machine it worked fine. When I started up a Browser and an MUA the 
problem occurred. This VM has 4 CPUs and 4Gb of memory and a load 
average around 0.4 right now. I'm a bit perplexed.


cheers

andrew




--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


[HACKERS] Python version dependency in plpython regression tests

2012-09-08 Thread Tom Lane
After reading the recent thread about python 2 vs python 3 support,
I thought I'd amuse myself by trying to get plpython3 supported in
the Fedora packages.  That turned out to be unreasonably painful
(which is something we oughta fix eventually), but it worked,
at least with F16/F17.  When I went to try to build it in rawhide,
I got these failures from the plpython_trigger regression test:

*** /builddir/build/BUILD/postgresql-9.2.0/src/pl/plpython/./python3/expected/pl
python_trigger.out  Sat Sep  8 16:20:55 2012
--- /builddir/build/BUILD/postgresql-9.2.0/src/pl/plpython/./python3/results/plp
ython_trigger.out   Sat Sep  8 16:21:07 2012
***
*** 483,489 
  BEFORE UPDATE ON trigger_test
  FOR EACH ROW EXECUTE PROCEDURE stupid7();
  UPDATE trigger_test SET v = 'null' WHERE i = 0;
! ERROR:  key a found in TD[new] does not exist as a column in the triggerin
g row
  CONTEXT:  while modifying trigger row
  PL/Python function stupid7
  DROP TRIGGER stupid_trigger7 ON trigger_test;
--- 483,489 
  BEFORE UPDATE ON trigger_test
  FOR EACH ROW EXECUTE PROCEDURE stupid7();
  UPDATE trigger_test SET v = 'null' WHERE i = 0;
! ERROR:  key b found in TD[new] does not exist as a column in the triggerin
g row
  CONTEXT:  while modifying trigger row
  PL/Python function stupid7
  DROP TRIGGER stupid_trigger7 ON trigger_test;
***
*** 497,503 
  BEFORE UPDATE ON trigger_test
  FOR EACH ROW EXECUTE PROCEDURE stupid7u();
  UPDATE trigger_test SET v = 'null' WHERE i = 0;
! ERROR:  key a found in TD[new] does not exist as a column in the triggerin
g row
  CONTEXT:  while modifying trigger row
  PL/Python function stupid7u
  DROP TRIGGER stupid_trigger7 ON trigger_test;
--- 497,503 
  BEFORE UPDATE ON trigger_test
  FOR EACH ROW EXECUTE PROCEDURE stupid7u();
  UPDATE trigger_test SET v = 'null' WHERE i = 0;
! ERROR:  key b found in TD[new] does not exist as a column in the triggerin
g row
  CONTEXT:  while modifying trigger row
  PL/Python function stupid7u
  DROP TRIGGER stupid_trigger7 ON trigger_test;


The test cases being complained of look like

-- TD keys not corresponding to row columns

CREATE FUNCTION stupid7() RETURNS trigger
AS $$
TD[new] = {'a': 'foo', 'b': 'bar'}
return MODIFY;
$$ LANGUAGE plpythonu;

and obviously, python is iterating through the hash's keys in a
different order than it was a minor version or two back.  (The failure
is occurring with 3.3.0-0.4.rc1.fc19, whereas I saw no failure with
3.2.3-7.fc17.)

I think probably the best thing is to change the test case so it has one
valid key and one not-valid one, rather than assuming that the same key
will always be complained of when there's more than one not-valid one.

Any objections, or different analysis?

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Python version dependency in plpython regression tests

2012-09-08 Thread Peter Eisentraut
On Sat, 2012-09-08 at 16:35 -0400, Tom Lane wrote:
 and obviously, python is iterating through the hash's keys in a
 different order than it was a minor version or two back.  (The failure
 is occurring with 3.3.0-0.4.rc1.fc19, whereas I saw no failure with
 3.2.3-7.fc17.)

Yes, known problem with 3.3.
 
 I think probably the best thing is to change the test case so it has
 one
 valid key and one not-valid one, rather than assuming that the same
 key
 will always be complained of when there's more than one not-valid
 one. 

That would probably work.  We dealt with a similar problem in
2cfb1c6f77734db81b6e74bcae630f93b94f69be, if you want some additional
inspiration.  Not sure why we didn't see this case then.



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] build farm machine using make -j 8 mixed results

2012-09-08 Thread Tom Lane
Andrew Dunstan and...@dunslane.net writes:
 Scratch that theory, that was just a transient. If anything it looks 
 like it is related to system load. When almost nothing was running on 
 the machine it worked fine. When I started up a Browser and an MUA the 
 problem occurred. This VM has 4 CPUs and 4Gb of memory and a load 
 average around 0.4 right now. I'm a bit perplexed.

Hm ... you weren't using the -l (--max-load) option were you?  That
would make system load affect gmake's scheduling.  Although it's clear
when I test it that it is waiting for the bison run to finish before
launching the dependent builds, so it still seems like it must be a bug
if your copy isn't waiting for that.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] build farm machine using make -j 8 mixed results

2012-09-08 Thread Tom Lane
Andrew Dunstan and...@dunslane.net writes:
 And it's the stock Fedora build of make.

Which Fedora branch exactly?

The package version I was trying to reproduce with here is
make-3.82-8.fc16.x86_64.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Python version dependency in plpython regression tests

2012-09-08 Thread Tom Lane
Peter Eisentraut pete...@gmx.net writes:
 On Sat, 2012-09-08 at 16:35 -0400, Tom Lane wrote:
 I think probably the best thing is to change the test case so it has
 one valid key and one not-valid one, rather than assuming that the
 same key will always be complained of when there's more than one
 not-valid one.

 That would probably work.  We dealt with a similar problem in
 2cfb1c6f77734db81b6e74bcae630f93b94f69be, if you want some additional
 inspiration.  Not sure why we didn't see this case then.

If it's got anything to do with hashing, platform dependency wouldn't be
a bit surprising.  Or they might have tweaked the algorithm some more
since May.

How come you did not back-patch that commit ... are we not supporting
3.3 in branches before 9.2 for some reason?

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] build farm machine using make -j 8 mixed results

2012-09-08 Thread Andrew Dunstan


On 09/08/2012 04:54 PM, Tom Lane wrote:

Andrew Dunstan and...@dunslane.net writes:

And it's the stock Fedora build of make.

Which Fedora branch exactly?

The package version I was trying to reproduce with here is
make-3.82-8.fc16.x86_64.


Same:

   $ rpm -q make
   make-3.82-8.fc16.x86_64

cheers

andrew



--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] build farm machine using make -j 8 mixed results

2012-09-08 Thread Andrew Dunstan


On 09/08/2012 04:46 PM, Tom Lane wrote:

Andrew Dunstan and...@dunslane.net writes:

Scratch that theory, that was just a transient. If anything it looks
like it is related to system load. When almost nothing was running on
the machine it worked fine. When I started up a Browser and an MUA the
problem occurred. This VM has 4 CPUs and 4Gb of memory and a load
average around 0.4 right now. I'm a bit perplexed.

Hm ... you weren't using the -l (--max-load) option were you?  That
would make system load affect gmake's scheduling.  Although it's clear
when I test it that it is waiting for the bison run to finish before
launching the dependent builds, so it still seems like it must be a bug
if your copy isn't waiting for that.




No. just make -j 4

And it's the stock Fedora build of make.

cheers

andrew



--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


[HACKERS] pg_test_fsync output and commit_delay

2012-09-08 Thread Peter Geoghegan
I propose that we try and develop better commit_delay advice, to make
it easier to set the parameter in a way that actually helps
performance. I have been researching a way to make commit_delay
adaptive, though have yet to develop any further insight that is worth
sharing. This may change.

The attached simple patch alters the output produced by pg_test_fsync,
so that we also see the average sync time per op. The pg_test_fsync
docs have had minor alterations too. Hopefully, another doc patch will
follow that builds upon this, and actually firmly recommends taking a
particular course of action when setting commit_delay - my previous
observations about what helped throughput, though supported by Greg
Smith, have not been scrutinised enough just yet, I feel. For now, the
equivocated wording of my doc alterations (that raw wal_sync_method
file sync time might *somehow* be useful here) seems appropriate.

-- 
Peter Geoghegan   http://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training and Services


pg_test_fsync.v1.2012_09_08.patch
Description: Binary data

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


[HACKERS] pg_upgrade testing for MSVC

2012-09-08 Thread Andrew Dunstan
Here's a patch to provide for the pg_upgrade tests to be run in MSVC 
builds. I want to get this running on the buildfarm before long on HEAD 
and REL9_2_STABLE.


cheers

andrew
diff --git a/src/tools/msvc/Install.pm b/src/tools/msvc/Install.pm
index 3923532..235a150 100644
--- a/src/tools/msvc/Install.pm
+++ b/src/tools/msvc/Install.pm
@@ -37,9 +37,16 @@ sub Install
 	$| = 1;
 
 	my $target = shift;
-	our $config;
-	require config_default.pl;
-	require config.pl if (-f config.pl);
+	# if called from vcregress, the config will be passed to us
+	# so no need to re-include these
+	our $config = shift;
+	unless ($config)
+	{
+		# suppress warning about harmless redeclaration of $config
+		no warnings 'misc'; 
+		require config_default.pl;
+		require config.pl if (-f config.pl);
+	}
 
 	chdir(../../..)if (-f ../../../configure);
 	chdir(../../../..) if (-f ../../../../configure);
diff --git a/src/tools/msvc/vcregress.pl b/src/tools/msvc/vcregress.pl
index 530770c..fed4916 100644
--- a/src/tools/msvc/vcregress.pl
+++ b/src/tools/msvc/vcregress.pl
@@ -9,15 +9,19 @@ our $config;
 use Cwd;
 use File::Copy;
 
+use Install qw(Install);
+
 my $startdir = getcwd();
 
 chdir ../../.. if (-d ../../../src/tools/msvc);
 
+my $topdir = getcwd();
+
 require 'src/tools/msvc/config_default.pl';
 require 'src/tools/msvc/config.pl' if (-f 'src/tools/msvc/config.pl');
 
 # buildenv.pl is for specifying the build environment settings
-# it should contian lines like:
+# it should contain lines like:
 # $ENV{PATH} = c:/path/to/bison/bin;$ENV{PATH};
 
 if (-e src/tools/msvc/buildenv.pl)
@@ -27,7 +31,7 @@ if (-e src/tools/msvc/buildenv.pl)
 
 my $what = shift || ;
 if ($what =~
-	/^(check|installcheck|plcheck|contribcheck|ecpgcheck|isolationcheck)$/i)
+	/^(check|installcheck|plcheck|contribcheck|ecpgcheck|isolationcheck|upgradecheck)$/i)
 {
 	$what = uc $what;
 }
@@ -73,7 +77,8 @@ my %command = (
 	INSTALLCHECK   = \installcheck,
 	ECPGCHECK  = \ecpgcheck,
 	CONTRIBCHECK   = \contribcheck,
-	ISOLATIONCHECK = \isolationcheck,);
+	ISOLATIONCHECK = \isolationcheck,
+UPGRADECHECK   = \upgradecheck,);
 
 my $proc = $command{$what};
 
@@ -231,6 +236,76 @@ sub contribcheck
 	exit $mstat if $mstat;
 }
 
+sub upgradecheck
+{
+	my $status;
+	my $cwd = getcwd();
+
+	# Much of this comes from the pg_upgrade test.sh script,
+	# but it only covers the --install case, and not the case
+	# where the old and new source or bin dirs are different.
+	# i.e. only the this version to this version check. That's
+	# what pg_upgrade's make check does.
+
+	$ENV{PGPORT} ||= 50432;
+	my $tmp_root = $topdir/contrib/pg_upgrade/tmp_check;
+	mkdir $tmp_root unless -d $tmp_root;
+	my $tmp_install = $tmp_root/install;
+	print Setting up temp install\n\n;
+	Install($tmp_install, $config);
+	# Install does a chdir, so change back after that
+	chdir $cwd;
+	my ($bindir,$libdir,$oldsrc,$newsrc) = 
+	  ($tmp_install/bin, $tmp_install/lib, $topdir, $topdir);
+	$ENV{PATH} = $bindir;$ENV{PATH};
+	my $data = $tmp_root/data;
+	$ENV{PGDATA} = $data;
+	my $logdir = $topdir/contrib/pg_upgrade/log;
+	mkdir $logdir unless -d $logdir;
+	print \nRunning initdb on old cluster\n\n;
+	system(initdb);
+	print \nStarting old cluster\n\n;
+	system(pg_ctl start -l $logdir/postmaster1.log -w);
+	print \nSetting up data for upgrading\n\n;
+	installcheck();
+	# now we can chdir into the source dir
+	chdir $topdir/contrib/pg_upgrade;
+	print \nDuming old cluster\n\n;
+	system(pg_dumpall -f $tmp_root/dump1.sql);
+	print \nStopping old cluster\n\n;
+	system(pg_ctl -m fast stop);
+	rename $data, $data.old;
+	print \nSetting up new cluster\n\n;
+	system(initdb);
+	print \nRunning pg_upgrade\n\n;
+	system(pg_upgrade -d $data.old -D $data -b $bindir -B $bindir);
+	print \nStarting new cluster\n\n;
+	system(pg_ctl -l $logdir/postmaster2.log -w start);
+	print \nSetting up stats on new cluster\n\n;
+	system(.\\analyze_new_cluster.bat);
+	print \nDumping new cluster\n\n;
+	system(pg_dumpall -f $tmp_root/dump2.sql);
+	print \nStopping new cluster\n\n;
+	system(pg_ctl -m fast stop);
+	print \nDeleting old cluster\n\n;
+	system(.\\delete_old_cluster.bat);
+	print \nComparing old and new cluster dumps\n\n;
+
+	# If any of the above failed, the following will also fail, so
+	# this should be a sufficient check of the whole thing.
+	system(diff -q $tmp_root/dump1.sql $tmp_root/dump2.sql);
+	$status = $?;
+	if (!$status)
+	{
+		print PASSED\n;
+	}
+	else
+	{
+		print dumps not identical!\n;
+		exit(1);
+	}
+}
+
 sub fetchRegressOpts
 {
 	my $handle;

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] pg_test_fsync output and commit_delay

2012-09-08 Thread Josh Berkus

 The attached simple patch alters the output produced by pg_test_fsync,
 so that we also see the average sync time per op. 

Good idea.  A median would be even better, but harder to calculate, I
imagine.  You might consider providing a maximum, too.

 The pg_test_fsync
 docs have had minor alterations too. Hopefully, another doc patch will
 follow that builds upon this, and actually firmly recommends taking a
 particular course of action when setting commit_delay - my previous
 observations about what helped throughput, though supported by Greg
 Smith, have not been scrutinised enough just yet, I feel. For now, the
 equivocated wording of my doc alterations (that raw wal_sync_method
 file sync time might *somehow* be useful here) seems appropriate.

I'll see what I can do.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


[HACKERS] Supporting plpython 2+3 builds better

2012-09-08 Thread Tom Lane
I wrote:
 After reading the recent thread about python 2 vs python 3 support,
 I thought I'd amuse myself by trying to get plpython3 supported in
 the Fedora packages.  That turned out to be unreasonably painful
 (which is something we oughta fix eventually), but it worked,

To give you an idea of what unreasonably painful means, attached is
the specfile diff needed to make this happen.  I will not comment on
the fragility of this beyond observing that the touch -r commands
are *necessary*, else you get incorrect results.

I think we really need to do something to make this type of build less
painful, because as far as I can see most distros are going to be
wanting to support both python 2 and 3 for awhile to come.

A sketch of a solution might go like this:

* Extend configure to have two switches, say --with-python and
--with-python3, which you can select either or both of.  If you
want to pick executables that are not named python and python3,
perhaps write it like --with-python3=/usr/bin/python3.2mu.  (I'd
be inclined to remove the dependency on a PYTHON envvar altogether.)

* Make configure output two independent sets of variables into
Makefile.global, viz

python2_includespec = ... 
python2_libdir  = ... 
python2_libspec = ... 
python2_additional_libs = ... 
python2_configdir   = ... 
python2_majorversion= ... 
python2_version = ... 

python3_includespec = ... 
python3_libdir  = ... 
python3_libspec = ... 
python3_additional_libs = ... 
python3_configdir   = ... 
python3_majorversion= ... 
python3_version = ... 

* Make plpython's Makefile build, test, install plpython2 and/or
plpython3 depending on what's set in Makefile.global.

I'm not sure yet whether it's possible to do this without duplicating a
lot of logic in config/python.m4 and plpython's Makefile.

Another problem is that Makefile.shlib isn't designed to build more than
one shared library per directory, which means that we might end up
having to copy all the source files into a new plpython3 subdirectory
anyway.  We're already doing some of that with the regression test
files, though.

Thoughts?

regards, tom lane

diff --git a/postgresql.spec b/postgresql.spec
index 524d81c..62ce352 100644
*** a/postgresql.spec
--- b/postgresql.spec
***
*** 35,40 
--- 35,41 
  %{!?test:%global test 1}
  %{!?upgrade:%global upgrade 1}
  %{!?plpython:%global plpython 1}
+ %{!?plpython3:%global plpython3 1}
  %{!?pltcl:%global pltcl 1}
  %{!?plperl:%global plperl 1}
  %{!?ssl:%global ssl 1}
*** BuildRequires: systemd-units
*** 107,112 
--- 108,117 
  BuildRequires: python-devel
  %endif
  
+ %if %plpython3
+ BuildRequires: python3-devel
+ %endif
+ 
  %if %pltcl
  BuildRequires: tcl-devel
  %endif
*** Install this if you want to write databa
*** 265,278 
  
  %if %plpython
  %package plpython
! Summary: The Python procedural language for PostgreSQL
  Group: Applications/Databases
  Requires: %{name}-server%{?_isa} = %{version}-%{release}
  
  %description plpython
  The postgresql-plpython package contains the PL/Python procedural language,
  which is an extension to the PostgreSQL database server.
! Install this if you want to write database functions in Python.
  %endif
  
  %if %pltcl
--- 270,295 
  
  %if %plpython
  %package plpython
! Summary: The Python2 procedural language for PostgreSQL
  Group: Applications/Databases
  Requires: %{name}-server%{?_isa} = %{version}-%{release}
  
  %description plpython
  The postgresql-plpython package contains the PL/Python procedural language,
  which is an extension to the PostgreSQL database server.
! Install this if you want to write database functions in Python 2.
! %endif
! 
! %if %plpython3
! %package plpython3
! Summary: The Python3 procedural language for PostgreSQL
! Group: Applications/Databases
! Requires: %{name}-server%{?_isa} = %{version}-%{release}
! 
! %description plpython3
! The postgresql-plpython3 package contains the PL/Python3 procedural language,
! which is an extension to the PostgreSQL database server.
! Install this if you want to write database functions in Python 3.
  %endif
  
  %if %pltcl
*** CFLAGS=$CFLAGS -DLINUX_OOM_SCORE_ADJ=0
*** 346,351 
--- 363,447 
  # CFLAGS=`echo $CFLAGS| sed -e s|-O2|-O1|g `
  # %%endif
  
+ # plpython requires separate configure/build runs to build against python 2
+ # versus python 3.  Our strategy is to do the python 3 run first, then make
+ # distclean and do it again for the normal build.  Note that the installed
+ # Makefile.global will reflect the python 2 build, which seems appropriate
+ # since that's still considered the default plpython version.
+ %if %plpython3
+ 
+ export PYTHON=/usr/bin/python3
+ 
+ # These configure options must match main build
+ %configure 

Re: [HACKERS] build farm machine using make -j 8 mixed results

2012-09-08 Thread Andrew Dunstan


On 09/08/2012 04:52 PM, Andrew Dunstan wrote:


On 09/08/2012 04:46 PM, Tom Lane wrote:

Andrew Dunstan and...@dunslane.net writes:

Scratch that theory, that was just a transient. If anything it looks
like it is related to system load. When almost nothing was running on
the machine it worked fine. When I started up a Browser and an MUA the
problem occurred. This VM has 4 CPUs and 4Gb of memory and a load
average around 0.4 right now. I'm a bit perplexed.

Hm ... you weren't using the -l (--max-load) option were you? That
would make system load affect gmake's scheduling.  Although it's clear
when I test it that it is waiting for the bison run to finish before
launching the dependent builds, so it still seems like it must be a bug
if your copy isn't waiting for that.




No. just make -j 4

And it's the stock Fedora build of make.



I have just repeated this on an absolutely fresh up to date F17 machine, 
with no symlink stuff in play.


Steps to recreate:

   CC=ccache gcc ../postgres/configure --enable-depend --enable-debug
   --enable-cassert --with-perl --with-python --with-tcl --with-libxml
   --with-libxslt -with-openssl --with-gssapi --with-pam --with-ldap
   make -j 4


cheers

andrew

q


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] build farm machine using make -j 8 mixed results

2012-09-08 Thread Tom Lane
Andrew Dunstan and...@dunslane.net writes:
 I have just repeated this on an absolutely fresh up to date F17 machine, 
 with no symlink stuff in play.

 Steps to recreate:

 CC=ccache gcc ../postgres/configure --enable-depend --enable-debug
 --enable-cassert --with-perl --with-python --with-tcl --with-libxml
 --with-libxslt -with-openssl --with-gssapi --with-pam --with-ldap
 make -j 4

Huh ... that recipe works (er, fails) for me too, at least some of the
time.  I wonder what exactly is the key difference between the working
and failing cases?

Anyway, what I notice is that I get different types of failures, but
they are all under ecpg/.  What I think we need to do is insert
.NOTPARALLEL in ecpg/Makefile, because there are several reasons not
to run its sub-makes in parallel:

* preproc/Makefile casually does this:

../ecpglib/typename.o: ../ecpglib/typename.c
$(MAKE) -C $(dir $@) $(notdir $@)

which is very likely to screw up any make proceeding in parallel in
ecpglib.

* compatlib and ecpglib will equally casually invoke make all in
other subdirectories; ditto.

And that's not even counting the bison-output problem you were seeing.
I'm not entirely sure what's causing that, but I'm suspicious that the
ultimate cause is the extra rules for the all...recurse targets in
ecpg/Makefile, which look like they could result in additional instances
of multiple make processes running in the same subdirectory.

After adding the .NOTPARALLEL marker, I don't see these failures
anymore.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] build farm machine using make -j 8 mixed results

2012-09-08 Thread Andrew Dunstan


On 09/08/2012 07:54 PM, Tom Lane wrote:

Andrew Dunstan and...@dunslane.net writes:

I have just repeated this on an absolutely fresh up to date F17 machine,
with no symlink stuff in play.
Steps to recreate:
 CC=ccache gcc ../postgres/configure --enable-depend --enable-debug
 --enable-cassert --with-perl --with-python --with-tcl --with-libxml
 --with-libxslt -with-openssl --with-gssapi --with-pam --with-ldap
 make -j 4

Huh ... that recipe works (er, fails) for me too, at least some of the
time.  I wonder what exactly is the key difference between the working
and failing cases?

Anyway, what I notice is that I get different types of failures, but
they are all under ecpg/.  What I think we need to do is insert
.NOTPARALLEL in ecpg/Makefile, because there are several reasons not
to run its sub-makes in parallel:

* preproc/Makefile casually does this:

../ecpglib/typename.o: ../ecpglib/typename.c
$(MAKE) -C $(dir $@) $(notdir $@)

which is very likely to screw up any make proceeding in parallel in
ecpglib.

* compatlib and ecpglib will equally casually invoke make all in
other subdirectories; ditto.

And that's not even counting the bison-output problem you were seeing.
I'm not entirely sure what's causing that, but I'm suspicious that the
ultimate cause is the extra rules for the all...recurse targets in
ecpg/Makefile, which look like they could result in additional instances
of multiple make processes running in the same subdirectory.

After adding the .NOTPARALLEL marker, I don't see these failures
anymore.




Well, I'm glad it's not just me. :-)

This fix works for me too.

I guess it should be applied back to 9.1, when it looks like we started 
using .NOTPARALLEL



cheers

andrew


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Python version dependency in plpython regression tests

2012-09-08 Thread Peter Eisentraut
On Sat, 2012-09-08 at 16:52 -0400, Tom Lane wrote:
 How come you did not back-patch that commit ... are we not supporting
 3.3 in branches before 9.2 for some reason? 

Python 3.3 isn't even released yet, much less so back then, so it seemed
premature.

Also, it's a fairly big change just to make the regression tests pass.



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] build farm machine using make -j 8 mixed results

2012-09-08 Thread Peter Eisentraut
On Sat, 2012-09-08 at 19:54 -0400, Tom Lane wrote:
 Anyway, what I notice is that I get different types of failures, but
 they are all under ecpg/.  What I think we need to do is insert
 .NOTPARALLEL in ecpg/Makefile,

I'd hate that, because the ecpg build is one of the slowest parts of the
build, so de-parallelizing it would slow down everything quite a bit.

  because there are several reasons not
 to run its sub-makes in parallel:
 
 * preproc/Makefile casually does this:
 
 ../ecpglib/typename.o: ../ecpglib/typename.c
   $(MAKE) -C $(dir $@) $(notdir $@)
 
 which is very likely to screw up any make proceeding in parallel in
 ecpglib.

That should probably be fixed by symlinking the source file and building
it in the preproc directory.

 And that's not even counting the bison-output problem you were seeing.
 I'm not entirely sure what's causing that, but I'm suspicious that the
 ultimate cause is the extra rules for the all...recurse targets in
 ecpg/Makefile, which look like they could result in additional instances
 of multiple make processes running in the same subdirectory.

I think the point of these targets is exactly to prevent that.




-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Supporting plpython 2+3 builds better

2012-09-08 Thread Peter Eisentraut
On Sat, 2012-09-08 at 19:18 -0400, Tom Lane wrote:
 To give you an idea of what unreasonably painful means, attached is
 the specfile diff needed to make this happen.  I will not comment on
 the fragility of this beyond observing that the touch -r commands
 are *necessary*, else you get incorrect results.

I think an easier way is to configure two vpath builds and install the
pieces you want from each one. yo

 Another problem is that Makefile.shlib isn't designed to build more
 than one shared library per directory,

That's the main problem, but fixing it would be very useful in other
places as well.  I had it on my radar to do something about that.

  which means that we might end up having to copy all the source files
 into a new plpython3 subdirectory anyway.  We're already doing some of
 that with the regression test files, though.

Doing it with the test files is already annoying enough.  For instance,
we don't have a static list of all the files we need to copy (because of
expected file variants that are not tracked in the makefiles), so we
need to copy the files again on each run.  If you extend this to all
source files, things would get totally bizarre.

A less invasive method might be to set up a second directory
pl/plpython3 which vpath-builds from pl/plpython.

But frankly, I'd like to work on the multiple-shlibs-in-one-directory
issue, because all of the above issues and more are projected in their
own strange ways on something like the transforms feature.  For example,
how would you organize the source code for an extension module that
provides a new data type and adapters for plpython{2,3}u and plperl{,u},
with tests and all that?




-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers