Re: [HACKERS] Range Types and extensions

2011-06-20 Thread Jeff Davis
On Sun, 2011-06-19 at 21:29 +0200, Florian Pflug wrote:
 If I'm not mistaken about this, that would imply that we also cannot
 have two range types with the same base type, the same opclass,
 but different collations. Which seems rather unfortunate... In fact,
 if that's true, maybe restricing range types to the database collation
 would be best...

Yes, we cannot have two range types with the same base type. That is a
consequence of the polymorphic type system, which needs to be able to
determine the range type given the base type.

A workaround is to use domains. That is effective, but awkward. For
instance, given:
  CREATE DOMAIN textdomain AS text;
  CREATE TYPE textdomainrange AS RANGE (subtype=textdomain);
then:
  '[a,z)'::textdomainrange @ 'b'::textdomain
would work, but:
  '[a,z)'::textdomainrange @ 'b'
would not, which would be annoying.

I don't see a way around this. It's not a collation problem, but a
general multiple range types with the same subtype problem.

I don't think there's much benefit in restricting it to the DB
collation. If someone really needs a different collation (or opclass,
for that matter), it might as well be allowed, even if you have to do
extra type annotations.

Regards,
Jeff Davis


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Patch - Debug builds without optimization

2011-06-20 Thread Radosław Smogura
Greg Stark st...@mit.edu Monday 20 of June 2011 03:39:12
 On Thu, Jun 16, 2011 at 9:00 PM, Tom Lane t...@sss.pgh.pa.us wrote:
  BTW, if you're hacking Postgres code and don't already have a
  reinstall script, you need one.  Mine is basically
  
 pg_ctl stop
 cd $PGBLDROOT/src/backend
 make install-bin
 pg_ctl start
 
 I've always wondered what other people do to iterate quickly. It's a
 bit of a pain that you can't just run the binary out of the build
 tree. This looks a lot safer than some of the things I was considering
 doing with symlinks.
I actually go to installation directory
and call in one line (simple because up arrow helps).

pg_ctl -D db stop; gmake -C ../postgresql -j5 install; pg_ctl -D db start

Regards,
Radek

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] WIP: Fast GiST index build

2011-06-20 Thread Alexander Korotkov
New version of patch. There are some bugfixes, minor refactoring, comments
(unfortunatelly, not all the code is covered by comments yet). Also
fastbuild parameter was added to the GiST index. It allows to test index
building with and without fast build without postgres recompile.

--
With best regards,
Alexander Korotkov.

On Thu, Jun 16, 2011 at 10:35 PM, Alexander Korotkov
aekorot...@gmail.comwrote:

 Oh, actually it's so easy. Thanks.

 --
 With best regards,
 Alexander Korotkov.

 On Thu, Jun 16, 2011 at 10:26 PM, Heikki Linnakangas 
 heikki.linnakan...@enterprisedb.com wrote:

 On 16.06.2011 21:13, Alexander Korotkov wrote:

 My current idea is to measure number of IO accesses by pg_stat_statements
 and measure CPU usage by /proc/PID/stat. Any thoughts?


 Actually, you get both of those very easily with:

 set log_statement_stats=on

 LOG:  QUERY STATISTICS
 DETAIL:  ! system usage stats:
!   0.000990 elapsed 0.00 user 0.00 system sec
!   [0.00 user 0.008000 sys total]
!   0/0 [32/0] filesystem blocks in/out
!   0/0 [0/959] page faults/reclaims, 0 [0] swaps
!   0 [0] signals rcvd, 0/0 [0/0] messages rcvd/sent
!   0/0 [10/1] voluntary/involuntary context switches
 STATEMENT:  SELECT generate_series(1,100);



 --
  Heikki Linnakangas
  EnterpriseDB   http://www.enterprisedb.com





gist_fast_build-0.2.0.patch.gz
Description: GNU Zip compressed data

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] ALTER TABLE lock strength reduction patch is unsafe

2011-06-20 Thread Simon Riggs
On Sun, Jun 19, 2011 at 10:13 PM, Simon Riggs si...@2ndquadrant.com wrote:

 pre-existing bug

I've been able to reproduce the bug on 9.0 stable as well, using the
OP's test script.
Multiple errors like this...

ERROR:  relation pgbench_accounts does not exist
STATEMENT:  alter table pgbench_accounts set (fillfactor = 100);

Presume we want to fix both bugs, not just the most recent one so will
continue with both patches.

-- 
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] pika buildfarm member failure on isolationCheck tests

2011-06-20 Thread Dan Ports
On Sun, Jun 19, 2011 at 09:10:02PM -0400, Robert Haas wrote:
 Is this an open item for 9.1beta3?

Yes. I've put it on the list.

The SxactGlobalXmin and its refcount were getting out of sync with the
actual transaction state. This is timing-dependent but I can reproduce it
fairly reliably under concurrent workloads.

It looks the problem comes from the change a couple days ago that
removed the SXACT_FLAG_ROLLED_BACK flag and changed the
SxactIsRolledBack checks to SxactIsDoomed. That's the correct thing to
do everywhere else, but gets us in trouble here. We shouldn't be
ignoring doomed transactions in SetNewSxactGlobalXmin, because they're
not eligible for cleanup until the associated backend aborts the
transaction and calls ReleasePredicateLocks.

However, it isn't as simple as just removing the SxactIsDoomed check
from SetNewSxactGlobalXmin. ReleasePredicateLocks calls
SetNewSxactGlobalXmin *before* it releases the aborted transaction's
SerializableXact (it pretty much has to, because we must drop and
reacquire SerializableXactHashLock in between). But we don't want the
aborted transaction included in the SxactGlobalXmin computation.

It seems like we do need that SXACT_FLAG_ROLLED_BACK after all, even
though it's only set for this brief interval. We need to be able to
distinguish a transaction that's just been marked for death (doomed)
from one that's already called ReleasePredicateLocks.

Dan

-- 
Dan R. K. Ports  MIT CSAILhttp://drkp.net/

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


[HACKERS] patch: plpgsql - remove unnecessary ccache search when a array variable is updated

2011-06-20 Thread Pavel Stehule
Hello

this patch significantly reduce a ccache searching. On my test - bubble sort

postgres=# \sf buble
CREATE OR REPLACE FUNCTION public.buble(integer[])
 RETURNS integer[]
 LANGUAGE plpgsql
AS $function$
declare
  unsorted bool := true;
  aux int;
begin
  while unsorted
  loop
unsorted := false;
for i in array_lower($1,1) .. array_upper($1,1) - 1
loop
  if $1[i]  $1[i+1] then
aux := $1[i];
$1[i] := $1[i+1]; $1[i+1] := aux;
unsorted := true;
  end if;
end loop;
  end loop;
  return $1;
end;
$function$ immutable

it decrease evaluation time about 15%.

Regards

Pavel Stehule

p.s. I know so bubble sort is not effective for large arrays. This
algorithm was used because a array is intensive modified.
*** ./pl_exec.c.orig	2011-06-20 08:55:37.0 +0200
--- ./pl_exec.c	2011-06-20 11:38:55.988650907 +0200
***
*** 153,158 
--- 153,170 
  static void exec_assign_value(PLpgSQL_execstate *estate,
    PLpgSQL_datum *target,
    Datum value, Oid valtype, bool *isNull);
+ static void
+ exec_eval_array_datum(PLpgSQL_execstate *estate,
+ PLpgSQL_datum *datum,
+ Oid *arraytypoid,
+ int32 *arraytypmod,
+ int16 *arraytyplen,
+ Oid *elemtypeid,
+ int16 *elemtyplen,
+ bool *elemtybyval,
+ char *elemtypalign,
+ Datum *value,
+ bool *isnull);
  static void exec_eval_datum(PLpgSQL_execstate *estate,
  PLpgSQL_datum *datum,
  Oid *typeid,
***
*** 3981,4001 
  arraytypeid, arraytypmod,
  oldarraydatum, oldarrayisnull);
  
! /* If target is domain over array, reduce to base type */
! arraytypeid = getBaseTypeAndTypmod(arraytypeid, arraytypmod);
! 
! /* ... and identify the element type */
! arrayelemtypeid = get_element_type(arraytypeid);
! if (!OidIsValid(arrayelemtypeid))
! 	ereport(ERROR,
! 			(errcode(ERRCODE_DATATYPE_MISMATCH),
! 			 errmsg(subscripted object is not an array)));
! 
! get_typlenbyvalalign(arrayelemtypeid,
! 	 elemtyplen,
! 	 elemtypbyval,
! 	 elemtypalign);
! arraytyplen = get_typlen(arraytypeid);
  
  /*
   * Evaluate the subscripts, switch into left-to-right order.
--- 3993,4009 
  arraytypeid, arraytypmod,
  oldarraydatum, oldarrayisnull);
  
! /* Fetch current value of array datum and metadata */
! exec_eval_array_datum(estate, target,
! arraytypeid,
! arraytypmod,
! arraytyplen,
! 		 arrayelemtypeid,
! 		 elemtyplen,
! 		 elemtypbyval,
! 		 elemtypalign,
! oldarraydatum,
! oldarrayisnull);
  
  /*
   * Evaluate the subscripts, switch into left-to-right order.
***
*** 4099,4104 
--- 4107,4178 
  	}
  }
  
+ 
+ /*
+  * exec_eval_array_datum			Get current value of PLpgSQL_datum
+  *
+  * The type oid, typmod, value in Datum format, and null flag are returned.
+  * The arraytyplen, elemtyplen, elemtypbyval, elemtypalign are returned.
+  *
+  * This routine is is used for array type variables - returns additional info
+  * about array elements - removes useless search inside ccache.
+  */ 
+ static void
+ exec_eval_array_datum(PLpgSQL_execstate *estate,
+ PLpgSQL_datum *datum,
+ Oid *arraytypoid,
+ int32 *arraytypmod,
+ int16 *arraytyplen,
+ Oid *elemtypeid,
+ int16 *elemtyplen,
+ bool *elemtybyval,
+ char *elemtypalign,
+ Datum *value,
+ bool *isnull)
+ {
+ 	PLpgSQL_var *var = (PLpgSQL_var *) datum;
+ 
+ 	Assert(datum-dtype == PLPGSQL_DTYPE_VAR);
+ 
+ 	if (var-array_desc == NULL)
+ 	{
+ 		PLpgSQL_array_desc *array_desc = (PLpgSQL_array_desc *) palloc(sizeof(PLpgSQL_array_desc));
+ 
+ 		array_desc-arraytypoid = var-datatype-typoid;
+ 		array_desc-arraytypmod = var-datatype-atttypmod;
+ 
+ 		/* If target is domain over array, reduce to base type */
+ 		array_desc-arraytypoid = getBaseTypeAndTypmod(array_desc-arraytypoid, 
+ 	array_desc-arraytypmod);
+ 
+ 		array_desc-elemtypoid = get_element_type(array_desc-arraytypoid);
+ 		if (!OidIsValid(array_desc-elemtypoid))
+ 			ereport(ERROR,
+ 	(errcode(ERRCODE_DATATYPE_MISMATCH),
+ 	 errmsg(subscripted object is not an array)));
+ 
+ 		get_typlenbyvalalign(array_desc-elemtypoid,
+ 			 array_desc-elemtyplen,
+ 			 array_desc-elemtypbyval,
+ 			 array_desc-elemtypalign);
+ 		array_desc-arraytyplen = get_typlen(array_desc-arraytypoid);
+ 
+ 		var-array_desc = array_desc;
+ 	}
+ 
+ 	*arraytypoid = var-array_desc-arraytypoid;
+ 	*arraytypmod = var-array_desc-arraytypmod;
+ 	*arraytyplen = var-array_desc-arraytyplen;
+ 
+ 	*elemtypeid = var-array_desc-elemtypoid;
+ 	*elemtyplen = var-array_desc-elemtyplen;
+ 	*elemtybyval = var-array_desc-elemtypbyval;
+ 	*elemtypalign = var-array_desc-elemtypalign;
+ 
+ 	*value = var-value;
+ 	*isnull = var-isnull;
+ }
+ 
  /*
   * exec_eval_datumGet current value of a PLpgSQL_datum
   *
*** 

[HACKERS] date_part for infinity intervals

2011-06-20 Thread Vlad Arkhipov
The behaviour of date_part function is opaque for infinity intervals. 
For example
date_part('epoch', 'infinity'::date) and date_part('year', 
'infinity'::date) return zero but is supposed to return 'infinity',
date_part('day', 'infinity'::date) returns zero, should it return 'NaN' 
instead?


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] patch: plpgsql - remove unnecessary ccache search when a array variable is updated

2011-06-20 Thread Simon Riggs
On Mon, Jun 20, 2011 at 10:49 AM, Pavel Stehule pavel.steh...@gmail.com wrote:

 this patch significantly reduce a ccache searching. On my test - bubble sort

It sounds good, but also somewhat worrying.

The first cache is slow, so we add another cache to avoid searching
the first cache.

What is making the first cache so slow?

-- 
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Boolean operators without commutators vs. ALL/ANY

2011-06-20 Thread Florian Pflug
On Jun20, 2011, at 03:16 , Greg Stark wrote:
 On Fri, Jun 17, 2011 at 3:49 PM, Florian Pflug f...@phlo.org wrote:
 The regex is always to the right of the operator.
 
 Which is something you have to remember... It's not in any
 way deducible from foo ~ bar alone.
 
 Except that it's always been this way, going back to perl4 or tcl or
 their predecessors. The regexp binding operator always has the regexp
 on the right.

Yeah. The strength of that argument really depends on one's
prior exposure to these languages, though...


 How is that worse than the situation with =~ and ~=?
 
 With =~ it is to the right, with ~= it is to the left.
 
 It's always where the tilde is. Yeah, you have to remember that.
 
 And when you get it wrong it will fail silently. No errors, just wrong 
 results.

Yeah, but this is hardly the only case where you'll get
unintended results if you mix up operator names. 

Now, one might argue, I guess, that mixing up =~ and ~=
or more likely than mixing up, say, ~ and ~~. But ultimately,
whether or not that is highly dependent on one's personal background,
so we're unlikely to ever reach agreement on that...

 While I've never accidentally written /foo/ =~ $_ in perl I have
 *frequently* forgotten whether the operator is ~= or =~. Actually I
 forget that pretty much every time I start writing some perl. I just
 put whichever comes first and if I get an error I reverse it.

Yeah, the nice thing in perl (and ruby also, which is *my* background)
is that regexp's and strings are distinguished by the type system,
and also by the parser.

The latter (i.e. regexp literals enclosed by /../) probably isn't
desirably for postgres, but the former definitely is (i.e. distinguishing
regexp's and text in the type system). Please see the thread
Adding a distinct pattern type to resolve the ~ commutator stalemate
for the details of the proposal. 

 I can see the temptation to make it symmetric but it's going to cause
 an awful lot of confusion.

I do believe that by adding a distinct type we can actually *reduce*
confusion. It makes text ~ pattern readable even for people who
don't intuitively know that the pattern always goes on the right.

best regards,
Florian Pflug


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] patch: plpgsql - remove unnecessary ccache search when a array variable is updated

2011-06-20 Thread Pavel Stehule
Hello

2011/6/20 Simon Riggs si...@2ndquadrant.com:
 On Mon, Jun 20, 2011 at 10:49 AM, Pavel Stehule pavel.steh...@gmail.com 
 wrote:

 this patch significantly reduce a ccache searching. On my test - bubble sort

 It sounds good, but also somewhat worrying.

 The first cache is slow, so we add another cache to avoid searching
 the first cache.

 What is making the first cache so slow?

a using of general cache should be slower than direct access to
memory. The slow down is based on catalog operations - hash
calculations, hash searching and cache validations. I don't know if it
is possible to optimize general cache.

you can compare profile of original pg


3008 13.0493  SearchCatCache
1306  5.6657  ExecEvalParamExtern
1143  4.9586  GetSnapshotData
1122  4.8675  AllocSetAlloc
1058  4.5898  MemoryContextAllocZero
1002  4.3469  ExecMakeFunctionResultNoSets
986   4.2775  ExecEvalArrayRef
851   3.6918  LWLockAcquire
783   3.3968  LWLockRelease
664   2.8806  RevalidateCachedPlan
646   2.8025  AllocSetFree
568   2.4641  array_ref
551   2.3904  CopySnapshot
519   2.2515  AllocSetReset
510   2.2125  array_set
492   2.1344  PopActiveSnapshot
381   1.6529  ArrayGetOffset
369   1.6008  AcquireExecutorLocks
348   1.5097  pfree
347   1.5054  MemoryContextAlloc
313   1.3579  bms_is_member
285   1.2364  CatalogCacheComputeHashValue
267   1.1583  PushActiveSnapshot
266   1.1540  hash_uint32
253   1.0976  pgstat_init_function_usage
233   1.0108  array_seek.clone.0

and patched postgresql's profile

3151  7.2135  AllocSetAlloc
2887  6.6091  ExecEvalParamExtern
2844  6.5107  list_member_ptr
2353  5.3867  AllocSetFree
2318  5.3065  GetSnapshotData
2201  5.0387  ExecMakeFunctionResultNoSets
2153  4.9288  LWLockAcquire
2055  4.7045  ExecEvalArrayRef
1879  4.3015  LWLockRelease
1675  3.8345  MemoryContextAllocZero
1463  3.3492  AcquireExecutorLocks
1375  3.1477  pfree
1356  3.1043  RevalidateCachedPlan
1261  2.8868  AllocSetCheck
1257  2.8776  PopActiveSnapshot
1115  2.5525  array_set
1102  2.5228  AllocSetReset
966   2.2114  CopySnapshot
938   2.1473  MemoryContextAlloc
875   2.0031  array_ref
772   1.7673  ResourceOwnerForgetPlanCacheRef
632   1.4468  array_seek.clone.0
554   1.2683  PushActiveSnapshot
499   1.1423  check_list_invariants
475   1.0874  ExecEvalConst
473   1.0828  bms_is_member
444   1.0164  ArrayGetNItems

so the most slow operation is SearchCatCache - but I am not a man who
can optimize this routine :)

Regards

Pavel Stehule



 --
  Simon Riggs   http://www.2ndQuadrant.com/
  PostgreSQL Development, 24x7 Support, Training  Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


[HACKERS] Re: [COMMITTERS] pgsql: Fixed string in German translation that causes segfault.

2011-06-20 Thread Devrim GÜNDÜZ
On Mon, 2011-06-20 at 11:58 +, Michael Meskes wrote:
 
 Fixed string in German translation that causes segfault.
 
 Applied patch by Christoph Berg c...@df7cb.de to replace placeholder
 %s by correct string.

AFAIK this won't have any effect. This change needs to go to through
pgtranslation project.
-- 
Devrim GÜNDÜZ
Principal Systems Engineer @ EnterpriseDB: http://www.enterprisedb.com
PostgreSQL Danışmanı/Consultant, Red Hat Certified Engineer
Community: devrim~PostgreSQL.org, devrim.gunduz~linux.org.tr
http://www.gunduz.org  Twitter: http://twitter.com/devrimgunduz


signature.asc
Description: This is a digitally signed message part


Re: [HACKERS] [BUG] Denormal float values break backup/restore

2011-06-20 Thread Marti Raudsepp
Hi, thanks for chiming in.

On Sat, Jun 11, 2011 at 22:08, Jeroen Vermeulen j...@xs4all.nl wrote:
 AIUI that is defined to be a little vague, but includes denormalized numbers
 that would undergo any rounding at all.  It says that on overflow the
 conversion should return the appropriate HUGE_VAL variant, and set ERANGE.
  On underflow it returns a reasonably appropriate value (and either may or
 must set ERANGE, which is the part that isn't clear to me).

Which standard is that? Does IEEE 754 itself define strtod() or is
there another relevant standard?

Regards,
Marti

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Grouping Sets

2011-06-20 Thread Pavel Stehule
Hello

I have not any newest patch related to GROUPING SETS. The last version
of this patch is probably correct, but it is not well tested.
Actually, this patch has not quality to production usage :(. It is
just concept. You can test it.

Regards

Pavel Stehule

2011/6/18 Mariano Mara mariano.m...@gmail.com:
 Hi hackers (and specially Pavel Stehule),
  I could really use the grouping set feature for some complex queries I'm
  migrating from other db vendor. If my WEB searching is precise, this
  wiki page [1] and this thread[2] are the last updates on the subject.
  I'm willing to test how these functions in my project but some questions 
 first:
  1- is there an up-to-date version of the patch that I should be aware of?
  2- Can I apply that patch to 8.4.8?
  3- any extra recommendations?

 TIA,
 Mariano

 [1] http://wiki.postgresql.org/wiki/Grouping_Sets
 [2] http://archives.postgresql.org/pgsql-hackers/2010-08/msg00647.php

 --
 Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
 To make changes to your subscription:
 http://www.postgresql.org/mailpref/pgsql-hackers


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


[HACKERS] Re: [COMMITTERS] pgsql: Fixed string in German translation that causes segfault.

2011-06-20 Thread Michael Meskes
On Mon, Jun 20, 2011 at 03:03:37PM +0300, Devrim GÜNDÜZ wrote:
 On Mon, 2011-06-20 at 11:58 +, Michael Meskes wrote:
  
  Fixed string in German translation that causes segfault.
  
  Applied patch by Christoph Berg c...@df7cb.de to replace placeholder
  %s by correct string.
 
 AFAIK this won't have any effect. This change needs to go to through
 pgtranslation project.

Oops, didn't know that. Does that mean the files in our main git are not the
canonical versions? 

Michael

-- 
Michael Meskes
Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org)
Michael at BorussiaFan dot De, Meskes at (Debian|Postgresql) dot Org
Jabber: michael.meskes at googlemail dot com
VfL Borussia! Força Barça! Go SF 49ers! Use Debian GNU/Linux, PostgreSQL

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] ALTER TABLE lock strength reduction patch is unsafe

2011-06-20 Thread Robert Haas
On Sun, Jun 19, 2011 at 5:13 PM, Simon Riggs si...@2ndquadrant.com wrote:
 On Fri, Jun 17, 2011 at 11:41 PM, Tom Lane t...@sss.pgh.pa.us wrote:
 Robert Haas robertmh...@gmail.com writes:
 On Thu, Jun 16, 2011 at 6:54 PM, Tom Lane t...@sss.pgh.pa.us wrote:
 4. Backend #2 visits the new, about-to-be-committed version of
 pgbench_accounts' pg_class row just before backend #3 commits.
 It sees the row as not good and keeps scanning.  By the time it
 reaches the previous version of the row, however, backend #3
 *has* committed.  So that version isn't good according to SnapshotNow
 either.

 thinks some more

 Why isn't this a danger for every pg_class update?  For example, it
 would seem that if VACUUM updates relpages/reltuples, it would be
 prone to this same hazard.

 VACUUM does that with an in-place, nontransactional update.  But yes,
 this is a risk for every transactional catalog update.

 Well, after various efforts to fix the problem, I notice that there
 are two distinct problems brought out by your test case.

 One of them is caused by my patch, one of them was already there in
 the code - this latter one is actually the hardest to fix.

 It took me about an hour to fix the first bug, but its taken a while
 of thinking about the second before I realised it was a pre-existing
 bug.

 The core problem is, as you observed that a pg_class update can cause
 rows to be lost with concurrent scans.
 We scan pg_class in two ways: to rebuild a relcache entry based on a
 relation's oid (easy fix). We also scan pg_class to resolve the name
 to oid mapping. The name to oid mapping is performed *without* a lock
 on the relation, since we don't know which relation to lock. So the
 name lookup can fail if we are in the middle of a pg_class update.
 This is an existing potential bug in Postgres unrelated to my patch.
 Ref: SearchCatCache()

 I've been looking at ways to lock the relation name and namespace
 prior to the lookup (or more precisely the hash), but its worth
 discussing whether we want that at all?

If this is a pre-existing bug, then it's not clear to me why we need
to do anything about it at all right now.  I mean, it would be nice to
have a fix, but it's hard to imagine that any proposed fix will be
low-risk, and I don't remember user complaints about this.  I continue
to think that the root of the problem here is that SnapshotNow is Evil
(TM).  If we get rid of that, then this problem goes away, but that
strikes me as a long-term project.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Re: [COMMITTERS] pgsql: Fixed string in German translation that causes segfault.

2011-06-20 Thread Robert Haas
2011/6/20 Michael Meskes mes...@postgresql.org:
 On Mon, Jun 20, 2011 at 03:03:37PM +0300, Devrim GÜNDÜZ wrote:
 On Mon, 2011-06-20 at 11:58 +, Michael Meskes wrote:
 
  Fixed string in German translation that causes segfault.
 
  Applied patch by Christoph Berg c...@df7cb.de to replace placeholder
  %s by correct string.

 AFAIK this won't have any effect. This change needs to go to through
 pgtranslation project.

 Oops, didn't know that. Does that mean the files in our main git are not the
 canonical versions?

Yep.  Peter overrides them just before each release.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Re: patch review : Add ability to constrain backend temporary file space

2011-06-20 Thread Cédric Villemain
2011/6/17 Cédric Villemain cedric.villemain.deb...@gmail.com:
 2011/6/17 Mark Kirkwood mark.kirkw...@catalyst.net.nz:
 On 17/06/11 13:08, Mark Kirkwood wrote:

 On 17/06/11 09:49, Cédric Villemain wrote:

 I have issues applying it.
 Please can you remove trailing space?
 Also, you can generate a cool patch like this :

 get git-external-diff from postgres/src/tools to /usr/lib/git-core/
 chmod +x it
 export GIT_EXTERNAL_DIFF=git-external-diff
 git format-patch --ext-diff origin

 I think I have the trailing spaces removed, and patch is updated for the
 variable renaming recently done in fd.c

 I have no idea why I can't get the git apply to work (obviously I have
 exceeded by git foo by quite a ways), but it should apply for you I hope (as
 it patches fine).


 If I didn't made mistake the attached patch does not have trailling
 space anymore and I did a minor cosmetic in FileClose. It is not in
 the expected format required by postgresql commiters but can be
 applyed with git apply...
 It looks like the issue is that patch generated with the git-ext-diff
 can not be git applyed (they need to use patch).

 Either I did something wrong or git-ext-diff format is not so great.


 I didn't test and all yet. From reading, the patch looks sane. I'll
 review it later this day or this week-end.



Submission review
===

I have updated the patch for cosmetic.
(I attach 2 files: 1 patch for 'git-apply' and 1 for 'patch')

I have also updated the guc.c entry (lacks of a NULL for the last
Hooks handler and make the pg_settings comment more like other GUC)

The patch includes doc and test.

Usability review
=

The patch implements what it claims modulo the BLK size used by
PostgreSQL: the filewrite operation happens before the test on file
limit and PostgreSQL write per 8kB. The result is that a
temp_file_limit=0 does not prevent temp_file writing or creation, but
stop the writing at 8192 bytes.
I've already argue to keep the value as a BLKSZ, but I don't care to
have kB by default.
The doc may need to be updated to reflect this behavior.
Maybe we want to disable the temp_file writing completely with the
value set to 0 ? (it looks useless to me atm but someone may have a
usecase for that ?!)

I don't believe it follows any SQL specs, and so far the community did
not refuse the idea, so looks in the good way.
I believe we want it, my only grippe is that a large value don't
prevent server to use a lot of IO before the operation is aborted, I
don't see real alternative at this level though.

After initial testing some corner cases have been outline (in previous
versions of the patch), so I believe the current code fix them well as
I didn't have issue again.

 Feature test


The feature does not work exactly as expected because the write limit
is rounded per 8kB because we write before checking. I believe if one
write a file of 1GB in one pass (instead of repetitive 8kB increment),
and the temp_file_limit is 0, then the server will write the 1GB
before aborting.
Also, the patch does not handle a !(returnCode =0) in FileWrite. I
don't know if it is needed or not.
As the counter is per session and we can have very long session, it
may hurtbut the problem will be larger than just this counter in
such case so...is it worth the trouble to handle it ?

Performance review
===

not done.

Coding review
=

I have done some update to the patch to make it follow the coding guidelines.
I don't think it can exist a portability issue.

Tom did say: I'd think MB would be a practical unit size, and would
avoid (at least for the near term) the need to make the parameter a
float.

Architecture review


IMO, the code used to log_temp_file is partially deprecated by this
feature: the system call to stat() looks now useles and can be removed
as well as the multiple if () around temp_file_limit or log_temp_file
can be merged.

Review review
=

I didn't test for performances, I have not tryed to overflow
temporary_files_size but looks hard to raise. (this might be possible
if the temp_file_limit is not used but the temporary_files_size
incremented, but current code does not allow to increment one without
using the feature, so it is good I think)

Mark, please double check that my little updates of your patch did not
remove anything or modify the behavior in a wrong way.

Thanks for your good job,
-- 
Cédric Villemain               2ndQuadrant
http://2ndQuadrant.fr/     PostgreSQL : Expertise, Formation et Support
diff --git a/doc/src/sgml/config.sgml b/doc/src/sgml/config.sgml
index 794aef4..19dc440 100644
--- a/doc/src/sgml/config.sgml
+++ b/doc/src/sgml/config.sgml
@@ -1025,6 +1025,43 @@ SET ENABLE_SEQSCAN TO OFF;
  /variablelist
  /sect2
 
+ sect2 id=runtime-config-resource-disk
+ titleDisk/title
+ variablelist
+
+ varlistentry id=guc-temp-file-limit xreflabel=temp_file_limit
+  termvarnametemp_file_limit/varname 

Re: [HACKERS] Range Types and extensions

2011-06-20 Thread Robert Haas
On Mon, Jun 20, 2011 at 2:33 AM, Jeff Davis pg...@j-davis.com wrote:
 On Sun, 2011-06-19 at 21:29 +0200, Florian Pflug wrote:
 If I'm not mistaken about this, that would imply that we also cannot
 have two range types with the same base type, the same opclass,
 but different collations. Which seems rather unfortunate... In fact,
 if that's true, maybe restricing range types to the database collation
 would be best...

 Yes, we cannot have two range types with the same base type. That is a
 consequence of the polymorphic type system, which needs to be able to
 determine the range type given the base type.

Boy, that's an unfortunate limitation.  :-(

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Re: patch review : Add ability to constrain backend temporary file space

2011-06-20 Thread Robert Haas
On Mon, Jun 20, 2011 at 9:15 AM, Cédric Villemain
cedric.villemain.deb...@gmail.com wrote:
 The feature does not work exactly as expected because the write limit
 is rounded per 8kB because we write before checking. I believe if one
 write a file of 1GB in one pass (instead of repetitive 8kB increment),
 and the temp_file_limit is 0, then the server will write the 1GB
 before aborting.

Can we rearrange thing so we check first, and then write?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] ALTER TABLE lock strength reduction patch is unsafe

2011-06-20 Thread Simon Riggs
On Mon, Jun 20, 2011 at 2:14 PM, Robert Haas robertmh...@gmail.com wrote:

 If this is a pre-existing bug, then it's not clear to me why we need
 to do anything about it at all right now.  I mean, it would be nice to
 have a fix, but it's hard to imagine that any proposed fix will be
 low-risk, and I don't remember user complaints about this.  I continue
 to think that the root of the problem here is that SnapshotNow is Evil
 (TM).  If we get rid of that, then this problem goes away, but that
 strikes me as a long-term project.

There are 2 bugs, one caused by my patch in 9.1, one that is pre-existing.

The 9.1 bug can be fixed easily. I will edit my patch down and repost
here shortly.

The pre-existing bug is slightly harder/contentious because we have to
lock the name of a possible relation, even before we know it exists.
I've been looking to implement that as a lock on the uint32 hash of
the relation's name and namespace. I'm looking for opinions ranging
from fix-now-and-backpatch thru to ignore and discuss for 9.2.

-- 
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


[HACKERS] POSIX question

2011-06-20 Thread Radosław Smogura

Hello,

I had some idea with hugepagse, and I read why PostgreSQL doesn't 
support POSIX (need of nattach). During read about POSIX/SysV I found 
this (thread about dynamic chunking shared memory).


http://archives.postgresql.org/pgsql-hackers/2010-08/msg00586.php

When playing with mmap I done some approach how to deal with growing 
files, so...


Maybe this approach could resolve both of above problems (POSIX and 
dynamic shared memory). Here is idea:


1. mmap some large amount of anonymous virtual memory (this will be 
maximum size of shared memory).
2. init small SysV chunk for shmem header (to keep fallout 
requirements)
3. SysV remap is Linux specific so unmap few 1st vm pages of step 1. 
and attach there (2.)
3. a. Lock header when adding chunks (1st chunk is header) (we don't 
want concurrent chunk allocation)
4. allocate some other chunks of shared memory (POSIX is the best way) 
and put it in shmem header, put there information like chunk id/name, is 
this POSIX or SysV, some useful flags (hugepage?) needed by reattaching, 
attach those in 1.

4b. unlock 3a

Point 1. will no eat memory, as memory allocation is delayed and in 
64bit platforms you may reserve quite huge chunk of this, and in future 
it may be possible using mmap / munmap to concat chunks / defrag it etc.


Mmap guarants that mmaping with mmap_fixed over already mmaped area 
will unmap old.


A working preview changeset applied for sysv_memory.c maybe quite 
small.


If someone will want to extend memory, he may add new chunk (ofcourse 
to keep header memory continuous number of chunks is limited).


What do you think about this?

Regards,
Radek

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] ALTER TABLE lock strength reduction patch is unsafe

2011-06-20 Thread Kevin Grittner
Simon Riggs si...@2ndquadrant.com wrote:
 
 I'm looking for opinions ranging from fix-now-and-backpatch thru
 to ignore and discuss for 9.2.
 
If it's a pre-existing bug I would think that one option would be to
put it into the next bug-fix release of each supported major release
in which it is manifest.  Of course, if it is *safe* to work it into
9.1, that'd be great.
 
-Kevin

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Range Types and extensions

2011-06-20 Thread David Fetter
On Sun, Jun 19, 2011 at 11:33:02PM -0700, Jeff Davis wrote:
 On Sun, 2011-06-19 at 21:29 +0200, Florian Pflug wrote:
  If I'm not mistaken about this, that would imply that we also
  cannot have two range types with the same base type, the same
  opclass, but different collations. Which seems rather
  unfortunate... In fact, if that's true, maybe restricing range
  types to the database collation would be best...
 
 Yes, we cannot have two range types with the same base type. That is
 a consequence of the polymorphic type system, which needs to be able
 to determine the range type given the base type.
 
 A workaround is to use domains. That is effective, but awkward. For
 instance, given:
   CREATE DOMAIN textdomain AS text;
   CREATE TYPE textdomainrange AS RANGE (subtype=textdomain);
 then:
   '[a,z)'::textdomainrange @ 'b'::textdomain
 would work, but:
   '[a,z)'::textdomainrange @ 'b'
 would not, which would be annoying.
 
 I don't see a way around this. It's not a collation problem, but a
 general multiple range types with the same subtype problem.

How might you address that problem, assuming you had the needed
resources to do it?

Cheers,
David.
-- 
David Fetter da...@fetter.org http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter  XMPP: david.fet...@gmail.com
iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Re: [COMMITTERS] pgsql: Fixed string in German translation that causes segfault.

2011-06-20 Thread Michael Meskes
On Mon, Jun 20, 2011 at 09:15:52AM -0400, Robert Haas wrote:
 Yep.  Peter overrides them just before each release.

Aren't there better ways to implement this, like git submodules? This
redundancy seem awkward to me. 

Michael
-- 
Michael Meskes
Michael at Fam-Meskes dot De, Michael at Meskes dot (De|Com|Net|Org)
Michael at BorussiaFan dot De, Meskes at (Debian|Postgresql) dot Org
Jabber: michael.meskes at googlemail dot com
VfL Borussia! Força Barça! Go SF 49ers! Use Debian GNU/Linux, PostgreSQL

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Range Types and extensions

2011-06-20 Thread Florian Pflug
On Jun20, 2011, at 15:19 , Robert Haas wrote:
 On Mon, Jun 20, 2011 at 2:33 AM, Jeff Davis pg...@j-davis.com wrote:
 On Sun, 2011-06-19 at 21:29 +0200, Florian Pflug wrote:
 If I'm not mistaken about this, that would imply that we also cannot
 have two range types with the same base type, the same opclass,
 but different collations. Which seems rather unfortunate... In fact,
 if that's true, maybe restricing range types to the database collation
 would be best...
 
 Yes, we cannot have two range types with the same base type. That is a
 consequence of the polymorphic type system, which needs to be able to
 determine the range type given the base type.
 
 Boy, that's an unfortunate limitation.  :-(

Hm, I'm starting to wonder if there isn't a way around that. It seems that
this restriction comes from the desire to allow functions with the
polymorphic signature
  (ANYELEMENT, ANYELEMENT) - ANYRANGE.

The only such function I can currently come up with is the generic
range constructor. Is having that worth the restriction to one
range type per base type?

Another option might be to extend polymorphic argument matching
to allow functions with the signature
  (non-polymorphic args) - polymorphic type
but to require the concrete output type to be specified with a cast
at the call site. For the generic range constructor, you'd then
have to write
  RANGE(lower, upper)::range_type

(If we had that, we could also (finally) provide functions to
set and get fields of composite types by name. As it stands,
doing that cleanly is hard because the desired signature of
the get function, namely
  (record, fieldname text) - anyelement
is not supported.)

A third approach might be to first define a PAIR type and then
define ranges on top of that. Since PAIR types wouldn't include
a comparison operators, the restriction to one PAIR type per
base type wouldn't matter. Instead of a generic RANGE constructor
you'd then use the generic PAIR constructor and cast the resulting
PAIR to whatever range you desire, i.e. write
  PAIR(lower, upper)::range_type.

best regards,
Florian Pflug

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] POSIX question

2011-06-20 Thread Florian Pflug
On Jun20, 2011, at 15:27 , Radosław Smogura wrote:
 1. mmap some large amount of anonymous virtual memory (this will be maximum 
 size of shared memory).
 ...
 Point 1. will no eat memory, as memory allocation is delayed and in 64bit 
 platforms you may reserve quite huge chunk of this, and in future it may be 
 possible using mmap / munmap to concat chunks / defrag it etc.

I think this breaks with strict overcommit settings (i.e. vm.overcommit_memory 
= 2 on linux). To fix that, you'd need a way to tell the kernel (or glibc) to 
simply reserve a chunk of virtual address space for further user. Not sure if 
there's a API for that...

best regards,
Florian Pflug


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Range Types and extensions

2011-06-20 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes:
 On Mon, Jun 20, 2011 at 2:33 AM, Jeff Davis pg...@j-davis.com wrote:
 Yes, we cannot have two range types with the same base type. That is a
 consequence of the polymorphic type system, which needs to be able to
 determine the range type given the base type.

 Boy, that's an unfortunate limitation.  :-(

Given the need to deal with multiple collations for collatable types,
I'd say it's not so much unfortunate as utterly unworkable.  At
least unless you give up the notion of binding the collation into the
type definition ... which has other issues, per discussion a few days
ago.  Even ignoring collations, I really think we want to allow multiple
range types for base types that have multiple btree sort orderings.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] POSIX question

2011-06-20 Thread Radosław Smogura
Florian Pflug f...@phlo.org Monday 20 of June 2011 16:16:58
 On Jun20, 2011, at 15:27 , Radosław Smogura wrote:
  1. mmap some large amount of anonymous virtual memory (this will be
  maximum size of shared memory). ...
  Point 1. will no eat memory, as memory allocation is delayed and in 64bit
  platforms you may reserve quite huge chunk of this, and in future it may
  be possible using mmap / munmap to concat chunks / defrag it etc.
 
 I think this breaks with strict overcommit settings (i.e.
 vm.overcommit_memory = 2 on linux). To fix that, you'd need a way to tell
 the kernel (or glibc) to simply reserve a chunk of virtual address space
 for further user. Not sure if there's a API for that...
 
 best regards,
 Florian Pflug

This may be achived by many other things, like mmap /dev/null.

Regards,
Radek

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Re: patch review : Add ability to constrain backend temporary file space

2011-06-20 Thread Cédric Villemain
2011/6/20 Robert Haas robertmh...@gmail.com:
 On Mon, Jun 20, 2011 at 9:15 AM, Cédric Villemain
 cedric.villemain.deb...@gmail.com wrote:
 The feature does not work exactly as expected because the write limit
 is rounded per 8kB because we write before checking. I believe if one
 write a file of 1GB in one pass (instead of repetitive 8kB increment),
 and the temp_file_limit is 0, then the server will write the 1GB
 before aborting.

 Can we rearrange thing so we check first, and then write?

probably but it needs more work to catch corner cases. We may be safe
to just document that (and also in the code). The only way I see so
far to have a larger value than 8kB here is to have a plugin doing the
sort instead of the postgresql core sort algo.




 --
 Robert Haas
 EnterpriseDB: http://www.enterprisedb.com
 The Enterprise PostgreSQL Company




-- 
Cédric Villemain               2ndQuadrant
http://2ndQuadrant.fr/     PostgreSQL : Expertise, Formation et Support

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] pika buildfarm member failure on isolationCheck tests

2011-06-20 Thread Kevin Grittner
Dan Ports d...@csail.mit.edu wrote:
 
 It looks the problem comes from the change a couple days ago that
 removed the SXACT_FLAG_ROLLED_BACK flag and changed the
 SxactIsRolledBack checks to SxactIsDoomed. That's the correct
 thing to do everywhere else, but gets us in trouble here. We
 shouldn't be ignoring doomed transactions in
 SetNewSxactGlobalXmin, because they're not eligible for cleanup
 until the associated backend aborts the transaction and calls
 ReleasePredicateLocks.
 
 However, it isn't as simple as just removing the SxactIsDoomed
 check from SetNewSxactGlobalXmin. ReleasePredicateLocks calls
 SetNewSxactGlobalXmin *before* it releases the aborted
 transaction's SerializableXact (it pretty much has to, because we
 must drop and reacquire SerializableXactHashLock in between). But
 we don't want the aborted transaction included in the
 SxactGlobalXmin computation.
 
 It seems like we do need that SXACT_FLAG_ROLLED_BACK after all,
 even though it's only set for this brief interval. We need to be
 able to distinguish a transaction that's just been marked for
 death (doomed) from one that's already called
 ReleasePredicateLocks.
 
I just looked this over and I agree.  It looks to me like we can set
this flag in ReleasePredicateLocks (along with the doomed flag) and
test it only in SetNewSxactGlobalXmin (instead of the doomed flag). 
Fundamentally, the problem is that while it's enough to know that a
transaction *will be* canceled in order to ignore it during
detection of rw-conflicts and dangerous structures, it's not enough
when calculating the new serializable xmin -- we need to know that a
transaction actually *has been* canceled to ignore it there.
 
Essentially, the fix is to revert a very small part of
http://git.postgresql.org/gitweb?p=postgresql.git;a=commit;h=264a6b127a918800d9f8bac80b5f4a8a8799d0f1
 
Either of us could easily fix this, but since you have the test
environment which can reproduce the problem, it's probably best that
you do it.
 
Since Heikki took the trouble to put the flags in a nice logical
order, we should probably put this right after the doomed flag.
 
-Kevin

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] POSIX question

2011-06-20 Thread Florian Weimer
* Florian Pflug:

 I think this breaks with strict overcommit settings
 (i.e. vm.overcommit_memory = 2 on linux). To fix that, you'd need a
 way to tell the kernel (or glibc) to simply reserve a chunk of virtual
 address space for further user. Not sure if there's a API for that...

mmap with PROT_NONE and subsequent update with mprotect does this on
Linux.

(It's not clear to me what this is trying to solve, though.)

-- 
Florian Weimerfwei...@bfk.de
BFK edv-consulting GmbH   http://www.bfk.de/
Kriegsstraße 100  tel: +49-721-96201-1
D-76133 Karlsruhe fax: +49-721-96201-99

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] POSIX question

2011-06-20 Thread Florian Pflug
On Jun20, 2011, at 16:39 , Radosław Smogura wrote:
 Florian Pflug f...@phlo.org Monday 20 of June 2011 16:16:58
 On Jun20, 2011, at 15:27 , Radosław Smogura wrote:
 1. mmap some large amount of anonymous virtual memory (this will be
 maximum size of shared memory). ...
 Point 1. will no eat memory, as memory allocation is delayed and in 64bit
 platforms you may reserve quite huge chunk of this, and in future it may
 be possible using mmap / munmap to concat chunks / defrag it etc.
 
 I think this breaks with strict overcommit settings (i.e.
 vm.overcommit_memory = 2 on linux). To fix that, you'd need a way to tell
 the kernel (or glibc) to simply reserve a chunk of virtual address space
 for further user. Not sure if there's a API for that...
 
 best regards,
 Florian Pflug
 
 This may be achived by many other things, like mmap /dev/null.

Are you sure? Isn't mmap()ing /dev/null a way to *allocate* memory?

Or at least this is what I always thought glibc does when you malloc()
are large block at once. (This allows it to actually return the memory
to the kernel once you free() it, which isn't possible if the memory
was allocated simply by extending the heap).

You can work around this by mmap()ing an actual file, because then
the kernel knows it can use the file as backing store and thus doesn't
need to reserve actual physical memory. (In a way, this just adds
additional swap space). Doesn't seem very clean though...

Even if there's a way to work around a strict overcommit setting, unless
the workaround is a syscall *explicitly* designed for that purpose, I'd
be very careful with using it. You might just as well be exploiting a
bug in the overcommit accounting logic and future kernel versions may
simply choose to fix the bug...

best regards,
Florian Pflug


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] proposal: a validator for configuration files

2011-06-20 Thread Alexey Klyukin
Florian,

On Jun 18, 2011, at 5:40 PM, Florian Pflug wrote:

 On Jun16, 2011, at 22:34 , Alexey Klyukin wrote:
 Attached is the v2 of the patch to show all parse errors at postgresql.conf.
 Changes (per review and suggestions from Florian):
 
 - do not stop on the first error during postmaster's start.
 - fix errors in processing files from include directives.
 - show only a single syntax error per line, i.e. fast forward to the EOL 
 after coming across the first one.
 - additional comments/error messages, code cleanup
 
 Looks mostly good.
 
 I found one issue which I missed earlier. As it stands, the behaviour
 of ProcessConfigFile() changes subtly when IsUnderPostmaster because of
 the early abort on errors. Now, in theory the outcome should still be
 the same, since we don't apply any settings if there's an error in one
 of them. But still, there's a risk that this code isn't 100% waterproof,
 and then we'd end up with different settings in the postmaster compared
 to the backends. The benefit seems also quite small - since the backends
 emit their messages at DEBUG2, you usually won't see the difference
 anyway. 

I don't think it has changed at all. Previously, we did goto cleanup_list (or
cleanup_exit in ParseConfigFp) right after the first error, no matter whether
that was a postmaster or its child. What I did in my patch is removing the
goto for the postmaster's case. It was my intention to exit after the initial
error for the postmaster's child, to avoid complaining about all errors both
in the postmaster and in the normal backend (imagine seeing 100 errors from
the postmaster and the same 100 from each of the backends if your log level is
DEBUG2). I think the postmaster's child case won't cause any problems, since
we do exactly what we used to do before.


 
 The elevel setting at the start of ProcessConfigFile() also seemed
 needlessly complex, since we cannot have IsUnderPostmaster and 
 context == PGCS_POSTMASTER at the same time.

Agreed. 

 
 I figured it'd be harder to explain the fixes I have in mind than
 simply do them and let the code speak. Attached you'll find an updated
 version of your v2 patch (called v2a) as well as an incremental patch
 against your v2 (called v2a_delta).
 
 The main changes are the removal of the early aborts when
 IsUnderPostmaster and the simplification of the elevel setting
 logic in ProcessConfigFile().


Attached is the new patch and the delta. It includes some of the changes from
your patch, while leaving the early abort stuff for postmaster's children.

Thank you,
Alexey.

--
Command Prompt, Inc.  http://www.CommandPrompt.com
PostgreSQL Replication, Consulting, Custom Development, 24x7 support




pg_parser_continue_on_error_v2b.patch
Description: Binary data


pg_parser_continue_on_error_v2b_delta.patch
Description: Binary data

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] POSIX question

2011-06-20 Thread Radosław Smogura
Florian Pflug f...@phlo.org Monday 20 of June 2011 17:01:40
 On Jun20, 2011, at 16:39 , Radosław Smogura wrote:
  Florian Pflug f...@phlo.org Monday 20 of June 2011 16:16:58
  
  On Jun20, 2011, at 15:27 , Radosław Smogura wrote:
  1. mmap some large amount of anonymous virtual memory (this will be
  maximum size of shared memory). ...
  Point 1. will no eat memory, as memory allocation is delayed and in
  64bit platforms you may reserve quite huge chunk of this, and in
  future it may be possible using mmap / munmap to concat chunks /
  defrag it etc.
  
  I think this breaks with strict overcommit settings (i.e.
  vm.overcommit_memory = 2 on linux). To fix that, you'd need a way to
  tell the kernel (or glibc) to simply reserve a chunk of virtual address
  space for further user. Not sure if there's a API for that...
  
  best regards,
  Florian Pflug
  
  This may be achived by many other things, like mmap /dev/null.
 
 Are you sure? Isn't mmap()ing /dev/null a way to *allocate* memory?
 
 Or at least this is what I always thought glibc does when you malloc()
 are large block at once. (This allows it to actually return the memory
 to the kernel once you free() it, which isn't possible if the memory
 was allocated simply by extending the heap).
 
 You can work around this by mmap()ing an actual file, because then
 the kernel knows it can use the file as backing store and thus doesn't
 need to reserve actual physical memory. (In a way, this just adds
 additional swap space). Doesn't seem very clean though...
 
 Even if there's a way to work around a strict overcommit setting, unless
 the workaround is a syscall *explicitly* designed for that purpose, I'd
 be very careful with using it. You might just as well be exploiting a
 bug in the overcommit accounting logic and future kernel versions may
 simply choose to fix the bug...
 
 best regards,
 Florian Pflug

I'm sure at 99%. When I ware playing with mmap I preallocated, probably, 
about 100GB of memory.

Regards,
Radek

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] POSIX question

2011-06-20 Thread Florian Pflug
On Jun20, 2011, at 17:05 , Radosław Smogura wrote:
 I'm sure at 99%. When I ware playing with mmap I preallocated, probably, 
 about 100GB of memory.

You need to set vm.overcommit_memory to 2 to see the difference. Did
you do that?

You can do that either with echo 2  /proc/sys/vm/overcommit_memory
or by editing /etc/sysctl.conf and issuing sysctl -p.

best regards,
Florian Pflug


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] POSIX question

2011-06-20 Thread Andres Freund
On Monday, June 20, 2011 17:05:48 Radosław Smogura wrote:
 Florian Pflug f...@phlo.org Monday 20 of June 2011 17:01:40
 
  On Jun20, 2011, at 16:39 , Radosław Smogura wrote:
   Florian Pflug f...@phlo.org Monday 20 of June 2011 16:16:58
   
   On Jun20, 2011, at 15:27 , Radosław Smogura wrote:
   1. mmap some large amount of anonymous virtual memory (this will be
   maximum size of shared memory). ...
   Point 1. will no eat memory, as memory allocation is delayed and in
   64bit platforms you may reserve quite huge chunk of this, and in
   future it may be possible using mmap / munmap to concat chunks /
   defrag it etc.
   
   I think this breaks with strict overcommit settings (i.e.
   vm.overcommit_memory = 2 on linux). To fix that, you'd need a way to
   tell the kernel (or glibc) to simply reserve a chunk of virtual
   address space for further user. Not sure if there's a API for that...
   
   best regards,
   Florian Pflug
   
   This may be achived by many other things, like mmap /dev/null.
  
  Are you sure? Isn't mmap()ing /dev/null a way to *allocate* memory?
  
  Or at least this is what I always thought glibc does when you malloc()
  are large block at once. (This allows it to actually return the memory
  to the kernel once you free() it, which isn't possible if the memory
  was allocated simply by extending the heap).
  
  You can work around this by mmap()ing an actual file, because then
  the kernel knows it can use the file as backing store and thus doesn't
  need to reserve actual physical memory. (In a way, this just adds
  additional swap space). Doesn't seem very clean though...
  
  Even if there's a way to work around a strict overcommit setting, unless
  the workaround is a syscall *explicitly* designed for that purpose, I'd
  be very careful with using it. You might just as well be exploiting a
  bug in the overcommit accounting logic and future kernel versions may
  simply choose to fix the bug...
  
  best regards,
  Florian Pflug
 
 I'm sure at 99%. When I ware playing with mmap I preallocated, probably,
 about 100GB of memory.
The default setting is to allow overcommit.

Andres

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Range Types and extensions

2011-06-20 Thread Greg Stark
On Mon, Jun 20, 2011 at 3:17 PM, Tom Lane t...@sss.pgh.pa.us wrote:
 Given the need to deal with multiple collations for collatable types,
 I'd say it's not so much unfortunate as utterly unworkable.  At
 least unless you give up the notion of binding the collation into the
 type definition ... which has other issues, per discussion a few days
 ago.  Even ignoring collations, I really think we want to allow multiple
 range types for base types that have multiple btree sort orderings.

I was imagining it would be not part of the type but part of the
internal data in the range type. The dumped representation would look
something like ['bar','baz',''en_US'] and input forms like
['bar','baz'] would just default to the database default collation or
the session's default collation or whatever.

The most disturbing thing about this is that it would make
unrestorable dumps if any of those collation names change or are not
installed before the data is loaded. It's kind of like having your
table names embedded in a text column in your tables. It could make
things awkward to manage later.



-- 
greg

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] POSIX question

2011-06-20 Thread Greg Stark
On Mon, Jun 20, 2011 at 4:01 PM, Florian Pflug f...@phlo.org wrote:
 Are you sure? Isn't mmap()ing /dev/null a way to *allocate* memory?

 Or at least this is what I always thought glibc does when you malloc()

It mmaps /dev/zero actually.


-- 
greg

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] POSIX question

2011-06-20 Thread Roger Leigh
On Mon, Jun 20, 2011 at 04:16:58PM +0200, Florian Pflug wrote:
 On Jun20, 2011, at 15:27 , Radosław Smogura wrote:
  1. mmap some large amount of anonymous virtual memory (this will be maximum 
  size of shared memory).
  ...
  Point 1. will no eat memory, as memory allocation is delayed and in 64bit 
  platforms you may reserve quite huge chunk of this, and in future it may be 
  possible using mmap / munmap to concat chunks / defrag it etc.
 
 I think this breaks with strict overcommit settings (i.e. 
 vm.overcommit_memory = 2 on linux). To fix that, you'd need a way to tell the 
 kernel (or glibc) to simply reserve a chunk of virtual address space for 
 further user. Not sure if there's a API for that...

I run discless swapless cluster systems with zero overcommit (i.e.
it's entirely disabled), which means that all operations are
strict success/fail due to allocation being immediate.  mmap of a
large amount of anonymous memory would almost certainly fail on
such a setup--you definitely can't assume that a large anonymous
mmap will always succeed, since there is no delayed allocation.

[we do in reality have a small overcommit allowance to permit
efficient fork(2), but it's tiny and (in this context) irrelevant]

Regards,
Roger

-- 
  .''`.  Roger Leigh
 : :' :  Debian GNU/Linux http://people.debian.org/~rleigh/
 `. `'   Printing on GNU/Linux?   http://gutenprint.sourceforge.net/
   `-GPG Public Key: 0x25BFB848   Please GPG sign your mail.


signature.asc
Description: Digital signature


Re: [HACKERS] proposal: a validator for configuration files

2011-06-20 Thread Florian Pflug
On Jun20, 2011, at 17:02 , Alexey Klyukin wrote:
 On Jun 18, 2011, at 5:40 PM, Florian Pflug wrote:
 On Jun16, 2011, at 22:34 , Alexey Klyukin wrote:
 Attached is the v2 of the patch to show all parse errors at postgresql.conf.
 Changes (per review and suggestions from Florian):
 
 - do not stop on the first error during postmaster's start.
 - fix errors in processing files from include directives.
 - show only a single syntax error per line, i.e. fast forward to the EOL 
 after coming across the first one.
 - additional comments/error messages, code cleanup
 
 Looks mostly good.
 
 I found one issue which I missed earlier. As it stands, the behaviour
 of ProcessConfigFile() changes subtly when IsUnderPostmaster because of
 the early abort on errors. Now, in theory the outcome should still be
 the same, since we don't apply any settings if there's an error in one
 of them. But still, there's a risk that this code isn't 100% waterproof,
 and then we'd end up with different settings in the postmaster compared
 to the backends. The benefit seems also quite small - since the backends
 emit their messages at DEBUG2, you usually won't see the difference
 anyway. 
 
 I don't think it has changed at all. Previously, we did goto cleanup_list (or
 cleanup_exit in ParseConfigFp) right after the first error, no matter whether
 that was a postmaster or its child. What I did in my patch is removing the
 goto for the postmaster's case. It was my intention to exit after the initial
 error for the postmaster's child, to avoid complaining about all errors both
 in the postmaster and in the normal backend (imagine seeing 100 errors from
 the postmaster and the same 100 from each of the backends if your log level is
 DEBUG2). I think the postmaster's child case won't cause any problems, since
 we do exactly what we used to do before.

Hm, I think you miss-understood what I was trying to say, probably because I
explained it badly. Let me try again.

I fully agree that there *shouldn't* be any difference in behaviour, because
it *shouldn't* matter whether we abort early or not - we won't have applied
any of the settings anway.

But.

The code the actually implements the check settings first, apply later logic
isn't easy to read. Now, assume that this code has a bug. Then, with your
patch applied, we might end up with the postmaster applying a setting (because
it didn't abort early) but the backend ignoring it (because they did abort 
early).
This is obviously bad. Depending on the setting, the consequences may range
from slightly confusing behaviour to outright crashes I guess...

Now, the risk of that happening might be very small. But on the other hand,
the benefit is pretty small also - you get a little less output for log level
DEBUG2, that's it. A level that people probably don't use for the production
databases anyway. This convinced me that the risk/benefit ratio isn't high 
enough
to warrant the early abort.

Another benefit of removing the check is that it reduces code complexity. Maybe
not when measured in line counts, but it's one less outside factor that changes
ProcessConfigFiles()'s behaviour and thus one thing less you need to think when
you modify that part again in the future. Again, it's a small benefit, but IMHO
it still outweights the benefit.

Having said that, this is my personal opinion and whoever will eventually
commit this may very will assess the cost/benefit ratio differently. So, if
after this more detailed explanations of my reasoning, you still feel that
it makes sense to keep the early abort, then feel free to mark the
patch Ready for Committer nevertheless.

best regards,
Florian Pflug


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] POSIX question

2011-06-20 Thread Radosław Smogura
Florian Pflug f...@phlo.org Monday 20 of June 2011 17:07:55
 On Jun20, 2011, at 17:05 , Radosław Smogura wrote:
  I'm sure at 99%. When I ware playing with mmap I preallocated,
  probably, about 100GB of memory.
 
 You need to set vm.overcommit_memory to 2 to see the difference. Did
 you do that?
 
 You can do that either with echo 2  /proc/sys/vm/overcommit_memory
 or by editing /etc/sysctl.conf and issuing sysctl -p.
 
 best regards,
 Florian Pflug
I've just created 127TB mapping in Linux - maximum allowed by VM. Trying 
overcommit with 0,1,2.

Regards,
Radek

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Range Types and extensions

2011-06-20 Thread Tom Lane
Greg Stark st...@mit.edu writes:
 On Mon, Jun 20, 2011 at 3:17 PM, Tom Lane t...@sss.pgh.pa.us wrote:
 Given the need to deal with multiple collations for collatable types,
 I'd say it's not so much unfortunate as utterly unworkable.  At
 least unless you give up the notion of binding the collation into the
 type definition ... which has other issues, per discussion a few days
 ago.  Even ignoring collations, I really think we want to allow multiple
 range types for base types that have multiple btree sort orderings.

 I was imagining it would be not part of the type but part of the
 internal data in the range type. The dumped representation would look
 something like ['bar','baz',''en_US'] and input forms like
 ['bar','baz'] would just default to the database default collation or
 the session's default collation or whatever.

 The most disturbing thing about this is that it would make
 unrestorable dumps if any of those collation names change or are not
 installed before the data is loaded. It's kind of like having your
 table names embedded in a text column in your tables. It could make
 things awkward to manage later.

Yeah.  In particular this would cause issues for pg_upgrade, which would
have to somehow ensure that collation OIDs didn't change between old and
new installations, which is just about impossible given the current
method for assigning them.  I think we need to avoid that, really.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Fwd: Keywords in pg_hba.conf should be field-specific

2011-06-20 Thread Pavel Stehule
Hello Brendan,

I checked  your patch, it is applied cleanly and I don't see any mayor
problem. This patch does all what is expected.

I have two minor comments

a) you don't use macro token_matches consistently

func: parse_hba_line

--if (strcmp(token-string, local) == 0)

should be
  if (token_is_keyword(token, local))
 ...

I don't see any sense when somebody use a quotes there.

b) probably you can simplify a memory management using own two
persistent memory context - and you can swap it. Then functions like
free_hba_record, clean_hba_list, free_lines should be removed.

Regards

Pavel Stehule



2011/6/18 Brendan Jurd dire...@gmail.com:
 On 18 June 2011 13:43, Alvaro Herrera alvhe...@commandprompt.com wrote:
 Is this really a WIP patch?  I'm playing a bit with it currently, seems
 fairly sane.


 In this case, the WIP designation is meant to convey warning: only
 casual testing has beeen done.  I tried it out with various
 permutations of pg_hba.conf, and it worked as advertised in those
 tests, but I have not made any attempt to formulate a more rigorous
 testing regimen.

 In particular I haven't tested that the more exotic authentication
 methods still work properly, and I can't recall whether I tested
 recursive file inclusion and group membership.

 Is that a wrongful use of the WIP designation?

 Cheers,
 BJ


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Fwd: Keywords in pg_hba.conf should be field-specific

2011-06-20 Thread Pavel Stehule
sorry

 a) you don't use macro token_matches consistently

should be

a) you don't use macro token_is_keyword consistently

it should be used for all keywords

Regards

Pavel Stehule

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] POSIX question

2011-06-20 Thread Andres Freund
On Monday, June 20, 2011 17:11:14 Greg Stark wrote:
 On Mon, Jun 20, 2011 at 4:01 PM, Florian Pflug f...@phlo.org wrote:
  Are you sure? Isn't mmap()ing /dev/null a way to *allocate* memory?
  
  Or at least this is what I always thought glibc does when you malloc()
 
 It mmaps /dev/zero actually.
As the nitpicking has already started: Afair its just passing -1 as fd and 
uses the MAP_ANONYMOUS flag argument ;)

Andres

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


[HACKERS] Auto Start second postgres 8.3.15-1 instance MAC OS X

2011-06-20 Thread Diogo Santos
Hi, I'm used to work with PostgreSQL on Windows but now I've moved to OS X
and I'm having problems to create a service to auto start a new server
(instance) of PostgreSQL.
Firstly I used the PostgreSQL installer to create the first server and the
postgres user, then I used initdb to create another server and until this
step, everything turned fine.
After that, I created a folder on StartupItems with StartupParameters and
the other script.
Here are the contents of the two files:

#walkinsense-pgsql


#!/bin/sh

. /etc/rc.common

# Postgres Plus Service script for OS/X

StartService ()
{
ConsoleMessage Starting Walkinsense PostgreSQL
su - postgres -c /Library/PostgreSQL/8.3/bin/pg_ctl -D
/Users/iMac1/Library/Teste/data -l
/Library/Walkinsense/data/pg_log/startup.log -o \-p 5440\ start
 if [ -e /Library/Walkinsense/data/postmaster.pid ]
then
ConsoleMessage PostgreSQL 8.3 started successfully
else
ConsoleMessage PostgreSQL 8.3 did not start in a timely fashion, please see
/Library/Walkinsense/data/pg_log/startup.log for details
fi
}

StopService()
{
ConsoleMessage Stopping PostgreSQL 8.3
su - postgres -c /Library/PostgreSQL/8.3/bin/pg_ctl stop -m fast -w -D
/Users/iMac1/Library/Teste/data
}

RestartService ()
{
StopService
sleep 2
StartService
}


RunService $1



#SetupParameters

{
  Description   = Walkinsense-pgsql;
  Provides  = (walkinsense-pgsql);
  Requires  = (Resolver);
  Preference= Late;
  Messages =
  {
start = Starting Walkinsense PostgreSQL;
stop  = Stopping Walkinsense PostgreSQL;
  };
}

Then, I changed the directory permissions doing sudo chown root:wheel
/Library/StartupItems/walkinsense-pgsql at the terminal.


When I do sudo SystemStarter start walkinsense-pgsql at the terminal, it
gives back the message:
Starting Walkinsense PostgreSQL
pg_ctl: could not open PID file
/Users/iMac1/Library/Teste/data/postmaster.pid: Permission denied

As you see, the server doesn't start.

I hope I explained it right.
Thanks!!

Diogo


Re: [HACKERS] [WIP] cache estimates, cache access cost

2011-06-20 Thread Kevin Grittner
Greg Smith g...@2ndquadrant.com wrote:
 On 06/19/2011 06:15 PM, Kevin Grittner wrote:
 I think the point is that if, on a fresh system, the first access
 to a table is something which uses a tables scan -- like select
 count(*) -- that all indexed access would then  tend to be
 suppressed for that table.  After all, for each individual query,
 selfishly looking at its own needs in isolation, it likely
 *would* be faster to use the cached heap data.
 
 If those accesses can compete with other activity, such that the
 data really does stay in the cache rather than being evicted, then
 what's wrong with that?
 
The problem is that if somehow the index *does* find its way into
cache, the queries might all run an order of magnitude faster by
using it.  The *first* query to bite the bullet and read through the
index wouldn't, of course, since it would have all that random disk
access.  But its not hard to imagine an application mix where this
feature could cause a surprising ten-fold performance drop after
someone does a table scan which could persist indefinitely.  I'm not
risking that in production without a clear mechanism to
automatically recover from that sort of cache skew.
 
-Kevin

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] proposal: a validator for configuration files

2011-06-20 Thread Tom Lane
Florian Pflug f...@phlo.org writes:
 The code the actually implements the check settings first, apply later logic
 isn't easy to read. Now, assume that this code has a bug. Then, with your
 patch applied, we might end up with the postmaster applying a setting (because
 it didn't abort early) but the backend ignoring it (because they did abort 
 early).
 This is obviously bad. Depending on the setting, the consequences may range
 from slightly confusing behaviour to outright crashes I guess...

This is already known to happen: there are cases where the postmaster
and a backend can come to different conclusions about whether a setting
is valid (eg, because it depends on database encoding).  Whether that's
a bug or not isn't completely clear, but if this patch is critically
dependent on the situation never happening, I don't think we can accept
it.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Fwd: Keywords in pg_hba.conf should be field-specific

2011-06-20 Thread Alvaro Herrera
Excerpts from Pavel Stehule's message of lun jun 20 11:34:25 -0400 2011:

 b) probably you can simplify a memory management using own two
 persistent memory context - and you can swap it. Then functions like
 free_hba_record, clean_hba_list, free_lines should be removed.

Yeah, I reworked the patch with something like that over the weekend.
Not all of it though.  I'll send an updated version shortly.

-- 
Álvaro Herrera alvhe...@commandprompt.com
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Range Types and extensions

2011-06-20 Thread Jeff Davis
On Mon, 2011-06-20 at 16:01 +0200, Florian Pflug wrote:
 Hm, I'm starting to wonder if there isn't a way around that. It seems that
 this restriction comes from the desire to allow functions with the
 polymorphic signature
   (ANYELEMENT, ANYELEMENT) - ANYRANGE.
 
 The only such function I can currently come up with is the generic
 range constructor. Is having that worth the restriction to one
 range type per base type?

Good point.

Having constructors is obviously important, but perhaps they don't have
to be generic. We could generate catalog entries for each constructor
for each range type, and name them after the range type itself. So,
instead of:
  range(1, 10)
you'd write:
  int4range(1,10)

That actually might be better anyway, because relying on the polymorphic
version is not perfect now anyway. For instance, if you want an
int8range using the generic range() constructor, you need a cast.

We'd still need to get the polymorphic type system to work the way we
want in this case. I'll look into that.

 Another option might be to extend polymorphic argument matching
 to allow functions with the signature
   (non-polymorphic args) - polymorphic type
 but to require the concrete output type to be specified with a cast
 at the call site. For the generic range constructor, you'd then
 have to write
   RANGE(lower, upper)::range_type

Interesting idea. 

 A third approach might be to first define a PAIR type and then
 define ranges on top of that. Since PAIR types wouldn't include
 a comparison operators, the restriction to one PAIR type per
 base type wouldn't matter. Instead of a generic RANGE constructor
 you'd then use the generic PAIR constructor and cast the resulting
 PAIR to whatever range you desire, i.e. write
   PAIR(lower, upper)::range_type.

Another interesting idea. A little awkward though, and doesn't offer
much opportunity to specify inclusivity/exclusivity of the bounds.

Regards,
Jeff Davis


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Adding a distinct pattern type to resolve the ~ commutator stalemate

2011-06-20 Thread David E. Wheeler
On Jun 19, 2011, at 4:56 PM, Florian Pflug wrote:

 Hm, it seems we either all have different idea about how such
 a pattern type would be be defined, or have grown so accustomed to
 pg's type system that we've forgotten how powerful it really
 is ;-) (For me, the latter is surely true...).
 
 I've now created a primitive prototype that uses a composite
 type for pattern. That changes the input syntax for patterns
 (you need to enclose them in brackets), but should model all
 the implicit and explicit casting rules and operator selection
 correctly. It also uses ~~~ in place of ~, for obvious
 reasons and again without changing the casting and overloading
 rules.

Ew.

 The prototype defines
  text ~~~ text
  text ~~~ pattern
  pattern ~~~ text
 and can be found at end of this mail.
 
 With that prototype, *all* the cases (even unknown ~~~ unknown)
 work as today, i.e. all of the statements below return true

Florian++ Very nice, thanks!

I don't suppose there's a special quoting to be had for patterns? Perhaps one 
of these (modulo SQL parsing issues);

/pattern/
{pattern}
qr/pattern/
qr'pattern'
R/pattern/
R'pattern'

Mike bikeshed is scarlet,

David


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Adding a distinct pattern type to resolve the ~ commutator stalemate

2011-06-20 Thread Florian Pflug
On Jun20, 2011, at 18:28 , David E. Wheeler wrote:
 I don't suppose there's a special quoting to be had for patterns? Perhaps one 
 of these (modulo SQL parsing issues);
 
/pattern/
{pattern}
qr/pattern/
qr'pattern'
R/pattern/
R'pattern'

Pretty daring suggestion, I must say ;-)

I think regexp's are nearly prominent enough in SQL to warrant this.
Also, the main reason why this is such a huge deal for most programming
languages is that it avoids having to double-escape backslashes.

At least with standard_conforming_strings=on, however, that isn't a problem
in SQL because backslashes in literals aren't treated specially. For example
writing
  'test' ~ '^\w+$'
Just Works (TM) if standard_conforming_strings=on, whereas in C you'd
have to write
  regexp_match(test, ^\\w+$)
to give the regexp engine a chance to even see the \.

best regards,
Florian Pflug


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] proposal: a validator for configuration files

2011-06-20 Thread Florian Pflug
On Jun20, 2011, at 18:16 , Tom Lane wrote:
 Florian Pflug f...@phlo.org writes:
 The code the actually implements the check settings first, apply later 
 logic
 isn't easy to read. Now, assume that this code has a bug. Then, with your
 patch applied, we might end up with the postmaster applying a setting 
 (because
 it didn't abort early) but the backend ignoring it (because they did abort 
 early).
 This is obviously bad. Depending on the setting, the consequences may range
 from slightly confusing behaviour to outright crashes I guess...
 
 This is already known to happen: there are cases where the postmaster
 and a backend can come to different conclusions about whether a setting
 is valid (eg, because it depends on database encoding).  Whether that's
 a bug or not isn't completely clear, but if this patch is critically
 dependent on the situation never happening, I don't think we can accept
 it.

Does that mean that some backends might currently choose to ignore an
updated config file wholesale on SIGUP (because some settings are invalid)
while others happily apply it? Meaning that they'll afterwards disagree
even on modified settings which *would* be valid for both backends?

Or do these kinds of setting rejections happen late enough to fall out
of the all-or-nothing logic in ProcessConfigFile?

Anyway, the patch *doesn't* depend on all backends's setting being in sync.
The issue we were discussion was whether it's OK to add another small risk
of them getting out of sync by aborting early on errors in backends but
not in the postmaster. I was arguing against that, while Alexey was in favour
of it, on the grounds that it reduces log traffic (but only at DEBUG2 or
beyond).

best regards,
Florian Pflug


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] ALTER TABLE lock strength reduction patch is unsafe

2011-06-20 Thread Robert Haas
On Mon, Jun 20, 2011 at 9:42 AM, Kevin Grittner
kevin.gritt...@wicourts.gov wrote:
 Simon Riggs si...@2ndquadrant.com wrote:

 I'm looking for opinions ranging from fix-now-and-backpatch thru
 to ignore and discuss for 9.2.

 If it's a pre-existing bug I would think that one option would be to
 put it into the next bug-fix release of each supported major release
 in which it is manifest.  Of course, if it is *safe* to work it into
 9.1, that'd be great.

I'm currently on the other end of the spectrum: ignore and consider for 9.2.

But that's mostly based on the belief that there isn't going to be a
way of fixing this that isn't far too invasive to back-patch.  Should
that turn out to be incorrect, that's a different matter, of course...

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Re: [COMMITTERS] pgsql: Fixed string in German translation that causes segfault.

2011-06-20 Thread Robert Haas
On Mon, Jun 20, 2011 at 9:54 AM, Michael Meskes mes...@postgresql.org wrote:
 On Mon, Jun 20, 2011 at 09:15:52AM -0400, Robert Haas wrote:
 Yep.  Peter overrides them just before each release.

 Aren't there better ways to implement this, like git submodules? This
 redundancy seem awkward to me.

Dunno.  I just work here.  :-)

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Re: [COMMITTERS] pgsql: Fixed string in German translation that causes segfault.

2011-06-20 Thread Alvaro Herrera
Excerpts from Michael Meskes's message of lun jun 20 09:54:36 -0400 2011:
 On Mon, Jun 20, 2011 at 09:15:52AM -0400, Robert Haas wrote:
  Yep.  Peter overrides them just before each release.
 
 Aren't there better ways to implement this, like git submodules? This
 redundancy seem awkward to me. 

There might be, but currently the translations are still using the CVS
machinery on pgfoundry.  This stuff predates our move to Git.  It's
possible that Peter already changed the msgstr in pgtranslation ...

Peter is working on moving that CVS stuff to Git, but AFAIR it will
happen once 9.1 has released.

-- 
Álvaro Herrera alvhe...@commandprompt.com
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Range Types and extensions

2011-06-20 Thread Merlin Moncure
On Mon, Jun 20, 2011 at 11:21 AM, Jeff Davis pg...@j-davis.com wrote:
 On Mon, 2011-06-20 at 16:01 +0200, Florian Pflug wrote:
 Hm, I'm starting to wonder if there isn't a way around that. It seems that
 this restriction comes from the desire to allow functions with the
 polymorphic signature
   (ANYELEMENT, ANYELEMENT) - ANYRANGE.

 The only such function I can currently come up with is the generic
 range constructor. Is having that worth the restriction to one
 range type per base type?

 Good point.

 Having constructors is obviously important, but perhaps they don't have
 to be generic. We could generate catalog entries for each constructor
 for each range type, and name them after the range type itself. So,
 instead of:
  range(1, 10)
 you'd write:
  int4range(1,10)

 That actually might be better anyway, because relying on the polymorphic
 version is not perfect now anyway. For instance, if you want an
 int8range using the generic range() constructor, you need a cast.

 We'd still need to get the polymorphic type system to work the way we
 want in this case. I'll look into that.

hm, what if there *was( only one range type per base type, but in the
various contexts where specific ordering and collation was important
you could optionally pass them in?  Meaning, the specific ordering was
not bound rigidly to the type, but to the operation?

merlin

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] ALTER TABLE lock strength reduction patch is unsafe

2011-06-20 Thread Kevin Grittner
Robert Haas robertmh...@gmail.com wrote:
 
 I'm currently on the other end of the spectrum: ignore and
 consider for 9.2.
 
I guess part of my point was that if we can't safely get something
into the initial 9.1 release, it doesn't mean we necessarily need to
wait for 9.2.  Bugs can be fixed along the way in minor releases.  A
9.1.1 fix might give us more time to work through details and ensure
that it is a safe and well-targeted fix.
 
-Kevin

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] proposal: a validator for configuration files

2011-06-20 Thread Tom Lane
Florian Pflug f...@phlo.org writes:
 On Jun20, 2011, at 18:16 , Tom Lane wrote:
 This is already known to happen: there are cases where the postmaster
 and a backend can come to different conclusions about whether a setting
 is valid (eg, because it depends on database encoding).  Whether that's
 a bug or not isn't completely clear, but if this patch is critically
 dependent on the situation never happening, I don't think we can accept
 it.

 Does that mean that some backends might currently choose to ignore an
 updated config file wholesale on SIGUP (because some settings are invalid)
 while others happily apply it? Meaning that they'll afterwards disagree
 even on modified settings which *would* be valid for both backends?

Yes.  I complained about that before:
http://archives.postgresql.org/pgsql-hackers/2011-04/msg00330.php
but we didn't come to any consensus about fixing it.  This patch might
be a good vehicle for revisiting the issue, though.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Boolean operators without commutators vs. ALL/ANY

2011-06-20 Thread Alvaro Herrera
Excerpts from Florian Pflug's message of lun jun 20 06:55:42 -0400 2011:

 The latter (i.e. regexp literals enclosed by /../) probably isn't
 desirably for postgres, but the former definitely is (i.e. distinguishing
 regexp's and text in the type system). Please see the thread
 Adding a distinct pattern type to resolve the ~ commutator stalemate
 for the details of the proposal. 

'your text' ~ regexp 'your.*foo'
 column ~ regexp 'your.*foo'

So you could do

regexp 'foo.*bar' ~ 'your text'

and it's immediately clear what's up.

The question is what to do wrt implicit casting of text to regexp.
If we don't, there's a backwards compatibility hit.

-- 
Álvaro Herrera alvhe...@commandprompt.com
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] possible connection leak in dblink?

2011-06-20 Thread Bruce Momjian
Joe Conway wrote:
-- Start of PGP signed section.
 On 06/17/2011 01:05 PM, Tom Lane wrote:
  Peter Eisentraut pete...@gmx.net writes:
  Is this a bug fix that should be backpatched?
  
  I pinged Joe Conway about this.  He is jetlagged from a trip to the Far
  East but promised to take care of it soon.  I think we can wait for his
  review.
 
 Sorry for the delay. I'm finally caught up on most of my obligations,
 and have plowed through the 1500 or so pgsql mailing list messages that
 I was behind. But if everyone is OK with it I would like to aim to
 commit a fix mid next week, because I still have to get through my
 daughter's high school graduation tomorrow, and an out of state funeral
 for my father-in-law Sunday/Monday.
 
 That said, I really would like to commit this myself, as I have yet to
 be brave enough to commit anything under git :-(

Sounds good.  We will be here to help.

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Patch - Debug builds without optimization

2011-06-20 Thread Alvaro Herrera
Excerpts from Greg Smith's message of lun jun 20 00:25:08 -0400 2011:
 Greg Stark wrote:
  I've always wondered what other people do to iterate quickly.
 
 I'd have bet money you had an elisp program for this by now!

Yeah :-)

 The peg utility script I use makes a reinstall as simple as:
 
 stop
 peg build

But you're building the entire server there, which was Tom's point --
you only need to build and reinstall the backend.

I have my own runpg utility which does a lot of these things too ...
The main difference (to Tom's approach) is that I don't use pg_ctl to
start/stop the server, because I always keep that running in a terminal,
which makes for easier debugging because the logs are always there and I
can ctrl-c it ...  Well I guess it's pretty much the same thing, because
Tom probably has a script to stop the server.

-- 
Álvaro Herrera alvhe...@commandprompt.com
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Boolean operators without commutators vs. ALL/ANY

2011-06-20 Thread Florian Pflug
On Jun20, 2011, at 19:22 , Alvaro Herrera wrote:
 Excerpts from Florian Pflug's message of lun jun 20 06:55:42 -0400 2011:
 The latter (i.e. regexp literals enclosed by /../) probably isn't
 desirably for postgres, but the former definitely is (i.e. distinguishing
 regexp's and text in the type system). Please see the thread
 Adding a distinct pattern type to resolve the ~ commutator stalemate
 for the details of the proposal. 
 
 'your text' ~ regexp 'your.*foo'
 column ~ regexp 'your.*foo'
 
 So you could do
 
 regexp 'foo.*bar' ~ 'your text'
 
 and it's immediately clear what's up.
 
 The question is what to do wrt implicit casting of text to regexp.
 If we don't, there's a backwards compatibility hit.


No, we certainly musn't allow text to be implicitly converted to
regexp, for otherwise e.g. varchar ~ varchar becomes ambiguous.

I posted a primitive prototype for a pattern type on said thread,
which seems to do everything we require without causing compatibility
problems.

best regards,
Florian Pflug


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Range Types and extensions

2011-06-20 Thread Florian Pflug
On Jun20, 2011, at 19:16 , Merlin Moncure wrote:
 On Mon, Jun 20, 2011 at 11:21 AM, Jeff Davis pg...@j-davis.com wrote:
 hm, what if there *was( only one range type per base type, but in the
 various contexts where specific ordering and collation was important
 you could optionally pass them in?  Meaning, the specific ordering was
 not bound rigidly to the type, but to the operation?

I suggested that previously here
  http://archives.postgresql.org/pgsql-hackers/2011-06/msg00846.php

In the ensuing discussion, however, it became clear that by doing so
range types become little more than a pair of values. More specifically,
a range then *doesn't* represent a set of values, because whether or
not a value is in the range depends on a specific sort order.

Actually, you'd probably even loose the possibility of having a
normalization function for discrete base types (which makes sure
that we know that [1,2] is the same as [1,3)), because it's
hard to image one normalization function that works sensibly for
two different orderings.

So by doing that, you effectively turn a RANGE into a quadruple
(lower type, lower_included bool, upper type, upper_included bool).

best regards,
Florian Pflug


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Patch - Debug builds without optimization

2011-06-20 Thread Tom Lane
Alvaro Herrera alvhe...@commandprompt.com writes:
 Excerpts from Greg Smith's message of lun jun 20 00:25:08 -0400 2011:
 The peg utility script I use makes a reinstall as simple as:
 
 stop
 peg build

 But you're building the entire server there, which was Tom's point --
 you only need to build and reinstall the backend.

Right, I was trying to illustrate how to have minimal turnaround time
when testing a small code change.  Rebuilding from scratch is slow
enough that you lose focus while waiting.  (Or I do, anyway.)

Granted, stuff like ccache can help with that, but why not adopt a
process that's not slow in the first place?

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Range Types and extensions

2011-06-20 Thread Tom Lane
Florian Pflug f...@phlo.org writes:
 On Jun20, 2011, at 19:16 , Merlin Moncure wrote:
 On Mon, Jun 20, 2011 at 11:21 AM, Jeff Davis pg...@j-davis.com wrote:
 hm, what if there *was( only one range type per base type, but in the
 various contexts where specific ordering and collation was important
 you could optionally pass them in?  Meaning, the specific ordering was
 not bound rigidly to the type, but to the operation?

 I suggested that previously here
   http://archives.postgresql.org/pgsql-hackers/2011-06/msg00846.php

 In the ensuing discussion, however, it became clear that by doing so
 range types become little more than a pair of values. More specifically,
 a range then *doesn't* represent a set of values, because whether or
 not a value is in the range depends on a specific sort order.

Yeah, that doesn't seem like the way to go.  If a range value doesn't
represent a well-defined set of base-type values, we lose a lot of the
mathematical underpinnings for range operations.

So ... just how awful would it be if we hard-wired range types to always
use their base type's default btree sort ordering and the database's
default collation?  In principle that sucks, but I'm not sure how wide
the use-cases actually will be for other choices.

The other viable alternative seems to be to require those two properties
(btree opclass and collation) to be part of a specific range type
definition.  The complaint about that seemed to be that we couldn't
infer an ANYRANGE type given only ANYELEMENT, but could we alleviate
that by identifying one range type as the default for the base type,
and then using that one in cases where we have no ANYRANGE input?

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Another issue with invalid XML values

2011-06-20 Thread Noah Misch
Hi Florian,

I tested this patch using two systems, a CentOS 5 box with
libxml2-2.6.26-2.1.2.8.el5_5.1 and an Ubuntu 8.04 box with libxml2
2.6.31.dfsg-2ubuntu1.6.  Both failed to build with this error:

xml.c: In function `pg_xml_init':
xml.c:934: error: `xmlStructuredErrorContext' undeclared (first use in this 
function)
xml.c:934: error: (Each undeclared identifier is reported only once
xml.c:934: error: for each function it appears in.)
make[4]: *** [xml.o] Error 1

It seems `xmlStructuredErrorContext' was added rather recently.  It's not
obvious which version added it, but 2.7.8 has it and 2.7.2 does not.  Looking
at 2.6.26's sources, that version seems to use `xmlGenericErrorContext' for
both structured and generic handler functions.  Based on that, I tried with a
change from `xmlStructuredErrorContext' to `xmlGenericErrorContext' in our
xml.c.  I ran the test suite and hit some failures in the xml test; see
attached regression.diffs.  I received warnings where the tests expected
nothing or expected errors.  Looks like my version of libxml2 (2.6.26 in this
case) classifies some of these messages differently.

I suggest testing the next version with both libxml2 2.6.23, the minimum
supported version, and libxml2 2.7.8, the latest version.

On Thu, Jun 09, 2011 at 06:11:46PM +0200, Florian Pflug wrote:
 On Jun2, 2011, at 01:34 , Florian Pflug wrote:
  On Jun2, 2011, at 00:02 , Noah Misch wrote:
  On Wed, Jun 01, 2011 at 06:16:21PM +0200, Florian Pflug wrote:
  Anyway, I'll try to come up with a patch that replaces 
  xmlSetGenericErrorFunc() with xmlSetStructuredErrorFunc().
  
  Sounds sensible.  Will this impose any new libxml2 version dependency?
  
  xmlSetStructuredErrorFunc() seems to be available starting with libxml 
  2.6.0,
  release on Oct 20, 2003. Since we already require the version to be = 
  2.6.23,
  we should be OK.
  
  I won't have access to my PC the next few days, but I'll try to come up with
  a patch some time next week.
 
 Phew... I did manage to produce a patch, but it was way more work than I
 had intended to put into this.
 
 As it turns out, you loose the nicely formatted context information that
 libxml2 provides via the generic error func once you switch to structured
 error reporting. Registering handlers for both doesn't help either, since
 the generic error handler isn't called once you register a structured one.
 
 Fortunately, libxml does export xmlParserPrintFileContext() which
 generates these context messages. It, however, doesn't return a string,
 but instead passes them to the generic error handler (this time, independent
 from whether a structural error handler is registered or not).

Interesting API, there.

 As it stood, the code assumed that all third-party library re-install
 their libxml error handlers before each library call, and thus didn't
 bother to restore the old error handler itself. Since I revamped the
 error handling anyway, I removed that requirement. There is now a
 function pg_xml_done() which restores the original error handler that
 we overwrote in pg_xml_init().

Seems reasonable.

 I also realized that some libxml error (like undefined namespace prefixes)
 must be ignored during xmlparse() and friends. Otherwise, it becomes
 impossible to build XML documents from individual fragments. pg_xml_init()
 therefore now takes an argument which specifies how strict the error
 checking is supposed to be. For the moment, only XPATH() uses the strict
 mode in which we report all errors. XMLPARSE() and friends only report
 parse errors, not namespace errors.

Seems fair offhand.  We must preserve the invariant that any object with type
xml can be passed through xml_out() and then be accepted by xml_in().  I'm
not seeing any specific risks there, but I figure I'd mention it.

I couldn't reconcile this description with the code.  I do see that
xpath_internal() uses XML_PURPOSE_OTHER, but so does xmlelement().  Why is
xmlelement() also ready for the strictest error reporting?

Also note that we may be able to be more strict when xmloption = document.

 
 Finally, I had to adjust contrib/xml2 because it uses some parts of
 the core XML support like pg_xml_init().
 
 Heres the indended behaviour with the patch applied:
 
 
 We always use structured error handling. For now, the error messages
 pretty much resemble the old ones, but it's now easy to add additional
 information.
 
 XMLPARSE() and casting to XML check for parse errors only, like they do
 without the patch. They're also capable of reporting warnings, but I
 didn't find a case where the libxml parser generates a warning.
 
 XPATH() reports all errors and warnings. Trying to use XPATH() on
 a document with e.g. inconsistent namespace usage or invalid
 namespace URIs therefore now raises an error. This is *necessary*
 because libxml's XPath evaluator gets confused if it encounters
 e.g. invalid namespace URI and outputs invalid XML in response.


Re: [HACKERS] Range Types and extensions

2011-06-20 Thread Darren Duncan

Tom Lane wrote:

Florian Pflug f...@phlo.org writes:

On Jun20, 2011, at 19:16 , Merlin Moncure wrote:

On Mon, Jun 20, 2011 at 11:21 AM, Jeff Davis pg...@j-davis.com wrote:
hm, what if there *was( only one range type per base type, but in the
various contexts where specific ordering and collation was important
you could optionally pass them in?  Meaning, the specific ordering was
not bound rigidly to the type, but to the operation?



I suggested that previously here
  http://archives.postgresql.org/pgsql-hackers/2011-06/msg00846.php



In the ensuing discussion, however, it became clear that by doing so
range types become little more than a pair of values. More specifically,
a range then *doesn't* represent a set of values, because whether or
not a value is in the range depends on a specific sort order.


Yeah, that doesn't seem like the way to go.  If a range value doesn't
represent a well-defined set of base-type values, we lose a lot of the
mathematical underpinnings for range operations.

So ... just how awful would it be if we hard-wired range types to always
use their base type's default btree sort ordering and the database's
default collation?  In principle that sucks, but I'm not sure how wide
the use-cases actually will be for other choices.

The other viable alternative seems to be to require those two properties
(btree opclass and collation) to be part of a specific range type
definition.  The complaint about that seemed to be that we couldn't
infer an ANYRANGE type given only ANYELEMENT, but could we alleviate
that by identifying one range type as the default for the base type,
and then using that one in cases where we have no ANYRANGE input?

regards, tom lane


I still think that the most elegant solution is for stuff like collation to just 
be built-in to the base types that the range is ranging over, meaning we have a 
separate text base type for each text collation, and the text operators are 
polymorphic over all those base types.  Having collations and stuff as something 
off to the side not built-in to text/etc types is the root of the problem.  The 
range-specific stuff can remain ANYELEMENT and no special-casing is required. 
Also, besides range constructors, a generic membership test like value in 
range is polymorphic. -- Darren Duncan


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] pgbench--new transaction type

2011-06-20 Thread Greg Smith

Itagaki Takahiro wrote:

Anyway, I'm not sure we need to include the query mode into the pgbench's
codes. Instead, how about providing a sample script as a separate sql
file? pgbench can execute any script files with -f option.
  


When you execute using -f, it doesn't correctly detect database 
scale.  Also, the really valuable thing here is seeing the higher 
selects/second number come out in the report.  I just realized neither 
Jeff nor myself ever included an example of the output in the new mode, 
which helps explain some of why the patch is built the way it is:


$ pgbench -c 12 -j 4 -T 30 -P pgbench
plgsql function created.
starting vacuum...end.
transaction type: SELECT only via plpgsql
scaling factor: 100
query mode: simple
number of clients: 12
number of threads: 4
duration: 30 s
number of transactions actually processed: 9342
tps = 311.056293 (including connections establishing)
tps = 311.117886 (excluding connections establishing)
selects per second = 159260.822100 (including connections establishing)
selects per second = 159292.357672 (excluding connections establishing)

--
Greg Smith   2ndQuadrant USg...@2ndquadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support  www.2ndQuadrant.us



--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] [WIP] cache estimates, cache access cost

2011-06-20 Thread Greg Smith

Kevin Grittner wrote:

But its not hard to imagine an application mix where this
feature could cause a surprising ten-fold performance drop after
someone does a table scan which could persist indefinitely.  I'm not
risking that in production without a clear mechanism to
automatically recover from that sort of cache skew


The idea that any of this will run automatically is a dream at this 
point, so saying you want to automatically recover from problems with 
the mechanism that doesn't even exist yet is a bit premature.  Some of 
the implementation ideas here might eventually lead to where real-time 
cache information is used, and that is where the really scary feedback 
loops you are right to be worried about come into play.  The idea for 
now is that you'll run this new type of ANALYZE CACHE operation 
manually, supervised and at a time where recent activity reflects the 
sort of workload you want to optimize for.  And then you should review 
its results to make sure the conclusions it drew about your cache 
population aren't really strange.


To help with that, I was thinking of writing a sanity check tool that 
showed how the cached percentages this discovers compare against the 
historical block hit percentages for the relation.  An example of how 
values changed from what they were already set to after a second ANALYZE 
CACHE is probably useful too.


--
Greg Smith   2ndQuadrant USg...@2ndquadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support  www.2ndQuadrant.us



--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Range Types and extensions

2011-06-20 Thread Tom Lane
Darren Duncan dar...@darrenduncan.net writes:
 I still think that the most elegant solution is for stuff like collation to 
 just 
 be built-in to the base types that the range is ranging over, meaning we have 
 a 
 separate text base type for each text collation, and the text operators are 
 polymorphic over all those base types.  Having collations and stuff as 
 something 
 off to the side not built-in to text/etc types is the root of the
 problem.

I tend to agree that this aspect of the SQL standard isn't terribly well
designed, but it's the standard and we're stuck with it.  We're not
going to support two parallel methods of dealing with collations.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] [WIP] cache estimates, cache access cost

2011-06-20 Thread Kevin Grittner
Greg Smith g...@2ndquadrant.com wrote:
 
 The idea that any of this will run automatically is a dream at
 this point, so saying you want to automatically recover from
 problems with the mechanism that doesn't even exist yet is a bit
 premature.
 
Well, I certainly didn't mean it to be a reason not to move forward
with development -- I wouldn't have raised the issue had you not
said this upthread:
 
 I don't see how sequential scan vs. index costing will be any
 different on a fresh system than it is now.
 
All I was saying is: I do; here's how...
 
Carry on.
 
-Kevin

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Another issue with invalid XML values

2011-06-20 Thread Florian Pflug
Hi

Thanks for the extensive review, it's very much appreciated!

On Jun20, 2011, at 19:57 , Noah Misch wrote:
 I tested this patch using two systems, a CentOS 5 box with
 libxml2-2.6.26-2.1.2.8.el5_5.1 and an Ubuntu 8.04 box with libxml2
 2.6.31.dfsg-2ubuntu1.6.  Both failed to build with this error:
 
 xml.c: In function `pg_xml_init':
 xml.c:934: error: `xmlStructuredErrorContext' undeclared (first use in this 
 function)
 xml.c:934: error: (Each undeclared identifier is reported only once
 xml.c:934: error: for each function it appears in.)
 make[4]: *** [xml.o] Error 1
 
 It seems `xmlStructuredErrorContext' was added rather recently.  It's not
 obvious which version added it, but 2.7.8 has it and 2.7.2 does not.  Looking
 at 2.6.26's sources, that version seems to use `xmlGenericErrorContext' for
 both structured and generic handler functions.  Based on that, I tried with a
 change from `xmlStructuredErrorContext' to `xmlGenericErrorContext' in our
 xml.c.  I ran the test suite and hit some failures in the xml test; see
 attached regression.diffs.  I received warnings where the tests expected
 nothing or expected errors. 

Great :-(. I wonder if maybe I should simply remove the part of the patch
which try to restore the error handler and context in pg_xml_done(). I've
added that because I feared that some 3rd-party libraries setup their libxml
error handle just once, not before every library class. The code previously
didn't care about this either, but since we're using structured instead of
generic error reporting now, we'd now collide with 3-rd party libs which
also use structured error handling, whereas previously that would have worked.
OTOH, once there's more than one such library...

Whats your stand on this?

  Looks like my version of libxml2 (2.6.26 in this
 case) classifies some of these messages differently.

Hm, that's exactly what I hoped that this patch would *prevent* from happening..
Will look into this.

 On Thu, Jun 09, 2011 at 06:11:46PM +0200, Florian Pflug wrote:
 On Jun2, 2011, at 01:34 , Florian Pflug wrote:
 I also realized that some libxml error (like undefined namespace prefixes)
 must be ignored during xmlparse() and friends. Otherwise, it becomes
 impossible to build XML documents from individual fragments. pg_xml_init()
 therefore now takes an argument which specifies how strict the error
 checking is supposed to be. For the moment, only XPATH() uses the strict
 mode in which we report all errors. XMLPARSE() and friends only report
 parse errors, not namespace errors.
 
 Seems fair offhand.  We must preserve the invariant that any object with type
 xml can be passed through xml_out() and then be accepted by xml_in().  I'm
 not seeing any specific risks there, but I figure I'd mention it.
 
 I couldn't reconcile this description with the code.  I do see that
 xpath_internal() uses XML_PURPOSE_OTHER, but so does xmlelement().  Why is
 xmlelement() also ready for the strictest error reporting?

xmlelement() doesn't use libxml's parser, only the xml writer. I didn't
want to suppress any errors the writer might raise (I'm not sure it raises
error at all, though).

 Also note that we may be able to be more strict when xmloption = document.

Yeah, probably. I think that better done in a separate patch, though - I've
tried not to change existing behaviour with this patch except where absolutely
necessary (i.e. for XPATH)

 Ideally, an invalid namespace URI should always be an error.  While namespace
 prefixes could become valid as you assemble a larger document, a namespace
 URI's validity is context-free.

I agree. That, however, would require xmlelement() to check all xmlns*
attributes, and I didn't find a way to do that with libxml() (Well,
other than to parse the element after creating it, but that's a huge
kludge). So I left that issue for a later time...

 The patch adds some trailing whitespace.

Ups, sorry for that. Will fix.

 Remembering to put a pg_xml_done() in every relevant elog(ERROR, ...) path
 seems fragile.  How about using RegisterXactCallback/RegisterSubXactCallback
 in pgxml_parser_init() to handle those cases?  You can also use it to Assert
 at non-abort transaction shutdown that pg_xml_done() was called as needed.

Hm, isn't that a bit fragile too? It seems that PG_TRY/PG_CATCH blocks don't
necessarily create sub-transactions, do they?

Also, most elog() paths already contained xmlFree() calls, because libxml
seemingly cannot be made to use context-based memory management. So one
already needs to be pretty careful when touching these...

Anyway, should we decide to give up on trying to restore the error handler
and context, we could get rid of pg_xml_done() again... (See above for the
details on that)

 Consider renaming XML_REPORT_ERROR it to something like XML_CHECK_ERROR,
 emphasizing that it wraps a conditional.

Hm, I think it was named XML_CHECK_ERROR at one point, and I changed it
because I felt it didn't convey the fact that it's actually ereport()

Re: [HACKERS] Range Types and extensions

2011-06-20 Thread Florian Pflug
On Jun20, 2011, at 20:58 , Tom Lane wrote:
 Darren Duncan dar...@darrenduncan.net writes:
 I still think that the most elegant solution is for stuff like collation to 
 just 
 be built-in to the base types that the range is ranging over, meaning we 
 have a 
 separate text base type for each text collation, and the text operators are 
 polymorphic over all those base types.  Having collations and stuff as 
 something 
 off to the side not built-in to text/etc types is the root of the
 problem.
 
 I tend to agree that this aspect of the SQL standard isn't terribly well
 designed, but it's the standard and we're stuck with it.  We're not
 going to support two parallel methods of dealing with collations.

Plus, you can always define a DOMAIN for every collation you intent to use,
and stay clear of COLLATE clauses except as part of these domain definitions.

Most interestingly, this is also the workaround Jeff Davis suggested for
those who absolutely need two range types over the same base type (i.e.
define one of the ranges over a domain).

best regards,
Florian Pflug


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] patch for 9.2: enhanced errors

2011-06-20 Thread Pavel Stehule
Hello

I am sending a updated patch


 Coding Review
 -


 In tupdesc.c

 line 202 the existing code is performing a deep copy of ConstrCheck.  Do you
 need to copy nkeys and conkey here as well?

 Then at line 250 ccname is freed but not conkey


fixed


 postgres_ext.h line 55
 + #define PG_DIAG_SCHEMA_NAME    's'
 + #define PG_DIAG_TABLE_NAME    't'
 + #define PG_DIAG_COLUMN_NAMES    'c'
 + #define PG_DIAG_CONSTRAINT_NAME 'n'

 The assignment of letters to parameters seems arbitrary to me, I don't have
 a different non-arbitrary mapping in mind but if anyone else does they
 should speak up.  I think it will be difficult to change this after 9.2 goes
 out.


 elog.c:
 ***
 *** 2197,2202 
 --- 2299,2319 
      if (application_name)
          appendCSVLiteral(buf, application_name);

 +     /* constraint_name */
 +     appendCSVLiteral(buf, edata-constraint_name);
 +     appendStringInfoChar(buf, ',');
 +
 +     /* schema name */
 +     appendCSVLiteral(buf, edata-schema_name);
 +     appendStringInfoChar(buf, ',');

 You need to update config.sgml at the same time you update this format.
 You need to append a , after application name but before constraintName.
 As it stands the CSV log has something like:
 .nbtinsert.c:433,psqla_pkey,public,a,a

fixed



 nbtinsert.c

 pg_get_indrelation is named differently than everything else in this file
 (ie _bt...).  My guess is that this function belongs somewhere else but I
 don't know the code well enough to say where you should move it too.


I renamed this function to IndexRelationGetParentRelation and muved to
relcache.c

I don't call a quote_identifier on only data error properties like
table_name or schema_name (but I am open to arguments for it or
against it). The quote_identifier is used for column names, because
there should be a more names and comma should be used inside name -
and this is consistent with pg_get_indexdef_columns.

Regards

Pavel Stehule
*** ./doc/src/sgml/config.sgml.orig	2011-06-20 18:08:39.0 +0200
--- ./doc/src/sgml/config.sgml	2011-06-20 21:12:31.688165497 +0200
***
*** 3919,3927 
  user query that led to the error (if any and enabled by
  varnamelog_min_error_statement/),
  character count of the error position therein,
! location of the error in the PostgreSQL source code
  (if varnamelog_error_verbosity/ is set to literalverbose/),
! and application name.
  Here is a sample table definition for storing CSV-format log output:
  
  programlisting
--- 3919,3933 
  user query that led to the error (if any and enabled by
  varnamelog_min_error_statement/),
  character count of the error position therein,
! location of the error in the PostgreSQL source code,
  (if varnamelog_error_verbosity/ is set to literalverbose/),
! application name,
! (following fields to end are filled when varnamelog_error_verbosity/ is
! set to literalverbose/)
! constraint name,
! schema name,
! table name,
! and column names.
  Here is a sample table definition for storing CSV-format log output:
  
  programlisting
***
*** 3950,3955 
--- 3956,3965 
query_pos integer,
location text,
application_name text,
+   constraint_name text,
+   schema_name text,
+   table_name text,
+   column_names text,
PRIMARY KEY (session_id, session_line_num)
  );
  /programlisting
*** ./src/backend/access/common/tupdesc.c.orig	2011-06-20 18:08:39.0 +0200
--- ./src/backend/access/common/tupdesc.c	2011-06-20 20:30:15.477883102 +0200
***
*** 200,205 
--- 200,217 
  	cpy-check[i].ccname = pstrdup(constr-check[i].ccname);
  if (constr-check[i].ccbin)
  	cpy-check[i].ccbin = pstrdup(constr-check[i].ccbin);
+ if (constr-check[i].nkeys  0)
+ {
+ 	cpy-check[i].conkey = palloc(sizeof(int16) * constr-check[i].nkeys);
+ 	memcpy(cpy-check[i].conkey, constr-check[i].conkey,
+ 		sizeof(int16) * constr-check[i].nkeys);
+ 	cpy-check[i].nkeys = constr-check[i].nkeys;
+ }
+ else
+ {
+ 	cpy-check[i].conkey = NULL;
+ 	constr-check[i].nkeys = 0;
+ }
  			}
  		}
  
***
*** 249,254 
--- 261,268 
  	pfree(check[i].ccname);
  if (check[i].ccbin)
  	pfree(check[i].ccbin);
+ if (check[i].conkey)
+ 	pfree(check[i].conkey);
  			}
  			pfree(check);
  		}
***
*** 409,414 
--- 423,431 
  			 * Similarly, don't assume that the checks are always read in the
  			 * same order; match them up by name and contents. (The name
  			 * *should* be unique, but...)
+ 			 *
+ 			 * nkeys and conkey depends on ccbin, and there are not neccessary
+ 			 * to compare it.
  			 */
  			for (j = 0; j  n; check2++, j++)
  			{
*** ./src/backend/access/nbtree/nbtinsert.c.orig	2011-06-20 18:08:39.0 +0200
--- 

Re: [HACKERS] Range Types and extensions

2011-06-20 Thread Darren Duncan

Florian Pflug wrote:

On Jun20, 2011, at 20:58 , Tom Lane wrote:

Darren Duncan dar...@darrenduncan.net writes:
I still think that the most elegant solution is for stuff like collation to just 
be built-in to the base types that the range is ranging over, meaning we have a 
separate text base type for each text collation, and the text operators are 
polymorphic over all those base types.  Having collations and stuff as something 
off to the side not built-in to text/etc types is the root of the

problem.

I tend to agree that this aspect of the SQL standard isn't terribly well
designed, but it's the standard and we're stuck with it.  We're not
going to support two parallel methods of dealing with collations.


Plus, you can always define a DOMAIN for every collation you intent to use,
and stay clear of COLLATE clauses except as part of these domain definitions.

Most interestingly, this is also the workaround Jeff Davis suggested for
those who absolutely need two range types over the same base type (i.e.
define one of the ranges over a domain).

best regards,
Florian Pflug


That DOMAIN-based solution ostensibly sounds like a good one then, under the 
circumstances.  What I *don't* want to see is for things like ranges to have 
their own collations and the like.  From the perspective of all range-specific 
things, the types over which they're defined like text should just have their 
own native ordering, which defines the range's sense of before and after. 
If DOMAIN effectively does that for text types, then that is the way to go. -- 
Darren Duncan


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


[HACKERS] REMINDER: Participation Requested: Survey about Open-Source Software Development

2011-06-20 Thread Jeffrey Carver
Hi,

Apologies for any inconvenience and thank you to those who have already
completed the survey. We will keep the survey open for another couple of
weeks. But, we do hope you will consider responding to the email request
below (sent 2 weeks ago).

Thanks,

Dr. Jeffrey Carver
Assistant Professor
University of Alabama
(v) 205-348-9829  (f) 205-348-0219
http://www.cs.ua.edu/~carver

-Original Message-
From: Jeffrey Carver [mailto:opensourcesur...@cs.ua.edu] 
Sent: Monday, June 13, 2011 11:58 AM
To: 'pgsql-hackers@postgresql.org'
Subject: Participation Requested: Survey about Open-Source Software
Development

Hi,

Drs. Jeffrey Carver, Rosanna Guadagno, Debra McCallum, and Mr. Amiangshu
Bosu,  University of Alabama, and Dr. Lorin Hochstein, University of
Southern California, are conducting a survey of open-source software
developers. This survey seeks to understand how developers on distributed,
virtual teams, like open-source projects, interact with each other to
accomplish their tasks. You must be at least 19 years of age to complete the
survey. The survey should take approximately 15 minutes to complete.

If you are actively participating as a developer, please consider completing
our survey.
 
Here is the link to the survey:   http://goo.gl/HQnux

We apologize for inconvenience and if you receive multiple copies of this
email. This survey has been approved by The University of Alabama IRB board.

Thanks,

Dr. Jeffrey Carver
Assistant Professor
University of Alabama
(v) 205-348-9829  (f) 205-348-0219
http://www.cs.ua.edu/~carver




-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] FOR KEY LOCK foreign keys

2011-06-20 Thread Noah Misch
On Sun, Jun 19, 2011 at 06:30:41PM +0200, Jesper Krogh wrote:
 I hope this hasn't been forgotten. But I cant see it has been committed  
 or moved
 into the commitfest process?

If you're asking about that main patch for $SUBJECT rather than those
isolationtester changes specifically, I can't speak to the plans for it.  I
wasn't planning to move the test suite work forward independent of the core
patch it serves, but we could do that if there's another application.

Thanks,
nm

 On 2011-03-11 16:51, Noah Misch wrote:
 On Fri, Feb 11, 2011 at 02:13:22AM -0500, Noah Misch wrote:
 Automated tests would go a long way toward building confidence that this 
 patch
 does the right thing.  Thanks to the SSI patch, we now have an in-tree test
 framework for testing interleaved transactions.  The only thing it needs to 
 be
 suitable for this work is a way to handle blocked commands.  If you like, I 
 can
 try to whip something up for that.
 [off-list ACK followed]

 Here's a patch implementing that.  It applies to master, with or without your
 KEY LOCK patch also applied, though the expected outputs reflect the
 improvements from your patch.  I add three isolation test specs:

fk-contention: blocking-only test case from your blog post
fk-deadlock: the deadlocking test case I used during patch review
fk-deadlock2: Joel Jacobson's deadlocking test case

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Re: [COMMITTERS] pgsql: Fixed string in German translation that causes segfault.

2011-06-20 Thread Peter Eisentraut
On mån, 2011-06-20 at 13:13 -0400, Alvaro Herrera wrote:
 Excerpts from Michael Meskes's message of lun jun 20 09:54:36 -0400 2011:
  On Mon, Jun 20, 2011 at 09:15:52AM -0400, Robert Haas wrote:
   Yep.  Peter overrides them just before each release.
  
  Aren't there better ways to implement this, like git submodules? This
  redundancy seem awkward to me. 
 
 There might be, but currently the translations are still using the CVS
 machinery on pgfoundry.  This stuff predates our move to Git.  It's
 possible that Peter already changed the msgstr in pgtranslation ...
 
 Peter is working on moving that CVS stuff to Git, but AFAIR it will
 happen once 9.1 has released.

A better way might be that translators simply work on a clone of the
source repository, which is then merged (as in, git merge) at release
time.  There are some issues with that to figure out, but it sounds like
the obviously right thing, from an interface point of view.



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Re: [COMMITTERS] pgsql: Fixed string in German translation that causes segfault.

2011-06-20 Thread Magnus Hagander
On Mon, Jun 20, 2011 at 22:20, Peter Eisentraut pete...@gmx.net wrote:
 On mån, 2011-06-20 at 13:13 -0400, Alvaro Herrera wrote:
 Excerpts from Michael Meskes's message of lun jun 20 09:54:36 -0400 2011:
  On Mon, Jun 20, 2011 at 09:15:52AM -0400, Robert Haas wrote:
   Yep.  Peter overrides them just before each release.
 
  Aren't there better ways to implement this, like git submodules? This
  redundancy seem awkward to me.

 There might be, but currently the translations are still using the CVS
 machinery on pgfoundry.  This stuff predates our move to Git.  It's
 possible that Peter already changed the msgstr in pgtranslation ...

 Peter is working on moving that CVS stuff to Git, but AFAIR it will
 happen once 9.1 has released.

 A better way might be that translators simply work on a clone of the
 source repository, which is then merged (as in, git merge) at release
 time.  There are some issues with that to figure out, but it sounds like
 the obviously right thing, from an interface point of view.

I don't think we want to track every single translation update as
commits in the main repository - we don't do that for non-translation
stuff... If it's a squash-merge, that's a different thing, of
course...

Other than that, yes, keeping translations in git branches seems like
a good interface.

-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Re: [COMMITTERS] pgsql: Don't use cp -i in the example WAL archive_command.

2011-06-20 Thread Tom Lane
Fujii Masao masao.fu...@gmail.com writes:
 Anyway, you seem to have forgotten to fix the example of archive_command
 in 24.3.5.1. Standalone Hot Backups.

  archive_command = 'test ! -f /var/lib/pgsql/backup_in_progress ||
 cp -i %p /var/lib/pgsql/archive/%f  /dev/null'

You're right, I didn't see that one.  Fixed, thanks.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Re: [COMMITTERS] pgsql: Fixed string in German translation that causes segfault.

2011-06-20 Thread Tom Lane
Magnus Hagander mag...@hagander.net writes:
 On Mon, Jun 20, 2011 at 22:20, Peter Eisentraut pete...@gmx.net wrote:
 A better way might be that translators simply work on a clone of the
 source repository, which is then merged (as in, git merge) at release
 time.  There are some issues with that to figure out, but it sounds like
 the obviously right thing, from an interface point of view.

 I don't think we want to track every single translation update as
 commits in the main repository - we don't do that for non-translation
 stuff... If it's a squash-merge, that's a different thing, of
 course...

 Other than that, yes, keeping translations in git branches seems like
 a good interface.

My recollection is that the current setup was created mainly so that
translators wouldn't need to be given commit privileges on the main
repo.  Giving them a separate repo to work in might be all right, but
of course whoever does the merges would have to be careful to only
accept changes made to the .po files and not anything else.

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Another issue with invalid XML values

2011-06-20 Thread Noah Misch
On Mon, Jun 20, 2011 at 09:15:51PM +0200, Florian Pflug wrote:
 On Jun20, 2011, at 19:57 , Noah Misch wrote:
  I tested this patch using two systems, a CentOS 5 box with
  libxml2-2.6.26-2.1.2.8.el5_5.1 and an Ubuntu 8.04 box with libxml2
  2.6.31.dfsg-2ubuntu1.6.  Both failed to build with this error:
  
  xml.c: In function `pg_xml_init':
  xml.c:934: error: `xmlStructuredErrorContext' undeclared (first use in this 
  function)
  xml.c:934: error: (Each undeclared identifier is reported only once
  xml.c:934: error: for each function it appears in.)
  make[4]: *** [xml.o] Error 1
  
  It seems `xmlStructuredErrorContext' was added rather recently.  It's not
  obvious which version added it, but 2.7.8 has it and 2.7.2 does not.  
  Looking
  at 2.6.26's sources, that version seems to use `xmlGenericErrorContext' for
  both structured and generic handler functions.  Based on that, I tried with 
  a
  change from `xmlStructuredErrorContext' to `xmlGenericErrorContext' in our
  xml.c.  I ran the test suite and hit some failures in the xml test; see
  attached regression.diffs.  I received warnings where the tests expected
  nothing or expected errors. 
 
 Great :-(. I wonder if maybe I should simply remove the part of the patch
 which try to restore the error handler and context in pg_xml_done(). I've
 added that because I feared that some 3rd-party libraries setup their libxml
 error handle just once, not before every library class. The code previously
 didn't care about this either, but since we're using structured instead of
 generic error reporting now, we'd now collide with 3-rd party libs which
 also use structured error handling, whereas previously that would have worked.
 OTOH, once there's more than one such library...
 
 Whats your stand on this?

Assuming it's sufficient to save/restore xmlStructuredErrorContext when it's
available and xmlGenericErrorContext otherwise, I would just add an Autoconf
check to identify which one to use.

  I couldn't reconcile this description with the code.  I do see that
  xpath_internal() uses XML_PURPOSE_OTHER, but so does xmlelement().  Why is
  xmlelement() also ready for the strictest error reporting?
 
 xmlelement() doesn't use libxml's parser, only the xml writer. I didn't
 want to suppress any errors the writer might raise (I'm not sure it raises
 error at all, though).

Makes sense.

  Also note that we may be able to be more strict when xmloption = document.
 
 Yeah, probably. I think that better done in a separate patch, though - I've
 tried not to change existing behaviour with this patch except where absolutely
 necessary (i.e. for XPATH)

Likewise.

  Ideally, an invalid namespace URI should always be an error.  While 
  namespace
  prefixes could become valid as you assemble a larger document, a namespace
  URI's validity is context-free.
 
 I agree. That, however, would require xmlelement() to check all xmlns*
 attributes, and I didn't find a way to do that with libxml() (Well,
 other than to parse the element after creating it, but that's a huge
 kludge). So I left that issue for a later time...

Likewise.

  Remembering to put a pg_xml_done() in every relevant elog(ERROR, ...) path
  seems fragile.  How about using RegisterXactCallback/RegisterSubXactCallback
  in pgxml_parser_init() to handle those cases?  You can also use it to Assert
  at non-abort transaction shutdown that pg_xml_done() was called as needed.
 
 Hm, isn't that a bit fragile too? It seems that PG_TRY/PG_CATCH blocks don't
 necessarily create sub-transactions, do they?

PG_TRY never creates a subtransaction.  However, elog(ERROR, ...) aborts
either the subtransaction, or if none, the top-level transaction.  Therefore,
registering the same callback with both RegisterXactCallback and
RegisterSubXactCallback ensures that it gets called for any elog/ereport
non-local exit.  Then you only explicitly call pg_xml_done() in the non-error
path.

However, I see now that in the core xml.c, every error-condition pg_xml_done()
appears in a PG_CATCH block.  That's plenty reliable ...

 Also, most elog() paths already contained xmlFree() calls, because libxml
 seemingly cannot be made to use context-based memory management. So one
 already needs to be pretty careful when touching these...

... and this is a good reason to keep doing it that way.  So, scratch that idea.

  Consider renaming XML_REPORT_ERROR it to something like XML_CHECK_ERROR,
  emphasizing that it wraps a conditional.
 
 Hm, I think it was named XML_CHECK_ERROR at one point, and I changed it
 because I felt it didn't convey the fact that it's actually ereport()
 and not just return false on error. But I agree that XML_REPORT_ERROR isn't
 very self-explanatory, either. What about XML_REPORT_PENDING_ERROR()
 or XML_CHECK_AND_REPORT_ERROR()?

XML_CHECK_AND_REPORT_ERROR() seems fine.  Or XML_CHECK_AND_EREPORT()?

  On Thu, Jun 09, 2011 at 06:11:46PM +0200, Florian Pflug wrote:
  *** xml_parse(text *data, XmlOptionType 

Re: [HACKERS] [BUG] Denormal float values break backup/restore

2011-06-20 Thread Jeroen Vermeulen

On 2011-06-20 19:22, Marti Raudsepp wrote:


AIUI that is defined to be a little vague, but includes denormalized numbers
that would undergo any rounding at all.  It says that on overflow the
conversion should return the appropriate HUGE_VAL variant, and set ERANGE.
  On underflow it returns a reasonably appropriate value (and either may or
must set ERANGE, which is the part that isn't clear to me).


Which standard is that? Does IEEE 754 itself define strtod() or is
there another relevant standard?


Urr.  No, this is C and/or Unix standards, not IEEE 754.

I did some more research into this.  The postgres docs do specify the 
range error, but seemingly based on a different interpretation of 
underflow than what I found in some of the instances of strtod() 
documentation:


Numbers too close to zero that are not representable as
distinct from zero will cause an underflow error.

This talks about denormals that get _all_ their significant digits 
rounded away, but some of the documents I saw specify an underflow for 
denormals that get _any_ of their significant digits rounded away (and 
thus have an abnormally high relative rounding error).


The latter would happen for any number that is small enough to be 
denormal, and is also not representable (note: that's not the same thing 
as not representable as distinct from zero!).  It's easy to get 
non-representable numbers when dumping binary floats in a decimal 
format.  For instance 0.1 is not representable, nor are 0.2, 0.01, and 
so on.  The inherent rounding of non-representable values produces 
weirdness like 0.1 + 0.2 - 0.3 != 0.


I made a quick round of the strtod() specifications I could find, and 
they seem to disagree wildly:


  SourceERANGE when Return what
-
PostgreSQL docs All digits lost zero
Linux programmer's manual   All digits lost zero
My GNU/Linux strtod()   Any digits lost rounded number
SunOS 5 Any digits lost rounded number
GNU documentation   All digits lost zero
IEEE 1003.1 (Open Group 2004)   Any digits lost denormal
JTC1/SC22/WG14 N794 Any digits lost denormal
Sun Studio (C99)Implementation-defined  ?
ISO/IEC 9899:TC2Implementation-defined  denormal
C99 Draft N869 (1999)   Implementation-defined  denormal

We can't guarantee very much, then.  It looks like C99 disagrees with 
the postgres interpretation, but also leaves a lot up to the compiler.


I've got a few ideas for solving this, but none of them are very good:

(a) Ignore underflow errors.

This could hurt anyone who relies on knowing their floating-point 
implementation and the underflow error to keep their rounding errors in 
check.  It also leaves a kind of gap in the predictability of the 
database's floating-point behaviour.


Worst hit, or possibly the only real problem, would be algorithms that 
divide other numbers, small enough not to produce infinities, by rounded 
denormals.


(b) Dump REAL and DOUBLE PRECISION in hex.

With this change, the representation problem goes away and ERANGE would 
reliably mean this was written in a precision that I can't reproduce. 
We could sensibly provide an option to ignore that error for 
cross-platform dump/restores.


This trick does raise a bunch of compatibility concerns: it's a new 
format of data to restore, it may not work on pre-C99 compilers, and so 
on.  Also, output for human consumption would have to differ from 
pg_dump output.


(c) Have pg_dump produce calculations, not literals, for denormals.

Did I mention how these were not great ideas?  If your database dump 
contains 1e-308, pg_dump could recognize that this value can be 
calculated in the database but possibly not entered directly, and dump 
e.g. 1e-307::float / 10 instead.


(d) Make pg_dump set some ignore underflows option.

This may make dumps unusable for older postgres versions.  Moreover, it 
doesn't help ORMs and applications that are currently unable to store 
the problem numbers.


(e) Do what the documentation promises.

Actually I have no idea how we could guarantee this.

(f) Ignore ERANGE unless strtod() returns ±0 or ±HUGE_VAL.

This is probably a reasonable stab at common sense.  It does have the 
nasty property that it doesn't give a full guarantee either way: 
restores could still break on pre-C99 systems that return 0 on 
underflow, but C99 doesn't guarantee a particularly accurate denormal. 
In practice though, implementations seem to do their best to give you 
the most appropriate rounded number.



Jeroen

--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Re: [COMMITTERS] pgsql: Fixed string in German translation that causes segfault.

2011-06-20 Thread Alvaro Herrera
Excerpts from Tom Lane's message of lun jun 20 16:44:20 -0400 2011:
 Magnus Hagander mag...@hagander.net writes:
  On Mon, Jun 20, 2011 at 22:20, Peter Eisentraut pete...@gmx.net wrote:
  A better way might be that translators simply work on a clone of the
  source repository, which is then merged (as in, git merge) at release
  time. There are some issues with that to figure out, but it sounds like
  the obviously right thing, from an interface point of view.
 
  I don't think we want to track every single translation update as
  commits in the main repository - we don't do that for non-translation
  stuff... If it's a squash-merge, that's a different thing, of
  course...
 
  Other than that, yes, keeping translations in git branches seems like
  a good interface.
 
 My recollection is that the current setup was created mainly so that
 translators wouldn't need to be given commit privileges on the main
 repo.

Yep.

 Giving them a separate repo to work in might be all right, but
 of course whoever does the merges would have to be careful to only
 accept changes made to the .po files and not anything else.

Honestly it doesn't seem all that good of an idea to me.  We would have
to merge changes from upstream PG repo into pgtranslation and the
history would look awful on the translation side.  I prefer something
similar to the current system, where the two repos are kept separate and
the files from pgtranslation are imported wholesale before each release,
without any kind of merge.  That keeps both histories clean.

Maybe we could have a way to prevent commits to the .po files that don't
come from the pgtranslation repository; probably we could have something
with Git hooks for this (so you have to set an environment variable
before being allowed the push, which wouldn't occur accidentally, or
something like that.)

(I admit I don't look that frequently into pgtranslation history,
though.)

-- 
Álvaro Herrera alvhe...@commandprompt.com
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Re: [COMMITTERS] pgsql: Fixed string in German translation that causes segfault.

2011-06-20 Thread Magnus Hagander
On Mon, Jun 20, 2011 at 22:44, Tom Lane t...@sss.pgh.pa.us wrote:
 Magnus Hagander mag...@hagander.net writes:
 On Mon, Jun 20, 2011 at 22:20, Peter Eisentraut pete...@gmx.net wrote:
 A better way might be that translators simply work on a clone of the
 source repository, which is then merged (as in, git merge) at release
 time.  There are some issues with that to figure out, but it sounds like
 the obviously right thing, from an interface point of view.

 I don't think we want to track every single translation update as
 commits in the main repository - we don't do that for non-translation
 stuff... If it's a squash-merge, that's a different thing, of
 course...

 Other than that, yes, keeping translations in git branches seems like
 a good interface.

 My recollection is that the current setup was created mainly so that
 translators wouldn't need to be given commit privileges on the main
 repo.  Giving them a separate repo to work in might be all right, but
 of course whoever does the merges would have to be careful to only
 accept changes made to the .po files and not anything else.

That should be easy enough to script - have the system automatically
merge the branches requied with --squash, and then make sure that no
non-.po files are modified.

-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


[HACKERS] Table Partitioning

2011-06-20 Thread David Fetter
Folks,

I noticed that we have some nice new speed optimizations (more
properly, de-pessimizations) for partitioned tables in 9.1.

Anybody care to look over the table partitioning stuff on the wiki and
check it for relevance?

http://wiki.postgresql.org/wiki/Table_partitioning

I think I may be able to get my employer to put some people on this...

Cheers,
David.
-- 
David Fetter da...@fetter.org http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter  XMPP: david.fet...@gmail.com
iCal: webcal://www.tripit.com/feed/ical/people/david74/tripit.ics

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] ALTER TABLE lock strength reduction patch is unsafe

2011-06-20 Thread Tom Lane
Robert Haas robertmh...@gmail.com writes:
 On Sun, Jun 19, 2011 at 5:13 PM, Simon Riggs si...@2ndquadrant.com wrote:
 We scan pg_class in two ways: to rebuild a relcache entry based on a
 relation's oid (easy fix). We also scan pg_class to resolve the name
 to oid mapping. The name to oid mapping is performed *without* a lock
 on the relation, since we don't know which relation to lock. So the
 name lookup can fail if we are in the middle of a pg_class update.
 This is an existing potential bug in Postgres unrelated to my patch.

 If this is a pre-existing bug, then it's not clear to me why we need
 to do anything about it at all right now.

Yeah.  This behavior has been there since day zero, and there have been
very few complaints about it.  But note that there's only a risk for
pg_class updates, not any other catalog, and there is exactly one kind
of failure with very predictable consequences.  The ALTER TABLE patch
has greatly expanded the scope of the issue, and that *is* a regression
compared to prior releases.

BTW, it seems to me that this issue is closely related to what Noah is
trying to fix here:
http://archives.postgresql.org/message-id/20110612191843.gf21...@tornado.leadboat.com

regards, tom lane

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] ALTER TABLE lock strength reduction patch is unsafe

2011-06-20 Thread Simon Riggs
On Mon, Jun 20, 2011 at 10:56 PM, Tom Lane t...@sss.pgh.pa.us wrote:
 Robert Haas robertmh...@gmail.com writes:
 On Sun, Jun 19, 2011 at 5:13 PM, Simon Riggs si...@2ndquadrant.com wrote:
 We scan pg_class in two ways: to rebuild a relcache entry based on a
 relation's oid (easy fix). We also scan pg_class to resolve the name
 to oid mapping. The name to oid mapping is performed *without* a lock
 on the relation, since we don't know which relation to lock. So the
 name lookup can fail if we are in the middle of a pg_class update.
 This is an existing potential bug in Postgres unrelated to my patch.

 If this is a pre-existing bug, then it's not clear to me why we need
 to do anything about it at all right now.

 Yeah.  This behavior has been there since day zero, and there have been
 very few complaints about it.  But note that there's only a risk for
 pg_class updates, not any other catalog, and there is exactly one kind
 of failure with very predictable consequences.

I agree we shouldn't do anything about the name lookups for 9.1
That is SearchCatCache using RELNAMENSP lookups, to be precise, as
well as triggers and few other similar call types.

 The ALTER TABLE patch
 has greatly expanded the scope of the issue, and that *is* a regression
 compared to prior releases.

I agree the scope for RELOID errors increased with my 9.1 patch. I'm
now happy with the locking patch (attached), which significantly
reduces the scope - back to the original error scope, in my testing.

I tried to solve both, but I think that's a step too far given the timing.

It seems likely that there will be objections to this patch. All I
would say is that issuing a stream of ALTER TABLEs against the same
table is not a common situation; if it were we would have seen more of
the pre-existing bug. ALTER TABLE command encompasses many subcommands
and we should evaluate each subcommand differently when we decide what
to do.

-- 
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services


relation_definition_lock.v3.patch
Description: Binary data

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] ALTER TABLE lock strength reduction patch is unsafe

2011-06-20 Thread Simon Riggs
On Mon, Jun 20, 2011 at 11:55 PM, Simon Riggs si...@2ndquadrant.com wrote:

 I agree we shouldn't do anything about the name lookups for 9.1
 That is SearchCatCache using RELNAMENSP lookups, to be precise, as
 well as triggers and few other similar call types.

Name lookups give ERRORs that look like this...

relation pgbench_accounts does not exist

 The ALTER TABLE patch
 has greatly expanded the scope of the issue, and that *is* a regression
 compared to prior releases.

 I agree the scope for RELOID errors increased with my 9.1 patch.

Which originally generated ERRORs like

cache lookup failed for relation 16390
or
could not open relation with OID 16390

which should now be absent from the server log.

-- 
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Fwd: Keywords in pg_hba.conf should be field-specific

2011-06-20 Thread Brendan Jurd
On 21 June 2011 06:06, Alvaro Herrera alvhe...@commandprompt.com wrote:
 Excerpts from Alvaro Herrera's message of lun jun 20 12:19:37 -0400 2011:
 Excerpts from Pavel Stehule's message of lun jun 20 11:34:25 -0400 2011:

  b) probably you can simplify a memory management using own two
  persistent memory context - and you can swap it. Then functions like
  free_hba_record, clean_hba_list, free_lines should be removed.

 Yeah, I reworked the patch with something like that over the weekend.
 Not all of it though.  I'll send an updated version shortly.

 Here it is, please let me know what you think.  I took the liberty of
 cleaning up some things that were clearly historical leftovers.


Okay, yeah, the MemoryContext approach is far more elegant than what I
had.  Boy was I ever barking up the wrong tree.

Cheers,
BJ

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Fwd: Keywords in pg_hba.conf should be field-specific

2011-06-20 Thread Alvaro Herrera
Excerpts from Brendan Jurd's message of lun jun 20 20:06:39 -0400 2011:
 On 21 June 2011 06:06, Alvaro Herrera alvhe...@commandprompt.com wrote:
  Excerpts from Alvaro Herrera's message of lun jun 20 12:19:37 -0400 2011:
  Excerpts from Pavel Stehule's message of lun jun 20 11:34:25 -0400 2011:
 
   b) probably you can simplify a memory management using own two
   persistent memory context - and you can swap it. Then functions like
   free_hba_record, clean_hba_list, free_lines should be removed.
 
  Yeah, I reworked the patch with something like that over the weekend.
  Not all of it though.  I'll send an updated version shortly.
 
  Here it is, please let me know what you think.  I took the liberty of
  cleaning up some things that were clearly historical leftovers.
 
 
 Okay, yeah, the MemoryContext approach is far more elegant than what I
 had.  Boy was I ever barking up the wrong tree.

Eh, whoever wrote the original code was barking up the same tree, so I
don't blame you for following the well-trodden path.

I realize I took out most of the fun of this patch from you, but -- are
you still planning to do some more exhaustive testing of it?  I checked
some funny scenarios (including include files and groups) but it's not
all that unlikely that I missed some cases.  I also didn't check any
auth method other than ident, md5 and trust, 'cause I don't have what's
required for anything else.  (It's a pity that the regression tests
don't exercise anything else.)

-- 
Álvaro Herrera alvhe...@commandprompt.com
The PostgreSQL Company - Command Prompt, Inc.
PostgreSQL Replication, Consulting, Custom Development, 24x7 support

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Fwd: Keywords in pg_hba.conf should be field-specific

2011-06-20 Thread Brendan Jurd
On 21 June 2011 11:11, Alvaro Herrera alvhe...@commandprompt.com wrote:
 I realize I took out most of the fun of this patch from you, but -- are
 you still planning to do some more exhaustive testing of it?  I checked
 some funny scenarios (including include files and groups) but it's not
 all that unlikely that I missed some cases.  I also didn't check any
 auth method other than ident, md5 and trust, 'cause I don't have what's
 required for anything else.  (It's a pity that the regression tests
 don't exercise anything else.)

I've pretty much tested all the things I can think to test, and I'm in
the same boat as you re auth methods.  If there are bugs, I think they
will be of the obscure kind, that only reveal themselves under
peculiar circumstances.

Cheers,
BJ

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] pika buildfarm member failure on isolationCheck tests

2011-06-20 Thread Dan Ports
While testing the fix for this one, I found another bug. Patches for
both are attached.

The first patch addresses this bug by re-adding SXACT_FLAG_ROLLED_BACK,
in a more limited form than its previous incarnation.

We need to be able to distinguish transactions that have already
called ReleasePredicateLocks and are thus eligible for cleanup from
those that have been merely marked for abort by other
backends. Transactions that are ROLLED_BACK are excluded from
SxactGlobalXmin calculations, but those that are merely DOOMED need to
be included.

Also update a couple of assertions to ensure we only try to clean up
ROLLED_BACK transactions.


The second patch fixes a bug in PreCommit_CheckForSerializationFailure.
This function checks whether there's a dangerous structure of the form
 far --- near --- me
where neither the far or near transactions have committed. If so, 
it aborts the near transaction by marking it as DOOMED. However, that
transaction might already be PREPARED. We need to check whether that's
the case and, if so, abort the transaction that's trying to commit
instead.

One of the prepared_xacts regression tests actually hits this bug.
I removed the anomaly from the duplicate-gids test so that it fails in
the intended way, and added a new test to check serialization failures
with a prepared transaction.

Dan

-- 
Dan R. K. Ports  MIT CSAILhttp://drkp.net/
diff --git a/src/backend/storage/lmgr/predicate.c b/src/backend/storage/lmgr/predicate.c
index 6c55211..3678878 100644
--- a/src/backend/storage/lmgr/predicate.c
+++ b/src/backend/storage/lmgr/predicate.c
@@ -246,7 +246,6 @@
 
 #define SxactIsCommitted(sxact) (((sxact)-flags  SXACT_FLAG_COMMITTED) != 0)
 #define SxactIsPrepared(sxact) (((sxact)-flags  SXACT_FLAG_PREPARED) != 0)
-#define SxactIsRolledBack(sxact) (((sxact)-flags  SXACT_FLAG_ROLLED_BACK) != 0)
 #define SxactIsDoomed(sxact) (((sxact)-flags  SXACT_FLAG_DOOMED) != 0)
 #define SxactIsReadOnly(sxact) (((sxact)-flags  SXACT_FLAG_READ_ONLY) != 0)
 #define SxactHasSummaryConflictIn(sxact) (((sxact)-flags  SXACT_FLAG_SUMMARY_CONFLICT_IN) != 0)
@@ -3047,7 +3046,7 @@ SetNewSxactGlobalXmin(void)
 
 	for (sxact = FirstPredXact(); sxact != NULL; sxact = NextPredXact(sxact))
 	{
-		if (!SxactIsRolledBack(sxact)
+		if (!SxactIsDoomed(sxact)
 			 !SxactIsCommitted(sxact)
 			 sxact != OldCommittedSxact)
 		{
@@ -3114,7 +3113,6 @@ ReleasePredicateLocks(const bool isCommit)
 	Assert(!isCommit || SxactIsPrepared(MySerializableXact));
 	Assert(!isCommit || !SxactIsDoomed(MySerializableXact));
 	Assert(!SxactIsCommitted(MySerializableXact));
-	Assert(!SxactIsRolledBack(MySerializableXact));
 
 	/* may not be serializable during COMMIT/ROLLBACK PREPARED */
 	if (MySerializableXact-pid != 0)
@@ -3153,22 +3151,7 @@ ReleasePredicateLocks(const bool isCommit)
 			MySerializableXact-flags |= SXACT_FLAG_READ_ONLY;
 	}
 	else
-	{
-		/*
-		 * The DOOMED flag indicates that we intend to roll back this
-		 * transaction and so it should not cause serialization
-		 * failures for other transactions that conflict with
-		 * it. Note that this flag might already be set, if another
-		 * backend marked this transaction for abort.
-		 *
-		 * The ROLLED_BACK flag further indicates that
-		 * ReleasePredicateLocks has been called, and so the
-		 * SerializableXact is eligible for cleanup. This means it
-		 * should not be considered when calculating SxactGlobalXmin.
-		 */
 		MySerializableXact-flags |= SXACT_FLAG_DOOMED;
-		MySerializableXact-flags |= SXACT_FLAG_ROLLED_BACK;
-	}
 
 	if (!topLevelIsDeclaredReadOnly)
 	{
@@ -3544,7 +3527,7 @@ ReleaseOneSerializableXact(SERIALIZABLEXACT *sxact, bool partial,
 nextConflict;
 
 	Assert(sxact != NULL);
-	Assert(SxactIsRolledBack(sxact) || SxactIsCommitted(sxact));
+	Assert(SxactIsDoomed(sxact) || SxactIsCommitted(sxact));
 	Assert(LWLockHeldByMe(SerializableFinishedListLock));
 
 	/*
diff --git a/src/include/storage/predicate_internals.h b/src/include/storage/predicate_internals.h
index 34c661d..495983f 100644
--- a/src/include/storage/predicate_internals.h
+++ b/src/include/storage/predicate_internals.h
@@ -90,22 +90,21 @@ typedef struct SERIALIZABLEXACT
 	int			pid;			/* pid of associated process */
 } SERIALIZABLEXACT;
 
-#define SXACT_FLAG_COMMITTED			0x0001	/* already committed */
-#define SXACT_FLAG_PREPARED0x0002	/* about to commit */
-#define SXACT_FLAG_ROLLED_BACK			0x0004	/* already rolled back */
-#define SXACT_FLAG_DOOMED0x0008	/* will roll back */
+#define SXACT_FLAG_COMMITTED0x0001	/* already committed */
+#define SXACT_FLAG_PREPARED	0x0002	/* about to commit */
+#define SXACT_FLAG_DOOMED	0x0004	/* will roll back */
 /*
  * The following flag actually means that the flagged transaction has a
  * conflict out *to a transaction which committed ahead of it*.  It's hard
  * to get that into a name of a reasonable length.
  */
-#define SXACT_FLAG_CONFLICT_OUT			0x0010
-#define 

Re: [HACKERS] Fwd: Keywords in pg_hba.conf should be field-specific

2011-06-20 Thread Pavel Stehule
2011/6/20 Alvaro Herrera alvhe...@commandprompt.com:
 Excerpts from Alvaro Herrera's message of lun jun 20 12:19:37 -0400 2011:
 Excerpts from Pavel Stehule's message of lun jun 20 11:34:25 -0400 2011:

  b) probably you can simplify a memory management using own two
  persistent memory context - and you can swap it. Then functions like
  free_hba_record, clean_hba_list, free_lines should be removed.

 Yeah, I reworked the patch with something like that over the weekend.
 Not all of it though.  I'll send an updated version shortly.

 Here it is, please let me know what you think.  I took the liberty of
 cleaning up some things that were clearly historical leftovers.


I have one question. I can't find any rules for work with tokens, etc,
where is quotes allowed and disallowed?

I don't see any other issues.

Regards

Pavel Stehule


 --
 Álvaro Herrera alvhe...@commandprompt.com
 The PostgreSQL Company - Command Prompt, Inc.
 PostgreSQL Replication, Consulting, Custom Development, 24x7 support


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Patch - Debug builds without optimization

2011-06-20 Thread Greg Smith

On 06/20/2011 01:34 PM, Tom Lane wrote:

I was trying to illustrate how to have minimal turnaround time
when testing a small code change.  Rebuilding from scratch is slow
enough that you lose focus while waiting.  (Or I do, anyway.)
   


I just keep upgrading to the fastest CPU I can possibly justify to avoid 
losing focus; it goes fast with 8 cores.  I was trying to demonstrate 
that peg makes this very high level now, and I was more jousting at the 
idea that everyone should bother to write their own individual reinstall 
script.


The peg code makes it easy to assimilate whatever other neat 
optimization ideas one might come across.  I just pushed an update out 
that absorbed this one, so now if you do:


stop
peg rebuild

It uses the install-bin trick you suggested.  It even does a couple of 
sanity checks so that it will probably fall back to a regular build if 
it doesn't look like you have a good install and binary tree already.  
Maybe I'll make a reinstall alias that does this combination next.


I don't expect to improve your workflow.  But people who haven't already 
invested a good chunk of work in automating things already will probably 
take some time to catch up with where peg puts them on day one.


--
Greg Smith   2ndQuadrant USg...@2ndquadrant.com   Baltimore, MD
PostgreSQL Training, Services, and 24x7 Support  www.2ndQuadrant.us



--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Fwd: Keywords in pg_hba.conf should be field-specific

2011-06-20 Thread Brendan Jurd
On 21 June 2011 13:51, Pavel Stehule pavel.steh...@gmail.com wrote:
 I have one question. I can't find any rules for work with tokens, etc,
 where is quotes allowed and disallowed?

 I don't see any other issues.

I'm not sure I understand your question, but quotes are allowed
anywhere and they always act to remove any special meaning the token
might otherwise have had.  It's much like quoting a column name in
SQL.

I didn't change any of the hba parsing rules, so escaping and whatnot
should all work the way it did before.  The only difference should be
that after parsing, keywords will only be evaluated for fields where
they matter.

Cheers,
BJ

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Fwd: Keywords in pg_hba.conf should be field-specific

2011-06-20 Thread Pavel Stehule
2011/6/21 Brendan Jurd dire...@gmail.com:
 On 21 June 2011 13:51, Pavel Stehule pavel.steh...@gmail.com wrote:
 I have one question. I can't find any rules for work with tokens, etc,
 where is quotes allowed and disallowed?

 I don't see any other issues.

 I'm not sure I understand your question, but quotes are allowed
 anywhere and they always act to remove any special meaning the token
 might otherwise have had.  It's much like quoting a column name in
 SQL.

I don't understand to using a macro

#define token_is_keyword(t, k)  (!t-quoted  strcmp(t-string, k) == 0)

because you disallowed a quoting?

Regards

Pavel


 I didn't change any of the hba parsing rules, so escaping and whatnot
 should all work the way it did before.  The only difference should be
 that after parsing, keywords will only be evaluated for fields where
 they matter.

 Cheers,
 BJ


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Fwd: Keywords in pg_hba.conf should be field-specific

2011-06-20 Thread Brendan Jurd
On 21 June 2011 14:34, Pavel Stehule pavel.steh...@gmail.com wrote:
 I don't understand to using a macro

 #define token_is_keyword(t, k)  (!t-quoted  strcmp(t-string, k) == 0)

 because you disallowed a quoting?

Well, a token can only be treated as a special keyword if it is unquoted.

As an example, in the 'database' field, the bare token 'replication'
is a keyword meaning the pseudo-database for streaming rep.  Whereas
the quoted token replication would mean a real database which is
called 'replication'.

Likewise, the bare token 'all' in the username field is a keyword --
it matches any username.  Whereas the quoted token all would only
match a user named 'all'.

Therefore, token_is_keyword only returns true where the token matches
the given string as is also unquoted.

Does that make sense?

Cheers,
BJ

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] FOR KEY LOCK foreign keys

2011-06-20 Thread Jesper Krogh

On 2011-06-20 22:11, Noah Misch wrote:

On Sun, Jun 19, 2011 at 06:30:41PM +0200, Jesper Krogh wrote:

I hope this hasn't been forgotten. But I cant see it has been committed
or moved
into the commitfest process?

If you're asking about that main patch for $SUBJECT rather than those
isolationtester changes specifically, I can't speak to the plans for it.  I
wasn't planning to move the test suite work forward independent of the core
patch it serves, but we could do that if there's another application.

Yes, I was actually asking about the main patch for foreign key locks.

Jesper
--
Jesper


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


  1   2   >