Re: [HACKERS] 9.3 Beta1 status report

2013-04-22 Thread Boszormenyi Zoltan

2013-04-21 15:10 keltezéssel, Bruce Momjian írta:

On Sun, Apr 21, 2013 at 09:34:10AM +0200, Boszormenyi Zoltan wrote:

2013-04-21 07:02 keltezéssel, Bruce Momjian írta:

I am not sure if Tom shared yet, but we are planning to package 9.3
beta1 on April 29, with a release on May 2.  Those dates might change,
but that is the current plan.  I have completed a draft 9.3 release
notes, which you can view here:

http://momjian.us/pgsql_docs/release-9-3.html

I will be working on polishing them for the next ten days, so any
feedback, patches, or commits are welcome.  I still need to add lots of
SGML markup.

How comes Álvaro's name comes out right in your page but not at
http://www.postgresql.org/docs/devel/static/release-9-3.html ?

Anyway, I attached a patch to fix my name in your page using markups.

Thanks, applied.


Thank you.


   I had not yet gotten to doing the SGML markup for
non-ASCII characters.




--
--
Zoltán Böszörményi
Cybertec Schönig  Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt, Austria
Web: http://www.postgresql-support.de
 http://www.postgresql.at/



--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


[HACKERS] Fast promotion, loose ends

2013-04-22 Thread Heikki Linnakangas
We never reached a consensus on the user interface of the new 'fast 
promotion'. We should settle that before beta. The thread died here:


http://www.postgresql.org/message-id/CA+U5nMKmDD7hGCYzOo=iFM=eK5OPDXCEzmq79fgLWr0TJk=s...@mail.gmail.com

Simon said in that email that he's waiting for further comments, so let 
me reiterate my view on this:


The current situation is that to do 'fast promotion', you have to use 
pg_ctl promote -m fast. 'slow' promotion is the default.


I didn't like that, because:

1. The slow method has no advantage over the fast method. Therefore I 
think the fast method should be the default, when promoting a standby. 
To be conservative, just in case there are bugs in the fast method, we'd 
still use the slow method for crash recovery and end of point-in-time 
recovery. That provides an escape hatch, so that if the fast method 
fails because of a bug, you can still start up the database.


2. There is no way to perform 'fast promotion' using the trigger file. 
That feature is only available using pg_ctl promote. When pg_ctl 
promote was introduced, it was not meant to replace the trigger file 
method, as that is also very useful in many situations. (as explained 
here: http://www.postgresql.org/message-id/5112a54b.8090...@vmware.com)


Putting points 1 and 2 together, I think the best solution is to always 
perform 'fast' promotion when in standby mode, and remove pg_ctl -m 
fast/smart option altogether. There is no need to expose that to users.


- Heikki


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Fast promotion, loose ends

2013-04-22 Thread Simon Riggs
On 22 April 2013 08:13, Heikki Linnakangas hlinnakan...@vmware.com wrote:
 We never reached a consensus on the user interface of the new 'fast
 promotion'. We should settle that before beta. The thread died here:

 http://www.postgresql.org/message-id/CA+U5nMKmDD7hGCYzOo=iFM=eK5OPDXCEzmq79fgLWr0TJk=s...@mail.gmail.com

 Simon said in that email that he's waiting for further comments, so let me
 reiterate my view on this:

I'm very happy to discuss this some more, thanks for bringing it up.

 The current situation is that to do 'fast promotion', you have to use
 pg_ctl promote -m fast. 'slow' promotion is the default.

 I didn't like that, because:

 1. The slow method has no advantage over the fast method. Therefore I think
 the fast method should be the default, when promoting a standby. To be
 conservative, just in case there are bugs in the fast method, we'd still use
 the slow method for crash recovery and end of point-in-time recovery. That
 provides an escape hatch, so that if the fast method fails because of a bug,
 you can still start up the database.

In general, I agree. My only advantage for slow is that we know it works.

 2. There is no way to perform 'fast promotion' using the trigger file. That
 feature is only available using pg_ctl promote. When pg_ctl promote was
 introduced, it was not meant to replace the trigger file method, as that is
 also very useful in many situations. (as explained here:
 http://www.postgresql.org/message-id/5112a54b.8090...@vmware.com)

I realise that it *is* possible to trigger fast promotion using a
file, since that is exactly the method we use to implement the feature
by pg_ctl.

In fact, pg_ctl promote is just a nice paint job over the trigger file concept.

So, to initiate promotion, you can create a file called
$DATADIR/fast_promote or $DATADIR/promote

Does that solve the issue?

 Putting points 1 and 2 together, I think the best solution is to always
 perform 'fast' promotion when in standby mode, and remove pg_ctl -m
 fast/smart option altogether. There is no need to expose that to users.

We can make pg_ctl promote write the fast_promote file by default and
remove the command line option altogether.

I would like to retain the option to promote with a full checkpoint,
for safety reasons, so continue to use both fast_promote and promote
files under the covers.

--
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Fast promotion, loose ends

2013-04-22 Thread Heikki Linnakangas

On 22.04.2013 10:58, Simon Riggs wrote:

On 22 April 2013 08:13, Heikki Linnakangashlinnakan...@vmware.com  wrote:

2. There is no way to perform 'fast promotion' using the trigger file. That
feature is only available using pg_ctl promote. When pg_ctl promote was
introduced, it was not meant to replace the trigger file method, as that is
also very useful in many situations. (as explained here:
http://www.postgresql.org/message-id/5112a54b.8090...@vmware.com)


I realise that it *is* possible to trigger fast promotion using a
file, since that is exactly the method we use to implement the feature
by pg_ctl.

In fact, pg_ctl promote is just a nice paint job over the trigger file concept.

So, to initiate promotion, you can create a file called
$DATADIR/fast_promote or $DATADIR/promote

Does that solve the issue?


Hmm. That requires write access to $DATADIR, so that's not quite the 
same thing as the trigger_file recovery.conf option. But if you really 
want to keep the 'slow' option, we could add an option to recovery.conf 
for it. I'm thinking something like this:


trigger_file = '/mnt/foo/trigger'# file to watch for
trigger_file_method = 'slow' # slow / fast, 'fast' is default.

You would have to decide which method to use when you create 
recovery.conf file, rather than when you promote, but I think that would OK.



Putting points 1 and 2 together, I think the best solution is to always
perform 'fast' promotion when in standby mode, and remove pg_ctl -m
fast/smart option altogether. There is no need to expose that to users.


We can make pg_ctl promote write the fast_promote file by default and
remove the command line option altogether.


Sounds good to me.


I would like to retain the option to promote with a full checkpoint,
for safety reasons, so continue to use both fast_promote and promote
files under the covers.


Ok. As long as 'fast' is the default and you get the same functionality 
with the trigger file method and pg_ctl promote, that works for me.


Thanks!

- Heikki


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


[HACKERS] REFRESH MATERIALIZED VIEW command in PL block hitting Assert

2013-04-22 Thread Jeevan Chalke
Hi,

I have observed that following sequence is causing server crash.

CREATE MATERIALIZED VIEW temp_class_mv AS
  SELECT * FROM pg_class
  WITH NO DATA;

CREATE OR REPLACE FUNCTION test_refresh_mv()
RETURNS int
AS $$
BEGIN
 REFRESH MATERIALIZED VIEW temp_class_mv;
 return 1;
END; $$ LANGUAGE plpgsql;

SELECT test_refresh_mv();


I had a quick look over the crash and it is hitting following Assert in
spi.c:

else if (IsA(stmt, RefreshMatViewStmt))
{
Assert(strncmp(completionTag,
   REFRESH MATERIALIZED VIEW , 23) == 0);
_SPI_current-processed = strtoul(completionTag + 23,
  NULL, 10);
}

It seems like we are missing expected value for completionTag in
ExecRefreshMatView()

Thanks

-- 
Jeevan B Chalke
Senior Software Engineer, RD
EnterpriseDB Corporation
The Enterprise PostgreSQL Company

Phone: +91 20 30589500

Website: www.enterprisedb.com
EnterpriseDB Blog: http://blogs.enterprisedb.com/
Follow us on Twitter: http://www.twitter.com/enterprisedb

This e-mail message (and any attachment) is intended for the use of the
individual or entity to whom it is addressed. This message contains
information from EnterpriseDB Corporation that may be privileged,
confidential, or exempt from disclosure under applicable law. If you are
not the intended recipient or authorized to receive this for the intended
recipient, any use, dissemination, distribution, retention, archiving, or
copying of this communication is strictly prohibited. If you have received
this e-mail in error, please notify the sender immediately by reply e-mail
and delete this message.


Re: [HACKERS] REFRESH MATERIALIZED VIEW command in PL block hitting Assert

2013-04-22 Thread Andres Freund
On 2013-04-22 18:35:04 +0530, Jeevan Chalke wrote:
 Hi,
 
 I have observed that following sequence is causing server crash.
 
 CREATE MATERIALIZED VIEW temp_class_mv AS
   SELECT * FROM pg_class
   WITH NO DATA;
 
 CREATE OR REPLACE FUNCTION test_refresh_mv()
 RETURNS int
 AS $$
 BEGIN
  REFRESH MATERIALIZED VIEW temp_class_mv;
  return 1;
 END; $$ LANGUAGE plpgsql;
 
 SELECT test_refresh_mv();
 
 
 I had a quick look over the crash and it is hitting following Assert in
 spi.c:
 
 else if (IsA(stmt, RefreshMatViewStmt))
 {
 Assert(strncmp(completionTag,
REFRESH MATERIALIZED VIEW , 23) == 0);
 _SPI_current-processed = strtoul(completionTag + 23,
   NULL, 10);
 }
 
 It seems like we are missing expected value for completionTag in
 ExecRefreshMatView()

Possibly independent from this issue, but where did that 23 come from?
ISTM we're strtoul()ing EW somenumber here.

Greetings,

Andres Freund

-- 
 Andres Freund http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] REFRESH MATERIALIZED VIEW command in PL block hitting Assert

2013-04-22 Thread Jeevan Chalke
On Mon, Apr 22, 2013 at 6:41 PM, Andres Freund and...@2ndquadrant.comwrote:

 On 2013-04-22 18:35:04 +0530, Jeevan Chalke wrote:
  Hi,
 
  I have observed that following sequence is causing server crash.
 
  CREATE MATERIALIZED VIEW temp_class_mv AS
SELECT * FROM pg_class
WITH NO DATA;
 
  CREATE OR REPLACE FUNCTION test_refresh_mv()
  RETURNS int
  AS $$
  BEGIN
   REFRESH MATERIALIZED VIEW temp_class_mv;
   return 1;
  END; $$ LANGUAGE plpgsql;
 
  SELECT test_refresh_mv();
 
 
  I had a quick look over the crash and it is hitting following Assert in
  spi.c:
 
  else if (IsA(stmt, RefreshMatViewStmt))
  {
  Assert(strncmp(completionTag,
 REFRESH MATERIALIZED VIEW , 23) ==
 0);
  _SPI_current-processed = strtoul(completionTag + 23,
NULL, 10);
  }
 
  It seems like we are missing expected value for completionTag in
  ExecRefreshMatView()

 Possibly independent from this issue, but where did that 23 come from?


23 is also bogus here.
It should be 26 i.e. length of REFRESH MATERIALIZED VIEW 

BTW, attached is the patch which works well for me, but need details review.

Thanks


 ISTM we're strtoul()ing EW somenumber here.

 Greetings,

 Andres Freund

 --
  Andres Freund http://www.2ndQuadrant.com/
  PostgreSQL Development, 24x7 Support, Training  Services




-- 
Jeevan B Chalke
Senior Software Engineer, RD
EnterpriseDB Corporation
The Enterprise PostgreSQL Company

Phone: +91 20 30589500

Website: www.enterprisedb.com
EnterpriseDB Blog: http://blogs.enterprisedb.com/
Follow us on Twitter: http://www.twitter.com/enterprisedb

This e-mail message (and any attachment) is intended for the use of the
individual or entity to whom it is addressed. This message contains
information from EnterpriseDB Corporation that may be privileged,
confidential, or exempt from disclosure under applicable law. If you are
not the intended recipient or authorized to receive this for the intended
recipient, any use, dissemination, distribution, retention, archiving, or
copying of this communication is strictly prohibited. If you have received
this e-mail in error, please notify the sender immediately by reply e-mail
and delete this message.


mv_crash.patch
Description: Binary data

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Fast promotion, loose ends

2013-04-22 Thread Shaun Thomas

On 04/22/2013 02:58 AM, Simon Riggs wrote:


So, to initiate promotion, you can create a file called
$DATADIR/fast_promote or $DATADIR/promote


Pardon my naiveté, but could it also be an option to read the method
from the promotion file?

echo slow  /my/promotion/path

That would work without any default naming scheme, and only incurs a
read on the file-handle.

--
Shaun Thomas
OptionsHouse | 141 W. Jackson Blvd. | Suite 500 | Chicago IL, 60604
312-676-8870
stho...@optionshouse.com

__

See http://www.peak6.com/email_disclaimer/ for terms and conditions related to 
this email


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] erroneous restore into pg_catalog schema

2013-04-22 Thread Robert Haas
On Wed, Apr 17, 2013 at 2:06 PM, Tom Lane t...@sss.pgh.pa.us wrote:
 I think this breaks contrib/adminpack, and perhaps other extensions.
 They'd not be hard to fix with script changes, but they'd be broken.

 In general, we would now have a situation where relocatable extensions
 could never be installed into pg_catalog.  That might be OK, but at
 least it would need to be documented.

 Also, I think we'd be pretty much hard-wiring the decision that pg_dump
 will never dump objects in pg_catalog, because its method for selecting
 the creation schema won't work in that case.  That probably is all right
 too, but we need to realize it's a consequence of this.

These are all good points.  I'm uncertain whether they are sufficient
justification for abandoning this idea and looking for another
solution, or whether we should live with them.  Any thoughts?

 As far as the code goes, OK except I strongly disapprove of removing
 the comment about temp_missing at line 3512.  The coding is not any less
 a hack in that respect for having been pushed into a subroutine.  If
 you want to rewrite the comment, fine, but failing to point out that
 something funny is going on is not a service to readers.

OK, how about something like this: Choose default creation namespace
(but note that temp_missing, if set, will trump this value).

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Enabling Checksums

2013-04-22 Thread Robert Haas
On Wed, Apr 17, 2013 at 8:21 PM, Greg Smith g...@2ndquadrant.com wrote:
 The more I read of this thread, the more unhappy I get.  It appears that
 the entire design process is being driven by micro-optimization for CPUs
 being built by Intel in 2013.

 And that's not going to get anyone past review, since all the tests I've
 been doing the last two weeks are on how fast an AMD Opteron 6234 with OS
 cache  shared_buffers can run this.  The main thing I'm still worried
 about is what happens when you have a fast machine that can move memory
 around very quickly and an in-memory workload, but it's hamstrung by the
 checksum computation--and it's not a 2013 Intel machine.

This is a good point.  However, I don't completely agree with the
conclusion that we shouldn't be worrying about any of this right now.
While I agree with Tom that it's far too late to think about any
CPU-specific optimizations for 9.3, I have a lot of concern, based on
Ants's numbers, that we've picked a checksum algorithm which is hard
to optimize for performance.  If we don't get that fixed for 9.3,
we're potentially looking at inflicting many years of serious
suffering on our user base.  If we at least get the *algorithm* right
now, we can worry about optimizing it later.  If we get it wrong,
we'll be living with the consequence of that for a really long time.

I wish that we had not scheduled beta quite so soon, as I am sure
there will be even more resistance to changing this after beta.  But
I'm having a hard time escaping the conclusion that we're on the edge
of shipping something we will later regret quite deeply.  Maybe I'm
wrong?

...Robert


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Enabling Checksums

2013-04-22 Thread Andres Freund
On 2013-04-22 11:27:25 -0400, Robert Haas wrote:
 On Wed, Apr 17, 2013 at 8:21 PM, Greg Smith g...@2ndquadrant.com wrote:
  The more I read of this thread, the more unhappy I get.  It appears that
  the entire design process is being driven by micro-optimization for CPUs
  being built by Intel in 2013.
 
  And that's not going to get anyone past review, since all the tests I've
  been doing the last two weeks are on how fast an AMD Opteron 6234 with OS
  cache  shared_buffers can run this.  The main thing I'm still worried
  about is what happens when you have a fast machine that can move memory
  around very quickly and an in-memory workload, but it's hamstrung by the
  checksum computation--and it's not a 2013 Intel machine.
 
 This is a good point.  However, I don't completely agree with the
 conclusion that we shouldn't be worrying about any of this right now.
 While I agree with Tom that it's far too late to think about any
 CPU-specific optimizations for 9.3, I have a lot of concern, based on
 Ants's numbers, that we've picked a checksum algorithm which is hard
 to optimize for performance.  If we don't get that fixed for 9.3,
 we're potentially looking at inflicting many years of serious
 suffering on our user base.  If we at least get the *algorithm* right
 now, we can worry about optimizing it later.  If we get it wrong,
 we'll be living with the consequence of that for a really long time.
 
 I wish that we had not scheduled beta quite so soon, as I am sure
 there will be even more resistance to changing this after beta.  But
 I'm having a hard time escaping the conclusion that we're on the edge
 of shipping something we will later regret quite deeply.  Maybe I'm
 wrong?

I don't see us changing away from CRCs anymore either by now. But I
think at least changing the polynom to something that
a) has higher error detection properties
b) can noticeably sped up on a a good part of the hardware pg is run on

If we are feeling really adventurous we can switch to a faster CRC
implementation, there are enough ones around and I know that at least my
proposed patch from some years ago (which is by far not the fastest that
is doable) is in production usage some places.

Greetings,

Andres Freund

-- 
 Andres Freund http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Fast promotion, loose ends

2013-04-22 Thread Simon Riggs
On 22 April 2013 09:29, Heikki Linnakangas hlinnakan...@vmware.com wrote:

 Hmm. That requires write access to $DATADIR, so that's not quite the same
 thing as the trigger_file recovery.conf option.

Well, you also (elsewhere) requested that we must keep recovery.conf
in $DATADIR, so it needs to be writable.

--
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Fast promotion, loose ends

2013-04-22 Thread Simon Riggs
On 22 April 2013 16:09, Shaun Thomas stho...@optionshouse.com wrote:
 On 04/22/2013 02:58 AM, Simon Riggs wrote:

 So, to initiate promotion, you can create a file called
 $DATADIR/fast_promote or $DATADIR/promote


 Pardon my naiveté, but could it also be an option to read the method from
 the promotion file?

 echo slow  /my/promotion/path

 That would work without any default naming scheme, and only incurs a read on
 the file-handle.

We could do that and a similar mechanism existed in pg_standby, but
was removed in 9.0.

Given where we are and the shape of the code I'd rather not do that.

--
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] event trigger API documentation?

2013-04-22 Thread Robert Haas
On Wed, Apr 17, 2013 at 3:20 PM, Dimitri Fontaine
dimi...@2ndquadrant.fr wrote:
 Please also note that the first series of patches did include the
 support code for all the core PL, but Robert didn't feel like commiting
 that and no other commiter did step up.

Of course, the chances of that would have been a lot higher had you
actually resubmitted the patches for other PLs as separate patches as
I suggested.  It's asking a bit much to suppose that some other
committer was going to go root back through the old patches, extract
the portion that applied to the PL they know something about, revise
it to match the other changes that were made subsequent to the last
patch version that incorporated those changes, and then commit it,
without so much as a CommitFest entry to point them in the right
direction.

 I'm struggling to understand how to properly solve the problem here from
 an organisation perspective. Before beta was not the good time for the
 people involved, and was not the good time for other people to get
 involved. Beta is not the good time to fix what couldn't be done before.

Event triggers are suffering from the same problem as materialized
views and CRCs: stuff that gets committed in the last CommitFest tends
to be big and buggy.  We could fix that problem by having one more
CommitFest where we only accepted small patches, but then people would
just try to force large patches into it after all, on the theory that
they were almost done in the previous CommitFest or weren't, really,
all that big (attachment: 30kB patch file).

No matter what we do, there's always going to be some pain around the
end of the development cycle.  It's painful to see work that feels
nearly done get its release date bumped by a year.  And yet it's also
painful to squeeze it in and find that there are still loose ends that
you just don't quite have time to fix.  You can phrase that problem in
a way that makes it sound like it's about project policy or committers
being jerks, but I think it's mostly just that deadlines suck.  I have
yet to work in a development organization - EnterpriseDB included -
that suffered any less agita around release deliverables than the
PostgreSQL community does.  There are always - always! - people who
want to slip the schedule to fit more into the release, people who
want to slip more into the release WITHOUT slipping the schedule (thus
upping the defect count), and people who want to stick to the schedule
at any cost (and even if it means dumping the feature personally
requested by God Himself).  And, all the intermediate positions, too.
All of those camps are as well-represented on pgsql-hackers as
anywhere else.

It is not as if any patches submitted now are going to go away.  We
will presumably have a CommitFest sometime in the next couple of
months during which whatever didn't make it into 9.3 can go into 9.4
(or maybe, as I suspect, 10.0).  Peter's complaint is legitimate, but
it's not a stop-ship issue for 9.3, and the next train will be along
shortly.  I know that you're disappointed in how much got done with
this feature for 9.3, but I think it will have more use cases than you
realize, and the next version can and will be even better.  Sure, in
some ideal world, we could have done more, but you, I, and Alvaro all
put a hell of a lot of work into this feature just to get it where it
is, and I think we should take some significant pride in making as
much progress as we did.  If you'd asked me 2 years ago when
PostgreSQL would have a feature of this type, my money would have been
on NEVER.

...Robert


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Enabling Checksums

2013-04-22 Thread Ants Aasma
On Mon, Apr 22, 2013 at 6:27 PM, Robert Haas robertmh...@gmail.com wrote:
 On Wed, Apr 17, 2013 at 8:21 PM, Greg Smith g...@2ndquadrant.com wrote:
 The more I read of this thread, the more unhappy I get.  It appears that
 the entire design process is being driven by micro-optimization for CPUs
 being built by Intel in 2013.

 And that's not going to get anyone past review, since all the tests I've
 been doing the last two weeks are on how fast an AMD Opteron 6234 with OS
 cache  shared_buffers can run this.  The main thing I'm still worried
 about is what happens when you have a fast machine that can move memory
 around very quickly and an in-memory workload, but it's hamstrung by the
 checksum computation--and it's not a 2013 Intel machine.

 This is a good point.  However, I don't completely agree with the
 conclusion that we shouldn't be worrying about any of this right now.
 While I agree with Tom that it's far too late to think about any
 CPU-specific optimizations for 9.3, I have a lot of concern, based on
 Ants's numbers, that we've picked a checksum algorithm which is hard
 to optimize for performance.  If we don't get that fixed for 9.3,
 we're potentially looking at inflicting many years of serious
 suffering on our user base.  If we at least get the *algorithm* right
 now, we can worry about optimizing it later.  If we get it wrong,
 we'll be living with the consequence of that for a really long time.

I was just now writing up a generic C based patch based on the
parallel FNV-1a + shift that we discussed with Florian with an added
round of mixing. Testing the performance in isolation indicates that:
1) it is about an order of magnitude faster than the Sarwate CRC
method used in Postgresql.
2) it is about 2x faster than fastest software based CRC method.
3) by using -msse4.1 -funroll-loops -ftree-vectorize compilation
options the performance improves 5x. (within 20% of handcoded ASM)

This leaves lingering doubts about the quality of the checksum. It's
hard if not impossible to prove absence of interesting patterns that
would trigger collisions. I do know the checksum quality is miles
ahead of the Fletcher sum originally proposed and during the last week
I haven't been able to think of a way to make the collision rate
significantly differ from CRC.

 I wish that we had not scheduled beta quite so soon, as I am sure
 there will be even more resistance to changing this after beta.  But
 I'm having a hard time escaping the conclusion that we're on the edge
 of shipping something we will later regret quite deeply.  Maybe I'm
 wrong?

Its unfortunate that this got delayed by so long. The performance side
of the argument was clear a month ago.

Regards,
Ants Aasma
-- 
Cybertec Schönig  Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt
Web: http://www.postgresql-support.de


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Enabling Checksums

2013-04-22 Thread Ants Aasma
On Mon, Apr 22, 2013 at 6:33 PM, Andres Freund and...@2ndquadrant.com wrote:
 I don't see us changing away from CRCs anymore either by now. But I
 think at least changing the polynom to something that
 a) has higher error detection properties
 b) can noticeably sped up on a a good part of the hardware pg is run on

+1 of changing the polynomial if we stick with CRC, but I think the
differences in error detection capability are mostly academic for
PostgreSQL usecase. Or does anyone have an experience with seeing
multiple random bit errors per page.

 If we are feeling really adventurous we can switch to a faster CRC
 implementation, there are enough ones around and I know that at least my
 proposed patch from some years ago (which is by far not the fastest that
 is doable) is in production usage some places.

The faster CRC implementation just use parallel lookup tables of more
bytes in parallel. Performance results from [1] show that doing 4
bytes in parallel will yield a 2.8x speedup, and 8 bytes in parallel
yields another 1.7x on top of that at the cost of using a 8kB lookup
table. And the end result is still over 3x slower than the code in the
original patch, where Greg's performance results prompted me to look
at what would have a lower overhead.

[1] http://create.stephan-brumme.com/crc32/

Regards,
Ants Aasma
-- 
Cybertec Schönig  Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt
Web: http://www.postgresql-support.de


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] 9.3 Beta1 status report

2013-04-22 Thread Alvaro Herrera
Bruce Momjian wrote:
 I am not sure if Tom shared yet, but we are planning to package 9.3
 beta1 on April 29, with a release on May 2.  Those dates might change,
 but that is the current plan.  I have completed a draft 9.3 release
 notes, which you can view here:
 
   http://momjian.us/pgsql_docs/release-9-3.html
 
 I will be working on polishing them for the next ten days, so any
 feedback, patches, or commits are welcome.  I still need to add lots of
 SGML markup.

Can you please clarify the policy on attaching people's names to items?

This item
  Have DROP OWNED only remove user-matching GRANTs on shared objects, e.g.
  databases, tablespaces (Álvaro Herrera)
was a backpatched bugfix; I don't think it should be listed.


This item
  Throw an error if expiring tuple is again updated or deleted (Kevin
  Grittner) KEEP?

not only needs to be kept, but is also a backward-incompatible change,
so I think it warrants a more verbose explanation.


This item
  Improve the ability to detect indexable prefixes in regular
  expressions (Tom Lane)
I'm not really sure about it.  Isn't it about the new pg_trgm code to
support regex indexes?  I think they either belong together, or perhaps
the one in optimizer shouldn't be listed.

This item
  Implement a generic binary heap and use it for Merge-Append operations
  (Abhijit Menon-Sen)

A generic binary heap was implemented; but merge-append was already
using their own binary heap.  So this is not a performance optimization.
I think the item should be moved down to the source code section.

There's an extra double quote here:
  Allow in-memory sorts to use their full memory allocation (Jeff Janes)


This item:
  Allow heap-only tuple updates on system tables (Andres Freund)
was a bug fix; item should be removed.


Shouldn't this one
  Add function to report the size of the GIN pending index insertion
  list (Fujii Masao)
be in the additional modules section?


In this item
  Add support to event triggers (Dimitri Fontaine, Tom Lane)
I am not sure why you list Tom.  I think Robert should be listed
instead.


In this this
  Internally store default foreign key matches (non-FULL, non-PARTIAL) as
  simple (Tom Lane)
there is something funny going on with  chars around unspecified.

-- 
Álvaro Herrerahttp://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training  Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


[HACKERS] Performance with the new security release?

2013-04-22 Thread Anne Rosset
Hi,
We are seeing some overall performance degradation in our application 
 since we installed the security release. Other commits were also done 
at the same time in the application so we don't know yet if the 
 degradation has any relationship with the security release.

 While we are digging into this, I would like to know if it is possible 
 that the release has some impact on performance. After reading this 
 It was created as a side effect of a refactoring effort to make 
 establishing new connections to a PostgreSQL server faster, and the 
 associated code more maintainable., I am thinking it is quite possible.

 Please let me know. Thanks,

 Anne


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Performance with the new security release?

2013-04-22 Thread Josh Berkus
On 04/22/2013 10:38 AM, Anne Rosset wrote:
  While we are digging into this, I would like to know if it is possible 
  that the release has some impact on performance. After reading this 
  It was created as a side effect of a refactoring effort to make 
  establishing new connections to a PostgreSQL server faster, and the 
  associated code more maintainable., I am thinking it is quite possible.

Does your application do a lot of rapidfire reconnection to PostgreSQL?
 i.e. hundreds of new connections per minute?

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Performance with the new security release?

2013-04-22 Thread Josh Berkus
On 04/22/2013 10:38 AM, Anne Rosset wrote:
  While we are digging into this, I would like to know if it is possible 
  that the release has some impact on performance. After reading this 
  It was created as a side effect of a refactoring effort to make 
  establishing new connections to a PostgreSQL server faster, and the 
  associated code more maintainable., I am thinking it is quite possible.

Does your application do a lot of rapidfire reconnection to the
database?  As in hundreds of new connections per minute?

Mind you, if it does, I strongly recommend pgbouncer ...

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Enabling Checksums

2013-04-22 Thread Josh Berkus
On 04/22/2013 09:25 AM, Ants Aasma wrote:
 This leaves lingering doubts about the quality of the checksum. It's
 hard if not impossible to prove absence of interesting patterns that
 would trigger collisions. I do know the checksum quality is miles
 ahead of the Fletcher sum originally proposed and during the last week
 I haven't been able to think of a way to make the collision rate
 significantly differ from CRC.

When we originally discussed this feature, we were potentially
discussing a checksum algo which produced collisions for 1 out of 256
pages.  That approach was considered acceptable, since it would be very
unlikely for such a collision to occur across multiple corrupted pages,
and fairly rare to have only one corrupted page.

So my perspective is, if we're doing better than 1 in 256, it's good enough.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Fast promotion, loose ends

2013-04-22 Thread Heikki Linnakangas

On 22.04.2013 18:45, Simon Riggs wrote:

On 22 April 2013 16:09, Shaun Thomasstho...@optionshouse.com  wrote:

On 04/22/2013 02:58 AM, Simon Riggs wrote:


So, to initiate promotion, you can create a file called
$DATADIR/fast_promote or $DATADIR/promote



Pardon my naiveté, but could it also be an option to read the method from
the promotion file?

echo slow  /my/promotion/path

That would work without any default naming scheme, and only incurs a read on
the file-handle.


Yeah, that would be one way to do it.

There's a little race condition if you create the file like above; 
postgres might read it just when it's created, but before the slow 
word has been written to it. But that would probably be acceptable, and 
an application could always do create and write the file and then rename 
it into place, if that's a problem.



We could do that and a similar mechanism existed in pg_standby, but
was removed in 9.0.


That code is still in pg_standby. Maybe you were thinking of some other 
feature?


- Heikki


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Enabling Checksums

2013-04-22 Thread Florian Pflug
On Apr22, 2013, at 18:25 , Ants Aasma a...@cybertec.at wrote:
 I was just now writing up a generic C based patch based on the
 parallel FNV-1a + shift that we discussed with Florian with an added
 round of mixing. Testing the performance in isolation indicates that:
 1) it is about an order of magnitude faster than the Sarwate CRC
 method used in Postgresql.
 2) it is about 2x faster than fastest software based CRC method.
 3) by using -msse4.1 -funroll-loops -ftree-vectorize compilation
 options the performance improves 5x. (within 20% of handcoded ASM)
 
 This leaves lingering doubts about the quality of the checksum. It's
 hard if not impossible to prove absence of interesting patterns that
 would trigger collisions. I do know the checksum quality is miles
 ahead of the Fletcher sum originally proposed and during the last week
 I haven't been able to think of a way to make the collision rate
 significantly differ from CRC.

Note though that CRCs may very well have similar interesting
corruption patterns which don't cause the checksum to change, though.
The only guarantee they really give is that those patterns will involve
more than N-1 flipped bits, where N is the hamming distance of the
CRC. For 16-bit checksums, N can at most be 16 (since XOR-ing the data
with a shifted version of the CRC polynomial will not cause the checksum
to change).

Thus, once more than two bytes on a page get corrupted, CRCs may not
have any advantage over fnv1+shift or similar approaches. They may even
work worse, since detecting some forms of corruption with 100% certainty
means missing others with a probability of more than 2^-16. Some CRC
polynomials for example detect all corruptions which affect an odd number
of bits, but in turn have a probability of 2^-15 of missing ones which
affect an even number of bits.

Since we're mostly attempting to protect against disk, not memory
corruption here, I'm not convinced at all that errors in only a few
bits are all that common, and certainly not that they are more likely
than other forms of corruption. I'd expect, for example, that blocks of
512 bytes (i.e. one sector) suddenly reading 0 is at least as likely
as a single flipped bit.

The one downside of the fnv1+shift approach is that it's built around
the assumption that processing 64-bytes at once is the sweet spot. That
might be true for x86 and x86_64 today, but it won't stay that way for
long, and quite surely isn't true for other architectures. That doesn't
necessarily rule it out, but it certainly weakens the argument that
slipping it into 9.3 avoids having the change the algorithm later...

best regards,
Florian Pflug



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Performance with the new security release?

2013-04-22 Thread Anne Rosset
Hi Josh,
Thanks for your reply. 
I don't think this is the case since we are using jboss/jdbc driver with a 
connection pool. 

Thanks,
Anne





-Original Message-
From: Josh Berkus [mailto:j...@agliodbs.com] 
Sent: Monday, April 22, 2013 10:58 AM
To: Anne Rosset
Cc: pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Performance with the new security release?

On 04/22/2013 10:38 AM, Anne Rosset wrote:
  While we are digging into this, I would like to know if it is 
 possible  that the release has some impact on performance. After 
 reading this  It was created as a side effect of a refactoring effort 
 to make  establishing new connections to a PostgreSQL server faster, 
 and the  associated code more maintainable., I am thinking it is quite 
 possible.

Does your application do a lot of rapidfire reconnection to the database?  As 
in hundreds of new connections per minute?

Mind you, if it does, I strongly recommend pgbouncer ...

--
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] minimizing the target list for foreign data wrappers

2013-04-22 Thread David Gudeman
Re-reading my first email I thought it was a little confusing, so here
is some clarification. In GetForeignPlan, tlist seems to be a target
list for a basic select * from the foreign table. For the ith
TargetEntry te in tlist, it seems that te-expr is a var with
varattno=i. I was mis-remembering and calling varattno attrno in the
original email.

My assumption is that the plan elements that use the output of the FDW
plan node will access columns indirectly using tlist. In other words,
I'm assuming that if there is a reference to a column c of the foreign
table, this column will be represented as a Var with varattno being an
offset into tlist. So if c is column number 3, for example, you get
its value by looking up TargetEntry number 3 in tlist and evaluate the
expr column for that TargetEntry. So if I change the  Var in the expr
column so the varattno points to a different column in the output
tuple, then everything will work.

The two risky assumptions I'm making are 1. that it actually uses this
indirect way of looking up columns in a foreign table and 2. that it
actually uses the tlist that I pass in when I call make_foreignscan().

Can anyone confirm or deny these assumptions?

Thanks.

On Sun, Apr 21, 2013 at 6:57 PM, David Gudeman dave.gude...@gmail.com wrote:

 A few years ago I wrote a roll-your-own foreign-data-wrapper system for 
 Postgres because Postgres didn't have one at the time (some details here 
 (http://unobtainabol.blogspot.com/2013/04/dave-foreign-data-introuction.html) 
 if anyone is interested). Now I'm being tasked to move it to Postgres 9.2.x 
 and I'd like to use FDW if possible.

 One of the problems I'm having is that in my application, the foreign tables 
 typically have hundreds of columns while typical queries only access a dozen 
 or so (the foreign server is a columnar SQL database). Furthermore, there is 
 no size optimization for NULL values passed back from the foreign server, so 
 if I return all of the columns from the table --even as NULLs-- the returned 
 data size will be several times the size that it needs to be. My application 
 cannot tolerate this level of inefficiency, so I need to return minimal 
 columns from the foreign table.

 The documentation doesn't say how to do this, but looking at the code I think 
 it is possible. In GetForeignPlan() you have to pass on the tlist argument, 
 which I presume means that the query plan will use the tlist that I pass in, 
 right? If so, then it should be possible for me to write a function that 
 takes tlist and baserel-reltargetlist and return a version of tlist that 
 knows which foreign-table columns are actually used, and replaces the rest 
 with a NULL constant.

 For example, suppose the original tlist is this: [VAR(attrno=1), 
 VAR(attrno=2), VAR(attrno=3)] and reltarget list says that I only need args 1 
 and 3. Then the new tlist would look like this: [VAR(attrno=1), 
 CONST(val=NULL), VAR(attrno=2)] where the attrno of the last VAR has been 
 reduced by one because the 2 column is no longer there.

 I did something very much like this in my roll-your-own version of FDW so I 
 know basically how to do it, but I did it at the pre-planning stage and I'm 
 not sure how much is already packed into the other plan nodes at this point. 
 Maybe it's too late to change the target list?

 Can anyone give me some advice or warnings on this? I'd hate to go to the 
 trouble of implementing and testing it only to find that I'm making some 
 bogus assumptions.

 Thanks,
 David Gudeman



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] 9.3 Beta1 status report

2013-04-22 Thread Bruce Momjian
On Mon, Apr 22, 2013 at 01:54:03PM -0300, Alvaro Herrera wrote:
 Bruce Momjian wrote:
  I am not sure if Tom shared yet, but we are planning to package 9.3
  beta1 on April 29, with a release on May 2.  Those dates might change,
  but that is the current plan.  I have completed a draft 9.3 release
  notes, which you can view here:
  
  http://momjian.us/pgsql_docs/release-9-3.html
  
  I will be working on polishing them for the next ten days, so any
  feedback, patches, or commits are welcome.  I still need to add lots of
  SGML markup.
 
 Can you please clarify the policy on attaching people's names to items?

Well, I just pull them from the commit message, of it no one is
mentioned in the commit message, I use the committer's name.

 This item
   Have DROP OWNED only remove user-matching GRANTs on shared objects, e.g.
   databases, tablespaces (Álvaro Herrera)
 was a backpatched bugfix; I don't think it should be listed.

OK, removed.  I now see there was a backpatch mention in the commit
messsage.

 This item
   Throw an error if expiring tuple is again updated or deleted (Kevin
   Grittner) KEEP?
 
 not only needs to be kept, but is also a backward-incompatible change,
 so I think it warrants a more verbose explanation.

OK, I don't understand myself, so I will need details.  I marked it as
backward-incompatible.

 This item
   Improve the ability to detect indexable prefixes in regular
   expressions (Tom Lane)
 I'm not really sure about it.  Isn't it about the new pg_trgm code to
 support regex indexes?  I think they either belong together, or perhaps
 the one in optimizer shouldn't be listed.

I have no idea.  I certainly see it affecting more than pg_trgm;  I see
backend regression test additions with the patch, 
628cbb50ba80c83917b07a7609ddec12cda172d0.

 This item
   Implement a generic binary heap and use it for Merge-Append operations
   (Abhijit Menon-Sen)
 
 A generic binary heap was implemented; but merge-append was already
 using their own binary heap.  So this is not a performance optimization.
 I think the item should be moved down to the source code section.

OK.


 There's an extra double quote here:
   Allow in-memory sorts to use their full memory allocation (Jeff Janes)

Fixed.
 
 This item:
   Allow heap-only tuple updates on system tables (Andres Freund)
 was a bug fix; item should be removed.

OK.

 Shouldn't this one
   Add function to report the size of the GIN pending index insertion
   list (Fujii Masao)
 be in the additional modules section?

Yes, moved.

 In this item
   Add support to event triggers (Dimitri Fontaine, Tom Lane)
 I am not sure why you list Tom.  I think Robert should be listed
 instead.

Tom did a massive fix/cleanup of that code.  I have added Robert.

 In this this
   Internally store default foreign key matches (non-FULL, non-PARTIAL) as
   simple (Tom Lane)
 there is something funny going on with  chars around unspecified.

Fixed, and applied.

Thanks!

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Fast promotion, loose ends

2013-04-22 Thread Simon Riggs
On 22 April 2013 19:04, Heikki Linnakangas hlinnakan...@vmware.com wrote:

 We could do that and a similar mechanism existed in pg_standby, but
 was removed in 9.0.


 That code is still in pg_standby. Maybe you were thinking of some other
 feature?

It wasn't removed from pg_standby. But since all the features of
pg_standby except that have been moved into the main server and/or new
extensions, I described that situation as being a feature removal.
Just a passing comment, no huge loss.

--
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


[HACKERS] libpq COPY handling

2013-04-22 Thread Robert Haas
Noah Misch pointed out something interesting to me:

/*
 * PQputCopyEnd - send EOF indication to the backend during COPY IN
 *
 * After calling this, use PQgetResult() to check command completion status.
 *
 * Returns 1 if successful, 0 if data could not be sent (only possible
 * in nonblock mode), or -1 if an error occurs.
 */

The comment alleges that 0 is a possible return value, but the only
return statements in the code for that function return literal values
of either 1 or -1.  I'm not sure whether that's a bug in the code or
the documentation.

Also, I noticed that there are a few places in fe-protocol3.c that
seem not to know about COPY-BOTH mode.  I'm not sure that any of these
are actually bugs right now given the current very limited use of
COPY-BOTH mode, but I'm wondering whether it wouldn't be better to
minimize the chance of future surprises.  Patch attached.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


libpq-copy-both.patch
Description: Binary data

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Enabling Checksums

2013-04-22 Thread Ants Aasma
On Mon, Apr 22, 2013 at 9:04 PM, Florian Pflug f...@phlo.org wrote:
 The one downside of the fnv1+shift approach is that it's built around
 the assumption that processing 64-bytes at once is the sweet spot. That
 might be true for x86 and x86_64 today, but it won't stay that way for
 long, and quite surely isn't true for other architectures. That doesn't
 necessarily rule it out, but it certainly weakens the argument that
 slipping it into 9.3 avoids having the change the algorithm later...

It's actually 128 bytes as it was tested. The ideal shape depends on
multiplication latency, multiplication throughput and amount of
registers available. Specifically BLCKSZ/mul_throughput_in_bytes needs
to be larger than BLCKSZ/(N_SUMS*sizeof(uint32))*(mul latency + 2*xor
latency). For latest Intel the values are 8192/16 = 512 and
8192/(32*4)*(5 + 2*1) = 448. 128 bytes is also 8 registers which is
the highest power of two fitting into architectural registers (16).
This means that the value chosen is indeed the sweet spot for x86
today. For future processors we can expect the multiplication width to
increase and possibly the latency too shifting the sweet spot into
higher widths. In fact, Haswell coming out later this year should have
AVX2 instructions that introduce integer ops on 256bit registers,
making the current choice already suboptimal.

All that said, having a lower width won't make the algorithm slower on
future processors, it will just leave some parallelism on the table
that could be used to make it even faster. The line in the sand needed
to be drawn somewhere, I chose the maximum comfortable width today
fearing that even that would be shot down based on code size.
Coincidentally 32 elements is also the internal parallelism that GPUs
have settled on. We could bump the width up by one notch to buy some
future safety, but after that I'm skeptical we will see any
conventional processors that would benefit from a higher width. I just
tested that the auto-vectorized version runs at basically identical
speed as GCC's inability to do good register allocation means that it
juggles values between registers and the stack one way or the other.

So to recap, I don't know of any CPUs where a lower value would be
better. Raising the width by one notch would mean better performance
on future processors, but raising it further would just bloat the size
of the inner loop without much benefit in sight.

Regards,
Ants Aasma

-- 
Cybertec Schönig  Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt
Web: http://www.postgresql-support.de


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] 9.3 Beta1 status report

2013-04-22 Thread Heikki Linnakangas

Allow tooling like pg_receivexlog to run on computers with different 
architectures (Heikki Linnakangas)


This probably should be mentioned in the backwards-compatibility 
section. Any 3rd party tools that speak the streaming replication 
protocol are affected.



E.1.3.2.1. Write-Ahead Log (WAL)

Store WAL in a continuous stream, rather than skipping the last 16MB 
segment every 4GB (Heikki Linnakangas) BACKWARD COMPATIBLE BREAK

Restructure WAL files to better handle timeline changes during recovery 
(Heikki Linnakangas)

Restructure WAL files to use a more compact storage format (Heikki 
Linnakangas)


Can you clarify which commits these came from? The first one is clear 
(dfda6eba), and I think the 3rd covers commits 20ba5ca6 and 061e7efb1. 
But what is that second entry?


- Heikki


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Enabling Checksums

2013-04-22 Thread Jeff Davis
On Mon, 2013-04-22 at 20:04 +0200, Florian Pflug wrote:
 The one downside of the fnv1+shift approach is that it's built around
 the assumption that processing 64-bytes at once is the sweet spot. That
 might be true for x86 and x86_64 today, but it won't stay that way for
 long, and quite surely isn't true for other architectures. That doesn't
 necessarily rule it out, but it certainly weakens the argument that
 slipping it into 9.3 avoids having the change the algorithm later...

I think you are setting the bar way too high. Right now, we have a slow
algorithm. According to Ants's tests, FNV-1a is much, much faster. Do
you think that it will still be such a bottleneck that we will want to
change it again later for purely performance reasons?

The only time this is likely to matter is in the situation Greg Smith
describes, where shared buffers is much smaller than memory, and the
working set of buffers is near the size of memory (in other words, a lot
of buffers moving to and from shared memory, but not much to or from
disk). And it's already significantly faster than algorithm in the
original tests (Fletcher), so it's not clear that it's still even a
serious problem.

(Also remember that checksum users already accept a WAL penalty.)

The biggest problem now is getting one of these faster algorithms (FNV
or even a faster CRC) into shape that is acceptable to
reviewers/committers. If we don't do that, we will be missing out on a
lot of potential checksum users for whom the existing CRC algorithm is
just too slow.

Regards,
Jeff Davis




-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Enabling Checksums

2013-04-22 Thread Robert Haas
On Mon, Apr 22, 2013 at 3:14 PM, Jeff Davis pg...@j-davis.com wrote:
 The biggest problem now is getting one of these faster algorithms (FNV
 or even a faster CRC) into shape that is acceptable to
 reviewers/committers. If we don't do that, we will be missing out on a
 lot of potential checksum users for whom the existing CRC algorithm is
 just too slow.

+1.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] 9.3 Beta1 status report

2013-04-22 Thread Robert Haas
On Mon, Apr 22, 2013 at 2:33 PM, Bruce Momjian br...@momjian.us wrote:
 In this item
   Add support to event triggers (Dimitri Fontaine, Tom Lane)
 I am not sure why you list Tom.  I think Robert should be listed
 instead.

 Tom did a massive fix/cleanup of that code.  I have added Robert.

I do not think that is true.  To what commit IDs are you referring?  I
think this item should credit Dimitri, myself, and Alvaro, probably in
that order.  The only commit by Tom to event_triggers.c is
cd3413ec3683918c9cb9cfb39ae5b2c32f231e8b, which is hardly a massive
fix/cleanup.

--
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Enabling Checksums

2013-04-22 Thread Jeff Davis
On Mon, 2013-04-22 at 19:25 +0300, Ants Aasma wrote:
 I was just now writing up a generic C based patch based on the
 parallel FNV-1a + shift that we discussed with Florian with an added
 round of mixing. Testing the performance in isolation indicates that:
 1) it is about an order of magnitude faster than the Sarwate CRC
 method used in Postgresql.
 2) it is about 2x faster than fastest software based CRC method.
 3) by using -msse4.1 -funroll-loops -ftree-vectorize compilation
 options the performance improves 5x. (within 20% of handcoded ASM)

That's great news!

This means that we can have a simple C implementation in a separate
file, and pass a few build flags when compiling just that file (so it
doesn't affect other code). That should make reviewers/committers happy
(including me).

FWIW, that was my last real concern about FNV (reviewability). I'm not
worried about the performance based on your analysis; nor am I worried
about the error detection rate.

Regards,
Jeff Davis




-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Enabling Checksums

2013-04-22 Thread Florian Pflug
On Apr22, 2013, at 21:14 , Jeff Davis pg...@j-davis.com wrote:
 On Mon, 2013-04-22 at 20:04 +0200, Florian Pflug wrote:
 The one downside of the fnv1+shift approach is that it's built around
 the assumption that processing 64-bytes at once is the sweet spot. That
 might be true for x86 and x86_64 today, but it won't stay that way for
 long, and quite surely isn't true for other architectures. That doesn't
 necessarily rule it out, but it certainly weakens the argument that
 slipping it into 9.3 avoids having the change the algorithm later...
 
 I think you are setting the bar way too high. Right now, we have a slow
 algorithm. According to Ants's tests, FNV-1a is much, much faster. Do
 you think that it will still be such a bottleneck that we will want to
 change it again later for purely performance reasons?

To clarify, it wasn't my intent to argue against shipping FNV1+SHIFT
in 9.3 - in fact I'd like to see us do exactly that. I was merely trying
to be unbiased, and hence stated not only arguments in favour or FNV1+SHIFT
(the ones about CRCs theoretical advantages in error detection being 
not really relevant to us), but also the one downside of FNV1+SHIFT.

Seems like I could have done a better job expressing myself though.

 The biggest problem now is getting one of these faster algorithms (FNV
 or even a faster CRC) into shape that is acceptable to
 reviewers/committers. If we don't do that, we will be missing out on a
 lot of potential checksum users for whom the existing CRC algorithm is
 just too slow.

Assuming that we only ship a plain C implementation with 9.3, what
are we missing on that front? The C implementation of FNV1+SHIFT is
only a few dozen lines or so.

best regards,
Florian Pflug



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] 9.3 Beta1 status report

2013-04-22 Thread Alvaro Herrera
Bruce Momjian wrote:
 On Mon, Apr 22, 2013 at 01:54:03PM -0300, Alvaro Herrera wrote:
  Bruce Momjian wrote:
   I am not sure if Tom shared yet, but we are planning to package 9.3
   beta1 on April 29, with a release on May 2.  Those dates might change,
   but that is the current plan.  I have completed a draft 9.3 release
   notes, which you can view here:
   
 http://momjian.us/pgsql_docs/release-9-3.html
   
   I will be working on polishing them for the next ten days, so any
   feedback, patches, or commits are welcome.  I still need to add lots of
   SGML markup.
  
  Can you please clarify the policy on attaching people's names to items?
 
 Well, I just pull them from the commit message, of it no one is
 mentioned in the commit message, I use the committer's name.

Hm, I listed code authors in roughly chronological order in 0ac5ad51
(fklocks).  I think that patch should list me, Noah, Andres, Alex,
Marti.

  This item
Improve the ability to detect indexable prefixes in regular
expressions (Tom Lane)
  I'm not really sure about it.  Isn't it about the new pg_trgm code to
  support regex indexes?  I think they either belong together, or perhaps
  the one in optimizer shouldn't be listed.
 
 I have no idea.  I certainly see it affecting more than pg_trgm;  I see
 backend regression test additions with the patch, 
 628cbb50ba80c83917b07a7609ddec12cda172d0.

Ah, yeah, it's unrelated to pg_trgm indexing.  It's a (backpatched) bug
fix, though.

-- 
Álvaro Herrerahttp://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training  Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Enabling Checksums

2013-04-22 Thread Simon Riggs
On 22 April 2013 20:32, Florian Pflug f...@phlo.org wrote:

 Assuming that we only ship a plain C implementation with 9.3, what
 are we missing on that front? The C implementation of FNV1+SHIFT is
 only a few dozen lines or so.

Forgive me, I can't seem to locate the patch for this? Re-post please,
just for clarity.

--
 Simon Riggs   http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] 9.3 Beta1 status report

2013-04-22 Thread Bruce Momjian
On Mon, Apr 22, 2013 at 03:19:36PM -0400, Robert Haas wrote:
 On Mon, Apr 22, 2013 at 2:33 PM, Bruce Momjian br...@momjian.us wrote:
  In this item
Add support to event triggers (Dimitri Fontaine, Tom Lane)
  I am not sure why you list Tom.  I think Robert should be listed
  instead.
 
  Tom did a massive fix/cleanup of that code.  I have added Robert.
 
 I do not think that is true.  To what commit IDs are you referring?  I
 think this item should credit Dimitri, myself, and Alvaro, probably in
 that order.  The only commit by Tom to event_triggers.c is
 cd3413ec3683918c9cb9cfb39ae5b2c32f231e8b, which is hardly a massive
 fix/cleanup.

OK, changed.

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Enabling Checksums

2013-04-22 Thread Ants Aasma
On Mon, Apr 22, 2013 at 10:54 PM, Simon Riggs si...@2ndquadrant.com wrote:
 On 22 April 2013 20:32, Florian Pflug f...@phlo.org wrote:

 Assuming that we only ship a plain C implementation with 9.3, what
 are we missing on that front? The C implementation of FNV1+SHIFT is
 only a few dozen lines or so.

 Forgive me, I can't seem to locate the patch for this? Re-post please,
 just for clarity.

Not posted yet. I'm writing it as we speak. Will post within half an hour or so.

Regards,
Ants Aasma
-- 
Cybertec Schönig  Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt
Web: http://www.postgresql-support.de


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Performance with the new security release?

2013-04-22 Thread Steve Singer

On 13-04-22 01:38 PM, Anne Rosset wrote:

Hi,
We are seeing some overall performance degradation in our application
  since we installed the security release. Other commits were also done
at the same time in the application so we don't know yet if the
  degradation has any relationship with the security release.

  While we are digging into this, I would like to know if it is possible
  that the release has some impact on performance. After reading this
  It was created as a side effect of a refactoring effort to make
  establishing new connections to a PostgreSQL server faster, and the
  associated code more maintainable., I am thinking it is quite possible.

  Please let me know. Thanks,


Exactly which version of PostgreSQL are you running? (we released 
security update releases for multiple PG versions).  Also which version 
were you running before?


There were some changes to analyze/vacuum in the previous set of minor 
releases that could cause performance issues in some cases (ie if 
statistics are no longer being updated because analyze can't get the 
exclusive lock for truncation).   There might be other unintended 
performance related changes.


Are all queries taking longer or only some?  Can you find any sort of 
pattern that might help narrow the issue?


Steve


  Anne






--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] 9.3 Beta1 status report

2013-04-22 Thread Bruce Momjian
On Mon, Apr 22, 2013 at 04:48:58PM -0300, Alvaro Herrera wrote:
 Bruce Momjian wrote:
  On Mon, Apr 22, 2013 at 01:54:03PM -0300, Alvaro Herrera wrote:
   Bruce Momjian wrote:
I am not sure if Tom shared yet, but we are planning to package 9.3
beta1 on April 29, with a release on May 2.  Those dates might change,
but that is the current plan.  I have completed a draft 9.3 release
notes, which you can view here:

http://momjian.us/pgsql_docs/release-9-3.html

I will be working on polishing them for the next ten days, so any
feedback, patches, or commits are welcome.  I still need to add lots of
SGML markup.
   
   Can you please clarify the policy on attaching people's names to items?
  
  Well, I just pull them from the commit message, of it no one is
  mentioned in the commit message, I use the committer's name.
 
 Hm, I listed code authors in roughly chronological order in 0ac5ad51
 (fklocks).  I think that patch should list me, Noah, Andres, Alex,
 Marti.

OK, I have now listed them in the order you specified.

   This item
 Improve the ability to detect indexable prefixes in regular
 expressions (Tom Lane)
   I'm not really sure about it.  Isn't it about the new pg_trgm code to
   support regex indexes?  I think they either belong together, or perhaps
   the one in optimizer shouldn't be listed.
  
  I have no idea.  I certainly see it affecting more than pg_trgm;  I see
  backend regression test additions with the patch, 
  628cbb50ba80c83917b07a7609ddec12cda172d0.
 
 Ah, yeah, it's unrelated to pg_trgm indexing.  It's a (backpatched) bug
 fix, though.

OK, removed.

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] 9.3 Beta1 status report

2013-04-22 Thread Bruce Momjian
On Mon, Apr 22, 2013 at 10:11:48PM +0300, Heikki Linnakangas wrote:
 Allow tooling like pg_receivexlog to run on computers with different 
 architectures (Heikki Linnakangas)
 
 This probably should be mentioned in the backwards-compatibility
 section. Any 3rd party tools that speak the streaming replication
 protocol are affected.
 
 E.1.3.2.1. Write-Ahead Log (WAL)
 
 Store WAL in a continuous stream, rather than skipping the last 16MB 
  segment every 4GB (Heikki Linnakangas) BACKWARD COMPATIBLE BREAK
 
 Restructure WAL files to better handle timeline changes during recovery 
  (Heikki Linnakangas)
 
 Restructure WAL files to use a more compact storage format (Heikki 
  Linnakangas)
 
 Can you clarify which commits these came from? The first one is
 clear (dfda6eba), and I think the 3rd covers commits 20ba5ca6 and
 061e7efb1. But what is that second entry?

Frankly, I found the WAL and timeline commits all over the place and
could hardly make sense of it.  I tried to collapse entries into
meaningful items, but I need help.  Can you suggest changes?

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Performance with the new security release?

2013-04-22 Thread Anne Rosset
Hi Steve,
Thanks for your reply.
We are now running  9.0.13. Before it was 9.0.7.
How can I find out if we are running into this issue: ie if statistics are no 
longer being updated because analyze can't get the 
exclusive lock for truncation?

I will dig into our logs to see for the query times.
Thanks,
Anne

-Original Message-
From: Steve Singer [mailto:st...@ssinger.info] 
Sent: Monday, April 22, 2013 12:59 PM
To: Anne Rosset
Cc: pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Performance with the new security release?

On 13-04-22 01:38 PM, Anne Rosset wrote:
 Hi,
 We are seeing some overall performance degradation in our application
   since we installed the security release. Other commits were also 
 done at the same time in the application so we don't know yet if the
   degradation has any relationship with the security release.

   While we are digging into this, I would like to know if it is possible
   that the release has some impact on performance. After reading this
   It was created as a side effect of a refactoring effort to make
   establishing new connections to a PostgreSQL server faster, and the
   associated code more maintainable., I am thinking it is quite possible.

   Please let me know. Thanks,

Exactly which version of PostgreSQL are you running? (we released security 
update releases for multiple PG versions).  Also which version were you running 
before?

There were some changes to analyze/vacuum in the previous set of minor releases 
that could cause performance issues in some cases (ie if statistics are no 
longer being updated because analyze can't get the 
exclusive lock for truncation).   There might be other unintended 
performance related changes.

Are all queries taking longer or only some?  Can you find any sort of pattern 
that might help narrow the issue?

Steve

   Anne





-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Performance with the new security release?

2013-04-22 Thread Steve Singer

On 13-04-22 04:15 PM, Anne Rosset wrote:

Hi Steve,
Thanks for your reply.
We are now running  9.0.13. Before it was 9.0.7.
How can I find out if we are running into this issue: ie if statistics are no 
longer being updated because analyze can't get the
exclusive lock for truncation?


This issue is discussed in the thread 
http://www.postgresql.org/message-id/CAMkU=1xYXOJp=jLAASPdSAqab-HwhA_tnRhy+JUe=4=b=v3...@mail.gmail.com


If your seeing messages in your logs of the form:

automatic vacuum of table XXX.YYY cannot (re)acquire exclusive lock for 
truncate scan


then you might be hitting this issue.



I will dig into our logs to see for the query times.
Thanks,
Anne

-Original Message-
From: Steve Singer [mailto:st...@ssinger.info]
Sent: Monday, April 22, 2013 12:59 PM
To: Anne Rosset
Cc: pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Performance with the new security release?

On 13-04-22 01:38 PM, Anne Rosset wrote:

Hi,
We are seeing some overall performance degradation in our application
   since we installed the security release. Other commits were also
done at the same time in the application so we don't know yet if the
   degradation has any relationship with the security release.

   While we are digging into this, I would like to know if it is possible
   that the release has some impact on performance. After reading this
   It was created as a side effect of a refactoring effort to make
   establishing new connections to a PostgreSQL server faster, and the
   associated code more maintainable., I am thinking it is quite possible.

   Please let me know. Thanks,

Exactly which version of PostgreSQL are you running? (we released security 
update releases for multiple PG versions).  Also which version were you running 
before?

There were some changes to analyze/vacuum in the previous set of minor releases 
that could cause performance issues in some cases (ie if statistics are no 
longer being updated because analyze can't get the
exclusive lock for truncation).   There might be other unintended
performance related changes.

Are all queries taking longer or only some?  Can you find any sort of pattern 
that might help narrow the issue?

Steve


   Anne









--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Performance with the new security release?

2013-04-22 Thread Anne Rosset
Hi Steve,
Yes I see these messages in our log. Is there a solution to this? 
Thanks,
Anne
-Original Message-
From: Steve Singer [mailto:st...@ssinger.info] 
Sent: Monday, April 22, 2013 1:26 PM
To: Anne Rosset
Cc: pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Performance with the new security release?

On 13-04-22 04:15 PM, Anne Rosset wrote:
 Hi Steve,
 Thanks for your reply.
 We are now running  9.0.13. Before it was 9.0.7.
 How can I find out if we are running into this issue: ie if 
 statistics are no longer being updated because analyze can't get the 
 exclusive lock for truncation?

This issue is discussed in the thread
http://www.postgresql.org/message-id/CAMkU=1xYXOJp=jLAASPdSAqab-HwhA_tnRhy+JUe=4=b=v3...@mail.gmail.com

If your seeing messages in your logs of the form:

automatic vacuum of table XXX.YYY cannot (re)acquire exclusive lock for 
truncate scan

then you might be hitting this issue.


 I will dig into our logs to see for the query times.
 Thanks,
 Anne

 -Original Message-
 From: Steve Singer [mailto:st...@ssinger.info]
 Sent: Monday, April 22, 2013 12:59 PM
 To: Anne Rosset
 Cc: pgsql-hackers@postgresql.org
 Subject: Re: [HACKERS] Performance with the new security release?

 On 13-04-22 01:38 PM, Anne Rosset wrote:
 Hi,
 We are seeing some overall performance degradation in our application
since we installed the security release. Other commits were also 
 done at the same time in the application so we don't know yet if the
degradation has any relationship with the security release.

While we are digging into this, I would like to know if it is possible
that the release has some impact on performance. After reading this
It was created as a side effect of a refactoring effort to make
establishing new connections to a PostgreSQL server faster, and the
associated code more maintainable., I am thinking it is quite possible.

Please let me know. Thanks,
 Exactly which version of PostgreSQL are you running? (we released security 
 update releases for multiple PG versions).  Also which version were you 
 running before?

 There were some changes to analyze/vacuum in the previous set of minor 
 releases that could cause performance issues in some cases (ie if statistics 
 are no longer being updated because analyze can't get the
 exclusive lock for truncation).   There might be other unintended
 performance related changes.

 Are all queries taking longer or only some?  Can you find any sort of pattern 
 that might help narrow the issue?

 Steve

Anne







-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] 9.3 Beta1 status report

2013-04-22 Thread Alvaro Herrera

Some more diacritics ..

-- 
Álvaro Herrerahttp://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training  Services
diff --git a/doc/src/sgml/release-9.3.sgml b/doc/src/sgml/release-9.3.sgml
index 7c46bd3..68d04a7 100644
--- a/doc/src/sgml/release-9.3.sgml
+++ b/doc/src/sgml/release-9.3.sgml
@@ -336,7 +336,7 @@
   listitem
para
 Allow the postmaster to listen on multiple Unix-domain sockets
-(Honza Horak)
+(Honza Horaacute;k)
/para
 
para
@@ -1064,7 +1064,7 @@
   listitem
para
 Add libpq function PQconninfo() to return connection information
-(Zoltan Boszormenyi, Magnus Hagander)
+(Zoltaacute;n Bouml;szouml;rmeacute;nyi, Magnus Hagander)
/para
   /listitem
 

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


[HACKERS] GSOC Student Project Idea

2013-04-22 Thread Michael Schuh
Greetings,

Hello, my name is Michael Schuh and I am a PhD student in Computer Science
at Montana State University. I have never participated in GSOC before, but
I am very excited to propose a project to PostgreSQL that I feel would be a
great follow-up to last year's project by Alexander Korotkov (
http://www.google-melange.com/gsoc/project/google/gsoc2012/akorotkov/53002).
I contacted Mr. Korotkov's mentor from last year, Mr. Heikki Linnakangas,
and he suggested I email this mailing list with my idea.

In brief, I would like to implement a state-of-the-art indexing algorithm
(named iDistance) directly in PostgreSQL using GiST or SP-GiST trees and
whatever means necessary. It is an ideal follow-up to last year's project
with Mr. Korotkov, which implemented classical indexing structures for
range queries. I strongly believe the community would greatly benefit from
the inclusion of iDistance, which has been shown to be dramatically more
effective than R-trees and KD-trees, especially for knn queries and above
10-20 dimensions.

A major focus of my current PhD thesis is high-dimensional data indexing
and retrieval, with an emphasis towards applied use in CBIR systems.
Recently, I published work which introduced a new open source
implementation of iDistance in C++ (and some Python), which I believe makes
me highly qualified and motivated for this opportunity. I have been
strongly considering a PostgreSQL implementation for an easy plug-and-play
use in existing applications, but with academic grant funding, the priority
is low. Below are links to my google code repository and recent
publication. I am happy to discuss any of this in further detail if you'd
like.

https://code.google.com/p/idistance/
http://www.cs.montana.edu/~timothy.wylie/files/bncod13.pdf

Although I do not have a lot of experience with PostgreSQL development, I
am eager to learn and commit my summer to enabling another fantastic
feature for the community. Since iDistance is a non-recursive, data-driven,
space-based partitioning strategy which builds directly onto a B+-tree, I
believe the implementation should be possible using only GiST support.
Please let me know if this is of any interest, or if you have any
additional questions. Unfortunately, I will be unavailable most of the day,
but I plan to fill out the GSOC application later this evening.

Thank you for your time,
Mike Schuh


Re: [HACKERS] 9.3 Beta1 status report

2013-04-22 Thread Bruce Momjian
On Mon, Apr 22, 2013 at 05:53:43PM -0300, Alvaro Herrera wrote:
 
 Some more diacritics ..

Thanks, applied.

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + It's impossible for everything to be true. +


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Enabling Checksums

2013-04-22 Thread Ants Aasma
On Mon, Apr 22, 2013 at 10:57 PM, Ants Aasma a...@cybertec.at wrote:
 On Mon, Apr 22, 2013 at 10:54 PM, Simon Riggs si...@2ndquadrant.com wrote:
 On 22 April 2013 20:32, Florian Pflug f...@phlo.org wrote:

 Assuming that we only ship a plain C implementation with 9.3, what
 are we missing on that front? The C implementation of FNV1+SHIFT is
 only a few dozen lines or so.

 Forgive me, I can't seem to locate the patch for this? Re-post please,
 just for clarity.

 Not posted yet. I'm writing it as we speak. Will post within half an hour or 
 so.

A slight delay, but here it is. I didn't lift the checksum part into a
separate file as I didn't have a great idea what I would call it. The
code is reasonably compact so I don't see a great need for this right
now. It would be more worth the effort when/if we add non-generic
variants. I'm not particularly attached to the method I used to mask
out pd_checksum field, this could be improved if someone has a better
idea how to structure the code.

I confirmed with objdump that compiling on GCC 4.7 with -msse4.1
-funroll-loops -ftree-vectorize does in fact vectorize that loop.
Simple way to verify: objdump -d src/backend/storage/page/bufpage.o |
grep pmulld | wc -l should output 16.

Unfortunately I can't work on this patch for about a week. Postgresql
9.3 will have to wait for me as I need to tend to the release of Ants
v2.0.

Regards,
Ants Aasma
-- 
Cybertec Schönig  Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt
Web: http://www.postgresql-support.de


parallel-fnv-checksum.patch
Description: Binary data

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] high io BUT huge amount of free memory

2013-04-22 Thread Merlin Moncure
On Mon, Apr 22, 2013 at 1:22 PM, Миша Тюрин tmih...@bk.ru wrote:

 My first message has been banned for too many latters.


 Hi all
 There is something wrong and ugly.

 1)
 Intel 32 core = 2*8 *2threads

 Linux avi-sql09 2.6.32-5-amd64 #1 SMP Sun May 6 04:00:17 UTC 2012 x86_64 
 GNU/Linux

 PostgreSQL 9.2.2 on x86_64-unknown-linux-gnu, compiled by gcc-4.4.real 
 (Debian 4.4.5-8) 4.4.5, 64-bit
 shared_buffers 64GB / constant hit rate - 99,18
 max_connections 160 / with pgbouncer pools there could not be more than 120 
 connections at all
 work_mem 32M
 checkpoint 1h 1.0
 swap off
 numa off,  interleaving on

 24*128GB HDD (RAID10) with 2GB bbu (1,5w+0,5r)

 2)
 free -g
  total   used   free sharedbuffers cached
 Mem:   378250128  0  0229
 -/+ buffers/cache: 20357

 and
 ! disks usage 100%  (free 128GB! WHY?)

 disk throughput - up-to 30MB/s (24r+6w)
 io - up-to 2,5-3K/s (0,5w + 2-2,5r)

 3) so maybe I've got something like this
 http://www.databasesoup.com/2012/04/red-hat-kernel-cache-clearing-issue.html
 or this
 http://comments.gmane.org/gmane.comp.db.sqlite.general/79457

 4) now i think
 a) upgrade linux core or
 b) set buffers to something like 300-320Gb
 my warm work set is about 300-400GB
 db at all - 700GB

 typical work load - pk-index-scans

 --
 looking forward
 thanks

this topic is more suitable for -performance.

check out this:

http://frosty-postgres.blogspot.com/2012/08/postgresql-numa-and-zone-reclaim-mode.html

merlin


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Performance with the new security release?

2013-04-22 Thread Steve Singer

On 13-04-22 04:41 PM, Anne Rosset wrote:

Hi Steve,
Yes I see these messages in our log. Is there a solution to this?
Thanks,
Anne


A manual analyze of the effected tables should work and give you updated 
statistics.  If your problem is just statistics then that should help.
A manual vacuum will , unfortunately, behave like the auto-vacuum. The 
only way to get vacuum past this (until this issue is fixed) is for 
vacuum to be able to get that exclusive lock.   If there are times of 
the day your database is less busy you might have some luck turning off 
auto-vacuum on these tables  and doing manual vacuums during those times.






-Original Message-
From: Steve Singer [mailto:st...@ssinger.info]
Sent: Monday, April 22, 2013 1:26 PM
To: Anne Rosset
Cc: pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Performance with the new security release?

On 13-04-22 04:15 PM, Anne Rosset wrote:

Hi Steve,
Thanks for your reply.
We are now running  9.0.13. Before it was 9.0.7.
How can I find out if we are running into this issue: ie if
statistics are no longer being updated because analyze can't get the exclusive lock 
for truncation?

This issue is discussed in the thread
http://www.postgresql.org/message-id/CAMkU=1xYXOJp=jLAASPdSAqab-HwhA_tnRhy+JUe=4=b=v3...@mail.gmail.com

If your seeing messages in your logs of the form:

automatic vacuum of table XXX.YYY cannot (re)acquire exclusive lock for truncate 
scan

then you might be hitting this issue.



I will dig into our logs to see for the query times.
Thanks,
Anne

-Original Message-
From: Steve Singer [mailto:st...@ssinger.info]
Sent: Monday, April 22, 2013 12:59 PM
To: Anne Rosset
Cc: pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Performance with the new security release?

On 13-04-22 01:38 PM, Anne Rosset wrote:

Hi,
We are seeing some overall performance degradation in our application
since we installed the security release. Other commits were also
done at the same time in the application so we don't know yet if the
degradation has any relationship with the security release.

While we are digging into this, I would like to know if it is possible
that the release has some impact on performance. After reading this
It was created as a side effect of a refactoring effort to make
establishing new connections to a PostgreSQL server faster, and the
associated code more maintainable., I am thinking it is quite possible.

Please let me know. Thanks,

Exactly which version of PostgreSQL are you running? (we released security 
update releases for multiple PG versions).  Also which version were you running 
before?

There were some changes to analyze/vacuum in the previous set of minor releases 
that could cause performance issues in some cases (ie if statistics are no 
longer being updated because analyze can't get the
exclusive lock for truncation).   There might be other unintended
performance related changes.

Are all queries taking longer or only some?  Can you find any sort of pattern 
that might help narrow the issue?

Steve


Anne











--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] high io BUT huge amount of free memory

2013-04-22 Thread Sergey Konoplev
On Mon, Apr 22, 2013 at 11:22 AM, Миша Тюрин tmih...@bk.ru wrote:
 free -g
  total   used   free sharedbuffers cached
 Mem:   378250128  0  0229
 -/+ buffers/cache: 20357

 and
 ! disks usage 100%  (free 128GB! WHY?)

 disk throughput - up-to 30MB/s (24r+6w)
 io - up-to 2,5-3K/s (0,5w + 2-2,5r)

What do iostat -xk 10 and vmstat -SM 10 show?



 3) so maybe I've got something like this
 http://www.databasesoup.com/2012/04/red-hat-kernel-cache-clearing-issue.html
 or this
 http://comments.gmane.org/gmane.comp.db.sqlite.general/79457

 4) now i think
 a) upgrade linux core or
 b) set buffers to something like 300-320Gb
 my warm work set is about 300-400GB
 db at all - 700GB

 typical work load - pk-index-scans

 --
 looking forward
 thanks

 Mikhail


 --
 Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
 To make changes to your subscription:
 http://www.postgresql.org/mailpref/pgsql-hackers




--
Kind regards,
Sergey Konoplev
Database and Software Consultant

Profile: http://www.linkedin.com/in/grayhemp
Phone: USA +1 (415) 867-9984, Russia +7 (901) 903-0499, +7 (988) 888-1979
Skype: gray-hemp
Jabber: gray...@gmail.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Enabling Checksums

2013-04-22 Thread Jeff Davis
On Tue, 2013-04-23 at 01:08 +0300, Ants Aasma wrote:
 A slight delay, but here it is. I didn't lift the checksum part into a
 separate file as I didn't have a great idea what I would call it. The
 code is reasonably compact so I don't see a great need for this right
 now. It would be more worth the effort when/if we add non-generic
 variants. I'm not particularly attached to the method I used to mask
 out pd_checksum field, this could be improved if someone has a better
 idea how to structure the code.

Thank you. A few initial comments:

I have attached (for illustration purposes only) a patch on top of yours
that divides the responsibilities a little more cleanly.

* easier to move into a separate file, and use your recommended compiler
flags without affecting other routines in bufpage.c
* makes the checksum algorithm itself simpler
* leaves the data-page-specific aspects (mixing in the page number,
ignoring pd_checksum, reducing to 16 bits) to PageCalcChecksum16
* overall easier to review and understand

I'm not sure what we should call the separate file or where we should
put it, though. How about src/backend/utils/checksum/checksum_fnv.c? Is
there a clean way to override the compiler flags for a single file so we
don't need to put it in its own directory?

Regards,
Jeff Davis

*** a/src/backend/storage/page/bufpage.c
--- b/src/backend/storage/page/bufpage.c
***
*** 23,28  static char pageCopyData[BLCKSZ];	/* for checksum calculation */
--- 23,29 
  static Page pageCopy = pageCopyData;
  
  static uint16 PageCalcChecksum16(Page page, BlockNumber blkno);
+ static uint32 checksum_fnv(char *data, uint32 size);
  
  /* 
   *		Page support functions
***
*** 986,1017  static const uint32 checksumBaseOffsets[N_SUMS] = {
  static uint16
  PageCalcChecksum16(Page page, BlockNumber blkno)
  {
! 	uint32 sums[N_SUMS];
! 	uint32 (*pageArr)[N_SUMS] = (uint32 (*)[N_SUMS]) page;
! 	uint32 result = blkno;
! 	int i, j;
! 	int pd_checksum_word = offsetof(PageHeaderData, pd_checksum)/sizeof(uint32);
! 	int pd_checksum_half = (offsetof(PageHeaderData, pd_checksum) % sizeof(uint32)) / sizeof(uint16);
  
  	/* only calculate the checksum for properly-initialized pages */
  	Assert(!PageIsNew(page));
  
  	/* initialize partial checksums to their corresponding offsets */
  	memcpy(sums, checksumBaseOffsets, sizeof(checksumBaseOffsets));
  
! 	/* first iteration needs to mask out pd_checksum field itself with zero */
! 	for (j = 0; j  N_SUMS; j++)
! 	{
! 		uint32 value = pageArr[0][j];
! 		if (j == pd_checksum_word)
! 			((uint16*) value)[pd_checksum_half] = 0;
! 		CHECKSUM_COMP(sums[j], value);
! 	}
! 
! 	/* now add in the rest of the page */
! 	for (i = 1; i  BLCKSZ/sizeof(uint32)/N_SUMS; i++)
  		for (j = 0; j  N_SUMS; j++)
! 			CHECKSUM_COMP(sums[j], pageArr[i][j]);
  
  	/* finally add in one round of zeroes for one more layer of mixing */
  	for (j = 0; j  N_SUMS; j++)
--- 987,1035 
  static uint16
  PageCalcChecksum16(Page page, BlockNumber blkno)
  {
! 	PageHeader	phdr   = (PageHeader) page;
! 	uint16		save_checksum;
! 	uint32		fnv_checksum;
  
  	/* only calculate the checksum for properly-initialized pages */
  	Assert(!PageIsNew(page));
  
+ 	/*
+ 	 * Save pd_checksum and set it to zero, so that the checksum calculation
+ 	 * isn't affected by the checksum stored on the page.
+ 	 */
+ 	save_checksum = phdr-pd_checksum;
+ 	phdr-pd_checksum = 0;
+ 	fnv_checksum = checksum_fnv(page, BLCKSZ);
+ 	phdr-pd_checksum = save_checksum;
+ 
+ 	/* mix in the block number to detect transposed pages */
+ 	fnv_checksum ^= blkno;
+ 
+ 	/*
+ 	 * Reduce to a uint16 (to fit in the pd_checksum field) with an offset of
+ 	 * one. That avoids checksums of zero, which seems like a good idea.
+ 	 */
+ 	return (fnv_checksum % 65535) + 1;
+ }
+ 
+ static uint32
+ checksum_fnv(char *data, uint32 size)
+ {
+ 	uint32 sums[N_SUMS];
+ 	uint32 (*dataArr)[N_SUMS] = (uint32 (*)[N_SUMS]) data;
+ 	uint32 result;
+ 	int i, j;
+ 
+ 	Assert((size % (sizeof(uint32)*N_SUMS)) == 0);
+ 
  	/* initialize partial checksums to their corresponding offsets */
  	memcpy(sums, checksumBaseOffsets, sizeof(checksumBaseOffsets));
  
! 	/* main checksum calculation */
! 	for (i = 0; i  size/sizeof(uint32)/N_SUMS; i++)
  		for (j = 0; j  N_SUMS; j++)
! 			CHECKSUM_COMP(sums[j], dataArr[i][j]);
  
  	/* finally add in one round of zeroes for one more layer of mixing */
  	for (j = 0; j  N_SUMS; j++)
***
*** 1021,1026  PageCalcChecksum16(Page page, BlockNumber blkno)
  	for (i = 0; i  N_SUMS; i++)
  		result ^= sums[i];
  
! 	/* use mod mapping to map the value into 16 bits and offset from zero */
! 	return (result % 65535) + 1;
  }
--- 1039,1043 
  	for (i = 0; i  N_SUMS; i++)
  		result ^= sums[i];
  
! 	return result;
  }

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:

[HACKERS] boolean abuse in geo_ops.c

2013-04-22 Thread Peter Eisentraut
The function

static char *
path_encode(bool closed, int npts, Point *pt)

is actually occasionally passed -1 as its first parameter to mean
something like neither or don't print any delimiters.  That doesn't
seem like good style.  Maybe this should be split off into another bool
parameter named delimited or something?  Or the argument could be made
into an integer.





-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] GSOC Student Project Idea

2013-04-22 Thread Stephen Frost
Michael,

* Michael Schuh (schuh.m...@gmail.com) wrote:
 In brief, I would like to implement a state-of-the-art indexing algorithm
 (named iDistance) directly in PostgreSQL using GiST or SP-GiST trees and
 whatever means necessary. 

For my 2c, this sounds fantastic.  Excellent project idea and it sounds
like you could actually get it done.

Thanks,

Stephen


signature.asc
Description: Digital signature


Re: [HACKERS] Performance with the new security release?

2013-04-22 Thread Anne Rosset
Thanks Steve.
I have read that a fix has been put in release 9.2.3 for this issue. Is that 
right?
Thanks,
Anne

-Original Message-
From: Steve Singer [mailto:st...@ssinger.info] 
Sent: Monday, April 22, 2013 4:35 PM
To: Anne Rosset
Cc: pgsql-hackers@postgresql.org
Subject: Re: [HACKERS] Performance with the new security release?

On 13-04-22 04:41 PM, Anne Rosset wrote:
 Hi Steve,
 Yes I see these messages in our log. Is there a solution to this?
 Thanks,
 Anne

A manual analyze of the effected tables should work and give you updated 
statistics.  If your problem is just statistics then that should help.
A manual vacuum will , unfortunately, behave like the auto-vacuum. The only way 
to get vacuum past this (until this issue is fixed) is for 
vacuum to be able to get that exclusive lock.   If there are times of 
the day your database is less busy you might have some luck turning off 
auto-vacuum on these tables  and doing manual vacuums during those times.




 -Original Message-
 From: Steve Singer [mailto:st...@ssinger.info]
 Sent: Monday, April 22, 2013 1:26 PM
 To: Anne Rosset
 Cc: pgsql-hackers@postgresql.org
 Subject: Re: [HACKERS] Performance with the new security release?

 On 13-04-22 04:15 PM, Anne Rosset wrote:
 Hi Steve,
 Thanks for your reply.
 We are now running  9.0.13. Before it was 9.0.7.
 How can I find out if we are running into this issue: ie if 
 statistics are no longer being updated because analyze can't get the 
 exclusive lock for truncation?
 This issue is discussed in the thread
 http://www.postgresql.org/message-id/CAMkU=1xYXOJp=jLAASPdSAqab-HwhA_t
 nRhy+JUe=4=b=v3...@mail.gmail.com

 If your seeing messages in your logs of the form:

 automatic vacuum of table XXX.YYY cannot (re)acquire exclusive lock for 
 truncate scan

 then you might be hitting this issue.


 I will dig into our logs to see for the query times.
 Thanks,
 Anne

 -Original Message-
 From: Steve Singer [mailto:st...@ssinger.info]
 Sent: Monday, April 22, 2013 12:59 PM
 To: Anne Rosset
 Cc: pgsql-hackers@postgresql.org
 Subject: Re: [HACKERS] Performance with the new security release?

 On 13-04-22 01:38 PM, Anne Rosset wrote:
 Hi,
 We are seeing some overall performance degradation in our application
 since we installed the security release. Other commits were also 
 done at the same time in the application so we don't know yet if the
 degradation has any relationship with the security release.

 While we are digging into this, I would like to know if it is possible
 that the release has some impact on performance. After reading this
 It was created as a side effect of a refactoring effort to make
 establishing new connections to a PostgreSQL server faster, and the
 associated code more maintainable., I am thinking it is quite possible.

 Please let me know. Thanks,
 Exactly which version of PostgreSQL are you running? (we released security 
 update releases for multiple PG versions).  Also which version were you 
 running before?

 There were some changes to analyze/vacuum in the previous set of minor 
 releases that could cause performance issues in some cases (ie if statistics 
 are no longer being updated because analyze can't get the
 exclusive lock for truncation).   There might be other unintended
 performance related changes.

 Are all queries taking longer or only some?  Can you find any sort of 
 pattern that might help narrow the issue?

 Steve

 Anne








-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Enabling Checksums

2013-04-22 Thread Alvaro Herrera
Jeff Davis escribió:

 I'm not sure what we should call the separate file or where we should
 put it, though. How about src/backend/utils/checksum/checksum_fnv.c? Is
 there a clean way to override the compiler flags for a single file so we
 don't need to put it in its own directory?

Sure, see src/backend/parser/Makefile about gram.o.

-- 
Álvaro Herrerahttp://www.2ndQuadrant.com/
PostgreSQL Development, 24x7 Support, Training  Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers