Re: [HACKERS] reducing our reliance on MD5

2015-02-11 Thread José Luis Tallón

On 02/11/2015 03:55 PM, Claudio Freire wrote:

On 02/11/2015 03:39 PM, Claudio Freire wrote:

[snip]
Seems the risk of someone either lifting pg_authid from disk or by hacking
the system and being postgres, thereby accessing passwords stored
somewhere
else, is actually the bigger problem. But also one that should be
reasonably
easy (TM) to fix in a backwards compatible way? (just rewrite with a new
hash whenever the password is changed, but keep reading md5 until they are
all replaced.
Problem with all challenge-response authentication protocols, is that
the hash you have stored has to match the hash you use on the wire
protocol.

It's not like you can store a SHA and provide MD5 authentication.

Yes, except that you can do fallback to plaintext if the client requests
(S)CRAM-SHA and you have (S)CRAM-MD5 instead, allowing for some
interoperability and backwards compatibility in the process: pre-change
libpq/JDBC could authenticate using password to a server with just
SCRAM-SHA512 credentials.

In any case, just storing the password BLOB(text or base64 encoded) along
with a mechanism identifier would go a long way towards making this part
pluggable... just like we do with LDAP/RADIUS/Kerberos/PAM today.


Wait... you mean falling back to storing plaintext or falling back to
transmitting plaintext over the wire?


Fallback to authentication using plaintext ... if the client is old. And 
might as well require SSL before allowing that --- the sane solution, 
definitively.
But let us not forget that there exist environments where even 
non-SSL plaintext is acceptable when the network is secure/trusted. We 
should not prevent that usage if users need it (performance or 
management reasons) if at all possible, I think.


This would imply adding a fallback or plaintext negotiation when 
possible to the new mechanisms.

We could even restrict that behaviour to hostssl entries in pg_hba.conf


Both seem a step backwards IMO.
Hmmm... as opposed to breaking applications innecesarily when simply 
enabling SSL/TLS would not make it insecure? or when users don't really 
need it?


There are many organizations out there happily using 3rd party 
applications that they can not modify easily and this change would break 
them gratuitously.

- New policy. Introduce new hashes, update credentials (when possible!)
- New applications can benefit from the newer/better security 
immediately
- Client app that uses older libpq/JDBC. Doesn't understand the new 
mech

- ... and now the user is left with a broken app and no easy way back

alternative: old app has to authenticate itself just as if 
pg_hba.conf said password, possibly restricted to doing so over an 
SSL-secured connection.



Moreover, requiring everybody to change all passwords and clients *at 
once* seems like a very poor decision towards allowing for graceful 
upgrades and make rolling changes back possible, right?


Additionally, there are cases where passwords are not stored in 
plaintext anywhere (required to be able to generate new credentials) and 
updating all clients at once simply isn't possible.
Please note that, AFAICS, adding a second entry to pg_hba.conf with the 
fallback mechanism won't work.


Now I come to think of it, the behaviour I just proposed would help 
users and maintainers of connection poolers to achieve seamless upgrades 
while increasing security (the pooler would negotiate using the newer 
mechanism with postgres using the client-provided password)




Thanks,

J.L.



--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Manipulating complex types as non-contiguous structures in-memory

2015-02-11 Thread Stephen Frost
* Tom Lane (t...@sss.pgh.pa.us) wrote:
 I think restricting this feature to varlena types is just fine.

Agreed.

Thanks,

Stephen


signature.asc
Description: Digital signature


Re: [HACKERS] reducing our reliance on MD5

2015-02-11 Thread Bruce Momjian
On Tue, Feb 10, 2015 at 09:30:37PM -0500, Tom Lane wrote:
 I think it would be wise to take two steps back and think about what
 the threat model is here, and what we actually need to improve.
 Offhand I can remember two distinct things we might wish to have more
 protection against:
 
 * scraping of passwords off the wire protocol (but is that still
 a threat in an SSL world?).  Better salting practice would do more
 than replacing the algorithm as such for this, IMO.

Agreed.  In 2004 Greg Stark estimated that it would take only 64k
connection attempts to get a server-supplied reply of a salt already
seen that can be replayed:

  
http://www.postgresql.org/message-id/flat/200410071728.i97hs1a16...@candle.pha.pa.us#200410071728.i97hs1a16...@candle.pha.pa.us

If you have a few salts the number goes down further.  I think the
32-bit salt length is the greatest risk to our existing MD5
implementation.  While leaving MD5 has a theoretical benefit, using a
64-bit salt has a practical benefit.

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + Everyone has their own god. +


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] [pgsql-advocacy] GSoC 2015 - mentors, students and admins.

2015-02-11 Thread Stephen Frost
Thom,

* Thom Brown (t...@linux.com) wrote:
 Before I do that, I'd like to have an idea of how many people are
 interested in being either a student or a mentor.

I'm happy to participate as a mentor again this year.

Thanks!

Stephen


signature.asc
Description: Digital signature


Re: [HACKERS] reducing our reliance on MD5

2015-02-11 Thread Stephen Frost
All,

* Heikki Linnakangas (hlinnakan...@vmware.com) wrote:
 On 02/11/2015 05:57 AM, Tom Lane wrote:
 In any case, my larger point was that given the pain that we're going to
 incur here, and the certainly years-long transition interval involved,
 it would be foolish to think only about replacing the MD5 algorithm and
 not about reconsidering the context we use it in.  Stuff like unreasonably
 short salt values should be dealt with at the same time.
 
 +1. We should pick a well-documented mechanism with well-understood
 properties, rather than roll our own again.

Agreed (with this and with the overall proposal from Heikki).

 4. Implement the SASL mechanism SCRAM. That's a challenge/response
 password scheme, similar to the current MD5 authentication, but with
 better salt and more expensive computation (= more resistance to
 dictionary attacks).

Note that Kerberos/GSSAPI can be implemented through SASL too, which
means we might be able to deprecate and eventually remove the code we
have for that (the only caveat to that is that some the options we
provide may not all be available through SASL; that said, some of the
options we provide shouldn't ever be used in production environments,
so..).

Thanks,

Stephen


signature.asc
Description: Digital signature


Re: [HACKERS] reducing our reliance on MD5

2015-02-11 Thread Stephen Frost
* Heikki Linnakangas (hlinnakan...@vmware.com) wrote:
 Let's do this properly. I can help with that,
 although I don't know if I'll find the time and enthusiasm to do all
 of it alone.

Agreed- let's do it properly.  This is definitely an area of interest
for me and if I can ever get out from under the current things going on,
I'd be happy to help also.

Thanks,

Stephen


signature.asc
Description: Digital signature


Re: [HACKERS] libpq's multi-threaded SSL callback handling is busted

2015-02-11 Thread Jan Urbański
Jan Urbański writes:

 I did some more digging on bug
 http://www.postgresql.org/message-id/cahul3dpwyfnugdgo95ohydq4kugdnbkptjq0mnbtubhcmg4...@mail.gmail.com
 which describes a deadlock when using libpq with SSL in a multi-threaded
 environment with other threads doing SSL independently.

 [reproducing instructions]

 I posit we should remove all CRYPTO_set_*_callback functions and associated
 cruft from libpq.

 I could submit a patch to get rid of the crazy CRYPTO_*_callback dance in
 libpq, but at the very least this will require a warning in the release notes

Attached is a patch doing just that.

 I would very much like to have this change back-patched, since setting and
 resetting the callback makes using libpq in a threaded OpenSSL-enabled app
 arguably less safe than if it didn't use any locking.

Also attached is a patch for 9.4 and all previous supported releases, which is
the same thing, but adjusted for when we didn't have a separate fe-secure.c and
fe-secure-openssl.c

If committed, this change will warrant a notice in the release notes. I could
try drafting it, if that'd be helpful.

Cheers,
Jan
commit 62f7433f697d49ab005ad22822dc943661a4e48a
Author: Jan Urbański wulc...@wulczer.org
Date:   Wed Feb 11 17:47:09 2015 +0100

Do not attempt to manage OpenSSL locking callbacks in libpq.

According to OpenSSL documentation, threaded programs using SSL should register
locking callbacks before starting work. In 6daf396879b6502c41146cdb1013568654f52ff6
libpq started doing that, registering its callbacks when the connection would
switch to SSL mode.

This would lead to overwriting any locking callback a properly written threaded
SSL application might have already set, but only got uncovered as a bug in
http://www.postgresql.org/message-id/48620925.6070...@pws.com.au where it
turned out that unloading libpq would leave a dangling function
pointer. Specifically, the PHP runtime would segfault after unloading libpq, as
reported in https://bugs.php.net/bug.php?id=40926

Commit 4e816286533dd34c10b368487d4079595a3e1418 made libpq unset the callback
after the last SSL connection was closed. This solved the segfault issue, but
introduced a potential for deadlocks when one thread using libpq with SSL and
another doing an unrelated SSL operation.

T1 (libpq) T2 (other)

start libpq connection
add locking callback

   start SSL operation
   take lock

finish libpq connection
remove locking callback

   (!) release lock, noop since no callback

This got reported in bug
http://www.postgresql.org/message-id/cahul3dpwyfnugdgo95ohydq4kugdnbkptjq0mnbtubhcmg4...@mail.gmail.com

The reality is that it's not libpq's responsibility to manage those OpenSSL
callbacks. It's up to application code to set them up before using OpenSSL in
threads and trying to handle them in libpq is just stomping on the ones that
should already by in place by the time we start using shared OpenSSL
structures.

This commit gets rid of all OpenSSL callback handling from libpq. For
well-behaved applications, this should increase reliability, since they will
now have certainty that the callbacks they set up before attempting to use
OpenSSL will be used throughout the program execution.

Applications that relied on libpq to set up locking callbacks will need to be
fixed.

diff --git a/src/interfaces/libpq/fe-secure-openssl.c b/src/interfaces/libpq/fe-secure-openssl.c
index a32af34..5d0b132 100644
--- a/src/interfaces/libpq/fe-secure-openssl.c
+++ b/src/interfaces/libpq/fe-secure-openssl.c
@@ -67,7 +67,6 @@ static int	verify_cb(int ok, X509_STORE_CTX *ctx);
 static int	verify_peer_name_matches_certificate_name(PGconn *conn,
   ASN1_STRING *name,
   char **store_name);
-static void destroy_ssl_system(void);
 static int	initialize_SSL(PGconn *conn);
 static PostgresPollingStatusType open_client_SSL(PGconn *);
 static char *SSLerrmessage(void);
@@ -90,8 +89,6 @@ static bool pq_init_crypto_lib = true;
 static SSL_CTX *SSL_context = NULL;
 
 #ifdef ENABLE_THREAD_SAFETY
-static long ssl_open_connections = 0;
-
 #ifndef WIN32
 static pthread_mutex_t ssl_config_mutex = PTHREAD_MUTEX_INITIALIZER;
 #else
@@ -112,16 +109,6 @@ static long win32_ssl_create_mutex = 0;
 void
 pgtls_init_library(bool do_ssl, int do_crypto)
 {
-#ifdef ENABLE_THREAD_SAFETY
-
-	/*
-	 * Disallow changing the flags while we have open connections, else we'd
-	 * get completely confused.
-	 */
-	if (ssl_open_connections != 0)
-		return;
-#endif
-
 	pq_init_ssl_lib = do_ssl;
 	pq_init_crypto_lib = do_crypto;
 }
@@ -705,40 +692,6 @@ verify_peer_name_matches_certificate(PGconn *conn)
 	return found_match  !got_error;
 }
 
-#ifdef ENABLE_THREAD_SAFETY
-/*
- *	Callback functions 

Re: [HACKERS] reducing our reliance on MD5

2015-02-11 Thread Joshua D. Drake


On 02/11/2015 07:54 AM, José Luis Tallón wrote:


On 02/11/2015 04:40 PM, Tom Lane wrote:

=?UTF-8?B?Sm9zw6kgTHVpcyBUYWxsw7Nu?= jltal...@adv-solutions.net writes:

In any case, just storing the password BLOB(text or base64 encoded)
along with a mechanism identifier would go a long way towards making
this part pluggable... just like we do with LDAP/RADIUS/Kerberos/PAM
today.

That's exactly the direction we must NOT go.


From a practitioners and one step at a time perspective, why don't we 
just offer SHA-2 as an alternative to MD5?


As a longer term approach, it seems something like key based auth (ala 
SSH) which proved popular when I brought it up before seems like a 
reasonable solution.


Sincerely,

Joshua D. Drake


--
Command Prompt, Inc. - http://www.commandprompt.com/  503-667-4564
PostgreSQL Support, Training, Professional Services and Development
High Availability, Oracle Conversion, @cmdpromptinc
If we send our children to Caesar for their education, we should
 not be surprised when they come back as Romans.


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] pgbench -f and vacuum

2015-02-11 Thread Robert Haas
On Wed, Feb 11, 2015 at 2:00 PM, Jeff Janes jeff.ja...@gmail.com wrote:
  I would rather just learn to add the -n when I use -f
 and don't have the default tables in place, than have to learn new methods
 for saying no really, I left -n off on purpose when I have a custom file
 which does use the default tables and I want them vacuumed.

Quite.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


[HACKERS] Standby receiving part of missing WAL segment

2015-02-11 Thread Thom Brown
Hi,

Today I witnessed a situation which appears to have gone down like this:

- The primary server starting streaming WAL data from segment 00A8 to the
standby
- The standby server started receiving that data
- Before 00A8 is finished, the wal sender process dies on the primary, but
the archiver process continues, and 00A8 ends up being archived as usual
- The primary continues to generate WAL and cleans up old WAL files from
pg_xlog until 00A8 is gone.
- The primary is restarted and the wal sender process is back up and running
- The standby says waiting for 00A8, which it can no longer get from the
primary
- 00A8 is in the standby's archive directory, but the standby is waiting
for the rest of the segment from the primary via streaming replication, so
doesn't check the archive
- The standby is restarted
- The standby goes back into recovery and eventually replays 00A8 and
continues as normal.

Should the standby be able to get feedback from the primary that the
requested segment is no longer available, and therefore know to check its
archive?

Or should it check the archive anyway if it hasn't received any further WAL
data via the streaming replication connection after a certain amount of
time?

At the moment, the standby gets stuck forever in this situation, even
though it has access to the WAL it needs.

Thom


Re: [HACKERS] ibm system z in the buildfarm

2015-02-11 Thread Mark Wong
On 02/11/2015 05:48 AM, David Fetter wrote:
 On Tue, Feb 10, 2015 at 05:49:18PM -0800, Mark Wong wrote:
 Hi everyone,

 I've seen in the archive a call for more architecture coverage so I just
 wanted to send a quick note that there is now Linux on System Z in the
 buildfarm now:

 http://pgbuildfarm.org/cgi-bin/show_history.pl?nm=nudibranchbr=HEAD
 
 That's super awesome!
 
 Do you have access to put a z/OS animal there, too?  The compilers
 available for that OS, as well as the OS itself, should make for some
 very interesting results.

Hmm, I don't think so, but we'll do it if the opportunity comes around.

Regards,
Mark



-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] GSoC 2015 - mentors, students and admins.

2015-02-11 Thread Josh Berkus
On 02/11/2015 04:33 AM, Anastasia Lubennikova wrote:
 I'm interested to participate as student again.

Let's *really* hope for no US sanctions, then.

-- 
Josh Berkus
PostgreSQL Experts Inc.
http://pgexperts.com


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] pgbench -f and vacuum

2015-02-11 Thread Jeff Janes
On Tue, Dec 23, 2014 at 7:42 AM, Alvaro Herrera alvhe...@2ndquadrant.com
wrote:

 Robert Haas wrote:
  On Mon, Dec 22, 2014 at 6:55 PM, Alvaro Herrera
  alvhe...@2ndquadrant.com wrote:
   Here's a completely different idea.  How about we add an option that
   means vacuum this table before running the test (can be given several
   times); by default the set of vacuumed tables is the current pgbench_*
   list, but if -f is specified then the default set is cleared.  So if
 you
   have a -f script and want to vacuum the default tables, you're forced
 to
   give a few --vacuum-table=foo options.  But this gives the option to
   vacuum some other table before the test, not just the pgbench default
   ones.
 
  Well, really, you might want arbitrary initialization steps, not just
  vacuums.  We could have --init-script=FILENAME.

 Init (pgbench -i) is the step that creates the tables and populates
 them, so I think this would need a different name, maybe startup, but
 otherwise yeah.

  Although that might be taking this thread rather far off-topic.

 Not really sure about that, because the only outstanding objection to
 this discussion is what happens in the startup stage if you specify -f.
 Right now vacuum is attempted on the standard tables, which is probably
 not the right thing in the vast majority of cases.  But if we turn that
 off, how do we reinstate it for the rare cases that want it?  Personally
 I would just leave it turned off and be done with it, but if we want to
 provide some way to re-enable it, this --startup-script=FILE gadget
 sounds like a pretty decent idea.


There are two (or more?) possible meanings of a startup script.  One would
be run a single time at the start of a pgbench run, and one would be run at
the start of each connection, in the case of -C or -c.  Vacuums would
presumably go in the first category, while something like tweaking a
work_mem or enable_* setting would use the second.  I'd find more use for
the second way.

I had a patch to do this on a per connection basis a while ago, but it took
the command as a string to --startup.  Robert suggested it be a filename
rather than a string, and I agreed but never followed up with a different
patch, as I couldn't figure out how to refactor the code that parses -f
files so that it could be used for this without undo replication of the
code.

See 
http://www.postgresql.org/message-id/CAMkU=1xV3tYKoHD8U2mQzfC5Kbn_bdcVf8br-EnUvy-6Z=b...@mail.gmail.com


I was wondering if we could't invent three new backslash commands.

One would precede an SQL command to be run during -i, and ignored any other
time (and then during -i any unbackslashed commands would be ignored)

One would precede an SQL command to be run upon starting up a pgbench run.

One would precede an SQL command to be run upon starting up a benchmarking
connection.

That way you could have a single file that would record its own
initialization requirements.

One problem is I don't know how you would merge together multiple -f
arguments.  Another is I would want to be able to override the
per-connection command without having to use sed or something to
edit-in-place the SQL file.

But as far as what has been discussed on the central topic of this thread,
I think that doing the vacuum and making the failure for non-existent
tables be non-fatal when -f is provided would be an improvement.  Or maybe
just making it non-fatal at all times--if the table is needed and not
present, the session will fail quite soon anyway.  I don't see the other
changes as being improvements.  I would rather just learn to add the -n
when I use -f and don't have the default tables in place, than have to
learn new methods for saying no really, I left -n off on purpose when I
have a custom file which does use the default tables and I want them
vacuumed.

Cheers,

Jeff


Re: [HACKERS] [pgsql-advocacy] GSoC 2015 - mentors, students and admins.

2015-02-11 Thread Grzegorz Parka
I'm interested to participate as student.

Regards,
Grzegorz Parka

2015-02-11 17:55 GMT+01:00 Stephen Frost sfr...@snowman.net:

 Thom,

 * Thom Brown (t...@linux.com) wrote:
  Before I do that, I'd like to have an idea of how many people are
  interested in being either a student or a mentor.

 I'm happy to participate as a mentor again this year.

 Thanks!

 Stephen



Re: [HACKERS] parallel mode and parallel contexts

2015-02-11 Thread Robert Haas
On Wed, Feb 11, 2015 at 1:59 PM, Robert Haas robertmh...@gmail.com wrote:
 I'm not sure what you mean by the severity of the lock.  How about
 marking the locks that the worker inherited from the parallel master
 and throwing an error if it tries to lock one of those objects in a
 mode that it does not already hold?  That'd probably catch a sizeable
 number of programming errors without tripping up many legitimate use
 cases (and we can always relax or modify the prohibition if it later
 turns out to be problematic).

Or maybe do that only when the new lock mode is stronger than one it
already holds.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] reducing our reliance on MD5

2015-02-11 Thread Claudio Freire
On Wed, Feb 11, 2015 at 10:31 AM, Magnus Hagander mag...@hagander.net wrote:
 On Wed, Feb 11, 2015 at 4:57 AM, Tom Lane t...@sss.pgh.pa.us wrote:

 Robert Haas robertmh...@gmail.com writes:
  On Tue, Feb 10, 2015 at 9:30 PM, Tom Lane t...@sss.pgh.pa.us wrote:
  Another thing we need to keep in mind besides client compatibility
  is dump/reload compatibility.

  I don't think there's an issue with dump/reload compatibility as far
  as what I proposed, since it only has to do with the authentication
  procedure, not what gets stored in pg_authid.  We might have reasons
  for moving that away from MD5 as well, but it's a separate project.

 Hm, well, that doesn't really square with your other expressed opinion:

  Are there other goals?

  I think the goal is stop using MD5, or at least have an option to not
  use MD5, because people think that's insecure.

 As you say, it's quite debatable whether MD5 is or isn't secure enough
 given the way we use it, but what's not debatable is that the optics of it
 are not very good anymore.  However, if we want to shut up the peanut
 gallery on this point, we have to get rid of MD5 usage in pg_authid not
 just the on-the-wire protocol --- I seriously doubt that the knee jerk
 MD5-is-insecure crowd will make any distinction there.  So I'm not
 following how you're satisfied with a proposal for just the latter.

 In any case, my larger point was that given the pain that we're going to
 incur here, and the certainly years-long transition interval involved,
 it would be foolish to think only about replacing the MD5 algorithm and
 not about reconsidering the context we use it in.  Stuff like unreasonably
 short salt values should be dealt with at the same time.



 All discussion seems to be about the protocol, which is also the harder
 problem, isn't it?

 ISTM that the more *important* thing to fix is the on-disk storage in
 pg_authid.

 If you actually worry about someone sniffing your network and mounting an
 attack against a password, you can use SSL. That means we already have a way
 to deal with that, whereas we don't have a way to mitigate the pg_authid
 thing. And also, if you are actually worried about someone sniffing an
 MD5'ed password, shouldn't you also worry about that same person sniffing
 all the *contents* of your database, which the password is there to protect?

 (and in that regard, LDAP and such encryption algorithms send the password
 in clear text in that case)

 Seems the risk of someone either lifting pg_authid from disk or by hacking
 the system and being postgres, thereby accessing passwords stored somewhere
 else, is actually the bigger problem. But also one that should be reasonably
 easy (TM) to fix in a backwards compatible way? (just rewrite with a new
 hash whenever the password is changed, but keep reading md5 until they are
 all replaced.

Problem with all challenge-response authentication protocols, is that
the hash you have stored has to match the hash you use on the wire
protocol.

It's not like you can store a SHA and provide MD5 authentication.

In essence, it makes making it pluggable or doing both changes
separately as two separate improvements more difficult.


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] reducing our reliance on MD5

2015-02-11 Thread José Luis Tallón

On 02/11/2015 03:39 PM, Claudio Freire wrote:

[snip]
Seems the risk of someone either lifting pg_authid from disk or by hacking
the system and being postgres, thereby accessing passwords stored somewhere
else, is actually the bigger problem. But also one that should be reasonably
easy (TM) to fix in a backwards compatible way? (just rewrite with a new
hash whenever the password is changed, but keep reading md5 until they are
all replaced.
Problem with all challenge-response authentication protocols, is that
the hash you have stored has to match the hash you use on the wire
protocol.

It's not like you can store a SHA and provide MD5 authentication.


Yes, except that you can do fallback to plaintext if the client 
requests (S)CRAM-SHA and you have (S)CRAM-MD5 instead, allowing for some 
interoperability and backwards compatibility in the process: pre-change 
libpq/JDBC could authenticate using password to a server with just 
SCRAM-SHA512 credentials.


In any case, just storing the password BLOB(text or base64 encoded) 
along with a mechanism identifier would go a long way towards making 
this part pluggable... just like we do with LDAP/RADIUS/Kerberos/PAM today.



/ J.L.





--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] reducing our reliance on MD5

2015-02-11 Thread Magnus Hagander
On Wed, Feb 11, 2015 at 3:10 PM, José Luis Tallón 
jltal...@adv-solutions.net wrote:

  On 02/11/2015 02:31 PM, Magnus Hagander wrote:


  In any case, my larger point was that given the pain that we're going to
 incur here, and the certainly years-long transition interval involved,
 it would be foolish to think only about replacing the MD5 algorithm and
 not about reconsidering the context we use it in.  Stuff like unreasonably
 short salt values should be dealt with at the same time.



  All discussion seems to be about the protocol, which is also the harder
 problem, isn't it?

  ISTM that the more *important* thing to fix is the on-disk storage in
 pg_authid.


 At least, looks like it would be the most urgent (and with no need for
 clients breaking in the process, AFAICT)

   [snip]
 Seems the risk of someone either lifting pg_authid from disk or by hacking
 the system and being postgres, thereby accessing passwords stored somewhere
 else, is actually the bigger problem. But also one that should be
 reasonably easy (TM) to fix in a backwards compatible way? (just rewrite
 with a new hash whenever the password is changed, but keep reading md5
 until they are all replaced.


 Adding a new system column with a text or enum representing the algorithm
 that created the hash would go a lot towards fixing this situation.
 When/If the column isn't there, just assume md5. This would allow for
 transparent pg_upgrade.


The hash value in pg_authid already contains md5 as a prefix. No need for
another column.


-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/


Re: [HACKERS] reducing our reliance on MD5

2015-02-11 Thread José Luis Tallón

On 02/11/2015 03:14 PM, Magnus Hagander wrote:


[snip]
The hash value in pg_authid already contains md5 as a prefix. No 
need for another column.


Yes, but for variable length mechanism names (i.e. not just 3 chars) it 
would become increasingly difficult to differentiate between the algo 
name and the stored credentials especially if we delegated the list 
of available mechanisms to an external library and/or in the case of 
upgrades.
(variable-length matching based on a table of available mechs and 
using strncmp isn't complicated, admittedly  but why bother?)


... plus we have already added many new columns to store the new 
capabilities in, as opposed to a bitmask.


I might well be overlooking something else, of course.


Regards,

/ J.L.



--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] reducing our reliance on MD5

2015-02-11 Thread Claudio Freire
On Wed, Feb 11, 2015 at 11:48 AM, José Luis Tallón
jltal...@adv-solutions.net wrote:
 On 02/11/2015 03:39 PM, Claudio Freire wrote:

 [snip]
 Seems the risk of someone either lifting pg_authid from disk or by hacking
 the system and being postgres, thereby accessing passwords stored
 somewhere
 else, is actually the bigger problem. But also one that should be
 reasonably
 easy (TM) to fix in a backwards compatible way? (just rewrite with a new
 hash whenever the password is changed, but keep reading md5 until they are
 all replaced.
 Problem with all challenge-response authentication protocols, is that
 the hash you have stored has to match the hash you use on the wire
 protocol.

 It's not like you can store a SHA and provide MD5 authentication.


 Yes, except that you can do fallback to plaintext if the client requests
 (S)CRAM-SHA and you have (S)CRAM-MD5 instead, allowing for some
 interoperability and backwards compatibility in the process: pre-change
 libpq/JDBC could authenticate using password to a server with just
 SCRAM-SHA512 credentials.

 In any case, just storing the password BLOB(text or base64 encoded) along
 with a mechanism identifier would go a long way towards making this part
 pluggable... just like we do with LDAP/RADIUS/Kerberos/PAM today.


Wait... you mean falling back to storing plaintext or falling back to
transmitting plaintext over the wire?

Both seem a step backwards IMO.


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] reducing our reliance on MD5

2015-02-11 Thread Claudio Freire
On Wed, Feb 11, 2015 at 12:20 PM, José Luis Tallón
jltal...@adv-solutions.net wrote:
 Both seem a step backwards IMO.

 Hmmm... as opposed to breaking applications innecesarily when simply
 enabling SSL/TLS would not make it insecure? or when users don't really need
 it?

No, as opposed to cases where people are already using md5
authentication effectively.

 Moreover, requiring everybody to change all passwords and clients *at once*
 seems like a very poor decision towards allowing for graceful upgrades and
 make rolling changes back possible, right?

I wasn't advocating for what. I was just pointing out that
implementation of SCRAM was an all-encompassing endeavour, in that you
have to attack both password storage and the transmission protocol.

For instance, if you tell me I can do SCRAM, I'll be happy to enable
it if my lib has support for it. But if you add oh... to use it, you
must store plaintext on pg_authid, I'd change my mind. Because you
cannot do SCRAM while storing md5.

And there lies the issue I'm pointing out. I'm not giving solutions.

Except, maybe, that if it were to be explicit for user creation:

CREATE ROLE someone WITH ENCRYPTED PASSWORD '1234' USING 'pbkdf2-hmac-sha1' ;

This would preclude authentication using md5, of course, but it would
be expectable and under admin control. And I myself would use it.

 Additionally, there are cases where passwords are not stored in plaintext
 anywhere (required to be able to generate new credentials) and updating all
 clients at once simply isn't possible.

I agree


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] reducing our reliance on MD5

2015-02-11 Thread José Luis Tallón

On 02/11/2015 04:40 PM, Tom Lane wrote:

=?UTF-8?B?Sm9zw6kgTHVpcyBUYWxsw7Nu?= jltal...@adv-solutions.net writes:

In any case, just storing the password BLOB(text or base64 encoded)
along with a mechanism identifier would go a long way towards making
this part pluggable... just like we do with LDAP/RADIUS/Kerberos/PAM today.

That's exactly the direction we must NOT go.

Upgrading the security of stored passwords in pg_authid is at least as
important as upgrading the wire protocol security; very possibly more so.
Any solution that requires cleartext passwords to be kept by the server
is simply not going to be accepted.


I definitively haven't explained myself properly.
I *never* suggested storing plaintext in pg_authid, but using plaintext 
authentication (which can always be matched against an on-disk hash, 
whatever the type) as a fallback to allow for seamless upgrades of security.
(once you are authenticated by using the old credentials, the 
server can transparently store the new hash)


When I referred to a text or base64 encoded I never implied on-disk 
plaintext (unless the user specifically requires that, which they might).



To avoid ambiguities, my proposal closely mimicks Dovecot's 
implementation of password schemes and credential upgrades

http://wiki2.dovecot.org/Authentication/PasswordSchemes




Thanks,

J.L.




--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] reducing our reliance on MD5

2015-02-11 Thread Robert Haas
On Wed, Feb 11, 2015 at 8:02 AM, Heikki Linnakangas
hlinnakan...@vmware.com wrote:
 On 02/11/2015 02:49 PM, Robert Haas wrote:
 So, this all sounds fairly nice if somebody's willing to do the work,
 but I can't help noticing that you originally proposed adopting SCRAM
 in 2012, and it's 2015 now.  So I wonder if anyone's really going to
 do all this work, and if not, whether we should go for something
 simpler.  Just plugging something else in for MD5 would be a lot less
 work for us to implement and for clients to support, even if it is (as
 it unarguably is) less elegant.

 Just plugging something else in for MD5 would still be a fair amount of
 work. Not that much less than the full program I proposed.

 Well, I guess it's easier if you immediately stop supporting MD5, have a
 flag day in all clients to implement the replacement, and break
 pg_dump/restore of passwords in existing databases. That sounds horrible.
 Let's do this properly. I can help with that, although I don't know if I'll
 find the time and enthusiasm to do all of it alone.

So are you thinking to integrate with the Cyrus SASL library, or do
you have another thought?

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] reducing our reliance on MD5

2015-02-11 Thread Tom Lane
=?UTF-8?B?Sm9zw6kgTHVpcyBUYWxsw7Nu?= jltal...@adv-solutions.net writes:
 In any case, just storing the password BLOB(text or base64 encoded) 
 along with a mechanism identifier would go a long way towards making 
 this part pluggable... just like we do with LDAP/RADIUS/Kerberos/PAM today.

That's exactly the direction we must NOT go.

Upgrading the security of stored passwords in pg_authid is at least as
important as upgrading the wire protocol security; very possibly more so.
Any solution that requires cleartext passwords to be kept by the server
is simply not going to be accepted.

Because of this constraint, I really suspect that we have zero chance of
achieving pluggability or farming out the problem to some third party
library.

Or in short: we've done that before, with LDAP/RADIUS/Kerberos/PAM,
and none of those solutions have proven very satisfactory; they certainly
have not replaced passwords to any measurable degree.  Expecting the next
external solution to do so is the definition of insanity.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] assessing parallel-safety

2015-02-11 Thread Robert Haas
On Wed, Feb 11, 2015 at 9:39 AM, Robert Haas robertmh...@gmail.com wrote:
 On Sun, Feb 8, 2015 at 12:28 PM, Robert Haas robertmh...@gmail.com wrote:
 On Sun, Feb 8, 2015 at 11:31 AM, Noah Misch n...@leadboat.com wrote:
 On Sat, Feb 07, 2015 at 08:18:55PM -0500, Robert Haas wrote:
 There are a few problems with this design that I don't immediately
 know how to solve:

 1. I'm concerned that the query-rewrite step could substitute a query
 that is not parallel-safe for one that is.  The upper Query might
 still be flagged as safe, and that's all that planner() looks at.

 I would look at determining the query's parallel safety early in the planner
 instead; simplify_function() might be a cheap place to check.  Besides
 avoiding rewriter trouble, this allows one to alter parallel safety of a
 function without invalidating Query nodes serialized in the system catalogs.

 Thanks, I'll investigate that approach.

 This does not seem to work out nicely.  The problem here is that
 simplify_function() gets called from eval_const_expressions() which
 gets called from a variety of places, but the principal one seems to
 be subquery_planner().  So if you have a query with two subqueries,
 and the second one contains something parallel-unsafe, you might by
 that time have already generated a parallel plan for the first one,
 which won't do.  Unless we want to rejigger this so that we do a
 complete eval_const_expressions() pass over the entire query tree
 (including all subqueries) FIRST, and then only after that go back and
 plan all of those subqueries, I don't see how to make this work; and
 I'm guessing that there are good reasons not to do that.

And ... while I was just talking with Jan about this, I realized
something that should have been blindingly obvious to me from the
beginning, which is that anything we do at parse time is doomed to
failure, because the properties of a function that we use to determine
parallel-safety can be modified later:

rhaas=# alter function random() immutable;  -- no it isn't
ALTER FUNCTION

I think we may want a dedicated parallel-safe property for functions
rather than piggybacking on provolatile, but that will probably also
be changeable via ALTER FUNCTION, and stored rules won't get
miraculously updated.  So this definitely can't be something we figure
out at parse-time ... it's got to be determined later.  But at the
moment I see no way to do that without an extra pass over the whole
rewritten query tree.  :-(

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] reducing our reliance on MD5

2015-02-11 Thread Claudio Freire
On Wed, Feb 11, 2015 at 6:30 PM, Claudio Freire klaussfre...@gmail.com wrote:
 On Wed, Feb 11, 2015 at 5:25 PM, Heikki Linnakangas
 hlinnakan...@vmware.com wrote:
 On 02/11/2015 06:35 AM, Claudio Freire wrote:

 Usually because handshakes use a random salt on both sides. Not sure
 about pg's though, but in general collision strength is required but
 not slowness, they're not bruteforceable.


 To be precise: collision resistance is usually not important for hashes used
 in authentication handshakes. Not for our MD5 authentication method anyway;
 otherwise we'd be screwed. What you need is resistance to pre-image attacks.

 AFAIK, if I find a colliding string to the MD5 stored in pg_authid, I
 can specify that to libpq and get authenticated.

 Am I missing something?

Oh, right, that's called pre-image.

Never mind then


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] reducing our reliance on MD5

2015-02-11 Thread Heikki Linnakangas

On 02/11/2015 05:34 PM, Claudio Freire wrote:

On Wed, Feb 11, 2015 at 12:20 PM, José Luis Tallón
jltal...@adv-solutions.net wrote:

Moreover, requiring everybody to change all passwords and clients *at once*
seems like a very poor decision towards allowing for graceful upgrades and
make rolling changes back possible, right?


I wasn't advocating for what. I was just pointing out that
implementation of SCRAM was an all-encompassing endeavour, in that you
have to attack both password storage and the transmission protocol.

For instance, if you tell me I can do SCRAM, I'll be happy to enable
it if my lib has support for it. But if you add oh... to use it, you
must store plaintext on pg_authid, I'd change my mind. Because you
cannot do SCRAM while storing md5.


You can't do plain SCRAM while storing md5, but you could implement a 
variant where the stored md5 hash is treated like the plaintext 
password. The client first calculates the MD5(plaintext password), and 
then proceeds with the SCRAM authentication with that. The MD5 doesn't 
add any extra security (nor reduce it AFAICS), but it would allow doing 
SCRAM, using the MD5 hashes already stored in pg_authid.


You'd still be just as vulnerable to someone stealing the MD5 hashes 
from pg_authid, of course. So if you don't want to continue supporting 
MD5 authentication for backwards-compatibility, you'd want to transform 
all the hashes to SCRAM stored keys.


- Heikki



--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] reducing our reliance on MD5

2015-02-11 Thread Claudio Freire
On Wed, Feb 11, 2015 at 5:25 PM, Heikki Linnakangas
hlinnakan...@vmware.com wrote:
 On 02/11/2015 06:35 AM, Claudio Freire wrote:

 Usually because handshakes use a random salt on both sides. Not sure
 about pg's though, but in general collision strength is required but
 not slowness, they're not bruteforceable.


 To be precise: collision resistance is usually not important for hashes used
 in authentication handshakes. Not for our MD5 authentication method anyway;
 otherwise we'd be screwed. What you need is resistance to pre-image attacks.

AFAIK, if I find a colliding string to the MD5 stored in pg_authid, I
can specify that to libpq and get authenticated.

Am I missing something?


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] reducing our reliance on MD5

2015-02-11 Thread Heikki Linnakangas

On 02/11/2015 11:30 PM, Claudio Freire wrote:

On Wed, Feb 11, 2015 at 5:25 PM, Heikki Linnakangas
hlinnakan...@vmware.com wrote:

On 02/11/2015 06:35 AM, Claudio Freire wrote:


Usually because handshakes use a random salt on both sides. Not sure
about pg's though, but in general collision strength is required but
not slowness, they're not bruteforceable.


To be precise: collision resistance is usually not important for hashes used
in authentication handshakes. Not for our MD5 authentication method anyway;
otherwise we'd be screwed. What you need is resistance to pre-image attacks.


AFAIK, if I find a colliding string to the MD5 stored in pg_authid, I
can specify that to libpq and get authenticated.

Am I missing something?


If you know the MD5 stored in pg_authid, you can use that directly to 
authenticate. No need to find the original password, or another 
colliding string, that hashes to the same MD5.


- Heikki



--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] reducing our reliance on MD5

2015-02-11 Thread Heikki Linnakangas

On 02/11/2015 06:35 AM, Claudio Freire wrote:

Usually because handshakes use a random salt on both sides. Not sure
about pg's though, but in general collision strength is required but
not slowness, they're not bruteforceable.


To be precise: collision resistance is usually not important for hashes 
used in authentication handshakes. Not for our MD5 authentication method 
anyway; otherwise we'd be screwed. What you need is resistance to 
pre-image attacks.


See https://en.wikipedia.org/wiki/Cryptographic_hash_function#Properties

- Heikki



--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] reducing our reliance on MD5

2015-02-11 Thread Heikki Linnakangas

On 02/11/2015 03:52 PM, Robert Haas wrote:

On Wed, Feb 11, 2015 at 8:02 AM, Heikki Linnakangas
hlinnakan...@vmware.com wrote:

On 02/11/2015 02:49 PM, Robert Haas wrote:

So, this all sounds fairly nice if somebody's willing to do the work,
but I can't help noticing that you originally proposed adopting SCRAM
in 2012, and it's 2015 now.  So I wonder if anyone's really going to
do all this work, and if not, whether we should go for something
simpler.  Just plugging something else in for MD5 would be a lot less
work for us to implement and for clients to support, even if it is (as
it unarguably is) less elegant.


Just plugging something else in for MD5 would still be a fair amount of
work. Not that much less than the full program I proposed.

Well, I guess it's easier if you immediately stop supporting MD5, have a
flag day in all clients to implement the replacement, and break
pg_dump/restore of passwords in existing databases. That sounds horrible.
Let's do this properly. I can help with that, although I don't know if I'll
find the time and enthusiasm to do all of it alone.


So are you thinking to integrate with the Cyrus SASL library, or do
you have another thought?


I think we need to implement the primary MD5 replacement ourselves, so 
that it's always available without extra libraries. Otherwise it will 
not get much adoption, or the extra dependency will be a hassle anyway. 
It's not that complicated, after all.


We could also support using a library like that for additional 
authentication mechanisms, though, for those who really need them.


- Heikki


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Parallel Seq Scan

2015-02-11 Thread Robert Haas
On Tue, Feb 10, 2015 at 3:56 PM, Andres Freund and...@2ndquadrant.com wrote:
 On 2015-02-10 09:23:02 -0500, Robert Haas wrote:
 On Tue, Feb 10, 2015 at 9:08 AM, Andres Freund and...@2ndquadrant.com 
 wrote:
  And good chunk sizes et al depend on higher layers,
  selectivity estimates and such. And that's planner/executor work, not
  the physical layer (which heapam.c pretty much is).

 If it's true that a good chunk size depends on the higher layers, then
 that would be a good argument for doing this differently, or at least
 exposing an API for the higher layers to tell heapam.c what chunk size
 they want.  I hadn't considered that possibility - can you elaborate
 on why you think we might want to vary the chunk size?

 Because things like chunk size depend on the shape of the entire
 plan. If you have a 1TB table and want to sequentially scan it in
 parallel with 10 workers you better use some rather large chunks. That
 way readahead will be efficient in a cpu/socket local manner,
 i.e. directly reading in the pages into the directly connected memory of
 that cpu. Important for performance on a NUMA system, otherwise you'll
 constantly have everything go over the shared bus.  But if you instead
 have a plan where the sequential scan goes over a 1GB table, perhaps
 with some relatively expensive filters, you'll really want a small
 chunks size to avoid waiting.

I see.  That makes sense.

 The chunk size will also really depend on
 what other nodes are doing, at least if they can run in the same worker.

Example?

 Even without things like NUMA and readahead I'm pretty sure that you'll
 want a chunk size a good bit above one page. The locks we acquire for
 the buffercache lookup and for reading the page are already quite bad
 for performance/scalability; even if we don't always/often hit the same
 lock. Making 20 processes that scan pages in parallel acquire yet a
 another lock (that's shared between all of them!) for every single page
 won't be fun, especially without or fast filters.

This is possible, but I'm skeptical.  If the amount of other work we
have to do that page is so little that the additional spinlock cycle
per page causes meaningful contention, I doubt we should be
parallelizing in the first place.

 No, and I said so upthread. I started commenting because you argued that
 architecturally parallelism belongs in heapam.c instead of upper layers,
 and I can't agree with that.  I now have, and it looks less bad than I
 had assumed, sorry.

OK, that's something.

 Unfortunately I still think it's wrong approach, also sorry.

 As pointed out above (moved there after reading the patch...) I don't
 think a chunk size of 1 or any other constant size can make sense. I
 don't even believe it'll necessarily be constant across an entire query
 execution (big initially, small at the end).  Now, we could move
 determining that before the query execution into executor
 initialization, but then we won't yet know how many workers we're going
 to get. We could add a function setting that at runtime, but that'd mix
 up responsibilities quite a bit.

I still think this belongs in heapam.c somehow or other.  If the logic
is all in the executor, then it becomes impossible for any code that
doensn't use the executor to do a parallel heap scan, and that's
probably bad.  It's not hard to imagine something like CLUSTER wanting
to reuse that code, and that won't be possible if the logic is up in
some higher layer.  If the logic we want is to start with a large
chunk size and then switch to a small chunk size when there's not much
of the relation left to scan, there's still no reason that can't be
encapsulated in heapam.c.

 Btw, using a atomic uint32 you'd end up without the spinlock and just
 about the same amount of code... Just do a atomic_fetch_add_until32(var,
 1, InvalidBlockNumber)... ;)

I thought of that, but I think there's an overflow hazard.

 Where the 'coordinated seqscan' scans a relation so that each tuple
 eventually gets returned once across all nodes, but the nested loop (and
 through it the index scan) will just run normally, without any
 coordination and parallelism. But everything below --- would happen
 multiple nodes. If you agree, yes, then we're in violent agreement
 ;). The single node that gets copied bit above makes me a bit unsure
 whether we are though.

Yeah, I think we're talking about the same thing.

 To me, given the existing executor code, it seems easiest to achieve
 that by having the ParallelismDrivingNode above having a dynamic number
 of nestloop children in different backends and point the coordinated
 seqscan to some shared state.  As you point out, the number of these
 children cannot be certainly known (just targeted for) at plan time;
 that puts a certain limit on how independent they are.  But since a
 large number of them can be independent between workers it seems awkward
 to generally treat them as being the same node across workers. But maybe
 that's just an issue 

Re: [HACKERS] Proposal : REINDEX xxx VERBOSE

2015-02-11 Thread Jim Nasby

On 2/5/15 12:01 PM, Tom Lane wrote:

Robert Haas robertmh...@gmail.com writes:

We've got a mix of styles for extensible options right now:


That we do.


So COPY puts the options at the very end, but EXPLAIN and VACUUM put
them right after the command name.  I prefer the latter style and
would vote to adopt it here.


Meh.  Options-at-the-end seems by far the most sensible style to me.
The options-right-after-the-keyword style is a mess, both logically
and from a parsing standpoint, and the only reason we have it at all
is historical artifact (ask Bruce about the origins of VACUUM ANALYZE
over a beer sometime).


I suspect at least some of this stems from how command line programs 
tend to process options before arguments. I tend to agree with you Tom, 
but I think what's more important is that we're consistent. COPY is 
already a bit of an oddball because it uses WITH, but both EXPLAIN and 
VACUUM use parenthesis immediately after the first verb. Introducing a 
parenthesis version that goes at the end instead of the beginning is 
just going to make this worse.


If we're going to take a stand on this, we need to do it NOW, before we 
have even more commands that use ().


I know you were worried about accepting options anywhere because it 
leads to reserved words, but perhaps we could support it just for 
EXPLAIN and VACUUM, and then switch to trailing options if people think 
that would be better.

--
Jim Nasby, Data Architect, Blue Treble Consulting
Data in Trouble? Get it in Treble! http://BlueTreble.com


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Corner case for add_path_precheck

2015-02-11 Thread Tom Lane
Antonin Houska a...@cybertec.at writes:
 Tom Lane t...@sss.pgh.pa.us wrote:
 Antonin Houska a...@cybertec.at writes:
 The special case is that the path passed to add_path_precheck() has costs
 *equal to* those of the old_path. If pathkeys, outer rells and costs are the
 same, as summarized in the comment above, I expect add_path_precheck() to
 return false. Do I misread anything?

 It does, so I don't see your point?

 Just that pre-check is - in this special (and very rare?) case - more
 stringent than the proper check would be.

It's assuming that a nonzero amount of cost will be added on before the
real check happens.  Even if none was added, discarding the new path is
the way we'd break the tie that would result, so I still don't see your
point.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Index-only scans for GiST.

2015-02-11 Thread Thom Brown
On 11 February 2015 at 22:50, Anastasia Lubennikova lubennikov...@gmail.com
 wrote:

 Finally there is a new version of patch (in attachments).
 It provides multicolumn index-only scan for GiST indexes.

 - Memory leak is fixed.
 - little code cleanup
 - example of performance test in attachmens
 - function OIDs have debugging values (*) just to avoid merge
 conflicts while testing patch

 Wiki page of the project is

 https://wiki.postgresql.org/wiki/Support_for_Index-only_scans_for_GIST_GSoC_2014

 Waiting for feedback.


Hi Anastasia.  Thanks for the updated patch.  I've just tried applying it
to head and it doesn't appear to apply cleanly.

$ patch -p1  ~/Downloads/indexonlyscan_gist_2.0.patch
(Stripping trailing CRs from patch; use --binary to disable.)
patching file src/backend/access/gist/gist.c
Hunk #1 succeeded at 1404 (offset 9 lines).
Hunk #2 succeeded at 1434 (offset 9 lines).
(Stripping trailing CRs from patch; use --binary to disable.)
patching file src/backend/access/gist/gistget.c
Hunk #1 succeeded at 227 (offset 1 line).
Hunk #2 succeeded at 243 (offset 1 line).
Hunk #3 succeeded at 293 (offset -4 lines).
Hunk #4 succeeded at 330 (offset -4 lines).
Hunk #5 succeeded at 365 (offset -5 lines).
Hunk #6 succeeded at 444 (offset -27 lines).
Hunk #7 succeeded at 474 (offset -27 lines).
Hunk #8 FAILED at 518.
Hunk #9 succeeded at 507 (offset -28 lines).
Hunk #10 succeeded at 549 with fuzz 1 (offset -28 lines).
Hunk #11 FAILED at 601.
2 out of 11 hunks FAILED -- saving rejects to file
src/backend/access/gist/gistget.c.rej
...

-- 
Thom


Re: [HACKERS] SSL renegotiation and other related woes

2015-02-11 Thread Andres Freund
On 2015-02-11 14:54:03 +0200, Heikki Linnakangas wrote:
 Thoughts? Can you reproduce any errors with this?

I'm on battery right now, so I can't really test much.

Did you test having an actual standby instead of pg_receivexlog? I saw
some slightly different errors there.

Does this patch alone work for you or did you test it together with the
earlier one to fix the renegotiation handling server side when
nonblocking? Because that frequently seemed to error out for me, at
least over actual network. A latch loop + checking for the states seemed
to fix that for me.

I'm pretty darn happy that you seem to have found a much less ugly
solution than mine. Although it's only pretty by comparison ;)

 + /*
 +  * To work-around an issue with OpenSSL and renegotiation, we don't
 +  * let SSL_write() to read any incoming data.

superflous to.

 +  * NB: This relies on the calling code to call pqsecure_read(), 
 completing
 +  * the renegotiation handshake, if pqsecure_write() returns 0. Otherwise
 +  * we'll never make progress.
 +  */

I think this should a) refer to the fact that pqSendSome does that in
blocking scenarios b) expand the comment around the pqReadData to
reference that fact.

How are we going to deal with callers using libpq in nonblocking mode. A
client doing a large COPY FROM STDIN to the server using a nonblocking
connection (not a stupid thing to do) could hit this IIUC.

Could we perhaps call SSL_peek() when SSL_write() hits WANTS_READ? In
combination with your fix I think that should fix the possibility of
such a deadlock? Especially on the serverside where the socket most of
the time is is in, although emulated, blocking mode that seems easier -
we can just retry afterwards.

Greetings,

Andres Freund

-- 
 Andres Freund http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] SSL renegotiation and other related woes

2015-02-11 Thread Andres Freund
Hi,

On 2015-02-05 23:03:02 +0200, Heikki Linnakangas wrote:
 On 01/26/2015 12:14 PM, Andres Freund wrote:
 Hi,
 
 When working on getting rid of ImmediateInterruptOK I wanted to verify
 that ssl still works correctly. Turned out it didn't. But neither did it
 in master.
 
 Turns out there's two major things we do wrong:
 
 1) We ignore the rule that once called and returning
 SSL_ERROR_WANTS_(READ|WRITE) SSL_read()/write() have to be called
 again with the same parameters. Unfortunately that rule doesn't mean
 just that the same parameters have to be passed in, but also that we
 can't just constantly switch between _read()/write(). Especially
 nonblocking backend code (i.e. walsender) and the whole frontend code
 violate this rule.
 
 I don't buy that. Googling around, and looking at the man pages, I just
 don't see anything to support that claim. I believe it is supported to
 switch between reads and writes.

There's at least two implementations that seem to have workaround to
achieve something like that. Apache's mod_ssl and the ssl layer for
libevent.

 2) We start renegotiations in be_tls_write() while in nonblocking mode,
 but don't properly retry to handle socket readyness. There's a loop
 that retries handshakes twenty times (???), but what actually is
 needed is to call SSL_get_error() and ensure that there's actually
 data available.
 
 Yeah, that's just crazy.
 
 Actually, I don't think we need to call SSL_do_handshake() at all. At least
 on modern OpenSSL versions. OpenSSL internally uses a state machine, and
 SSL_read() and SSL_write() calls will complete the handshake just as well.

Some blaming seems to show that that's been the case for a long time.

 2) is easy enough to fix, but 1) is pretty hard. Before anybody says
 that 1) isn't an important rule: It reliably causes connection aborts
 within a couple renegotiations. The higher the latency the higher the
 likelihood of aborts. Even locally it doesn't take very long to
 abort. Errors usually are something like SSL connection has been closed
 unexpectedly or SSL Error: sslv3 alert unexpected message and a host
 of other similar messages. There's a couple reports of those in the
 archives and I've seen many more in client logs.
 
 Yeah, I can reproduce that. There's clearly something wrong.

 I believe this is an OpenSSL bug. I traced the unexpected message error to
 this code in OpenSSL's s3_read_bytes() function:

Yep, I got to that as well.

 case SSL3_RT_APPLICATION_DATA:
 /*
  * At this point, we were expecting handshake data, but have
  * application data.  If the library was running inside ssl3_read()
  * (i.e. in_read_app_data is set) and it makes sense to read
  * application data at this point (session renegotiation not yet
  * started), we will indulge it.
  */
 if (s-s3-in_read_app_data 
 (s-s3-total_renegotiations != 0) 
 (((s-state  SSL_ST_CONNECT) 
   (s-state = SSL3_ST_CW_CLNT_HELLO_A) 
   (s-state = SSL3_ST_CR_SRVR_HELLO_A)
  ) || ((s-state  SSL_ST_ACCEPT) 
(s-state = SSL3_ST_SW_HELLO_REQ_A) 
(s-state = SSL3_ST_SR_CLNT_HELLO_A)
  )
 )) {
 s-s3-in_read_app_data = 2;
 return (-1);
 } else {
 al = SSL_AD_UNEXPECTED_MESSAGE;
  SSLerr(SSL_F_SSL3_READ_BYTES, SSL_R_UNEXPECTED_RECORD);
 goto f_err;
 }
 
 So that quite clearly throws an error, *unless* the original application
 call was SSL_read(). As you also concluded, OpenSSL doesn't like it when
 SSL_write() is called when it's in renegotiation. I think that's OpenSSL's
 fault and it should indulge the application data message even in
 SSL_write().

That'd be nice, but I think there were multiple reports to them about
this and there wasn't much of a response. Maybe it's time to try again.

 In theory, I guess we should do similar hacks in the server, and always call
 SSL_read() before SSL_write(), but it seems to work without it. Or maybe
 not; OpenSSL server and client code is not symmetric, so perhaps it works in
 server mode without that.

I think we should do it on the server side - got some server side errors
when I cranked up the pg_receivexlog's status interval and set the wal
sender timeout very low.

Greetings,

Andres Freund


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] reducing our reliance on MD5

2015-02-11 Thread Tom Lane
Heikki Linnakangas hlinnakan...@vmware.com writes:
 On 02/11/2015 03:52 PM, Robert Haas wrote:
 So are you thinking to integrate with the Cyrus SASL library, or do
 you have another thought?

 I think we need to implement the primary MD5 replacement ourselves, so 
 that it's always available without extra libraries. Otherwise it will 
 not get much adoption, or the extra dependency will be a hassle anyway. 

+1

 We could also support using a library like that for additional 
 authentication mechanisms, though, for those who really need them.

We've already got a sufficiency of external authentication mechanisms.
If people wanted to use non-built-in authentication, we'd not be having
this discussion.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Corner case for add_path_precheck

2015-02-11 Thread Antonin Houska
Tom Lane t...@sss.pgh.pa.us wrote:
 Antonin Houska a...@cybertec.at writes:
  The special case is that the path passed to add_path_precheck() has costs
  *equal to* those of the old_path. If pathkeys, outer rells and costs are the
  same, as summarized in the comment above, I expect add_path_precheck() to
  return false. Do I misread anything?
 
 It does, so I don't see your point?

Just that pre-check is - in this special (and very rare?) case - more
stringent than the proper check would be.

-- 
Antonin Houska
Cybertec Schönig  Schönig GmbH
Gröhrmühlgasse 26
A-2700 Wiener Neustadt
Web: http://www.postgresql-support.de, http://www.cybertec.at


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] libpq's multi-threaded SSL callback handling is busted

2015-02-11 Thread Andres Freund
On 2015-02-09 18:17:14 +0100, Jan Urbański wrote:
 We added unsetting the locking callback in
 4e816286533dd34c10b368487d4079595a3e1418 due to this bug report:
 http://www.postgresql.org/message-id/48620925.6070...@pws.com.au
 
 Indeed, commenting out the CRYPTO_set_locking_callback(NULL) call in
 fe-secure-openssl.c gets rid of the deadlock. However, it makes php segfault
 with the (attached) reproduction script from the original 2008 bug report. If
 your php.ini loads both the pgsql and curl extensions, reproduce the segfault 
 with:
 
 php -f pg_segfault.php
 
 The most difficult part about fixing this bug is to determine *who's at
 fault*. I now lean towards the opinion that we shouldn't be messing with
 OpenSSL callbacks *at all*.
 
 First of all, the current behaviour is crazy. We're setting and unsetting the
 locking callback every time a connection is made/closed, which is not how
 OpenSSL is supposed to be used. The *application* using libpq should set a
 callback before it starts threads, it's no business of the library's.

I don't buy this argument at all. Delivering a libpq that's not
threadsafe in some circumstances is a really bad idea.

 The old behaviour was slightly less insane (set callbacks first time we're
 engaging OpenSSL code, never touch them again). The real sane solution is to
 leave it up to the application.

The real solution would be for openssl to do it itself.

 I posit we should remove all CRYPTO_set_*_callback functions and associated
 cruft from libpq. This unfortunately means that multi-threaded applications
 using libpq and SSL will break if they haven't been setting their own 
 callbacks
 (if they have, well, tough luck! libpq will just stomp over them the first 
 time
 it connects to Postgres, but at least *some* callbacks are left present after
 that).

I think there's no chance in hell we can do that. Breaking a noticeable
amount of libpq users in a minor upgrade is imo a cure *FAR* worse than
the cure.

 Note that the PHP pgsql extension doesn't set the OpenSSL callbacks, because
 libpq was setting them on its own. If we remove the callback handling from
 libpq, PHP will need to add them. By the way, the MySQL extension for PHP also
 does not set those callbacks.

I think this shows pretty clearly how insane it would be to change this
in a minor release. Do you really expect people to update libpq and php
in unison. After a minor release?

Note that we *already* provide applications with the ability to set the
callbacks themselves and prevent us from doing so: PQinitSSL(false).

 Let me reiterate: I now believe the callbacks should be set by the 
 application,
 libraries should not touch them, since they don't know what will they be
 stomping on. If the application is run through a VM like Python or PHP, it's
 the VM that should make sure the callbacks are set.

I fail to see why it's any better to do it in the VM. That relies on
either always loading the VM's openssl module, even if you don't
actually need it because the library you use deals with openssl. It
prevents other implementations of openssl in the VM.

What I think we should do is the following:

* Emphasize the fact that it's important to use PQinitSSL(false) if you
  did things yourself.
* If there's already callbacks set: Remember that fact and don't
  overwrite. In the next major version: warn.
* Assert or something if the callback when unregistering isn't the same
  as it was when registering. That's pretty much guaranteed to cause
  issues.

 I would very much like to have this change back-patched, since setting and
 resetting the callback makes using libpq in a threaded OpenSSL-enabled app
 arguably less safe than if it didn't use any locking.

Meh^2

Greetings,

Andres Freund

-- 
 Andres Freund http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] relid of non user created tables

2015-02-11 Thread Jim Nasby

On 2/7/15 5:55 AM, Tom Lane wrote:

Deepak S in.live...@live.in writes:

I wish to know if it is safe to assume that non user created tables always have 
a relid of less than 10,000 (I chose this value after trial and error).


Depends what you consider user created.  System catalogs with
manually-assigned OIDs will have OIDs below 10K, but initdb creates
numerous things that don't have manually-assigned OIDs.  And what of,
say, TOAST tables attached to user tables?

You might find the comments for FirstNormalObjectId in
src/include/access/transam.h enlightening.


I think the better question is Why do you want to know this? I suspect 
there's a better way to do whatever you're trying to do.

--
Jim Nasby, Data Architect, Blue Treble Consulting
Data in Trouble? Get it in Treble! http://BlueTreble.com


--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


[HACKERS] Index-only scans for GiST.

2015-02-11 Thread Anastasia Lubennikova
Finally there is a new version of patch (in attachments).
It provides multicolumn index-only scan for GiST indexes.

- Memory leak is fixed.
- little code cleanup
- example of performance test in attachmens
- function OIDs have debugging values (*) just to avoid merge conflicts
while testing patch

Wiki page of the project is
https://wiki.postgresql.org/wiki/Support_for_Index-only_scans_for_GIST_GSoC_2014

Waiting for feedback.
-- 
Best regards,
Lubennikova Anastasia


indexonlyscan_gist_2.0.patch
Description: Binary data


test_performance.sql
Description: Binary data


indexonlyscan_gist_docs.patch
Description: Binary data

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


[HACKERS] Re: [HACKERS] Streaming replication and WAL archive interactions

2015-02-11 Thread Миша Тюрин

  This should be a very common setup in the field, so how are  people doing it 
in practice?

One of possible workaround with archive and streaming was to use pg_receivexlog 
from standby to copy/save WALs to archive. but with pg_receivexlog was also 
issue with fsync.


[ master ] -- streaming -- [ standby ] -- pg_receivexlog -- [ /archive ]


In that case archive is always in pre standby state and it could be better than 
had archive broken on promote.
--
Misha

Re: [HACKERS] Typo in logicaldecoding.sgml

2015-02-11 Thread Andres Freund
On 2015-02-10 20:34:33 -0500, Robert Haas wrote:
 On Tue, Feb 10, 2015 at 6:57 PM, Andres Freund and...@2ndquadrant.com wrote:
  I think one of them could be a fix. Any advice?
 
  I slightly prefer the second form...
 
 I think it uses is definitely more standard English usage in this context.

Pushed it uses.

Thanks to you two.

Andres Freund

-- 
 Andres Freund http://www.2ndQuadrant.com/
 PostgreSQL Development, 24x7 Support, Training  Services


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] reducing our reliance on MD5

2015-02-11 Thread Tom Lane
Stephen Frost sfr...@snowman.net writes:
 * Tom Lane (t...@sss.pgh.pa.us) wrote:
 We've already got a sufficiency of external authentication mechanisms.
 If people wanted to use non-built-in authentication, we'd not be having
 this discussion.

 Just to be clear- lots of people *do* use the external authentication
 mechanisms we provide, particularly Kerberos/GSSAPI.  SASL would bring
 us quite a few additional mechanisms (SQL-based, Berkley DB, one-time
 passwords, RSA SecurID, etc..) and would mean we might be able to
 eventually drop direct GSSAPI and LDAP support and have a better
 alternative for those who want to use password-based auth.  

My point is that we already have got a lot of external authentication
mechanisms, and it's completely unclear (to me anyway) that there is
any demand for another one.  The unsatisfied demand is for a *built in*
mechanism, specifically one that people have more faith in than MD5.
Those who worry about that and don't mind having additional moving parts
have probably already migrated to one or another of the existing external
solutions.

While I won't stand in the way of somebody adding support for an external
SASL library, I think such work has got basically zero to do with the
actual problem.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] reducing our reliance on MD5

2015-02-11 Thread Stephen Frost
* Tom Lane (t...@sss.pgh.pa.us) wrote:
 Stephen Frost sfr...@snowman.net writes:
  * Tom Lane (t...@sss.pgh.pa.us) wrote:
  We've already got a sufficiency of external authentication mechanisms.
  If people wanted to use non-built-in authentication, we'd not be having
  this discussion.
 
  Just to be clear- lots of people *do* use the external authentication
  mechanisms we provide, particularly Kerberos/GSSAPI.  SASL would bring
  us quite a few additional mechanisms (SQL-based, Berkley DB, one-time
  passwords, RSA SecurID, etc..) and would mean we might be able to
  eventually drop direct GSSAPI and LDAP support and have a better
  alternative for those who want to use password-based auth.  
 
 My point is that we already have got a lot of external authentication
 mechanisms, and it's completely unclear (to me anyway) that there is
 any demand for another one.  

To this point, I have had requests for various things which SASL would
provide- ability to use the same password as the Unix host, SecurID
support, and SQL-based (with another PG database) all come to mind.
I agree that such isn't particularly relevant to this discussion though.

 The unsatisfied demand is for a *built in*
 mechanism, specifically one that people have more faith in than MD5.

I agree that there is demand for a new mechanism which isn't MD5.  I
don't think *built in* is at all a requirement however- *easy to use*
is.

 Those who worry about that and don't mind having additional moving parts
 have probably already migrated to one or another of the existing external
 solutions.

Certainly we'd want to make SASL-with-SCRAM easy for users to use.  To
the extent that the moving parts in that arrangement are hidden from
view, hopefully users will use it and we get comfortable enough with it
for distros to set it as the default.

I am catagorically *not* worried about non-libpq client libraries and
applications in this regard.  For starters, those aren't end users,
those are other developers who have a vested interest in supporting
whatever we do.  Further, Java certainly supports SASL, as do quite a
few other languages (Python, PHP, Ruby, there's a few examples in Go it
seems).

 While I won't stand in the way of somebody adding support for an external
 SASL library, I think such work has got basically zero to do with the
 actual problem.

I don't think that's accurate.  If we provide a better mechanism for
password-based authentication, even if it is through SASL, I feel pretty
confident that at least the Debian based distros (which universally
include SASL support for applications that support it) would support it
and might even switch to it for the default (once they were comfortable
with it and enough clients supported it), unless we told them not to.

That doesn't address existing installations, of course, and we'd have to
wait for non-libpq client to update, but things would certainly move
faster if we actually had something for things to move *to*, even if
that thing is SASL-based SCRAM, imv.

Thanks!

Stephen


signature.asc
Description: Digital signature


Re: [HACKERS] pg_upgrade bug in handling postgres/template1 databases

2015-02-11 Thread Bruce Momjian
On Tue, Feb 10, 2015 at 12:52:33PM -0500, Bruce Momjian wrote:
 The attached patch fixes the problem, and I have reproduced the bug and
 verified the patch fixes it.  It would have been nice if I had figured
 this out two weeks ago, before we released minor version upgrades.  This
 bug has existed back to 9.0 or earlier, so it isn't new.

Patch applied back through 9.0.  This bug was reported by an EDB
customer.

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + Everyone has their own god. +


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] reducing our reliance on MD5

2015-02-11 Thread Stephen Frost
* Tom Lane (t...@sss.pgh.pa.us) wrote:
 Heikki Linnakangas hlinnakan...@vmware.com writes:
  We could also support using a library like that for additional 
  authentication mechanisms, though, for those who really need them.
 
 We've already got a sufficiency of external authentication mechanisms.
 If people wanted to use non-built-in authentication, we'd not be having
 this discussion.

Just to be clear- lots of people *do* use the external authentication
mechanisms we provide, particularly Kerberos/GSSAPI.  SASL would bring
us quite a few additional mechanisms (SQL-based, Berkley DB, one-time
passwords, RSA SecurID, etc..) and would mean we might be able to
eventually drop direct GSSAPI and LDAP support and have a better
alternative for those who want to use password-based auth.  

Thanks,

Stephen


signature.asc
Description: Digital signature


Re: [HACKERS] Better error message on pg_upgrade checksum mismatches

2015-02-11 Thread Bruce Momjian
On Tue, Feb 10, 2015 at 10:55:07AM -0500, Greg Sabino Mullane wrote:
 Just a little thing that's been bugging me. If one side of the 
 pg_upgrade has checksums and the other does not, give a less 
 cryptic error message.

Thanks.  Slightly adjusted patch attached and applied to head.

-- 
  Bruce Momjian  br...@momjian.ushttp://momjian.us
  EnterpriseDB http://enterprisedb.com

  + Everyone has their own god. +
diff --git a/contrib/pg_upgrade/controldata.c b/contrib/pg_upgrade/controldata.c
new file mode 100644
index a02a8ec..0e70b6f
*** a/contrib/pg_upgrade/controldata.c
--- b/contrib/pg_upgrade/controldata.c
*** check_control_data(ControlData *oldctrl,
*** 572,581 
  	 * We might eventually allow upgrades from checksum to no-checksum
  	 * clusters.
  	 */
! 	if (oldctrl-data_checksum_version != newctrl-data_checksum_version)
! 	{
! 		pg_fatal(old and new pg_controldata checksum versions are invalid or do not match\n);
! 	}
  }
  
  
--- 572,585 
  	 * We might eventually allow upgrades from checksum to no-checksum
  	 * clusters.
  	 */
! 	if (oldctrl-data_checksum_version == 0 
! 		newctrl-data_checksum_version != 0)
! 		pg_fatal(old cluster does not use data checksums but the new one does\n);
! 	else if (oldctrl-data_checksum_version != 0 
! 			 newctrl-data_checksum_version == 0)
! 		pg_fatal(old cluster uses data checksums but the new one does not\n);
! 	else if (oldctrl-data_checksum_version != newctrl-data_checksum_version)
! 		pg_fatal(old and new cluster pg_controldata checksum versions do not match\n);
  }
  
  

-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] ExplainModifyTarget doesn't work as expected

2015-02-11 Thread Etsuro Fujita
On 2015/02/10 14:49, Etsuro Fujita wrote:
 On 2015/02/07 1:09, Tom Lane wrote:
 IIRC, this code was written at a time when we didn't have NO INHERIT check
 constraints and so it was impossible for the parent table to get optimized
 away while leaving children.  So the comment in ExplainModifyTarget was
 good at the time.  But it no longer is.

 I think your basic idea of preserving the original parent table's relid
 is correct; but instead of doing it like this patch does, I'd be inclined
 to make ModifyTable inherit from Scan not Plan, and use the scan.scanrelid
 field to carry the parent RTI.  Then you would probably end up with a net
 savings of code rather than net addition; certainly ExplainModifyTarget
 would go away entirely since you'd just treat ModifyTable like any other
 Scan in this part of EXPLAIN.
 
 Will follow your revision.

Done.  Attached is an updated version of the patch.

Best regards,
Etsuro Fujita
*** a/src/backend/commands/explain.c
--- b/src/backend/commands/explain.c
***
*** 95,101  static const char *explain_get_index_name(Oid indexId);
  static void ExplainIndexScanDetails(Oid indexid, ScanDirection indexorderdir,
  		ExplainState *es);
  static void ExplainScanTarget(Scan *plan, ExplainState *es);
- static void ExplainModifyTarget(ModifyTable *plan, ExplainState *es);
  static void ExplainTargetRel(Plan *plan, Index rti, ExplainState *es);
  static void show_modifytable_info(ModifyTableState *mtstate, ExplainState *es);
  static void ExplainMemberNodes(List *plans, PlanState **planstates,
--- 95,100 
***
*** 724,736  ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
  		case T_WorkTableScan:
  		case T_ForeignScan:
  		case T_CustomScan:
- 			*rels_used = bms_add_member(*rels_used,
- 		((Scan *) plan)-scanrelid);
- 			break;
  		case T_ModifyTable:
- 			/* cf ExplainModifyTarget */
  			*rels_used = bms_add_member(*rels_used,
! 	  linitial_int(((ModifyTable *) plan)-resultRelations));
  			break;
  		default:
  			break;
--- 723,731 
  		case T_WorkTableScan:
  		case T_ForeignScan:
  		case T_CustomScan:
  		case T_ModifyTable:
  			*rels_used = bms_add_member(*rels_used,
! 		((Scan *) plan)-scanrelid);
  			break;
  		default:
  			break;
***
*** 1067,1072  ExplainNode(PlanState *planstate, List *ancestors,
--- 1062,1068 
  		case T_WorkTableScan:
  		case T_ForeignScan:
  		case T_CustomScan:
+ 		case T_ModifyTable:
  			ExplainScanTarget((Scan *) plan, es);
  			break;
  		case T_IndexScan:
***
*** 1101,1109  ExplainNode(PlanState *planstate, List *ancestors,
  	ExplainPropertyText(Index Name, indexname, es);
  			}
  			break;
- 		case T_ModifyTable:
- 			ExplainModifyTarget((ModifyTable *) plan, es);
- 			break;
  		case T_NestLoop:
  		case T_MergeJoin:
  		case T_HashJoin:
--- 1097,1102 
***
*** 2104,2127  ExplainScanTarget(Scan *plan, ExplainState *es)
  }
  
  /*
-  * Show the target of a ModifyTable node
-  */
- static void
- ExplainModifyTarget(ModifyTable *plan, ExplainState *es)
- {
- 	Index		rti;
- 
- 	/*
- 	 * We show the name of the first target relation.  In multi-target-table
- 	 * cases this should always be the parent of the inheritance tree.
- 	 */
- 	Assert(plan-resultRelations != NIL);
- 	rti = linitial_int(plan-resultRelations);
- 
- 	ExplainTargetRel((Plan *) plan, rti, es);
- }
- 
- /*
   * Show the target relation of a scan or modify node
   */
  static void
--- 2097,2102 
*** a/src/backend/nodes/copyfuncs.c
--- b/src/backend/nodes/copyfuncs.c
***
*** 120,125  CopyPlanFields(const Plan *from, Plan *newnode)
--- 120,140 
  }
  
  /*
+  * CopyScanFields
+  *
+  *		This function copies the fields of the Scan node.  It is used by
+  *		all the copy functions for classes which inherit from Scan.
+  */
+ static void
+ CopyScanFields(const Scan *from, Scan *newnode)
+ {
+ 	CopyPlanFields((const Plan *) from, (Plan *) newnode);
+ 
+ 	COPY_SCALAR_FIELD(scanrelid);
+ }
+ 
+ 
+ /*
   * _copyPlan
   */
  static Plan *
***
*** 168,174  _copyModifyTable(const ModifyTable *from)
  	/*
  	 * copy node superclass fields
  	 */
! 	CopyPlanFields((const Plan *) from, (Plan *) newnode);
  
  	/*
  	 * copy remainder of node
--- 183,189 
  	/*
  	 * copy node superclass fields
  	 */
! 	CopyScanFields((const Scan *) from, (Scan *) newnode);
  
  	/*
  	 * copy remainder of node
***
*** 306,325  _copyBitmapOr(const BitmapOr *from)
  
  
  /*
-  * CopyScanFields
-  *
-  *		This function copies the fields of the Scan node.  It is used by
-  *		all the copy functions for classes which inherit from Scan.
-  */
- static void
- CopyScanFields(const Scan *from, Scan *newnode)
- {
- 	CopyPlanFields((const Plan *) from, (Plan *) newnode);
- 
- 	COPY_SCALAR_FIELD(scanrelid);
- }
- 
- /*
   * _copyScan
   */
  static Scan *
--- 321,326 
*** a/src/backend/nodes/outfuncs.c
--- 

Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes

2015-02-11 Thread Michael Paquier
On Wed, Feb 11, 2015 at 11:03 PM, Syed, Rahila rahila.s...@nttdata.com
wrote:

 IMO, we should add details about how this new field is used in the
 comments on top of XLogRecordBlockImageHeader, meaning that when a page
 hole is present we use the compression info structure and when there is no
 hole, we are sure that the FPW raw length is BLCKSZ meaning that the two
 bytes of the CompressionInfo stuff is unnecessary.
 This comment is included in the patch attached.

  For correctness with_hole should be set even for uncompressed pages. I
 think that we should as well use it for sanity checks in xlogreader.c when
 decoding records.
 This change is made in the attached patch. Following sanity checks have
 been added in xlogreader.c

 if (!(blk-with_hole)  blk-hole_offset != 0 || blk-with_hole 
 blk-hole_offset = 0))

 if (blk-with_hole  blk-bkp_len = BLCKSZ)

 if (!(blk-with_hole)  blk-bkp_len != BLCKSZ)


Cool, thanks!

This patch fails to compile:
xlogreader.c:1049:46: error: extraneous ')' after condition, expected a
statement
blk-with_hole  blk-hole_offset
= 0))

Note as well that at least clang does not like much how the sanity check
with with_hole are done. You should place parentheses around the ''
expressions. Also, I would rather define with_hole == 0 or with_hole == 1
explicitly int those checks.

There is a typo:
s/true,see/true, see/

[nitpicky]Be as well aware of the 80-character limit per line that is
usually normally by comment blocks.[/]

+ * with_hole is used to identify the presence of hole in a block.
+ * As mentioned above, length of block cannnot be more than 15-bit long.
+ * So, the free bit in the length field is used by with_hole to identify
presence of
+ * XLogRecordBlockImageCompressionInfo. If hole is not present ,the raw
size of
+ * a compressed block is equal to BLCKSZ therefore
XLogRecordBlockImageCompressionInfo
+ * for the corresponding compressed block need not be stored in header.
+ * If hole is present raw size is stored.
I would rewrite this paragraph as follows, fixing the multiple typos:
with_hole is used to identify the presence of a hole in a block image. As
the length of a block cannot be more than 15-bit long, the extra bit in the
length field is used for this identification purpose. If the block image
has no hole, it is ensured that the raw size of a compressed block image is
equal to BLCKSZ, hence the contents of XLogRecordBlockImageCompressionInfo
are not necessary.

+   /* Followed by the data related to compression if block is
compressed */
This comment needs to be updated to if block image is compressed and has a
hole.

+   lp_off and lp_len fields in ItemIdData (see include/storage/itemid.h)
and
+   XLogRecordBlockImageHeader where page hole offset and length is limited
to 15-bit
+   length (see src/include/access/xlogrecord.h).
80-character limit...

Regards
-- 
Michael


Re: [HACKERS] The return value of allocate_recordbuf()

2015-02-11 Thread Michael Paquier
On Wed, Feb 11, 2015 at 2:13 AM, Robert Haas robertmh...@gmail.com wrote:

 On Mon, Feb 9, 2015 at 7:02 AM, Michael Paquier
 michael.paqu...@gmail.com wrote:
  On Mon, Feb 9, 2015 at 7:58 PM, Fujii Masao masao.fu...@gmail.com
 wrote:
  MemoryContextAllocExtended() was added, so isn't it time to replace
 palloc()
  with MemoryContextAllocExtended(CurrentMemoryContext, MCXT_ALLOC_NO_OOM)
  in allocate_recordbuf()?
 
  Yeah, let's move on here, but not with this patch I am afraid as this
  breaks any client utility using xlogreader.c in frontend, like
  pg_xlogdump or pg_rewind because MemoryContextAllocExtended is not
  available in frontend, only in backend. We are going to need something
  like palloc_noerror, defined on both backend (mcxt.c) and frontend
  (fe_memutils.c) side.

 Unfortunately, that gets us back into the exact terminological dispute
 that we were hoping to avoid.  Perhaps we could compromise on
 palloc_extended().


Yes, why not using palloc_extended instead of palloc_noerror that has been
clearly rejected in the other thread. Now, for palloc_extended we should
copy the flags of MemoryContextAllocExtended to fe_memutils.h and have the
same interface between frontend and backend (note that MCXT_ALLOC_HUGE has
no real utility for frontends). Attached is a patch to achieve that,
completed with a second patch for the problem related to this thread. Note
that in the second patch I have added an ERROR in logical.c after calling
XLogReaderAllocate, this was missing..

 Btw, the huge flag should be used as well as palloc only allows
  allocation up to 1GB, and this is incompatible with ~9.4 where malloc
  is used.

 I think that flag should be used only if it's needed, not just on the
 grounds that 9.4 has no such limit.  In general, limiting allocations
 to 1GB has been good to us; for example, it prevents a corrupted TOAST
 length from causing a memory allocation large enough to crash the
 machine (yes, there are machines you can crash with a giant memory
 allocation!).  We should bypass that limit only where it is clearly
 necessary.


Fine for me to error out in this code path if we try more than 1GB of
allocation.
Regards,
-- 
Michael
From b9ff4e9ffdc9b02d1ee372c63ee36bfe7659ee9f Mon Sep 17 00:00:00 2001
From: Michael Paquier mich...@otacoo.com
Date: Thu, 12 Feb 2015 03:36:55 +0900
Subject: [PATCH 1/2] Add palloc_extended for frontend and backend

This interface can be used to control at a lower level memory allocation
using an interface similar to MemoryContextAllocExtended, but this time
applied to CurrentMemoryContext on backend side, while on frontend side
a similar interface is available.
---
 src/backend/utils/mmgr/mcxt.c| 37 +
 src/common/fe_memutils.c | 36 ++--
 src/include/common/fe_memutils.h | 11 ++-
 src/include/utils/palloc.h   |  1 +
 4 files changed, 74 insertions(+), 11 deletions(-)

diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index 202bc78..2589f6a 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -811,6 +811,43 @@ palloc0(Size size)
 	return ret;
 }
 
+void *
+palloc_extended(Size size, int flags)
+{
+	/* duplicates MemoryContextAllocExtended to avoid increased overhead */
+	void	   *ret;
+
+	AssertArg(MemoryContextIsValid(CurrentMemoryContext));
+	AssertNotInCriticalSection(CurrentMemoryContext);
+
+	if (((flags  MCXT_ALLOC_HUGE) != 0  !AllocHugeSizeIsValid(size)) ||
+		((flags  MCXT_ALLOC_HUGE) == 0  !AllocSizeIsValid(size)))
+		elog(ERROR, invalid memory alloc request size %zu, size);
+
+	CurrentMemoryContext-isReset = false;
+
+	ret = (*CurrentMemoryContext-methods-alloc) (CurrentMemoryContext, size);
+	if (ret == NULL)
+	{
+		if ((flags  MCXT_ALLOC_NO_OOM) == 0)
+		{
+			MemoryContextStats(TopMemoryContext);
+			ereport(ERROR,
+	(errcode(ERRCODE_OUT_OF_MEMORY),
+	 errmsg(out of memory),
+	 errdetail(Failed on request of size %zu., size)));
+		}
+		return NULL;
+	}
+
+	VALGRIND_MEMPOOL_ALLOC(CurrentMemoryContext, ret, size);
+
+	if ((flags  MCXT_ALLOC_ZERO) != 0)
+		MemSetAligned(ret, 0, size);
+
+	return ret;
+}
+
 /*
  * pfree
  *		Release an allocated chunk.
diff --git a/src/common/fe_memutils.c b/src/common/fe_memutils.c
index 345221e..bd60cc2 100644
--- a/src/common/fe_memutils.c
+++ b/src/common/fe_memutils.c
@@ -19,8 +19,8 @@
 
 #include postgres_fe.h
 
-void *
-pg_malloc(size_t size)
+static inline void *
+pg_malloc_extended(size_t size, int flags)
 {
 	void	   *tmp;
 
@@ -28,22 +28,32 @@ pg_malloc(size_t size)
 	if (size == 0)
 		size = 1;
 	tmp = malloc(size);
-	if (!tmp)
+
+	if (tmp == NULL)
 	{
-		fprintf(stderr, _(out of memory\n));
-		exit(EXIT_FAILURE);
+		if ((flags  MCXT_ALLOC_NO_OOM) == 0)
+		{
+			fprintf(stderr, _(out of memory\n));
+			exit(EXIT_FAILURE);
+		}
+		return NULL;
 	}
+
+	if ((flags  MCXT_ALLOC_ZERO) != 0)
+		MemSet(tmp, 0, size);
 	return tmp;
 }
 
 void *

Re: [HACKERS] reducing our reliance on MD5

2015-02-11 Thread Heikki Linnakangas

On 02/11/2015 05:57 AM, Tom Lane wrote:

In any case, my larger point was that given the pain that we're going to
incur here, and the certainly years-long transition interval involved,
it would be foolish to think only about replacing the MD5 algorithm and
not about reconsidering the context we use it in.  Stuff like unreasonably
short salt values should be dealt with at the same time.


+1. We should pick a well-documented mechanism with well-understood 
properties, rather than roll our own again.


Apart from the weaknesses in the MD5 authentication method itself, our 
protocol doesn't make it easy to add new methods and still support old 
clients gracefully. We should address that too.


For the first cut, I propose:

1. Implement SASL. The server sends a new AuthenticationSASLRequest 
message, to which the client responds with AuthenticationSASLResponse 
message. These messages carry the SASL challenge/response messages, 
until AuthenticationOK is received. Similar to our GSSAPI 
implementation. (Note that this doesn't require the use of any 3rd party 
libraries or such. The specification for SASL itself is very short and 
high-level.)


2. Add a way for the client and server to negotiate which authentication 
mechanism to use. Currently, the server dictates it to the client, and 
if it doesn't support that mechanism, it has to disconnect. To make it 
possible to add new mechanisms gracefully, with a long transition 
period, we need a way to negotiate. The server should send a list of 
allowed authentication methods in the first AuthenticationSASLRequest 
message, and the client picks one.


3. To support old clients that don't understand the new 
AuthenticationSASLRequest message, the client tells that it supports it 
before the authentication begins. It does that by tacking a special 
option protocol.extensions=sasl in the startup packet (more extensions 
could be added there in the future, as a comma-separated list). With 9.2 
and above servers, the server will ignore unrecognized options 
containing a dot. 9.1 and earlier will throw an error, but the client 
can then retry without the option, like libpq does for application_name.


4. Implement the SASL mechanism SCRAM. That's a challenge/response 
password scheme, similar to the current MD5 authentication, but with 
better salt and more expensive computation (= more resistance to 
dictionary attacks).


5. Modify/add syntax to ALTER USER SET PASSWORD to allow storing the new 
SCRAM, and other, authentication information in pg_authid.


That's it for starters. The SCRAM mechanism is a fairly straightforward 
replacement for MD5. This scheme makes it possible to add more 
mechanisms later, without requiring all clients to immediately support them.


- Heikki



--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] GSoC 2015 - mentors, students and admins.

2015-02-11 Thread Вадим Грибанов
Hi Thom!

I'm interested to participate as student.
On 9 Feb 2015 23:54, Thom Brown t...@linux.com wrote:

 Hi all,

 Google Summer of Code 2015 is approaching.  I'm intending on registering
 PostgreSQL again this year.

 Before I do that, I'd like to have an idea of how many people are
 interested in being either a student or a mentor.

 I've volunteered to be admin again, but if anyone else has a strong
 interest of seeing the projects through this year, please let yourself be
 known.

 Who would be willing to mentor projects this year?

 Project ideas are welcome, and I've copied many from last year's
 submissions into this year's wiki page.  Please feel free to add more (or
 delete any that stand no chance or are redundant):
 https://wiki.postgresql.org/wiki/GSoC_2015

 Students can find more information about GSoC here:
 http://www.postgresql.org/developer/summerofcode

 Thanks

 Thom



Re: [HACKERS] reducing our reliance on MD5

2015-02-11 Thread Heikki Linnakangas

On 02/11/2015 04:38 AM, Tom Lane wrote:

Peter Eisentraut pete...@gmx.net writes:

On 2/10/15 8:28 PM, Robert Haas wrote:

I don't actually care which algorithm we use, and I dowannahafta care.
What I do want to do is provide a framework so that, when somebody
discovers that X is better than Y because Z, somebody who knows about
cryptography and not much about PostgreSQL ca add support for X in a
relatively small number of lines of code.



sounds like SASL


Sounds like pie in the sky really :-(.


SASL won't automatically solve any of ourproblems. We still have to 
write the support for all the authentication mechanisms we wanted to 
support. The nice thing about SASL is there is a list of well-understood 
and documented mechanisms that go with it, with canonical names like 
SCRAM-SHA-1. If we use those, it's well-defined what the mechanism is, 
with less design work for us. And who knows, it just might make it 
easier to add client-support, if there's an existing SASL library for 
language X.


So +1 for implementing SASL. It's simple to implement, and AFAICS isn't 
any more difficult than rolling our own.



 We could make the server turn on
a dime perhaps, but the client-side population will not come along nearly
that quickly, nor with small effort.  Stored passwords won't migrate to a
new scheme transparently either.


Yep.


I think it's probably reasonable to think about a more modern password
auth method, but not to imagine that it will be pluggable or that the
adoption time for any revision will be less than years long.


It makes sense to make it pluggable if it's simple, but in reality most 
drivers will support only a few most common mechanisms. That's OK. We 
need to design the protocol so that multiple mechanisms can be used 
side-by-side for years.


- Heikki



--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] What exactly is our CRC algorithm?

2015-02-11 Thread Abhijit Menon-Sen
At 2015-02-10 14:30:51 +0200, hlinnakan...@vmware.com wrote:

 I propose that we add a new header file in src/port, let's call it
 crc_instructions.h […] the point of the crc_instructions.h header
 is to hide the differences between compilers and architectures.

Moving the assembly code/compiler intrinsics to a separate header is
fine with me, but I really don't like the proposed implementation at
all. Here are some comments:

1. I don't mind moving platform-specific tests for CRC32C instructions
   to configure, but if we only define PG_HAVE_CRC32C_INSTRUCTIONS, we
   would anyway have to reproduce all that platform-specific stuff in
   the header file. To do it properly, I think we should generate the
   right version of crc_instructions.h for the platform. Otherwise,
   what's the point? Might as well have only the complicated header.

2. At this point, I think we should stick with the _sse function rather
   than a generic _hw one to drive the platform-specific instructions.
   The structure of that function (n/8*(8bytes)+n%8*(bytes)) is specific
   to what works best for the SSE instructions. On another platform, we
   may need something very similar, or very different.
   
   With the proposed structure, _hw would inevitably become #ifdef'ed
   non-trivially, e.g. to run a single-byte loop at the start to align
   the main loop, but only for certain combinations of #defines.
   
   I think we should consider what makes the most sense when someone
   actually wants to add support for another platform/compiler, and
   not before.

3. I dislike everything about pg_crc32_instructions_runtime_check()—the
   name, the separate PG_CRC32C_INSTRUCTIONS_NEED_RUNTIME_CHECK #define,
   the fact that you try to do the test inline rather than at startup…

   Maybe you're trying to avoid checking separately at startup so that a
   program that links with code from src/common doesn't need to do its
   own test. But I don't think the results are worth it.

   As for omitting the slicing-by-8 tables, if there's a more compelling
   reason to do that than Gentoo users might like that ;-), I think it
   should be done with a straightforward -DUSE_CRC_SSE style arrangement
   rather than figuring out that you HAVE_CRC32C_INSTRUCTIONS but don't
   NEED_RUNTIME_CHECK. If you want to build special-purpose binaries,
   just pick whichever CRC implementation you like, done.

 I believe the CRC instructions are also available in 32-bit mode, so
 the check for __x86_64__ is not needed. Right?

I'm not sure, but I guess you're right.

 It would be nice to use GCC's builtin intrinsics,
 __builtin_ia32_crc32* where available, but I guess those are only
 available if you build with -msse4.2, and that would defeat the
 point of the runtime check.

Exactly.

-- Abhijit


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] What exactly is our CRC algorithm?

2015-02-11 Thread Heikki Linnakangas

On 02/11/2015 11:28 AM, Abhijit Menon-Sen wrote:

1. I don't mind moving platform-specific tests for CRC32C instructions
to configure, but if we only define PG_HAVE_CRC32C_INSTRUCTIONS, we
would anyway have to reproduce all that platform-specific stuff in
the header file. To do it properly, I think we should generate the
right version of crc_instructions.h for the platform. Otherwise,
what's the point? Might as well have only the complicated header.


I don't follow. I didn't change configure at all, compared to your patch.


2. At this point, I think we should stick with the _sse function rather
than a generic _hw one to drive the platform-specific instructions.
The structure of that function (n/8*(8bytes)+n%8*(bytes)) is specific
to what works best for the SSE instructions. On another platform, we
may need something very similar, or very different.

With the proposed structure, _hw would inevitably become #ifdef'ed
non-trivially, e.g. to run a single-byte loop at the start to align
the main loop, but only for certain combinations of #defines.

I think we should consider what makes the most sense when someone
actually wants to add support for another platform/compiler, and
not before.


Hmm, ok. The ARM CRC32C instruction is very similar to the SSE4.2 one, 
but I presume it does require alignment.



3. I dislike everything about pg_crc32_instructions_runtime_check()—the
name, the separate PG_CRC32C_INSTRUCTIONS_NEED_RUNTIME_CHECK #define,
the fact that you try to do the test inline rather than at startup…

Maybe you're trying to avoid checking separately at startup so that a
program that links with code from src/common doesn't need to do its
own test. But I don't think the results are worth it.


As a case in point, with your patch pg_xlogdump would not have used the 
CRC instruction, because it never called pg_choose_crc_impl(). Choose 
on first use is much more convenient than requiring every program to 
call a function.



As for omitting the slicing-by-8 tables, if there's a more compelling
reason to do that than Gentoo users might like that ;-), I think it
should be done with a straightforward -DUSE_CRC_SSE style arrangement
rather than figuring out that you HAVE_CRC32C_INSTRUCTIONS but don't
NEED_RUNTIME_CHECK. If you want to build special-purpose binaries,
just pick whichever CRC implementation you like, done.


I stopped short of actually allowing you to force the use of SSE, e.g. 
with -DUSE_CRC_SSE, but my thinking was that that would force 
PG_HAVE_CRC32C_INSTRUCTIONS to be set, and NEEDS_RUNTIME_CHECK to be 
unset. I.e. those two #defines control what code is generated, but in 
the header file. I think what you're suggesting is that we'd instead 
have two mutually exclusive #defines, something like 
USE_CRC_SSE_ALWAYS and USE_CRC_SSE_WITH_RUNTIME_CHECK. It wouldn't 
be much different, but would require some places to check for both.



It would be nice to use GCC's builtin intrinsics,
__builtin_ia32_crc32* where available, but I guess those are only
available if you build with -msse4.2, and that would defeat the
point of the runtime check.


Exactly.


Hmm. Perhaps we should check for the built-in functions, and if they're 
available, use them and not set NEEDS_RUNTIME_CHECK. If you compile with 
-msse4.2, the code won't run without SSE4.2 anyway (or at least isn't 
guaranteed to run).


- Heikki



--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] reducing our reliance on MD5

2015-02-11 Thread Robert Haas
On Wed, Feb 11, 2015 at 4:13 AM, Heikki Linnakangas
hlinnakan...@vmware.com wrote:
 For the first cut, I propose:

 1. Implement SASL. The server sends a new AuthenticationSASLRequest message,
 to which the client responds with AuthenticationSASLResponse message. These
 messages carry the SASL challenge/response messages, until AuthenticationOK
 is received. Similar to our GSSAPI implementation. (Note that this doesn't
 require the use of any 3rd party libraries or such. The specification for
 SASL itself is very short and high-level.)

 2. Add a way for the client and server to negotiate which authentication
 mechanism to use. Currently, the server dictates it to the client, and if it
 doesn't support that mechanism, it has to disconnect. To make it possible to
 add new mechanisms gracefully, with a long transition period, we need a way
 to negotiate. The server should send a list of allowed authentication
 methods in the first AuthenticationSASLRequest message, and the client picks
 one.

 3. To support old clients that don't understand the new
 AuthenticationSASLRequest message, the client tells that it supports it
 before the authentication begins. It does that by tacking a special option
 protocol.extensions=sasl in the startup packet (more extensions could be
 added there in the future, as a comma-separated list). With 9.2 and above
 servers, the server will ignore unrecognized options containing a dot. 9.1
 and earlier will throw an error, but the client can then retry without the
 option, like libpq does for application_name.

 4. Implement the SASL mechanism SCRAM. That's a challenge/response
 password scheme, similar to the current MD5 authentication, but with better
 salt and more expensive computation (= more resistance to dictionary
 attacks).

 5. Modify/add syntax to ALTER USER SET PASSWORD to allow storing the new
 SCRAM, and other, authentication information in pg_authid.

 That's it for starters. The SCRAM mechanism is a fairly straightforward
 replacement for MD5. This scheme makes it possible to add more mechanisms
 later, without requiring all clients to immediately support them.

So, this all sounds fairly nice if somebody's willing to do the work,
but I can't help noticing that you originally proposed adopting SCRAM
in 2012, and it's 2015 now.  So I wonder if anyone's really going to
do all this work, and if not, whether we should go for something
simpler.  Just plugging something else in for MD5 would be a lot less
work for us to implement and for clients to support, even if it is (as
it unarguably is) less elegant.

-- 
Robert Haas
EnterpriseDB: http://www.enterprisedb.com
The Enterprise PostgreSQL Company


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] reducing our reliance on MD5

2015-02-11 Thread Álvaro Hernández Tortosa


On 11/02/15 02:30, Tom Lane wrote:

[...]


I think it would be wise to take two steps back and think about what
the threat model is here, and what we actually need to improve.
Offhand I can remember two distinct things we might wish to have more
protection against:

* scraping of passwords off the wire protocol (but is that still
a threat in an SSL world?).  Better salting practice would do more
than replacing the algorithm as such for this, IMO.


mitm

We might consider it our problem or not, but in general terms 
man-in-the-middle attacks, which are easy to implement in many 
scenarios, are a scraping problem. In particular, I have seen tons of 
developers turn off SSL validation during development and not turning 
back it on for production, leaving servers vulnerable to password 
scraping under mitm attacks. So I would always considering hashing anyway.


SCRAM seems to be a good solution anyway.

Regards,

Álvaro




--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] GSoC 2015 - mentors, students and admins.

2015-02-11 Thread Anastasia Lubennikova
I'm interested to participate as student again.


-- 
Best regards,
Lubennikova Anastasia


Re: [HACKERS] SSL renegotiation and other related woes

2015-02-11 Thread Heikki Linnakangas

On 02/05/2015 11:03 PM, Heikki Linnakangas wrote:

On 01/26/2015 12:14 PM, Andres Freund wrote:
Can we work-around that easily? I believe we can. The crucial part is
that we mustn't let SSL_write() to process any incoming application
data. We can achieve that if we always call SSL_read() to drain such
data, before calling SSL_write(). But that's not quite enough; if we're
in renegotiation, SSL_write() might still try to read more data from the
socket that has arrived after the SSL_read() call. Fortunately, we can
easily prevent that by hacking pqsecure_raw_read() to always return
EWOULDBLOCK, when it's called through SSL_write().

The attached patch does that. I've been running your pg_receivexlog test
case with this for about 15 minutes without any errors now, with
ssl_renegotiation_limit=50kB, while before it errored out within seconds.


Here is a simplified version of this patch. It seems to be enough to not 
let SSL_write() to read any data from the socket. No need to call 
SSL_read() first, and the server-side changes I made were only needed 
because of the other patch I had applied.


Thoughts? Can you reproduce any errors with this?


In theory, I guess we should do similar hacks in the server, and always
call SSL_read() before SSL_write(), but it seems to work without it. Or
maybe not; OpenSSL server and client code is not symmetric, so perhaps
it works in server mode without that.


Not included in this patch, but I believe we apply a similar patch to 
the server-side too.


- Heikki

From b78bfbb70127cbe3f360458eb2a97dcc28194fbc Mon Sep 17 00:00:00 2001
From: Heikki Linnakangas heikki.linnakangas@iki.fi
Date: Thu, 5 Feb 2015 22:44:58 +0200
Subject: [PATCH 1/1] Fix sslv3 alert unexpected message errors in SSL
 renegotiation.

There seems to be a bug in OpenSSL renegotiation, so that when SSL_write()
needs to read data to complete the handshake, but it receives application
data instead, it gets confused and throws an unexpected message error. To
work around that, don't let SSL_write() to read data from the socket, even
if some is available. Arrange so that SSL_read() processes all incoming
messages instead.
---
 src/interfaces/libpq/fe-misc.c   | 14 +-
 src/interfaces/libpq/fe-secure-openssl.c | 20 
 src/interfaces/libpq/fe-secure.c |  9 +
 src/interfaces/libpq/libpq-int.h |  2 ++
 4 files changed, 40 insertions(+), 5 deletions(-)

diff --git a/src/interfaces/libpq/fe-misc.c b/src/interfaces/libpq/fe-misc.c
index 945e283..9dd02d5 100644
--- a/src/interfaces/libpq/fe-misc.c
+++ b/src/interfaces/libpq/fe-misc.c
@@ -920,11 +920,15 @@ pqSendSome(PGconn *conn, int len)
 			 * to clear the channel eventually because it's blocked trying to
 			 * send data to us.  (This can happen when we are sending a large
 			 * amount of COPY data, and the server has generated lots of
-			 * NOTICE responses.)  To avoid a deadlock situation, we must be
-			 * prepared to accept and buffer incoming data before we try
-			 * again.  Furthermore, it is possible that such incoming data
-			 * might not arrive until after we've gone to sleep.  Therefore,
-			 * we wait for either read ready or write ready.
+			 * NOTICE responses.)  Another scenario is that SSL renegotiation
+			 * is in progress, and the SSL library needs to read a message
+			 * to complete the handshake, before it can send more data.
+			 *
+			 * To avoid a deadlock situation, we must be prepared to accept
+			 * and buffer incoming data before we try again.  Furthermore, it
+			 * is possible that such incoming data might not arrive until
+			 * after we've gone to sleep.  Therefore, we wait for either read
+			 * ready or write ready.
 			 */
 			if (pqReadData(conn)  0)
 			{
diff --git a/src/interfaces/libpq/fe-secure-openssl.c b/src/interfaces/libpq/fe-secure-openssl.c
index a32af34..2e75f3c 100644
--- a/src/interfaces/libpq/fe-secure-openssl.c
+++ b/src/interfaces/libpq/fe-secure-openssl.c
@@ -319,8 +319,28 @@ pgtls_write(PGconn *conn, const void *ptr, size_t len)
 	char		sebuf[256];
 	int			err;
 
+	/*
+	 * To work-around an issue with OpenSSL and renegotiation, we don't
+	 * let SSL_write() to read any incoming data.
+	 *
+	 * If SSL_write() is called, and renegotiation has just been inititated,
+	 * SSL_write() might try to read from the socket to complete the
+	 * handshake. If there was some application data in-flight, it might
+	 * receive the application data instead. That confuses it, and it throws
+	 * an sslv3 alert unexpected message error.
+	 *
+	 * We avoid that setting a kill-switch, pqsecure_block_raw_read, which
+	 * tells pqsecure_raw_read() to not return any data to the caller, even
+	 * if some is available.
+	 *
+	 * NB: This relies on the calling code to call pqsecure_read(), completing
+	 * the renegotiation handshake, if pqsecure_write() returns 0. Otherwise
+	 * we'll never make progress.
+	 */
 	SOCK_ERRNO_SET(0);
+	pqsecure_block_raw_read = true;
 	n = 

Re: [HACKERS] reducing our reliance on MD5

2015-02-11 Thread Heikki Linnakangas

On 02/11/2015 02:49 PM, Robert Haas wrote:

So, this all sounds fairly nice if somebody's willing to do the work,
but I can't help noticing that you originally proposed adopting SCRAM
in 2012, and it's 2015 now.  So I wonder if anyone's really going to
do all this work, and if not, whether we should go for something
simpler.  Just plugging something else in for MD5 would be a lot less
work for us to implement and for clients to support, even if it is (as
it unarguably is) less elegant.


Just plugging something else in for MD5 would still be a fair amount 
of work. Not that much less than the full program I proposed.


Well, I guess it's easier if you immediately stop supporting MD5, have a 
flag day in all clients to implement the replacement, and break 
pg_dump/restore of passwords in existing databases. That sounds 
horrible. Let's do this properly. I can help with that, although I don't 
know if I'll find the time and enthusiasm to do all of it alone.


- Heikki



--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] Manipulating complex types as non-contiguous structures in-memory

2015-02-11 Thread Tom Lane
Jim Nasby jim.na...@bluetreble.com writes:
 On 2/10/15 5:19 PM, Tom Lane wrote:
 What do you mean by non-variant?

 Ugh, sorry, brainfart. I meant to say non-varlena.

 I can't think of any non-varlena types we'd want this for, but maybe 
 someone else can think of a case. If there is a use-case I wouldn't 
 handle it with this patch, but we'd want to consider it...

There isn't any practical way to interpose TOAST pointers for non-varlena
types, since we make no assumptions about the bit contents of fixed-length
types.  But I'm having a hard time thinking of a fixed-length type in
which you'd have any need for a deserialized representation, either.
I think restricting this feature to varlena types is just fine.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] reducing our reliance on MD5

2015-02-11 Thread Magnus Hagander
On Wed, Feb 11, 2015 at 4:57 AM, Tom Lane t...@sss.pgh.pa.us wrote:

 Robert Haas robertmh...@gmail.com writes:
  On Tue, Feb 10, 2015 at 9:30 PM, Tom Lane t...@sss.pgh.pa.us wrote:
  Another thing we need to keep in mind besides client compatibility
  is dump/reload compatibility.

  I don't think there's an issue with dump/reload compatibility as far
  as what I proposed, since it only has to do with the authentication
  procedure, not what gets stored in pg_authid.  We might have reasons
  for moving that away from MD5 as well, but it's a separate project.

 Hm, well, that doesn't really square with your other expressed opinion:

  Are there other goals?

  I think the goal is stop using MD5, or at least have an option to not
  use MD5, because people think that's insecure.

 As you say, it's quite debatable whether MD5 is or isn't secure enough
 given the way we use it, but what's not debatable is that the optics of it
 are not very good anymore.  However, if we want to shut up the peanut
 gallery on this point, we have to get rid of MD5 usage in pg_authid not
 just the on-the-wire protocol --- I seriously doubt that the knee jerk
 MD5-is-insecure crowd will make any distinction there.  So I'm not
 following how you're satisfied with a proposal for just the latter.

 In any case, my larger point was that given the pain that we're going to
 incur here, and the certainly years-long transition interval involved,
 it would be foolish to think only about replacing the MD5 algorithm and
 not about reconsidering the context we use it in.  Stuff like unreasonably
 short salt values should be dealt with at the same time.



All discussion seems to be about the protocol, which is also the harder
problem, isn't it?

ISTM that the more *important* thing to fix is the on-disk storage in
pg_authid.

If you actually worry about someone sniffing your network and mounting an
attack against a password, you can use SSL. That means we already have a
way to deal with that, whereas we don't have a way to mitigate the
pg_authid thing. And also, if you are actually worried about someone
sniffing an MD5'ed password, shouldn't you also worry about that same
person sniffing all the *contents* of your database, which the password is
there to protect?

(and in that regard, LDAP and such encryption algorithms send the password
in clear text in that case)

Seems the risk of someone either lifting pg_authid from disk or by hacking
the system and being postgres, thereby accessing passwords stored somewhere
else, is actually the bigger problem. But also one that should be
reasonably easy (TM) to fix in a backwards compatible way? (just rewrite
with a new hash whenever the password is changed, but keep reading md5
until they are all replaced.


-- 
 Magnus Hagander
 Me: http://www.hagander.net/
 Work: http://www.redpill-linpro.com/


Re: [HACKERS] GRANT USAGE on FOREIGN SERVER exposes passwords

2015-02-11 Thread Tom Lane
Jim Nasby jim.na...@bluetreble.com writes:
 On 2/5/15 10:48 AM, Tom Lane wrote:
 The dblink example is entirely uncompelling, given that as you said
 somebody with access to a dblink connection could execute ALTER USER on
 the far end.

 Actually, you can eliminate that by not granting direct access to dblink 
 functions. Instead you create a SECURITY DEFINER function that sanity 
 checks the SQL you're trying to run and rejects things like ALTER USER. 
 While you're doing that, you can also lock away the connection 
 information. A former coworker actually built a system that does this, 
 at least to a limited degree.

... but if you aren't giving the untrusted user direct access to the
connection, then he also doesn't get to see its options in the view.
So this still isn't compelling, so far as dblink goes.

regards, tom lane


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] 9.6 Feature help requested: Inclusion Constraints

2015-02-11 Thread David Fetter
On Tue, Feb 10, 2015 at 09:47:22PM -0500, Peter Eisentraut wrote:
 On 2/9/15 3:12 AM, Jeff Davis wrote:
  On Sat, 2015-02-07 at 16:08 -0800, Jeff Davis wrote:
  I believe Inclusion Constraints will be important for postgres.
  
  I forgot to credit Darren Duncan with the name of this feature:
  
  http://www.postgresql.org/message-id/4f8bb9b0.5090...@darrenduncan.net
 
 I think it would be confusing to name something inclusion constraint
 that is not somehow the opposite of an exclusion constraint.

To my view, this opposition is there.

In both cases, permission for a tuple to exist is based on another
other tuple's presence with some generalized equality criterion.  In
the exclusion case, it's prevented.  In the inclusion case, it's
required.

Cheers,
David.
-- 
David Fetter da...@fetter.org http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter  XMPP: david.fet...@gmail.com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] ibm system z in the buildfarm

2015-02-11 Thread David Fetter
On Tue, Feb 10, 2015 at 05:49:18PM -0800, Mark Wong wrote:
 Hi everyone,
 
 I've seen in the archive a call for more architecture coverage so I just
 wanted to send a quick note that there is now Linux on System Z in the
 buildfarm now:
 
 http://pgbuildfarm.org/cgi-bin/show_history.pl?nm=nudibranchbr=HEAD

That's super awesome!

Do you have access to put a z/OS animal there, too?  The compilers
available for that OS, as well as the OS itself, should make for some
very interesting results.

Cheers,
David.
-- 
David Fetter da...@fetter.org http://fetter.org/
Phone: +1 415 235 3778  AIM: dfetter666  Yahoo!: dfetter
Skype: davidfetter  XMPP: david.fet...@gmail.com

Remember to vote!
Consider donating to Postgres: http://www.postgresql.org/about/donate


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers


Re: [HACKERS] assessing parallel-safety

2015-02-11 Thread Noah Misch
On Wed, Feb 11, 2015 at 03:21:12PM -0500, Robert Haas wrote:
 On Wed, Feb 11, 2015 at 9:39 AM, Robert Haas robertmh...@gmail.com wrote:
  This does not seem to work out nicely.  The problem here is that
  simplify_function() gets called from eval_const_expressions() which
  gets called from a variety of places, but the principal one seems to
  be subquery_planner().  So if you have a query with two subqueries,
  and the second one contains something parallel-unsafe, you might by
  that time have already generated a parallel plan for the first one,
  which won't do.

That is a major mark against putting the check in simplify_function(), agreed.

  Unless we want to rejigger this so that we do a
  complete eval_const_expressions() pass over the entire query tree
  (including all subqueries) FIRST, and then only after that go back and
  plan all of those subqueries, I don't see how to make this work; and
  I'm guessing that there are good reasons not to do that.

I expect that would work fine, but I do think it premature to venture that far
out of your way to optimize this new tree examination.  The cost may just not
matter.  Other parts of the planner use code like contain_volatile_functions()
and contain_nonstrict_functions(), which have the same kind of inefficiency
you're looking to avoid here.

 I think we may want a dedicated parallel-safe property for functions
 rather than piggybacking on provolatile, but that will probably also
 be changeable via ALTER FUNCTION, and stored rules won't get
 miraculously updated.  So this definitely can't be something we figure
 out at parse-time ... it's got to be determined later.

Pretty much.

 But at the
 moment I see no way to do that without an extra pass over the whole
 rewritten query tree.  :-(

+1 for doing that.


-- 
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers