Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes
On Wed, Feb 11, 2015 at 11:03 PM, Syed, Rahila wrote: > >IMO, we should add details about how this new field is used in the > comments on top of XLogRecordBlockImageHeader, meaning that when a page > hole is present we use the compression info structure and when there is no > hole, we are sure that the FPW raw length is BLCKSZ meaning that the two > bytes of the CompressionInfo stuff is unnecessary. > This comment is included in the patch attached. > > > For correctness with_hole should be set even for uncompressed pages. I > think that we should as well use it for sanity checks in xlogreader.c when > decoding records. > This change is made in the attached patch. Following sanity checks have > been added in xlogreader.c > > if (!(blk->with_hole) && blk->hole_offset != 0 || blk->with_hole && > blk->hole_offset <= 0)) > > if (blk->with_hole && blk->bkp_len >= BLCKSZ) > > if (!(blk->with_hole) && blk->bkp_len != BLCKSZ) > Cool, thanks! This patch fails to compile: xlogreader.c:1049:46: error: extraneous ')' after condition, expected a statement blk->with_hole && blk->hole_offset <= 0)) Note as well that at least clang does not like much how the sanity check with with_hole are done. You should place parentheses around the '&&' expressions. Also, I would rather define with_hole == 0 or with_hole == 1 explicitly int those checks. There is a typo: s/true,see/true, see/ [nitpicky]Be as well aware of the 80-character limit per line that is usually normally by comment blocks.[/] + * "with_hole" is used to identify the presence of hole in a block. + * As mentioned above, length of block cannnot be more than 15-bit long. + * So, the free bit in the length field is used by "with_hole" to identify presence of + * XLogRecordBlockImageCompressionInfo. If hole is not present ,the raw size of + * a compressed block is equal to BLCKSZ therefore XLogRecordBlockImageCompressionInfo + * for the corresponding compressed block need not be stored in header. + * If hole is present raw size is stored. I would rewrite this paragraph as follows, fixing the multiple typos: "with_hole" is used to identify the presence of a hole in a block image. As the length of a block cannot be more than 15-bit long, the extra bit in the length field is used for this identification purpose. If the block image has no hole, it is ensured that the raw size of a compressed block image is equal to BLCKSZ, hence the contents of XLogRecordBlockImageCompressionInfo are not necessary. + /* Followed by the data related to compression if block is compressed */ This comment needs to be updated to "if block image is compressed and has a hole". + lp_off and lp_len fields in ItemIdData (see include/storage/itemid.h) and + XLogRecordBlockImageHeader where page hole offset and length is limited to 15-bit + length (see src/include/access/xlogrecord.h). 80-character limit... Regards -- Michael
Re: [HACKERS] The return value of allocate_recordbuf()
On Wed, Feb 11, 2015 at 2:13 AM, Robert Haas wrote:
> On Mon, Feb 9, 2015 at 7:02 AM, Michael Paquier
> wrote:
> > On Mon, Feb 9, 2015 at 7:58 PM, Fujii Masao
> wrote:
> >> MemoryContextAllocExtended() was added, so isn't it time to replace
> palloc()
> >> with MemoryContextAllocExtended(CurrentMemoryContext, MCXT_ALLOC_NO_OOM)
> >> in allocate_recordbuf()?
> >
> > Yeah, let's move on here, but not with this patch I am afraid as this
> > breaks any client utility using xlogreader.c in frontend, like
> > pg_xlogdump or pg_rewind because MemoryContextAllocExtended is not
> > available in frontend, only in backend. We are going to need something
> > like palloc_noerror, defined on both backend (mcxt.c) and frontend
> > (fe_memutils.c) side.
>
> Unfortunately, that gets us back into the exact terminological dispute
> that we were hoping to avoid. Perhaps we could compromise on
> palloc_extended().
>
Yes, why not using palloc_extended instead of palloc_noerror that has been
clearly rejected in the other thread. Now, for palloc_extended we should
copy the flags of MemoryContextAllocExtended to fe_memutils.h and have the
same interface between frontend and backend (note that MCXT_ALLOC_HUGE has
no real utility for frontends). Attached is a patch to achieve that,
completed with a second patch for the problem related to this thread. Note
that in the second patch I have added an ERROR in logical.c after calling
XLogReaderAllocate, this was missing..
> Btw, the huge flag should be used as well as palloc only allows
> > allocation up to 1GB, and this is incompatible with ~9.4 where malloc
> > is used.
>
> I think that flag should be used only if it's needed, not just on the
> grounds that 9.4 has no such limit. In general, limiting allocations
> to 1GB has been good to us; for example, it prevents a corrupted TOAST
> length from causing a memory allocation large enough to crash the
> machine (yes, there are machines you can crash with a giant memory
> allocation!). We should bypass that limit only where it is clearly
> necessary.
>
Fine for me to error out in this code path if we try more than 1GB of
allocation.
Regards,
--
Michael
From b9ff4e9ffdc9b02d1ee372c63ee36bfe7659ee9f Mon Sep 17 00:00:00 2001
From: Michael Paquier
Date: Thu, 12 Feb 2015 03:36:55 +0900
Subject: [PATCH 1/2] Add palloc_extended for frontend and backend
This interface can be used to control at a lower level memory allocation
using an interface similar to MemoryContextAllocExtended, but this time
applied to CurrentMemoryContext on backend side, while on frontend side
a similar interface is available.
---
src/backend/utils/mmgr/mcxt.c| 37 +
src/common/fe_memutils.c | 36 ++--
src/include/common/fe_memutils.h | 11 ++-
src/include/utils/palloc.h | 1 +
4 files changed, 74 insertions(+), 11 deletions(-)
diff --git a/src/backend/utils/mmgr/mcxt.c b/src/backend/utils/mmgr/mcxt.c
index 202bc78..2589f6a 100644
--- a/src/backend/utils/mmgr/mcxt.c
+++ b/src/backend/utils/mmgr/mcxt.c
@@ -811,6 +811,43 @@ palloc0(Size size)
return ret;
}
+void *
+palloc_extended(Size size, int flags)
+{
+ /* duplicates MemoryContextAllocExtended to avoid increased overhead */
+ void *ret;
+
+ AssertArg(MemoryContextIsValid(CurrentMemoryContext));
+ AssertNotInCriticalSection(CurrentMemoryContext);
+
+ if (((flags & MCXT_ALLOC_HUGE) != 0 && !AllocHugeSizeIsValid(size)) ||
+ ((flags & MCXT_ALLOC_HUGE) == 0 && !AllocSizeIsValid(size)))
+ elog(ERROR, "invalid memory alloc request size %zu", size);
+
+ CurrentMemoryContext->isReset = false;
+
+ ret = (*CurrentMemoryContext->methods->alloc) (CurrentMemoryContext, size);
+ if (ret == NULL)
+ {
+ if ((flags & MCXT_ALLOC_NO_OOM) == 0)
+ {
+ MemoryContextStats(TopMemoryContext);
+ ereport(ERROR,
+ (errcode(ERRCODE_OUT_OF_MEMORY),
+ errmsg("out of memory"),
+ errdetail("Failed on request of size %zu.", size)));
+ }
+ return NULL;
+ }
+
+ VALGRIND_MEMPOOL_ALLOC(CurrentMemoryContext, ret, size);
+
+ if ((flags & MCXT_ALLOC_ZERO) != 0)
+ MemSetAligned(ret, 0, size);
+
+ return ret;
+}
+
/*
* pfree
* Release an allocated chunk.
diff --git a/src/common/fe_memutils.c b/src/common/fe_memutils.c
index 345221e..bd60cc2 100644
--- a/src/common/fe_memutils.c
+++ b/src/common/fe_memutils.c
@@ -19,8 +19,8 @@
#include "postgres_fe.h"
-void *
-pg_malloc(size_t size)
+static inline void *
+pg_malloc_extended(size_t size, int flags)
{
void *tmp;
@@ -28,22 +28,32 @@ pg_malloc(size_t size)
if (size == 0)
size = 1;
tmp = malloc(size);
- if (!tmp)
+
+ if (tmp == NULL)
{
- fprintf(stderr, _("out of memory\n"));
- exit(EXIT_FAILURE);
+ if ((flags & MCXT_ALLOC_NO_OOM) == 0)
+ {
+ fprintf(stderr, _("out of memory\n"));
+ exit(EXIT_FAILURE);
+ }
+ return NULL;
}
+
+ if ((flags & MCXT_ALLOC_ZERO) != 0)
+ MemSet(tmp, 0, size);
return tmp;
}
void *
-pg_malloc0(size_t size)
+pg
Re: [HACKERS] assessing parallel-safety
On Wed, Feb 11, 2015 at 03:21:12PM -0500, Robert Haas wrote: > On Wed, Feb 11, 2015 at 9:39 AM, Robert Haas wrote: > > This does not seem to work out nicely. The problem here is that > > simplify_function() gets called from eval_const_expressions() which > > gets called from a variety of places, but the principal one seems to > > be subquery_planner(). So if you have a query with two subqueries, > > and the second one contains something parallel-unsafe, you might by > > that time have already generated a parallel plan for the first one, > > which won't do. That is a major mark against putting the check in simplify_function(), agreed. > > Unless we want to rejigger this so that we do a > > complete eval_const_expressions() pass over the entire query tree > > (including all subqueries) FIRST, and then only after that go back and > > plan all of those subqueries, I don't see how to make this work; and > > I'm guessing that there are good reasons not to do that. I expect that would work fine, but I do think it premature to venture that far out of your way to optimize this new tree examination. The cost may just not matter. Other parts of the planner use code like contain_volatile_functions() and contain_nonstrict_functions(), which have the same kind of inefficiency you're looking to avoid here. > I think we may want a dedicated parallel-safe property for functions > rather than piggybacking on provolatile, but that will probably also > be changeable via ALTER FUNCTION, and stored rules won't get > miraculously updated. So this definitely can't be something we figure > out at parse-time ... it's got to be determined later. Pretty much. > But at the > moment I see no way to do that without an extra pass over the whole > rewritten query tree. :-( +1 for doing that. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ExplainModifyTarget doesn't work as expected
On 2015/02/10 14:49, Etsuro Fujita wrote:
> On 2015/02/07 1:09, Tom Lane wrote:
>> IIRC, this code was written at a time when we didn't have NO INHERIT check
>> constraints and so it was impossible for the parent table to get optimized
>> away while leaving children. So the comment in ExplainModifyTarget was
>> good at the time. But it no longer is.
>>
>> I think your basic idea of preserving the original parent table's relid
>> is correct; but instead of doing it like this patch does, I'd be inclined
>> to make ModifyTable inherit from Scan not Plan, and use the scan.scanrelid
>> field to carry the parent RTI. Then you would probably end up with a net
>> savings of code rather than net addition; certainly ExplainModifyTarget
>> would go away entirely since you'd just treat ModifyTable like any other
>> Scan in this part of EXPLAIN.
>
> Will follow your revision.
Done. Attached is an updated version of the patch.
Best regards,
Etsuro Fujita
*** a/src/backend/commands/explain.c
--- b/src/backend/commands/explain.c
***
*** 95,101 static const char *explain_get_index_name(Oid indexId);
static void ExplainIndexScanDetails(Oid indexid, ScanDirection indexorderdir,
ExplainState *es);
static void ExplainScanTarget(Scan *plan, ExplainState *es);
- static void ExplainModifyTarget(ModifyTable *plan, ExplainState *es);
static void ExplainTargetRel(Plan *plan, Index rti, ExplainState *es);
static void show_modifytable_info(ModifyTableState *mtstate, ExplainState *es);
static void ExplainMemberNodes(List *plans, PlanState **planstates,
--- 95,100
***
*** 724,736 ExplainPreScanNode(PlanState *planstate, Bitmapset **rels_used)
case T_WorkTableScan:
case T_ForeignScan:
case T_CustomScan:
- *rels_used = bms_add_member(*rels_used,
- ((Scan *) plan)->scanrelid);
- break;
case T_ModifyTable:
- /* cf ExplainModifyTarget */
*rels_used = bms_add_member(*rels_used,
! linitial_int(((ModifyTable *) plan)->resultRelations));
break;
default:
break;
--- 723,731
case T_WorkTableScan:
case T_ForeignScan:
case T_CustomScan:
case T_ModifyTable:
*rels_used = bms_add_member(*rels_used,
! ((Scan *) plan)->scanrelid);
break;
default:
break;
***
*** 1067,1072 ExplainNode(PlanState *planstate, List *ancestors,
--- 1062,1068
case T_WorkTableScan:
case T_ForeignScan:
case T_CustomScan:
+ case T_ModifyTable:
ExplainScanTarget((Scan *) plan, es);
break;
case T_IndexScan:
***
*** 1101,1109 ExplainNode(PlanState *planstate, List *ancestors,
ExplainPropertyText("Index Name", indexname, es);
}
break;
- case T_ModifyTable:
- ExplainModifyTarget((ModifyTable *) plan, es);
- break;
case T_NestLoop:
case T_MergeJoin:
case T_HashJoin:
--- 1097,1102
***
*** 2104,2127 ExplainScanTarget(Scan *plan, ExplainState *es)
}
/*
- * Show the target of a ModifyTable node
- */
- static void
- ExplainModifyTarget(ModifyTable *plan, ExplainState *es)
- {
- Index rti;
-
- /*
- * We show the name of the first target relation. In multi-target-table
- * cases this should always be the parent of the inheritance tree.
- */
- Assert(plan->resultRelations != NIL);
- rti = linitial_int(plan->resultRelations);
-
- ExplainTargetRel((Plan *) plan, rti, es);
- }
-
- /*
* Show the target relation of a scan or modify node
*/
static void
--- 2097,2102
*** a/src/backend/nodes/copyfuncs.c
--- b/src/backend/nodes/copyfuncs.c
***
*** 120,125 CopyPlanFields(const Plan *from, Plan *newnode)
--- 120,140
}
/*
+ * CopyScanFields
+ *
+ * This function copies the fields of the Scan node. It is used by
+ * all the copy functions for classes which inherit from Scan.
+ */
+ static void
+ CopyScanFields(const Scan *from, Scan *newnode)
+ {
+ CopyPlanFields((const Plan *) from, (Plan *) newnode);
+
+ COPY_SCALAR_FIELD(scanrelid);
+ }
+
+
+ /*
* _copyPlan
*/
static Plan *
***
*** 168,174 _copyModifyTable(const ModifyTable *from)
/*
* copy node superclass fields
*/
! CopyPlanFields((const Plan *) from, (Plan *) newnode);
/*
* copy remainder of node
--- 183,189
/*
* copy node superclass fields
*/
! CopyScanFields((const Scan *) from, (Scan *) newnode);
/*
* copy remainder of node
***
*** 306,325 _copyBitmapOr(const BitmapOr *from)
/*
- * CopyScanFields
- *
- * This function copies the fields of the Scan node. It is used by
- * all the copy functions for classes which inherit from Scan.
- */
- static void
- CopyScanFields(const Scan *from, Scan *newnode)
- {
- CopyPlanFields((const Plan *) from, (Plan *) newnode);
-
- COPY_SCALAR_FIELD(scanrelid);
- }
-
- /*
* _copyScan
*/
static Scan *
--- 321,326
*** a/src/bac
Re: [HACKERS] Better error message on pg_upgrade checksum mismatches
On Tue, Feb 10, 2015 at 10:55:07AM -0500, Greg Sabino Mullane wrote:
> Just a little thing that's been bugging me. If one side of the
> pg_upgrade has checksums and the other does not, give a less
> cryptic error message.
Thanks. Slightly adjusted patch attached and applied to head.
--
Bruce Momjian http://momjian.us
EnterpriseDB http://enterprisedb.com
+ Everyone has their own god. +
diff --git a/contrib/pg_upgrade/controldata.c b/contrib/pg_upgrade/controldata.c
new file mode 100644
index a02a8ec..0e70b6f
*** a/contrib/pg_upgrade/controldata.c
--- b/contrib/pg_upgrade/controldata.c
*** check_control_data(ControlData *oldctrl,
*** 572,581
* We might eventually allow upgrades from checksum to no-checksum
* clusters.
*/
! if (oldctrl->data_checksum_version != newctrl->data_checksum_version)
! {
! pg_fatal("old and new pg_controldata checksum versions are invalid or do not match\n");
! }
}
--- 572,585
* We might eventually allow upgrades from checksum to no-checksum
* clusters.
*/
! if (oldctrl->data_checksum_version == 0 &&
! newctrl->data_checksum_version != 0)
! pg_fatal("old cluster does not use data checksums but the new one does\n");
! else if (oldctrl->data_checksum_version != 0 &&
! newctrl->data_checksum_version == 0)
! pg_fatal("old cluster uses data checksums but the new one does not\n");
! else if (oldctrl->data_checksum_version != newctrl->data_checksum_version)
! pg_fatal("old and new cluster pg_controldata checksum versions do not match\n");
}
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pg_upgrade bug in handling postgres/template1 databases
On Tue, Feb 10, 2015 at 12:52:33PM -0500, Bruce Momjian wrote: > The attached patch fixes the problem, and I have reproduced the bug and > verified the patch fixes it. It would have been nice if I had figured > this out two weeks ago, before we released minor version upgrades. This > bug has existed back to 9.0 or earlier, so it isn't new. Patch applied back through 9.0. This bug was reported by an EDB customer. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + Everyone has their own god. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] reducing our reliance on MD5
* Tom Lane (t...@sss.pgh.pa.us) wrote: > Stephen Frost writes: > > * Tom Lane (t...@sss.pgh.pa.us) wrote: > >> We've already got a sufficiency of external authentication mechanisms. > >> If people wanted to use non-built-in authentication, we'd not be having > >> this discussion. > > > Just to be clear- lots of people *do* use the external authentication > > mechanisms we provide, particularly Kerberos/GSSAPI. SASL would bring > > us quite a few additional mechanisms (SQL-based, Berkley DB, one-time > > passwords, RSA SecurID, etc..) and would mean we might be able to > > eventually drop direct GSSAPI and LDAP support and have a better > > alternative for those who want to use password-based auth. > > My point is that we already have got a lot of external authentication > mechanisms, and it's completely unclear (to me anyway) that there is > any demand for another one. To this point, I have had requests for various things which SASL would provide- ability to use the same password as the Unix host, SecurID support, and SQL-based (with another PG database) all come to mind. I agree that such isn't particularly relevant to this discussion though. > The unsatisfied demand is for a *built in* > mechanism, specifically one that people have more faith in than MD5. I agree that there is demand for a new mechanism which isn't MD5. I don't think *built in* is at all a requirement however- *easy to use* is. > Those who worry about that and don't mind having additional moving parts > have probably already migrated to one or another of the existing external > solutions. Certainly we'd want to make SASL-with-SCRAM easy for users to use. To the extent that the moving parts in that arrangement are hidden from view, hopefully users will use it and we get comfortable enough with it for distros to set it as the default. I am catagorically *not* worried about non-libpq client libraries and applications in this regard. For starters, those aren't end users, those are other developers who have a vested interest in supporting whatever we do. Further, Java certainly supports SASL, as do quite a few other languages (Python, PHP, Ruby, there's a few examples in Go it seems). > While I won't stand in the way of somebody adding support for an external > SASL library, I think such work has got basically zero to do with the > actual problem. I don't think that's accurate. If we provide a better mechanism for password-based authentication, even if it is through SASL, I feel pretty confident that at least the Debian based distros (which universally include SASL support for applications that support it) would support it and might even switch to it for the default (once they were comfortable with it and enough clients supported it), unless we told them not to. That doesn't address existing installations, of course, and we'd have to wait for non-libpq client to update, but things would certainly move faster if we actually had something for things to move *to*, even if that thing is SASL-based SCRAM, imv. Thanks! Stephen signature.asc Description: Digital signature
Re: [HACKERS] reducing our reliance on MD5
Stephen Frost writes: > * Tom Lane (t...@sss.pgh.pa.us) wrote: >> We've already got a sufficiency of external authentication mechanisms. >> If people wanted to use non-built-in authentication, we'd not be having >> this discussion. > Just to be clear- lots of people *do* use the external authentication > mechanisms we provide, particularly Kerberos/GSSAPI. SASL would bring > us quite a few additional mechanisms (SQL-based, Berkley DB, one-time > passwords, RSA SecurID, etc..) and would mean we might be able to > eventually drop direct GSSAPI and LDAP support and have a better > alternative for those who want to use password-based auth. My point is that we already have got a lot of external authentication mechanisms, and it's completely unclear (to me anyway) that there is any demand for another one. The unsatisfied demand is for a *built in* mechanism, specifically one that people have more faith in than MD5. Those who worry about that and don't mind having additional moving parts have probably already migrated to one or another of the existing external solutions. While I won't stand in the way of somebody adding support for an external SASL library, I think such work has got basically zero to do with the actual problem. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] reducing our reliance on MD5
* Tom Lane (t...@sss.pgh.pa.us) wrote: > Heikki Linnakangas writes: > > We could also support using a library like that for additional > > authentication mechanisms, though, for those who really need them. > > We've already got a sufficiency of external authentication mechanisms. > If people wanted to use non-built-in authentication, we'd not be having > this discussion. Just to be clear- lots of people *do* use the external authentication mechanisms we provide, particularly Kerberos/GSSAPI. SASL would bring us quite a few additional mechanisms (SQL-based, Berkley DB, one-time passwords, RSA SecurID, etc..) and would mean we might be able to eventually drop direct GSSAPI and LDAP support and have a better alternative for those who want to use password-based auth. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] Typo in logicaldecoding.sgml
On 2015-02-10 20:34:33 -0500, Robert Haas wrote: > On Tue, Feb 10, 2015 at 6:57 PM, Andres Freund wrote: > >> I think one of them could be a fix. Any advice? > > > > I slightly prefer the second form... > > I think "it uses" is definitely more standard English usage in this context. Pushed "it uses". Thanks to you two. Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
[HACKERS] Re: [HACKERS] Streaming replication and WAL archive interactions
> This should be a very common setup in the field, so how are people doing it >in practice? One of possible workaround with archive and streaming was to use pg_receivexlog from standby to copy/save WALs to archive. but with pg_receivexlog was also issue with fsync. [ master ] -- streaming --> [ standby ] -- pg_receivexlog --> [ /archive ] In that case archive is always in pre standby state and it could be better than had archive broken on promote. -- Misha
Re: [HACKERS] libpq's multi-threaded SSL callback handling is busted
On 2015-02-09 18:17:14 +0100, Jan Urbański wrote: > We added unsetting the locking callback in > 4e816286533dd34c10b368487d4079595a3e1418 due to this bug report: > http://www.postgresql.org/message-id/48620925.6070...@pws.com.au > > Indeed, commenting out the CRYPTO_set_locking_callback(NULL) call in > fe-secure-openssl.c gets rid of the deadlock. However, it makes php segfault > with the (attached) reproduction script from the original 2008 bug report. If > your php.ini loads both the pgsql and curl extensions, reproduce the segfault > with: > > php -f pg_segfault.php > > The most difficult part about fixing this bug is to determine *who's at > fault*. I now lean towards the opinion that we shouldn't be messing with > OpenSSL callbacks *at all*. > > First of all, the current behaviour is crazy. We're setting and unsetting the > locking callback every time a connection is made/closed, which is not how > OpenSSL is supposed to be used. The *application* using libpq should set a > callback before it starts threads, it's no business of the library's. I don't buy this argument at all. Delivering a libpq that's not threadsafe in some circumstances is a really bad idea. > The old behaviour was slightly less insane (set callbacks first time we're > engaging OpenSSL code, never touch them again). The real sane solution is to > leave it up to the application. The real solution would be for openssl to do it itself. > I posit we should remove all CRYPTO_set_*_callback functions and associated > cruft from libpq. This unfortunately means that multi-threaded applications > using libpq and SSL will break if they haven't been setting their own > callbacks > (if they have, well, tough luck! libpq will just stomp over them the first > time > it connects to Postgres, but at least *some* callbacks are left present after > that). I think there's no chance in hell we can do that. Breaking a noticeable amount of libpq users in a minor upgrade is imo a cure *FAR* worse than the cure. > Note that the PHP pgsql extension doesn't set the OpenSSL callbacks, because > libpq was setting them on its own. If we remove the callback handling from > libpq, PHP will need to add them. By the way, the MySQL extension for PHP also > does not set those callbacks. I think this shows pretty clearly how insane it would be to change this in a minor release. Do you really expect people to update libpq and php in unison. After a minor release? Note that we *already* provide applications with the ability to set the callbacks themselves and prevent us from doing so: PQinitSSL(false). > Let me reiterate: I now believe the callbacks should be set by the > application, > libraries should not touch them, since they don't know what will they be > stomping on. If the application is run through a VM like Python or PHP, it's > the VM that should make sure the callbacks are set. I fail to see why it's any better to do it in the VM. That relies on either always loading the VM's openssl module, even if you don't actually need it because the library you use deals with openssl. It prevents other implementations of openssl in the VM. What I think we should do is the following: * Emphasize the fact that it's important to use PQinitSSL(false) if you did things yourself. * If there's already callbacks set: Remember that fact and don't overwrite. In the next major version: warn. * Assert or something if the callback when unregistering isn't the same as it was when registering. That's pretty much guaranteed to cause issues. > I would very much like to have this change back-patched, since setting and > resetting the callback makes using libpq in a threaded OpenSSL-enabled app > arguably less safe than if it didn't use any locking. Meh^2 Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SSL renegotiation and other related woes
Hi,
On 2015-02-05 23:03:02 +0200, Heikki Linnakangas wrote:
> On 01/26/2015 12:14 PM, Andres Freund wrote:
> >Hi,
> >
> >When working on getting rid of ImmediateInterruptOK I wanted to verify
> >that ssl still works correctly. Turned out it didn't. But neither did it
> >in master.
> >
> >Turns out there's two major things we do wrong:
> >
> >1) We ignore the rule that once called and returning
> >SSL_ERROR_WANTS_(READ|WRITE) SSL_read()/write() have to be called
> >again with the same parameters. Unfortunately that rule doesn't mean
> >just that the same parameters have to be passed in, but also that we
> >can't just constantly switch between _read()/write(). Especially
> >nonblocking backend code (i.e. walsender) and the whole frontend code
> >violate this rule.
>
> I don't buy that. Googling around, and looking at the man pages, I just
> don't see anything to support that claim. I believe it is supported to
> switch between reads and writes.
There's at least two implementations that seem to have workaround to
achieve something like that. Apache's mod_ssl and the ssl layer for
libevent.
> >2) We start renegotiations in be_tls_write() while in nonblocking mode,
> >but don't properly retry to handle socket readyness. There's a loop
> >that retries handshakes twenty times (???), but what actually is
> >needed is to call SSL_get_error() and ensure that there's actually
> >data available.
>
> Yeah, that's just crazy.
>
> Actually, I don't think we need to call SSL_do_handshake() at all. At least
> on modern OpenSSL versions. OpenSSL internally uses a state machine, and
> SSL_read() and SSL_write() calls will complete the handshake just as well.
Some blaming seems to show that that's been the case for a long time.
> >2) is easy enough to fix, but 1) is pretty hard. Before anybody says
> >that 1) isn't an important rule: It reliably causes connection aborts
> >within a couple renegotiations. The higher the latency the higher the
> >likelihood of aborts. Even locally it doesn't take very long to
> >abort. Errors usually are something like "SSL connection has been closed
> >unexpectedly" or "SSL Error: sslv3 alert unexpected message" and a host
> >of other similar messages. There's a couple reports of those in the
> >archives and I've seen many more in client logs.
>
> Yeah, I can reproduce that. There's clearly something wrong.
> I believe this is an OpenSSL bug. I traced the "unexpected message" error to
> this code in OpenSSL's s3_read_bytes() function:
Yep, I got to that as well.
> >case SSL3_RT_APPLICATION_DATA:
> >/*
> > * At this point, we were expecting handshake data, but have
> > * application data. If the library was running inside ssl3_read()
> > * (i.e. in_read_app_data is set) and it makes sense to read
> > * application data at this point (session renegotiation not yet
> > * started), we will indulge it.
> > */
> >if (s->s3->in_read_app_data &&
> >(s->s3->total_renegotiations != 0) &&
> >(((s->state & SSL_ST_CONNECT) &&
> > (s->state >= SSL3_ST_CW_CLNT_HELLO_A) &&
> > (s->state <= SSL3_ST_CR_SRVR_HELLO_A)
> > ) || ((s->state & SSL_ST_ACCEPT) &&
> > (s->state <= SSL3_ST_SW_HELLO_REQ_A) &&
> > (s->state >= SSL3_ST_SR_CLNT_HELLO_A)
> > )
> >)) {
> >s->s3->in_read_app_data = 2;
> >return (-1);
> >} else {
> >al = SSL_AD_UNEXPECTED_MESSAGE;
> > SSLerr(SSL_F_SSL3_READ_BYTES, SSL_R_UNEXPECTED_RECORD);
> >goto f_err;
> >}
>
> So that quite clearly throws an error, *unless* the original application
> call was SSL_read(). As you also concluded, OpenSSL doesn't like it when
> SSL_write() is called when it's in renegotiation. I think that's OpenSSL's
> fault and it should "indulge" the application data message even in
> SSL_write().
That'd be nice, but I think there were multiple reports to them about
this and there wasn't much of a response. Maybe it's time to try again.
> In theory, I guess we should do similar hacks in the server, and always call
> SSL_read() before SSL_write(), but it seems to work without it. Or maybe
> not; OpenSSL server and client code is not symmetric, so perhaps it works in
> server mode without that.
I think we should do it on the server side - got some server side errors
when I cranked up the pg_receivexlog's status interval and set the wal
sender timeout very low.
Greetings,
Andres Freund
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SSL renegotiation and other related woes
On 2015-02-11 14:54:03 +0200, Heikki Linnakangas wrote: > Thoughts? Can you reproduce any errors with this? I'm on battery right now, so I can't really test much. Did you test having an actual standby instead of pg_receivexlog? I saw some slightly different errors there. Does this patch alone work for you or did you test it together with the earlier one to fix the renegotiation handling server side when nonblocking? Because that frequently seemed to error out for me, at least over actual network. A latch loop + checking for the states seemed to fix that for me. I'm pretty darn happy that you seem to have found a much less ugly solution than mine. Although it's only pretty by comparison ;) > + /* > + * To work-around an issue with OpenSSL and renegotiation, we don't > + * let SSL_write() to read any incoming data. superflous "to". > + * NB: This relies on the calling code to call pqsecure_read(), > completing > + * the renegotiation handshake, if pqsecure_write() returns 0. Otherwise > + * we'll never make progress. > + */ I think this should a) refer to the fact that pqSendSome does that in blocking scenarios b) expand the comment around the pqReadData to reference that fact. How are we going to deal with callers using libpq in nonblocking mode. A client doing a large COPY FROM STDIN to the server using a nonblocking connection (not a stupid thing to do) could hit this IIUC. Could we perhaps call SSL_peek() when SSL_write() hits WANTS_READ? In combination with your fix I think that should fix the possibility of such a deadlock? Especially on the serverside where the socket most of the time is is in, although emulated, blocking mode that seems easier - we can just retry afterwards. Greetings, Andres Freund -- Andres Freund http://www.2ndQuadrant.com/ PostgreSQL Development, 24x7 Support, Training & Services -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Index-only scans for GiST.
On 11 February 2015 at 22:50, Anastasia Lubennikova wrote: > Finally there is a new version of patch (in attachments). > It provides multicolumn index-only scan for GiST indexes. > > - Memory leak is fixed. > - little code cleanup > - example of performance test in attachmens > - function OIDs have debugging values (*) just to avoid merge > conflicts while testing patch > > Wiki page of the project is > > https://wiki.postgresql.org/wiki/Support_for_Index-only_scans_for_GIST_GSoC_2014 > > Waiting for feedback. > Hi Anastasia. Thanks for the updated patch. I've just tried applying it to head and it doesn't appear to apply cleanly. $ patch -p1 < ~/Downloads/indexonlyscan_gist_2.0.patch (Stripping trailing CRs from patch; use --binary to disable.) patching file src/backend/access/gist/gist.c Hunk #1 succeeded at 1404 (offset 9 lines). Hunk #2 succeeded at 1434 (offset 9 lines). (Stripping trailing CRs from patch; use --binary to disable.) patching file src/backend/access/gist/gistget.c Hunk #1 succeeded at 227 (offset 1 line). Hunk #2 succeeded at 243 (offset 1 line). Hunk #3 succeeded at 293 (offset -4 lines). Hunk #4 succeeded at 330 (offset -4 lines). Hunk #5 succeeded at 365 (offset -5 lines). Hunk #6 succeeded at 444 (offset -27 lines). Hunk #7 succeeded at 474 (offset -27 lines). Hunk #8 FAILED at 518. Hunk #9 succeeded at 507 (offset -28 lines). Hunk #10 succeeded at 549 with fuzz 1 (offset -28 lines). Hunk #11 FAILED at 601. 2 out of 11 hunks FAILED -- saving rejects to file src/backend/access/gist/gistget.c.rej ... -- Thom
[HACKERS] Index-only scans for GiST.
Finally there is a new version of patch (in attachments). It provides multicolumn index-only scan for GiST indexes. - Memory leak is fixed. - little code cleanup - example of performance test in attachmens - function OIDs have debugging values (*) just to avoid merge conflicts while testing patch Wiki page of the project is https://wiki.postgresql.org/wiki/Support_for_Index-only_scans_for_GIST_GSoC_2014 Waiting for feedback. -- Best regards, Lubennikova Anastasia indexonlyscan_gist_2.0.patch Description: Binary data test_performance.sql Description: Binary data indexonlyscan_gist_docs.patch Description: Binary data -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Corner case for add_path_precheck
Antonin Houska writes: > Tom Lane wrote: >> Antonin Houska writes: >>> The special case is that the path passed to add_path_precheck() has costs >>> *equal to* those of the old_path. If pathkeys, outer rells and costs are the >>> same, as summarized in the comment above, I expect add_path_precheck() to >>> return false. Do I misread anything? >> It does, so I don't see your point? > Just that pre-check is - in this special (and very rare?) case - more > stringent than the proper check would be. It's assuming that a nonzero amount of cost will be added on before the real check happens. Even if none was added, discarding the new path is the way we'd break the tie that would result, so I still don't see your point. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Corner case for add_path_precheck
Tom Lane wrote: > Antonin Houska writes: > > The special case is that the path passed to add_path_precheck() has costs > > *equal to* those of the old_path. If pathkeys, outer rells and costs are the > > same, as summarized in the comment above, I expect add_path_precheck() to > > return false. Do I misread anything? > > It does, so I don't see your point? Just that pre-check is - in this special (and very rare?) case - more stringent than the proper check would be. -- Antonin Houska Cybertec Schönig & Schönig GmbH Gröhrmühlgasse 26 A-2700 Wiener Neustadt Web: http://www.postgresql-support.de, http://www.cybertec.at -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] relid of non user created tables
On 2/7/15 5:55 AM, Tom Lane wrote: Deepak S writes: I wish to know if it is safe to assume that non user created tables always have a relid of less than 10,000 (I chose this value after trial and error). Depends what you consider "user created". System catalogs with manually-assigned OIDs will have OIDs below 10K, but initdb creates numerous things that don't have manually-assigned OIDs. And what of, say, TOAST tables attached to user tables? You might find the comments for FirstNormalObjectId in src/include/access/transam.h enlightening. I think the better question is "Why do you want to know this?" I suspect there's a better way to do whatever you're trying to do. -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Proposal : REINDEX xxx VERBOSE
On 2/5/15 12:01 PM, Tom Lane wrote: Robert Haas writes: We've got a mix of styles for extensible options right now: That we do. So COPY puts the options at the very end, but EXPLAIN and VACUUM put them right after the command name. I prefer the latter style and would vote to adopt it here. Meh. Options-at-the-end seems by far the most sensible style to me. The options-right-after-the-keyword style is a mess, both logically and from a parsing standpoint, and the only reason we have it at all is historical artifact (ask Bruce about the origins of VACUUM ANALYZE over a beer sometime). I suspect at least some of this stems from how command line programs tend to process options before arguments. I tend to agree with you Tom, but I think what's more important is that we're consistent. COPY is already a bit of an oddball because it uses WITH, but both EXPLAIN and VACUUM use parenthesis immediately after the first verb. Introducing a parenthesis version that goes at the end instead of the beginning is just going to make this worse. If we're going to take a stand on this, we need to do it NOW, before we have even more commands that use (). I know you were worried about accepting options anywhere because it leads to reserved words, but perhaps we could support it just for EXPLAIN and VACUUM, and then switch to trailing options if people think that would be better. -- Jim Nasby, Data Architect, Blue Treble Consulting Data in Trouble? Get it in Treble! http://BlueTreble.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] reducing our reliance on MD5
Heikki Linnakangas writes: > On 02/11/2015 03:52 PM, Robert Haas wrote: >> So are you thinking to integrate with the Cyrus SASL library, or do >> you have another thought? > I think we need to implement the primary MD5 replacement ourselves, so > that it's always available without extra libraries. Otherwise it will > not get much adoption, or the extra dependency will be a hassle anyway. +1 > We could also support using a library like that for additional > authentication mechanisms, though, for those who really need them. We've already got a sufficiency of external authentication mechanisms. If people wanted to use non-built-in authentication, we'd not be having this discussion. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] reducing our reliance on MD5
On 02/11/2015 11:30 PM, Claudio Freire wrote: On Wed, Feb 11, 2015 at 5:25 PM, Heikki Linnakangas wrote: On 02/11/2015 06:35 AM, Claudio Freire wrote: Usually because handshakes use a random salt on both sides. Not sure about pg's though, but in general collision strength is required but not slowness, they're not bruteforceable. To be precise: collision resistance is usually not important for hashes used in authentication handshakes. Not for our MD5 authentication method anyway; otherwise we'd be screwed. What you need is resistance to pre-image attacks. AFAIK, if I find a colliding string to the MD5 stored in pg_authid, I can specify that to libpq and get authenticated. Am I missing something? If you know the MD5 stored in pg_authid, you can use that directly to authenticate. No need to find the original password, or another colliding string, that hashes to the same MD5. - Heikki -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] reducing our reliance on MD5
On Wed, Feb 11, 2015 at 6:30 PM, Claudio Freire wrote: > On Wed, Feb 11, 2015 at 5:25 PM, Heikki Linnakangas > wrote: >> On 02/11/2015 06:35 AM, Claudio Freire wrote: >>> >>> Usually because handshakes use a random salt on both sides. Not sure >>> about pg's though, but in general collision strength is required but >>> not slowness, they're not bruteforceable. >> >> >> To be precise: collision resistance is usually not important for hashes used >> in authentication handshakes. Not for our MD5 authentication method anyway; >> otherwise we'd be screwed. What you need is resistance to pre-image attacks. > > AFAIK, if I find a colliding string to the MD5 stored in pg_authid, I > can specify that to libpq and get authenticated. > > Am I missing something? Oh, right, that's called pre-image. Never mind then -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] reducing our reliance on MD5
On Wed, Feb 11, 2015 at 5:25 PM, Heikki Linnakangas wrote: > On 02/11/2015 06:35 AM, Claudio Freire wrote: >> >> Usually because handshakes use a random salt on both sides. Not sure >> about pg's though, but in general collision strength is required but >> not slowness, they're not bruteforceable. > > > To be precise: collision resistance is usually not important for hashes used > in authentication handshakes. Not for our MD5 authentication method anyway; > otherwise we'd be screwed. What you need is resistance to pre-image attacks. AFAIK, if I find a colliding string to the MD5 stored in pg_authid, I can specify that to libpq and get authenticated. Am I missing something? -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] Parallel Seq Scan
On Tue, Feb 10, 2015 at 3:56 PM, Andres Freund wrote: > On 2015-02-10 09:23:02 -0500, Robert Haas wrote: >> On Tue, Feb 10, 2015 at 9:08 AM, Andres Freund >> wrote: >> > And good chunk sizes et al depend on higher layers, >> > selectivity estimates and such. And that's planner/executor work, not >> > the physical layer (which heapam.c pretty much is). >> >> If it's true that a good chunk size depends on the higher layers, then >> that would be a good argument for doing this differently, or at least >> exposing an API for the higher layers to tell heapam.c what chunk size >> they want. I hadn't considered that possibility - can you elaborate >> on why you think we might want to vary the chunk size? > > Because things like chunk size depend on the shape of the entire > plan. If you have a 1TB table and want to sequentially scan it in > parallel with 10 workers you better use some rather large chunks. That > way readahead will be efficient in a cpu/socket local manner, > i.e. directly reading in the pages into the directly connected memory of > that cpu. Important for performance on a NUMA system, otherwise you'll > constantly have everything go over the shared bus. But if you instead > have a plan where the sequential scan goes over a 1GB table, perhaps > with some relatively expensive filters, you'll really want a small > chunks size to avoid waiting. I see. That makes sense. > The chunk size will also really depend on > what other nodes are doing, at least if they can run in the same worker. Example? > Even without things like NUMA and readahead I'm pretty sure that you'll > want a chunk size a good bit above one page. The locks we acquire for > the buffercache lookup and for reading the page are already quite bad > for performance/scalability; even if we don't always/often hit the same > lock. Making 20 processes that scan pages in parallel acquire yet a > another lock (that's shared between all of them!) for every single page > won't be fun, especially without or fast filters. This is possible, but I'm skeptical. If the amount of other work we have to do that page is so little that the additional spinlock cycle per page causes meaningful contention, I doubt we should be parallelizing in the first place. > No, and I said so upthread. I started commenting because you argued that > architecturally parallelism belongs in heapam.c instead of upper layers, > and I can't agree with that. I now have, and it looks less bad than I > had assumed, sorry. OK, that's something. > Unfortunately I still think it's wrong approach, also sorry. > > As pointed out above (moved there after reading the patch...) I don't > think a chunk size of 1 or any other constant size can make sense. I > don't even believe it'll necessarily be constant across an entire query > execution (big initially, small at the end). Now, we could move > determining that before the query execution into executor > initialization, but then we won't yet know how many workers we're going > to get. We could add a function setting that at runtime, but that'd mix > up responsibilities quite a bit. I still think this belongs in heapam.c somehow or other. If the logic is all in the executor, then it becomes impossible for any code that doensn't use the executor to do a parallel heap scan, and that's probably bad. It's not hard to imagine something like CLUSTER wanting to reuse that code, and that won't be possible if the logic is up in some higher layer. If the logic we want is to start with a large chunk size and then switch to a small chunk size when there's not much of the relation left to scan, there's still no reason that can't be encapsulated in heapam.c. > Btw, using a atomic uint32 you'd end up without the spinlock and just > about the same amount of code... Just do a atomic_fetch_add_until32(var, > 1, InvalidBlockNumber)... ;) I thought of that, but I think there's an overflow hazard. > Where the 'coordinated seqscan' scans a relation so that each tuple > eventually gets returned once across all nodes, but the nested loop (and > through it the index scan) will just run normally, without any > coordination and parallelism. But everything below --- would happen > multiple nodes. If you agree, yes, then we're in violent agreement > ;). The "single node that gets copied" bit above makes me a bit unsure > whether we are though. Yeah, I think we're talking about the same thing. > To me, given the existing executor code, it seems easiest to achieve > that by having the ParallelismDrivingNode above having a dynamic number > of nestloop children in different backends and point the coordinated > seqscan to some shared state. As you point out, the number of these > children cannot be certainly known (just targeted for) at plan time; > that puts a certain limit on how independent they are. But since a > large number of them can be independent between workers it seems awkward > to generally treat them as being the same node across worke
Re: [HACKERS] reducing our reliance on MD5
On 02/11/2015 06:35 AM, Claudio Freire wrote: Usually because handshakes use a random salt on both sides. Not sure about pg's though, but in general collision strength is required but not slowness, they're not bruteforceable. To be precise: collision resistance is usually not important for hashes used in authentication handshakes. Not for our MD5 authentication method anyway; otherwise we'd be screwed. What you need is resistance to pre-image attacks. See https://en.wikipedia.org/wiki/Cryptographic_hash_function#Properties - Heikki -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] assessing parallel-safety
On Wed, Feb 11, 2015 at 9:39 AM, Robert Haas wrote: > On Sun, Feb 8, 2015 at 12:28 PM, Robert Haas wrote: >> On Sun, Feb 8, 2015 at 11:31 AM, Noah Misch wrote: >>> On Sat, Feb 07, 2015 at 08:18:55PM -0500, Robert Haas wrote: There are a few problems with this design that I don't immediately know how to solve: 1. I'm concerned that the query-rewrite step could substitute a query that is not parallel-safe for one that is. The upper Query might still be flagged as safe, and that's all that planner() looks at. >>> >>> I would look at determining the query's parallel safety early in the planner >>> instead; simplify_function() might be a cheap place to check. Besides >>> avoiding rewriter trouble, this allows one to alter parallel safety of a >>> function without invalidating Query nodes serialized in the system catalogs. >> >> Thanks, I'll investigate that approach. > > This does not seem to work out nicely. The problem here is that > simplify_function() gets called from eval_const_expressions() which > gets called from a variety of places, but the principal one seems to > be subquery_planner(). So if you have a query with two subqueries, > and the second one contains something parallel-unsafe, you might by > that time have already generated a parallel plan for the first one, > which won't do. Unless we want to rejigger this so that we do a > complete eval_const_expressions() pass over the entire query tree > (including all subqueries) FIRST, and then only after that go back and > plan all of those subqueries, I don't see how to make this work; and > I'm guessing that there are good reasons not to do that. And ... while I was just talking with Jan about this, I realized something that should have been blindingly obvious to me from the beginning, which is that anything we do at parse time is doomed to failure, because the properties of a function that we use to determine parallel-safety can be modified later: rhaas=# alter function random() immutable; -- no it isn't ALTER FUNCTION I think we may want a dedicated parallel-safe property for functions rather than piggybacking on provolatile, but that will probably also be changeable via ALTER FUNCTION, and stored rules won't get miraculously updated. So this definitely can't be something we figure out at parse-time ... it's got to be determined later. But at the moment I see no way to do that without an extra pass over the whole rewritten query tree. :-( -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] reducing our reliance on MD5
On 02/11/2015 03:52 PM, Robert Haas wrote: On Wed, Feb 11, 2015 at 8:02 AM, Heikki Linnakangas wrote: On 02/11/2015 02:49 PM, Robert Haas wrote: So, this all sounds fairly nice if somebody's willing to do the work, but I can't help noticing that you originally proposed adopting SCRAM in 2012, and it's 2015 now. So I wonder if anyone's really going to do all this work, and if not, whether we should go for something simpler. Just plugging something else in for MD5 would be a lot less work for us to implement and for clients to support, even if it is (as it unarguably is) less elegant. "Just plugging something else in for MD5" would still be a fair amount of work. Not that much less than the full program I proposed. Well, I guess it's easier if you immediately stop supporting MD5, have a "flag day" in all clients to implement the replacement, and break pg_dump/restore of passwords in existing databases. That sounds horrible. Let's do this properly. I can help with that, although I don't know if I'll find the time and enthusiasm to do all of it alone. So are you thinking to integrate with the Cyrus SASL library, or do you have another thought? I think we need to implement the primary MD5 replacement ourselves, so that it's always available without extra libraries. Otherwise it will not get much adoption, or the extra dependency will be a hassle anyway. It's not that complicated, after all. We could also support using a library like that for additional authentication mechanisms, though, for those who really need them. - Heikki -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] reducing our reliance on MD5
On 02/11/2015 05:34 PM, Claudio Freire wrote: On Wed, Feb 11, 2015 at 12:20 PM, José Luis Tallón wrote: Moreover, requiring everybody to change all passwords and clients *at once* seems like a very poor decision towards allowing for graceful upgrades and make rolling changes back possible, right? I wasn't advocating for what. I was just pointing out that implementation of SCRAM was an all-encompassing endeavour, in that you have to attack both password storage and the transmission protocol. For instance, if you tell me I can do SCRAM, I'll be happy to enable it if my lib has support for it. But if you add "oh... to use it, you must store plaintext on pg_authid", I'd change my mind. Because you cannot do SCRAM while storing md5. You can't do plain SCRAM while storing md5, but you could implement a variant where the stored md5 hash is treated like the plaintext password. The client first calculates the MD5(plaintext password), and then proceeds with the SCRAM authentication with that. The MD5 doesn't add any extra security (nor reduce it AFAICS), but it would allow doing SCRAM, using the MD5 hashes already stored in pg_authid. You'd still be just as vulnerable to someone stealing the MD5 hashes from pg_authid, of course. So if you don't want to continue supporting MD5 authentication for backwards-compatibility, you'd want to transform all the hashes to SCRAM stored keys. - Heikki -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pgbench -f and vacuum
On Wed, Feb 11, 2015 at 2:00 PM, Jeff Janes wrote: > I would rather just learn to add the -n when I use -f > and don't have the default tables in place, than have to learn new methods > for saying "no really, I left -n off on purpose" when I have a custom file > which does use the default tables and I want them vacuumed. Quite. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ibm system z in the buildfarm
On 02/11/2015 05:48 AM, David Fetter wrote: > On Tue, Feb 10, 2015 at 05:49:18PM -0800, Mark Wong wrote: >> Hi everyone, >> >> I've seen in the archive a call for more architecture coverage so I just >> wanted to send a quick note that there is now Linux on System Z in the >> buildfarm now: >> >> http://pgbuildfarm.org/cgi-bin/show_history.pl?nm=nudibranch&br=HEAD > > That's super awesome! > > Do you have access to put a z/OS animal there, too? The compilers > available for that OS, as well as the OS itself, should make for some > very interesting results. Hmm, I don't think so, but we'll do it if the opportunity comes around. Regards, Mark -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] parallel mode and parallel contexts
On Wed, Feb 11, 2015 at 1:59 PM, Robert Haas wrote: > I'm not sure what you mean by the "severity" of the lock. How about > marking the locks that the worker inherited from the parallel master > and throwing an error if it tries to lock one of those objects in a > mode that it does not already hold? That'd probably catch a sizeable > number of programming errors without tripping up many legitimate use > cases (and we can always relax or modify the prohibition if it later > turns out to be problematic). Or maybe do that only when the new lock mode is stronger than one it already holds. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] pgbench -f and vacuum
On Tue, Dec 23, 2014 at 7:42 AM, Alvaro Herrera wrote: > Robert Haas wrote: > > On Mon, Dec 22, 2014 at 6:55 PM, Alvaro Herrera > > wrote: > > > Here's a completely different idea. How about we add an option that > > > means "vacuum this table before running the test" (can be given several > > > times); by default the set of vacuumed tables is the current pgbench_* > > > list, but if -f is specified then the default set is cleared. So if > you > > > have a -f script and want to vacuum the default tables, you're forced > to > > > give a few --vacuum-table=foo options. But this gives the option to > > > vacuum some other table before the test, not just the pgbench default > > > ones. > > > > Well, really, you might want arbitrary initialization steps, not just > > vacuums. We could have --init-script=FILENAME. > > "Init" (pgbench -i) is the step that creates the tables and populates > them, so I think this would need a different name, maybe "startup," but > otherwise yeah. > > > Although that might be taking this thread rather far off-topic. > > Not really sure about that, because the only outstanding objection to > this discussion is what happens in the startup stage if you specify -f. > Right now vacuum is attempted on the standard tables, which is probably > not the right thing in the vast majority of cases. But if we turn that > off, how do we reinstate it for the rare cases that want it? Personally > I would just leave it turned off and be done with it, but if we want to > provide some way to re-enable it, this --startup-script=FILE gadget > sounds like a pretty decent idea. > There are two (or more?) possible meanings of a startup script. One would be run a single time at the start of a pgbench run, and one would be run at the start of each connection, in the case of -C or -c. Vacuums would presumably go in the first category, while something like tweaking a work_mem or enable_* setting would use the second. I'd find more use for the second way. I had a patch to do this on a per connection basis a while ago, but it took the command as a string to --startup. Robert suggested it be a filename rather than a string, and I agreed but never followed up with a different patch, as I couldn't figure out how to refactor the code that parses -f files so that it could be used for this without undo replication of the code. See < http://www.postgresql.org/message-id/CAMkU=1xV3tYKoHD8U2mQzfC5Kbn_bdcVf8br-EnUvy-6Z=b...@mail.gmail.com > I was wondering if we could't invent three new backslash commands. One would precede an SQL command to be run during -i, and ignored any other time (and then during -i any unbackslashed commands would be ignored) One would precede an SQL command to be run upon starting up a pgbench run. One would precede an SQL command to be run upon starting up a benchmarking connection. That way you could have a single file that would record its own initialization requirements. One problem is I don't know how you would merge together multiple -f arguments. Another is I would want to be able to override the per-connection command without having to use sed or something to edit-in-place the SQL file. But as far as what has been discussed on the central topic of this thread, I think that doing the vacuum and making the failure for non-existent tables be non-fatal when -f is provided would be an improvement. Or maybe just making it non-fatal at all times--if the table is needed and not present, the session will fail quite soon anyway. I don't see the other changes as being improvements. I would rather just learn to add the -n when I use -f and don't have the default tables in place, than have to learn new methods for saying "no really, I left -n off on purpose" when I have a custom file which does use the default tables and I want them vacuumed. Cheers, Jeff
Re: [HACKERS] GSoC 2015 - mentors, students and admins.
On 02/11/2015 04:33 AM, Anastasia Lubennikova wrote: > I'm interested to participate as student again. Let's *really* hope for no US sanctions, then. -- Josh Berkus PostgreSQL Experts Inc. http://pgexperts.com -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [pgsql-advocacy] GSoC 2015 - mentors, students and admins.
I'm interested to participate as student. Regards, Grzegorz Parka 2015-02-11 17:55 GMT+01:00 Stephen Frost : > Thom, > > * Thom Brown (t...@linux.com) wrote: > > Before I do that, I'd like to have an idea of how many people are > > interested in being either a student or a mentor. > > I'm happy to participate as a mentor again this year. > > Thanks! > > Stephen >
[HACKERS] Standby receiving part of missing WAL segment
Hi, Today I witnessed a situation which appears to have gone down like this: - The primary server starting streaming WAL data from segment 00A8 to the standby - The standby server started receiving that data - Before 00A8 is finished, the wal sender process dies on the primary, but the archiver process continues, and 00A8 ends up being archived as usual - The primary continues to generate WAL and cleans up old WAL files from pg_xlog until 00A8 is gone. - The primary is restarted and the wal sender process is back up and running - The standby says "waiting for 00A8", which it can no longer get from the primary - 00A8 is in the standby's archive directory, but the standby is waiting for the rest of the segment from the primary via streaming replication, so doesn't check the archive - The standby is restarted - The standby goes back into recovery and eventually replays 00A8 and continues as normal. Should the standby be able to get feedback from the primary that the requested segment is no longer available, and therefore know to check its archive? Or should it check the archive anyway if it hasn't received any further WAL data via the streaming replication connection after a certain amount of time? At the moment, the standby gets stuck forever in this situation, even though it has access to the WAL it needs. Thom
Re: [HACKERS] libpq's multi-threaded SSL callback handling is busted
Jan Urbański writes:
> I did some more digging on bug
> http://www.postgresql.org/message-id/cahul3dpwyfnugdgo95ohydq4kugdnbkptjq0mnbtubhcmg4...@mail.gmail.com
> which describes a deadlock when using libpq with SSL in a multi-threaded
> environment with other threads doing SSL independently.
>
> [reproducing instructions]
>
> I posit we should remove all CRYPTO_set_*_callback functions and associated
> cruft from libpq.
>
> I could submit a patch to get rid of the crazy CRYPTO_*_callback dance in
> libpq, but at the very least this will require a warning in the release notes
Attached is a patch doing just that.
> I would very much like to have this change back-patched, since setting and
> resetting the callback makes using libpq in a threaded OpenSSL-enabled app
> arguably less safe than if it didn't use any locking.
Also attached is a patch for 9.4 and all previous supported releases, which is
the same thing, but adjusted for when we didn't have a separate fe-secure.c and
fe-secure-openssl.c
If committed, this change will warrant a notice in the release notes. I could
try drafting it, if that'd be helpful.
Cheers,
Jan
commit 62f7433f697d49ab005ad22822dc943661a4e48a
Author: Jan Urbański
Date: Wed Feb 11 17:47:09 2015 +0100
Do not attempt to manage OpenSSL locking callbacks in libpq.
According to OpenSSL documentation, threaded programs using SSL should register
locking callbacks before starting work. In 6daf396879b6502c41146cdb1013568654f52ff6
libpq started doing that, registering its callbacks when the connection would
switch to SSL mode.
This would lead to overwriting any locking callback a properly written threaded
SSL application might have already set, but only got uncovered as a bug in
http://www.postgresql.org/message-id/48620925.6070...@pws.com.au where it
turned out that unloading libpq would leave a dangling function
pointer. Specifically, the PHP runtime would segfault after unloading libpq, as
reported in https://bugs.php.net/bug.php?id=40926
Commit 4e816286533dd34c10b368487d4079595a3e1418 made libpq unset the callback
after the last SSL connection was closed. This solved the segfault issue, but
introduced a potential for deadlocks when one thread using libpq with SSL and
another doing an unrelated SSL operation.
T1 (libpq) T2 (other)
start libpq connection
add locking callback
start SSL operation
take lock
finish libpq connection
remove locking callback
(!) release lock, noop since no callback
This got reported in bug
http://www.postgresql.org/message-id/cahul3dpwyfnugdgo95ohydq4kugdnbkptjq0mnbtubhcmg4...@mail.gmail.com
The reality is that it's not libpq's responsibility to manage those OpenSSL
callbacks. It's up to application code to set them up before using OpenSSL in
threads and trying to handle them in libpq is just stomping on the ones that
should already by in place by the time we start using shared OpenSSL
structures.
This commit gets rid of all OpenSSL callback handling from libpq. For
well-behaved applications, this should increase reliability, since they will
now have certainty that the callbacks they set up before attempting to use
OpenSSL will be used throughout the program execution.
Applications that relied on libpq to set up locking callbacks will need to be
fixed.
diff --git a/src/interfaces/libpq/fe-secure-openssl.c b/src/interfaces/libpq/fe-secure-openssl.c
index a32af34..5d0b132 100644
--- a/src/interfaces/libpq/fe-secure-openssl.c
+++ b/src/interfaces/libpq/fe-secure-openssl.c
@@ -67,7 +67,6 @@ static int verify_cb(int ok, X509_STORE_CTX *ctx);
static int verify_peer_name_matches_certificate_name(PGconn *conn,
ASN1_STRING *name,
char **store_name);
-static void destroy_ssl_system(void);
static int initialize_SSL(PGconn *conn);
static PostgresPollingStatusType open_client_SSL(PGconn *);
static char *SSLerrmessage(void);
@@ -90,8 +89,6 @@ static bool pq_init_crypto_lib = true;
static SSL_CTX *SSL_context = NULL;
#ifdef ENABLE_THREAD_SAFETY
-static long ssl_open_connections = 0;
-
#ifndef WIN32
static pthread_mutex_t ssl_config_mutex = PTHREAD_MUTEX_INITIALIZER;
#else
@@ -112,16 +109,6 @@ static long win32_ssl_create_mutex = 0;
void
pgtls_init_library(bool do_ssl, int do_crypto)
{
-#ifdef ENABLE_THREAD_SAFETY
-
- /*
- * Disallow changing the flags while we have open connections, else we'd
- * get completely confused.
- */
- if (ssl_open_connections != 0)
- return;
-#endif
-
pq_init_ssl_lib = do_ssl;
pq_init_crypto_lib = do_crypto;
}
@@ -705,40 +692,6 @@ verify_peer_name_matches_certificate(PGconn *conn)
return found_match && !got_error;
}
-#ifdef ENABLE_THREAD_SAFETY
-/*
- * Callback functions for
Re: [HACKERS] reducing our reliance on MD5
* Heikki Linnakangas (hlinnakan...@vmware.com) wrote: > Let's do this properly. I can help with that, > although I don't know if I'll find the time and enthusiasm to do all > of it alone. Agreed- let's do it properly. This is definitely an area of interest for me and if I can ever get out from under the current things going on, I'd be happy to help also. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] reducing our reliance on MD5
All, * Heikki Linnakangas (hlinnakan...@vmware.com) wrote: > On 02/11/2015 05:57 AM, Tom Lane wrote: > >In any case, my larger point was that given the pain that we're going to > >incur here, and the certainly years-long transition interval involved, > >it would be foolish to think only about replacing the MD5 algorithm and > >not about reconsidering the context we use it in. Stuff like unreasonably > >short salt values should be dealt with at the same time. > > +1. We should pick a well-documented mechanism with well-understood > properties, rather than roll our own again. Agreed (with this and with the overall proposal from Heikki). > 4. Implement the SASL mechanism "SCRAM". That's a challenge/response > password scheme, similar to the current MD5 authentication, but with > better salt and more expensive computation (= more resistance to > dictionary attacks). Note that Kerberos/GSSAPI can be implemented through SASL too, which means we might be able to deprecate and eventually remove the code we have for that (the only caveat to that is that some the options we provide may not all be available through SASL; that said, some of the options we provide shouldn't ever be used in production environments, so..). Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] reducing our reliance on MD5
On Tue, Feb 10, 2015 at 09:30:37PM -0500, Tom Lane wrote: > I think it would be wise to take two steps back and think about what > the threat model is here, and what we actually need to improve. > Offhand I can remember two distinct things we might wish to have more > protection against: > > * scraping of passwords off the wire protocol (but is that still > a threat in an SSL world?). Better salting practice would do more > than replacing the algorithm as such for this, IMO. Agreed. In 2004 Greg Stark estimated that it would take only 64k connection attempts to get a server-supplied reply of a salt already seen that can be replayed: http://www.postgresql.org/message-id/flat/200410071728.i97hs1a16...@candle.pha.pa.us#200410071728.i97hs1a16...@candle.pha.pa.us If you have a few salts the number goes down further. I think the 32-bit salt length is the greatest risk to our existing MD5 implementation. While leaving MD5 has a theoretical benefit, using a 64-bit salt has a practical benefit. -- Bruce Momjian http://momjian.us EnterpriseDB http://enterprisedb.com + Everyone has their own god. + -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] [pgsql-advocacy] GSoC 2015 - mentors, students and admins.
Thom, * Thom Brown (t...@linux.com) wrote: > Before I do that, I'd like to have an idea of how many people are > interested in being either a student or a mentor. I'm happy to participate as a mentor again this year. Thanks! Stephen signature.asc Description: Digital signature
Re: [HACKERS] Manipulating complex types as non-contiguous structures in-memory
* Tom Lane (t...@sss.pgh.pa.us) wrote: > I think restricting this feature to varlena types is just fine. Agreed. Thanks, Stephen signature.asc Description: Digital signature
Re: [HACKERS] reducing our reliance on MD5
On 02/11/2015 07:54 AM, José Luis Tallón wrote: On 02/11/2015 04:40 PM, Tom Lane wrote: =?UTF-8?B?Sm9zw6kgTHVpcyBUYWxsw7Nu?= writes: In any case, just storing the "password BLOB"(text or base64 encoded) along with a mechanism identifier would go a long way towards making this part pluggable... just like we do with LDAP/RADIUS/Kerberos/PAM today. That's exactly the direction we must NOT go. From a practitioners and one step at a time perspective, why don't we just offer SHA-2 as an alternative to MD5? As a longer term approach, it seems something like key based auth (ala SSH) which proved popular when I brought it up before seems like a reasonable solution. Sincerely, Joshua D. Drake -- Command Prompt, Inc. - http://www.commandprompt.com/ 503-667-4564 PostgreSQL Support, Training, Professional Services and Development High Availability, Oracle Conversion, @cmdpromptinc "If we send our children to Caesar for their education, we should not be surprised when they come back as Romans." -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] reducing our reliance on MD5
On 02/11/2015 04:40 PM, Tom Lane wrote: =?UTF-8?B?Sm9zw6kgTHVpcyBUYWxsw7Nu?= writes: In any case, just storing the "password BLOB"(text or base64 encoded) along with a mechanism identifier would go a long way towards making this part pluggable... just like we do with LDAP/RADIUS/Kerberos/PAM today. That's exactly the direction we must NOT go. Upgrading the security of stored passwords in pg_authid is at least as important as upgrading the wire protocol security; very possibly more so. Any solution that requires cleartext passwords to be kept by the server is simply not going to be accepted. I definitively haven't explained myself properly. I *never* suggested storing plaintext in pg_authid, but using plaintext authentication (which can always be matched against an on-disk hash, whatever the type) as a fallback to allow for seamless upgrades of security. (once you are authenticated by using the old credentials, the server can transparently store the new hash) When I referred to a "text or base64 encoded" I never implied on-disk plaintext (unless the user specifically requires that, which they might). To avoid ambiguities, my proposal closely mimicks Dovecot's implementation of password schemes and credential upgrades http://wiki2.dovecot.org/Authentication/PasswordSchemes Thanks, J.L. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] reducing our reliance on MD5
=?UTF-8?B?Sm9zw6kgTHVpcyBUYWxsw7Nu?= writes: > In any case, just storing the "password BLOB"(text or base64 encoded) > along with a mechanism identifier would go a long way towards making > this part pluggable... just like we do with LDAP/RADIUS/Kerberos/PAM today. That's exactly the direction we must NOT go. Upgrading the security of stored passwords in pg_authid is at least as important as upgrading the wire protocol security; very possibly more so. Any solution that requires cleartext passwords to be kept by the server is simply not going to be accepted. Because of this constraint, I really suspect that we have zero chance of achieving pluggability or farming out the problem to some third party library. Or in short: we've done that before, with LDAP/RADIUS/Kerberos/PAM, and none of those solutions have proven very satisfactory; they certainly have not replaced passwords to any measurable degree. Expecting the next external solution to do so is the definition of insanity. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] reducing our reliance on MD5
On Wed, Feb 11, 2015 at 12:20 PM, José Luis Tallón wrote: >> Both seem a step backwards IMO. > > Hmmm... as opposed to breaking applications innecesarily when simply > enabling SSL/TLS would not make it insecure? or when users don't really need > it? No, as opposed to cases where people are already using md5 authentication effectively. > Moreover, requiring everybody to change all passwords and clients *at once* > seems like a very poor decision towards allowing for graceful upgrades and > make rolling changes back possible, right? I wasn't advocating for what. I was just pointing out that implementation of SCRAM was an all-encompassing endeavour, in that you have to attack both password storage and the transmission protocol. For instance, if you tell me I can do SCRAM, I'll be happy to enable it if my lib has support for it. But if you add "oh... to use it, you must store plaintext on pg_authid", I'd change my mind. Because you cannot do SCRAM while storing md5. And there lies the issue I'm pointing out. I'm not giving solutions. Except, maybe, that if it were to be explicit for user creation: CREATE ROLE someone WITH ENCRYPTED PASSWORD '1234' USING 'pbkdf2-hmac-sha1' ; This would preclude authentication using md5, of course, but it would be expectable and under admin control. And I myself would use it. > Additionally, there are cases where passwords are not stored in plaintext > anywhere (required to be able to generate new credentials) and updating all > clients at once simply isn't possible. I agree -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] reducing our reliance on MD5
On 02/11/2015 03:55 PM, Claudio Freire wrote: On 02/11/2015 03:39 PM, Claudio Freire wrote: [snip] Seems the risk of someone either lifting pg_authid from disk or by hacking the system and being postgres, thereby accessing passwords stored somewhere else, is actually the bigger problem. But also one that should be reasonably easy (TM) to fix in a backwards compatible way? (just rewrite with a new hash whenever the password is changed, but keep reading md5 until they are all replaced. Problem with all challenge-response authentication protocols, is that the hash you have stored has to match the hash you use on the wire protocol. It's not like you can store a SHA and provide MD5 authentication. Yes, except that you can do "fallback to plaintext" if the client requests (S)CRAM-SHA and you have (S)CRAM-MD5 instead, allowing for some interoperability and backwards compatibility in the process: pre-change libpq/JDBC could authenticate using password to a server with just SCRAM-SHA512 credentials. In any case, just storing the "password BLOB"(text or base64 encoded) along with a mechanism identifier would go a long way towards making this part pluggable... just like we do with LDAP/RADIUS/Kerberos/PAM today. Wait... you mean falling back to storing plaintext or falling back to transmitting plaintext over the wire? Fallback to authentication using plaintext ... if the client is old. And might as well require SSL before allowing that --- the sane solution, definitively. But let us not forget that there exist environments where even non-SSL plaintext is acceptable when the network is secure/trusted. We should not prevent that usage if users need it (performance or management reasons) if at all possible, I think. This would imply adding a fallback "or plaintext negotiation when possible" to the new mechanisms. We could even restrict that behaviour to "hostssl" entries in pg_hba.conf Both seem a step backwards IMO. Hmmm... as opposed to breaking applications innecesarily when simply enabling SSL/TLS would not make it insecure? or when users don't really need it? There are many organizations out there happily using 3rd party applications that they can not modify easily and this change would break them gratuitously. - New policy. Introduce new hashes, update credentials (when possible!) - New applications can benefit from the newer/better security immediately - Client app that uses older libpq/JDBC. Doesn't understand the new mech - ... and now the user is left with a broken app and no easy way back alternative: old app has to authenticate itself just as if pg_hba.conf said "password", possibly restricted to doing so over an SSL-secured connection. Moreover, requiring everybody to change all passwords and clients *at once* seems like a very poor decision towards allowing for graceful upgrades and make rolling changes back possible, right? Additionally, there are cases where passwords are not stored in plaintext anywhere (required to be able to generate new credentials) and updating all clients at once simply isn't possible. Please note that, AFAICS, adding a second entry to pg_hba.conf with the fallback mechanism won't work. Now I come to think of it, the behaviour I just proposed would help users and maintainers of connection poolers to achieve seamless upgrades while increasing security (the pooler would negotiate using the newer mechanism with postgres using the client-provided password) Thanks, J.L. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] reducing our reliance on MD5
On Wed, Feb 11, 2015 at 11:48 AM, José Luis Tallón wrote: > On 02/11/2015 03:39 PM, Claudio Freire wrote: >> >> [snip] >> Seems the risk of someone either lifting pg_authid from disk or by hacking >> the system and being postgres, thereby accessing passwords stored >> somewhere >> else, is actually the bigger problem. But also one that should be >> reasonably >> easy (TM) to fix in a backwards compatible way? (just rewrite with a new >> hash whenever the password is changed, but keep reading md5 until they are >> all replaced. >> Problem with all challenge-response authentication protocols, is that >> the hash you have stored has to match the hash you use on the wire >> protocol. >> >> It's not like you can store a SHA and provide MD5 authentication. > > > Yes, except that you can do "fallback to plaintext" if the client requests > (S)CRAM-SHA and you have (S)CRAM-MD5 instead, allowing for some > interoperability and backwards compatibility in the process: pre-change > libpq/JDBC could authenticate using password to a server with just > SCRAM-SHA512 credentials. > > In any case, just storing the "password BLOB"(text or base64 encoded) along > with a mechanism identifier would go a long way towards making this part > pluggable... just like we do with LDAP/RADIUS/Kerberos/PAM today. Wait... you mean falling back to storing plaintext or falling back to transmitting plaintext over the wire? Both seem a step backwards IMO. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] reducing our reliance on MD5
On 02/11/2015 03:39 PM, Claudio Freire wrote: [snip] Seems the risk of someone either lifting pg_authid from disk or by hacking the system and being postgres, thereby accessing passwords stored somewhere else, is actually the bigger problem. But also one that should be reasonably easy (TM) to fix in a backwards compatible way? (just rewrite with a new hash whenever the password is changed, but keep reading md5 until they are all replaced. Problem with all challenge-response authentication protocols, is that the hash you have stored has to match the hash you use on the wire protocol. It's not like you can store a SHA and provide MD5 authentication. Yes, except that you can do "fallback to plaintext" if the client requests (S)CRAM-SHA and you have (S)CRAM-MD5 instead, allowing for some interoperability and backwards compatibility in the process: pre-change libpq/JDBC could authenticate using password to a server with just SCRAM-SHA512 credentials. In any case, just storing the "password BLOB"(text or base64 encoded) along with a mechanism identifier would go a long way towards making this part pluggable... just like we do with LDAP/RADIUS/Kerberos/PAM today. / J.L. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] reducing our reliance on MD5
On 02/11/2015 03:14 PM, Magnus Hagander wrote: [snip] The hash value in pg_authid already contains "md5" as a prefix. No need for another column. Yes, but for variable length mechanism names (i.e. not just 3 chars) it would become increasingly difficult to differentiate between the algo name and the stored credentials especially if we delegated the list of available mechanisms to an external library and/or in the case of upgrades. (variable-length matching based on a table of available mechs and using strncmp isn't complicated, admittedly but why bother?) ... plus we have already added many new columns to store the new "capabilities" in, as opposed to a bitmask. I might well be overlooking something else, of course. Regards, / J.L. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] reducing our reliance on MD5
On Wed, Feb 11, 2015 at 10:31 AM, Magnus Hagander wrote: > On Wed, Feb 11, 2015 at 4:57 AM, Tom Lane wrote: >> >> Robert Haas writes: >> > On Tue, Feb 10, 2015 at 9:30 PM, Tom Lane wrote: >> >> Another thing we need to keep in mind besides client compatibility >> >> is dump/reload compatibility. >> >> > I don't think there's an issue with dump/reload compatibility as far >> > as what I proposed, since it only has to do with the authentication >> > procedure, not what gets stored in pg_authid. We might have reasons >> > for moving that away from MD5 as well, but it's a separate project. >> >> Hm, well, that doesn't really square with your other expressed opinion: >> >> >> Are there other goals? >> >> > I think the goal is "stop using MD5, or at least have an option to not >> > use MD5, because people think that's insecure". >> >> As you say, it's quite debatable whether MD5 is or isn't secure enough >> given the way we use it, but what's not debatable is that the optics of it >> are not very good anymore. However, if we want to shut up the peanut >> gallery on this point, we have to get rid of MD5 usage in pg_authid not >> just the on-the-wire protocol --- I seriously doubt that the knee jerk >> MD5-is-insecure crowd will make any distinction there. So I'm not >> following how you're satisfied with a proposal for just the latter. >> >> In any case, my larger point was that given the pain that we're going to >> incur here, and the certainly years-long transition interval involved, >> it would be foolish to think only about replacing the MD5 algorithm and >> not about reconsidering the context we use it in. Stuff like unreasonably >> short salt values should be dealt with at the same time. > > > > All discussion seems to be about the protocol, which is also the harder > problem, isn't it? > > ISTM that the more *important* thing to fix is the on-disk storage in > pg_authid. > > If you actually worry about someone sniffing your network and mounting an > attack against a password, you can use SSL. That means we already have a way > to deal with that, whereas we don't have a way to mitigate the pg_authid > thing. And also, if you are actually worried about someone sniffing an > MD5'ed password, shouldn't you also worry about that same person sniffing > all the *contents* of your database, which the password is there to protect? > > (and in that regard, LDAP and such encryption algorithms send the password > in clear text in that case) > > Seems the risk of someone either lifting pg_authid from disk or by hacking > the system and being postgres, thereby accessing passwords stored somewhere > else, is actually the bigger problem. But also one that should be reasonably > easy (TM) to fix in a backwards compatible way? (just rewrite with a new > hash whenever the password is changed, but keep reading md5 until they are > all replaced. Problem with all challenge-response authentication protocols, is that the hash you have stored has to match the hash you use on the wire protocol. It's not like you can store a SHA and provide MD5 authentication. In essence, it makes making it pluggable or doing both changes separately as two separate improvements more difficult. -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] assessing parallel-safety
On Sun, Feb 8, 2015 at 12:28 PM, Robert Haas wrote: > On Sun, Feb 8, 2015 at 11:31 AM, Noah Misch wrote: >> On Sat, Feb 07, 2015 at 08:18:55PM -0500, Robert Haas wrote: >>> There are a few problems with this design that I don't immediately >>> know how to solve: >>> >>> 1. I'm concerned that the query-rewrite step could substitute a query >>> that is not parallel-safe for one that is. The upper Query might >>> still be flagged as safe, and that's all that planner() looks at. >> >> I would look at determining the query's parallel safety early in the planner >> instead; simplify_function() might be a cheap place to check. Besides >> avoiding rewriter trouble, this allows one to alter parallel safety of a >> function without invalidating Query nodes serialized in the system catalogs. > > Thanks, I'll investigate that approach. This does not seem to work out nicely. The problem here is that simplify_function() gets called from eval_const_expressions() which gets called from a variety of places, but the principal one seems to be subquery_planner(). So if you have a query with two subqueries, and the second one contains something parallel-unsafe, you might by that time have already generated a parallel plan for the first one, which won't do. Unless we want to rejigger this so that we do a complete eval_const_expressions() pass over the entire query tree (including all subqueries) FIRST, and then only after that go back and plan all of those subqueries, I don't see how to make this work; and I'm guessing that there are good reasons not to do that. Ideas? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] reducing our reliance on MD5
On Wed, Feb 11, 2015 at 3:26 PM, Robert Haas wrote: > On Wed, Feb 11, 2015 at 9:14 AM, Magnus Hagander > wrote: > > On Wed, Feb 11, 2015 at 3:10 PM, José Luis Tallón > > wrote: > >> > >> On 02/11/2015 02:31 PM, Magnus Hagander wrote: > >> > >> > >>> In any case, my larger point was that given the pain that we're going > to > >>> incur here, and the certainly years-long transition interval involved, > >>> it would be foolish to think only about replacing the MD5 algorithm and > >>> not about reconsidering the context we use it in. Stuff like > >>> unreasonably > >>> short salt values should be dealt with at the same time. > >> > >> > >> > >> All discussion seems to be about the protocol, which is also the harder > >> problem, isn't it? > >> > >> ISTM that the more *important* thing to fix is the on-disk storage in > >> pg_authid. > >> > >> > >> At least, looks like it would be the most urgent (and with no need for > >> clients breaking in the process, AFAICT) > >> > >> [snip] > >> Seems the risk of someone either lifting pg_authid from disk or by > hacking > >> the system and being postgres, thereby accessing passwords stored > somewhere > >> else, is actually the bigger problem. But also one that should be > reasonably > >> easy (TM) to fix in a backwards compatible way? (just rewrite with a new > >> hash whenever the password is changed, but keep reading md5 until they > are > >> all replaced. > >> > >> > >> Adding a new system column with a text or enum representing the > algorithm > >> that created the "hash" would go a lot towards fixing this situation. > >> When/If the column isn't there, just assume "md5". This would allow for > >> transparent pg_upgrade. > > > > > > The hash value in pg_authid already contains "md5" as a prefix. No need > for > > another column. > > How do we distinguish that from a password that actually starts with md5? > > You mean one stored in cleatext? I don't know offhand :) If it's actually stored with md5 (which it really should be) it's still the hash, but prefixed with the constant md5. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/
Re: [HACKERS] 9.6 Feature help requested: Inclusion Constraints
On Tue, Feb 10, 2015 at 9:47 PM, Peter Eisentraut wrote: > On 2/9/15 3:12 AM, Jeff Davis wrote: >> On Sat, 2015-02-07 at 16:08 -0800, Jeff Davis wrote: >>> I believe Inclusion Constraints will be important for postgres. >> >> I forgot to credit Darren Duncan with the name of this feature: >> >> http://www.postgresql.org/message-id/4f8bb9b0.5090...@darrenduncan.net > > I think it would be confusing to name something inclusion constraint > that is not somehow the opposite of an exclusion constraint. It sorta is. UNIQUE is to exclusion constrant as FOREIGN KEY is to inclusion constraint. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] reducing our reliance on MD5
On Wed, Feb 11, 2015 at 9:14 AM, Magnus Hagander wrote: > On Wed, Feb 11, 2015 at 3:10 PM, José Luis Tallón > wrote: >> >> On 02/11/2015 02:31 PM, Magnus Hagander wrote: >> >> >>> In any case, my larger point was that given the pain that we're going to >>> incur here, and the certainly years-long transition interval involved, >>> it would be foolish to think only about replacing the MD5 algorithm and >>> not about reconsidering the context we use it in. Stuff like >>> unreasonably >>> short salt values should be dealt with at the same time. >> >> >> >> All discussion seems to be about the protocol, which is also the harder >> problem, isn't it? >> >> ISTM that the more *important* thing to fix is the on-disk storage in >> pg_authid. >> >> >> At least, looks like it would be the most urgent (and with no need for >> clients breaking in the process, AFAICT) >> >> [snip] >> Seems the risk of someone either lifting pg_authid from disk or by hacking >> the system and being postgres, thereby accessing passwords stored somewhere >> else, is actually the bigger problem. But also one that should be reasonably >> easy (TM) to fix in a backwards compatible way? (just rewrite with a new >> hash whenever the password is changed, but keep reading md5 until they are >> all replaced. >> >> >> Adding a new system column with a text or enum representing the algorithm >> that created the "hash" would go a lot towards fixing this situation. >> When/If the column isn't there, just assume "md5". This would allow for >> transparent pg_upgrade. > > > The hash value in pg_authid already contains "md5" as a prefix. No need for > another column. How do we distinguish that from a password that actually starts with md5? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] What exactly is our CRC algorithm?
At 2015-02-11 13:20:29 +0200, hlinnakan...@vmware.com wrote: > > I don't follow. I didn't change configure at all, compared to your > patch. OK, I extrapolated a little too much. Your patch didn't actually include crc_instructions.h; from the name of the #define, I imagined you planned to move the check to configure. But now I guess from your response that PG_HAVE_CRC32C_INSTRUCTIONS would be #defined by the header (as you did say it would be). So let's forget that part. > As a case in point, with your patch pg_xlogdump would not have used > the CRC instruction, because it never called pg_choose_crc_impl(). > "Choose on first use" is much more convenient than requiring every > program to call a function. I can see your point. I'm not fond of the code, but I guess it's not a huge deal in the larger scheme of things. > I think what you're suggesting is that we'd instead have two mutually > exclusive #defines, something like "USE_CRC_SSE_ALWAYS" and > "USE_CRC_SSE_WITH_RUNTIME_CHECK". I was thinking of only one knob: USE_CRC_SSE would (try to) build the code to use the SSE instructions without any test. If it (or another USE_CRC_XXX flag) isn't set, we'd build all applicable variants and test at runtime to decide what to use. Just give me a little while to think through what that would look like, I'll post a patch. -- Abhijit -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SSL renegotiation and other related woes
Heikki Linnakangas wrote:
>> Can we work-around that easily? I believe we can. The crucial part is
>> that we mustn't let SSL_write() to process any incoming application
>> data. We can achieve that if we always call SSL_read() to drain such
>> data, before calling SSL_write(). But that's not quite enough; if we're
>> in renegotiation, SSL_write() might still try to read more data from the
>> socket that has arrived after the SSL_read() call. Fortunately, we can
>> easily prevent that by hacking pqsecure_raw_read() to always return
>> EWOULDBLOCK, when it's called through SSL_write().
>>
>> The attached patch does that. I've been running your pg_receivexlog test
>> case with this for about 15 minutes without any errors now, with
>> ssl_renegotiation_limit=50kB, while before it errored out within seconds.
>
> Here is a simplified version of this patch. It seems to be enough to not
> let SSL_write() to read any data from the socket. No need to call
> SSL_read() first, and the server-side changes I made were only needed
> because of the other patch I had applied.
>
> Thoughts? Can you reproduce any errors with this?
>
>> In theory, I guess we should do similar hacks in the server, and always
>> call SSL_read() before SSL_write(), but it seems to work without it. Or
>> maybe not; OpenSSL server and client code is not symmetric, so perhaps
>> it works in server mode without that.
>
> Not included in this patch, but I believe we apply a similar patch to
> the server-side too.
It seems to me that there is a bug in the server part of your first patch
(not included in your second patch):
--- a/src/backend/libpq/be-secure-openssl.c
+++ b/src/backend/libpq/be-secure-openssl.c
@@ -576,11 +576,11 @@ be_tls_write(Port *port, void *ptr, size_t len, int
*waitfor)
/*
* If SSL renegotiations are enabled and we're getting close to the
* limit, start one now; but avoid it if there's one already in
-* progress. Request the renegotiation 1kB before the limit has
+* progress. Request the renegotiation 32kB before the limit has
* actually expired.
*/
if (ssl_renegotiation_limit && !in_ssl_renegotiation &&
- port->count > (ssl_renegotiation_limit - 1) * 1024L)
+ port->count > (ssl_renegotiation_limit - 32) * 1024L)
{
in_ssl_renegotiation = true;
Experiment shows that for values of ssl_renegotiation_limit less than 32,
no renegotiation takes place at all with this change applied.
port->count is an unsigned long.
Yours,
Laurenz Albe
--
Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org)
To make changes to your subscription:
http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] reducing our reliance on MD5
On Wed, Feb 11, 2015 at 3:10 PM, José Luis Tallón < jltal...@adv-solutions.net> wrote: > On 02/11/2015 02:31 PM, Magnus Hagander wrote: > > > In any case, my larger point was that given the pain that we're going to >> incur here, and the certainly years-long transition interval involved, >> it would be foolish to think only about replacing the MD5 algorithm and >> not about reconsidering the context we use it in. Stuff like unreasonably >> short salt values should be dealt with at the same time. >> > > > All discussion seems to be about the protocol, which is also the harder > problem, isn't it? > > ISTM that the more *important* thing to fix is the on-disk storage in > pg_authid. > > > At least, looks like it would be the most urgent (and with no need for > clients breaking in the process, AFAICT) > > [snip] > Seems the risk of someone either lifting pg_authid from disk or by hacking > the system and being postgres, thereby accessing passwords stored somewhere > else, is actually the bigger problem. But also one that should be > reasonably easy (TM) to fix in a backwards compatible way? (just rewrite > with a new hash whenever the password is changed, but keep reading md5 > until they are all replaced. > > > Adding a new system column with a text or enum representing the algorithm > that created the "hash" would go a lot towards fixing this situation. > When/If the column isn't there, just assume "md5". This would allow for > transparent pg_upgrade. > The hash value in pg_authid already contains "md5" as a prefix. No need for another column. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/
Re: [HACKERS] reducing our reliance on MD5
On 02/11/2015 02:31 PM, Magnus Hagander wrote: In any case, my larger point was that given the pain that we're going to incur here, and the certainly years-long transition interval involved, it would be foolish to think only about replacing the MD5 algorithm and not about reconsidering the context we use it in. Stuff like unreasonably short salt values should be dealt with at the same time. All discussion seems to be about the protocol, which is also the harder problem, isn't it? ISTM that the more *important* thing to fix is the on-disk storage in pg_authid. At least, looks like it would be the most urgent (and with no need for clients breaking in the process, AFAICT) [snip] Seems the risk of someone either lifting pg_authid from disk or by hacking the system and being postgres, thereby accessing passwords stored somewhere else, is actually the bigger problem. But also one that should be reasonably easy (TM) to fix in a backwards compatible way? (just rewrite with a new hash whenever the password is changed, but keep reading md5 until they are all replaced. Adding a new system column with a text or enum representing the algorithm that created the "hash" would go a lot towards fixing this situation. When/If the column isn't there, just assume "md5". This would allow for transparent pg_upgrade. Then, based on the "default_hash" (or something to the same effect) GUC, automatically rewrite the hashes (and set the corresponding 'hash_type') upon password change. Please note that this allows for clients supporting it (be it natively or by relying on an external SASL lib or the like) to request some challenge-response mechanism ---such as SCRAM-SHA256--- when available. In fact, some forms of challenge-response-ready hashes also support "password" (plaintext) with the same authenticator, thereby perfectly replacing the functionality we have today. Existing implementation: Dovecot's CRAM-MD5 mechanism Thanks, / J.L.
Re: [HACKERS] [REVIEW] Re: Compression of full-page-writes
>IMO, we should add details about how this new field is used in the comments on >top of XLogRecordBlockImageHeader, meaning that when a page hole is present we >use the compression info structure and when there is no hole, we are sure that >the FPW raw length is BLCKSZ meaning that the two bytes of the CompressionInfo >stuff is unnecessary. This comment is included in the patch attached. > For correctness with_hole should be set even for uncompressed pages. I think > that we should as well use it for sanity checks in xlogreader.c when decoding > records. This change is made in the attached patch. Following sanity checks have been added in xlogreader.c if (!(blk->with_hole) && blk->hole_offset != 0 || blk->with_hole && blk->hole_offset <= 0)) if (blk->with_hole && blk->bkp_len >= BLCKSZ) if (!(blk->with_hole) && blk->bkp_len != BLCKSZ) Thank you, Rahila Syed __ Disclaimer: This email and any attachments are sent in strictest confidence for the sole use of the addressee and may contain legally privileged, confidential, and proprietary data. If you are not the intended recipient, please advise the sender by replying promptly to this email and then delete and destroy this email and any attachments without any further use, copying or forwarding. Support-compression-for-full-page-writes-in-WAL_v18.patch Description: Support-compression-for-full-page-writes-in-WAL_v18.patch -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] reducing our reliance on MD5
On Wed, Feb 11, 2015 at 8:02 AM, Heikki Linnakangas wrote: > On 02/11/2015 02:49 PM, Robert Haas wrote: >> So, this all sounds fairly nice if somebody's willing to do the work, >> but I can't help noticing that you originally proposed adopting SCRAM >> in 2012, and it's 2015 now. So I wonder if anyone's really going to >> do all this work, and if not, whether we should go for something >> simpler. Just plugging something else in for MD5 would be a lot less >> work for us to implement and for clients to support, even if it is (as >> it unarguably is) less elegant. > > "Just plugging something else in for MD5" would still be a fair amount of > work. Not that much less than the full program I proposed. > > Well, I guess it's easier if you immediately stop supporting MD5, have a > "flag day" in all clients to implement the replacement, and break > pg_dump/restore of passwords in existing databases. That sounds horrible. > Let's do this properly. I can help with that, although I don't know if I'll > find the time and enthusiasm to do all of it alone. So are you thinking to integrate with the Cyrus SASL library, or do you have another thought? -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] ibm system z in the buildfarm
On Tue, Feb 10, 2015 at 05:49:18PM -0800, Mark Wong wrote: > Hi everyone, > > I've seen in the archive a call for more architecture coverage so I just > wanted to send a quick note that there is now Linux on System Z in the > buildfarm now: > > http://pgbuildfarm.org/cgi-bin/show_history.pl?nm=nudibranch&br=HEAD That's super awesome! Do you have access to put a z/OS animal there, too? The compilers available for that OS, as well as the OS itself, should make for some very interesting results. Cheers, David. -- David Fetter http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] 9.6 Feature help requested: Inclusion Constraints
On Tue, Feb 10, 2015 at 09:47:22PM -0500, Peter Eisentraut wrote: > On 2/9/15 3:12 AM, Jeff Davis wrote: > > On Sat, 2015-02-07 at 16:08 -0800, Jeff Davis wrote: > >> I believe Inclusion Constraints will be important for postgres. > > > > I forgot to credit Darren Duncan with the name of this feature: > > > > http://www.postgresql.org/message-id/4f8bb9b0.5090...@darrenduncan.net > > I think it would be confusing to name something inclusion constraint > that is not somehow the opposite of an exclusion constraint. To my view, this opposition is there. In both cases, permission for a tuple to exist is based on another other tuple's presence with some generalized equality criterion. In the exclusion case, it's prevented. In the inclusion case, it's required. Cheers, David. -- David Fetter http://fetter.org/ Phone: +1 415 235 3778 AIM: dfetter666 Yahoo!: dfetter Skype: davidfetter XMPP: david.fet...@gmail.com Remember to vote! Consider donating to Postgres: http://www.postgresql.org/about/donate -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] GRANT USAGE on FOREIGN SERVER exposes passwords
Jim Nasby writes: > On 2/5/15 10:48 AM, Tom Lane wrote: >> The dblink example is entirely uncompelling, given that as you said >> somebody with access to a dblink connection could execute ALTER USER on >> the far end. > Actually, you can eliminate that by not granting direct access to dblink > functions. Instead you create a SECURITY DEFINER function that sanity > checks the SQL you're trying to run and rejects things like ALTER USER. > While you're doing that, you can also lock away the connection > information. A former coworker actually built a system that does this, > at least to a limited degree. ... but if you aren't giving the untrusted user direct access to the connection, then he also doesn't get to see its options in the view. So this still isn't compelling, so far as dblink goes. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] reducing our reliance on MD5
On Wed, Feb 11, 2015 at 4:57 AM, Tom Lane wrote: > Robert Haas writes: > > On Tue, Feb 10, 2015 at 9:30 PM, Tom Lane wrote: > >> Another thing we need to keep in mind besides client compatibility > >> is dump/reload compatibility. > > > I don't think there's an issue with dump/reload compatibility as far > > as what I proposed, since it only has to do with the authentication > > procedure, not what gets stored in pg_authid. We might have reasons > > for moving that away from MD5 as well, but it's a separate project. > > Hm, well, that doesn't really square with your other expressed opinion: > > >> Are there other goals? > > > I think the goal is "stop using MD5, or at least have an option to not > > use MD5, because people think that's insecure". > > As you say, it's quite debatable whether MD5 is or isn't secure enough > given the way we use it, but what's not debatable is that the optics of it > are not very good anymore. However, if we want to shut up the peanut > gallery on this point, we have to get rid of MD5 usage in pg_authid not > just the on-the-wire protocol --- I seriously doubt that the knee jerk > MD5-is-insecure crowd will make any distinction there. So I'm not > following how you're satisfied with a proposal for just the latter. > > In any case, my larger point was that given the pain that we're going to > incur here, and the certainly years-long transition interval involved, > it would be foolish to think only about replacing the MD5 algorithm and > not about reconsidering the context we use it in. Stuff like unreasonably > short salt values should be dealt with at the same time. > All discussion seems to be about the protocol, which is also the harder problem, isn't it? ISTM that the more *important* thing to fix is the on-disk storage in pg_authid. If you actually worry about someone sniffing your network and mounting an attack against a password, you can use SSL. That means we already have a way to deal with that, whereas we don't have a way to mitigate the pg_authid thing. And also, if you are actually worried about someone sniffing an MD5'ed password, shouldn't you also worry about that same person sniffing all the *contents* of your database, which the password is there to protect? (and in that regard, LDAP and such encryption algorithms send the password in clear text in that case) Seems the risk of someone either lifting pg_authid from disk or by hacking the system and being postgres, thereby accessing passwords stored somewhere else, is actually the bigger problem. But also one that should be reasonably easy (TM) to fix in a backwards compatible way? (just rewrite with a new hash whenever the password is changed, but keep reading md5 until they are all replaced. -- Magnus Hagander Me: http://www.hagander.net/ Work: http://www.redpill-linpro.com/
Re: [HACKERS] Manipulating complex types as non-contiguous structures in-memory
Jim Nasby writes: > On 2/10/15 5:19 PM, Tom Lane wrote: >> What do you mean by non-variant? > Ugh, sorry, brainfart. I meant to say non-varlena. > I can't think of any non-varlena types we'd want this for, but maybe > someone else can think of a case. If there is a use-case I wouldn't > handle it with this patch, but we'd want to consider it... There isn't any practical way to interpose TOAST pointers for non-varlena types, since we make no assumptions about the bit contents of fixed-length types. But I'm having a hard time thinking of a fixed-length type in which you'd have any need for a deserialized representation, either. I think restricting this feature to varlena types is just fine. regards, tom lane -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] reducing our reliance on MD5
On 11/02/15 02:30, Tom Lane wrote: [...] I think it would be wise to take two steps back and think about what the threat model is here, and what we actually need to improve. Offhand I can remember two distinct things we might wish to have more protection against: * scraping of passwords off the wire protocol (but is that still a threat in an SSL world?). Better salting practice would do more than replacing the algorithm as such for this, IMO. mitm We might consider it our problem or not, but in general terms man-in-the-middle attacks, which are easy to implement in many scenarios, are a scraping problem. In particular, I have seen tons of developers turn off SSL validation during development and not turning back it on for production, leaving servers vulnerable to password scraping under mitm attacks. So I would always considering hashing anyway. SCRAM seems to be a good solution anyway. Regards, Álvaro -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] reducing our reliance on MD5
On 02/11/2015 02:49 PM, Robert Haas wrote: So, this all sounds fairly nice if somebody's willing to do the work, but I can't help noticing that you originally proposed adopting SCRAM in 2012, and it's 2015 now. So I wonder if anyone's really going to do all this work, and if not, whether we should go for something simpler. Just plugging something else in for MD5 would be a lot less work for us to implement and for clients to support, even if it is (as it unarguably is) less elegant. "Just plugging something else in for MD5" would still be a fair amount of work. Not that much less than the full program I proposed. Well, I guess it's easier if you immediately stop supporting MD5, have a "flag day" in all clients to implement the replacement, and break pg_dump/restore of passwords in existing databases. That sounds horrible. Let's do this properly. I can help with that, although I don't know if I'll find the time and enthusiasm to do all of it alone. - Heikki -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] SSL renegotiation and other related woes
On 02/05/2015 11:03 PM, Heikki Linnakangas wrote:
On 01/26/2015 12:14 PM, Andres Freund wrote:
Can we work-around that easily? I believe we can. The crucial part is
that we mustn't let SSL_write() to process any incoming application
data. We can achieve that if we always call SSL_read() to drain such
data, before calling SSL_write(). But that's not quite enough; if we're
in renegotiation, SSL_write() might still try to read more data from the
socket that has arrived after the SSL_read() call. Fortunately, we can
easily prevent that by hacking pqsecure_raw_read() to always return
EWOULDBLOCK, when it's called through SSL_write().
The attached patch does that. I've been running your pg_receivexlog test
case with this for about 15 minutes without any errors now, with
ssl_renegotiation_limit=50kB, while before it errored out within seconds.
Here is a simplified version of this patch. It seems to be enough to not
let SSL_write() to read any data from the socket. No need to call
SSL_read() first, and the server-side changes I made were only needed
because of the other patch I had applied.
Thoughts? Can you reproduce any errors with this?
In theory, I guess we should do similar hacks in the server, and always
call SSL_read() before SSL_write(), but it seems to work without it. Or
maybe not; OpenSSL server and client code is not symmetric, so perhaps
it works in server mode without that.
Not included in this patch, but I believe we apply a similar patch to
the server-side too.
- Heikki
From b78bfbb70127cbe3f360458eb2a97dcc28194fbc Mon Sep 17 00:00:00 2001
From: Heikki Linnakangas
Date: Thu, 5 Feb 2015 22:44:58 +0200
Subject: [PATCH 1/1] Fix "sslv3 alert unexpected message" errors in SSL
renegotiation.
There seems to be a bug in OpenSSL renegotiation, so that when SSL_write()
needs to read data to complete the handshake, but it receives application
data instead, it gets confused and throws an "unexpected message" error. To
work around that, don't let SSL_write() to read data from the socket, even
if some is available. Arrange so that SSL_read() processes all incoming
messages instead.
---
src/interfaces/libpq/fe-misc.c | 14 +-
src/interfaces/libpq/fe-secure-openssl.c | 20
src/interfaces/libpq/fe-secure.c | 9 +
src/interfaces/libpq/libpq-int.h | 2 ++
4 files changed, 40 insertions(+), 5 deletions(-)
diff --git a/src/interfaces/libpq/fe-misc.c b/src/interfaces/libpq/fe-misc.c
index 945e283..9dd02d5 100644
--- a/src/interfaces/libpq/fe-misc.c
+++ b/src/interfaces/libpq/fe-misc.c
@@ -920,11 +920,15 @@ pqSendSome(PGconn *conn, int len)
* to clear the channel eventually because it's blocked trying to
* send data to us. (This can happen when we are sending a large
* amount of COPY data, and the server has generated lots of
- * NOTICE responses.) To avoid a deadlock situation, we must be
- * prepared to accept and buffer incoming data before we try
- * again. Furthermore, it is possible that such incoming data
- * might not arrive until after we've gone to sleep. Therefore,
- * we wait for either read ready or write ready.
+ * NOTICE responses.) Another scenario is that SSL renegotiation
+ * is in progress, and the SSL library needs to read a message
+ * to complete the handshake, before it can send more data.
+ *
+ * To avoid a deadlock situation, we must be prepared to accept
+ * and buffer incoming data before we try again. Furthermore, it
+ * is possible that such incoming data might not arrive until
+ * after we've gone to sleep. Therefore, we wait for either read
+ * ready or write ready.
*/
if (pqReadData(conn) < 0)
{
diff --git a/src/interfaces/libpq/fe-secure-openssl.c b/src/interfaces/libpq/fe-secure-openssl.c
index a32af34..2e75f3c 100644
--- a/src/interfaces/libpq/fe-secure-openssl.c
+++ b/src/interfaces/libpq/fe-secure-openssl.c
@@ -319,8 +319,28 @@ pgtls_write(PGconn *conn, const void *ptr, size_t len)
char sebuf[256];
int err;
+ /*
+ * To work-around an issue with OpenSSL and renegotiation, we don't
+ * let SSL_write() to read any incoming data.
+ *
+ * If SSL_write() is called, and renegotiation has just been inititated,
+ * SSL_write() might try to read from the socket to complete the
+ * handshake. If there was some application data in-flight, it might
+ * receive the application data instead. That confuses it, and it throws
+ * an "sslv3 alert unexpected message" error.
+ *
+ * We avoid that setting a kill-switch, pqsecure_block_raw_read, which
+ * tells pqsecure_raw_read() to not return any data to the caller, even
+ * if some is available.
+ *
+ * NB: This relies on the calling code to call pqsecure_read(), completing
+ * the renegotiation handshake, if pqsecure_write() returns 0. Otherwise
+ * we'll never make progress.
+ */
SOCK_ERRNO_SET(0);
+ pqsecure_block_raw_read = true;
n = SSL_write(conn->ssl,
Re: [HACKERS] reducing our reliance on MD5
On Wed, Feb 11, 2015 at 4:13 AM, Heikki Linnakangas wrote: > For the first cut, I propose: > > 1. Implement SASL. The server sends a new AuthenticationSASLRequest message, > to which the client responds with AuthenticationSASLResponse message. These > messages carry the SASL challenge/response messages, until AuthenticationOK > is received. Similar to our GSSAPI implementation. (Note that this doesn't > require the use of any 3rd party libraries or such. The specification for > SASL itself is very short and high-level.) > > 2. Add a way for the client and server to negotiate which authentication > mechanism to use. Currently, the server dictates it to the client, and if it > doesn't support that mechanism, it has to disconnect. To make it possible to > add new mechanisms gracefully, with a long transition period, we need a way > to negotiate. The server should send a list of allowed authentication > methods in the first AuthenticationSASLRequest message, and the client picks > one. > > 3. To support old clients that don't understand the new > AuthenticationSASLRequest message, the client tells that it supports it > before the authentication begins. It does that by tacking a special option > "protocol.extensions=sasl" in the startup packet (more extensions could be > added there in the future, as a comma-separated list). With 9.2 and above > servers, the server will ignore unrecognized options containing a dot. 9.1 > and earlier will throw an error, but the client can then retry without the > option, like libpq does for application_name. > > 4. Implement the SASL mechanism "SCRAM". That's a challenge/response > password scheme, similar to the current MD5 authentication, but with better > salt and more expensive computation (= more resistance to dictionary > attacks). > > 5. Modify/add syntax to ALTER USER SET PASSWORD to allow storing the new > SCRAM, and other, authentication information in pg_authid. > > That's it for starters. The SCRAM mechanism is a fairly straightforward > replacement for MD5. This scheme makes it possible to add more mechanisms > later, without requiring all clients to immediately support them. So, this all sounds fairly nice if somebody's willing to do the work, but I can't help noticing that you originally proposed adopting SCRAM in 2012, and it's 2015 now. So I wonder if anyone's really going to do all this work, and if not, whether we should go for something simpler. Just plugging something else in for MD5 would be a lot less work for us to implement and for clients to support, even if it is (as it unarguably is) less elegant. -- Robert Haas EnterpriseDB: http://www.enterprisedb.com The Enterprise PostgreSQL Company -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] GSoC 2015 - mentors, students and admins.
I'm interested to participate as student again. -- Best regards, Lubennikova Anastasia
Re: [HACKERS] What exactly is our CRC algorithm?
On 02/11/2015 11:28 AM, Abhijit Menon-Sen wrote: 1. I don't mind moving platform-specific tests for CRC32C instructions to configure, but if we only define PG_HAVE_CRC32C_INSTRUCTIONS, we would anyway have to reproduce all that platform-specific stuff in the header file. To do it properly, I think we should generate the right version of crc_instructions.h for the platform. Otherwise, what's the point? Might as well have only the complicated header. I don't follow. I didn't change configure at all, compared to your patch. 2. At this point, I think we should stick with the _sse function rather than a generic _hw one to "drive" the platform-specific instructions. The structure of that function (n/8*(8bytes)+n%8*(bytes)) is specific to what works best for the SSE instructions. On another platform, we may need something very similar, or very different. With the proposed structure, _hw would inevitably become #ifdef'ed non-trivially, e.g. to run a single-byte loop at the start to align the main loop, but only for certain combinations of #defines. I think we should consider what makes the most sense when someone actually wants to add support for another platform/compiler, and not before. Hmm, ok. The ARM CRC32C instruction is very similar to the SSE4.2 one, but I presume it does require alignment. 3. I dislike everything about pg_crc32_instructions_runtime_check()—the name, the separate PG_CRC32C_INSTRUCTIONS_NEED_RUNTIME_CHECK #define, the fact that you try to do the test "inline" rather than at startup… Maybe you're trying to avoid checking separately at startup so that a program that links with code from src/common doesn't need to do its own test. But I don't think the results are worth it. As a case in point, with your patch pg_xlogdump would not have used the CRC instruction, because it never called pg_choose_crc_impl(). "Choose on first use" is much more convenient than requiring every program to call a function. As for omitting the slicing-by-8 tables, if there's a more compelling reason to do that than "Gentoo users might like that ;-)", I think it should be done with a straightforward -DUSE_CRC_SSE style arrangement rather than figuring out that you HAVE_CRC32C_INSTRUCTIONS but don't NEED_RUNTIME_CHECK. If you want to build special-purpose binaries, just pick whichever CRC implementation you like, done. I stopped short of actually allowing you to force the use of SSE, e.g. with -DUSE_CRC_SSE, but my thinking was that that would force PG_HAVE_CRC32C_INSTRUCTIONS to be set, and NEEDS_RUNTIME_CHECK to be unset. I.e. those two #defines control what code is generated, but in the header file. I think what you're suggesting is that we'd instead have two mutually exclusive #defines, something like "USE_CRC_SSE_ALWAYS" and "USE_CRC_SSE_WITH_RUNTIME_CHECK". It wouldn't be much different, but would require some places to check for both. It would be nice to use GCC's builtin intrinsics, __builtin_ia32_crc32* where available, but I guess those are only available if you build with -msse4.2, and that would defeat the point of the runtime check. Exactly. Hmm. Perhaps we should check for the built-in functions, and if they're available, use them and not set NEEDS_RUNTIME_CHECK. If you compile with -msse4.2, the code won't run without SSE4.2 anyway (or at least isn't guaranteed to run). - Heikki -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] What exactly is our CRC algorithm?
At 2015-02-10 14:30:51 +0200, hlinnakan...@vmware.com wrote: > > I propose that we add a new header file in src/port, let's call it > crc_instructions.h […] the point of the crc_instructions.h header > is to hide the differences between compilers and architectures. Moving the assembly code/compiler intrinsics to a separate header is fine with me, but I really don't like the proposed implementation at all. Here are some comments: 1. I don't mind moving platform-specific tests for CRC32C instructions to configure, but if we only define PG_HAVE_CRC32C_INSTRUCTIONS, we would anyway have to reproduce all that platform-specific stuff in the header file. To do it properly, I think we should generate the right version of crc_instructions.h for the platform. Otherwise, what's the point? Might as well have only the complicated header. 2. At this point, I think we should stick with the _sse function rather than a generic _hw one to "drive" the platform-specific instructions. The structure of that function (n/8*(8bytes)+n%8*(bytes)) is specific to what works best for the SSE instructions. On another platform, we may need something very similar, or very different. With the proposed structure, _hw would inevitably become #ifdef'ed non-trivially, e.g. to run a single-byte loop at the start to align the main loop, but only for certain combinations of #defines. I think we should consider what makes the most sense when someone actually wants to add support for another platform/compiler, and not before. 3. I dislike everything about pg_crc32_instructions_runtime_check()—the name, the separate PG_CRC32C_INSTRUCTIONS_NEED_RUNTIME_CHECK #define, the fact that you try to do the test "inline" rather than at startup… Maybe you're trying to avoid checking separately at startup so that a program that links with code from src/common doesn't need to do its own test. But I don't think the results are worth it. As for omitting the slicing-by-8 tables, if there's a more compelling reason to do that than "Gentoo users might like that ;-)", I think it should be done with a straightforward -DUSE_CRC_SSE style arrangement rather than figuring out that you HAVE_CRC32C_INSTRUCTIONS but don't NEED_RUNTIME_CHECK. If you want to build special-purpose binaries, just pick whichever CRC implementation you like, done. > I believe the CRC instructions are also available in 32-bit mode, so > the check for __x86_64__ is not needed. Right? I'm not sure, but I guess you're right. > It would be nice to use GCC's builtin intrinsics, > __builtin_ia32_crc32* where available, but I guess those are only > available if you build with -msse4.2, and that would defeat the > point of the runtime check. Exactly. -- Abhijit -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] reducing our reliance on MD5
On 02/11/2015 05:57 AM, Tom Lane wrote: In any case, my larger point was that given the pain that we're going to incur here, and the certainly years-long transition interval involved, it would be foolish to think only about replacing the MD5 algorithm and not about reconsidering the context we use it in. Stuff like unreasonably short salt values should be dealt with at the same time. +1. We should pick a well-documented mechanism with well-understood properties, rather than roll our own again. Apart from the weaknesses in the MD5 authentication method itself, our protocol doesn't make it easy to add new methods and still support old clients gracefully. We should address that too. For the first cut, I propose: 1. Implement SASL. The server sends a new AuthenticationSASLRequest message, to which the client responds with AuthenticationSASLResponse message. These messages carry the SASL challenge/response messages, until AuthenticationOK is received. Similar to our GSSAPI implementation. (Note that this doesn't require the use of any 3rd party libraries or such. The specification for SASL itself is very short and high-level.) 2. Add a way for the client and server to negotiate which authentication mechanism to use. Currently, the server dictates it to the client, and if it doesn't support that mechanism, it has to disconnect. To make it possible to add new mechanisms gracefully, with a long transition period, we need a way to negotiate. The server should send a list of allowed authentication methods in the first AuthenticationSASLRequest message, and the client picks one. 3. To support old clients that don't understand the new AuthenticationSASLRequest message, the client tells that it supports it before the authentication begins. It does that by tacking a special option "protocol.extensions=sasl" in the startup packet (more extensions could be added there in the future, as a comma-separated list). With 9.2 and above servers, the server will ignore unrecognized options containing a dot. 9.1 and earlier will throw an error, but the client can then retry without the option, like libpq does for application_name. 4. Implement the SASL mechanism "SCRAM". That's a challenge/response password scheme, similar to the current MD5 authentication, but with better salt and more expensive computation (= more resistance to dictionary attacks). 5. Modify/add syntax to ALTER USER SET PASSWORD to allow storing the new SCRAM, and other, authentication information in pg_authid. That's it for starters. The SCRAM mechanism is a fairly straightforward replacement for MD5. This scheme makes it possible to add more mechanisms later, without requiring all clients to immediately support them. - Heikki -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] reducing our reliance on MD5
On 02/11/2015 04:38 AM, Tom Lane wrote: Peter Eisentraut writes: On 2/10/15 8:28 PM, Robert Haas wrote: I don't actually care which algorithm we use, and I dowannahafta care. What I do want to do is provide a framework so that, when somebody discovers that X is better than Y because Z, somebody who knows about cryptography and not much about PostgreSQL ca add support for X in a relatively small number of lines of code. sounds like SASL Sounds like pie in the sky really :-(. SASL won't automatically solve any of ourproblems. We still have to write the support for all the authentication mechanisms we wanted to support. The nice thing about SASL is there is a list of well-understood and documented mechanisms that go with it, with canonical names like "SCRAM-SHA-1". If we use those, it's well-defined what the mechanism is, with less design work for us. And who knows, it just might make it easier to add client-support, if there's an existing SASL library for language X. So +1 for implementing SASL. It's simple to implement, and AFAICS isn't any more difficult than rolling our own. We could make the server turn on a dime perhaps, but the client-side population will not come along nearly that quickly, nor with small effort. Stored passwords won't migrate to a new scheme transparently either. Yep. I think it's probably reasonable to think about a more modern password auth method, but not to imagine that it will be pluggable or that the adoption time for any revision will be less than years long. It makes sense to make it pluggable if it's simple, but in reality most drivers will support only a few most common mechanisms. That's OK. We need to design the protocol so that multiple mechanisms can be used side-by-side for years. - Heikki -- Sent via pgsql-hackers mailing list (pgsql-hackers@postgresql.org) To make changes to your subscription: http://www.postgresql.org/mailpref/pgsql-hackers
Re: [HACKERS] GSoC 2015 - mentors, students and admins.
Hi Thom! I'm interested to participate as student. On 9 Feb 2015 23:54, "Thom Brown" wrote: > Hi all, > > Google Summer of Code 2015 is approaching. I'm intending on registering > PostgreSQL again this year. > > Before I do that, I'd like to have an idea of how many people are > interested in being either a student or a mentor. > > I've volunteered to be admin again, but if anyone else has a strong > interest of seeing the projects through this year, please let yourself be > known. > > Who would be willing to mentor projects this year? > > Project ideas are welcome, and I've copied many from last year's > submissions into this year's wiki page. Please feel free to add more (or > delete any that stand no chance or are redundant): > https://wiki.postgresql.org/wiki/GSoC_2015 > > Students can find more information about GSoC here: > http://www.postgresql.org/developer/summerofcode > > Thanks > > Thom >
