Re: Trying to improve the stability of the OpenBSD build nodes
On Tue, Jun 19, 2012 at 1:43 AM, Amos Jeffries squ...@treenet.co.nz wrote: On 19/06/2012 3:03 a.m., Kinkie wrote: Hi all, I'm currently trying to improve the stability of the OpenBSD build nodes; I'm trying to figure out whether the issue can be worked around by changing the way those nodes connect to the master. the 4.8 node has successfully failed a build (that is: it has failed a trunk build for legitimate reasons). I'll run some more experiments, but I hope that the build failures on those nodes will start making sense again, so I invite who's interested to start tracking them again. OpenBSD is not unsupported past 5.0 nowdays. I've dropped 4.8 from the stable branch matrix, left 4.9 there. But was wondering if we should continue to test on 4.8/4.9 at all on 3.2+ Well, the build issues there are squid-caused from what I can see, caused by the unavailability of Shm and failures to properly link in the wake of its absence - it may be the only platform we have which doesn't support shm and this may be its value. Apart from that, I'll just set up a new OpenBSD node and drop the older ones, no problem. -- /kinkie
squid md5 and fips mode
Hi, I have been looking at FIPS issues with squid in both RHEL5 and RHEL6. In fips mode, MD5 is not allowed for securing web traffic (with some TLS exceptions) nor user authentication. It is allowed for other things, such as hashes for disk objects. The problem in older versions of squid was that the cache object code used md5, and since openssl did not allow it, it would die. This was fixed with a patch by using the squid buildin md5 functions for the cache object hashes. See https://bugzilla.redhat.com/show_bug.cgi?id=705097 Now for recent versions of squid, I have the reverse problem. The openssl md5 code is never used, so I had to patch it back to using openssl with the exception for the cache object id where I used private_MD5_Init() It basically undoes most of these commits: - Changing 'xMD5' function name to 'SquidMD5' - Changing 'MD5_CTX' typedef to 'SquidMD5_CTX' - Changing 'MD5_DIGEST_CHARS' define to 'SQUID_MD5_DIGEST_LENGTH' - Changing 'MD5_DIGEST_LENGTH' define to 'SQUID_MD5_DIGEST_LENGTH' - Removing messy #ifdef logic in include/md5.h that tries to use the system libraries if available. We'll always use the Squid MD5 routines. My request is to basically undo this change, and to use openssl again where possible, so that fips enforcement does not fail with the custom crypto code that goes undetected. A rought patch (that does not take into account systems with no crypt()) is attached at: https://bugzilla.redhat.com/show_bug.cgi?id=833086 Thanks, Paul
Re: squid md5 and fips mode
On Tue, Jun 19, 2012 at 9:10 AM, Paul Wouters pwout...@redhat.com wrote: Hi, I have been looking at FIPS issues with squid in both RHEL5 and RHEL6. In fips mode, MD5 is not allowed for securing web traffic (with some TLS exceptions) nor user authentication. It is allowed for other things, such as hashes for disk objects. The problem in older versions of squid was that the cache object code used md5, and since openssl did not allow it, it would die. This was fixed with a patch by using the squid buildin md5 functions for the cache object hashes. See https://bugzilla.redhat.com/show_bug.cgi?id=705097 Now for recent versions of squid, I have the reverse problem. The openssl md5 code is never used, so I had to patch it back to using openssl with the exception for the cache object id where I used private_MD5_Init() It basically undoes most of these commits: - Changing 'xMD5' function name to 'SquidMD5' - Changing 'MD5_CTX' typedef to 'SquidMD5_CTX' - Changing 'MD5_DIGEST_CHARS' define to 'SQUID_MD5_DIGEST_LENGTH' - Changing 'MD5_DIGEST_LENGTH' define to 'SQUID_MD5_DIGEST_LENGTH' - Removing messy #ifdef logic in include/md5.h that tries to use the system libraries if available. We'll always use the Squid MD5 routines. My request is to basically undo this change, and to use openssl again where possible, so that fips enforcement does not fail with the custom crypto code that goes undetected. A rought patch (that does not take into account systems with no crypt()) is attached at: https://bugzilla.redhat.com/show_bug.cgi?id=833086 A few quick thoughts: - We'd love patches that make squid explicitly aware of e.g. FIPS mode, so that we can enforce it ourselves. We've no idea today when we change something whether its going to impact on such external needs, and frankly, tracking it is going to be tedious and error prone, leading to the sort of flip-flop situation you have. - as we don't have an OpenSSL exception in our copyright (and its -hard- to add one today), you can't legally ship a squid binary linked against openSSL anyway. http://gplv3.fsf.org/wiki/index.php/Compatible_licenses#GPLv2-incompatible_licenses. Note that our COPYING file says 'version 2' not 'version 2 or later', though at least some source files say 'version 2 or later' - we'll need to figure out what to do about *that* in due course. So, basically every build of squid that uses openSSL has to have been built by the end user locally, anyway. Yes, this kindof sucks. That said, if the FIPS standard doesn't like MD5, there is no need for use to use it at all, we could use sha1 as a build time option for cache keys (or take the first N bits of sha1), if that helps: that would allow us to be entirely MD5 free when desired. -Rob
Re: Trying to improve the stability of the OpenBSD build nodes
OpenBSD is not unsupported past 5.0 nowdays. I've dropped 4.8 from the stable branch matrix, left 4.9 there. But was wondering if we should continue to test on 4.8/4.9 at all on 3.2+ Well, the build issues there are squid-caused from what I can see, caused by the unavailability of Shm and failures to properly link in the wake of its absence - it may be the only platform we have which doesn't support shm and this may be its value. Apart from that, I'll just set up a new OpenBSD node and drop the older ones, no problem. Hm.. OpenBSD 5.1 on amd64 is having kernel crash issues on kvm. I'll try x86. -- /kinkie
Jenkins build is back to normal : 3.2-matrix » obsd-49-x86 #240
See http://build.squid-cache.org/job/3.2-matrix/./label=obsd-49-x86/240/changes
Re: Trying to improve the stability of the OpenBSD build nodes
On Tue, Jun 19, 2012 at 1:55 PM, Kinkie gkin...@gmail.com wrote: OpenBSD is not unsupported past 5.0 nowdays. I've dropped 4.8 from the stable branch matrix, left 4.9 there. But was wondering if we should continue to test on 4.8/4.9 at all on 3.2+ Well, the build issues there are squid-caused from what I can see, caused by the unavailability of Shm and failures to properly link in the wake of its absence - it may be the only platform we have which doesn't support shm and this may be its value. Apart from that, I'll just set up a new OpenBSD node and drop the older ones, no problem. Hm.. OpenBSD 5.1 on amd64 is having kernel crash issues on kvm. I'll try x86. The x86 OpenBSD 5.1 node seems stable; if nobody objects I'll gradually dismiss the other nodes. -- /kinkie
Re: squid md5 and fips mode
On 06/19/2012 05:05 AM, Robert Collins wrote:
A few quick thoughts:
- We'd love patches that make squid explicitly aware of e.g. FIPS
mode, so that we can enforce it ourselves. We've no idea today when we
change something whether its going to impact on such external needs,
and frankly, tracking it is going to be tedious and error prone,
leading to the sort of flip-flop situation you have.
I've attached a first patch. Note that I have not been able to test the
helpers yet.
Since the md5 functions are called in void functions that cannot return
errors up the chain, I had to go up the call chain a little bit. I did
put in an assert() in case I missed a callchain somewhere.
I've added some comments in the patch on how you can test this without
actually running fully in fips mode.
Let me know if this is something along the lines of what you were
thinking of.
That said, if the FIPS standard doesn't like MD5, there is no need for
use to use it at all, we could use sha1 as a build time option for
cache keys (or take the first N bits of sha1), if that helps: that
would allow us to be entirely MD5 free when desired.
I thought about doing that, but it would also require some tricks when
people update squid with a cache based on md5. For now I added another
entrance function into the MD5 code that you call when you know there is
no fips issue. This is what the store_key/MemObject code now calls.
I'm interested in feedback and would love something like this to get
merged into upstream
Paul
diff -aur squid-3.2.0.16-orig/helpers/basic_auth/NCSA/basic_ncsa_auth.cc squid-3.2.0.16/helpers/basic_auth/NCSA/basic_ncsa_auth.cc
--- squid-3.2.0.16-orig/helpers/basic_auth/NCSA/basic_ncsa_auth.cc 2012-03-06 21:42:55.0 -0500
+++ squid-3.2.0.16/helpers/basic_auth/NCSA/basic_ncsa_auth.cc 2012-06-19 18:49:12.889646542 -0400
@@ -42,6 +42,8 @@
#include errno.h
#endif
+extern int GetFIPSEnabled(void);
+
static hash_table *hash = NULL;
static HASHFREE my_free;
@@ -52,6 +54,41 @@
char *passwd;
} user_data;
+/*
+ * Return 1 if we are in fips mode and we should disallow MD5 for any
+ * network and authentication use
+ * Quick and dirty way to test functionality in fips mode:
+ * echo 1 /tmp/fips_enabled
+ * sudo mount --bind /tmp/fips_enabled /proc/sys/crypto/fips_enabled
+ * and umount to disable fips again
+ */
+static int
+my_GetFIPSEnabled(void)
+{
+#ifdef LINUX
+FILE *f;
+char d;
+size_t size;
+
+f = fopen(/proc/sys/crypto/fips_enabled, r);
+if (!f)
+return 0;
+
+size = fread(d, 1, 1, f);
+fclose(f);
+if (size != 1)
+return 0;
+if (d != '1')
+return 0;
+
+return 1;
+#else
+return 0; /* we only know about Linux fips mode */
+#endif
+}
+
+
+
static void
my_free(void *p)
{
@@ -139,21 +176,29 @@
u = (user_data *) hash_lookup(hash, user);
if (u == NULL) {
SEND_ERR(No such user);
+ }
#if HAVE_CRYPT
-} else if (strlen(passwd) = 8 strcmp(u-passwd, (char *) crypt(passwd, u-passwd)) == 0) {
-// Bug 3107: crypt() DES functionality silently truncates long passwords.
-SEND_OK();
-} else if (strlen(passwd) 8 strcmp(u-passwd, (char *) crypt(passwd, u-passwd)) == 0) {
-// Bug 3107: crypt() DES functionality silently truncates long passwords.
-SEND_ERR(Password too long. Only 8 characters accepted.);
+ /* using the real crypt() call with return NULL for non-allowed algos passed via the salt */
+else {
+ char *crypted = crypt(passwd, u-passwd); /* avoid strcmp() on NULL */
+ if (crypted strlen(passwd) = 8 strcmp(u-passwd, crypted) == 0) {
+ // Bug 3107: crypt() DES functionality silently truncates long passwords.
+ SEND_OK();
+ } else if (strlen(passwd) 8 strcmp(u-passwd, crypted) == 0) {
+ // Bug 3107: crypt() DES functionality silently truncates long passwords.
+ SEND_ERR(Password too long. Only 8 characters accepted.);
+ }
+#endif
+ else if (!my_GetFIPSEnabled() strcmp(u-passwd, (char *) crypt_md5(passwd, u-passwd)) == 0) {
+ SEND_OK();
+ } else if (!my_GetFIPSEnabled() strcmp(u-passwd, (char *) md5sum(passwd)) == 0) {
+SEND_OK();
+} else {
+ SEND_ERR(Wrong password);
+ }
+#if HAVE_CRYPT
+ }
#endif
-} else if (strcmp(u-passwd, (char *) crypt_md5(passwd, u-passwd)) == 0) {
-SEND_OK();
-} else if (strcmp(u-passwd, (char *) md5sum(passwd)) == 0) {
-SEND_OK();
-} else {
-SEND_ERR(Wrong password);
-}
}
if (hash != NULL) {
hashFreeItems(hash, my_free);
diff -aur squid-3.2.0.16-orig/helpers/basic_auth/NCSA/crypt_md5.h squid-3.2.0.16/helpers/basic_auth/NCSA/crypt_md5.h
--- squid-3.2.0.16-orig/helpers/basic_auth/NCSA/crypt_md5.h 2012-03-06 21:42:55.0 -0500
+++ squid-3.2.0.16/helpers/basic_auth/NCSA/crypt_md5.h 2012-06-19 18:47:05.979122455 -0400
@@ -20,4 +20,6 @@
/* MD5 hash without salt */
Re: Trying to improve the stability of the OpenBSD build nodes
On 19.06.2012 23:55, Kinkie wrote: OpenBSD is not unsupported past 5.0 nowdays. I've dropped 4.8 from the stable branch matrix, left 4.9 there. But was wondering if we should continue to test on 4.8/4.9 at all on 3.2+ Well, the build issues there are squid-caused from what I can see, caused by the unavailability of Shm and failures to properly link in the wake of its absence - it may be the only platform we have which doesn't support shm and this may be its value. Apart from that, I'll just set up a new OpenBSD node and drop the older ones, no problem. Hm.. OpenBSD 5.1 on amd64 is having kernel crash issues on kvm. I'll try x86. :-(. I looked at the normal build issues on 4.8/4.9 again. It's that dependency loop between libipc.la and libmgr.la. see previous thread about them, which I followed up with a cross-module dependency analysis. We need to erase all up dependencies which libipc.la has on libmgr.la, by moving them to inline code, or the definitions into libipc. And swap the order of the libraries to libmgr.la libipc.la in Makefiles. Previous proposal which still needs implementing properly was to create a Subscription API on libipc which libmgr subscribed a callback to. Amos
Re: Trying to improve the stability of the OpenBSD build nodes
On 20.06.2012 05:16, Kinkie wrote: On Tue, Jun 19, 2012 at 1:55 PM, Kinkie gkin...@gmail.com wrote: OpenBSD is not unsupported past 5.0 nowdays. I've dropped 4.8 from the stable branch matrix, left 4.9 there. But was wondering if we should continue to test on 4.8/4.9 at all on 3.2+ Well, the build issues there are squid-caused from what I can see, caused by the unavailability of Shm and failures to properly link in the wake of its absence - it may be the only platform we have which doesn't support shm and this may be its value. Apart from that, I'll just set up a new OpenBSD node and drop the older ones, no problem. Hm.. OpenBSD 5.1 on amd64 is having kernel crash issues on kvm. I'll try x86. The x86 OpenBSD 5.1 node seems stable; if nobody objects I'll gradually dismiss the other nodes. 4.8 can go IMO. I'm kind of in favour of keeping 4.9 around just to maintain 3.1 support. Amos
Squid-3.2 Stabilizing on FreeBSD 9.0
The current build failures on FreeBSD 9.0 are due to a broken unit-test in testRock binary file. 1) The current unit-test is not linked to tools.cc for a fatalf() function call made by an exception handler. ** I'm starting on a patch to correct that right now. 2) it only needs the fatalf() function to report that SHM is failing to open file ./testRock__testRockSearch.shm due to lack of privileges. Alex / Kinkie : can you two please work out whether FreeBSD slave node needs any special administration changes or whether the SHM needs to be re-written to support FreeBSD 9 better? Thank you Amos
Jenkins build is back to normal : 3.1-matrix » obsd-49-x86 #142
See http://build.squid-cache.org/job/3.1-matrix/./label=obsd-49-x86/142/
