Re: DO NOT REPLY [Bug 41364] chunk-size contains space
On Friday 01 August 2008 19:51:48 William A. Rowe, Jr. wrote: [EMAIL PROTECTED] wrote: https://issues.apache.org/bugzilla/show_bug.cgi?id=41364 Roy T. Fielding [EMAIL PROTECTED] changed: What|Removed |Added - --- Status|RESOLVED|CLOSED I really wonder if RESOLVED vs CLOSED makes any sense to an open source community. While this is a sensible division of labor between dev and QA, we don't have two distinct groups of individuals. Should we simply dump CLOSED from the the list (or dump RESOLVED) and answer the question only once when we believe it to be fixed? It's not as though the user can't confirm the fix themselves and reopen, when appropriate. Or allow status to be moved directly to CLOSED without passing through RESOLVED first. I've seen bugzilla scenarios in which tickets have been RESOLVED awaiting the owner to CLOSE them, and others where the relevant person has CLOSED it because that's what makes sense. I agree with the observation, but I think RESOLVED and CLOSED are sometimes useful distinctions, so mightn't it be best to let the ticket-editor choose? Cheers, Geoff -- Un terrien, c'est un singe avec des clefs de char...
overview of MPMs?
Hi all, Just wondering if anyone has a link or howto that would give me some background info on the interface with the different MPM modes/implementations? I'm not even sure where the different implementations are in the source tree, but I'm curious to take a look if someone could point me in the right direction. TIA. Cheers, Geoff
Re: overview of MPMs?
On Fri, 2008-04-11 at 13:52 +1000, Graham Dumpleton wrote: 2008/4/11 Geoff Thorpe [EMAIL PROTECTED]: Hi all, Just wondering if anyone has a link or howto that would give me some background info on the interface with the different MPM modes/implementations? I'm not even sure where the different implementations are in the source tree, but I'm curious to take a look if someone could point me in the right direction. TIA. If wanting to understand how each work, then you might start by looking at: http://www.fmc-modeling.org/category/projects/apache/amp/4_3Multitasking_server.html This was based on Apache 2.0, but should principally be the same. Thanks! Following some clues from that, I managed to find the code too. Cheers, Geoff
Re: --enable-distcache
Hi Philip, Sifting through a tonne of mail that has been spooling away on the side for the last few months. In particular, I saw this post of yours w.r.t. distcache last november, and I didn't see any reply on the list; On Thu, 2007-11-15 at 06:00 -0500, Philip M. Gollucci wrote: [snip] ./configure \ --prefix=/software/httpd/2.2.6/prefork \ --with-expat=/software/expat/2.0.0 \ --with-perl=/software/perl/5.5.8/bin/perl \ --with-apr=/software/apr/1.2.8/bin/apr-1-config \ --with-apr-util=/software/apr-util/1.2.8/bin/apu-1-config \ --with-mpm=prefork \ --with-ssl=/usr \ --enable-ssl --enable-distcache \ [snip] checking whether Distcache is required... yes (specified) checking distcache/dc_client.h usability... no checking distcache/dc_client.h presence... no checking for distcache/dc_client.h... no configure: error: distcache support failed: can't include distcache headers Sure enough, this file is not in the tarball or SVN. Has it been this way since 2003 ? http://marc.info/?l=apache-httpd-devm=106984835501531w=2 FWIW, distcache is available at www.distcache.org, the dc session cache implementation in httpd communicates with distcache tools by linking with its libs. (Hence the need to enable it in configure and for it to be pre-installed.) You may have already figured this out since posting, but FWIW. Cheers, Geoff
Re: r382799 - /httpd/httpd/trunk/modules/ssl/ssl_scache_shmcb.c
On March 3, 2006 08:11 am, Joe Orton wrote:
[snip]
Log:
* modules/ssl/ssl_scache_shmcb.c (shmcb_safe_clear): Mark with
noinline attribute for GCC 3.
[snip]
This reminded me that I had a rewrite of shmcb lurking around that needed
dusting off and updating due to changes in CVS. I've attached a new
version that is a drop-in replacement for 2.2.0 - I think it addresses
all the safe memcpy issues and would *finally* allow Ken Coar to remove
that shmcb item from STATUS. It also obviates the need for the gcc-fear
your commit aptly illustrated :-) (The new version is no longer
vulnerable to agressive memset optimisation - the only memsets should be
for binary data).
I've given this version a quick workover on x86_64 using sslswamp, but
obviously it would be good to submit it to some heavier testing,
particularly on the most alignment-vulnerable archs (eg. sparc). Unless I
suddenly become the victim of a drive-by hardware donation, it won't be
me doing this. This version is also easier on the eyes (smaller too) BTW.
Cheers,
Geoff
--
Geoff Thorpe
[EMAIL PROTECTED]
http://www.geoffthorpe.net/
Self-interest and materialistic desire are parts of who we are, but
not all. To base a social and economic system on these traits is
dangerously fundamentalist.
-- Joel Bakan
/* Copyright 2001-2004 The Apache Software Foundation
*
* Licensed under the Apache License, Version 2.0 (the License);
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an AS IS BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/
/* _ _
* _ __ ___ ___ __| |___ ___| | mod_ssl
* | '_ ` _ \ / _ \ / _` | / __/ __| | Apache Interface to OpenSSL
* | | | | | | (_) | (_| | \__ \__ \ |
* |_| |_| |_|\___/ \__,_|___|___/___/_|
* |_|
* ssl_scache_shmcb.c
* Session Cache via Shared Memory (Cyclic Buffer Variant)
*/
#include ssl_private.h
/*
* This shared memory based SSL session cache implementation was
* originally written by Geoff Thorpe geoff geoffthorpe.net for C2Net
* Europe as a contribution to Ralf Engelschall's mod_ssl project.
*
* Since rewritten by GT to not use alignment-fudging memcpys and reduce
* complexity.
*/
/*
* Header structure - the start of the shared-mem segment
*/
typedef struct {
/* Stats for cache operations */
unsigned long stat_stores;
unsigned long stat_expiries;
unsigned long stat_scrolled;
unsigned long stat_retrieves_hit;
unsigned long stat_retrieves_miss;
unsigned long stat_removes_hit;
unsigned long stat_removes_miss;
/* Number of subcaches */
unsigned int subcache_num;
/* How many indexes each subcache's queue has */
unsigned int index_num;
/* How large each subcache is, including the queue and data */
unsigned int subcache_size;
/* How far into each subcache the data area is (optimisation) */
unsigned int subcache_data_offset;
/* How large the data area in each subcache is (optimisation) */
unsigned int subcache_data_size;
} SHMCBHeader;
/*
* Subcache structure - the start of each subcache, followed by indexes then data
*/
typedef struct {
/* The start position and length of the cyclic buffer of indexes */
unsigned int idx_pos, idx_used;
/* Same for the data area */
unsigned int data_pos, data_used;
} SHMCBSubcache;
/*
* Index structure - each subcache has an array of these
*/
typedef struct {
/* absolute time this entry expires */
time_t expires;
/* location within the subcache's data area */
unsigned int data_pos;
/* size (most logic ignores this, we keep it only to minimise memcpy) */
unsigned int data_used;
/* Optimisation to prevent ASN decoding unless a match is likely */
unsigned char s_id2;
/* Used to mark explicitly-removed sessions */
unsigned char removed;
} SHMCBIndex;
/* This macro takes a pointer to the header and a zero-based index and returns
* a pointer to the corresponding subcache. */
#define SHMCB_SUBCACHE(pHeader, num) \
(SHMCBSubcache *)(((unsigned char *)(pHeader)) + \
sizeof(SHMCBHeader) + \
(num) * ((pHeader)-subcache_size))
/* This macro takes a pointer to the header and a session id and returns a
* pointer to the corresponding subcache. */
#define SHMCB_MASK(pHeader, id) \
SHMCB_SUBCACHE((pHeader), *(id) ((pHeader)-subcache_num - 1))
/* This macro takes the same params as the last, generating two outputs for use
* in ap_log_error(...). */
#define SHMCB_MASK_DBG(pHeader, id) \
*(id), (*(id) ((pHeader)-subcache_num - 1))
/* This macro takes a pointer
Re: [STATUS] (httpd-2.1) Wed Apr 13 23:46:37 2005
On April 15, 2005 11:28 am, Joe Orton wrote: I've attached the copy of the replacement ssl_scache_shmcb.c which you sent out for review. I have not had a chance to review or test though, sorry. Thanks Joe - heh, I'd forgotten how complex this code was ;-) I'll attempt to take another look through this and retest in the near future. If anyone else tries it out, I'd appreciate any feedback. FWIW, this aligns all data structures in the shared-mem segment to a byte-alignment corresponding to sizeof(unsigned long) - are there any architectures on which that could be unsufficient? Cheers, Geoff -- Geoff Thorpe [EMAIL PROTECTED] http://www.geoffthorpe.net/ Greedy Genghis George, Guru of God and Guns.
Re: [STATUS] (httpd-2.1) Wed Apr 13 23:46:37 2005
On April 13, 2005 11:46 pm, Rodent of Unusual Size wrote: TODO ISSUES REMAINING IN MOD_SSL: [snip] * the shmcb code should just align its memory segment rather than jumping through all the safe memcpy and memset hoops FWIW, I wrote a mockup for this and left it with a couple of devs to look at (from memory, Joe and Madhu, and perhaps one or two others?). Joe, do you still have that patch? Did you have any positive/negative experiences with it? I've probably got a copy buried somewhere, but I'm not sure offhand and am too busy to go sifting. If you have it not too far away, I'd love a copy :-) Cheers, Geoff -- Geoff Thorpe [EMAIL PROTECTED] http://www.geoffthorpe.net/ Greedy Genghis George, Guru of God and Guns.
Re: Apache with Security Processor - Interesting
On October 21, 2004 01:05 pm, Madhusudan Mathihalli wrote: On Wed, 20 Oct 2004 20:10:53 -0400, Geoff Thorpe [EMAIL PROTECTED] wrote: Did the control-command support ever make it into a cvs-worthy form? Nope - I don't believe it is in the CVS. Can you re-send the patch ? Um, all I have is an old version for mod_ssl (apache 1.3) on my website (in the crypto page) - it adds the SSLCryptoDeviceCtrl directive. IIRC, one of the Bill's on this list ported this to 2.* form, you may be able to find it in the archives I guess (I'm at work and so not able to trawl for this right now). Note, I wasn't terribly pleased with how I implemented it, so I don't pretend to suggest this is the Right Way(tm). That's a far more general way to squash this issue. Well.. I'm not sure of that - it'll probably be more clear after seeing your patch. AFAICT, we have so some how force Apache to load the dynamic engine - and that's pretty much what my patch does. Yes, and in your implementation you're doing this by issuing various control-commands exposed by the dynamic engine to achieve it. This magic may be a suitable transparent behaviour to use when the specified engine can't be initialised directly, but it's still limited to whatever logic you hard-code into the existing SSLCryptoDevice directive. Any other commands supported by the dynamic engine (let alone any supported by the targetted engine once it's loaded) would be inaccessible unless someone goes in and adds further C logic. Eg. if an engine requires configuration/authorisation, it can expose those details as control-commands and as long as there's a mechanism to pass arbitrary commands from the configuration file through, then Apache wouldn't need to be any the wiser. Eg. the existing autoloading magic, plus additional device-specific setup/control, could be achieved with; SSLCryptoDevice dynamic SSLCryptoDeviceCtrl SO_PATH:/lib/.../foobar.so SSLCryptoDeviceCtrl LOAD [... the dynamic engine is now replaced by the loaded engine...] SSLCryptoDeviceCtrl FOOBAR_LOGFILE:[...] SSLCryptoDeviceCtrl FOOBAR_AUTH:[...] ... [... appending 'post' executes commands after ENGINE_init() ...] SSLCryptoDeviceCtrl FOOBAR_LOCKpost ... Without something like this, the functionality is limited to whatever is explicitly coded into apache, which I think is a little limiting. Also useful will be some alterations that allow HSM keys to be used - I can help with that if you like, but it'll definitely need control-command support first. NB: Another way forward is to try and hook into Steve Henson's revisions to the configuration code. This essentially offloads all of this sort of stuff to an openssl-specific configuration file that is executed at the appropriate moment(s). We'd need to get Steve involved in that discussion though if you're interested (there are potential pros and cons to this). Cheers, Geoff -- Geoff Thorpe [EMAIL PROTECTED] http://www.geoffthorpe.net/
Re: Apache with Security Processor - Interesting
On October 20, 2004 02:44 am, Madhusudan Mathihalli wrote: If ppl think it'll be a good addition to Apache, I can clean it up and try to commit it sometime tomorrow. Did the control-command support ever make it into a cvs-worthy form? That's a far more general way to squash this issue. I could understand that you might want the single-line configuration semantics to be a bit smarter by default too, but I think that if you're going to this trouble, it would make sense to also expose engine control-commands to the configuration to let users do this sort of cleverness directly (and in any other ways they come up with without needing the code changed). Let me know if this isn't already in CVS because I had a diff floating around somewhere that did this. Cheers, Geoff -- Geoff Thorpe [EMAIL PROTECTED] http://www.geoffthorpe.net/
Re: OCSP addition
On May 11, 2004 02:23 am, Hotmail wrote: I plan to add OCSP support to mod_ssl. Cool, this will probably make quite a few people happy(ier). [snip] 3. In ssl_callback_SSLVerify_Validity( ): - if the parameter UseOCSP is on, try an OCSP check - if the OCSP check failed because the certificate is revoked = return error - if the OCSP check succeeded = return ok (ok is an input parameter, don't know what it is exactly) - call ssl_callback_SSLVerify_CRL( ) and return result Do you see any problem with that ? The only thing I can see is the blocking-vs-non-blocking semantics of the CRL lookup. For traditional apache, this isn't a problem as you have an entire child process dedicated to serving the current request, and so you can block all you need to. However the desire to have the modules plug-and-go in different MPM scenarios could run up against problems if the CRL lookup is latent (which is quite likely, as the raison d'etre for PKI and OSCP is to permit authentication *at a distance*). I assume this wouldn't be a problem with kernel-threading, but it would definitely cause speed-bumps for green/user-threading. Then again, maybe this isn't a problem. In the worst case, it could be left as a known-limitation - presumably anyone wanting to use OSCP on client-authentication already has a fairly clear idea of the configuration and architecture they are after and so can live with any additional rules you impose. Is somebody interesting in testing that code, or even work on it ? I really can't help here, but I wish you the best with it. Cheers, Geoff -- Geoff Thorpe [EMAIL PROTECTED] http://www.geoffthorpe.net/
Re: [PATCH] Fix SEGV in ssl_scache_shmcb.c
Hi all,
On May 10, 2004 05:04 pm, Mathihalli, Madhusudan wrote:
mod_ssl dumps core when you specify a low cache size (Ex. 1)
OR in a manner similar to Bug 27751. In both the cases, the problem
arises because of a incorrect/incomplete assumption about the size of
the session data in the cache. The session when stored in the cache can
be a maximum of SSL_SESSION_MAX_DER bytes - however, it's NOT safe to
copy SSL_SESSION_MAX_DER bytes back from the cache when we're trying to
retrieve the session id.
The following patch fixes the assumption by including a new 'size'
variable in the cache to store the correct size of the session data -
so that it can be used for retrieval.
Any comments ?
Just one :-) I hadn't been particularly clear about something so wires may
have got crossed, there is a second patch lurking around and it's purpose
is overlapped with the one you posted. The patch you sent reduces the
memcpy() overhead to the minimum required whereas previously it was
pegged at the maximum possible. The cost for that is the addition of
another member variable in the index structure. However the use of
maximal memcpy over minimum memcpy was not the bug, just an
inelegance of the code. The real bug was that no check was being made
that the size of the desired memcpy was less than the size of the
(sub-)cache, no matter whether it was maximal or minimal! :-) I think the
bug would have been triggered by maximal and minimal scenarios, provided
you used small enough cache sizes (less than 256kb) and waited long
enough.
That second patch is attached to this mail - it is the necessary fix to
the bug. The other patch is a slight improvement in efficiency (and code
quality) and would also be useful if it's considered solid enough, but it
should be independent of the fix.
Cheers,
Geoff
--
Geoff Thorpe
[EMAIL PROTECTED]
http://www.geoffthorpe.net/
diff -urN httpd-2.0.49/modules/ssl/ssl_scache_shmcb.c httpd-2.0.49-patched/modules/ssl/ssl_scache_shmcb.c
--- httpd-2.0.49/modules/ssl/ssl_scache_shmcb.c 2004-02-09 15:53:20.0 -0500
+++ httpd-2.0.49-patched/modules/ssl/ssl_scache_shmcb.c 2004-03-25 21:41:45.0 -0500
@@ -848,6 +848,9 @@
unsigned int dest_offset,
unsigned char *src, unsigned int src_len)
{
+/* Cover the case that src_len buf_size */
+if(src_len buf_size)
+src_len = buf_size;
/* Can it be copied all in one go? */
if (dest_offset + src_len buf_size)
/* yes */
@@ -871,6 +874,9 @@
unsigned int src_offset,
unsigned int src_len)
{
+/* Cover the case that src_len buf_size */
+if(src_len buf_size)
+src_len = buf_size;
/* Can it be copied all in one go? */
if (src_offset + src_len buf_size)
/* yes */
Re: [STATUS] (httpd-2.0) Wed Mar 24 23:45:11 EST 2004
On March 24, 2004 11:45 pm, Rodent of Unusual Size wrote: APACHE 2.0 STATUS: [snip] TODO ISSUES REMAINING IN MOD_SSL: [snip] * the shmcb code should just align its memory segment rather than jumping through all the safe memcpy and memset hoops I've asked on more than one occasion and had no response. Why is this comment here? From who and where does it come? Could it please be either (a) discussed, or (b) removed? It happens to make very little sense, but I'd certainly be keen to hear if someone has any rational logic to the contrary. Cheers, Geoff -- Geoff Thorpe [EMAIL PROTECTED] http://www.geoffthorpe.net/
Re: [STATUS] (httpd-2.0) Wed Mar 24 23:45:11 EST 2004
On March 25, 2004 10:50 am, Joe Orton wrote: I've asked on more than one occasion and had no response. Why is this comment here? From who and where does it come? Could it please be either (a) discussed, or (b) removed? It happens to make very little sense, but I'd certainly be keen to hear if someone has any rational logic to the contrary. I'll try (a): why are the _safe_ memcpy wrappers needed if not simply to cope with the fact that the shmcb structures are not kept word-aligned within the shmem segment? Well that *is* exactly why the wrappers exist. :-) The question is more how to modify the design to obviate this issue without bloat and waste, because the segment is partitioned up into a geometry of divisions, each with meta-data and cache data compacted end to end. That said, a couple of 'char' primitives have already moved to larger types, and another couple probably should too (I have a sneaking suspicion this is part of the problem in that bug report from HP). When this is done, the space/geometry arguments may go out the window (though I'm still worried about what a 64-bit architecture and tool-chain might do if we rely on alignment). Hmm, perhaps after all I could remove the safe stuff by removing the use of the overlays. ponders. This is *not* just a question of aligning structures though, it's a question of changing the way data is managed and accessed, which is why the flippant STATUS note had been bugging me. However, that more fundamental change may be a worthwhile anyway ... sigh OK, I will report back on this once I get down into the bowels of this code again chasing Ken's bug. Does this mean I've got some volunteers to help test this if I do undertake to rewire the cache internals? :-) Cheers, Geoff -- Geoff Thorpe [EMAIL PROTECTED] http://www.geoffthorpe.net/
Re: [STATUS] (httpd-2.0) Wed Mar 24 23:45:11 EST 2004
On March 25, 2004 12:46 pm, Mathihalli, Madhusudan wrote: Now that we're discussing about shmcb, I had another question - I see the following in shmcb.c 608 /* Work on the basis that you need 10 bytes index for each session 609 * (approx 150 bytes), which is to divide temp by 160 - and then 610 * make sure we err on having too index space to burn even when 611 * the cache is full, which is a lot less stupid than having 612 * having not enough index space to utilise the whole cache!. */ 613 temp /= 120; Is the comment on line 609 wrong OR is line 613 wrong ? Neither. Line 610 is the reason, even if I wasn't able to rite my english proper like at a time the riting did. :-) Cheers, Geoff -- Geoff Thorpe [EMAIL PROTECTED] http://www.geoffthorpe.net/
Re: [PATCH] Flag OpenSSL to NOT store sessions
On March 25, 2004 06:28 pm, Mathihalli, Madhusudan wrote: Apart from flagging OpenSSL to NOT lookup the internal cache for session-id's, we should ALSO tell OpenSSL to NOT store the sessions ! This fixes my problem where the httpd process size keeps increasing when SSLVerifyClient is enabled along with SSLSessionCache. Makes sense, the server should not use any per-process caching. If your patch lets session caching work but doesn't leak, that would indicate you've got the right flags. Cheers, Geoff -- Geoff Thorpe [EMAIL PROTECTED] http://www.geoffthorpe.net/
Re: Error messages from ssl_scache_shmcb
Um ...
On March 24, 2004 05:30 pm, Mathihalli, Madhusudan wrote:
Ever noticed the following set of messages in the error_log - they can
really fill up the log file pretty quickly! I can't really get much
useful information (and I don't even know what 'internal error' means
?)
It's my way of saying a bug.
---
[Wed Mar 24 13:55:46 2004] [error] shmcb_insert_encoded_session
internal error [Wed Mar 24 13:55:46 2004] [error] can't store a
session!
[Wed Mar 24 13:55:46 2004] [error] 'shmcb' code was unable to store a
session in the cache.
---
This is not supposed to happen...
@@ -396,14 +399,20 @@
{
SSLModConfigRec *mc = myModConfig(s);
BOOL to_return = FALSE;
+int rv;
ssl_mutex_on(s);
-if (!shmcb_store_session(s, mc-tSessionCacheDataTable, id, idlen,
- pSession, timeout))
+rv = shmcb_store_session(s, mc-tSessionCacheDataTable, id, idlen,
+ pSession, timeout);
+if (rv != TRUE) {
/* in this cache engine, stores should never fail. */
ap_log_error(APLOG_MARK, APLOG_ERR, 0, s,
- 'shmcb' code was unable to store a
- session in the cache.);
+ 'shmcb' was unable to store a session in the
cache
+ (cause: %s),
+(rv == SSL_INTERNAL_ERROR) ? internal error
+ : (rv == SSL_SESSION_TOO_BIG) ? session too
big
+ : unknown error);
+}
else {
ap_log_error(APLOG_MARK, APLOG_DEBUG, 0, s,
shmcb_store successful);
Note the comment in the source. Rather than trimming this log message to
be less of an eye-sore, I think we should probably fix the bug that's
triggering it. It's been a long time since I last looked at shmcb in any
detail, and it's possible that there were still bugs waiting to turn up
eventually just as it's also possible that some of the porting to apache2
might have introduced some niggles along the way. As it happens, I'm
going to be delving in the next few days anyway because of bug 27751[1] -
perhaps this is related? If you want to get some context from the mement
this error occurs, we could try and dig in. I expect to be able to start
hacking in earnest on this some time this weekend.
Cheers,
Geoff
[1] BTW Madhu, this is someone from your neck of the woods (based on the
domain name, at least). Just in case you feel like syncing up about this.
--
Geoff Thorpe
[EMAIL PROTECTED]
http://www.geoffthorpe.net/
Re: Error messages from ssl_scache_shmcb
On March 24, 2004 10:37 pm, Mathihalli, Madhusudan wrote: This is not supposed to happen... Well. It's fairly easy to reproduce the problem - I used sslswamp (new session-id everytime) with a shmcb memory size of 64000, and 300 sec timeout. You can get a ton of those error messages in the error_log. I thought it to be fair enough to run out of cache :) .. or maybe I was wrong ! Well I am now more certain than ever that some glitches have crept in somehow, because this very same test (and numerous others) were pretty robust way back when I was banging on it. I'll look into this and let you know what I find (though don't hold out much hope until early next week at the earliest, the next few days promise to be pretty busy). Cheers, Geoff -- Geoff Thorpe [EMAIL PROTECTED] http://www.geoffthorpe.net/
Re: [PROPOSAL] Move httpd to the subversion repository
On March 16, 2004 09:52 pm, Kean Johnston wrote: Do we need to buy a license? No but if you send us money we'll donate it to the End Sarcasm Campaign. Is that a SCO project or some godless communist movement? I ask only information... Then some smart-ass thought it would be funny to throw in a jibe to me about licensing (some people have no self control). Dude lighten up, I'm waging no license jihad here. I thought my little joke pretty mild and subdued given the nature of the (albeit accidental) posting, ie. proclamations about packaging and redistribution of open source tools coming from the sco.com domain. Anyway, irony is better than flaming, surely? (I'll avoid comments about it being a free world, as the courts have yet to decide that one.) Cheers, Geoff PS: Smile, boys will be Boies. -- Geoff Thorpe [EMAIL PROTECTED] http://www.geoffthorpe.net/
Re: [OT] sco stuff (was: [PROPOSAL] Move httpd to the subversion repository)
Oh Greg you big pooper, There you go getting all balanced and objective on us ... :-) On March 17, 2004 12:42 pm, Greg Stein wrote: One thing that I'd like to point out here: the ASF cares about end users who are on the SCO platform. Yes, the httpd PMC considered making some sort of statement or removing support or whatever for SCO when this stuff first started, but we came to the right end position: end users matter, and if end users are on SCO, then we'll help them. Retaliation is indeed the wrong way forward. SCO's IP crusade is unbridled opportunism, (IMHO, IANAL, FWIW, $0.02, [etc]) and to react by trying to disadvantage their platform would stink of the same rot, and as you're hinting, this could actually drive users (who are largely caught in the middle of it) to support the cause their OS overlords are pursuing. And if Microsoft arrives here and starts to participate, then we'll accept them just as much as SCO or anybody else. Because it isn't the people at the companies that we're here for, but the end user. It's that uncanny Microsoft/SCO analogy rearing its ugly head again, go figure. At the end of the day, it's the portability of tools like apache that allows users to wander freely from platforms that lose their relevance over to those that surpass them. And isn't that, in reality, the real bug-up-the-u-no-where for SCO, Microsoft, and their ilk? Their relevance is questioned, their opportunism threatened, and their lock-in dissolved. As IP hawks have been preaching for so long; adapt or die. I finally agree. I apologise to Kean and the good open source hackers at SCO or Microsoft - if I seemed cynical or ungrateful before, I am a healed man and you now have my upmost admiration. Please keep up the excellent work maintaining a portable stick-free teflon OSS base on your platforms. This gives your users plenty of escape routes if they see better value (and even, shudder, principles?!) elsewhere, and I can hardly stoop to criticise you for such honest measures. Happy hacking. Please keep that in mind. Amen, Geoff PS: If SCO loses their case, would that be a Gargantuan Profiteering Loss? -- Geoff Thorpe [EMAIL PROTECTED] http://www.geoffthorpe.net/
Re: [PROPOSAL] Move httpd to the subversion repository
On March 16, 2004 09:10 pm, Kean Johnston wrote: You can get the latest from ftp://ftp.sco.com/pub/openserver5/opensrc Its one-stop shopping for most of the useful open source libraries. Do we need to buy a license? Cheers, Geoff -- Geoff Thorpe [EMAIL PROTECTED] http://www.geoffthorpe.net/
Re: [PATCH] SSLCryptoDevCtl support
Hi Serge, Thanks for porting this. In my defense, I knew that firsttime hack stuff would never wash with Ralf (for mod_ssl) or the ASF (for apache2), but I simply wanted to quickly wash my hands of that whole weirdness of double config processing (IMHO brokenness may be tolerated, but must never be embraced :-). Cheers, Geoff On February 23, 2004 03:02 pm, Serge Hallyn wrote: thanks for your reply. Attached is a new patch against HEAD taking your comments into consideration. Note that the static int firsttime hack can go away as soon as the repeated config reads in main() are consolidated. -- Geoff Thorpe [EMAIL PROTECTED] http://www.geoffthorpe.net/
STATUS (was [PATCH] OpenSSL dynamic engines under 2.0.48)
On February 17, 2004 04:21 pm, Sander Temme wrote: I haven't reviewed the above, but isn't a bunch of logic for using hardware crypto devices already in HEAD? This is up for backport at line 151 of the httpd-2.0 STATUS. I took a quick look and you are maybe thinking of something else. If its the ENGINE_init() versus SSL_EXPERIMENTAL_ENGINE stuff you're referring to, then that's more of an underlying autoconf-level issue. However, I did notice something else - both the httpd-2.0 and httpd-2.1 STATUS files seem to have this comment; * the shmcb code should just align its memory segment rather than jumping through all the safe memcpy and memset hoops Could this please be removed, as it makes no sense. Anyone who has read and understood the internals would know why. There are certainly things that could be done to improve the code however (and I really should find time to do that), but the current comment is likely to lead someone to doing something very painful to themselves. Cheers, Geoff -- Geoff Thorpe [EMAIL PROTECTED] http://www.geoffthorpe.net/
Re: [PATCH] OpenSSL dynamic engines under 2.0.48
Hi there, On February 17, 2004 03:26 pm, Serge Hallyn wrote: We have a need to use a dynamic OpenSSL engine under apache. The attached patch (against 2.0.48) directs apache to accept ssl.conf directives such as: SSLCryptoDevice dynamic SSLCryptoLibpath /usr/local/lib/hw_ibmca.so SSLCryptoDevID ibmca directing openssl to load hw_ibmca.so dynamically as engine id ibmca. Is there a fundamental complaint against incorporation of this feature? Not a fundamental one, but something more aesthetic, yeah :-) I wrote a patch ages ago (that hasn't been updated since) to add more generalised control-command support to Apache. That was for mod_ssl/apache-1.3 though, so it has only illustrative value for now. You can link to the patch and the explanatory email down in the news section of; http://www.geoffthorpe.net/crypto/ (yes, another page that hasn't been updated in a while). The idea of control-commands is more general than supplying a shared-library path or an ID, so I think it makes sense to add the more general mechanism if we are going to go this route at all. Eg. your above example would then be; SSLCryptoDevice dynamic SSLCryptoDeviceCtrl SO_PATH:/usr/local/lib/hw_ibmca.so SSLCryptoDeviceCtrl ID:ibmca As another example of where this can go, using HSM keys (or smart-cards, usb-keys, etc) via custom control commands could be entirely supported in this way - without needing to recompile openssl or apache. (An engine can expose control commands to identify and load private keys, and the private key file loaded by Apache could instead contain stub data interpreted by the engine implementation to refer to the HSM keys). It's a bit of an ugly hack, but it works. The point being; the only limit on what you can do from the configuration file is what the engine implementation exposes in the way of control commands - neither apache nor openssl need to be adjusted for each feature. Note also that as of openssl 0.9.8, platforms that have DSO_METHOD support implemented will be able automatically load shared-library engines without any of this. ENGINE_by_id(foo) will attempt to automatically load and bind libfoo.so (dlfcn), foo.dll (win32), [etc] from a specific sub-directory of the openssl installation if the foo ENGINE doesn't already exist. So if Apache doesn't want to go the route of generalised control commands, they could wait until 0.9.8 is out and then this basic use-case will be available transparently, without any change to what's there now. Cheers, Geoff -- Geoff Thorpe [EMAIL PROTECTED] http://www.geoffthorpe.net/
Re: mod_ssl not sending Alert upon close ?
On February 4, 2004 04:39 pm, Mathihalli, Madhusudan wrote: Hi, I was playing with ssldump for the data transferred b/w browser and Apache (2.0.48) - and realized that the Apache2 (+ mod_ssl) does not send the Alert message to the client before closing the connection. Funnily enough, I was just stewing on a similar problem with openssl's builtin s_server application - in that case, the braindamage is in s_server.c's use of SSL_CTX_set_quiet_shutdown(ctx,1). Perhaps apache2 is doing the same thing? Cheers, Geoff -- Geoff Thorpe [EMAIL PROTECTED] http://www.geoffthorpe.net/
Re: distcache support in mod_ssl for 2.1
On November 26, 2003 07:05 am, Joe Orton wrote: Hiya, I'd like to integrate distcache support into mod_ssl on HEAD: are the copyright holders willing to contribute copyright ownership to the ASF for ssl_scache_dc.c and the other changes? This is good news. IIRC, I already put notes into the mod_ssl (apache1.3) and httpd2 support saying that the ASF (and Ralf, in the 1.3 case) can totally subsume the glue code under their own license/copyright if they want to integrate. But if that wasn't already spelled out correctly, you can consider this the go-ahead. I've rediffed the published patch for 2.0.48 against HEAD, attached for review. Changes relative to the original patch are: - don't add #if 0 around #if APR_HAVE_UNISTD_H - style fixes in ssl_scache_dc.c: if( - if (, untabify cool. - don't cast the void * to DC_CTX * in ssl_scache_dc.c Ah this is a consequence of your trashing shmht? - rediff ssl_scache.c against HEAD - adjust ssl_scache_dc.c for new status API in HEAD OK - how come this was working as-is before? Has the previous technique been deprecated but still compiles/links/runs in the mean time? - move distcache checks into modules/ssl/config.m4 Biensur. :-) Cheers, Geoff -- Geoff Thorpe [EMAIL PROTECTED] http://www.geoffthorpe.net/
Re: distcache support in mod_ssl for 2.1
On November 26, 2003 11:34 am, Joe Orton wrote: Can you fill in the CLA http://www.apache.org/licenses/cla.pdf and fax/post it to the included number, to make that official? done [by fax]. Now to clarify, this covers anything you find in the support-apache2 and support-apache1.3-modssl packages on the distcache site (or respectively, the distcache-httpd-2 and distcache-mod_ssl modules in CVS). Ie. the autoconf support for apache, the ssl_scache_dc.c file, patch to modules/ssl/, and the installation README - it's all yours. - don't cast the void * to DC_CTX * in ssl_scache_dc.c Ah this is a consequence of your trashing shmht? yup Right. Which reminds me, there were some questions lingering about the read/write wrappers in shmcb, IIRC. Last time this was discussed, I was hacking on apache's openssl autoconf macros, with subsequent insertability of distcache's autoconf macros and glue code being my next priority. Seeing as you're now helping with that bit, is there something I could do to reciprocate w.r.t. shmcb now? Any pointers to past or present issues, comments, bugs, etc would be welcome - I'll try to find some time shortly to go in there and clean things up. (Yes, I promise to turn off the tabs too :-). Cheers, Geoff -- Geoff Thorpe [EMAIL PROTECTED] http://www.geoffthorpe.net/
distcache (was RE: consider reopening 1.3)
On November 17, 2003 02:22 pm, Bill Stoddard wrote: application environments. Being able to eliminate 1 machine in 3 due to scalability improvements in 2.0 probably won't be a sufficient return on investment for most folks. A really kick-ass load balancing/active fail-over feature in mod_proxy might generate some interest in 2.0 deployed in the DMZ (features like this are significantly easier to implement in threaded webservers). Which reminds me, I must find someone who would be prepared to discuss our distributed session caching stuff (ww.distcache.org) for use with apache2/mod_ssl. Any takers? There are companies out there flogging pretty lame and expensive SSL load-balancers and if you're running more than one machine for https, you're almost obliged to go down that route even if it hurts all your rational sense. If scalability and architectural flexibility are part of the argument for 2.0, would it be advantageous to get distcache support included into apache? Mandrake and Redhat already have (optional) support in their SRPMS, FWIW. The apache glue to the distcache API is a pretty small patch - an autoconf check plus an additional caching module/syntax in modules/ssl/. Use of the alternative session cache mode is via an alternative syntax to the SSLSessionCache configuration parameter. We upgraded the httpd patch to 2.0.48 recently, but on the distcache side we are in the midst of trying to move the development branch to stable before chasing down integration too hard. Last time I looked into getting our patch into the upstream code, there were one or two fairly important unresolved bugs in distcache and apache's autoconf support for openssl/ssl-c needed quite a bit of reorganisation. I helped with the latter at that time, and we sorted out the remaining issues in distcache shortly after. So ... if there's anyone on the apache side who'd be prepared to look deeper into this and discuss integration with apache, please make contact with me off the list. Cheers, Geoff -- Geoff Thorpe [EMAIL PROTECTED] http://www.geoffthorpe.net/
Re: mod_ssl broken as DSO in HEAD
Hi there, On June 16, 2003 10:14 pm, William A. Rowe, Jr. wrote: At 08:13 PM 6/16/2003, Justin Erenkrantz wrote: The change to move OpenSSL's initialization to ssl_hook_pre_config is busted when mod_ssl is a DSO. It will try to call all of the OpenSSL init sequences twice which I don't believe OpenSSL supports. We may need to register proper 'cleanups' to 'unload' all of the engines, as you point out in this problem; If things really are supposed to initialise (and unload) two times on startup, then I guess this is what you should do for openssl too. Right. What is more interesting (as a DSO) is that our module, and ergo the SSL library, should have been unloaded and reinitialized. Apparently we can't count on that behavior, though. Depending on what gets initialised, there are a variety of things you might need to unload from openssl's point of view. However, this is usually an issue for people from a memory leak point of view, not a double initialisation. Potential issue: in a few places, openssl uses a global is_set variable to handle first-time initialisation of callbacks (eg. malloc). The idea is that once they're set, they can't be overriden. If you don't need to do *that* twice on startup, there's no problem. If you do, then it shouldn't matter *provided* you can live without error checking - if it fails then the (correct) global is still set from the previous initialisation, and if it succeeds then the globals needed setting anyway. Of course if it fails on the first initialisation then you have a bug that you'll never spot ... BTW: this is only for stuff where you are providing callback overrides for memory management, thread locking, error stacks, etc. Well, if it isn't done before pre-config, the entire ENGINE support code in the command handler for SSLCryptoDevice is borked. I know, I already spent two days there. gulp SSLCryptoDevice ... I don't like the sound of that ;-) If we force the SSL Library to stay loaded (reliably, across platforms) between startup phases (with the SSL module itself unloading and reloading - ick), then saving the initialization in pprocess might be a viable alternative, but where? register_hooks is no better than pre_config in this regard. Or we simply work out an ENGINE_unload_engines() as a cleanup. But of course, OpenSSL doesn't really like to be cleared and restarted from scratch, so I don't know if this is really trivial or a nightmare. This can be done easily enough, it's just a case of knowing what bits you need to clean up. What is beyond my reach however is a way of convincing Apache to run in one process and start up and close down cleanly - this is normally the easiest way to check for missing cleanups because they show up as leaks. I could otherwise give you some tips on cleanup sequences, but without a way of checking the cleanup code for leaks we'd be acting on faith that the sequence is (a) complete, and (b) the minimum necessary (no point putting redundant calls in and risking extra static linking). Let me know if, and how, I can help. If you want to hunt around yourself, check out the apps_startup() and apps_shutdown() macro definitions inside apps/apps.h in the openssl source. You may not need all of those cleanups, and of course YMMV :-) Cheers, Geoff -- Geoff Thorpe [EMAIL PROTECTED] http://www.geoffthorpe.net/
Re: OpenSSL locking was Re: mod_ssl broken as DSO in HEAD
Hi,
On June 18, 2003 02:28 am, Justin Erenkrantz wrote:
Finally got a chance to dig some more and found that if I add:
CRYPTO_set_id_callback(openssl_id);
to ssl_hook_pre_config before the ENGINE_load_builtin_engines and where
openssl_id is defined as:
static unsigned long openssl_id(void)
{
/* FIXME: This is lame and not portable. -aaron */
return (unsigned long) apr_os_thread_current();
}
(The above is used in flood - hence Aaron's comment.)
The SEGVs go away if we do this. Does this make sense given the
internals of OpenSSL?
I'm not the best person to talk about with respect to locking in openssl
(I more or less detest threads, an Alan Cox quote about state-machines
springs to mind...). That said, :-), I took another look at the backtrace
you posted and it certainly seems odd. Digging ... the implementation of
CRYPTO_thread_id() is;
unsigned long CRYPTO_thread_id(void)
{
unsigned long ret=0;
if (id_callback == NULL)
{
#ifdef OPENSSL_SYS_WIN16
ret=(unsigned long)GetCurrentTask();
#elif defined(OPENSSL_SYS_WIN32)
ret=(unsigned long)GetCurrentThreadId();
#elif defined(GETPID_IS_MEANINGLESS)
ret=1L;
#else
ret=(unsigned long)getpid();
#endif
}
else
ret=id_callback();
return(ret);
}
The segfault is (or appears to be, stack smashing possibilities aside)
occuring in a function called from this, and the address is less than the
other openssl library functions in the trace, but not by much. Which begs
the question - is id_callback set to something (and if so, what?) or does
the address correspond to getpid()? Unfortunately, I think the easiest
way to see this would be to hack the above function (and perhaps others
in crypto/cryptlib.c that deal with id_callback) to watch what is going
on.
On this same tangent, do we need to be doing all of the CRYPTO_lock
stuff? I don't believe we are doing that. And, I know in flood, we had
lots of problems until we called them. So, I think mod_ssl should be
passing the lock structures - especially for worker MPM builds...
Whenever I build Apache, I typically build a version of openssl configured
with no-threads so this sort of issue is implicitly ruled out. It also
removes a few meaningless function calls given that (multi-forked,
traditional) Apache doesn't need any thread-safe locking. However I know
that isn't the answer to your question in general, as many people will be
using the openssl libs bundled with their system and/or trying to use
Apache2 in a threaded setup.
While I've got someone's attention, I have some real issues in that
both of the OpenSSL lock implementations (CRYPTO_set_locking_callback
and CRYPTO_set_dynlock_*) require global variables to implement. I
don't think I can say 'ick' loud enough.
What would the possibility/feasibility of passing application-specific
opaque values through the CRYPTO_lock callbacks? Say
static void openssl_lock(int mode, int n, void *app, const char *file,
int line)
static CRYPTO_dynlock_value *ssl_dyn_create(void *app, const char*
file, int line)
We can have app passed into the CRYPTO_set functions - in the case of
httpd, it would be the pools for ssl_dyn_create *or* the global array
of locks that openssl_lock would require. You don't know how much
happier that would make me if we could do that. globals are just so
icky. -- justin
For this and the previous question, the short answer is maybe, but it
doesn't solve the problem of working with existing versions.
In both cases, ie. the segfault you're seeing *and* the issue about
attaching an opaque pointer to caller-provide locking callbacks, it would
make sense to take that conversation over to openssl-dev. In particular,
we should get any appropriate details into the request-tracking system.
BTW: Once we're over there, Richard Levitte may be able to provide better
comments than I on this locking stuff, particularly as the dynlock stuff
was (IIRC) his creation.
Cheers,
Geoff
--
Geoff Thorpe
[EMAIL PROTECTED]
http://www.geoffthorpe.net/
Re: mod_ssl broken as DSO in HEAD
Hi, On June 17, 2003 04:06 pm, Justin Erenkrantz wrote: --On Tuesday, June 17, 2003 12:46 PM -0500 William A. Rowe, Jr. [EMAIL PROTECTED] wrote: If folks would be so kind as to cvs up their httpd-2.1 trees and check out the latest cleanup reordering and introduction of ENGINE_ and CONF_ cleanups - I would be most grateful. Geoff, if you could review the latest commit for sanity, I'd appreciate that, too. Nope, still SEGVs in the same location even though the cleanups are invoked. -- justin Well first off, the original backtrace showed a segfault down inside the bowels of the error handlers, something to do with CRYPTO_thread_id() IIRC. That you were getting this, combined with the fact you couldn't get a segfault under a different debugger and/or other variations of running it (again, IIRC), would suggest that at least part of what you're seeing is environmental and/or platform specific. You would still have been getting library errors of course, but it doesn't explain away the segfaults. But with respect to the missing cleanups; it's quite possible that adding those cleanup calls won't guarantee complete unloading if there are references left lingering by the application code. If there are objects allocated (or references held) by Apache code, these should all be released prior to calling the cleanup functions otherwise things will be left lingering. The cleanup functions should only release the references held by the library's internal state - the application must do it's part too. This includes any of the following objects mod_ssl may have lying around; RSA, DSA, EVP_PKEY, SSL, SSL_CTX, ENGINE, X509, ... (you get the picture). Note that these things can have references between them, eg. an SSL object may hold a reference to an RSA key which in turn may hold a reference to an ENGINE, etc. This is why the easiest way usually to track down this sort of thing is to run the cleanup, exit(0), and then check for memory leaks using valgrind or whatever. Other than those vague thoughts, I can't suggest much else. If this continues to resist your efforts, please let me know how I can reproduce and debug this (in linux) from the apache point of view. I'm a bit shallow in my knowledge of apache's internals and how to track its loading, unloading, memory handling, etc. Cheers, Geoff -- Geoff Thorpe [EMAIL PROTECTED] http://www.geoffthorpe.net/
Update on distcache support
Hi all, I've attached a patch for the current head of CVS that represents the latest state of distcache support. This includes the autoconf hooks so that, if mod_ssl is configured to be built, distcache support will be probed for and enabled if found. If --enable-distcache is specified then failure to autodetect distcache support will be considered an error, and if --disable-distcache is specified then the distcache probes and configuration steps are bypassed. If anyone feels motivated to help review this, please grab the appropriate bundle of distcache version 0.4pre2 from www.distcache.org. There are (experimental) precompiled RPMs of distcache available for x86, as well as source code of course. The support attached to this email supercedes the apache-2 support packages at www.distcache.org so please don't try to combine them - only distcache itself is required. Once you have distcache's tools, libraries and headers installed, you can hook things up with apache as follows; distcache - - start a cache server, eg; # dc_server -daemon -listen IP:9001 - start a local proxy for the web-server to use, eg; # dc_client -daemon -listen UNIX:/tmp/dc_client \ -server IP:localhost:9001 apache -- - configure, compile, and install apache using the attached patch *after* you've installed the distcache headers and libraries. - change the SSLSessionCache directive to use distcache, eg; SSLSessionCache dc:UNIX:/tmp/dc_client That should be all that's required beyond the normal apache/mod_ssl setup. I've tested this as best I can on the systems I have access to and have observed no problems, ie. no known bugs - I'm keen to get results from any out of the ordinary environments. If you set up additional servers to test shared caching, give each its own instance of dc_client with the -server switch pointing to the machine (and port) that dc_server is listening on. It should work with hostname resolution as well as dotted-numeric IP addresses (eg. IP:192.168.0.1:9001, IP:cache-server.localnet:9001, etc). If everything is running on the same host, you can elect to stick with unix domain addresses instead of IPv4 for dc_server if you prefer. If you're at a lost for testing apache's session caching behaviour, you can try swamp at; http://www.geoffthorpe.net/crypto/swamp/ It is written for this kind of thing and it supports multiple destination addresses using arbitrary distribution patterns and arbitrary session resume patterns. Eg; # swamp -cipher RC4-MD5 -num 10 -count 2 -update 10 \ -connect server1.localnet:443 \ -connect server2.localnet:443 \ -connect server3.localnet:443 \ -session srrrsrr \ -distribute 1,2,3,2 The above example will, in each of the 10 parallel contexts, send each subsequent request to servers 1, 2, 3, and then 2 again before starting over - so server2 gets twice as many requests as server1 and server3. Moreover, this pattern cycles every 4 requests, whereas the session-resumption pattern srrrsrr has period 7 so things should mix up nicely (s means neogitate a new session, r means try to resume the last session we successfully negotiated). This will cause each web server to try resuming SSL/TLS sessions previously negotiated by other servers, and the statistics generated by swamp will display how many of these resume attempts succeed. Another aspect of this I would like help with is the stability of the whole architecture as/when various services are interrupted and/or restarted. The distcache side *should* be robust against crashes and/or network interruptions, eg. if dc_client and/or dc_server are killed off and then restarted at some point things should automatically re-organise themselves on the fly. Apache's session caching should fail gracefully if one of the components is shut-off or the distcache networking disappears (which swamp will show with failed resume stats), and should start working again automatically once the problem is resolved. Any behaviour to the contrary is a bug. If anyone is interested in this, please consider popping over to the distcache-users mail list. Cheers, Geoff -- Geoff Thorpe [EMAIL PROTECTED] http://www.geoffthorpe.net/ Index: LAYOUT === RCS file: /home/cvspublic/httpd-2.0/LAYOUT,v retrieving revision 1.2 diff -u -r1.2 LAYOUT --- LAYOUT 8 Dec 2002 21:05:55 - 1.2 +++ LAYOUT 3 Apr 2003 16:50:55 - @@ -129,6 +129,7 @@ ssl_scache_dbm.c session cache via DBM file ssl_scache_shmcb.c .. session cache via shared memory cyclic buffer ssl_scache_shmht.c .. session cache via shared memory hash table +ssl_scache_dc.c . session cache via Distcache (distributed) ssl_util.c .. utility functions ssl_util_ssl.c .. the OpenSSL companion source ssl_util_ssl.h .. the OpenSSL companion header
Re: [PATCH] remove shmht from mod_ssl
Hi there, * William A. Rowe, Jr. ([EMAIL PROTECTED]) wrote: At 10:45 AM 3/20/2003, you wrote: Has anyone done any testing of shmht in 2.0? It quickly stops caching new sessions for me using Geoff Thorpe's swamp tool. If anyone does want to put effort into getting it working, I've attached a patch which contains some fixes: the conversion to the RMM code was not finished. +1, as long as this code remains in the tree, it's good for us to have the very best code available. Might I suggest that anyone interested in shmcb get in contact and we talk about cleaning that up first? As was discussed recently in another thread, there seems to be some concern about the use of the alignment-fudging accessor functions I put in there, and rather than trying to align shmcb's geometry (which is what is suggested in the STATUS file) I suggested that I rejig the code itself to blast the header structures on and off the shm segment directly - simplifying the code, removing alignment hassles, and (I believe) not affecting performance. For my part, I'm happy to do this and I would *certainly* feel better about doing it prior to shmht being unplugged from apache altogether. Call me paranoid if you like. Anyone following openssl lists will know why I've been a bit distracted of late. Apache's autoconf hooks for openssl was my first priority up until it was recently committed, and my second priority in turn is to submit distcache(.org) glue code for 2.1-dev too. In short, I'm ready to roll on shmcb work once the distcache hooks are sorted out - and this (BTW) may also be wise before culling shmht, because it provides another high-performance (if I might be so bold to say) cache alternative to shmcb. I don't necessarily disagree for 2.1-dev, but we've sort of concurred that users shouldn't have to switch modules or configuration significantly in a given minor version (e.g. 2.0). Now I'm not arguing that's it's very unlikely a user successfully used shmht.c, but as a matter of principle, we shouldn't drop this away from 2.0. I agree, for the reasons you stated as well as the reason I mentioned above. There were lingering reports IIRC that shmcb was broken, and in all my dealings with shmcb since writing it, this has generally meant gcc has found another way on another platform to butcher the 'safe' accessor functions. All the more reason to think about getting the shmcb code more robust against such issues before pulling the plug on shmht. Cheers, Geoff -- Geoff Thorpe [EMAIL PROTECTED] http://www.geoffthorpe.net/
Re: [PATCH] openssl versions?
Hi there, Thanks for filling in the SSL-C bits Madhu, looks like a clean fit. * MATHIHALLI,MADHUSUDAN (HP-Cupertino,ex1) ([EMAIL PROTECTED]) wrote: -Original Message- From: William A. Rowe, Jr. [mailto:[EMAIL PROTECTED] [snip] Anyways, nice patch - I'd prefer if you would follow the Fix One Thing rule of committing this patch; e.g. take it back apart and have each commit labeled as to it's single purpose. But at least here, I'm +1 for this to go into 2.1-dev and I'll help continue to review/improve it in-tree. Oh yes definitely - I'll break it into small patches and commit them seperately. I had no intention of committing it to the 2.0-dev. I would, for my part, be quite cautious about putting the new detection code into any existing stable branch. There's a certain amount of apples and oranges involved here - the new detection is essentially autoconf all the way and the existing stuff contains lots of hard-coded paths and *file* checks (as opposed to $(CPP)/$(CC) tests). The autoconf approach is clearly preferable, but whether this has the capacity to bite anyone whose build system or installation target is dependant on the oddities of the existing behaviour is another question. Cheers, Geoff -- Geoff Thorpe [EMAIL PROTECTED] http://www.geoffthorpe.net/
Re: [PATCH] openssl configuration (v2)
Hi Madhu, Thanks for giving the latest incarnation a review. * MATHIHALLI,MADHUSUDAN (HP-Cupertino,ex1) ([EMAIL PROTECTED]) wrote: I tried the patch, and it seemed to work fine for me (and it's more cleaner than what we have today). - I don't know how SSL-C will be broken with this patch - Anybody out there using SSL-C ? - I have a patch to get SSL-C to work with mod_ssl.. I'll have to dust it out, before posting the patch. If it helps move things along I'd be happy to sift through your existing patch to see what it does. Although the two patches are unlikely to cohabitate, it should be obvious enough from your patch what (if anything) needs changing in mine to achieve SSL-C's needs, and I could send an updated version with whatever I find? Ball, your court, etc :-) As for testing SSL-C, can't we just approach Eric or Tim? Or do Covalent or any other commecial Apache vendor use SSL-C? These would seem the obvious avenues to me, but I don't want to pre-empt whatever communications are already going w.r.t. SSL-C support. Just let me know if I can help in some way. - any objections Not from me, but then that should come as no surprise ... Cheers, Geoff -- Geoff Thorpe [EMAIL PROTECTED] http://www.geoffthorpe.net/
Re: [PATCH] openssl configuration (v2)
Hi there, * Justin Erenkrantz ([EMAIL PROTECTED]) wrote: Yes, the former is what I would expect as well. I would add a third clause to your AC_CHECK_LIB which appends -lLIBRARY to saved_LIBS. Hmm. I wonder where saved_LIBS will end up in relationship to AP_LIBS. Ah, it seems that EXTRA_LIBS will be before AP_LIBS - Solaris and other one-pass linkers should be happy. Yeah I've changed this to manually add -lssl -lcrypto to LIBS after it's restored from saved_LIBS and that makes things work. I also added a missing AC_MSG_ERROR() if the libs weren't found, as this is clearly a fatal SSL/TLS configuration error I should have been catching already. So I guess we make do with that then? I know this is what is currently done so I presume this is OK, though I admit to being less than pleased with manually added linker flags given that we're using AC_CHECK_LIB() which is supposed to hide all that (and handle whatever the correct syntax is on the host). NB: If it's later decided that AC_CHECK_LIB() should take care of adding the linker flags internally, the save/restore trick with LIBS would have to go, which in turn means that the other linker flags would have to be configured in advance using AC_CHECK_LIB macros (rather than using $apr_config --libs). in portability problems. Unless someone tells me the Apache-Approved(tm) way to do this, I'd rather just leave the comment there to guide someone else if they feel so moved afterwards. ap_ssltk_version=`$p/openssl version` I kinda like that approach (i.e. what we're currently doing...). I'm not sure I like it but I'll go with the flow if you insist. The thing is, it requires the openssl executable be installed which on package-management systems will be the openssl package containing programs, documents, scripts, certificates, etc. rather than just a libopenssl package contains libs and headers. This also requires us to test for the openssl program and work around the case that it's not found (the current code seems to consider this fatal but that seems dramatic under the circumstances). At the least, can we leave that as a second step once the configuration is tidied up? Right now the version check in configure is OK even if there's no obvious way to propogate a text form of the version to the console. I can't see the implications of requiring the presence of the openssl binary to be worth the effort for a purely aesthetic issue. If you want to be cute, you could do something with AC_EGREP_HEADER, but I'm not totally clear what the syntax would be. I took a look because this idea seemed promising at first look, but the code generated by AC_EGREP_HEADER sends both stdout and stderr to /dev/null, so the regexp seems useful only for a yes/no pattern-match - I can't see any obvious way to capture what comes out of the egrep. Perhaps there's another macro lurking somewhere that would solve all this? - Seeing as CPPFLAGS seems ideal for header *and* compilation checks, is INCLUDES still the appropriate place to APR_ADDTO() any required include path once the tests are done? Yeah, I thought about that after I sent my original message. We really should be adding the -I's to INCLUDES. But, autoconf will only temporarily use CPPFLAGS, so we should add to CPPFLAGS for the header checks, then if it works, add it to INCLUDES (which gets morphed to EXTRA_INCLUDES later on). Our build system should use INCLUDES properly. -- justin OK, I'll stick with INCLUDES then. So AFAICS, if we accept the following; (1) explicitly adding -lssl -lcrypto into LIBS if the library checks succeed (using APR_ADDTO), (2) don't try to grep the version text yet for output from configure nor assume the openssl binary to generate it, (3) put include paths into INCLUDES (using APR_ADDTO) then the patch attached to this mail should be OK? Does this seem reasonable? (Note, I've still left the old version stubbed in - I prefer to leave the diff readable until someone's ready to start using the word commit). Cheers, Geoff -- Geoff Thorpe [EMAIL PROTECTED] http://www.geoffthorpe.net/ Index: acconfig.h === RCS file: /home/cvspublic/httpd-2.0/acconfig.h,v retrieving revision 1.1 diff -u -r1.1 acconfig.h --- acconfig.h 31 Jan 2002 14:51:37 - 1.1 +++ acconfig.h 7 Mar 2003 20:54:37 - @@ -1,2 +1,8 @@ /* Define this if struct tm has a field tm_gmtoff */ #undef HAVE_GMTOFF + +/* Define this if we are building with OpenSSL */ +#undef HAVE_OPENSSL + +/* Define this if we are building with SSL-C */ +#undef HAVE_SSLC Index: acinclude.m4 === RCS file: /home/cvspublic/httpd-2.0/acinclude.m4,v retrieving revision 1.136 diff -u -r1.136 acinclude.m4 --- acinclude.m417 Feb 2003 02:32:19 - 1.136 +++ acinclude.m47 Mar 2003 20:54:37 - @@ -312,7 +312,7 @@ ]) dnl -dnl
Re: mod_ssl to-do items
Hi (again :-), * Justin Erenkrantz ([EMAIL PROTECTED]) wrote: --On Thursday, March 6, 2003 12:42 PM -0500 Geoff Thorpe [EMAIL PROTECTED] wrote: [snip] * session cache store should be pluggable I'd like to speak with the person who's behind this, as I'm waiting for (Pardon me while I go off on a tangent...) Note that there really isn't a single person who is responsible for mod_ssl. The only person who may claim that is Ralf and he hasn't been involved for years. All of us on this list are implicitly responsible. Some people are more involved than others though - Madhu and DougM have been beneficial to keeping it running though (among others that I have forgotten right now - sorry!) - but the point of adding mod_ssl to our tree was so that it could be group-maintained rather than a specific individual. So, I wouldn't go look for a specific person to talk to. Whomever that person would be should be reading this list. Just start having a conversation about what you'd like to see happen on this list and people will magically start to chime in. And, keep submitting patches about what you'd like to see fixed - it's the surefire way of getting our attention. (And, simply repost if you don't hear anything in a week...) Thanks for the tangent, it's good to know what the conventional wisdom is w.r.t. code management (it seems contributing to apache2's mod_ssl might be less troubled by long periods of silence than has been the case with apache1.3 mod_ssl sometimes :-). Nonetheless, I'd like to know who the person(s) are that are behind the statement in the STATUS file so that I can find out what it is (exactly) that they mean and what (if any) history there is to the idea. session cache store should be pluggable has an encouraging sound to it, but is somewhat unburdened by any concrete meaning that I can see. :-) * the shmcb code should just align its memory segment rather than jumping through all the safe memcpy and memset hoops I wrote shmcb and I don't think the person who wrote this quite understands why this jumping through hoops is done. Trying to align all the relevant data will cost more than just a few wasted bytes of storage, and I respectfully suggest that the safe accessors (which aren't speed-critical) would be easier to maintain. Again, I'm available to bash these ideas out if anyone's interested. It's probably under the idea that we'd save a few cycles and make the code more understandable if we relied upon alignment. In fact, I believe the second argument is the more compelling one. I find the safe calls a tremendous difficulty in understanding the shmcb code (which, AIUI, is broken right now). -- justin Broken? How so? As for the safe calls - if they're a difficulty then they should be clarified because the principle is very simple. However to be more constructive than that; they can probably be done away with completely without losing the design if I get approval to restructure ssl_scache_shmcb.c a bit. Namely, the various structures used in the layout/geometry could just be memcpy()d to/from the relevant locations rather than having references to them in which individual data elements are safely accessed one-by-one. The geometry/layout shmcb places on the shared-memory segment was reasonably-well thought out, but the code that implements it is more organic - it grew into its current shape as I attempted to prototype the storage model. A little bit of rewriting and rewiring would (I suspect) greatly simplify things and I'm reasonably confident they would obviate the need for the safe functions - the alignment correction would be implicit by memcpy()ing the larger non-primitive structures between non-aligned shared memory and aligned local variables. This would slightly increase the amount of memcpy() activity (which was the naive reason that the safe approach was used to begin with) but it may just as likely be an improvement if a couple of bigger memcpy()s turn out faster than lots of little ones. Certainly this wouldn't affect the handling of the raw session data itself, and so I doubt any amount of profiling would show the slightest difference one way or the other. Anyway, I can put that on my TODO list if there are people who'd accommodate it by reviewing and helping me test. Cheers, Geoff -- Geoff Thorpe [EMAIL PROTECTED] http://www.geoffthorpe.net/
Re: mod_ssl to-do items
Hi, * John K. Sterling ([EMAIL PROTECTED]) wrote: As justin said, this is something that has been randomly discussed from time to time for many years now. MANY people see the need for this for obvious reasons. As for the STATUS file entry, the concrete meaning (from a high level): 1) create a pluggable session storage interface (ala perl's tied hash) 2) expose it to apache modules 3) implement some standard back-ends 4) convert the ssl code to use it for session storage. Yup, makes sense. I'm still wondering if the right way to do this is in openssl though, because it's a generic problem for SSL/TLS rather than anything specific to HTTPS (and in theory there is no reason why Apache shouldn't be able to share a session cache with other SSL/TLS software). I have heard of some pluggable c session storage code out there, but to add another external project dependency would be lame. perhaps it could get added to apr-util... otherwise mod_ssl would have to compile it in locally. I don't know if this is/was distcache, but distcache is about half my interest in this subject (the other half being openssl). I'm waiting to finish the openssl configuration stuff in http-2.0 before trying to submit a patch for distcache support - so if modularising might affect this I could look at it at the same time and perhaps save myself some effort later on. I think however that the current caching interface is not a great base to be generalising from (which to be fair is essentially inherited from the same limiting interface in OpenSSL that hasn't been substantially touched since SSLeay days). Is anyone working on this already or ready to do so now? If so please make contact so we can avoid duplication of effort. If not, I'll start giving it some thought, although I'll probably ask that I be allowed to sort out and contribute my distcache hooks into the existing modules/ssl first. I need to clear a few things off my TODO list before embarking on investigations of this sort. One of the issues about modularising session caching is that it would make a lot of sense to allow caches to be stacked. This already happens in some sense with openssl managing a per-process stack and apache's SSLSessionCache allowing an application cache to be plugged in on top. That limiting interface I spoke of already causes problems at this point, because there's no has_session type of hook that allows the inner cache to quickly check with the external cache in case a locally-cached session has been *cancelled* by the external cache. This created a vulnerability that was fixed in Apache-2 when I raised the subject on the modssl-users list, but as far as I know is still lurking unchanged in mod_ssl/apache-1.3. The fix of course has no choice but to shut off the internal cache completely and use the external cache directly. So this same interface is not going to lead to any clean way to layer caching mechanisms. (This is stuff I've been dealing with in distcache BTW, but that's another spiel for another day). Thanks for getting back to me about this. Cheers, Geoff -- Geoff Thorpe [EMAIL PROTECTED] http://www.geoffthorpe.net/
Re: [STATUS] (httpd-2.0) Wed Mar 5 23:45:12 EST 2003
I noticed a couple of these notes apply to me in one way or another so figured I'd chime in with some thoughts; * Rodent of Unusual Size ([EMAIL PROTECTED]) wrote: APACHE 2.1 STATUS: -*-text-*- [snip] TODO ISSUES REMAINING IN MOD_SSL: * In order to use a DSO version of mod_ssl we have to link with -lssl and -lcrypto. A workaround is in place right now where the entire EXTRA_LIBS macro is being appended to the objects list, but this is a hack. We should either revamp the APACHE_CHECK_SSL_TOOLKIT autoconf function or come up with some other autoconf checks to search for libssl and libcrypto and properly add them to mod_ssl's link flags. I think the patch I sent to the list (which is awaiting comments BTW) will address this, but this is unconfirmed. Namely, my patch uses only LDFLAGS tweaks for library paths, and then only if --with-ssl=path is specified. For the remainder, autoconf's normal mechanisms for library probes and linker configuration are used. Ie. the actual probing for libssl and libcrypto is handled by the AC_CHECK_LIB() macro and so autoconf alone works out what flags are required from then on for linking apache. I'm wondering if the above problem is a consequence of us explicitly setting linker syntax and not letting autotools do their job properly? Anyway, the patch is there if someone wants to see if it helps with this issue as well. * session cache store should be pluggable I'd like to speak with the person who's behind this, as I'm waiting for my current openssl-configuration changes to be reviewed and ironed out before I submit a new session cache module supporting 'distcache' (www.distcache.org). I'm also wondering if on the openssl-side we need some kind of modularised approach to session caching rather than this duplicating of effort in openssl-based apps. Unfortunately that would probably raise the issue that the basic session caching interface is inadequate and some rewiring would be required first. I see worms oozing out of cans. Anyway, I'm happy to discuss this issue with whoever is behind it in case I can be of any help. * the shmcb code should just align its memory segment rather than jumping through all the safe memcpy and memset hoops I wrote shmcb and I don't think the person who wrote this quite understands why this jumping through hoops is done. Trying to align all the relevant data will cost more than just a few wasted bytes of storage, and I respectfully suggest that the safe accessors (which aren't speed-critical) would be easier to maintain. Again, I'm available to bash these ideas out if anyone's interested. Cheers, Geoff -- Geoff Thorpe [EMAIL PROTECTED] http://www.geoffthorpe.net/
Re: [PATCH] openssl configuration (v2)
not supposed to assume anything about its location). I imagine the proper way is to have a compile test spit the string out and then capture it somehow. No doubt an autoconf-guru knows the relevant macro trick, but on my own I'm more likely to wildly overcomplicate this code and bring in portability problems. Unless someone tells me the Apache-Approved(tm) way to do this, I'd rather just leave the comment there to guide someone else if they feel so moved afterwards. dnl # hook module into the Autoconf mechanism (--enable-ssl option) APACHE_MODULE(ssl, [SSL/TLS support (mod_ssl)], $ssl_objs, , no, [ APACHE_CHECK_SSL_TOOLKIT -AC_CHECK_FUNCS(SSL_set_state) -AC_CHECK_FUNCS(SSL_set_cert_store) +dnl These checks aren't really useful and could fail for silly reasons Just remove them altogether. Agreed. The rest of it looks good. -- justin Thanks for the feedback. I've incorporated all the above and attached a new patch. Does this seem OK? There are a couple of notes I want to leave you with; - I've incorporated the use of `$apr_config --libs` as you suggested and so my patch is currently broken, but I want to head in the right direction. Any ideas why the successfully reported -lssl -lcrypto flags disappear from the generated Makefile's? - I've moved the version checks into the header check steps so I don't have to save and restore CPPFLAGS twice. - Seeing as CPPFLAGS seems ideal for header *and* compilation checks, is INCLUDES still the appropriate place to APR_ADDTO() any required include path once the tests are done? Cheers, Geoff -- Geoff Thorpe [EMAIL PROTECTED] http://www.geoffthorpe.net/ Index: acconfig.h === RCS file: /home/cvspublic/httpd-2.0/acconfig.h,v retrieving revision 1.1 diff -u -r1.1 acconfig.h --- acconfig.h 31 Jan 2002 14:51:37 - 1.1 +++ acconfig.h 6 Mar 2003 21:38:19 - @@ -1,2 +1,8 @@ /* Define this if struct tm has a field tm_gmtoff */ #undef HAVE_GMTOFF + +/* Define this if we are building with OpenSSL */ +#undef HAVE_OPENSSL + +/* Define this if we are building with SSL-C */ +#undef HAVE_SSLC Index: acinclude.m4 === RCS file: /home/cvspublic/httpd-2.0/acinclude.m4,v retrieving revision 1.136 diff -u -r1.136 acinclude.m4 --- acinclude.m417 Feb 2003 02:32:19 - 1.136 +++ acinclude.m46 Mar 2003 21:38:19 - @@ -312,7 +312,7 @@ ]) dnl -dnl APACHE_CHECK_SSL_TOOLKIT +dnl APACHE_CHECK_SSL_TOOLKIT (old version) dnl dnl Find the openssl toolkit installation and check it for the right dnl version, then add its flags to INCLUDES and LIBS. This should @@ -320,7 +320,7 @@ dnl and then AC_TRY_LINK to test the libraries directly for the version, dnl but that will require someone who knows how to program openssl. dnl -AC_DEFUN(APACHE_CHECK_SSL_TOOLKIT,[ +AC_DEFUN(APACHE_CHECK_SSL_TOOLKIT_OLD,[ if test x$ap_ssltk_base = x; then AC_MSG_CHECKING(for SSL/TLS toolkit base) ap_ssltk_base= @@ -421,6 +421,108 @@ fi APR_ADDTO(LIBS, [-lssl -lcrypto]) ap_cv_ssltk=$ap_ssltk_base +fi +]) + +dnl +dnl APACHE_CHECK_SSL_TOOLKIT (new version) +dnl +dnl Configure for the detected openssl/ssl-c toolkit installation, giving +dnl preference to --with-ssl=path if it was specified. +dnl +AC_DEFUN(APACHE_CHECK_SSL_TOOLKIT,[ +if test x$ap_ssltk_configured = x; then + dnl initialise the variables we use + ap_ssltk_base= + ap_ssltk_inc= + ap_ssltk_lib= + ap_ssltk_type= + + dnl Determine the SSL/TLS toolkit's base directory, if any + AC_MSG_CHECKING(for SSL/TLS toolkit base) + AC_ARG_WITH(ssl, APACHE_HELP_STRING(--with-ssl=DIR,SSL/TLS toolkit), [ +dnl If --with-ssl specifies a directory, we use that directory or fail +if test x$withval != xyes -a x$withval != x; then + dnl This ensures $withval is actually a directory and that it is absolute + ap_ssltk_base=`cd $withval ; pwd` +fi + ]) + if test x$ap_ssltk_base = x; then +AC_MSG_RESULT(none) + else +AC_MSG_RESULT($ap_ssltk_base) + fi + + dnl Run header and version checks + saved_CPPFLAGS=$CPPFLAGS + if test x$ap_ssltk_base != x; then +ap_ssltk_inc=-I$ap_ssltk_base/include +CPPFLAGS=$CPPFLAGS $ap_ssltk_inc + fi + AC_CHECK_HEADERS([sslc.h], [ap_ssltk_type=sslc], []) + if test x$ap_ssltk_type = x; then +AC_CHECK_HEADERS([openssl/opensslv.h openssl/ssl.h], [ap_ssltk_type=openssl], []) +if test x$ap_ssltk_type = x; then + AC_MSG_ERROR([No SSL/TLS headers were available]) +fi +dnl so it's OpenSSL - report, then test for a good version +echo ... SSL/TLS support configuring for OpenSSL +AC_MSG_CHECKING(for OpenSSL version) +AC_TRY_COMPILE([#include openssl/opensslv.h], +[#if !defined(OPENSSL_VERSION_NUMBER) || OPENSSL_VERSION_NUMBER 0x0090609f +#error invalid openssl version +#endif], + [dnl Replace this with OPENSSL_VERSION_TEXT from opensslv.h
[PATCH] openssl configuration (v2)
Hi all, Thanks to those who replied to my earlier post on this subject, especially Madhu. Here's the next incarnation of the patch based on that feedback and some ferretting of my own. Patch notes; - This now probes for sslc.h to distinguish between OpenSSL and SSL-C. - Existing version checking only made sense for OpenSSL so is now only done for OpenSSL. - The version error is now just a warning. - HAVE_OPENSSL and HAVE_SSLC are set up from these tests so that modules/ssl/ code can be clearer and less ambiguous. - Libraries are now checked by AC_CHECK_LIB instead of AC_TRY_LINK as in my previous patch. This lets autoconf/automake deal with adding the appropriate -lssl -lcrypto linker flags, so this should be a little more portable. - inclusion of openssl or ssl-c headers now takes place in ssl_toolkit_compat.h instead of mod_ssl.h. This results in cleaner handling of the [HAVE|NO]_SSL_X509V3_H stuff for SSL-C (and which isn't relevant for OpenSSL at all). Also, paths are now correct for headers, so an installed version of OpenSSL or SSL-C will usable as-is without needing any -I flags generated from the configure script. Questions for apache gurus/code-reviewers; - AC_CHECK_HEADERS() appears difficult to coax into accepting additional include paths, so if --with-ssl=path is specified there appears no obvious way to have AC_CHECK_HEADERS() pick up those headers in (particularly if versions exist in system locations too and we want autoconf's tests to find the path versions in preference to any auto-detectable ones). I've left some comments in the acinclude.m4 changes about this. For now, I've made do with adding -I... to CFLAGS prior to AC_TRY_COMPILE, but I'm sure autoconf intended some other way of handling this. For one thing, is -I actually portable anyway? The existing code depends utterly on it but it would be nice to do away with it altogether. - My changes use autoconf tests for openssl/ssl-c headers and libraries (existing code just looks for files but doesn't actually try to use them). As a result, linker flags like -ldl, -lsocket, -lnsl, -ldld, etc are needed in advance of these tests. I've added the obvious ones I know about so that this patch can be tested as-is, but ideally Apache's builtin tests (which are obviously OK because Apache links fine) should occur before the AC_CHECK_LIB()s for ssl and crypto. See Step 3 of my changes to acinclude.m4. - The adjustments made to LDFLAGS at the end of the testing has been written to try and match the existing stuff, but I don't confess to know what the significance of $ap_platform_runtime_link_flag is so I'm working blind there. - I'm tagging -DHAVE_OPENSSL or -DHAVE_SSLC directly onto CFLAGS rather than using anything like AC_DEFINE because the latter possibility would require HAVE_OPENSSL and HAVE_SSLC to be stubbed into an appropriate something.h.in file. If you prefer not to have such stuff polluting CFLAGS then please suggest an appropriate .in file for me to hook into. Note that (I think) the problems of relying on syntax like -I, -L, etc all apply to the existing code anyway, so they're more questions of style than whether my changes will work. The only things I can think of that my changes my break are that I've dumped the builtin paths for searching for headers and libraries and instead let autoconf probes (combined with --with-ssl=path) find whatever they find. In other words, perhaps some people might have been relying on the unconventional anti-autoconf nature of the existing checks that my changes remove? Any/all feedback most welcome. Cheers, Geoff -- Geoff Thorpe [EMAIL PROTECTED] http://www.geoffthorpe.net/ Index: acinclude.m4 === RCS file: /home/cvspublic/httpd-2.0/acinclude.m4,v retrieving revision 1.136 diff -u -r1.136 acinclude.m4 --- acinclude.m417 Feb 2003 02:32:19 - 1.136 +++ acinclude.m44 Mar 2003 23:00:03 - @@ -312,7 +312,7 @@ ]) dnl -dnl APACHE_CHECK_SSL_TOOLKIT +dnl APACHE_CHECK_SSL_TOOLKIT (old version) dnl dnl Find the openssl toolkit installation and check it for the right dnl version, then add its flags to INCLUDES and LIBS. This should @@ -320,7 +320,7 @@ dnl and then AC_TRY_LINK to test the libraries directly for the version, dnl but that will require someone who knows how to program openssl. dnl -AC_DEFUN(APACHE_CHECK_SSL_TOOLKIT,[ +AC_DEFUN(APACHE_CHECK_SSL_TOOLKIT_OLD,[ if test x$ap_ssltk_base = x; then AC_MSG_CHECKING(for SSL/TLS toolkit base) ap_ssltk_base= @@ -421,6 +421,140 @@ fi APR_ADDTO(LIBS, [-lssl -lcrypto]) ap_cv_ssltk=$ap_ssltk_base +fi +]) + +dnl +dnl APACHE_CHECK_SSL_TOOLKIT (new version) +dnl +dnl Configure for the detected openssl/ssl-c toolkit installation, giving +dnl preference to --with-ssl=path if it was specified. +dnl +AC_DEFUN(APACHE_CHECK_SSL_TOOLKIT,[ +if test x
Re: [PATCH] openssl configuration
Hi Thom, * Thom May ([EMAIL PROTECTED]) wrote: * Geoff Thorpe ([EMAIL PROTECTED]) wrote : It would perhaps make sense to provide a --force-ssl-ver type of option that would bypass version checks, and then have any version checking failure text point out the existence of --force-ssl-ver. This way, the more determined users can force configure to bypass that, whilst it still provides a certain safety-net for the more naive and less intrepid against accidently meddling with known-to-be-out-of-date support libraries. (un)?fortunately most vendors prefer to backport security fixes rather than release new versions of software into stable releases since backports are far less likely to interfere with already tested and correctly integrated software. Thus the average user is unlikely to *know* that they would need to force an ssl version. Less intrepid users are far more likely to be following vendor security updates ;-) Well, this is an issue for the httpd developers to decide on, not me. I put the version checks in because (a) to me (again, outside the httpd sphere of view) it seemed logical, but more importantly (b) the existing autoconf checks did essentially the same version checking but in a more fragile form and for a now out-of-date threshold. I'm just as happy to axe the version check or set up warnings in its place. My goal here is to fix the openssl checks so that the currently-incorrect path, include, and linker handling is corrected. Eg. if I set up my system with non-standard PATH, INCLUDES, ld.so.conf, etc - anything following normal autoconf practice will be fine but apache's ssl/tls handling will not. Likewise if --with-ssl=dir is used with a relative path it will succeed the configure checks but fail compilation. W.r.t the version checks, I don't feel passionately about it one way or the other - by all means tell me what the consensus is and I'll rejig the patch for that. I don't think we should have enforced version checks for this; if we do detect an old version I think the most we should do is to suggest that the user checks with their vendor that they have the most uptodate release for their OS; and that said release fixes the (known) security holes. If that's what people want, that's what I'll do. Should I simply leave in a version check equivalent to the existing one (0.9.6e) and not rock the boat? Or should I turn the version error into a version warning? Cheers, Geoff -- Geoff Thorpe [EMAIL PROTECTED] http://www.geoffthorpe.net/
Re: [PATCH] openssl configuration
for this if there's agreement with the approach. I assume the includes required for SSL-C match those that are in the current mod_ssl.h? I don't know how many others have tried to do this, but based on my experience, most of the changes went into a header file.. There were very little modifications required to the .c files. However, I do echo your concern regarding how long will it be, before the two go in different ways. It might become more difficult to maintain one code base for supporting both the toolkits. Well the approach in ssl_toolkit_compat.h looks good, I wish I'd spotted it before (it shouldn't surprise you that I was more used to looking at mod_ssl/apache-1.3 source so didn't expect to find that). As long as compatiblity can be maintained merely by pre-compiler tricks and stub functions/macros, then why not? I think the error is in trying to do this transparently by trying to keep configure checks and source code confined to areas of commonality between different toolkits. Those areas will continue to shrink :-) Cheers, Geoff -- Geoff Thorpe [EMAIL PROTECTED] http://www.geoffthorpe.net/
Re: [PATCH] openssl configuration
Hi Madhu, * MATHIHALLI,MADHUSUDAN (HP-Cupertino,ex1) ([EMAIL PROTECTED]) wrote: I haven't gone through the entire stuff, but some quick questions, based on your patch : Thanks for the initial feedback, it's certainly useful to know this code has been on other people's minds, and that I didn't just stumble onto something old and forgotten. 1. I thought we should not be enforcing openssl version number checks (something like - openssl version SHOULD be 0.9.6i) - mainly because ppl. can apply patches to their previous versions of OpenSSL, and thus avoid security problems.. (ofcourse, I know you're coming from the OpenSSL background, but that was the message I got here when I tried doing something like that) I put the version checks in simply because there are version checks in the existing M4 stuff and I would have thought it unacceptable to the ASF for me to remove them! :-) The current version checks are implemented in a cock-eyed fashion and are also out of date (0.9.6e used to be a meaningful cut-off point, but that has changed more recently). It would perhaps make sense to provide a --force-ssl-ver type of option that would bypass version checks, and then have any version checking failure text point out the existence of --force-ssl-ver. This way, the more determined users can force configure to bypass that, whilst it still provides a certain safety-net for the more naive and less intrepid against accidently meddling with known-to-be-out-of-date support libraries. 2. Regarding hardcoding openssl/some_header_name.h, I think it's a bad idea - how about ppl. who want to use other SSL toolkits along with apache ?. Erm, like what? The only candidate I can think of would be the RSA-C toolkit? If so, I would say that it doesn't make much sense to try and deliberately reduce the focus (or increase the blur, if you prefer) of the configuration checking so that something other than openssl still slips through the configuration looking like openssl. The complexity doesn't seem worth the trouble to me. To support other SSL toolkits, the more sensible approach would be to introduce alternative autoconf checks that are tailored to those toolkits. But anyway, I find it difficult to imagine that the existing version checks in Apache's configure was letting through alternative toolkits anyway?? So perhaps something like --with-rsa-c could be implemented, and it would be able to target RSA-C directly without needing to be fudged to only check things common to both RSA-C and OpenSSL. Likewise, the includes in mod_ssl.h should be #ifdef,#elif,#endif'd so that you get precisely the headers you need for precisely the toolkit you are using (and using precisely the header paths defined in the respective API). The more things go, the more OpenSSL and RSA-C are likely to diverge - I strongly suspect they've already diverged too far for the code in its current form, but perhaps I'm mistaken there. If the only issue right now between the two is what prefix we do or don't add to the include paths, then that should be considered extremely good fortune. Sooner or later though, we'd end up needing to separating things with precompiler logic to handle differences between toolkits, syntax, symbol definitions, enhancements, optional extras, etc. Failing to do that in the autoconf support or the SSL/TLS module code puts Apache in a position of being dependant on two entirely independant development projects remaining similar enough. That's not a solid position to maintain, and you lose any ability to tune to (or benefit from) a toolkit's evolution. It also goes against the grain of why things like autoconf exist. Of course, I'm hardly about to donate hours or days of my own time to further RSA's commercial cause. :-) But if doing some re-organisation work will benefit apache and openssl as well, I wouldn't begrudge helping create targetted support mechanisms for both OpenSSL and RSA-C. It certainly seems a better way to go than taking a hands-off approach to the use of fork()d toolkits - nobody can give Apache or its users any guarantees that they'll be able to rely on continued similarities. Or were you referring to SSLeay? Or compability layers built on top of something else? Cheers, Geoff -- Geoff Thorpe [EMAIL PROTECTED] http://www.geoffthorpe.net/
[PATCH] openssl configuration
Hi, I'll be mailing back here at another time about my work on integrating distcache (www.distcache.org) support into Apache 2. I first need to run a bunch of checks on my autoconf hooks into the Apache 2 build system though ... ... which leads nicely on to why I *am* posting now. The distcache exercise put me in direct contact with the openssl checks in apache (acinclude.m4 + modules/ssl/config.m4), and so I noticed that the code had been waiting in a very strange state for someone to find time to sort it out. so I've been hacking away at it and have attached a (unified) diff of where I'm at. This isn't ready for CVS yet, but it needs (now) the input of an apache hacker to verify a few things and fix an issue to do with how the rest of the build system is tied together. Also, given some of the weirdness of the existing code, any Right Way of doing things is necessarily going to be a big enough change that regressions have to be watched for. Eg. the existing openssl checks just cycle through some preset commonly-used system paths, and don't use any of the normal autoconf mechanisms to find or test anything. It's obviously not supposed to be this way, but I can't guess how disruptive it might be to suddenly commit a correction to it. Well I'll leave off by logging some info about the patch contents. Any and all feedback would most welcome. Cheers, Geoff Patch notes: - I've left the existing APACHE_CHECK_SSL_TOOLKIT implementation but simply renamed it out of the way - this is mainly to make sure the diff is nice and clean and not an abomination caused by diffing two *almost* entirely different blocks of code against one another. - All the hard-coded path checking (where on earth did that come from?) that was used before is gone and the autoconf macros AC_TRY_COMPILE and AC_TRY_LINK are used instead to verify headers and libraries. This needs a regression once-over, but can't possibly be a good thing to leave the way it was. - The existing implementation doesn't try any compile or link tests, because right now any linking test would fail because the basic system library checks hadn't been performed in advance. I've put a FIXME there so someone can take a look at the issue - whatever Apache's existing library checks are is obviously sufficient, because Apache links :-) So it's just a matter of rearranging or reproducing all the standard -ldl -lnsl -lsocket [etc] checks to occur before APACHE_CHECK_SSL_TOOLKIT runs. - Relative paths to --with-ssl were broken but now work. Before they would pass the configure checks if the paths were valid relative to the top-level directory, but would then fail (not surprisingly) when compiling inside modules/ssl/. Linking was also broken for the same reason. - A path supplied to --with-ssl might (before) have been overriden by another linker path containing openssl libraries and yet silently succeeded. This can lead to mismatches between headers and libraries (with all the obvious linker or run-time problems that can lead to), and also has the failing that it succeeds whilst contradicting what was asked for. This can be pretty painful if there are important reasons why --with-ssl was specifying a specific build. - --with-ssl supports both installation bases (where include and lib are subdirectories) as well as compiled openssl source trees (just the lib subdirectory). I think this was the case before too, but because of the other items mentioned, this now actually works. - openssl version isn't used any more, because it requires that the openssl binary be installed. More accurate (and much easier) version checking is possible from openssl's opensslv.h header, and this has the added advantage that RPM type systems only need the libopenssl*** package installed rather than the utils as well. - The version check is less fragile (no regexp string games) and has been bumped to 0.9.6i from 0.9.6e because of the recent SSL/TLS vulnerability. - openssl headers are now included with the openssl/ directory prefix. - modules/ssl/config.m4 no longer needs to check SSL_set_state and SSL_set_cert_store functions - the only possible use for these was because nothing up until that point had run any compiler or link tests on the guesses made by configure. That's now changed so these checks are pointless. -- Geoff Thorpe [EMAIL PROTECTED] http://www.geoffthorpe.net/ Index: acinclude.m4 === RCS file: /home/cvspublic/httpd-2.0/acinclude.m4,v retrieving revision 1.136 diff -u -r1.136 acinclude.m4 --- acinclude.m417 Feb 2003 02:32:19 - 1.136 +++ acinclude.m427 Feb 2003 05:14:01 - @@ -312,7 +312,7 @@ ]) dnl -dnl APACHE_CHECK_SSL_TOOLKIT +dnl APACHE_CHECK_SSL_TOOLKIT (old version) dnl dnl Find the openssl toolkit installation and check it for the right dnl version, then add its flags to INCLUDES and LIBS. This should
