OpenSSL and LSB Build Tools
Using the attached patch, OpenSSL can be built using lsbcc3 on Linux:
$ ./Configure linux-generic32-lsb
$ make CC=lsbcc3
Since this might be of general interest, could you please consider the
attached patch for future versions of OpenSSL?
Regards, Remo
diff -ru openssl-0.9.8g/Configure openssl-0.9.8g-lsb/Configure
--- openssl-0.9.8g/Configure 2007-09-16 14:24:17.0 +0200
+++ openssl-0.9.8g-lsb/Configure 2008-04-23 10:45:58.0 +0200
@@ -314,6 +314,7 @@
# *-generic* is endian-neutral target, but ./config is free to
# throw in -D[BL]_ENDIAN, whichever appropriate...
linux-generic32,gcc:-DTERMIO -O3 -fomit-frame-pointer -Wall::-D_REENTRANT::-ldl:BN_LLONG RC4_CHAR RC4_CHUNK DES_INT DES_UNROLL BF_PTR:${no_asm}:dlfcn:linux-shared:-fPIC::.so.\$(SHLIB_MAJOR).\$(SHLIB_MINOR),
+linux-generic32-lsb,gcc:-DTERMIOS -O3 -fomit-frame-pointer -Wall::-D_REENTRANT::-ldl:BN_LLONG RC4_CHAR RC4_CHUNK DES_INT DES_UNROLL BF_PTR:${no_asm}:dlfcn:linux-shared:-fPIC::.so.\$(SHLIB_MAJOR).\$(SHLIB_MINOR),
linux-ppc, gcc:-DB_ENDIAN -DTERMIO -O3 -Wall::-D_REENTRANT::-ldl:BN_LLONG RC4_CHAR RC4_CHUNK DES_RISC1 DES_UNROLL::linux_ppc32.o::dlfcn:linux-shared:-fPIC::.so.\$(SHLIB_MAJOR).\$(SHLIB_MINOR),
IA-32 targets...
linux-ia32-icc, icc:-DL_ENDIAN -DTERMIO -O2 -no_cpprt::-D_REENTRANT::-ldl:BN_LLONG ${x86_gcc_des} ${x86_gcc_opts}:${x86_elf_asm}:dlfcn:linux-shared:-KPIC::.so.\$(SHLIB_MAJOR).\$(SHLIB_MINOR),
@@ -321,6 +322,7 @@
linux-aout, gcc:-DL_ENDIAN -DTERMIO -O3 -fomit-frame-pointer -march=i486 -Wall::(unknown):::BN_LLONG ${x86_gcc_des} ${x86_gcc_opts}:${x86_out_asm},
linux-generic64,gcc:-DTERMIO -O3 -Wall::-D_REENTRANT::-ldl:SIXTY_FOUR_BIT_LONG RC4_CHAR RC4_CHUNK DES_INT DES_UNROLL BF_PTR:${no_asm}:dlfcn:linux-shared:-fPIC::.so.\$(SHLIB_MAJOR).\$(SHLIB_MINOR),
+linux-generic64-lsb,gcc:-DTERMIOS -O3 -Wall::-D_REENTRANT::-ldl:SIXTY_FOUR_BIT_LONG RC4_CHAR RC4_CHUNK DES_INT DES_UNROLL BF_PTR:${no_asm}:dlfcn:linux-shared:-fPIC::.so.\$(SHLIB_MAJOR).\$(SHLIB_MINOR),
linux-ppc64, gcc:-m64 -DB_ENDIAN -DTERMIO -O3 -Wall::-D_REENTRANT::-ldl:SIXTY_FOUR_BIT_LONG RC4_CHAR RC4_CHUNK DES_RISC1 DES_UNROLL::linux_ppc64.o::dlfcn:linux-shared:-fPIC:-m64:.so.\$(SHLIB_MAJOR).\$(SHLIB_MINOR),
linux-ia64, gcc:-DL_ENDIAN -DTERMIO -O3 -Wall::-D_REENTRANT::-ldl:SIXTY_FOUR_BIT_LONG RC4_CHUNK:${ia64_asm}:dlfcn:linux-shared:-fPIC::.so.\$(SHLIB_MAJOR).\$(SHLIB_MINOR),
linux-ia64-ecc,ecc:-DL_ENDIAN -DTERMIO -O2 -Wall -no_cpprt::-D_REENTRANT::-ldl:SIXTY_FOUR_BIT_LONG RC4_CHUNK:${ia64_asm}:dlfcn:linux-shared:-fPIC::.so.\$(SHLIB_MAJOR).\$(SHLIB_MINOR),
diff -ru openssl-0.9.8g/crypto/ui/ui_openssl.c openssl-0.9.8g-lsb/crypto/ui/ui_openssl.c
--- openssl-0.9.8g/crypto/ui/ui_openssl.c 2005-09-26 04:40:18.0 +0200
+++ openssl-0.9.8g-lsb/crypto/ui/ui_openssl.c 2008-04-23 10:37:21.0 +0200
@@ -182,7 +182,7 @@
# undef SGTTY
#endif
-#if defined(linux) !defined(TERMIO)
+#if defined(linux) !defined(TERMIO) !defined(TERMIOS)
# undef TERMIOS
# define TERMIO
# undef SGTTY
Re: OpenSSL and LSB
Theodore Tso wrote: I would suggest that the best way to do this is to *add* new mutator functions (and accessor functions, where necessary) which applications who care about ABI stability can use, and then document a set of interfaces for which ABI stability is guaranteed. That could be a relatively small set initially --- enough for applications that aren't doing anything strange, and then this list can gradually be expanded out, with some new ABI stable functions getting added as necessary. Possibly. There are (or have been) analogous developments that could contribute to your thinking of how to address LSB concerns. Eg. the FIPS work has certification, signature-checking, and other fixed interface considerations (I think, I haven't been involved in this at all). Also, OPENSSL_NO_DEPRECATED is used as a mechanism to nudge away from backwards-compatible interfaces to their preferred replacements (more of this coming to the cvs list in a day or three, BTW). If you define this symbol when building your code against openssl, deprecated interfaces will not be exposed from the headers. If you define this symbol when building openssl itself, deprecated interfaces will not be compiled (I know this doesn't address structures/macros/etc that are not compiled/linkable interfaces, but you get the general idea). There appear to be a few possible avenues to explore; * define LSB-specific wrapper libraries/interfaces to implement a stable ABI, * define only a small subset of functionality in LSB, with the intention of openssl keeping *those* interfaces fixed, * look for ways that a FIPS/NO_DEPRECATED-style pre-processor configuration could compile openssl in some LSB-friendly manner, * define LSB relative to an openssl stable branch. The last point should perhaps not be scoffed at. This is how compatibility is handled in the real world anyway - until openssl reaches some kind of v1.0 state, 0.9.7 is considered a different library to 0.9.8, such that fixes and maintenance for each branch *do* maintain ABI compatibility. Ie. distributions typically install distinct shared-libraries for each, and track them with distinct packages. Having said all that, I'll have to leave this issue to others who have the motivation and time for it. Cheers, Geoff __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: OpenSSL and LSB
and is written in C++. I doubt that will happen. You'd alienate a lot of developers out there and lose support of some systems, especially embedded systems, which may not even have a C++ compiler. I don't think there is any justification for C++. -Brad __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: OpenSSL and LSB
On Sun, Mar 16, 2008 at 09:21:05PM -0700, Michael Sierchio wrote: It is *so* difficult to critique something without seeming to criticize the work of others, so the following disclaimer applies. MUCH is owed to the developers and maintainers of OpenSSL -- Mark, Ralf, Stephen, Ben, Lutz, Nils, Richard, Bodo, Ulf, Andy, Geoff -- and a host of others. OpenSSL is ubiquitous, thanks in large part to them. 108 bows to each of you. Indeed; for myself, I was trying very hard *not* to critique, or at least voice any value judgements, but instead to try to describe the current status of the codebase. Having once been a steward of such a codebase (MIT Kerberos v5, from circa 1993 to 1999), and I know how much work it is to maintain backwards compatibility, even at an API level, while worrying about trying to make it more maintainable going forward. We were also under-resourced for most of my time there --- sound familiar? :-) If one were really serious, it calls for a rewrite -- one that replaces the dreadful BIO-stuff, develops a strictly modular separation of crypto libraries (which are used so many places other than for SSL/TLS), etc. and is written in C++. Well, there *are* other SSL libraries out there, and in fact some people have suggested that the LSB standardize one of the alternate libraries. But, all aside from the licensing issues, the fact of the matter is that for better or for worse, the vast majority of the applications out there are using OpenSSL; it *is* the defacto standard. A rewrite wouldn't help all of the exsting applications which are using the current API. And where are we going to find the resources to do a rewrite? I have talked to one a developer from one ISV (which I won't name pending getting permission from him that it's OK) that the ABI situation has gotten painful enough that he was thinking about creating a wrapper library that would provide a stable ABI, and push the distro's to ship that so that his (fairly widely downloaded and used) binary application could link against a stable library that he knew was guaranteed to work across multiple Linux distro's. So this is not just an theoretical problem, or something which is LSB-specific, but it is something that is affecting OpenSSL client applications. So the question is what's the lowest cost method, which is still maintainable in the long run, for providing this compatibility? I would suggest that the best way to do this is to *add* new mutator functions (and accessor functions, where necessary) which applications who care about ABI stability can use, and then document a set of interfaces for which ABI stability is guaranteed. That could be a relatively small set initially --- enough for applications that aren't doing anything strange, and then this list can gradually be expanded out, with some new ABI stable functions getting added as necessary. Does that make sense? - Ted __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: OpenSSL and LSB
Theodore Ts'o wrote: Reading through the mail archives, the problem, as I understand it, is that OpenSSL is derived from a very old legacy codebase, with an interface which relies on publically visible data structures which must be accessed either directly, or via accessor macros. In some cases, those macros could be changed to accessor functions, and it looks like the easy cases have been done --- but in other cases, the macros couldn't be replaced with accessor functions without causing an API change which would breaking applications. Is that a fair summary of the situation? It is *so* difficult to critique something without seeming to criticize the work of others, so the following disclaimer applies. MUCH is owed to the developers and maintainers of OpenSSL -- Mark, Ralf, Stephen, Ben, Lutz, Nils, Richard, Bodo, Ulf, Andy, Geoff -- and a host of others. OpenSSL is ubiquitous, thanks in large part to them. 108 bows to each of you. OpenSSL derives from Eric Young's work of many, many years ago. It has come to resemble a tarmac that is mostly patches. Yes, yours is a fair assessment. My first impression (On first looking into Chapman's Homer) was -- wow, it really *is* painful to write C++ in C. That was more than a decade ago. If one were really serious, it calls for a rewrite -- one that replaces the dreadful BIO-stuff, develops a strictly modular separation of crypto libraries (which are used so many places other than for SSL/TLS), etc. and is written in C++. Just my USD 0.02^H1^H05 - Michael __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: OpenSSL and LSB (patch)
Camp, TracyX E wrote: ... Comments, etc. welcomed. I've committed one part of the patch (to OpenSSL_0_9_8-stable and the cvs head). The changes in crypto/x509/ are little bit problematic IMHO as it could require code changes in application using the existing macros. For example if you have something like: const X509 *cert = ; ... const ASN1_TIME *atime = X509_get_notBefore(cert); this should at least create a warning as your function prototype is ASN1_TIME *X509_get_notBefore(X509 *); I would prefer const ASN1_TIME *X509_get_notBefore(const X509 *); but this will cause problems if the code relies on the fact the returned object is not const ... For 0.9.9-dev I would prefer making the parameter (and hence the return type) in the X509_foo_get() functions const but I'm not sure if we should change this behaviour in 0.9.8-stable. Comments are welcome. Cheers, Nils __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
RE: OpenSSL and LSB (patch)
For 0.9.9-dev I would prefer making the parameter (and hence the return type) in the X509_foo_get() functions const but I'm not sure if we should change this behaviour in 0.9.8-stable. Comments are welcome. I think I had experimented with making things const at one point but had concluded that might break things in the manner you describe since macros are implicily always non-const. For the purposes of the LSB exersize, making the change in 0.9.9 with const declarations sounds like a reasonable thing to do, since the timing of LSB probably indicates a 0.9.9 release will be available (and hopefully in distros) before LSB 4.0. Thanks, Tracy Camp Cheers, Nils __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED] __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: OpenSSL and LSB (patch)
Ben Laurie wrote: Camp, TracyX E wrote: I really didn't hear much back on this topic, but what I did hear seemed to support the approach I have been taking thus far. Please see my previous message in this thread for the details. So in hopes of moving this topic along in a practical sense, I have attached two patches that implement all the macro to function conversion that I believe would be necessary to support OpenSSL ABI stability from an LSB perspective going forward by allowing applications to always treat OpenSSL data structures as opaque types. One patch is against the released OpenSSL 0.9.8d and another is against the development snapshot from October 16th. As far as I can tell, these conversions should not present any sort of noticeable performance impact since they are almost all related to protocol setup in some manner. I also believe the changes should not impact binary compatibility with applications that where build using the macro versions of these routines. Where possible I've also attempted to preserve the coding style in use. To get OpenSSL into the LSB either a different approach would need to be proposed, or some form of the attached patches would need to make it into an OpenSSL release, followed by a significant number of Linux distributions picking up that release or later. There is an upcoming early next-year LSB 3.2 release that potentially could be the release vehicle for OpenSSL in LSB. Comments, etc. welcomed. I'm very much in favour of this. so am I Cheers, Nils __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: OpenSSL and LSB (patch)
Camp, TracyX E wrote: I really didn't hear much back on this topic, but what I did hear seemed to support the approach I have been taking thus far. Please see my previous message in this thread for the details. So in hopes of moving this topic along in a practical sense, I have attached two patches that implement all the macro to function conversion that I believe would be necessary to support OpenSSL ABI stability from an LSB perspective going forward by allowing applications to always treat OpenSSL data structures as opaque types. One patch is against the released OpenSSL 0.9.8d and another is against the development snapshot from October 16th. As far as I can tell, these conversions should not present any sort of noticeable performance impact since they are almost all related to protocol setup in some manner. I also believe the changes should not impact binary compatibility with applications that where build using the macro versions of these routines. Where possible I've also attempted to preserve the coding style in use. To get OpenSSL into the LSB either a different approach would need to be proposed, or some form of the attached patches would need to make it into an OpenSSL release, followed by a significant number of Linux distributions picking up that release or later. There is an upcoming early next-year LSB 3.2 release that potentially could be the release vehicle for OpenSSL in LSB. Comments, etc. welcomed. I'm very much in favour of this. One comment, though - if you're going to make structure opaque, then why not make them truly opaque by removing their definitions from the public headers? Thanks, Tracy Camp -- http://www.apache-ssl.org/ben.html http://www.links.org/ There is no limit to what a man can do or how far he can go if he doesn't mind who gets the credit. - Robert Woodruff __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
RE: OpenSSL and LSB (patch)
I'm very much in favour of this. One comment, though - if you're going to make structure opaque, then why not make them truly opaque by removing their definitions from the public headers? In LSB they would be truly opaque. LSB works by producing a set of stub libraries and generated headers that contain the defined interfaces. In those generated headers, the types would be opaque. These stub libs and headers however are only used by applications attempting to build with the LSB environment, the real openssl remains unchanged and is utilized at runtime. I don't really see changing all of openssl directly as a reasonable solution however since a vast mountain of software is already written with those structures defined. Tracy Camp -- http://www.apache-ssl.org/ben.html http://www.links.org/ There is no limit to what a man can do or how far he can go if he doesn't mind who gets the credit. - Robert Woodruff __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED] __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
Re: OpenSSL and LSB (patch)
Camp, TracyX E wrote: I'm very much in favour of this. One comment, though - if you're going to make structure opaque, then why not make them truly opaque by removing their definitions from the public headers? In LSB they would be truly opaque. LSB works by producing a set of stub libraries and generated headers that contain the defined interfaces. In those generated headers, the types would be opaque. These stub libs and headers however are only used by applications attempting to build with the LSB environment, the real openssl remains unchanged and is utilized at runtime. I don't really see changing all of openssl directly as a reasonable solution however since a vast mountain of software is already written with those structures defined. How is that vast amount of software going to work under LSB, then? In any case, ABI stability is not a problem unique to LSB, and I'm not sure why we'd want to solve it for only that project. Cheers, Ben. -- http://www.apache-ssl.org/ben.html http://www.links.org/ There is no limit to what a man can do or how far he can go if he doesn't mind who gets the credit. - Robert Woodruff __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
RE: OpenSSL and LSB (patch)
How is that vast amount of software going to work under LSB, then? LSB does not support all applications anyways (for example, anything that has a direct kernel dependency is not a LSB application), and we are not trying to do so here either. Instead I'm attempting to be able to support something close to 70% of existing applications (i.e. they will 'just work' or only require minor tweaks to be LSB compliant), whereas it might be a bit rude to entirely break the last 30% in the more general openssl distribution case. In any case, ABI stability is not a problem unique to LSB, and I'm not sure why we'd want to solve it for only that project. I'm personally totally okay with that, but I'm also really motived from an LSB perspective and presume minimally invasive change would be the most acceptable in the upstread OpenSSL distributions. The difference really is that with this patch you are more able to create ABI stable applications by following good practice, but the choice is really yours, but when using LSB you have to because LSB would force that upon you. Tracy Camp Cheers, Ben. -- http://www.apache-ssl.org/ben.html http://www.links.org/ There is no limit to what a man can do or how far he can go if he doesn't mind who gets the credit. - Robert Woodruff __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED] __ OpenSSL Project http://www.openssl.org Development Mailing List openssl-dev@openssl.org Automated List Manager [EMAIL PROTECTED]
OpenSSL and LSB
Id like to re-start some discussion about including OpenSSL in LSB (acronym expansion: Linux Standards Base see: http://www.freestandards.org). There is apparently general interest in seeing OpenSSL in LSB both for its SSL functionality as well as its utility as a general cryptographic library. In particular applications that wish to be LSB compliant would benefit from security bug fixes. Currently applications that use OpenSSL and wish to be LSB compliant are forced to statically link OpenSSL and in theory at least, may need to re-release binaries for each OpenSSL release (or at least evaluate if a re-release is necessary), something that is typically just not practical for most LSB applications. The general issues with incorporating OpenSSL into LSB are ABI stability since the magic 1.0.0 release (while closer) has not happened yet, and just the shear size of the OpenSSL API. I spent quiet a bit of time conducting an analysis of 0.7.x-0.9.8d OpenSSL releases. (documented at http://www.freestandards.org/en/OpenSSL_stability_analysis) The analysis attempted to determine three things: What APIs are actually used by released applications and can a sub-set of the total OpenSSL API be created that is functionally useful and finally is that API subset ABI stable across many releases of OpenSSL. The answers are generally, yes it is possible to determine an API subset that will work for most applications that make use of OpenSSL (language bindings and the like are ignored here), and more-or-less yes that subset is ABI stable. Further experiments attempted to determine if by treating all or most OpenSSL structure types as opaque types (i.e. no stack allocations or direct structure manipulations) would give ABI stability. The quick answer there again is yes this provided almost perfect ABI stability, but with a loss of functionality. The loss of functionality was because a fairly large portion of the OpenSSL API is implemented as macros that perform trivial manipulations on OpenSSL data types. However since these are macros, it is not also possible to force the data types they manipulate to be opaque. In almost all cases it is fairly trivial to convert the macros into functions and remove the dependency on fully typed data types. Which brings me to my questions for the OpenSSL developer community: 1) First and most fundamentally: Is the general idea of attempting to only incorporate an API subset into the LSB ABI a good or acceptable one? LSB ABIs typically do not include every last interface of most libraries that are standardized either because the non-included APIs are really library internals, the API is not stable or is only useful in specialized cases. In this case the APIs I didnt include are as far as I can tell only useful in specialized cases, or have higher level APIs that provide the same functionality (EVP vs. DES). 2) Is my concern about API stability going forward using fully typed structures unfounded? 3) Are there concerns about converting large numbers of macros into functions? (Performance etc.) Thanks, Tracy Camp
Re: OpenSSL and LSB
IBM uses OpenSSL internally for crypto, though we have our own SSL library. To make it consumable for IBM products we did most of the things you've mentioned in a wrapper layer and provide essentially only the EVP layer to IBM applications. Anything where we've needed access to the underlying data more generally we've added accessor functions, there's no noticeable performance impact. Where this has been needed it's generally been for something that's relatively slow anyway - i.e. Diffie-Hellman. The opaque data types have generally been fine for normal use, the only times we've been forced to cheat and peek under the covers has been when we've been coding up some of the NIST test cases. The only thing that's been really inconvenient to deal with in this way is ASN.1 parsing, and again that's generally only been when writing test cases or working around bugs - i.e. the recent PKCS#1.5 padding problem. 1) First and most fundamentally: Is the general idea of attempting to only incorporate an API subset into the LSB ABI a good or acceptable one? LSB ABIs typically do not include every last interface of most libraries that are standardized either because the non-included APIs are really library internals, the API is not stable or is only useful in specialized cases. In this case the APIs I didn’t include are as far as I can tell only useful in specialized cases, or have higher level APIs that provide the same functionality (EVP vs. DES). 2) Is my concern about API stability going forward using fully typed structures unfounded? 3) Are there concerns about converting large numbers of macros into functions? (Performance etc.) 1) EVP seems sufficient with a few small tweaks. OpenSSL's SSL layer may not yet sit cleanly on that but it could. 2) Opaque data structures is the way to go in our experience. Some apps. may claim they need more, but the same can be said of the data structures within the kernel. 3) ASN.1 is the only obvious problem I've noticed, performance is unlikely to be an issue. There are a couple of other issues you may need to deal with: Callbacks: everyone +dog setting the threading and memory allocation callbacks doesn't work where you have multiple independent shared libraries using OpenSSL within the same process. Various parts of the application end up stealing one anothers callbacks and/or crashing because the shared library where the callbacks point was unloaded while some other part of the application was using OpenSSL. I'd guess you'd hit this if you started using OpenSSL in a big way within Gnome or KDE. We just set them to a sane default when the shared library containing our wrapper code gets loaded and ignore further attempts to set them, this probably needs to be a build option. We also added namespacing to OpenSSL to avoid crashes when customer application code linked with other versions of OpenSSL ended up in the same process - for LSB you may not need that since the objective is to have everyone use one common version. Cheers Peter Peter Waltenberg Camp, TracyX E [EMAIL PROTECTED] tel.com To Sent by: openssl-dev@openssl.org owner-openssl-dev cc @openssl.org Subject OpenSSL and LSB 26/10/06 05:53 AM Please respond to openssl-dev I’d like to re-start some discussion about including OpenSSL in LSB (acronym expansion: Linux Standards Base see: http://www.freestandards.org ). There is apparently general interest in seeing OpenSSL in LSB both for its SSL functionality as well as its utility as a general cryptographic library. In particular applications that wish to be LSB compliant would benefit from security bug fixes. Currently applications that use OpenSSL and wish to be LSB compliant are forced to statically link OpenSSL and in theory at least, may need to re-release binaries for each OpenSSL release (or at least evaluate if a re-release
