Hi All, I helped Wei Dai wrestle with a similar problem for Crypto++. We wrote a couple audit tools, one of which is attached. I just completed an audit on my /usr/lib. The following OpenSSL modules crashed during a simple load/unload cycle:
/usr/lib/debug/lib/libcrypto.so.0.9.8 /usr/lib/debug/lib/libssl.so.0.9.8 /usr/lib/debug/usr/lib/ssl/engines/lib4758cca.so /usr/lib/debug/usr/lib/ssl/engines/libaep.so /usr/lib/debug/usr/lib/ssl/engines/libatalla.so /usr/lib/debug/usr/lib/ssl/engines/libcapi.so /usr/lib/debug/usr/lib/ssl/engines/libchil.so Since OpenSSL is a C library, a packager could be doing something with side effects. Or if there are globals and an Uninitialize function, the function might be whacking data that's still in use. In the Crypto++ case, there was a global object with a destructor that would destroy an object during unload. No problem, since that's what destructors do. However, if you want to catch a C++ exception across a module boundary, you need RTLD_GLOBAL. But RTLD_GLOBAL causes the runtime linker-loader to fold non-private symbols into one. So if two processes have the shared library open, both processes will call the dtor on the same object. The first will exit cleanly, the second process will continue to run [for a while] with a bad object. The interactions described above are specified in the Linux ABI, and result in a One Definition Rule (ODR) violation. The attached tool stemmed from a feature request [1] (and another similar problem which materialized while I was making the request [2]) on the GCC mailing list. I requested a warning for 'global variables' in compilation units since they [globals] are the cause of the crash under certain circumstances. So far, the response has been RTFM. Unfortunately, nearly 300 additional modules have not yet read the manual, including a couple of GTK modules. Jeffrey Walton Baltimore, MD, US [1] Switch to warn of global variables in a C++ shared object, http://gcc.gnu.org/bugzilla/show_bug.cgi?id=46097 [2] Global variable in static library - double free or corruption error, http://gcc.gnu.org/ml/gcc-help/2010-10/msg00248.html ======================================= jeff...@bruno:~/$ uname -a Linux bruno 2.6.32-25-generic #45-Ubuntu SMP Sat Oct 16 19:52:42 UTC 2010 x86_64 GNU/Linux jeff...@bruno:~/$ apt-cache pkgnames | grep openssl libcrypt-openssl-x509-perl libcrypt-openssl-rsa-perl libcurl3-openssl-dev libcrypt-openssl-random-perl python-openssl libcrypt-openssl-dsa-perl python2.3-pyopenssl openssl-blacklist-extra libglobus-openssl-module0 python-openssl-dbg libopensc-openssl libopenssl-ruby1.9.1 libengine-pkcs11-openssl libglobus-openssl-module-dev python-openssl-doc libcurl4-openssl-dev libglobus-openssl-module-doc libopenssl-ruby python2.4-pyopenssl globus-openssl-progs python2.6-openssl libxmlsec1-openssl libglobus-openssl-dev openssl-blacklist python2-pyopenssl openssl openssl-doc python-pyopenssl libglobus-gsi-openssl-error-dev libopenssl-ruby1.8 libopenssl-ruby1.9 libglobus-gsi-openssl-error-doc libengine-tpm-openssl aolserver4-nsopenssl libcrypt-openssl-bignum-perl libglobus-gsi-openssl-error0 libglobus-openssl jeff...@bruno:~/$ apt-cache showpkg openssl Package: openssl Versions: 0.9.8k-7ubuntu8.3 (/var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_lucid-updates_main_binary-amd64_Packages) (/var/lib/apt/lists/security.ubuntu.com_ubuntu_dists_lucid-security_main_binary-amd64_Packages) (/var/lib/dpkg/status) Description Language: File: /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_lucid-updates_main_binary-amd64_Packages MD5: 977022bc5545601176b69704acc5df9b 0.9.8k-7ubuntu8 (/var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_lucid_main_binary-amd64_Packages) Description Language: File: /var/lib/apt/lists/us.archive.ubuntu.com_ubuntu_dists_lucid_main_binary-amd64_Packages MD5: 977022bc5545601176b69704acc5df9b Reverse Depends: apache2-threaded-dev,openssl apache2-prefork-dev,openssl slurm-llnl,openssl 0.9.8g-9 quassel-core,openssl ganeti2,openssl openvpn,openssl libssl0.9.8,openssl 0.9.6-2 dovecot-common,openssl apache2-threaded-dev,openssl apache2-prefork-dev,openssl docbookwiki,openssl yaws,openssl xmail,openssl x11vnc,openssl wl-beta,openssl wl,openssl uw-imapd,openssl tunneldigger,openssl topal,openssl tinyca,openssl 0.9.7e telnetd-ssl,openssl 0.9.8g-9 sympa,openssl 0.9.5a stunnel4,openssl stone,openssl ssvnc,openssl slurm-llnl,openssl 0.9.8g-9 sisu,openssl sendmail-bin,openssl quassel-core,openssl pyca,openssl 0.9.6 pyca,openssl 0.9.7 prosody,openssl proftpd-basic,openssl osptoolkit,openssl openvas-server,openssl openswan,openssl nufw,openssl nuauth,openssl lighttpd,openssl libstrongswan,openssl libapache2-mod-shib2,openssl kvpnc,openssl kolabd,openssl jabberd14,openssl ipopd,openssl httping,openssl gsoap,openssl gnus,openssl gnumed-server,openssl globus-openssl-progs,openssl ganeti2,openssl ftpd-ssl,openssl 0.9.2b etpan-ng,openssl ensymble,openssl ejabberd,openssl ebox-openvpn,openssl ebox,openssl 0.9.7e dtc-xen,openssl dtc-common,openssl dsniff,openssl dkimproxy,openssl cryptmount,openssl courier-ssl,openssl courier-pop-ssl,openssl courier-imap-ssl,openssl citadel-webcit,openssl citadel-server,openssl boxbackup-server,openssl boxbackup-client,openssl bincimap,openssl bcfg2-server,openssl aolserver4-nsopenssl,openssl 0.9.6 ssl-cert,openssl 0.9.8g-9 openvpn,openssl openssl-blacklist,openssl 0.9.8g-9 mutt,openssl libssl0.9.8,openssl 0.9.6-2 libpam-mount,openssl exim4-base,openssl dovecot-common,openssl ca-certificates,openssl ca-certificates,openssl apache2-threaded-dev,openssl apache2-prefork-dev,openssl Dependencies: 0.9.8k-7ubuntu8.3 - libc6 (2 2.7) libssl0.9.8 (2 0.9.8k-1) zlib1g (2 1:1.1.4) ca-certificates (0 (null)) openssl-doc (0 (null)) ssleay (3 0.9.2b) 0.9.8k-7ubuntu8 - libc6 (2 2.7) libssl0.9.8 (2 0.9.8k-1) zlib1g (2 1:1.1.4) ca-certificates (0 (null)) openssl-doc (0 (null)) ssleay (3 0.9.2b) Provides: 0.9.8k-7ubuntu8.3 - 0.9.8k-7ubuntu8 - Reverse Provides: jeff...@bruno:~/$
// load-unload.exe - load and unload shared objects to see which developers // and packagers did not pay attention to *ALL* the details of a C++ shared // object, RTLD_GLOBAL (so exceptions can be caught), the ABI, and ODR. // // This program is free software: you can redistribute it and/or modify // it under the terms of the GNU General Public License as published by // the Free Software Foundation, either version 3 of the License, or // (at your option) any later version. // // This program is distributed in the hope that it will be useful, // but WITHOUT ANY WARRANTY; without even the implied warranty of // MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See the // GNU General Public License for more details. // // You should have received a copy of the GNU General Public License // along with this program. If not, see <http://www.gnu.org/licenses/>. // // This notice must not be removed from any files, and original // attribution must remain with the files. // // Copyright (C) 2010, Jeffrey Walton (noloa...@gmail.com) // Need pthreads so gdb can follow the call to dlopen(2) // g++ -g -ggdb -O0 -Wall -Wextra load-unload.cpp -o load-unload.exe -ldl -lpthread #include <iostream> using std::cout; using std::cerr; using std::endl; #include <fstream> using std::ifstream; #include <string> using std::string; #include <vector> using std::vector; #include <stdexcept> using std::runtime_error; #include <cstdlib> using std::exit; #include <cctype> using std::isdigit; #include <cstring> using std::strerror; #include <algorithm> using std::find; using std::sort; using std::unique; using std::binary_search; // Include after the standard c/c++ stuff #include <errno.h> #include <dirent.h> #include <dlfcn.h> #include <link.h> #include <elf.h> #include <sys/types.h> #include <sys/stat.h> typedef vector<string> FileList; typedef vector<string> DirList; typedef vector<string> ErrList; typedef size_t MaxDepth; MaxDepth Unlimited = static_cast<MaxDepth>(-1); MaxDepth RootOnly = static_cast<MaxDepth>(1); // See APPLICATION BINARY INTERFACE, Chapter 4. #if !defined(ELFMAG0) # define ELFMAG0 0x7f #endif #if !defined(ELFMAG1) # define ELFMAG1 'E' #endif #if !defined(ELFMAG2) # define ELFMAG2 'L' #endif #if !defined(ELFMAG3) # define ELFMAG3 'F' #endif class ProgramError : public runtime_error { public: ProgramError(const char* msg, int err = -1) : runtime_error(msg), m_err(err) { } ProgramError(const string& msg, int err = -1) : runtime_error(msg.c_str()), m_err(err) { } int error() const { return m_err; } private: int m_err; }; class AutoDir { public: AutoDir(DIR*& dir) : m_dir(dir) { } virtual ~AutoDir() { if(m_dir) { closedir(m_dir); m_dir = NULL; } } private: DIR*& m_dir; }; class AutoLib { public: AutoLib(void*& lib) : m_lib(lib) { } virtual ~AutoLib() { if(m_lib) { dlclose(m_lib); m_lib = NULL; } } private: void*& m_lib; }; void DoDirectoryWalk(const string& entry, FileList& files, ErrList errata, MaxDepth = Unlimited); void DoLoadUnload(const FileList& files, ErrList errata); bool IsSharedObject(const string& file); bool IsElfFormat(const string& file); bool IsKnownToCrash(const string& file); int main(int argc, char* argv[]) { ErrList errata; // non fatal, such as access denied string libdir = "/usr/lib"; // assume /usr/lib if(argc >= 2) libdir = argv[1]; cout << "Testing shared objects in " << libdir << endl; cout << "Press CTLT-C to early exit" << endl; cout << endl; size_t processed = 0; try { FileList files; // MaxDepth depth = RootOnly; MaxDepth depth = Unlimited; if(depth == Unlimited) { cout << "Recursion depth = unlimited" << endl; } else if(depth == RootOnly) { cout << "Recursion depth = root directory only" << endl; } else { cout << "Recursion depth = " << depth << endl; } // Errata is non-fatal errors, such as EACCES. // Recursion depth is RootOnly, Unlimited, or any number in between. DoDirectoryWalk(libdir, files, errata, depth /*recursion*/); char* ldpath = getenv("LD_PATH"); if(ldpath) { // Since LD_PATH was set, recurse all directories // to test all SOs on the alternate path DoDirectoryWalk(ldpath, files, errata, Unlimited /*recursion depth*/); } // Sort for sane viewing processed = files.size(); sort(files.begin(), files.end()); // Remove any duplicates (should there be any duplicates?) FileList::iterator last = unique(files.begin(), files.end()); files.erase(last, files.end()); if(processed) { cout << "Attempting to process " << processed << " files" << endl; DoLoadUnload(files, errata); } } catch(const ProgramError& err) { cerr << err.what() << endl; exit(err.error()); } if(errata.size()) { cout << endl; cout << "Errata:" << endl; for(size_t i = 0; i < errata.size(); i++) { cout << errata[i] << endl; } } if(processed) { cout << endl; cout << "Processed " << processed << " files" << endl; } return 0; } void DoLoadUnload(const FileList& files, ErrList errata) { for(size_t i = 0; i < files.size(); i++) { cout << "Loading/unloading " << files[i] << endl; if(IsKnownToCrash(files[i])) { cout << " * known to crash (skipping): " << files[i] << endl; continue; } void* lib = NULL; AutoLib cleanup(lib); lib = dlopen(files[i].c_str(), RTLD_GLOBAL); if(!lib) lib = dlopen(files[i].c_str(), RTLD_GLOBAL | RTLD_LAZY); if(!lib) { errata.push_back(dlerror()); } } } // Adapted from W. Richard Stevens, Advanced Programming in the Unix Environment void DoDirectoryWalk(const string& entry, FileList& files, ErrList errata, MaxDepth maxDepth) { if(!entry.length()) { throw ProgramError("Entry is not valid", EINVAL); } DirList dirs; struct stat sbuf; struct dirent* dbuf; int ret = 0; // http://linux.die.net/man/2/lstat ret = lstat(entry.c_str(), &sbuf); if(ret < 0) { string err; if(EACCES == errno) { err = string("lstat failed for ") + entry + " (EACCES)"; errata.push_back(err); return; } err = string("lstat failed for ") + entry + ": "; throw ProgramError(err + strerror(ret), errno); } if(S_ISREG(sbuf.st_mode)) { // if(IsSharedObject(entry)) // files.push_back(entry); if(IsElfFormat(entry)) files.push_back(entry); return; } if(!S_ISDIR(sbuf.st_mode)) return; string path = entry; if(path[path.size()-1] != '/') path += '/'; DIR* dp = NULL; AutoDir cleanup(dp); dp = opendir(path.c_str()); if(!dp) throw ProgramError(strerror(ret), errno); if(maxDepth--) { // Only add directories if we are still recursing while(NULL != (dbuf = readdir(dp))) { if(0 == strcmp(".", dbuf->d_name)) continue; if(0 == strcmp("..", dbuf->d_name)) continue; dirs.push_back(path + dbuf->d_name); } for(size_t i = 0; i < dirs.size(); i++) { DoDirectoryWalk(dirs[i], files, errata, maxDepth); } } } bool IsKnownToCrash(const string& file) { // How large is this list going to get? Should it be sorted and searched? if(file == "/usr/lib/libwxsmithlib.so.0.0.1") return true; if(file == "/usr/lib/libwx_gtk2u_core-2.8.so.0") return true; if(file == "/usr/lib/debug/usr/lib/libwxsmithlib.so.0.0.1") return true; if(file == "/usr/lib/debug/usr/lib/ssl/engines/libcswift.so") return true; if(file == "/usr/lib/debug/usr/lib/ssl/engines/libgmp.so") return true; if(file == "/usr/lib/debug/usr/lib/ssl/engines/libnuron.so") return true; if(file == "/usr/lib/debug/usr/lib/ssl/engines/libsureware.so") return true; if(file == "/usr/lib/debug/usr/lib/ssl/engines/libubsec.so") return true; // And beloved crypto libraries and its ssl companion.... if(file == "/usr/lib/debug/lib/libcrypto.so.0.9.8") return true; if(file == "/usr/lib/debug/lib/libssl.so.0.9.8") return true; if(file == "/usr/lib/debug/usr/lib/libcrypto++.so.8.0.0") return true; if(file == "/usr/lib/debug/usr/lib/ssl/engines/lib4758cca.so") return true; if(file == "/usr/lib/debug/usr/lib/ssl/engines/libaep.so") return true; if(file == "/usr/lib/debug/usr/lib/ssl/engines/libatalla.so") return true; if(file == "/usr/lib/debug/usr/lib/ssl/engines/libcapi.so") return true; if(file == "/usr/lib/debug/usr/lib/ssl/engines/libchil.so") return true; // It looks like every thing from CodeBlocks is broken.... if(string::npos != file.find("/usr/lib/codeblocks/plugins/")) return true; if(file == "/usr/lib/debug/usr/lib/libcodeblocks.so.0.0.1") return true; // And about half the stuff from OpenOffice if(string::npos != file.find("/usr/lib/openoffice/basis3.2/program/")) return true; return false; } // Cheap, but keeps us from opening each file looking for an elf signature and other attributes. bool IsSharedObject(const string& file) { static const string ext = "so"; string::size_type pos = file.rfind(ext); if(string::npos == pos) return false; // This catches all files that end with '.so' if(file.substr(pos) == ext) return true; // This catches all files that end with '.so.W',, '.so.W.X', ..., '.so.W.X.Y.Z'. It will // probably miss some interesting SOs like OpenSSL since a letter is used in versioning. pos += ext.length(); while(pos < file.length()) { const char c = file[pos]; if(c != '.' && !isdigit(c)) return false; pos++; } return true; } // Heavier weight, but more reliable. bool IsElfFormat(const string& file) { try { // We are only interested in the first four bytes of Elf32_Ehdr. // ELFMAG0 can take on a few values. See APPLICATION BINARY INTERFACE, Chapter 4. static const char elf1[] = { 0x7F, ELFMAG1, ELFMAG2, ELFMAG3 }; static const char elf2[] = { 0xB1, ELFMAG1, ELFMAG2, ELFMAG3 }; std::ifstream stream; stream.open(file.c_str()); char ident[4] = {0,0,0,0}; stream.read(ident, sizeof(ident)); if(0 == memcmp(elf1, ident, 4) || 0 == memcmp(elf2, ident, 4)) return true; } catch(std::exception& e) { } return false; }
