---- Rainer Jung <rainer.j...@kippdata.de> wrote:
> On 22.06.2012 14:16, oh...@cox.net wrote:
> >
> > ---- oh...@cox.net wrote:
> >>
> >> ---- Rainer Jung <rainer.j...@kippdata.de> wrote:
> >>> On 22.06.2012 06:10, Joe Lewis wrote:
> >>>> On 6/21/12 10:02 PM, oh...@cox.net wrote:
> >>>>> ---- Joe Lewis<j...@joe-lewis.com> wrote:
> >>>>>> On 6/21/12 9:39 PM, oh...@cox.net wrote:
> >>>>>>> ---- oh...@cox.net wrote:
> >>>>>>>> ---- oh...@cox.net wrote:
> >>>>>>>>> ---- Joe Lewis<j...@joe-lewis.com> wrote:
> >>>>>>>>>> On 6/21/12 7:32 PM, oh...@cox.net wrote:
> >>>>>>>>>>> ---- oh...@cox.net wrote:
> >>>>>>>>>>>> ---- Joe Lewis<j...@joe-lewis.com> wrote:
> >>>>>>>>>>>>> On 6/21/12 6:46 PM, oh...@cox.net wrote:
> >>>>>>>>>>>>>> ---- Joe Lewis<j...@joe-lewis.com> wrote:
> >>>>>>>>>>>>>>> On 6/21/12 5:49 PM, oh...@cox.net wrote:
> >>>>>>>>>>>>>>>> ---- oh...@cox.net wrote:
> >>>>>>>>>>>>>>>>> ---- Sorin Manolache<sor...@gmail.com> wrote:
> >>>>>>>>>>>>>>>>>> And I forgot to say: run gdb in some sort of environment
> >>>>>>>>>>>>>>>>>> where you see
> >>>>>>>>>>>>>>>>>> your current source code line and a couple of surrounding
> >>>>>>>>>>>>>>>>>> lines. You
> >>>>>>>>>>>>>>>>>> could achieve this with the "list" command, but I prefer
> >>>>>>>>>>>>>>>>>> running gdb in
> >>>>>>>>>>>>>>>>>> emacs and let emacs do the nice listing of source code in
> >>>>>>>>>>>>>>>>>> a different panel.
> >>>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>> S
> >>>>>>>>>>>>>>>>> Here's the function from my source. It's the original
> >>>>>>>>>>>>>>>>> from mod_headers.c, plus my printf:
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> static int header_post_config(apr_pool_t *pconf,
> >>>>>>>>>>>>>>>>> apr_pool_t *plog,
> >>>>>>>>>>>>>>>>> apr_pool_t *ptemp,
> >>>>>>>>>>>>>>>>> server_rec *s)
> >>>>>>>>>>>>>>>>> {
> >>>>>>>>>>>>>>>>> printf("In header_post_config\n");
> >>>>>>>>>>>>>>>>> header_ssl_lookup =
> >>>>>>>>>>>>>>>>> APR_RETRIEVE_OPTIONAL_FN(ssl_var_lookup);
> >>>>>>>>>>>>>>>>> return OK;
> >>>>>>>>>>>>>>>>> }
> >>>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>>> Jim
> >>>>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> I was able to get the segfault to go away. Here's what I
> >>>>>>>>>>>>>>>> had to do:
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> - Created /etc/ld.so.conf.d/my.conf, and added the
> >>>>>>>>>>>>>>>> directory where my libobaccess.so was
> >>>>>>>>>>>>>>>> - Run 'ldconfig' to activate.
> >>>>>>>>>>>>>>>> - In the apxs command, DON'T include the -L and -l arguments
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> After that, Apache appears to start ok, without segfault :)!!
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Thanks for all of the great help, esp. the suggestion about
> >>>>>>>>>>>>>>>> checking "ldconfig -p". I still don't understand why, but
> >>>>>>>>>>>>>>>> I'm just glad that I can get past this piece so now I can
> >>>>>>>>>>>>>>>> debug my module :)...
> >>>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>>> Later,
> >>>>>>>>>>>>>>>> Jim
> >>>>>>>>>>>>>>> I'm just glad this list is as good as it is!
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> FYI, the ldconfig is the dynamic linker control, and those
> >>>>>>>>>>>>>>> /etc/ld.so.conf.d files provide additional search
> >>>>>>>>>>>>>>> directories for the
> >>>>>>>>>>>>>>> linker to check in when loading a library.
> >>>>>>>>>>>>>>>
> >>>>>>>>>>>>>>> Joe
> >>>>>>>>>>>>>>> --
> >>>>>>>>>>>>>>> http://www.silverhawk.net/
> >>>>>>>>>>>>>> Hi,
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Sorry to report, but my earlier report was a "false positive"
> >>>>>>>>>>>>>> :)...
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I forgot that the mod_headers.c that I was doing the earlier
> >>>>>>>>>>>>>> testing with had all references to the libobaccess.so removed
> >>>>>>>>>>>>>> :(!!
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> So, I'm still stuck with basically the same problem, now,
> >>>>>>>>>>>>>> working with my "full" code, with the calls in it:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> - If I compile with -L and -l, Apache segfaults when it starts
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> - If I compile without -L and -l, then I get "undefined
> >>>>>>>>>>>>>> symbol" errors when I try to start Apache, e.g.:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> [root@apachemodule build-mod_headers]#
> >>>>>>>>>>>>>> /apps/httpd2222/bin/apachectl -k start -X
> >>>>>>>>>>>>>> httpd: Syntax error on line 84 of
> >>>>>>>>>>>>>> /apps/httpd2222/conf/httpd.conf: Cannot load
> >>>>>>>>>>>>>> /apps/httpd2222/modules/mod_headers.so into server:
> >>>>>>>>>>>>>> /apps/httpd2222/modules/mod_headers.so: undefined symbol:
> >>>>>>>>>>>>>> ObResource_isProtected
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> That "ObResource_isProtected" should be a symbol in
> >>>>>>>>>>>>>> libobaccess.so, and in fact, if I do "nm --dynamic", I get:
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> [root@apachemodule build-mod_headers]# nm --dynamic
> >>>>>>>>>>>>>> /apps/netpoint/lib64/libobaccess.so | grep
> >>>>>>>>>>>>>> "ObResource_isProtected"
> >>>>>>>>>>>>>> 00000000000a6d80 T ObResource_isProtected
> >>>>>>>>>>>>>> [root@apachemodule build-mod_headers]#
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> I'm *assuming* that the reason for the "undefined symbol"
> >>>>>>>>>>>>>> error is that libobaccess.so is actually not being loaded,
> >>>>>>>>>>>>>> but then when I try to load libobaccess.so, either via -L and
> >>>>>>>>>>>>>> -l in the apxs, or using LoadFile in httpd.conf, I get the
> >>>>>>>>>>>>>> segfault (same gdb info, BTW).
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Catch-22?
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Sorry for the false alarm :(!!
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>> Jim
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>>>
> >>>>>>>>>>>>> Not a catch-22. The -L and -l specify linker options when
> >>>>>>>>>>>>> assembling
> >>>>>>>>>>>>> the code. The ldconfig is a run-time thing. If you are
> >>>>>>>>>>>>> getting the
> >>>>>>>>>>>>> stderr messages, you are making it all the way into your
> >>>>>>>>>>>>> library. I'd
> >>>>>>>>>>>>> suggest commenting out the following line and see if you get
> >>>>>>>>>>>>> farther :
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> header_ssl_lookup = APR_RETRIEVE_OPTIONAL_FN(ssl_var_lookup);
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> That should tell you if the problem is the ssl_var_lookup.
> >>>>>>>>>>>>>
> >>>>>>>>>>>>> Joe
> >>>>>>>>>>>>> --
> >>>>>>>>>>>>> http://www.silverhawk.net/
> >>>>>>>>>>>> Hi,
> >>>>>>>>>>>>
> >>>>>>>>>>>> Thanks for the suggestion. I just tried what you suggested,
> >>>>>>>>>>>> and got a segfault when I started Apache with the modified
> >>>>>>>>>>>> module.
> >>>>>>>>>>>>
> >>>>>>>>>>>> Jim
> >>>>>>>>>>>>
> >>>>>>>>>>>>
> >>>>>>>>>>> Hi,
> >>>>>>>>>>>
> >>>>>>>>>>> As a reminder, here's the gdb with the library loaded:
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> (gdb) b header_post_config
> >>>>>>>>>>> Function "header_post_config" not defined.
> >>>>>>>>>>> Make breakpoint pending on future shared library load? (y or [n])
> >>>>>>>>>>> y
> >>>>>>>>>>> Breakpoint 1 (header_post_config) pending.
> >>>>>>>>>>> (gdb) run -d /apps/httpd2222/ -f /apps/httpd2222/conf/httpd.conf
> >>>>>>>>>>> Starting program: /apps/httpd2222/bin/httpd -d /apps/httpd2222/
> >>>>>>>>>>> -f /apps/httpd2222/conf/httpd.conf
> >>>>>>>>>>> [Thread debugging using libthread_db enabled]
> >>>>>>>>>>> [New Thread 182897610272 (LWP 11317)]
> >>>>>>>>>>> Breakpoint 2 at 0x2a9751ea90: file mod_headers.c, line 1121.
> >>>>>>>>>>> Pending breakpoint "header_post_config" resolved
> >>>>>>>>>>> mod_headers-jl V0.09 - start calling OAM API
> >>>>>>>>>>> In register_hooks
> >>>>>>>>>>> In create_headers_dir_config
> >>>>>>>>>>> In create_headers_dir_config
> >>>>>>>>>>> In header_cmd
> >>>>>>>>>>> In header_inout_cmd
> >>>>>>>>>>> In parse_format_tag
> >>>>>>>>>>> In parse_misc_string
> >>>>>>>>>>> In create_headers_dir_config
> >>>>>>>>>>> In header_cmd
> >>>>>>>>>>> In header_inout_cmd
> >>>>>>>>>>> In parse_format_tag
> >>>>>>>>>>> In parse_misc_string
> >>>>>>>>>>> [Switching to Thread 182897610272 (LWP 11317)]
> >>>>>>>>>>>
> >>>>>>>>>>> Breakpoint 2, header_post_config (pconf=0x573138, plog=0x5a52c8,
> >>>>>>>>>>> ptemp=0x5a72d8, s=0x59d3a8) at mod_headers.c:1121
> >>>>>>>>>>> 1121 printf("In header_post_config\n");
> >>>>>>>>>>> (gdb) n
> >>>>>>>>>>> 1120 {
> >>>>>>>>>>> (gdb) n
> >>>>>>>>>>> 1121 printf("In header_post_config\n");
> >>>>>>>>>>> (gdb) n
> >>>>>>>>>>> In header_post_config
> >>>>>>>>>>> 1122 header_ssl_lookup =
> >>>>>>>>>>> APR_RETRIEVE_OPTIONAL_FN(ssl_var_lookup);
> >>>>>>>>>>> (gdb) n
> >>>>>>>>>>> 1124 }
> >>>>>>>>>>> (gdb) n
> >>>>>>>>>>> 0x00000000004360c7 in ap_run_post_config (pconf=0x573138,
> >>>>>>>>>>> plog=0x5a52c8, ptemp=0x5a72d8, s=0x59d3a8) at config.c:91
> >>>>>>>>>>> 91 AP_IMPLEMENT_HOOK_RUN_ALL(int, post_config,
> >>>>>>>>>>> (gdb) n
> >>>>>>>>>>>
> >>>>>>>>>>> Program received signal SIGSEGV, Segmentation fault.
> >>>>>>>>>>> 0x0000003518d6c1e1 in BN_num_bits () from /lib64/libcrypto.so.4
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>>> So, it's actually blowing up in "BN_num_bits()" in
> >>>>>>>>>>> /lib64/libcrypto.so.4?
> >>>>>>>>>>>
> >>>>>>>>>>> Jim
> >>>>>>>>>> When you see the :
> >>>>>>>>>>
> >>>>>>>>>> Program received signal SIGSEGV, Segmentation fault.
> >>>>>>>>>> 0x0000003518d6c1e1 in BN_num_bits () from /lib64/libcrypto.so.4
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> in gdb, type "bt" and hit enter. It should show the back trace
> >>>>>>>>>> and how
> >>>>>>>>>> you got to the BN_num_bits() function.
> >>>>>>>>>>
> >>>>>>>>>> Joe
> >>>>>>>>>> --
> >>>>>>>>>> http://www.silverhawk.net/
> >>>>>>>>> Hi,
> >>>>>>>>>
> >>>>>>>>> Here's the bt full:
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> In header_inout_cmd
> >>>>>>>>> In parse_format_tag
> >>>>>>>>> In parse_misc_string
> >>>>>>>>> [Switching to Thread 182897610272 (LWP 6676)]
> >>>>>>>>>
> >>>>>>>>> Breakpoint 2, header_post_config (pconf=0x573138, plog=0x5a52c8,
> >>>>>>>>> ptemp=0x5a72d8, s=0x59d3a8) at mod_headers.c:1121
> >>>>>>>>> 1121 printf("In header_post_config\n");
> >>>>>>>>> (gdb) n
> >>>>>>>>> 1120 {
> >>>>>>>>> (gdb) n
> >>>>>>>>> 1121 printf("In header_post_config\n");
> >>>>>>>>> (gdb) n
> >>>>>>>>> In header_post_config
> >>>>>>>>> 1122 header_ssl_lookup =
> >>>>>>>>> APR_RETRIEVE_OPTIONAL_FN(ssl_var_lookup);
> >>>>>>>>> (gdb) n
> >>>>>>>>> 1124 }
> >>>>>>>>> (gdb) n
> >>>>>>>>> 0x00000000004360c7 in ap_run_post_config (pconf=0x573138,
> >>>>>>>>> plog=0x5a52c8, ptemp=0x5a72d8, s=0x59d3a8) at config.c:91
> >>>>>>>>> 91 AP_IMPLEMENT_HOOK_RUN_ALL(int, post_config,
> >>>>>>>>> (gdb) n
> >>>>>>>>>
> >>>>>>>>> Program received signal SIGSEGV, Segmentation fault.
> >>>>>>>>> 0x0000003518d6c1e1 in BN_num_bits () from /lib64/libcrypto.so.4
> >>>>>>>>> (gdb) bt full
> >>>>>>>>> #0 0x0000003518d6c1e1 in BN_num_bits () from /lib64/libcrypto.so.4
> >>>>>>>>> No symbol table info available.
> >>>>>>>>> #1 0x0000003518da8f4e in X509_ATTRIBUTE_create () from
> >>>>>>>>> /lib64/libcrypto.so.4
> >>>>>>>>> No symbol table info available.
> >>>>>>>>> #2 0x0000003518dadea2 in asn1_ex_i2c () from /lib64/libcrypto.so.4
> >>>>>>>>> No symbol table info available.
> >>>>>>>>> #3 0x0000003518dadf79 in asn1_ex_i2c () from /lib64/libcrypto.so.4
> >>>>>>>>> No symbol table info available.
> >>>>>>>>> #4 0x0000003518dae0e1 in ASN1_item_ex_i2d () from
> >>>>>>>>> /lib64/libcrypto.so.4
> >>>>>>>>> No symbol table info available.
> >>>>>>>>> #5 0x0000003518dae5f2 in ASN1_template_i2d () from
> >>>>>>>>> /lib64/libcrypto.so.4
> >>>>>>>>> No symbol table info available.
> >>>>>>>>> #6 0x0000003518dae28e in ASN1_item_ex_i2d () from
> >>>>>>>>> /lib64/libcrypto.so.4
> >>>>>>>>> No symbol table info available.
> >>>>>>>>> #7 0x0000003518dae3c6 in ASN1_item_i2d () from
> >>>>>>>>> /lib64/libcrypto.so.4
> >>>>>>>>> No symbol table info available.
> >>>>>>>>> #8 0x0000002a987d9d3a in ssl_pphrase_Handle (s=0x59d3a8,
> >>>>>>>>> p=0x5a72d8) at ssl_engine_pphrase.c:505
> >>>>>>>>> mc = (SSLModConfigRec *) 0x571738
> >>>>>>>>> sc = (SSLSrvConfigRec *) 0x668c38
> >>>>>>>>> pServ = (server_rec *) 0x65fa48
> >>>>>>>>> cpVHostID = 0x60add0 "www.example.com:443"
> >>>>>>>>> szPath =
> >>>>>>>>> "/apps/httpd2222/conf/certs/apache1.whatever.com.key\000\177\000\000\000Ü׿\177\000\000\000à׿\177",
> >>>>>>>>> '\0'<rep eats 15
> >>>>>>>>> times>,
> >>>>>>>>> "\001\000\000\000°å¿\177\000\000\000;\000\000\000+\000\000\000\020Û¿\177",
> >>>>>>>>> '\0'<repeats 35 times>, "à\224k",
> >>>>>>>>> ' ---Type<return>
> >>>>>>>>> to continue, or q<return> to quit---
> >>>>>>>>> \0'<repeats 13 times>,
> >>>>>>>>> "øé¿\177\000\000\000\020\000\000\000;\000\000\000\016\000\000\000\001\000\001\000\001",
> >>>>>>>>> '\0'<repeats 15 times>, "n\000\000\000\000\004\000\000°å¿\177",
> >>>>>>>>> '\0'<repeats 11 times>, "\020Û¿\177\000\000\000+\000\000\000;",
> >>>>>>>>> '\0'<repeats 35 times>...
> >>>>>>>>> pPrivateKey = (EVP_PKEY *) 0x6ba670
> >>>>>>>>> asn1 = Variable "asn1" is not available.
> >>>>>>>>> (gdb)
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> Not really sure what to make of that though :(...
> >>>>>>>>>
> >>>>>>>>> Jim
> >>>>>>>> Hi,
> >>>>>>>>
> >>>>>>>> Ok, something interesting. I noticed that the servername in the
> >>>>>>>> gdb output was example.whatever.com, so I changed ServerName in
> >>>>>>>> extras/httpd-ssl.conf to "apache1.whatever.com". Apache still
> >>>>>>>> segfaulted after that.
> >>>>>>>>
> >>>>>>>> I noticed that igdb was showing my router IP address for
> >>>>>>>> "mod_unique_id" (whatever that is), so I thought it was because I
> >>>>>>>> didn't have the hostname in /etc/hosts. So, I added
> >>>>>>>> "apache1.whatever.com" to /etc/hosts, and also turned Apache
> >>>>>>>> LogLevel to debug, and now I get a different error:
> >>>>>>>>
> >>>>>>>> [root@apachemodule dev]# /apps/httpd2222/bin/apachectl -k start -X
> >>>>>>>> mod_headers-jl V0.09 - start calling OAM API
> >>>>>>>> In register_hooks
> >>>>>>>> In create_headers_dir_config
> >>>>>>>> In create_headers_dir_config
> >>>>>>>> In header_cmd
> >>>>>>>> In header_inout_cmd
> >>>>>>>> In parse_format_tag
> >>>>>>>> In parse_misc_string
> >>>>>>>> In create_headers_dir_config
> >>>>>>>> In header_cmd
> >>>>>>>> In header_inout_cmd
> >>>>>>>> In parse_format_tag
> >>>>>>>> In parse_misc_string
> >>>>>>>> In header_post_config
> >>>>>>>> *** glibc detected *** corrupted double-linked list:
> >>>>>>>> 0x00000000006b9710 ***
> >>>>>>>> /apps/httpd2222/bin/apachectl: line 78: 7599
> >>>>>>>> Aborted $HTTPD $ARGV
> >>>>>>>> [root@apachemodule dev]#
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Here's the Apache error_log:
> >>>>>>>>
> >>>>>>>> [Thu Jun 21 23:24:11 2012] [info] mod_unique_id: using ip addr
> >>>>>>>> 72.215.225.9
> >>>>>>>> [Thu Jun 21 23:24:12 2012] [info] Init: Seeding PRNG with 144 bytes
> >>>>>>>> of entropy
> >>>>>>>> [Thu Jun 21 23:24:12 2012] [info] Loading certificate& private
> >>>>>>>> key of SSL-aware server
> >>>>>>>> [Thu Jun 21 23:29:51 2012] [info] mod_unique_id: using ip addr
> >>>>>>>> 72.215.225.9
> >>>>>>>> [Thu Jun 21 23:29:52 2012] [info] Init: Seeding PRNG with 144 bytes
> >>>>>>>> of entropy
> >>>>>>>> [Thu Jun 21 23:29:52 2012] [info] Loading certificate& private
> >>>>>>>> key of SSL-aware server
> >>>>>>>> [Thu Jun 21 23:29:56 2012] [info] mod_unique_id: using ip addr
> >>>>>>>> 72.215.225.9
> >>>>>>>> [Thu Jun 21 23:29:57 2012] [info] Init: Seeding PRNG with 144 bytes
> >>>>>>>> of entropy
> >>>>>>>> [Thu Jun 21 23:29:57 2012] [info] Loading certificate& private
> >>>>>>>> key of SSL-aware server
> >>>>>>>> [root@apachemodule dev]#
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> and here's the gdb backtrace:
> >>>>>>>>
> >>>>>>>> (gdb) b header_post_config
> >>>>>>>> Function "header_post_config" not defined.
> >>>>>>>> Make breakpoint pending on future shared library load? (y or [n]) y
> >>>>>>>> Breakpoint 1 (header_post_config) pending.
> >>>>>>>> (gdb) run -d /apps/httpd2222/ -f /apps/httpd2222/conf/httpd.conf
> >>>>>>>> Starting program: /apps/httpd2222/bin/httpd -d /apps/httpd2222/ -f
> >>>>>>>> /apps/httpd2222/conf/httpd.conf
> >>>>>>>> [Thread debugging using libthread_db enabled]
> >>>>>>>> [New Thread 182897610272 (LWP 7644)]
> >>>>>>>> Breakpoint 2 at 0x2a9751ea90: file mod_headers.c, line 1121.
> >>>>>>>> Pending breakpoint "header_post_config" resolved
> >>>>>>>> mod_headers-jl V0.09 - start calling OAM API
> >>>>>>>> In register_hooks
> >>>>>>>> In create_headers_dir_config
> >>>>>>>> In create_headers_dir_config
> >>>>>>>> In header_cmd
> >>>>>>>> In header_inout_cmd
> >>>>>>>> In parse_format_tag
> >>>>>>>> In parse_misc_string
> >>>>>>>> In create_headers_dir_config
> >>>>>>>> In header_cmd
> >>>>>>>> In header_inout_cmd
> >>>>>>>> In parse_format_tag
> >>>>>>>> In parse_misc_string
> >>>>>>>> [Switching to Thread 182897610272 (LWP 7644)]
> >>>>>>>>
> >>>>>>>> Breakpoint 2, header_post_config (pconf=0x573138, plog=0x5a52c8,
> >>>>>>>> ptemp=0x5a72d8, s=0x59d3a8) at mod_headers.c:1121
> >>>>>>>> 1121 printf("In header_post_config\n");
> >>>>>>>> (gdb) n
> >>>>>>>> 1120 {
> >>>>>>>> (gdb) n
> >>>>>>>> 1121 printf("In header_post_config\n");
> >>>>>>>> (gdb) n
> >>>>>>>> In header_post_config
> >>>>>>>> 1122 header_ssl_lookup =
> >>>>>>>> APR_RETRIEVE_OPTIONAL_FN(ssl_var_lookup);
> >>>>>>>> (gdb) n
> >>>>>>>> 1124 }
> >>>>>>>> (gdb) n
> >>>>>>>> 0x00000000004360c7 in ap_run_post_config (pconf=0x573138,
> >>>>>>>> plog=0x5a52c8, ptemp=0x5a72d8, s=0x59d3a8) at config.c:91
> >>>>>>>> 91 AP_IMPLEMENT_HOOK_RUN_ALL(int, post_config,
> >>>>>>>> (gdb) n
> >>>>>>>> *** glibc detected *** corrupted double-linked list:
> >>>>>>>> 0x00000000006b9710 ***
> >>>>>>>>
> >>>>>>>> Program received signal SIGABRT, Aborted.
> >>>>>>>> 0x000000351432e26d in raise () from /lib64/tls/libc.so.6
> >>>>>>>> (gdb) bt full
> >>>>>>>> #0 0x000000351432e26d in raise () from /lib64/tls/libc.so.6
> >>>>>>>> No symbol table info available.
> >>>>>>>> #1 0x000000351432fa6e in abort () from /lib64/tls/libc.so.6
> >>>>>>>> No symbol table info available.
> >>>>>>>> #2 0x0000003514363641 in __libc_message () from /lib64/tls/libc.so.6
> >>>>>>>> No symbol table info available.
> >>>>>>>> #3 0x0000003514369512 in _int_free () from /lib64/tls/libc.so.6
> >>>>>>>> No symbol table info available.
> >>>>>>>> #4 0x0000003514369846 in free () from /lib64/tls/libc.so.6
> >>>>>>>> No symbol table info available.
> >>>>>>>> #5 0x0000002a9790b6ba in R_free () from
> >>>>>>>> /apps/netpoint/lib64/libobaccess.so
> >>>>>>>> No symbol table info available.
> >>>>>>>> #6 0x0000002a9792bc41 in X509_CINF_free () from
> >>>>>>>> /apps/netpoint/lib64/libobaccess.so
> >>>>>>>> No symbol table info available.
> >>>>>>>> #7 0x0000002a9790ed98 in X509_free () from
> >>>>>>>> /apps/netpoint/lib64/libobaccess.so
> >>>>>>>> No symbol table info available.
> >>>>>>>> #8 0x0000002a987d97b6 in ssl_pphrase_Handle (s=0x59d3a8,
> >>>>>>>> p=0x5a72d8) at ssl_engine_pphrase.c:243
> >>>>>>>> mc = (SSLModConfigRec *) 0x571738
> >>>>>>>> sc = (SSLSrvConfigRec *) 0x668c38
> >>>>>>>> pServ = (server_rec *) 0x65fa48
> >>>>>>>> cpVHostID = 0x60add0 "apache1.whatever.com:443"
> >>>>>>>> szPath =
> >>>>>>>> "/apps/httpd2222/conf/certs/apache1.whatever.com.crt\000\177\000\000\000Ü׿\177\000\000\000à׿\177",
> >>>>>>>> '\0'<repeats 15 times>,
> >>>>>>>> "\001\000\000\000°å¿\177\000\000\000;\000\000\000+\000\000\000\020Û¿\177",
> >>>>>>>> '\0'<repeats 35 times>, "à\224k", '\0'<repeats 13 times>,
> >>>>>>>> "øé¿\177\000\000\000\020\000\000\000;\000\000\000\016\000\000\000\001\000\001\000\001",
> >>>>>>>> '\0'<repeats 15 times>, "n\000\000\000\000\004\000\000°å¿\177",
> >>>>>>>> '\0'<repeats 11 times>, "\020Û¿\177\000\000\000+\000\000\000;",
> >>>>>>>> '\0'<repeats 35 times>...
> >>>>>>>> pPrivateKey = Variable "pPrivateKey" is not available.
> >>>>>>>> (gdb)
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> Jim
> >>>>>>>>
> >>>>>>> I guess the thing that I'm puzzled about is why it goes from
> >>>>>>> ssl_pphrase_Handle (which is part of Apache code?) to X509_free ()
> >>>>>>> in the libobaccess.so? Apache shouldn't even be aware of the
> >>>>>>> libobaccess stuff yet (since I think that Apache is still trying to
> >>>>>>> initialize SSL processing)?
> >>>>>>>
> >>>>>>> Jim
> >>>>>> It shouldn't unless you've declared a shutdown hook or a clean up hook
> >>>>>> of sorts. The looks of that stack trace look strikingly like an
> >>>>>> exception handler firing off. What are all of the hooks you have
> >>>>>> running?
> >>>>>>
> >>>>>> Joe
> >>>>>> --
> >>>>>> http://www.silverhawk.net/
> >>>>>>
> >>>>>
> >>>>> Hi,
> >>>>>
> >>>>> I'm basically using the original mod_headers.c as a starter code, and
> >>>>> stuck some stuff in there. Haven't done anything with the hooks etc.
> >>>>> yet, so whatever was there is there:
> >>>>>
> >>>>> static void register_hooks(apr_pool_t *p)
> >>>>> {
> >>>>> printf("mod_headers-jl V0.09 - start calling OAM API\n");
> >>>>> printf("In register_hooks\n");
> >>>>> ap_register_output_filter("FIXUP_HEADERS_OUT",
> >>>>> ap_headers_output_filter,
> >>>>> NULL, AP_FTYPE_CONTENT_SET);
> >>>>> ap_register_output_filter("FIXUP_HEADERS_ERR",
> >>>>> ap_headers_error_filter,
> >>>>> NULL, AP_FTYPE_CONTENT_SET);
> >>>>> ap_hook_pre_config(header_pre_config,NULL,NULL,APR_HOOK_MIDDLE);
> >>>>> ap_hook_post_config(header_post_config,NULL,NULL,APR_HOOK_MIDDLE);
> >>>>> ap_hook_insert_filter(ap_headers_insert_output_filter, NULL,
> >>>>> NULL, APR_HOOK_LAST);
> >>>>> ap_hook_insert_error_filter(ap_headers_insert_error_filter,
> >>>>> NULL, NULL, APR_HOOK_LAST);
> >>>>> ap_hook_fixups(ap_headers_fixup, NULL, NULL, APR_HOOK_LAST);
> >>>>> ap_hook_post_read_request(ap_headers_early, NULL, NULL,
> >>>>> APR_HOOK_FIRST);
> >>>>> }
> >>>>>
> >>>>> module AP_MODULE_DECLARE_DATA headers_module =
> >>>>> {
> >>>>> STANDARD20_MODULE_STUFF,
> >>>>> create_headers_dir_config, /* dir config creater */
> >>>>> merge_headers_config, /* dir merger --- default is to
> >>>>> override */
> >>>>> NULL, /* server config */
> >>>>> NULL, /* merge server configs */
> >>>>> headers_cmds, /* command apr_table_t */
> >>>>> register_hooks /* register hooks */
> >>>>> };
> >>>>>
> >>>>> I guess that there is a:
> >>>>>
> >>>>> ap_hook_post_config(header_post_config,NULL,NULL,APR_HOOK_MIDDLE);
> >>>>>
> >>>>> in that register_hooks(), but it was there from the original code.
> >>>>>
> >>>>> BTW, I was checking, and it looks like there's an X509_free() function
> >>>>> in both openssl and in libobaccess.so (used 'nm --dynamic<file>), so
> >>>>> I'm wondering if things are confused, and maybe Apache code expects to
> >>>>> call the openssl X509_free, but somehow ends up calling the
> >>>>> X509_free() in libobaccess.so?
> >>>>>
> >>>>> What would cause something like that? If two .so files have the same
> >>>>> function, how to avoid conflicts?
> >>>>>
> >>>>> Jim
> >>>>
> >>>> Jim, you may have hit that on the head. I'm not sure of the load order
> >>>> and looking up symbols, but it could very well change things. Try
> >>>> re-ordering or adding a LoadFile before the SSL module and see if that
> >>>> changes things.
> >>>
> >>> Look up order usually is load order. So if mod_ssl loads first and loads
> >>> its dependency libcrypto, the symbol will always be found there even if
> >>> the implementation in libobaccess.so is needed - and vice versa.
> >>>
> >>> On Solaris there is a -Bdirect linker flag which changes runtime search
> >>> order so that each module would find a needed symbol first inits direct
> >>> dependencies. Unfortunately AFAIK notihing similar exists for Linux.
> >>>
> >>> You would need to find ssl libs and libobaccess with compatible symbols.
> >>> Best would be a libobaccess which is dynamically linked against a
> >>> compatible version of libcrypto instead of - as it seems - being
> >>> statically linked against it.
> >>>
> >>> Regards,
> >>>
> >>> Rainer
> >>>
> >>
> >>
> >> Hi Rainer,
> >>
> >> Here's exactly what I found:
> >>
> >> [root@apachemodule ~]# ldd /apps/netpoint/lib64/libobaccess.so
> >> libnsl.so.1 => /lib64/libnsl.so.1 (0x0000002a95ab8000)
> >> libdl.so.2 => /lib64/libdl.so.2 (0x0000002a95bd0000)
> >> libpthread.so.0 => /lib64/tls/libpthread.so.0 (0x0000002a95cd3000)
> >> libstdc++.so.6 => /lib64/libstdc++.so.6 (0x0000002a95de9000)
> >> libm.so.6 => /lib64/tls/libm.so.6 (0x0000002a95fd9000)
> >> libgcc_s.so.1 => /lib64/libgcc_s.so.1 (0x0000002a9615f000)
> >> libc.so.6 => /lib64/tls/libc.so.6 (0x0000002a9626d000)
> >> /lib64/ld-linux-x86-64.so.2 (0x000000552aaaa000)
> >> [root@apachemodule ~]#
> >> [root@apachemodule ~]# nm --dynamic /apps/netpoint/lib64/libobaccess.so |
> >> grep "X509_free"
> >> 00000000002d8d60 T X509_free
> >>
> >>
> >> [root@apachemodule ~]# ldd /lib64/libssl.so.0.9.7a
> >> libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2
> >> (0x0000003518700000)
> >> libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x0000003519000000)
> >> libcom_err.so.2 => /lib64/libcom_err.so.2 (0x0000003518500000)
> >> libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3
> >> (0x0000003518900000)
> >> libcrypto.so.4 => /lib64/libcrypto.so.4 (0x0000003518d00000)
> >> libdl.so.2 => /lib64/libdl.so.2 (0x0000003514800000)
> >> libz.so.1 => /usr/lib64/libz.so.1 (0x0000003514c00000)
> >> libc.so.6 => /lib64/tls/libc.so.6 (0x0000003514300000)
> >> libresolv.so.2 => /lib64/libresolv.so.2 (0x0000003518300000)
> >> /lib64/ld-linux-x86-64.so.2 (0x000000552aaaa000)
> >> [root@apachemodule ~]#
> >> [root@apachemodule ~]# nm --dynamic /lib64/libssl.so.0.9.7a | grep
> >> "X509_free"
> >> U X509_free
> >>
> >>
> >> [root@apachemodule ~]# ldd /lib64/libcrypto.so.0.9.7a
> >> libgssapi_krb5.so.2 => /usr/lib64/libgssapi_krb5.so.2
> >> (0x0000003518700000)
> >> libkrb5.so.3 => /usr/lib64/libkrb5.so.3 (0x0000003519000000)
> >> libcom_err.so.2 => /lib64/libcom_err.so.2 (0x0000003518500000)
> >> libk5crypto.so.3 => /usr/lib64/libk5crypto.so.3
> >> (0x0000003518900000)
> >> libdl.so.2 => /lib64/libdl.so.2 (0x0000003514800000)
> >> libz.so.1 => /usr/lib64/libz.so.1 (0x0000003514c00000)
> >> libc.so.6 => /lib64/tls/libc.so.6 (0x0000003514300000)
> >> libresolv.so.2 => /lib64/libresolv.so.2 (0x0000003518300000)
> >> /lib64/ld-linux-x86-64.so.2 (0x000000552aaaa000)
> >> [root@apachemodule ~]#
> >> [root@apachemodule ~]# nm --dynamic /lib64/libcrypto.so.0.9.7a | grep
> >> "X509_free"
> >> 0000003518da9860 T X509_free
> >>
> >>
> >> So:
> >>
> >> - liboaccess.so doesn't REFERENCE libcrypto, but it HAS an X509_free()
> >> function in it.
> >> - libssl.so.REFERENCES libcrypto, AND it HAS an X509_free function in it.
> >> - libcrypto also has an X509_free function in it.
> >>
> >> From your msg above, are you saying that this situation can't be resolved
> >> by trying to change load order?
> >>
> >> And the only way is what you said in your last paragraph (find another
> >> libobccess.so)?
>
> I don't know a way how to fix this apart from using mod_ssl and
> libobccess which have both bin build against the same major version of
> OpenSSL. In your case mod_ssl seems to use the platform OpenSSL, so
> 0.9.7(a) (which is pretty old), and we don't know what OpenSSL version
> was linked in statically into your libobccess.so.
>
> Load order most likely doesn't help. Either mod_header/liboccess.so will
> habe a problem or mod_ssl.
>
> > Also, BTW, the call to X509_free is apparently because in
> > ssl_engine_pphrase.c, which it seems is part of mod_ssl, it has:
> >
> > /*
> > * Free the X509 structure
> > */
> > X509_free(pX509Cert);
> >
> > but that's calling the X509_free() in libobaccess.so, rather than the
> > X509_free() in libcrypt.so?
>
> It's just calling X509_free(). The runtime linker will search through
> the httpd binary, all loaded modules and libraries in load order and
> call the symbol wherever it found it first. So you can change load order
> and load mod_ssl and OpenSSL libs before libobccess.so. But then when
> libobccess runs it will itself call symbols from within OpenSSL, e.g.
> X509_free() and again the runtime linker will search for the symbol
> through the loaded files *and not dircetly use the one inside libobccess
> - even if it is caled form inside this file*.
>
> Regards,
>
> Rainer
Rainer and Joe,
SUCCESS (though I'm not calling it that yet because of earlier false positive
:))!
Moving the LoadFile obaccess.so after didn't work because then got undefined
symbol.
So, I moved the LoadModule mod_ssl *in front of* the LoadModule libobaccess.so
(so I guess that makes the libobaccess.so "first"???).
Anyway, after that, Apache starts without aborting.
I've done some quick testing and client-authenticated SSL still seems to be
functioning (that was what I was afraid I'd break by moving the LoadModule for
mod_ssl).
Here's part of my httpd.conf, for the record:
LoadModule log_forensic_module modules/mod_log_forensic.so
LoadModule logio_module modules/mod_logio.so
LoadModule env_module modules/mod_env.so
LoadModule mime_magic_module modules/mod_mime_magic.so
LoadModule cern_meta_module modules/mod_cern_meta.so
LoadModule expires_module modules/mod_expires.so
#LoadFile /lib64/libcrypto.so.4
LoadModule ssl_module modules/mod_ssl.so
#LoadFile /lib64/libcrypto.so.4
#LoadFile /apps/netpoint/AccessServerSDK/oblix/lib/libxmlengine.so
LoadFile /apps/netpoint/lib64/libobaccess.so
LoadModule headers_module modules/mod_headers.so
#LoadFile /apps/netpoint/lib64/libobaccess.so
LoadModule ident_module modules/mod_ident.so
LoadModule usertrack_module modules/mod_usertrack.so
LoadModule unique_id_module modules/mod_unique_id.so
LoadModule setenvif_module modules/mod_setenvif.so
LoadModule version_module modules/mod_version.so
LoadModule proxy_module modules/mod_proxy.so
LoadModule proxy_connect_module modules/mod_proxy_connect.so
LoadModule proxy_ftp_module modules/mod_proxy_ftp.so
LoadModule proxy_http_module modules/mod_proxy_http.so
LoadModule proxy_scgi_module modules/mod_proxy_scgi.so
LoadModule proxy_ajp_module modules/mod_proxy_ajp.so
LoadModule proxy_balancer_module modules/mod_proxy_balancer.so
# ORIGINAL LOCATION OF LOADMODULE mod_ssl.so
#LoadFile /lib64/libcrypto.so.4
#LoadModule ssl_module modules/mod_ssl.so
#LoadFile /lib64/libcrypto.so.4
LoadModule mime_module modules/mod_mime.so
LoadModule dav_module modules/mod_dav.so
LoadModule status_module modules/mod_status.so
LoadModule autoindex_module modules/mod_autoindex.so
LoadModule asis_module modules/mod_asis.so
LoadModule info_module modules/mod_info.so
LoadModule cgi_module modules/mod_cgi.so
LoadModule dav_fs_module modules/mod_dav_fs.so
LoadModule vhost_alias_module modules/mod_vhost_alias.so
LoadModule negotiation_module modules/mod_negotiation.so
LoadModule dir_module modules/mod_dir.so
LoadModule imagemap_module modules/mod_imagemap.so
LoadModule actions_module modules/mod_actions.so
LoadModule speling_module modules/mod_speling.so
LoadModule userdir_module modules/mod_userdir.so
So:
- LoadModule for mod_ssl in front of LoadModule for libobaccess.so, then
- LoadFile libaccess.so, then
- LoadModule for mod_headers.so
(A preliminary) THANKS!!
Now I gotta debug my calls...
Jim
