Re: Upgrading to 12.1S 362003 - a few issues
>> I've placed the ktrace (sq3.kt.xz) and LD_DEBUG output (sq.deb.xz) at >> http://www.heuristicsystems.com/squid4K > No need, the problem occurs somewhere in user code. Try to build the > binary with debug symbols and look into the coredump with gdb. > >> >> Pity about ldd, but I appreciate the complexity & the effort. I'm also >> comforted that it isn't an error, rather the evolution of ldd :) > ldd should be fixed after https://reviews.freebsd.org/D25257 is committed. > ___ > [email protected] mailing list > https://lists.freebsd.org/mailman/listinfo/freebsd-stable > To unsubscribe, send any mail to "[email protected]" Thank-you Konstantin, I appreciate your advice. Though that will be difficult without a functioning pkg so I'll wait a while to see if others have a problem. It is however, very odd that a "fat" squid et al worked for months prior to the FreeBSD update (the production jail is very rarely changed ie years, except for FreeBSD updates). I'll see if there is something in the libc logs that shed some light. Kind regards, Dewayne. PS I noticed segmentation faults in bare metal i386 hardware a few months ago, but wasn't able to find the problem. Guess not many build i386 systems on amd64... ___ [email protected] mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[email protected]"
Re: Upgrading to 12.1S 362003 - a few issues
On Sat, Jun 13, 2020 at 12:34:46PM +1000, Dewayne Geraghty wrote:
> Hi Konstantin,
> I did try ktrace/kdump but kdump complained of "data too short". Using
> your suggestion about LD_DEBUG nearly caused me to fall off my chair. I
> think this is most relevant:
>
> # setenv LD_DEBUG 1
> # ktrace -f /tmp/sq3.kt /usr/local/sbin/squid start
> /libexec/ld-elf.so.1 is initialized, base address = 0x8aded000
> RTLD dynamic = 0x8ae0f6c0
> RTLD pltgot = 0
> initializing thread locks
> _rtld_thread_init: done
> processing main program's program header
> note osrel 1201517
> note fctl0 0
> note crt_no_init
> AT_EXECPATH 0xdfe0 /usr/bin/ktrace
> obj_main path /usr/bin/ktrace
> Filling in DT_DEBUG entry
> Ignoring d_tag 1879048186 = 0x6ffa
> /usr/bin/ktrace valid_hash_sysv 1 valid_hash_gnu 0 dynsymcount 30
> lm_init("(null)")
> loading LD_PRELOAD libraries
> loading needed objects
> Searching for "libc.so.7"
> search_library_pathfds('libc.so.7', '(null)', fdp)
> lm_find("/usr/bin/ktrace", "/lib")
> lmp_find("/usr/bin/ktrace")
> lmp_find("$DEFAULT$")
> Trying "/lib/libc.so.7"
> Opened "/lib/libc.so.7", fd 3
> loading "/lib/libc.so.7"
> Ignoring d_tag 1879048186 = 0x6ffa
> /lib/libc.so.7 valid_hash_sysv 1 valid_hash_gnu 1 dynsymcount 3126
> 0x6d20f000 .. 0x6d3eafff: /lib/libc.so.7
> checking for required versions
> initializing initial thread local storage offsets
> relocating "/usr/bin/ktrace"
> reloc_jmpslot: *0x4b177c04 = 0x6d39bac0
> reloc_jmpslot: *0x4b177c08 = 0x6d3b6480
> ... [ Lots of these ]
> reloc_jmpslot: *0x9d4859fc = 0x9d46e560
> reloc_jmpslot: *0x9d485a00 = 0x9d47a428
> reloc_jmpslot: *0x9d485a04 = 0x9d47a448
> relocating "/usr/local/lib/heimdal/libasn1.so.8"
> relocating "/usr/local/lib/heimdal/libwind.so.0"
> relocating "/usr/local/lib/heimdal/libheimbase.so.1"
> relocating "/usr/local/lib/heimdal/libhx509.so.5"
> relocating "/usr/local/lib/heimdal/libhcrypto.so.4"
> relocating "/usr/local/lib/heimdal/libheimsqlite.so.0"
> relocating "/usr/local/lib/heimdal/libcom_err.so.1"
> relocating "/usr/local/lib/heimdal/libroken.so.18"
> relocating "/usr/local/lib/libintl.so.8"
> relocating "/usr/local/lib/heimdal/libheimntlm.so.0"
> doing copy relocations
> initializing initial thread local storage
> initializing key program variables
> "__progname": *0x73be38b8 <-- 0xdd44
> "environ": *0x73cbbc08 <-- 0xfffc69cc
> "__elf_aux_vector": *0x9d4a6a5c <-- 0xfffc6a4c
> resolving ifuncs
> reloc_jmpslot: *0x9d484d88 = 0x9d352cc0
> reloc_jmpslot: *0x9d484d8c = 0x9d352c70
> reloc_jmpslot: *0x9cb16e04 = 0x9d352c70
> reloc_jmpslot: *0x9cb16e0c = 0x9d352cc0
> calling init function for /lib/libc.so.7 at 0x9d47c440
> calling init function for /lib/libc.so.7 at 0x9d40fc20
> calling init function for /lib/libc.so.7 at 0x9d3f0f40
> calling init function for /lib/libthr.so.3 at 0x9cb15210
> calling init function for /lib/libthr.so.3 at 0x9cb10cb0
> _rtld_thread_init: done
> calling init function for /lib/libgcc_s.so.1 at 0x9bfad540
> calling init function for /lib/libgcc_s.so.1 at 0x9bfa3de0
> calling init function for /lib/libcxxrt.so.1 at 0x9b698480
> calling init function for /usr/lib/libc++.so.1 at 0x9ab43d90
> calling init function for /usr/lib/libc++.so.1 at 0x9aae3e40
> calling init function for /lib/libcrypt.so.5 at 0x965d3210
> calling init function for /usr/local/lib/libcrypto.so.11 at 0x96e9f420
> calling init function for /usr/local/lib/heimdal/libroken.so.18 at
> 0xa32b6f60
> calling init function for /usr/local/lib/libintl.so.8 at 0xa23cb104
> calling init function for /usr/local/lib/heimdal/libheimbase.so.1 at
> 0x9ed15de0
> calling init function for /usr/local/lib/heimdal/libcom_err.so.1 at
> 0x9fec3768
> calling init function for /usr/local/lib/heimdal/libasn1.so.8 at 0x9e23f054
> calling init function for /usr/local/lib/heimdal/libhcrypto.so.4 at
> 0x9fb5641c
> calling init function for /usr/local/lib/heimdal/libheimsqlite.so.0 at
> 0xa1b9603c
> calling init function for /usr/local/lib/heimdal/libwind.so.0 at 0x9f77a990
> calling init function for /usr/local/lib/heimdal/libhx509.so.5 at 0xa0c4e3b4
> calling init function for /usr/local/lib/heimdal/libkrb5.so.26 at 0x99527188
> calling init function for /usr/local/lib/heimdal/libheimntlm.so.0 at
> 0xa498c814
> calling init function for /usr/local/lib/heimdal/libgssapi.so.3 at
> 0x9a45bfd0
> calling init function for /usr/local/lib/libpcre.so.1 at 0x97550ca8
> calling init function for /usr/local/lib/libpcreposix.so.0 at 0x9728e518
> calling init function for /lib/libm.so.5 at 0x97139f10
> calling init function for /usr/local/lib/libssl.so.11 at 0x98579078
> calling init function for /usr/lib/libregex.so.1 at 0x95e8f0a0
> calling init function for /usr/lib/librt.so.1 at 0x953ca240
> loading filtees
> enforcing main obj relro
> transferring control to program entry point = 0x73757360
So the app code started executing.
> "atexit" in "squid" ==> 0x9d457ac0 in "libc.so.7"
> reloc_jmpslot: *0x73be32b4 = 0x9d457ac0
> "_ZSt13set_terminatePFvvE" in
Re: Upgrading to 12.1S 362003 - a few issues
Hi Konstantin,
I did try ktrace/kdump but kdump complained of "data too short". Using
your suggestion about LD_DEBUG nearly caused me to fall off my chair. I
think this is most relevant:
# setenv LD_DEBUG 1
# ktrace -f /tmp/sq3.kt /usr/local/sbin/squid start
/libexec/ld-elf.so.1 is initialized, base address = 0x8aded000
RTLD dynamic = 0x8ae0f6c0
RTLD pltgot = 0
initializing thread locks
_rtld_thread_init: done
processing main program's program header
note osrel 1201517
note fctl0 0
note crt_no_init
AT_EXECPATH 0xdfe0 /usr/bin/ktrace
obj_main path /usr/bin/ktrace
Filling in DT_DEBUG entry
Ignoring d_tag 1879048186 = 0x6ffa
/usr/bin/ktrace valid_hash_sysv 1 valid_hash_gnu 0 dynsymcount 30
lm_init("(null)")
loading LD_PRELOAD libraries
loading needed objects
Searching for "libc.so.7"
search_library_pathfds('libc.so.7', '(null)', fdp)
lm_find("/usr/bin/ktrace", "/lib")
lmp_find("/usr/bin/ktrace")
lmp_find("$DEFAULT$")
Trying "/lib/libc.so.7"
Opened "/lib/libc.so.7", fd 3
loading "/lib/libc.so.7"
Ignoring d_tag 1879048186 = 0x6ffa
/lib/libc.so.7 valid_hash_sysv 1 valid_hash_gnu 1 dynsymcount 3126
0x6d20f000 .. 0x6d3eafff: /lib/libc.so.7
checking for required versions
initializing initial thread local storage offsets
relocating "/usr/bin/ktrace"
reloc_jmpslot: *0x4b177c04 = 0x6d39bac0
reloc_jmpslot: *0x4b177c08 = 0x6d3b6480
... [ Lots of these ]
reloc_jmpslot: *0x9d4859fc = 0x9d46e560
reloc_jmpslot: *0x9d485a00 = 0x9d47a428
reloc_jmpslot: *0x9d485a04 = 0x9d47a448
relocating "/usr/local/lib/heimdal/libasn1.so.8"
relocating "/usr/local/lib/heimdal/libwind.so.0"
relocating "/usr/local/lib/heimdal/libheimbase.so.1"
relocating "/usr/local/lib/heimdal/libhx509.so.5"
relocating "/usr/local/lib/heimdal/libhcrypto.so.4"
relocating "/usr/local/lib/heimdal/libheimsqlite.so.0"
relocating "/usr/local/lib/heimdal/libcom_err.so.1"
relocating "/usr/local/lib/heimdal/libroken.so.18"
relocating "/usr/local/lib/libintl.so.8"
relocating "/usr/local/lib/heimdal/libheimntlm.so.0"
doing copy relocations
initializing initial thread local storage
initializing key program variables
"__progname": *0x73be38b8 <-- 0xdd44
"environ": *0x73cbbc08 <-- 0xfffc69cc
"__elf_aux_vector": *0x9d4a6a5c <-- 0xfffc6a4c
resolving ifuncs
reloc_jmpslot: *0x9d484d88 = 0x9d352cc0
reloc_jmpslot: *0x9d484d8c = 0x9d352c70
reloc_jmpslot: *0x9cb16e04 = 0x9d352c70
reloc_jmpslot: *0x9cb16e0c = 0x9d352cc0
calling init function for /lib/libc.so.7 at 0x9d47c440
calling init function for /lib/libc.so.7 at 0x9d40fc20
calling init function for /lib/libc.so.7 at 0x9d3f0f40
calling init function for /lib/libthr.so.3 at 0x9cb15210
calling init function for /lib/libthr.so.3 at 0x9cb10cb0
_rtld_thread_init: done
calling init function for /lib/libgcc_s.so.1 at 0x9bfad540
calling init function for /lib/libgcc_s.so.1 at 0x9bfa3de0
calling init function for /lib/libcxxrt.so.1 at 0x9b698480
calling init function for /usr/lib/libc++.so.1 at 0x9ab43d90
calling init function for /usr/lib/libc++.so.1 at 0x9aae3e40
calling init function for /lib/libcrypt.so.5 at 0x965d3210
calling init function for /usr/local/lib/libcrypto.so.11 at 0x96e9f420
calling init function for /usr/local/lib/heimdal/libroken.so.18 at
0xa32b6f60
calling init function for /usr/local/lib/libintl.so.8 at 0xa23cb104
calling init function for /usr/local/lib/heimdal/libheimbase.so.1 at
0x9ed15de0
calling init function for /usr/local/lib/heimdal/libcom_err.so.1 at
0x9fec3768
calling init function for /usr/local/lib/heimdal/libasn1.so.8 at 0x9e23f054
calling init function for /usr/local/lib/heimdal/libhcrypto.so.4 at
0x9fb5641c
calling init function for /usr/local/lib/heimdal/libheimsqlite.so.0 at
0xa1b9603c
calling init function for /usr/local/lib/heimdal/libwind.so.0 at 0x9f77a990
calling init function for /usr/local/lib/heimdal/libhx509.so.5 at 0xa0c4e3b4
calling init function for /usr/local/lib/heimdal/libkrb5.so.26 at 0x99527188
calling init function for /usr/local/lib/heimdal/libheimntlm.so.0 at
0xa498c814
calling init function for /usr/local/lib/heimdal/libgssapi.so.3 at
0x9a45bfd0
calling init function for /usr/local/lib/libpcre.so.1 at 0x97550ca8
calling init function for /usr/local/lib/libpcreposix.so.0 at 0x9728e518
calling init function for /lib/libm.so.5 at 0x97139f10
calling init function for /usr/local/lib/libssl.so.11 at 0x98579078
calling init function for /usr/lib/libregex.so.1 at 0x95e8f0a0
calling init function for /usr/lib/librt.so.1 at 0x953ca240
loading filtees
enforcing main obj relro
transferring control to program entry point = 0x73757360
"atexit" in "squid" ==> 0x9d457ac0 in "libc.so.7"
reloc_jmpslot: *0x73be32b4 = 0x9d457ac0
"_ZSt13set_terminatePFvvE" in "squid" ==> 0x9b697a30 in "libcxxrt.so.1"
reloc_jmpslot: *0x73be3794 = 0x9b697a30
"strlen" in "squid" ==> 0x9d4750b0 in "libc.so.7"
reloc_jmpslot: *0x73be3108 = 0x9d4750b0
"malloc" in "squid" ==> 0x9d3e6620 in "libc.so.7"
reloc_jmpslot: *0x73be3820 = 0x9d3e6620
"memcpy" in "squid" ==> 0x9d47aa20 in "libc.so.7"
reloc_j
Re: Upgrading to 12.1S 362003 - a few issues
On Sat, Jun 13, 2020 at 10:24:49AM +1000, Dewayne Geraghty wrote: > After upgrading to 12.1Stable as of June 11: > 1) squid - fails with segmentation fault, ldd "Cannot load PIE binary" > 2) gcc9 - suffers a cc1 internal compiler error > 3) pkg-static - issues "failed" messages, unable to package or install > > Environment Xeon E3, ufs2 only, previously running FreeBSD 12.1 dated > 1st May (from kernel.old). > Prior to the upgrade all ports were rebuilt without issue, but NOT > installed as they were a fall-back, in the event that clang 10 caused > issues (the concern). There are multiple jails on this system, both > amd64 and i386 - some for building, testing and production use. One of > the production i386 jails runs squid, unchanged since Sept 2019. > > /etc/src.conf contains > WITH_PIE=YES > WITH_BIND_NOW=YES > > Most of our 1400+ ports are built and run with relro, now, pie and where > possible with noexecstack &/or no-common. These functioned in an ASLR > environment. (ASLR is only disabled during builds (gcc9 complains), or > when there's a problem, now). > > Note: NONE of the ports were rebuilt after the upgrade. However as part > of resolution, beep and squid were rebuilt. > > === Sequence of thigns === > > Upgrade performed. System rebooted without incident to > FreeBSD 12.1-STABLE #0 r362003M: Thu Jun 11 23:07:00 AEST 2020 i386 > hqdev-amd64-smp-vga 1201517 1201517 > but some port/application failures: > > Problem 1 > - > > i386 jail demonstrated: > > # /usr/local/etc/rc.d/squid start > Starting squid. > Segmentation fault Segmentation fault means that image was activated and kernel handed control to usermode. Try to debug it some, for instance use ktrace and LD_DEBUG=1 to see how much things progressed. > > # ldd /usr/local/sbin/squid > /usr/local/sbin/squid: > ldd: /usr/local/sbin/squid: Cannot load PIE binary /usr/local/sbin/squid > as DSO > /usr/local/sbin/squid: exit status 1 This is cosmetics, the problem is in ldd(1) mis-detecting PIE binary as DSO. Before some changes in rtld it was innocent, but since from recent times rtld refuses to dlopen(3) PIE binaries, method that ldd uses for DSO no longer works. It will take some time to fix ldd, because it needs to start parsing dynamic segment for DF_1_FLAGS. ___ [email protected] mailing list https://lists.freebsd.org/mailman/listinfo/freebsd-stable To unsubscribe, send any mail to "[email protected]"
