Bug#1027952: pcre2 causes crashes in link-grammar
On Sun, 22 Jan 2023 at 17:07:05 +, Simon McVittie wrote: > On Sat, 07 Jan 2023 at 22:07:31 +, Matthew Vernon wrote: > > I'm struggling a bit here; I wanted to try and bisect pcre2 upstream commits > > to see where this bug might have been introduced (or get to the bottom of > > what link-grammar's test is doing wrong, I see they've been troublesome in > > the past cf #975696). > > # apt install git > # git clone https://github.com/PCRE2Project/pcre2 > # cd /root/pcre2 > # git checkout pcre2-10.42 > # ./autogen.sh > # ./configure CFLAGS="-fsanitize=address -fno-omit-frame-pointer > -fsanitize=undefined" --prefix=/usr > # make > # ( cd /root/link-grammar-5.11.0~dfsg && debuild > -eASAN_OPTIONS=detect_leaks=0 -eDEB_BUILD_OPTIONS="noopt > sanitize=+address,+undefined" -eLD_PRELOAD=libasan.so.8 > -eLD_LIBRARY_PATH=/root/pcre2/.libs -us -uc -b ) > ... tests fail ... Bisection suggests that the first bad commit is d90fb238 "Refactor match_data() to always use the heap instead of having an initial frames vector on the stack; some consequential adjustmentsneeded.", which seems like a plausible place to have introduced a heap overflow. smcv
Bug#1027952: pcre2 causes crashes in link-grammar
On Sat, 07 Jan 2023 at 22:07:31 +, Matthew Vernon wrote: > I'm struggling a bit here; I wanted to try and bisect pcre2 upstream commits > to see where this bug might have been introduced (or get to the bottom of > what link-grammar's test is doing wrong, I see they've been troublesome in > the past cf #975696). I tried using AddressSanitizer to get more information, which might be helpful. I used a bookworm podman container for this, and for simplicity I'm using uid 0 in the container, but you could probably do the same in a Docker container, a VM or a chroot and as an unprivileged user. $ podman run -it --rm debian:bookworm-slim # apt install --no-install-recommends devscripts build-essential # echo "deb-src http://deb.debian.org/debian bookworm main" >> /etc/apt/sources.list # cd /root # apt update # apt source pcre2 link-grammar # echo "deb-src http://deb.debian.org/debian sid main" >> /etc/apt/sources.list # apt source pcre2 # apt build-dep pcre2 link-grammar # cd /root/pcre2-10.40 # debuild -eASAN_OPTIONS=detect_leaks=0 -eDEB_BUILD_OPTIONS="noopt sanitize=+address,+undefined" -us -uc -b # cd /root/pcre2-10.42 # debuild -eASAN_OPTIONS=detect_leaks=0 -eDEB_BUILD_OPTIONS="noopt sanitize=+address,+undefined" -us -uc -b # cd /root/link-grammar-5.11.0~dfsg # debuild -eASAN_OPTIONS=detect_leaks=0 -eDEB_BUILD_OPTIONS="noopt sanitize=+address,+undefined" -eLD_PRELOAD=libasan.so.8 -eLD_LIBRARY_PATH=/root/pcre2-10.40/debian/tmp/usr/lib/x86_64-linux-gnu -us -uc -b ... tests pass ... # debuild -eASAN_OPTIONS=detect_leaks=0 -eDEB_BUILD_OPTIONS="noopt sanitize=+address,+undefined" -eLD_PRELOAD=libasan.so.8 -eLD_LIBRARY_PATH=/root/pcre2-10.42/debian/tmp/usr/lib/x86_64-linux-gnu -us -uc -b ... tests fail ... (Note that the order of options to debuild is significant, debuild option -e must come before dpkg-buildpackage options like -us, -uc, -b.) With those steps, I get AddressSanitizer reporting a heap buffer overflow when the test tries to match a regular expression, and a similar error in the multi-java test. A roughly equivalent setup with the upstream libpcre also "works" (by which I mean, fails). AddressSanitizer output below. # apt install git # git clone https://github.com/PCRE2Project/pcre2 # cd /root/pcre2 # git checkout pcre2-10.42 # ./autogen.sh # ./configure CFLAGS="-fsanitize=address -fno-omit-frame-pointer -fsanitize=undefined" --prefix=/usr # make # ( cd /root/link-grammar-5.11.0~dfsg && debuild -eASAN_OPTIONS=detect_leaks=0 -eDEB_BUILD_OPTIONS="noopt sanitize=+address,+undefined" -eLD_PRELOAD=libasan.so.8 -eLD_LIBRARY_PATH=/root/pcre2/.libs -us -uc -b ) ... tests fail ... The same setup with pcre2-10.40 passes tests. The link-grammar build later fails when using the sanitizers, because it uses d-shlibs which is basically an anti-pattern (#902605), but that's enough to be able to bisect (in progress). smcv === link-grammar 5.11.0: tests/test-suite.log === # TOTAL: 5 # PASS: 3 # SKIP: 0 # XFAIL: 0 # FAIL: 2 # XPASS: 0 # ERROR: 0 .. contents:: :depth: 2 FAIL: multi-thread == = ==107299==ERROR: AddressSanitizer: heap-buffer-overflow on address 0x60e00708 at pc 0x7f8c46092bb4 bp 0x7f8c3cb87800 sp 0x7f8c3cb877f8 READ of size 1 at 0x60e00708 thread T10 #0 0x7f8c46092bb3 in match (/root/pcre2/.libs/libpcre2-8.so.0+0x492bb3) #1 0x7f8c460b4eac in pcre2_match_8 (/root/pcre2/.libs/libpcre2-8.so.0+0x4b4eac) #2 0x7f8c474930e3 in reg_match dict-common/regex-morph.c:219 #3 0x7f8c47494753 in match_regex dict-common/regex-morph.c:405 #4 0x7f8c475b2997 in regex_guess tokenize/tokenize.c:413 #5 0x7f8c475ca258 in separate_word tokenize/tokenize.c:2709 #6 0x7f8c475cdf73 in separate_sentence tokenize/tokenize.c:3116 #7 0x7f8c4745fc07 in sentence_split link-grammar/api.c:494 #8 0x55b9169165b8 in parse_one_sent tests/multi-thread.cc:34 #9 0x55b9169177e9 in parse_sents tests/multi-thread.cc:119 #10 0x55b9169209c2 in void std::__invoke_impl(std::__invoke_other, void (*&&)(Dictionary_s*, Parse_Options_s*, int, int), Dictionary_s*&&, Parse_Options_s*&&, int&&, int&&) /usr/include/c++/12/bits/invoke.h:61 #11 0x55b91692056f in std::__invoke_result::type std::__invoke(void (*&&)(Dictionary_s*, Parse_Options_s*, int, int), Dictionary_s*&&, Parse_Options_s*&&, int&&, int&&) /usr/include/c++/12/bits/invoke.h:96 #12 0x55b91691fffb in void std::thread::_Invoker >::_M_invoke<0ul, 1ul, 2ul, 3ul, 4ul>(std::_Index_tuple<0ul, 1ul, 2ul, 3ul, 4ul>) /usr/include/c++/12/bits/std_thread.h:252 #13 0x55b91691fc59 in std::thread::_Invoker >::operator()() /usr/include/c++/12/bits/std_thread.h:259 #14 0x55b91691fc11 in std::thread::_State_impl > >::_M_run() /usr/include/c++/12/bits/std_thread.h:210 #15 0x7f8c46ed44a2 (/lib/x86_64-linux-gnu/libstdc++.so.6+0xd44a
Bug#1027952: pcre2 causes crashes in link-grammar
Dear Maintainer, I tried to reproduce this segfault. Unfortunately this did not work as expected. The segfault in multi-thread did not show up. But I received one in multi-dict (in e.g. 3 of 6 runs). This was done in a up-to-date unstable amd64 VM, running at a AMD cpu, 16 threads given to the VM. The segfault did manifest in each case in libhunspell. This seems to be caused by the static variable "utf_tbl" possibly being shared between threads, and being freed and initialized while maybe used in another thread. Kind regards, Bernhard (gdb) bt #0 0x7f56b5b3159b in unicodetolower(unsigned short, int) () from /lib/x86_64-linux-gnu/libhunspell-1.7.so.0 #1 0x7f56b5b317ae in get_captype_utf8(std::vector > const&, int) () from /lib/x86_64-linux-gnu/libhunspell-1.7.so.0 #2 0x7f56b5b385bc in ?? () from /lib/x86_64-linux-gnu/libhunspell-1.7.so.0 ... (gdb) bt #0 0x7f56b5b3159b in unicodetolower (c=c@entry=116, langnum=langnum@entry=0) at ./src/hunspell/csutil.cxx:2482 #1 0x7f56b5b317ae in get_captype_utf8 (word=std::vector of length 6, capacity 64 = {...}, langnum=0) at ./src/hunspell/csutil.cxx:2538 #2 0x7f56b5b385bc in HashMgr::get_clen_and_captype (workbuf=std::vector of length 6, capacity 64 = {...}, captype=, word="tenant", this=0x7f568eb64d90) at ./src/hunspell/hashmgr.cxx:468 #3 HashMgr::get_clen_and_captype (workbuf=..., captype=, word=..., this=0x7f568eb64d90) at ./src/hunspell/hashmgr.cxx:464 #4 HashMgr::load_tables (this=0x7f568eb64d90, tpath=, key=) at ./src/hunspell/hashmgr.cxx:690 #5 0x7f56b5b38dfc in HashMgr::HashMgr (this=this@entry=0x7f568eb64d90, tpath=tpath@entry=0x7f56adffa800 "/usr/share/hunspell/en_US.dic", apath=apath@entry=0x7f56adffa400 "/usr/share/hunspell/en_US.aff", key=key@entry=0x0) at ./src/hunspell/hashmgr.cxx:101 #6 0x7f56b5b43616 in HunspellImpl::HunspellImpl (this=0x7f568fb8c850, affpath=0x7f56adffa400 "/usr/share/hunspell/en_US.aff", dpath=0x7f56adffa800 "/usr/share/hunspell/en_US.dic", key=0x0) at ./src/hunspell/hunspell.cxx:179 #7 0x7f56b5b43a27 in Hunspell_create (affpath=affpath@entry=0x7f56adffa400 "/usr/share/hunspell/en_US.aff", dpath=dpath@entry=0x7f56adffa800 "/usr/share/hunspell/en_US.dic") at ./src/hunspell/hunspell.cxx:2164 #8 0x7f56b60decc1 in spellcheck_create (lang=0x7f568d0e22c0 "en") at tokenize/spellcheck-hun.c:101 #9 0x7f56b60af6e5 in dictionary_six_str (lang=lang@entry=0x560b8f8ac37a "en", input=input@entry=0x7f568f9bc460 "\n %", '*' , "%\n %", ' ' , "%\n % Copyright (C) 1991-1998 Daniel "..., dict_name=dict_name@entry=0x7f568fa3e630 "en/4.0.dict", pp_name=pp_name@entry=0x7f568fa3e6f0 "en/4.0.knowledge", cons_name=cons_name@entry=0x7f568fa3e8e0 "en/4.0.constituent-knowledge", affix_name=affix_name@entry=0x7f568fa3e930 "en/4.0.affix", regex_name=0x7f568fa3e070 "en/4.0.regex") at dict-file/dictionary.c:159 #10 0x7f56b60afc28 in dictionary_six (regex_name=0x7f568fa3e070 "en/4.0.regex", affix_name=0x7f568fa3e930 "en/4.0.affix", cons_name=0x7f568fa3e8e0 "en/4.0.constituent-knowledge", pp_name=0x7f568fa3e6f0 "en/4.0.knowledge", dict_name=0x7f568fa3e630 "en/4.0.dict", lang=0x560b8f8ac37a "en") at dict-file/dictionary.c:281 #11 dictionary_create_from_file (lang=0x560b8f8ac37a "en") at dict-file/dictionary.c:307 #12 0x560b8f8ab598 in parse_one_sent (sent_str=0x560b8f8ac248 "The line extends 10 miles offshore.") at ./tests/multi-dict.cc:28 #13 parse_sents (thread_id=, niter=30) at ./tests/multi-dict.cc:82 #14 0x7f56b5ed44a3 in std::execute_native_thread_routine (__p=0x560b90a39a70) at ../../../../../src/libstdc++-v3/src/c++11/thread.cc:82 #15 0x7f56b5ca7fd4 in start_thread (arg=) at ./nptl/pthread_create.c:442 #16 0x7f56b5d2866c in clone3 () at ../sysdeps/unix/sysv/linux/x86_64/clone3.S:81 2482 return (utf_tbl) ? utf_tbl[c].clower : c; => 0x7f56b5b3159b <_Z14unicodetolowerti+27>:movzwl 0x4(%rdx,%rax,2),%eax (gdb) print/x $rdx $49 = 0x7f56ac614010 (gdb) print/x $rax $50 = 0x15c (gdb) x/1xg ($rdx + $rax * 2) + 0x4 0x7f56ac6142cc: Cannot access memory at address 0x7f56ac6142cc (gdb) print utf_tbl $51 = (unicode_info2 *) 0x0 # Unstable amd64 qemu VM 2023-01-20 apt update apt dist-upgrade apt install systemd-coredump mc gdb valgrind rr autopkgtest dpkg-dev libhunspell-1.7-0-dbgsym libstdc++6-dbgsym apt build-dep link-grammar autopkgtest --shell-fail link-grammar -- null PASS: mem-leak PASS: multi-java PASS: dict-reopen ../test-driver: Zeile 112: 12435 Speicherzugriffsfehler (Speicherabzug geschrieben) "$@" >> "$log_file" 2>&1 FAIL: multi-dict PASS: multi-thread === link-grammar 5.12.0: tests/test-suite.log === # TOTAL: 5 # PASS: 4 # SKIP: 0 # XFAIL: 0 # FAIL: 1 # XPASS: 0 # ERROR: 0 .. contents:: :depth: 2 FAIL: multi-dict [ 348.182545] multi-dict[12511]: segfault at 7f56ac6142cc ip 7f56b5b3159b sp 7f56adffa188 error 4 in libhun
Bug#1027952: pcre2 causes crashes in link-grammar
Hi, I'm struggling a bit here; I wanted to try and bisect pcre2 upstream commits to see where this bug might have been introduced (or get to the bottom of what link-grammar's test is doing wrong, I see they've been troublesome in the past cf #975696). I took unstable's link-grammar source, and built it thus: ./autogen.sh #the python test fails for some reason ./configure --disable-python-bindings --disable-java-bindings make make check I then cloned pcre2 upstream, checked out pcre2-10.42 ./autogen.sh ./configure --prefix=/home/matthew/junk/pcre2bug/install make make install Then I go back to my link-grammar tree and do cd tests LD_LIBRARY_PATH=/home/matthew/junk/pcre2bug/install/lib make check which works similarly LD_LIBRARY_PATH=/home/matthew/junk/pcre2bug/install/lib ./multi-thread works With a slightly horrible hack: lsof -p $(ps waux | grep multi-thread | grep pcre | cut -f 4 -d ' ') I can confirm the running multi-thread test is definitely running my locally-built (and thus 10.42) pcre2 library. ...so I'm not currently able to reproduce this in any way that might help me track down what if anything in pcre2 changed. And I'm not sure upstream would be very happy with "this test suite from another package now fails in Debian CI" as a bug report... Regards, Matthew
Bug#1027952: pcre2 causes crashes in link-grammar
On Thu, Jan 05, 2023 at 03:07:58AM +0200, Adrian Bunk wrote: >... > #0 0x7f46a39011b1 in ?? () from /lib/x86_64-linux-gnu/libpcre2-8.so.0 > #1 0x7f46a38ebb85 in ?? () from /lib/x86_64-linux-gnu/libpcre2-8.so.0 > #2 0x7f46a38fa10f in pcre2_match_8 () >from /lib/x86_64-linux-gnu/libpcre2-8.so.0 >... I should have double-checked that all relevant -dbgsym were installed, here is a better backtrace: #0 0x7f392ac891b1 in _pcre2_xclass_8 (c=1350565888, data=, utf=utf@entry=1) at src/pcre2_xclass.c:136 #1 0x7f392ac73b85 in match ( start_eptr=start_eptr@entry=0x7f38d8431aa0 "his", start_ecode=, top_bracket=, frame_size=frame_size@entry=128, match_data=match_data@entry=0x55fe226640d0, mb=mb@entry=0x7f39278df4e0) at src/pcre2_match.c:2171 #2 0x7f392ac8210f in pcre2_match_8 (code=, subject=subject@entry=0x7f38d8431aa0 "his", length=3, length@entry=18446744073709551615, start_offset=start_offset@entry=0, options=options@entry=1073741824, match_data=, mcontext=) at src/pcre2_match.c:7289 #3 0x7f392b134a86 in reg_match (rn=0x55fe22663e00, s=0x7f38d8431aa0 "his") at dict-common/regex-morph.c:219 #4 match_regex (rn=0x55fe22663e00, s=s@entry=0x7f38d8431aa0 "his") at dict-common/regex-morph.c:405 #5 0x7f392b16af63 in regex_guess (gword=0x7f38d8450160, word=, dict=0x55fe2265fa00) at tokenize/tokenize.c:413 #6 separate_word (sent=sent@entry=0x7f38d8430070, unsplit_word=unsplit_word@entry=0x7f38d8450160, opts=opts@entry=0x55fe252c60a0) at tokenize/tokenize.c:2709 #7 0x7f392b16c55a in separate_sentence (sent=sent@entry=0x7f38d8430070, opts=opts@entry=0x55fe252c60a0) at tokenize/tokenize.c:3116 #8 0x7f392b1261ec in sentence_split (sent=0x7f38d8430070, opts=0x55fe252c60a0) at ./link-grammar/api.c:494 #9 0x55fe20ead670 in parse_one_sent (dict=, opts=0x55fe252c60a0, sent_str=) at ./tests/multi-thread.cc:34 #10 0x55fe20ead9aa in parse_sents (dict=0x55fe2265fa00, opts=0x55fe252c60a0, thread_id=, niter=500) at ./tests/multi-thread.cc:119 #11 0x7f392af8c4a3 in std::execute_native_thread_routine ( __p=0x55fe252b7f30) at ../../../../../src/libstdc++-v3/src/c++11/thread.cc:82 #12 0x7f392ad3ffd4 in start_thread (arg=) at ./nptl/pthread_create.c:442 #13 0x7f392adbf8d0 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100 cu Adrian
Bug#1027952: pcre2 causes crashes in link-grammar
On Thu, Jan 05, 2023 at 09:14:48AM +, Matthew Vernon wrote: > Hi, > > On 05/01/2023 01:07, Adrian Bunk wrote: > > > https://tracker.debian.org/pkg/pcre2 > > Issues preventing migration: > > ∙ ∙ autopkgtest for link-grammar/5.11.0~dfsg-1: amd64: Pass, arm64: > > Regression ♻ (reference ♻), armel: Regression ♻ (reference ♻), armhf: > > Regression ♻ (reference ♻), i386: Pass, ppc64el: Regression ♻ (reference > > ♻), s390x: Pass > > Am I reading that right that it works on amd64, but fails on some other > architectures? No, it only says that it once succeeded: https://buildd.debian.org/status/logs.php?pkg=link-grammar&arch=amd64 https://ci.debian.net/packages/l/link-grammar/testing/amd64/ For the autopkgtest that's success on 4th attempt, after 3 failures. > > #0 0x7f46a39011b1 in ?? () from /lib/x86_64-linux-gnu/libpcre2-8.so.0 > > ...but it then fails in your local test on amd64? The name of the failuing test is "multi-thread", and wherever the problem is my guess would be a race condition. > [but link-grammar compiles OK with the previous pcre2 version?] That's what my local build tests say. And also what the link-grammar autopkgtest says. > Regards, > > Matthew cu Adrian
Bug#1027952: pcre2 causes crashes in link-grammar
Hi, On 05/01/2023 01:07, Adrian Bunk wrote: https://tracker.debian.org/pkg/pcre2 Issues preventing migration: ∙ ∙ autopkgtest for link-grammar/5.11.0~dfsg-1: amd64: Pass, arm64: Regression ♻ (reference ♻), armel: Regression ♻ (reference ♻), armhf: Regression ♻ (reference ♻), i386: Pass, ppc64el: Regression ♻ (reference ♻), s390x: Pass Am I reading that right that it works on amd64, but fails on some other architectures? #0 0x7f46a39011b1 in ?? () from /lib/x86_64-linux-gnu/libpcre2-8.so.0 ...but it then fails in your local test on amd64? [but link-grammar compiles OK with the previous pcre2 version?] Regards, Matthew
Bug#1027952: pcre2 causes crashes in link-grammar
Package: libpcre2-8-0 Version: 10.42-1 Severity: serious Tags: ftbfs Control: affects -1 src:link-grammar https://tracker.debian.org/pkg/pcre2 Issues preventing migration: ∙ ∙ autopkgtest for link-grammar/5.11.0~dfsg-1: amd64: Pass, arm64: Regression ♻ (reference ♻), armel: Regression ♻ (reference ♻), armhf: Regression ♻ (reference ♻), i386: Pass, ppc64el: Regression ♻ (reference ♻), s390x: Pass https://buildd.debian.org/status/logs.php?pkg=link-grammar&ver=5.11.0~dfsg-1%2Bb1 ... make check-TESTS make[4]: Entering directory '/<>/tests' make[5]: Entering directory '/<>/tests' PASS: mem-leak PASS: multi-java ../test-driver: line 112: 3264343 Segmentation fault "$@" >> "$log_file" 2>&1 FAIL: multi-thread PASS: multi-dict PASS: dict-reopen === link-grammar 5.11.0: tests/test-suite.log === # TOTAL: 5 # PASS: 4 # SKIP: 0 # XFAIL: 0 # FAIL: 1 # XPASS: 0 # ERROR: 0 .. contents:: :depth: 2 FAIL: multi-thread == FAIL multi-thread (exit status: 139) Testsuite summary for link-grammar 5.11.0 # TOTAL: 5 # PASS: 4 # SKIP: 0 # XFAIL: 0 # FAIL: 1 # XPASS: 0 # ERROR: 0 See tests/test-suite.log Please report to https://github.com/opencog/link-grammar make[5]: *** [Makefile:806: test-suite.log] Error 1 A backtrace from a coredump of a local build of link-grammar also suggests that pcre might be the problem: Core was generated by `/tmp/link-grammar-5.11.0~dfsg/tests/.libs/multi-thread'. Program terminated with signal SIGSEGV, Segmentation fault. #0 0x7f46a39011b1 in ?? () from /lib/x86_64-linux-gnu/libpcre2-8.so.0 [Current thread is 1 (Thread 0x7f469e5546c0 (LWP 17344))] (gdb) bt #0 0x7f46a39011b1 in ?? () from /lib/x86_64-linux-gnu/libpcre2-8.so.0 #1 0x7f46a38ebb85 in ?? () from /lib/x86_64-linux-gnu/libpcre2-8.so.0 #2 0x7f46a38fa10f in pcre2_match_8 () from /lib/x86_64-linux-gnu/libpcre2-8.so.0 #3 0x7f46a3da9a86 in reg_match (rn=0x562bf27ace00, s=0x7f467b5b2100 "Bill") at dict-common/regex-morph.c:219 #4 match_regex (rn=0x562bf27ace00, s=s@entry=0x7f467b5b2100 "Bill") at dict-common/regex-morph.c:405 #5 0x7f46a3ddff63 in regex_guess (gword=0x7f467b59b3b0, word=, dict=0x562bf27a8a00) at tokenize/tokenize.c:413 #6 separate_word (sent=sent@entry=0x7f467b5b59e0, unsplit_word=unsplit_word@entry=0x7f467b59b3b0, opts=opts@entry=0x562bf547ae60) at tokenize/tokenize.c:2709 #7 0x7f46a3de155a in separate_sentence (sent=sent@entry=0x7f467b5b59e0, opts=opts@entry=0x562bf547ae60) at tokenize/tokenize.c:3116 #8 0x7f46a3d9b1ec in sentence_split (sent=0x7f467b5b59e0, opts=0x562bf547ae60) at ./link-grammar/api.c:494 #9 0x562bf0419670 in parse_one_sent (dict=, opts=0x562bf547ae60, sent_str=) at ./tests/multi-thread.cc:34 #10 0x562bf04199aa in parse_sents (dict=0x562bf27a8a00, opts=0x562bf547ae60, thread_id=, niter=500) at ./tests/multi-thread.cc:119 #11 0x7f46a3c044a3 in ?? () from /lib/x86_64-linux-gnu/libstdc++.so.6 #12 0x7f46a39b7fd4 in start_thread (arg=) at ./nptl/pthread_create.c:442 #13 0x7f46a3a378d0 in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:100 (gdb)