Bug#1016811: libwebkit2gtk-4.0-37: bullseye backport crashes a lot on arm64
On Fri, Oct 28, 2022 at 08:02:19PM +0200, Loïc Yhuel wrote: > The crash in operationArithNegateProfiledOptimize is due to a bug in > clang 10/11. > On AArch64, the frame pointer might be set incorrectly depending on > which registers are saved on the stack. > This is fixed in > https://github.com/llvm/llvm-project/commit/e9d9a61208, which applies > easily on clang 11. Backports are going to start using gcc again (except for amd64, where gcc is broken) so hopefully this helps. Berto
Bug#1016811: libwebkit2gtk-4.0-37: bullseye backport crashes a lot on arm64
The crash in operationArithNegateProfiledOptimize is due to a bug in clang 10/11. On AArch64, the frame pointer might be set incorrectly depending on which registers are saved on the stack. This is fixed in https://github.com/llvm/llvm-project/commit/e9d9a61208, which applies easily on clang 11. WebKit uses __builtin_frame_address(1) in DECLARE_CALL_FRAME macro, which due to the bug reads a saved floating point register, instead of the frame pointer of the caller.
Bug#1016811: libwebkit2gtk-4.0-37: bullseye backport crashes a lot on arm64
On Mon, Oct 03, 2022 at 01:07:09PM +0900, Dominique MARTINET wrote: > Since gcc build appears to fix the issue I think that's the way to > go for bullseye, at least until we understand where that makes a > difference... gcc is the default compiler that we use for WebKit in Debian, the only reason why it wasn't being used in bullseye in this case it's because of this bug: https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=1008098 I narrowed this one down as much as possible, we should probably report it upstream but I didn't have the chance to do it yet. Berto
Bug#1016811: libwebkit2gtk-4.0-37: bullseye backport crashes a lot on arm64
Hi Alberto, Sebastian, Sebastian Krzyszkowiak wrote on Thu, Sep 29, 2022 at 06:15:56PM +0200: > the patch that sets setAllowsServerPreconnect does not fix the > original issue, however, this one does: > https://github.com/WebKit/WebKit/pull/4790 > > With this applied on top of your recent backport of 2.38.0, WebKit > becomes somewhat usable on arm64 in bullseye. It still crashes on some > pages (such as Twitter), but in a different way (somewhere in JSC) > which probably deserves a separate issue. Thank you for finding this other PR -- I can confirm that fixes the original issue which crashed almost immediately on any page, and also confirm there are leftover crashes with a clang build on this patch. Here's the backtrace I get: (gdb) bt #0 codeBlock () at ../Source/JavaScriptCore/interpreter/RegisterInlines.h:41 #1 codeBlock () at ../Source/JavaScriptCore/interpreter/CallFrameInlines.h:62 #2 operationArithNegateProfiledOptimize () at ../Source/JavaScriptCore/jit/JITOperations.cpp:3361 #3 0x74148358 in ?? () #4 0x2ed44080 in ?? () Backtrace stopped: previous frame inner to this frame (corrupt stack?) unfortunately it looks like gdb doesn't manage to unwind it properly (or indeed corrupted), so that might be difficult to debug. Since gcc build appears to fix the issue I think that's the way to go for bullseye, at least until we understand where that makes a difference... Sebastian Krzyszkowiak wrote on Fri, Sep 30, 2022 at 01:01:55PM +0200: > I can still see some backtraces showing up in logs due to preconnect > attempts, so applying PR 4790 will likely still be a good idea, but it > doesn't bring down the whole process anymore. I agree on principle; out of curiousity where did you see these logs? I do not see anything on stdout/stderr with the gcc build, but I would assume this to be logged elsewhere or perhaps only if some magic env var is set? Thanks, -- Dominique
Bug#1016811: libwebkit2gtk-4.0-37: bullseye backport crashes a lot on arm64
Control: tags -1 pending On Fri, Sep 30, 2022 at 01:01:55PM +0200, Sebastian Krzyszkowiak wrote: > > Can you check if it works for you? If it does then I'll switch > > back to gcc for the next upload. > > I have tested your unpatched-gcc build on my machines and it appears > perfectly usable - haven't managed to make it crash so far. Ok, then the next backport will use gcc instead of clang for arm64 (unfortunately gcc still crashes on amd64). Thanks! Berto
Bug#1016811: libwebkit2gtk-4.0-37: bullseye backport crashes a lot on arm64
On Fri, Sep 30, 2022 at 11:05 AM Alberto Garcia wrote: > Can you check if it works for you? If it does then I'll switch back to > gcc for the next upload. I have tested your unpatched-gcc build on my machines and it appears perfectly usable - haven't managed to make it crash so far. I can still see some backtraces showing up in logs due to preconnect attempts, so applying PR 4790 will likely still be a good idea, but it doesn't bring down the whole process anymore. Cheers, Sebastian
Bug#1016811: libwebkit2gtk-4.0-37: bullseye backport crashes a lot on arm64
On Fri, Sep 30, 2022 at 04:57:33PM +0900, Dominique MARTINET wrote: > The build has been quite stable with gcc for me, so while I trust > Sebastian the patch he pointed at fixes the immediate issue I'll try > to rebuild with it as well and start looking at the next crashes. So let me get this straight... if the packages are built with clang (that's how bullseye backports are currently being built) then it crashes, but with gcc it is stable, right? I made a new build using all the standard Debian tools but with gcc instead of clang, and this is the result: https://people.debian.org/~berto/files/webkitgtk-2.38.0-arm64/unpatched-gcc/ Can you check if it works for you? If it does then I'll switch back to gcc for the next upload. Berto
Bug#1016811: libwebkit2gtk-4.0-37: bullseye backport crashes a lot on arm64
Dominique MARTINET wrote on Fri, Sep 30, 2022 at 01:26:03PM +0900: > Alberto Garcia wrote on Thu, Sep 29, 2022 at 11:19:04AM +: > > Hmmm... ok, this may be a bug in the compiler. The official Debian > > builds are made using clang (because of bug #1008098), did you use > > gcc? > > Hm, that is not specified in the control or rule file? > > I've triggered a fresh build to check and can confirm it's using gcc... > Interesting that my build didn't fail on this (well, I guess ldd > segfault was just as bad :-D) > > I've had a quick look but wasn't able to figure how to build with clang > to check -- would building the package with clang in CC/CXX suffice, > e.g. `CC=clang CXX=clang++ dpkg-buildpackage` ? > (that appears to do the job, I'll send a new mail on Monday if build > didn't finish in time to test today) build finished, tested -- I can reproduce ! :) The build has been quite stable with gcc for me, so while I trust Sebastian the patch he pointed at fixes the immediate issue I'll try to rebuild with it as well and start looking at the next crashes. (It's a bit annoying to take 3+ hours to build though so I might have a stab at running from build dir again first though...) Cheers, -- Dominique
Bug#1016811: libwebkit2gtk-4.0-37: bullseye backport crashes a lot on arm64
Alberto Garcia wrote on Thu, Sep 29, 2022 at 11:19:04AM +: > On Wed, Sep 28, 2022 at 08:58:28AM +0900, Dominique MARTINET wrote: > > So I can confirm the same package as sid's 2.38.0-2 without any > > extra patch works just fine if I recompile it -- so it must be a > > difference in our build environment. > > Hmmm... ok, this may be a bug in the compiler. The official Debian > builds are made using clang (because of bug #1008098), did you use > gcc? Hm, that is not specified in the control or rule file? I've triggered a fresh build to check and can confirm it's using gcc... Interesting that my build didn't fail on this (well, I guess ldd segfault was just as bad :-D) I've had a quick look but wasn't able to figure how to build with clang to check -- would building the package with clang in CC/CXX suffice, e.g. `CC=clang CXX=clang++ dpkg-buildpackage` ? (that appears to do the job, I'll send a new mail on Monday if build didn't finish in time to test today) > > Happy to send you the list of packages I have installed or anything > > else that might be useful to track this down further! > > Yes, that would be helpful. Here's the full list (no more useful text after it): adduser 3.118 adwaita-icon-theme 3.38.0-1 apt 2.2.4 at-spi2-core2.38.0-4 autoconf2.69-14 automake1:1.16.3-2 autopoint 0.21-4 autotools-dev 20180224.1+nmu1 base-files 11.1+deb11u5 base-passwd 3.5.51 bash5.1-2+deb11u1 binutils2.35.2-2 binutils-aarch64-linux-gnu 2.35.2-2 binutils-common:arm64 2.35.2-2 bison 2:3.7.5+dfsg-1 bsdextrautils 2.36.1-8+deb11u1 bsdutils1:2.36.1-8+deb11u1 bubblewrap 0.4.1-3 build-essential 12.9 bzip2 1.0.8-4 ca-certificates 20210119 ccache 4.2-1 clang 1:11.0-51+nmu5 clang-111:11.0.1-2 cmake 3.18.4-2+deb11u1 cmake-data 3.18.4-2+deb11u1 coreutils 8.32-4 cpp 4:10.2.1-1 cpp-10 10.2.1-6 dash0.5.11+git20200708+dd9ef66-5 dbus1.12.20-2 dbus-user-session 1.12.20-2 dconf-gsettings-backend:arm64 0.38.0-2 dconf-service 0.38.0-2 debconf 1.5.77 debhelper 13.3.4 debian-archive-keyring 2021.1.1 debianutils 4.11.2 dh-autoreconf 20 dh-strip-nondeterminism 1.12.0-1 dictionaries-common 1.28.4 diffutils 1:3.7-5 dirmngr 2.2.27-2+deb11u2 dmsetup 2:1.02.175-2.1 docbook 4.5-6 docbook-to-man 1:2.0.0-45 docbook-xml 4.5-9 docbook-xsl 1.79.2+dfsg-1 dpkg1.20.12 dpkg-dev1.20.12 dwz 0.13+20210201-1 e2fsprogs 1.46.2-2 eatmydata 105-9 emacsen-common 3.0.4 fakeroot1.25.3-1.1 file1:5.39-3 findutils 4.8.0-1 flex2.6.4-8 fontconfig 2.13.1-4.2 fontconfig-config 2.13.1-4.2 fonts-cantarell 0.111-3 fonts-dejavu-core 2.37-2 g++ 4:10.2.1-1 g++-10 10.2.1-6 gcc 4:10.2.1-1 gcc-10 10.2.1-6 gcc-10-base:arm64 10.2.1-6 gcc-9-base:arm649.3.0-22 gdb 10.1-1.7 gettext 0.21-4 gettext-base0.21-4 gir1.2-atk-1.0:arm642.36.0-2 gir1.2-atspi-2.0:arm64 2.38.0-4 gir1.2-cloudproviders-0.3.0:arm64 0.3.0-3 gir1.2-colord-1.0:arm64 1.4.5-3 gir1.2-freedesktop:arm641.66.1-1+b1
Bug#1016811: libwebkit2gtk-4.0-37: bullseye backport crashes a lot on arm64
On Tue, Sep 20, 2022 at 5:44 PM Alberto Garcia wrote: > By the way 2.38.0 just came out, it'll probably be backported soon to > bullseye. Hi Alberto, the patch that sets setAllowsServerPreconnect does not fix the original issue, however, this one does: https://github.com/WebKit/WebKit/pull/4790 With this applied on top of your recent backport of 2.38.0, WebKit becomes somewhat usable on arm64 in bullseye. It still crashes on some pages (such as Twitter), but in a different way (somewhere in JSC) which probably deserves a separate issue. Cheers, Sebastian
Bug#1016811: libwebkit2gtk-4.0-37: bullseye backport crashes a lot on arm64
On Wed, Sep 28, 2022 at 08:58:28AM +0900, Dominique MARTINET wrote: > > I will rebuild the package again without the patch, to check if I can > > reproduce the issue without it in my build environment. > > > > That takes quite a while, so will probably only report back tomorrow. > > So I can confirm the same package as sid's 2.38.0-2 without any > extra patch works just fine if I recompile it -- so it must be a > difference in our build environment. Hmmm... ok, this may be a bug in the compiler. The official Debian builds are made using clang (because of bug #1008098), did you use gcc? > Happy to send you the list of packages I have installed or anything > else that might be useful to track this down further! Yes, that would be helpful. Berto
Bug#1016811: libwebkit2gtk-4.0-37: bullseye backport crashes a lot on arm64
Hello, Dominique MARTINET wrote on Tue, Sep 27, 2022 at 01:58:40PM +0900: > I will rebuild the package again without the patch, to check if I can > reproduce the issue without it in my build environment. > > That takes quite a while, so will probably only report back tomorrow. So I can confirm the same package as sid's 2.38.0-2 without any extra patch works just fine if I recompile it -- so it must be a difference in our build environment. Happy to send you the list of packages I have installed or anything else that might be useful to track this down further! (As a side note, it turns out that even when this does works, gdk opengl context creation fails on my system because it apparently always try to initialize the context as opengl first regardless of GLES setting so this isn't as good as I was hoping... In particular the fallback code rotates the rendered pixels by 180° for some reason so it's upside down, but removing that as well is perfect if just a bit slow. Hopefully I can figure that out next... Did I already say I hate closed drivers? But anyway that's unrelated to this bug: epiphany (and MiniBrowser) work just fine at this point; and I'll open a bug upstream directly for the fallback code) -- Dominique
Bug#1016811: libwebkit2gtk-4.0-37: bullseye backport crashes a lot on arm64
Dominique MARTINET wrote on Tue, Sep 27, 2022 at 10:10:40AM +0900: > (Or I guess I could actually build the deb package now I have ldd figured > out, but allowing partial builds would be great for actual development...) hmpf! So I have rebuilt as a package, and it .. just works? At this point I'm starting to think the build environment matters quite a bit, would you mind comparing with what I'm doing: - I do not have any beefy aarch64 machine, so building from a x86_64 host with qemu-user-static and binfmt - in order to get the sysroot I just run inside a container: podman run docker.io/arm64v8/debian:bullseye (using a volume for build data) - in that container I intalled all required dependencies with apt build-dep in build tree, I don't think there is anything else, so it should be similar to pdebuild - there is a bug with ldd and qemu user: I replace ldd with the following script --- #!/bin/sh lddtree "$@" 2>/dev/null \ | sed -e '/^[^ ]/d' -e 's/^ */\t/' \ -e 's/ld-linux-aarch64.so.1.*/ld-linux-aarch64.so.1/' \ -e 's/$/ (0x005501acc000)/' --- This is used by g-ir-scanner afaiu; build fails for me without it. - I used the upstream tarball https://webkitgtk.org/releases/webkitgtk-2.38.0.tar.xz - I used the debian directory from sid, with a few adjustments: http://deb.debian.org/debian/pool/main/w/webkit2gtk/webkit2gtk_2.38.0-2.debian.tar.xz diff -ur orig/debian/patches/series webkitgtk-2.38.0/debian/patches/series --- orig/debian/patches/series 2022-09-26 18:25:41.0 +0900 +++ webkitgtk-2.38.0/debian/patches/series 2022-09-27 10:27:48.811885275 +0900 @@ -5,3 +5,4 @@ dont-detect-sse2.patch reduce-memory-overheads.patch fix-ftbfs-hurd.patch +fix-bullseye.patch diff -ur orig/debian/rules webkitgtk-2.38.0/debian/rules --- orig/debian/rules 2022-09-26 18:25:41.0 +0900 +++ webkitgtk-2.38.0/debian/rules 2022-09-27 11:09:31.391250044 +0900 @@ -12,8 +12,8 @@ # soup3 build -> 4.1 API packages # gtk4 build -> 5.0 API packages ENABLE_SOUP2=YES -ENABLE_SOUP3=YES -ENABLE_GTK4=YES +ENABLE_SOUP3=NO +ENABLE_GTK4=NO # Set to YES to use the docs from the tarball instead of generating them. # Use only if gi-docgen is not available. USE_PREBUILT_DOCS=NO diff -ur orig/debian/control.in webkitgtk-2.38.0/debian/control.in --- orig/debian/control.in 2022-09-26 18:25:41.0 +0900 +++ webkitgtk-2.38.0/debian/control.in 2022-09-27 11:11:07.441672820 +0900 @@ -51,7 +51,7 @@ libegl1-mesa-dev, libgl-dev, libgles-dev -Build-Depends-Indep: gi-docgen, libglib2.0-doc, libgtk-3-doc, libsoup2.4-doc +Build-Depends-Indep: libglib2.0-doc, libgtk-3-doc, libsoup2.4-doc Standards-Version: 4.6.1.0 Rules-Requires-Root: no Vcs-Browser: https://salsa.debian.org/webkit-team/webkit - + fix-bullseye.patch as per link - --- a/Source/WebKit/UIProcess/API/glib/WebKitWebsiteDataManager.cpp +++ a/Source/WebKit/UIProcess/API/glib/WebKitWebsiteDataManager.cpp @@ -498,6 +498,9 @@ WebKit::WebsiteDataStore& webkitWebsiteDataManagerGetDataStore(WebKitWebsiteData configuration->setServiceWorkerRegistrationDirectory(FileSystem::stringFromFileSystemRepresentation(priv->swRegistrationsDirectory.get())); if (priv->domCacheDirectory) configuration->setCacheStorageDirectory(FileSystem::stringFromFileSystemRepresentation(priv->domCacheDirectory.get())); +#if !ENABLE(SERVER_PRECONNECT) +configuration->setAllowsServerPreconnect(false); +#endif priv->websiteDataStore = WebKit::WebsiteDataStore::create(WTFMove(configuration), PAL::SessionID::generatePersistentSessionID()); priv->websiteDataStore->setIgnoreTLSErrors(priv->tlsErrorsPolicy == WEBKIT_TLS_ERRORS_POLICY_IGNORE); } - - I built without documentation because no gi-docgen: DEB_BUILD_OPTIONS=nodoc dpkg-buildpackage With that package, epiphany works just fine. I will rebuild the package again without the patch, to check if I can reproduce the issue without it in my build environment. That takes quite a while, so will probably only report back tomorrow. Thanks, -- Dominique
Bug#1016811: libwebkit2gtk-4.0-37: bullseye backport crashes a lot on arm64
Alberto Garcia wrote on Mon, Sep 26, 2022 at 07:44:24AM +: > > I've tried the patched variant, it seems to fail in the same place? > > Ok, thanks. I uploaded the unpatched version, although I suspect it's > not going to make any difference. I agree it shouldn't make any difference : I tried anyway, and can confirm I hit the same trace. I've also finally finished building yesterday (the last failure I was referring to wasn't the badly escaped argument -- that was just a warning -- but ldd segfaulting on qemu-user aarch64 on the webkit so, presumably because it is too big? I've replaced it with a wrapper around lddtree and could finish building...), but unfortunately I run into a different crash before I can test this; I suspect this is due to my build options, I will try to get closer to the deb package options... (Or I guess I could actually build the deb package now I have ldd figured out, but allowing partial builds would be great for actual development...) I'll report again when/if I make progress, thanks for the follow up and packages :) -- Dominique
Bug#1016811: libwebkit2gtk-4.0-37: bullseye backport crashes a lot on arm64
On Mon, Sep 26, 2022 at 09:21:45AM +0900, Dominique MARTINET wrote: > Alberto Garcia wrote on Mon, Sep 26, 2022 at 12:00:04AM +: > > I just uploaded the packages built for bullseye, same URL, the patched > > ones are available already, the unpatched ones soon. > > Thanks! > > I've tried the patched variant, it seems to fail in the same place? Ok, thanks. I uploaded the unpatched version, although I suspect it's not going to make any difference. Berto
Bug#1016811: libwebkit2gtk-4.0-37: bullseye backport crashes a lot on arm64
Alberto Garcia wrote on Mon, Sep 26, 2022 at 12:00:04AM +: > I just uploaded the packages built for bullseye, same URL, the patched > ones are available already, the unpatched ones soon. Thanks! I've tried the patched variant, it seems to fail in the same place? Here's the new trace with adjusted lines: /usr/lib/aarch64-linux-gnu/webkit2gtk-4.0/WebKitNetworkProcess #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 #1 0x817b3aa0 in __GI_abort () at abort.c:79 #2 0x8427958c in WTFCrashWithInfo(int, char const*, char const*, int) () at WTF/Headers/wtf/Assertions.h:754 #3 0x8525db20 in captureStackTrace () at ../Source/WTF/wtf/StackTrace.cpp:79 #4 0x85232d18 in WTFReleaseLogStackTrace () at ../Source/WTF/wtf/Assertions.cpp:602 #5 0x884e8934 in internalError () at ../Source/WebCore/platform/network/ResourceErrorBase.cpp:97 #6 0x869e6c14 in preconnectTo () at ../Source/WebKit/NetworkProcess/NetworkConnectionToWebProcess.cpp:733 #7 0x8689baf8 in callMemberFunctionImpl >, WebKit::NetworkResourceLoadParameters&&), std::tuple >, WebKit::NetworkResourceLoadParameters>, 0, 1> () at ../Source/WebKit/Platform/IPC/HandleMessage.h:131 #8 callMemberFunction >, WebKit::NetworkResourceLoadParameters&&), std::tuple >, WebKit::NetworkResourceLoadParameters>, std::integer_sequence > () at ../Source/WebKit/Platform/IPC/HandleMessage.h:137 #9 handleMessage >, WebKit::NetworkResourceLoadParameters&&)> () at ../Source/WebKit/Platform/IPC/HandleMessage.h:259 #10 didReceiveNetworkConnectionToWebProcessMessage () at DerivedSources/WebKit/NetworkConnectionToWebProcessMessageReceiver.cpp:357 #11 0x86b3ff1c in dispatchMessage () at ../Source/WebKit/Platform/IPC/Connection.cpp:1150 #12 0x86b402b4 in dispatchOneIncomingMessage () at ../Source/WebKit/Platform/IPC/Connection.cpp:1219 #13 0x8525c4b0 in operator() () at ../Source/WTF/wtf/Function.h:82 #14 performWork () at ../Source/WTF/wtf/RunLoop.cpp:133 #15 0x852b5570 in operator() () at ../Source/WTF/wtf/glib/RunLoopGLib.cpp:80 #16 __invoke () at ../Source/WTF/wtf/glib/RunLoopGLib.cpp:79 #17 0x852b48f4 in operator() () at ../Source/WTF/wtf/glib/RunLoopGLib.cpp:53 #18 __invoke () at ../Source/WTF/wtf/glib/RunLoopGLib.cpp:45 #19 0x81d25ab4 in g_main_context_dispatch () from /usr/lib/aarch64-linux-gnu/libglib-2.0.so.0 #20 0x81d25e5c in ?? () from /usr/lib/aarch64-linux-gnu/libglib-2.0.so.0 #21 0x81d261b0 in g_main_loop_run () from /usr/lib/aarch64-linux-gnu/libglib-2.0.so.0 #22 0x852b4ef8 in run () at ../Source/WTF/wtf/glib/RunLoopGLib.cpp:108 #23 0x86b11004 in run () at ../Source/WebKit/Shared/AuxiliaryProcessMain.h:71 #24 AuxiliaryProcessMain () at ../Source/WebKit/Shared/AuxiliaryProcessMain.h:97 #25 0x817b3e18 in __libc_start_main (main=0x400878 <__wrap_main>, argc=3, argv=0xcfa17918, init=, fini=, rtld_fini=, stack_end=) at ../csu/libc-start.c:308 #26 0x00400874 in _start () /usr/lib/aarch64-linux-gnu/webkit2gtk-4.0/WebKitWebProcess #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 #1 0xa0afdaa0 in __GI_abort () at abort.c:79 #2 0xa35c358c in WTFCrashWithInfo(int, char const*, char const*, int) () at WTF/Headers/wtf/Assertions.h:754 #3 0xa45a7b20 in captureStackTrace () at ../Source/WTF/wtf/StackTrace.cpp:79 #4 0xa457cd18 in WTFReleaseLogStackTrace () at ../Source/WTF/wtf/Assertions.cpp:602 #5 0xa7832934 in internalError () at ../Source/WebCore/platform/network/ResourceErrorBase.cpp:97 #6 0xa61e6f6c in internallyFailedLoadTimerFired () at ../Source/WebKit/WebProcess/Network/WebLoaderStrategy.cpp:503 #7 0xa45ff61c in operator() () at ../Source/WTF/wtf/glib/RunLoopGLib.cpp:177 #8 __invoke () at ../Source/WTF/wtf/glib/RunLoopGLib.cpp:169 #9 0xa45fe8f4 in operator() () at ../Source/WTF/wtf/glib/RunLoopGLib.cpp:53 #10 __invoke () at ../Source/WTF/wtf/glib/RunLoopGLib.cpp:45 #11 0xa106fab4 in g_main_context_dispatch () from /usr/lib/aarch64-linux-gnu/libglib-2.0.so.0 #12 0xa106fe5c in ?? () from /usr/lib/aarch64-linux-gnu/libglib-2.0.so.0 #13 0xa10701b0 in g_main_loop_run () from /usr/lib/aarch64-linux-gnu/libglib-2.0.so.0 #14 0xa45feef8 in run () at ../Source/WTF/wtf/glib/RunLoopGLib.cpp:108 #15 0xa62a1aac in run () at ../Source/WebKit/Shared/AuxiliaryProcessMain.h:71 #16 AuxiliaryProcessMain () at ../Source/WebKit/Shared/AuxiliaryProcessMain.h:97 #17 0xa0afde18 in __libc_start_main (main=0x400878 <__wrap_main>, argc=3, argv=0xc73c1cc8, init=, fini=, rtld_fini=, stack_end=) at ../csu/libc-start.c:308 #18 0x00400874 in _start () Since the patched variant does not resolve the problem I'm not going to bother with the unpatched -- please ask if you'd like me to try anyway. Meanwhile
Bug#1016811: libwebkit2gtk-4.0-37: bullseye backport crashes a lot on arm64
On Sun, Sep 25, 2022 at 01:40:32PM +0900, Dominique MARTINET wrote: > > https://people.debian.org/~berto/files/webkitgtk-2.38.0-arm64/ > You were indeed faster than me... But it looks like these packages were > built for debian testing/bookworm (require libc6 >= 2.34), so I cannot > test easily due to proprietary libGL and other drivers woes. Argh, you're right, my fault!! I just uploaded the packages built for bullseye, same URL, the patched ones are available already, the unpatched ones soon. Berto
Bug#1016811: libwebkit2gtk-4.0-37: bullseye backport crashes a lot on arm64
Hi, Alberto Garcia wrote on Fri, Sep 23, 2022 at 12:38:58AM +0200: > I built two sets of packages, the unpatched vanilla version of > 2.38.0 and the one with the patch I mentioned in my previous comment: > > https://people.debian.org/~berto/files/webkitgtk-2.38.0-arm64/ > > Can you test them both and tell me if things get better? Thanks! You were indeed faster than me... But it looks like these packages were built for debian testing/bookworm (require libc6 >= 2.34), so I cannot test easily due to proprietary libGL and other drivers woes. (I actually do have some frankendebian version for testing, I'll give it a try on Monday, but this will not necessarily have the same problems as what I had reported here on stable/bullseye) I should get access to a beefier machine at work and will resume my building attempts there, and report with both tests (the packages you provided on bookworm and stable package with patch) hopefully by the end of Monday. -- Dominique
Bug#1016811: libwebkit2gtk-4.0-37: bullseye backport crashes a lot on arm64
On Thu, Sep 22, 2022 at 07:45:29AM +0900, Dominique MARTINET wrote: > If you have time to build a package you might be faster than me, so > I'd appreciate it :) Hi, I built two sets of packages, the unpatched vanilla version of 2.38.0 and the one with the patch I mentioned in my previous comment: https://people.debian.org/~berto/files/webkitgtk-2.38.0-arm64/ Can you test them both and tell me if things get better? Thanks! Berto
Bug#1016811: libwebkit2gtk-4.0-37: bullseye backport crashes a lot on arm64
Hi Alberto, Alberto Garcia wrote on Wed, Sep 21, 2022 at 12:54:22PM +0200: > On Fri, Sep 16, 2022 at 10:18:32AM +0900, Dominique MARTINET wrote: > > The traces are slightly different: > > > > /usr/lib/aarch64-linux-gnu/webkit2gtk-4.0/WebKitNetworkProcess > > (gdb) bt > > #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 > > #1 0x7cfdfaa0 in __GI_abort () at abort.c:79 > > #2 0x7fa8ac50 in WTFCrashWithInfo(int, char const*, char const*, > > int) () at WTF/Headers/wtf/Assertions.h:741 > > #3 0x80a2d5a8 in captureStackTrace () at > > ../Source/WTF/wtf/StackTrace.cpp:79 > > #4 0x80a08ea0 in WTFReleaseLogStackTrace () at > > ../Source/WTF/wtf/Assertions.cpp:592 > > #5 0x83c06550 in internalError () at > > ../Source/WebCore/platform/network/ResourceErrorBase.cpp:97 > > #6 0x820e8d1c in preconnectTo () at > > ../Source/WebKit/NetworkProcess/NetworkConnectionToWebProcess.cpp:735 > > do you think you can try this patch? > > > https://bug-244026-attachments.webkit.org/attachment.cgi?id=462492=diff=raw=1 Thanks! > If you have problems / are unsure about how to build WebKit I can > provide packages for you. Just let me know. I'm definitely unsure about how to build WebKit, but that can be fixed... I might need to find somewhere with a bit more memory though, been trying to build the 2.38 release yesterday and build got OOM killed even with -j1 on a VM with 6GB of ram :/ If you have time to build a package you might be faster than me, so I'd appreciate it :) -- Dominique
Bug#1016811: libwebkit2gtk-4.0-37: bullseye backport crashes a lot on arm64
On Fri, Sep 16, 2022 at 10:18:32AM +0900, Dominique MARTINET wrote: > The traces are slightly different: > > /usr/lib/aarch64-linux-gnu/webkit2gtk-4.0/WebKitNetworkProcess > (gdb) bt > #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 > #1 0x7cfdfaa0 in __GI_abort () at abort.c:79 > #2 0x7fa8ac50 in WTFCrashWithInfo(int, char const*, char const*, > int) () at WTF/Headers/wtf/Assertions.h:741 > #3 0x80a2d5a8 in captureStackTrace () at > ../Source/WTF/wtf/StackTrace.cpp:79 > #4 0x80a08ea0 in WTFReleaseLogStackTrace () at > ../Source/WTF/wtf/Assertions.cpp:592 > #5 0x83c06550 in internalError () at > ../Source/WebCore/platform/network/ResourceErrorBase.cpp:97 > #6 0x820e8d1c in preconnectTo () at > ../Source/WebKit/NetworkProcess/NetworkConnectionToWebProcess.cpp:735 Hello Dominique, do you think you can try this patch? https://bug-244026-attachments.webkit.org/attachment.cgi?id=462492=diff=raw=1 If you have problems / are unsure about how to build WebKit I can provide packages for you. Just let me know. Berto
Bug#1016811: libwebkit2gtk-4.0-37: bullseye backport crashes a lot on arm64
On Fri, Sep 16, 2022 at 10:18:32AM +0900, Dominique MARTINET wrote: > Alberto Garcia wrote on Wed, Aug 17, 2022 at 10:31:22AM +: > > Thanks, I just forwarded the bug upstream, I'll try to reproduce it > > myself this or next week. > > I've also been observing similar crashes on aarch64 bullseye package > and using bookworm is not an option for me (thanks to proprietary > drivers...) > > > The traces are slightly different: Sorry for the late reply, and thanks! I think the traces are useful, I added them to the upstream bug report and I'll try to discuss them with the development team. By the way 2.38.0 just came out, it'll probably be backported soon to bullseye. Berto
Bug#1016811: libwebkit2gtk-4.0-37: bullseye backport crashes a lot on arm64
Hi, Alberto Garcia wrote on Wed, Aug 17, 2022 at 10:31:22AM +: > Thanks, I just forwarded the bug upstream, I'll try to reproduce it > myself this or next week. I've also been observing similar crashes on aarch64 bullseye package and using bookworm is not an option for me (thanks to proprietary drivers...) The traces are slightly different: /usr/lib/aarch64-linux-gnu/webkit2gtk-4.0/WebKitNetworkProcess (gdb) bt #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 #1 0x7cfdfaa0 in __GI_abort () at abort.c:79 #2 0x7fa8ac50 in WTFCrashWithInfo(int, char const*, char const*, int) () at WTF/Headers/wtf/Assertions.h:741 #3 0x80a2d5a8 in captureStackTrace () at ../Source/WTF/wtf/StackTrace.cpp:79 #4 0x80a08ea0 in WTFReleaseLogStackTrace () at ../Source/WTF/wtf/Assertions.cpp:592 #5 0x83c06550 in internalError () at ../Source/WebCore/platform/network/ResourceErrorBase.cpp:97 #6 0x820e8d1c in preconnectTo () at ../Source/WebKit/NetworkProcess/NetworkConnectionToWebProcess.cpp:735 #7 0x81fc62f4 in callMemberFunctionImpl >, WebKit::NetworkResourceLoadParameters&&), std::tuple >, WebKit::NetworkResourceLoadParameters>, 0, 1> () at ../Source/WebKit/Platform/IPC/HandleMessage.h:125 #8 callMemberFunction >, WebKit::NetworkResourceLoadParameters&&), std::tuple >, WebKit::NetworkResourceLoadParameters>, std::integer_sequence > () at ../Source/WebKit/Platform/IPC/HandleMessage.h:131 #9 handleMessage >, WebKit::NetworkResourceLoadParameters&&)> () at ../Source/WebKit/Platform/IPC/HandleMessage.h:196 #10 didReceiveNetworkConnectionToWebProcessMessage () at DerivedSources/WebKit/NetworkConnectionToWebProcessMessageReceiver.cpp:479 #11 0x822543d0 in dispatchMessage () at ../Source/WebKit/Platform/IPC/Connection.cpp:1134 #12 0x82254768 in dispatchOneIncomingMessage () at ../Source/WebKit/Platform/IPC/Connection.cpp:1203 #13 0x80a2bf40 in operator() () at ../Source/WTF/wtf/Function.h:82 #14 performWork () at ../Source/WTF/wtf/RunLoop.cpp:133 #15 0x80a85190 in operator() () at ../Source/WTF/wtf/glib/RunLoopGLib.cpp:80 #16 __invoke () at ../Source/WTF/wtf/glib/RunLoopGLib.cpp:79 #17 0x80a84524 in operator() () at ../Source/WTF/wtf/glib/RunLoopGLib.cpp:53 #18 __invoke () at ../Source/WTF/wtf/glib/RunLoopGLib.cpp:45 #19 0x7d551ab4 in g_main_context_dispatch () from /usr/lib/aarch64-linux-gnu/libglib-2.0.so.0 #20 0x7d551e5c in ?? () from /usr/lib/aarch64-linux-gnu/libglib-2.0.so.0 #21 0x7d5521b0 in g_main_loop_run () from /usr/lib/aarch64-linux-gnu/libglib-2.0.so.0 #22 0x80a84b20 in run () at ../Source/WTF/wtf/glib/RunLoopGLib.cpp:108 #23 0x822280d8 in run () at ../Source/WebKit/Shared/AuxiliaryProcessMain.h:70 #24 AuxiliaryProcessMain () at ../Source/WebKit/Shared/AuxiliaryProcessMain.h:96 #25 0x7cfdfe18 in __libc_start_main (main=0x400878 <__wrap_main>, argc=3, argv=0xf1c90058, init=, fini=, rtld_fini=, stack_end=) at ../csu/libc-start.c:308 #26 0x00400874 in _start () /usr/lib/aarch64-linux-gnu/webkit2gtk-4.0/WebKitWebProcess (gdb) bt #0 __GI_raise (sig=sig@entry=6) at ../sysdeps/unix/sysv/linux/raise.c:50 #1 0x99831aa0 in __GI_abort () at abort.c:79 #2 0x9c2dcc50 in WTFCrashWithInfo(int, char const*, char const*, int) () at WTF/Headers/wtf/Assertions.h:741 #3 0x9d27f5a8 in captureStackTrace () at ../Source/WTF/wtf/StackTrace.cpp:79 #4 0x9d25aea0 in WTFReleaseLogStackTrace () at ../Source/WTF/wtf/Assertions.cpp:592 #5 0xa0458550 in internalError () at ../Source/WebCore/platform/network/ResourceErrorBase.cpp:97 #6 0x9edead30 in internallyFailedLoadTimerFired () at ../Source/WebKit/WebProcess/Network/WebLoaderStrategy.cpp:495 #7 0x9d2d723c in operator() () at ../Source/WTF/wtf/glib/RunLoopGLib.cpp:177 #8 __invoke () at ../Source/WTF/wtf/glib/RunLoopGLib.cpp:169 #9 0x9d2d6524 in operator() () at ../Source/WTF/wtf/glib/RunLoopGLib.cpp:53 #10 __invoke () at ../Source/WTF/wtf/glib/RunLoopGLib.cpp:45 #11 0x99da3ab4 in g_main_context_dispatch () from /usr/lib/aarch64-linux-gnu/libglib-2.0.so.0 #12 0x99da3e5c in ?? () from /usr/lib/aarch64-linux-gnu/libglib-2.0.so.0 #13 0x99da41b0 in g_main_loop_run () from /usr/lib/aarch64-linux-gnu/libglib-2.0.so.0 #14 0x9d2d6b20 in run () at ../Source/WTF/wtf/glib/RunLoopGLib.cpp:108 #15 0x9eea47c4 in run () at ../Source/WebKit/Shared/AuxiliaryProcessMain.h:70 #16 AuxiliaryProcessMain () at ../Source/WebKit/Shared/AuxiliaryProcessMain.h:96 #17 0x99831e18 in __libc_start_main (main=0x400878 <__wrap_main>, argc=3, argv=0xf7b85168, init=, fini=, rtld_fini=, stack_end=) at ../csu/libc-start.c:308 #18 0x00400874 in _start () hopefully the root of the problem is the same and having debug symbols installed will help pinpoint
Bug#1016811: libwebkit2gtk-4.0-37: bullseye backport crashes a lot on arm64
Control: forwarded -1 https://bugs.webkit.org/show_bug.cgi?id=244026 On Wed, Aug 17, 2022 at 01:01:46AM +0200, Sebastian Krzyszkowiak wrote: > > I'm going to publish 2.36.6-1~deb11u1 this week. It contains a number > > of crash fixes and I would appreciate if you could verify whether the > > problem still happens with it. > > > I've just tried 2.36.6-1~deb11u1 and it still crashes the same way. > > Note that these crashes seem exclusive to backports; as far as I'm > aware they don't happen on bookworm at all. Thanks, I just forwarded the bug upstream, I'll try to reproduce it myself this or next week. Berto
Bug#1016811: libwebkit2gtk-4.0-37: bullseye backport crashes a lot on arm64
On 8/15/22, Alberto Garcia wrote: > On Sun, Aug 07, 2022 at 10:24:45PM +0200, Sebastian Krzyszkowiak wrote: > >> since 2.36.0-3~deb11u1, webkit2gtk in bullseye has been almost >> unusable on arm64 computers, crashing a lot when navigating between >> pages. > > Hello, > > I'm going to publish 2.36.6-1~deb11u1 this week. It contains a number > of crash fixes and I would appreciate if you could verify whether the > problem still happens with it. > > Thanks, > > Berto Hi Berto, I've just tried 2.36.6-1~deb11u1 and it still crashes the same way. Note that these crashes seem exclusive to backports; as far as I'm aware they don't happen on bookworm at all. Cheers, Sebastian
Bug#1016811: libwebkit2gtk-4.0-37: bullseye backport crashes a lot on arm64
On Sun, Aug 07, 2022 at 10:24:45PM +0200, Sebastian Krzyszkowiak wrote: > since 2.36.0-3~deb11u1, webkit2gtk in bullseye has been almost > unusable on arm64 computers, crashing a lot when navigating between > pages. Hello, I'm going to publish 2.36.6-1~deb11u1 this week. It contains a number of crash fixes and I would appreciate if you could verify whether the problem still happens with it. Thanks, Berto
Bug#1016811: libwebkit2gtk-4.0-37: bullseye backport crashes a lot on arm64
For the record, I've just realized that the system I reported it from was lacking bullseye-security updates, but I've checked it now on a fully up-to-date arm64 system with webkit2gtk 2.36.4-1~deb11u1 and it's still crashing in the same way.
Bug#1016811: libwebkit2gtk-4.0-37: bullseye backport crashes a lot on arm64
Package: libwebkit2gtk-4.0-37 Version: 2.36.3-1~deb11u1 Severity: important X-Debbugs-Cc: d...@dosowisko.net, t...@security.debian.org Dear Maintainer, since 2.36.0-3~deb11u1, webkit2gtk in bullseye has been almost unusable on arm64 computers, crashing a lot when navigating between pages. An easy way to reproduce is to navigate to https://news.ycombinator.com/ and click some link (for example, "new" or "login"); or navigate to https://mobile.twitter.com/ which makes it crash right away. Crashes are reproducible in both GNOME Web and in MiniBrowser shipped by webkit2gtk package. This has been seen on various machines and in downstream distros like PureOS as well (https://source.puri.sm/Librem5/debs/epiphany/-/issues/38) Example backtrace: [] Process 14397 (WebKitNetworkPr) of user 1000 dumped core. Stack trace of thread 14397: #0 0x914f92ac __GI_raise (libc.so.6 + 0x372ac) #1 0x914e5ea0 __GI_abort (libc.so.6 + 0x23ea0) #2 0x93f89c50 _Z16WTFCrashWithInfoiPKcS0_i (libjavascriptcoregtk-4.0.so.18 + 0x198c50) #3 0x94f2a138 _ZN3WTF10StackTrace17captureStackTraceEii (libjavascriptcoregtk-4.0.so.18 + 0x1139138) #4 0x94f05a30 WTFReleaseLogStackTrace (libjavascriptcoregtk-4.0.so.18 + 0x1114a30) #5 0x97f02988 _ZN7WebCore13internalErrorERKN3WTF3URLE (libwebkit2gtk-4.0.so.37 + 0x214a988) #6 0x966038e0 _ZN6WebKit29NetworkConnectionToWebProcess12preconnectToESt8optionalIN3WTF16ObjectIdentifierIN7WebCore14ResourceLoaderONS_29NetworkResourceLoadParametersE (libwebkit2gtk-4.0.so.37 + 0x84b8e0) #7 0x964e0eb8 _ZN6WebKit29NetworkConnectionToWebProcess46didReceiveNetworkConnectionToWebProcessMessageERN3IPC10ConnectionERNS1_7DecoderE (libwebkit2gtk-4.0.so.37 + 0x728eb8) #8 0x9676eb70 _ZN3IPC10Connection15dispatchMessageESt10unique_ptrINS_7DecoderESt14default_deleteIS2_EE (libwebkit2gtk-4.0.so.37 + 0x9b6b70) #9 0x9676ef08 _ZN3IPC10Connection26dispatchOneIncomingMessageEv (libwebkit2gtk-4.0.so.37 + 0x9b6f08) #10 0x94f28ad0 _ZN3WTF7RunLoop11performWorkEv (libjavascriptcoregtk-4.0.so.18 + 0x1137ad0) #11 0x94f819f4 _ZZN3WTF7RunLoopC1EvEN3$_18__invokeEPv (libjavascriptcoregtk-4.0.so.18 + 0x11909f4) #12 0x94f80d88 _ZN3WTF7RunLoop3$_08__invokeEP8_GSourcePFiPvES4_ (libjavascriptcoregtk-4.0.so.18 + 0x118fd88) #13 0x91a53ab4 g_main_dispatch (libglib-2.0.so.0 + 0x53ab4) #14 0x91a53e5c g_main_context_iterate (libglib-2.0.so.0 + 0x53e5c) #15 0x91a541b0 g_main_loop_run (libglib-2.0.so.0 + 0x541b0) #16 0x94f81384 _ZN3WTF7RunLoop3runEv (libjavascriptcoregtk-4.0.so.18 + 0x1190384) #17 0x9674287c _ZN6WebKit20AuxiliaryProcessMainINS_22NetworkProcessMainSoupEEEiiPPc (libwebkit2gtk-4.0.so.37 + 0x98a87c) #18 0x914e6218 __libc_start_main (libc.so.6 + 0x24218) #19 0x00400874 $x (WebKitNetworkProcess + 0x874) #20 0x00400874 $x (WebKitNetworkProcess + 0x874) [] Process 14382 (WebKitWebProces) of user 1000 dumped core. Stack trace of thread 2: #0 0x854082ac __GI_raise (libc.so.6 + 0x372ac) #1 0x853f4ea0 __GI_abort (libc.so.6 + 0x23ea0) #2 0x87e98c50 _Z16WTFCrashWithInfoiPKcS0_i (libjavascriptcoregtk-4.0.so.18 + 0x198c50) #3 0x88e39138 _ZN3WTF10StackTrace17captureStackTraceEii (libjavascriptcoregtk-4.0.so.18 + 0x1139138) #4 0x88e14a30 WTFReleaseLogStackTrace (libjavascriptcoregtk-4.0.so.18 + 0x1114a30) #5 0x8be11988 _ZN7WebCore13internalErrorERKN3WTF3URLE (libwebkit2gtk-4.0.so.37 + 0x214a988) #6 0x8a9c1824 _ZN6WebKit17WebLoaderStrategy30internallyFailedLoadTimerFiredEv (libwebkit2gtk-4.0.so.37 + 0xcfa824) #7 0x88e90aa0 _ZZN3WTF7RunLoop9TimerBaseC1ERS0_EN3$_38__invokeEPv (libjavascriptcoregtk-4.0.so.18 + 0x1190aa0) #8 0x88e8fd88 _ZN3WTF7RunLoop3$_08__invokeEP8_GSourcePFiPvES4_ (libjavascriptcoregtk-4.0.so.18 + 0x118fd88) #9 0x85962ab4 g_main_dispatch (libglib-2.0.so.0 + 0x53ab4) #10 0x85962e5c g_main_context_iterate (libglib-2.0.so.0 + 0x53e5c) #11 0x859631b0 g_main_loop_run (libglib-2.0.so.0 + 0x541b0) #12 0x88e90384 _ZN3WTF7RunLoop3runEv (libjavascriptcoregtk-4.0.so.18 + 0x1190384) #13 0x8aa7b2b4 _ZN6WebKit20AuxiliaryProcessMainINS_17WebProcessMainGtkEEEiiPPc (libwebkit2gtk-4.0.so.37 + 0xdb42b4) #14 0x853f5218 __libc_start_main (libc.so.6 + 0x24218) #15 0x00400874 $x (WebKitWebProcess + 0x874) #16 0x00400874 $x (WebKitWebProcess + 0x874) -- System Information: Debian Release: 11.4 APT prefers stable APT policy: (990, 'stable'), (500, 'stable-updates'), (500, 'stable-security') Architecture: arm64 (aarch64) Foreign Architectures: armhf Kernel: Linux 5.15.52-rockchip64 (SMP w/6 CPU threads; PREEMPT) Kernel taint flags: TAINT_CRAP Locale: LANG=pl_PL.UTF-8, LC_CTYPE=pl_PL.UTF-8
