Re: Debugging Epiphany/Midori (webkit-gtk based) on earmv6hf (RPI 2)
After looking at the code, I´m pretty certain it´s the JavaScriptCore JIT compiler* who generates some assembly on a heap segment which then calls library functions with a misaligned stack pointer. The version of webkit-gtk which I built from a recent pkgsrc release is 2.4.8. The best thing to do first is trying the current version (2.10) of webkit-gtk I guess. * https://trac.webkit.org/browser/trunk/Source/JavaScriptCore/assembler/ARMAssembler.h 2015-10-14 7:56 GMT+02:00 Stephan: > 2015-10-13 21:34 GMT+02:00 Nick Hudson : >> On 10/13/15 17:58, Stephan wrote: >> >>> >>> Breakpoint 1, 0x46213ff4 in g_dpgettext2 () from >>> /usr/pkg/lib/libglib-2.0.so.0 >>> (gdb) i r $r12 >>> r120x7fffb8c8 2147465416 >>> >>> Breakpoint 1, 0x46213ff4 in g_dpgettext2 () from >>> /usr/pkg/lib/libglib-2.0.so.0 >>> (gdb) i r $r12 >>> r120x7fffb870 2147465328 >>> >>> Contrary, sp is broken in the non-working case: >>> >>> (gdb) i r $r12 >>> r120x7fffa414 2147460116 >>> >>> Unfortunately, the call trace is incomplete in that case: >> >> r13 is sp. > > Sure. When I use gdb, I set a breakpoint on . In practise, > the breakpoint hits after the stack frame was set up by that function, > e.g. when sp was already lowered to allocate space for local variables > (or the stack pointer was potentially aligned like gcc on x86 does). > On gcc/ARM, ip (r12) gets a copy of sp on the first instruction of the > function prologue. That´s why I monitored r12 (another solution could > be to break on *function+0, but I haven´t tested that). > >> >> debug symbols (compile flag -g) will help a lot here. >> > > Uff... ATM it looks like Webkit is at fault. It took 2 days to compile > on the RPI2 so it could take a long time to get the symbols ready. > Also the last thing I´ve seen yesterday was that the frame where gdb > stopped being able to follow the stack was special. Basically > speaking, I think the condition that makes this crash happen is when a > new tab/page is supposed to open from a JavaScript event - so it´s > likely that the JS interpreter is on the call trace. The code to the > said frame where gdb stops was executable memory (rwx) residing in an > [anon] region. I´m not sure whether symbols can help in this case nor > what else can... ;) > >> >> Nick
Re: Debugging Epiphany/Midori (webkit-gtk based) on earmv6hf (RPI 2)
Just a side note: The official Pi folks have been working on optimizations specific to the Pi / armv6 architecture: http://blog.barisione.org/2014-09/rpi-browser/ Among these are "JavaScript JIT fixes for ARMv6". Perhaps that version works better (also in terms of performace and stability), though I don´t know where the source code is. Fixes solving our issues might even be included in a more recent version of webkit-gtk, so it might make sense to check that first. 2015-10-14 7:56 GMT+02:00 Stephan: > 2015-10-13 21:34 GMT+02:00 Nick Hudson : >> On 10/13/15 17:58, Stephan wrote: >> >>> >>> Breakpoint 1, 0x46213ff4 in g_dpgettext2 () from >>> /usr/pkg/lib/libglib-2.0.so.0 >>> (gdb) i r $r12 >>> r120x7fffb8c8 2147465416 >>> >>> Breakpoint 1, 0x46213ff4 in g_dpgettext2 () from >>> /usr/pkg/lib/libglib-2.0.so.0 >>> (gdb) i r $r12 >>> r120x7fffb870 2147465328 >>> >>> Contrary, sp is broken in the non-working case: >>> >>> (gdb) i r $r12 >>> r120x7fffa414 2147460116 >>> >>> Unfortunately, the call trace is incomplete in that case: >> >> r13 is sp. > > Sure. When I use gdb, I set a breakpoint on . In practise, > the breakpoint hits after the stack frame was set up by that function, > e.g. when sp was already lowered to allocate space for local variables > (or the stack pointer was potentially aligned like gcc on x86 does). > On gcc/ARM, ip (r12) gets a copy of sp on the first instruction of the > function prologue. That´s why I monitored r12 (another solution could > be to break on *function+0, but I haven´t tested that). > >> >> debug symbols (compile flag -g) will help a lot here. >> > > Uff... ATM it looks like Webkit is at fault. It took 2 days to compile > on the RPI2 so it could take a long time to get the symbols ready. > Also the last thing I´ve seen yesterday was that the frame where gdb > stopped being able to follow the stack was special. Basically > speaking, I think the condition that makes this crash happen is when a > new tab/page is supposed to open from a JavaScript event - so it´s > likely that the JS interpreter is on the call trace. The code to the > said frame where gdb stops was executable memory (rwx) residing in an > [anon] region. I´m not sure whether symbols can help in this case nor > what else can... ;) > >> >> Nick
Re: Debugging Epiphany/Midori (webkit-gtk based) on earmv6hf (RPI 2)
2015-10-13 21:34 GMT+02:00 Nick Hudson: > On 10/13/15 17:58, Stephan wrote: > >> >> Breakpoint 1, 0x46213ff4 in g_dpgettext2 () from >> /usr/pkg/lib/libglib-2.0.so.0 >> (gdb) i r $r12 >> r120x7fffb8c8 2147465416 >> >> Breakpoint 1, 0x46213ff4 in g_dpgettext2 () from >> /usr/pkg/lib/libglib-2.0.so.0 >> (gdb) i r $r12 >> r120x7fffb870 2147465328 >> >> Contrary, sp is broken in the non-working case: >> >> (gdb) i r $r12 >> r120x7fffa414 2147460116 >> >> Unfortunately, the call trace is incomplete in that case: > > r13 is sp. Sure. When I use gdb, I set a breakpoint on . In practise, the breakpoint hits after the stack frame was set up by that function, e.g. when sp was already lowered to allocate space for local variables (or the stack pointer was potentially aligned like gcc on x86 does). On gcc/ARM, ip (r12) gets a copy of sp on the first instruction of the function prologue. That´s why I monitored r12 (another solution could be to break on *function+0, but I haven´t tested that). > > debug symbols (compile flag -g) will help a lot here. > Uff... ATM it looks like Webkit is at fault. It took 2 days to compile on the RPI2 so it could take a long time to get the symbols ready. Also the last thing I´ve seen yesterday was that the frame where gdb stopped being able to follow the stack was special. Basically speaking, I think the condition that makes this crash happen is when a new tab/page is supposed to open from a JavaScript event - so it´s likely that the JS interpreter is on the call trace. The code to the said frame where gdb stops was executable memory (rwx) residing in an [anon] region. I´m not sure whether symbols can help in this case nor what else can... ;) > > Nick
Re: Debugging Epiphany/Midori (webkit-gtk based) on earmv6hf (RPI 2)
On 05/31/15 18:07, Stephan wrote: Hi folks, I am currently testing some applications on the RPI 2. Some work pretty well, others not yet. As for webkit-gtk based browsers, I am experiencing crashes from time to time. One problem that occurs often seems to be related to g_dpgettext2 () from glib2. The top of the stack looks like this: (gdb) bt #0 0x636f7452 in ?? () #1 0x45ff3fa8 in g_dpgettext2 () from /usr/pkg/lib/libglib-2.0.so.0 #2 0x42ad6030 in gtk_stock_lookup () from /usr/pkg/lib/libgtk-x11-2.0.so.0 #3 0x42987b98 in gtk_action_set_stock_id () from /usr/pkg/lib/libgtk-x11-2.0.so.0 #4 0x45f55cfc in g_object_set_valist () from /usr/pkg/lib/libgobject-2.0.so.0 #5 0x45f5642c in g_object_set () from /usr/pkg/lib/libgobject-2.0.so.0 #6 0x4298a27c in gtk_action_group_add_actions_full () from /usr/pkg/lib/libgtk-x11-2.0.so.0 #7 0x4298a388 in gtk_action_group_add_actions () from /usr/pkg/lib/libgtk-x11-2.0.so.0 #8 0x0004238c in ?? () #9 0x45f5322c in g_object_new_internal () from /usr/pkg/lib/libgobject-2.0.so.0 #10 0x45f5587c in g_object_new_valist () from /usr/pkg/lib/libgobject-2.0.so.0 #11 0x45f55a24 in g_object_new () from /usr/pkg/lib/libgobject-2.0.so.0 #12 0x00043ff0 in ephy_window_new_with_chrome () #13 0x0003ac94 in ephy_shell_new_tab_full () #14 0x0003f81c in ?? () #15 0x40a090e8 in webkit_marshal_OBJECT__OBJECT () from /usr/pkg/lib/libwebkitgtk-1.0.so.0 #16 0x45f4e070 in g_closure_invoke () from /usr/pkg/lib/libgobject-2.0.so.0 #17 0x45f6154c in signal_emit_unlocked_R () from /usr/pkg/lib/libgobject-2.0.so.0 #18 0x45f69278 in g_signal_emit_valist () from /usr/pkg/lib/libgobject-2.0.so.0 #19 0x45f69cac in g_signal_emit_by_name () from /usr/pkg/lib/libgobject-2.0.so.0 #20 0x409d9074 in WebKit::FrameLoaderClient::dispatchCreatePage(WebCore::NavigationAction const&) () from /usr/pkg/lib/libwebkitgtk-1.0.so.0 ... I'm pretty sure the problem is that somewhere in the call stack the sp isn't 8-byte aligned and the alloca in g_dpgettext2 falls over this Move up the frames doing up; info frame (or similar) Nick
Re: Debugging Epiphany/Midori (webkit-gtk based) on earmv6hf (RPI 2)
Thanks all for your input. Nick was right that the stack pointer needs to be aligned on a 8-byte boundary. I was totally unaware that the calling convention requires this. The g_dpgettext2() function runs successfully a couple of times when epiphany launches. It is then called with an appropriately aligned stack pointer: Breakpoint 1, 0x46213ff4 in g_dpgettext2 () from /usr/pkg/lib/libglib-2.0.so.0 (gdb) i r $r12 r120x7fffb8c8 2147465416 Breakpoint 1, 0x46213ff4 in g_dpgettext2 () from /usr/pkg/lib/libglib-2.0.so.0 (gdb) i r $r12 r120x7fffb870 2147465328 Contrary, sp is broken in the non-working case: (gdb) i r $r12 r120x7fffa414 2147460116 Unfortunately, the call trace is incomplete in that case: #35 0x40c42178 in WebCore::EventTarget::dispatchEvent(WTF::PassRefPtr, int&) () from /usr/pkg/lib/libwebkitgtk-1.0.so.0 #36 0x4147fd84 in WebCore::jsNodePrototypeFunctionDispatchEvent(JSC::ExecState*) () from /usr/pkg/lib/libwebkitgtk-1.0.so.0 #37 0x43fc81a4 in ?? () Backtrace stopped: previous frame identical to this frame (corrupt stack?) The only thing I can say just now is that sp is already misaligned in frame 36 when it comes out of this obfuscated frame 37. I set a separate breakpoint as it seems not too easy to get the value of sp for each frame on a full call trace: Breakpoint 1, 0x4147fc94 in WebCore::jsNodePrototypeFunctionDispatchEvent(JSC::ExecState*) () from /usr/pkg/lib/libwebkitgtk-1.0.so.0 (gdb) bt #0 0x4147fc94 in WebCore::jsNodePrototypeFunctionDispatchEvent(JSC::ExecState*) () from /usr/pkg/lib/libwebkitgtk-1.0.so.0 #1 0x50add124 in ?? () (gdb) i r $r12 r120x7fffb88c 2147465356 This is not too easy :) 2015-10-13 8:17 GMT+00:00 Nick Hudson: > On 05/31/15 18:07, Stephan wrote: >> >> Hi folks, >> >> I am currently testing some applications on the RPI 2. Some work >> pretty well, others not yet. As for webkit-gtk based browsers, I am >> experiencing crashes from time to time. >> >> One problem that occurs often seems to be related to g_dpgettext2 () >> from glib2. The top of the stack looks like this: >> >> (gdb) bt >> #0 0x636f7452 in ?? () >> #1 0x45ff3fa8 in g_dpgettext2 () from /usr/pkg/lib/libglib-2.0.so.0 >> #2 0x42ad6030 in gtk_stock_lookup () from >> /usr/pkg/lib/libgtk-x11-2.0.so.0 >> #3 0x42987b98 in gtk_action_set_stock_id () from >> /usr/pkg/lib/libgtk-x11-2.0.so.0 >> #4 0x45f55cfc in g_object_set_valist () from >> /usr/pkg/lib/libgobject-2.0.so.0 >> #5 0x45f5642c in g_object_set () from /usr/pkg/lib/libgobject-2.0.so.0 >> #6 0x4298a27c in gtk_action_group_add_actions_full () from >> /usr/pkg/lib/libgtk-x11-2.0.so.0 >> #7 0x4298a388 in gtk_action_group_add_actions () from >> /usr/pkg/lib/libgtk-x11-2.0.so.0 >> #8 0x0004238c in ?? () >> #9 0x45f5322c in g_object_new_internal () from >> /usr/pkg/lib/libgobject-2.0.so.0 >> #10 0x45f5587c in g_object_new_valist () from >> /usr/pkg/lib/libgobject-2.0.so.0 >> #11 0x45f55a24 in g_object_new () from /usr/pkg/lib/libgobject-2.0.so.0 >> #12 0x00043ff0 in ephy_window_new_with_chrome () >> #13 0x0003ac94 in ephy_shell_new_tab_full () >> #14 0x0003f81c in ?? () >> #15 0x40a090e8 in webkit_marshal_OBJECT__OBJECT () from >> /usr/pkg/lib/libwebkitgtk-1.0.so.0 >> #16 0x45f4e070 in g_closure_invoke () from >> /usr/pkg/lib/libgobject-2.0.so.0 >> #17 0x45f6154c in signal_emit_unlocked_R () from >> /usr/pkg/lib/libgobject-2.0.so.0 >> #18 0x45f69278 in g_signal_emit_valist () from >> /usr/pkg/lib/libgobject-2.0.so.0 >> #19 0x45f69cac in g_signal_emit_by_name () from >> /usr/pkg/lib/libgobject-2.0.so.0 >> #20 0x409d9074 in >> WebKit::FrameLoaderClient::dispatchCreatePage(WebCore::NavigationAction >> const&) () from /usr/pkg/lib/libwebkitgtk-1.0.so.0 >> ... > > > I'm pretty sure the problem is that somewhere in the call stack the > sp isn't 8-byte aligned and the alloca in g_dpgettext2 falls over this > > Move up the frames doing up; info frame (or similar) > > Nick
Re: Debugging Epiphany/Midori (webkit-gtk based) on earmv6hf (RPI 2)
On 10/13/15 17:58, Stephan wrote: Breakpoint 1, 0x46213ff4 in g_dpgettext2 () from /usr/pkg/lib/libglib-2.0.so.0 (gdb) i r $r12 r120x7fffb8c8 2147465416 Breakpoint 1, 0x46213ff4 in g_dpgettext2 () from /usr/pkg/lib/libglib-2.0.so.0 (gdb) i r $r12 r120x7fffb870 2147465328 Contrary, sp is broken in the non-working case: (gdb) i r $r12 r120x7fffa414 2147460116 Unfortunately, the call trace is incomplete in that case: r13 is sp. debug symbols (compile flag -g) will help a lot here. Nick
Re: Debugging Epiphany/Midori (webkit-gtk based) on earmv6hf (RPI 2)
On Thu 08 Oct 2015 at 17:59:22 +, Stephan wrote: > IMHO it would be valid to pass the value of sp as the first argument > (mov r0, sp). However, the way how r0 is computed looks quite > obfuscated to me: > >0x4621402c: mov r3, sp=> ? >0x46214030 : lsr r7, r3, #3 => ? >0x46214034 : lsl r4, r7, #3 => ? ... >0x46214040 : mov r0, r4 => ? I don't really know any ARM assembly but it seems that the value from sp is shifted right 3 bits and then left again. In effect clearing the low 3 bits, or aligning to 8. Which by itself is a very sensible thing to do for an allocator. Unfortunately (since it rounds down) it can have the effect of allocating the memory lower than sp, which is bound to fail at some point. Probably the value should also go back into sp. There is also an alloca in libc: cvs/src/lib/libc/arch/arm/gen/alloca.S. It uses a similar shifting trick but it uses it to round up the allocation size to multiples of 8, and it does end up with the same value in r0 and sp. -Olaf. -- ___ Olaf 'Rhialto' Seibert -- The Doctor: No, 'eureka' is Greek for \X/ rhialto/at/xs4all.nl-- 'this bath is too hot.' pgpFCM8YmMK22.pgp Description: PGP signature
Re: Debugging Epiphany/Midori (webkit-gtk based) on earmv6hf (RPI 2)
Folks, more on this topic: This -8<-- (gdb) x/5i 0x45fd3efc 0x45fd3efc: add r12, pc, #1048576 ; 0x10 0x45fd3f00: add r12, r12, #4096 ; 0x1000 0x45fd3f04: ldr pc, [r12, #1652]! ; 0x674 0x45fd3f08: add r12, pc, #1048576 ; 0x10 0x45fd3f0c: add r12, r12, #4096 ; 0x1000 --- is not garbage. It's kind of a link table (plt) inside of glib2. It makes the first 2 calls in g_dpgettext2() go to strlen() and the third go to memcpy(). memcpy() saves the r0 and lr regisers at the very beginning and pops them into r0 and pc at a later time in order to return to the caller. However somewhere in the function, the stack data seems to become corrupted and a junk value is loaded into pc, leading to the crash. Either its value was overwritten or sp is pointing to the wrong place. (gdb) x/50i 0x45e80f98 0x45e80f98 : push {r0, lr}=> SAVE REGISTERS 0x45e80f9c: subs r2, r2, #4 0x45e80fa0 : blt 0x45e81028 0x45e80fa4 : ands r12, r0, #3 0x45e80fa8 : bne 0x45e81050 0x45e80fac : ands r12, r1, #3 0x45e80fb0 : bne 0x45e81080 0x45e80fb4 : subs r2, r2, #8 0x45e80fb8 : blt 0x45e81008 0x45e80fbc : subs r2, r2, #20 0x45e80fc0 : blt 0x45e80ff4 0x45e80fc4 : push {r4} ; (str r4, [sp, #-4]!) 0x45e80fc8 : ldm r1!, {r3, r4, r12, lr} 0x45e80fcc : stmia r0!, {r3, r4, r12, lr} 0x45e80fd0 : ldm r1!, {r3, r4, r12, lr} 0x45e80fd4 : stmia r0!, {r3, r4, r12, lr} 0x45e80fd8 : subs r2, r2, #32 0x45e80fdc : bge 0x45e80fc8 0x45e80fe0 : cmn r2, #16 0x45e80fe4 : ldmge r1!, {r3, r4, r12, lr} 0x45e80fe8 : stmiage r0!, {r3, r4, r12, lr} 0x45e80fec : subge r2, r2, #16 0x45e80ff0 : pop {r4} ; (ldr r4, [sp], #4) 0x45e80ff4 : adds r2, r2, #20 0x45e80ff8 : ldmge r1!, {r3, r12, lr} 0x45e80ffc : stmiage r0!, {r3, r12, lr} 0x45e81000 : subsge r2, r2, #12 0x45e81004 : bge 0x45e80ff8 0x45e81008 : adds r2, r2, #8 0x45e8100c : blt 0x45e81028 0x45e81010 : subs r2, r2, #4 0x45e81014 : ldrlt r3, [r1], #4 0x45e81018 : strlt r3, [r0], #4 0x45e8101c : ldmge r1!, {r3, r12} 0x45e81020 : stmiage r0!, {r3, r12} 0x45e81024 : subge r2, r2, #4 0x45e81028 : adds r2, r2, #4 0x45e8102c : popeq {r0, pc} 0x45e81030 : cmp r2, #2 0x45e81034 : ldrb r3, [r1], #1 0x45e81038 : strb r3, [r0], #1 0x45e8103c : ldrbge r3, [r1], #1 0x45e81040 : strbge r3, [r0], #1 0x45e81044 : ldrbgt r3, [r1], #1 0x45e81048 : strbgt r3, [r0], #1 => 0x45e8104c : pop {r0, pc} ==> CRASH: PC gets corrupted value 0x45e81050 : rsb r12, r12, #4 Further investigation is to come ;) 2015-05-31 19:05 GMT+00:00 Jared McNeill : > I saw the same issue a while back, don't remember exactly which version of > glib2 it was. I LD_PRELOAD'd a version of g_dpgettext2 that always returned > NULL and it made the crashes go away. g_dpgettext2 uses alloca internally, > so I increased stack size and the crashes went away again. > > https://git.gnome.org/browse/glib/tree/glib/ggettext.c#n271 > > > On Sun, 31 May 2015, Stephan wrote: > >> >> That does not help unfortunately - just tested again with ulimit -s >> unlimited. I dont think we are hitting the stack boundary >> because sp is still inside the stack region outlined by pmap. Have you >> seen the very same issue as I do? >> >> Am 31.05.2015 19:48 schrieb "Jared McNeill" : >> Default stack size limits on evbarm are too low for webkit-gtk. Bump >> it up with ulimit -s and the g_dpgettext2 >> problem should go away. >> >> >> >> On Sun, 31 May 2015, Stephan wrote: >> >> Hi folks, >> >> I am currently testing some applications on the RPI 2. Some >> work >> pretty well, others not yet. As for webkit-gtk based browsers, >> I am >> experiencing crashes from time to time. >> >> One problem that occurs often seems to be related to >> g_dpgettext2 () >> from glib2. The top of the stack looks like this: >> >> (gdb) bt >> #0 0x636f7452 in ?? () >> #1 0x45ff3fa8 in g_dpgettext2 () from >> /usr/pkg/lib/libglib-2.0.so.0 >> #2 0x42ad6030 in gtk_stock_lookup () from >> /usr/pkg/lib/libgtk-x11-2.0.so.0 >> #3 0x42987b98 in
Re: Debugging Epiphany/Midori (webkit-gtk based) on earmv6hf (RPI 2)
Default stack size limits on evbarm are too low for webkit-gtk. Bump it up with ulimit -s and the g_dpgettext2 problem should go away. On Sun, 31 May 2015, Stephan wrote: Hi folks, I am currently testing some applications on the RPI 2. Some work pretty well, others not yet. As for webkit-gtk based browsers, I am experiencing crashes from time to time. One problem that occurs often seems to be related to g_dpgettext2 () from glib2. The top of the stack looks like this: (gdb) bt #0 0x636f7452 in ?? () #1 0x45ff3fa8 in g_dpgettext2 () from /usr/pkg/lib/libglib-2.0.so.0 #2 0x42ad6030 in gtk_stock_lookup () from /usr/pkg/lib/libgtk-x11-2.0.so.0 #3 0x42987b98 in gtk_action_set_stock_id () from /usr/pkg/lib/libgtk-x11-2.0.so.0 #4 0x45f55cfc in g_object_set_valist () from /usr/pkg/lib/libgobject-2.0.so.0 #5 0x45f5642c in g_object_set () from /usr/pkg/lib/libgobject-2.0.so.0 #6 0x4298a27c in gtk_action_group_add_actions_full () from /usr/pkg/lib/libgtk-x11-2.0.so.0 #7 0x4298a388 in gtk_action_group_add_actions () from /usr/pkg/lib/libgtk-x11-2.0.so.0 #8 0x0004238c in ?? () #9 0x45f5322c in g_object_new_internal () from /usr/pkg/lib/libgobject-2.0.so.0 #10 0x45f5587c in g_object_new_valist () from /usr/pkg/lib/libgobject-2.0.so.0 #11 0x45f55a24 in g_object_new () from /usr/pkg/lib/libgobject-2.0.so.0 #12 0x00043ff0 in ephy_window_new_with_chrome () #13 0x0003ac94 in ephy_shell_new_tab_full () #14 0x0003f81c in ?? () #15 0x40a090e8 in webkit_marshal_OBJECT__OBJECT () from /usr/pkg/lib/libwebkitgtk-1.0.so.0 #16 0x45f4e070 in g_closure_invoke () from /usr/pkg/lib/libgobject-2.0.so.0 #17 0x45f6154c in signal_emit_unlocked_R () from /usr/pkg/lib/libgobject-2.0.so.0 #18 0x45f69278 in g_signal_emit_valist () from /usr/pkg/lib/libgobject-2.0.so.0 #19 0x45f69cac in g_signal_emit_by_name () from /usr/pkg/lib/libgobject-2.0.so.0 #20 0x409d9074 in WebKit::FrameLoaderClient::dispatchCreatePage(WebCore::NavigationAction const) () from /usr/pkg/lib/libwebkitgtk-1.0.so.0 ... The call to the topmost frame happens from 0x45ff3fa4 in the said funtion. The disassembly from within gdb loos like this: (gdb) disass g_dpgettext2 Dump of assembler code for function g_dpgettext2: 0x45ff3f44 +0: mov r12, sp 0x45ff3f48 +4: push{r4, r5, r6, r7, r8, r9, r10, r11, r12, lr, pc} 0x45ff3f4c +8: sub r11, r12, #4 0x45ff3f50 +12:sub sp, sp, #12 0x45ff3f54 +16:mov r9, r0 0x45ff3f58 +20:mov r0, r1 0x45ff3f5c +24:mov r6, r2 0x45ff3f60 +28:str r1, [r11, #-48] ; 0x30 0x45ff3f64 +32:bl 0x45fd3f44 0x45ff3f68 +36:mov r5, r0 0x45ff3f6c +40:mov r0, r6 0x45ff3f70 +44:bl 0x45fd3f44 0x45ff3f74 +48:add r10, r5, #1 0x45ff3f78 +52:add r8, r0, #1 0x45ff3f7c +56:add r3, r8, r10 0x45ff3f80 +60:add r3, r3, #14 0x45ff3f84 +64:bic r3, r3, #7 0x45ff3f88 +68:sub sp, sp, r3 0x45ff3f8c +72:mov r3, sp 0x45ff3f90 +76:lsr r7, r3, #3 0x45ff3f94 +80:lsl r4, r7, #3 0x45ff3f98 +84:ldr r1, [r11, #-48] ; 0x30 0x45ff3f9c +88:mov r2, r5 0x45ff3fa0 +92:mov r0, r4 0x45ff3fa4 +96:bl 0x45fd3efc 0x45ff3fa8 +100: mov r3, #4 0x45ff3fac +104: mov r2, r8 0x45ff3fb0 +108: strbr3, [r5, r7, lsl #3] ... Now the jump target actually seems to be some random memory: (gdb) info symbol 0x45fd3efc No symbol matches 0x45fd3efc. At least it is mapped to the text section of libglib2. It decodes like follows which just looks like random garbage: (gdb) x/5i 0x45fd3efc 0x45fd3efc: add r12, pc, #1048576 ; 0x10 0x45fd3f00: add r12, r12, #4096 ; 0x1000 0x45fd3f04: ldr pc, [r12, #1652]! ; 0x674 0x45fd3f08: add r12, pc, #1048576 ; 0x10 0x45fd3f0c: add r12, r12, #4096 ; 0x1000 The disassembly of g_dpgettext2() from objdump looks weird as well: 00033f44 g_dpgettext2: 33f44: e1a0c00dmov ip, sp 33f48: e92ddff0push{r4, r5, r6, r7, r8, r9, sl, fp, ip, lr, pc} 33f4c: e24cb004sub fp, ip, #4 33f50: e24dd00csub sp, sp, #12 33f54: e1a09000mov r9, r0 33f58: e1a1mov r0, r1 33f5c: e1a06002mov r6, r2 33f60: e50b1030str r1, [fp, #-48] ; 0x30 33f64: ebff7ff6bl 13f44 g_mem_chunk_new-0x7c 33f68: e1a05000mov r5, r0 33f6c: e1a6mov r0, r6 33f70: ebff7ff3bl 13f44 g_mem_chunk_new-0x7c 33f74: e285a001add sl, r5, #1 33f78: e2808001add r8, r0, #1 33f7c: e088300aadd r3, r8, sl 33f80: e283300eadd r3, r3, #14 33f84: e3c33007bic r3, r3, #7 33f88: e04dd003sub sp, sp, r3 33f8c: e1a0300dmov r3, sp 33f90:
Re: Debugging Epiphany/Midori (webkit-gtk based) on earmv6hf (RPI 2)
I saw the same issue a while back, don't remember exactly which version of glib2 it was. I LD_PRELOAD'd a version of g_dpgettext2 that always returned NULL and it made the crashes go away. g_dpgettext2 uses alloca internally, so I increased stack size and the crashes went away again. https://git.gnome.org/browse/glib/tree/glib/ggettext.c#n271 On Sun, 31 May 2015, Stephan wrote: That does not help unfortunately - just tested again with ulimit -s unlimited. I dont think we are hitting the stack boundary because sp is still inside the stack region outlined by pmap. Have you seen the very same issue as I do? Am 31.05.2015 19:48 schrieb Jared McNeill jmcne...@invisible.ca: Default stack size limits on evbarm are too low for webkit-gtk. Bump it up with ulimit -s and the g_dpgettext2 problem should go away. On Sun, 31 May 2015, Stephan wrote: Hi folks, I am currently testing some applications on the RPI 2. Some work pretty well, others not yet. As for webkit-gtk based browsers, I am experiencing crashes from time to time. One problem that occurs often seems to be related to g_dpgettext2 () from glib2. The top of the stack looks like this: (gdb) bt #0 0x636f7452 in ?? () #1 0x45ff3fa8 in g_dpgettext2 () from /usr/pkg/lib/libglib-2.0.so.0 #2 0x42ad6030 in gtk_stock_lookup () from /usr/pkg/lib/libgtk-x11-2.0.so.0 #3 0x42987b98 in gtk_action_set_stock_id () from /usr/pkg/lib/libgtk-x11-2.0.so.0 #4 0x45f55cfc in g_object_set_valist () from /usr/pkg/lib/libgobject-2.0.so.0 #5 0x45f5642c in g_object_set () from /usr/pkg/lib/libgobject-2.0.so.0 #6 0x4298a27c in gtk_action_group_add_actions_full () from /usr/pkg/lib/libgtk-x11-2.0.so.0 #7 0x4298a388 in gtk_action_group_add_actions () from /usr/pkg/lib/libgtk-x11-2.0.so.0 #8 0x0004238c in ?? () #9 0x45f5322c in g_object_new_internal () from /usr/pkg/lib/libgobject-2.0.so.0 #10 0x45f5587c in g_object_new_valist () from /usr/pkg/lib/libgobject-2.0.so.0 #11 0x45f55a24 in g_object_new () from /usr/pkg/lib/libgobject-2.0.so.0 #12 0x00043ff0 in ephy_window_new_with_chrome () #13 0x0003ac94 in ephy_shell_new_tab_full () #14 0x0003f81c in ?? () #15 0x40a090e8 in webkit_marshal_OBJECT__OBJECT () from /usr/pkg/lib/libwebkitgtk-1.0.so.0 #16 0x45f4e070 in g_closure_invoke () from /usr/pkg/lib/libgobject-2.0.so.0 #17 0x45f6154c in signal_emit_unlocked_R () from /usr/pkg/lib/libgobject-2.0.so.0 #18 0x45f69278 in g_signal_emit_valist () from /usr/pkg/lib/libgobject-2.0.so.0 #19 0x45f69cac in g_signal_emit_by_name () from /usr/pkg/lib/libgobject-2.0.so.0 #20 0x409d9074 in WebKit::FrameLoaderClient::dispatchCreatePage(WebCore::NavigationAction const) () from /usr/pkg/lib/libwebkitgtk-1.0.so.0 ... The call to the topmost frame happens from 0x45ff3fa4 in the said funtion. The disassembly from within gdb loos like this: (gdb) disass g_dpgettext2 Dump of assembler code for function g_dpgettext2: 0x45ff3f44 +0: mov r12, sp 0x45ff3f48 +4: push {r4, r5, r6, r7, r8, r9, r10, r11, r12, lr, pc} 0x45ff3f4c +8: sub r11, r12, #4 0x45ff3f50 +12: sub sp, sp, #12 0x45ff3f54 +16: mov r9, r0 0x45ff3f58 +20: mov r0, r1 0x45ff3f5c +24: mov r6, r2 0x45ff3f60 +28: str r1, [r11, #-48] ; 0x30 0x45ff3f64 +32: bl 0x45fd3f44 0x45ff3f68 +36: mov r5, r0 0x45ff3f6c +40: mov r0, r6 0x45ff3f70 +44: bl 0x45fd3f44 0x45ff3f74 +48: add r10, r5, #1 0x45ff3f78 +52: add r8, r0, #1 0x45ff3f7c +56: add r3, r8, r10 0x45ff3f80 +60: add r3, r3, #14 0x45ff3f84 +64: bic r3, r3, #7 0x45ff3f88 +68: sub sp, sp, r3 0x45ff3f8c +72: mov r3, sp 0x45ff3f90 +76: lsr r7, r3, #3 0x45ff3f94 +80: lsl r4, r7, #3 0x45ff3f98 +84: ldr r1, [r11, #-48] ; 0x30 0x45ff3f9c +88: mov r2, r5 0x45ff3fa0 +92: mov r0, r4 0x45ff3fa4 +96: bl 0x45fd3efc 0x45ff3fa8 +100: mov r3, #4 0x45ff3fac +104: mov r2, r8 0x45ff3fb0 +108: strb r3, [r5, r7, lsl #3] ... Now the jump target actually seems to be some random memory:
Debugging Epiphany/Midori (webkit-gtk based) on earmv6hf (RPI 2)
Hi folks, I am currently testing some applications on the RPI 2. Some work pretty well, others not yet. As for webkit-gtk based browsers, I am experiencing crashes from time to time. One problem that occurs often seems to be related to g_dpgettext2 () from glib2. The top of the stack looks like this: (gdb) bt #0 0x636f7452 in ?? () #1 0x45ff3fa8 in g_dpgettext2 () from /usr/pkg/lib/libglib-2.0.so.0 #2 0x42ad6030 in gtk_stock_lookup () from /usr/pkg/lib/libgtk-x11-2.0.so.0 #3 0x42987b98 in gtk_action_set_stock_id () from /usr/pkg/lib/libgtk-x11-2.0.so.0 #4 0x45f55cfc in g_object_set_valist () from /usr/pkg/lib/libgobject-2.0.so.0 #5 0x45f5642c in g_object_set () from /usr/pkg/lib/libgobject-2.0.so.0 #6 0x4298a27c in gtk_action_group_add_actions_full () from /usr/pkg/lib/libgtk-x11-2.0.so.0 #7 0x4298a388 in gtk_action_group_add_actions () from /usr/pkg/lib/libgtk-x11-2.0.so.0 #8 0x0004238c in ?? () #9 0x45f5322c in g_object_new_internal () from /usr/pkg/lib/libgobject-2.0.so.0 #10 0x45f5587c in g_object_new_valist () from /usr/pkg/lib/libgobject-2.0.so.0 #11 0x45f55a24 in g_object_new () from /usr/pkg/lib/libgobject-2.0.so.0 #12 0x00043ff0 in ephy_window_new_with_chrome () #13 0x0003ac94 in ephy_shell_new_tab_full () #14 0x0003f81c in ?? () #15 0x40a090e8 in webkit_marshal_OBJECT__OBJECT () from /usr/pkg/lib/libwebkitgtk-1.0.so.0 #16 0x45f4e070 in g_closure_invoke () from /usr/pkg/lib/libgobject-2.0.so.0 #17 0x45f6154c in signal_emit_unlocked_R () from /usr/pkg/lib/libgobject-2.0.so.0 #18 0x45f69278 in g_signal_emit_valist () from /usr/pkg/lib/libgobject-2.0.so.0 #19 0x45f69cac in g_signal_emit_by_name () from /usr/pkg/lib/libgobject-2.0.so.0 #20 0x409d9074 in WebKit::FrameLoaderClient::dispatchCreatePage(WebCore::NavigationAction const) () from /usr/pkg/lib/libwebkitgtk-1.0.so.0 ... The call to the topmost frame happens from 0x45ff3fa4 in the said funtion. The disassembly from within gdb loos like this: (gdb) disass g_dpgettext2 Dump of assembler code for function g_dpgettext2: 0x45ff3f44 +0: mov r12, sp 0x45ff3f48 +4: push{r4, r5, r6, r7, r8, r9, r10, r11, r12, lr, pc} 0x45ff3f4c +8: sub r11, r12, #4 0x45ff3f50 +12:sub sp, sp, #12 0x45ff3f54 +16:mov r9, r0 0x45ff3f58 +20:mov r0, r1 0x45ff3f5c +24:mov r6, r2 0x45ff3f60 +28:str r1, [r11, #-48] ; 0x30 0x45ff3f64 +32:bl 0x45fd3f44 0x45ff3f68 +36:mov r5, r0 0x45ff3f6c +40:mov r0, r6 0x45ff3f70 +44:bl 0x45fd3f44 0x45ff3f74 +48:add r10, r5, #1 0x45ff3f78 +52:add r8, r0, #1 0x45ff3f7c +56:add r3, r8, r10 0x45ff3f80 +60:add r3, r3, #14 0x45ff3f84 +64:bic r3, r3, #7 0x45ff3f88 +68:sub sp, sp, r3 0x45ff3f8c +72:mov r3, sp 0x45ff3f90 +76:lsr r7, r3, #3 0x45ff3f94 +80:lsl r4, r7, #3 0x45ff3f98 +84:ldr r1, [r11, #-48] ; 0x30 0x45ff3f9c +88:mov r2, r5 0x45ff3fa0 +92:mov r0, r4 0x45ff3fa4 +96:bl 0x45fd3efc 0x45ff3fa8 +100: mov r3, #4 0x45ff3fac +104: mov r2, r8 0x45ff3fb0 +108: strbr3, [r5, r7, lsl #3] ... Now the jump target actually seems to be some random memory: (gdb) info symbol 0x45fd3efc No symbol matches 0x45fd3efc. At least it is mapped to the text section of libglib2. It decodes like follows which just looks like random garbage: (gdb) x/5i 0x45fd3efc 0x45fd3efc: add r12, pc, #1048576 ; 0x10 0x45fd3f00: add r12, r12, #4096 ; 0x1000 0x45fd3f04: ldr pc, [r12, #1652]! ; 0x674 0x45fd3f08: add r12, pc, #1048576 ; 0x10 0x45fd3f0c: add r12, r12, #4096 ; 0x1000 The disassembly of g_dpgettext2() from objdump looks weird as well: 00033f44 g_dpgettext2: 33f44: e1a0c00dmov ip, sp 33f48: e92ddff0push{r4, r5, r6, r7, r8, r9, sl, fp, ip, lr, pc} 33f4c: e24cb004sub fp, ip, #4 33f50: e24dd00csub sp, sp, #12 33f54: e1a09000mov r9, r0 33f58: e1a1mov r0, r1 33f5c: e1a06002mov r6, r2 33f60: e50b1030str r1, [fp, #-48] ; 0x30 33f64: ebff7ff6bl 13f44 g_mem_chunk_new-0x7c 33f68: e1a05000mov r5, r0 33f6c: e1a6mov r0, r6 33f70: ebff7ff3bl 13f44 g_mem_chunk_new-0x7c 33f74: e285a001add sl, r5, #1 33f78: e2808001add r8, r0, #1 33f7c: e088300aadd r3, r8, sl 33f80: e283300eadd r3, r3, #14 33f84: e3c33007bic r3, r3, #7 33f88: e04dd003sub sp, sp, r3 33f8c: e1a0300dmov r3, sp 33f90: e1a071a3lsr r7, r3, #3 33f94: e1a04187lsl r4, r7, #3 33f98: e51b1030