On Mon, Jan 11, 2021 at 9:37 AM Deepak Krishnan <[email protected]> wrote:
> Hi, > > Running OSv with some Linux binaries and had a crash that I couldn't > completely understand: > > (gdb) where > #0 processor::cli_hlt () at arch/x64/processor.hh:248 > #1 0x0000000040208fe6 in arch::halt_no_interrupts () at > arch/x64/arch.hh:48 > #2 0x0000000040487679 in osv::halt () at arch/x64/power.cc:26 > #3 0x000000004021f9e9 in abort (fmt=0x40989a90 "Assertion failed: %s (%s: > %s: %d)\n") at runtime.cc:132 > #4 0x000000004021fa24 in __assert_fail (expr=0x409d5613 > "sched::preemptable()", file=0x409d5603 "arch/x64/mmu.cc", line=37, > func=0x409d5840 <page_fault::__func__> "page_fault") at runtime.cc:139 > #5 0x0000000040476cea in page_fault (ef=0xffff8000089c5068) at > arch/x64/mmu.cc:37 > #6 <signal handler called> > #7 thread_main () at arch/x64/entry.S:113 > > It looks like what is happening is OSv thread #36 is executing a > pthread_create -> pthread_run sequence, and the pthread run function is > causing a page_fault in thread_main(). Since the threads start with > preemption disabled, we are tripping on the assert there. > Right. > 36 (0xffff800005cc5040) /usr/local/nu cpu0 queued > osv::rcu_lock_type::unlock() at include/osv/rcu.hh:179 vruntime 2.45014e+16 > 37 (0xffff8000089c0040) >/usr/local/n cpu0 running ?? at > arch/x64/entry.S:112 vruntime -8.38882e+14 > > (gdb) osv thread 37 > (gdb) where > #0 sched::thread::switch_to (this=0xffff8000089c0040) at > arch/x64/arch-switch.hh:108 > #1 0x00000000405b9e4c in sched::cpu::reschedule_from_interrupt > (this=0xffff80000001f040, called_from_yield=false, preempt_after=...) at > core/sched.cc:339 > #2 0x00000000405b9728 in sched::cpu::schedule () at core/sched.cc:228 > #3 0x000000004024cc8e in sched::preempt_enable () at > include/osv/sched.hh:1017 > #4 0x0000000040256c62 in osv::rcu_lock_type::unlock () at > include/osv/rcu.hh:179 > #5 0x000000004025776d in std::lock_guard<osv::rcu_lock_type>::~lock_guard > (this=0x2000006ffaa8, __in_chrg=<optimized out>) at > /misc/cross/el7.5-x86_64/gcc-7.3.0/x86_64-redhat-linux/include/c++/7.3.0/bits/std_mutex.h:168 > #6 0x000000004025725c in > lock_guard_for_with_lock<osv::rcu_lock_type>::~lock_guard_for_with_lock > (this=0x2000006ffaa8, __in_chrg=<optimized out>) at include/osv/mutex.h:88 > #7 0x00000000405bd16c in sched::thread::wake (this=0xffff8000089c0040) at > core/sched.cc:1164 > #8 0x00000000405bcd8f in sched::thread::start (this=0xffff8000089c0040) > at core/sched.cc:1079 > > gdb) osv thread 37 > (gdb) where > #0 thread_main () at arch/x64/entry.S:112 > I'm not sure we're seeing here the *same* thread. Maybe you're starting two threads at the same time? > > I do get the reasoning behind threads starting with preemption disabled ( > https://github.com/cloudius-systems/osv/commit/695375f65303e13df1b9de798577ee9a4f8f9892), > but what is being observed here is indeed a valid scenario that needs to be > handled? If not, please do share any pointers to correct my understanding. > Interesting. Theoretically, there is no reason why we didn't reach preempt_enable() and enabled preemption. If we didn't, there is one possible explanation: The thread's stack was allocated with mmap() without pre-population (MAP_STACK or MAP_POPULATE), so when the thread began to execute it had to page-fault its own stack, which failed. I'll try to prepare a test and a patch for this issue - if I can reproduce it (the patch itself should be trivial). Stay tuned. > > Thanks > Deepak > > -- > You received this message because you are subscribed to the Google Groups > "OSv Development" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/osv-dev/4b814e35-7384-4123-ac42-7523e292b29bn%40googlegroups.com > <https://groups.google.com/d/msgid/osv-dev/4b814e35-7384-4123-ac42-7523e292b29bn%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "OSv Development" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/CANEVyjt1NHQ5W6OSuixp2VCcAai-wSb%2BVuAHzdq0wyx6NUZQMw%40mail.gmail.com.
