native-example and python image works with qemu. It seems it only hangs on my custom tensorflow image. I'm only passing --verbose /python3 to run.py
On Thu, Jan 23, 2020 at 4:34 PM Waldek Kozaczuk <[email protected]> wrote: > Are you passing any parameters to ./scripts/run.py? I would also suggest > to specify just single vCPU - it should make debugging easier. > > Also does it hang with this app only or others as well? Can you try this: > ./scripts/build image=native-example > ./scripts/run.py > > Waldek > > On Thursday, January 23, 2020 at 5:28:43 PM UTC-5, zhiting zhu wrote: >> >> Unfortunately, I can't boot the vm with qemu. It's hanging at the >> beginning. >> >> I'm seeing this: >> bsd: initializing - done >> VFS: mounting ramfs at / >> VFS: mounting devfs at /dev >> net: initializing - done >> vga: Add VGA device instance >> eth0: ethernet address: 52:54:00:12:34:56 >> virtio-blk: Add blk device instances 0 as vblk0, devsize=1192516096 >> random: virtio-rng registered as a source. >> random: intel drng, rdrand registered as a source. >> random: <Software, Yarrow> initialized >> VFS: unmounting /dev >> VFS: mounting rofs at /rofs >> random: device unblocked. >> >> >> >> On Thu, Jan 23, 2020 at 3:25 PM zhiting zhu <[email protected]> >> wrote: >> >>> Here's the output log. See the file attached. >>> >>> On Thu, Jan 23, 2020 at 7:19 AM Nadav Har'El <[email protected]> wrote: >>> >>>> >>>> On Thu, Jan 23, 2020 at 2:23 PM Waldek Kozaczuk <[email protected]> >>>> wrote: >>>> >>>>> Can you send us the full output? I wonder if there are any warnings >>>>> before it? >>>>> >>>>> I see you are running this on firecracker. Can you run it under qemu >>>>> and connect to it with gdb and see if you get better stacktrace? >>>>> >>>>> You can add another debug statement like that: >>>>> >>>>> if (strcmp(" /lib/python3.6/google/protobuf/pyext/_message. >>>>> cpython-36m-x86_64-linux-gnu.so",pathname)==0 && i == 28) { >>>>> ... >>>>> // Put breakpoint here >>>>> } >>>>> >>>>> and try to see what statement causes the fault. Make sure to do 'osv >>>>> syms" to get as much debug info resolved as possible. (see >>>>> https://github.com/cloudius-systems/osv/wiki/Debugging-OSv). >>>>> >>>> >>>> I hope that "osv syms" will find the >>>> newly-loaded-but-not-yet-completely-loaded libraries. If it doesn't, maybe >>>> we can fix the order of when the array that "osv syms" uses gets written >>>> during loaded. >>>> >>>>> >>>>> I wonder if this has to with the order of initializing the ELF objects >>>>> when called by dlopen(). >>>>> >>>>> Waldek >>>>> >>>>> PS. If no more clue next step would be to add an app so we can build >>>>> it and reproduce it? >>>>> >>>>> On Wednesday, January 22, 2020 at 6:23:30 PM UTC-5, zhiting zhu wrote: >>>>>> >>>>>> Hey, >>>>>> >>>>>> I'm hitting this error: >>>>>> >>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing DT_INIT function >>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Finished executing DT_INIT >>>>>> function >>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 56 >>>>>> DT_INIT_ARRAYSZ functions >>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 0 init func >>>>>> 0x1000388ad500 >>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 1 init func >>>>>> 0x1000388ac8a0 >>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 2 init func >>>>>> 0x1000388ac8d0 >>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 3 init func >>>>>> 0x1000388ac900 >>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 4 init func >>>>>> 0x1000388ac930 >>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 5 init func >>>>>> 0x1000388ac960 >>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 6 init func >>>>>> 0x1000388ac990 >>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 7 init func >>>>>> 0x1000388ac9c0 >>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 8 init func >>>>>> 0x1000388ac9f0 >>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 9 init func >>>>>> 0x1000388aca20 >>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 10 init func >>>>>> 0x1000388aca50 >>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 11 init func >>>>>> 0x1000388aca80 >>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 12 init func >>>>>> 0x1000388acab0 >>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 13 init func >>>>>> 0x1000388acae0 >>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 14 init func >>>>>> 0x1000388acb10 >>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 15 init func >>>>>> 0x1000388acb40 >>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 16 init func >>>>>> 0x1000388acb70 >>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 17 init func >>>>>> 0x1000388acc50 >>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 18 init func >>>>>> 0x1000388acc80 >>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 19 init func >>>>>> 0x1000388accb0 >>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 20 init func >>>>>> 0x1000388acce0 >>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 21 init func >>>>>> 0x1000388acd10 >>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 22 init func >>>>>> 0x1000388acd40 >>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 23 init func >>>>>> 0x1000388acd70 >>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 24 init func >>>>>> 0x1000388acda0 >>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 25 init func >>>>>> 0x1000388acdd0 >>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 26 init func >>>>>> 0x1000388ace00 >>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 27 init func >>>>>> 0x1000388ace50 >>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 28 init func >>>>>> 0x1000388ace80 >>>>>> Aborted >>>>>> >>>>>> [backtrace] >>>>>> 0x0000000040463abb <osv::generate_signal(siginfo&, >>>>>> exception_frame*)+59> >>>>>> 0x0000000040463b2a <osv::handle_mmap_fault(unsigned long, int, >>>>>> exception_frame*)+26> >>>>>> 0x000000004032f3e9 <mmu::vm_fault(unsigned long, >>>>>> exception_frame*)+185> >>>>>> 0x000000004038f7b6 <page_fault+166> >>>>>> 0x000000004038e5f6 <???+1077470710> >>>>>> 0x0000000040341042 <elf::program::init_library(int, char**)+402> >>>>>> 0x00000000403473db >>>>>> <elf::program::get_library(std::__cxx11::basic_string<char, >>>>>> std::char_traits<char>, std::allocator<char> >, >>>>>> std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, >>>>>> std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, >>>>>> std::char_traits<char>, std::allocator<char> > > >, bool)+715> >>>>>> 0x0000000040462f19 <dlopen+153> >>>>>> 0x0000100000937228 <_PyImport_FindSharedFuncptr+376> >>>>>> 0x006567617373656c <???+1936942444> >>>>>> 2020-01-22T17:13:51.345740567 [anonymous-instance:ERROR:vmm/src/ >>>>>> lib.rs:1658] Failed to log metrics: Logger was not initialized. >>>>>> >>>>>> Is there any clue how to debug this? The function pointer seems to >>>>>> point to a valid address but I get a seg fault when executing it. >>>>>> >>>>>> Best, >>>>>> Zhiting >>>>>> >>>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "OSv Development" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/osv-dev/ecdc93a7-2a7e-4d39-87e9-6de15578b7df%40googlegroups.com >>>>> <https://groups.google.com/d/msgid/osv-dev/ecdc93a7-2a7e-4d39-87e9-6de15578b7df%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "OSv Development" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/osv-dev/CANEVyjsYXNroc3%3DeXB7Z0cV-rfeO9yUGpkDGZP87LDNunXHocQ%40mail.gmail.com >>>> <https://groups.google.com/d/msgid/osv-dev/CANEVyjsYXNroc3%3DeXB7Z0cV-rfeO9yUGpkDGZP87LDNunXHocQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- > You received this message because you are subscribed to the Google Groups > "OSv Development" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/osv-dev/096465a1-ffbc-45f9-b469-fa5c97c752f7%40googlegroups.com > <https://groups.google.com/d/msgid/osv-dev/096465a1-ffbc-45f9-b469-fa5c97c752f7%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "OSv Development" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/CA%2B3q14wqekD_SyadpYfmMxdgYcbK2g6ZUrn8m7mwmM0VAvFUCg%40mail.gmail.com.
