It seems like it got stuck while trying to mount the filesystem. The next boot message would normally be 'VFS: mounting devfs at /dev".
I wonder if the image (usr.img) is somehow locked or something. Have you tried to rebuild the image? Try zfs. Waldek On Thursday, January 23, 2020 at 5:54:08 PM UTC-5, zhiting zhu wrote: > > native-example and python image works with qemu. It seems it only hangs on > my custom tensorflow image. I'm only passing --verbose /python3 to run.py > > > On Thu, Jan 23, 2020 at 4:34 PM Waldek Kozaczuk <[email protected] > <javascript:>> wrote: > >> Are you passing any parameters to ./scripts/run.py? I would also suggest >> to specify just single vCPU - it should make debugging easier. >> >> Also does it hang with this app only or others as well? Can you try this: >> ./scripts/build image=native-example >> ./scripts/run.py >> >> Waldek >> >> On Thursday, January 23, 2020 at 5:28:43 PM UTC-5, zhiting zhu wrote: >>> >>> Unfortunately, I can't boot the vm with qemu. It's hanging at the >>> beginning. >>> >>> I'm seeing this: >>> bsd: initializing - done >>> VFS: mounting ramfs at / >>> VFS: mounting devfs at /dev >>> net: initializing - done >>> vga: Add VGA device instance >>> eth0: ethernet address: 52:54:00:12:34:56 >>> virtio-blk: Add blk device instances 0 as vblk0, devsize=1192516096 >>> random: virtio-rng registered as a source. >>> random: intel drng, rdrand registered as a source. >>> random: <Software, Yarrow> initialized >>> VFS: unmounting /dev >>> VFS: mounting rofs at /rofs >>> random: device unblocked. >>> >>> >>> >>> On Thu, Jan 23, 2020 at 3:25 PM zhiting zhu <[email protected]> >>> wrote: >>> >>>> Here's the output log. See the file attached. >>>> >>>> On Thu, Jan 23, 2020 at 7:19 AM Nadav Har'El <[email protected]> wrote: >>>> >>>>> >>>>> On Thu, Jan 23, 2020 at 2:23 PM Waldek Kozaczuk <[email protected]> >>>>> wrote: >>>>> >>>>>> Can you send us the full output? I wonder if there are any warnings >>>>>> before it? >>>>>> >>>>>> I see you are running this on firecracker. Can you run it under qemu >>>>>> and connect to it with gdb and see if you get better stacktrace? >>>>>> >>>>>> You can add another debug statement like that: >>>>>> >>>>>> if (strcmp(" /lib/python3.6/google/protobuf/pyext/_message. >>>>>> cpython-36m-x86_64-linux-gnu.so",pathname)==0 && i == 28) { >>>>>> ... >>>>>> // Put breakpoint here >>>>>> } >>>>>> >>>>>> and try to see what statement causes the fault. Make sure to do 'osv >>>>>> syms" to get as much debug info resolved as possible. (see >>>>>> https://github.com/cloudius-systems/osv/wiki/Debugging-OSv). >>>>>> >>>>> >>>>> I hope that "osv syms" will find the >>>>> newly-loaded-but-not-yet-completely-loaded libraries. If it doesn't, >>>>> maybe >>>>> we can fix the order of when the array that "osv syms" uses gets written >>>>> during loaded. >>>>> >>>>>> >>>>>> I wonder if this has to with the order of initializing the ELF >>>>>> objects when called by dlopen(). >>>>>> >>>>>> Waldek >>>>>> >>>>>> PS. If no more clue next step would be to add an app so we can build >>>>>> it and reproduce it? >>>>>> >>>>>> On Wednesday, January 22, 2020 at 6:23:30 PM UTC-5, zhiting zhu wrote: >>>>>>> >>>>>>> Hey, >>>>>>> >>>>>>> I'm hitting this error: >>>>>>> >>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing DT_INIT function >>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Finished executing >>>>>>> DT_INIT function >>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 56 >>>>>>> DT_INIT_ARRAYSZ functions >>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 0 init func >>>>>>> 0x1000388ad500 >>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 1 init func >>>>>>> 0x1000388ac8a0 >>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 2 init func >>>>>>> 0x1000388ac8d0 >>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 3 init func >>>>>>> 0x1000388ac900 >>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 4 init func >>>>>>> 0x1000388ac930 >>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 5 init func >>>>>>> 0x1000388ac960 >>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 6 init func >>>>>>> 0x1000388ac990 >>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 7 init func >>>>>>> 0x1000388ac9c0 >>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 8 init func >>>>>>> 0x1000388ac9f0 >>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 9 init func >>>>>>> 0x1000388aca20 >>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 10 init func >>>>>>> 0x1000388aca50 >>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 11 init func >>>>>>> 0x1000388aca80 >>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 12 init func >>>>>>> 0x1000388acab0 >>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 13 init func >>>>>>> 0x1000388acae0 >>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 14 init func >>>>>>> 0x1000388acb10 >>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 15 init func >>>>>>> 0x1000388acb40 >>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 16 init func >>>>>>> 0x1000388acb70 >>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 17 init func >>>>>>> 0x1000388acc50 >>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 18 init func >>>>>>> 0x1000388acc80 >>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 19 init func >>>>>>> 0x1000388accb0 >>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 20 init func >>>>>>> 0x1000388acce0 >>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 21 init func >>>>>>> 0x1000388acd10 >>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 22 init func >>>>>>> 0x1000388acd40 >>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 23 init func >>>>>>> 0x1000388acd70 >>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 24 init func >>>>>>> 0x1000388acda0 >>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 25 init func >>>>>>> 0x1000388acdd0 >>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 26 init func >>>>>>> 0x1000388ace00 >>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 27 init func >>>>>>> 0x1000388ace50 >>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 28 init func >>>>>>> 0x1000388ace80 >>>>>>> Aborted >>>>>>> >>>>>>> [backtrace] >>>>>>> 0x0000000040463abb <osv::generate_signal(siginfo&, >>>>>>> exception_frame*)+59> >>>>>>> 0x0000000040463b2a <osv::handle_mmap_fault(unsigned long, int, >>>>>>> exception_frame*)+26> >>>>>>> 0x000000004032f3e9 <mmu::vm_fault(unsigned long, >>>>>>> exception_frame*)+185> >>>>>>> 0x000000004038f7b6 <page_fault+166> >>>>>>> 0x000000004038e5f6 <???+1077470710> >>>>>>> 0x0000000040341042 <elf::program::init_library(int, char**)+402> >>>>>>> 0x00000000403473db >>>>>>> <elf::program::get_library(std::__cxx11::basic_string<char, >>>>>>> std::char_traits<char>, std::allocator<char> >, >>>>>>> std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, >>>>>>> std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, >>>>>>> std::char_traits<char>, std::allocator<char> > > >, bool)+715> >>>>>>> 0x0000000040462f19 <dlopen+153> >>>>>>> 0x0000100000937228 <_PyImport_FindSharedFuncptr+376> >>>>>>> 0x006567617373656c <???+1936942444> >>>>>>> 2020-01-22T17:13:51.345740567 [anonymous-instance:ERROR:vmm/src/ >>>>>>> lib.rs:1658] Failed to log metrics: Logger was not initialized. >>>>>>> >>>>>>> Is there any clue how to debug this? The function pointer seems to >>>>>>> point to a valid address but I get a seg fault when executing it. >>>>>>> >>>>>>> Best, >>>>>>> Zhiting >>>>>>> >>>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "OSv Development" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/osv-dev/ecdc93a7-2a7e-4d39-87e9-6de15578b7df%40googlegroups.com >>>>>> >>>>>> <https://groups.google.com/d/msgid/osv-dev/ecdc93a7-2a7e-4d39-87e9-6de15578b7df%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "OSv Development" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/osv-dev/CANEVyjsYXNroc3%3DeXB7Z0cV-rfeO9yUGpkDGZP87LDNunXHocQ%40mail.gmail.com >>>>> >>>>> <https://groups.google.com/d/msgid/osv-dev/CANEVyjsYXNroc3%3DeXB7Z0cV-rfeO9yUGpkDGZP87LDNunXHocQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- >> You received this message because you are subscribed to the Google Groups >> "OSv Development" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/osv-dev/096465a1-ffbc-45f9-b469-fa5c97c752f7%40googlegroups.com >> >> <https://groups.google.com/d/msgid/osv-dev/096465a1-ffbc-45f9-b469-fa5c97c752f7%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "OSv Development" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/173619bc-4cdf-4408-8b5d-541693d08de6%40googlegroups.com.
