Yeah, Zfs image doesn't hang. I don't know why rofs image hangs. I need to increase the qemu memory in upload_manifest.py otherwise it hangs on building zfs images.
On Thu, Jan 23, 2020 at 5:17 PM Waldek Kozaczuk <[email protected]> wrote: > It seems like it got stuck while trying to mount the filesystem. The next > boot message would normally be 'VFS: mounting devfs at /dev". > > I wonder if the image (usr.img) is somehow locked or something. Have you > tried to rebuild the image? Try zfs. > > Waldek > > On Thursday, January 23, 2020 at 5:54:08 PM UTC-5, zhiting zhu wrote: >> >> native-example and python image works with qemu. It seems it only hangs >> on my custom tensorflow image. I'm only passing --verbose /python3 to >> run.py >> >> >> On Thu, Jan 23, 2020 at 4:34 PM Waldek Kozaczuk <[email protected]> >> wrote: >> >>> Are you passing any parameters to ./scripts/run.py? I would also suggest >>> to specify just single vCPU - it should make debugging easier. >>> >>> Also does it hang with this app only or others as well? Can you try this: >>> ./scripts/build image=native-example >>> ./scripts/run.py >>> >>> Waldek >>> >>> On Thursday, January 23, 2020 at 5:28:43 PM UTC-5, zhiting zhu wrote: >>>> >>>> Unfortunately, I can't boot the vm with qemu. It's hanging at the >>>> beginning. >>>> >>>> I'm seeing this: >>>> bsd: initializing - done >>>> VFS: mounting ramfs at / >>>> VFS: mounting devfs at /dev >>>> net: initializing - done >>>> vga: Add VGA device instance >>>> eth0: ethernet address: 52:54:00:12:34:56 >>>> virtio-blk: Add blk device instances 0 as vblk0, devsize=1192516096 >>>> random: virtio-rng registered as a source. >>>> random: intel drng, rdrand registered as a source. >>>> random: <Software, Yarrow> initialized >>>> VFS: unmounting /dev >>>> VFS: mounting rofs at /rofs >>>> random: device unblocked. >>>> >>>> >>>> >>>> On Thu, Jan 23, 2020 at 3:25 PM zhiting zhu <[email protected]> >>>> wrote: >>>> >>>>> Here's the output log. See the file attached. >>>>> >>>>> On Thu, Jan 23, 2020 at 7:19 AM Nadav Har'El <[email protected]> >>>>> wrote: >>>>> >>>>>> >>>>>> On Thu, Jan 23, 2020 at 2:23 PM Waldek Kozaczuk <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Can you send us the full output? I wonder if there are any warnings >>>>>>> before it? >>>>>>> >>>>>>> I see you are running this on firecracker. Can you run it under qemu >>>>>>> and connect to it with gdb and see if you get better stacktrace? >>>>>>> >>>>>>> You can add another debug statement like that: >>>>>>> >>>>>>> if (strcmp(" /lib/python3.6/google/protobuf/pyext/_message. >>>>>>> cpython-36m-x86_64-linux-gnu.so",pathname)==0 && i == 28) { >>>>>>> ... >>>>>>> // Put breakpoint here >>>>>>> } >>>>>>> >>>>>>> and try to see what statement causes the fault. Make sure to do 'osv >>>>>>> syms" to get as much debug info resolved as possible. (see >>>>>>> https://github.com/cloudius-systems/osv/wiki/Debugging-OSv). >>>>>>> >>>>>> >>>>>> I hope that "osv syms" will find the >>>>>> newly-loaded-but-not-yet-completely-loaded libraries. If it doesn't, >>>>>> maybe >>>>>> we can fix the order of when the array that "osv syms" uses gets written >>>>>> during loaded. >>>>>> >>>>>>> >>>>>>> I wonder if this has to with the order of initializing the ELF >>>>>>> objects when called by dlopen(). >>>>>>> >>>>>>> Waldek >>>>>>> >>>>>>> PS. If no more clue next step would be to add an app so we can build >>>>>>> it and reproduce it? >>>>>>> >>>>>>> On Wednesday, January 22, 2020 at 6:23:30 PM UTC-5, zhiting zhu >>>>>>> wrote: >>>>>>>> >>>>>>>> Hey, >>>>>>>> >>>>>>>> I'm hitting this error: >>>>>>>> >>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing DT_INIT >>>>>>>> function >>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Finished executing >>>>>>>> DT_INIT function >>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 56 >>>>>>>> DT_INIT_ARRAYSZ functions >>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 0 init func >>>>>>>> 0x1000388ad500 >>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 1 init func >>>>>>>> 0x1000388ac8a0 >>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 2 init func >>>>>>>> 0x1000388ac8d0 >>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 3 init func >>>>>>>> 0x1000388ac900 >>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 4 init func >>>>>>>> 0x1000388ac930 >>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 5 init func >>>>>>>> 0x1000388ac960 >>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 6 init func >>>>>>>> 0x1000388ac990 >>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 7 init func >>>>>>>> 0x1000388ac9c0 >>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 8 init func >>>>>>>> 0x1000388ac9f0 >>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 9 init func >>>>>>>> 0x1000388aca20 >>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 10 init func >>>>>>>> 0x1000388aca50 >>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 11 init func >>>>>>>> 0x1000388aca80 >>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 12 init func >>>>>>>> 0x1000388acab0 >>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 13 init func >>>>>>>> 0x1000388acae0 >>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 14 init func >>>>>>>> 0x1000388acb10 >>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 15 init func >>>>>>>> 0x1000388acb40 >>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 16 init func >>>>>>>> 0x1000388acb70 >>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 17 init func >>>>>>>> 0x1000388acc50 >>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 18 init func >>>>>>>> 0x1000388acc80 >>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 19 init func >>>>>>>> 0x1000388accb0 >>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 20 init func >>>>>>>> 0x1000388acce0 >>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 21 init func >>>>>>>> 0x1000388acd10 >>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 22 init func >>>>>>>> 0x1000388acd40 >>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 23 init func >>>>>>>> 0x1000388acd70 >>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 24 init func >>>>>>>> 0x1000388acda0 >>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 25 init func >>>>>>>> 0x1000388acdd0 >>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 26 init func >>>>>>>> 0x1000388ace00 >>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 27 init func >>>>>>>> 0x1000388ace50 >>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 28 init func >>>>>>>> 0x1000388ace80 >>>>>>>> Aborted >>>>>>>> >>>>>>>> [backtrace] >>>>>>>> 0x0000000040463abb <osv::generate_signal(siginfo&, >>>>>>>> exception_frame*)+59> >>>>>>>> 0x0000000040463b2a <osv::handle_mmap_fault(unsigned long, int, >>>>>>>> exception_frame*)+26> >>>>>>>> 0x000000004032f3e9 <mmu::vm_fault(unsigned long, >>>>>>>> exception_frame*)+185> >>>>>>>> 0x000000004038f7b6 <page_fault+166> >>>>>>>> 0x000000004038e5f6 <???+1077470710> >>>>>>>> 0x0000000040341042 <elf::program::init_library(int, char**)+402> >>>>>>>> 0x00000000403473db >>>>>>>> <elf::program::get_library(std::__cxx11::basic_string<char, >>>>>>>> std::char_traits<char>, std::allocator<char> >, >>>>>>>> std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, >>>>>>>> std::allocator<char> >, std::allocator<std::__cxx11::basic_string<char, >>>>>>>> std::char_traits<char>, std::allocator<char> > > >, bool)+715> >>>>>>>> 0x0000000040462f19 <dlopen+153> >>>>>>>> 0x0000100000937228 <_PyImport_FindSharedFuncptr+376> >>>>>>>> 0x006567617373656c <???+1936942444> >>>>>>>> 2020-01-22T17:13:51.345740567 [anonymous-instance:ERROR:vmm/src/ >>>>>>>> lib.rs:1658] Failed to log metrics: Logger was not initialized. >>>>>>>> >>>>>>>> Is there any clue how to debug this? The function pointer seems to >>>>>>>> point to a valid address but I get a seg fault when executing it. >>>>>>>> >>>>>>>> Best, >>>>>>>> Zhiting >>>>>>>> >>>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "OSv Development" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/osv-dev/ecdc93a7-2a7e-4d39-87e9-6de15578b7df%40googlegroups.com >>>>>>> <https://groups.google.com/d/msgid/osv-dev/ecdc93a7-2a7e-4d39-87e9-6de15578b7df%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "OSv Development" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/osv-dev/CANEVyjsYXNroc3%3DeXB7Z0cV-rfeO9yUGpkDGZP87LDNunXHocQ%40mail.gmail.com >>>>>> <https://groups.google.com/d/msgid/osv-dev/CANEVyjsYXNroc3%3DeXB7Z0cV-rfeO9yUGpkDGZP87LDNunXHocQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "OSv Development" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/osv-dev/096465a1-ffbc-45f9-b469-fa5c97c752f7%40googlegroups.com >>> <https://groups.google.com/d/msgid/osv-dev/096465a1-ffbc-45f9-b469-fa5c97c752f7%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to the Google Groups > "OSv Development" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/osv-dev/173619bc-4cdf-4408-8b5d-541693d08de6%40googlegroups.com > <https://groups.google.com/d/msgid/osv-dev/173619bc-4cdf-4408-8b5d-541693d08de6%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "OSv Development" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/CA%2B3q14yaCeFeSgQQMC7rYFqc4aWh%2Bt1fjEP4dU-aHKW3%2B7Y5-Q%40mail.gmail.com.
