BTW how big is you image? I wounder if ROFS hangs because it runs out of memory when trying to load files into memory.
On Thursday, January 23, 2020 at 6:24:21 PM UTC-5, zhiting zhu wrote: > > Yeah, Zfs image doesn't hang. I don't know why rofs image hangs. I need to > increase the qemu memory in upload_manifest.py otherwise it hangs on > building zfs images. > > On Thu, Jan 23, 2020 at 5:17 PM Waldek Kozaczuk <[email protected] > <javascript:>> wrote: > >> It seems like it got stuck while trying to mount the filesystem. The next >> boot message would normally be 'VFS: mounting devfs at /dev". >> >> I wonder if the image (usr.img) is somehow locked or something. Have you >> tried to rebuild the image? Try zfs. >> >> Waldek >> >> On Thursday, January 23, 2020 at 5:54:08 PM UTC-5, zhiting zhu wrote: >>> >>> native-example and python image works with qemu. It seems it only hangs >>> on my custom tensorflow image. I'm only passing --verbose /python3 to >>> run.py >>> >>> >>> On Thu, Jan 23, 2020 at 4:34 PM Waldek Kozaczuk <[email protected]> >>> wrote: >>> >>>> Are you passing any parameters to ./scripts/run.py? I would also >>>> suggest to specify just single vCPU - it should make debugging easier. >>>> >>>> Also does it hang with this app only or others as well? Can you try >>>> this: >>>> ./scripts/build image=native-example >>>> ./scripts/run.py >>>> >>>> Waldek >>>> >>>> On Thursday, January 23, 2020 at 5:28:43 PM UTC-5, zhiting zhu wrote: >>>>> >>>>> Unfortunately, I can't boot the vm with qemu. It's hanging at the >>>>> beginning. >>>>> >>>>> I'm seeing this: >>>>> bsd: initializing - done >>>>> VFS: mounting ramfs at / >>>>> VFS: mounting devfs at /dev >>>>> net: initializing - done >>>>> vga: Add VGA device instance >>>>> eth0: ethernet address: 52:54:00:12:34:56 >>>>> virtio-blk: Add blk device instances 0 as vblk0, devsize=1192516096 >>>>> random: virtio-rng registered as a source. >>>>> random: intel drng, rdrand registered as a source. >>>>> random: <Software, Yarrow> initialized >>>>> VFS: unmounting /dev >>>>> VFS: mounting rofs at /rofs >>>>> random: device unblocked. >>>>> >>>>> >>>>> >>>>> On Thu, Jan 23, 2020 at 3:25 PM zhiting zhu <[email protected]> >>>>> wrote: >>>>> >>>>>> Here's the output log. See the file attached. >>>>>> >>>>>> On Thu, Jan 23, 2020 at 7:19 AM Nadav Har'El <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> >>>>>>> On Thu, Jan 23, 2020 at 2:23 PM Waldek Kozaczuk <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Can you send us the full output? I wonder if there are any warnings >>>>>>>> before it? >>>>>>>> >>>>>>>> I see you are running this on firecracker. Can you run it under >>>>>>>> qemu and connect to it with gdb and see if you get better stacktrace? >>>>>>>> >>>>>>>> You can add another debug statement like that: >>>>>>>> >>>>>>>> if (strcmp(" /lib/python3.6/google/protobuf/pyext/_message. >>>>>>>> cpython-36m-x86_64-linux-gnu.so",pathname)==0 && i == 28) { >>>>>>>> ... >>>>>>>> // Put breakpoint here >>>>>>>> } >>>>>>>> >>>>>>>> and try to see what statement causes the fault. Make sure to do >>>>>>>> 'osv syms" to get as much debug info resolved as possible. (see >>>>>>>> https://github.com/cloudius-systems/osv/wiki/Debugging-OSv). >>>>>>>> >>>>>>> >>>>>>> I hope that "osv syms" will find the >>>>>>> newly-loaded-but-not-yet-completely-loaded libraries. If it doesn't, >>>>>>> maybe >>>>>>> we can fix the order of when the array that "osv syms" uses gets >>>>>>> written >>>>>>> during loaded. >>>>>>> >>>>>>>> >>>>>>>> I wonder if this has to with the order of initializing the ELF >>>>>>>> objects when called by dlopen(). >>>>>>>> >>>>>>>> Waldek >>>>>>>> >>>>>>>> PS. If no more clue next step would be to add an app so we can >>>>>>>> build it and reproduce it? >>>>>>>> >>>>>>>> On Wednesday, January 22, 2020 at 6:23:30 PM UTC-5, zhiting zhu >>>>>>>> wrote: >>>>>>>>> >>>>>>>>> Hey, >>>>>>>>> >>>>>>>>> I'm hitting this error: >>>>>>>>> >>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing DT_INIT >>>>>>>>> function >>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Finished executing >>>>>>>>> DT_INIT function >>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 56 >>>>>>>>> DT_INIT_ARRAYSZ functions >>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 0 init func >>>>>>>>> 0x1000388ad500 >>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 1 init func >>>>>>>>> 0x1000388ac8a0 >>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 2 init func >>>>>>>>> 0x1000388ac8d0 >>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 3 init func >>>>>>>>> 0x1000388ac900 >>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 4 init func >>>>>>>>> 0x1000388ac930 >>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 5 init func >>>>>>>>> 0x1000388ac960 >>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 6 init func >>>>>>>>> 0x1000388ac990 >>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 7 init func >>>>>>>>> 0x1000388ac9c0 >>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 8 init func >>>>>>>>> 0x1000388ac9f0 >>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 9 init func >>>>>>>>> 0x1000388aca20 >>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 10 init func >>>>>>>>> 0x1000388aca50 >>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 11 init func >>>>>>>>> 0x1000388aca80 >>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 12 init func >>>>>>>>> 0x1000388acab0 >>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 13 init func >>>>>>>>> 0x1000388acae0 >>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 14 init func >>>>>>>>> 0x1000388acb10 >>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 15 init func >>>>>>>>> 0x1000388acb40 >>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 16 init func >>>>>>>>> 0x1000388acb70 >>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 17 init func >>>>>>>>> 0x1000388acc50 >>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 18 init func >>>>>>>>> 0x1000388acc80 >>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 19 init func >>>>>>>>> 0x1000388accb0 >>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 20 init func >>>>>>>>> 0x1000388acce0 >>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 21 init func >>>>>>>>> 0x1000388acd10 >>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 22 init func >>>>>>>>> 0x1000388acd40 >>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 23 init func >>>>>>>>> 0x1000388acd70 >>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 24 init func >>>>>>>>> 0x1000388acda0 >>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 25 init func >>>>>>>>> 0x1000388acdd0 >>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 26 init func >>>>>>>>> 0x1000388ace00 >>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 27 init func >>>>>>>>> 0x1000388ace50 >>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 28 init func >>>>>>>>> 0x1000388ace80 >>>>>>>>> Aborted >>>>>>>>> >>>>>>>>> [backtrace] >>>>>>>>> 0x0000000040463abb <osv::generate_signal(siginfo&, >>>>>>>>> exception_frame*)+59> >>>>>>>>> 0x0000000040463b2a <osv::handle_mmap_fault(unsigned long, int, >>>>>>>>> exception_frame*)+26> >>>>>>>>> 0x000000004032f3e9 <mmu::vm_fault(unsigned long, >>>>>>>>> exception_frame*)+185> >>>>>>>>> 0x000000004038f7b6 <page_fault+166> >>>>>>>>> 0x000000004038e5f6 <???+1077470710> >>>>>>>>> 0x0000000040341042 <elf::program::init_library(int, char**)+402> >>>>>>>>> 0x00000000403473db >>>>>>>>> <elf::program::get_library(std::__cxx11::basic_string<char, >>>>>>>>> std::char_traits<char>, std::allocator<char> >, >>>>>>>>> std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, >>>>>>>>> std::allocator<char> >, >>>>>>>>> std::allocator<std::__cxx11::basic_string<char, >>>>>>>>> std::char_traits<char>, std::allocator<char> > > >, bool)+715> >>>>>>>>> 0x0000000040462f19 <dlopen+153> >>>>>>>>> 0x0000100000937228 <_PyImport_FindSharedFuncptr+376> >>>>>>>>> 0x006567617373656c <???+1936942444> >>>>>>>>> 2020-01-22T17:13:51.345740567 [anonymous-instance:ERROR:vmm/src/ >>>>>>>>> lib.rs:1658] Failed to log metrics: Logger was not initialized. >>>>>>>>> >>>>>>>>> Is there any clue how to debug this? The function pointer seems to >>>>>>>>> point to a valid address but I get a seg fault when executing it. >>>>>>>>> >>>>>>>>> Best, >>>>>>>>> Zhiting >>>>>>>>> >>>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "OSv Development" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/osv-dev/ecdc93a7-2a7e-4d39-87e9-6de15578b7df%40googlegroups.com >>>>>>>> >>>>>>>> <https://groups.google.com/d/msgid/osv-dev/ecdc93a7-2a7e-4d39-87e9-6de15578b7df%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> >>>>>>> -- >>>>>>> You received this message because you are subscribed to the Google >>>>>>> Groups "OSv Development" group. >>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>> send an email to [email protected]. >>>>>>> To view this discussion on the web visit >>>>>>> https://groups.google.com/d/msgid/osv-dev/CANEVyjsYXNroc3%3DeXB7Z0cV-rfeO9yUGpkDGZP87LDNunXHocQ%40mail.gmail.com >>>>>>> >>>>>>> <https://groups.google.com/d/msgid/osv-dev/CANEVyjsYXNroc3%3DeXB7Z0cV-rfeO9yUGpkDGZP87LDNunXHocQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>> . >>>>>>> >>>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "OSv Development" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/osv-dev/096465a1-ffbc-45f9-b469-fa5c97c752f7%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/osv-dev/096465a1-ffbc-45f9-b469-fa5c97c752f7%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "OSv Development" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/osv-dev/173619bc-4cdf-4408-8b5d-541693d08de6%40googlegroups.com >> >> <https://groups.google.com/d/msgid/osv-dev/173619bc-4cdf-4408-8b5d-541693d08de6%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "OSv Development" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/36872cb0-9673-4756-ab6d-df94a19dba4b%40googlegroups.com.
