Yeah that is pretty big image given that the default memory size if 4GB. I wonder if increasing the memory to something like 8GB would make rofs work. rofs is not very memory efficient - see https://github.com/cloudius-systems/osv/issues/979. This is one of the issues I would want to work next.
On Thursday, January 23, 2020 at 6:28:38 PM UTC-5, zhiting zhu wrote: > > The image is 2.5 G. > > On Thu, Jan 23, 2020 at 5:26 PM Waldek Kozaczuk <[email protected] > <javascript:>> wrote: > >> BTW how big is you image? >> >> I wounder if ROFS hangs because it runs out of memory when trying to load >> files into memory. >> >> On Thursday, January 23, 2020 at 6:24:21 PM UTC-5, zhiting zhu wrote: >>> >>> Yeah, Zfs image doesn't hang. I don't know why rofs image hangs. I need >>> to increase the qemu memory in upload_manifest.py otherwise it hangs on >>> building zfs images. >>> >>> On Thu, Jan 23, 2020 at 5:17 PM Waldek Kozaczuk <[email protected]> >>> wrote: >>> >>>> It seems like it got stuck while trying to mount the filesystem. The >>>> next boot message would normally be 'VFS: mounting devfs at /dev". >>>> >>>> I wonder if the image (usr.img) is somehow locked or something. Have >>>> you tried to rebuild the image? Try zfs. >>>> >>>> Waldek >>>> >>>> On Thursday, January 23, 2020 at 5:54:08 PM UTC-5, zhiting zhu wrote: >>>>> >>>>> native-example and python image works with qemu. It seems it only >>>>> hangs on my custom tensorflow image. I'm only passing --verbose /python3 >>>>> to >>>>> run.py >>>>> >>>>> >>>>> On Thu, Jan 23, 2020 at 4:34 PM Waldek Kozaczuk <[email protected]> >>>>> wrote: >>>>> >>>>>> Are you passing any parameters to ./scripts/run.py? I would also >>>>>> suggest to specify just single vCPU - it should make debugging easier. >>>>>> >>>>>> Also does it hang with this app only or others as well? Can you try >>>>>> this: >>>>>> ./scripts/build image=native-example >>>>>> ./scripts/run.py >>>>>> >>>>>> Waldek >>>>>> >>>>>> On Thursday, January 23, 2020 at 5:28:43 PM UTC-5, zhiting zhu wrote: >>>>>>> >>>>>>> Unfortunately, I can't boot the vm with qemu. It's hanging at the >>>>>>> beginning. >>>>>>> >>>>>>> I'm seeing this: >>>>>>> bsd: initializing - done >>>>>>> VFS: mounting ramfs at / >>>>>>> VFS: mounting devfs at /dev >>>>>>> net: initializing - done >>>>>>> vga: Add VGA device instance >>>>>>> eth0: ethernet address: 52:54:00:12:34:56 >>>>>>> virtio-blk: Add blk device instances 0 as vblk0, devsize=1192516096 >>>>>>> random: virtio-rng registered as a source. >>>>>>> random: intel drng, rdrand registered as a source. >>>>>>> random: <Software, Yarrow> initialized >>>>>>> VFS: unmounting /dev >>>>>>> VFS: mounting rofs at /rofs >>>>>>> random: device unblocked. >>>>>>> >>>>>>> >>>>>>> >>>>>>> On Thu, Jan 23, 2020 at 3:25 PM zhiting zhu <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> Here's the output log. See the file attached. >>>>>>>> >>>>>>>> On Thu, Jan 23, 2020 at 7:19 AM Nadav Har'El <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> >>>>>>>>> On Thu, Jan 23, 2020 at 2:23 PM Waldek Kozaczuk < >>>>>>>>> [email protected]> wrote: >>>>>>>>> >>>>>>>>>> Can you send us the full output? I wonder if there are any >>>>>>>>>> warnings before it? >>>>>>>>>> >>>>>>>>>> I see you are running this on firecracker. Can you run it under >>>>>>>>>> qemu and connect to it with gdb and see if you get better stacktrace? >>>>>>>>>> >>>>>>>>>> You can add another debug statement like that: >>>>>>>>>> >>>>>>>>>> if (strcmp(" /lib/python3.6/google/protobuf/pyext/_message. >>>>>>>>>> cpython-36m-x86_64-linux-gnu.so",pathname)==0 && i == 28) { >>>>>>>>>> ... >>>>>>>>>> // Put breakpoint here >>>>>>>>>> } >>>>>>>>>> >>>>>>>>>> and try to see what statement causes the fault. Make sure to do >>>>>>>>>> 'osv syms" to get as much debug info resolved as possible. (see >>>>>>>>>> https://github.com/cloudius-systems/osv/wiki/Debugging-OSv). >>>>>>>>>> >>>>>>>>> >>>>>>>>> I hope that "osv syms" will find the >>>>>>>>> newly-loaded-but-not-yet-completely-loaded libraries. If it doesn't, >>>>>>>>> maybe >>>>>>>>> we can fix the order of when the array that "osv syms" uses gets >>>>>>>>> written >>>>>>>>> during loaded. >>>>>>>>> >>>>>>>>>> >>>>>>>>>> I wonder if this has to with the order of initializing the ELF >>>>>>>>>> objects when called by dlopen(). >>>>>>>>>> >>>>>>>>>> Waldek >>>>>>>>>> >>>>>>>>>> PS. If no more clue next step would be to add an app so we can >>>>>>>>>> build it and reproduce it? >>>>>>>>>> >>>>>>>>>> On Wednesday, January 22, 2020 at 6:23:30 PM UTC-5, zhiting zhu >>>>>>>>>> wrote: >>>>>>>>>>> >>>>>>>>>>> Hey, >>>>>>>>>>> >>>>>>>>>>> I'm hitting this error: >>>>>>>>>>> >>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing DT_INIT >>>>>>>>>>> function >>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Finished executing >>>>>>>>>>> DT_INIT function >>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 56 >>>>>>>>>>> DT_INIT_ARRAYSZ functions >>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 0 init func >>>>>>>>>>> 0x1000388ad500 >>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 1 init func >>>>>>>>>>> 0x1000388ac8a0 >>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 2 init func >>>>>>>>>>> 0x1000388ac8d0 >>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 3 init func >>>>>>>>>>> 0x1000388ac900 >>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 4 init func >>>>>>>>>>> 0x1000388ac930 >>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 5 init func >>>>>>>>>>> 0x1000388ac960 >>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 6 init func >>>>>>>>>>> 0x1000388ac990 >>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 7 init func >>>>>>>>>>> 0x1000388ac9c0 >>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 8 init func >>>>>>>>>>> 0x1000388ac9f0 >>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 9 init func >>>>>>>>>>> 0x1000388aca20 >>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 10 init >>>>>>>>>>> func 0x1000388aca50 >>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 11 init >>>>>>>>>>> func 0x1000388aca80 >>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 12 init >>>>>>>>>>> func 0x1000388acab0 >>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 13 init >>>>>>>>>>> func 0x1000388acae0 >>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 14 init >>>>>>>>>>> func 0x1000388acb10 >>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 15 init >>>>>>>>>>> func 0x1000388acb40 >>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 16 init >>>>>>>>>>> func 0x1000388acb70 >>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 17 init >>>>>>>>>>> func 0x1000388acc50 >>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 18 init >>>>>>>>>>> func 0x1000388acc80 >>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 19 init >>>>>>>>>>> func 0x1000388accb0 >>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 20 init >>>>>>>>>>> func 0x1000388acce0 >>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 21 init >>>>>>>>>>> func 0x1000388acd10 >>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 22 init >>>>>>>>>>> func 0x1000388acd40 >>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 23 init >>>>>>>>>>> func 0x1000388acd70 >>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 24 init >>>>>>>>>>> func 0x1000388acda0 >>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 25 init >>>>>>>>>>> func 0x1000388acdd0 >>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 26 init >>>>>>>>>>> func 0x1000388ace00 >>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 27 init >>>>>>>>>>> func 0x1000388ace50 >>>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 28 init >>>>>>>>>>> func 0x1000388ace80 >>>>>>>>>>> Aborted >>>>>>>>>>> >>>>>>>>>>> [backtrace] >>>>>>>>>>> 0x0000000040463abb <osv::generate_signal(siginfo&, >>>>>>>>>>> exception_frame*)+59> >>>>>>>>>>> 0x0000000040463b2a <osv::handle_mmap_fault(unsigned long, int, >>>>>>>>>>> exception_frame*)+26> >>>>>>>>>>> 0x000000004032f3e9 <mmu::vm_fault(unsigned long, >>>>>>>>>>> exception_frame*)+185> >>>>>>>>>>> 0x000000004038f7b6 <page_fault+166> >>>>>>>>>>> 0x000000004038e5f6 <???+1077470710> >>>>>>>>>>> 0x0000000040341042 <elf::program::init_library(int, char**)+402> >>>>>>>>>>> 0x00000000403473db >>>>>>>>>>> <elf::program::get_library(std::__cxx11::basic_string<char, >>>>>>>>>>> std::char_traits<char>, std::allocator<char> >, >>>>>>>>>>> std::vector<std::__cxx11::basic_string<char, >>>>>>>>>>> std::char_traits<char>, >>>>>>>>>>> std::allocator<char> >, >>>>>>>>>>> std::allocator<std::__cxx11::basic_string<char, >>>>>>>>>>> std::char_traits<char>, std::allocator<char> > > >, bool)+715> >>>>>>>>>>> 0x0000000040462f19 <dlopen+153> >>>>>>>>>>> 0x0000100000937228 <_PyImport_FindSharedFuncptr+376> >>>>>>>>>>> 0x006567617373656c <???+1936942444> >>>>>>>>>>> 2020-01-22T17:13:51.345740567 [anonymous-instance:ERROR:vmm/src/ >>>>>>>>>>> lib.rs:1658] Failed to log metrics: Logger was not initialized. >>>>>>>>>>> >>>>>>>>>>> Is there any clue how to debug this? The function pointer seems >>>>>>>>>>> to point to a valid address but I get a seg fault when executing >>>>>>>>>>> it. >>>>>>>>>>> >>>>>>>>>>> Best, >>>>>>>>>>> Zhiting >>>>>>>>>>> >>>>>>>>>>> -- >>>>>>>>>> You received this message because you are subscribed to the >>>>>>>>>> Google Groups "OSv Development" group. >>>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>>> send an email to [email protected]. >>>>>>>>>> To view this discussion on the web visit >>>>>>>>>> https://groups.google.com/d/msgid/osv-dev/ecdc93a7-2a7e-4d39-87e9-6de15578b7df%40googlegroups.com >>>>>>>>>> >>>>>>>>>> <https://groups.google.com/d/msgid/osv-dev/ecdc93a7-2a7e-4d39-87e9-6de15578b7df%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>>> . >>>>>>>>>> >>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "OSv Development" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to [email protected]. >>>>>>>>> To view this discussion on the web visit >>>>>>>>> https://groups.google.com/d/msgid/osv-dev/CANEVyjsYXNroc3%3DeXB7Z0cV-rfeO9yUGpkDGZP87LDNunXHocQ%40mail.gmail.com >>>>>>>>> >>>>>>>>> <https://groups.google.com/d/msgid/osv-dev/CANEVyjsYXNroc3%3DeXB7Z0cV-rfeO9yUGpkDGZP87LDNunXHocQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>>> . >>>>>>>>> >>>>>>>> -- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "OSv Development" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/osv-dev/096465a1-ffbc-45f9-b469-fa5c97c752f7%40googlegroups.com >>>>>> >>>>>> <https://groups.google.com/d/msgid/osv-dev/096465a1-ffbc-45f9-b469-fa5c97c752f7%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> -- >>>> You received this message because you are subscribed to the Google >>>> Groups "OSv Development" group. >>>> To unsubscribe from this group and stop receiving emails from it, send >>>> an email to [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/osv-dev/173619bc-4cdf-4408-8b5d-541693d08de6%40googlegroups.com >>>> >>>> <https://groups.google.com/d/msgid/osv-dev/173619bc-4cdf-4408-8b5d-541693d08de6%40googlegroups.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>> -- >> You received this message because you are subscribed to the Google Groups >> "OSv Development" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/osv-dev/36872cb0-9673-4756-ab6d-df94a19dba4b%40googlegroups.com >> >> <https://groups.google.com/d/msgid/osv-dev/36872cb0-9673-4756-ab6d-df94a19dba4b%40googlegroups.com?utm_medium=email&utm_source=footer> >> . >> > -- You received this message because you are subscribed to the Google Groups "OSv Development" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/34332f2f-7a70-4711-bc34-ce19bccaedcd%40googlegroups.com.
