The image is 2.5 G. On Thu, Jan 23, 2020 at 5:26 PM Waldek Kozaczuk <[email protected]> wrote:
> BTW how big is you image? > > I wounder if ROFS hangs because it runs out of memory when trying to load > files into memory. > > On Thursday, January 23, 2020 at 6:24:21 PM UTC-5, zhiting zhu wrote: >> >> Yeah, Zfs image doesn't hang. I don't know why rofs image hangs. I need >> to increase the qemu memory in upload_manifest.py otherwise it hangs on >> building zfs images. >> >> On Thu, Jan 23, 2020 at 5:17 PM Waldek Kozaczuk <[email protected]> >> wrote: >> >>> It seems like it got stuck while trying to mount the filesystem. The >>> next boot message would normally be 'VFS: mounting devfs at /dev". >>> >>> I wonder if the image (usr.img) is somehow locked or something. Have you >>> tried to rebuild the image? Try zfs. >>> >>> Waldek >>> >>> On Thursday, January 23, 2020 at 5:54:08 PM UTC-5, zhiting zhu wrote: >>>> >>>> native-example and python image works with qemu. It seems it only hangs >>>> on my custom tensorflow image. I'm only passing --verbose /python3 to >>>> run.py >>>> >>>> >>>> On Thu, Jan 23, 2020 at 4:34 PM Waldek Kozaczuk <[email protected]> >>>> wrote: >>>> >>>>> Are you passing any parameters to ./scripts/run.py? I would also >>>>> suggest to specify just single vCPU - it should make debugging easier. >>>>> >>>>> Also does it hang with this app only or others as well? Can you try >>>>> this: >>>>> ./scripts/build image=native-example >>>>> ./scripts/run.py >>>>> >>>>> Waldek >>>>> >>>>> On Thursday, January 23, 2020 at 5:28:43 PM UTC-5, zhiting zhu wrote: >>>>>> >>>>>> Unfortunately, I can't boot the vm with qemu. It's hanging at the >>>>>> beginning. >>>>>> >>>>>> I'm seeing this: >>>>>> bsd: initializing - done >>>>>> VFS: mounting ramfs at / >>>>>> VFS: mounting devfs at /dev >>>>>> net: initializing - done >>>>>> vga: Add VGA device instance >>>>>> eth0: ethernet address: 52:54:00:12:34:56 >>>>>> virtio-blk: Add blk device instances 0 as vblk0, devsize=1192516096 >>>>>> random: virtio-rng registered as a source. >>>>>> random: intel drng, rdrand registered as a source. >>>>>> random: <Software, Yarrow> initialized >>>>>> VFS: unmounting /dev >>>>>> VFS: mounting rofs at /rofs >>>>>> random: device unblocked. >>>>>> >>>>>> >>>>>> >>>>>> On Thu, Jan 23, 2020 at 3:25 PM zhiting zhu <[email protected]> >>>>>> wrote: >>>>>> >>>>>>> Here's the output log. See the file attached. >>>>>>> >>>>>>> On Thu, Jan 23, 2020 at 7:19 AM Nadav Har'El <[email protected]> >>>>>>> wrote: >>>>>>> >>>>>>>> >>>>>>>> On Thu, Jan 23, 2020 at 2:23 PM Waldek Kozaczuk <[email protected]> >>>>>>>> wrote: >>>>>>>> >>>>>>>>> Can you send us the full output? I wonder if there are any >>>>>>>>> warnings before it? >>>>>>>>> >>>>>>>>> I see you are running this on firecracker. Can you run it under >>>>>>>>> qemu and connect to it with gdb and see if you get better stacktrace? >>>>>>>>> >>>>>>>>> You can add another debug statement like that: >>>>>>>>> >>>>>>>>> if (strcmp(" /lib/python3.6/google/protobuf/pyext/_message. >>>>>>>>> cpython-36m-x86_64-linux-gnu.so",pathname)==0 && i == 28) { >>>>>>>>> ... >>>>>>>>> // Put breakpoint here >>>>>>>>> } >>>>>>>>> >>>>>>>>> and try to see what statement causes the fault. Make sure to do >>>>>>>>> 'osv syms" to get as much debug info resolved as possible. (see >>>>>>>>> https://github.com/cloudius-systems/osv/wiki/Debugging-OSv). >>>>>>>>> >>>>>>>> >>>>>>>> I hope that "osv syms" will find the >>>>>>>> newly-loaded-but-not-yet-completely-loaded libraries. If it doesn't, >>>>>>>> maybe >>>>>>>> we can fix the order of when the array that "osv syms" uses gets >>>>>>>> written >>>>>>>> during loaded. >>>>>>>> >>>>>>>>> >>>>>>>>> I wonder if this has to with the order of initializing the ELF >>>>>>>>> objects when called by dlopen(). >>>>>>>>> >>>>>>>>> Waldek >>>>>>>>> >>>>>>>>> PS. If no more clue next step would be to add an app so we can >>>>>>>>> build it and reproduce it? >>>>>>>>> >>>>>>>>> On Wednesday, January 22, 2020 at 6:23:30 PM UTC-5, zhiting zhu >>>>>>>>> wrote: >>>>>>>>>> >>>>>>>>>> Hey, >>>>>>>>>> >>>>>>>>>> I'm hitting this error: >>>>>>>>>> >>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing DT_INIT >>>>>>>>>> function >>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Finished executing >>>>>>>>>> DT_INIT function >>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 56 >>>>>>>>>> DT_INIT_ARRAYSZ functions >>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 0 init func >>>>>>>>>> 0x1000388ad500 >>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 1 init func >>>>>>>>>> 0x1000388ac8a0 >>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 2 init func >>>>>>>>>> 0x1000388ac8d0 >>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 3 init func >>>>>>>>>> 0x1000388ac900 >>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 4 init func >>>>>>>>>> 0x1000388ac930 >>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 5 init func >>>>>>>>>> 0x1000388ac960 >>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 6 init func >>>>>>>>>> 0x1000388ac990 >>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 7 init func >>>>>>>>>> 0x1000388ac9c0 >>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 8 init func >>>>>>>>>> 0x1000388ac9f0 >>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 9 init func >>>>>>>>>> 0x1000388aca20 >>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 10 init func >>>>>>>>>> 0x1000388aca50 >>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 11 init func >>>>>>>>>> 0x1000388aca80 >>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 12 init func >>>>>>>>>> 0x1000388acab0 >>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 13 init func >>>>>>>>>> 0x1000388acae0 >>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 14 init func >>>>>>>>>> 0x1000388acb10 >>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 15 init func >>>>>>>>>> 0x1000388acb40 >>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 16 init func >>>>>>>>>> 0x1000388acb70 >>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 17 init func >>>>>>>>>> 0x1000388acc50 >>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 18 init func >>>>>>>>>> 0x1000388acc80 >>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 19 init func >>>>>>>>>> 0x1000388accb0 >>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 20 init func >>>>>>>>>> 0x1000388acce0 >>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 21 init func >>>>>>>>>> 0x1000388acd10 >>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 22 init func >>>>>>>>>> 0x1000388acd40 >>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 23 init func >>>>>>>>>> 0x1000388acd70 >>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 24 init func >>>>>>>>>> 0x1000388acda0 >>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 25 init func >>>>>>>>>> 0x1000388acdd0 >>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 26 init func >>>>>>>>>> 0x1000388ace00 >>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 27 init func >>>>>>>>>> 0x1000388ace50 >>>>>>>>>> ELF [tid:51, /lib/python3.6/google/protobuf/pyext/_ >>>>>>>>>> message.cpython-36m-x86_64-linux-gnu.so]: Executing 28 init func >>>>>>>>>> 0x1000388ace80 >>>>>>>>>> Aborted >>>>>>>>>> >>>>>>>>>> [backtrace] >>>>>>>>>> 0x0000000040463abb <osv::generate_signal(siginfo&, >>>>>>>>>> exception_frame*)+59> >>>>>>>>>> 0x0000000040463b2a <osv::handle_mmap_fault(unsigned long, int, >>>>>>>>>> exception_frame*)+26> >>>>>>>>>> 0x000000004032f3e9 <mmu::vm_fault(unsigned long, >>>>>>>>>> exception_frame*)+185> >>>>>>>>>> 0x000000004038f7b6 <page_fault+166> >>>>>>>>>> 0x000000004038e5f6 <???+1077470710> >>>>>>>>>> 0x0000000040341042 <elf::program::init_library(int, char**)+402> >>>>>>>>>> 0x00000000403473db >>>>>>>>>> <elf::program::get_library(std::__cxx11::basic_string<char, >>>>>>>>>> std::char_traits<char>, std::allocator<char> >, >>>>>>>>>> std::vector<std::__cxx11::basic_string<char, std::char_traits<char>, >>>>>>>>>> std::allocator<char> >, >>>>>>>>>> std::allocator<std::__cxx11::basic_string<char, >>>>>>>>>> std::char_traits<char>, std::allocator<char> > > >, bool)+715> >>>>>>>>>> 0x0000000040462f19 <dlopen+153> >>>>>>>>>> 0x0000100000937228 <_PyImport_FindSharedFuncptr+376> >>>>>>>>>> 0x006567617373656c <???+1936942444> >>>>>>>>>> 2020-01-22T17:13:51.345740567 [anonymous-instance:ERROR:vmm/src/ >>>>>>>>>> lib.rs:1658] Failed to log metrics: Logger was not initialized. >>>>>>>>>> >>>>>>>>>> Is there any clue how to debug this? The function pointer seems >>>>>>>>>> to point to a valid address but I get a seg fault when executing it. >>>>>>>>>> >>>>>>>>>> Best, >>>>>>>>>> Zhiting >>>>>>>>>> >>>>>>>>>> -- >>>>>>>>> You received this message because you are subscribed to the Google >>>>>>>>> Groups "OSv Development" group. >>>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>>> send an email to [email protected]. >>>>>>>>> To view this discussion on the web visit >>>>>>>>> https://groups.google.com/d/msgid/osv-dev/ecdc93a7-2a7e-4d39-87e9-6de15578b7df%40googlegroups.com >>>>>>>>> <https://groups.google.com/d/msgid/osv-dev/ecdc93a7-2a7e-4d39-87e9-6de15578b7df%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>>>>> . >>>>>>>>> >>>>>>>> -- >>>>>>>> You received this message because you are subscribed to the Google >>>>>>>> Groups "OSv Development" group. >>>>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>>>> send an email to [email protected]. >>>>>>>> To view this discussion on the web visit >>>>>>>> https://groups.google.com/d/msgid/osv-dev/CANEVyjsYXNroc3%3DeXB7Z0cV-rfeO9yUGpkDGZP87LDNunXHocQ%40mail.gmail.com >>>>>>>> <https://groups.google.com/d/msgid/osv-dev/CANEVyjsYXNroc3%3DeXB7Z0cV-rfeO9yUGpkDGZP87LDNunXHocQ%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>>>>>> . >>>>>>>> >>>>>>> -- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "OSv Development" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to [email protected]. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/osv-dev/096465a1-ffbc-45f9-b469-fa5c97c752f7%40googlegroups.com >>>>> <https://groups.google.com/d/msgid/osv-dev/096465a1-ffbc-45f9-b469-fa5c97c752f7%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- >>> You received this message because you are subscribed to the Google >>> Groups "OSv Development" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/osv-dev/173619bc-4cdf-4408-8b5d-541693d08de6%40googlegroups.com >>> <https://groups.google.com/d/msgid/osv-dev/173619bc-4cdf-4408-8b5d-541693d08de6%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >> -- > You received this message because you are subscribed to the Google Groups > "OSv Development" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/osv-dev/36872cb0-9673-4756-ab6d-df94a19dba4b%40googlegroups.com > <https://groups.google.com/d/msgid/osv-dev/36872cb0-9673-4756-ab6d-df94a19dba4b%40googlegroups.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "OSv Development" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/osv-dev/CA%2B3q14wj%2BTXKdNrxH4PAG%3DbH%3D3uzxXW3NwNCuDVN57czk2xQZw%40mail.gmail.com.
