On Monday, February 11, 2019 at 2:00:59 AM UTC-5, Nadav Har'El wrote: > > On Mon, Feb 11, 2019 at 1:33 AM Waldek Kozaczuk <[email protected] > <javascript:>> wrote: > >> Here is the scenario: >> 1) Build image with some app X and httpserver API module. >> 2) Start app X using REST API /app call with new_program=false (same >> namespace) >> 3) OSv loads app library and any dependent ones from disk and executes it. >> 4) Start app X again same way as in 2) over and over again (unlimited >> times) BUT OSv skips load any libraries from disk instead re-uses relevant >> ELF objects already loaded/mmaped in memory. >> >> This actually works using simple native-example. But fails on a second >> run with ffmpeg app where I get this type of stack trace: >> 0x00000000003a9482 in processor::cli_hlt () at arch/x64/processor.hh:247 >> 247 asm volatile ("cli; hlt" : : : "memory"); >> (gdb) bt >> #0 0x00000000003a9482 in processor::cli_hlt () at >> arch/x64/processor.hh:247 >> #1 arch::halt_no_interrupts () at arch/x64/arch.hh:48 >> #2 osv::halt () at arch/x64/power.cc:24 >> #3 0x000000000023f144 in abort (fmt=fmt@entry=0x63335b "Aborted\n") at >> runtime.cc:132 >> #4 0x0000000000202765 in abort () at runtime.cc:98 >> #5 0x000000000045dc9c in osv::generate_signal (siginfo=..., >> ef=0xffff800003099068) at libc/signal.cc:124 >> #6 0x000000000045dd0b in osv::handle_mmap_fault (addr=<optimized out>, >> sig=<optimized out>, ef=<optimized out>) at libc/signal.cc:139 >> #7 0x0000000000347bf3 in mmu::vm_fault (addr=0, addr@entry=56, >> ef=ef@entry=0xffff800003099068) at core/mmu.cc:1338 >> #8 0x00000000003a32de in page_fault (ef=0xffff800003099068) at >> arch/x64/mmu.cc:38 >> #9 <signal handler called> >> #10 0x0000100003d39d6c in ?? () >> #11 0x00002000003ff220 in ?? () >> #12 0xffffa00002ce7ada in ?? () >> #13 0x0000000000000000 in ?? () >> >> If I set new_program=true ffmpeg runs fine every time up to 32 times. >> >> So my motivation is to be able to run given application on OSv >> sequentially one by one without having to restart OSv and only pay price >> for loading code in memory once on the first run. >> > > osv::application:run() and osv::run() have accumulated so many layers over > the year that I forget all their details and you may need to look at the > actual code (the REST API calls osv::application:run() to do its work). I > do believe that after the application finishes running, it indeed unloads > the library by default. > One thing you could do is after app= osv::application::run(...), you can > do something like > > auto lib = app.lib(); > // as long as this "lib" exists, the library will not be unloaded. > You can even create an eternal copy with something like: > new std::shared_ptr<elf::object>(lib); > > However, *not* unloading the library after the run finishes may not make > it easier to run it again - it might even be the opposite, e.g., if the > program needs some global variable to be set on load to some default > values. Some programs may not be possible to run a second time if they > leave behind too stuff they don't clean, etc, but this is probably not your > case. I wonder if in your case somehow the library did *not* unload, and > that caused the change from the "new_program" case. > I agree with you that messing at this level (loading/unloading elf) would only complicate things more. Given that most expensive part is loading code from the disk, maybe we could somehow optimize things at the lower level. We would still unload object in terms of destructing how its is represented, but we would not unmap() from memory and make it stay in memory wherever the ELF file segments were loaded to. So that when the same program (same path) gets loaded, we would still create it representation state, run init functions, etc, but we would skip loading corresponding file fragments from disk. Obviously it may be not as trivial given some parts of ELF get relocated which means that corresponding ELF segments in memory get re-written. Any optimization would have to take this into account as well and somehow know to skip relocation on second run. So all in all the main idea is to keep the parts of ELF file loaded in memory so that we do not have to load it again from disk on subsequent runs.
> > Can you please try to look at the backtrace with gdb, instead of the > above, maybe we'll get more information? > > >> >> I have not analyzed OSv code well enough to understand if that is what >> already happens. But as the ffmpeg example shows it comes with some quirks. >> Also this would be best effort and only work if app cleans itself after >> completion which I think is the case for ffmpeg but may not the case for >> all kinds of apps. >> > > Indeed. > > >> Looking at elf.cc it looks like the elf segments get unmapped in >> remove_objects() that is setup to be called once they are no longer >> referenced. So ideally we would want to have an option of skipping >> unmapping the ELF so that when app is re-executed again we skip the mmaping >> part as code would be still in memory. >> >> I do not understand ELF lifecycle well enough to know if that is even >> possible. >> > > >> -- >> You received this message because you are subscribed to the Google Groups >> "OSv Development" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected] <javascript:>. >> For more options, visit https://groups.google.com/d/optout. >> > -- You received this message because you are subscribed to the Google Groups "OSv Development" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. For more options, visit https://groups.google.com/d/optout.
