Re: [Qemu-devel] QEMU to generate host binary
Dr. David Alan Gilbert dgilb...@redhat.com writes: * Ayaz Akram (aaq...@gmail.com) wrote: Thanks for your answers. The thing that i still do not get is once we have host assembly code (output assembly log generated for compiled TBs ), should we still worry about problems like self modifying code and other dynamic conditions? Moreover, assuming static linking, will not this code be enough to generate .text section of an executable that could be run directly on host (if somehow other sections of that host executable can be generated, which is itself difficult) ? It can certainly help, but you can still walk into a piece of code that you've not previously translated, and a shared library for example might change, and your code might change due to other things such as randomisation of library loading addresses. snip You say 'should we still worry about problems like self modifying code and other dynamic conditions?' - well you hope that for most 'normal' programs that self modification doesn't happen; but it keeps turning up even when you don't expect it, e.g. loading/unloading of plugins or if you happen to be emulating a JIT (such as qemu!). I think PLT patching is another case of self modifying code you often come across. -- Alex Bennée
Re: [Qemu-devel] QEMU to generate host binary
* Ayaz Akram (aaq...@gmail.com) wrote: Thanks for your answers. The thing that i still do not get is once we have host assembly code (output assembly log generated for compiled TBs ), should we still worry about problems like self modifying code and other dynamic conditions? Moreover, assuming static linking, will not this code be enough to generate .text section of an executable that could be run directly on host (if somehow other sections of that host executable can be generated, which is itself difficult) ? It can certainly help, but you can still walk into a piece of code that you've not previously translated, and a shared library for example might change, and your code might change due to other things such as randomisation of library loading addresses. Various emulators in the past have dumped some part of the translation state to disk in an attempt to save on costs when they next emulate the code; others have tried to do background analysis to optimise the translated code into a pretranslated blob - but they can never guarantee to get the whole set of code translated; and that's even for a user-mode emulation where at least you know which binary file you're trying to emulate. In system emulation it's much more difficult because the emulator doesn't have a view of which executable the page it's currently emulating corresponds to; indeed that page could be shared between multiple different executables, or in the opposite direction is often patched during runtime linking. You say 'should we still worry about problems like self modifying code and other dynamic conditions?' - well you hope that for most 'normal' programs that self modification doesn't happen; but it keeps turning up even when you don't expect it, e.g. loading/unloading of plugins or if you happen to be emulating a JIT (such as qemu!). And QEMU (especially in system mode) doesn't know whether the binary that's being run is a nice safe friendly static binary or something that's going to do some mad optimisation itself. One example of the type of thing you can do is save statistics information about a piece of executable, e.g. 'this code is hot' so you know it's a hot loop that's worth optimising; that can pay off if you have multiple levels of interpreter/optimisation in your emulator. Dave On Mon, Jun 29, 2015 at 1:04 PM, Peter Crosthwaite peter.crosthwa...@xilinx.com wrote: On Mon, Jun 29, 2015 at 8:13 AM, Stefan Hajnoczi stefa...@gmail.com wrote: On Sun, Jun 28, 2015 at 07:29:39PM -0400, Ayaz Akram wrote: Let's say qemu is running in System Emulation Mode, when it runs guest's System emulation makes the problem even harder, as a system mode binary (usually an OS or some sort) will have difficult porting from one CPU-types system arch to another. This is more realistic (but still very difficult and not generally solvable) in user-mode emulation. binary, it can log the translated code for host. Is it possible to merge that translated code and other sections of guest's binary to make a binary which can be run directly on host. No, because of self-modifying code, run-time code loading, etc. Ruling these two out for the moment ... It is not possible to statically translate an executable (in the general case). There are architectures where it is possible due to restrictions (e.g. no code loading, all jump destinations are known in advance, etc) but Debug info with function information might give you a crude approximation of jump targets coming from fn pointers. That + the statically determinable jump targets might give you something for apps that don't do anything wierd. I'm wondering if the jump problem can be crudely solved by a fully single-step translation. The result binary would be huge an inefficient. But could you keep two translations around? One that uses the statically determinable best guess of the jump dest table I describe above, and a second defensive translation of the entire app in single-step? There are more complications however. Another one I can think of is instructions that change runtime state and affect (re)translation (e.g. the arm setend instruction which switches CPU endianness). Regards, Peter the popular x86, ARM, etc architectures allow too much freedom to be amenable to static translation. Stefan -- Dr. David Alan Gilbert / dgilb...@redhat.com / Manchester, UK
Re: [Qemu-devel] QEMU to generate host binary
On Sun, Jun 28, 2015 at 07:29:39PM -0400, Ayaz Akram wrote: Let's say qemu is running in System Emulation Mode, when it runs guest's binary, it can log the translated code for host. Is it possible to merge that translated code and other sections of guest's binary to make a binary which can be run directly on host. No, because of self-modifying code, run-time code loading, etc. It is not possible to statically translate an executable (in the general case). There are architectures where it is possible due to restrictions (e.g. no code loading, all jump destinations are known in advance, etc) but the popular x86, ARM, etc architectures allow too much freedom to be amenable to static translation. Stefan pgpL9CyDGvCjP.pgp Description: PGP signature
Re: [Qemu-devel] QEMU to generate host binary
On Mon, Jun 29, 2015 at 8:13 AM, Stefan Hajnoczi stefa...@gmail.com wrote: On Sun, Jun 28, 2015 at 07:29:39PM -0400, Ayaz Akram wrote: Let's say qemu is running in System Emulation Mode, when it runs guest's System emulation makes the problem even harder, as a system mode binary (usually an OS or some sort) will have difficult porting from one CPU-types system arch to another. This is more realistic (but still very difficult and not generally solvable) in user-mode emulation. binary, it can log the translated code for host. Is it possible to merge that translated code and other sections of guest's binary to make a binary which can be run directly on host. No, because of self-modifying code, run-time code loading, etc. Ruling these two out for the moment ... It is not possible to statically translate an executable (in the general case). There are architectures where it is possible due to restrictions (e.g. no code loading, all jump destinations are known in advance, etc) but Debug info with function information might give you a crude approximation of jump targets coming from fn pointers. That + the statically determinable jump targets might give you something for apps that don't do anything wierd. I'm wondering if the jump problem can be crudely solved by a fully single-step translation. The result binary would be huge an inefficient. But could you keep two translations around? One that uses the statically determinable best guess of the jump dest table I describe above, and a second defensive translation of the entire app in single-step? There are more complications however. Another one I can think of is instructions that change runtime state and affect (re)translation (e.g. the arm setend instruction which switches CPU endianness). Regards, Peter the popular x86, ARM, etc architectures allow too much freedom to be amenable to static translation. Stefan
Re: [Qemu-devel] QEMU to generate host binary
Thanks for your answers. The thing that i still do not get is once we have host assembly code (output assembly log generated for compiled TBs ), should we still worry about problems like self modifying code and other dynamic conditions? Moreover, assuming static linking, will not this code be enough to generate .text section of an executable that could be run directly on host (if somehow other sections of that host executable can be generated, which is itself difficult) ? On Mon, Jun 29, 2015 at 1:04 PM, Peter Crosthwaite peter.crosthwa...@xilinx.com wrote: On Mon, Jun 29, 2015 at 8:13 AM, Stefan Hajnoczi stefa...@gmail.com wrote: On Sun, Jun 28, 2015 at 07:29:39PM -0400, Ayaz Akram wrote: Let's say qemu is running in System Emulation Mode, when it runs guest's System emulation makes the problem even harder, as a system mode binary (usually an OS or some sort) will have difficult porting from one CPU-types system arch to another. This is more realistic (but still very difficult and not generally solvable) in user-mode emulation. binary, it can log the translated code for host. Is it possible to merge that translated code and other sections of guest's binary to make a binary which can be run directly on host. No, because of self-modifying code, run-time code loading, etc. Ruling these two out for the moment ... It is not possible to statically translate an executable (in the general case). There are architectures where it is possible due to restrictions (e.g. no code loading, all jump destinations are known in advance, etc) but Debug info with function information might give you a crude approximation of jump targets coming from fn pointers. That + the statically determinable jump targets might give you something for apps that don't do anything wierd. I'm wondering if the jump problem can be crudely solved by a fully single-step translation. The result binary would be huge an inefficient. But could you keep two translations around? One that uses the statically determinable best guess of the jump dest table I describe above, and a second defensive translation of the entire app in single-step? There are more complications however. Another one I can think of is instructions that change runtime state and affect (re)translation (e.g. the arm setend instruction which switches CPU endianness). Regards, Peter the popular x86, ARM, etc architectures allow too much freedom to be amenable to static translation. Stefan
Re: [Qemu-devel] QEMU to generate host binary
Let's say qemu is running in System Emulation Mode, when it runs guest's binary, it can log the translated code for host. Is it possible to merge that translated code and other sections of guest's binary to make a binary which can be run directly on host. Thanks On Fri, Jun 26, 2015 at 11:34 PM, Peter Crosthwaite peter.crosthwa...@xilinx.com wrote: On Fri, Jun 26, 2015 at 12:33 PM, Ayaz Akram aaq...@gmail.com wrote: Hello ! Is anyone aware of an effort to produce an executable binary for host using qemu. I mean is it possible that qemu generate a binary for whatever application it is emulating, which can later be run directly on host? I'm not sure what this binary would mean just yet. Are you extracting just the guest + its runtime state to a binary that picks up where the guest left off? Or are you including the machine emulator (i.e. QEMU itself) in this new binary to avoid having to load the guest it while picking up where left off? Regards, Peter Thanks
[Qemu-devel] QEMU to generate host binary
Hello ! Is anyone aware of an effort to produce an executable binary for host using qemu. I mean is it possible that qemu generate a binary for whatever application it is emulating, which can later be run directly on host? Thanks
Re: [Qemu-devel] QEMU to generate host binary
On Fri, Jun 26, 2015 at 12:33 PM, Ayaz Akram aaq...@gmail.com wrote: Hello ! Is anyone aware of an effort to produce an executable binary for host using qemu. I mean is it possible that qemu generate a binary for whatever application it is emulating, which can later be run directly on host? I'm not sure what this binary would mean just yet. Are you extracting just the guest + its runtime state to a binary that picks up where the guest left off? Or are you including the machine emulator (i.e. QEMU itself) in this new binary to avoid having to load the guest it while picking up where left off? Regards, Peter Thanks
