Re: [Qemu-devel] qemu-user: relocating target code weakness
On 01/25/2011 02:36 AM, Richard Henderson wrote: On 01/24/2011 01:44 PM, Stefano Bonifazi wrote: Wow wonderful! So you fixed the code for PIC (ET_DYN) support? Yes. how can I get your sources? I was mistaken -- a later version of the patch set was in fact merged. I simply forgot to delete my working branch afterward. r~ Hi! I tested succesfully the sources with your fixes (though it remains the little problem of ld.so.1 path)! Even my final goal of having more than one instance of qemu-user running in the same address space worked fine using pie code after your fixes! Man I really own you a big thank you! :) Though I was on the right way, I'd surely take very long for doing the same fixes, and I have got a very short deadline! You almost saved me! :) I dunno where you are from, but if you happen to visit Rome you have a dinner paid! :D Again many thanks and best regards! Stefano B. P.S. Please just answer that last question, whether it is possible to have a variable showing the upper bound of heap (some brk_end) for a target process
Re: [Qemu-devel] qemu-user: relocating target code weakness
On 01/26/2011 04:38 PM, Richard Henderson wrote: On 01/26/2011 03:07 AM, Stefano Bonifazi wrote: P.S. Please just answer that last question, whether it is possible to have a variable showing the upper bound of heap (some brk_end) for a target process No, the heap grows until it reaches some other memory mapped entity. r~ Thank you again! :) Best regards! Stefano B.
Re: [Qemu-devel] qemu-user: relocating target code weakness
On 01/26/2011 09:19 PM, Richard Henderson wrote: On 01/26/2011 12:17 PM, Lluís wrote: Richard Henderson writes: On 01/26/2011 03:07 AM, Stefano Bonifazi wrote: P.S. Please just answer that last question, whether it is possible to have a variable showing the upper bound of heap (some brk_end) for a target process No, the heap grows until it reaches some other memory mapped entity. From man brk(2) : sbrk() increments the program's data space by increment bytes. Calling sbrk() with an increment of 0 can be used to find the current location of the program break. I already sent this to the list in a previous mail, but it seems you overlooked it as you were not an explicit recipient. That's the current top of the heap. I answered the question as if it was asking for the maximum top of the heap. r~ Yup! I am concerned what can be the highest address used as heap by a target binary, because I am creating multiple instances of qemu-user, and I wanna prevent the target process images and heaps to overlap.. Good enough to know the mmu will stop giving memory when meeting another memory mapped region.. Moreover I think I'd have to call sbrk from within the target binary for resizing its heap, whereas I wanna control it from qemu-user that is loading, then translating and executing it.. @Luis: Yup sorry for not answering the previous post of yours, I thought I'd receive emails always when somebody answered a post I created at the beginning! Thank you! Regards, Stefano B.
Re: [Qemu-devel] qemu-user: relocating target code weakness
On 01/25/2011 01:06 AM, Mike Frysinger wrote: On Mon, Jan 24, 2011 at 11:29, Mulyadi Santosa wrote: I wrote an article about understanding ELF years ago, here is the URL: http://www.linuxforums.org/articles/understanding-elf-using-readelf-and-objdump_125.html i also highly recommend Linkers Loaders: http://linker.iecc.com/ some pieces might be a little dated, but buy large, it's still relevant -mike Thank you for this suggestion! Of course it was also one of the first I got: http://www.iecc.com/linker/ Here I got it for free! Best regards! Stefano B.
Re: [Qemu-devel] qemu-user: relocating target code weakness
On 01/25/2011 12:32 AM, Mike Frysinger wrote: On Mon, Jan 24, 2011 at 16:44, Stefano Bonifazi wrote: http://lists.gnu.org/archive/html/qemu-devel/2010-07/msg01626.html A noob question, how can I get your sources? Is there a simpler solution than copypaste all the code from your messages into patches and then applying them? Can you just send your sources by email? Or can I download them from a site? weird ... usually the gnu archives have a download raw text option so you can get a mbox file to import into your mailer you could use the gmane.org nntp gateway to get all the raw e-mails though: nntp://news.gmane.org/gmane.comp.emulators.qemu -mike Hi! I didn't mean the messages sources, I am already using nntp with thunderbird, I meant whether there is an option for getting the .c, .h sources, instead of copying the patches and apply them .. sorry but I am really new with this Thank you! Regards! Stefano B.
Re: [Qemu-devel] qemu-user: relocating target code weakness
On 01/25/2011 02:36 AM, Richard Henderson wrote: On 01/24/2011 01:44 PM, Stefano Bonifazi wrote: Wow wonderful! So you fixed the code for PIC (ET_DYN) support? Yes. how can I get your sources? I was mistaken -- a later version of the patch set was in fact merged. I simply forgot to delete my working branch afterward. r~ Sorry.. merged with qemu? What version? I have qemu.0.13.0 and there are no your fixes.. How can I get the fixed qemu sources?
Re: [Qemu-devel] qemu-user: relocating target code weakness
On 01/25/2011 01:18 AM, Mike Frysinger wrote: On Mon, Jan 24, 2011 at 19:06, Mike Frysinger wrote: On Mon, Jan 24, 2011 at 11:29, Mulyadi Santosa wrote: I wrote an article about understanding ELF years ago, here is the URL: http://www.linuxforums.org/articles/understanding-elf-using-readelf-and-objdump_125.html i also highly recommend Linkers Loaders: http://linker.iecc.com/ some pieces might be a little dated, but buy large, it's still relevant Ian Lance Taylor (author of the GOLD linker) maintains a blog where he posts well written and in-depth articles on various toolchain pieces: http://www.airs.com/blog/archives/category/programming this one talks about the GOT/PLT for ELF on a MMU: http://www.airs.com/blog/archives/41 -mike Great! Thank you! Good that I ordered new ink for my printer :) Best regards! Stefano B.
Re: [Qemu-devel] TCG flow vs dyngen
That said, QEMU's currently working fairly well on this front too, so studying either should work pretty well... Mr Richard Henderson's patch on elfload.c says I was right.. at least the version I am working on (qemu-0.13.0) had some bugs and weaknesses though it worked smoothly for most cases.. And to be honest, the best way to get up to speed on this is to read this: http://www.muppetlabs.com/~breadbox/software/tiny/teensy.html uhmm seems a good piece.. maybe one of the last I still didn't have :) Thank you!! Stefano B.
Re: [Qemu-devel] TCG flow vs dyngen
Again wow!! Is that really possible? Some sort of callback triggered at every instruction execution? Yes, this mechanism works. I have written a code to count different kinds of instructions. Great! that opens a lot of possibilities!. It exists in file qemu/target-i386/translate.c Ops right! I checked target-ppc/translate.c as I need Power-PC as target.. I wonder what function replaces it there.. You are also talking about qemu source code privided here http://wiki.qemu.org/Download, right? Yes I am using this http://wiki.qemu.org/download/qemu-0.13.0.tar.gz If you need, I can give the source code of counting implementation with some documentation. Hope this helps. Wow that would be awesome! I'd really appreciate it very much! Thank you! :) You are free of sending it to my address! :) Best regards!! Stefano B.
Re: [Qemu-devel] TCG flow vs dyngen
On 01/25/2011 10:05 AM, Edgar E. Iglesias wrote: On Tue, Jan 25, 2011 at 10:04:39AM +0100, Stefano Bonifazi wrote: Again wow!! Is that really possible? Some sort of callback triggered at every instruction execution? Yes, this mechanism works. I have written a code to count different kinds of instructions. Great! that opens a lot of possibilities!. It exists in file qemu/target-i386/translate.c Ops right! I checked target-ppc/translate.c as I need Power-PC as target.. I wonder what function replaces it there.. You are also talking about qemu source code privided here http://wiki.qemu.org/Download, right? Yes I am using this http://wiki.qemu.org/download/qemu-0.13.0.tar.gz If you need, I can give the source code of counting implementation with some documentation. Hope this helps. Wow that would be awesome! I'd really appreciate it very much! Thank you! :) You are free of sending it to my address! :) Hi, If you are interested in instruction counting maybe you should take a look at the -icount option as well. Cheers Thank you! Already tried long ago, it doesn't work with qemu-user..If I remember fine its core was in files not used in qemu-user :( Regards, Stefano B.
Re: [Qemu-devel] qemu-user: relocating target code weakness
Sorry.. merged with qemu? What version? I have qemu.0.13.0 and there are no your fixes.. How can I get the fixed qemu sources? you probably want to use the latest git tree http://git.qemu.org/qemu.git/ -mike Wow man! I got your work through the git! Very good job! Now everything is much clearer there! I am happy to see that you did many edits I also did without knowing your work :) Surely you had the sureness of touch doing everything new.. Instead I was always afraid to destroy what worked before, so I moved very slowly :[ Anyway thank you again! If something keeps being wrong I know who to ask for help to! :) Best regards! Stefano B.
Re: [Qemu-devel] qemu-user: relocating target code weakness
On 01/25/2011 09:53 AM, Mike Frysinger wrote: On Tue, Jan 25, 2011 at 03:47, Stefano Bonifazi wrote: On 01/25/2011 02:36 AM, Richard Henderson wrote: On 01/24/2011 01:44 PM, Stefano Bonifazi wrote: Wow wonderful! So you fixed the code for PIC (ET_DYN) support? Yes. how can I get your sources? I was mistaken -- a later version of the patch set was in fact merged. I simply forgot to delete my working branch afterward. Sorry.. merged with qemu? What version? I have qemu.0.13.0 and there are no your fixes.. How can I get the fixed qemu sources? you probably want to use the latest git tree http://git.qemu.org/qemu.git/ -mike Hi! I think there is still a bug I corrected: You keep getting the name of the dynamic linker from the PT_INTERP program segment, but that gives you the absolute position of the dynamic linker inside the machine the target binary was created in. Here qemu-user is an emultaor used for running that binary into another machine.. Qemu default to usr/gnemul/ the path where the libs for the target machines are stored... So we need to patch the absolute dynamic linker name with the proper path.. I dunno maybe this can be my first little contribute to qemu family ;) Regards, Stefano B.
Re: [Qemu-devel] qemu-user: relocating target code weakness
On 01/25/2011 09:53 AM, Mike Frysinger wrote: On Tue, Jan 25, 2011 at 03:47, Stefano Bonifazi wrote: On 01/25/2011 02:36 AM, Richard Henderson wrote: On 01/24/2011 01:44 PM, Stefano Bonifazi wrote: Wow wonderful! So you fixed the code for PIC (ET_DYN) support? Yes. how can I get your sources? I was mistaken -- a later version of the patch set was in fact merged. I simply forgot to delete my working branch afterward. Sorry.. merged with qemu? What version? I have qemu.0.13.0 and there are no your fixes.. How can I get the fixed qemu sources? you probably want to use the latest git tree http://git.qemu.org/qemu.git/ -mike Hi again! :) debugging a test pie code using qemu-ppc with your (and my little one) fixes I got this: start_brk 0x end_code0x400102e0 start_code 0x4000 start_data 0x4001024c end_data0x400102e0 start_stack 0x40811438 brk 0x400102e4 entry 0x40828c24 that is start_brk is 0 As far as I understood brk is the .bss section, that is unitialized data area, am I right? If not pls could you be so kind to explain me what it is, or pointing me to some document where it is explained.. among all material I am studying about linkers, relocations, I never found brk other than in qemu.. Once i am sure what it is I'll try to fix (if needed.. for me it is too strange it is zero) it.. Thank you again! Stefano B.
Re: [Qemu-devel] qemu-user: relocating target code weakness
On 01/25/2011 05:22 PM, Richard Henderson wrote: On 01/25/2011 02:47 AM, Stefano Bonifazi wrote: You keep getting the name of the dynamic linker from the PT_INTERP program segment, but that gives you the absolute position of the dynamic linker inside the machine the target binary was created in. Here qemu-user is an emultaor used for running that binary into another machine.. Qemu default to usr/gnemul/ the path where the libs for the target machines are stored... So we need to patch the absolute dynamic linker name with the proper path.. In load_elf_interp, fd = open(path(filename), O_RDONLY); Notice the path function call. That does the translation into gnemul, given the proper configure option, or -L command-line option. r~ Hi! Strange, I have all the target libs in the default host usr/gnemul folder, but your fixed qemu still complained for not finding ld.so.1 until I fixed the code.. I'll try to check why path doesn't work! Thank you! Best regards! Stefano B.
Re: [Qemu-devel] qemu-user: relocating target code weakness
On 01/25/2011 05:26 PM, Richard Henderson wrote: On 01/25/2011 03:06 AM, Stefano Bonifazi wrote: start_brk 0x end_code0x400102e0 start_code 0x4000 start_data 0x4001024c end_data0x400102e0 start_stack 0x40811438 brk 0x400102e4 entry 0x40828c24 that is start_brk is 0 As far as I understood brk is the .bss section, that is unitialized data area, am I right? Not quite. It's normally the beginning of the heap, after the bss section. That said, it looks like start_brk is a dead variable. It's written only by the FLAT loader, and nothing at all reads it, except for this debugging dump. The real value is the brk variable, which is indeed set to a plausible looking value. r~ Thank you!! I really missed that.. I've always wondered where the heap was! So brk and start_brk are the same .. the latter just being used previously and now forgotten there? Is there a heap end address? Is it possible to set it someway? It would be really very helpful for me!! Thank you again! Best regards! Stefano B.
[Qemu-devel] qemu-user: relocating target code weakness
Hi! I am working on a project based on qemu-user. More exactly it is qemu-ppc (version 0.13.0) with x86 host. All the project and documentation about qemu will be open for everybody as it is a project for my university that is a public one.. I have the need to relocate the target code in the memory space to some other starting address. So I went inside linux-user/elfload.c: load_elf_binary and there I found many things that according to me are someway buggy or just weak .. Of course I am only a student, and all of this was far beyond my knowledge .. I am studying like a crazy for understanding all of this.. elf stuff, linking and loading .. and I may be just confused.. nevertheless I think it is worth of some attention by some of you gurus outta there :) First of all: info-start_mmap = (abi_ulong)ELF_START_MMAP; What is this? what is start_mmap supposed to point at at the end? Why that static value is chosen at the beginning? Then there is a block of code protected by the define: #if defined(CONFIG_USE_GUEST_BASE) this define appears to be enabled in my code though I do not use any special parameter for qemu-ppc.. What is it supposed to do? I mean I guess it wants to reserve a memory area for the target code, starting at a user defined address (guest_base).. and all of that is done in linux-user/main.c: main, through a proper command line parameter, but then the guest_base address is not more used in the code of qemu-user o.O Inside the code, protected by that define in load_elf_binary, seems to check if the system can reserve the requested area of program address space, otherwise find a suitable address and set again guest_base to that address.. but oddly that variable is never more taken in any consideration then!!! Going down the code I notice that for executable target binaries the flag MAP_FIXED for mmap is set.. and in the requested starting address the variable load_bias is always zero for executable files, so that the starting address is simply the p_vaddr of the various program segments..what does it mean? Simple! A target executable code is ALWAYS loaded at the same address of the program address space it would be if run on a target machine (not on the host machine through qemu)..far enough? No for me!! When executing in the original machine the binary was created for, its OS creates an address space only for the target.. that means all the address are free for it, and it can be loaded at any address chosen by the link editor when creating the binary.. but here in qemu-user it shares the address space with qemu-user code itself (and in my case with all other I added to it!) so those addresses can be not free at all!! Isn't that a big weakness?? I went a little forward, though with my poor skills.. I forced a bias, created assembly PPC simple test binaries and had a look at what happens when a shift of the start address is required (after fixing some variables that did not consider fine such possibility): The target global variables addresses are hard coded into the target binary.. nothing in the process of elf loading or then in TCG do anything for patching those addresses when there is a shift of the starting code.. the result is simply got: segmentation fault when trying to access those variables! Moreover the way the various variables like start_code, start_data, end_data, elf_brk were set was very odd to me: It was like while cycling through all the program segments they could be set for any new segment checked! Now I must admit, through all my studying of ELF, I did not understand yet (I'd appreciate very much if you can tell me it!!!) if it is possible to have multiple code segments or multiple data segments in an executable binary.. What I did was to check if the code had the PF_X flag or PF_W flag for distinguishing whether it was a code segment or a data segment (hoping there was only one of each), and set those variables accordingly.. This approach fails anyway when I use PIC code (I am trying to use PIC code now as a solution for relocating code), it seems there are many code segments and many writable segments, and I can't understand how to set properly start_code, end_code.. etc there. Another thing I had to edit while setting those variable was making them consider as address not more p_vaddr, but the actual loading address (error) .. If any of you is interested in solving this problem I can provide all my logs for various tests, source code for the test ppc binaries for testing everything and of course my elfload.c Thank you in advance for any support! Best regards, Stefano B.
Re: [Qemu-devel] TCG flow vs dyngen
On 01/24/2011 12:40 AM, Rob Landley wrote: On 01/23/2011 04:25 PM, Stefano Bonifazi wrote: I am trying to shift in memory the target executable .. now the code is supposed to be loaded by the elfloader at the exact start address set at link time .. Ah, elf loading. That's a whole 'nother bag of worms. Oddly enough, I was deling with this last year trying to debug the uClibc dynamic linker. I blogged a bit about it at the time: http://landley.net/notes-2010.html#12-07-2010 (And the next few days. Sigh, I never did go back and fill in the holes, did I?) Inside elfloader there is even a check for verifying whether that address range is busy.. but no action is taken in that case o.O Maybe I'll post a new thread about this problem (bug?) .. anyway if you think you can help me anyway I'll give you further details.. Tired right now, but if you post a clearer question (what are you trying to _do_) and cc: me on it I'll try to respond. Maybe I can find some decent documentation to point you at, or maybe I'll write some... Rob Thank you! I read your post, and yup you also noticed the weird of load_bias.. and wondered how it can work on x86.. But I think your work was on qemu-system.. I am working on qemu-user.. Yup better to post a new thread, I'll cc: you there! Thank you very much! Stefano B
[Qemu-devel] Re: TCG flow vs dyngen
On 01/17/2011 12:59 PM, Lluís wrote: Stefano Bonifazi writes: Hi! In case you are interested in helping me, I'll give you a big piece of news I've just got (even my teacher is not informed yet! :) ) I still don't understand what is your high-level objective... Lluis Hi! Sorry I've noticed your reply only know (dunno why I was not notified by email!) Do you mean what is my final goal?
[Qemu-devel] Re: TCG flow vs dyngen
On 01/16/2011 10:08 PM, Raphaël Lefèvre wrote: 2011/1/17 Stefano Bonifazistefboombas...@gmail.com: Hi! In case you are interested in helping me, I'll give you a big piece of news I've just got (even my teacher is not informed yet! :) ) I've just managed to make more than one instance of qemu-user run at the same time linking the target code with a specified address for the code section (-Ttext address of ld). It works fine and this proves my idea that the problem is within the elf loader.. Making it relocate the target code properly would fix the problem ;) Now let's work on it :) Regards, Stefano B. Congratulation~ just keep going on~! Raphaël Lefèvre Thank you! Working on the elf loader I found out many problems on that code.. If you are interested you can have a look to my last post! Best regards! Stefano B.
[Qemu-devel] Re: TCG flow vs dyngen
On 01/16/2011 08:24 PM, Peter Maydell wrote: 2011/1/16 Stefano Bonifazistefboombas...@gmail.com: I need to make the different instances of qemu-user exchange data .. obviously keeping all of them in the same address space would be the easiest way (unless I have to change all qemu code ;) ) The problem is that you're trying to break a fundamental assumption made by a lot of qemu code. That's a large job which involves understanding, checking and possibly changing lots of already written code. In contrast, the code you need to exchange data between the instances is going to be fairly small and self contained and you'll already understand it because you've written it/will write it. I think it's pretty clear which one is going to be easier. Running each qemu as its own process and using interprocess communication for whatever coordination you need between the various instances seems more likely to be workable to me. Exactly, it was the easiest way also for me.. and I've already done it, works smoothly .. the only big problem is that it is not good for my teacher.. he says it should work the dynamic library way o.O I think he's wrong. (You might like to think about what happens if the program being emulated in qemu user-mode does a fork()). Basically you're trying to do things the hard way; maybe you can get something that sort of works in the subset of cases you care about, but why on earth put in that much time and effort on something irrelevant to the actual problem you're trying to work on? -- PMM Well my teacher's answer was that it is useless doing that, as there are already plenty of solutions based on IPC .. they are interested in this other approach, testing it .. They are not interested on how difficult it can be for a student, how long it can take.. :( Best regards, Stefano B.
[Qemu-devel] Re: [RFC/PATCH] elfload: add FDPIC support
On 01/09/2011 09:48 AM, Mike Frysinger wrote:
This is a PoC at this point, but it seems to be working for me. At
least, all the current crashes I'm seeing are due to my Blackfin port
being incomplete. All of the FDPIC table parsing seems to be OK ...
If someone with a more functional target would like to try this, that'd
be cool. Or if people want to give feedback on how to approach this
problem so I can adjust the details now.
Signed-off-by: Mike Frysingervap...@gentoo.org
---
elf.h| 19 ++
linux-user/elfload.c | 67 ++
linux-user/qemu.h|8 ++
3 files changed, 94 insertions(+), 0 deletions(-)
diff --git a/elf.h b/elf.h
index 7067c90..d2f24f4 100644
--- a/elf.h
+++ b/elf.h
@@ -1191,6 +1191,25 @@ typedef struct elf64_note {
Elf64_Word n_type; /* Content type */
} Elf64_Nhdr;
+
+/* This data structure represents a PT_LOAD segment. */
+struct elf32_fdpic_loadseg {
+ /* Core address to which the segment is mapped. */
+ Elf32_Addr addr;
+ /* VMA recorded in the program header. */
+ Elf32_Addr p_vaddr;
+ /* Size of this segment in memory. */
+ Elf32_Word p_memsz;
+};
+struct elf32_fdpic_loadmap {
+ /* Protocol version number, must be zero. */
+ Elf32_Half version;
+ /* Number of segments in this map. */
+ Elf32_Half nsegs;
+ /* The actual memory map. */
+ struct elf32_fdpic_loadseg segs[/*nsegs*/];
+};
+
#ifdef ELF_CLASS
#if ELF_CLASS == ELFCLASS32
diff --git a/linux-user/elfload.c b/linux-user/elfload.c
index 33d776d..8100ffd 100644
--- a/linux-user/elfload.c
+++ b/linux-user/elfload.c
@@ -1075,6 +1075,32 @@ static void zero_bss(abi_ulong elf_bss, abi_ulong
last_bss, int prot)
}
}
+#ifdef CONFIG_USE_FDPIC
+static abi_ulong loader_build_fdpic_loadmap(struct image_info *info, abi_ulong
sp)
+{
+uint16_t n;
+struct elf32_fdpic_loadseg *loadsegs = info-loadsegs;
+
+/* elf32_fdpic_loadseg */
+for (n = 0; n info-nsegs; ++n) {
+sp -= 12;
+put_user_u32(loadsegs[n].addr, sp+0);
+put_user_u32(loadsegs[n].p_vaddr, sp+4);
+put_user_u32(loadsegs[n].p_memsz, sp+8);
+}
+
+/* elf32_fdpic_loadmap */
+sp -= 4;
+put_user_u16(0, sp+0); /* version */
+put_user_u16(info-nsegs, sp+2); /* nsegs */
+
+info-personality = PER_LINUX_FDPIC;
+info-loadmap_addr = sp;
+
+return sp;
+}
+#endif
+
static abi_ulong create_elf_tables(abi_ulong p, int argc, int envc,
struct elfhdr *exec,
struct image_info *info,
@@ -1087,6 +1113,21 @@ static abi_ulong create_elf_tables(abi_ulong p, int
argc, int envc,
const int n = sizeof(elf_addr_t);
sp = p;
+
+#ifdef CONFIG_USE_FDPIC
+/* Needs to be before we load the env/argc/... */
+if (elf_is_fdpic(exec)) {
+/* Need 4 byte alignment for these structs */
+sp= ~3;
+sp = loader_build_fdpic_loadmap(info, sp);
+info-other_info = interp_info;
+if (interp_info) {
+interp_info-other_info = info;
+sp = loader_build_fdpic_loadmap(interp_info, sp);
+}
+}
+#endif
+
u_platform = 0;
k_platform = ELF_PLATFORM;
if (k_platform) {
@@ -1197,6 +1238,11 @@ static void load_elf_image(const char *image_name, int
image_fd,
}
bswap_phdr(phdr, ehdr-e_phnum);
+#ifdef CONFIG_USE_FDPIC
+info-nsegs = 0;
+info-pt_dynamic_addr = 0;
+#endif
+
/* Find the maximum size of the image and allocate an appropriate
amount of memory to handle that. */
loaddr = -1, hiaddr = 0;
@@ -1210,6 +1256,11 @@ static void load_elf_image(const char *image_name, int
image_fd,
if (a hiaddr) {
hiaddr = a;
}
+#ifdef CONFIG_USE_FDPIC
+++info-nsegs;
+} else if (phdr[i].p_type == PT_DYNAMIC) {
+info-pt_dynamic_addr = phdr[i].p_vaddr;
+#endif
}
}
@@ -1290,6 +1341,22 @@ static void load_elf_image(const char *image_name, int
image_fd,
}
load_bias = load_addr - loaddr;
+#ifdef CONFIG_USE_FDPIC
+{
+struct elf32_fdpic_loadseg *loadsegs = info-loadsegs =
+qemu_malloc(sizeof(*loadsegs) * info-nsegs);
+
+for (i = 0; i ehdr-e_phnum; ++i) {
+if (phdr[i].p_type != PT_LOAD)
+continue;
+loadsegs-addr = phdr[i].p_vaddr + load_bias;
+loadsegs-p_vaddr = phdr[i].p_vaddr;
+loadsegs-p_memsz = phdr[i].p_memsz;
+ ++loadsegs;
+}
+}
+#endif
+
info-load_bias = load_bias;
info-load_addr = load_addr;
info-entry = ehdr-e_entry + load_bias;
diff --git a/linux-user/qemu.h b/linux-user/qemu.h
index 32de241..0924a1a 100644
--- a/linux-user/qemu.h
+++ b/linux-user/qemu.h
@@ -51,6 +51,14 @@ struct image_info {
abi_ulong arg_start;
abi_ulong arg_end;
int
Re: [Qemu-devel] Re: TCG flow vs dyngen
On 01/24/2011 02:36 PM, Lluís wrote: Stefano Bonifazi writes: Do you mean what is my final goal? Exactly. A higher level perspective of what is our ultimate goal might help others figure out better ways to do it. Right now I don't remember what you posted your where technically trying to do, but I do remember it looked convoluted to me. Lluis Sorry if I could not explain it better before, but it was not totally clear for me too since the beginning, as I get new specs from my teacher on the way, according what I manage to do, and where I find big obstacles! Now, the final goal is to get multiple instances of qemu-ppc driven by a systemc project executing on a x86 machine, with the different qemu-ppc instances used as emulators for power-pc binaries.. I would get the results of the run of the various ppc binaries back to the systemc project and work with the results then. I've already managed to integrate systemc with qemu-ppc, and I managed to load multiple instances of qemu together, by loading it as a dynamic library. I think much confusion about my goals was originated by the fact that the first attempt (failed) was to use qemu-user for loading many target binaries one after the other.. Then I changed for having many instances of qemu-user at the same time inside the same process.. The actual problem is letting qemu-user able of loading target code at a different address than the one chosen by the link editor when creating the binary.. If you are interested in that I've just created a new post about it: http://lists.nongnu.org/archive/html/qemu-devel/2011-01/msg02361.html Best regards, Stefano B.
Re: [Qemu-devel] TCG flow vs dyngen
On 01/24/2011 03:32 PM, Peter Maydell wrote: Being a JIT doesn't prohibit counting target instructions executed. It just means that counting them generally requires generating code to do the counting at runtime, so it's a more complicated change to make than it would be in a non-JIT emulator. What do you mean? Should I change the code of qemu-user for counting the instructions, or should I add code into the target binaries? The major reason for not counting cycles is that for an emulation of a modern CPU this is pretty nearly impossible: the number of cycles an instruction takes can depend on whether it causes a cache miss, which CPU internal pipeline it uses, whether it needs to stall waiting for a result from an earlier insn, whether the CPU correctly predicted the branch leading up to it or not, and on and on. You would need to precisely model all the internals of each variant of each CPU, which would be a mammoth undertaking requiring probably unpublished internal data, and if you ever managed to finish it then it would run incredibly slowly and would probably contain enough bugs you couldn't trust the data it gave you anyway. Yup, I think it was just a silly mistake of mine when in the first post I wrote cycles.. that was because for me anything that can estimate how long it takes to do the work would be fine.. I can't simply check the time because that is host machine dependent... Number of executed instructions would be fine.. This means that QEMU can no longer run on a type of host it can't execute target code for This isn't correct; for instance there's hppa support in TCG for hppa hosts but no hppa target support, and there's sh4 target support but no TCG backend for it. The two ends are cleanly separated in qemu and don't generally depend on each other. Well I experienced a strange behavior some time ago that initially made me think mr Rob was right on that though I knew host support and target support were separated in qemu: I tried to make directly qemu-ppc on a x86_64 machine from inside ppc-linux-user folder (i can do fine onto x86 machine) and it failed because there was no tgc/x86_64/tcg_target.h, whereas doing the make from within the main folder worked. So I do not understand very well.. is there some required headers fix when using the main make file? Best regards! Stefano B.
Re: [Qemu-devel] Re: TCG flow vs dyngen
Hi! Thank you for answering me! If I understand this correctly, the execution of one of your PPC cores is oblivious of the others (they share no guest physical memory). No! They do share the same address space.. the way I am loading the different qemu-ppc instances divides their namespaces allowing them to coexist, but they share the same address space anyway (that is the same of the caller process too), that is what I want for a communication. The problem is that they are at the same time oblivious of the others and each of them wants to map its target binary at the same unique virtual address (again see my last post about relocating target code).. I tried successfully the way of IPC (interprocess communication) having a different qemu-ppc spawned by systemc as a process, then using shared memory and signals for communicating.. pretty easy and well working, but the specs of my project (university) do not let me using IPC.. [1] http://sites.google.com/site/hplabscotson/ Thank you, I am a student of digital electronics, with not big knowledge about developing in linux but this project is very interesting for my field.. some sort of alternative to systemc if I understand fine! Thanks surely I'll have a look at that! Best regards! Stefano B.
Re: [Qemu-devel] qemu-user: relocating target code weakness
Hi! Thank you for answering me! Start address in which address the ELF code section, in reality (not under qemu-user) should be mapped. in x86 32 bit it's 08048000, you can check it by yourself by executing: cat /proc/self/maps Wait, like that I don't understand anything more.. As far as I understood when calling mmap from within a process, its result, when no error happen, would be the virtual address where the mapping starts.. so if I get 0x1000 from mmap why should it be 08048000 instead?? Inside load_elf_binary the call to mmap has elf_ppnt-p_vaddr as required starting address (plus alignment) and MAP_FIXED flag.. so what I get is elf_ppnt-p_vaddr as starting map address (if the call does not fail) .. so I do not understand where does that address ELF_START_MMAP get any role? however, as you already know, it might be relocated elsewhere if the code is PIC or PIE (Position Independent Executable). Well PIC target code, inside load_elf_binary would have elf_ex.e_type == ET_DYN so the mmap would be not flagged with MAP_FIXED, and the starting address of the mapping would be chosen by the memory manager .. again ELF_START_MMAP is not used... I wrote an article about understanding ELF years ago, here is the URL: http://www.linuxforums.org/articles/understanding-elf-using-readelf-and-objdump_125.html Will you believe me if I tell you I printed that document some hours ago without noticing you were the author? :) I downloaded and read so much material from the Internet for studying this stuff!! what is start_mmap supposed to point at at the end? Why that static value is chosen at the beginning? in qemu 0.13.0, linux-user/elfload.c, lines around 1728 to 1761, you will find that start_mmap is kinda an address that is the result of mmap operation that will be filled with the code in code section. It will be a hint on which address the guest code really points when dealing with memory address. No, not for my understanding: that code for me searches for a suitable mapping area, I can't see info-start_mmap being set anywhere else.. surely not among those lines of code I dare to guess that, every code executed in the guest...when referring to virtual address, will be get substracted by the delta of ELF_START_MMAP and real_base. It's like hey, it's written A in the code, but it's mapped in B, ok so the delta is A-B, call it X. Then every address in the code should be substracted (or added, depending on how you see it) with X, then it will point in qemu mmaped VMA Wrong, at least for what I did understand, and from the tests I did.. I compared different input - output assembly code, hardcoded target binary addresses like those of static global variables (not of shared modules) are not changed at all, producing segmentation faults when forcing a shift of the mapping.. See the codes, read it slowly, you wil get the idea. I did, I do, I am changing the code for testing it.. fixing it.. but I can't see any relocation PS: IMHO the real guru is still the one and only Fabrice Bellard, too bad he's out of qemu. How is it possible that the creator of all of this is out? Thank you very much for your help! Best regards! Stefano B.
Re: [Qemu-devel] TCG flow vs dyngen
On 01/24/2011 07:02 PM, Dushyant Bansal wrote: On Monday 24 January 2011 08:26 PM, Stefano Bonifazi wrote: On 01/24/2011 03:32 PM, Peter Maydell wrote: Being a JIT doesn't prohibit counting target instructions executed. It just means that counting them generally requires generating code to do the counting at runtime, so it's a more complicated change to make than it would be in a non-JIT emulator. What do you mean? Should I change the code of qemu-user for counting the instructions, or should I add code into the target binaries? You should see this pdf (www.ecs.syr.edu/faculty/yin/Teaching/TC2010/Proj4.pdf). It talks about tracing the instructions. -- Dushyant Wow thank you! It sounds incredibly interesting!! What we really need is to insert a function call into the translated code, so when each instruction is executed at runtime, our inserted function will be executed. Again wow!! Is that really possible? Some sort of callback triggered at every instruction execution? Do you have any another document explaining that? This pdf just gives instructions on how to do it on an old version of qemu (disas_insn doesn't exist at all on my code now), and does not explain what it is, what's behind that suggested code .. Also the code for single step would be of great help to me! I really needed that.. but when I tried it on qemu-user didn't work at all.. Thank you very much! Best regards, Stefano B.
Re: [Qemu-devel] qemu-user: relocating target code weakness
Hi! Thank you for your answer! he is telling you what ELF_START_MMAP is all about. it is the base address that the linux kernel for that architecture will start giving out addresses. so when running Linux on an x86 system, the first mmap() a process does will start at 0x8000 and move up. although looking at the elfload code quickly, it doesnt seem that this is really used anymore. so it probably can be ignored. Uhmm that makes sense and explains many things like why they can simply consider always possible that the target is allocated at elf_ppnt-p_vaddr, without risking it to clash with qemu-user code.. Inside load_elf_binary the call to mmap has elf_ppnt-p_vaddr as required starting address (plus alignment) and MAP_FIXED flag. review the mmap() man page ... MAP_FIXED is always a *suggestion* and never a requirement. the app must check the return value to see what the kernel actually gave it. Sorry, wrong! *MAP_FIXED* Don't interpret/addr/ as a hint: place the mapping at exactly that address. [] If the specified address cannot be used,*mmap*() will fail.[] http://www.kernel.org/doc/man-pages/online/pages/man2/mmap.2.html Thank you! Best regards! Stefano B.
Re: [Qemu-devel] Re: [RFC/PATCH] elfload: add FDPIC support
On 01/24/2011 08:11 PM, Mike Frysinger wrote: On Mon, Jan 24, 2011 at 08:34, Stefano Bonifazi wrote: Is FDPIC something different than simply PIC code (position independent code)? FDPIC ELF is the ELF PIE format used on NOMMU systems so that both the text and data regions may be located anywhere. it is the only ELF format supported under NOMMU systems. I am also trying to fight with the problem of changing the starting address of target code for qemu-user, and I was just moving into the option of using PIC target code .. but the original qemu-user load_elf_binary does not work on them.. and I was just about to try to edit it.. i dont believe my patch accomplishes that in any way. i will need to force all files to be loaded at a higher address than zero for the Blackfin arch (as the first ~4KiB is reserved for the ABI), but that is independent of FDPIC ELF support. -mike I don't understand.. what is the difference between pie binary for pcc and for your architecture? As far as I understood pie code is independent from addresses, so it should not care if the OS running them would have an mmu at all.. it should be just the task of the dynamic linker to relocate it properly, am I wrong? Thank you! Regards, Stefano B.
Re: [Qemu-devel] TCG flow vs dyngen
Hi! Thanks for replying me! The thing is, the kernel currently _does_ work, so studying the relevant kernel code (and possibly the dynamic loader code) is one way to learn how it currently works. Sorry what kernel? Qemu's? Linux's?
Re: [Qemu-devel] qemu-user: relocating target code weakness
but that can happen with the app running natively too, so any app not handling MAP_FIXED failures is buggy and not qemu's problem. -mike How? For what I learned each process executing on a OS with an mmu sees its virtual address space, and normally only its code is loaded there (well I am learning that the dynamic linker also inject into that space shared library code used by the process code, but of course a good dynamic linker would prevent clashes!) so how can it happen that it can clash with anything?? If I remember fine, at the time of DOS there were some addresses reserved for the OS, and then only one executable could run at time, but in modern time with virtual addressing I don't think it is possible other than in scenarios alike ours.. Regards, Stefano B.
Re: [Qemu-devel] Re: [RFC/PATCH] elfload: add FDPIC support
On 01/24/2011 10:27 PM, Mike Frysinger wrote: On Mon, Jan 24, 2011 at 16:06, Stefano Bonifazi wrote: I don't understand.. what is the difference between pie binary for pcc and for your architecture? as i said, i think this is all irrelevant to what you want to do. but since you asked and i feel like writing ... i have no idea what pcc is. there are really two stark differences between FDPIC ELF and a PIE ELF on say x86. since the data/text sections can be relocated independently of each other, the PIC cannot assume a fixed offset between its text and GOT. so every PLT entry is actually two sets of addresses -- the function address and the function's GOT address. the other big difference is that an FDPIC ELF app must first do a little relocation processing of itself as soon as it starts ... in order to do so, the kernel provides a loadmap (made up of multiple loadsegs) which describes the executable's PT_LOADs (the addr encoded in the program header and the addr the chunk was actually relocated to) as well as the (optional) executable's interpreter's PT_LOADs. As far as I understood pie code is independent from addresses, so it should not care if the OS running them would have an mmu at all.. it should be just the task of the dynamic linker to relocate it properly, am I wrong? but the PIE code still has fixed offsets between its text and its data. so FDPIC ELF is even more flexible than a PIE ELF. -mike Thank you very much! As a student, understanding how things work is even more precious than making things work ;) You know? I was just trying to figure out today how the code could locate the .got in x86 when relocation is needed.. I am studying ELF and relocation now, reading all possible material online, but I did not find yet about the fixed offset! PPC stands for PowerPC. Now also your code is much clearer for me, though not useful in my case.. Thank you again! Best regards! Stefano B.
Re: [Qemu-devel] qemu-user: relocating target code weakness
how can the ldso possibly prevent clashes ? it has no idea what addresses an app will ask for at runtime. plus, if the kernel is employing ASLR (which isnt uncommon nowadays), the load addresses could be anywhere. -mike Well not alone, in my mind ld.so asks the memory manager through calling mmap where it can map the shared modules inside the process address space.. the memory manager should know what addresses are free and which are taken .. then when the process code requires dynamic memory allocation, it does it again through the memory manager (i.e. malloc) avoiding of allocating memory where shared modules have been loaded into.. Again with ASLR I think the mmu should be aware of all used memory slots, avoiding conflicts.. Well your view of random possible clashes maybe is what happens or happened in Windows systems :D But Linux is supposed to be different, isn't it? Best regards, Stefano B.
Re: [Qemu-devel] qemu-user: relocating target code weakness
Yes. Have a look at http://lists.gnu.org/archive/html/qemu-devel/2010-07/msg01626.html where I tried to clean this up last year. The patch never got properly reviewed, however. All that said, unless you have an executable that's been properly prepared for relocation, e.g. an ET_DYN binary instead of a normal ET_EXEC binary, you will *not* have enough information to do what you're suggesting. r~ Wow wonderful! So you fixed the code for PIC (ET_DYN) support? I won't have any problem in producing the target code into pie format, as I am writing, compiling and linking my target code by my own ;) A noob question, how can I get your sources? Is there a simpler solution than copypaste all the code from your messages into patches and then applying them? Can you just send your sources by email? Or can I download them from a site? I am very eager to study your code and to try it :) Thank you very much!! Stefano B.
Re: [Qemu-devel] TCG flow vs dyngen
On 01/23/2011 10:50 PM, Rob Landley wrote: On 01/16/2011 10:01 AM, Raphaël Lefèvre wrote: On Sun, Jan 16, 2011 at 11:21 PM, Stefano Bonifazi stefboombas...@gmail.com wrote: 2. how can I check the number of target cpu cycles or target instructions executed inside qemu-user (i.e. qemu-ppc)? Is there any variable I can inspect for such informations? at Dec, 2010 Keep in mind I'm a bit rusty and not an expert, but I'll give a stab at answering: You can't, because QEMU doesn't work that way. QEMU isn't an instruction level emulator, it's closer to a Java JIT. It doesn't translate one instruction at a time but instead translates large blocks of code all at once, and keeps a cache of translated blocks around. Execution jumps into each block and either waits for it to exit again (meaning it jumped out of that page and QEMU's main execution loop has to look up what page to execute next, possibly translating it first if it's not in the cache yet), or else QEMU interrupts it after while to fake an IRQ of some kind (such as a timer interrupt). You may want to read Fabrice Bellard's original paper on the QEMU design: http://www.usenix.org/event/usenix05/tech/freenix/full_papers/bellard/bellard.pdf Since that was written, dyngen was replaced with tcg, but that does the same thing in a slightly different way. Building a QEMU with dyngen support used to use the host compiler to compile chunks of code corresponding to the target operations it would see at runtime, and then strip the machine language out of the resulting .o files and save them in a table. Then at runtime dyngen could generate translated pages by gluing together the resulting saved machine language snippets the host compiler had produced when qemu was built. The problem was, beating the right kind of machine language snippets out of the .o files the compiler produced from the example code turned out to be VERY COMPILER DEPENDENT. This is why you couldn't build qemu with gcc 4.x for the longest time, gcc's code generator and the layout of the .o files changed in a bunch of subtle ways which broke dyngen's ability to extract usable machine code snippets to put 'em into the table so it could translate pages at runtime. TCG stands for Tiny Code Generator. It just hardwires a code generator into QEMU. They wrote a mini-compiler in C, which knows what instructions to output for each host qemu supports. If QEMU understands target instructions well enough to _read_ them, it's not a big stretch to be able to _write_ them when running on that kind of host. (It's more or less the same operation in reverse.) This means that QEMU can no longer run on a type of host it can't execute target code for, but the solution is to just add support for all the interesting machines out there, on both sides. So, when QEMU executes code, the virtual MMU faults a new page into the virtual TLB, and goes I can't execute this, fix it up! And the fixup handler looks for a translation of the page in the cache of translated pages, and if it can't find it it calls the translator to convert the target code into a page of corresponding host code. Which may involve discarding an existing entry out of the cache, but this is how instruction caches work on real hardware anyway so the delays in QEMU are where they'd be on real hardware anyway, and optimizing for one is pretty close to optimizing for the other, so life is good. The chunk you found earlier is a function pointer typecast: #define tcg_qemu_tb_exec(tb_ptr) \ ((long REGPARM (*)(void *))code_gen_prologue)(tb_ptr) Which looks like it's calling code_gen_prologue() with tp_ptr as its argument (typecast to a void *), and it returns a long. That calls a translated page, and when the function returns that means the page of code needs to jump to code somewhere outside of that page, and we go back to the main loop to figure out where to go next. The reason QEMU is as fast as it is is because once it has a page of translated code, actually _running_ it is entirely native. It jumps into the page, and executes natively until it leaves the page. Control only goes back to QEMU to switch pages or to handle I/O and interrupts and such. So when you ask how many clock cycles did that instruction take, the answer is it doesn't work that way. QEMU emulates at memory page level (generally 4k of target code), not at individual instruction level. (Oh, and the worst thing you can do to QEMU from a performance perspective is self-modifying code. Because the virtual MMU has to strip the executable bit off the TLB entry and re-translate the entire page next time something tries to execute it. It _works_, it's just slow. But again, real hardware can hiccup a bit on this too.) Does that answer your question? Rob Wow! Thank you! That's an ANSWER! Gold for who's studying all of that! Though at the stage of my work I had to understand almost all of it, your perfect summary make everything much clearer.. About counting
[Qemu-devel] Changing the content of target cpu registers
Hi all! I am working on qemu-user (qemu-ppc). I'd like to edit the values of target registers during the execution. Can I do that by simply changing the content of env-gpr[] or do these only contain a copy of the values of the registers? In this last case, where are the real values of the target registers stored so that by modifying them I can alter the behavior of the target code execution? Thank you in advance! Stefano B.
Re: [Qemu-devel] Changing the content of target cpu registers
On 01/18/2011 06:17 PM, Blue Swirl wrote: On Tue, Jan 18, 2011 at 9:29 AM, Stefano Bonifazi stefboombas...@gmail.com wrote: Hi all! I am working on qemu-user (qemu-ppc). I'd like to edit the values of target registers during the execution. Can I do that by simply changing the content of env-gpr[] or do these only contain a copy of the values of the registers? In this last case, where are the real values of the target registers stored so that by modifying them I can alter the behavior of the target code execution? env-gpr is the canonical location, but the translator assigns TCG variables to them (cpu_gpr[] in translate.c), so GPR contents may be cached to these. But when helpers are called or the TB finishes, env-gpr should be valid again. Hi! Thank you for your answer! So if I understand well if I set env-gpr in a code section where there is no TCG translation on progress, I can edit directly the target CPU register right? Best Regards! Stefano B.
Re: [Qemu-devel] TCG flow vs dyngen
On 01/16/2011 03:46 PM, Raphael Lefevre wrote: On Wed, Dec 15, 2010 at 4:17 AM, Stefano Bonifazi stefboombas...@gmail.com wrote: On 12/11/2010 03:44 PM, Blue Swirl wrote: Hi! Thank you very much! Knowing exactly where I should check, in a so big project helped me very much!! Anyway after having spent more than 2 days on that code I still can't understand how it works the real execution: in cpu-exec.c : cpu_exec_nocache i find: /* execute the generated code */ next_tb = tcg_qemu_tb_exec(tb-tc_ptr); and in cpu-exec.c : cpu_exec /* execute the generated code */ next_tb = tcg_qemu_tb_exec(tc_ptr); so I thought tcg_qemu_tb_exec function should do the work of executing the translated binary in the host. But then I found out it is just a define in tcg.h: #define tcg_qemu_tb_exec(tb_ptr) ((long REGPARM (*)(void *))code_gen_prologue)(tb_ptr) and again in exec.c uint8_t code_gen_prologue[1024] code_gen_section; Maybe I have some problems with that C syntax, but I really don't understand what happens there.. how the execution happens! Here instead with QEMU/TCG I understood that at runtime the target binary is translated into host binary (somehow) .. but then.. how can this new host binary be run? Shall the host code at runtime do some sort of (assembly speaking) branch jump to an area of memory with new host binary instructions .. and then jump back to the old process binary code? 1. As I know, the host codes translated from the target instructions exist by the format of object file, that’s why they can be executed directly. 2. I think you catch the right concept in some point of view, one part of the internal of QEMU does such jump back works certainly. If so, can you explain me how this happens in those lines of code? I only can give a rough profile, the code you listed do a simple thing: Modify the pointer of the host code execution to point the next address that the host processor should continue to execute. I am just a student.. unluckily at university they just tell you that a cpu follows some sort of fetch -decode-execute flow .. but then you open QEMU.. and wow there is a huge gap for understanding it, and no books where to study it! ;) The QEMU is not used to simulate the every details of the processor should behave, it just try to approximate the necessary operations what a machine should be! “fetch-decode-execute” flow only need to be concerned when you involve into the hardware design. Raphaël Lefèvre Thank you very much! I've already solved this problem.. Right now I am fighting with the possibility of changing qemu-user code for making it run several binaries in succession .. But it seems to remember the first translated code.. Nobody answered to my post about it, do you have any idea?
Re: [Qemu-devel] TCG flow vs dyngen
Sorry for my belated on this discussion, after I searched for the topics you posted, it seems two main problems are unsolved? (Am I right?? I'm not sure...) 1. I edited QEMU user, more exactly qemu-ppc launching the main function (inside main.c) from another c function I created, passing it the appropriate parameters. ...balabala at Jan, 2011 2. how can I check the number of target cpu cycles or target instructions executed inside qemu-user (i.e. qemu-ppc)? Is there any variable I can inspect for such informations? at Dec, 2010 If I'm not correct, please let me know where the problem is. Raphaël Lefèvre Hi! Thank you very much for Your concern! Honestly I had lost hope in any help, I even contacted directly some developers in this mailing list without luck! I am a student who needs to use qemu for a project where it will be used for its capabilities of running PowerPC code. As you can imagine qemu goes far beyond the knowledge in electronics and computer science of a student. Nevertheless I have to do that! I have been studying all the possible technical documents available in the internet, but it is really not much at all , not sufficient for getting the code and being able of understanding it .. It is in C, even not modular C++ Anyway with some help from this mailing list, and a lot of studying about assembly, loaders, compilers.. I am going on, though there are still big problems due of the nature of the QEMU code.. First of all, I am starting from qemu-user, more specifically, qemu-ppc as I don't need the full system capabilities, and it is easier for me to control the binary target memory with qemu-user. Originally I started with a lot of work on libqemu .. until some developer here told me it was deprecated (though still in the source) and not working fine. I edited the code of qemu-ppc so that another function of mine calls qemu-user main, with the appropriate parameters.. The pursued goal was to launch it several times with different target binaries in succession.. For some reason, I still can't find out, qemu code remembers the old code, running it instead of the new loaded binary.. and if I flush the cache of translated code before loading a new binary it stops and can't go on! My workaround to this problem was compiling qemu-ppc as a dynamic library and load it at runtime.. I also managed to load multiple copies of it (with dlmopen each at a different address space) ..in fact I need to run more than one qemu-ppc at the same time but a new big problem popped up now: the target binary is loaded always at a fixed address.. no matter if another qemu-ppc already loaded code there.. it is like the internal elf loader can't understand those addresses are not available, and then relocate them .. I tried to link (ld) the binary target elf as position independent code, but then qemu-ppc complains it can't find /usr/lib/libc.so.1 and /usr/lib/ld.so.1 To sum up the problems are (in order of importance): - making the elf loader relocate the target code into other addresses when the default ones (I guess those embedded into the target binary when it is not compiled as position independent code) are taken - making qemu-user able of running more than one target binary in succession - counting qemu-user executed instructions My university is a public one, so my project will be open to the community, I will also upload the documentation I am writing about qemu coming from the knowledge I am acquiring working on it, so that, I hope, other people will find less frustrating the first steps into developing qemu! Any help will be more than welcome! Thank you in advance! Stefano B.
Re: [Qemu-devel] TCG flow vs dyngen
Thank you very much for Your fast reply! On 01/16/2011 07:29 PM, Peter Maydell wrote: Linux doesn't seem to have dlmopen http://www.unix.com/man-page/All/3c/dlmopen/ #define __USE_GNU #include dlfcn.h lib_handle1 = dlmopen(LM_ID_NEWLM,./libqemu-ppc.so, RTLD_NOW); I am developing that on a clean ubuntu 10.10 but google suggests that it puts the library in its own namespace but not its own address space. I need to make the different instances of qemu-user exchange data .. obviously keeping all of them in the same address space would be the easiest way (unless I have to change all qemu code ;) ) Running each qemu as its own process and using interprocess communication for whatever coordination you need between the various instances seems more likely to be workable to me. This will also fix your can't run more than one binary in succession problem, because you can just have the first qemu run and exit as normal and launch a second qemu to run the second binary. -- PMM Exactly, it was the easiest way also for me.. and I've already done it, works smoothly .. the only big problem is that it is not good for my teacher.. he says it should work the dynamic library way o.O Working with libraries even solved the problem of consecutive runs, though according to me it is not good a software when you must reboot it for making it run again fine.. sounds more Windows style :D Clearly it makes memory dirty and do not clean after the target process completes its execution.. leaving the OS care about it. I tried zeroing all global variables before starting a new execution without results (other than making it stall) .. After very long time spent trying to find a solution I think the problem should be with the mmap' ings stuff in the loader .. the same reason why 2 different libraries with their own namespaces clash according to me.. the elf loaders work globally within the unique address space .. I think for a guru of loaders-linkers should not be so difficult to patch it.. but not for a student who almost heard about them for the first time ;) Any help is very appreciated :) Thank you again! Stefano B.
Re: [Qemu-devel] TCG flow vs dyngen
Hi! In case you are interested in helping me, I'll give you a big piece of news I've just got (even my teacher is not informed yet! :) ) I've just managed to make more than one instance of qemu-user run at the same time linking the target code with a specified address for the code section (-Ttext address of ld). It works fine and this proves my idea that the problem is within the elf loader.. Making it relocate the target code properly would fix the problem ;) Now let's work on it :) Regards, Stefano B. On 01/16/2011 08:02 PM, Stefano Bonifazi wrote: Thank you very much for Your fast reply! On 01/16/2011 07:29 PM, Peter Maydell wrote: Linux doesn't seem to have dlmopen http://www.unix.com/man-page/All/3c/dlmopen/ #define __USE_GNU #include dlfcn.h lib_handle1 = dlmopen(LM_ID_NEWLM,./libqemu-ppc.so, RTLD_NOW); I am developing that on a clean ubuntu 10.10 but google suggests that it puts the library in its own namespace but not its own address space. I need to make the different instances of qemu-user exchange data .. obviously keeping all of them in the same address space would be the easiest way (unless I have to change all qemu code ;) ) Running each qemu as its own process and using interprocess communication for whatever coordination you need between the various instances seems more likely to be workable to me. This will also fix your can't run more than one binary in succession problem, because you can just have the first qemu run and exit as normal and launch a second qemu to run the second binary. -- PMM Exactly, it was the easiest way also for me.. and I've already done it, works smoothly .. the only big problem is that it is not good for my teacher.. he says it should work the dynamic library way o.O Working with libraries even solved the problem of consecutive runs, though according to me it is not good a software when you must reboot it for making it run again fine.. sounds more Windows style :D Clearly it makes memory dirty and do not clean after the target process completes its execution.. leaving the OS care about it. I tried zeroing all global variables before starting a new execution without results (other than making it stall) .. After very long time spent trying to find a solution I think the problem should be with the mmap' ings stuff in the loader .. the same reason why 2 different libraries with their own namespaces clash according to me.. the elf loaders work globally within the unique address space .. I think for a guru of loaders-linkers should not be so difficult to patch it.. but not for a student who almost heard about them for the first time ;) Any help is very appreciated :) Thank you again! Stefano B.
[Qemu-devel] HELP PLEASE! Consecutive runs of qemu-user
Hi! First of all Happy new year to everybody! :) Wish you all to realize all your dreams! :) I edited QEMU user, more exactly qemu-ppc launching the main function (inside main.c) from another c function I created, passing it the appropriate parameters. I also caught the syscall 1 preventing it to abort the process, and simply returning from cpu_loop function, so that this function is not more endless, and main returns to my user defined function. The problem is that launching a second time main with other parameters fails: leaving the code like it was, the previous translated binary was kept in memory and run again, no matter what binary parameter I passed to main. I added then tb_flush(env); after cpu_loop inside main.. But with this edit launching main with other parameters blocks the execution at tcg_qemu_tb_exec(tc_ptr); ..It sounds like QEMU stores the state of the translation even after main function is over :( Is it possible to reset completely the state, allowing consecutive runs of different binaries? What are the state variables I should clean between different runs? Thank you very much for your help! Best regards! Stefano B.
Re: [Qemu-devel] checking the number of target cpu cycles or instructions executed
On 12/23/2010 06:42 PM, Andreas Färber wrote: Hi, Am 22.12.2010 um 13:19 schrieb Stefano Bonifazi: how can I check the number of target cpu cycles or target instructions executed inside qemu-user (i.e. qemu-ppc)? Is there any variable I can inspect for such informations? QEMU's emulation is not cycle-accurate, so you will not be able to retrieve CPU cycle info to my knowledge. As for instructions, take a look at the -icount option. Andreas Hi! Thank You very much! I understood it was not a cycle accurate emulator.. In my case I am more interested in how many instructions the target machine would execute rather than how many host cycles or instructions..so I think counted fetched target instructions can be good.. Anyway how to switch on -icount option for user emulation? Setting use_icount to 1 (or2) into exec.c does not work: the execution hangs, and qemu_icount is not incremented.. Thank you! Best Regards, Stefano B.
[Qemu-devel] checking the number of target cpu cycles or instructions executed
Hi all! :) how can I check the number of target cpu cycles or target instructions executed inside qemu-user (i.e. qemu-ppc)? Is there any variable I can inspect for such informations? Thank you very much in advance! Stefano B.
Re: [Qemu-devel] libqemu.a not made with version 0.13
On 12/20/2010 07:31 PM, Blue Swirl wrote: Are you aware of QEMU-SystemC: http://www.greensocs.com/projects/QEMUSystemC Perhaps that would be a better starting point than plain QEMU. Hi! Thank you! Sure I already checked this project! I googled all the web for QEMU technical documents:) Even Chinese pages :D QEMU-SystemC is great for electronic engineers, offering mixed simulation capabilities between QEMU devices and early SystemC designs (modules).. Engineers can test interaction of new devices since their first design with full systems .. Anyway that project simply allows QEMU to plug in a full architecture new SystemC modules through a bridge .. My goal is opposite, I want SystemC to use a little part of QEMU only as a binary translation engine between PPC and i386 machines .. Moreover that project is not updated to last QEMU version.. Surely I'll go back to it in case I don't manage to make SystemC sources compile fine with QEMU stuff .. Thank You! Best Regards! Stefano B.
[Qemu-devel] Re: Problems executing qemu-ppc
Hi! I am answering myself hoping that my solution may help somebody other who has to face the same problem: In QEMU home page, under downloads section you can find QEMU Linux user mode tests: http://wiki.qemu.org/download/linux-user-test-0.3.tar.gz Inside there are also examples for qemu-ppc Anyway what one probably wants to obtain with qemu-ppc is not just running the tests programs provided there. Instead probably you want to test PPC programs you compile for your needs. Unless you compile your programs with the same PPC Linux kernel whose libs are contained in the downloaded archive, they won't run fine with qemu-ppc using the gnemul libs provided in the downloaded archive. I don't have any powerPC machine, so I use a QEMU virtual machine (qemu-system-ppc) with installed Debian 5.07 PowerPC for compiling PowerPC programs. So what I did for making qemu-ppc run fine my PPC programs was simply to copy all the following files: /lib/ld* /lib/lib* from the Debian PPC linux (virtual machine) into /usr/gnemul/qemu-ppc/lib/ of my real i386 Linux system (ubuntu 10.10, i386) and /etc/ld* from the Debian PPC Linux into /usr/gnemul/qemu-ppc/etc/ of my real i386 Linux system After that running qemu-ppc myproggy in my real i386 Linux where proggy is a simple program compiled into the Debian PPC linux VM works perfectly :) Feel free of asking for explanations, though I must confess I do not master the topic.. still I managed to run it :) Best Regards! Stefano B.
[Qemu-devel] libqemu.a not made with version 0.13
Hi all! version 0.13 of qemu does not make libqemu.a Is this some choice of developers, or due to any problem? How to fix that? Thank you in advance! Stefano B.
Re: [Qemu-devel] libqemu.a not made with version 0.13
On 12/19/2010 05:10 PM, Blue Swirl wrote: On Sun, Dec 19, 2010 at 2:29 PM, Stefano Bonifazi stefboombas...@gmail.com wrote: Hi all! version 0.13 of qemu does not make libqemu.a Is this some choice of developers, or due to any problem? The API provided by libqemu.a is not supported, so when the compile system was improved so that libqemu.a was no longer needed, it was decided to be dropped. This was during development of 0.11 I think. These days, the only supported interface for integrating QEMU with other applications is QMP. Hi! Thank you! You saved me from a lot of useless work then! Is QMP some sort of server-client system? I mean the only way I might control qemu would be remotely, isn't it? So is there no more way of building a new application with inside qemu-capabilities? I need to create an application that execute a PPC binary on a i386 host with some input, and get the result from that binary.. I thought I could use libqemu in some user mode way (i do not need the full system emulation) .. I can't simply call qemu-ppc, as I need everything to be in the same process .. I needed to do something like the qruncom test but with powerPC binaries .. What do you suggest? Thank you! Best Regards! Stefano B.
Re: [Qemu-devel] libqemu.a not made with version 0.13
On 12/19/2010 05:51 PM, Andreas Färber wrote: Am 19.12.2010 um 17:32 schrieb Stefano Bonifazi: I need to create an application that execute a PPC binary on a i386 host with some input, and get the result from that binary.. I thought I could use libqemu in some user mode way (i do not need the full system emulation) .. I can't simply call qemu-ppc, as I need everything to be in the same process .. Why? Can't you just pass the input via command line? Regards, Andreas No I can't! First of all it is a specification of my project .. this application would be a SystemC TLM2 loosely timed module so it should be as fast as possible, it can't rely on process intercommunication all should be part of the same SystemC process.. moreover the inputs to be passed to the binary may be complex structs .. I'll need full access to the binaries address space for passing them someway the inputs .. Thank you for your interest! :) Best Regards, Stefano B.
Re: [Qemu-devel] libqemu.a not made with version 0.13
On 12/19/2010 06:03 PM, Andreas Färber wrote: In particular, on some platforms libqemu.a would seem to compile okay but the resulting QEMU executable would simply crash. We got around these --whole-archive issues by putting together lists of object files in Makefile.objs et al. that we can link into QEMU's executables directly. It is theoretically possible for your application to integrate with our Makefiles and link to such objects directly. But again this is not supported and may break at any time. If you just need it for one assignment it may work - but don't expect instructions, you'd probably be the first one trying! Regards, Andreas Thank You for your warnings! I am just a student, it is already very hard for me to understand all the mechanisms of QEMU.. I can't base my project on bugged sources, I would be not able of fixing them while developing my project upon them.. Any idea how to go on? Maybe editing qemu-ppc, inserting inside my own code? Thank you in advance! Stefano B.
[Qemu-devel] Re: Problems executing qemu-ppc
On 12/19/2010 04:38 PM, Alexander Graf wrote: On 19.12.2010, at 15:19, Mulyadi Santosa wrote: Hi :) On Sun, Dec 19, 2010 at 03:29, Stefano Bonifazistefboombas...@email.it wrote: Hi! I am answering myself hoping that my solution may help somebody other who has to face the same problem: I am not PPC user by myself, but I think it is a good candidate to be written in either Qemu web forum or Qemu wiki (do we have one?) :) www.qemu.org is a wiki :). And yes, please, that sounds like it really should be on there. As a hint: gcc -static would have also worked. Alex As part of my project I am writing a good QEMU technical documentation, .. I want to insert also this in it someway.. Then if my teacher's policy allows it I'll make my documentation public..I am experiencing big difficulties due to the lack of good updated documentation.. it is a huge C project, and C is not as readable as OO projects.. moreover one needs so many notions about emulators and computer architectures that can't be taught at university .. maybe some document written by who had to learn everything for understanding QEMU can be useful for novices more than one written by those who master all those topics for whom everything is easy and not worth to be said ;) Best Regards! Stefano B.
[Qemu-devel] Problems executing qemu-ppc
Hi all! May someone help me running qemu-ppc? Executing: qemu-ppc hello with hello a PPC binary, I get: /lib/ld.so.1: No such file or directory executing qemu-ppc without argument I get this: [...] -L path set the elf interpreter prefix (default=/usr/gnemul/qemu-ppc) [...] but /usr/gnemul/qemu-ppc does not exist on my system. I don't know what an elf interpreter prefix is :[ my uname: Linux Stefano-PC 2.6.35-23-generic-pae #41-Ubuntu SMP Wed Nov 24 10:35:46 UTC 2010 i686 GNU/Linux Thank you in advance! Stefano B.
Re: [Qemu-devel] classic emulator Vs QEMU-TCG
On 12/16/2010 04:57 PM, Mulyadi Santosa wrote: With my limited C knowledge, I saw that as a instruction jump (to tb_ptr). The code_gen_prologue seems to me like a cast. casting each opcode in tb_ptr as uint8_t with maximum length=1024 Hi! Thank you for your reply! I've got an explanation from a C guru :) ((long REGPARM (*)(void *))code_gen_prologue)(tb_ptr) - (long REGPARM (*)(void *)) is a type: a pointer pointing to a function, which takes one (void*) parameter. code_gen_prologue is anarray, array's name when used is considered to be a pointer to its first element, thus you are casting here pointer to the first byte ofarray to pointer to function (...). Ellipsis with tb_ptr mean 'call function under this address and pass there whatever tb_ptr is' Now everything is very clear for me :) I do know pointer to functions, and if I had got: long REGPARM (*myfunc)(void *) I would have recognized it.. but removing the function pointer name, leaving only the * was enough for me to be lost :[ also the definition of code_gen_prologue was tricky: uint8_t code_gen_prologue[1024] code_gen_section; that code_gen_section at first confused my idea of a normal variable definition as type identifier .. until I found out it was a define for a compiler directive (alignment) :[ So inside code_gen_prologue array there is stored some function (in binary code) that takes a tb_ptr as argument and returns a long .. I have to check what it is inside there for understanding how the translated target code is run now ;) I hope that's the right interpretation...I must admit Qemu is full of gcc and C tricks here and there... Yes! I've only had some experience in OO programming, really always avoided defines and odd compiler directives in my code before.. Best Regards! Stefano B.
Re: [Qemu-devel] classic emulator Vs QEMU-TCG
On 12/16/2010 04:41 PM, Peter Maydell wrote: Some hints: * go and look up the C syntax for function pointers and casting things to function pointers Yup! See the reply to Mr. Santosa, thank you! * code_gen_prologue[] contains code which has been generated once on startup -- go and find the function which is doing this, which ought to tell you what the prologue code actually does... Is that the following? /* init global prologue and epilogue */ s-code_buf = code_gen_prologue; s-code_ptr = s-code_buf; tcg_target_qemu_prologue(s); Trying to understand the pseudo-assembly in tcg_target_qemu_prologue (in file tcg-target.c), I think it builds an assembly function scheleton storying it inside code_gen_prologue array.. Cosidering the implementation of that function for i386 I think the jmp *%eax is the actual code that jumps to the host binary produced by TCG from the target binary.. in fact, if I am not wrong,this binary function is what is actually called by tcg_qemu_tb_exec(tb_ptr) macro with tb_ptr passed to the function in %eax, thus jmp *%eax starts the execution of the binary code .. am I wrong? * try single stepping individual machine instructions in the debugger as you go through tcg_qemu_tb_exec() and matching this up with what is really happening here and with the bits of qemu which generated that code. -- PMM I would have already done that.. unluckily I have always used IDE with integrated debuggers, and I can't find an IDE for loading this project.. I guess I have no other choice than learning also gdb Thank you for your tips! :) Best Regards! Stefano B.
[Qemu-devel] classic emulator Vs QEMU-TCG
Hi all!
I am a student, trying to understand QEMU, specifically TCG
translation/execution.
After spending much time on the code I still have big doubts. I think my
doubts are due to the classic idea I have of an emulator.
Actually as a student, I've never developed even a simple classic
emulator myself, but in my idea it should follow this flow:
1) Fetch target instruction
i.e. PC(0x532652) : 0x104265 (I am just inventing)
2) Decode
Opcode 0x10 : ADD, R1: 0x42, R2: 0x65
3) Look up instruction function table:
switch(opcode)
case add :
add(R1, R2)
break;
4) Execution
void add(int R1, int R2)
{ env-reg[R1] = env-reg[R1] + env[R2];}
Now all of that would be compiled offline for the host machine and at
runtime the host macine would just execute the binary host code for the
instruction env-reg[R1] = env-reg[R1] + env[R2]; (its host binary
translation)
In QEMU/TCG, thanks to the help of Mr. Blue Swirl, I understood there is
a runtime creation of host binary, starting from the loaded target binary..
My big doubt is, how can I execute that new binary? .. Shall TCG put it
in some memory location, and then make the process branch to that
address (and then back) ?
I really can't see how that happens in the code :(
in cpu-exec.c : cpu_exec_nocache i find:
/* execute the generated code */
next_tb = tcg_qemu_tb_exec(tb-tc_ptr);
and in cpu-exec.c : cpu_exec
/* execute the generated code */
next_tb = tcg_qemu_tb_exec(tc_ptr);
so I thought tcg_qemu_tb_exec function should do the work of executing
the translated binary in the host.
But then I found out it is just a define in tcg.h:
#define tcg_qemu_tb_exec(tb_ptr) ((long REGPARM (*)(void
*))code_gen_prologue)(tb_ptr)
and again in exec.c
uint8_t code_gen_prologue[1024] code_gen_section;
Maybe I have some problems with that C syntax, but I really don't
understand what happens there.. how the execution happens!
I think for all of you working for so long on QEMU, with a long
successful experience in this field should be very easy.. but atm I
really can't figure it out alone.. I can't find good documents
explaining it, and I can't understand myself from the code!
Thank you very very much for any help! :)
Stefano B.
Re: [Qemu-devel] TCG flow vs dyngen
On 12/11/2010 03:44 PM, Blue Swirl wrote: On Sat, Dec 11, 2010 at 2:32 PM, Stefano Bonifazi stefboombas...@gmail.com wrote: Where does the execution of host binary take place in the previous list of events? Between point 5) and 6) ? After 6) ? In what QEMU source code file/function does the final execution of host binary take place? In the previous list of events, when does the translator try to chain the current TB with previous ones? Before TCG generates the binary in order to feed it with linked micro code? All of this happens in cpu-exec.c:581 to 618. Hi! Thank you very much! Knowing exactly where I should check, in a so big project helped me very much!! Anyway after having spent more than 2 days on that code I still can't understand how it works the real execution: in cpu-exec.c : cpu_exec_nocache i find: /* execute the generated code */ next_tb = tcg_qemu_tb_exec(tb-tc_ptr); and in cpu-exec.c : cpu_exec /* execute the generated code */ next_tb = tcg_qemu_tb_exec(tc_ptr); so I thought tcg_qemu_tb_exec function should do the work of executing the translated binary in the host. But then I found out it is just a define in tcg.h: #define tcg_qemu_tb_exec(tb_ptr) ((long REGPARM (*)(void *))code_gen_prologue)(tb_ptr) and again in exec.c uint8_t code_gen_prologue[1024] code_gen_section; Maybe I have some problems with that C syntax, but I really don't understand what happens there.. how the execution happens! Maybe I am too stuck to my idea of a common emulator fetch - decode - execute where an addition would be implemented simply as env-regC = env-regA +env-regB ... where this C instruction would be compiled offline into host machine binary by host compiler.. so the emulator would be a monolith block of host code just with branches for the different opcodes that would come from the target binary loaded at runtime.. Here instead with QEMU/TCG I understood that at runtime the target binary is translated into host binary (somehow) .. but then.. how can this new host binary be run? Shall the host code at runtime do some sort of (assembly speaking) branch jump to an area of memory with new host binary instructions .. and then jump back to the old process binary code? If so, can you explain me how this happens in those lines of code? I am just a student.. unluckily at university they just tell you that a cpu follows some sort of fetch -decode-execute flow .. but then you open QEMU.. and wow there is a huge gap for understanding it, and no books where to study it! ;) Please help me understanding it :) Thank you very very much in advance! Stefano B.
Re: [Qemu-devel] TCG flow vs dyngen
Thank you very very much! I'd take months for understanding everything myself from the source code! :) On 12/11/2010 12:02 PM, Blue Swirl wrote: On Fri, Dec 10, 2010 at 9:26 PM, Stefano Bonifazi stefboombas...@gmail.com wrote: [..] - So, I think that the technical documentation is now obsolete, isn't it? At least we shouldn't link to that paper anymore. There's also documentation generated from qemu-tech.texi that should be up to date. Do you mean this: http://www.weilnetz.de/qemu-tech.html ? - If I understand well, TCG runtime flow is the following: - TCG takes the target binary, and splits it into target blocks - if the TB is not cached, TCG translates it (or better the target instructions it is composed by) into TCG micro ops, The above is not the job of TCG (which is host specific), but the target specific translators (target-*/translate.c). Ok, then considering QEMU flow instead of simply TCG, do those steps take place in the order I considered? - TCG caches the TB, - TCG tries to chain the block with others, The above is part of the CPU execution loop (cpu-exec.c), TCG is not involved anymore. Ok! Thank you, now I have a clearer idea of where different operations are implemented.. but again considering the whole QEMU flow, are the steps I reported executed in the order I put them? - TCG copies the TB into the execution buffer There is no copying. Does that mean TCG produces the host object code directly into the emulator's memory for it to fetch? Or does TCG make the emulator even execute that object code as soon as it is produced? But, if the object code is consumed on the fly, it means there is no cashing of it, is it there? What is actually cached? Only target blocks? Their translation into TCG uops? Host binary code generated by TCG? Again many many thanks!!! Stefano B.
RE: [Qemu-devel] TCG flow vs dyngen
-Original Message- From: Blue Swirl [mailto:blauwir...@gmail.com] Sent: sabato 11 dicembre 2010 14:12 To: Stefano Bonifazi Cc: qemu-devel@nongnu.org Subject: Re: [Qemu-devel] TCG flow vs dyngen There's a large buffer for generated code, allocated in exec.c. This is filled with host code by TCG, when full it is flushed. The CPU execution loop generates new TBs when needed, otherwise the old code can be executed. TCG also uses intermediate ops but those are used only once during translation. So if I understand well the flow is the following: 1) the CPU execution loop at runtime takes a new TB from the target code 2) I guess some hash function is computed on this TB for getting a key for searching into the buffer of generated code that probably should store the binary as a map key-binary 3) if the search is successful the binary is given to the translator(how? You said no copy involved) and we return to point 1) otherwise: 4) the target specific translator generates TCG uops from the TB 5) TCG uses uops for generating host binary code 6) this new binary code is cached by TGC if there is enough storage place Is that all correct? Where does the execution of host binary take place in the previous list of events? Between point 5) and 6) ? After 6) ? In what QEMU source code file/function does the final execution of host binary take place? In the previous list of events, when does the translator try to chain the current TB with previous ones? Before TCG generates the binary in order to feed it with linked micro code? Thank you very very much! :) Stefano B.
[Qemu-devel] RE: [PATCH] fix qruncom compilation problems
-Original Message- From: Paolo Bonzini [mailto:pbonz...@redhat.com] Sent: venerdì 10 dicembre 2010 22:49 To: Stefano Bonifazi Subject: Re: [PATCH] fix qruncom compilation problems For runcom (without the q) this wouldn't work, because it runs the code in vm86 mode. It's possible that this is ok for qruncom with other changes to let the TCG backend know about vm86_mem. So informing the interpreter with page_set_flags(0x+vm86_mem, 0x11+vm86_mem, PAGE_WRITE | PAGE_READ | PAGE_EXEC | PAGE_VALID); Would be not enough? I was looking at this in my free time and it seriously shrunk later, so I'm afraid I cannot help. Surely I do understand you! Your help has been very very useful and appreciated already thank you! May you direct me to somebody who's working on it? Some TCG guru who could understand immediately what's wrong? :) I noticed, far now, that each question on this mailing list is answered only by one QEMU developer, is that a sort of policy or just a coincidence? Again thank you! Best regards! Stefano B.
[Qemu-devel] TCG flow vs dyngen
Hi all! From the technical documentation (http://www.usenix.org/publications/library/proceedings/usenix05/tech/freenix/bellard.html) I read: The first step is to split each target CPU instruction into fewer simpler instructions called /micro operations/. Each micro operation is implemented by a small piece of C code. This small C source code is compiled by GCC to an object file. The micro operations are chosen so that their number is much smaller (typically a few hundreds) than all the combinations of instructions and operands of the target CPU. The translation from target CPU instructions to micro operations is done entirely with hand coded code. A compile time tool called dyngen uses the object file containing the micro operations as input to generate a dynamic code generator. This dynamic code generator is invoked at runtime to generate a complete host function which concatenates several micro operations. instead from wikipedia(http://en.wikipedia.org/wiki/QEMU) and other sources I read: The Tiny Code Generator (TCG) aims to remove the shortcoming of relying on a particular version of GCC http://en.wikipedia.org/wiki/GNU_Compiler_Collection or any compiler, instead incorporating the compiler (code generator) into other tasks performed by QEMU in run-time. The whole translation task thus consists of two parts: blocks of target code (/TBs/) being rewritten in *TCG ops* - a kind of machine-independent intermediate notation, and subsequently this notation being compiled for the host's architecture by TCG. Optional optimisation passes are performed between them. - So, I think that the technical documentation is now obsolete, isn't it? - The old way used much offline (compile time) work compiling the micro operations into host machine code, while if I understand well, TCG does everything in run-time(please correct me if I am wrong!).. so I wonder, how can it be as fast as the previous method (or even faster)? - If I understand well, TGC runtime flow is the following: - TCG takes the target binary, and splits it into target blocks - if the TB is not cached, TGC translates it (or better the target instructions it is composed by) into TCG micro ops, - TGC compiles TGC uops into host object code, - TGC caches the TB, - TGC tries to chain the block with others, - TGC copies the TB into the execution buffer - TGC runs it Am I right? Please correct me, whether I am wrong, as I wanna use that flow scheme for trying to understand the code.. Thank you very much in advance! Stefano B.
[Qemu-devel] Re: [PATCH] fix qruncom compilation problems
On 12/10/2010 09:53 AM, Paolo Bonzini wrote: On 12/09/2010 06:29 PM, Stefano Bonifazi wrote: how can one think that addresses around zero are free for a mapping?? Addresses around zero are always free, because if they weren't you couldn't detect NULL pointer dereferences reliably. mmap-ing at zero thus is a tricky operation, because it removes the possibility to detect NULL pointer dereferences. What's worse, such ability would be lost even for _kernel_ dereferences of NULL, thus opening a large security hole for privilege-escalation or kernel exploits. So, mmap-ing addresses close to zero is restricted to root. Paolo Hi! Thank you! Very clear explanation! :) - So why can't I simply change the following: vm86_mem = mmap((void *)0x, 0x11, PROT_WRITE | PROT_READ | PROT_EXEC, MAP_FIXED|MAP_ANON | MAP_PRIVATE, -1, 0); page_set_flags(0x, 0x11, PAGE_WRITE | PAGE_READ | PAGE_EXEC | PAGE_VALID); into something like: vm86_mem = mmap((void *)0x, 0x11, PROT_WRITE | PROT_READ | PROT_EXEC, MAP_ANON | MAP_PRIVATE, -1, 0); page_set_flags(vm86_mem, 0x11+vm86_mem, PAGE_WRITE | PAGE_READ | PAGE_EXEC | PAGE_VALID); ? - Any luck with the tcg fatal error? I am trying to understand how tcg works for fixing the error.. but it is so complicated! :) Thank You again! Best Regards! Stefano B.
[Qemu-devel] Re: [PATCH] fix qruncom compilation problems
On 12/08/2010 01:49 PM, Paolo Bonzini wrote:
Signed-off-by: Paolo Bonzinipbonz...@redhat.com
---
I had this patch lying around but I don't think I ever got
qruncom to work completely.
Makefile.target |3 ++
tests/Makefile |7 ++--
tests/qruncom.c | 93 +++---
3 files changed, 67 insertions(+), 36 deletions(-)
diff --git a/Makefile.target b/Makefile.target
index 5784844..4ac8f6f 100644
--- a/Makefile.target
+++ b/Makefile.target
@@ -339,6 +339,9 @@ obj-y += $(addprefix ../libdis/, $(libdis-y))
obj-y += $(libobj-y)
obj-y += $(addprefix $(HWDIR)/, $(hw-obj-y))
+else # !CONFIG_SOFTMMU
+libqemu.a: $(addprefix ../, $(common-obj-y)) $(libobj-y) $(addprefix
../libdis/, $(libdis-y))
+ ar rc $@ $^
endif # CONFIG_SOFTMMU
obj-y += $(addprefix ../, $(trace-obj-y))
diff --git a/tests/Makefile b/tests/Makefile
index e43ec70..6dbeb6f 100644
--- a/tests/Makefile
+++ b/tests/Makefile
@@ -116,9 +116,10 @@ speed: sha1 sha1-i386
# broken test
# NOTE: -fomit-frame-pointer is currently needed : this is a bug in libqemu
-qruncom: qruncom.c ../ioport-user.c ../i386-user/libqemu.a
- $(CC) $(CFLAGS) -fomit-frame-pointer $(LDFLAGS) -I../target-i386 -I..
-I../i386-user -I../fpu \
- -o $@ $(filter %.c, $^) -L../i386-user -lqemu -lm
+qruncom: qruncom.c
+ #$(MAKE) -C ../i386-linux-user libqemu.a
+ $(CC) $(CFLAGS) -fomit-frame-pointer $(LDFLAGS) -I../target-i386 -I..
-I../linux-user -I../i386-linux-user -I../fpu \
+ -o $@ $(filter %.c, $^) -L../i386-linux-user -lqemu -lm
# arm test
hello-arm: hello-arm.o
diff --git a/tests/qruncom.c b/tests/qruncom.c
index 079f7a2..66fc223 100644
--- a/tests/qruncom.c
+++ b/tests/qruncom.c
@@ -12,10 +12,68 @@
#includesignal.h
#includemalloc.h
+#define NEED_CPU_H 1
#include cpu.h
//#define SIGTEST
+unsigned long guest_base = 0;
+int have_guest_base = 0;
+int singlestep = 0;
+unsigned long last_brk = 0;
+
+void cpu_outb(uint32_t addr, uint8_t val)
+{
+fprintf(stderr, outb: port=0x%04PRIx32, data=%02PRIx8\n,
+addr, val);
+}
+
+void cpu_outw(uint32_t addr, uint16_t val)
+{
+fprintf(stderr, outw: port=0x%04PRIx32, data=%04PRIx16\n,
+addr, val);
+}
+
+void cpu_outl(uint32_t addr, uint32_t val)
+{
+fprintf(stderr, outl: port=0x%04PRIx32, data=%08PRIx32\n,
+addr, val);
+}
+
+uint8_t cpu_inb(uint32_t addr)
+{
+fprintf(stderr, inb: port=0x%04PRIx32\n, addr);
+return 0;
+}
+
+uint16_t cpu_inw(uint32_t addr)
+{
+fprintf(stderr, inw: port=0x%04PRIx32\n, addr);
+return 0;
+}
+
+uint32_t cpu_inl(uint32_t addr)
+{
+fprintf(stderr, inl: port=0x%04PRIx32\n, addr);
+return 0;
+}
+
+void cpu_list_lock(void)
+{
+}
+
+void cpu_list_unlock(void)
+{
+}
+
+void mmap_lock(void)
+{
+}
+
+void mmap_unlock(void)
+{
+}
+
int cpu_get_pic_interrupt(CPUState *env)
{
return -1;
@@ -44,26 +102,6 @@ static void set_idt(int n, unsigned int dpl)
set_gate(idt_table + n, 0, dpl, 0, 0);
}
-void qemu_free(void *ptr)
-{
-free(ptr);
-}
-
-void *qemu_malloc(size_t size)
-{
-return malloc(size);
-}
-
-void *qemu_mallocz(size_t size)
-{
-void *ptr;
-ptr = qemu_malloc(size);
-if (!ptr)
-return NULL;
-memset(ptr, 0, size);
-return ptr;
-}
-
void *qemu_vmalloc(size_t size)
{
return memalign(4096, size);
@@ -74,17 +112,6 @@ void qemu_vfree(void *ptr)
free(ptr);
}
-void qemu_printf(const char *fmt, ...)
-{
-va_list ap;
-va_start(ap, fmt);
-vprintf(fmt, ap);
-va_end(ap);
-}
-
-/* XXX: this is a bug in helper2.c */
-int errno;
-
/**/
#define COM_BASE_ADDR0x10100
@@ -99,7 +126,7 @@ static void usage(void)
static inline uint8_t *seg_to_linear(unsigned int seg, unsigned int reg)
{
-return (uint8_t *)((seg 4) + (reg 0x));
+return (uint8_t *)(uintptr_t) ((seg 4) + (reg 0x));
}
static inline void pushw(CPUState *env, int val)
@@ -241,7 +268,7 @@ int main(int argc, char **argv)
case EXCP0D_GPF:
{
int int_num, ah;
-int_num = *(uint8_t *)(env-segs[R_CS].base + env-eip + 1);
+int_num = *(uint8_t *)(uintptr_t) (env-segs[R_CS].base +
env-eip + 1);
if (int_num != 0x21)
goto unknown_int;
ah = (env-regs[R_EAX] 8) 0xff;
Hi!
Thank you for your help!
I've linked qemu-malloc.o and cutils.o together with qruncom.c and I
managed to succesfully make it!
here the make line:
#$(MAKE) -C ../i386-linux-user libqemu.a
$(CC) $(CFLAGS) -fomit-frame-pointer $(LDFLAGS) -I../target-i386
-I.. -I../linux-user -I../i386-linux-user -I../fpu \
-o $@ ../qemu-malloc.o ../cutils.o $(filter %.c, $^)
-L../i386-linux-user -lqemu -lm
Anyway running it with a com file as argument gave the error:
mmap: Operation not permitted
I
[Qemu-devel] Re: [PATCH] fix qruncom compilation problems
On 12/09/2010 08:16 AM, Paolo Bonzini wrote: On 12/08/2010 10:43 PM, Stefano Bonifazi wrote: Anyway running it with a com file as argument gave the error: mmap: Operation not permitted You have to run it as root I think. Paolo Thank you! Running as root worked, though it raises then the following error (from gdb) I am currently trying to understand: /home/stefano/LinuxDev/qemu-0.12.5/tcg/tcg.c:1367: tcg fatal error Program received signal SIGABRT, Aborted. 0x0012e416 in __kernel_vsyscall () Surely any hint on how to to fix this will be very welcome :) I wish I could understand also what was wrong before, I mean /mmap/.. I understand you can't babysit me, but the gap between what one studies at university and the real world is very big and I feel lost :( I've read pretty much about mmap trying to figure out myself but understanding how to map a file (what I could find in every article about mmap online) is not the same as understanding how it works inside QEMU .. I know each process gets its own logical address space, if I understood fine mmap should take a portion of qruncom address space and give it to the emulator that should then see that as its own address space (please correct me if I am wrong!) .. Now if I got fine the flag MAP_FIXED, obliges the process to give that portion of address space starting at its /addr/ parameter (the first).. or if it is not possible to give an error.. My big doubt is how can the process give exactly that portion of address space starting at zero by just running it as root?.. I am expecting that area of address space to be taken by I dunno, code, data of the process itself.. honestly I don't know how things are allocated when a process is run(and I wish I could learn that).. but how can one think that addresses around zero are free for a mapping?? I'll appreciate very much any explanation, or links where to learn those topics! :) Thank you very much! Best Regards! Stefano B.
[Qemu-devel] Using the mailing list for asking questions about the source code
Hi All! I am new in QEMU developing and I am not sure if I can use this mailing list for asking general questions about QEMU source code as I could not find any guidelines about it. I noticed that, usually, questions about the source code in the QEMU forum never receive answers. Surely the best help I can receive is from you, and it is essential due to the lack of good technical documentation, but I don't know if the mailing list is only for sharing patches and fixes. In this case I ask for an apology for my intrusion, and I'd be very glad of being addressed to a proper help. Best Regards, Stefano B.
[Qemu-devel] Using the mailing list for asking questions about the source code
Hi All! I am new in QEMU developing and I am not sure if I can use this mailing list for asking general questions about QEMU source code as I could not find any guidelines about it. I noticed that, usually, questions about the source code in the QEMU forum never receive answers. Surely the best help I can receive is from you, and it is essential due to the lack of good technical documentation, but I don't know if the mailing list is only for sharing patches and fixes. In this case I ask for an apology for my intrusion, and I'd be very glad of being addressed to a proper help. Best Regards, Stefano B.
Re: [Qemu-devel] Using the mailing list for asking questions about the source code
On 12/07/2010 10:01 PM, Stefan Weil wrote: Hi Stefano, you found this mailing list, so I assume you read everything which is available on http://wiki.qemu.org/Main_Page, and you also tried hard to find the answers to your questions yourself, didn't you? If there remain concrete questions, you should collect them and send them to qemu-devel or add them to http://wiki.qemu.org/Talk:Manual with a remark missing documentation. Precise questions have a higher probability to get an answer than abstract ones. Kind regards, Stefan W. Hi! Thank you for your quick answer! :) Yes, I've spent many days collecting and studying all possible documents about QEMU in the internet. Unluckily there is much about using it, but few about its code. The few documents are often old and not updated. The project is huge and complex and seems very very hard for a student to face it all alone. I am sure that few words from who has spent on it so much time, efforts, experience, passion will speed up my work very much :) Thank you again! Best regards! Stefano B.
