This is tremendously helpful. Thanks a lot. My work is focusing on kernel, base on arm architecture, I want have whole view of the running kernel.
With qemu & qemu monitor & gdb & crash tools, I can stop kernel anywhere, read phy/virt address, registers even system register(with last qemu release). Thanks for sharing your experience, It does greate help. Hope to hear that you successfully run last AOSP on upstream QEMU. 在 2020年7月4日星期六 UTC+8上午12:35:32，Julien Robin写道： > > Hi 郁进, > > Recently I focused on making the untouched Android code base working with > standard/upstream QEMU (which is a little bit different from the AOSP's > emulator, which is a QEMU derivative), focusing on x86_64 builds and > "android-x86" project build configurations; the work is still in progress. > But from what I know : > > When building aosp_XXXX-eng project, the "emulator" command line is made > available so that just typing "emulator" into your terminal is (normally) > enough to get it starting with the correct parameters set. Most of the time > I use "emulator -verbose -show-kernel -no-snapshot" when running the AOSP's > emulator. So that you have all the default parameters shown into your > terminal, then kernel debug information, then Android's "init" output into > your terminal. The kernel used to run those builds is called > "kernel-ranchu" : it's a customised and prebuilt version of the 4.14 > kernel, intended for use with the aosp's emulator virtual hardware and > drivers (called goldfish in its original version, then ranchu). > > If you build aosp then close the terminal window, and you want the > "emulator" command to be available again, you should just type "source > build/envsetup.sh && lunch" commands again (no need to rebuild everything). > > But for arm64 builds its horribly slow, like several minutes for booting > (upstream qemu is making the following option "-accel tcg,thread=multi" > available so that the performance using an AMD 3700X is almost similar to > using KVM on a native arm64 Raspberry Pi 4. But unfortunately this accel > option does not seem to be used by the "emulator" command, and seems not to > work on AOSP's emulator, at least last time I tried). Be aware that > emulator is a derivative of QEMU, recognizing some AOSP specific commands, > and not recognizing some orginal QEMU commands. > > *About having the kernel finding and starting "init" :* > > Lot of things can be confusing about having your emulator or qemu finding > and running "init" process. I will give you everything I discovered so that > hopefully you will be able to understand and debug anything that happens to > you > > On really old versions of Android, "init" executable was only available > into the ramdisk, and "system" partition was just containing the /system > subfolder (this is the opposite of "system as root"). In this case, Linux > kernel absolutely needed to load the ramdisk, because init was into this > ramdisk. I guess fstab file was into the ramdisk too, in order to find and > mount partitions once init started. > > Some more recent version of android using "system as root" (8 or 9 maybe) > gave some ramdisk file as parameter to the emulator, despite the fact this > ramdisk wasn't actually used anymore by the emulator : the content of the > ramdisk was already into system.img. And the kernel was already able to > access the system.img partition without the need of anything into the > ramdisk - so the ramdisk wasn't used anymore. "init" file was at the root > of the system.img partition image. In this case, you should ensure your > machine and kernel are able to find your ext4 root (when in trouble, look > into the kernel messages to find an ext4 file system at /dev/sda, or > /dev/vda, or even sda1, vda1, etc, so that you can be sure about the kernel > ability to see the partition you need). > > But Android 10 introduced a new partition/image format that can be > dynamically expanded and physically fragmented (so that devices sold with a > too small system partition for being updated, can be updated anyway). This > is cool feature, but a little bit more complex to handle. When using this > kind of system images, android requires a brand new specific ramdisk with a > specific "init" executable in order to read and mount this new system > partition format created by android. I believe emulator is using it with > Android 10 (to be confirmed). But in order to focus on my work, I'm still > using the previous way : an ext4 system.img with everything into it ;) so > that qemu boots it directly. > > > For example, on my x86_64 custom builds using "system as root", and having > the vendor files included into system.img, as a standard ext4 image, I run > the following command to start it using QEMU : > > kvm -m 4096 -smp 4 -kernel bzImage -append "root=/dev/sda rootfstype=ext4 > ro init=/init selinux=1 checkreqprot=1 androidboot.selinux=permissive > console=ttyS0 androidboot.hardware=luoss loglevel=8" -serial mon:stdio > -drive format=raw,file=system.img -vga std > -m 4096 ask for 4GB RAM > bzImage is a custom 5.4 kernel > root=/dev/sda rootfstype=ext4 ro init=/init is telling my kernel to mount > what it sees as "/dev/sda" and to run the "init" executable which is > located into it > selinux=1 checkreqprot=1 androidboot.selinux=permissive are selinux > parameters : when you want to disable selinux, you can't (init absolutely > wants your kernel to have selinux running) but you set it to > androidboot.selinux=permissive, so that no operation is actually blocked. > console=ttyS0 ask the kernel to print its messages into the serial port > androidboot.hardware=HW_NAME is used to tell init process to load > init.HW_NAME.rc configuration file (that should be into /init.HW_NAME.rc > or /vendor/etc/init/hw/init.HW_NAME.rc). For aosp emulator builds, it is > /vendor/etc/init/hw/init.ranchu.rc > loglevel=8 ask for the kernel, then init, to be as verbose as possible > > -serial mon:stdio is a qemu parameter I use to redirect the virtual serial > port (on which the kernel talks) to the terminal in which qemu is running. > So that I can use this terminal as a read/write terminal with the virtual > machine. > > > The system image i'm using to do so is an ext4 partition image, on which i > placed everything needed, and an init.luoss.rc and fstab.luoss files - > mainly just to mount a /data tmpfs instead of using a real userdata.img > partiton - yes, I'm cheating ;) normally system.img is not enough, and in > most case, /vendor subfolder is into a vendor.img partition. > > I guess that when I'll be trying to run arm64 builds on qemu, some > parameters will need to be changed. > > I hope that all this information will help you to successfully dig, > understand and solve your issues! > > Best regards > > Julien > > Le lundi 29 juin 2020 06:41:01 UTC+2, 郁进 a écrit : >> >> Hi, >> >> I found another tickets you submitted to run arm64 android via qemu. >> >> I tried to lunch aosp_arm64-eng project, run with android emulator, with >> qemu args: >> >> emulator -kernel >> ../Android_kernel/out/android-mainline/common/arch/arm64/boot/Image -system >> xxx ...... *-qemu **--monitor tcp:127.0.0.1:2222 >> <http://127.0.0.1:2222>,server,nowait -gdb tcp::4444 -S* >> >> But the Android kernel panic occurred since no init found, and I try to >> dump guest memory via qemu monitor, can't parsed by crash tools. >> >> Have you build and run arm64 android via qemu successfully? How ? >> >> Looking forward to your reply, many thanks. >> >> >> 在 2020年3月3日星期二 UTC+8下午11:18:37，Julien ROBIN写道： >>> >>> Many thanks for your answers >>> >>> In fact what I'm trying to do is using QEMU files into AOSP source code >>> to get Android (with GUI) built and running on a standard/upstream QEMU >>> version. >>> >>> This would give the ability of running Android and its future releases >>> without real "porting", on any device running a Linux kernel (be it >>> upstream or not) and having KVM working on it. Which would be extremely >>> interesting. >>> >>> From what you write, it seems qemu-trusty-arm64 is not really for me. So >>> what I'm trying to do is probably more about using device/generic/qemu/ >>> qemu_arm64.mk (and others archs) files included into AOSP. So I'll >>> continue into this other subject : >>> https://groups.google.com/forum/?hl=bn#!topic/android-building/kbiEg5OxBGU >>> >>> >>> While talking with you, I noticed that AVD Emulator is different from >>> upstream QEMU, probably about goldfish and ranchu related things. Things >>> working fine with the emulator (aosp_x86_64-eng for example and >>> android-goldfish-4.14-dev which seems mandatory to get Android 10 working >>> with AVD Emulator) does not seems to be working with upstream QEMU (at >>> least I can't get the GUI). Therefore the documentation about AVD Emulator >>> builds https://source.android.com/setup/create/avd didn't really help >>> me for upstream QEMU. >>> >>> Can you confirm it's possible to use modern versions of AOSP on upstream >>> QEMU? Someone did succeed in the past with Android 6 / untouched AOSP and a >>> extended compatibility kernel (I have one based on >>> android-goldfish-4.14-dev), this is why I'm trying with newer versions of >>> AOSP. This was documented here, but it's not up to date anymore : >>> https://www.collabora.com/news-and-blog/blog/2016/09/02/building-android-for-qemu-a-step-by-step-guide/ >>> >>> Thank you in advance >>> >>> Julien >>> >>> >>> On 3/2/20 10:51 PM, 'Matthew Maurer' via Android Building wrote: >>> >>> This target is a testing target used for evaluating our reference >>> TrustZone implementation. "storageproxyd" is continuously resetting there >>> because it is not being launched by our test script >>> <https://android.googlesource.com/trusty/device/arm/generic-arm64/+/refs/heads/master/project/qemu/qemu.py> >>> which >>> would attach a virtual RPMB resource. >>> >>> A few notes about this target: >>> * It does not have a GUI >>> * Running Java is not supported and unlikely to work >>> * It requires a custom EL3 and Trusty running alongside it to work >>> >>> The manifest for these components is at >>> https://android.googlesource.com/trusty/manifest, and that manifest >>> contains a checked in prebuilt of the qemu_trusty_arm64-userdebug target >>> that we are currently testing against. >>> >>> If you tell me more about what you're trying to make the target do, I >>> may be able to give you further help with getting it to do what you want. >>> If you aren't trying to use or inspect the Trusty TZ implementation, this >>> target probably isn't the one you're looking for. >>> >>> On Mon, Mar 2, 2020 at 1:34 PM Dan Willemsen <dwill...@google.com> >>> wrote: >>> >>>> +Matthew Maurer Do we have any documentation on this target? I think >>>> the last time we talked about this target it was only minimally functional. >>>> >>>> - Dan >>>> >>>> On Mon, Mar 2, 2020 at 12:51 PM Julien Robin <julien...@gmail.com> >>>> wrote: >>>> >>>>> I succeeded to find a workaround for the issue I signaled here, >>>>> editing device/generic/trusty/BoardConfig.mk and putting >>>>> BUILD_QEMU_IMAGES >>>>> to false, which still gives vendor.img, system.img and userdata.img >>>>> >>>>> I then found this kernel >>>>> https://android.googlesource.com/kernel/common/+/refs/heads/android-trusty-4.19, >>>>> >>>>> and using arch/arm64/configs/trusty_qemu_defconfig, I successfully built >>>>> it. >>>>> >>>>> Also, I'm able to start most of this Android build on Debian 10 >>>>> provided qemu-system-aarch64 the following way : >>>>> >>>>> qemu-system-aarch64 >>>>> -M virt >>>>> -cpu cortex-a57 >>>>> -smp 1 >>>>> -m 2048 >>>>> -monitor none >>>>> -parallel none >>>>> -serial mon:stdio >>>>> -kernel Image -append "root=/dev/vda rootfstype=ext4 ro >>>>> init=/init loglevel=8 selinux=1 checkreqprot=1 >>>>> androidboot.selinux=permissive androidboot.hardware=qemu_trusty >>>>> console=ttyAMA0" >>>>> -device virtio-gpu-pci >>>>> -drive format=raw,file=system-custom.img >>>>> -drive format=raw,file=userdata-custom.img >>>>> >>>>> userdata-custom.img is just a bigger userdata.img (1024MB instead of >>>>> 4) in which I used resize2fs, system-custom.img is just a system.img in >>>>> which the placeholder folder "vendor" now embeds the content of >>>>> vendor.img, >>>>> copied by "cp -a". Finally, still into system-custom.img, >>>>> fstab.qemu_trusty >>>>> only contains >>>>> LABEL=data /data ext4 noatime,nosuid, >>>>> nodev,errors=panic wait >>>>> So that /data (available at /dev/vdb) is mounted by its label, while >>>>> system-custom.img embeding root system and vendor is already mounted by >>>>> kernel cmdline (by /dev/vda). ramdisk.img is not used as system.img >>>>> already >>>>> contains everything needed to start init. >>>>> >>>>> However, I guess I should use a slightly modified version of qemu, >>>>> available here >>>>> https://android.googlesource.com/trusty/external/qemu/+/refs/heads/master >>>>> Because using Debian 10 qemu version, I get the following output, and >>>>> no display : >>>>> >>>>> init: init first stage started! >>>>> [...] >>>>> init: Loading SELinux policy >>>>> [...] >>>>> selinux: SELinux: Loaded policy from /vendor/etc/selinux/ >>>>> precompiled_sepolicy >>>>> >>>>> selinux: SELinux: Loaded file_contexts >>>>> >>>>> random: init: uninitialized urandom read (40 bytes read) >>>>> random: init: uninitialized urandom read (40 bytes read) >>>>> init: init second stage started! >>>>> [...] >>>>> ueventd: Coldboot took 1.344 seconds >>>>> [...] >>>>> init: starting service 'storageproxyd'... >>>>> libprocessgroup: CgroupMap::FindController called for [1] failed, RC >>>>> file was not initialized properly >>>>> libprocessgroup: Failed to make and chown /uid_0: Read-only file >>>>> system >>>>> libprocessgroup: CgroupMap::FindController called for [829] failed, >>>>> RC file was not initialized properly >>>>> init: createProcessGroup(0, 829) failed for service 'storageproxyd': >>>>> Read-only file system >>>>> init: cpuset cgroup controller is not mounted! >>>>> init: Service 'storageproxyd' (pid 829) exited with status 1 >>>>> init: Sending signal 9 to service 'storageproxyd' (pid 829) process >>>>> group... >>>>> libprocessgroup: CgroupMap::FindController called for [1] failed, RC >>>>> file was not initialized properly >>>>> libprocessgroup: CgroupMap::FindController called for [1] failed, RC >>>>> file was not initialized properly >>>>> >>>>> >>>>> init: starting service 'storageproxyd'... >>>>> libprocessgroup: CgroupMap::FindController called for [1] failed, RC >>>>> file was not initialized properly >>>>> libprocessgroup: Failed to make and chown /uid_0: Read-only file >>>>> system >>>>> init: createProcessGroup(0, 830) failed for service 'storageproxyd': >>>>> Read-only file system >>>>> libprocessgroup: CgroupMap::FindController called for [830] failed, >>>>> RC file was not initialized properly >>>>> init: cpuset cgroup controller is not mounted! >>>>> init: Service 'storageproxyd' (pid 830) exited with status 1 >>>>> init: Sending signal 9 to service 'storageproxyd' (pid 830) process >>>>> group... >>>>> >>>>> ...and so on in loop >>>>> >>>>> However I'm still not sure to proceed correctly, as I'm doing >>>>> everything alone with no documentation (I may have missed it?) >>>>> >>>>> Is here some documentation available? If there isn't, is someone able >>>>> to provide an example of working qemu command line (even old or >>>>> unoptimized), or just few explanations so that I can be placed on the >>>>> right >>>>> track? >>>>> >>>>> Thank you very much in advance >>>>> >>>>> >>>>> On 3/1/20 2:14 PM, Julien Robin wrote: >>>>>> >>>>>> Hi there, >>>>>> >>>>>> I see that starting from branch android-10.0.0_r1, lunch offers a >>>>>> choice called qemu_trusty_arm64-userdebug, in which I'm interested >>>>>> >>>>>> However, I got the following error, no matter which OS I'm using for >>>>>> building (be it Ubuntu 18.048 or Debian 10). I have this error on r1, >>>>>> r20 >>>>>> and r29. >>>>>> >>>>>> [ 62% 16150/26047] Create system-qemu.img now >>>>>> FAILED: out/target/product/trusty/system-qemu.img >>>>>> /bin/bash -c "(export SGDISK=out/host/linux-x86/bin/sgdisk >>>>>> SIMG2IMG=out/host/linux-x86/bin/simg2img; >>>>>> device/generic/goldfish/tools/mk_combined_img.py -i >>>>>> out/target/product/trusty/system-qemu-config.txt -o >>>>>> out/target/product/trusty/system-qemu.img)" >>>>>> 1+0 records in >>>>>> 2048+0 records out >>>>>> 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0063128 s, 166 MB/s >>>>>> Traceback (most recent call last): >>>>>> File "device/generic/goldfish/tools/mk_combined_img.py", line 163, >>>>>> in <module> >>>>>> main() >>>>>> File "device/generic/goldfish/tools/mk_combined_img.py", line 135, >>>>>> in main >>>>>> if check_sparse(partition["path"]): >>>>>> File "device/generic/goldfish/tools/mk_combined_img.py", line 12, >>>>>> in check_sparse >>>>>> with open(filename, 'rb') as i: >>>>>> IOError: [Errno 2] No such file or directory: >>>>>> 'out/target/product/trusty/vbmeta.img' >>>>>> 13:44:11 ninja failed with: exit status 1 >>>>>> >>>>>> #### failed to build some targets (19:41 (mm:ss)) #### >>>>>> >>>>>> What should I do to get qemu_trusty_arm64-userdebug building >>>>>> successfully? >>>>>> >>>>>> Thank you in advance! >>>>>> >>>>>> Best regards, >>>>>> Julien ROBIN >>>>>> >>>>> -- >>>>> -- >>>>> You received this message because you are subscribed to the "Android >>>>> Building" mailing list. >>>>> To post to this group, send email to android-...@googlegroups.com >>>>> To unsubscribe from this group, send email to >>>>> android-...@googlegroups.com >>>>> For more options, visit this group at >>>>> http://groups.google.com/group/android-building?hl=en >>>>> >>>>> --- >>>>> You received this message because you are subscribed to the Google >>>>> Groups "Android Building" group. >>>>> To unsubscribe from this group and stop receiving emails from it, send >>>>> an email to android-...@googlegroups.com. >>>>> To view this discussion on the web visit >>>>> https://groups.google.com/d/msgid/android-building/a30c5a05-6134-43a5-8f3b-32313e32bac0%40googlegroups.com >>>>> >>>>> <https://groups.google.com/d/msgid/android-building/a30c5a05-6134-43a5-8f3b-32313e32bac0%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>> . >>>>> >>>> -- >>> -- >>> You received this message because you are subscribed to the "Android >>> Building" mailing list. >>> To post to this group, send email to android-...@googlegroups.com >>> To unsubscribe from this group, send email to >>> android-...@googlegroups.com >>> For more options, visit this group at >>> http://groups.google.com/group/android-building?hl=en >>> >>> --- >>> You received this message because you are subscribed to a topic in the >>> Google Groups "Android Building" group. >>> To unsubscribe from this topic, visit >>> https://groups.google.com/d/topic/android-building/fY-gdZQ-67M/unsubscribe >>> . >>> To unsubscribe from this group and all its topics, send an email to >>> android-...@googlegroups.com. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/android-building/CAGSQo01HoD%2B8b1LdOrqaqmnQK5Fa1Ys9REVZ9Rt-85E4WrYRaw%40mail.gmail.com >>> >>> <https://groups.google.com/d/msgid/android-building/CAGSQo01HoD%2B8b1LdOrqaqmnQK5Fa1Ys9REVZ9Rt-85E4WrYRaw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> >>> -- -- You received this message because you are subscribed to the "Android Building" mailing list. To post to this group, send email to android-building@googlegroups.com To unsubscribe from this group, send email to android-building+unsubscr...@googlegroups.com For more options, visit this group at http://groups.google.com/group/android-building?hl=en --- You received this message because you are subscribed to the Google Groups "Android Building" group. To unsubscribe from this group and stop receiving emails from it, send an email to android-building+unsubscr...@googlegroups.com. To view this discussion on the web visit https://groups.google.com/d/msgid/android-building/9ffaf8c5-828c-40ef-8dd1-54479ba99ac6o%40googlegroups.com.
