Hi Julien, 郁进, Did either of you manage to run AOSP on upstream QEMU?
Thanks On Saturday, July 25, 2020 at 12:21:35 AM UTC+9 [email protected] wrote: > This is tremendously helpful. Thanks a lot. > > My work is focusing on kernel, base on arm architecture, I want have > whole view of the running kernel. > > With qemu & qemu monitor & gdb & crash tools, I can stop kernel anywhere, > read phy/virt address, registers even system register(with last qemu > release). > > Thanks for sharing your experience, It does greate help. > > Hope to hear that you successfully run last AOSP on upstream QEMU. > > 在 2020年7月4日星期六 UTC+8上午12:35:32，Julien Robin写道： > >> Hi 郁进, >> >> Recently I focused on making the untouched Android code base working with >> standard/upstream QEMU (which is a little bit different from the AOSP's >> emulator, which is a QEMU derivative), focusing on x86_64 builds and >> "android-x86" project build configurations; the work is still in progress. >> But from what I know : >> >> When building aosp_XXXX-eng project, the "emulator" command line is made >> available so that just typing "emulator" into your terminal is (normally) >> enough to get it starting with the correct parameters set. Most of the time >> I use "emulator -verbose -show-kernel -no-snapshot" when running the AOSP's >> emulator. So that you have all the default parameters shown into your >> terminal, then kernel debug information, then Android's "init" output into >> your terminal. The kernel used to run those builds is called >> "kernel-ranchu" : it's a customised and prebuilt version of the 4.14 >> kernel, intended for use with the aosp's emulator virtual hardware and >> drivers (called goldfish in its original version, then ranchu). >> >> If you build aosp then close the terminal window, and you want the >> "emulator" command to be available again, you should just type "source >> build/envsetup.sh && lunch" commands again (no need to rebuild everything). >> >> But for arm64 builds its horribly slow, like several minutes for booting >> (upstream qemu is making the following option "-accel tcg,thread=multi" >> available so that the performance using an AMD 3700X is almost similar to >> using KVM on a native arm64 Raspberry Pi 4. But unfortunately this accel >> option does not seem to be used by the "emulator" command, and seems not to >> work on AOSP's emulator, at least last time I tried). Be aware that >> emulator is a derivative of QEMU, recognizing some AOSP specific commands, >> and not recognizing some orginal QEMU commands. >> >> *About having the kernel finding and starting "init" :* >> >> Lot of things can be confusing about having your emulator or qemu finding >> and running "init" process. I will give you everything I discovered so that >> hopefully you will be able to understand and debug anything that happens to >> you >> >> On really old versions of Android, "init" executable was only available >> into the ramdisk, and "system" partition was just containing the /system >> subfolder (this is the opposite of "system as root"). In this case, Linux >> kernel absolutely needed to load the ramdisk, because init was into this >> ramdisk. I guess fstab file was into the ramdisk too, in order to find and >> mount partitions once init started. >> >> Some more recent version of android using "system as root" (8 or 9 maybe) >> gave some ramdisk file as parameter to the emulator, despite the fact this >> ramdisk wasn't actually used anymore by the emulator : the content of the >> ramdisk was already into system.img. And the kernel was already able to >> access the system.img partition without the need of anything into the >> ramdisk - so the ramdisk wasn't used anymore. "init" file was at the root >> of the system.img partition image. In this case, you should ensure your >> machine and kernel are able to find your ext4 root (when in trouble, look >> into the kernel messages to find an ext4 file system at /dev/sda, or >> /dev/vda, or even sda1, vda1, etc, so that you can be sure about the kernel >> ability to see the partition you need). >> >> But Android 10 introduced a new partition/image format that can be >> dynamically expanded and physically fragmented (so that devices sold with a >> too small system partition for being updated, can be updated anyway). This >> is cool feature, but a little bit more complex to handle. When using this >> kind of system images, android requires a brand new specific ramdisk with a >> specific "init" executable in order to read and mount this new system >> partition format created by android. I believe emulator is using it with >> Android 10 (to be confirmed). But in order to focus on my work, I'm still >> using the previous way : an ext4 system.img with everything into it ;) so >> that qemu boots it directly. >> >> >> For example, on my x86_64 custom builds using "system as root", and >> having the vendor files included into system.img, as a standard ext4 image, >> I run the following command to start it using QEMU : >> >> kvm -m 4096 -smp 4 -kernel bzImage -append "root=/dev/sda >> rootfstype=ext4 ro init=/init selinux=1 checkreqprot=1 >> androidboot.selinux=permissive console=ttyS0 androidboot.hardware=luoss >> loglevel=8" -serial mon:stdio -drive format=raw,file=system.img -vga std >> -m 4096 ask for 4GB RAM >> bzImage is a custom 5.4 kernel >> root=/dev/sda rootfstype=ext4 ro init=/init is telling my kernel to mount >> what it sees as "/dev/sda" and to run the "init" executable which is >> located into it >> selinux=1 checkreqprot=1 androidboot.selinux=permissive are selinux >> parameters : when you want to disable selinux, you can't (init absolutely >> wants your kernel to have selinux running) but you set it to >> androidboot.selinux=permissive, so that no operation is actually blocked. >> console=ttyS0 ask the kernel to print its messages into the serial port >> androidboot.hardware=HW_NAME is used to tell init process to load >> init.HW_NAME.rc configuration file (that should be into /init.HW_NAME.rc >> or /vendor/etc/init/hw/init.HW_NAME.rc). For aosp emulator builds, it is >> /vendor/etc/init/hw/init.ranchu.rc >> loglevel=8 ask for the kernel, then init, to be as verbose as possible >> >> -serial mon:stdio is a qemu parameter I use to redirect the virtual >> serial port (on which the kernel talks) to the terminal in which qemu is >> running. So that I can use this terminal as a read/write terminal with the >> virtual machine. >> >> >> The system image i'm using to do so is an ext4 partition image, on which >> i placed everything needed, and an init.luoss.rc and fstab.luoss files - >> mainly just to mount a /data tmpfs instead of using a real userdata.img >> partiton - yes, I'm cheating ;) normally system.img is not enough, and in >> most case, /vendor subfolder is into a vendor.img partition. >> >> I guess that when I'll be trying to run arm64 builds on qemu, some >> parameters will need to be changed. >> >> I hope that all this information will help you to successfully dig, >> understand and solve your issues! >> >> Best regards >> >> Julien >> >> Le lundi 29 juin 2020 06:41:01 UTC+2, 郁进 a écrit : >>> >>> Hi, >>> >>> I found another tickets you submitted to run arm64 android via qemu. >>> >>> I tried to lunch aosp_arm64-eng project, run with android emulator, with >>> qemu args: >>> >>> emulator -kernel >>> ../Android_kernel/out/android-mainline/common/arch/arm64/boot/Image -system >>> xxx ...... *-qemu **--monitor tcp:127.0.0.1:2222 >>> <http://127.0.0.1:2222>,server,nowait -gdb tcp::4444 -S* >>> >>> But the Android kernel panic occurred since no init found, and I try to >>> dump guest memory via qemu monitor, can't parsed by crash tools. >>> >>> Have you build and run arm64 android via qemu successfully? How ? >>> >>> Looking forward to your reply, many thanks. >>> >>> >>> 在 2020年3月3日星期二 UTC+8下午11:18:37，Julien ROBIN写道： >>>> >>>> Many thanks for your answers >>>> >>>> In fact what I'm trying to do is using QEMU files into AOSP source code >>>> to get Android (with GUI) built and running on a standard/upstream QEMU >>>> version. >>>> >>>> This would give the ability of running Android and its future releases >>>> without real "porting", on any device running a Linux kernel (be it >>>> upstream or not) and having KVM working on it. Which would be extremely >>>> interesting. >>>> >>>> From what you write, it seems qemu-trusty-arm64 is not really for me. >>>> So what I'm trying to do is probably more about using >>>> device/generic/qemu/qemu_arm64.mk (and others archs) files included >>>> into AOSP. So I'll continue into this other subject : >>>> https://groups.google.com/forum/?hl=bn#!topic/android-building/kbiEg5OxBGU >>>> >>>> >>>> While talking with you, I noticed that AVD Emulator is different from >>>> upstream QEMU, probably about goldfish and ranchu related things. Things >>>> working fine with the emulator (aosp_x86_64-eng for example and >>>> android-goldfish-4.14-dev which seems mandatory to get Android 10 working >>>> with AVD Emulator) does not seems to be working with upstream QEMU (at >>>> least I can't get the GUI). Therefore the documentation about AVD Emulator >>>> builds https://source.android.com/setup/create/avd didn't really help >>>> me for upstream QEMU. >>>> >>>> Can you confirm it's possible to use modern versions of AOSP on >>>> upstream QEMU? Someone did succeed in the past with Android 6 / untouched >>>> AOSP and a extended compatibility kernel (I have one based on >>>> android-goldfish-4.14-dev), this is why I'm trying with newer versions of >>>> AOSP. This was documented here, but it's not up to date anymore : >>>> https://www.collabora.com/news-and-blog/blog/2016/09/02/building-android-for-qemu-a-step-by-step-guide/ >>>> >>>> Thank you in advance >>>> >>>> Julien >>>> >>>> >>>> On 3/2/20 10:51 PM, 'Matthew Maurer' via Android Building wrote: >>>> >>>> This target is a testing target used for evaluating our reference >>>> TrustZone implementation. "storageproxyd" is continuously resetting there >>>> because it is not being launched by our test script >>>> <https://android.googlesource.com/trusty/device/arm/generic-arm64/+/refs/heads/master/project/qemu/qemu.py> >>>> which >>>> would attach a virtual RPMB resource. >>>> >>>> A few notes about this target: >>>> * It does not have a GUI >>>> * Running Java is not supported and unlikely to work >>>> * It requires a custom EL3 and Trusty running alongside it to work >>>> >>>> The manifest for these components is at >>>> https://android.googlesource.com/trusty/manifest, and that manifest >>>> contains a checked in prebuilt of the qemu_trusty_arm64-userdebug target >>>> that we are currently testing against. >>>> >>>> If you tell me more about what you're trying to make the target do, I >>>> may be able to give you further help with getting it to do what you want. >>>> If you aren't trying to use or inspect the Trusty TZ implementation, this >>>> target probably isn't the one you're looking for. >>>> >>>> On Mon, Mar 2, 2020 at 1:34 PM Dan Willemsen <[email protected]> >>>> wrote: >>>> >>>>> +Matthew Maurer Do we have any documentation on this target? I think >>>>> the last time we talked about this target it was only minimally >>>>> functional. >>>>> >>>>> - Dan >>>>> >>>>> On Mon, Mar 2, 2020 at 12:51 PM Julien Robin <[email protected]> >>>>> wrote: >>>>> >>>>>> I succeeded to find a workaround for the issue I signaled here, >>>>>> editing device/generic/trusty/BoardConfig.mk and putting >>>>>> BUILD_QEMU_IMAGES >>>>>> to false, which still gives vendor.img, system.img and userdata.img >>>>>> >>>>>> I then found this kernel >>>>>> https://android.googlesource.com/kernel/common/+/refs/heads/android-trusty-4.19, >>>>>> >>>>>> and using arch/arm64/configs/trusty_qemu_defconfig, I successfully built >>>>>> it. >>>>>> >>>>>> Also, I'm able to start most of this Android build on Debian 10 >>>>>> provided qemu-system-aarch64 the following way : >>>>>> >>>>>> qemu-system-aarch64 >>>>>> -M virt >>>>>> -cpu cortex-a57 >>>>>> -smp 1 >>>>>> -m 2048 >>>>>> -monitor none >>>>>> -parallel none >>>>>> -serial mon:stdio >>>>>> -kernel Image -append "root=/dev/vda rootfstype=ext4 ro >>>>>> init=/init loglevel=8 selinux=1 checkreqprot=1 >>>>>> androidboot.selinux=permissive androidboot.hardware=qemu_trusty >>>>>> console=ttyAMA0" >>>>>> -device virtio-gpu-pci >>>>>> -drive format=raw,file=system-custom.img >>>>>> -drive format=raw,file=userdata-custom.img >>>>>> >>>>>> userdata-custom.img is just a bigger userdata.img (1024MB instead of >>>>>> 4) in which I used resize2fs, system-custom.img is just a system.img in >>>>>> which the placeholder folder "vendor" now embeds the content of >>>>>> vendor.img, >>>>>> copied by "cp -a". Finally, still into system-custom.img, >>>>>> fstab.qemu_trusty >>>>>> only contains >>>>>> LABEL=data /data ext4 noatime,nosuid, >>>>>> nodev,errors=panic wait >>>>>> So that /data (available at /dev/vdb) is mounted by its label, while >>>>>> system-custom.img embeding root system and vendor is already mounted by >>>>>> kernel cmdline (by /dev/vda). ramdisk.img is not used as system.img >>>>>> already >>>>>> contains everything needed to start init. >>>>>> >>>>>> However, I guess I should use a slightly modified version of qemu, >>>>>> available here >>>>>> https://android.googlesource.com/trusty/external/qemu/+/refs/heads/master >>>>>> Because using Debian 10 qemu version, I get the following output, and >>>>>> no display : >>>>>> >>>>>> init: init first stage started! >>>>>> [...] >>>>>> init: Loading SELinux policy >>>>>> [...] >>>>>> selinux: SELinux: Loaded policy from /vendor/etc/selinux/ >>>>>> precompiled_sepolicy >>>>>> >>>>>> selinux: SELinux: Loaded file_contexts >>>>>> >>>>>> random: init: uninitialized urandom read (40 bytes read) >>>>>> random: init: uninitialized urandom read (40 bytes read) >>>>>> init: init second stage started! >>>>>> [...] >>>>>> ueventd: Coldboot took 1.344 seconds >>>>>> [...] >>>>>> init: starting service 'storageproxyd'... >>>>>> libprocessgroup: CgroupMap::FindController called for [1] failed, RC >>>>>> file was not initialized properly >>>>>> libprocessgroup: Failed to make and chown /uid_0: Read-only file >>>>>> system >>>>>> libprocessgroup: CgroupMap::FindController called for [829] failed, >>>>>> RC file was not initialized properly >>>>>> init: createProcessGroup(0, 829) failed for service 'storageproxyd': >>>>>> Read-only file system >>>>>> init: cpuset cgroup controller is not mounted! >>>>>> init: Service 'storageproxyd' (pid 829) exited with status 1 >>>>>> init: Sending signal 9 to service 'storageproxyd' (pid 829) process >>>>>> group... >>>>>> libprocessgroup: CgroupMap::FindController called for [1] failed, RC >>>>>> file was not initialized properly >>>>>> libprocessgroup: CgroupMap::FindController called for [1] failed, RC >>>>>> file was not initialized properly >>>>>> >>>>>> >>>>>> init: starting service 'storageproxyd'... >>>>>> libprocessgroup: CgroupMap::FindController called for [1] failed, RC >>>>>> file was not initialized properly >>>>>> libprocessgroup: Failed to make and chown /uid_0: Read-only file >>>>>> system >>>>>> init: createProcessGroup(0, 830) failed for service 'storageproxyd': >>>>>> Read-only file system >>>>>> libprocessgroup: CgroupMap::FindController called for [830] failed, >>>>>> RC file was not initialized properly >>>>>> init: cpuset cgroup controller is not mounted! >>>>>> init: Service 'storageproxyd' (pid 830) exited with status 1 >>>>>> init: Sending signal 9 to service 'storageproxyd' (pid 830) process >>>>>> group... >>>>>> >>>>>> ...and so on in loop >>>>>> >>>>>> However I'm still not sure to proceed correctly, as I'm doing >>>>>> everything alone with no documentation (I may have missed it?) >>>>>> >>>>>> Is here some documentation available? If there isn't, is someone able >>>>>> to provide an example of working qemu command line (even old or >>>>>> unoptimized), or just few explanations so that I can be placed on the >>>>>> right >>>>>> track? >>>>>> >>>>>> Thank you very much in advance >>>>>> >>>>>> >>>>>> On 3/1/20 2:14 PM, Julien Robin wrote: >>>>>>> >>>>>>> Hi there, >>>>>>> >>>>>>> I see that starting from branch android-10.0.0_r1, lunch offers a >>>>>>> choice called qemu_trusty_arm64-userdebug, in which I'm interested >>>>>>> >>>>>>> However, I got the following error, no matter which OS I'm using for >>>>>>> building (be it Ubuntu 18.048 or Debian 10). I have this error on r1, >>>>>>> r20 >>>>>>> and r29. >>>>>>> >>>>>>> [ 62% 16150/26047] Create system-qemu.img now >>>>>>> FAILED: out/target/product/trusty/system-qemu.img >>>>>>> /bin/bash -c "(export SGDISK=out/host/linux-x86/bin/sgdisk >>>>>>> SIMG2IMG=out/host/linux-x86/bin/simg2img; >>>>>>> device/generic/goldfish/tools/mk_combined_img.py -i >>>>>>> out/target/product/trusty/system-qemu-config.txt -o >>>>>>> out/target/product/trusty/system-qemu.img)" >>>>>>> 1+0 records in >>>>>>> 2048+0 records out >>>>>>> 1048576 bytes (1.0 MB, 1.0 MiB) copied, 0.0063128 s, 166 MB/s >>>>>>> Traceback (most recent call last): >>>>>>> File "device/generic/goldfish/tools/mk_combined_img.py", line 163, >>>>>>> in <module> >>>>>>> main() >>>>>>> File "device/generic/goldfish/tools/mk_combined_img.py", line 135, >>>>>>> in main >>>>>>> if check_sparse(partition["path"]): >>>>>>> File "device/generic/goldfish/tools/mk_combined_img.py", line 12, >>>>>>> in check_sparse >>>>>>> with open(filename, 'rb') as i: >>>>>>> IOError: [Errno 2] No such file or directory: >>>>>>> 'out/target/product/trusty/vbmeta.img' >>>>>>> 13:44:11 ninja failed with: exit status 1 >>>>>>> >>>>>>> #### failed to build some targets (19:41 (mm:ss)) #### >>>>>>> >>>>>>> What should I do to get qemu_trusty_arm64-userdebug building >>>>>>> successfully? >>>>>>> >>>>>>> Thank you in advance! >>>>>>> >>>>>>> Best regards, >>>>>>> Julien ROBIN >>>>>>> >>>>>> -- >>>>>> -- >>>>>> You received this message because you are subscribed to the "Android >>>>>> Building" mailing list. >>>>>> To post to this group, send email to [email protected] >>>>>> To unsubscribe from this group, send email to >>>>>> [email protected] >>>>>> For more options, visit this group at >>>>>> http://groups.google.com/group/android-building?hl=en >>>>>> >>>>>> --- >>>>>> You received this message because you are subscribed to the Google >>>>>> Groups "Android Building" group. >>>>>> To unsubscribe from this group and stop receiving emails from it, >>>>>> send an email to [email protected]. >>>>>> To view this discussion on the web visit >>>>>> https://groups.google.com/d/msgid/android-building/a30c5a05-6134-43a5-8f3b-32313e32bac0%40googlegroups.com >>>>>> >>>>>> <https://groups.google.com/d/msgid/android-building/a30c5a05-6134-43a5-8f3b-32313e32bac0%40googlegroups.com?utm_medium=email&utm_source=footer> >>>>>> . >>>>>> >>>>> -- >>>> -- >>>> You received this message because you are subscribed to the "Android >>>> Building" mailing list. >>>> To post to this group, send email to [email protected] >>>> To unsubscribe from this group, send email to >>>> [email protected] >>>> For more options, visit this group at >>>> http://groups.google.com/group/android-building?hl=en >>>> >>>> --- >>>> You received this message because you are subscribed to a topic in the >>>> Google Groups "Android Building" group. >>>> To unsubscribe from this topic, visit >>>> https://groups.google.com/d/topic/android-building/fY-gdZQ-67M/unsubscribe >>>> . >>>> To unsubscribe from this group and all its topics, send an email to >>>> [email protected]. >>>> To view this discussion on the web visit >>>> https://groups.google.com/d/msgid/android-building/CAGSQo01HoD%2B8b1LdOrqaqmnQK5Fa1Ys9REVZ9Rt-85E4WrYRaw%40mail.gmail.com >>>> >>>> <https://groups.google.com/d/msgid/android-building/CAGSQo01HoD%2B8b1LdOrqaqmnQK5Fa1Ys9REVZ9Rt-85E4WrYRaw%40mail.gmail.com?utm_medium=email&utm_source=footer> >>>> . >>>> >>>> -- -- You received this message because you are subscribed to the "Android Building" mailing list. To post to this group, send email to [email protected] To unsubscribe from this group, send email to [email protected] For more options, visit this group at http://groups.google.com/group/android-building?hl=en --- You received this message because you are subscribed to the Google Groups "Android Building" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/android-building/a52e9878-7432-4b28-b011-3e58d706491cn%40googlegroups.com.
