Re: [PATCH 1/5] tests/boot_linux_console: Add initrd test for the Exynos4210
On 10/10/19 3:43 PM, Philippe Mathieu-Daudé wrote: On 10/9/19 9:07 PM, Cleber Rosa wrote: On Wed, Oct 09, 2019 at 02:38:02PM +0100, Peter Maydell wrote: On Tue, 8 Oct 2019 at 22:49, Cleber Rosa wrote: On Mon, Oct 07, 2019 at 05:28:49PM +0100, Peter Maydell wrote: >>> Do our other acceptance tests download random third-party (ie "not a well-known distro") binaries for the tests ? It seems a bit hazardous for reproducability and trustability reasons... [...] I find it hard to judge precisely how much of a third-party some of these are. I remember Philippe mentioning that one of them, I guess the images used on linux_ssh_mips_malta.py, were "as official as it gets" (my words, from my often misleading memory). Reproducibility is definitely an issue, in the sense given that some of these can indeed go away, but as long as they're available the hash recorded in the test should guarantee that we're running the same images. So this thread is a follow up on: https://www.mail-archive.com/qemu-devel@nongnu.org/msg546430.html For Open Source software I can understand we want to be able to rebuild them, and should provide a link about how to rebuild. I don't want to rebuild images myself, I want to focus on testing QEMU. I tried once to build a MIPSsim kernel and it was a total nightmare: https://lists.nongnu.org/archive/html/qemu-devel/2018-04/msg04071.html (Thomas Huth eventually succeeded using buildroot). Do you think we should do something different here? I'm not sure, which is why I asked whether this new test was in line with what we've done previously. Since these are just test cases and we don't redistribute them to other people there's less of a traceability/reproducibility worry, and if we check hashes on download that cuts off a lot of "fail to notice if the image changes for some reason" possible problems. Yep, because I have no clue how to do improve on this (redistributing the binaries is definitely not on the improvement side, and neither is not testing some machine types), the current approach seems good. QEMU machines are not restricted to running Linux or other Open Source software. It seems important to be able to test for regressions with closed-source code too, because it usually has been well tested on real hardware. I just posted a firmware test: https://www.mail-archive.com/qemu-devel@nongnu.org/msg651012.html I could find it stored compressed and uuencoded on the Wayback Machine. Since I don't want to abuse from this amazing service, and other script to decode/uncompress it, I stored it on a new repository with its SHA-1 (the default hash used by Avocado): https://github.com/philmd/qemu-testing-blob/tree/master/hppa/hp9000/712 Is this acceptable? Incorrect? I'v been wondering about this during the week-end. If this is problematic to add the tests in the QEMU repository, we can add the tests in another repository. The problem will be to keep it sync with QEMU. If we add it as a QEMU submodule, it is the same as continuing adding the tests in the main tree. Avocado don't have problem browsing tests in various directories, the problem is we need to import avocado_qemu (tests/acceptance/avocado_qemu) which imports QEMUMachine from qemu.machine. Cleber, we once talked about having the main repository producing python-qemu and python-qemu-acceptance-testing packages. How do you want to proceed? Extract theses files from the main repo and have it consumes the packages? I'm afraid that doesn't scale properly with distributions. Also, updating them to improve the testing will be slow/painful. I now realize this is probably not the clever path to continue, but I'm currently out of ideas to unblock the testing process. Peter what do you think, do you think of other alternatives we can investigate? Thanks, Phil. Regarding Avocado tests, maybe we can simply add a "untrusted" or "closedsource" tag, so people willing to test untrusted binaries could still run the tests using 'avocado --tag untrusted_software', and we could skip these tests by default. I am very interested because I already experimented with: - AIX on PReP/40p - some RTOS on Canon PowerShot A1100 - VxWorks on MIPS and SPARC - closed-source bootloader for Raspi3/4 Regards, Phil.
Re: [PATCH 1/5] tests/boot_linux_console: Add initrd test for the Exynos4210
On 10/9/19 9:07 PM, Cleber Rosa wrote: On Wed, Oct 09, 2019 at 02:38:02PM +0100, Peter Maydell wrote: On Tue, 8 Oct 2019 at 22:49, Cleber Rosa wrote: On Mon, Oct 07, 2019 at 05:28:49PM +0100, Peter Maydell wrote: >>> Do our other acceptance tests download random third-party (ie "not a well-known distro") binaries for the tests ? It seems a bit hazardous for reproducability and trustability reasons... [...] I find it hard to judge precisely how much of a third-party some of these are. I remember Philippe mentioning that one of them, I guess the images used on linux_ssh_mips_malta.py, were "as official as it gets" (my words, from my often misleading memory). Reproducibility is definitely an issue, in the sense given that some of these can indeed go away, but as long as they're available the hash recorded in the test should guarantee that we're running the same images. So this thread is a follow up on: https://www.mail-archive.com/qemu-devel@nongnu.org/msg546430.html For Open Source software I can understand we want to be able to rebuild them, and should provide a link about how to rebuild. I don't want to rebuild images myself, I want to focus on testing QEMU. I tried once to build a MIPSsim kernel and it was a total nightmare: https://lists.nongnu.org/archive/html/qemu-devel/2018-04/msg04071.html (Thomas Huth eventually succeeded using buildroot). Do you think we should do something different here? I'm not sure, which is why I asked whether this new test was in line with what we've done previously. Since these are just test cases and we don't redistribute them to other people there's less of a traceability/reproducibility worry, and if we check hashes on download that cuts off a lot of "fail to notice if the image changes for some reason" possible problems. Yep, because I have no clue how to do improve on this (redistributing the binaries is definitely not on the improvement side, and neither is not testing some machine types), the current approach seems good. QEMU machines are not restricted to running Linux or other Open Source software. It seems important to be able to test for regressions with closed-source code too, because it usually has been well tested on real hardware. I just posted a firmware test: https://www.mail-archive.com/qemu-devel@nongnu.org/msg651012.html I could find it stored compressed and uuencoded on the Wayback Machine. Since I don't want to abuse from this amazing service, and other script to decode/uncompress it, I stored it on a new repository with its SHA-1 (the default hash used by Avocado): https://github.com/philmd/qemu-testing-blob/tree/master/hppa/hp9000/712 Is this acceptable? Incorrect? Regarding Avocado tests, maybe we can simply add a "untrusted" or "closedsource" tag, so people willing to test untrusted binaries could still run the tests using 'avocado --tag untrusted_software', and we could skip these tests by default. I am very interested because I already experimented with: - AIX on PReP/40p - some RTOS on Canon PowerShot A1100 - VxWorks on MIPS and SPARC - closed-source bootloader for Raspi3/4 Regards, Phil.
Re: [PATCH 1/5] tests/boot_linux_console: Add initrd test for the Exynos4210
On Wed, Oct 09, 2019 at 02:38:02PM +0100, Peter Maydell wrote: > On Tue, 8 Oct 2019 at 22:49, Cleber Rosa wrote: > > > > I find it hard to judge precisely how much of a third-party some of > > these are. I remember Philippe mentioning that one of them, I guess > > the images used on linux_ssh_mips_malta.py, were "as official as it > > gets" (my words, from my often misleading memory). > > > > Reproducibility is definitely an issue, in the sense given that some > > of these can indeed go away, but as long as they're available the hash > > recorded in the test should guarantee that we're running the same > > images. > > > > Do you think we should do something different here? > > I'm not sure, which is why I asked whether this new test > was in line with what we've done previously. Since these > are just test cases and we don't redistribute them to > other people there's less of a traceability/reproducibility > worry, and if we check hashes on download that cuts off > a lot of "fail to notice if the image changes for some > reason" possible problems. > > thanks > -- PMM > Yep, because I have no clue how to do improve on this (redistributing the binaries is definitely not on the improvement side, and neither is not testing some machine types), the current approach seems good. Thanks for checking in and giving feedback! - Cleber.
Re: [PATCH 1/5] tests/boot_linux_console: Add initrd test for the Exynos4210
On Tue, 8 Oct 2019 at 22:49, Cleber Rosa wrote:
> On Mon, Oct 07, 2019 at 05:28:49PM +0100, Peter Maydell wrote:
> > Do our other acceptance tests download random third-party
> > (ie "not a well-known distro") binaries for the tests ?
> > It seems a bit hazardous for reproducability and trustability
> > reasons...
> A quick and dirty grep shows (excluding comments and docs):
>
>boot_linux_console.py:kernel_url =
> ('https://archives.fedoraproject.org/pub/archive/fedora'
>boot_linux_console.py:deb_url =
> ('http://snapshot.debian.org/archive/debian/'
>boot_linux_console.py:deb_url =
> ('http://snapshot.debian.org/archive/debian/'
>boot_linux_console.py:deb_url =
> ('http://snapshot.debian.org/archive/debian/'
>boot_linux_console.py:initrd_url =
> ('https://github.com/groeck/linux-build-test/raw/'
>boot_linux_console.py:kernel_url =
> ('https://mipsdistros.mips.com/LinuxDistro/nanomips/'
>boot_linux_console.py:kernel_url =
> ('https://mipsdistros.mips.com/LinuxDistro/nanomips/'
>boot_linux_console.py:kernel_url =
> ('https://mipsdistros.mips.com/LinuxDistro/nanomips/'
>boot_linux_console.py:kernel_url =
> ('https://archives.fedoraproject.org/pub/archive/fedora'
>boot_linux_console.py:kernel_url =
> ('https://archives.fedoraproject.org/pub/archive/fedora'
>boot_linux_console.py:uboot_url =
> ('https://raw.githubusercontent.com/'
>boot_linux_console.py:spi_url =
> ('https://raw.githubusercontent.com/'
>boot_linux_console.py:kernel_url =
> ('https://archives.fedoraproject.org/pub/archive'
>boot_linux_console.py:kernel_url =
> ('http://archive.debian.org/debian/dists/lenny/main/'
>boot_linux_console.py:kernel_url =
> ('https://archives.fedoraproject.org/pub/archive'
>linux_initrd.py:kernel_url =
> ('https://archives.fedoraproject.org/pub/archive/fedora/li'
>linux_initrd.py:kernel_url =
> ('https://archives.fedoraproject.org/pub/archive/fedora'
>linux_ssh_mips_malta.py:'be': {'image_url':
> ('https://people.debian.org/~aurel32/qemu/mips/'
>linux_ssh_mips_malta.py:'le': {'image_url':
> ('https://people.debian.org/~aurel32/qemu/mipsel/'
>linux_ssh_mips_malta.py:kernel_url =
> ('https://people.debian.org/~aurel32/qemu/mips/'
>linux_ssh_mips_malta.py:kernel_url =
> ('https://people.debian.org/~aurel32/qemu/mipsel/'
>linux_ssh_mips_malta.py:kernel_url =
> ('https://people.debian.org/~aurel32/qemu/mips/'
>linux_ssh_mips_malta.py:kernel_url =
> ('https://people.debian.org/~aurel32/qemu/mipsel/'
>machine_m68k_nextcube.py:rom_url =
> ('http://www.nextcomputers.org/NeXTfiles/Software/ROM_Files/'
>
> I find it hard to judge precisely how much of a third-party some of
> these are. I remember Philippe mentioning that one of them, I guess
> the images used on linux_ssh_mips_malta.py, were "as official as it
> gets" (my words, from my often misleading memory).
>
> Reproducibility is definitely an issue, in the sense given that some
> of these can indeed go away, but as long as they're available the hash
> recorded in the test should guarantee that we're running the same
> images.
>
> Do you think we should do something different here?
I'm not sure, which is why I asked whether this new test
was in line with what we've done previously. Since these
are just test cases and we don't redistribute them to
other people there's less of a traceability/reproducibility
worry, and if we check hashes on download that cuts off
a lot of "fail to notice if the image changes for some
reason" possible problems.
thanks
-- PMM
Re: [PATCH 1/5] tests/boot_linux_console: Add initrd test for the Exynos4210
On Tue, Oct 08, 2019 at 05:49:07PM -0400, Cleber Rosa wrote:
> On Mon, Oct 07, 2019 at 05:28:49PM +0100, Peter Maydell wrote:
> > On Sat, 5 Oct 2019 at 16:47, Philippe Mathieu-Daudé wrote:
> > >
> > > This test boots a Linux kernel on a smdkc210 board and verify
> > > the serial output is working.
> > >
> > > The cpio image used comes from the linux-build-test project:
> > > https://github.com/groeck/linux-build-test
> >
> > Thanks for putting this test case together, very helpful.
> >
> > > +initrd_url = ('https://github.com/groeck/linux-build-test/raw/'
> > > + '2eb0a73b5d5a28df3170c546ddaaa9757e1e0848/rootfs/'
> > > + 'arm/rootfs-armv5.cpio.gz')
> >
> > Do our other acceptance tests download random third-party
> > (ie "not a well-known distro") binaries for the tests ?
> > It seems a bit hazardous for reproducability and trustability
> > reasons...
> >
FWIW, the root file system was generated with buildroot, with
minor modifications. It should be possible to create a clean
root file system from there.
Guenter
Re: [PATCH 1/5] tests/boot_linux_console: Add initrd test for the Exynos4210
On Mon, Oct 07, 2019 at 05:28:49PM +0100, Peter Maydell wrote:
> On Sat, 5 Oct 2019 at 16:47, Philippe Mathieu-Daudé wrote:
> >
> > This test boots a Linux kernel on a smdkc210 board and verify
> > the serial output is working.
> >
> > The cpio image used comes from the linux-build-test project:
> > https://github.com/groeck/linux-build-test
>
> Thanks for putting this test case together, very helpful.
>
> > +initrd_url = ('https://github.com/groeck/linux-build-test/raw/'
> > + '2eb0a73b5d5a28df3170c546ddaaa9757e1e0848/rootfs/'
> > + 'arm/rootfs-armv5.cpio.gz')
>
> Do our other acceptance tests download random third-party
> (ie "not a well-known distro") binaries for the tests ?
> It seems a bit hazardous for reproducability and trustability
> reasons...
>
> thanks
> -- PMM
A quick and dirty grep shows (excluding comments and docs):
boot_linux_console.py:kernel_url =
('https://archives.fedoraproject.org/pub/archive/fedora'
boot_linux_console.py:deb_url =
('http://snapshot.debian.org/archive/debian/'
boot_linux_console.py:deb_url =
('http://snapshot.debian.org/archive/debian/'
boot_linux_console.py:deb_url =
('http://snapshot.debian.org/archive/debian/'
boot_linux_console.py:initrd_url =
('https://github.com/groeck/linux-build-test/raw/'
boot_linux_console.py:kernel_url =
('https://mipsdistros.mips.com/LinuxDistro/nanomips/'
boot_linux_console.py:kernel_url =
('https://mipsdistros.mips.com/LinuxDistro/nanomips/'
boot_linux_console.py:kernel_url =
('https://mipsdistros.mips.com/LinuxDistro/nanomips/'
boot_linux_console.py:kernel_url =
('https://archives.fedoraproject.org/pub/archive/fedora'
boot_linux_console.py:kernel_url =
('https://archives.fedoraproject.org/pub/archive/fedora'
boot_linux_console.py:uboot_url =
('https://raw.githubusercontent.com/'
boot_linux_console.py:spi_url = ('https://raw.githubusercontent.com/'
boot_linux_console.py:kernel_url =
('https://archives.fedoraproject.org/pub/archive'
boot_linux_console.py:kernel_url =
('http://archive.debian.org/debian/dists/lenny/main/'
boot_linux_console.py:kernel_url =
('https://archives.fedoraproject.org/pub/archive'
linux_initrd.py:kernel_url =
('https://archives.fedoraproject.org/pub/archive/fedora/li'
linux_initrd.py:kernel_url =
('https://archives.fedoraproject.org/pub/archive/fedora'
linux_ssh_mips_malta.py:'be': {'image_url':
('https://people.debian.org/~aurel32/qemu/mips/'
linux_ssh_mips_malta.py:'le': {'image_url':
('https://people.debian.org/~aurel32/qemu/mipsel/'
linux_ssh_mips_malta.py:kernel_url =
('https://people.debian.org/~aurel32/qemu/mips/'
linux_ssh_mips_malta.py:kernel_url =
('https://people.debian.org/~aurel32/qemu/mipsel/'
linux_ssh_mips_malta.py:kernel_url =
('https://people.debian.org/~aurel32/qemu/mips/'
linux_ssh_mips_malta.py:kernel_url =
('https://people.debian.org/~aurel32/qemu/mipsel/'
machine_m68k_nextcube.py:rom_url =
('http://www.nextcomputers.org/NeXTfiles/Software/ROM_Files/'
I find it hard to judge precisely how much of a third-party some of
these are. I remember Philippe mentioning that one of them, I guess
the images used on linux_ssh_mips_malta.py, were "as official as it
gets" (my words, from my often misleading memory).
Reproducibility is definitely an issue, in the sense given that some
of these can indeed go away, but as long as they're available the hash
recorded in the test should guarantee that we're running the same
images.
Do you think we should do something different here?
Thanks!
- Cleber.
Re: [PATCH 1/5] tests/boot_linux_console: Add initrd test for the Exynos4210
On Sat, Oct 05, 2019 at 05:47:44PM +0200, Philippe Mathieu-Daudé wrote: > This test boots a Linux kernel on a smdkc210 board and verify > the serial output is working. > > The cpio image used comes from the linux-build-test project: > https://github.com/groeck/linux-build-test > > If ARM is a target being built, "make check-acceptance" will > automatically include this test by the use of the "arch:arm" tags. > > This test can be run using: > > $ avocado --show=app,console run -t machine:smdkc210 > tests/acceptance/boot_linux_console.py > console: Booting Linux on physical CPU 0x900 > console: Linux version 4.19.0-6-armmp (debian-ker...@lists.debian.org) (gcc > version 8.3.0 (Debian 8.3.0-6)) #1 SMP Debian 4.19.67-2+deb10u1 (2019-09-20) > console: CPU: ARMv7 Processor [410fc090] revision 0 (ARMv7), cr=10c5387d > console: CPU: PIPT / VIPT nonaliasing data cache, VIPT nonaliasing > instruction cache > console: OF: fdt: Machine model: Samsung smdkv310 evaluation board based on > Exynos4210 > [...] > console: Samsung CPU ID: 0x43210211 > console: random: get_random_bytes called from start_kernel+0xa0/0x504 with > crng_init=0 > console: percpu: Embedded 17 pages/cpu s39756 r8192 d21684 u69632 > console: Built 1 zonelists, mobility grouping on. Total pages: 249152 > console: Kernel command line: printk.time=0 console=ttySAC0,115200n8 > earlyprintk random.trust_cpu=off cryptomgr.notests cpuidle.off=1 panic=-1 > noreboot > [...] > console: L2C: platform modifies aux control register: 0x0202 -> > 0x3e420001 > console: L2C: platform provided aux values permit register corruption. > console: L2C: DT/platform modifies aux control register: 0x0202 -> > 0x3e420001 > console: L2C-310 erratum 769419 enabled > console: L2C-310 enabling early BRESP for Cortex-A9 > console: L2C-310: enabling full line of zeros but not enabled in Cortex-A9 > console: L2C-310 ID prefetch enabled, offset 1 lines > console: L2C-310 dynamic clock gating disabled, standby mode disabled > console: L2C-310 cache controller enabled, 8 ways, 128 kB > console: L2C-310: CACHE_ID 0x41c8, AUX_CTRL 0x7e420001 > console: Exynos4210 clocks: sclk_apll = 1200, sclk_mpll = 1200 > console: sclk_epll = 1200, sclk_vpll = 1200, arm_clk = 1200 > [...] > console: s3c-i2c 1386.i2c: slave address 0x00 > console: s3c-i2c 1386.i2c: bus frequency set to 93 KHz > console: s3c-i2c 1386.i2c: i2c-0: S3C I2C adapter > [...] > console: dma-pl330 1268.pdma: Loaded driver for PL330 DMAC-241330 > console: dma-pl330 1268.pdma: DBUFF-256x8bytes Num_Chans-8 > Num_Peri-32 Num_Events-16 > console: dma-pl330 1269.pdma: Loaded driver for PL330 DMAC-241330 > console: dma-pl330 1269.pdma: DBUFF-256x8bytes Num_Chans-8 > Num_Peri-32 Num_Events-16 > console: dma-pl330 1285.mdma: Loaded driver for PL330 DMAC-241330 > console: dma-pl330 1285.mdma: DBUFF-256x8bytes Num_Chans-8 > Num_Peri-1 Num_Events-16 > console: dma-pl330 1285.mdma: PM domain LCD0 will not be powered off > console: Serial: 8250/16550 driver, 4 ports, IRQ sharing disabled > console: Serial: AMBA driver > console: 1380.serial: ttySAC0 at MMIO 0x1380 (irq = 40, base_baud = > 0) is a S3C6400/10 > console: console [ttySAC0] enabled > console: 1381.serial: ttySAC1 at MMIO 0x1381 (irq = 41, base_baud = > 0) is a S3C6400/10 > console: 1382.serial: ttySAC2 at MMIO 0x1382 (irq = 42, base_baud = > 0) is a S3C6400/10 > console: 1383.serial: ttySAC3 at MMIO 0x1383 (irq = 43, base_baud = > 0) is a S3C6400/10 > [...] > console: Freeing unused kernel memory: 2048K > console: Run /init as init process > console: mount: mounting devtmpfs on /dev failed: Device or resource busy > console: Starting logging: OK > console: Initializing random number generator... random: dd: uninitialized > urandom read (512 bytes read) > console: done. > console: Starting network: OK > console: Found console ttySAC0 > console: Linux version 4.19.0-6-armmp (debian-ker...@lists.debian.org) (gcc > version 8.3.0 (Debian 8.3.0-6)) #1 SMP Debian 4.19.67-2+deb10u1 (2019-09-20) > console: Boot successful. > PASS (37.98 s) > > Signed-off-by: Philippe Mathieu-Daudé > --- > serial input is not working :( > > I sometime get (not always): > > Starting network: OK > [ 70.403690] rcu: INFO: rcu_preempt detected stalls on CPUs/tasks: > [ 70.423212] rcu: 0-...!: (36 GPs behind) idle=c7a/1/0x4000 > softirq=287/288 fqs=1 > [ 70.428209] rcu: (detected by 1, t=2602 jiffies, g=-443, q=2209) > [ 70.432826] Sending NMI from CPU 1 to CPUs 0: > [ 70.473866] NMI backtrace for cpu 0 > [ 70.476621] CPU: 0 PID: 112 Comm: cat Not tainted 4.19.0 #1 > [ 70.476711] Hardware name: SAMSUNG EXYNOS (Flattened Device Tree) > [ 70.476916] PC is at mntput_no_expire+0x88/0x464 > [ 70.476996] LR is at
Re: [PATCH 1/5] tests/boot_linux_console: Add initrd test for the Exynos4210
On Sat, 5 Oct 2019 at 16:47, Philippe Mathieu-Daudé wrote:
>
> This test boots a Linux kernel on a smdkc210 board and verify
> the serial output is working.
>
> The cpio image used comes from the linux-build-test project:
> https://github.com/groeck/linux-build-test
Thanks for putting this test case together, very helpful.
> +initrd_url = ('https://github.com/groeck/linux-build-test/raw/'
> + '2eb0a73b5d5a28df3170c546ddaaa9757e1e0848/rootfs/'
> + 'arm/rootfs-armv5.cpio.gz')
Do our other acceptance tests download random third-party
(ie "not a well-known distro") binaries for the tests ?
It seems a bit hazardous for reproducability and trustability
reasons...
thanks
-- PMM
