Re: Can we use emulation of other architectures to run integration tests?
On 30. 07. 20 14:14, Miro Hrončok wrote: On 30. 07. 20 12:24, Aleksandra Fedorova wrote: As COPR has recently got support for s390 builds, the question is: if emulation is good enough for building packages, can we use it for testing? What are the limitations there? Is it worth it? I don't have that many experience with this but every time I've attempted to build some Python package via mock's forcearch emulation on s390x, I got some segfaults and gave up. Seems like I am not the only one, see: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org/message/RPUH24RWLJQLXLD7VQ35NUPQD52RQAYQ/ -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Can we use emulation of other architectures to run integration tests?
On Thu, Jul 30, 2020 at 01:06:46PM -0400, Robbie Harwood wrote: > "Richard W.M. Jones" writes: > > > On Thu, Jul 30, 2020 at 11:46:08AM -0400, Robbie Harwood wrote: > >> Aleksandra Fedorova writes: > >> > >>> As COPR has recently got support for s390 builds, the question is: > >>> if emulation is good enough for building packages, can we use it for > >>> testing? What are the limitations there? Is it worth it? > >> > >> Cross-architecture emulation is unbelievably slow in the general > >> case. While it helps for some specific use cases, it's not a > >> substitute for actually getting hardware. > > > > Since qemu TCG now supports host thread per vCPU you can usually throw > > lots of vCPUs at the problem, assuming your builds can be parallelised > > and your x86 hardware has plenty of cores. > > That's good to know, and definitely I could see that helping with builds > themselves, but won't help with many (most?) test suites :) This is sadly true for many :-( For nbdkit though, our test suite will use as many ‘make -j #cores’ as you can throw at it (I have tested 64) :-) Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Can we use emulation of other architectures to run integration tests?
"Richard W.M. Jones" writes: > On Thu, Jul 30, 2020 at 11:46:08AM -0400, Robbie Harwood wrote: >> Aleksandra Fedorova writes: >> >>> As COPR has recently got support for s390 builds, the question is: >>> if emulation is good enough for building packages, can we use it for >>> testing? What are the limitations there? Is it worth it? >> >> Cross-architecture emulation is unbelievably slow in the general >> case. While it helps for some specific use cases, it's not a >> substitute for actually getting hardware. > > Since qemu TCG now supports host thread per vCPU you can usually throw > lots of vCPUs at the problem, assuming your builds can be parallelised > and your x86 hardware has plenty of cores. That's good to know, and definitely I could see that helping with builds themselves, but won't help with many (most?) test suites :) Thanks, --Robbie signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Can we use emulation of other architectures to run integration tests?
On Thu, Jul 30, 2020 at 11:46:08AM -0400, Robbie Harwood wrote: > Aleksandra Fedorova writes: > > > As COPR has recently got support for s390 builds, the question is: if > > emulation is good enough for building packages, can we use it for > > testing? What are the limitations there? Is it worth it? > > Cross-architecture emulation is unbelievably slow in the general case. > While it helps for some specific use cases, it's not a substitute for > actually getting hardware. Since qemu TCG now supports host thread per vCPU you can usually throw lots of vCPUs at the problem, assuming your builds can be parallelised and your x86 hardware has plenty of cores. We use it reasonably successfully to build a lot of Fedora/RISC-V packages, along with a limited amount of actual hardware. Rich. https://wiki.qemu.org/Features/tcg-multithread -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com Fedora Windows cross-compiler. Compile Windows programs, test, and build Windows installers. Over 100 libraries supported. http://fedoraproject.org/wiki/MinGW ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Can we use emulation of other architectures to run integration tests?
On Thu, Jul 30, 2020 at 02:10:58PM +0200, Aleksandra Fedorova wrote: > The scenario we use for current x86 dist-git tests is based on vms: we > spin up a full vm with qemu-kvm, ssh inside it, install a package and > run the test script. I guess you could do something with virt-builder, but ... > Should it be possible to do the same with qemu-system-s390x? Maybe > even with nested virtualization, with s390x inside x86? qemu-system-s390x uses TCG to emulate instructions (so does qemu-user). qemu emulates a full System-z machine, at least enough to boot and run Linux. Nested virtualization would indicate something to do with KVM, but that is never possible when the guest arch != host arch. In the past support for both POWER and s390x has varied between what Fedora (especially the kernel) requires and what qemu can emulate. I haven't actually tried it that recently, so I'm not sure if it works right now. Generally if it boots it will usually work very reliably. But when it breaks it can be painful to try to discover what new feature is missing or what qemu command line option is required to enable/disable some machine feature to get it to work. So I guess yes, it's the way to go, and it'll often work until it doesn't and then you'll have some debugging on your hands. Rich. -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-top is 'top' for virtual machines. Tiny program with many powerful monitoring features, net stats, disk stats, logging, etc. http://people.redhat.com/~rjones/virt-top ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Can we use emulation of other architectures to run integration tests?
Aleksandra Fedorova writes: > As COPR has recently got support for s390 builds, the question is: if > emulation is good enough for building packages, can we use it for > testing? What are the limitations there? Is it worth it? Cross-architecture emulation is unbelievably slow in the general case. While it helps for some specific use cases, it's not a substitute for actually getting hardware. Thanks, --Robbie signature.asc Description: PGP signature ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Can we use emulation of other architectures to run integration tests?
Hi Aleksandra, On Thu, 2020-07-30 at 12:24 +0200, Aleksandra Fedorova wrote: > I'd like to get some understanding on the current state of emulation > of other architectures. > > In the current CI infra we have infinite(*) access to x86_64 compute > resources, but we haven't yet got our hands on any non x86_64 > hardware. > > As COPR has recently got support for s390 builds, the question is: if > emulation is good enough for building packages, can we use it for > testing? What are the limitations there? Is it worth it? > > If yes, can you suggest some links, tips on the topic? I tried the copr emulation of arm32 and s390x with the valgrind package. But although it seems to be able to produce binaries it was unable to run the tests. Which makes it hard to know whether the produced binaries are OK. And they make the build fail because we do require at least some tests to succeed in %check. Cheers, Mark ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Can we use emulation of other architectures to run integration tests?
On Thu, 2020-07-30 at 13:48 +0200, Nikos Mavrogiannopoulos wrote: > On Thu, Jul 30, 2020 at 12:25 PM Aleksandra Fedorova > wrote: > > Hi, all, > > > > I'd like to get some understanding on the current state of emulation > > of other architectures. > > > > In the current CI infra we have infinite(*) access to x86_64 compute > > resources, but we haven't yet got our hands on any non x86_64 > > hardware. > > > > As COPR has recently got support for s390 builds, the question is: if > > emulation is good enough for building packages, can we use it for > > testing? What are the limitations there? Is it worth it? > > Few years ago we transformed the gnutls' upstream CI from baremetal > h/w to qemu-user [0] (reasoning was pretty much what you mention, we > had x86-64 systems for free, and we had to pay for everything else). > This eliminated the need for such dedicated hardware, and in practice > the years it was in use I believe it eliminated issues in > compatibility with non-x86-64 architectures and also helped catch > problems in new code (such as alignment issues). For an upstream test > suite it was totally worth it, and I believe it eliminated all issues > we were getting with non-x86 hardware support. The only drawback that > was noticed is that it could not be used to test some special features > of these CPUs, but that's also a problem with dedicated hardware > (e.g., when it doesn't support the particular instruction set you'd > like to introduce). We use it heavily in the GCC upstream tester for ancient targets (think m68k, hppa, alpha and the like). It works amazingly well. Interestingly enough I've found those old targets to work better than, say, s390. Jeff ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Can we use emulation of other architectures to run integration tests?
Hi, Nikos, On Thu, Jul 30, 2020 at 1:48 PM Nikos Mavrogiannopoulos wrote: > > On Thu, Jul 30, 2020 at 12:25 PM Aleksandra Fedorova > wrote: > > > > Hi, all, > > > > I'd like to get some understanding on the current state of emulation > > of other architectures. > > > > In the current CI infra we have infinite(*) access to x86_64 compute > > resources, but we haven't yet got our hands on any non x86_64 > > hardware. > > > > As COPR has recently got support for s390 builds, the question is: if > > emulation is good enough for building packages, can we use it for > > testing? What are the limitations there? Is it worth it? > > Few years ago we transformed the gnutls' upstream CI from baremetal > h/w to qemu-user [0] (reasoning was pretty much what you mention, we > had x86-64 systems for free, and we had to pay for everything else). > This eliminated the need for such dedicated hardware, and in practice > the years it was in use I believe it eliminated issues in > compatibility with non-x86-64 architectures and also helped catch > problems in new code (such as alignment issues). For an upstream test > suite it was totally worth it, and I believe it eliminated all issues > we were getting with non-x86 hardware support. The only drawback that > was noticed is that it could not be used to test some special features > of these CPUs, but that's also a problem with dedicated hardware > (e.g., when it doesn't support the particular instruction set you'd > like to introduce). Thanks for the example. If I understand correctly in the emulated environment you run _build_ of the gnutls. Do you also run any arch-specific integration tests? Basically I am trying to find value we could bring on top of the existing s390x-scratch builds, which are already done by Koji. > regards, > Nikos > > [0]. https://gitlab.com/gnutls/gnutls/-/blob/master/.gitlab-ci.yml#L718 > ___ > CI mailing list -- c...@lists.fedoraproject.org > To unsubscribe send an email to ci-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/c...@lists.fedoraproject.org -- Aleksandra Fedorova bookwar ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Can we use emulation of other architectures to run integration tests?
On 30. 07. 20 12:24, Aleksandra Fedorova wrote: As COPR has recently got support for s390 builds, the question is: if emulation is good enough for building packages, can we use it for testing? What are the limitations there? Is it worth it? I don't have that many experience with this but every time I've attempted to build some Python package via mock's forcearch emulation on s390x, I got some segfaults and gave up. -- Miro Hrončok -- Phone: +420777974800 IRC: mhroncok ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Can we use emulation of other architectures to run integration tests?
Hi, Rich, On Thu, Jul 30, 2020 at 1:10 PM Richard W.M. Jones wrote: > > On Thu, Jul 30, 2020 at 12:24:19PM +0200, Aleksandra Fedorova wrote: > > Hi, all, > > > > I'd like to get some understanding on the current state of emulation > > of other architectures. > > > > In the current CI infra we have infinite(*) access to x86_64 compute > > resources, but we haven't yet got our hands on any non x86_64 > > hardware. > > > > As COPR has recently got support for s390 builds, the question is: if > > emulation is good enough for building packages, can we use it for > > testing? What are the limitations there? Is it worth it? > > > > If yes, can you suggest some links, tips on the topic? > > To be clear, are we talking qemu-system-X, qemu-user, or something > else entirely? Are you planning to do the tests in a full VM (eg. in > qemu-system-X)? Thank you for the question. I think I am actually interested in both. The scenario we use for current x86 dist-git tests is based on vms: we spin up a full vm with qemu-kvm, ssh inside it, install a package and run the test script. Should it be possible to do the same with qemu-system-s390x? Maybe even with nested virtualization, with s390x inside x86? > Rich. > > > > > (*) Not really, and some of our infra services are not yet good at > > scaling up, but we are getting better at it. > > > > -- > > Aleksandra Fedorova > > bookwar > > ___ > > devel mailing list -- devel@lists.fedoraproject.org > > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > > Fedora Code of Conduct: > > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > > List Archives: > > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org -- Aleksandra Fedorova bookwar ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Can we use emulation of other architectures to run integration tests?
* Daniel P. Berrangé: >> For emulating 32-bit targets, we have a broken readdir/telldir/seekdir >> implementation in glibc on 64 bit host kernels because we try to use >> d_ino directly, which is 64 bit and does not fit into the long value recte: d_off >> that POSIX requires. A kernel patch with a new interface has been >> posted which would work around this has been proposed, but it is not >> going anywhere. > >> The second issue also affects full-system virtualization if p9fs (not >> sure what the right name is, it's the older pass-through file system) is >> used. But it's specific to 32 bit, so maybe not that important after >> all. > > Yep, 9p, its generally not great POSIX emulation no matter what. > > The more modern virtio-fs is a more promising solution for the > future, based on FUSE. The second problem is not a matter of old vs modern interfaces, it's that POSIX requires us to fit the 64-bit d_off value we get from the kernel into a 32-bit long, because that's what telldir returns. This is true even for builds with 64-bit off_t values. The kernel cannot really fix this on our behalf because it would have to perform a side-translation for every directory entry. That's problematic for large directories. But in glibc, we know when we need the translation (when the application calls telldir), and we can defer the translation until then. It's just a little bit awkward to implement because telldir must not fail, so we need to preallocate space for the translation during the readdir call. But it's not impossible. Thanks, Florian ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Can we use emulation of other architectures to run integration tests?
On Thu, Jul 30, 2020 at 01:38:41PM +0200, Florian Weimer wrote: > * Daniel P. Berrangé: > > > I'm not familiar with what COPR is doing for s39x0 ? Is it using the > > simple QEMU linux-user syscall emulation, or is it running a proper > > QEMU s390x VM. > > > > I'm guessing probably the former. The linux-user syscall emulation is > > truely amazing, but it is certainly not feature complete or fully > > robust. > > It still implements vfork using fork, right? This means we should > likely fix posix_spawn in glibc to support this deviation from the > kernel interface. Other Linux emulations have exactly the same problem. Yeah, all of fork(), vfork() and clone(), end up getting mapped onto fork() or pthread_create() depending the precise set of CLONE_* flags required. Common helper for all: https://gitlab.com/qemu-project/qemu/-/blob/master/linux-user/syscall.c#L172 https://gitlab.com/qemu-project/qemu/-/blob/master/linux-user/syscall.c#L6086 And then the syscall wiring vfork: https://gitlab.com/qemu-project/qemu/-/blob/master/linux-user/syscall.c#L10792 fork: https://gitlab.com/qemu-project/qemu/-/blob/master/linux-user/syscall.c#L7882 clone: https://gitlab.com/qemu-project/qemu/-/blob/master/linux-user/syscall.c#L9792 there's no clone2/clone3 support at all either. > For emulating 32-bit targets, we have a broken readdir/telldir/seekdir > implementation in glibc on 64 bit host kernels because we try to use > d_ino directly, which is 64 bit and does not fit into the long value > that POSIX requires. A kernel patch with a new interface has been > posted which would work around this has been proposed, but it is not > going anywhere. > The second issue also affects full-system virtualization if p9fs (not > sure what the right name is, it's the older pass-through file system) is > used. But it's specific to 32 bit, so maybe not that important after > all. Yep, 9p, its generally not great POSIX emulation no matter what. The more modern virtio-fs is a more promising solution for the future, based on FUSE. Regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :| ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Can we use emulation of other architectures to run integration tests?
On Thu, Jul 30, 2020 at 12:25 PM Aleksandra Fedorova wrote: > > Hi, all, > > I'd like to get some understanding on the current state of emulation > of other architectures. > > In the current CI infra we have infinite(*) access to x86_64 compute > resources, but we haven't yet got our hands on any non x86_64 > hardware. > > As COPR has recently got support for s390 builds, the question is: if > emulation is good enough for building packages, can we use it for > testing? What are the limitations there? Is it worth it? Few years ago we transformed the gnutls' upstream CI from baremetal h/w to qemu-user [0] (reasoning was pretty much what you mention, we had x86-64 systems for free, and we had to pay for everything else). This eliminated the need for such dedicated hardware, and in practice the years it was in use I believe it eliminated issues in compatibility with non-x86-64 architectures and also helped catch problems in new code (such as alignment issues). For an upstream test suite it was totally worth it, and I believe it eliminated all issues we were getting with non-x86 hardware support. The only drawback that was noticed is that it could not be used to test some special features of these CPUs, but that's also a problem with dedicated hardware (e.g., when it doesn't support the particular instruction set you'd like to introduce). regards, Nikos [0]. https://gitlab.com/gnutls/gnutls/-/blob/master/.gitlab-ci.yml#L718 ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Can we use emulation of other architectures to run integration tests?
On Thursday, July 30, 2020 1:28:49 PM CEST Daniel P. Berrangé wrote: > On Thu, Jul 30, 2020 at 12:24:19PM +0200, Aleksandra Fedorova wrote: > > Hi, all, > > > > I'd like to get some understanding on the current state of emulation > > of other architectures. > > > > In the current CI infra we have infinite(*) access to x86_64 compute > > resources, but we haven't yet got our hands on any non x86_64 > > hardware. > > > > As COPR has recently got support for s390 builds, the question is: if > > emulation is good enough for building packages, can we use it for > > testing? What are the limitations there? Is it worth it? > > I'm not familiar with what COPR is doing for s39x0 ? Is it using the > simple QEMU linux-user syscall emulation, or is it running a proper > QEMU s390x VM. > > I'm guessing probably the former. The linux-user syscall emulation is > truely amazing, but it is certainly not feature complete or fully > robust. Yes, it is just running `mock -r fedora--s390x` on x86_64 VM. That uses dnf's --forcearch feature (qemu-user-static emulation). Pavel > Applications that are multi-threaded are especially likely to expose > bugs and/or design limitations that are often hard to fix. If the > app uses common/simple syscalls it'll probably be OK in general; if it > uses more obscure stuff it'll come up against bugs / missing features. > It doesn't emulate ptrace, or robust futexes, and doesn't separate > rlimits for the emulation layer from the application lkayer. QEMU > isn't especially prompt about adding support for newly exposed kernel > syscalls. The quality can also vary depending on what target > architecture is involved. > > Overall it is pretty decent at being able to run common toolchains for > performing builds. Once you get into running a broader class of > applications, things can get more hairy. > > Basically if it works for an given app, that is great, but don't be > too surprised if apps experiance odd behaviour or hits bugs. > > I certainly won't expect to be able to blindly throw any Fedora package > at it and expect to run arbitrary integration tests with the same level > of quality you'd experiance with bare metal for that particular arch. > > The QEMU VM emulation will get a more reliable environment for integration > testing, since that's just emulating the CPU + hardware, with Linux still > providing the syscall ABI. The cost is that VM is higher overhead than > linux-user. > > As something that individual packagers can opt-in to QEMU linux-user > emulation might be viable. Packagers would just have to try it and see > if it works well enough for their particular package's testing use cases. > I'm not sure if that makes it compelling enough to invest time into > though from a Fedora infra POV. > > Regards, > Daniel > -- > |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| > |: https://libvirt.org -o-https://fstop138.berrange.com :| > |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :| > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org > ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Can we use emulation of other architectures to run integration tests?
* Daniel P. Berrangé: > I'm not familiar with what COPR is doing for s39x0 ? Is it using the > simple QEMU linux-user syscall emulation, or is it running a proper > QEMU s390x VM. > > I'm guessing probably the former. The linux-user syscall emulation is > truely amazing, but it is certainly not feature complete or fully > robust. It still implements vfork using fork, right? This means we should likely fix posix_spawn in glibc to support this deviation from the kernel interface. Other Linux emulations have exactly the same problem. For emulating 32-bit targets, we have a broken readdir/telldir/seekdir implementation in glibc on 64 bit host kernels because we try to use d_ino directly, which is 64 bit and does not fit into the long value that POSIX requires. A kernel patch with a new interface has been posted which would work around this has been proposed, but it is not going anywhere. The second issue also affects full-system virtualization if p9fs (not sure what the right name is, it's the older pass-through file system) is used. But it's specific to 32 bit, so maybe not that important after all. Thanks, Florian ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Can we use emulation of other architectures to run integration tests?
On Thu, Jul 30, 2020 at 12:24:19PM +0200, Aleksandra Fedorova wrote: > Hi, all, > > I'd like to get some understanding on the current state of emulation > of other architectures. > > In the current CI infra we have infinite(*) access to x86_64 compute > resources, but we haven't yet got our hands on any non x86_64 > hardware. > > As COPR has recently got support for s390 builds, the question is: if > emulation is good enough for building packages, can we use it for > testing? What are the limitations there? Is it worth it? I'm not familiar with what COPR is doing for s39x0 ? Is it using the simple QEMU linux-user syscall emulation, or is it running a proper QEMU s390x VM. I'm guessing probably the former. The linux-user syscall emulation is truely amazing, but it is certainly not feature complete or fully robust. Applications that are multi-threaded are especially likely to expose bugs and/or design limitations that are often hard to fix. If the app uses common/simple syscalls it'll probably be OK in general; if it uses more obscure stuff it'll come up against bugs / missing features. It doesn't emulate ptrace, or robust futexes, and doesn't separate rlimits for the emulation layer from the application lkayer. QEMU isn't especially prompt about adding support for newly exposed kernel syscalls. The quality can also vary depending on what target architecture is involved. Overall it is pretty decent at being able to run common toolchains for performing builds. Once you get into running a broader class of applications, things can get more hairy. Basically if it works for an given app, that is great, but don't be too surprised if apps experiance odd behaviour or hits bugs. I certainly won't expect to be able to blindly throw any Fedora package at it and expect to run arbitrary integration tests with the same level of quality you'd experiance with bare metal for that particular arch. The QEMU VM emulation will get a more reliable environment for integration testing, since that's just emulating the CPU + hardware, with Linux still providing the syscall ABI. The cost is that VM is higher overhead than linux-user. As something that individual packagers can opt-in to QEMU linux-user emulation might be viable. Packagers would just have to try it and see if it works well enough for their particular package's testing use cases. I'm not sure if that makes it compelling enough to invest time into though from a Fedora infra POV. Regards, Daniel -- |: https://berrange.com -o-https://www.flickr.com/photos/dberrange :| |: https://libvirt.org -o-https://fstop138.berrange.com :| |: https://entangle-photo.org-o-https://www.instagram.com/dberrange :| ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Re: Can we use emulation of other architectures to run integration tests?
On Thu, Jul 30, 2020 at 12:24:19PM +0200, Aleksandra Fedorova wrote: > Hi, all, > > I'd like to get some understanding on the current state of emulation > of other architectures. > > In the current CI infra we have infinite(*) access to x86_64 compute > resources, but we haven't yet got our hands on any non x86_64 > hardware. > > As COPR has recently got support for s390 builds, the question is: if > emulation is good enough for building packages, can we use it for > testing? What are the limitations there? Is it worth it? > > If yes, can you suggest some links, tips on the topic? To be clear, are we talking qemu-system-X, qemu-user, or something else entirely? Are you planning to do the tests in a full VM (eg. in qemu-system-X)? Rich. > > (*) Not really, and some of our infra services are not yet good at > scaling up, but we are getting better at it. > > -- > Aleksandra Fedorova > bookwar > ___ > devel mailing list -- devel@lists.fedoraproject.org > To unsubscribe send an email to devel-le...@lists.fedoraproject.org > Fedora Code of Conduct: > https://docs.fedoraproject.org/en-US/project/code-of-conduct/ > List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines > List Archives: > https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org -- Richard Jones, Virtualization Group, Red Hat http://people.redhat.com/~rjones Read my programming and virtualization blog: http://rwmj.wordpress.com virt-builder quickly builds VMs from scratch http://libguestfs.org/virt-builder.1.html ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
Can we use emulation of other architectures to run integration tests?
Hi, all, I'd like to get some understanding on the current state of emulation of other architectures. In the current CI infra we have infinite(*) access to x86_64 compute resources, but we haven't yet got our hands on any non x86_64 hardware. As COPR has recently got support for s390 builds, the question is: if emulation is good enough for building packages, can we use it for testing? What are the limitations there? Is it worth it? If yes, can you suggest some links, tips on the topic? (*) Not really, and some of our infra services are not yet good at scaling up, but we are getting better at it. -- Aleksandra Fedorova bookwar ___ devel mailing list -- devel@lists.fedoraproject.org To unsubscribe send an email to devel-le...@lists.fedoraproject.org Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/ List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines List Archives: https://lists.fedoraproject.org/archives/list/devel@lists.fedoraproject.org
