On Oct 24, 2019, at 20:08, Rich Persaud <pers...@gmail.com> wrote: > > (adding the automated-testing list) > >> On Oct 24, 2019, at 18:52, Richard Purdie >> <richard.pur...@linuxfoundation.org> wrote: >> I'm posting this to openembedded-architecture since it probably is the >> best place for discussions about OE/Yocto planning. >> >> The Yocto Project TSC believes one of the things needed for YP and for >> OE is more information being pulled together about how an LTS release >> could work. The document linked to below is the information it was able >> to collate together on the subject. Its the result of some long >> discussions by the TSC. >> ... >> The key thing needed to make this happen is a resource commitment to >> it. Without such a commitment it is simply not feasible. > > Vendors who provide commercial support for OE/Yocto-based products may > already be investing downstream resources for equivalent "LTS" support. May > be useful for the document to show the financial benefits of shifting a > percentage of downstream LTS validation resources to upstream Yocto LTS. > >> There may be >> new companies willing to join the project in order to make it happen, >> if so we should talk further. >> >> The document: >> >> https://docs.google.com/document/d/1AwAFDf52f_FoXksbHEVUMlu4hpcI0JMGVG-Kj_sUkyc/ > > > The project components to be covered would need to match those included in > > our standard release process and should be clearly defined. Those > > components would be: > > Bitbake > > OE-Core > > Meta-yocto > > yocto-docs > ... > > For simplicity, it is also proposed that only automated testing be used for > > testing the LTS and that any current manual QA is not performed. > ... > > Only run virtualized tests > > Only support one host distro, an Ubuntu LTS as it has right lifespan > > When the first Yocto LTS release is available, could that become the host > distro for automated testing? That would provide Yocto automated testing > with supply chain integrity benefits from Yocto. If we take that path, with > automated testing based on virtualization, could we add meta-virtualization > to the list of Yocto LTS release components? > > Yocto-on-Yocto automated testing could reduce risks [1] that impact software > supply chains. A potential topic for the ATS event [2] next week. > > Rich > > [1] https://www.platformsecuritysummit.com/2019/speaker/sherman/ > > [2] https://elinux.org/Automated_Testing_Summit_2019
There may be lessons in Microsoft's move from diverse hardware to VM-based testing: https://www.ghacks.net/2019/09/23/former-microsoft-employee-explains-why-bugs-in-windows-updates-increased/ > Back in 2014/2015, Microsoft employed an entire team that was dedicated to > testing the operating system, builds, updates, drivers, and other code. The > team consisted of multiple groups that would run tests and discuss bugs and > issues in daily meetings. Tests were conducted manually by the team and > through automated testing, and if tests were passed, would give the okay to > integrate the code into Windows. > > The teams ran the tests on "real" hardware in a lab through automated > testing. The machines had different hardware components, e.g. processors, > hard drives, video and sound cards, and other components to cover a wide > range of system configurations, and this meant that bugs that affected only > certain hardware components or configurations were detected in the process. > > Microsoft laid off almost the entire Windows Test team as it moved the focus > from three different systems -- Windows, Windows Mobile and Xbox -- to a > single system. The company moved most of the testing to virtual machines and > this meant according to Berg that tests were no longer conducted on real and > diverse hardware configurations for the most part. Business requirements for "LTS releases" include security fixes and non-regression of production systems, i.e. the software supply chain security topics discussed in [1] above. There were CKI [3] discussions about sharing test resources (human, machine) for kernel testing. Some CKI challenges and potential solutions [4] may be applicable to Yocto LTS testing of packages beyond the Linux kernel. VM-based testing could be supplemented with pooled test results from distributed testing on diverse hardware, across the Yocto contributor community. Even with VM-based testing, IOMMU PCI passthrough can be used to give a VM direct access to a PCI device, for device driver testing. With firmware attacks, the behavior of hardware-under-test could be altered between test runs, which may invalidate test results and certification claims that identify a DUT. There are "boot integrity" projects with tamper detection and firmware measurements on TPM-enabled hardware. These hashes can be recorded [5] alongside test results, when using an OS with build and boot integrity on test and logging systems. Rich [3] https://cki-project.org/ [4] https://docs.google.com/document/d/1EIU-GEJpChfB2TLzi3ebXQqUnXQ1CQ2gyl48FE-dfQI [5] https://platformsecuritysummit.com/2019/speaker/loucaides
_______________________________________________ Openembedded-architecture mailing list Openembedded-architecture@lists.openembedded.org http://lists.openembedded.org/mailman/listinfo/openembedded-architecture
