Hi, Can the downloads be turned into normal do_fetch() SRC_URI downloads including caches in yocto infrastructure?
There are many issues around CVE checking that it's really a process where a lot of details and uncertainties just need to be accepted. It's far from a perfect and users just need to accept this. Cheers, -Mikko On Wed, Oct 12, 2022 at 04:54:57PM +0900, Marta Rybczynska wrote: > I'll be looking into how to fix it. My current idea is to make the download > a transaction, so do not update the database until we're sure the download > is complete. Plus a warning that we have had an issue, I think. > > In a next step we can retry a number of times. > > Regards > Marta > > On Wed, 12 Oct 2022, 16:50 Alexander Kanavin, <[email protected]> > wrote: > > > Thanks for the information, can you send a patch? > > > > Alex > > > > On Wed, 12 Oct 2022 at 09:25, Alberto Pianon <[email protected]> wrote: > > > > > > All, Marta, Richard, > > > > > > while implementing stats aggregation for CVE metadata in the Oniro > > > project, I encountered a severe issue in Yocto's CVE checker, apparently > > > due to this: > > > > > https://git.yoctoproject.org/poky/tree/meta/recipes-core/meta/cve-update-db-native.bb#n81 > > > > > > It appears that when cve-update-db-native fails to fetch some years of > > > NIST CVE db, it just issues a warning but goes on anyway. > > > > > > The result is that in some builds, randomly (depending on NIST webserver > > > timeouts or other connection problems), CVE db is not complete, so the > > > CVE check returns false negatives (i.e. no vulnerabilities found in some > > > components even if such vulnerabilities do exist) > > > > > > I ran into such problem because in Oniro we need aggregate data from > > > different builds for a large target matrix; I added a check to check > > > that CVE metadata for each component are the same in all builds, and it > > > failed, so I tried to figure out the cause and I found this: > > > > > > - in a build where cve-check found a vulnerability for acl: > > > $ sqlite3 build-linux-clang-qemuarm-efi/tmp/CVE_CHECK/nvdcve_1.1.db > > > sqlite> SELECT DISTINCT ID FROM PRODUCTS WHERE PRODUCT IS "acl"; > > > CVE-2009-4411 > > > sqlite> > > > > > > - in another build where cve-check did not found any vulnerability for > > > the very same version of acl: > > > $ sqlite3 build-linux-clang-qemux86/tmp/CVE_CHECK/nvdcve_1.1.db > > > sqlite> SELECT DISTINCT ID FROM PRODUCTS WHERE PRODUCT IS "acl"; > > > sqlite> > > > > > > so I listed both CVE db files in those two builds and this is what I > > > got: > > > > > > $ ls -ll build-linux-clang-qemuarm-efi/tmp/CVE_CHECK/nvdcve_1.1.db > > > build-linux-clang-qemux86/tmp/CVE_CHECK/nvdcve_1.1.db > > > -rw-r--r-- 1 ubuntu ubuntu 215093248 Oct 11 10:56 > > > build-linux-clang-qemuarm-efi/tmp/CVE_CHECK/nvdcve_1.1.db > > > -rw-r--r-- 1 ubuntu ubuntu 28672 Oct 11 10:00 > > > build-linux-clang-qemux86/tmp/CVE_CHECK/nvdcve_1.1.db > > > > > > The two CVE db files were fetched just about 1h apart, but the second > > > file is apparently incomplete. > > > > > > I checked the log for the second build, and I found this: > > > > > > WARNING: cve-update-db-native-1.0-r0 do_fetch: Failed to fetch CVE > > > data ([Errno 99] Cannot assign requested address) > > > NOTE: recipe cve-update-db-native-1.0-r0: task do_fetch: Succeeded > > > > > > Fetch fails, but do_fetch task succeeds. > > > > > > So I looked into the recipe's code, and I found this: > > > > > https://git.yoctoproject.org/poky/tree/meta/recipes-core/meta/cve-update-db-native.bb#n81 > > > > > > It iterates over NIST CVE db years, but if some year fail to download, > > > it goes on anyway, and it still merges the successful downloads into > > > nvdcve_1.1.db, without returning error. > > > > > > IMHO this is a severe issue because it may silently lead to false > > > negatives in the CVE check. If some downloads fail due to timeouts or > > > other connection problems, cve-check should retry a number of times, and > > > if any download still fails, cve-update-db-native do_fecth should fail, > > > and it turn all do_cve_check tasks should fail, since doing a CVE check > > > against a corrupted/incomplete CVE database is clearly useless > > > > > > Regards, > > > > > > Alberto > > > > > > > > > > > > > > > > > > > >
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#171659): https://lists.openembedded.org/g/openembedded-core/message/171659 Mute This Topic: https://lists.openembedded.org/mt/94276393/21656 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
