> -----Original Message----- > From: Michael Halstead <[email protected]> > Sent: Tuesday, November 11, 2025 6:31 > To: Marko, Peter (FT D EU SK BFS1) <[email protected]> > Cc: Marta Rybczynska <[email protected]>; Richard Purdie > <[email protected]>; [email protected]; openembedded- > [email protected]; [email protected] > Subject: Re: [OE-core] [PATCH 5/5] cve-update: Keep mtime stamp in the > database itself > > On Mon, Nov 10, 2025 at 7:31 AM Peter Marko via lists.openembedded.org > <[email protected]> wrote: > > > > > > > > > -----Original Message----- > > > From: Marta Rybczynska <[email protected]> > > > Sent: Monday, November 10, 2025 16:29 > > > To: Marko, Peter (FT D EU SK BFS1) <[email protected]> > > > Cc: Richard Purdie <[email protected]>; > > > [email protected]; [email protected]; > > > [email protected] > > > Subject: Re: [OE-core] [PATCH 5/5] cve-update: Keep mtime stamp in the > > > database itself > > > > > > > > > > > > On Mon, Nov 10, 2025 at 4:18 PM Marko, Peter <[email protected] > > > <mailto:[email protected]> > wrote: > > > > > > > > > > > > > > > > -----Original Message----- > > > > From: Marta Rybczynska <[email protected] > > > <mailto:[email protected]> > > > > > Sent: Monday, November 10, 2025 16:00 > > > > To: Richard Purdie <[email protected] > > > <mailto:[email protected]> > > > > > Cc: [email protected] <mailto:[email protected]> ; > > > [email protected] <mailto:openembedded- > > > [email protected]> ; > > > > [email protected] <mailto:[email protected]> ; Marko, Peter > > > (FT D EU SK BFS1) > > > > <[email protected] <mailto:[email protected]> > > > > > Subject: Re: [OE-core] [PATCH 5/5] cve-update: Keep mtime stamp in > > > the > > > > database itself > > > > > > > > > > > > > > > > On Mon, Nov 10, 2025 at 3:16 PM Richard Purdie > > > > <[email protected] > > > <mailto:[email protected]> > > > <mailto:[email protected] > > > <mailto:[email protected]> > > > > > > wrote: > > > > > > > > > > > > On Mon, 2025-11-03 at 14:31 +0000, Niko Mauno via > > > > lists.openembedded.org <http://lists.openembedded.org> > > > <http://lists.openembedded.org> wrote: > > > > > This should help to avoid problems that will occur if the > > > modification > > > > > time of database file itself is altered e.g. by > > > unassociated > > > > > process(es) on the file system which hosts the database > > > file. > > > > > > > > > > Since this change updates the database structure by > > > adding a new > > > table, > > > > > bump the 'minor' version number in database file names to > > > enforce > > > full > > > > > database fetch. This should also iron out e.g. situation > > > where the > > > > > database might have inconspicuously omitted entries due > > > to way > > > in > > > > which > > > > > the mtime of database file itself was relied upon. > > > > > > > > > > Signed-off-by: Niko Mauno <[email protected] > > > <mailto:[email protected]> > > > > <mailto:[email protected] <mailto:[email protected]> > > > > > > > > > > --- > > > > > meta/classes/cve-check.bbclass | 2 +- > > > > > .../recipes-core/meta/cve-update-db-native.bb > > > <http://cve-update- > > > db-native.bb> <http://cve-update-db- > > > > native.bb <http://native.bb> > | 3 + > > > > > meta/recipes-core/meta/cve-update-native.inc | 59 > > > ++++++++++++------- > > > > > .../meta/cve-update-nvd2-native.bb > > > <http://cve-update-nvd2- > > > native.bb> <http://cve-update-nvd2-native.bb> > > > > | 3 + > > > > > 4 files changed, 46 insertions(+), 21 deletions(-) > > > > > > > > I am a bit worried about this since it takes what is a > > > simple mtime > > > > comparison and means that to check the database you now > > > have to > > > lock > > > > and open it which is comparatively expensive. > > > > > > > > What kind of "unassociated processes" are updating the > > > mtime? > > > > > > > > > > > > > > > > > > > > I second Richard here. If we would really want to keep the date > > > somewhere else > > > > than the file > > > > timestamp, we could create a simple file that stores it, what is > > > way > > > simpler and > > > > doesn't require > > > > to open the database just to see if you want to update it. > > > > > > > > I second Richard's question: in what kind of a situation have you > > > seen an > > > issue? > > > > > > The issue is on Yocto AB. > > > It has not updated statistics for more than 5 months now. > > > > > > Looking at the logs, something is updating mtime very frequently... > > > I don't know the AB infrastructure, but it looks like there is some > > > filesystem > > > crawler, maybe backup of sync process. > > On 2024-12-26 we set workers in the > autobulder.yoctoproject.org/valkyrie cluster to run "touch > /srv/autobuilder/valkyrie.yocto.io/current_sources/CVE_CHECK/*.db > /srv/autobuilder/valkyrie.yocto.io/current_sources/CVE_CHECK2/*.db" > every 3 minute to ensure no stale NFS cache existed for these files.
This explains a lot. Touching file while we rely on mtime is obviously breaking cve-check code. Note that stopping the touch service is not enough, at least not for LTS branches, because NVD2 code will update new CVEs only since the mtime timestamp. Could you also delete the databases now so it can start fresh without 5 months of missing data? Peter > That job was disabled for all builders on 2025-09-02. The performance > testing workers were still running the job though. I've disabled it on > those now. > > The issue of stale cached NFS data may return now. Something to keep an eye > on. > > If we want to keep the timestamp in a separate file I suggest creating > a new file named timestamp_$(date +%s) for each timestamp and deleting > previous timestamp_ files. That should avoid most caching issues and > keep overhead minimal. > > > > Or the nfs mount is plain broken there? > > > > > > Example is here: > > > https://valkyrie.yocto.io/pub/non-release/20251109- > > > 43/testresults/metrics/db-update/ > > > NOTE: Current time: Sun Nov 9 07:17:52 2025; DB time: Sun Nov 9 > > > 07:14:51 2025 > > > NOTE: Current time: Sun Nov 9 07:19:22 2025; DB time: Sun Nov 9 > > > 07:18:14 2025 > > > 2 jobs are 90 seconds from each other, but something managed to > > > change > > > the timestamp in between and only 3.5 minutes after last change! > > > > > > > > > > > > > > > The autobuilder was expected to force the complete download every day. > > > Something > > > has changed? The code does open the database after checking the > timestamp, so > > > frequent jobs couldn't cause it. I think that machine requires some > > > monitoring > to > > > see > > > what is actually changing those timestamps, because we do not know what it > is > > > also > > > changing. > > > > 1 day doesn't beat the 3.5 minutes crawler... > > This is probably AB issue only. > > > > And maybe other example: > > https://valkyrie.yocto.io/pub/non-release/20251110-45/testresults/metrics/db- > update/ > > NOTE: Current time: Mon Nov 10 07:17:47 2025; DB time: Mon Nov 10 07:16:33 > 2025 > > NOTE: Current time: Mon Nov 10 07:19:17 2025; DB time: Mon Nov 10 07:16:33 > 2025 > > > > This shows that the first job does not modify it (at least not in the first > > 90 > seconds). > > I guess it fit the 3.5 minutes window of opportunity. > > And my local build does not show any timestamp manipulation either. > > > > Peter > > > > > > > > I haven't seen such an effect on my systems. > > > > > > Kind regards, > > > Marta > > > > > > > > > > > -- > Michael Halstead > Linux Foundation / Yocto Project > Systems Operations Engineer
-=-=-=-=-=-=-=-=-=-=-=- Links: You receive all messages sent to this group. View/Reply Online (#226148): https://lists.openembedded.org/g/openembedded-core/message/226148 Mute This Topic: https://lists.openembedded.org/mt/116098343/21656 Group Owner: [email protected] Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub [[email protected]] -=-=-=-=-=-=-=-=-=-=-=-
