On 11/10/25 16:59, Marta Rybczynska wrote:
On Mon, Nov 10, 2025 at 3:16 PM Richard Purdie <[email protected]
<mailto:[email protected]>> wrote:
On Mon, 2025-11-03 at 14:31 +0000, Niko Mauno via lists.openembedded.org
<http://lists.openembedded.org/> wrote:
> This should help to avoid problems that will occur if the modification
> time of database file itself is altered e.g. by unassociated
> process(es) on the file system which hosts the database file.
>
> Since this change updates the database structure by adding a new table,
> bump the 'minor' version number in database file names to enforce full
> database fetch. This should also iron out e.g. situation where the
> database might have inconspicuously omitted entries due to way in which
> the mtime of database file itself was relied upon.
>
> Signed-off-by: Niko Mauno <[email protected]
<mailto:[email protected]>>
> ---
> meta/classes/cve-check.bbclass | 2 +-
> .../recipes-core/meta/cve-update-db-native.bb
<http://cve-update-db-native.bb/> | 3 +
> meta/recipes-core/meta/cve-update-native.inc | 59 ++++++++++++-------
> .../meta/cve-update-nvd2-native.bb <http://cve-update-nvd2-native.bb/>
| 3 +
> 4 files changed, 46 insertions(+), 21 deletions(-)
I am a bit worried about this since it takes what is a simple mtime
comparison and means that to check the database you now have to lock
and open it which is comparatively expensive.
Hello Richard,
To me it seemed that the overhead of the additional database access which is
performed once or twice during BitBake execution lifecycle is really neglible,
and in relative terms, very cheap compared to producing false negative CVE
checks.
This change also hopes to close a window of opportunity by sampling the
database mtime timestamp value before the fetch operation is performed, in
contrast to former when the file mtime is (supposedly) updated only after the
fetch has been performed and the database write completed.
What kind of "unassociated processes" are updating the mtime?
Seems this question was kindly filled in by Peter and Michael on AB behalf -- for my own
behalf, please see the response below in Marta's "seconding" question.
I second Richard here. If we would really want to keep the date somewhere else
than the file
timestamp, we could create a simple file that stores it, what is way simpler
and doesn't require
to open the database just to see if you want to update it.
Hello Marta,
I fail to see what is the real benefit in using a separate file for the
purpose, and somehow I doubt the operations involved would be that much
simpler, or provide any significant edge performance wise. With this change we
just move away from using file mtime and still keep the timestamp along in the
one and same database file.
I second Richard's question: in what kind of a situation have you seen an issue?
Personally I was bitten by the file mtime dependency about an year ago when
there were serious issues with downloading the database updates from the NVD
website. I tried to mitigate the problem by using a centralized job for
downloading the database (with very lenient settings so it wouldn't time out or
so) and then distributed the database file to several build servers. But with
this approach the mtime value was always recently updated before the database
update attempt, and it took a while for me to realize the mtime check prevented
the database update from occurring. I don't recall what was the cause of the
file mtime getting updated at that time, the database file was
compressed+decompressed as part of the flow and resided behind NFS mount.
Later I also noticed that others seemed to have encountered issues which could also be
related to use of "external" file mtime provided timestamp
- https://lists.openembedded.org/g/openembedded-core/topic/110179038#msg208869
-
https://git.yoctoproject.org/poky/commit/?id=9d2dcd798322748cfdead0db320a5242e4b2cf31
These made me wonder if there could be some less flaky alternative to the file
mtime reliance, which seemed under some circumstances to lead to inconspicuous
issues in form of false negative CVE checks due to omitted CVEs in the database.
Kind regards,
Marta
Finally, considering Marta's advise in her other reply in this thread, quoting
her comment here:
So for people who want to use the NVD data directly from the source still, I
think it makes sense to implement the download
of the JSON 2.0 feeds (should be closer to the nvd-native than nvd2-native, and
*faster*) and then decommission both nvd1
and the current nvd2 (API based).
If it helps I can e.g. rework the series such that only the changes in the topmost
"keep mtime in database itself" commit would be applied in the two recipes
independently, which would still help to avoid the current mtime check from occasionally
biting unsuspecting users.
-Niko
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#226186):
https://lists.openembedded.org/g/openembedded-core/message/226186
Mute This Topic: https://lists.openembedded.org/mt/116098343/21656
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
