On Thu Jun 25, 2026 at 2:29 PM CEST, Awais Belal wrote:
>> I was questionning why CVE_DB_RESULTS_PER_PAGE*_MAX* (notice the _MAX)
>> should be. Seems to me that this is a limit imposed by NVD itself and I
>> don't see a reason to set it to a different value than 2000 (as given by
>> NVD).
>
> Ah, sorry for misreading your comment earlier. I went with a configurable
> max for flexibility purposes as well. Consider a scenario where someone
> would want to limit the max as well for whatever reason. Even in this
> scenario if the NVD limits change all we'd need to do is update a single
> variable rather than having to change multiple locations, we could also use
> a local var for that but then it won't cover point 1. Still if you think
> this is over engineered please let me know.
I still think this is over engineered. I can't imagine a reason in
"whatever reason".
To be clear, what I suggest is:
RESULTS_PER_PAGE_MAX = 2000 # imposed by NVD
if results_per_page > RESULTS_PER_PAGE_MAX:
...
(2000 only writen once, no "multiple locations" to update)
>
> On Thu, Jun 25, 2026 at 5:22 PM Yoann Congal <[email protected]> wrote:
>
>> On Thu Jun 25, 2026 at 1:54 PM CEST, Awais Belal wrote:
>> > Hi Yoann,
>> >
>> > Thanks for picking this up
>> >
>> >> What value are you using? Especially in the last few days...
>> >
>> > I've seen much better results with a cap of 500/1000 than the default of
>> > 2000.
>> >
>> >> I'm not sure CVE_DB_RESULTS_PER_PAGE_MAX should be a bitbake variable (I
>> >> don't see a reason for it to be configurable). Can you simplify the code
>> >> a little by making it a constant in the code instead? We already have a
>> >> handfull of hardcoded limits from NVD.
>> >
>> > Well it is a valid request parameter for starters. This allows the users
>> to
>> > control it from the configurations rather than hard coding to any limits
>> > someone sees fitting for their environment so more flexibility. Not
>> setting
>> > the variable defaults to the limits recommended by NVD however my
>> analysis
>> > shows that these days due to bulk updates on the NVD side the fetch has
>> > failed most of the time due to an "incomplete read" error.
>>
>> I agree that a configurable CVE_DB_RESULTS_PER_PAGE looks like a good idea.
>>
>> I was questionning why CVE_DB_RESULTS_PER_PAGE*_MAX* (notice the _MAX)
>> should be. Seems to me that this is a limit imposed by NVD itself and I
>> don't see a reason to set it to a different value than 2000 (as given by
>> NVD).
>>
>> >
>> > On Thu, Jun 25, 2026 at 4:43 PM Yoann Congal <[email protected]>
>> wrote:
>> >
>> >> Hello,
>> >>
>> >> Thanks for looking at this!
>> >>
>> >> On Thu Jun 25, 2026 at 1:15 PM CEST, Awais Belal via
>> >> lists.openembedded.org wrote:
>> >> > From: Awais B <[email protected]>
>> >> >
>> >> > It is seen that during bulk updates on the NVD side the server
>> >> > struggles to keep up with the default/max of 2000 entries per
>> >> > page and we see a lot of incomplete read errors resulting in
>> >> > proper db sync failures most of the times. Lowering the per
>> >> > page value noticably increases the reliability of the process
>> >> > and hence should ideally be configurable.
>> >>
>> >> What value are you using? Especially in the last few days...
>> >>
>> >> >
>> >> > Signed-off-by: Awais B <[email protected]>
>> >> > ---
>> >> > meta/recipes-core/meta/cve-update-nvd2-native.bb | 14 ++++++++++++++
>> >> > 1 file changed, 14 insertions(+)
>> >> >
>> >> > diff --git a/meta/recipes-core/meta/cve-update-nvd2-native.bb
>> >> b/meta/recipes-core/meta/cve-update-nvd2-native.bb
>> >> > index 945bd1d927..5d8b76e62c 100644
>> >> > --- a/meta/recipes-core/meta/cve-update-nvd2-native.bb
>> >> > +++ b/meta/recipes-core/meta/cve-update-nvd2-native.bb
>> >> > @@ -34,6 +34,11 @@ CVE_DB_INCR_UPDATE_AGE_THRES ?= "10368000"
>> >> > # Number of attempts for each http query to nvd server before giving
>> up
>> >> > CVE_DB_UPDATE_ATTEMPTS ?= "5"
>> >> >
>> >> > +# Maximum number of CVE records per API response. Default/max is
>> 2000.
>> >> > +# Lowering this value can help avoid incomplete read errors during
>> bulk
>> >> NVD updates.
>> >> > +CVE_DB_RESULTS_PER_PAGE ?= ""
>> >> > +CVE_DB_RESULTS_PER_PAGE_MAX ?= "2000"
>> >> > +
>> >> > CVE_CHECK_DB_DLDIR_FILE ?=
>> >> "${DL_DIR}/CVE_CHECK/${CVE_CHECK_DB_FILENAME}"
>> >> > CVE_CHECK_DB_DLDIR_LOCK ?= "${CVE_CHECK_DB_DLDIR_FILE}.lock"
>> >> > CVE_CHECK_DB_TEMP_FILE ?= "${CVE_CHECK_DB_FILE}.tmp"
>> >> > @@ -217,6 +222,15 @@ def update_db_file(db_tmp_file, d,
>> database_time):
>> >> > api_key = d.getVar("NVDCVE_API_KEY") or None
>> >> > attempts = int(d.getVar("CVE_DB_UPDATE_ATTEMPTS"))
>> >> >
>> >> > + results_per_page = d.getVar("CVE_DB_RESULTS_PER_PAGE")
>> >> > + results_per_page_max =
>> >> int(d.getVar("CVE_DB_RESULTS_PER_PAGE_MAX"))
>> >>
>> >> I'm not sure CVE_DB_RESULTS_PER_PAGE_MAX should be a bitbake variable (I
>> >> don't see a reason for it to be configurable). Can you simplify the code
>> >> a little by making it a constant in the code instead? We already have a
>> >> handfull of hardcoded limits from NVD.
>> >>
>> >> > + if results_per_page:
>> >> > + results_per_page = int(results_per_page)
>> >> > + if results_per_page > results_per_page_max:
>> >> > + bb.warn("CVE_DB_RESULTS_PER_PAGE exceeds maximum of
>> %d,
>> >> capping" % results_per_page_max)
>> >> > + results_per_page = results_per_page_max
>> >> > + req_args['resultsPerPage'] = results_per_page
>> >> > +
>> >> > # Recommended by NVD
>> >> > wait_time = 6
>> >> > if api_key:
>> >>
>> >>
>> >> --
>> >> Yoann Congal
>> >> Smile ECS
>> >>
>> >>
>>
>>
>> --
>> Yoann Congal
>> Smile ECS
>>
>>
--
Yoann Congal
Smile ECS
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#239559):
https://lists.openembedded.org/g/openembedded-core/message/239559
Mute This Topic: https://lists.openembedded.org/mt/119971554/21656
Group Owner: [email protected]
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[[email protected]]
-=-=-=-=-=-=-=-=-=-=-=-
