On 2023/12/12 11:54, Yuta Hayama via lists.openembedded.org wrote:
> I think you are right that HTTP 403 does not look like reaching the rate limit
> (As Yoann noted, 429 would feel right). The HTTP 403 error returned may be
> because the request is being sent from a single PC, so I will try
> cve-update-nvd2-native:do_fetch on three different PCs. This would be closer
> to the actual use case I have indicated.
I have been trying to test running cve-update-nvd2-native:do_fetch on three
PCs at the same time. The result was that only one machine continued to make
successful requests until do_fetch completed, while the other two failed with
<urlopen error [Errno 101] Network is unreachable>.
It seems that only one person can fetch in my assumed situation, no matter how
much we try to adjust the delay time...
>>>> As a specific example, multiple cve-update-nvd2-native:do_fetch runs
>>>> in parallel can easily reach the rate limit. It can be assumed that
>>>> this situation will occur if several people run bitbake in one office.
>>>> (often unaware of each
>>>> other...)
Also, the following was my misunderstanding.
> Perhaps someone has not read the documentation
> about rate limiting, so the server is putting a delay before returning a
> response.
> That is, I think a single bitbake will not cause the issue.
The reason it takes a few seconds for the server to respond is probably that
the response is too long and the server is struggling.
In fact, if we send requests repeatedly that shorten the response, the response
will come back in a relatively short time, eventually reaching the rate limit.
And the HTTP error code at that time seems to be 403 for some reason. This is
the operation that clearly causes the rate limit to be reached, but it still
does not look like it will be 429.
Below is an example of test code:
-------------------------------------------------------------------------------
import urllib.request
import time
cves = \
['CVE-2019-14899',
'CVE-2021-3714',
'CVE-2021-3864',
'CVE-2022-0400',
'CVE-2022-1247',
'CVE-2022-3219',
'CVE-2022-36402',
'CVE-2022-38096',
'CVE-2022-4543',
'CVE-2022-46456',
'CVE-2023-0687',
'CVE-2023-1386',
'CVE-2023-25584',
'CVE-2023-3019',
'CVE-2023-3397',
'CVE-2023-3640',
'CVE-2023-38559',
'CVE-2023-40030',
'CVE-2023-4010',
'CVE-2023-4039',
'CVE-2023-42363',
'CVE-2023-42364',
'CVE-2023-42365',
'CVE-2023-42366',
'CVE-2023-46407',
'CVE-2023-47100',
'CVE-2023-49292',
'CVE-2023-5088',
'CVE-2023-5156',
'CVE-2023-6238',]
for cve in cves:
url = 'https://services.nvd.nist.gov/rest/json/cves/2.0?cveId=' + cve
print('URL:', url)
res = urllib.request.urlopen(url)
print('.urlopen() done.')
#time.sleep(6)
-------------------------------------------------------------------------------
Anyway, the question of HTTP error codes still remains, but unfortunately the
situation of multiple people fetching at the same time seems to be difficult
to deal with.
Regards,
Yuta Hayama
-=-=-=-=-=-=-=-=-=-=-=-
Links: You receive all messages sent to this group.
View/Reply Online (#192254):
https://lists.openembedded.org/g/openembedded-core/message/192254
Mute This Topic: https://lists.openembedded.org/mt/103049968/21656
Group Owner: openembedded-core+ow...@lists.openembedded.org
Unsubscribe: https://lists.openembedded.org/g/openembedded-core/unsub
[arch...@mail-archive.com]
-=-=-=-=-=-=-=-=-=-=-=-
