Enrique Sánchez has proposed merging
~enriqueesanchz/launchpad:importvulnerabilityjob-urlfetch-timeout into
launchpad:master.
Commit message:
Add override getBlob request timeout
Catch RequestExceptionWrapper, a non-requests exception that can occur
during the git hosting API call.
Requested reviews:
Launchpad code reviewers (launchpad-reviewers)
For more details, see:
https://code.launchpad.net/~enriqueesanchz/launchpad/+git/launchpad/+merge/492534
During the import job execution we can face this exception:
lp.code.model.githosting.RequestExceptionWrapper: Attempted to set connect
timeout to 0, but the timeout cannot be set to a value less than or equal to 0.
This change aims to set a custom value to that timeout instead of using the
default one. Also we handle the `RequestExceptionWrapper` so we stop making API
calls if they are failing.
--
Your team Launchpad code reviewers is requested to review the proposed merge of
~enriqueesanchz/launchpad:importvulnerabilityjob-urlfetch-timeout into
launchpad:master.
diff --git a/lib/lp/bugs/model/importvulnerabilityjob.py b/lib/lp/bugs/model/importvulnerabilityjob.py
index db7b2f0..0b758bd 100644
--- a/lib/lp/bugs/model/importvulnerabilityjob.py
+++ b/lib/lp/bugs/model/importvulnerabilityjob.py
@@ -30,10 +30,12 @@ from lp.bugs.scripts.soss.sossimport import SOSSImporter
from lp.code.errors import GitRepositoryBlobNotFound, GitRepositoryScanFault
from lp.code.interfaces.githosting import IGitHostingClient
from lp.code.interfaces.gitlookup import IGitLookup
+from lp.code.model.githosting import RequestExceptionWrapper
from lp.services.config import config
from lp.services.database.interfaces import IPrimaryStore, IStore
from lp.services.job.interfaces.job import JobStatus
from lp.services.job.model.job import Job
+from lp.services.timeout import override_timeout
CVE_PATTERN = re.compile(r"^CVE-\d{4}-\d+$")
logger = logging.getLogger(__name__)
@@ -250,7 +252,9 @@ class ImportVulnerabilityJob(VulnerabilityJobDerived):
success = False
try:
logger.debug(f"[ImportVulnerabilityJob] Getting {file}")
- blob = ref.getBlob(file)
+
+ with override_timeout(self.config.getblob_timeout):
+ blob = ref.getBlob(file)
logger.debug(
f"[ImportVulnerabilityJob] Parsing {cve_sequence}"
@@ -262,11 +266,17 @@ class ImportVulnerabilityJob(VulnerabilityJobDerived):
if bug and vulnerability:
success = True
self.metadata["result"]["succeeded"].append(cve_sequence)
- except (GitRepositoryScanFault, GitRepositoryBlobNotFound):
- # If there is a problem with git repository, we stop as we
- # won't be able to fetch more files
+ except (
+ GitRepositoryScanFault,
+ GitRepositoryBlobNotFound,
+ RequestExceptionWrapper,
+ ) as e:
+ # If there is a problem with git repository API call, we stop
+ # as we won't be able to fetch more files
+ self.notifyUserError(f"{cve_sequence}: {e}")
raise VulnerabilityJobException(
- f"Failed to get file from Git repository: {file}"
+ f"Failed to get file from Git repository: {file}."
+ "Aborting."
)
except Exception as e:
self.notifyUserError(f"{cve_sequence}: {e}")
diff --git a/lib/lp/services/config/schema-lazr.conf b/lib/lp/services/config/schema-lazr.conf
index d3c2892..7dbf78a 100644
--- a/lib/lp/services/config/schema-lazr.conf
+++ b/lib/lp/services/config/schema-lazr.conf
@@ -2253,6 +2253,7 @@ link: IBranchMergeProposalJobSource
[IImportVulnerabilityJobSource]
module: lp.bugs.interfaces.vulnerabilityjob
dbuser: launchpad_main
+getblob_timeout: 10
[IExportVulnerabilityJobSource]
module: lp.bugs.interfaces.vulnerabilityjob
_______________________________________________
Mailing list: https://launchpad.net/~launchpad-reviewers
Post to : launchpad-reviewers@lists.launchpad.net
Unsubscribe : https://launchpad.net/~launchpad-reviewers
More help : https://help.launchpad.net/ListHelp
