[
https://issues.apache.org/jira/browse/BEAM-10381?focusedWorklogId=453080&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-453080
]
ASF GitHub Bot logged work on BEAM-10381:
-----------------------------------------
Author: ASF GitHub Bot
Created on: 30/Jun/20 19:06
Start Date: 30/Jun/20 19:06
Worklog Time Spent: 10m
Work Description: ajamato commented on a change in pull request #12130:
URL: https://github.com/apache/beam/pull/12130#discussion_r447916417
##########
File path: sdks/java/container/license_scripts/pull_licenses_java.py
##########
@@ -39,7 +39,7 @@
from urllib.request import urlopen, URLError, HTTPError
SOURCE_CODE_REQUIRED_LICENSES = ['lgpl', 'glp', 'cddl', 'mpl']
-RETRY_NUM = 3
+RETRY_NUM = 9
Review comment:
I suggest we use wait_fixed(5) instead of the wait_exponential. Since we
were only setting retries to 3 before it should have been waiting:
2s, 4s, 8s.
wait_fixed(5) will spend about a minute max on each url. I don't want to
make it wait much more. I believe the test will time out if we make it too long.
wait: 5s, 5s, 5s, 5s, 5s, 5s, 5s, 5s, 5s
I suggest we try this first. WDYT? Here are the tenacity docs:
https://tenacity.readthedocs.io/en/latest/
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 453080)
Time Spent: 0.5h (was: 20m)
> ./sdks/java/container/license_scripts/license_script.sh failing in python PR
> ----------------------------------------------------------------------------
>
> Key: BEAM-10381
> URL: https://issues.apache.org/jira/browse/BEAM-10381
> Project: Beam
> Issue Type: Test
> Components: test-failures
> Reporter: Alex Amato
> Priority: P2
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> This seems odd as this PR doesn't modify any java code or deps. Is this just
> broken at master?
> PR [https://github.com/apache/beam/pull/12084]
> Fails the *Python2_PVR_Flink ("Run Python2_PVR_Flink PreCommit")*
> [https://scans.gradle.com/s/rqx6h5cqyvvfo]
> I added some logging locally to determine the URL it fails to pull from:
> INFO:root:Pulling from url:
> https://repo1.maven.org/maven2/org/apache/hadoop/hadoop-mapreduce-client-jobclient/2.8.5/hadoop-mapreduce-client-jobclient-2.8.5.jar
> Traceback (most recent call last):
> File
> "/usr/local/google/home/ajamato/beam/sdks/java/container/license_scripts/pull_licenses_java.py",
> line 250, in <module>
> pool.map(execute, dependencies['dependencies'])
> File "/usr/lib/python3.7/multiprocessing/pool.py", line 268, in map
> return self._map_async(func, iterable, mapstar, chunksize).get()
> File "/usr/lib/python3.7/multiprocessing/pool.py", line 657, in get
> raise self._value
> File "/usr/lib/python3.7/multiprocessing/pool.py", line 121, in worker
> result = (True, func(*args, **kwds))
> File "/usr/lib/python3.7/multiprocessing/pool.py", line 44, in mapstar
> return list(map(*args))
> File
> "/usr/local/google/home/ajamato/beam/sdks/java/container/license_scripts/pull_licenses_java.py",
> line 203, in execute
> pull_source_code(base_url, dir_name, name_version)
> File
> "/usr/local/google/home/ajamato/beam/sdks/java/container/license_scripts/pull_licenses_java.py",
> line 105, in pull_source_code
> soup = BeautifulSoup(urlopen(base_url).read(), "html.parser")
> File "/usr/lib/python3.7/urllib/request.py", line 222, in urlopen
> return opener.open(url, data, timeout)
> File "/usr/lib/python3.7/urllib/request.py", line 531, in open
> response = meth(req, response)
> File "/usr/lib/python3.7/urllib/request.py", line 641, in http_response
> 'http', request, response, code, msg, hdrs)
> File "/usr/lib/python3.7/urllib/request.py", line 569, in error
> return self._call_chain(*args)
> File "/usr/lib/python3.7/urllib/request.py", line 503, in _call_chain
> result = func(*args)
> File "/usr/lib/python3.7/urllib/request.py", line 649, in http_error_default
> raise HTTPError(req.full_url, code, msg, hdrs, fp)
> urllib.error.HTTPError: HTTP Error 404: Not Found
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
