[
https://issues.apache.org/jira/browse/BEAM-9764?focusedWorklogId=425529&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-425529
]
ASF GitHub Bot logged work on BEAM-9764:
----------------------------------------
Author: ASF GitHub Bot
Created on: 21/Apr/20 03:04
Start Date: 21/Apr/20 03:04
Worklog Time Spent: 10m
Work Description: Hannah-Jiang commented on a change in pull request
#11428:
URL: https://github.com/apache/beam/pull/11428#discussion_r411834018
##########
File path: sdks/java/container/license_scripts/pull_licenses_java.py
##########
@@ -108,89 +244,27 @@ def write_to_csv(csv_dict):
maven_url_temp = 'https://repo1.maven.org/maven2/{module}/{version}'
- for dep in dependencies['dependencies']:
- '''
- An example of a Json blob.
- {
- "moduleName": "antlr:antlr",
- "moduleUrl": "http://www.antlr.org/";,
- "moduleVersion": "2.7.7",
- "moduleLicense": "BSD License",
- "moduleLicenseUrl": "http://www.antlr.org/license.html";
- }
- '''
- name = dep['moduleName'].split(':')[1].lower()
- version = dep['moduleVersion']
- name_version = name + '-' + version
- dir_name = '{license_dir}/{name_version}.jar'.format(
- license_dir=LICENSE_DIR, name_version=name_version)
- # if auto pulled, directory is existing at {license_dir}
- if not os.path.isdir(dir_name):
- # skip self dependencies
- if dep['moduleName'].startswith('beam'):
- print('Skippig', name + '-' + version)
- continue
- os.mkdir(dir_name)
- # pull license
- try:
- license_url = dep_config[name][version]['license']
- except:
- license_url = dep['moduleLicenseUrl']
- pull_from_url(dir_name + '/LICENSE', license_url, name_version,
- no_licenses)
- # pull notice
- try:
- notice_url = dep_config[name][version]['notice']
- pull_from_url(dir_name + '/NOTICE', notice_url, name_version,
- no_licenses)
- except:
- notice_url = None
- else:
- try:
- license_url = dep['moduleLicenseUrl']
- except:
- license_url = ''
- print(
- 'License/notice for {name_version} were pulled automatically.'.
- format(name_version=name_version))
-
- # get license_type to decide if pull source code.
- try:
- license_type = dep['moduleLicense']
- except:
- try:
- license_type = dep_config[name][version]['type']
- except:
- no_license_type.add(name_version)
- license_type = ''
- continue
+ csv_list = []
+ no_licenses = []
+ no_license_type = []
+ incorrect_source_url = []
- # pull source code if license_type is one of
SOURCE_CODE_REQUIRED_LICENSES.
- if any(x in license_type.lower()
- for x in SOURCE_CODE_REQUIRED_LICENSES):
- try:
- base_url = dep_config[name][version]['source']
- except:
- module = dep['moduleName'].split(':')[0].replace('.', '/')
- base_url = maven_url_temp.format(module=module + '/' + name,
- version=version)
- pull_source_code(base_url, dir_name, name_version,
- incorrect_source_url)
- source_included = True
- else:
- source_included = False
-
- csv_dict[name_version] = {
- 'url_to_license': license_url,
- 'license_type': license_type,
- 'source_included': source_included
- }
+ queue = Queue()
+ for x in range(THREADS):
+ worker = Worker(queue)
+ worker.daemon = True
+ worker.start()
+ for dep in dependencies['dependencies']:
+ queue.put(dep)
+ queue.join()
Review comment:
Thanks Udi. Yes, I agree `ThreadPoolExecutor` is better here, but it
(concurrent.future) only works with Py3. In order to use virtual env, we need
to use tox and it brings unnecessary complexity. I think it's good enough for
now and we can change to `ThreadPoolExecutor` when we sunset down Py2.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
Issue Time Tracking
-------------------
Worklog Id: (was: 425529)
Time Spent: 3.5h (was: 3h 20m)
> :sdks:java:container:generateThirdPartyLicenses failing
> -------------------------------------------------------
>
> Key: BEAM-9764
> URL: https://issues.apache.org/jira/browse/BEAM-9764
> Project: Beam
> Issue Type: Bug
> Components: sdk-java-core, test-failures
> Reporter: Udi Meiri
> Assignee: Hannah Jiang
> Priority: Major
> Time Spent: 3.5h
> Remaining Estimate: 0h
>
> https://builds.apache.org/job/beam_PreCommit_Python2_PVR_Flink_Cron/774/console
> The traceback is interspersed with other logs:
> {code}
> Traceback (most recent call last):
> Successfully pulled
> java_third_party_licenses/protobuf-java-util-3.11.1.jar/LICENSE from
> https://opensource.org/licenses/BSD-3-Clause
> Successfully pulled java_third_party_licenses/protoc-3.11.0.jar/LICENSE from
> http://www.apache.org/licenses/LICENSE-2.0.txt
> File "sdks/java/container/license_scripts/pull_licenses_java.py", line 138,
> in <module>
> Successfully pulled java_third_party_licenses/protoc-3.11.1.jar/LICENSE from
> http://www.apache.org/licenses/LICENSE-2.0.txt
> license_url = dep['moduleLicenseUrl']
> Successfully pulled java_third_party_licenses/zetasketch-0.1.0.jar/LICENSE
> from http://www.apache.org/licenses/LICENSE-2.0.txt
> KeyError: 'moduleLicenseUrl'
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
