chamikaramj commented on code in PR #38348:
URL: https://github.com/apache/beam/pull/38348#discussion_r3191463960
##########
sdks/python/apache_beam/runners/portability/stager.py:
##########
@@ -701,14 +701,29 @@ def _remove_dependency_from_requirements(
return tmp_requirements_filename
@staticmethod
- def _extract_local_packages(requirements_file):
+ def _extract_local_packages(requirements_file, temp_dir):
local_deps = []
pypi_deps = []
with open(requirements_file, 'r') as fin:
+ staging_temp_dir = tempfile.mkdtemp(dir=temp_dir)
for line in fin:
dep = line.strip()
if os.path.exists(dep):
local_deps.append(dep)
+ elif urlparse(dep).scheme:
+ last_component = dep.rsplit('/', 1)[-1]
+ if not last_component:
+ _LOGGER.warning(
+ 'Extra package %s has an unexpected format hence will not be
downloaded locally.'
+ % dep)
+ continue
+ # Download remote package.
+ _LOGGER.info(
+ 'Downloading remote extra package: %s locally before staging',
+ dep)
+ local_file_path = FileSystems.join(staging_temp_dir, last_component)
+ Stager._download_file(dep, local_file_path)
+ local_deps.append(local_file_path)
Review Comment:
I don't think this analysis is correct. AFAICT
`Stager._remove_dependency_from_requirements` is called one time specifically
to remove the "apache-beam" package. Not as a way to trim the requirements file
based on local deps.
https://github.com/apache/beam/blob/efe4e941939a77275146ecceb01cd74b28555286/sdks/python/apache_beam/runners/portability/stager.py#L755
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
