uranusjr commented on a change in pull request #17349:
URL: https://github.com/apache/airflow/pull/17349#discussion_r680677265
##########
File path: airflow/operators/python.py
##########
@@ -333,20 +337,31 @@ def __init__(
templates_exts=templates_exts,
**kwargs,
)
- self.requirements = list(requirements or [])
+ if not isinstance(requirements, str):
+ self.requirements = list(requirements or [])
+ else:
+ self.requirements = requirements
self.string_args = string_args or []
self.python_version = python_version
self.use_dill = use_dill
self.system_site_packages = system_site_packages
- if not self.system_site_packages and self.use_dill and 'dill' not in
self.requirements:
- self.requirements.append('dill')
self.pickling_library = dill if self.use_dill else pickle
+ def pre_execute(self, context: Any):
+ if isinstance(self.requirements, list):
+ return
+
+ import pkg_resources
+ self.requirements = [str(req) for req in
pkg_resources.parse_requirements(self.requirements)]
Review comment:
The requirements file format can do more than listing requirements, and
those cannot be parsed by `parse_requirements` (I don’t remember whether
setuptools would throw or just silently drop them.) So instead of trying to
pre-parse the content, I would try to parse it to pip instead.
* Remove `pre_execute`
* Change `prepare_virtualenv` to accept `requirements` as str (plus the
currently supported list form)
* When doing `pip install`, write the str type `requirements` to a temporary
file for `pip intall -r`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
