gaoyibin0001 commented on issue #15286:
URL: https://github.com/apache/airflow/issues/15286#issuecomment-1046497988
> @ManikandanUV We are doing it the following way for now:
>
> ```python
> env = vars_dict.get("conda_env", None)
> path_to_python = f"/home/username/.conda{'/envs/'+env if env is not None
else ''}/bin/python"
>
> parse_files = BashOperator(
> task_id='parse-files',
> bash_command=f"{path_to_python}
{abs_path_code}/my_repo/parse.py {files_to_parse}",
> env={"PATH": os.environ["PATH"],
> "DB_CONN": db_conn}
> )
> ```
>
> We have an environment variable containing the conda-env name which is
used to get the full path to the Python executable. Then, using a BashOperator,
we can use the same environment again for different Tasks.
>
> Additionally, we run an update to the environment if requirements changed
(note that we are using poetry as package manager):
>
> ```python
> update_repo = BashOperator(
> task_id=f"update-repo-{folder}",
> bash_command=f"cd {abs_path_code}/{folder}; "
> "git checkout master; git stash; git stash drop; git pull"
> )
> install_dependencies = BashOperator(
> task_id=f"install-dependencies-{folder}",
> bash_command=f"cd {abs_path_code}/{folder}; conda activate
{env_name}; poetry install "
> )
> update_repo >> install_dependencies
> `
> ``
> ```
may use "conda run -n env_name python xxx.py
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
