GitHub user stemarks edited a discussion: Airflow with virtualenv and private repositories
Hi All, We are new to Airflow and have setup some basic DAGs with little effort, however we are now looking at more complex DAGs and have hit an issue with the dependencies clashing with airflow. so now we want to use virtual env's, the issue is we have all our python packages live in private github repositories. We need to be able to run all our dags in their own virtual env, but the packages we use for our tasks are stored in private github repos, we are a GCP house so could put the wheels onto Artifact Registry if that makes it easier. The biggest thing that im not sure how to get around is the authentication to our github repositories, is there an ENV variable that i can set to tell it to use for authentication when it gets a git+https://github.com in the requiremenets? The documentation i have read doesnt really give a clear example of doing this. I would also want it to be cached so that it doesnt re-install every time the dag runs unless a version has changed. I'm sure that this is a common use case but struggling to progress due to lack of knowledge We are using the official Apache Airflow Helm chart for our deployment GitHub link: https://github.com/apache/airflow/discussions/44812 ---- This is an automatically sent email for [email protected]. To unsubscribe, please send an email to: [email protected]
