feluelle edited a comment on issue #8605:
URL: https://github.com/apache/airflow/issues/8605#issuecomment-643470106
My `docker-compose.yaml` file for running _LocalExecutor_ with meta-db
_postgres_ using apache airflow prod official Docker file.
Additional notes:
* using Airflow (`1-orchestration`) for _ELT_ with great_expectations
(`2-expectations`) and dbt (`3-transformations`).
* using AWS System Manager - Parameter Store for Airflow variables and
connections (using volume `.aws` folder to authenticate)
* using my edited requirements due to latest Airflow and Great Expectations
incompatibilities of `marshmallow` dependency. (ge uses `marshmallow>3.0` and
airflow uses `marshmallow==2.21.0`)
* using `.dbt` folder to mount dbt profiles used for authentication and add
additional port `38081:8081` to use for `dbt docs serve`
```yaml
version: '3.7'
x-airflow-environment:
&airflow-common
environment:
- AIRFLOW_HOME=/opt/airflow/1-orchestration
- AIRFLOW__CORE__EXECUTOR=LocalExecutor
-
AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgresql+psycopg2://airflow:airflow@airflow_db:5432/airflow
- AIRFLOW__CORE__FERNET_KEY=FB0o_zt4e3Ziq3LdUUO7F2Z95cvFFx16hU8jTeR1ASM=
-
AIRFLOW__SECRETS__BACKEND=airflow.providers.amazon.aws.secrets.systems_manager.SystemsManagerParameterStoreBackend
image: apache/airflow:master-python3.7
build:
context: https://github.com/apache/airflow.git
args:
CONSTRAINT_REQUIREMENTS:
https://raw.githubusercontent.com/feluelle/airflow/marshmallow-update/requirements/requirements-python3.7.txt
AIRFLOW_EXTRAS: amazon,postgres,google
ADDITIONAL_PYTHON_DEPS: dbt great_expectations
services:
airflow_db:
image: library/postgres:10
container_name: airflow_db
environment:
- POSTGRES_USER=airflow
- POSTGRES_PASSWORD=airflow
- POSTGRES_DB=airflow
ports:
- 35432:5432
restart: always
airflow_db_init:
<<: *airflow-common
container_name: airflow_db_init
command: db upgrade
depends_on:
- airflow_db
airflow_users_init:
<<: *airflow-common
container_name: airflow_users_init
command: users create --role Admin --username admin --email
[email protected] --firstname Admin --lastname Airflow --password airflow
depends_on:
- airflow_db_init
airflow_webserver:
<<: *airflow-common
container_name: airflow_webserver
command: webserver
ports:
- 38080:8080
- 38081:8081
volumes:
- ~/.aws:/home/airflow/.aws
- ~/.dbt:/home/airflow/.dbt
- ../1-orchestration:/opt/airflow/1-orchestration
- ../2-expectations:/opt/airflow/2-expectations
- ../3-transformations:/opt/airflow/3-transformations
depends_on:
- airflow_db
restart: always
airflow_scheduler:
<<: *airflow-common
container_name: airflow_scheduler
command: scheduler
volumes:
- ~/.aws:/home/airflow/.aws
- ~/.dbt:/home/airflow/.dbt
- ../1-orchestration:/opt/airflow/1-orchestration
- ../2-expectations:/opt/airflow/2-expectations
- ../3-transformations:/opt/airflow/3-transformations
depends_on:
- airflow_db
restart: always
```
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
