hi,
Based on some feedback, we've decided to change the way we deploy liminal
locally (how "liminal deploy" works).
The deploy needs to get the following tasks done:
1. Deploy liminal/runners/airflow/dag/liminal_dags.py to the airflow docker
/dags folder.
this is the generic code which loads user dags from yaml
2. Deploy the user yamls files to a location in airflow docker where (1) can
load them
3. Ensure airflow docker process includes any python dependency which (1) needs
which is not already bundled with the airflow package (e.g. pyyaml)
Before the change, liminal deploy would call a bash script which:
1. Creates a LIMINAL_HOME folder
2. Creates a monolytic zip, in which it packages liminal_dags, and all of the
liminal and user dependnecies found in the different requirements-*.txt
3. Copies the user yamls into it
4. Docker compose mounts this folder into it's /dags folder
There were a few issues with this:
1. installation was error prone - any import could break it
2. Installation had superflous packages not needed for loading DAGS
After the change:
1. Create LIMINAL_HOME folder
2. Docker compose starts airflow from a Dockerfile we provide in /scripts (vs.
airflow image)
3. This dockerfile contains instructions on how to install liminal from pypi
(or local files, or github)
with it's minimal dependencies
so liminal deploy-> docker compose-> docker build from dockerfile -> pip
install liminal
The specific version which will be installed in the docker can be controlled by
an environment parameter LIMINAL_VERSION
which can hold any valid string which completes the following command:
pip install ${LIMINAL_VERSION}
This includes
- A standard pip version like apache-liminal==0.0.1dev1 avail from pypi
- A URL for git e.g. git+https://github.com/apache/incubator-liminal.git
- A string indicating where to get the package from like --index <url>
apache-liminal==xyz
- A local path to a .whl which is available inside the docker (e.g. which you
placed
in the scripts/ folder)
This is useful if you are making changes in liminal locally and want to test
them.
If you don't specify this variable, liminal attempts to discover how to install
itself
by running a pip freeze and looking at the result.
This covers pip repositories, files and installtion from URL.
The fallback in case no string is found, is simply 'apache-liminal' assuming
your .pypirc contains an
index which has this package.
In addition, we moved to the new official airflow docker (previously we were
using a community unofficial docker, before airflow released this one)
I will ping some folks on the slack channel to give this a go, as we need to
validate this on several setups.
Assaf
