request for testing: #check that you have docker running #check that you have a local kubernetes in your docker service
create fresh project with .yaml file create a fresh virtualenv pip install -i https://test.pypi.org/simple/ apache-liminal #this is a test pypi make sure docker is running Make sure old airflows containers aren’t running (after doing this once, the following command takes care of this) liminal build liminal deploy --clean #this may take a while on the first time liminal start you should be able to go to localhost:8080, enable the DAG and trigger it . On 2020/11/01 19:52:10, Assaf Pinhasi <[email protected]> wrote: > hi, > > Based on some feedback, we've decided to change the way we deploy liminal > locally (how "liminal deploy" works). > > The deploy needs to get the following tasks done: > 1. Deploy liminal/runners/airflow/dag/liminal_dags.py to the airflow docker > /dags folder. > this is the generic code which loads user dags from yaml > 2. Deploy the user yamls files to a location in airflow docker where (1) can > load them > 3. Ensure airflow docker process includes any python dependency which (1) > needs which is not already bundled with the airflow package (e.g. pyyaml) > > Before the change, liminal deploy would call a bash script which: > 1. Creates a LIMINAL_HOME folder > 2. Creates a monolytic zip, in which it packages liminal_dags, and all of the > liminal and user dependnecies found in the different requirements-*.txt > 3. Copies the user yamls into it > 4. Docker compose mounts this folder into it's /dags folder > > There were a few issues with this: > 1. installation was error prone - any import could break it > 2. Installation had superflous packages not needed for loading DAGS > > After the change: > 1. Create LIMINAL_HOME folder > 2. Docker compose starts airflow from a Dockerfile we provide in /scripts > (vs. airflow image) > 3. This dockerfile contains instructions on how to install liminal from pypi > (or local files, or github) > with it's minimal dependencies > > so liminal deploy-> docker compose-> docker build from dockerfile -> pip > install liminal > > The specific version which will be installed in the docker can be controlled > by an environment parameter LIMINAL_VERSION > > which can hold any valid string which completes the following command: > pip install ${LIMINAL_VERSION} > > This includes > - A standard pip version like apache-liminal==0.0.1dev1 avail from pypi > - A URL for git e.g. git+https://github.com/apache/incubator-liminal.git > - A string indicating where to get the package from like --index <url> > apache-liminal==xyz > - A local path to a .whl which is available inside the docker (e.g. which you > placed > in the scripts/ folder) > > This is useful if you are making changes in liminal locally and want to test > them. > > If you don't specify this variable, liminal attempts to discover how to > install itself > by running a pip freeze and looking at the result. > This covers pip repositories, files and installtion from URL. > > The fallback in case no string is found, is simply 'apache-liminal' assuming > your .pypirc contains an > index which has this package. > > In addition, we moved to the new official airflow docker (previously we were > using a community unofficial docker, before airflow released this one) > > I will ping some folks on the slack channel to give this a go, as we need to > validate this on several setups. > > Assaf > > > >
