This is an automated email from the ASF dual-hosted git repository.
assafpinhasi pushed a commit to branch official_docker
in repository https://gitbox.apache.org/repos/asf/incubator-liminal.git
The following commit(s) were added to refs/heads/official_docker by this push:
new 1802a5c new packaging and deployment by baking liminal into the
airflow docker dynamically
new 7936d02 Merge branch 'official_docker' of
github.com:apache/incubator-liminal into official_docker
1802a5c is described below
commit 1802a5c48a6f14db63d51a96e74f053fc2c5ee3e
Author: Assaf Pinhasi <[email protected]>
AuthorDate: Fri Oct 30 12:51:23 2020 +0200
new packaging and deployment by baking liminal into the airflow docker
dynamically
add notice file
support deploying local/unofficial vesions of liminal inside airflow
some instructions on how to build and test liminal locally
RELEASE updates
few deployment tweaks
fixing missing logs and making deploy --clean reset the DB first
---
MANIFEST.in | 1 -
NOTICE | 6 +
README.md | 9 +
dev/RELEASE.md | 698 +++++++++++++++++++++
MANIFEST.in => dev/sign.sh | 19 +-
liminal/core/environment.py | 21 +
scripts/Dockerfile-airflow | 15 +
scripts/__init__.py | 0
scripts/docker-compose.yml | 65 +-
scripts/liminal | 57 +-
scripts/package.sh | 60 --
.../requirements-airflow.txt | 0
setup.py | 17 +-
13 files changed, 878 insertions(+), 90 deletions(-)
diff --git a/MANIFEST.in b/MANIFEST.in
index 6e3a162..c6db8dd 100644
--- a/MANIFEST.in
+++ b/MANIFEST.in
@@ -17,5 +17,4 @@
# under the License.
include scripts/*
-include requirements-airflow.txt
recursive-include liminal/build/ *
\ No newline at end of file
diff --git a/NOTICE b/NOTICE
new file mode 100644
index 0000000..a3c5a7e
--- /dev/null
+++ b/NOTICE
@@ -0,0 +1,6 @@
+Apache Liminal
+Copyright 2016-2020 The Apache Software Foundation
+
+This product includes software developed at The Apache Software
+Foundation (http://www.apache.org/).
+
diff --git a/README.md b/README.md
index 8d1fd47..d283f13 100644
--- a/README.md
+++ b/README.md
@@ -101,6 +101,7 @@ services:
```bash
pip install liminal
```
+
2. Optional: set LIMINAL_HOME to path of your choice (if not set, will default
to ~/liminal_home)
```bash
echo 'export LIMINAL_HOME=</path/to/some/folder>' >> ~/.bash_profile && source
~/.bash_profile
@@ -124,6 +125,8 @@ When your pipeline code is ready, you can test it by
running it locally on your
1. Ensure you have The Docker engine running locally, and enable a local
Kubernetes cluster:
+And allocate it at least 3 CPUs (under "Resources" in the Docker preference
UI).
+
If you want to execute your pipeline on a remote kubernetes cluster, make sure
the cluster is configured
using :
```bash
@@ -152,6 +155,12 @@ You'll see that a number of outputs indicating various
docker images built.
cd </path/to/your/liminal/code>
liminal deploy
```
+Note: after upgrading liminal, it's recommended to issue the command
+```bash
+liminal deploy --clean
+```
+
+This will rebuild the airlfow docker containers from scratch with a fresh
version of liminal, ensuring consistency.
4. Start the server
```bash
diff --git a/dev/RELEASE.md b/dev/RELEASE.md
new file mode 100644
index 0000000..97acde4
--- /dev/null
+++ b/dev/RELEASE.md
@@ -0,0 +1,698 @@
+<!--
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+<!-- START doctoc generated TOC please keep comment here to allow auto update
-->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+
+
+- [Apache Liminal source releases](#apache-liminal-source-releases)
+ - [Apache Liminal Package](#apache-liminal-package)
+- [Prerequisites for the release manager preparing the
release](#prerequisites-for-the-release-manager-preparing-the-release)
+ - [version IDs](#version-ids)
+ - [Testing the release locally](#testing-the-release-locally)
+ - [Upload Public keys to id.apache.org](#upload-public-keys-to-idapacheorg)
+ - [Configure PyPI uploads](#configure-pypi-uploads)
+ - [Hardware used to prepare and verify the
packages](#hardware-used-to-prepare-and-verify-the-packages)
+- [Apache Liminal packages](#apache-liminal-packages)
+ - [Prepare the Apache Liminal Package
RC](#prepare-the-apache-liminal-package-rc)
+ - [Build RC artifacts (both source packages and convenience
packages)](#build-rc-artifacts-both-source-packages-and-convenience-packages)
+ - [Prepare PyPI convenience "snapshot"
packages](#prepare-pypi-convenience-snapshot-packages)
+ - [Vote and verify the Apache Liminal release
candidate](#vote-and-verify-the-apache-liminal-release-candidate)
+ - [Prepare Vote email on the Apache Liminal release
candidate](#prepare-vote-email-on-the-apache-liminal-release-candidate)
+ - [Verify the release candidate by PMCs
(legal)](#verify-the-release-candidate-by-pmcs-legal)
+ - [PMC responsibilities](#pmc-responsibilities)
+ - [SVN check](#svn-check)
+ - [Verify the licences](#verify-the-licences)
+ - [Verify the signatures](#verify-the-signatures)
+ - [Verify the SHA512 sum](#verify-the-sha512-sum)
+ - [Verify if the release candidate "works" by
Contributors](#verify-if-the-release-candidate-works-by-contributors)
+ - [Publish the final Apache Liminal
release](#publish-the-final-apache-liminal-release)
+ - [Summarize the voting for the Apache Liminal
release](#summarize-the-voting-for-the-apache-liminal-release)
+ - [Publish release to SVN](#publish-release-to-svn)
+ - [Prepare PyPI "release" packages](#prepare-pypi-release-packages)
+ - [Update CHANGELOG.md](#update-changelogmd)
+ - [Notify developers of release](#notify-developers-of-release)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
+# Apache Liminal source releases
+
+## Apache Liminal Package
+
+This package contains sources that allow the user building fully-functional
Apache Liminal package.
+They contain sources for "apache-liminal" python package that installs
"liminal" Python package and includes
+all the assets required to release the webserver UI coming with Apache Liminal
+
+The Source releases are the only "official" Apache Software Foundation
releases, and they are distributed
+via [Official Apache Download sources](https://downloads.apache.org/)
+
+Following source releases Apache Liminal release manager also distributes
convenience packages:
+
+* PyPI packages released via https://pypi.org/project/apache-liminal/
+* Docker Images released via
https://hub.docker.com/repository/docker/apache/liminal (coming soon...)
+
+Those convenience packages are not "official releases" of Apache Liminal, but
the users who
+cannot or do not want to build the packages themselves can use them as a
convenient way of installing
+Apache Liminal, however they are not considered as "official source releases".
You can read more
+details about it in the [ASF Release
Policy](http://www.apache.org/legal/release-policy.html).
+
+This document describes the process of releasing both - official source
packages and convenience
+packages for Apache Liminal packages.
+
+# Prerequisites for the release manager preparing the release
+
+The person acting as release manager has to fulfill certain pre-requisites.
More details and FAQs are
+available in the [ASF Release
Policy](http://www.apache.org/legal/release-policy.html) but here some important
+pre-requisites are listed below. Note that release manager does not have to be
a PMC - it is enough
+to be committer to assume the release manager role, but there are final steps
in the process (uploading
+final releases to SVN) that can only be done by PMC member. If needed, the
release manager
+can ask PMC to perform that final step of release.
+
+## version IDs
+
+Should follow https://www.python.org/dev/peps/pep-0440/
+
+## Testing the release locally
+
+- Build a local version of liminal:
+```bash
+# Set Version
+export LIMINAL_BUILD_VERSION=0.0.1RC1
+python setup.py sdist bdist_wheel
+```
+
+- Start a test project with a liminal .yml file in it
+
+- Clear folder $LIMINAL_HOME
+
+- Install liminal in the test project
+
+```bash
+pip install <path to lminal .whl created by setup.py>
+liminal build
+liminal deploy --clean
+liminal start
+```
+
+Note:
+When you run liminal deploy, a liminal installs itself inside airflow
container.
+You can control which version of liminal gets installed inside docker using
LIMINAL_VERSION environment variable.
+
+This can be any string which results in a legal call to:
+```bash
+pip install ${LIMINAL_VERSION}
+```
+
+This includes
+- A standard pip version like apache-liminal==0.0.1dev1 avail from pypi
+- A URL for git e.g. git+https://github.com/apache/incubator-liminal.git
+- A string indicating where to get the package from like --index <url>
apache-liminal==xyz
+- A local path to a .whl which is available inside the docker (e.g. which you
placed
+in the scripts/ folder)
+
+This is useful if you are making changes in liminal locally and want to test
them.
+
+If you don't specify this variable, liminal attempts to discover how to
install itself
+by running a pip freeze and looking at the result.
+This covers pip repositories, files and installtion from URL.
+
+The fallback in case no string is found, is simply 'apache-liminal' assuming
your .pypirc contains an
+index which has this package.
+
+## Upload Public keys to id.apache.org
+
+Make sure your public key is on id.apache.org and in KEYS. You will need to
sign the release artifacts
+with your pgp key. After you have created a key, make sure you:
+
+- Add your GPG pub key to
https://dist.apache.org/repos/dist/release/liminal/KEYS , follow the
instructions at the top of that file. Upload your GPG public key to
https://pgp.mit.edu
+- Add your key fingerprint to https://id.apache.org/ (login with your apache
credentials, paste your fingerprint into the pgp fingerprint field and hit
save).
+
+```shell script
+# Create PGP Key
+gpg --gen-key
+
+# Checkout ASF dist repo
+svn checkout https://dist.apache.org/repos/dist/release/incubator/liminal
+cd liminal
+
+
+# Add your GPG pub key to KEYS file. Replace "Aviem Zur" with your name
+(gpg --list-sigs "Aviem Zur" && gpg --armor --export "Aviem Zur" ) >> KEYS
+
+
+# Commit the changes
+svn commit -m "Add PGP keys of Liminal developers"
+```
+
+See this for more detail on creating keys and what is required for signing
releases.
+
+http://www.apache.org/dev/release-signing.html#basic-facts
+
+## Configure PyPI uploads
+
+In order to not reveal your password in plain text, it's best if you create
and configure API Upload tokens.
+You can add and copy the tokens here:
+
+* [Test PyPI](https://test.pypi.org/manage/account/token/)
+* [Prod PyPI](https://pypi.org/manage/account/token/)
+
+
+Create a `~/.pypirc` file:
+
+```ini
+[distutils]
+index-servers =
+ pypi
+ pypitest
+
+[pypi]
+username=__token__
+password=<API Upload Token>
+
+[pypitest]
+repository=https://test.pypi.org/legacy/
+username=__token__
+password=<API Upload Token>
+```
+
+Set proper permissions for the pypirc file:
+
+```shell script
+$ chmod 600 ~/.pypirc
+```
+
+- Install [twine](https://pypi.org/project/twine/) if you do not have it
already (it can be done
+ in a separate virtual environment).
+
+```shell script
+pip install twine
+```
+
+(more details [here](https://peterdowns.com/posts/first-time-with-pypi.html).)
+
+- Set proper permissions for the pypirc file:
+`$ chmod 600 ~/.pypirc`
+
+## Hardware used to prepare and verify the packages
+
+The best way to prepare and verify the releases is to prepare them on a
hardware owned and controlled
+by the committer acting as release manager. While strictly speaking, releases
must only be verified
+on hardware owned and controlled by the committer, for practical reasons it's
best if the packages are
+prepared using such hardware. More information can be found in this
+[FAQ](http://www.apache.org/legal/release-policy.html#owned-controlled-hardware)
+
+# Apache Liminal packages
+
+## Prepare the Apache Liminal Package RC
+
+### Build RC artifacts (both source packages and convenience packages)
+
+The Release Candidate artifacts we vote upon should be the exact ones we vote
against, without any modification than renaming – i.e. the contents of the
files must be the same between voted release canidate and final release.
Because of this the version in the built artifacts that will become the
official Apache releases must not include the rcN suffix.
+
+- Set environment variables
+
+```bash
+# Set Version
+export LIMINAL_BUILD_VERSION=0.0.1RC1
+
+# Example after cloning
+git clone https://github.com/apache/incubating-liminal.git
+cd incubating-liminal
+```
+
+- Tag your release
+
+```bash
+git tag -a ${LIMINAL_BUILD_VERSION} -m "some message"
+
+```
+
+- Clean the checkout: the sdist step below will
+
+```bash
+git clean -fxd
+```
+
+- Tarball the repo
+
+```bash
+git archive --format=tar.gz ${LIMINAL_BUILD_VERSION}
--prefix=apache-liminal-${LIMINAL_BUILD_VERSION}/ -o
apache-liminal-${LIMINAL_BUILD_VERSION}-source.tar.gz`
+```
+
+
+- Generate sdist
+
+ NOTE: Make sure your checkout is clean at this stage - any untracked or
changed files will otherwise be included
+ in the file produced.
+
+```bash
+python setup.py sdist bdist_wheel
+```
+
+- Generate SHA512/ASC (If you have not generated a key yet, generate it by
following instructions on
http://www.apache.org/dev/openpgp.html#key-gen-generate-key)
+
+```bash
+dev/sign.sh apache-liminal-${LIMINAL_BUILD_VERSION}-source.tar.gz
+dev/sign.sh dist/apache-liminal-${LIMINAL_BUILD_VERSION}.tar.gz
+dev/sign.sh dist/apache_liminal-${LIMINAL_BUILD_VERSION}-py3-none-any.whl
+```
+
+- Push Tags
+
+```bashs
+git push origin ${LIMINAL_BUILD_VERSION}
+```
+
+- Push the artifacts to ASF dev dist repo
+```
+# First clone the repo
+svn checkout https://dist.apache.org/repos/dist/dev/liminal liminal-dev
+
+# Create new folder for the release
+cd liminal-dev
+svn mkdir ${LIMINAL_BUILD_VERSION}
+
+# Move the artifacts to svn folder & commit
+mv apache{-,_}liminal-${LIMINAL_BUILD_VERSION}* ${LIMINAL_BUILD_VERSION}/
+cd ${LIMINAL_BUILD_VERSION}
+svn add *
+svn commit -m "Add artifacts for Liminal ${LIMINAL_BUILD_VERSION}"
+```
+
+### Prepare PyPI convenience "snapshot" packages
+
+At this point we have the artefact that we vote on, but as a convenience to
developers we also want to
+publish "snapshots" of the RC builds to pypi for installing via pip. To do
this we need to
+
+- Verify the artifacts that would be uploaded:
+
+```bash
+twine check dist/*
+```
+
+- Upload the package to PyPi's test environment:
+
+```bash
+twine upload -r pypitest dist/*
+```
+
+- Verify that the test package looks good by downloading it and installing it
into a virtual environment. The package download link is available at:
+https://test.pypi.org/project/apache-liminal/#files
+
+- Upload the package to PyPi's production environment:
+`twine upload -r pypi dist/*`
+
+- Again, confirm that the package is available here:
+https://pypi.python.org/pypi/apache-liminal
+
+
+It is important to stress that this snapshot should not be named "release",
and it
+is not supposed to be used by and advertised to the end-users who do not read
the devlist.
+
+## Vote and verify the Apache Liminal release candidate
+
+### Prepare Vote email on the Apache Liminal release candidate
+
+- Use the dev/liminal-jira script to generate a list of Liminal JIRAs that
were closed in the release.
+
+- Send out a vote to the [email protected] mailing list:
+
+Subject:
+```
+[VOTE] Liminal 1.10.2rc3
+```
+
+Body:
+
+```
+Hey all,
+
+I have cut Liminal 1.10.2 RC3. This email is calling a vote on the release,
+which will last for 72 hours. Consider this my (binding) +1.
+
+Liminal 1.10.2 RC3 is available at:
+https://dist.apache.org/repos/dist/dev/liminal/1.10.2rc3/
+
+*apache-liminal-1.10.2rc3-source.tar.gz* is a source release that comes
+with INSTALL instructions.
+*apache-liminal-1.10.2rc3-bin.tar.gz* is the binary Python "sdist" release.
+
+Public keys are available at:
+https://dist.apache.org/repos/dist/release/liminal/KEYS
+
+Only votes from PMC members are binding, but the release manager should
encourage members of the community
+to test the release and vote with "(non-binding)".
+
+The test procedure for PMCs and Contributors who would like to test this RC
are described in
+https://github.com/apache/liminal/blob/master/dev/README.md#vote-and-verify-the-apache-liminal-release-candidate
+
+Please note that the version number excludes the `rcX` string, so it's now
+simply 1.10.2. This will allow us to rename the artifact without modifying
+the artifact checksums when we actually release.
+
+
+Changes since 1.10.2rc2:
+*Bugs*:
+[LIMINAL-3732] ...
+...
+
+
+*Improvements*:
+[LIMINAL-3302] ...
+...
+
+
+*New features*:
+[LIMINAL-2874] ...
+...
+
+
+*Doc-only Change*:
+[LIMINAL-XXX] Fix Minor issues in Documentation
+...
+
+Cheers,
+<your name>
+```
+
+### Verify the release candidate by PMCs (legal)
+
+#### PMC responsibilities
+
+The PMCs should verify the releases in order to make sure the release is
following the
+[Apache Legal Release Policy](http://www.apache.org/legal/release-policy.html).
+
+At least 3 (+1) votes should be recorded in accordance to
+[Votes on Package
Releases](https://www.apache.org/foundation/voting.html#ReleaseVotes)
+
+The legal checks include:
+
+* checking if the packages are present in the right dist folder on svn
+* verifying if all the sources have correct licences
+* verifying if release manager signed the releases with the right key
+* verifying if all the checksums are valid for the release
+
+#### SVN check
+
+The files should be present in the sub-folder of
+[Liminal dist](https://dist.apache.org/repos/dist/dev/incubator/liminal/)
+
+The following files should be present (9 files):
+
+* .tar.gz + .asc + .sha512
+* -.whl + .asc + .sha512
+
+As a PMC you should be able to clone the SVN repository:
+
+```shell script
+ svn co https://dist.apache.org/repos/dist/dev/incubator/liminal
+```
+
+Or update it if you already checked it out:
+
+```shell script
+svn update .
+```
+
+#### Verify the licences
+
+This can be done with the Apache RAT tool.
+
+* Download the latest jar from https://creadur.apache.org/rat/download_rat.cgi
(unpack the sources,
+ the jar is inside)
+* Unpack the .tar.gz to a folder
+* Enter the folder and run the check (point to the place where you extracted
the .jar)
+
+```shell script
+java -jar ../../apache-rat-0.13/apache-rat-0.13.jar -E .rat-excludes -d .
+```
+
+#### Verify the signatures
+
+Make sure you have the key of person signed imported in your GPG. You can find
the valid keys in
+[KEYS](https://dist.apache.org/repos/dist/release/liminal/KEYS).
+
+You can import the whole KEYS file:
+
+```shell script
+gpg --import KEYS
+```
+
+You can also import the keys individually from a keyserver. The below one uses
Kaxil's key and
+retrieves it from the default GPG keyserver
+[OpenPGP.org](https://keys.openpgp.org):
+
+```shell script
+gpg --receive-keys 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+```
+
+You should choose to import the key when asked.
+
+Note that by being default, the OpenPGP server tends to be overloaded often
and might respond with
+errors or timeouts. Many of the release managers also uploaded their keys to
the
+[GNUPG.net](https://keys.gnupg.net) keyserver, and you can retrieve it from
there.
+
+```shell script
+gpg --keyserver keys.gnupg.net --receive-keys
12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+```
+
+Once you have the keys, the signatures can be verified by running this:
+
+```shell script
+for i in *.asc
+do
+ echo "Checking $i"; gpg --verify `basename $i .sha512 `
+done
+```
+
+This should produce results similar to the below. The "Good signature from
..." is indication
+that the signatures are correct. Do not worry about the "not certified with a
trusted signature"
+warning. Most of the certificates used by release managers are self signed,
that's why you get this
+warning. By importing the server in the previous step and importing it via ID
from
+[KEYS](https://dist.apache.org/repos/dist/release/liminal/KEYS) page, you know
that
+this is a valid Key already.
+
+```
+Checking apache-liminal-1.10.12rc4-bin.tar.gz.asc
+gpg: assuming signed data in 'apache-liminal-1.10.12rc4-bin.tar.gz'
+gpg: Signature made sob, 22 sie 2020, 20:28:28 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <[email protected]>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+Checking apache_liminal-1.10.12rc4-py2.py3-none-any.whl.asc
+gpg: assuming signed data in 'apache_liminal-1.10.12rc4-py2.py3-none-any.whl'
+gpg: Signature made sob, 22 sie 2020, 20:28:31 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <[email protected]>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+Checking apache-liminal-1.10.12rc4-source.tar.gz.asc
+gpg: assuming signed data in 'apache-liminal-1.10.12rc4-source.tar.gz'
+gpg: Signature made sob, 22 sie 2020, 20:28:25 CEST
+gpg: using RSA key 12717556040EEF2EEAF1B9C275FCCD0A25FA0E4B
+gpg: Good signature from "Kaxil Naik <[email protected]>" [unknown]
+gpg: WARNING: This key is not certified with a trusted signature!
+gpg: There is no indication that the signature belongs to the owner.
+Primary key fingerprint: 1271 7556 040E EF2E EAF1 B9C2 75FC CD0A 25FA 0E4B
+```
+
+#### Verify the SHA512 sum
+
+Run this:
+
+```shell script
+for i in *.sha512
+do
+ echo "Checking $i"; gpg --print-md SHA512 `basename $i .sha512 ` | diff -
$i
+done
+```
+
+You should get output similar to:
+
+```
+Checking apache_liminal-1.10.12rc4-py3-none-any.whl.sha512
+Checking apache-liminal-1.10.12rc4.tar.gz.sha512
+```
+
+### Verify if the release candidate "works" by Contributors
+
+This can be done (and we encourage to) by any of the Contributors. In fact,
it's best if the
+actual users of Apache Liminal test it in their own staging/test
installations. Each release candidate
+is available on PyPI apart from SVN packages, so everyone should be able to
install
+the release candidate version of Liminal via simply (<VERSION> is 1.10.12 for
example, and <X> is
+release candidate number 1,2,3,....).
+
+```shell script
+pip install apache-liminal==<LIMINAL_BUID_VERSION>rc<X>
+```
+
+## Publish the final Apache Liminal release
+
+### Summarize the voting for the Apache Liminal release
+
+Once the vote has been passed, you will need to send a result vote to
[email protected]:
+
+Subject:
+```
+[RESULT][VOTE] Liminal 1.10.2rc3
+```
+
+Message:
+
+```
+Hello,
+
+Apache Liminal 1.10.2 (based on RC3) has been accepted.
+
+3 “+1” binding votes received:
+....
+
+3 "+1" non-binding votes received:
+
+...
+
+Vote thread:
+...
+
+I'll continue with the release process, and the release announcement will
follow shortly.
+
+Cheers,
+<your name>
+```
+
+
+### Publish release to SVN
+
+You need to migrate the RC artifacts that passed to this repository:
+https://dist.apache.org/repos/dist/release/liminal/
+(The migration should include renaming the files so that they no longer have
the RC number in their filenames.)
+
+The best way of doing this is to svn cp between the two repos (this avoids
having to upload the binaries again, and gives a clearer history in the svn
commit logs):
+
+```shell script
+# First clone the repo
+export RC=1.10.4rc5
+export VERSION=${RC/rc?/}
+svn checkout https://dist.apache.org/repos/dist/release/liminal liminal-release
+
+# Create new folder for the release
+cd liminal-release
+svn mkdir ${VERSION}
+cd ${VERSION}
+
+# Move the artifacts to svn folder & commit
+for f in ../../liminal-dev/$RC/*; do svn cp $f ${$(basename $f)/rc?/}; done
+svn commit -m "Release Liminal ${VERSION} from ${RC}"
+
+# Remove old release
+# http://www.apache.org/legal/release-policy.html#when-to-archive
+cd ..
+export PREVIOUS_VERSION=1.10.1
+svn rm ${PREVIOUS_VERSION}
+svn commit -m "Remove old release: ${PREVIOUS_VERSION}"
+```
+
+Verify that the packages appear in
[liminal](https://dist.apache.org/repos/dist/release/liminal/)
+
+### Prepare PyPI "release" packages
+
+At this point we release an official package:
+
+- Build the package:
+
+ ```shell script
+ python setup.py sdist bdist_wheel`
+ ```
+
+- Verify the artifacts that would be uploaded:
+
+ ```shell script
+ twine check dist/*`
+ ```
+
+- Upload the package to PyPi's test environment:
+
+ ```shell script
+ twine upload -r pypitest dist/*
+ ```
+
+- Verify that the test package looks good by downloading it and installing it
into a virtual environment.
+ The package download link is available at:
https://test.pypi.org/project/apache-liminal/#files
+
+- Upload the package to PyPi's production environment:
+
+ ```shell script
+ twine upload -r pypi dist/*
+ ```
+
+- Again, confirm that the package is available here:
https://pypi.python.org/pypi/apache-liminal
+
+### Update CHANGELOG.md
+
+- Get a diff between the last version and the current version:
+
+ ```shell script
+ $ git log 1.8.0..1.9.0 --pretty=oneline
+ ```
+- Update CHANGELOG.md with the details, and commit it.
+
+### Notify developers of release
+
+- Notify [email protected] (cc'ing [email protected] and
[email protected]) that
+the artifacts have been published:
+
+Subject:
+```shell script
+cat <<EOF
+Liminal ${VERSION} is released
+EOF
+```
+
+Body:
+```shell script
+cat <<EOF
+Dear Liminal community,
+
+I'm happy to announce that Liminal ${VERSION} was just released.
+
+The source release, as well as the binary "sdist" release, are available
+here:
+
+https://dist.apache.org/repos/dist/release/liminal/${VERSION}/
+
+We also made this version available on PyPi for convenience (`pip install
apache-liminal`):
+
+https://pypi.python.org/pypi/apache-liminal
+
+The documentation is available on:
+https://liminal.apache.org/
+https://liminal.apache.org/1.10.2/
+https://liminal.readthedocs.io/en/1.10.2/
+https://liminal.readthedocs.io/en/stable/
+
+Find the CHANGELOG here for more details:
+
+https://liminal.apache.org/changelog.html#liminal-1-10-2-2019-01-19
+
+Cheers,
+<your name>
+EOF
+```
\ No newline at end of file
diff --git a/MANIFEST.in b/dev/sign.sh
old mode 100644
new mode 100755
similarity index 60%
copy from MANIFEST.in
copy to dev/sign.sh
index 6e3a162..2456271
--- a/MANIFEST.in
+++ b/dev/sign.sh
@@ -1,4 +1,4 @@
-#
+#!/usr/bin/env bash
# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements. See the NOTICE file
# distributed with this work for additional information
@@ -9,13 +9,22 @@
#
# http://www.apache.org/licenses/LICENSE-2.0
#
-# Unless required bgit y applicable law or agreed to in writing,
+# Unless required by applicable law or agreed to in writing,
# software distributed under the License is distributed on an
# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
# KIND, either express or implied. See the License for the
# specific language governing permissions and limitations
# under the License.
+set -euo pipefail
+
+# Use this to sign the tar balls generated from
+# python setup.py sdist --formats=gztar
+# ie. sign.sh <my_tar_ball>
+# you will still be required to type in your signing key password
+# or it needs to be available in your keychain
-include scripts/*
-include requirements-airflow.txt
-recursive-include liminal/build/ *
\ No newline at end of file
+for name in "${@}"
+do
+ gpg --armor --output "${name}.asc" --detach-sig "${name}"
+ gpg --print-md SHA512 "${name}" > "${name}.sha512"
+done
\ No newline at end of file
diff --git a/liminal/core/environment.py b/liminal/core/environment.py
index f335ce9..8491490 100644
--- a/liminal/core/environment.py
+++ b/liminal/core/environment.py
@@ -17,11 +17,14 @@
# under the License.
import os
+import subprocess
+from pathlib import Path
DEFAULT_DAGS_ZIP_NAME = 'liminal.zip'
DEFAULT_LIMINAL_HOME = os.path.expanduser('~/liminal_home')
DEFAULT_PIPELINES_SUBDIR = "pipelines"
LIMINAL_HOME_PARAM_NAME = "LIMINAL_HOME"
+LIMINAL_VERSION_PARAM_NAME = 'LIMINAL_VERSION'
def get_liminal_home():
@@ -36,3 +39,21 @@ def get_dags_dir():
# if we are inside airflow, we will take it from the configured dags folder
base_dir = os.environ.get("AIRFLOW__CORE__DAGS_FOLDER", get_liminal_home())
return os.path.join(base_dir, DEFAULT_PIPELINES_SUBDIR)
+
+
+def get_liminal_version():
+ result = os.environ.get(LIMINAL_VERSION_PARAM_NAME, None)
+ if not result:
+ output = subprocess.run(['pip freeze | grep \'apache-liminal\''],
capture_output=True, env=os.environ, shell=True)
+ pip_res = output.stdout.decode('UTF-8').strip()
+ value = None
+ if not pip_res:
+ value = 'apache-liminal'
+ if ' @ ' in pip_res:
+ value = pip_res[pip_res.index(' @ ')+3:]
+ else:
+ value= pip_res
+ print(f'LIMINAL_VERSION not set. Setting it to currently installed
version: {value}')
+ os.environ[LIMINAL_VERSION_PARAM_NAME] = value
+ return os.environ.get(LIMINAL_VERSION_PARAM_NAME, 'apache-liminal')
+
diff --git a/scripts/Dockerfile-airflow b/scripts/Dockerfile-airflow
new file mode 100644
index 0000000..4abb2a4
--- /dev/null
+++ b/scripts/Dockerfile-airflow
@@ -0,0 +1,15 @@
+FROM apache/airflow
+
+ARG LIMINAL_VERSION='apache-liminal'
+ARG DAG_FOLDER='/opt/airflow/dags/'
+ADD scripts/* ${DAG_FOLDER}
+
+RUN ls -ls ${DAG_FOLDER}
+ADD liminal/runners/airflow/dag/liminal_dags.py ${DAG_FOLDER}
+
+USER airflow
+
+RUN pip --disable-pip-version-check install -r
/opt/airflow/dags/requirements-airflow.txt --user
+RUN pip --disable-pip-version-check install --user --no-deps ${LIMINAL_VERSION}
+
+
diff --git a/scripts/__init__.py b/scripts/__init__.py
new file mode 100644
index 0000000..e69de29
diff --git a/scripts/docker-compose.yml b/scripts/docker-compose.yml
index 8cab3ea..17e7faa 100644
--- a/scripts/docker-compose.yml
+++ b/scripts/docker-compose.yml
@@ -6,34 +6,85 @@
- POSTGRES_USER=airflow
- POSTGRES_PASSWORD=airflow
- POSTGRES_DB=
-
+ restart: always
ports:
- "5432:5432"
+ volumes:
+ - ${LIMINAL_HOME}/db:/var/lib/postgresql/data
logging:
options:
max-size: 10m
max-file: "3"
webserver:
- image: puckel/docker-airflow:1.10.9
- restart: always
+ build:
+ context: .
+ dockerfile: scripts/Dockerfile-airflow
+ args:
+ LIMINAL_VERSION: ${LIMINAL_VERSION}
+ image: liminal-airflow
depends_on:
- postgres
environment:
- LOAD_EX=n
- EXECUTOR=Local
- - AIRFLOW__CORE__DAGS_FOLDER=/usr/local/airflow/dags
+ - KUBECONFIG=/home/airflow/kube/config
- AIRFLOW__WEBSERVER__WORKERS=1
+ - LIMINAL_HOME=/opt/airflow/dags
+ - AIRFLOW__CORE__LOAD_DEFAULT_CONNECTIONS=False
+ -
AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgres+psycopg2://airflow:airflow@postgres:5432/airflow
+ -
AIRFLOW_CONN_METADATA_DB=postgres+psycopg2://airflow:airflow@postgres:5432/airflow
+ - AIRFLOW_VAR__METADATA_DB_SCHEMA=airflow
logging:
options:
max-size: 10m
max-file: "3"
volumes:
- - ${LIMINAL_HOME}:/usr/local/airflow/dags
- - ${HOME}/.kube:/usr/local/airflow/.kube
+ - ${LIMINAL_HOME}:/opt/airflow/dags
+ - ${LIMINAL_HOME}/logs:/opt/airflow/logs
+ - ${HOME}/.kube:/home/airflow/kube
ports:
- "8080:8080"
- command: webserver
+ command: "webserver"
+ healthcheck:
+ test: ["CMD-SHELL", "[ -f
/usr/local/airflow/airflow-webserver.pid ]"]
+ interval: 30s
+ timeout: 30s
+ retries: 3
+
+
+ scheduler:
+ build:
+ context: .
+ dockerfile: scripts/Dockerfile-airflow
+ args:
+ LIMINAL_VERSION: ${LIMINAL_VERSION}
+ image: liminal-airflow
+ depends_on:
+ - postgres
+ - webserver
+ environment:
+ - LOAD_EX=n
+ - EXECUTOR=Local
+ - AIRFLOW__WEBSERVER__WORKERS=1
+ - KUBECONFIG=/home/airflow/kube/config
+ - LIMINAL_HOME=/opt/airflow/dags
+ - AIRFLOW__CORE__LOAD_DEFAULT_CONNECTIONS=False
+ -
AIRFLOW__CORE__SQL_ALCHEMY_CONN=postgres+psycopg2://airflow:airflow@postgres:5432/airflow
+ -
AIRFLOW_CONN_METADATA_DB=postgres+psycopg2://airflow:airflow@postgres:5432/airflow
+ - AIRFLOW_VAR__METADATA_DB_SCHEMA=airflow
+ - AIRFLOW__SCHEDULER__SCHEDULER_HEARTBEAT_SEC=10
+ logging:
+ options:
+ max-size: 10m
+ max-file: "3"
+ volumes:
+ - ${LIMINAL_HOME}:/opt/airflow/dags
+ - ${LIMINAL_HOME}/logs:/opt/airflow/logs
+ - ${HOME}/.kube:/home/airflow/kube
+ ports:
+ - "8793:8793"
+ command: "scheduler"
healthcheck:
test: ["CMD-SHELL", "[ -f
/usr/local/airflow/airflow-webserver.pid ]"]
interval: 30s
diff --git a/scripts/liminal b/scripts/liminal
index 5ea21ad..958a00e 100755
--- a/scripts/liminal
+++ b/scripts/liminal
@@ -18,6 +18,7 @@
# specific language governing permissions and limitations
# under the License.
import os
+import pathlib
import shutil
import subprocess
import sys
@@ -27,6 +28,13 @@ import click
from liminal.build import liminal_apps_builder
from liminal.core import environment
from liminal.core.util import files_util
+from liminal.runners.airflow import dag
+import scripts as s
+try:
+ import importlib.resources as pkg_resources
+except ImportError:
+ # Try backported to PY<37 `importlib_resources`.
+ import importlib_resources as pkg_resources
@click.group()
@@ -51,20 +59,44 @@ def build(path):
liminal_apps_builder.build_liminal_apps(path)
-def deploy_liminal_core_internal():
- click.echo("WARN: refreshing liminal core package")
- liminal_home = environment.get_liminal_home()
- subprocess.call([f'package.sh {liminal_home}'], shell=True)
-
+def deploy_liminal_core_internal(clean):
+ with pkg_resources.path(dag, 'liminal_dags.py') as p:
+ dags_path=p
+ os.makedirs(environment.get_dags_dir(), exist_ok=True)
+ dag_target_file = os.path.join(environment.get_liminal_home(),
'liminal_dags.py')
+ shutil.copyfile(dags_path, dag_target_file)
+ #initialize the env. variable which indicates to the docke compose which
liminal to install in airflow docker
+ liminal_version = environment.get_liminal_version()
+ # if liminal is installed from local file - teh developer needs to put it
in the /scripts folder, in which case
+ # it will end up inside the container during build
+ if liminal_version.find("file://")>-1:
+ local_file_name = os.path.basename(liminal_version)
+ full_path = os.path.join('/opt/airflow/dags',local_file_name)
+ print(f'Liminal was installed locally, changing the LIMINAL_VERSION
parameter to {full_path}')
+ os.environ[environment.LIMINAL_VERSION_PARAM_NAME]= full_path
+ if clean:
+ docker_compose_command('down', ['--rmi', 'local'])
+ docker_compose_command('build', ['--no-cache'])
+ docker_compose_command('run', ['webserver', 'resetdb', '-y'])
+ docker_compose_command('run', ['webserver','initdb'])
+ docker_compose_command('down', [])
+
+def docker_compose_command(command_name, args):
+ docker_compose_path, project_dir = get_docker_compose_paths()
+ concated_args = ' '.join(args)
+ subprocess.call(
+ [f'docker-compose -f "{docker_compose_path}" -p liminal
--project-directory {project_dir} {command_name} {concated_args}'],
+ env=os.environ, shell=True)
@cli.command("deploy", short_help="deploys your liminal.yaml files to
$LIMINAL_HOME folder")
@click.option('--path', default=os.getcwd(), help="folder containing
liminal.yaml files")
-def deploy_liminal_apps(path):
[email protected]('--clean', is_flag=True, default=False, help="re-deploy liminal
airflow components from scratch")
+def deploy_liminal_apps(path, clean):
click.echo("deploying liminal yaml files")
liminal_home = environment.get_liminal_home()
os.makedirs(liminal_home, exist_ok=True)
os.makedirs(environment.get_dags_dir(), exist_ok=True)
- deploy_liminal_core_internal()
+ deploy_liminal_core_internal(clean)
config_files = files_util.find_config_files(path)
for config_file in config_files:
click.echo(f"deploying liminal file: {config_file}")
@@ -80,10 +112,19 @@ def start():
if docker_is_running():
# initialize liminal home by default
environment.get_liminal_home()
+ docker_compose_path, project_dir = get_docker_compose_paths()
result = subprocess.call(
- [f'docker-compose -f
"{environment.get_liminal_home()}/docker-compose.yml" up'],
+ [f'docker-compose -f "{docker_compose_path}" --project-directory
{project_dir} up'],
env=os.environ, shell=True)
+def get_docker_compose_paths():
+ with pkg_resources.path(s, 'docker-compose.yml') as p:
+ docker_compose_path = p
+ project_dir = pathlib.Path(docker_compose_path).parent.parent.absolute()
+ print(f"starting docker compose from path {project_dir}")
+ return docker_compose_path, project_dir
+
+
if __name__ == '__main__':
cli()
diff --git a/scripts/package.sh b/scripts/package.sh
deleted file mode 100755
index de88f41..0000000
--- a/scripts/package.sh
+++ /dev/null
@@ -1,60 +0,0 @@
-#!/bin/bash
-#
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements. See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership. The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License. You may obtain a copy of the License at
-#
-# http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required bgit y applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied. See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-echo $1
-target_path="$1"
-
-echo "running from " $(PWD)
-echo "target path for liminal zip file is " $target_path
-
-echo "cleaning up the temp dirs $TMPDIR/liminal_build"
-rm -rf $TMPDIR/liminal_build-*/
-
-tmp_dir=$(mktemp -d -t liminal_build-)
-echo "creating temp directory $tmp_dir"
-
-docker_build_dir=$tmp_dir/docker_build
-mkdir -p $docker_build_dir
-echo "docker build directory :"$docker_build_dir
-
-mkdir $docker_build_dir/"zip_content"
-mkdir $docker_build_dir/"dags"
-
-#copy the content of the user project into the build folder
-rsync -a --exclude 'venv' $(PWD)/ $docker_build_dir/zip_content/
-
-# perform installation of external pacakges (framework-requirements and
user-requirements)
-# this is done inside a docker to 1) avoid requiring the user to install
stuff, and 2) to create a platform-compatible
-# package (install the native libraries in a flavour suitable for the docker
in which airflow runs, and not user machine)
-
-docker run --rm --name liminal_build -v
/private/"$docker_build_dir":/home/liminal/tmp --entrypoint="" -u 0 \
- puckel/docker-airflow:1.10.9 /bin/bash -c "cd
/home/liminal/tmp/zip_content &&
- pip install --no-deps --target=\"/home/liminal/tmp/zip_content\"
liminal==0.0.2dev5 &&
- rsync -avzh --ignore-errors
/home/liminal/tmp/zip_content/liminal-resources/* /home/liminal/tmp/zip_content/
- pip install --target=\"/home/liminal/tmp/zip_content\" -r
/home/liminal/tmp/zip_content/requirements-airflow.txt &&
- pip install --target=\"/home/liminal/tmp/zip_content\" -r
/home/liminal/tmp/zip_content/requirements.txt"
-
-
-# zip the content per
https://airflow.apache.org/docs/stable/concepts.html#packaged-dags
-cd $docker_build_dir/zip_content
-mv docker-compose.yml $target_path
-rm __init__.py
-
-zip -r ../dags/liminal.zip .
-cp ../dags/liminal.zip $target_path
diff --git a/requirements-airflow.txt b/scripts/requirements-airflow.txt
similarity index 100%
rename from requirements-airflow.txt
rename to scripts/requirements-airflow.txt
diff --git a/setup.py b/setup.py
index 0cf0e64..69d9a85 100644
--- a/setup.py
+++ b/setup.py
@@ -28,10 +28,12 @@ with open('requirements.txt') as f:
requirements = f.read().splitlines()
print(requirements)
+
setuptools.setup(
- name="liminal",
- version=os.environ["LIMINAL_BUILD_VERSION"],
- author="liminal team",
+ name="apache-liminal",
+ version=os.environ.get("LIMINAL_BUILD_VERSION",
os.environ.get('LIMINAL_VERSION', None)),
+ author="[email protected]",
+ author_email='[email protected]',
description="A package for authoring and deploying machine learning
workflows",
long_description=long_description,
long_description_content_type="text/markdown",
@@ -45,9 +47,6 @@ setuptools.setup(
license='Apache License, Version 2.0',
python_requires='>=3.6',
install_requires=requirements,
- scripts=['scripts/liminal', 'scripts/package.sh'],
- include_package_data=True,
- data_files=[('liminal-resources', ['scripts/docker-compose.yml',
- 'requirements-airflow.txt',
-
'liminal/runners/airflow/dag/liminal_dags.py'])]
-)
+ scripts=['scripts/liminal'],
+ include_package_data=True)
+
