An example of creating a deployment using the k8s model architecture can be
found here:
https://github.com/kubernetes-client/python/blob/master/examples/deployment_examples.py
def create_deployment_object():
# Configureate Pod template container
container = client.V1Container(
name="nginx",
image="nginx:1.7.9",
ports=[client.V1ContainerPort(container_port=80)])
# Create and configurate a spec section
template = client.V1PodTemplateSpec(
metadata=client.V1ObjectMeta(labels={"app": "nginx"}),
spec=client.V1PodSpec(containers=[container]))
# Create the specification of deployment
spec = client.ExtensionsV1beta1DeploymentSpec(
replicas=3,
template=template)
# Instantiate the deployment object
deployment = client.ExtensionsV1beta1Deployment(
api_version="extensions/v1beta1",
kind="Deployment",
metadata=client.V1ObjectMeta(name=DEPLOYMENT_NAME),
spec=spec)
return deployment
This would involve a more k8s knowledge from the user, but would have the
massive benefit that we would not have to maintain new features as the k8s
API updates (Would simply update version). A user would have to supply is a
deployment object and possibly a "success criteria" (i.e. an endpoint to
test).
Conversely, we could make the API a bit easier by only requiring a spec and
an optional metadata, after which we would handle a lot of the boilerplate.
On Tue, Jul 3, 2018 at 9:20 AM Daniel Imberman <[email protected]>
wrote:
> Hi all,
>
> Enclosed is a proposal for a kubernetes deployment management operator. I
> think this would be a good addition to current k8s offerings s.t. users can
> actually launch persistent applications from airflow DAGs.
>
> * What?*
> Add an operator that monitors a k8s deployment, declaring the task
> complete on proper deployment/accessibility of endpoint
>
> * Why?*
> Not all tasks are single pods, sometimes you would want to run one task
> that launches a service, and then a second task that smoke tests/stress
> tests/
> gives state to an application deployment. This would give airflow extra
> functionality as a CI/CD tool in the k8s ecosystem.
>
> * Fix:*
> Create a modification (or extension) of the k8sPodOperator that can
> handle entire deployments (possibly using the k8s model API to ensure
> full flexibility of users).
>
> Thank you.
>
