It's a really tough balancing act. On the one hand I agree that ideally we don't want to force people to user k8s's (ridiculously) verbose model. On the other hand the other solution is to hand-craft requests like this <https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/kubernetes/kubernetes_request_factory/kubernetes_request_factory.py> which I think will be really hard to maintain in the long-term (and I actually want to propose refactoring out in the next few days).
Perhaps what we can do is offer a "PodBuilder" and "DeploymentBuilder" class that allows users to enter in basic info, gives them back an official model to plug in to the operator, and if they want to customize it they would have the option to take the V1Pod or the V1Deployment object and customize as necessary. This would also future-proof both operators, as the system would be much easier to update as k8s starts offering new features. I also agree that a helm operator sounds like a pretty good idea, though no reason these two efforts can't be done in parallel :) On Thu, Jul 5, 2018 at 5:48 AM James Meickle <jmeic...@quantopian.com.invalid> wrote: > I think it's going to be an antipattern to write Python configuration in > Airflow to configure a Kubernetes deployment since even a "simple" > deployment would likely be more classes/objects than the DAG itself. I do > like the idea of having a more featured operator than the PodOperator, but > if I were to choose a second operator to implement, it would definitely be > a Helm operator instead of a Kube API operator. > > On Tue, Jul 3, 2018 at 12:47 PM, Daniel Imberman < > daniel.imber...@gmail.com> > wrote: > > > An example of creating a deployment using the k8s model architecture can > be > > found here: > > https://github.com/kubernetes-client/python/blob/master/ > > examples/deployment_examples.py > > > > def create_deployment_object(): > > # Configureate Pod template container > > container = client.V1Container( > > name="nginx", > > image="nginx:1.7.9", > > ports=[client.V1ContainerPort(container_port=80)]) > > # Create and configurate a spec section > > template = client.V1PodTemplateSpec( > > metadata=client.V1ObjectMeta(labels={"app": "nginx"}), > > spec=client.V1PodSpec(containers=[container])) > > # Create the specification of deployment > > spec = client.ExtensionsV1beta1DeploymentSpec( > > replicas=3, > > template=template) > > # Instantiate the deployment object > > deployment = client.ExtensionsV1beta1Deployment( > > api_version="extensions/v1beta1", > > kind="Deployment", > > metadata=client.V1ObjectMeta(name=DEPLOYMENT_NAME), > > spec=spec) > > > > return deployment > > > > This would involve a more k8s knowledge from the user, but would have the > > massive benefit that we would not have to maintain new features as the > k8s > > API updates (Would simply update version). A user would have to supply > is a > > deployment object and possibly a "success criteria" (i.e. an endpoint to > > test). > > > > Conversely, we could make the API a bit easier by only requiring a spec > and > > an optional metadata, after which we would handle a lot of the > boilerplate. > > > > On Tue, Jul 3, 2018 at 9:20 AM Daniel Imberman < > daniel.imber...@gmail.com> > > wrote: > > > > > Hi all, > > > > > > Enclosed is a proposal for a kubernetes deployment management > operator. I > > > think this would be a good addition to current k8s offerings s.t. users > > can > > > actually launch persistent applications from airflow DAGs. > > > > > > * What?* > > > Add an operator that monitors a k8s deployment, declaring the task > > > complete on proper deployment/accessibility of endpoint > > > > > > * Why?* > > > Not all tasks are single pods, sometimes you would want to run one > task > > > that launches a service, and then a second task that smoke tests/stress > > > tests/ > > > gives state to an application deployment. This would give airflow > extra > > > functionality as a CI/CD tool in the k8s ecosystem. > > > > > > * Fix:* > > > Create a modification (or extension) of the k8sPodOperator that can > > > handle entire deployments (possibly using the k8s model API to ensure > > > full flexibility of users). > > > > > > Thank you. > > > > > >
