@chris: Thank you! My wiki name is dimberman. @gerard: I've started writing out my reply but there's a fair amount to respond to so I'll need a few minutes :).
On Wed, Jul 5, 2017 at 1:17 PM Chris Riccomini <[email protected]> wrote: > @daniel, what's your wiki username? I can grant you access. > > On Wed, Jul 5, 2017 at 12:35 PM, Gerard Toonstra <[email protected]> > wrote: > > > Hey Daniel, > > > > Great work. We're looking at running airflow on AWS ECS inside docker > > containers and making great progress on this. > > We use redis and RDS as managed services to form a comms backbone and > then > > just spawn webserver, scheduler, worker and flower containers > > as needed on ECS. We deploy dags using an Elastic File System (shared > > across all instances), which then map this read-only into the docker > > container. > > We're now evaluating this setup going forward in more earnest. > > > > Good idea to use queues to separate dependencies or some other concerns > > (high-mem pods?), there are many ways this way that it's possible to > > customize where and on which hw a DAG is going to run. We're looking at > > Cycle scaling to temporarily increase resources in a morning run and > create > > larger worker containers for data science tasks and perhaps some other > > tasks. > > > > > > - In terms of tooling: The current airflow config is somewhat static in > > the sense that it does not reconfigure itself to the (now) dynamic > > environment. > > You'd think that airflow should have to query the environment to figure > > out parallellism instead of statically specifying this. > > > > - Sometimes DAGs import hooks or operators that import dependencies at > the > > top. The only reason, (I think) that a scheduler needs to physically > > import and parse a DAG is because there may be dynamically built > elements > > within the DAG. If there wouldn't be static elements, it is theoretically > > possible to optimize this. Your PDF sort of hints towards a system > > where a worker where a DAG will eventually run could parse the DAG and > > report > > back a meta description of the DAG, which could simplify and optimize > > performance of the scheduler at the cost of network roundtrips. > > > > - About redeploying instances: We see this as a potential issue for our > > setup. My take is that jobs simply shouldn't take that much time in > > principle to start with, > > which avoids having to worry about this. If that's ridiculous, > shouldn't > > it be a concern of the environment airflow runs in rather than airflow > > itself? I.e.... > > further tool out kubernetes CLI's / operators to query the environment > > to plan/deny/schedule this kind of work automatically. Beacuse k8s was > > probably > > built from the perspective of handling short-running queries, running > > anything long-term on that is going to naturally compete with the > > architecture. > > > > - About failures and instances disappearing on failure: it's not > desirable > > to keep those instances around for a long time, we really do need to > depend > > on > > client logging and other services available to tell us what happened. > > The difference in thinking is that a pod/container is just a temporary > > thing that runs a job > > and we should be interested in how the job did vs. how the > container/pod > > ran this. From my little experience with k8s though, I do see that it > tends > > to > > get rid of everything a little bit too quick on failure. One thing you > > could look into is to log onto a commonly shared volume with a specific > > 'key' for that container, > > so you can always refer back to the important log file and fish this > > out, with measures to clean up the shared filesystem on a regular basis. > > > > - About rescaling and starting jobs: it doesn't come for free as you > > mention. I think it's a great idea to be able to scale out on busy > > intervals (we intend to just use cycle scaling here), > > but a hint towards what policy or scaling strategy you intend to use on > > k8s is welcome there. > > > > > > Gerard > > > > > > On Wed, Jul 5, 2017 at 8:43 PM, Daniel Imberman < > [email protected] > > > > > wrote: > > > > > @amit > > > > > > I've added the proposal to the PR for now. Should make it easier for > > people > > > to get to it. Will delete once I add it to the wiki. > > > > https://github.com/bloomberg/airflow/blob/29694ae9903c4dad3f18fb8eb767c4 > > > 922dbef2e8/dimberman-KubernetesExecutorProposal-050717-1423-36.pdf > > > > > > Daniel > > > > > > On Wed, Jul 5, 2017 at 11:36 AM Daniel Imberman < > > [email protected] > > > > > > > wrote: > > > > > > > Hi Amit, > > > > > > > > For now the design doc is included as an attachment to the original > > > email. > > > > Once I am able to get permission to edit the wiki I would like add it > > > there > > > > but for now I figured that this would get the ball rolling. > > > > > > > > > > > > Daniel > > > > > > > > > > > > On Wed, Jul 5, 2017 at 11:33 AM Amit Kulkarni <[email protected]> > wrote: > > > > > > > >> Hi Daniel, > > > >> > > > >> I don't see link to design PDF. > > > >> > > > >> > > > >> Amit Kulkarni > > > >> Site Reliability Engineer > > > >> Mobile: (716)-352-3270 <(716)%20352-3270> <(716)%20352-3270> > > > >> > > > >> Payments partner to the platform economy > > > >> > > > >> On Wed, Jul 5, 2017 at 11:25 AM, Daniel Imberman < > > > >> [email protected]> > > > >> wrote: > > > >> > > > >> > Hello Airflow community! > > > >> > > > > >> > My name is Daniel Imberman, and I have been working on behalf of > > > >> Bloomberg > > > >> > LP to create an airflow kubernetes executor/operator. We wanted to > > > allow > > > >> > for maximum throughput/scalability, while keeping a lot of the > > > >> kubernetes > > > >> > details abstracted away from the users. Below I have a link to the > > WIP > > > >> PR > > > >> > and the PDF of the initial proposal. If anyone has any > > > >> comments/questions I > > > >> > would be glad to discuss this feature further. > > > >> > > > > >> > Thank you, > > > >> > > > > >> > Daniel > > > >> > > > > >> > https://github.com/apache/incubator-airflow/pull/2414 > > > >> > > > > >> > > > > > > > > > >
