Hi everyone, I'm investigating the possibility for our organization to use Airflow for workflow management.
Some requirements on our side regard resource management, and in particular the possibility for the system to run tasks on top of Apache Mesos. Airflow partially satisfies our requirements in that regard, meaning that after having a look at the docs and code, it appears to me (correct me if I'm wrong) that resources are determined for the whole system (via configuration) and cannot be determined on a per-task basis. We'd need this because some of our jobs are quite lightweight while others may require a lot of resources, making it a "one-size-fits-all" configuration quite wasteful. I had a look at the AirflowMesosScheduler and MesosExecutor and thought it would be nice to add this feature and perhaps I can add it myself. What I would need is some guidance on how to make this fit into the overall system design: is there an established way to explicitly ask for resources for a specific task in the DAG? If not, what could be a possible way to introduce it? And if this reveals itself to be outside of the scope of Airflow, how do you think I can make it meet our requirement? Thanks in advance. P.S.: if by any chance some of you are on the Mesos mailing list as well, you may know that I'm having issues in making Airflow run successfully using Mesos due to missing Python packages. I'm not sure whether this mailing list is an appropriate place for users to get help. If so, I could probably share that post here as well. Thanks! -- Stefano Baghino | TERALYTICS *software engineer* Teralytics AG | Zollstrasse 62 | 8005 Zurich | Switzerland phone: +41 43 508 24 57 email: stefano.bagh...@teralytics.ch www.teralytics.net Company registration number: CH-020.3.037.709-7 | Trade register Canton Zurich Board of directors: Georg Polzer, Luciano Franceschina, Mark Schmitz, Yann de Vries This e-mail message contains confidential information which is for the sole attention and use of the intended recipient. Please notify us at once if you think that it may not be intended for you and delete it immediately.
