As a follow up to my message this morning, I've been able to to finally make the whole thing work: Airflow is dockerized and deployed on Marathon and uses Mesos as an executor. To solve the problem, I copied a small portion of the official Mesos image to install Mesos on the container. I then copied the installed Python bindings to a directory where those could be found at runtime. I have updated the original gist and here <https://gist.github.com/stefanobaghino/50b2919eef6acb108d3647ccf570d937/revisions> you can see what I changed.
On Thu, Aug 3, 2017 at 12:13 PM, Hitec Adam Cecile <[email protected]> wrote: > Hello, > > There's a problem with the build system not guessing the WHL filename > correctly. Look at the JIRA bug, I reported it with a fix. > > Regards, Adam > > > Le 3 août 2017 08:47:38 GMT+02:00, Stefano Baghino < > [email protected]> a écrit : >> >> Hello everyone! >> >> I'm trying to use Apache Mesos to run jobs automated with Apache Airflow >> (workflow management). >> >> In order to run Airflow on top of Marathon (which we need to have Airflow >> automatically restart in case of failure), I created a container >> (Dockerfile here >> <https://gist.github.com/stefanobaghino/50b2919eef6acb108d3647ccf570d937#file-dockerfile>) >> and then provided the necessary environment variables to let it know to: 1) >> use Mesos and 2) where to find the Mesos master. The problem arises when I >> launch the container: apparently, `mesos.native` is missing (stacktrace >> here >> <https://gist.github.com/stefanobaghino/50b2919eef6acb108d3647ccf570d937#file-stderr>). >> I tried installing `mesos` or `mesos.native` via pip in the container but >> both appear to not be there (`mesos` is actually there when you search, but >> there is no actual package to install as it's just a placeholder). >> >> Is my diagnosis correct? How can I install the required Mesos packages? >> Both the container definition and the error stacktrace are available in >> this gist >> <https://gist.github.com/stefanobaghino/50b2919eef6acb108d3647ccf570d937#file-stderr>. >> Thanks! >> >> -- >> Stefano Baghino | TERALYTICS >> *software engineer* >> >> Teralytics AG | Zollstrasse 62 | 8005 Zurich | Switzerland >> phone: +41 43 508 24 57 >> email: [email protected] >> www.teralytics.net >> >> Company registration number: CH-020.3.037.709-7 | Trade register Canton >> Zurich >> Board of directors: Georg Polzer, Luciano Franceschina, Mark Schmitz, >> Yann de Vries >> >> This e-mail message contains confidential information which is for the >> sole attention and use of the intended recipient. Please notify us at once >> if you think that it may not be intended for you and delete it immediately. >> > > -- > Envoyé de mon appareil Android avec Courriel K-9 Mail. Veuillez excuser ma > brièveté. > -- Stefano Baghino | TERALYTICS *software engineer* Teralytics AG | Zollstrasse 62 | 8005 Zurich | Switzerland phone: +41 43 508 24 57 email: [email protected] www.teralytics.net Company registration number: CH-020.3.037.709-7 | Trade register Canton Zurich Board of directors: Georg Polzer, Luciano Franceschina, Mark Schmitz, Yann de Vries This e-mail message contains confidential information which is for the sole attention and use of the intended recipient. Please notify us at once if you think that it may not be intended for you and delete it immediately.
