Devs,
I've got an initial prototype going on this project. During this
process I noticed that much of what I needed to do ended up duplicating
code that is in the twill-yarn project. I'm curious if the community
would be open to refactoring that moved some of this code such that it
could be shared by different clustering control implementations. For
instance, much of the code in YarnTwillPreparer, YarnContainerMain,
YarnServiceMain, etc is not really yarn specific.
Thoughts?
Thanks,
Martin
On 02/01/2017 04:13 PM, Martin Serrano wrote:
Hey Devs,
I've been evangelizing Twill at here at my company. We have a for
several years had a basic clustering solution for running our system
and its components across multiple hosts. As we've worked to migrate
the platform to Hadoop and YARN, we have started to move our cluster
control to be Yarn-based. And my goal is to make it Twill-based.
However as we do this work we anticipate the need to support customers
that do not have and do not want to operate Hadoop infrastructure.
I'm working on an architecture which would use Twill for command and
control for all of our services. For customers that did not want the
full enterprise capabilities that come with Hadoop we could continue
to offer our basic clustering support (albeit with reduced
capabilities) by plugging our clustering solution into Twill. One of
the aspects of Twill that interested me from the start was that the
control API was abstracted from YARN.
A couple questions:
1) Do any plugins for other clustering backends already exist? Open
source? Commercial?
2) If we were to do this, would the Twill community be interested in a
donation of this code to the Twill project?
I recall that Henry and I had a conversation at the last Apache Big
Data that there had been talk of plugins for other clustering backends
but I don't know if anything ever came of that.
Cheers,
Martin Serrano
