Terence,

Our solution is completely in house (but doesn't need to stay that way).  It is 
much simpler than either Mesos or Yarn.  My initial POC  indicates it will 
work.  Our solution does not do any resource management but will do some load 
balancing.

Martin

Sent from my Verizon Wireless 4G LTE DROID
On Feb 6, 2017 8:32 PM, Terence Yim <cht...@gmail.com> wrote:
Hi Martin,

You are right that the twill API was designed to be as neutral as possible and 
not to be specific for YARN. In fact, there are some interests to create a 
Mesos runner in twill. Would you mind sharing a bit more about the cluster 
management infrastructure your company is using ? Is it completely in house or 
is it based on some open source project?

Terence

Sent from my iPhone

> On Feb 6, 2017, at 12:44 PM, Martin Serrano <mar...@attivio.com> wrote:
>
> Devs,
>
> I've got an initial prototype going on this project.  During this process I 
> noticed that much of what I needed to do ended up duplicating code that is in 
> the twill-yarn project.  I'm curious if the community would be open to 
> refactoring that moved some of this code such that it could be shared by 
> different clustering control implementations.  For instance, much of the code 
> in YarnTwillPreparer, YarnContainerMain, YarnServiceMain, etc is not really 
> yarn specific.
>
> Thoughts?
>
> Thanks,
> Martin
>
>> On 02/01/2017 04:13 PM, Martin Serrano wrote:
>> Hey Devs,
>>
>> I've been evangelizing Twill at here at my company.  We have a for several 
>> years had a basic clustering solution for running our system and its 
>> components across multiple hosts.  As we've worked to migrate the platform 
>> to Hadoop and YARN, we have started to move our cluster control to be 
>> Yarn-based.  And my goal is to make it Twill-based.  However as we do this 
>> work we anticipate the need to support customers that do not have and do not 
>> want to operate Hadoop infrastructure.
>>
>> I'm working on an architecture which would use Twill for command and control 
>> for all of our services.  For customers that did not want the full 
>> enterprise capabilities that come with Hadoop we could continue to offer our 
>> basic clustering support (albeit with reduced capabilities) by plugging our 
>> clustering solution into Twill.  One of the aspects of Twill that interested 
>> me from the start was that the control API was abstracted from YARN.
>>
>> A couple questions:
>>
>> 1) Do any plugins for other clustering backends already exist? Open source?  
>> Commercial?
>>
>> 2) If we were to do this, would the Twill community be interested in a 
>> donation of this code to the Twill project?
>>
>> I recall that Henry and I had a conversation at the last Apache Big Data 
>> that there had been talk of plugins for other clustering backends but I 
>> don't know if anything ever came of that.
>>
>> Cheers,
>> Martin Serrano
>>
>

Reply via email to