Thanks Jagadish. So, in essence, I should be looking at samza-11 branch for the final API against which I would have to write the Mesos integration pieces?
On Thu, Sep 1, 2016 at 12:56 AM, Jagadish Venkatraman < jagadish1...@gmail.com> wrote: > Hi Sriram, > > I had started prototyping it (purely to ensure that the Samza API makes > sense with Mesos). The exact API on the Samza-11 trunk is slightly > different, but hopefully there're similarities - > https://github.com/apache/samza/blob/master/samza-core/ > src/main/java/org/apache/samza/clustermanager/ClusterResourceManager.java > > Find a stub implementation here: (that encapsulates a fair bit of boiler > plate from Mesos driver creation etc.) > https://github.com/vjagadish/samza-clone/commit/ > 9e5ed9f1774dadf079ad33913ff7f20ed58bc8dc > > A version of the prototype with the Old API: here > <https://github.com/bringhurst/samza/tree/SAMZA-375/samza-mesos/src/main/ > scala/org/apache/samza/job/mesos> > > Some interesting implementation notes: > - Mesos did not (yet) support a preferred host request. However, that could > be implemented via dynamic reservations > <http://mesos.apache.org/documentation/latest/reservation/>. > - My discussions with the Mesos community here: > https://mail-archives.apache.org/mod_mbox/mesos-user/201602.mbox/% > 3ccamd3yjgxmhg4rtw4gbxgf9msmbv6abzbgpql6ejq5gwmot0...@mail.gmail.com%3E > - MESOS-4616 has more context. > > It'd be awesome you can take a stab at Mesos integration - I'm happy to > help out in whatever way I can. > > Thank you, > Jagadish > > On Wed, Aug 31, 2016 at 10:45 AM, Sriram Ramachandrasekaran < > sri.ram...@gmail.com> wrote: > > > Yi, > > That's a good amount of history to know. I will take a look at 680 and > then > > see if I can implement something as well. If there's some stuff that's > > already done, would be glad to re-use it too. > > Thanks again > > > > On Wed, Aug 31, 2016 at 10:58 PM, Yi Pan <nickpa...@gmail.com> wrote: > > > > > Hi, Sriram, > > > > > > The story behind delaying the integration of SAMZA-375 is that there > are > > > tons of repeated code in SamzaAppMaster that exist in both samza-yarn > and > > > Mesos. W/o the change we recently made in SAMZA-680, we are going to > copy > > > the SamzaAppMaster code for every distributed execution system that we > > > added support in Samza. Now, w/ the change in SAMZA-680, we have > inverted > > > the JobCoordinator and the AppMaster logic, which makes it much easier > to > > > have pluggable distributed cluster management system in Samza. As > stated > > in > > > the JIRA, all we need is now a Mesos-specific implementation of > > > ClusterResourceManager that can talk to Mesos for container > > > request/allocation. > > > > > > @Jagadish, I remember that you did some proto-type integration w/ Mesos > > > based on SAMZA-680. Would you mind to share some example code for that? > > > > > > Thanks! > > > > > > -Yi > > > > > > On Tue, Aug 30, 2016 at 8:35 PM, Sriram Ramachandrasekaran < > > > sri.ram...@gmail.com> wrote: > > > > > > > Folks, > > > > > > > > We've been using Samza in Production from beginning of this year. > It's > > > been > > > > quite stable for our needs, although, we don't use it heavily yet. > One > > of > > > > the things we would like to know is, where is Samza Mesos integration > > in > > > > the roadmap? I know, SAMZA-375 > > > > <https://issues.apache.org/jira/browse/SAMZA-375> is specifically > > > towards > > > > that, but, is there something stopping the community from integration > > > into > > > > mainline? > > > > > > > > I ask this because, we run our Samza jobs on YARN right now and we > use > > > > Mesos infra for other workloads. I really don't want to manage 2 > infra > > > > components which are supposed to do exactly the same thing. We've > built > > > > enough tooling around Mesos infra, so, wouldn't want to move away > from > > it > > > > too. > > > > > > > > The options we're evaluating are: > > > > 1. Move to KStreams and get away from YARN > > > > 2. Explore Samza-Mesos integration so that, we can reduce "explicit" > > > > dependency on Kafka. > > > > > > > > > > > > Some clarity on this would really help us. > > > > Sriram > > > > > > > > -- > > > > It's just about how deep your longing is! > > > > > > > > > > > > > > > -- > > It's just about how deep your longing is! > > > > > > -- > Jagadish V, > Graduate Student, > Department of Computer Science, > Stanford University > -- It's just about how deep your longing is!
