> On June 4, 2013, 12:33 a.m., Ben Mahler wrote: > > Good idea, but seems like a JSON endpoint would be better (this is how we > > expose statistics in mesos)? > > Brenden Matthews wrote: > That's fine, but then I'd have to do a bunch of JSON serialization in the > Hadoop JobTracker, which seems overkill to me. > > Ben Mahler wrote: > My concern is twofold: > > (1) An http endpoint seems nicer than a file, IMO. One can just curl the > endpoint and each response will contain the latest data, rather than a > periodically update file where each read returns a checkpointed version of > the scheduler state. > > (2) The idiomatic way of exposing state in mesos is through http > endpoints, so it would be nice to stay consistent there. I realize that it's > tough to add that into the hadoop scheduler given that we don't have any > abstractions in place to assist in exporting stats over an endpoint. But I'm > open to options! :)
I can investigate to see if there's a way to do it without having to add dependencies or make a bunch of changes elsewhere. I dislike the idea of having lots of patches against Hadoop, because it increases the complexity of maintaining our own distribution. - Brenden ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/11165/#review21384 ----------------------------------------------------------- On June 11, 2013, 7:33 p.m., Brenden Matthews wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/11165/ > ----------------------------------------------------------- > > (Updated June 11, 2013, 7:33 p.m.) > > > Review request for mesos. > > > Description > ------- > > Write Mesos JobTracker stats to state file. > > Review: https://reviews.apache.org/r/11165/ > > > Diffs > ----- > > hadoop/mesos/src/java/org/apache/hadoop/mapred/MesosScheduler.java > afe401f5265e3d9494af7eace42eec45943184a3 > > Diff: https://reviews.apache.org/r/11165/diff/ > > > Testing > ------- > > Used in production at airbnb. > > make -j10 check && cd hadoop && make hadoop-2.0.0-mr1-cdh4.2.1 && make > hadoop-0.20.205.0 && make hadoop-0.20.2-cdh3u3 > > > Thanks, > > Brenden Matthews > >
