> On July 17, 2013, 2:58 a.m., Ben Mahler wrote: > > src/sched/sched.cpp, line 78 > > <https://reviews.apache.org/r/12603/diff/2/?file=322317#file322317line78> > > > > CACHED_UPDATES_CAPACITY? > > > > Also, maybe add a note that this only works for smaller clusters. If > > there are more than 100 offers in 10 seconds, this will not eliminate any > > duplicates and will only be adding unnecessary overhead, right? > > > > Caching O(20-30) seconds of updates seems nicer, but we'd have to be > > careful of the memory / performance overhead. Thoughts? > > Vinod Kone wrote: > Changed it to "MAX_ACKED_UPDATES". I think thats more descriptive. > > Regarding the number, it was definitely pulled out of thin air. Thanks to > your nudge, here are my thoughts. > > As you alluded, the critical number here is the "number of unique status > updates" / "status update retry interval" generated by the cluster. > Currently, the retry interval is set to 10s. Note that if the rate of status > unique updates is high, the changes in this patch are irrelevant. The > scheduler would still get backed up processing the updates. > > Now regarding MAX_ACKED_UPDATES value, the higher the number the more > likely the driver can bypass the scheduler for duplicates. As far as I can > tell, the only concern would be the memory consumed. Back of envelope math > suggests it is roughly, 2 * 16 bytes (UUID size) * MAX_ACKED_UPDATES. For > MAX_ACKED_UPDATES=1000, this value is ~32KB whereas for 10,000 it is ~320KB. > IMO, ~320KB overhead for the driver is reasonable, so I will bump this number > to 10,000. > > Thoughts?
That's a much better description that what I said! Care to add a comment reflecting this (the part about the status update rate) in the code for posterity? SGTM, I'm ok caching O(1000-10,000) updates. This is useful for calculating the memory overhead: http://www.boost.org/doc/libs/1_54_0/doc/html/intrusive/unordered_set_unordered_multiset.html#intrusive.unordered_set_unordered_multiset.unordered_set_unordered_multiset_performance - Ben ----------------------------------------------------------- This is an automatically generated e-mail. To reply, visit: https://reviews.apache.org/r/12603/#review23230 ----------------------------------------------------------- On July 17, 2013, 6:32 a.m., Vinod Kone wrote: > > ----------------------------------------------------------- > This is an automatically generated e-mail. To reply, visit: > https://reviews.apache.org/r/12603/ > ----------------------------------------------------------- > > (Updated July 17, 2013, 6:32 a.m.) > > > Review request for mesos, Benjamin Hindman and Ben Mahler. > > > Bugs: MESOS-551 > https://issues.apache.org/jira/browse/MESOS-551 > > > Repository: mesos > > > Description > ------- > > See summary. > > > Diffs > ----- > > src/sched/sched.cpp 7ea82e547c612159c9fa24fb6d62e3d2b5f11982 > src/tests/status_update_manager_tests.cpp > 42395324dfe49659bee2229c6573ffef0874d923 > > Diff: https://reviews.apache.org/r/12603/diff/ > > > Testing > ------- > > make check (OSX and Linux) > > > Thanks, > > Vinod Kone > >
