Hi Joris, For `Scalars`, yes, it's also dynamic allocated in `Resources::mutable_scalar()`.
For `2x` number, do you have any patch to share? Tested cases this afternoon: add resources including 100 roles (1 CPU for each); the performance is downgrade a lot; so I agree with you to improve some algorithms in Resources/Sorter. For 'basic benchmarks', temporary tracking in my personal github ( https://github.com/k82cn/mesos/blob/resources_benchmark/src/tests/resources_tests.cpp). The following cases are in my mind to add: 1. simple resources, e.g. 1 cpu 2. resources with port, e.g. [1-2], [3-4], ... [101-102] 3. resources with reservation, cpus(r1):1;cpus(r2):1; .... cpus(r10):1 4. resources with diskInfo 5. resources with revocableInfo The operators will be +, +=, -, -=, cpus(), contains. I booked a weekly call to discuss for allocaiton performance and sent the invitation to the dev@. If any comments, please let me know. ---- Da (Klaus), Ma (马达) | PMP® | Software Architect Platform OpenSource Technology, STG, IBM GCG +86-10-8245 4084 | [email protected] | http://k82.me On Mon, Jul 11, 2016 at 3:28 PM, Joris Van Remoortere <[email protected]> wrote: > +Dev > > Hey Klaus, > > Using Stout's `Optional` to represent the `optional` concept of a message > in protobuf is definitely a step in the right direction. > Regarding your comment in slack yesterday: From my version of the protobuf > generated code there definitely is dynamic allocation even for scalars. > > It looks like in our case there is a minimum of 3 dynamic allocations per > Resource object: > >> void Resource::SharedDtor() { >> if (name_ != &::google::protobuf::internal::kEmptyString) { >> delete name_; >> } >> if (role_ != _default_role_) { >> delete role_; >> } >> if (this != default_instance_) { >> delete scalar_; >> delete ranges_; >> delete set_; >> delete reservation_; >> delete disk_; >> delete revocable_; >> } >> } > > > The 2x number I mentioned came from running some of the existing > benchmarks. I didn't explore further because it didn't have as big an > impact as I had hoped. The first battle is simplifying some of the > algorithms in the Sorter / Resources. Once that is done then the resource > arithmetic will be more of a bounding factor. > > I agree with Ben that we should focus on writing some basic benchmarks > that represent the common uses of Resources in the allocator. We should > scale these benchmarks to represent some of the more stressful environments > that could occur. For example, had we had such a benchmark, we would have > realized much earlier on that we needed to aggregate only quantities in the > Sorter, and that using the existing form of Resources would have led to a > grinding halt if a reservation were made on every machine. > > Is there a regular call that is scheduled to discuss this? I think there > are some other folks also working on benchmarks and interested in the > discussion. > > — > *Joris Van Remoortere* > Mesosphere > > On Sun, Jul 10, 2016 at 8:50 PM, Klaus Ma <[email protected]> wrote: > >> + more devs :). >> >> ---- >> Da (Klaus), Ma (马达) | PMP® | Software Architect >> Platform OpenSource Technology, STG, IBM GCG >> +86-10-8245 4084 | [email protected] | http://k82.me >> >> On Mon, Jul 11, 2016 at 7:43 AM, Klaus Ma <[email protected]> wrote: >> >>> Hi Joris, >>> >>> I think `Option` is helpful to the performance improvement, it used >>> `placement new` to avoid dynamic allocation. Suppose you're using Option >>> for optional member in protobuf, and using class instance directly >>> (operator=). >>> >>> I'm adding some benchmark for `Resources`, especially for the >>> `Resources` with Rang, DiskInfo and ReservationInfo >>> >>> Draft PR for Benchmark of Resources: >>> https://github.com/k82cn/mesos/commit/09ca215cb37b1f89eb7d622228a8cf2249eb641c >>> >>> >>> ---- >>> Da (Klaus), Ma (马达) | PMP® | Software Architect >>> Platform OpenSource Technology, STG, IBM GCG >>> +86-10-8245 4084 | [email protected] | http://k82.me >>> >> >> >
