Benjamin Mahler created MESOS-10026:
---------------------------------------
Summary: Improve v1 operator API read performance.
Key: MESOS-10026
URL: https://issues.apache.org/jira/browse/MESOS-10026
Project: Mesos
Issue Type: Improvement
Components: HTTP API
Reporter: Benjamin Mahler
Currently, the v1 operator API has poor performance relative to the v0 json
API. The following initial numbers were provided by [~Will Mahler] from our
state serving benchmark:
|OPTIMIZED - Master (baseline)| | | | |
|Test setup|1000 agents with a total of 10000 running tasks and 10000 completed
tasks|10000 agents with a total of 100000 running tasks and 100000 completed
tasks|20000 agents with a total of 200000 running tasks and 200000 completed
tasks|40000 agents with a total of 400000 running tasks and 400000 completed
tasks|
|v0 'state' response|0.17|1.66|8.96|12.42|
|v1 x-protobuf|0.35|3.21|9.47|19.09|
|v1 json|0.45|4.72|10.81|31.43|
There is quite a lot of variance, but v1 protobuf consistently slower than v0
(sometimes significantly so) and v1 json is consistently slower than v1
protobuf (sometimes significantly so).
The reason that the v1 operator API is slower is that it does the following:
(1) Construct temporary unversioned state response object by copying in-memory
un-versioned state into overall response object. (expensive!)
(2) Evolve it to v1: serialize, de-serialize into v1 overall state object.
(expensive!)
(3) Serialize the overall v1 state object to protobuf or json.
(4) Destruct the temporaries (expensive! but is done after response starts
serving)
On the other hand, the v0 jsonify approach does the following:
(1) Serialize the in-memory unversioned state into json, by traversing state
and accumulating the overall serialized json.
This means that v1 has substantial overhead vs v0, and we need to remove it to
bring v1 on-par or better than v0. v1 should serialize directly to json
(straightforward with jsonify) or protobuf (this can be done via a
io::CodedOutputStream).
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
