Vincent Roy created MESOS-7883: ---------------------------------- Summary: Quota heuristic check not accounting for mount volumes Key: MESOS-7883 URL: https://issues.apache.org/jira/browse/MESOS-7883 Project: Mesos Issue Type: Bug Reporter: Vincent Roy
This may be expected but came as a surprise to us. We are unable to create a quota bigger than the root disk space on slaves. Given two clusters with the same number of slaves and root disk size, but one that also has mount volumes, is what the disk resources look like: {noformat} [root@fin-fang-foom-master-1 ~]# curl -s master.mesos:5050/state | jq '.slaves[] .resources .disk' 28698 28699 28698 28698 28697 {noformat} {noformat} [root@hydra-master-1 ~]# curl -s master.mesos:5050/state | jq '.slaves[] .resources .disk' 50817 50817 50814 50819 50817 {noformat} In {{fin-fang-foom}}, I was able to create a quota for {{143490mb}} which is the total of available disk resources, root in this case, as reported by Mesos. For {{hydra}}, I am only able to create a quota for {{143489mb}}. This is equivalent to the total of root disks available in {{hydra}} rather than the total available disks reported by Mesos resources which is {{254084mb}}. With a modified Mesos that adds logging to {{quota_handler}}, we can see that only the {{disk(*)}} number increases in {{nonStaticClusterResources}} after every iteration. The final iteration is {{disk(*):143489}} which is the maximum quota I was able to create on {{hydra}}. We expected that quota heuristic check would also include resources such as {{disk(*)[MOUNT:/dcos/volume2]:7373}} {noformat} Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.763764 24902 quota_handler.cpp:71] Performing capacity heuristic check for a set quota request Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.763783 24902 quota_handler.cpp:87] heuristic: total quota 'disk(*):143489' Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.763870 24902 quota_handler.cpp:111] heuristic: nonStaticAgentResources = 'ports(*):[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, 8182-32000]; disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; disk(*):28698; cpus(*):4; mem(*):15023' Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.763923 24902 quota_handler.cpp:113] heuristic: nonStaticClusterResources = 'ports(*):[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, 8182-32000]; disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; disk(*):28698; cpus(*):4; mem(*):15023' Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.763989 24902 quota_handler.cpp:111] heuristic: nonStaticAgentResources = 'ports(*):[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, 8182-32000]; disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; disk(*):28698; cpus(*):4; mem(*):15023' Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.764022 24902 quota_handler.cpp:113] heuristic: nonStaticClusterResources = 'ports(*):[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, 8182-32000]; disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; disk(*):57396; cpus(*):8; mem(*):30046; disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373' Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.764077 24902 quota_handler.cpp:111] heuristic: nonStaticAgentResources = 'ports(*):[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, 8182-32000]; disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; disk(*):28695; cpus(*):4; mem(*):15023' Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.764119 24902 quota_handler.cpp:113] heuristic: nonStaticClusterResources = 'ports(*):[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, 8182-32000]; disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; disk(*):86091; cpus(*):12; mem(*):45069; disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373' Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.764225 24902 quota_handler.cpp:111] heuristic: nonStaticAgentResources = 'ports(*):[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, 8182-32000]; disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; disk(*):28700; cpus(*):4; mem(*):15023' Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.764307 24902 quota_handler.cpp:113] heuristic: nonStaticClusterResources = 'ports(*):[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, 8182-32000]; disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; disk(*):114791; cpus(*):16; mem(*):60092; disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373' Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.764434 24902 quota_handler.cpp:111] heuristic: nonStaticAgentResources = 'ports(*):[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, 8182-32000]; disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; disk(*):28698; cpus(*):4; mem(*):15023' Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.764492 24902 quota_handler.cpp:113] heuristic: nonStaticClusterResources = 'ports(*):[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, 8182-32000]; disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; disk(*):143489; cpus(*):20; mem(*):75115; disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373' Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.764562 24902 quota_handler.cpp:118] heuristic: nonStaticClusterResources.contains(totalQuota) {noformat} -- This message was sent by Atlassian JIRA (v6.4.14#64029)