Vincent Roy created MESOS-7883:
----------------------------------
Summary: Quota heuristic check not accounting for mount volumes
Key: MESOS-7883
URL: https://issues.apache.org/jira/browse/MESOS-7883
Project: Mesos
Issue Type: Bug
Reporter: Vincent Roy
This may be expected but came as a surprise to us. We are unable to create a
quota bigger than the root disk space on slaves.
Given two clusters with the same number of slaves and root disk size, but one
that also has mount volumes, is what the disk resources look like:
{noformat}
[root@fin-fang-foom-master-1 ~]# curl -s master.mesos:5050/state | jq
'.slaves[] .resources .disk'
28698
28699
28698
28698
28697
{noformat}
{noformat}
[root@hydra-master-1 ~]# curl -s master.mesos:5050/state | jq '.slaves[]
.resources .disk'
50817
50817
50814
50819
50817
{noformat}
In {{fin-fang-foom}}, I was able to create a quota for {{143490mb}} which is
the total of available disk resources, root in this case, as reported by Mesos.
For {{hydra}}, I am only able to create a quota for {{143489mb}}. This is
equivalent to the total of root disks available in {{hydra}} rather than the
total available disks reported by Mesos resources which is {{254084mb}}.
With a modified Mesos that adds logging to {{quota_handler}}, we can see that
only the {{disk(*)}} number increases in {{nonStaticClusterResources}} after
every iteration. The final iteration is {{disk(*):143489}} which is the maximum
quota I was able to create on {{hydra}}. We expected that quota heuristic check
would also include resources such as {{disk(*)[MOUNT:/dcos/volume2]:7373}}
{noformat}
Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.763764 24902
quota_handler.cpp:71] Performing capacity heuristic check for a set quota
request
Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.763783 24902
quota_handler.cpp:87] heuristic: total quota 'disk(*):143489'
Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.763870 24902
quota_handler.cpp:111] heuristic: nonStaticAgentResources =
'ports(*):[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, 8182-32000];
disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373;
disk(*)[MOUNT:/dcos/volume2]:7373; disk(*):28698; cpus(*):4; mem(*):15023'
Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.763923 24902
quota_handler.cpp:113] heuristic: nonStaticClusterResources =
'ports(*):[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, 8182-32000];
disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373;
disk(*)[MOUNT:/dcos/volume2]:7373; disk(*):28698; cpus(*):4; mem(*):15023'
Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.763989 24902
quota_handler.cpp:111] heuristic: nonStaticAgentResources =
'ports(*):[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, 8182-32000];
disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373;
disk(*)[MOUNT:/dcos/volume2]:7373; disk(*):28698; cpus(*):4; mem(*):15023'
Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.764022 24902
quota_handler.cpp:113] heuristic: nonStaticClusterResources =
'ports(*):[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, 8182-32000];
disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373;
disk(*)[MOUNT:/dcos/volume2]:7373; disk(*):57396; cpus(*):8; mem(*):30046;
disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373;
disk(*)[MOUNT:/dcos/volume2]:7373'
Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.764077 24902
quota_handler.cpp:111] heuristic: nonStaticAgentResources =
'ports(*):[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, 8182-32000];
disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373;
disk(*)[MOUNT:/dcos/volume2]:7373; disk(*):28695; cpus(*):4; mem(*):15023'
Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.764119 24902
quota_handler.cpp:113] heuristic: nonStaticClusterResources =
'ports(*):[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, 8182-32000];
disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373;
disk(*)[MOUNT:/dcos/volume2]:7373; disk(*):86091; cpus(*):12; mem(*):45069;
disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373;
disk(*)[MOUNT:/dcos/volume2]:7373; disk(*)[MOUNT:/dcos/volume0]:7373;
disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373'
Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.764225 24902
quota_handler.cpp:111] heuristic: nonStaticAgentResources =
'ports(*):[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, 8182-32000];
disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373;
disk(*)[MOUNT:/dcos/volume2]:7373; disk(*):28700; cpus(*):4; mem(*):15023'
Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.764307 24902
quota_handler.cpp:113] heuristic: nonStaticClusterResources =
'ports(*):[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, 8182-32000];
disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373;
disk(*)[MOUNT:/dcos/volume2]:7373; disk(*):114791; cpus(*):16; mem(*):60092;
disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373;
disk(*)[MOUNT:/dcos/volume2]:7373; disk(*)[MOUNT:/dcos/volume0]:7373;
disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373;
disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373;
disk(*)[MOUNT:/dcos/volume2]:7373'
Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.764434 24902
quota_handler.cpp:111] heuristic: nonStaticAgentResources =
'ports(*):[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, 8182-32000];
disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373;
disk(*)[MOUNT:/dcos/volume2]:7373; disk(*):28698; cpus(*):4; mem(*):15023'
Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.764492 24902
quota_handler.cpp:113] heuristic: nonStaticClusterResources =
'ports(*):[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, 8182-32000];
disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373;
disk(*)[MOUNT:/dcos/volume2]:7373; disk(*):143489; cpus(*):20; mem(*):75115;
disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373;
disk(*)[MOUNT:/dcos/volume2]:7373; disk(*)[MOUNT:/dcos/volume0]:7373;
disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373;
disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373;
disk(*)[MOUNT:/dcos/volume2]:7373; disk(*)[MOUNT:/dcos/volume0]:7373;
disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373'
Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.764562 24902
quota_handler.cpp:118] heuristic: nonStaticClusterResources.contains(totalQuota)
{noformat}
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
