[ https://issues.apache.org/jira/browse/MESOS-7883?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16779689#comment-16779689 ]
Benjamin Mahler commented on MESOS-7883: ---------------------------------------- Linking in quota "capacity heuristic" testing work. > Quota heuristic check not accounting for mount volumes > ------------------------------------------------------ > > Key: MESOS-7883 > URL: https://issues.apache.org/jira/browse/MESOS-7883 > Project: Mesos > Issue Type: Bug > Components: allocation > Reporter: Vincent Roy > Priority: Major > Labels: resource-management > > This may be expected but came as a surprise to us. We are unable to create a > quota bigger than the root disk space on slaves. > Given two clusters with the same number of slaves and root disk size, but one > that also has mount volumes, is what the disk resources look like: > {noformat} > [root@fin-fang-foom-master-1 ~]# curl -s master.mesos:5050/state | jq > '.slaves[] .resources .disk' > 28698 > 28699 > 28698 > 28698 > 28697 > {noformat} > {noformat} > [root@hydra-master-1 ~]# curl -s master.mesos:5050/state | jq '.slaves[] > .resources .disk' > 50817 > 50817 > 50814 > 50819 > 50817 > {noformat} > In {{fin-fang-foom}}, I was able to create a quota for {{143490mb}} which is > the total of available disk resources, root in this case, as reported by > Mesos. For {{hydra}}, I am only able to create a quota for {{143489mb}}. This > is equivalent to the total of root disks available in {{hydra}} rather than > the total available disks reported by Mesos resources which is {{254084mb}}. > With a modified Mesos that adds logging to {{quota_handler}}, we can see that > only the {{disk(*)}} number increases in {{nonStaticClusterResources}} after > every iteration. The final iteration is {{disk(*):143489}} which is the > maximum quota I was able to create on {{hydra}}. We expected that quota > heuristic check would also include resources such as > {{disk(*)[MOUNT:/dcos/volume2]:7373}} > {noformat} > Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.763764 > 24902 quota_handler.cpp:71] Performing capacity heuristic check for a set > quota request > Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.763783 > 24902 quota_handler.cpp:87] heuristic: total quota 'disk(*):143489' > Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.763870 > 24902 quota_handler.cpp:111] heuristic: nonStaticAgentResources = > 'ports(*):[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, > 8182-32000]; disk(*)[MOUNT:/dcos/volume0]:7373; > disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; > disk(*):28698; cpus(*):4; mem(*):15023' > Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.763923 > 24902 quota_handler.cpp:113] heuristic: nonStaticClusterResources = > 'ports(*):[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, > 8182-32000]; disk(*)[MOUNT:/dcos/volume0]:7373; > disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; > disk(*):28698; cpus(*):4; mem(*):15023' > Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.763989 > 24902 quota_handler.cpp:111] heuristic: nonStaticAgentResources = > 'ports(*):[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, > 8182-32000]; disk(*)[MOUNT:/dcos/volume0]:7373; > disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; > disk(*):28698; cpus(*):4; mem(*):15023' > Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.764022 > 24902 quota_handler.cpp:113] heuristic: nonStaticClusterResources = > 'ports(*):[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, > 8182-32000]; disk(*)[MOUNT:/dcos/volume0]:7373; > disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; > disk(*):57396; cpus(*):8; mem(*):30046; disk(*)[MOUNT:/dcos/volume0]:7373; > disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373' > Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.764077 > 24902 quota_handler.cpp:111] heuristic: nonStaticAgentResources = > 'ports(*):[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, > 8182-32000]; disk(*)[MOUNT:/dcos/volume0]:7373; > disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; > disk(*):28695; cpus(*):4; mem(*):15023' > Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.764119 > 24902 quota_handler.cpp:113] heuristic: nonStaticClusterResources = > 'ports(*):[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, > 8182-32000]; disk(*)[MOUNT:/dcos/volume0]:7373; > disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; > disk(*):86091; cpus(*):12; mem(*):45069; disk(*)[MOUNT:/dcos/volume0]:7373; > disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; > disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373; > disk(*)[MOUNT:/dcos/volume2]:7373' > Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.764225 > 24902 quota_handler.cpp:111] heuristic: nonStaticAgentResources = > 'ports(*):[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, > 8182-32000]; disk(*)[MOUNT:/dcos/volume0]:7373; > disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; > disk(*):28700; cpus(*):4; mem(*):15023' > Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.764307 > 24902 quota_handler.cpp:113] heuristic: nonStaticClusterResources = > 'ports(*):[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, > 8182-32000]; disk(*)[MOUNT:/dcos/volume0]:7373; > disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; > disk(*):114791; cpus(*):16; mem(*):60092; disk(*)[MOUNT:/dcos/volume0]:7373; > disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; > disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373; > disk(*)[MOUNT:/dcos/volume2]:7373; disk(*)[MOUNT:/dcos/volume0]:7373; > disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373' > Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.764434 > 24902 quota_handler.cpp:111] heuristic: nonStaticAgentResources = > 'ports(*):[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, > 8182-32000]; disk(*)[MOUNT:/dcos/volume0]:7373; > disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; > disk(*):28698; cpus(*):4; mem(*):15023' > Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.764492 > 24902 quota_handler.cpp:113] heuristic: nonStaticClusterResources = > 'ports(*):[1025-2180, 2182-3887, 3889-5049, 5052-8079, 8082-8180, > 8182-32000]; disk(*)[MOUNT:/dcos/volume0]:7373; > disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; > disk(*):143489; cpus(*):20; mem(*):75115; disk(*)[MOUNT:/dcos/volume0]:7373; > disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; > disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373; > disk(*)[MOUNT:/dcos/volume2]:7373; disk(*)[MOUNT:/dcos/volume0]:7373; > disk(*)[MOUNT:/dcos/volume1]:7373; disk(*)[MOUNT:/dcos/volume2]:7373; > disk(*)[MOUNT:/dcos/volume0]:7373; disk(*)[MOUNT:/dcos/volume1]:7373; > disk(*)[MOUNT:/dcos/volume2]:7373' > Aug 11 12:54:18 hydra-master-1 mesos-master[24896]: I0811 12:54:18.764562 > 24902 quota_handler.cpp:118] heuristic: > nonStaticClusterResources.contains(totalQuota) > {noformat} -- This message was sent by Atlassian JIRA (v7.6.3#76005)