[jira] [Commented] (MESOS-4553) Manage offers in allocator.
[
https://issues.apache.org/jira/browse/MESOS-4553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16387065#comment-16387065
]
Klaus Ma commented on MESOS-4553:
-
Got that; looking forward to see this feature :)
> Manage offers in allocator.
> ---
>
> Key: MESOS-4553
> URL: https://issues.apache.org/jira/browse/MESOS-4553
> Project: Mesos
> Issue Type: Epic
> Components: master
>Reporter: Klaus Ma
>Assignee: Joseph Wu
>Priority: Major
> Labels: multitenancy
>
> Currently, Offers are managed by {{Master}} while Resources are handled by
> the {{Allocator}}. This introduces a variety of races between the {{Master}}
> and {{Allocator}} actors; and limitations with regards to information that
> the {{Allocator}} can act upon. See the linked issues for some examples of
> races/limitations.
> The goal of this epic is to track a refactor of the {{Master}} and
> {{Allocator}}. The {{Master}} should continue to manage communication with
> {{Frameworks}}, including the act of sending Offers, but all state associated
> with Offers (primarily OfferIDs and Timers), will be off-loaded to the
> {{Allocator}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-4553) Manage offers in allocator.
[
https://issues.apache.org/jira/browse/MESOS-4553?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16386958#comment-16386958
]
Klaus Ma commented on MESOS-4553:
-
According to the previous comments, it's handled in -MESOS-3147- ; is that
correct? That'll be great if we can manage offer in allocator, it'll also give
more room for other to enhance allocator policy :)
> Manage offers in allocator.
> ---
>
> Key: MESOS-4553
> URL: https://issues.apache.org/jira/browse/MESOS-4553
> Project: Mesos
> Issue Type: Epic
> Components: master
>Reporter: Klaus Ma
>Assignee: Joseph Wu
>Priority: Major
> Labels: multitenancy
>
> Currently, Offers are managed by {{Master}} while Resources are handled by
> the {{Allocator}}. This introduces a variety of races between the {{Master}}
> and {{Allocator}} actors; and limitations with regards to information that
> the {{Allocator}} can act upon. See the linked issues for some examples of
> races/limitations.
> The goal of this epic is to track a refactor of the {{Master}} and
> {{Allocator}}. The {{Master}} should continue to manage communication with
> {{Frameworks}}, including the act of sending Offers, but all state associated
> with Offers (primarily OfferIDs and Timers), will be off-loaded to the
> {{Allocator}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (MESOS-907) Add Kerberos Authentication support
[ https://issues.apache.org/jira/browse/MESOS-907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15734795#comment-15734795 ] Klaus Ma commented on MESOS-907: I think Mesos prefer to delegate this feature to user to make a module :). > Add Kerberos Authentication support > --- > > Key: MESOS-907 > URL: https://issues.apache.org/jira/browse/MESOS-907 > Project: Mesos > Issue Type: Story > Components: general >Reporter: Adam B >Assignee: Tim Anderegg > Labels: security, twitter > > MESOS-704 added basic authentication support using CRAM-MD5 through SASL. Now > we should integrate Kerberos authentication using GSS-API, which is already > supported by SASL. Kerberos is a widely-used industry standard authentication > service, and integration with Mesos will make it easier for customers to > integrate their existing security process with Mesos. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-1718) Command executor can overcommit the agent.
[
https://issues.apache.org/jira/browse/MESOS-1718?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15733949#comment-15733949
]
Klaus Ma commented on MESOS-1718:
-
[~alexr] , anyway, I think exception is always bad :). It's better for us to
align CLI executor to others.
> Command executor can overcommit the agent.
> --
>
> Key: MESOS-1718
> URL: https://issues.apache.org/jira/browse/MESOS-1718
> Project: Mesos
> Issue Type: Bug
> Components: agent
>Reporter: Benjamin Mahler
>Priority: Critical
>
> Currently we give a small amount of resources to the command executor, in
> addition to resources used by the command task:
> https://github.com/apache/mesos/blob/0.20.0-rc1/src/slave/slave.cpp#L2448
> {code: title=}
> ExecutorInfo Slave::getExecutorInfo(
> const FrameworkID& frameworkId,
> const TaskInfo& task)
> {
> ...
> // Add an allowance for the command executor. This does lead to a
> // small overcommit of resources.
> executor.mutable_resources()->MergeFrom(
> Resources::parse(
> "cpus:" + stringify(DEFAULT_EXECUTOR_CPUS) + ";" +
> "mem:" + stringify(DEFAULT_EXECUTOR_MEM.megabytes())).get());
> ...
> }
> {code}
> This leads to an overcommit of the slave. Ideally, for command tasks we can
> "transfer" all of the task resources to the executor at the slave / isolation
> level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MESOS-6326) Build failed on Mac
Klaus Ma created MESOS-6326:
---
Summary: Build failed on Mac
Key: MESOS-6326
URL: https://issues.apache.org/jira/browse/MESOS-6326
Project: Mesos
Issue Type: Bug
Affects Versions: 1.0.1
Reporter: Klaus Ma
Priority: Minor
Built Mesos 1.0.1 failed on Mac:
{{uname -a}}: Darwin Klauss-MacBook-Pro.local 16.0.0 Darwin Kernel Version
16.0.0: Mon Aug 29 17:56:20 PDT 2016; root:xnu-3789.1.32~3/RELEASE_X86_64 x86_64
{code}
In file included from ../../src/appc/spec.cpp:19:
In file included from ../../3rdparty/stout/include/stout/protobuf.hpp:31:
In file included from
../3rdparty/protobuf-2.6.1/src/google/protobuf/repeated_field.h:58:
In file included from
../3rdparty/protobuf-2.6.1/src/google/protobuf/generated_message_util.h:44:
In file included from
../3rdparty/protobuf-2.6.1/src/google/protobuf/stubs/once.h:81:
In file included from
../3rdparty/protobuf-2.6.1/src/google/protobuf/stubs/atomicops.h:184:
../3rdparty/protobuf-2.6.1/src/google/protobuf/stubs/atomicops_internals_macosx.h:164:10:
error: 'OSAtomicAdd64Barrier' is deprecated:
first deprecated in macOS 10.12 - Use std::atomic_fetch_add() from
instead [-Werror,-Wdeprecated-declarations]
return OSAtomicAdd64Barrier(increment,
^
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/usr/include/libkern/OSAtomicDeprecated.h:247:9:
note:
'OSAtomicAdd64Barrier' has been explicitly marked deprecated here
int64_t OSAtomicAdd64Barrier( int64_t __theAmount,
^
In file included from ../../src/appc/spec.cpp:19:
In file included from ../../3rdparty/stout/include/stout/protobuf.hpp:31:
In file included from
../3rdparty/protobuf-2.6.1/src/google/protobuf/repeated_field.h:58:
In file included from
../3rdparty/protobuf-2.6.1/src/google/protobuf/generated_message_util.h:44:
In file included from
../3rdparty/protobuf-2.6.1/src/google/protobuf/stubs/once.h:81:
In file included from
../3rdparty/protobuf-2.6.1/src/google/protobuf/stubs/atomicops.h:184:
../3rdparty/protobuf-2.6.1/src/google/protobuf/stubs/atomicops_internals_macosx.h:173:9:
error: 'OSAtomicCompareAndSwap64Barrier' is
deprecated: first deprecated in macOS 10.12 - Use
std::atomic_compare_exchange_strong() from instead
[-Werror,-Wdeprecated-declarations]
if (OSAtomicCompareAndSwap64Barrier(
^
/Applications/Xcode.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX10.12.sdk/usr/include/libkern/OSAtomicDeprecated.h:645:9:
note:
'OSAtomicCompareAndSwap64Barrier' has been explicitly marked deprecated
here
boolOSAtomicCompareAndSwap64Barrier( int64_t __oldValue, int64_t __newValue,
^
12 errors generated.
make[2]: *** [appc/libmesos_no_3rdparty_la-spec.lo] Error 1
make[1]: *** [all] Error 2
make: *** [all-recursive] Error 1
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-6247) Enable Framework to set weight
[ https://issues.apache.org/jira/browse/MESOS-6247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15548683#comment-15548683 ] Klaus Ma commented on MESOS-6247: - [~jvanremoortere] , yes, they can not share the reserved resources with each other in different role. For the weight, it's better to let Mesos to allocate resources within a role. Because other frameworks may be deployed in this environment, e.g. Storm, it'll be a huge work to modify those frameworks one by one. I agree with you and BenM that hierarchical role is the long term solution; but any suggestion on the target date? BTW, how about other user's scenario about multiple frameworks? > Enable Framework to set weight > -- > > Key: MESOS-6247 > URL: https://issues.apache.org/jira/browse/MESOS-6247 > Project: Mesos > Issue Type: Bug > Components: allocation > Environment: all >Reporter: Klaus Ma >Priority: Critical > > We'd like to enable framework's weight when it register. So the framework can > share resources based on weight within the same role. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Issue Comment Deleted] (MESOS-6247) Enable Framework to set weight
[ https://issues.apache.org/jira/browse/MESOS-6247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Klaus Ma updated MESOS-6247: Comment: was deleted (was: They can not share resources with each other in two roles.) > Enable Framework to set weight > -- > > Key: MESOS-6247 > URL: https://issues.apache.org/jira/browse/MESOS-6247 > Project: Mesos > Issue Type: Bug > Components: allocation > Environment: all >Reporter: Klaus Ma >Priority: Critical > > We'd like to enable framework's weight when it register. So the framework can > share resources based on weight within the same role. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-6247) Enable Framework to set weight
[ https://issues.apache.org/jira/browse/MESOS-6247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15547250#comment-15547250 ] Klaus Ma commented on MESOS-6247: - They can not share resources with each other in two roles. > Enable Framework to set weight > -- > > Key: MESOS-6247 > URL: https://issues.apache.org/jira/browse/MESOS-6247 > Project: Mesos > Issue Type: Bug > Components: allocation > Environment: all >Reporter: Klaus Ma >Priority: Critical > > We'd like to enable framework's weight when it register. So the framework can > share resources based on weight within the same role. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-6247) Enable Framework to set weight
[ https://issues.apache.org/jira/browse/MESOS-6247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15547251#comment-15547251 ] Klaus Ma commented on MESOS-6247: - They can not share resources with each other in two roles. > Enable Framework to set weight > -- > > Key: MESOS-6247 > URL: https://issues.apache.org/jira/browse/MESOS-6247 > Project: Mesos > Issue Type: Bug > Components: allocation > Environment: all >Reporter: Klaus Ma >Priority: Critical > > We'd like to enable framework's weight when it register. So the framework can > share resources based on weight within the same role. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (MESOS-6247) Enable Framework to set weight
[ https://issues.apache.org/jira/browse/MESOS-6247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15547159#comment-15547159 ] Klaus Ma edited comment on MESOS-6247 at 10/5/16 12:27 AM: --- Yes, hierarchical role is the solution for this case; do you have target date for that feature? If not, set weight to framework is the fastest solution I can image. was (Author: klaus1982): Yes, hierarchical role is the target solution for this case; do you have target date for that feature? If not, set weight to framework is the fastest solution I can image. > Enable Framework to set weight > -- > > Key: MESOS-6247 > URL: https://issues.apache.org/jira/browse/MESOS-6247 > Project: Mesos > Issue Type: Bug > Components: allocation > Environment: all >Reporter: Klaus Ma >Priority: Critical > > We'd like to enable framework's weight when it register. So the framework can > share resources based on weight within the same role. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Comment Edited] (MESOS-6247) Enable Framework to set weight
[ https://issues.apache.org/jira/browse/MESOS-6247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15547159#comment-15547159 ] Klaus Ma edited comment on MESOS-6247 at 10/5/16 12:27 AM: --- Yes, hierarchical role is the solution for this case; do you have target date for that feature? If not, set weight to framework is the solution I can image for now. was (Author: klaus1982): Yes, hierarchical role is the solution for this case; do you have target date for that feature? If not, set weight to framework is the fastest solution I can image. > Enable Framework to set weight > -- > > Key: MESOS-6247 > URL: https://issues.apache.org/jira/browse/MESOS-6247 > Project: Mesos > Issue Type: Bug > Components: allocation > Environment: all >Reporter: Klaus Ma >Priority: Critical > > We'd like to enable framework's weight when it register. So the framework can > share resources based on weight within the same role. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-6247) Enable Framework to set weight
[ https://issues.apache.org/jira/browse/MESOS-6247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15547159#comment-15547159 ] Klaus Ma commented on MESOS-6247: - Yes, hierarchical role is the target solution for this case; do you have target date for that feature? If not, set weight to framework is the fastest solution I can image. > Enable Framework to set weight > -- > > Key: MESOS-6247 > URL: https://issues.apache.org/jira/browse/MESOS-6247 > Project: Mesos > Issue Type: Bug > Components: allocation > Environment: all >Reporter: Klaus Ma >Priority: Critical > > We'd like to enable framework's weight when it register. So the framework can > share resources based on weight within the same role. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-6247) Enable Framework to set weight
[ https://issues.apache.org/jira/browse/MESOS-6247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15547097#comment-15547097 ] Klaus Ma commented on MESOS-6247: - Wow, here's the case I'd like to resolve: 1. There're two types of workload in the cluster: long running service & BigData workload 2. The resources (HW) for BigData workload is better 3. The BigData workload includes MapReduce (YARN) & Spark Current design is to run YARN & Spark in the same role and reserved resources for them; so I'd like to set weight for the framework. > Enable Framework to set weight > -- > > Key: MESOS-6247 > URL: https://issues.apache.org/jira/browse/MESOS-6247 > Project: Mesos > Issue Type: Bug > Components: allocation > Environment: all >Reporter: Klaus Ma >Priority: Critical > > We'd like to enable framework's weight when it register. So the framework can > share resources based on weight within the same role. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-6247) Enable Framework to set weight
[ https://issues.apache.org/jira/browse/MESOS-6247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15544801#comment-15544801 ] Klaus Ma commented on MESOS-6247: - [~bmahler], any suggestion? > Enable Framework to set weight > -- > > Key: MESOS-6247 > URL: https://issues.apache.org/jira/browse/MESOS-6247 > Project: Mesos > Issue Type: Bug > Components: allocation > Environment: all >Reporter: Klaus Ma >Priority: Critical > > We'd like to enable framework's weight when it register. So the framework can > share resources based on weight within the same role. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-6247) Enable Framework to set weight
Klaus Ma created MESOS-6247: --- Summary: Enable Framework to set weight Key: MESOS-6247 URL: https://issues.apache.org/jira/browse/MESOS-6247 Project: Mesos Issue Type: Bug Components: allocation Environment: all Reporter: Klaus Ma Priority: Critical We'd like to enable framework's weight when it register. So the framework can share resources based on weight within the same role. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-5123) Docker task may fail if path to agent work_dir is relative.
[
https://issues.apache.org/jira/browse/MESOS-5123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15520577#comment-15520577
]
Klaus Ma commented on MESOS-5123:
-
[~alexr], I think we can close this PR. The diff will be different after
refactor {{Flags.load}}.
> Docker task may fail if path to agent work_dir is relative.
>
>
> Key: MESOS-5123
> URL: https://issues.apache.org/jira/browse/MESOS-5123
> Project: Mesos
> Issue Type: Improvement
> Components: docker
>Affects Versions: 1.0.0
>Reporter: Alexander Rukletsov
>Assignee: Klaus Ma
> Labels: docker, documentation, mesosphere
>
> When a local folder for agent’s {{\-\-work_dir}} is specified (e.g.,
> {{\-\-work_dir=w/s}}) docker complains that there are forbidden symbols in a
> *local* volume name. Specifying an absolute path (e.g.,
> {{\-\-work_dir=/tmp}}) solves the problem.
> Docker error observed:
> {noformat}
> docker: Error response from daemon: create
> w/s/slaves/33b8fe47-e9e0-468a-83a6-98c1e3537e59-S1/frameworks/33b8fe47-e9e0-468a-83a6-98c1e3537e59-0001/executors/docker-test/runs/3cc5cb04-d0a9-490e-94d5-d446b66c97cc:
> volume name invalid:
> "w/s/slaves/33b8fe47-e9e0-468a-83a6-98c1e3537e59-S1/frameworks/33b8fe47-e9e0-468a-83a6-98c1e3537e59-0001/executors/docker-test/runs/3cc5cb04-d0a9-490e-94d5-d446b66c97cc"
> includes invalid characters for a local volume name, only
> "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed.
> {noformat}
> First off, it is not obvious that Mesos always creates a volume for the
> sandbox. We may want to document it.
> Second, it's hard to understand that local {{work_dir}} can trigger forbidden
> symbols error in docker. Does it make sense to check it during agent launch
> if docker containerizer is enabled? Or reject docker tasks during task
> validation?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-6048) Docker execution fails if agent's workdir is a relative path
[
https://issues.apache.org/jira/browse/MESOS-6048?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15512345#comment-15512345
]
Klaus Ma commented on MESOS-6048:
-
[~greggomann] , I used to propose a PR; but after sync up JIe/Micheal, we'd
suggest to refactor {{Flags.load}} firstly, there're ~20 override functions
here.
> Docker execution fails if agent's workdir is a relative path
>
>
> Key: MESOS-6048
> URL: https://issues.apache.org/jira/browse/MESOS-6048
> Project: Mesos
> Issue Type: Bug
>Affects Versions: 1.0.0
> Environment: openSUSE Tumbleweed, Docker 1.11.2
>Reporter: Tobias Pfeiffer
>Assignee: Greg Mann
>Priority: Minor
>
> I start my mesos-agent.sh for development purposes like
> {code}
> sudo ./bin/mesos-agent.sh --master=127.0.0.1:5050 \
> --work_dir=../../tmp --containerizers=docker \
> --no-systemd_enable_support
> {code}
> And when executing a task via the Python API like
> {code}
> task.command.value = "echo foobar; hostname"
> task.container.type = 1 # docker
> task.container.docker.image = "phusion/baseimage:0.9.19"
> {code}
> then this fails with the following error message:
> {noformat}
> I0816 17:04:48.799820 9105 docker.cpp:809] Running docker -H
> unix:///var/run/docker.sock run --cpu-shares 1024 --memory 1234567 -e
> MESOS_SANDBOX=/mnt/mesos/sandbox -e
> MESOS_CONTAINER_NAME=mesos-bc638eef-caf2-4778-a762-58826ea40364-S3.8386e2e0-6f40-47e2-9022-069b9f52b051
> -v
> ../../tmp/slaves/bc638eef-caf2-4778-a762-58826ea40364-S3/frameworks/bc638eef-caf2-4778-a762-58826ea40364-0011/executors/0/runs/8386e2e0-6f40-47e2-9022-069b9f52b051:/mnt/mesos/sandbox
> --net host --entrypoint /bin/sh --name
> mesos-bc638eef-caf2-4778-a762-58826ea40364-S3.8386e2e0-6f40-47e2-9022-069b9f52b051
> phusion/baseimage:0.9.19 -c echo foobar; hostname
> docker: Error response from daemon: create
> ../../tmp/slaves/bc638eef-caf2-4778-a762-58826ea40364-S3/frameworks/bc638eef-caf2-4778-a762-58826ea40364-0011/executors/0/runs/8386e2e0-6f40-47e2-9022-069b9f52b051:
>
> "../../tmp/slaves/bc638eef-caf2-4778-a762-58826ea40364-S3/frameworks/bc638eef-caf2-4778-a762-58826ea40364-0011/executors/0/runs/8386e2e0-6f40-47e2-9022-069b9f52b051"
> includes invalid characters for a local volume name, only
> "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed.
> {noformat}
> In short: Relative paths are not allowed for mounting host directories into
> Docker containers. I suggest that the agent should turn the work directory
> into an abspath before passing it to the `-v` parameter of docker.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-4811) Reusable/Cacheable Offer
[ https://issues.apache.org/jira/browse/MESOS-4811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15469385#comment-15469385 ] Klaus Ma commented on MESOS-4811: - The requirement is different :). Although we can improve performance by batch, it's still some downgrade here; and it's heavy for master to let each workload go through Mesos master. > Reusable/Cacheable Offer > > > Key: MESOS-4811 > URL: https://issues.apache.org/jira/browse/MESOS-4811 > Project: Mesos > Issue Type: Bug > Components: allocation >Reporter: Klaus Ma >Assignee: Abhishek Dasgupta > Labels: tech-debt > > Currently, the resources are return back to allocator when task finished; and > those resources are not allocated to framework until next allocation cycle. > The performance is low for short running tasks (MESOS-3078). The proposed > solution is to let framework keep using the offer until allocator decide to > rescind it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-4988) Excluded reserved resources when got nonRevocable resources in stage 1.
[ https://issues.apache.org/jira/browse/MESOS-4988?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15469309#comment-15469309 ] Klaus Ma commented on MESOS-4988: - Did not get time to check the # of performance improvement, we can close it firstly. > Excluded reserved resources when got nonRevocable resources in stage 1. > --- > > Key: MESOS-4988 > URL: https://issues.apache.org/jira/browse/MESOS-4988 > Project: Mesos > Issue Type: Bug > Components: allocation >Reporter: Klaus Ma > > Allocator will only allocate non-revocable resources to satify quota. As the > reserved resources can not be revocable, it's not necessary to call > `nonRevocable()` for reserved resources. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-4808) Allocation in batch instead of execute it every-time when addSlave/addFramework.
[
https://issues.apache.org/jira/browse/MESOS-4808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15438538#comment-15438538
]
Klaus Ma commented on MESOS-4808:
-
OK to close.
> Allocation in batch instead of execute it every-time when
> addSlave/addFramework.
>
>
> Key: MESOS-4808
> URL: https://issues.apache.org/jira/browse/MESOS-4808
> Project: Mesos
> Issue Type: Bug
> Components: allocation
>Reporter: Klaus Ma
> Labels: master, tech-debt
>
> Currently, {{allocate()}} are executed every-time when a new slave/framework
> are registered; if there're lots of agent start all most the same time, the
> allocation will keep running for a while. It's acceptable behaviour to
> allocate resources in next allocation cycle. But when a task is finished,
> it's better to allocate ASAP although there's performances issues; refer to
> MESOS-3078 for more detail on short running tasks.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-6058) Register slave in deactivate mode
[ https://issues.apache.org/jira/browse/MESOS-6058?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Klaus Ma updated MESOS-6058: Description: In my cluster, I'd like to reserve some resource for one application, dynamic reservation feature is used because the reservation maybe changed. But when a slave register to the master, some tasks from other frameworks maybe dispatched to the new slave before reservation. The proposal is to enable slave register in deactivate mode, and activate it after configuration, e.g. dynamic reservation. cc [~kaysoky]/[~jvanremoortere] was: In my cluster, I'd like to reserve some resource for one application, dynamic reservation feature is used because the reservation maybe changed. But when a slave register to the master, some tasks from other frameworks maybe dispatched to the new slave before reservation. The proposal is to enable slave register in deactivate mode, and activate it after configuration, e.g. dynamic reservation. cc [~kaysoky] > Register slave in deactivate mode > - > > Key: MESOS-6058 > URL: https://issues.apache.org/jira/browse/MESOS-6058 > Project: Mesos > Issue Type: Improvement > Components: master, slave >Reporter: Klaus Ma > > In my cluster, I'd like to reserve some resource for one application, dynamic > reservation feature is used because the reservation maybe changed. But when > a slave register to the master, some tasks from other frameworks maybe > dispatched to the new slave before reservation. The proposal is to enable > slave register in deactivate mode, and activate it after configuration, e.g. > dynamic reservation. > cc [~kaysoky]/[~jvanremoortere] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-6058) Register slave in deactivate mode
Klaus Ma created MESOS-6058: --- Summary: Register slave in deactivate mode Key: MESOS-6058 URL: https://issues.apache.org/jira/browse/MESOS-6058 Project: Mesos Issue Type: Improvement Components: master, slave Reporter: Klaus Ma In my cluster, I'd like to reserve some resource for one application, dynamic reservation feature is used because the reservation maybe changed. But when a slave register to the master, some tasks from other frameworks maybe dispatched to the new slave before reservation. The proposal is to enable slave register in deactivate mode, and activate it after configuration, e.g. dynamic reservation. cc [~kaysoky] -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-5700) Benchmark for Resource class (protobuf vs. C++)
[ https://issues.apache.org/jira/browse/MESOS-5700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Klaus Ma updated MESOS-5700: Attachment: name_roleId.diff > Benchmark for Resource class (protobuf vs. C++) > --- > > Key: MESOS-5700 > URL: https://issues.apache.org/jira/browse/MESOS-5700 > Project: Mesos > Issue Type: Bug >Reporter: Klaus Ma >Assignee: Klaus Ma > Attachments: hashmap.diff, name_roleId.diff, port.perf.log, > reservation.perf.log > > > Add benchmark of Resource class for Allocation Performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-5700) Benchmark for Resource class (protobuf vs. C++)
[ https://issues.apache.org/jira/browse/MESOS-5700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Klaus Ma updated MESOS-5700: Attachment: hashmap.diff > Benchmark for Resource class (protobuf vs. C++) > --- > > Key: MESOS-5700 > URL: https://issues.apache.org/jira/browse/MESOS-5700 > Project: Mesos > Issue Type: Bug >Reporter: Klaus Ma >Assignee: Klaus Ma > Attachments: hashmap.diff, port.perf.log, reservation.perf.log > > > Add benchmark of Resource class for Allocation Performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-5700) Benchmark for Resource class (protobuf vs. C++)
[
https://issues.apache.org/jira/browse/MESOS-5700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15381362#comment-15381362
]
Klaus Ma commented on MESOS-5700:
-
Did more test based on the Resources benchmark:
1. make Resource's name & role to be enum/int will improve 400ms per 10
operator on 1000 resource.
2. using hashmap in Resources will improve 2 s per 10 operator on 4000
resource; it should be more improved if using name + role as key. The overhead
of hash map is 70ms per 5000 operators on 4 resources.
Please refer to the attachment for the code diff I used.
*Test result for Resource's name & role*:
Using name() & role() in {{addable()}}
{code}
Took 577.479396ms to perform 10 'total += r' operations on cpus(role_0,
principal_0, {key_0: value_0}):1; ...
Took 647.031984ms to perform 10 'total -= r' operations on cpus(role_0,
principal_0, {key_0: value_0}):1; ...
Took 620.093549ms to perform 10 'total = total + r' operations on cpus(role_0,
principal_0, {key_0: value_0}):1; ...
Took 693.93056ms to perform 10 'total = total - r' operations on cpus(role_0,
principal_0, {key_0: value_0}):1; ...
{code}
without name() but with role() in {{addable()}}
{code}
Took 279.432157ms to perform 10 'total += r' operations on cpus(role_0,
principal_0, {key_0: value_0}):1; ...
Took 647.879612ms to perform 10 'total -= r' operations on cpus(role_0,
principal_0, {key_0: value_0}):1; ...
Took 323.552389ms to perform 10 'total = total + r' operations on cpus(role_0,
principal_0, {key_0: value_0}):1; ...
Took 682.131788ms to perform 10 'total = total - r' operations on cpus(role_0,
principal_0, {key_0: value_0}):1; ...
{code}
without name() and with role_id() in {{addable()}}
{code}
Took 175.373644ms to perform 10 'total += r' operations on cpus(0, principal_0,
{key_0: value_0}):1; cpus(...
Took 598.86704ms to perform 10 'total -= r' operations on cpus(0, principal_0,
{key_0: value_0}):1; cpus(...
Took 222.053999ms to perform 10 'total = total + r' operations on cpus(0,
principal_0, {key_0: value_0}):1; cpus(...
Took 616.455333ms to perform 10 'total = total - r' operations on cpus(0,
principal_0, {key_0: value_0}):1; cpus(...
{code}
*Test result for hash map for Resources*:
With cpus/gpus/disk
{code}
Took 4.396598secs to perform 10 'total += r' operations on cpus(0, principal_0,
{key_0: value_0}):1; gpus(...
Took 4.73346secs to perform 10 'total -= r' operations on cpus(0, principal_0,
{key_0: value_0}):1; gpus(...
Took 4.437688secs to perform 10 'total = total + r' operations on cpus(0,
principal_0, {key_0: value_0}):1; gpus(...
Took 4.940758secs to perform 10 'total = total - r' operations on cpus(0,
principal_0, {key_0: value_0}):1; gpus(...
{code}
Using map for operator+=
{code}
Took 2.469885secs to perform 10 'total += r' operations on cpus(0, principal_0,
{key_0: value_0}):1; gpus(...
Took 4.806674secs to perform 10 'total -= r' operations on cpus(0, principal_0,
{key_0: value_0}):1; gpus(...
Took 3.0442secs to perform 10 'total = total + r' operations on cpus(0,
principal_0, {key_0: value_0}):1; gpus(...
Took 3.278178secs to perform 10 'total = total - r' operations on cpus(0,
principal_0, {key_0: value_0}):1; gpus(...
{code}
> Benchmark for Resource class (protobuf vs. C++)
> ---
>
> Key: MESOS-5700
> URL: https://issues.apache.org/jira/browse/MESOS-5700
> Project: Mesos
> Issue Type: Bug
>Reporter: Klaus Ma
>Assignee: Klaus Ma
> Attachments: port.perf.log, reservation.perf.log
>
>
> Add benchmark of Resource class for Allocation Performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-5700) Benchmark for Resource class (protobuf vs. C++)
[ https://issues.apache.org/jira/browse/MESOS-5700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Klaus Ma updated MESOS-5700: Attachment: port.perf.log > Benchmark for Resource class (protobuf vs. C++) > --- > > Key: MESOS-5700 > URL: https://issues.apache.org/jira/browse/MESOS-5700 > Project: Mesos > Issue Type: Bug >Reporter: Klaus Ma >Assignee: Klaus Ma > Attachments: port.perf.log, reservation.perf.log > > > Add benchmark of Resource class for Allocation Performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-5700) Benchmark for Resource class (protobuf vs. C++)
[ https://issues.apache.org/jira/browse/MESOS-5700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Klaus Ma updated MESOS-5700: Attachment: reservation.perf.log > Benchmark for Resource class (protobuf vs. C++) > --- > > Key: MESOS-5700 > URL: https://issues.apache.org/jira/browse/MESOS-5700 > Project: Mesos > Issue Type: Bug >Reporter: Klaus Ma >Assignee: Klaus Ma > Attachments: reservation.perf.log > > > Add benchmark of Resource class for Allocation Performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-5700) Benchmark for Resource class (protobuf vs. C++)
[
https://issues.apache.org/jira/browse/MESOS-5700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15380797#comment-15380797
]
Klaus Ma commented on MESOS-5700:
-
Just did a profiling for Resources class based on the benchmark, please refer
to the attachment for the detail. Here's suggestion to improve the performance:
1. In reservation.perf.log, it shows {{operator!=(string, string)}} in
{{addable/subtractable}} is the heavy function; so I'm thinking to
* Make name of Resource to be enum instread of string (as we discussed in
the doc)
* Introduce roleId (uint64_t), similar to uid in Linux; {{roleId}} is used
within Mesos master/agent
2. In port.perf.log, it shows {{ranges validation}} is the heavy function; no
conclusion for now, but it's better to refine {{Resource::validate}}:
* role validation is not necessary in every operator/loop
* ranges validation maybe not necessary in every operator/loop,
{{coalesce}} may handle it (investigating)
cc [~bmahler]/[~gyliu]/[~alexr]
> Benchmark for Resource class (protobuf vs. C++)
> ---
>
> Key: MESOS-5700
> URL: https://issues.apache.org/jira/browse/MESOS-5700
> Project: Mesos
> Issue Type: Bug
>Reporter: Klaus Ma
>Assignee: Klaus Ma
>
> Add benchmark of Resource class for Allocation Performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (MESOS-5123) Docker task may fail if path to agent work_dir is relative.
[
https://issues.apache.org/jira/browse/MESOS-5123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Klaus Ma reassigned MESOS-5123:
---
Assignee: Klaus Ma
> Docker task may fail if path to agent work_dir is relative.
>
>
> Key: MESOS-5123
> URL: https://issues.apache.org/jira/browse/MESOS-5123
> Project: Mesos
> Issue Type: Improvement
> Components: docker
>Affects Versions: 1.0.0
>Reporter: Alexander Rukletsov
>Assignee: Klaus Ma
> Labels: docker, documentation, mesosphere
>
> When a local folder for agent’s {{\-\-work_dir}} is specified (e.g.,
> {{\-\-work_dir=w/s}}) docker complains that there are forbidden symbols in a
> *local* volume name. Specifying an absolute path (e.g.,
> {{\-\-work_dir=/tmp}}) solves the problem.
> Docker error observed:
> {noformat}
> docker: Error response from daemon: create
> w/s/slaves/33b8fe47-e9e0-468a-83a6-98c1e3537e59-S1/frameworks/33b8fe47-e9e0-468a-83a6-98c1e3537e59-0001/executors/docker-test/runs/3cc5cb04-d0a9-490e-94d5-d446b66c97cc:
> volume name invalid:
> "w/s/slaves/33b8fe47-e9e0-468a-83a6-98c1e3537e59-S1/frameworks/33b8fe47-e9e0-468a-83a6-98c1e3537e59-0001/executors/docker-test/runs/3cc5cb04-d0a9-490e-94d5-d446b66c97cc"
> includes invalid characters for a local volume name, only
> "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed.
> {noformat}
> First off, it is not obvious that Mesos always creates a volume for the
> sandbox. We may want to document it.
> Second, it's hard to understand that local {{work_dir}} can trigger forbidden
> symbols error in docker. Does it make sense to check it during agent launch
> if docker containerizer is enabled? Or reject docker tasks during task
> validation?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-5841) Clean up `FlagsBase::add`
[
https://issues.apache.org/jira/browse/MESOS-5841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Klaus Ma updated MESOS-5841:
Shepherd: Michael Park
> Clean up `FlagsBase::add`
> -
>
> Key: MESOS-5841
> URL: https://issues.apache.org/jira/browse/MESOS-5841
> Project: Mesos
> Issue Type: Improvement
> Components: stout
>Affects Versions: 1.0.0
>Reporter: Greg Mann
>Assignee: Klaus Ma
> Labels: mesosphere
>
> In the definition for {{FlagsBase}}, we currently have 20 overloads for the
> {{FlagsBase::add}} function. This makes both the {{FlagsBase}} class
> definition and the {{flags.cpp}} files in Mesos difficult to read. We should
> clean up {{FlagsBase::add}} so that it does not require so many overloads.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (MESOS-5841) Clean up `FlagsBase::add`
[
https://issues.apache.org/jira/browse/MESOS-5841?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Klaus Ma reassigned MESOS-5841:
---
Assignee: Klaus Ma
> Clean up `FlagsBase::add`
> -
>
> Key: MESOS-5841
> URL: https://issues.apache.org/jira/browse/MESOS-5841
> Project: Mesos
> Issue Type: Improvement
> Components: stout
>Affects Versions: 1.0.0
>Reporter: Greg Mann
>Assignee: Klaus Ma
> Labels: mesosphere
>
> In the definition for {{FlagsBase}}, we currently have 20 overloads for the
> {{FlagsBase::add}} function. This makes both the {{FlagsBase}} class
> definition and the {{flags.cpp}} files in Mesos difficult to read. We should
> clean up {{FlagsBase::add}} so that it does not require so many overloads.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-4811) Reusable/Cacheable Offer
[ https://issues.apache.org/jira/browse/MESOS-4811?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15374550#comment-15374550 ] Klaus Ma commented on MESOS-4811: - [~a10gupta], thanks for your interesting on this EPIC; this EPIC dependent on several other EPICs, e.g. "Manage offer in allocator" (MESOS-4553), Revocable by default, performance improvement of Resources/Sorter. So I'd suggest to join the discussion other EPICs before the working on this one :). Currently, we're focus on performance improvement of Resources/Sorter as discussion in dev@. > Reusable/Cacheable Offer > > > Key: MESOS-4811 > URL: https://issues.apache.org/jira/browse/MESOS-4811 > Project: Mesos > Issue Type: Bug > Components: allocation >Reporter: Klaus Ma >Assignee: Abhishek Dasgupta > Labels: tech-debt > > Currently, the resources are return back to allocator when task finished; and > those resources are not allocated to framework until next allocation cycle. > The performance is low for short running tasks (MESOS-3078). The proposed > solution is to let framework keep using the offer until allocator decide to > rescind it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (MESOS-4923) Treat revocable resources as a separate pool when considering fairness
[
https://issues.apache.org/jira/browse/MESOS-4923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Klaus Ma reassigned MESOS-4923:
---
Assignee: Klaus Ma
> Treat revocable resources as a separate pool when considering fairness
> --
>
> Key: MESOS-4923
> URL: https://issues.apache.org/jira/browse/MESOS-4923
> Project: Mesos
> Issue Type: Bug
> Components: allocation
>Reporter: Guangya Liu
>Assignee: Klaus Ma
>
> The current logic of roleSorter is that when it do role sorter, the resources
> in it will include both regular resources and revocable resources, and this
> may not accurate for some cases, take the following case as an instance:
> 1) framework1 and framework2.
> 2) framework1 got 1 reserved cpu and 9 revocable cpu. cpu(r1):1;cpu(*){REV}:9
> 3) framework2 got 9 reserved cpus: cpu(r1):9
> When allocator allocate resources in next cycle, framework2 will be handled
> first as it has less SCALAR resources than framework1, but this may not be
> right for some cases as framework1 is using only 1 reserved resources and
> other resources are revocable which can be easily got evicted.
> A proposal here is treat revocable resources as a separate pool when
> considering fairness, this can be achieved by introducing a new sorter for
> revocable resources so as to distinguish the sorter for regular resources and
> revocable resources. To the built in allocator, the logic would be as this:
> 1) Quota Role Sorter
> 2) non-revocable Role Sorter
> 3) Revocable Role Sorter
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (MESOS-5603) Improve test cases in ValueTest
[
https://issues.apache.org/jira/browse/MESOS-5603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Klaus Ma reassigned MESOS-5603:
---
Assignee: Klaus Ma
> Improve test cases in ValueTest
> ---
>
> Key: MESOS-5603
> URL: https://issues.apache.org/jira/browse/MESOS-5603
> Project: Mesos
> Issue Type: Improvement
>Reporter: Klaus Ma
>Assignee: Klaus Ma
>
> In {{ValueTest.*}}, it did not include enough cases for positive & negative
> cases. It's better to add more cases to show which cases are allowed and
> which cases are disallowed.
> And the related document should be also updated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (MESOS-5700) Benchmark for Resource class (protobuf vs. C++)
[ https://issues.apache.org/jira/browse/MESOS-5700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Klaus Ma reassigned MESOS-5700: --- Assignee: Klaus Ma > Benchmark for Resource class (protobuf vs. C++) > --- > > Key: MESOS-5700 > URL: https://issues.apache.org/jira/browse/MESOS-5700 > Project: Mesos > Issue Type: Bug >Reporter: Klaus Ma >Assignee: Klaus Ma > > Add benchmark of Resource class for Allocation Performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (MESOS-4967) Oversubscription for reservation
[ https://issues.apache.org/jira/browse/MESOS-4967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Klaus Ma reassigned MESOS-4967: --- Assignee: Klaus Ma > Oversubscription for reservation > > > Key: MESOS-4967 > URL: https://issues.apache.org/jira/browse/MESOS-4967 > Project: Mesos > Issue Type: Epic > Components: allocation, framework, master >Reporter: Klaus Ma >Assignee: Klaus Ma > Labels: IBM, mesosphere > > Reserved resources allow frameworks and cluster operators to ensure > sufficient resources are available when needed. Reservations are usually > made to guarantee there are enough resources under peak loads. Often times, > reserved resources are not actually allocated; in other words, the frameworks > do not use those resources and they sit reserved, but idle. > This underutilization is either an opportunity cost or a direct cost, > particularly to the cluster operator. Reserved but unallocated resources > held by a Lender Framework could be optimistically offered to other > frameworks, which we refer to as Tenant Frameworks. When the resources are > requested back by the Lender Framework, some of the Tenant Framework’s tasks > are evicted and the original resource offer guarantee is preserved. > The first step is to identify when resources are reserved, but not allocated. > We then offer these reserved resources to other frameworks, but mark these > offered resources as revocable resources. This allows Tenant Frameworks to > use these resources temporarily in a 'best-effort' fashion, knowing that they > could be revoked or reclaimed at any time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (MESOS-5739) Fix Value parsing code to only accept the canonical formats
[
https://issues.apache.org/jira/browse/MESOS-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Klaus Ma reassigned MESOS-5739:
---
Assignee: Klaus Ma
> Fix Value parsing code to only accept the canonical formats
> ---
>
> Key: MESOS-5739
> URL: https://issues.apache.org/jira/browse/MESOS-5739
> Project: Mesos
> Issue Type: Bug
> Components: master
>Reporter: Klaus Ma
>Assignee: Klaus Ma
>
> We should fix the value parsing code to only accept the canonical formats as
> defined in http://mesos.apache.org/documentation/latest/attributes-resources/
> , the behaviour after the fix is as following:
> {code}
> 1. Did not support [1-2, [3-4]] as Ranges; it should be [1-2, 3-4].
> 2. Did not support {a{b, c}d} as Set; it should be {ab, cd}
> 3. Add check for Text against [a-zA-Z0-9_/.-]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-2845) Command tasks lead to a mixing of revocable / non-revocable cpus and memory within the container.
[
https://issues.apache.org/jira/browse/MESOS-2845?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Klaus Ma updated MESOS-2845:
Assignee: (was: Klaus Ma)
> Command tasks lead to a mixing of revocable / non-revocable cpus and memory
> within the container.
> -
>
> Key: MESOS-2845
> URL: https://issues.apache.org/jira/browse/MESOS-2845
> Project: Mesos
> Issue Type: Bug
> Components: slave
>Reporter: Benjamin Mahler
> Labels: twitter
>
> Due to the hack
> [here|https://github.com/apache/mesos/blob/9a5788801e7fc95fce99749a23803fc52c67c0ce/src/slave/slave.cpp#L3101],
> where we add a small set of resources into the command executor:
> {code}
> ExecutorInfo Slave::getExecutorInfo(
> const FrameworkID& frameworkId,
> const TaskInfo& task)
> {
> if (task.has_command()) {
> ...
> // XXX: These are always non-revocable.
> // Add an allowance for the command executor. This does lead to a
> // small overcommit of resources.
> executor.mutable_resources()->MergeFrom(
> Resources::parse(
> "cpus:" + stringify(DEFAULT_EXECUTOR_CPUS) + ";" +
> "mem:" + stringify(DEFAULT_EXECUTOR_MEM.megabytes())).get());
> }
> ...
> }
> {code}
> The obvious extension here would be to make these revocable, but would be
> great to remove this hack entirely.
> Seems to originate in [r/22251|https://reviews.apache.org/r/22251/] from
> MESOS-1417.
> FYI [~idownes] [~jieyu]
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-3283) Improve allocation performance especially with large number of slaves and frameworks.
[ https://issues.apache.org/jira/browse/MESOS-3283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Klaus Ma updated MESOS-3283: Assignee: (was: Klaus Ma) > Improve allocation performance especially with large number of slaves and > frameworks. > - > > Key: MESOS-3283 > URL: https://issues.apache.org/jira/browse/MESOS-3283 > Project: Mesos > Issue Type: Improvement > Components: allocation >Affects Versions: 0.23.0 >Reporter: Mandeep Chadha > Labels: mesosphere, tech-debt > > Improve batch allocations performance especially with large number of slaves > and frameworks. > e.g. these are the allocation timings for 10K slaves and varying number of > frameworks. > Using 1 slaves and 1 frameworks > Added 1 slaves in 14.50836112secs > Updated 1 slaves in 18.665093703secs > [ OK ] > SlaveCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/12 (34983 > ms) > [ RUN ] > SlaveCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/13 > Using 1 slaves and 50 frameworks > Added 1 slaves in 51.534229549secs > Updated 1 slaves in 57.131554303secs > [ OK ] > SlaveCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/13 (110449 > ms) > [ RUN ] > SlaveCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/14 > Using 1 slaves and 100 frameworks > Added 1 slaves in 1.5891310434mins > Updated 1 slaves in 1.80562078148333mins > [ OK ] > SlaveCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/14 (205467 > ms) > [ RUN ] > SlaveCount/HierarchicalAllocator_BENCHMARK_Test.AddAndUpdateSlave/15 > Using 1 slaves and 200 frameworks > Added 1 slaves in 3.0750647275mins > Updated 1 slaves in 3.85846762096667mins -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-3889) Modify Oversubscription documentation to explicitly forbid the QoS Controller from killing executors running on optimistically offered resources.
[
https://issues.apache.org/jira/browse/MESOS-3889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Klaus Ma updated MESOS-3889:
Assignee: (was: Klaus Ma)
> Modify Oversubscription documentation to explicitly forbid the QoS Controller
> from killing executors running on optimistically offered resources.
> -
>
> Key: MESOS-3889
> URL: https://issues.apache.org/jira/browse/MESOS-3889
> Project: Mesos
> Issue Type: Bug
>Reporter: Artem Harutyunyan
> Labels: mesosphere
>
> The oversubcription documentation currently assumes that oversubscribed
> resources ({{USAGE_SLACK}}) are the only type of revocable resources.
> Optimistic offers will add a second type of revocable resource
> ({{ALLOCATION_SLACK}}) that should not be acted upon by oversubscription
> components.
> For example, the [oversubscription
> doc|http://mesos.apache.org/documentation/latest/oversubscription/] says the
> following:
> {quote}
> NOTE: If any resource used by a task or executor is revocable, the whole
> container is treated as a revocable container and can therefore be killed or
> throttled by the QoS Controller.
> {quote}
> which we may amend to something like:
> {quote}
> NOTE: If any resource used by a task or executor is revocable usage slack,
> the whole container is treated as an oversubscribed container and can
> therefore be killed or throttled by the QoS Controller.
> {quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-5739) Fix Value parsing code to only accept the canonical formats
[
https://issues.apache.org/jira/browse/MESOS-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Klaus Ma updated MESOS-5739:
Assignee: (was: Klaus Ma)
> Fix Value parsing code to only accept the canonical formats
> ---
>
> Key: MESOS-5739
> URL: https://issues.apache.org/jira/browse/MESOS-5739
> Project: Mesos
> Issue Type: Bug
> Components: master
>Reporter: Klaus Ma
>
> We should fix the value parsing code to only accept the canonical formats as
> defined in http://mesos.apache.org/documentation/latest/attributes-resources/
> , the behaviour after the fix is as following:
> {code}
> 1. Did not support [1-2, [3-4]] as Ranges; it should be [1-2, 3-4].
> 2. Did not support {a{b, c}d} as Set; it should be {ab, cd}
> 3. Add check for Text against [a-zA-Z0-9_/.-]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-5619) Add task_num to mesos-execute
[
https://issues.apache.org/jira/browse/MESOS-5619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Klaus Ma updated MESOS-5619:
Assignee: (was: Klaus Ma)
> Add task_num to mesos-execute
> -
>
> Key: MESOS-5619
> URL: https://issues.apache.org/jira/browse/MESOS-5619
> Project: Mesos
> Issue Type: Bug
> Components: cli
>Reporter: Klaus Ma
>
> According to current code, {{mesos-execute}} will only launch one task. It's
> better to add a parameter to special how many task to launch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-5603) Improve test cases in ValueTest
[
https://issues.apache.org/jira/browse/MESOS-5603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Klaus Ma updated MESOS-5603:
Assignee: (was: Klaus Ma)
> Improve test cases in ValueTest
> ---
>
> Key: MESOS-5603
> URL: https://issues.apache.org/jira/browse/MESOS-5603
> Project: Mesos
> Issue Type: Improvement
>Reporter: Klaus Ma
>
> In {{ValueTest.*}}, it did not include enough cases for positive & negative
> cases. It's better to add more cases to show which cases are allowed and
> which cases are disallowed.
> And the related document should be also updated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-4838) Update unavailable in batch to avoid several allocate(slaveId) call
[
https://issues.apache.org/jira/browse/MESOS-4838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Klaus Ma updated MESOS-4838:
Assignee: (was: Klaus Ma)
> Update unavailable in batch to avoid several allocate(slaveId) call
> ---
>
> Key: MESOS-4838
> URL: https://issues.apache.org/jira/browse/MESOS-4838
> Project: Mesos
> Issue Type: Bug
>Reporter: Klaus Ma
>
> In "/machine/schedule", all machines in master will trigger a
> {{allocate(slaveId)}} which will increase the workload of master. The
> proposal of this JIRA is to update unavailable in batch to avoid several
> {{allocate(slaveId)}} call.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-5123) Docker task may fail if path to agent work_dir is relative.
[
https://issues.apache.org/jira/browse/MESOS-5123?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Klaus Ma updated MESOS-5123:
Assignee: (was: Klaus Ma)
> Docker task may fail if path to agent work_dir is relative.
>
>
> Key: MESOS-5123
> URL: https://issues.apache.org/jira/browse/MESOS-5123
> Project: Mesos
> Issue Type: Improvement
> Components: docker
>Affects Versions: 1.0.0
>Reporter: Alexander Rukletsov
> Labels: docker, documentation, mesosphere
>
> When a local folder for agent’s {{\-\-work_dir}} is specified (e.g.,
> {{\-\-work_dir=w/s}}) docker complains that there are forbidden symbols in a
> *local* volume name. Specifying an absolute path (e.g.,
> {{\-\-work_dir=/tmp}}) solves the problem.
> Docker error observed:
> {noformat}
> docker: Error response from daemon: create
> w/s/slaves/33b8fe47-e9e0-468a-83a6-98c1e3537e59-S1/frameworks/33b8fe47-e9e0-468a-83a6-98c1e3537e59-0001/executors/docker-test/runs/3cc5cb04-d0a9-490e-94d5-d446b66c97cc:
> volume name invalid:
> "w/s/slaves/33b8fe47-e9e0-468a-83a6-98c1e3537e59-S1/frameworks/33b8fe47-e9e0-468a-83a6-98c1e3537e59-0001/executors/docker-test/runs/3cc5cb04-d0a9-490e-94d5-d446b66c97cc"
> includes invalid characters for a local volume name, only
> "[a-zA-Z0-9][a-zA-Z0-9_.-]" are allowed.
> {noformat}
> First off, it is not obvious that Mesos always creates a volume for the
> sandbox. We may want to document it.
> Second, it's hard to understand that local {{work_dir}} can trigger forbidden
> symbols error in docker. Does it make sense to check it during agent launch
> if docker containerizer is enabled? Or reject docker tasks during task
> validation?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-3898) Identify and implement test cases for handling a race between optimistic lender and tenant offers.
[ https://issues.apache.org/jira/browse/MESOS-3898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Klaus Ma updated MESOS-3898: Assignee: (was: Klaus Ma) > Identify and implement test cases for handling a race between optimistic > lender and tenant offers. > -- > > Key: MESOS-3898 > URL: https://issues.apache.org/jira/browse/MESOS-3898 > Project: Mesos > Issue Type: Bug >Reporter: Artem Harutyunyan > Labels: mesosphere > > An example is the when lender launches the task on an agent followed by a > borrower launching a task on the same agent before the optimistic offer is > rescinded. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-5231) Create Design Doc for Manage offers in allocator
[ https://issues.apache.org/jira/browse/MESOS-5231?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Klaus Ma updated MESOS-5231: Assignee: (was: Klaus Ma) > Create Design Doc for Manage offers in allocator > > > Key: MESOS-5231 > URL: https://issues.apache.org/jira/browse/MESOS-5231 > Project: Mesos > Issue Type: Bug >Reporter: Klaus Ma > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-887) Scheduler driver should use exited() to detect disconnection with Master.
[ https://issues.apache.org/jira/browse/MESOS-887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Klaus Ma updated MESOS-887: --- Assignee: (was: Klaus Ma) > Scheduler driver should use exited() to detect disconnection with Master. > - > > Key: MESOS-887 > URL: https://issues.apache.org/jira/browse/MESOS-887 > Project: Mesos > Issue Type: Improvement > Components: framework, master >Affects Versions: 0.13.0, 0.14.0, 0.14.1, 0.14.2, 0.15.0, 0.16.0 >Reporter: Benjamin Mahler > Labels: reliability, twitter > > The Scheduler Driver already links with the master, but it does not use the > built in exited() notification from libprocess to detect socket closure. > Of particular concern is that, if the socket breaks and subsequent messages > are successfully sent on ephemeral sockets, then we don't re-register with > the master. Messages may have been dropped. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-4320) Did not rescind offer if offer did not include USAGE_SLACK
[ https://issues.apache.org/jira/browse/MESOS-4320?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Klaus Ma updated MESOS-4320: Assignee: (was: Klaus Ma) > Did not rescind offer if offer did not include USAGE_SLACK > -- > > Key: MESOS-4320 > URL: https://issues.apache.org/jira/browse/MESOS-4320 > Project: Mesos > Issue Type: Bug >Reporter: Klaus Ma > > When Slave update its USAGE_SLACK resource, the master will rescind offer if > the offer included revocable resource. But in Optimistic Offer, the offer > with ALLOCATION_SLACK resources should not rescind. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-4967) Oversubscription for reservation
[ https://issues.apache.org/jira/browse/MESOS-4967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Klaus Ma updated MESOS-4967: Assignee: (was: Klaus Ma) > Oversubscription for reservation > > > Key: MESOS-4967 > URL: https://issues.apache.org/jira/browse/MESOS-4967 > Project: Mesos > Issue Type: Epic > Components: allocation, framework, master >Reporter: Klaus Ma > Labels: IBM, mesosphere > > Reserved resources allow frameworks and cluster operators to ensure > sufficient resources are available when needed. Reservations are usually > made to guarantee there are enough resources under peak loads. Often times, > reserved resources are not actually allocated; in other words, the frameworks > do not use those resources and they sit reserved, but idle. > This underutilization is either an opportunity cost or a direct cost, > particularly to the cluster operator. Reserved but unallocated resources > held by a Lender Framework could be optimistically offered to other > frameworks, which we refer to as Tenant Frameworks. When the resources are > requested back by the Lender Framework, some of the Tenant Framework’s tasks > are evicted and the original resource offer guarantee is preserved. > The first step is to identify when resources are reserved, but not allocated. > We then offer these reserved resources to other frameworks, but mark these > offered resources as revocable resources. This allows Tenant Frameworks to > use these resources temporarily in a 'best-effort' fashion, knowing that they > could be revoked or reclaimed at any time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-5022) Provide LDAP as default authorisation
[
https://issues.apache.org/jira/browse/MESOS-5022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Klaus Ma updated MESOS-5022:
Assignee: (was: Klaus Ma)
> Provide LDAP as default authorisation
> -
>
> Key: MESOS-5022
> URL: https://issues.apache.org/jira/browse/MESOS-5022
> Project: Mesos
> Issue Type: Epic
>Reporter: Klaus Ma
>
> The default authorisation/ACL is configured by {{json}} file; operator has to
> restart master if any new user was added. It's better to provide LDAP as
> default ACL:
> 1. Provide an real example on ACL interface
> 2. Provide an default auth plugin for user to use
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-3897) Identify and implement test cases for verifying eviction logic in the agent
[ https://issues.apache.org/jira/browse/MESOS-3897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Klaus Ma updated MESOS-3897: Assignee: (was: Klaus Ma) > Identify and implement test cases for verifying eviction logic in the agent > --- > > Key: MESOS-3897 > URL: https://issues.apache.org/jira/browse/MESOS-3897 > Project: Mesos > Issue Type: Bug >Reporter: Artem Harutyunyan > Labels: mesosphere > -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-4811) Reusable/Cacheable Offer
[ https://issues.apache.org/jira/browse/MESOS-4811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Klaus Ma updated MESOS-4811: Assignee: (was: Klaus Ma) > Reusable/Cacheable Offer > > > Key: MESOS-4811 > URL: https://issues.apache.org/jira/browse/MESOS-4811 > Project: Mesos > Issue Type: Bug > Components: allocation >Reporter: Klaus Ma > Labels: tech-debt > > Currently, the resources are return back to allocator when task finished; and > those resources are not allocated to framework until next allocation cycle. > The performance is low for short running tasks (MESOS-3078). The proposed > solution is to let framework keep using the offer until allocator decide to > rescind it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-4265) Launch tasks after executors evicted
[ https://issues.apache.org/jira/browse/MESOS-4265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Klaus Ma updated MESOS-4265: Assignee: (was: Klaus Ma) > Launch tasks after executors evicted > > > Key: MESOS-4265 > URL: https://issues.apache.org/jira/browse/MESOS-4265 > Project: Mesos > Issue Type: Bug >Reporter: Klaus Ma > > Launch tasks after evicting executor which is used by other framework as > ALLOCATION_SLACK -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-3431) Refactor Protobuf tests
[
https://issues.apache.org/jira/browse/MESOS-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Klaus Ma updated MESOS-3431:
Assignee: (was: Klaus Ma)
> Refactor Protobuf tests
> ---
>
> Key: MESOS-3431
> URL: https://issues.apache.org/jira/browse/MESOS-3431
> Project: Mesos
> Issue Type: Task
> Components: test
>Reporter: Alexander Rukletsov
>Priority: Minor
>
> {{ProtobufTest.JSON}} test does several things simultaneously, including
> message instantiation, conversion, parsing. We should split this test into
> several independent ones that test just one thing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-4303) Support resources re-shuffle when new framework registered
[ https://issues.apache.org/jira/browse/MESOS-4303?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Klaus Ma updated MESOS-4303: Assignee: (was: Klaus Ma) > Support resources re-shuffle when new framework registered > -- > > Key: MESOS-4303 > URL: https://issues.apache.org/jira/browse/MESOS-4303 > Project: Mesos > Issue Type: Epic > Components: master >Reporter: Klaus Ma > > Currently, if one famework launched tasks in all resources, the other > framework has to wait those tasks completed to get deserved resources. That's > un-fair to the new framework, there're several EPICs on such case but still > can not resolve it: > - Quota: balance resouces based on un-allocated resources; the used > resources can not be reclaim back when new framework registered > - Optimistic Offer Phase 1 (oversubscription of reservation): the un-used > reserved resources can be used by the new framework; but still can not > reclaim un-reserved resources back. > - Optimistic Offer Phase 2: in phase 2, all resources are offered to all > framework; besides resources conflict, it also need to re-shuffle resources > if one framework over occupied. > The target of this EPIC is to > 1. Enhance DRF to identify which resources should be reclaim > 2. Define the protocol between Mesos & Framework to reclaim resources > gracefully > To keep the compatibility, this feature is controlled by master paramenter, > e.g. --enable-reclaim. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-4627) Improve Ranges parsing to handle single values.
[
https://issues.apache.org/jira/browse/MESOS-4627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Klaus Ma updated MESOS-4627:
Assignee: (was: Klaus Ma)
> Improve Ranges parsing to handle single values.
> ---
>
> Key: MESOS-4627
> URL: https://issues.apache.org/jira/browse/MESOS-4627
> Project: Mesos
> Issue Type: Improvement
>Reporter: Benjamin Mahler
> Labels: newbie
>
> Users expect to be able to write a single value entry when specifying ports:
> {noformat}
> ./bin/mesos-slave.sh --resources="ports:[80, 100-120]" --master=localhost:5050
> ...
> Failed to determine slave resources: Failed to parse resource ports value
> [80, 100-120] error Expecting one or more "ranges"
> {noformat}
> We should improve our parsing ability here. We should also consider
> stringifying using this more succinct format.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-3888) Support distinguishing revocable resources in the Resource protobuf.
[
https://issues.apache.org/jira/browse/MESOS-3888?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Klaus Ma updated MESOS-3888:
Assignee: (was: Klaus Ma)
> Support distinguishing revocable resources in the Resource protobuf.
>
>
> Key: MESOS-3888
> URL: https://issues.apache.org/jira/browse/MESOS-3888
> Project: Mesos
> Issue Type: Bug
>Reporter: Artem Harutyunyan
> Labels: mesosphere
>
> Add enum type into RevocableInfo:
> * Framework need to assign RevocableInfo when launching task; if it’s not
> assign, use reserved resources. Framework need to identify which resources
> it’s using
> * Oversubscription resources need to assign the type by Agent (MESOS-3930)
> * Update Oversubscription document that OO has over-subscribe the Allocation
> Slack and recommend QoS to handle the usage slack only. (MESOS-3889)
> {code}
> message Resource {
> ...
> message RevocableInfo {
>enum Type {
> // Under-utilized, allocated resources. Controlled by
> // oversubscription (QoSController & ResourceEstimator).
> USAGE_SLACK = 1;
> // Unallocated, reserved resources.
> // Controlled by optimistic offers (Allocator).
> ALLOCATION_SLACK = 2;
>}
>optional Type type = 1;
> }
> ...
> optional RevocableInfo revocable = 9;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-3896) Add accounting for reservation slack in the allocator.
[
https://issues.apache.org/jira/browse/MESOS-3896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Klaus Ma updated MESOS-3896:
Assignee: (was: Klaus Ma)
> Add accounting for reservation slack in the allocator.
> --
>
> Key: MESOS-3896
> URL: https://issues.apache.org/jira/browse/MESOS-3896
> Project: Mesos
> Issue Type: Bug
>Reporter: Artem Harutyunyan
> Labels: mesosphere
>
> MESOS-XXX: Optimsistic accounter
> {code}
> class HierarchicalAllocatorProcess
> {
> struct Slave
> {
> ...
> struct Optimistic
> {
> Resources total; // The total allocation slack resources
> Resources allocated; // The allocated allocation slack resources
> };
>
> Optimistic optimistic;
> };
> }
> {code}
> MESOS-4146: flatten & allocationSlack for Optimistic Offer
> {code}
> class Resources
> {
> // Returns a Resources object with the same amount of each resource
> // type as these Resources, but with all Resource objects marked as
> // the specified `RevocableInfo::Type`; the other attribute is not
> // affected.
> Resources flatten(Resource::RevocableInfo::Type type);
> // Return a Resources object that:
> // - if role is given, the resources did not include role's reserved
> // resources.
> // - the resources's revocable type is `ALLOCATION_SLACK`
> // - the role of resources is set to "*"
> Resources allocationSlack(Option role = None());
> }
> {code}
> MESOS-XXX: Allocate the allocation_slack resources to framework
> {code}
> void HierarchicalAllocatorProcess::allocate(
> const hashset& slaveIds_)
> {
> foreach slave; foreach role; foreach framework
> {
> Resource optimistic;
> if (framework.revocable) {
> Resources total =
> slaves[slaveId].optimistic.total.allocationSlack(role);
> optimistic = total - slaves[slaveId].optimistic.allocated;
> }
> ...
> offerable[frameworkId][slaveId] += resources + optimistic;
> ...
> slaves[slaveId].optimistic.allocated += optimistic;
> }
> }
> {code}
>
> Here's some consideration about `ALLOCATION_SLACK`:
> 1. 'Old' resources (available/total) did not include ALLOCATION_SLACK
> 2. After `Quota`, `remainingClusterResources.contains` should not check
> ALLOCATION_SLACK; if there no enough resources, master can still offer
> ALLOCATION_SALCK resources.
> 3. In sorter, it'll not include ALLOCATION_SLACK; as those resources are
> borrowed from other role/framework
> 4. If either normal resources or ALLOCATION_SLACK resources are
> allocable/!filtered, it can be offered to framework
> 5. Currently, allocator will assign all ALLOCATION_SALCK resources in slave
> to one framework
> MESOS-XXX: Update ALLOCATION_SLACK for dynamic reservation (updateAllocation)
> {code}
> void HierarchicalAllocatorProcess::updateAllocation(
> const FrameworkID& frameworkId,
> const SlaveID& slaveId,
> const vector& operations)
> {
> ...
> Try updatedOptimistic =
> slaves[slaveId].optimistic.total.apply(operations);
> CHECK_SOME(updatedTotal);
> slaves[slaveId].optimistic.total =
>
> updatedOptimistic.get().stateless().reserved().flatten(ALLOCATION_SLACK);
> ...
> }
> {code}
>
> MESOS-XXX: Add ALLOCATION_SLACK when slaver register/re-register (addSlave)
> {code}
> void HierarchicalAllocatorProcess::addSlave(
> const SlaveID& slaveId,
> const SlaveInfo& slaveInfo,
> const Option& unavailability,
> const Resources& total,
> const hashmap& used)
> {
> ...
> slaves[slaveId].optimistic.total =
> total.stateless().reserved().flatten(ALLOCATION_SLACK);
> ...
> }
> {code}
>
> No need to handle `removeSlave`, it'll all related info from `slaves`
> including `optimistic`.
> MESOS-XXX: return resources to allocator (recoverResources)
> {code}
> void HierarchicalAllocatorProcess::recoverResources(
> const FrameworkID& frameworkId,
> const SlaveID& slaveId,
> const Resources& resources,
> const Option& filters)
> {
> if (slaves.contains(slaveId))
> {
> ...
> slaves[slaveId].optimistic.allocated -= resources.allocationSlack();
> ...
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-4741) Add role information for static reservation in /master/roles
[
https://issues.apache.org/jira/browse/MESOS-4741?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Klaus Ma updated MESOS-4741:
Assignee: (was: Klaus Ma)
> Add role information for static reservation in /master/roles
>
>
> Key: MESOS-4741
> URL: https://issues.apache.org/jira/browse/MESOS-4741
> Project: Mesos
> Issue Type: Bug
> Components: HTTP API
>Reporter: Klaus Ma
>
> In {{/master/roles}}, it should show static reservation roles if there's no
> tasks.
> {code}
> Klauss-MacBook-Pro:mesos klaus$ curl http://localhost:5050/master/roles.json
> | python -m json.tool
> % Total% Received % Xferd Average Speed TimeTime Time
> Current
> Dload Upload Total SpentLeft Speed
> 10093 100930 0 13907 0 --:--:-- --:--:-- --:--:-- 15500
> {
> "roles": [
> {
> "frameworks": [],
> "name": "*",
> "resources": {
> "cpus": 0,
> "disk": 0,
> "mem": 0
> },
> "weight": 1.0
> }
> ]
> }
> {code}
> After submit tasks to r1, it'll show roles.
> {code}
> Klauss-MacBook-Pro:mesos klaus$ curl http://localhost:5050/master/roles |
> python -m json.tool
> % Total% Received % Xferd Average Speed TimeTime Time
> Current
> Dload Upload Total SpentLeft Speed
> 100 221 100 2210 0 32721 0 --:--:-- --:--:-- --:--:-- 36833
> {
> "roles": [
> {
> "frameworks": [],
> "name": "*",
> "resources": {
> "cpus": 0,
> "disk": 0,
> "mem": 0
> },
> "weight": 1.0
> },
> {
> "frameworks": [
> "b4f15a2e-5d9a-4d31-a29e-7737af41c8e4-0002"
> ],
> "name": "r1",
> "resources": {
> "cpus": 1.0,
> "disk": 0,
> "mem": 0
> },
> "weight": 1.0
> }
> ]
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-4553) Manage offers in allocator.
[
https://issues.apache.org/jira/browse/MESOS-4553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Klaus Ma updated MESOS-4553:
Assignee: (was: Klaus Ma)
> Manage offers in allocator.
> ---
>
> Key: MESOS-4553
> URL: https://issues.apache.org/jira/browse/MESOS-4553
> Project: Mesos
> Issue Type: Epic
> Components: master
>Reporter: Klaus Ma
>
> Currently, the {{offers}} are managed by {{Master}} which introduces two
> issues:
> 1. In Quota, master rescind more offers to address race condition
> 2. Allocator can not modify offers: resources return to allocator and offer
> again, that impact resources utilisation & performance (MESOS-3078)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-5234) Add helper function to simplify tokenize handling
[
https://issues.apache.org/jira/browse/MESOS-5234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Klaus Ma updated MESOS-5234:
Assignee: (was: Klaus Ma)
> Add helper function to simplify tokenize handling
> -
>
> Key: MESOS-5234
> URL: https://issues.apache.org/jira/browse/MESOS-5234
> Project: Mesos
> Issue Type: Bug
> Components: stout
>Reporter: Klaus Ma
>Priority: Minor
>
> Based on [~jvanremoortere]'s suggestion on patch of MESOS-4627, it's better
> to add a helper function
> {code}
> foreachtoken(temp, ",\n", [](const string& token) { ... });
> {code}
> to simplify {{tokenize()}} handling.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-3070) Master CHECK failure if a framework uses duplicated task id.
[
https://issues.apache.org/jira/browse/MESOS-3070?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Klaus Ma updated MESOS-3070:
Assignee: (was: Klaus Ma)
> Master CHECK failure if a framework uses duplicated task id.
>
>
> Key: MESOS-3070
> URL: https://issues.apache.org/jira/browse/MESOS-3070
> Project: Mesos
> Issue Type: Bug
> Components: master
>Affects Versions: 0.22.1
>Reporter: Jie Yu
>
> We observed this in one of our testing cluster.
> One framework (under development) keeps launching tasks using the same
> task_id. We don't expect the master to crash even if the framework is not
> doing what it's supposed to do. However, under a series of events, this could
> happen and keeps crashing the master.
> 1) frameworkA launches task 'task_id_1' on slaveA
> 2) master fails over
> 3) slaveA has not re-registered yet
> 4) frameworkA re-registered and launches task 'task_id_1' on slaveB
> 5) slaveA re-registering and add task "task_id_1' to frameworkA
> 6) CHECK failure in addTask
> {noformat}
> I0716 21:52:50.759305 28805 master.hpp:159] Adding task 'task_id_1' with
> resources cpus(*):4; mem(*):32768 on slave
> 20150417-232509-1735470090-5050-48870-S25 (hostname)
> ...
> ...
> F0716 21:52:50.760136 28805 master.hpp:362] Check failed:
> !tasks.contains(task->task_id()) Duplicate task 'task_id_1' of framework
>
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-4558) Reduce the running time of benchmark tests.
[
https://issues.apache.org/jira/browse/MESOS-4558?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Klaus Ma updated MESOS-4558:
Assignee: (was: Klaus Ma)
> Reduce the running time of benchmark tests.
> ---
>
> Key: MESOS-4558
> URL: https://issues.apache.org/jira/browse/MESOS-4558
> Project: Mesos
> Issue Type: Task
>Reporter: Vinod Kone
> Labels: newbie++
>
> Currently benchmark tests take a long time (>5 hours). It would be nice to
> reduce the total time taken by the benchmark tests to enable us to run them
> on ASF CI.
> Command to run only benchmark tests
> {code}
> MESOS_BENCHMARK=1 GTEST_FILTER="*BENCHMARK*" make check
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-1718) Command executor can overcommit the slave.
[
https://issues.apache.org/jira/browse/MESOS-1718?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Klaus Ma updated MESOS-1718:
Assignee: (was: Klaus Ma)
> Command executor can overcommit the slave.
> --
>
> Key: MESOS-1718
> URL: https://issues.apache.org/jira/browse/MESOS-1718
> Project: Mesos
> Issue Type: Bug
> Components: slave
>Reporter: Benjamin Mahler
>
> Currently we give a small amount of resources to the command executor, in
> addition to resources used by the command task:
> https://github.com/apache/mesos/blob/0.20.0-rc1/src/slave/slave.cpp#L2448
> {code: title=}
> ExecutorInfo Slave::getExecutorInfo(
> const FrameworkID& frameworkId,
> const TaskInfo& task)
> {
> ...
> // Add an allowance for the command executor. This does lead to a
> // small overcommit of resources.
> executor.mutable_resources()->MergeFrom(
> Resources::parse(
> "cpus:" + stringify(DEFAULT_EXECUTOR_CPUS) + ";" +
> "mem:" + stringify(DEFAULT_EXECUTOR_MEM.megabytes())).get());
> ...
> }
> {code}
> This leads to an overcommit of the slave. Ideally, for command tasks we can
> "transfer" all of the task resources to the executor at the slave / isolation
> level.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-3930) Set resource type as USAGE_SLACK for Oversubscription
[ https://issues.apache.org/jira/browse/MESOS-3930?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Klaus Ma updated MESOS-3930: Assignee: (was: Klaus Ma) > Set resource type as USAGE_SLACK for Oversubscription > - > > Key: MESOS-3930 > URL: https://issues.apache.org/jira/browse/MESOS-3930 > Project: Mesos > Issue Type: Bug >Reporter: Guangya Liu > > Set resource type as USAGE_SLACK for Oversubscription so as to distinguish > the revocable resource for allocation slack and usage slack -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-5700) Benchmark for Resource class (protobuf vs. C++)
[ https://issues.apache.org/jira/browse/MESOS-5700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Klaus Ma updated MESOS-5700: Assignee: (was: Klaus Ma) > Benchmark for Resource class (protobuf vs. C++) > --- > > Key: MESOS-5700 > URL: https://issues.apache.org/jira/browse/MESOS-5700 > Project: Mesos > Issue Type: Bug >Reporter: Klaus Ma > > Add benchmark of Resource class for Allocation Performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-5701) Add benchmark for sorter performance
[ https://issues.apache.org/jira/browse/MESOS-5701?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Klaus Ma updated MESOS-5701: Assignee: (was: Klaus Ma) > Add benchmark for sorter performance > > > Key: MESOS-5701 > URL: https://issues.apache.org/jira/browse/MESOS-5701 > Project: Mesos > Issue Type: Bug > Components: allocation >Reporter: Klaus Ma > > Add benchmark of sorter in allocation for Allocation Performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-4853) Considering using libcurl multi interface to implement 'curl' in Mesos.
[ https://issues.apache.org/jira/browse/MESOS-4853?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Klaus Ma updated MESOS-4853: Assignee: (was: Klaus Ma) > Considering using libcurl multi interface to implement 'curl' in Mesos. > --- > > Key: MESOS-4853 > URL: https://issues.apache.org/jira/browse/MESOS-4853 > Project: Mesos > Issue Type: Improvement >Reporter: Jie Yu > > Reference: > https://curl.haxx.se/libcurl/c/libcurl-multi.html > Currently, some URI fetchers rely on 'curl' command directly (using > subprocess). This is not ideal. The libcurl easy interface is blocking, so it > does not compose well with our async environment. However, the multi > interface seems to be suitable for our async environment. The tricky part is > that we need to hook the fd selecting logic with our underlying > libev/libevent runtime, but this should be doable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-4923) Treat revocable resources as a separate pool when considering fairness
[
https://issues.apache.org/jira/browse/MESOS-4923?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Klaus Ma updated MESOS-4923:
Assignee: (was: Klaus Ma)
> Treat revocable resources as a separate pool when considering fairness
> --
>
> Key: MESOS-4923
> URL: https://issues.apache.org/jira/browse/MESOS-4923
> Project: Mesos
> Issue Type: Bug
> Components: allocation
>Reporter: Guangya Liu
>
> The current logic of roleSorter is that when it do role sorter, the resources
> in it will include both regular resources and revocable resources, and this
> may not accurate for some cases, take the following case as an instance:
> 1) framework1 and framework2.
> 2) framework1 got 1 reserved cpu and 9 revocable cpu. cpu(r1):1;cpu(*){REV}:9
> 3) framework2 got 9 reserved cpus: cpu(r1):9
> When allocator allocate resources in next cycle, framework2 will be handled
> first as it has less SCALAR resources than framework1, but this may not be
> right for some cases as framework1 is using only 1 reserved resources and
> other resources are revocable which can be easily got evicted.
> A proposal here is treat revocable resources as a separate pool when
> considering fairness, this can be achieved by introducing a new sorter for
> revocable resources so as to distinguish the sorter for regular resources and
> revocable resources. To the built in allocator, the logic would be as this:
> 1) Quota Role Sorter
> 2) non-revocable Role Sorter
> 3) Revocable Role Sorter
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-3892) Add a helper function to the Agent to retrieve the list of executors that are using optimistically offered, revocable resources.
[
https://issues.apache.org/jira/browse/MESOS-3892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Klaus Ma updated MESOS-3892:
Assignee: (was: Klaus Ma)
> Add a helper function to the Agent to retrieve the list of executors that are
> using optimistically offered, revocable resources.
>
>
> Key: MESOS-3892
> URL: https://issues.apache.org/jira/browse/MESOS-3892
> Project: Mesos
> Issue Type: Bug
>Reporter: Artem Harutyunyan
> Labels: mesosphere
>
> In the agent, add a helper function to get the list of the exeuctor using
> ALLOCATION_SLACK.
> It's short term solution which is different the design document, because
> master did not have executor for command line executor. Send evicatble
> executors from master to slave will addess in post-MVP after MESOS-1718.
> {noformat}
> class Slave {
> ...
> // If the executor used revocable resources, add it into
> `evictableExecutors`
> // list.
> void addEvictableExecutor(Executor* executor);
> // If the executor used revocable resources, remove it from
> // `evictableExecutors` list.
> void removeEvictableExecutor(Executor* executor);
> // Get evictable executor ID list by `request resources`. The return value
> is `Result>`:
> // - if `isError()`, there's not enough resources to launch tasks
> // - if `isNone()`, no evictable exectuors need to be terminated
> // - if !`isNone()`, the list of executors that need to be evicted for
> resources
> Result> getEvictableExecutors(const Resources&
> request);
> ...
> // The map of evictable executor list. If there's not enough resources,
> // the evictable executor will be terminated by slave to release resources.
> hashmap evictableExecutors;
> ...
> }
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-4755) Update roleSorter when slave active/deactive
[
https://issues.apache.org/jira/browse/MESOS-4755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Klaus Ma updated MESOS-4755:
Assignee: (was: Klaus Ma)
> Update roleSorter when slave active/deactive
>
>
> Key: MESOS-4755
> URL: https://issues.apache.org/jira/browse/MESOS-4755
> Project: Mesos
> Issue Type: Bug
> Components: allocation
>Reporter: Klaus Ma
>
> Currently, the total resources of {{roleSorter}} are not updated when Agent
> active/deactive.
> It need to remove slave.total from roleSorter when deactive, and add it back
> when agent active again.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-3059) Allow http endpoint to dynamically change the slave attributes
[ https://issues.apache.org/jira/browse/MESOS-3059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Klaus Ma updated MESOS-3059: Assignee: (was: Klaus Ma) > Allow http endpoint to dynamically change the slave attributes > -- > > Key: MESOS-3059 > URL: https://issues.apache.org/jira/browse/MESOS-3059 > Project: Mesos > Issue Type: Wish >Reporter: Nitin > Labels: mesosphere > > This is well understood that - changing the attributes dynamically is not > safe without a restart because slave itself may not know which old framework > tasks are running on it that were dependent on previous attributes. > However, total restart makes lot of other history to delete. We need to > ensure a dynamic attribute changes with a soft restart. > It will be good to expose a rest endpoint either at slave or mesos-master > which directly changes the state in zookeeper. > USE-CASE > We use slave attributes/roles to direct the framework scheduling to use > specific slave as per it's requirements. Mesos scheduler only creates the > offer on the basis of some resources. > In our use case, we have some categorization of our spark frameworks or jobs > with framework(like marathon) based on multiple factors. We want job or > frameworks belonging to one category be running into their specific cluster > of resources. We want to dynamically manage the slaves into these logical > sub-clusters. > Since number of jobs that will be submitted or when it will be submitted is > very dynamic, it make sense to be able to dynamically assign roles or > attributes to slaves. It is not possible to gauge the requirements at time of > cluster provisioning. Static role or attribute assignment leads to > sub-optimal use of the cluster. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (MESOS-5603) Improve test cases in ValueTest
[
https://issues.apache.org/jira/browse/MESOS-5603?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Klaus Ma reassigned MESOS-5603:
---
Assignee: Klaus Ma
> Improve test cases in ValueTest
> ---
>
> Key: MESOS-5603
> URL: https://issues.apache.org/jira/browse/MESOS-5603
> Project: Mesos
> Issue Type: Improvement
>Reporter: Klaus Ma
>Assignee: Klaus Ma
>
> In {{ValueTest.*}}, it did not include enough cases for positive & negative
> cases. It's better to add more cases to show which cases are allowed and
> which cases are disallowed.
> And the related document should be also updated.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-5762) Add event stream in Mesos
[ https://issues.apache.org/jira/browse/MESOS-5762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360459#comment-15360459 ] Klaus Ma commented on MESOS-5762: - That's great, let me try it :). > Add event stream in Mesos > - > > Key: MESOS-5762 > URL: https://issues.apache.org/jira/browse/MESOS-5762 > Project: Mesos > Issue Type: Bug > Components: master >Reporter: Klaus Ma > Labels: tech-debt > > Currently, Mesos-DNS will retrieve host info (name <-> IP mapping) > periodically. In DCOS, it case data by nginx; but there's performance concern > when it's out of DCOS. It's better for Mesos to provide event stream to avoid > caching. > Anyway, it's hard to make event stream reliable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-5762) Add event stream in Mesos
[ https://issues.apache.org/jira/browse/MESOS-5762?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15360426#comment-15360426 ] Klaus Ma commented on MESOS-5762: - [~vinodkone], I'd like to get register a URL in Mesos to get tasks's status, not only the framework. Because I'd like make DNS server got tasks's status ASAP. > Add event stream in Mesos > - > > Key: MESOS-5762 > URL: https://issues.apache.org/jira/browse/MESOS-5762 > Project: Mesos > Issue Type: Bug > Components: master >Reporter: Klaus Ma > Labels: tech-debt > > Currently, Mesos-DNS will retrieve host info (name <-> IP mapping) > periodically. In DCOS, it case data by nginx; but there's performance concern > when it's out of DCOS. It's better for Mesos to provide event stream to avoid > caching. > Anyway, it's hard to make event stream reliable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-5762) Add event stream in Mesos
Klaus Ma created MESOS-5762: --- Summary: Add event stream in Mesos Key: MESOS-5762 URL: https://issues.apache.org/jira/browse/MESOS-5762 Project: Mesos Issue Type: Bug Components: master Reporter: Klaus Ma Currently, Mesos-DNS will retrieve host info (name <-> IP mapping) periodically. In DCOS, it case data by nginx; but there's performance concern when it's out of DCOS. It's better for Mesos to provide event stream to avoid caching. Anyway, it's hard to make event stream reliable. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-5700) Benchmark for Resource class (protobuf vs. C++)
[
https://issues.apache.org/jira/browse/MESOS-5700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15356615#comment-15356615
]
Klaus Ma commented on MESOS-5700:
-
Here's benchmark for {{operator+}} & {{operator+=}}
{code}
[ RUN ]
ResourcesOperatorCount/Resources_BENCHMARK_Test.Operator_AddAndAssign/5
Added 100 resources (cpus:1) in 437384us
[ OK ]
ResourcesOperatorCount/Resources_BENCHMARK_Test.Operator_AddAndAssign/5 (438 ms)
[ RUN ]
ResourcesOperatorCount/Resources_BENCHMARK_Test.Operator_AddAndAssign/11
Added 100 resources (cpus:1;mem:2) in 826587us
[ OK ]
ResourcesOperatorCount/Resources_BENCHMARK_Test.Operator_AddAndAssign/11 (826
ms)
[ RUN ]
ResourcesOperatorCount/Resources_BENCHMARK_Test.Operator_AddAndAssign/17
Added 100 resources (cpus:1;ports:[1-100]) in 1.944934secs
[ OK ]
ResourcesOperatorCount/Resources_BENCHMARK_Test.Operator_AddAndAssign/17 (1945
ms)
[ RUN ] ResourcesOperatorCount/Resources_BENCHMARK_Test.Operator_Add/5
Added 100 resources (cpus:1) in 1.368948secs
[ OK ] ResourcesOperatorCount/Resources_BENCHMARK_Test.Operator_Add/5
(1369 ms)
[ RUN ] ResourcesOperatorCount/Resources_BENCHMARK_Test.Operator_Add/11
Added 100 resources (cpus:1;mem:2) in 2.734078secs
[ OK ] ResourcesOperatorCount/Resources_BENCHMARK_Test.Operator_Add/11
(2734 ms)
[ RUN ] ResourcesOperatorCount/Resources_BENCHMARK_Test.Operator_Add/17
Added 100 resources (cpus:1;ports:[1-100]) in 4.410165secs
[ OK ] ResourcesOperatorCount/Resources_BENCHMARK_Test.Operator_Add/17
(4410 ms)
{code}
> Benchmark for Resource class (protobuf vs. C++)
> ---
>
> Key: MESOS-5700
> URL: https://issues.apache.org/jira/browse/MESOS-5700
> Project: Mesos
> Issue Type: Bug
>Reporter: Klaus Ma
>Assignee: Klaus Ma
>
> Add benchmark of Resource class for Allocation Performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-5700) Benchmark for Resource class (protobuf vs. C++)
[
https://issues.apache.org/jira/browse/MESOS-5700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15356374#comment-15356374
]
Klaus Ma commented on MESOS-5700:
-
Will add more benchmark for {{operator+(...)}}, according to the output of
{{calgrind}}, CopyFrom is the heavy operators.
> Benchmark for Resource class (protobuf vs. C++)
> ---
>
> Key: MESOS-5700
> URL: https://issues.apache.org/jira/browse/MESOS-5700
> Project: Mesos
> Issue Type: Bug
>Reporter: Klaus Ma
>Assignee: Klaus Ma
>
> Add benchmark of Resource class for Allocation Performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-4770) Investigate performance improvements for 'Resources' class.
[
https://issues.apache.org/jira/browse/MESOS-4770?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15356373#comment-15356373
]
Klaus Ma commented on MESOS-4770:
-
Add some benchmark for Resources operators; i'll hep to evaluate the
improvement.
> Investigate performance improvements for 'Resources' class.
> ---
>
> Key: MESOS-4770
> URL: https://issues.apache.org/jira/browse/MESOS-4770
> Project: Mesos
> Issue Type: Improvement
>Reporter: Benjamin Mahler
>Assignee: Michael Park
>Priority: Critical
>
> Currently we have some performance issues when we have heavy usage of the
> {{Resources}} class. Currently, we tend to work around these issues (e.g.
> reduce the amount of Resources arithmetic operations in the caller code).
> The implementation of {{Resources}} currently consists of wrapping underlying
> {{Resource}} protobuf objects and manipulating them. This is fairly expensive
> compared to doing things more directly with C++ objects.
> This ticket is to explore the performance improvements of using C++ objects
> more directly instead of working off of {{Resource}} objects.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-5700) Benchmark for Resource class (protobuf vs. C++)
[
https://issues.apache.org/jira/browse/MESOS-5700?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15355336#comment-15355336
]
Klaus Ma commented on MESOS-5700:
-
Keep add the same resource for 100 times:
{code}
[ RUN ] ResourcesOperatorCount/Resources_BENCHMARK_Test.Operator_Add/17
Added 100 resources (cpus:1;ports:[1-100]) in 1.895914secs
[ OK ] ResourcesOperatorCount/Resources_BENCHMARK_Test.Operator_Add/17
(1896 ms)
{code}
> Benchmark for Resource class (protobuf vs. C++)
> ---
>
> Key: MESOS-5700
> URL: https://issues.apache.org/jira/browse/MESOS-5700
> Project: Mesos
> Issue Type: Bug
>Reporter: Klaus Ma
>Assignee: Klaus Ma
>
> Add benchmark of Resource class for Allocation Performance.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MESOS-5739) Enhance Value parsing
Klaus Ma created MESOS-5739:
---
Summary: Enhance Value parsing
Key: MESOS-5739
URL: https://issues.apache.org/jira/browse/MESOS-5739
Project: Mesos
Issue Type: Bug
Components: master
Reporter: Klaus Ma
Assignee: Klaus Ma
Enhanced Value parsing:
{code}
1. Did not support [1-2, [3-4]] as Ranges; it should be [1-2, 3-4].
2. Did not support {a{b, c}d} as Set; it should be {ab, cd}
3. Add check for Text against [a-zA-Z0-9_/.-]
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-5739) Enhance Value parsing
[
https://issues.apache.org/jira/browse/MESOS-5739?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15353941#comment-15353941
]
Klaus Ma commented on MESOS-5739:
-
https://reviews.apache.org/r/49223/
> Enhance Value parsing
> -
>
> Key: MESOS-5739
> URL: https://issues.apache.org/jira/browse/MESOS-5739
> Project: Mesos
> Issue Type: Bug
> Components: master
>Reporter: Klaus Ma
>Assignee: Klaus Ma
>
> Enhanced Value parsing:
> {code}
> 1. Did not support [1-2, [3-4]] as Ranges; it should be [1-2, 3-4].
> 2. Did not support {a{b, c}d} as Set; it should be {ab, cd}
> 3. Add check for Text against [a-zA-Z0-9_/.-]
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-5715) Enhance startsWith/endsWith's performance
[
https://issues.apache.org/jira/browse/MESOS-5715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15350462#comment-15350462
]
Klaus Ma commented on MESOS-5715:
-
https://reviews.apache.org/r/49246/
> Enhance startsWith/endsWith's performance
> -
>
> Key: MESOS-5715
> URL: https://issues.apache.org/jira/browse/MESOS-5715
> Project: Mesos
> Issue Type: Bug
> Components: stout
>Reporter: Klaus Ma
>Assignee: Klaus Ma
>
> Enhanced startsWith/endsWith's performance by avoiding {{strings::find}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (MESOS-5692) Add helper function "begin_with/end_with" to strings
[ https://issues.apache.org/jira/browse/MESOS-5692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Klaus Ma reassigned MESOS-5692: --- Assignee: Klaus Ma > Add helper function "begin_with/end_with" to strings > > > Key: MESOS-5692 > URL: https://issues.apache.org/jira/browse/MESOS-5692 > Project: Mesos > Issue Type: Bug > Components: stout >Reporter: Klaus Ma >Assignee: Klaus Ma >Priority: Minor > Labels: newbie > > Add helper function to check whether a string is start/end with special char. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-5715) Enhance startsWith/endsWith's performance
Klaus Ma created MESOS-5715:
---
Summary: Enhance startsWith/endsWith's performance
Key: MESOS-5715
URL: https://issues.apache.org/jira/browse/MESOS-5715
Project: Mesos
Issue Type: Bug
Components: stout
Reporter: Klaus Ma
Assignee: Klaus Ma
Enhanced startsWith/endsWith's performance by avoiding {{strings::find}}.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-5700) Benchmark for Resource class (protobuf vs. C++)
[ https://issues.apache.org/jira/browse/MESOS-5700?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Klaus Ma updated MESOS-5700: Summary: Benchmark for Resource class (protobuf vs. C++) (was: Benchmark for Resource class) > Benchmark for Resource class (protobuf vs. C++) > --- > > Key: MESOS-5700 > URL: https://issues.apache.org/jira/browse/MESOS-5700 > Project: Mesos > Issue Type: Bug >Reporter: Klaus Ma >Assignee: Klaus Ma > > Add benchmark of Resource class for Allocation Performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-5692) Add helper function "begin_with/end_with" to strings
[ https://issues.apache.org/jira/browse/MESOS-5692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15350062#comment-15350062 ] Klaus Ma commented on MESOS-5692: - https://reviews.apache.org/r/49140/ > Add helper function "begin_with/end_with" to strings > > > Key: MESOS-5692 > URL: https://issues.apache.org/jira/browse/MESOS-5692 > Project: Mesos > Issue Type: Bug > Components: stout >Reporter: Klaus Ma >Priority: Minor > Labels: newbie > > Add helper function to check whether a string is start/end with special char. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-5692) Add helper function "begin_with/end_with" to strings
[
https://issues.apache.org/jira/browse/MESOS-5692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15347712#comment-15347712
]
Klaus Ma commented on MESOS-5692:
-
No special case in Mesos for now, one potential case is:
{code}
foreach (char c, str) {
if (strings::startsWith(temp, c)) {
...
}
}
{code}
> Add helper function "begin_with/end_with" to strings
>
>
> Key: MESOS-5692
> URL: https://issues.apache.org/jira/browse/MESOS-5692
> Project: Mesos
> Issue Type: Bug
> Components: stout
>Reporter: Klaus Ma
>Priority: Minor
> Labels: newbiee
>
> Add helper function to check whether a string is start/end with special char.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-5692) Add helper function "begin_with/end_with" to strings
[ https://issues.apache.org/jira/browse/MESOS-5692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Klaus Ma updated MESOS-5692: Priority: Minor (was: Major) > Add helper function "begin_with/end_with" to strings > > > Key: MESOS-5692 > URL: https://issues.apache.org/jira/browse/MESOS-5692 > Project: Mesos > Issue Type: Bug > Components: stout >Reporter: Klaus Ma >Priority: Minor > Labels: newbiee > > Add helper function to check whether a string is start/end with special char. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MESOS-5692) Add helper function "begin_with/end_with" to strings
[ https://issues.apache.org/jira/browse/MESOS-5692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Klaus Ma updated MESOS-5692: Labels: newbiee (was: ) > Add helper function "begin_with/end_with" to strings > > > Key: MESOS-5692 > URL: https://issues.apache.org/jira/browse/MESOS-5692 > Project: Mesos > Issue Type: Bug > Components: stout >Reporter: Klaus Ma >Priority: Minor > Labels: newbiee > > Add helper function to check whether a string is start/end with special char. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-5701) Add benchmark for sorter performance
Klaus Ma created MESOS-5701: --- Summary: Add benchmark for sorter performance Key: MESOS-5701 URL: https://issues.apache.org/jira/browse/MESOS-5701 Project: Mesos Issue Type: Bug Components: allocation Reporter: Klaus Ma Assignee: Klaus Ma Add benchmark of sorter in allocation for Allocation Performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-5700) Benchmark for Resource class
Klaus Ma created MESOS-5700: --- Summary: Benchmark for Resource class Key: MESOS-5700 URL: https://issues.apache.org/jira/browse/MESOS-5700 Project: Mesos Issue Type: Bug Reporter: Klaus Ma Assignee: Klaus Ma Add benchmark of Resource class for Allocation Performance. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MESOS-5692) Add helper function "begin_with/end_with" to strings
Klaus Ma created MESOS-5692: --- Summary: Add helper function "begin_with/end_with" to strings Key: MESOS-5692 URL: https://issues.apache.org/jira/browse/MESOS-5692 Project: Mesos Issue Type: Bug Components: stout Reporter: Klaus Ma Add helper function to check whether a string is start/end with special char. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-5565) Add logging when Offer::Operation::Launch message has no tasks.
[
https://issues.apache.org/jira/browse/MESOS-5565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15345514#comment-15345514
]
Klaus Ma commented on MESOS-5565:
-
sure, please help on this :). Thanks very much :).
> Add logging when Offer::Operation::Launch message has no tasks.
> ---
>
> Key: MESOS-5565
> URL: https://issues.apache.org/jira/browse/MESOS-5565
> Project: Mesos
> Issue Type: Improvement
>Reporter: Anand Mazumdar
>Priority: Minor
> Labels: newbie
>
> Currently, when a {{Offer::Accept::Launch}} message has no tasks specified,
> Mesos would treat such requests as implicitly declining all offers. This can
> be very counter-intuitive for framework developers since we do not have any
> logging on the Master around this behavior. It would be good to add some
> logging on the master to apprise the framework developers that all the offers
> have been implicitly declined.
> {code}
> if (operation.type() == Offer::Operation::LAUNCH) {
> if (operation.launch().task_infos().size() > 0) {
> ++metrics->messages_launch_tasks;
> } else {
> ++metrics->messages_decline_offers;
> }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-5565) Add logging when Offer::Operation::Launch message has no tasks.
[
https://issues.apache.org/jira/browse/MESOS-5565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Klaus Ma updated MESOS-5565:
Assignee: (was: Klaus Ma)
> Add logging when Offer::Operation::Launch message has no tasks.
> ---
>
> Key: MESOS-5565
> URL: https://issues.apache.org/jira/browse/MESOS-5565
> Project: Mesos
> Issue Type: Improvement
>Reporter: Anand Mazumdar
>Priority: Minor
> Labels: newbie
>
> Currently, when a {{Offer::Accept::Launch}} message has no tasks specified,
> Mesos would treat such requests as implicitly declining all offers. This can
> be very counter-intuitive for framework developers since we do not have any
> logging on the Master around this behavior. It would be good to add some
> logging on the master to apprise the framework developers that all the offers
> have been implicitly declined.
> {code}
> if (operation.type() == Offer::Operation::LAUNCH) {
> if (operation.launch().task_infos().size() > 0) {
> ++metrics->messages_launch_tasks;
> } else {
> ++metrics->messages_decline_offers;
> }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-5565) Add logging when Offer::Operation::Launch message has no tasks.
[
https://issues.apache.org/jira/browse/MESOS-5565?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Klaus Ma updated MESOS-5565:
Fix Version/s: (was: 1.0.0)
> Add logging when Offer::Operation::Launch message has no tasks.
> ---
>
> Key: MESOS-5565
> URL: https://issues.apache.org/jira/browse/MESOS-5565
> Project: Mesos
> Issue Type: Improvement
>Reporter: Anand Mazumdar
>Assignee: Klaus Ma
>Priority: Minor
> Labels: newbie
>
> Currently, when a {{Offer::Accept::Launch}} message has no tasks specified,
> Mesos would treat such requests as implicitly declining all offers. This can
> be very counter-intuitive for framework developers since we do not have any
> logging on the Master around this behavior. It would be good to add some
> logging on the master to apprise the framework developers that all the offers
> have been implicitly declined.
> {code}
> if (operation.type() == Offer::Operation::LAUNCH) {
> if (operation.launch().task_infos().size() > 0) {
> ++metrics->messages_launch_tasks;
> } else {
> ++metrics->messages_decline_offers;
> }
> }
> }
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (MESOS-4967) Oversubscription for reservation
[ https://issues.apache.org/jira/browse/MESOS-4967?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Klaus Ma reassigned MESOS-4967: --- Assignee: Klaus Ma > Oversubscription for reservation > > > Key: MESOS-4967 > URL: https://issues.apache.org/jira/browse/MESOS-4967 > Project: Mesos > Issue Type: Epic > Components: allocation, framework, master >Reporter: Klaus Ma >Assignee: Klaus Ma > Labels: IBM, mesosphere > > Reserved resources allow frameworks and cluster operators to ensure > sufficient resources are available when needed. Reservations are usually > made to guarantee there are enough resources under peak loads. Often times, > reserved resources are not actually allocated; in other words, the frameworks > do not use those resources and they sit reserved, but idle. > This underutilization is either an opportunity cost or a direct cost, > particularly to the cluster operator. Reserved but unallocated resources > held by a Lender Framework could be optimistically offered to other > frameworks, which we refer to as Tenant Frameworks. When the resources are > requested back by the Lender Framework, some of the Tenant Framework’s tasks > are evicted and the original resource offer guarantee is preserved. > The first step is to identify when resources are reserved, but not allocated. > We then offer these reserved resources to other frameworks, but mark these > offered resources as revocable resources. This allows Tenant Frameworks to > use these resources temporarily in a 'best-effort' fashion, knowing that they > could be revoked or reclaimed at any time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MESOS-5619) Add task_num to mesos-execute
[
https://issues.apache.org/jira/browse/MESOS-5619?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15335358#comment-15335358
]
Klaus Ma commented on MESOS-5619:
-
It's the {{mesos-execute}} (the command scheduler). Update the info accordingly.
> Add task_num to mesos-execute
> -
>
> Key: MESOS-5619
> URL: https://issues.apache.org/jira/browse/MESOS-5619
> Project: Mesos
> Issue Type: Bug
> Components: cli
>Reporter: Klaus Ma
>Assignee: Klaus Ma
>
> According to current code, {{mesos-execute}} will only launch one task. It's
> better to add a parameter to special how many task to launch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MESOS-5619) Add task_num to mesos-execute
[
https://issues.apache.org/jira/browse/MESOS-5619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Klaus Ma updated MESOS-5619:
Summary: Add task_num to mesos-execute (was: Add task_num to
mesos-executor)
> Add task_num to mesos-execute
> -
>
> Key: MESOS-5619
> URL: https://issues.apache.org/jira/browse/MESOS-5619
> Project: Mesos
> Issue Type: Bug
> Components: cli
>Reporter: Klaus Ma
>Assignee: Klaus Ma
>
> According to current code, {{mesos-executor}} will only launch one task. It's
> better to add a parameter to special how many task to launch.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
