[jira] [Commented] (MESOS-5037) foreachkey behaviour is not expected in multimap
[
https://issues.apache.org/jira/browse/MESOS-5037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883939#comment-16883939
]
haosdent commented on MESOS-5037:
-
[~bmahler] Sry for the delay. The context is we have an agent which has 1k
tasks, and when the agent reregister, it would trigger 1000,000 rounds on this
loop. I have a flamegraph last week, but it is cleared by incident.
> foreachkey behaviour is not expected in multimap
>
>
> Key: MESOS-5037
> URL: https://issues.apache.org/jira/browse/MESOS-5037
> Project: Mesos
> Issue Type: Bug
> Components: stout
>Reporter: haosdent
>Priority: Major
> Labels: foundations, stout
>
> Currently the {{foreachkey}} implementation is
> {code}
> #define foreachkey(VAR, COL)\
> foreachpair (VAR, __foreach__::ignore, COL)
> {code}
> This works in most structures. But in multimap, one key may map to multi
> values. This means there are multi pairs which have same key. So when call
> {{foreachkey}}, the {{key}} would duplicated when iteration. My idea to solve
> this is we prefer call {{foreach}} on {{(COL).keys()}} if {{keys()}} method
> exists in {{COL}}.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (MESOS-5037) foreachkey behaviour is not expected in multimap
[
https://issues.apache.org/jira/browse/MESOS-5037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884011#comment-16884011
]
Benjamin Mahler commented on MESOS-5037:
[~haosd...@gmail.com] can you post a link to the code in question?
> foreachkey behaviour is not expected in multimap
>
>
> Key: MESOS-5037
> URL: https://issues.apache.org/jira/browse/MESOS-5037
> Project: Mesos
> Issue Type: Bug
> Components: stout
>Reporter: haosdent
>Priority: Major
> Labels: foundations, stout
>
> Currently the {{foreachkey}} implementation is
> {code}
> #define foreachkey(VAR, COL)\
> foreachpair (VAR, __foreach__::ignore, COL)
> {code}
> This works in most structures. But in multimap, one key may map to multi
> values. This means there are multi pairs which have same key. So when call
> {{foreachkey}}, the {{key}} would duplicated when iteration. My idea to solve
> this is we prefer call {{foreach}} on {{(COL).keys()}} if {{keys()}} method
> exists in {{COL}}.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (MESOS-9889) Master CPU high due to unexpected foreachkey behaviour in Master::__reregisterSlave
haosdent created MESOS-9889:
---
Summary: Master CPU high due to unexpected foreachkey behaviour in
Master::__reregisterSlave
Key: MESOS-9889
URL: https://issues.apache.org/jira/browse/MESOS-9889
Project: Mesos
Issue Type: Bug
Reporter: haosdent
At
https://github.com/apache/mesos/blob/9932550e9632e7fbb9a45b217793c7f508f57001/src/master/master.cpp#L7707-L7708
{code}
void Master::__reregisterSlave(
...
foreachkey (FrameworkID frameworkId,
slaves.unreachableTasks.at(slaveInfo.id())) {
...
foreach (TaskID taskId,
slaves.unreachableTasks.at(slaveInfo.id()).get(frameworkId)) {
{code}
Our case is when network flapping, 3~4 agents reregister, then master would CPU
full and could not process any requests during that period.
After change
{code}
-foreachkey (FrameworkID frameworkId,
- slaves.unreachableTasks.at(slaveInfo.id())) {
+foreach (FrameworkID frameworkId,
+ slaves.unreachableTasks.at(slaveInfo.id()).keys()) {
{code}
The problem gone.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (MESOS-5037) foreachkey behaviour is not expected in multimap
[
https://issues.apache.org/jira/browse/MESOS-5037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884055#comment-16884055
]
haosdent commented on MESOS-5037:
-
[~bmahler] Create at https://issues.apache.org/jira/browse/MESOS-9889
> foreachkey behaviour is not expected in multimap
>
>
> Key: MESOS-5037
> URL: https://issues.apache.org/jira/browse/MESOS-5037
> Project: Mesos
> Issue Type: Bug
> Components: stout
>Reporter: haosdent
>Priority: Major
> Labels: foundations, stout
>
> Currently the {{foreachkey}} implementation is
> {code}
> #define foreachkey(VAR, COL)\
> foreachpair (VAR, __foreach__::ignore, COL)
> {code}
> This works in most structures. But in multimap, one key may map to multi
> values. This means there are multi pairs which have same key. So when call
> {{foreachkey}}, the {{key}} would duplicated when iteration. My idea to solve
> this is we prefer call {{foreach}} on {{(COL).keys()}} if {{keys()}} method
> exists in {{COL}}.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (MESOS-9890) /roles and GET_ROLES does not always expose parent roles.
Benjamin Mahler created MESOS-9890: -- Summary: /roles and GET_ROLES does not always expose parent roles. Key: MESOS-9890 URL: https://issues.apache.org/jira/browse/MESOS-9890 Project: Mesos Issue Type: Bug Reporter: Benjamin Mahler If some descendant roles are present in frameworks, then the parent roles will not be exposed in the /roles and GET_ROLES endpoints. This is because the tracking is currently based on frameworks being subscribed to the role. -- This message was sent by Atlassian JIRA (v7.6.14#76016)
[jira] [Comment Edited] (MESOS-5037) foreachkey behaviour is not expected in multimap
[
https://issues.apache.org/jira/browse/MESOS-5037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884044#comment-16884044
]
Benjamin Mahler edited comment on MESOS-5037 at 7/12/19 5:59 PM:
-
[~bmahler] Sure, it is
https://github.com/apache/mesos/blob/9932550e9632e7fbb9a45b217793c7f508f57001/src/master/master.cpp#L7707-L7708
{code}
void Master::__reregisterSlave(
...
foreachkey (FrameworkID frameworkId,
slaves.unreachableTasks.at(slaveInfo.id())) {
...
foreach (TaskID taskId,
slaves.unreachableTasks.at(slaveInfo.id()).get(frameworkId)) {
{code}
Our case is when network flapping, 3~4 agents reregister, then master would CPU
full and could not process any requests during that period.
was (Author: haosd...@gmail.com):
[~bmahler] Sure, it is
https://github.com/apache/mesos/blob/master/src/master/master.cpp#L7707-L7708
{code}
void Master::__reregisterSlave(
...
foreachkey (FrameworkID frameworkId,
slaves.unreachableTasks.at(slaveInfo.id())) {
...
foreach (TaskID taskId,
slaves.unreachableTasks.at(slaveInfo.id()).get(frameworkId)) {
{code}
Our case is when network flapping, 3~4 agents reregister, then master would CPU
full and could not process any requests during that period.
> foreachkey behaviour is not expected in multimap
>
>
> Key: MESOS-5037
> URL: https://issues.apache.org/jira/browse/MESOS-5037
> Project: Mesos
> Issue Type: Bug
> Components: stout
>Reporter: haosdent
>Priority: Major
> Labels: foundations, stout
>
> Currently the {{foreachkey}} implementation is
> {code}
> #define foreachkey(VAR, COL)\
> foreachpair (VAR, __foreach__::ignore, COL)
> {code}
> This works in most structures. But in multimap, one key may map to multi
> values. This means there are multi pairs which have same key. So when call
> {{foreachkey}}, the {{key}} would duplicated when iteration. My idea to solve
> this is we prefer call {{foreach}} on {{(COL).keys()}} if {{keys()}} method
> exists in {{COL}}.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (MESOS-5037) foreachkey behaviour is not expected in multimap
[
https://issues.apache.org/jira/browse/MESOS-5037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884044#comment-16884044
]
haosdent commented on MESOS-5037:
-
[~bmahler] Sure, it is
https://github.com/apache/mesos/blob/master/src/master/master.cpp#L7707-L7708
{code}
void Master::__reregisterSlave(
...
foreachkey (FrameworkID frameworkId,
slaves.unreachableTasks.at(slaveInfo.id())) {
...
foreach (TaskID taskId,
slaves.unreachableTasks.at(slaveInfo.id()).get(frameworkId)) {
{code}
Our case is when network flapping, 3~4 agents reregister, then master would CPU
full and could not process any requests during that period.
> foreachkey behaviour is not expected in multimap
>
>
> Key: MESOS-5037
> URL: https://issues.apache.org/jira/browse/MESOS-5037
> Project: Mesos
> Issue Type: Bug
> Components: stout
>Reporter: haosdent
>Priority: Major
> Labels: foundations, stout
>
> Currently the {{foreachkey}} implementation is
> {code}
> #define foreachkey(VAR, COL)\
> foreachpair (VAR, __foreach__::ignore, COL)
> {code}
> This works in most structures. But in multimap, one key may map to multi
> values. This means there are multi pairs which have same key. So when call
> {{foreachkey}}, the {{key}} would duplicated when iteration. My idea to solve
> this is we prefer call {{foreach}} on {{(COL).keys()}} if {{keys()}} method
> exists in {{COL}}.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (MESOS-5037) foreachkey behaviour is not expected in multimap
[
https://issues.apache.org/jira/browse/MESOS-5037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884045#comment-16884045
]
haosdent commented on MESOS-5037:
-
After change
{code}
-foreachkey (FrameworkID frameworkId,
- slaves.unreachableTasks.at(slaveInfo.id())) {
+foreach (FrameworkID frameworkId,
+ slaves.unreachableTasks.at(slaveInfo.id()).keys()) {
{code}
The problem gone.
> foreachkey behaviour is not expected in multimap
>
>
> Key: MESOS-5037
> URL: https://issues.apache.org/jira/browse/MESOS-5037
> Project: Mesos
> Issue Type: Bug
> Components: stout
>Reporter: haosdent
>Priority: Major
> Labels: foundations, stout
>
> Currently the {{foreachkey}} implementation is
> {code}
> #define foreachkey(VAR, COL)\
> foreachpair (VAR, __foreach__::ignore, COL)
> {code}
> This works in most structures. But in multimap, one key may map to multi
> values. This means there are multi pairs which have same key. So when call
> {{foreachkey}}, the {{key}} would duplicated when iteration. My idea to solve
> this is we prefer call {{foreach}} on {{(COL).keys()}} if {{keys()}} method
> exists in {{COL}}.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (MESOS-5037) foreachkey behaviour is not expected in multimap
[
https://issues.apache.org/jira/browse/MESOS-5037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884051#comment-16884051
]
haosdent commented on MESOS-5037:
-
[~bmahler] No problem.
> foreachkey behaviour is not expected in multimap
>
>
> Key: MESOS-5037
> URL: https://issues.apache.org/jira/browse/MESOS-5037
> Project: Mesos
> Issue Type: Bug
> Components: stout
>Reporter: haosdent
>Priority: Major
> Labels: foundations, stout
>
> Currently the {{foreachkey}} implementation is
> {code}
> #define foreachkey(VAR, COL)\
> foreachpair (VAR, __foreach__::ignore, COL)
> {code}
> This works in most structures. But in multimap, one key may map to multi
> values. This means there are multi pairs which have same key. So when call
> {{foreachkey}}, the {{key}} would duplicated when iteration. My idea to solve
> this is we prefer call {{foreach}} on {{(COL).keys()}} if {{keys()}} method
> exists in {{COL}}.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (MESOS-6566) The Docker executor should not leak task env variables in the Docker command cmd line.
[
https://issues.apache.org/jira/browse/MESOS-6566?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16883994#comment-16883994
]
Vinod Kone commented on MESOS-6566:
---
See the description in MESOS-6951 for a potential solution using `--env`
argument.
> The Docker executor should not leak task env variables in the Docker command
> cmd line.
> --
>
> Key: MESOS-6566
> URL: https://issues.apache.org/jira/browse/MESOS-6566
> Project: Mesos
> Issue Type: Bug
> Components: docker, security
>Reporter: Gastón Kleiman
>Assignee: Till Toenshoff
>Priority: Major
>
> Task environment variables are sensitive, as they might contain secrets.
> The Docker executor starts tasks by executing a {{docker run}} command, and
> it includes the env variables in the cmd line of the docker command, exposing
> them to all the users in the machine:
> {code}
> $ ./src/mesos-execute --command="sleep 200" --containerizer=docker
> --docker_image=alpine --env='{"foo": "bar"}' --master=10.0.2.15:5050
> --name=test
> $ ps aux | grep bar
> [...] docker -H unix:///var/run/docker.sock run [...] -e foo=bar [...] alpine
> -c sleep 200
> $
> {code}
> The Docker executor could pass Docker the {{--env-file}} flag, pointing it to
> a file with the environment variables.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Commented] (MESOS-5037) foreachkey behaviour is not expected in multimap
[
https://issues.apache.org/jira/browse/MESOS-5037?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16884048#comment-16884048
]
Benjamin Mahler commented on MESOS-5037:
[~haosd...@gmail.com] Can you file a separate ticket for the performance
problem? And we can keep this ticket as a foreachkey issue?
> foreachkey behaviour is not expected in multimap
>
>
> Key: MESOS-5037
> URL: https://issues.apache.org/jira/browse/MESOS-5037
> Project: Mesos
> Issue Type: Bug
> Components: stout
>Reporter: haosdent
>Priority: Major
> Labels: foundations, stout
>
> Currently the {{foreachkey}} implementation is
> {code}
> #define foreachkey(VAR, COL)\
> foreachpair (VAR, __foreach__::ignore, COL)
> {code}
> This works in most structures. But in multimap, one key may map to multi
> values. This means there are multi pairs which have same key. So when call
> {{foreachkey}}, the {{key}} would duplicated when iteration. My idea to solve
> this is we prefer call {{foreach}} on {{(COL).keys()}} if {{keys()}} method
> exists in {{COL}}.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Assigned] (MESOS-9816) Add draining state information to master event stream and state endpoints
[
https://issues.apache.org/jira/browse/MESOS-9816?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Greg Mann reassigned MESOS-9816:
Assignee: Greg Mann (was: Joseph Wu)
> Add draining state information to master event stream and state endpoints
> -
>
> Key: MESOS-9816
> URL: https://issues.apache.org/jira/browse/MESOS-9816
> Project: Mesos
> Issue Type: Task
> Components: master
>Reporter: Joseph Wu
>Assignee: Greg Mann
>Priority: Major
> Labels: foundations, mesosphere
>
> The response for {{GET_STATE}} and {{GET_AGENTS}} should include the new
> fields indicating deactivation or draining states:
> {code}
> message Response {
> . . .
> message GetAgents {
> message Agent {
> . . .
> optional bool deactivated = 12;
> optional DrainInfo drain_info = 13;
> . . .
> }
> }
> . . .
> }
> {code}
> Additionally, the master's event stream should get a new event whenever these
> states change:
> {code}
> message Event {
> . . .
> enum Type {
> . . .
> AGENT_UPDATED = 10;
> }
> message AgentUpdated {
> optional bool deactivated = 1;
> optional DrainInfo drain_info = 2;
> }
> . . .
> optional AgentUpdated agent_updated = 10;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
[jira] [Created] (MESOS-9891) Add AGENT_UPDATED event with drain state.
Greg Mann created MESOS-9891:
Summary: Add AGENT_UPDATED event with drain state.
Key: MESOS-9891
URL: https://issues.apache.org/jira/browse/MESOS-9891
Project: Mesos
Issue Type: Task
Reporter: Greg Mann
The master's event stream for schedulers and operators should get a new event
whenever an agent's drain or deactivation state changes:
{code}
message Event {
. . .
enum Type {
. . .
AGENT_UPDATED = 10;
}
message AgentUpdated {
optional bool deactivated = 1;
optional DrainInfo drain_info = 2;
}
. . .
optional AgentUpdated agent_updated = 10;
}
{code}
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
