[jira] [Commented] (MESOS-4353) Limit the number of processes created by libprocess
[
https://issues.apache.org/jira/browse/MESOS-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15148134#comment-15148134
]
Klaus Ma commented on MESOS-4353:
-
[~magedm], thanks for your feedback; I'm OK with that.
> Limit the number of processes created by libprocess
> ---
>
> Key: MESOS-4353
> URL: https://issues.apache.org/jira/browse/MESOS-4353
> Project: Mesos
> Issue Type: Improvement
> Components: libprocess
>Reporter: Qian Zhang
>Assignee: Qian Zhang
>
> Currently libprocess will create {{max(8, number of CPU cores)}} processes
> during the initialization, see
> https://github.com/apache/mesos/blob/0.26.0/3rdparty/libprocess/src/process.cpp#L2146
> for details. This should be OK for a normal machine which has no much cores
> (e.g., 16, 32), but for a powerful machine which may have a large number of
> cores (e.g., an IBM Power machine may have 192 cores), this will cause too
> much worker threads which are not necessary.
> And since libprocess is widely used in Mesos (master, agent, scheduler,
> executor), it may also cause some performance issue. For example, when user
> creates a Docker container via Mesos in a Mesos agent which is running on a
> powerful machine with 192 cores, the DockerContainerizer in Mesos agent will
> create a dedicated executor for the container, and there will be 192 worker
> threads in that executor. And if user creates 1000 Docker containers in that
> machine, then there will be 1000 executors, i.e., 1000 * 192 worker threads
> which is a large number and may thrash the OS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-4353) Limit the number of processes created by libprocess
[
https://issues.apache.org/jira/browse/MESOS-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15146321#comment-15146321
]
Maged Michael commented on MESOS-4353:
--
[Klaus]: "If the value is invalid, keep current behaviour, right?"
Yes. The number of worker threads will remain unchanged.
[Klaus]: "Are we also updating the document to support
LIBPROCESS_WORKER_THREADS for Master/Agent?"
I don't think there is need for separate update of the documentation for master
and agent. The only section that will need needs to be updated is "Libprocess
Options" (http://mesos.apache.org/documentation/latest/configuration/). If we
add the LIBPROCESS_WORKER_THREADS environment variable, then it will apply to
all types of Libprocess processes, including master and agents.
> Limit the number of processes created by libprocess
> ---
>
> Key: MESOS-4353
> URL: https://issues.apache.org/jira/browse/MESOS-4353
> Project: Mesos
> Issue Type: Improvement
> Components: libprocess
>Reporter: Qian Zhang
>Assignee: Qian Zhang
>
> Currently libprocess will create {{max(8, number of CPU cores)}} processes
> during the initialization, see
> https://github.com/apache/mesos/blob/0.26.0/3rdparty/libprocess/src/process.cpp#L2146
> for details. This should be OK for a normal machine which has no much cores
> (e.g., 16, 32), but for a powerful machine which may have a large number of
> cores (e.g., an IBM Power machine may have 192 cores), this will cause too
> much worker threads which are not necessary.
> And since libprocess is widely used in Mesos (master, agent, scheduler,
> executor), it may also cause some performance issue. For example, when user
> creates a Docker container via Mesos in a Mesos agent which is running on a
> powerful machine with 192 cores, the DockerContainerizer in Mesos agent will
> create a dedicated executor for the container, and there will be 192 worker
> threads in that executor. And if user creates 1000 Docker containers in that
> machine, then there will be 1000 executors, i.e., 1000 * 192 worker threads
> which is a large number and may thrash the OS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-4353) Limit the number of processes created by libprocess
[
https://issues.apache.org/jira/browse/MESOS-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15146313#comment-15146313
]
Klaus Ma commented on MESOS-4353:
-
"All other values are invalid and generate a warning."
[Klaus]: If the value is invalid, keep current behaviour, right?
"The proposed environment variable can be set directly for Mesos master, agents
(slaves), and tests."
"Update documentation of Mesos configuration to reflect the addition of this
libprocess environment variable"
[Klaus]: Are we also updating the document to support
{{LIBPROCESS_WORKER_THREADS}} for Master/Agent?
> Limit the number of processes created by libprocess
> ---
>
> Key: MESOS-4353
> URL: https://issues.apache.org/jira/browse/MESOS-4353
> Project: Mesos
> Issue Type: Improvement
> Components: libprocess
>Reporter: Qian Zhang
>Assignee: Qian Zhang
>
> Currently libprocess will create {{max(8, number of CPU cores)}} processes
> during the initialization, see
> https://github.com/apache/mesos/blob/0.26.0/3rdparty/libprocess/src/process.cpp#L2146
> for details. This should be OK for a normal machine which has no much cores
> (e.g., 16, 32), but for a powerful machine which may have a large number of
> cores (e.g., an IBM Power machine may have 192 cores), this will cause too
> much worker threads which are not necessary.
> And since libprocess is widely used in Mesos (master, agent, scheduler,
> executor), it may also cause some performance issue. For example, when user
> creates a Docker container via Mesos in a Mesos agent which is running on a
> powerful machine with 192 cores, the DockerContainerizer in Mesos agent will
> create a dedicated executor for the container, and there will be 192 worker
> threads in that executor. And if user creates 1000 Docker containers in that
> machine, then there will be 1000 executors, i.e., 1000 * 192 worker threads
> which is a large number and may thrash the OS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-4353) Limit the number of processes created by libprocess
[
https://issues.apache.org/jira/browse/MESOS-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15145367#comment-15145367
]
Maged Michael commented on MESOS-4353:
--
If this is OK, then I propose the following design:
* Introduce a new environment variable to allow the operator to set the number
of libprocess worker threads.
* The environment variable is named LIBPROCESS_WORKER_THREADS
* Valid values of the environment variable are integers in the range 1 to 1024.
* All other values are invalid and generate a warning.
* The proposed environment variable can be set directly for Mesos master,
agents (slaves), and tests.
* For executors, the proposed environment variable can be set indirectly by
including it in the setting of the agent (slave)
--executor_environment_variables option (See documentation of Mesos
configuration http://mesos.apache.org/documentation/latest/configuration/).
* Update documentation of Mesos configuration to reflect the addition of this
libprocess environment variable.
> Limit the number of processes created by libprocess
> ---
>
> Key: MESOS-4353
> URL: https://issues.apache.org/jira/browse/MESOS-4353
> Project: Mesos
> Issue Type: Improvement
> Components: libprocess
>Reporter: Qian Zhang
>Assignee: Qian Zhang
>
> Currently libprocess will create {{max(8, number of CPU cores)}} processes
> during the initialization, see
> https://github.com/apache/mesos/blob/0.26.0/3rdparty/libprocess/src/process.cpp#L2146
> for details. This should be OK for a normal machine which has no much cores
> (e.g., 16, 32), but for a powerful machine which may have a large number of
> cores (e.g., an IBM Power machine may have 192 cores), this will cause too
> much worker threads which are not necessary.
> And since libprocess is widely used in Mesos (master, agent, scheduler,
> executor), it may also cause some performance issue. For example, when user
> creates a Docker container via Mesos in a Mesos agent which is running on a
> powerful machine with 192 cores, the DockerContainerizer in Mesos agent will
> create a dedicated executor for the container, and there will be 192 worker
> threads in that executor. And if user creates 1000 Docker containers in that
> machine, then there will be 1000 executors, i.e., 1000 * 192 worker threads
> which is a large number and may thrash the OS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-4353) Limit the number of processes created by libprocess
[
https://issues.apache.org/jira/browse/MESOS-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15142699#comment-15142699
]
Maged Michael commented on MESOS-4353:
--
Replying to Joris:
> I don't think it makes sense to make this a maximum. Rather, it is just the
> number of libprocess_worker_threads.
My concern is that the number may be set to a very large value.How about we set
a hardwired maximum value to limit the given value if it is too large?
> Limit the number of processes created by libprocess
> ---
>
> Key: MESOS-4353
> URL: https://issues.apache.org/jira/browse/MESOS-4353
> Project: Mesos
> Issue Type: Improvement
> Components: libprocess
>Reporter: Qian Zhang
>Assignee: Qian Zhang
>
> Currently libprocess will create {{max(8, number of CPU cores)}} processes
> during the initialization, see
> https://github.com/apache/mesos/blob/0.26.0/3rdparty/libprocess/src/process.cpp#L2146
> for details. This should be OK for a normal machine which has no much cores
> (e.g., 16, 32), but for a powerful machine which may have a large number of
> cores (e.g., an IBM Power machine may have 192 cores), this will cause too
> much worker threads which are not necessary.
> And since libprocess is widely used in Mesos (master, agent, scheduler,
> executor), it may also cause some performance issue. For example, when user
> creates a Docker container via Mesos in a Mesos agent which is running on a
> powerful machine with 192 cores, the DockerContainerizer in Mesos agent will
> create a dedicated executor for the container, and there will be 192 worker
> threads in that executor. And if user creates 1000 Docker containers in that
> machine, then there will be 1000 executors, i.e., 1000 * 192 worker threads
> which is a large number and may thrash the OS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-4353) Limit the number of processes created by libprocess
[
https://issues.apache.org/jira/browse/MESOS-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15132967#comment-15132967
]
Maged Michael commented on MESOS-4353:
--
Patch https://reviews.apache.org/r/43144/
> Limit the number of processes created by libprocess
> ---
>
> Key: MESOS-4353
> URL: https://issues.apache.org/jira/browse/MESOS-4353
> Project: Mesos
> Issue Type: Improvement
> Components: libprocess
>Reporter: Qian Zhang
>Assignee: Qian Zhang
>
> Currently libprocess will create {{max(8, number of CPU cores)}} processes
> during the initialization, see
> https://github.com/apache/mesos/blob/0.26.0/3rdparty/libprocess/src/process.cpp#L2146
> for details. This should be OK for a normal machine which has no much cores
> (e.g., 16, 32), but for a powerful machine which may have a large number of
> cores (e.g., an IBM Power machine may have 192 cores), this will cause too
> much worker threads which are not necessary.
> And since libprocess is widely used in Mesos (master, agent, scheduler,
> executor), it may also cause some performance issue. For example, when user
> creates a Docker container via Mesos in a Mesos agent which is running on a
> powerful machine with 192 cores, the DockerContainerizer in Mesos agent will
> create a dedicated executor for the container, and there will be 192 worker
> threads in that executor. And if user creates 1000 Docker containers in that
> machine, then there will be 1000 executors, i.e., 1000 * 192 worker threads
> which is a large number and may thrash the OS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-4353) Limit the number of processes created by libprocess
[
https://issues.apache.org/jira/browse/MESOS-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15131116#comment-15131116
]
Maged Michael commented on MESOS-4353:
--
Draft design document
https://docs.google.com/document/d/1D34S0HOZOonu510305THPECioCZ4bH5Zqq-AeCq5gmQ
> Limit the number of processes created by libprocess
> ---
>
> Key: MESOS-4353
> URL: https://issues.apache.org/jira/browse/MESOS-4353
> Project: Mesos
> Issue Type: Improvement
> Components: libprocess
>Reporter: Qian Zhang
>Assignee: Qian Zhang
>
> Currently libprocess will create {{max(8, number of CPU cores)}} processes
> during the initialization, see
> https://github.com/apache/mesos/blob/0.26.0/3rdparty/libprocess/src/process.cpp#L2146
> for details. This should be OK for a normal machine which has no much cores
> (e.g., 16, 32), but for a powerful machine which may have a large number of
> cores (e.g., an IBM Power machine may have 192 cores), this will cause too
> much worker threads which are not necessary.
> And since libprocess is widely used in Mesos (master, agent, scheduler,
> executor), it may also cause some performance issue. For example, when user
> creates a Docker container via Mesos in a Mesos agent which is running on a
> powerful machine with 192 cores, the DockerContainerizer in Mesos agent will
> create a dedicated executor for the container, and there will be 192 worker
> threads in that executor. And if user creates 1000 Docker containers in that
> machine, then there will be 1000 executors, i.e., 1000 * 192 worker threads
> which is a large number and may thrash the OS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-4353) Limit the number of processes created by libprocess
[
https://issues.apache.org/jira/browse/MESOS-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15115256#comment-15115256
]
Maged Michael commented on MESOS-4353:
--
Currently, there is a lower bound of 8 on the number of worker threads. I
suggest that if the user provides an upper bound that is less than 8 (e.g.,
sets an environment variable LIBPROCESS_MAX_WORKER_THREADS=4), then the
provided upper bound overrides the lower bound (in this example, the number of
worker threads would be set to 4). The rationale is that not all types Mesos
processes require 8 worker threads.
> Limit the number of processes created by libprocess
> ---
>
> Key: MESOS-4353
> URL: https://issues.apache.org/jira/browse/MESOS-4353
> Project: Mesos
> Issue Type: Improvement
> Components: libprocess
>Reporter: Qian Zhang
>Assignee: Qian Zhang
>
> Currently libprocess will create {{max(8, number of CPU cores)}} processes
> during the initialization, see
> https://github.com/apache/mesos/blob/0.26.0/3rdparty/libprocess/src/process.cpp#L2146
> for details. This should be OK for a normal machine which has no much cores
> (e.g., 16, 32), but for a powerful machine which may have a large number of
> cores (e.g., an IBM Power machine may have 192 cores), this will cause too
> much worker threads which are not necessary.
> And since libprocess is widely used in Mesos (master, agent, scheduler,
> executor), it may also cause some performance issue. For example, when user
> creates a Docker container via Mesos in a Mesos agent which is running on a
> powerful machine with 192 cores, the DockerContainerizer in Mesos agent will
> create a dedicated executor for the container, and there will be 192 worker
> threads in that executor. And if user creates 1000 Docker containers in that
> machine, then there will be 1000 executors, i.e., 1000 * 192 worker threads
> which is a large number and may thrash the OS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-4353) Limit the number of processes created by libprocess
[
https://issues.apache.org/jira/browse/MESOS-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15110571#comment-15110571
]
Klaus Ma commented on MESOS-4353:
-
+1 for Option 1; it give a chance to operator to optimise it.
> Limit the number of processes created by libprocess
> ---
>
> Key: MESOS-4353
> URL: https://issues.apache.org/jira/browse/MESOS-4353
> Project: Mesos
> Issue Type: Improvement
> Components: libprocess
>Reporter: Qian Zhang
>Assignee: Qian Zhang
>
> Currently libprocess will create {{max(8, number of CPU cores)}} processes
> during the initialization, see
> https://github.com/apache/mesos/blob/0.26.0/3rdparty/libprocess/src/process.cpp#L2146
> for details. This should be OK for a normal machine which has no much cores
> (e.g., 16, 32), but for a powerful machine which may have a large number of
> cores (e.g., an IBM Power machine may have 192 cores), this will cause too
> much worker threads which are not necessary.
> And since libprocess is widely used in Mesos (master, agent, scheduler,
> executor), it may also cause some performance issue. For example, when user
> creates a Docker container via Mesos in a Mesos agent which is running on a
> powerful machine with 192 cores, the DockerContainerizer in Mesos agent will
> create a dedicated executor for the container, and there will be 192 worker
> threads in that executor. And if user creates 1000 Docker containers in that
> machine, then there will be 1000 executors, i.e., 1000 * 192 worker threads
> which is a large number and may thrash the OS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MESOS-4353) Limit the number of processes created by libprocess
[
https://issues.apache.org/jira/browse/MESOS-4353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15109695#comment-15109695
]
Maged Michael commented on MESOS-4353:
--
Some options:
(1) Use an environment variable to set the maximum number of libprocess worker
threads per process. Such value can be set according to the role of the Mesos
process. For example, a Mesos Master process may have a higher maximum number
of libprocess worker threads than a Docker Containerizer.
(2) Automatically detect the type of the Mesos process (master, executor, ...)
and set the maximum number of libprocess worker threads accordingly.
(3) Set a hardwired maximum number of worker threads regardless of the type of
the Mesos process.
I think option (1) is the most flexible.
> Limit the number of processes created by libprocess
> ---
>
> Key: MESOS-4353
> URL: https://issues.apache.org/jira/browse/MESOS-4353
> Project: Mesos
> Issue Type: Improvement
> Components: libprocess
>Reporter: Qian Zhang
>Assignee: Qian Zhang
>
> Currently libprocess will create {{max(8, number of CPU cores)}} processes
> during the initialization, see
> https://github.com/apache/mesos/blob/0.26.0/3rdparty/libprocess/src/process.cpp#L2146
> for details. This should be OK for a normal machine which has no much cores
> (e.g., 16, 32), but for a powerful machine which may have a large number of
> cores (e.g., an IBM Power machine may have 192 cores), this will cause too
> much worker threads which are not necessary.
> And since libprocess is widely used in Mesos (master, agent, scheduler,
> executor), it may also cause some performance issue. For example, when user
> creates a Docker container via Mesos in a Mesos agent which is running on a
> powerful machine with 192 cores, the DockerContainerizer in Mesos agent will
> create a dedicated executor for the container, and there will be 192 worker
> threads in that executor. And if user creates 1000 Docker containers in that
> machine, then there will be 1000 executors, i.e., 1000 * 192 worker threads
> which is a large number and may thrash the OS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
