[
https://issues.apache.org/jira/browse/GEARPUMP-149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Karol Brejna updated GEARPUMP-149:
----------------------------------
Description:
Let's assume that we have a cluster with some "specialized" nodes (for example
a worker on IoT device that is responsible for controlling some actuators,
dedicated workers to achieve data locality, some hardware specific - mem/cpu -
tasks, etc.).
There should be a way to "hint" Gearpump to prefer some nodes (workers) over
others for certain tasks (executors).
We already have "localities" feature. Developer can add a configuration (when
submitting the application) to bind a task with a worker:
{code}
gearpump {
streaming {
localities {
app1: { //This is the application name to be submitted
workerId: [ //The worker id where to launch Task0_0 and Task0_1
TaskId(0,0), TaskId(0,1)
]
}
}
}
}
{code}
The problem is: users have no idea what each Task's task ID is (especially if
there are lots of nodes in a DAG).
We could introduce more flexible "binding" mechanism, for example labeling
tasks and workers or using some other "awareness" algorithm.
was:
Let's assume that we have a cluster with some "specialized" nodes (for
example a worker on IoT device that is responsible for controlling some
actuators, dedicated workers to achieve data locality, some hardware
specific - mem/cpu - tasks, etc.).
There should be a way to "hint" Gearpump to prefer some nodes (workers) over
others for certain tasks (executors).
We already have "localities" feature. Developer can add a configuration (when
submitting the application) to bind a task with a worker:
{code}
gearpump {
streaming {
localities {
app1: { //This is the application name to be submitted
workerId: [ //The worker id where to launch Task0_0 and Task0_1
TaskId(0,0), TaskId(0,1)
]
}
}
}
}
{code}
The problem is: users have no idea what each Task's task ID is (especially if
there are lots of nodes in a DAG).
We could introduce more flexible "binding" mechanism, for example labeling
tasks and workers or using some other "awareness" algorithm.
> Binding tasks to workers
> ------------------------
>
> Key: GEARPUMP-149
> URL: https://issues.apache.org/jira/browse/GEARPUMP-149
> Project: Apache Gearpump
> Issue Type: Improvement
> Reporter: Karol Brejna
> Priority: Minor
>
> Let's assume that we have a cluster with some "specialized" nodes (for
> example a worker on IoT device that is responsible for controlling some
> actuators, dedicated workers to achieve data locality, some hardware specific
> - mem/cpu - tasks, etc.).
> There should be a way to "hint" Gearpump to prefer some nodes (workers) over
> others for certain tasks (executors).
> We already have "localities" feature. Developer can add a configuration (when
> submitting the application) to bind a task with a worker:
> {code}
> gearpump {
> streaming {
> localities {
> app1: { //This is the application name to be submitted
> workerId: [ //The worker id where to launch Task0_0 and Task0_1
> TaskId(0,0), TaskId(0,1)
> ]
> }
> }
> }
> }
> {code}
> The problem is: users have no idea what each Task's task ID is (especially if
> there are lots of nodes in a DAG).
> We could introduce more flexible "binding" mechanism, for example labeling
> tasks and workers or using some other "awareness" algorithm.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
