[
https://issues.apache.org/jira/browse/SPARK-16574?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15388657#comment-15388657
]
Norman He commented on SPARK-16574:
-----------------------------------
Here is how i solved it:
/**
* Using Redis to init and distribute gpuid evenly to all hosts
* @param gpuCount
* @return
*/
def initResourceForGPUDistribution(gpuCount:Int)= {
val r = new RedisClient("bdalab12", 6379)
for( host <- List("bdalab13", "bdalab12", "bdalab14", "bdalab15") ; i <-
Array.range(0, gpuCount)) {
r.rpush(host, i)
}
r.disconnect
}
during init() for each gpuWorkers just rpop()
> Distribute computing to each node based on certain hints
> --------------------------------------------------------
>
> Key: SPARK-16574
> URL: https://issues.apache.org/jira/browse/SPARK-16574
> Project: Spark
> Issue Type: Wish
> Reporter: Norman He
>
> 1) I have gpuWorkers RDD like(each node have 2 gpus)
> val nodes= 10
> val gpuCount = 2
> val cross: Array[(Int, Int)] = for( x <- Array.range(0, nodes); y <-
> Array.range(0, gpuCount ) ) yield (x, y)
> var gpuWorkers: RDD[(Int, Int)] = sc.parallelize(cross, nodes * gpuCount)
> 2) when executor runs, I would somehow like to distribute code to each nodes
> based on cross's gpu index(y) so that each machine 2 gpu can be used.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
