Hi Kazuaki,
I am looking at http://kiszk.github.io/spark-gpu/ , can you point me where is the kick-start scripts that I can give it a go? to be more specifically, what does *"off-loading"* mean? aims to reduce the copy overhead between CPU and GPU? I am a newbie for GPU, how can I specify how many GPU cores I want to use (like --executor-cores) ? At 2016-01-04 11:52:01, "Kazuaki Ishizaki" <ishiz...@jp.ibm.com> wrote: Dear all, We reopened the existing JIRA entry https://issues.apache.org/jira/browse/SPARK-3785to support off-loading computations to a GPU by adding a description for our prototype. We are working to effectively and easily exploit GPUs on Spark at http://github.com/kiszk/spark-gpu. Please also visit our project page http://kiszk.github.io/spark-gpu/. For now, we added a new format for a partition in an RDD, which is a column-based structure in an array format, in addition to the current Iterator[T] format with Seq[T]. This reduces data serialization/deserialization and copy overhead between CPU and GPU. Our prototype achieved more than 3x performance improvement for a simple logistic regression program using a NVIDIA K40 card. This JIRA entry (SPARK-3785) includes a link to a design document. We are very glad to hear valuable feedback/suggestions/comments and to have great discussions to exploit GPUs in Spark. Best Regards, Kazuaki Ishizaki
