[
https://issues.apache.org/jira/browse/YARN-6043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15952113#comment-15952113
]
Zhankun Tang commented on YARN-6043:
------------------------------------
Given our TensorFlow on YARN prototype is basically working . Here's the repo
and design(both is fast changing) for review:
Repo:
https://github.com/Intel-bigdata/TensorFlowOnYARN
Design:
https://docs.google.com/document/d/1EtAKg6-GwOa5-39GjccREm6y6DErEUUhNn4LfGzixjA/edit?usp=sharing
> [HDL] Tensorflow on YARN
> ------------------------
>
> Key: YARN-6043
> URL: https://issues.apache.org/jira/browse/YARN-6043
> Project: Hadoop YARN
> Issue Type: New Feature
> Reporter: Kai Zheng
> Attachments: TensorFlow_on_YARN_design.pdf
>
>
> As discussed in the umbrella HADOOP-13944, we'd like to work and support Deep
> Learning on Hadoop. As a beginning, we implemented a prototype running
> Tensorflow on YARN. Preliminarily the work provides a tool yarn-tf allowing
> users to submit and run a Tensorflow job (say mnist.py) in a YARN cluster. It
> allocates and launches a Tensorflow cluster in YARN dynamically, executing
> the job, and then destroys the cluster after the work is done. It doesn't
> require Python and Tensorflow binary installations be done previously on YARN
> nodes (on client host, Python is required if the job is written in Python).
> It doesn't go in the Docker approach. Given an existing Hadoop cluster, it's
> pretty easy to run a Tensorflow job using the provided yarn-tf.jar bundle
> (the TF core library and our JNI wrapper) and yarn-tf tool.
> In this jira we'll post our design documenting how we did it and the general
> approach. The prototype work is under polishing and will be public here soon.
> Filing this as unassigned as it's a team work. Your thoughts and feedback are
> welcome.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
