[jira] [Updated] (YARN-6043) [HDL] Tensorflow on YARN
[ https://issues.apache.org/jira/browse/YARN-6043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated YARN-6043: --- Attachment: TensorFlow_on_YARN_design.pdf > [HDL] Tensorflow on YARN > > > Key: YARN-6043 > URL: https://issues.apache.org/jira/browse/YARN-6043 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Kai Zheng > Attachments: TensorFlow_on_YARN_design.pdf > > > As discussed in the umbrella HADOOP-13944, we'd like to work and support Deep > Learning on Hadoop. As a beginning, we implemented a prototype running > Tensorflow on YARN. Preliminarily the work provides a tool yarn-tf allowing > users to submit and run a Tensorflow job (say mnist.py) in a YARN cluster. It > allocates and launches a Tensorflow cluster in YARN dynamically, executing > the job, and then destroys the cluster after the work is done. It doesn't > require Python and Tensorflow binary installations be done previously on YARN > nodes (on client host, Python is required if the job is written in Python). > It doesn't go in the Docker approach. Given an existing Hadoop cluster, it's > pretty easy to run a Tensorflow job using the provided yarn-tf.jar bundle > (the TF core library and our JNI wrapper) and yarn-tf tool. > In this jira we'll post our design documenting how we did it and the general > approach. The prototype work is under polishing and will be public here soon. > Filing this as unassigned as it's a team work. Your thoughts and feedback are > welcome. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6043) [HDL] Tensorflow on YARN
[ https://issues.apache.org/jira/browse/YARN-6043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated YARN-6043: --- Attachment: (was: TensorFlow_on_YARN_proposal.pdf) > [HDL] Tensorflow on YARN > > > Key: YARN-6043 > URL: https://issues.apache.org/jira/browse/YARN-6043 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Kai Zheng > > As discussed in the umbrella HADOOP-13944, we'd like to work and support Deep > Learning on Hadoop. As a beginning, we implemented a prototype running > Tensorflow on YARN. Preliminarily the work provides a tool yarn-tf allowing > users to submit and run a Tensorflow job (say mnist.py) in a YARN cluster. It > allocates and launches a Tensorflow cluster in YARN dynamically, executing > the job, and then destroys the cluster after the work is done. It doesn't > require Python and Tensorflow binary installations be done previously on YARN > nodes (on client host, Python is required if the job is written in Python). > It doesn't go in the Docker approach. Given an existing Hadoop cluster, it's > pretty easy to run a Tensorflow job using the provided yarn-tf.jar bundle > (the TF core library and our JNI wrapper) and yarn-tf tool. > In this jira we'll post our design documenting how we did it and the general > approach. The prototype work is under polishing and will be public here soon. > Filing this as unassigned as it's a team work. Your thoughts and feedback are > welcome. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Updated] (YARN-6043) [HDL] Tensorflow on YARN
[ https://issues.apache.org/jira/browse/YARN-6043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhankun Tang updated YARN-6043: --- Attachment: TensorFlow_on_YARN_proposal.pdf Proposal attached. And thanks [~drankye], [~jingcheng.du], [~HuafengWang], [~mauzhang], [~zhongmuzhong] > [HDL] Tensorflow on YARN > > > Key: YARN-6043 > URL: https://issues.apache.org/jira/browse/YARN-6043 > Project: Hadoop YARN > Issue Type: New Feature >Reporter: Kai Zheng > Attachments: TensorFlow_on_YARN_proposal.pdf > > > As discussed in the umbrella HADOOP-13944, we'd like to work and support Deep > Learning on Hadoop. As a beginning, we implemented a prototype running > Tensorflow on YARN. Preliminarily the work provides a tool yarn-tf allowing > users to submit and run a Tensorflow job (say mnist.py) in a YARN cluster. It > allocates and launches a Tensorflow cluster in YARN dynamically, executing > the job, and then destroys the cluster after the work is done. It doesn't > require Python and Tensorflow binary installations be done previously on YARN > nodes (on client host, Python is required if the job is written in Python). > It doesn't go in the Docker approach. Given an existing Hadoop cluster, it's > pretty easy to run a Tensorflow job using the provided yarn-tf.jar bundle > (the TF core library and our JNI wrapper) and yarn-tf tool. > In this jira we'll post our design documenting how we did it and the general > approach. The prototype work is under polishing and will be public here soon. > Filing this as unassigned as it's a team work. Your thoughts and feedback are > welcome. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
