[ https://issues.apache.org/jira/browse/YARN-8563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16592475#comment-16592475 ]
Zhankun Tang edited comment on YARN-8563 at 8/25/18 5:38 AM: ------------------------------------------------------------- [~leftnoteasy] One question: *When and how will the prebuilt Docker images be built?* Build-on-demand? Or we use an image hub that contains the various version combinations prebuilt in advance? In my mind it might work like this: When user run the job, we generate the Dockerfile with the specified version of TF/Python installation sentence and build a new image based on official images. And this needs a docker hub similar server to distribute the image(but I guess this is not a big issue?). This seems to be a more general and extensible way since the Dockerfile can also include user-specified python packages. In essence, mounting or downloading the python packages(.whl file) in docker and then pip install them seems the same way to a customized Docker build. And it seems the "--localizations"(YARN-8714) can be also solved in build-on-demand? We download remote file/directories to local and declare it into the Dockerfile to copy to the image. was (Author: tangzhankun): [~leftnoteasy] One question: *When and how will the prebuilt Docker images be built?* Build-on-demand? Or we use an image hub that contains the various version combinations prebuilt in advance? In my mind it might work like this: When user run the job, we generate the Dockerfile with the specified version of TF/Python installation sentence and build a new image based on official images. This seems to be a more general and extensible way since the Dockerfile can also include user-specified python packages. In essence, mounting or downloading the python packages(.whl file) in docker and then pip install them seems the same way to a customized Docker build. And it seems the "--localizations" can be also solved in build-on-demand? We download remote file/directories to local and declare it into the Dockerfile to copy to the image. > [Submarine] Support users to specify Python/TF package/version/dependencies > for training job. > --------------------------------------------------------------------------------------------- > > Key: YARN-8563 > URL: https://issues.apache.org/jira/browse/YARN-8563 > Project: Hadoop YARN > Issue Type: Sub-task > Reporter: Wangda Tan > Priority: Major > > YARN-8561 assumes all Python / Tensorflow dependencies will be packed to > docker image. In practice, user doesn't want to build docker image. Instead, > user can provide python package / dependencies (like .whl), Python and TF > version. And Submarine can localize specified dependencies to prebuilt base > Docker images. -- This message was sent by Atlassian JIRA (v7.6.3#76005) --------------------------------------------------------------------- To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org