[
https://issues.apache.org/jira/browse/YARN-8563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16592475#comment-16592475
]
Zhankun Tang edited comment on YARN-8563 at 8/25/18 5:41 AM:
-------------------------------------------------------------
[~leftnoteasy] One question:
*When and how will the prebuilt Docker images be built?*
Build-on-demand? Or we use an image hub that contains the various version
combinations prebuilt in advance?
In my mind it might work like this: When user run the job, we generate the
Dockerfile with the specified version of TF/Python installation sentence and
build a new image based on official images. And this needs a docker registry
similar server to distribute the image(but I guess this is not a big issue?).
This seems to be a more general and extensible way since the Dockerfile can
also include user-specified python packages. In essence, mounting or
downloading the python packages(.whl file) in docker and then pip install them
seems the same way to a customized Docker build.
And it seems the "--localizations"(YARN-8714) can be also solved in
build-on-demand? We download remote file/directories to local and declare it
into the Dockerfile to copy to the image. But utilize localization of YARN is
also a good choice.
was (Author: tangzhankun):
[~leftnoteasy] One question:
*When and how will the prebuilt Docker images be built?*
Build-on-demand? Or we use an image hub that contains the various version
combinations prebuilt in advance?
In my mind it might work like this: When user run the job, we generate the
Dockerfile with the specified version of TF/Python installation sentence and
build a new image based on official images. And this needs a docker registry
similar server to distribute the image(but I guess this is not a big issue?).
This seems to be a more general and extensible way since the Dockerfile can
also include user-specified python packages. In essence, mounting or
downloading the python packages(.whl file) in docker and then pip install them
seems the same way to a customized Docker build.
And it seems the "--localizations"(YARN-8714) can be also solved in
build-on-demand? We download remote file/directories to local and declare it
into the Dockerfile to copy to the image.
> [Submarine] Support users to specify Python/TF package/version/dependencies
> for training job.
> ---------------------------------------------------------------------------------------------
>
> Key: YARN-8563
> URL: https://issues.apache.org/jira/browse/YARN-8563
> Project: Hadoop YARN
> Issue Type: Sub-task
> Reporter: Wangda Tan
> Priority: Major
>
> YARN-8561 assumes all Python / Tensorflow dependencies will be packed to
> docker image. In practice, user doesn't want to build docker image. Instead,
> user can provide python package / dependencies (like .whl), Python and TF
> version. And Submarine can localize specified dependencies to prebuilt base
> Docker images.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
