[
https://issues.apache.org/jira/browse/SUBMARINE-507?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17117911#comment-17117911
]
Manikandan R commented on SUBMARINE-507:
----------------------------------------
Writing down my thoughts on storing environments..Please share your views.
1. Following tables can be created in Submarine Metastore:
a) Table Name: environments
Columns:
environment_id int primary key
name varchar(255) unique not null
description string
location string
docker_id int references docker_images(docker_id)
kernel_id int references kernel(kernel_id)
created_date timestamp
last_updated_date timestamp
"location" column captures hdfs path of the environment file.
b) Table Name: docker_images
docker_id int primary key
name varchar(255) unique not null
description string
created_date timestamp
last_updated_date timestamp
c) Table Name: kernel
kernel_id int primary key
name varchar(255) unique not null
description string
repository string
repository_type enum(''private', 'public')
created_date timestamp
last_updated_date timestamp
Having separate tables for docker_images and kernel give us lot of flexibility
while operating environments.
- docker and kernel images could be created only once and used for many
environments.
- Avoid creating the same images and kernel/conda again and again in registry
and repository respectively.
(If required, we can clean up these 2 tables if it grows very very big and
becomes a bottleneck, but very unlikely).
2. How to store environment file?
Create a directory hdfs://mycluster/submarine/environments/ if it doesn't
exists and use environment name as file name. For example,
hdfs://mycluster/submarine/environments/my_env.txt
3. How to store docker_images?
We could set up our own registry as part of starting up the server. Please
refer [https://www.docker.com/blog/how-to-use-your-own-registry/] for details.
There are several options for this storage as documented in
[https://docs.docker.com/registry/configuration/#storage]. For first cut, We
can begin with file system and can be iterated over next releases based on the
need.
4. How to store kernel/conda?
There are 2 types. 1. Private 2. Public.
For private repo, we will need to set up local repo's and can be used.
> Submarine Environment Management
> --------------------------------
>
> Key: SUBMARINE-507
> URL: https://issues.apache.org/jira/browse/SUBMARINE-507
> Project: Apache Submarine
> Issue Type: New Feature
> Reporter: Manikandan R
> Assignee: Manikandan R
> Priority: Major
> Labels: pull-request-available
>
> Scope of this JIRA is to support environment management. It includes the
> following:
> 1. Create Environment
> 2. Update Environment
> 3. Delete Environment
> 4. List Environments
> In addition, this JIRA should also ensures that environments has been
> persisted like experiments so that it can used for later use.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
