[
https://issues.apache.org/jira/browse/YARN-7129?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16541904#comment-16541904
]
Eric Yang edited comment on YARN-7129 at 7/13/18 10:55 PM:
-----------------------------------------------------------
[~Zian Chen] Thank you for reviewing this patch. Here is how I run the
application catalog:
1. Run mvn clean package
2. If docker present on the build machine. There is a docker image built with
name: hadoop/appcatalog-docker:3.2.0-SNAPSHOT
3. Transfer the image to Hadoop cluster with docker installed:
{code}
docker save hadoop/appcatalog-docker:3.2.0-SNAPSHOT -o appcatalog.tar
{code}
4. Import image to docker on cluster nodes:
{code}
docker load -i appcatalog.tar
{code}
5. Create a yarnfile with content:
{code}
{
"name": "catalog",
"kerberos_principal" : {
"principal_name" : "eyang/_h...@example.com",
"keytab" : "file:///etc/security/keytabs/eyang.service.keytab"
},
"version": "1",
"components" :
[
{
"name": "appcatalog",
"number_of_containers": 1,
"artifact": {
"id": "hadoop/appcatalog-docker:3.2.0-SNAPSHOT",
"type": "DOCKER"
},
"resource": {
"cpus": 1,
"memory": "256"
},
"run_privileged_container": true,
"configuration": {
"env": {
"YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE":"true",
"YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS":"/usr/local/hadoop-3.2.0-SNAPSHOT/etc/hadoop:/etc/hadoop/conf:ro,/etc/krb5.conf:/etc/krb5.conf:ro,/etc/security/keytabs/eyang.service.keytab:/etc/security/keytabs/eyang.service.keytab:ro",
"KEYTAB":"/etc/security/keytabs/eyang.service.keytab",
"PRINCIPAL":"ey...@example.com"
},
"properties": {
"docker.network": "host"
}
}
}
]
}
{code}
6. Launch the application with:
{code}
yarn app -launch appcatalog yarnfile
{code}
7. Look at the application master log file, and it will report where the
application is launched, and use web browser to visit port 8080 of the
application catalog.
Mount paths used in this yarn file:
| Source | Destination | Purpose |
| /usr/local/hadoop-3.2.0-SNAPSHOT/etc/hadoop | /etc/hadoop/conf | Read only
Hadoop configuration |
| /etc/krb5.conf | /etc/krb5.conf | Read only Kerberos configuration |
| /etc/security/keytabs/eyang.service.keytab |
/etc/security/keytabs/eyang.service.keytab | Kerberos keytab used by
application |
KEYTAB, and PRINCIPAL environment variables are used to generate jaas
configuration for application catalog.
8. Before deploying any sample application, container-executor.cfg may need to
include: {{docker.trusted.registries=hadoop,eboraas,jenkins}}.
was (Author: eyang):
[~Zian Chen] Thank you for reviewing this patch. Here is how I run the
application catalog:
1. Create a yarnfile with content:
{code}
{
"name": "catalog",
"kerberos_principal" : {
"principal_name" : "eyang/_h...@example.com",
"keytab" : "file:///etc/security/keytabs/eyang.service.keytab"
},
"version": "1",
"components" :
[
{
"name": "appcatalog",
"number_of_containers": 1,
"artifact": {
"id": "hadoop/appcatalog-docker:3.2.0-SNAPSHOT",
"type": "DOCKER"
},
"resource": {
"cpus": 1,
"memory": "256"
},
"run_privileged_container": true,
"configuration": {
"env": {
"YARN_CONTAINER_RUNTIME_DOCKER_RUN_OVERRIDE_DISABLE":"true",
"YARN_CONTAINER_RUNTIME_DOCKER_MOUNTS":"/usr/local/hadoop-3.2.0-SNAPSHOT/etc/hadoop:/etc/hadoop/conf:ro,/etc/krb5.conf:/etc/krb5.conf:ro,/etc/security/keytabs/eyang.service.keytab:/etc/security/keytabs/eyang.service.keytab:ro",
"KEYTAB":"/etc/security/keytabs/eyang.service.keytab",
"PRINCIPAL":"ey...@example.com"
},
"properties": {
"docker.network": "host"
}
}
}
]
}
{code}
2. Launch the application with:
{code}
yarn app -launch appcatalog yarnfile
{code}
3. Look at the application master log file, and it will report where the
application is launched, and use web browser to visit port 8080 of the
application catalog.
Mount paths used in this yarn file:
| Source | Destination | Purpose |
| /usr/local/hadoop-3.2.0-SNAPSHOT/etc/hadoop | /etc/hadoop/conf | Read only
Hadoop configuration |
| /etc/krb5.conf | /etc/krb5.conf | Read only Kerberos configuration |
| /etc/security/keytabs/eyang.service.keytab |
/etc/security/keytabs/eyang.service.keytab | Kerberos keytab used by
application |
KEYTAB, and PRINCIPAL environment variables are used to generate jaas
configuration for application catalog.
4. Before deploying any sample application, container-executor.cfg may need to
include: {{docker.trusted.registries=hadoop,eboraas,jenkins}}.
> Application Catalog for YARN applications
> -----------------------------------------
>
> Key: YARN-7129
> URL: https://issues.apache.org/jira/browse/YARN-7129
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: applications
> Reporter: Eric Yang
> Assignee: Eric Yang
> Priority: Major
> Attachments: YARN Appstore.pdf, YARN-7129.001.patch,
> YARN-7129.002.patch, YARN-7129.003.patch, YARN-7129.004.patch,
> YARN-7129.005.patch, YARN-7129.006.patch, YARN-7129.007.patch
>
>
> YARN native services provides web services API to improve usability of
> application deployment on Hadoop using collection of docker images. It would
> be nice to have an application catalog system which provides an editorial and
> search interface for YARN applications. This improves usability of YARN for
> manage the life cycle of applications.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
