[jira] [Commented] (YARN-8913) Add helper scripts to launch MaWo App to run Hadoop unit tests on Hadoop Cluster
[
https://issues.apache.org/jira/browse/YARN-8913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16844900#comment-16844900
]
Hadoop QA commented on YARN-8913:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} YARN-8913 does not apply to trunk. Rebase required? Wrong Branch?
See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | YARN-8913 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12946129/YARN-8913.003.patch |
| Console output |
https://builds.apache.org/job/PreCommit-YARN-Build/24124/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Add helper scripts to launch MaWo App to run Hadoop unit tests on Hadoop
> Cluster
>
>
> Key: YARN-8913
> URL: https://issues.apache.org/jira/browse/YARN-8913
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Yesha Vora
>Assignee: Yesha Vora
>Priority: Blocker
> Attachments: YARN-8913.001.patch, YARN-8913.002.patch,
> YARN-8913.003.patch
>
>
> MaWo application can be used to run Hadoop UT faster in a Hadoop cluster.
> Develop helper scripts to orchestrate end-to-end workflow for running Hadoop
> UT using MaWo app.
> Pre-requisite:
> * A Hadoop Cluster with HDFS and YARN installed
> * Enable Docker on YARN feature
>
> Helper-scripts
> * MaWo_Driver
> ** create a docker image with latest hadoop source code
> ** create payload to MaWo app (This is input to mawo app where Each MaWo
> Task = UT execution of each Hadoop Module)
> ** Upload payload file to HDFS
> ** Update MaWo-Launch.json to resolve RM_HOST / Docker Image etc dynamically
> ** Launch MaWo app in Hadoop cluster
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8913) Add helper scripts to launch MaWo App to run Hadoop unit tests on Hadoop Cluster
[ https://issues.apache.org/jira/browse/YARN-8913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714313#comment-16714313 ] Allen Wittenauer commented on YARN-8913: This should absolutely be using the dev-support/docker/Dockerfile and not something custom. As it stands, this Dockerfile is missing some absolutely critical parts (eg, everything native) and is downloading components that are very different than what the ASF actually uses for its tests. (Why is maven coming from Fedora instead of, you know, Apache?) > Add helper scripts to launch MaWo App to run Hadoop unit tests on Hadoop > Cluster > > > Key: YARN-8913 > URL: https://issues.apache.org/jira/browse/YARN-8913 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Yesha Vora >Assignee: Yesha Vora >Priority: Blocker > Attachments: YARN-8913.001.patch, YARN-8913.002.patch, > YARN-8913.003.patch > > > MaWo application can be used to run Hadoop UT faster in a Hadoop cluster. > Develop helper scripts to orchestrate end-to-end workflow for running Hadoop > UT using MaWo app. > Pre-requisite: > * A Hadoop Cluster with HDFS and YARN installed > * Enable Docker on YARN feature > > Helper-scripts > * MaWo_Driver > ** create a docker image with latest hadoop source code > ** create payload to MaWo app (This is input to mawo app where Each MaWo > Task = UT execution of each Hadoop Module) > ** Upload payload file to HDFS > ** Update MaWo-Launch.json to resolve RM_HOST / Docker Image etc dynamically > ** Launch MaWo app in Hadoop cluster -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8913) Add helper scripts to launch MaWo App to run Hadoop unit tests on Hadoop Cluster
[ https://issues.apache.org/jira/browse/YARN-8913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667888#comment-16667888 ] Yesha Vora commented on YARN-8913: -- [~eyang], Thanks for feedback. Please find attached patch#3 addressing your suggestions. > Add helper scripts to launch MaWo App to run Hadoop unit tests on Hadoop > Cluster > > > Key: YARN-8913 > URL: https://issues.apache.org/jira/browse/YARN-8913 > Project: Hadoop YARN > Issue Type: Sub-task >Reporter: Yesha Vora >Assignee: Yesha Vora >Priority: Blocker > Attachments: YARN-8913.001.patch, YARN-8913.002.patch, > YARN-8913.003.patch > > > MaWo application can be used to run Hadoop UT faster in a Hadoop cluster. > Develop helper scripts to orchestrate end-to-end workflow for running Hadoop > UT using MaWo app. > Pre-requisite: > * A Hadoop Cluster with HDFS and YARN installed > * Enable Docker on YARN feature > > Helper-scripts > * MaWo_Driver > ** create a docker image with latest hadoop source code > ** create payload to MaWo app (This is input to mawo app where Each MaWo > Task = UT execution of each Hadoop Module) > ** Upload payload file to HDFS > ** Update MaWo-Launch.json to resolve RM_HOST / Docker Image etc dynamically > ** Launch MaWo app in Hadoop cluster -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
[jira] [Commented] (YARN-8913) Add helper scripts to launch MaWo App to run Hadoop unit tests on Hadoop Cluster
[
https://issues.apache.org/jira/browse/YARN-8913?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16660947#comment-16660947
]
Eric Yang commented on YARN-8913:
-
Launch command looks like this in patch 002:
{code}
cd /tmp/mawo/hadoop-applications-mawo-0.0.1-SNAPSHOT/bin;export
JAVA_HOME=/usr/lib/jvm/java-1.8.0;/tmp/mawo/hadoop-applications-mawo-0.0.1-SNAPSHOT/bin/mawo
--config /tmp/mawo-config master -job_descripiton_file_path
/tmp/mawo/mawo-payload.json -mawo_app_name mawo-hadoop-ut
{code}
This is trying to do 3 things at the same time.
# Change working directory to
/tmp/mawo/hadoop-applications-mawo-0.0.1-SNAPSHOT/bin
# Set JAVA_HOME environment variable
# Run Mawo script with configuration parameters
It will greatly simplify the complexity by using an entry-point.sh to
bootstrap. Mawo script can source mawo-env.sh to locate JAVA_HOME location
like hadoop's libexec/hadoop-config.sh. The working directory can be computed
from basedir $0/.. in the entry-point.sh. This will change the launch command
to call a fixed entry-point.sh to improve readability and prevent path
traversal security exploits in the container.
> Add helper scripts to launch MaWo App to run Hadoop unit tests on Hadoop
> Cluster
>
>
> Key: YARN-8913
> URL: https://issues.apache.org/jira/browse/YARN-8913
> Project: Hadoop YARN
> Issue Type: Sub-task
>Reporter: Yesha Vora
>Assignee: Yesha Vora
>Priority: Blocker
> Attachments: YARN-8913.001.patch, YARN-8913.002.patch
>
>
> MaWo application can be used to run Hadoop UT faster in a Hadoop cluster.
> Develop helper scripts to orchestrate end-to-end workflow for running Hadoop
> UT using MaWo app.
> Pre-requisite:
> * A Hadoop Cluster with HDFS and YARN installed
> * Enable Docker on YARN feature
>
> Helper-scripts
> * MaWo_Driver
> ** create a docker image with latest hadoop source code
> ** create payload to MaWo app (This is input to mawo app where Each MaWo
> Task = UT execution of each Hadoop Module)
> ** Upload payload file to HDFS
> ** Update MaWo-Launch.json to resolve RM_HOST / Docker Image etc dynamically
> ** Launch MaWo app in Hadoop cluster
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: yarn-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: yarn-issues-h...@hadoop.apache.org
