[jira] [Commented] (HDFS-13383) Ozone: start-ozone.sh fail to start datanode because of incomplete classpaths
[
https://issues.apache.org/jira/browse/HDFS-13383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16425585#comment-16425585
]
Elek, Marton commented on HDFS-13383:
-
Adding ozone jar files to the classpath all the time even if the yarn/hdfs
component is not ozone relateed could be dangerous.
I suggest to solve it with setting environment variable
HADOOP_SHELL_PROFILES=ozone (didn't test it yet, but will do).
> Ozone: start-ozone.sh fail to start datanode because of incomplete classpaths
> -
>
> Key: HDFS-13383
> URL: https://issues.apache.org/jira/browse/HDFS-13383
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
>Priority: Major
> Fix For: HDFS-7240
>
> Attachments: HDFS-13383-HDFS-7240.001.patch,
> HDFS-13383-HDFS-7240.002.patch
>
>
> start-ozone.sh calls start-dfs.sh to start the NN and DN in a ozone cluster.
> Starting of datanode fails because of incomplete classpaths as datanode is
> unable to load all the plugins.
> Setting the class path to the following values does resolve the issue:
> {code}
> export
> HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/opt/hadoop/hadoop-3.2.0-SNAPSHOT/share/hadoop/ozone/*:/opt/hadoop/hadoop-3.2.0-SNAPSHOT/share/hadoop/hdsl/*:/opt/hadoop/hadoop-3.2.0-SNAPSHOT/share/hadoop/ozone/lib/*:/opt/hadoop/hadoop-3.2.0-SNAPSHOT/share/hadoop/hdsl/lib/*:/opt/hadoop/hadoop-3.2.0-SNAPSHOT/share/hadoop/cblock/*:/opt/hadoop/hadoop-3.2.0-SNAPSHOT/share/hadoop/cblock/lib/*
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13383) Ozone: start-ozone.sh fail to start datanode because of incomplete classpaths
[
https://issues.apache.org/jira/browse/HDFS-13383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424198#comment-16424198
]
genericqa commented on HDFS-13383:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 18m
0s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 27m
4s{color} | {color:green} HDFS-7240 passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
18s{color} | {color:red} common in HDFS-7240 failed. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 28s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
12s{color} | {color:red} common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
1s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
14s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 49s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 16s{color}
| {color:red} common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 69m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:b78c94f |
| JIRA Issue | HDFS-13383 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12917373/HDFS-13383-HDFS-7240.002.patch
|
| Optional Tests | asflicense mvnsite unit shellcheck shelldocs |
| uname | Linux 8dc6f6c291eb 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-7240 / ac77b18 |
| maven | version: Apache Maven 3.3.9 |
| mvnsite |
https://builds.apache.org/job/PreCommit-HDFS-Build/23761/artifact/out/branch-mvnsite-hadoop-ozone_common.txt
|
| shellcheck | v0.4.6 |
| mvnsite |
https://builds.apache.org/job/PreCommit-HDFS-Build/23761/artifact/out/patch-mvnsite-hadoop-ozone_common.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/23761/artifact/out/patch-unit-hadoop-ozone_common.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/23761/testReport/ |
| Max. process+thread count | 313 (vs. ulimit of 1) |
| modules | C: hadoop-ozone/common U: hadoop-ozone/common |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/23761/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Ozone: start-ozone.sh fail to start datanode because of incomplete classpaths
> -
>
> Key: HDFS-13383
> URL: https://issues.apache.org/jira/browse/HDFS-13383
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
>Priority: Major
> Fix For: HDFS-7240
>
> Attachments: HDFS-13383-HDFS-7240.001.patch,
> HDFS-13383-HDFS-7240.002.patch
>
>
> start-ozone.sh calls start-dfs.sh to start the NN and DN in a ozone cluster.
> Starting of datanode fails because of incomplete classpaths as
[jira] [Commented] (HDFS-13383) Ozone: start-ozone.sh fail to start datanode because of incomplete classpaths
[
https://issues.apache.org/jira/browse/HDFS-13383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16424098#comment-16424098
]
Nanda kumar commented on HDFS-13383:
Thanks [~msingh] for working on this.
+1 on patch v002, LGTM. I will commit this shortly.
> Ozone: start-ozone.sh fail to start datanode because of incomplete classpaths
> -
>
> Key: HDFS-13383
> URL: https://issues.apache.org/jira/browse/HDFS-13383
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
>Priority: Major
> Fix For: HDFS-7240
>
> Attachments: HDFS-13383-HDFS-7240.001.patch,
> HDFS-13383-HDFS-7240.002.patch
>
>
> start-ozone.sh calls start-dfs.sh to start the NN and DN in a ozone cluster.
> Starting of datanode fails because of incomplete classpaths as datanode is
> unable to load all the plugins.
> Setting the class path to the following values does resolve the issue:
> {code}
> export
> HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/opt/hadoop/hadoop-3.2.0-SNAPSHOT/share/hadoop/ozone/*:/opt/hadoop/hadoop-3.2.0-SNAPSHOT/share/hadoop/hdsl/*:/opt/hadoop/hadoop-3.2.0-SNAPSHOT/share/hadoop/ozone/lib/*:/opt/hadoop/hadoop-3.2.0-SNAPSHOT/share/hadoop/hdsl/lib/*:/opt/hadoop/hadoop-3.2.0-SNAPSHOT/share/hadoop/cblock/*:/opt/hadoop/hadoop-3.2.0-SNAPSHOT/share/hadoop/cblock/lib/*
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13383) Ozone: start-ozone.sh fail to start datanode because of incomplete classpaths
[
https://issues.apache.org/jira/browse/HDFS-13383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423538#comment-16423538
]
Mukul Kumar Singh commented on HDFS-13383:
--
Thanks for looking at the patch [~shashikant]. I just checked the latest repo
and found that the compilation doesn't generate the jars.
We are only generating the tests jars.
{code}
HW13605:ozone_review msingh$ ls
hadoop-dist/target/hadoop-3.2.0-SNAPSHOT/share/hadoop/hdsl/
hadoop-hdsl-common-3.2.0-SNAPSHOT-tests.jar
hadoop-hdsl-server-scm-3.2.0-SNAPSHOT-tests.jar sources
hadoop-hdsl-container-service-3.2.0-SNAPSHOT-tests.jar
hadoop-hdsl-tools-3.2.0-SNAPSHOT-tests.jar webapps
hadoop-hdsl-server-framework-3.2.0-SNAPSHOT-tests.jar
hadoop-ozone-tools-3.2.0-SNAPSHOT-tests.jar
{code}
> Ozone: start-ozone.sh fail to start datanode because of incomplete classpaths
> -
>
> Key: HDFS-13383
> URL: https://issues.apache.org/jira/browse/HDFS-13383
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
>Priority: Major
> Fix For: HDFS-7240
>
> Attachments: HDFS-13383-HDFS-7240.001.patch
>
>
> start-ozone.sh calls start-dfs.sh to start the NN and DN in a ozone cluster.
> Starting of datanode fails because of incomplete classpaths as datanode is
> unable to load all the plugins.
> Setting the class path to the following values does resolve the issue:
> {code}
> export
> HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/opt/hadoop/hadoop-3.2.0-SNAPSHOT/share/hadoop/ozone/*:/opt/hadoop/hadoop-3.2.0-SNAPSHOT/share/hadoop/hdsl/*:/opt/hadoop/hadoop-3.2.0-SNAPSHOT/share/hadoop/ozone/lib/*:/opt/hadoop/hadoop-3.2.0-SNAPSHOT/share/hadoop/hdsl/lib/*:/opt/hadoop/hadoop-3.2.0-SNAPSHOT/share/hadoop/cblock/*:/opt/hadoop/hadoop-3.2.0-SNAPSHOT/share/hadoop/cblock/lib/*
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13383) Ozone: start-ozone.sh fail to start datanode because of incomplete classpaths
[
https://issues.apache.org/jira/browse/HDFS-13383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423498#comment-16423498
]
Shashikant Banerjee commented on HDFS-13383:
Thanks [~msingh], for working on this. I just applied the patch and ran oz
script to start KSM ,SCM etc... The issue still seems to exist there.
HW15685:hadoop sbanerjee$ ./hadoop-dist/target/hadoop-3.2.0-SNAPSHOT/bin/oz
genesis -h
Error: Could not find or load main class org.apache.hadoop.ozone.genesis.Genesis
HW15685:hadoop sbanerjee$ ./hadoop-dist/target/hadoop-3.2.0-SNAPSHOT/bin/oz scm
WARNING: /Users/sbanerjee/hadoop/hadoop-dist/target/hadoop-3.2.0-SNAPSHOT/logs
does not exist. Creating.
Error: Could not find or load main class
org.apache.hadoop.ozone.scm.StorageContainerManager
> Ozone: start-ozone.sh fail to start datanode because of incomplete classpaths
> -
>
> Key: HDFS-13383
> URL: https://issues.apache.org/jira/browse/HDFS-13383
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
>Priority: Major
> Fix For: HDFS-7240
>
> Attachments: HDFS-13383-HDFS-7240.001.patch
>
>
> start-ozone.sh calls start-dfs.sh to start the NN and DN in a ozone cluster.
> Starting of datanode fails because of incomplete classpaths as datanode is
> unable to load all the plugins.
> Setting the class path to the following values does resolve the issue:
> {code}
> export
> HADOOP_CLASSPATH=$HADOOP_CLASSPATH:/opt/hadoop/hadoop-3.2.0-SNAPSHOT/share/hadoop/ozone/*:/opt/hadoop/hadoop-3.2.0-SNAPSHOT/share/hadoop/hdsl/*:/opt/hadoop/hadoop-3.2.0-SNAPSHOT/share/hadoop/ozone/lib/*:/opt/hadoop/hadoop-3.2.0-SNAPSHOT/share/hadoop/hdsl/lib/*:/opt/hadoop/hadoop-3.2.0-SNAPSHOT/share/hadoop/cblock/*:/opt/hadoop/hadoop-3.2.0-SNAPSHOT/share/hadoop/cblock/lib/*
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13383) Ozone: start-ozone.sh fail to start datanode because of incomplete classpaths
[
https://issues.apache.org/jira/browse/HDFS-13383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16423409#comment-16423409
]
genericqa commented on HDFS-13383:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} HDFS-7240 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
15s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} HDFS-7240 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 29s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
0s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
14s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 52s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
27s{color} | {color:green} common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
26s{color} | {color:red} The patch generated 4 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 42m 32s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:7a542fb |
| JIRA Issue | HDFS-13383 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12917290/HDFS-13383-HDFS-7240.001.patch
|
| Optional Tests | asflicense mvnsite unit shellcheck shelldocs |
| uname | Linux 4c7d0d28a000 3.13.0-143-generic #192-Ubuntu SMP Tue Feb 27
10:45:36 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-7240 / d0488c7 |
| maven | version: Apache Maven 3.3.9 |
| shellcheck | v0.4.6 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/23755/testReport/ |
| asflicense |
https://builds.apache.org/job/PreCommit-HDFS-Build/23755/artifact/out/patch-asflicense-problems.txt
|
| Max. process+thread count | 341 (vs. ulimit of 1) |
| modules | C: hadoop-ozone/common U: hadoop-ozone/common |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/23755/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Ozone: start-ozone.sh fail to start datanode because of incomplete classpaths
> -
>
> Key: HDFS-13383
> URL: https://issues.apache.org/jira/browse/HDFS-13383
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ozone
>Affects Versions: HDFS-7240
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
>Priority: Major
> Fix For: HDFS-7240
>
> Attachments: HDFS-13383-HDFS-7240.001.patch
>
>
> start-ozone.sh calls start-dfs.sh to start the NN and DN in a ozone cluster.
> Starting of datanode fails because of incomplete classpaths as datanode is
> unable to load all the plugins.
> Setting the class path to the following values does resolve the issue:
> {code}
> export
>
