[jira] [Comment Edited] (HDFS-14643) [Dynamometer] Merge extra commits from GitHub to Hadoop
[
https://issues.apache.org/jira/browse/HDFS-14643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967520#comment-16967520
]
Takanobu Asanuma edited comment on HDFS-14643 at 11/5/19 1:22 PM:
--
Some of them have been already merged.
* HDFS-14817: [PR #70|https://github.com/linkedin/dynamometer/pull/70]
* HDFS-14824: [PR #76|https://github.com/linkedin/dynamometer/pull/76], [PR
#92|https://github.com/linkedin/dynamometer/pull/92], [PR
#96|https://github.com/linkedin/dynamometer/pull/96]
* HDFS-14825: [PR #84|https://github.com/linkedin/dynamometer/pull/84]
was (Author: tasanuma0829):
Some of them have been already merged.
* HDFS-14817: [PR #90|https://github.com/linkedin/dynamometer/pull/90]
* HDFS-14824: [PR #76|https://github.com/linkedin/dynamometer/pull/76], [PR
#92|https://github.com/linkedin/dynamometer/pull/92], [PR
#96|https://github.com/linkedin/dynamometer/pull/96]
* HDFS-14825: [PR #84|https://github.com/linkedin/dynamometer/pull/84]
> [Dynamometer] Merge extra commits from GitHub to Hadoop
> ---
>
> Key: HDFS-14643
> URL: https://issues.apache.org/jira/browse/HDFS-14643
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
>
> While Dynamometer was in the process of being committed to Hadoop, a few
> patches went in to the GitHub version that haven't yet made it into the
> version committed here. Some of them are related to TravisCI and Bintray
> deployment, which can safely be ignored in a Hadoop context, but a few are
> relevant:
> {code}
> * 2d2591e 2019-05-24 Make XML parsing error message more explicit (PR #97)
> [lfengnan ]
> * 755a298 2019-04-04 Fix misimplemented CountTimeWritable setter and update
> the README docs regarding the output file (PR #96) [Christopher Gregorian
> ]
> * 66d3e19 2019-03-14 Modify AuditReplay workflow to output count and latency
> of operations (PR #92) [Christopher Gregorian ]
> * 5c1d8cd 2019-02-28 Fix issues with the start-workload.sh script (PR #84)
> [Erik Krogen ]
> {code}
> I will use this ticket to track porting these 4 commits into Hadoop's
> Dynamometer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14643) [Dynamometer] Merge extra commits from GitHub to Hadoop
[
https://issues.apache.org/jira/browse/HDFS-14643?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967520#comment-16967520
]
Takanobu Asanuma commented on HDFS-14643:
-
Some of them have been already merged.
* HDFS-14817: [PR #90|https://github.com/linkedin/dynamometer/pull/90]
* HDFS-14824: [PR #76|https://github.com/linkedin/dynamometer/pull/76], [PR
#92|https://github.com/linkedin/dynamometer/pull/92], [PR
#96|https://github.com/linkedin/dynamometer/pull/96]
* HDFS-14825: [PR #84|https://github.com/linkedin/dynamometer/pull/84]
> [Dynamometer] Merge extra commits from GitHub to Hadoop
> ---
>
> Key: HDFS-14643
> URL: https://issues.apache.org/jira/browse/HDFS-14643
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Erik Krogen
>Assignee: Erik Krogen
>Priority: Major
>
> While Dynamometer was in the process of being committed to Hadoop, a few
> patches went in to the GitHub version that haven't yet made it into the
> version committed here. Some of them are related to TravisCI and Bintray
> deployment, which can safely be ignored in a Hadoop context, but a few are
> relevant:
> {code}
> * 2d2591e 2019-05-24 Make XML parsing error message more explicit (PR #97)
> [lfengnan ]
> * 755a298 2019-04-04 Fix misimplemented CountTimeWritable setter and update
> the README docs regarding the output file (PR #96) [Christopher Gregorian
> ]
> * 66d3e19 2019-03-14 Modify AuditReplay workflow to output count and latency
> of operations (PR #92) [Christopher Gregorian ]
> * 5c1d8cd 2019-02-28 Fix issues with the start-workload.sh script (PR #84)
> [Erik Krogen ]
> {code}
> I will use this ticket to track porting these 4 commits into Hadoop's
> Dynamometer.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14891) RBF: namenode links in NameFederation Health page (federationhealth.html) cannot use https scheme
[ https://issues.apache.org/jira/browse/HDFS-14891?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HDFS-14891: Fix Version/s: 3.3.0 Resolution: Fixed Status: Resolved (was: Patch Available) Committed [^HDFS-14891.008.patch] to trunk. Thanks for your contribution, [~risyomei], and thanks for your review, [~elgoiri]! > RBF: namenode links in NameFederation Health page (federationhealth.html) > cannot use https scheme > -- > > Key: HDFS-14891 > URL: https://issues.apache.org/jira/browse/HDFS-14891 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf, ui >Reporter: Xieming Li >Assignee: Xieming Li >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-14891.001.patch, HDFS-14891.002.patch, > HDFS-14891.003.patch, HDFS-14891.004.patch, HDFS-14891.005.patch, > HDFS-14891.006.patch, HDFS-14891.007.patch, HDFS-14891.008.patch, > HDFS-14891.patch > > > The scheme of links in federationhealth.html are hard coded as 'http'. > It should be set to 'https' when dfs.http.policy is HTTPS_ONLY > (HTTP_AND_HTTPS also, maybe) > > [https://github.com/apache/hadoop/blob/c99a12167ff9566012ef32104a3964887d62c899/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/webapps/router/federationhealth.html#L168-L169] > [https://github.com/apache/hadoop/blob/c99a12167ff9566012ef32104a3964887d62c899/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/webapps/router/federationhealth.html#L236] > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14891) RBF: namenode links in NameFederation Health page (federationhealth.html) cannot use https scheme
[ https://issues.apache.org/jira/browse/HDFS-14891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967259#comment-16967259 ] Takanobu Asanuma commented on HDFS-14891: - Thanks for fixing the checkstyle issue. Will commit it soon. > RBF: namenode links in NameFederation Health page (federationhealth.html) > cannot use https scheme > -- > > Key: HDFS-14891 > URL: https://issues.apache.org/jira/browse/HDFS-14891 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf, ui >Reporter: Xieming Li >Assignee: Xieming Li >Priority: Major > Attachments: HDFS-14891.001.patch, HDFS-14891.002.patch, > HDFS-14891.003.patch, HDFS-14891.004.patch, HDFS-14891.005.patch, > HDFS-14891.006.patch, HDFS-14891.007.patch, HDFS-14891.008.patch, > HDFS-14891.patch > > > The scheme of links in federationhealth.html are hard coded as 'http'. > It should be set to 'https' when dfs.http.policy is HTTPS_ONLY > (HTTP_AND_HTTPS also, maybe) > > [https://github.com/apache/hadoop/blob/c99a12167ff9566012ef32104a3964887d62c899/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/webapps/router/federationhealth.html#L168-L169] > [https://github.com/apache/hadoop/blob/c99a12167ff9566012ef32104a3964887d62c899/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/webapps/router/federationhealth.html#L236] > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14891) RBF: namenode links in NameFederation Health page (federationhealth.html) cannot use https scheme
[ https://issues.apache.org/jira/browse/HDFS-14891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967157#comment-16967157 ] Takanobu Asanuma commented on HDFS-14891: - +1 on [^HDFS-14891.007.patch] from me. [~elgoiri] Could you review it again? > RBF: namenode links in NameFederation Health page (federationhealth.html) > cannot use https scheme > -- > > Key: HDFS-14891 > URL: https://issues.apache.org/jira/browse/HDFS-14891 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf, ui >Reporter: Xieming Li >Assignee: Xieming Li >Priority: Major > Attachments: HDFS-14891.001.patch, HDFS-14891.002.patch, > HDFS-14891.003.patch, HDFS-14891.004.patch, HDFS-14891.005.patch, > HDFS-14891.006.patch, HDFS-14891.007.patch, HDFS-14891.patch > > > The scheme of links in federationhealth.html are hard coded as 'http'. > It should be set to 'https' when dfs.http.policy is HTTPS_ONLY > (HTTP_AND_HTTPS also, maybe) > > [https://github.com/apache/hadoop/blob/c99a12167ff9566012ef32104a3964887d62c899/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/webapps/router/federationhealth.html#L168-L169] > [https://github.com/apache/hadoop/blob/c99a12167ff9566012ef32104a3964887d62c899/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/webapps/router/federationhealth.html#L236] > -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-14825) [Dynamometer] Workload doesn't start unless an absolute path of Mapper class given
[
https://issues.apache.org/jira/browse/HDFS-14825?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takanobu Asanuma reassigned HDFS-14825:
---
Assignee: Takanobu Asanuma
> [Dynamometer] Workload doesn't start unless an absolute path of Mapper class
> given
> --
>
> Key: HDFS-14825
> URL: https://issues.apache.org/jira/browse/HDFS-14825
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Soya Miyoshi
>Assignee: Takanobu Asanuma
>Priority: Major
>
> When starting a workload by start-workload.sh, unless an absolute path of
> Mapper is given, the workload doesn't start.
>
> {code:java}
> $ hadoop/tools/dynamometer/dynamometer-workload/bin/start-workload.sh - \
> Dauditreplay.input-path=hdfs:///user/souya/input/audit \
> -Dauditreplay.output-path=hdfs:///user/souya/results/ \
> -Dauditreplay.num-threads=50 -Dauditreplay.log-start-time.ms=5 \
> -nn_uri hdfs://namenode_address:port/ \
> -mapper_class_name AuditReplayMapper
> {code}
> results in
> {code:java}
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> Exception in thread "main" java.lang.ClassNotFoundException: Class
> org.apache.hadoop.tools.dynamometer.workloadgenerator.AuditReplayMapper not
> found
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2572)
> at
> org.apache.hadoop.tools.dynamometer.workloadgenerator.WorkloadDriver.getMapperClass(WorkloadDriver.java:183)
> at
> org.apache.hadoop.tools.dynamometer.workloadgenerator.WorkloadDriver.run(WorkloadDriver.java:127)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.hadoop.tools.dynamometer.workloadgenerator.WorkloadDriver.main(WorkloadDriver.java:172)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14825) [Dynamometer] Workload doesn't start unless an absolute path of Mapper class given
[
https://issues.apache.org/jira/browse/HDFS-14825?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16967152#comment-16967152
]
Takanobu Asanuma commented on HDFS-14825:
-
Thanks for reporting it, [~soyamiyoshi]. I'd like to work on it.
> [Dynamometer] Workload doesn't start unless an absolute path of Mapper class
> given
> --
>
> Key: HDFS-14825
> URL: https://issues.apache.org/jira/browse/HDFS-14825
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Soya Miyoshi
>Priority: Major
>
> When starting a workload by start-workload.sh, unless an absolute path of
> Mapper is given, the workload doesn't start.
>
> {code:java}
> $ hadoop/tools/dynamometer/dynamometer-workload/bin/start-workload.sh - \
> Dauditreplay.input-path=hdfs:///user/souya/input/audit \
> -Dauditreplay.output-path=hdfs:///user/souya/results/ \
> -Dauditreplay.num-threads=50 -Dauditreplay.log-start-time.ms=5 \
> -nn_uri hdfs://namenode_address:port/ \
> -mapper_class_name AuditReplayMapper
> {code}
> results in
> {code:java}
> SLF4J: Defaulting to no-operation (NOP) logger implementation
> SLF4J: See http://www.slf4j.org/codes.html#StaticLoggerBinder for further
> details.
> Exception in thread "main" java.lang.ClassNotFoundException: Class
> org.apache.hadoop.tools.dynamometer.workloadgenerator.AuditReplayMapper not
> found
> at
> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2572)
> at
> org.apache.hadoop.tools.dynamometer.workloadgenerator.WorkloadDriver.getMapperClass(WorkloadDriver.java:183)
> at
> org.apache.hadoop.tools.dynamometer.workloadgenerator.WorkloadDriver.run(WorkloadDriver.java:127)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at
> org.apache.hadoop.tools.dynamometer.workloadgenerator.WorkloadDriver.main(WorkloadDriver.java:172)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14824) [Dynamometer] Dynamometer in org.apache.hadoop.tools does not output the benchmark results.
[ https://issues.apache.org/jira/browse/HDFS-14824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16965259#comment-16965259 ] Takanobu Asanuma commented on HDFS-14824: - Thanks for committing it, [~xkrogen]! > [Dynamometer] Dynamometer in org.apache.hadoop.tools does not output the > benchmark results. > --- > > Key: HDFS-14824 > URL: https://issues.apache.org/jira/browse/HDFS-14824 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Soya Miyoshi >Assignee: Takanobu Asanuma >Priority: Major > Fix For: 3.3.0 > > > According to the latest > [document|https://aajisaka.github.io/hadoop-document/hadoop-project/hadoop-dynamometer/Dynamometer.html > ], the benchmark results will be written in `Dauditreplay.output-path`. > However, current org.apache.hadooop.tools hasn't merged [this pull > request|https://github.com/linkedin/dynamometer/pull/76 ], so it does not > output the benchmark results. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14891) RBF: namenode links in NameFederation Health page (federationhealth.html) cannot use https scheme
[
https://issues.apache.org/jira/browse/HDFS-14891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16963882#comment-16963882
]
Takanobu Asanuma commented on HDFS-14891:
-
Thanks for update the patch, [~risyomei]. Almost [^HDFS-14891.006.patch] looks
good to me. Only one comment,
* Line 238 in federationhealth.html, please use \{webScheme} instead of
\{scheme}.
> RBF: namenode links in NameFederation Health page (federationhealth.html)
> cannot use https scheme
> --
>
> Key: HDFS-14891
> URL: https://issues.apache.org/jira/browse/HDFS-14891
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf, ui
>Reporter: Xieming Li
>Assignee: Xieming Li
>Priority: Major
> Attachments: HDFS-14891.001.patch, HDFS-14891.002.patch,
> HDFS-14891.003.patch, HDFS-14891.004.patch, HDFS-14891.005.patch,
> HDFS-14891.006.patch, HDFS-14891.patch
>
>
> The scheme of links in federationhealth.html are hard coded as 'http'.
> It should be set to 'https' when dfs.http.policy is HTTPS_ONLY
> (HTTP_AND_HTTPS also, maybe)
>
> [https://github.com/apache/hadoop/blob/c99a12167ff9566012ef32104a3964887d62c899/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/webapps/router/federationhealth.html#L168-L169]
> [https://github.com/apache/hadoop/blob/c99a12167ff9566012ef32104a3964887d62c899/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/webapps/router/federationhealth.html#L236]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14824) [Dynamometer] Dynamometer in org.apache.hadoop.tools does not output the benchmark results.
[ https://issues.apache.org/jira/browse/HDFS-14824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16963243#comment-16963243 ] Takanobu Asanuma commented on HDFS-14824: - [~xkrogen] Thanks for your review and letting me know the PRs! I'll also merge them in this jira. > [Dynamometer] Dynamometer in org.apache.hadoop.tools does not output the > benchmark results. > --- > > Key: HDFS-14824 > URL: https://issues.apache.org/jira/browse/HDFS-14824 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Soya Miyoshi >Assignee: Takanobu Asanuma >Priority: Major > > According to the latest > [document|https://aajisaka.github.io/hadoop-document/hadoop-project/hadoop-dynamometer/Dynamometer.html > ], the benchmark results will be written in `Dauditreplay.output-path`. > However, current org.apache.hadooop.tools hasn't merged [this pull > request|https://github.com/linkedin/dynamometer/pull/76 ], so it does not > output the benchmark results. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14824) [Dynamometer] Dynamometer in org.apache.hadoop.tools does not output the benchmark results.
[ https://issues.apache.org/jira/browse/HDFS-14824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HDFS-14824: Status: Patch Available (was: Open) Sent a PR. I almost just merged [this original PR|https://github.com/linkedin/dynamometer/pull/76], fixed the license and checkstyle warns. > [Dynamometer] Dynamometer in org.apache.hadoop.tools does not output the > benchmark results. > --- > > Key: HDFS-14824 > URL: https://issues.apache.org/jira/browse/HDFS-14824 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Soya Miyoshi >Assignee: Takanobu Asanuma >Priority: Major > > According to the latest > [document|https://aajisaka.github.io/hadoop-document/hadoop-project/hadoop-dynamometer/Dynamometer.html > ], the benchmark results will be written in `Dauditreplay.output-path`. > However, current org.apache.hadooop.tools hasn't merged [this pull > request|https://github.com/linkedin/dynamometer/pull/76 ], so it does not > output the benchmark results. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14824) [Dynamometer] Dynamometer in org.apache.hadoop.tools does not output the benchmark results.
[ https://issues.apache.org/jira/browse/HDFS-14824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16962616#comment-16962616 ] Takanobu Asanuma commented on HDFS-14824: - Thanks for the confirmation. I'd like to assign this jira to myself. > [Dynamometer] Dynamometer in org.apache.hadoop.tools does not output the > benchmark results. > --- > > Key: HDFS-14824 > URL: https://issues.apache.org/jira/browse/HDFS-14824 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Soya Miyoshi >Priority: Major > > According to the latest > [document|https://aajisaka.github.io/hadoop-document/hadoop-project/hadoop-dynamometer/Dynamometer.html > ], the benchmark results will be written in `Dauditreplay.output-path`. > However, current org.apache.hadooop.tools hasn't merged [this pull > request|https://github.com/linkedin/dynamometer/pull/76 ], so it does not > output the benchmark results. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-14824) [Dynamometer] Dynamometer in org.apache.hadoop.tools does not output the benchmark results.
[ https://issues.apache.org/jira/browse/HDFS-14824?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma reassigned HDFS-14824: --- Assignee: Takanobu Asanuma > [Dynamometer] Dynamometer in org.apache.hadoop.tools does not output the > benchmark results. > --- > > Key: HDFS-14824 > URL: https://issues.apache.org/jira/browse/HDFS-14824 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Soya Miyoshi >Assignee: Takanobu Asanuma >Priority: Major > > According to the latest > [document|https://aajisaka.github.io/hadoop-document/hadoop-project/hadoop-dynamometer/Dynamometer.html > ], the benchmark results will be written in `Dauditreplay.output-path`. > However, current org.apache.hadooop.tools hasn't merged [this pull > request|https://github.com/linkedin/dynamometer/pull/76 ], so it does not > output the benchmark results. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14907) [Dynamometer] DataNode can't find junit jar when using Hadoop-3 binary
[
https://issues.apache.org/jira/browse/HDFS-14907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takanobu Asanuma updated HDFS-14907:
Resolution: Fixed
Status: Resolved (was: Patch Available)
[~xkrogen] Thanks for reviewing and committing it!
> [Dynamometer] DataNode can't find junit jar when using Hadoop-3 binary
> --
>
> Key: HDFS-14907
> URL: https://issues.apache.org/jira/browse/HDFS-14907
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Fix For: 3.3.0
>
>
> When executing {{start-dynamometer-cluster.sh}} with Hadoop-3 binary,
> datanodes fail to run with the following log and
> {{start-dynamometer-cluster.sh}} fails.
> {noformat}
> LogType:stderr
> LogLastModifiedTime:Wed Oct 09 15:03:09 +0900 2019
> LogLength:1386
> LogContents:
> Exception in thread "main" java.lang.NoClassDefFoundError: org/junit/Assert
> at
> org.apache.hadoop.test.GenericTestUtils.assertExists(GenericTestUtils.java:299)
> at
> org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:243)
> at
> org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:252)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.getBaseDirectory(MiniDFSCluster.java:2982)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.determineDfsBaseDir(MiniDFSCluster.java:2972)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.formatDataNodeDirs(MiniDFSCluster.java:2834)
> at
> org.apache.hadoop.tools.dynamometer.SimulatedDataNodes.run(SimulatedDataNodes.java:123)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at
> org.apache.hadoop.tools.dynamometer.SimulatedDataNodes.main(SimulatedDataNodes.java:88)
> Caused by: java.lang.ClassNotFoundException: org.junit.Assert
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 9 more
> ./start-component.sh: line 317: kill: (2261) - No such process
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14934) [SBN Read] Standby NN throws many InterruptedExceptions when dfs.ha.tail-edits.period is 0
[
https://issues.apache.org/jira/browse/HDFS-14934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16961814#comment-16961814
]
Takanobu Asanuma commented on HDFS-14934:
-
Sorry I haven't find the fix. If increasing
{{dfs.ha.tail-edits.period.backoff-max}} (like 10s), we can reduce the warn log
frequency.
> [SBN Read] Standby NN throws many InterruptedExceptions when
> dfs.ha.tail-edits.period is 0
> --
>
> Key: HDFS-14934
> URL: https://issues.apache.org/jira/browse/HDFS-14934
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Takanobu Asanuma
>Priority: Major
>
> When dfs.ha.tail-edits.period is 0ms (or very short-time), there are many
> warn logs in standby NN.
> {noformat}
> 2019-10-25 16:25:46,945 [Logger channel (from parallel executor) to hostname>/:] WARN concurrent.ExecutorHelper
> (ExecutorHelper.java:logThrowableFromAfterExecute(55)) - Thread
> (Thread[Logger channel (from parallel executor) to / address>:,5,main]) interrupted:
> java.lang.InterruptedException
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:509)
> at
> com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:82)
> at
> org.apache.hadoop.util.concurrent.ExecutorHelper.logThrowableFromAfterExecute(ExecutorHelper.java:48)
> at
> org.apache.hadoop.util.concurrent.HadoopThreadPoolExecutor.afterExecute(HadoopThreadPoolExecutor.java:90)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1157)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14934) [SBN Read] Standby NN throws many InterruptedExceptions when dfs.ha.tail-edits.period is 0
[
https://issues.apache.org/jira/browse/HDFS-14934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16961814#comment-16961814
]
Takanobu Asanuma edited comment on HDFS-14934 at 10/29/19 8:44 AM:
---
Sorry I haven't found the fix. If increasing
{{dfs.ha.tail-edits.period.backoff-max}} (like 10s), we can reduce the warn log
frequency.
was (Author: tasanuma0829):
Sorry I haven't find the fix. If increasing
{{dfs.ha.tail-edits.period.backoff-max}} (like 10s), we can reduce the warn log
frequency.
> [SBN Read] Standby NN throws many InterruptedExceptions when
> dfs.ha.tail-edits.period is 0
> --
>
> Key: HDFS-14934
> URL: https://issues.apache.org/jira/browse/HDFS-14934
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Takanobu Asanuma
>Priority: Major
>
> When dfs.ha.tail-edits.period is 0ms (or very short-time), there are many
> warn logs in standby NN.
> {noformat}
> 2019-10-25 16:25:46,945 [Logger channel (from parallel executor) to hostname>/:] WARN concurrent.ExecutorHelper
> (ExecutorHelper.java:logThrowableFromAfterExecute(55)) - Thread
> (Thread[Logger channel (from parallel executor) to / address>:,5,main]) interrupted:
> java.lang.InterruptedException
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:509)
> at
> com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:82)
> at
> org.apache.hadoop.util.concurrent.ExecutorHelper.logThrowableFromAfterExecute(ExecutorHelper.java:48)
> at
> org.apache.hadoop.util.concurrent.HadoopThreadPoolExecutor.afterExecute(HadoopThreadPoolExecutor.java:90)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1157)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14824) [Dynamometer] Dynamometer in org.apache.hadoop.tools does not output the benchmark results.
[ https://issues.apache.org/jira/browse/HDFS-14824?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16961676#comment-16961676 ] Takanobu Asanuma commented on HDFS-14824: - Hi [~xkrogen], seems the feature is important. Does LinkedIn have a plan to merge it to hadoop, or can I work on it? > [Dynamometer] Dynamometer in org.apache.hadoop.tools does not output the > benchmark results. > --- > > Key: HDFS-14824 > URL: https://issues.apache.org/jira/browse/HDFS-14824 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Soya Miyoshi >Priority: Major > > According to the latest > [document|https://aajisaka.github.io/hadoop-document/hadoop-project/hadoop-dynamometer/Dynamometer.html > ], the benchmark results will be written in `Dauditreplay.output-path`. > However, current org.apache.hadooop.tools hasn't merged [this pull > request|https://github.com/linkedin/dynamometer/pull/76 ], so it does not > output the benchmark results. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14907) [Dynamometer] DataNode can't find junit jar when using Hadoop-3 binary
[
https://issues.apache.org/jira/browse/HDFS-14907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16960845#comment-16960845
]
Takanobu Asanuma edited comment on HDFS-14907 at 10/28/19 8:07 AM:
---
Thanks for your advice, [~xkrogen].
start-component.sh looks the good place. I sent a PR. I've confirmed that
dyno-datanodes run successfully with the PR.
was (Author: tasanuma0829):
Thanks for your advice, [~xkrogen].
start-component.sh looks the good place. I sent a PR.
> [Dynamometer] DataNode can't find junit jar when using Hadoop-3 binary
> --
>
> Key: HDFS-14907
> URL: https://issues.apache.org/jira/browse/HDFS-14907
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
>
> When executing {{start-dynamometer-cluster.sh}} with Hadoop-3 binary,
> datanodes fail to run with the following log and
> {{start-dynamometer-cluster.sh}} fails.
> {noformat}
> LogType:stderr
> LogLastModifiedTime:Wed Oct 09 15:03:09 +0900 2019
> LogLength:1386
> LogContents:
> Exception in thread "main" java.lang.NoClassDefFoundError: org/junit/Assert
> at
> org.apache.hadoop.test.GenericTestUtils.assertExists(GenericTestUtils.java:299)
> at
> org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:243)
> at
> org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:252)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.getBaseDirectory(MiniDFSCluster.java:2982)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.determineDfsBaseDir(MiniDFSCluster.java:2972)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.formatDataNodeDirs(MiniDFSCluster.java:2834)
> at
> org.apache.hadoop.tools.dynamometer.SimulatedDataNodes.run(SimulatedDataNodes.java:123)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at
> org.apache.hadoop.tools.dynamometer.SimulatedDataNodes.main(SimulatedDataNodes.java:88)
> Caused by: java.lang.ClassNotFoundException: org.junit.Assert
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 9 more
> ./start-component.sh: line 317: kill: (2261) - No such process
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14907) [Dynamometer] DataNode can't find junit jar when using Hadoop-3 binary
[
https://issues.apache.org/jira/browse/HDFS-14907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takanobu Asanuma updated HDFS-14907:
Status: Patch Available (was: Open)
> [Dynamometer] DataNode can't find junit jar when using Hadoop-3 binary
> --
>
> Key: HDFS-14907
> URL: https://issues.apache.org/jira/browse/HDFS-14907
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
>
> When executing {{start-dynamometer-cluster.sh}} with Hadoop-3 binary,
> datanodes fail to run with the following log and
> {{start-dynamometer-cluster.sh}} fails.
> {noformat}
> LogType:stderr
> LogLastModifiedTime:Wed Oct 09 15:03:09 +0900 2019
> LogLength:1386
> LogContents:
> Exception in thread "main" java.lang.NoClassDefFoundError: org/junit/Assert
> at
> org.apache.hadoop.test.GenericTestUtils.assertExists(GenericTestUtils.java:299)
> at
> org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:243)
> at
> org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:252)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.getBaseDirectory(MiniDFSCluster.java:2982)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.determineDfsBaseDir(MiniDFSCluster.java:2972)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.formatDataNodeDirs(MiniDFSCluster.java:2834)
> at
> org.apache.hadoop.tools.dynamometer.SimulatedDataNodes.run(SimulatedDataNodes.java:123)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at
> org.apache.hadoop.tools.dynamometer.SimulatedDataNodes.main(SimulatedDataNodes.java:88)
> Caused by: java.lang.ClassNotFoundException: org.junit.Assert
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 9 more
> ./start-component.sh: line 317: kill: (2261) - No such process
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14907) [Dynamometer] DataNode can't find junit jar when using Hadoop-3 binary
[
https://issues.apache.org/jira/browse/HDFS-14907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16960845#comment-16960845
]
Takanobu Asanuma commented on HDFS-14907:
-
Thanks for your advice, [~xkrogen].
start-component.sh looks the good place. I sent a PR.
> [Dynamometer] DataNode can't find junit jar when using Hadoop-3 binary
> --
>
> Key: HDFS-14907
> URL: https://issues.apache.org/jira/browse/HDFS-14907
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
>
> When executing {{start-dynamometer-cluster.sh}} with Hadoop-3 binary,
> datanodes fail to run with the following log and
> {{start-dynamometer-cluster.sh}} fails.
> {noformat}
> LogType:stderr
> LogLastModifiedTime:Wed Oct 09 15:03:09 +0900 2019
> LogLength:1386
> LogContents:
> Exception in thread "main" java.lang.NoClassDefFoundError: org/junit/Assert
> at
> org.apache.hadoop.test.GenericTestUtils.assertExists(GenericTestUtils.java:299)
> at
> org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:243)
> at
> org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:252)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.getBaseDirectory(MiniDFSCluster.java:2982)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.determineDfsBaseDir(MiniDFSCluster.java:2972)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.formatDataNodeDirs(MiniDFSCluster.java:2834)
> at
> org.apache.hadoop.tools.dynamometer.SimulatedDataNodes.run(SimulatedDataNodes.java:123)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at
> org.apache.hadoop.tools.dynamometer.SimulatedDataNodes.main(SimulatedDataNodes.java:88)
> Caused by: java.lang.ClassNotFoundException: org.junit.Assert
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 9 more
> ./start-component.sh: line 317: kill: (2261) - No such process
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-14907) [Dynamometer] DataNode can't find junit jar when using Hadoop-3 binary
[
https://issues.apache.org/jira/browse/HDFS-14907?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takanobu Asanuma reassigned HDFS-14907:
---
Assignee: Takanobu Asanuma
> [Dynamometer] DataNode can't find junit jar when using Hadoop-3 binary
> --
>
> Key: HDFS-14907
> URL: https://issues.apache.org/jira/browse/HDFS-14907
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
>
> When executing {{start-dynamometer-cluster.sh}} with Hadoop-3 binary,
> datanodes fail to run with the following log and
> {{start-dynamometer-cluster.sh}} fails.
> {noformat}
> LogType:stderr
> LogLastModifiedTime:Wed Oct 09 15:03:09 +0900 2019
> LogLength:1386
> LogContents:
> Exception in thread "main" java.lang.NoClassDefFoundError: org/junit/Assert
> at
> org.apache.hadoop.test.GenericTestUtils.assertExists(GenericTestUtils.java:299)
> at
> org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:243)
> at
> org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:252)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.getBaseDirectory(MiniDFSCluster.java:2982)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.determineDfsBaseDir(MiniDFSCluster.java:2972)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.formatDataNodeDirs(MiniDFSCluster.java:2834)
> at
> org.apache.hadoop.tools.dynamometer.SimulatedDataNodes.run(SimulatedDataNodes.java:123)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at
> org.apache.hadoop.tools.dynamometer.SimulatedDataNodes.main(SimulatedDataNodes.java:88)
> Caused by: java.lang.ClassNotFoundException: org.junit.Assert
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 9 more
> ./start-component.sh: line 317: kill: (2261) - No such process
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14934) [SBN Read] Standby NN throws many InterruptedExceptions when dfs.ha.tail-edits.period is 0
[
https://issues.apache.org/jira/browse/HDFS-14934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16959563#comment-16959563
]
Takanobu Asanuma commented on HDFS-14934:
-
Thanks for your reply, [~ayushtkn] . We have seen the issue with
trunk(3f89084ac756c9296d412821d76ff2bee57d0c2f) which includes HDFS-14655.
> [SBN Read] Standby NN throws many InterruptedExceptions when
> dfs.ha.tail-edits.period is 0
> --
>
> Key: HDFS-14934
> URL: https://issues.apache.org/jira/browse/HDFS-14934
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Takanobu Asanuma
>Priority: Major
>
> When dfs.ha.tail-edits.period is 0ms (or very short-time), there are many
> warn logs in standby NN.
> {noformat}
> 2019-10-25 16:25:46,945 [Logger channel (from parallel executor) to hostname>/:] WARN concurrent.ExecutorHelper
> (ExecutorHelper.java:logThrowableFromAfterExecute(55)) - Thread
> (Thread[Logger channel (from parallel executor) to / address>:,5,main]) interrupted:
> java.lang.InterruptedException
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:509)
> at
> com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:82)
> at
> org.apache.hadoop.util.concurrent.ExecutorHelper.logThrowableFromAfterExecute(ExecutorHelper.java:48)
> at
> org.apache.hadoop.util.concurrent.HadoopThreadPoolExecutor.afterExecute(HadoopThreadPoolExecutor.java:90)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1157)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14934) [SBN Read] Standby NN throws many InterruptedExceptions when dfs.ha.tail-edits.period is 0
[
https://issues.apache.org/jira/browse/HDFS-14934?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16959530#comment-16959530
]
Takanobu Asanuma commented on HDFS-14934:
-
Seems it is related to HDFS-14655.
> [SBN Read] Standby NN throws many InterruptedExceptions when
> dfs.ha.tail-edits.period is 0
> --
>
> Key: HDFS-14934
> URL: https://issues.apache.org/jira/browse/HDFS-14934
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Takanobu Asanuma
>Priority: Major
>
> When dfs.ha.tail-edits.period is 0ms (or very short-time), there are many
> warn logs in standby NN.
> {noformat}
> 2019-10-25 16:25:46,945 [Logger channel (from parallel executor) to hostname>/:] WARN concurrent.ExecutorHelper
> (ExecutorHelper.java:logThrowableFromAfterExecute(55)) - Thread
> (Thread[Logger channel (from parallel executor) to / address>:,5,main]) interrupted:
> java.lang.InterruptedException
> at
> com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:509)
> at
> com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:82)
> at
> org.apache.hadoop.util.concurrent.ExecutorHelper.logThrowableFromAfterExecute(ExecutorHelper.java:48)
> at
> org.apache.hadoop.util.concurrent.HadoopThreadPoolExecutor.afterExecute(HadoopThreadPoolExecutor.java:90)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1157)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14655) [SBN Read] Namenode crashes if one of The JN is down
[
https://issues.apache.org/jira/browse/HDFS-14655?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16959529#comment-16959529
]
Takanobu Asanuma commented on HDFS-14655:
-
This issue may cause HDFS-14934.
> [SBN Read] Namenode crashes if one of The JN is down
>
>
> Key: HDFS-14655
> URL: https://issues.apache.org/jira/browse/HDFS-14655
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.3.0
>Reporter: Harshakiran Reddy
>Assignee: Ayush Saxena
>Priority: Critical
> Fix For: 2.10.0, 3.3.0, 3.1.4, 3.2.2
>
> Attachments: HDFS-14655-01.patch, HDFS-14655-02.patch,
> HDFS-14655-03.patch, HDFS-14655-04.patch, HDFS-14655-05.patch,
> HDFS-14655-06.patch, HDFS-14655-07.patch, HDFS-14655-08.patch,
> HDFS-14655-branch-2-01.patch, HDFS-14655-branch-2-02.patch,
> HDFS-14655.poc.patch
>
>
> {noformat}
> 2019-07-04 17:35:54,064 | INFO | Logger channel (from parallel executor) to
> XXX/XXX | Retrying connect to server: XXX/XXX. Already tried
> 9 time(s); retry policy is RetryUpToMaximumCountWithFixedSleep(maxRetries=10,
> sleepTime=1000 MILLISECONDS) | Client.java:975
> 2019-07-04 17:35:54,087 | FATAL | Edit log tailer | Unknown error encountered
> while tailing edits. Shutting down standby NN. | EditLogTailer.java:474
> java.lang.OutOfMemoryError: unable to create new native thread
> at java.lang.Thread.start0(Native Method)
> at java.lang.Thread.start(Thread.java:717)
> at
> java.util.concurrent.ThreadPoolExecutor.addWorker(ThreadPoolExecutor.java:957)
> at
> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:1378)
> at
> com.google.common.util.concurrent.MoreExecutors$ListeningDecorator.execute(MoreExecutors.java:440)
> at
> com.google.common.util.concurrent.AbstractListeningExecutorService.submit(AbstractListeningExecutorService.java:56)
> at
> org.apache.hadoop.hdfs.qjournal.client.IPCLoggerChannel.getJournaledEdits(IPCLoggerChannel.java:565)
> at
> org.apache.hadoop.hdfs.qjournal.client.AsyncLoggerSet.getJournaledEdits(AsyncLoggerSet.java:272)
> at
> org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.selectRpcInputStreams(QuorumJournalManager.java:533)
> at
> org.apache.hadoop.hdfs.qjournal.client.QuorumJournalManager.selectInputStreams(QuorumJournalManager.java:508)
> at
> org.apache.hadoop.hdfs.server.namenode.JournalSet.selectInputStreams(JournalSet.java:275)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1681)
> at
> org.apache.hadoop.hdfs.server.namenode.FSEditLog.selectInputStreams(FSEditLog.java:1714)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer.doTailEdits(EditLogTailer.java:307)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.doWork(EditLogTailer.java:460)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.access$300(EditLogTailer.java:410)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread$1.run(EditLogTailer.java:427)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:360)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1709)
> at
> org.apache.hadoop.security.SecurityUtil.doAsLoginUserOrFatal(SecurityUtil.java:483)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.EditLogTailer$EditLogTailerThread.run(EditLogTailer.java:423)
> 2019-07-04 17:35:54,112 | INFO | Edit log tailer | Exiting with status 1:
> java.lang.OutOfMemoryError: unable to create new native thread |
> ExitUtil.java:210
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14934) [SBN Read] Standby NN throws many InterruptedExceptions when dfs.ha.tail-edits.period is 0
Takanobu Asanuma created HDFS-14934:
---
Summary: [SBN Read] Standby NN throws many InterruptedExceptions
when dfs.ha.tail-edits.period is 0
Key: HDFS-14934
URL: https://issues.apache.org/jira/browse/HDFS-14934
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Takanobu Asanuma
When dfs.ha.tail-edits.period is 0ms (or very short-time), there are many warn
logs in standby NN.
{noformat}
2019-10-25 16:25:46,945 [Logger channel (from parallel executor) to /:] WARN concurrent.ExecutorHelper
(ExecutorHelper.java:logThrowableFromAfterExecute(55)) - Thread (Thread[Logger
channel (from parallel executor) to /:,5,main])
interrupted:
java.lang.InterruptedException
at
com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:509)
at
com.google.common.util.concurrent.FluentFuture$TrustedFuture.get(FluentFuture.java:82)
at
org.apache.hadoop.util.concurrent.ExecutorHelper.logThrowableFromAfterExecute(ExecutorHelper.java:48)
at
org.apache.hadoop.util.concurrent.HadoopThreadPoolExecutor.afterExecute(HadoopThreadPoolExecutor.java:90)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1157)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14933) Fixing a typo in documentation of Observer NameNode
[
https://issues.apache.org/jira/browse/HDFS-14933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takanobu Asanuma updated HDFS-14933:
Fix Version/s: 3.3.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed to trunk. Thanks for your contribution, [~risyomei]!
> Fixing a typo in documentation of Observer NameNode
> ---
>
> Key: HDFS-14933
> URL: https://issues.apache.org/jira/browse/HDFS-14933
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Reporter: Xieming Li
>Assignee: Xieming Li
>Priority: Trivial
> Fix For: 3.3.0
>
> Attachments: HDFS-14933.001.patch
>
>
> Fix a typo in documentation Observer NameNode
> https://aajisaka.github.io/hadoop-document/hadoop-project/hadoop-project-dist/hadoop-hdfs/ObserverNameNode.html
> This
> {code}
>
> dfs.ha.tail-edits.period
> 10s
>
> {code}
> should be changed to
> {code}
>
> dfs.ha.tail-edits.period.backoff-max
> 10s
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14933) Fixing a typo in documentation of Observer NameNode
[
https://issues.apache.org/jira/browse/HDFS-14933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takanobu Asanuma updated HDFS-14933:
Summary: Fixing a typo in documentation of Observer NameNode (was: Fixing
a typo in documentaion of Observer NameNode)
> Fixing a typo in documentation of Observer NameNode
> ---
>
> Key: HDFS-14933
> URL: https://issues.apache.org/jira/browse/HDFS-14933
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Reporter: Xieming Li
>Assignee: Xieming Li
>Priority: Trivial
> Attachments: HDFS-14933.001.patch
>
>
> Fix a typo in documentation Observer NameNode
> https://aajisaka.github.io/hadoop-document/hadoop-project/hadoop-project-dist/hadoop-hdfs/ObserverNameNode.html
> This
> {code}
>
> dfs.ha.tail-edits.period
> 10s
>
> {code}
> should be changed to
> {code}
>
> dfs.ha.tail-edits.period.backoff-max
> 10s
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14933) Fixing a typo in documentaion of Observer NameNode
[
https://issues.apache.org/jira/browse/HDFS-14933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16959461#comment-16959461
]
Takanobu Asanuma commented on HDFS-14933:
-
+1 on [^HDFS-14933.001.patch].
> Fixing a typo in documentaion of Observer NameNode
> --
>
> Key: HDFS-14933
> URL: https://issues.apache.org/jira/browse/HDFS-14933
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation
>Reporter: Xieming Li
>Assignee: Xieming Li
>Priority: Trivial
> Attachments: HDFS-14933.001.patch
>
>
> Fix a typo in documentation Observer NameNode
> https://aajisaka.github.io/hadoop-document/hadoop-project/hadoop-project-dist/hadoop-hdfs/ObserverNameNode.html
> This
> {code}
>
> dfs.ha.tail-edits.period
> 10s
>
> {code}
> should be changed to
> {code}
>
> dfs.ha.tail-edits.period.backoff-max
> 10s
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14902) RBF: NullPointer When Misconfigured
[
https://issues.apache.org/jira/browse/HDFS-14902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takanobu Asanuma updated HDFS-14902:
Parent: HDFS-14603
Issue Type: Sub-task (was: Improvement)
> RBF: NullPointer When Misconfigured
> ---
>
> Key: HDFS-14902
> URL: https://issues.apache.org/jira/browse/HDFS-14902
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: Takanobu Asanuma
>Priority: Minor
> Attachments: HDFS-14902.001.patch
>
>
> Admittedly the server was mis-configured, but this should be a bit more
> elegant.
> {code:none}
> 2019-10-08 11:19:52,505 ERROR router.NamenodeHeartbeatService: Unhandled
> exception updating NN registration for null:null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$NamenodeMembershipRecordProto$Builder.setServiceAddress(HdfsServerFederationProtos.java:3831)
> at
> org.apache.hadoop.hdfs.server.federation.store.records.impl.pb.MembershipStatePBImpl.setServiceAddress(MembershipStatePBImpl.java:119)
> at
> org.apache.hadoop.hdfs.server.federation.store.records.MembershipState.newInstance(MembershipState.java:108)
> at
> org.apache.hadoop.hdfs.server.federation.resolver.MembershipNamenodeResolver.registerNamenode(MembershipNamenodeResolver.java:259)
> at
> org.apache.hadoop.hdfs.server.federation.router.NamenodeHeartbeatService.updateState(NamenodeHeartbeatService.java:223)
> at
> org.apache.hadoop.hdfs.server.federation.router.NamenodeHeartbeatService.periodicInvoke(NamenodeHeartbeatService.java:159)
> at
> org.apache.hadoop.hdfs.server.federation.router.PeriodicService$1.run(PeriodicService.java:178)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14917) Change the ICON of "Decommissioned & dead" datanode on "dfshealth.html"
[ https://issues.apache.org/jira/browse/HDFS-14917?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HDFS-14917: Fix Version/s: 3.3.0 Resolution: Fixed Status: Resolved (was: Patch Available) Committed to trunk. Thanks for your contribution, [~risyomei], and thanks for your review, [~elgoiri]. > Change the ICON of "Decommissioned & dead" datanode on "dfshealth.html" > --- > > Key: HDFS-14917 > URL: https://issues.apache.org/jira/browse/HDFS-14917 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ui >Reporter: Xieming Li >Assignee: Xieming Li >Priority: Trivial > Fix For: 3.3.0 > > Attachments: HDFS-14917.patch, image-2019-10-21-17-49-10-635.png, > image-2019-10-21-17-49-58-759.png, image-2019-10-21-18-03-53-914.png, > image-2019-10-21-18-04-52-405.png, image-2019-10-21-18-05-19-160.png, > image-2019-10-21-18-13-01-884.png, image-2019-10-21-18-13-54-427.png > > > This is a really simple UI change proposal: > The icon of "Decommissioned & dead" datanode could be improved. It can be > changed from !image-2019-10-21-18-05-19-160.png|width=31,height=28! to > !image-2019-10-21-18-04-52-405.png|width=32,height=29! so that, > # icon " !image-2019-10-21-18-13-01-884.png|width=26,height=25! " can be > used for all status starts with "decommission" on dfshealth.html, > # icon " !image-2019-10-21-18-13-01-884.png|width=26,height=25! " can be > differentiated with icon " !image-2019-10-21-18-13-54-427.png! " on > federationhealth.html > |*DataNode Infomation Legend (now)* > dfshealth.html#tab-datanode > |!image-2019-10-21-17-49-10-635.png|width=516,height=55!| > |*DataNode* *Infomation* *Legend (proposed)* > dfshealth.html#tab-datanode > |!image-2019-10-21-18-03-53-914.png|width=589,height=60!| > |*NameService Legend* > > federationhealth.htm#tab-namenode|!image-2019-10-21-17-49-58-759.png|width=445,height=43!| -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14638) [Dynamometer] Fix scripts to refer to current build structure
[
https://issues.apache.org/jira/browse/HDFS-14638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takanobu Asanuma updated HDFS-14638:
Fix Version/s: 3.3.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
> [Dynamometer] Fix scripts to refer to current build structure
> -
>
> Key: HDFS-14638
> URL: https://issues.apache.org/jira/browse/HDFS-14638
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode, test
>Reporter: Erik Krogen
>Assignee: Takanobu Asanuma
>Priority: Major
> Fix For: 3.3.0
>
>
> The scripts within the Dynamometer build dirs all refer to the old
> distribution structure with a single {{bin}} directory and a single {{lib}}
> directory. We need to update them to refer to the Hadoop-standard layout.
> Also as pointed out by [~pingsutw]:
> {quote}
> Due to dynamometer rename to hadoop-dynamometer in hadoop-tools
> but we still use old name of jar inside the scripts
> {code}
> "$hadoop_cmd" jar "${script_pwd}"/lib/dynamometer-infra-*.jar
> org.apache.hadoop.tools.dynamometer.Client "$@"
> {code}
> We should rename these jar inside the scripts
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14930) [Dynamometer] EC files become missing block groups
Takanobu Asanuma created HDFS-14930:
---
Summary: [Dynamometer] EC files become missing block groups
Key: HDFS-14930
URL: https://issues.apache.org/jira/browse/HDFS-14930
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Takanobu Asanuma
If there are ec files in fsimage and simulating it with Dynamometer, all of
them become missing block groups.
{noformat}
$ sudo -u hdfs hdfs fsck /tmp/ec_test_data
...
Erasure Coded Block Groups:
Total size:52753424 B
Total files: 140
Total block groups (validated):140 (avg. block group size 376810 B)
UNRECOVERABLE BLOCK GROUPS: 140 (100.0 %)
CORRUPT FILES:140
MISSING BLOCK GROUPS: 140
MISSING SIZE: 52753424 B
Minimally erasure-coded block groups: 0 (0.0 %)
Over-erasure-coded block groups: 0 (0.0 %)
Under-erasure-coded block groups: 0 (0.0 %)
Unsatisfactory placement block groups: 0 (0.0 %)
Average block group size: 0.0
Missing block groups: 140
Corrupt block groups: 0
Missing internal blocks: 0
Blocks queued for replication: 0
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14917) Change the ICON of "Decommissioned & dead" datanode on "dfshealth.html"
[ https://issues.apache.org/jira/browse/HDFS-14917?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16958556#comment-16958556 ] Takanobu Asanuma commented on HDFS-14917: - Using ban-circle icon for all decommissioned statuses makes sense. And I agreed that creating the new jira for overall design. Can we proceed this jira separately? if so, +1 on HDFS-14917.patch. > Change the ICON of "Decommissioned & dead" datanode on "dfshealth.html" > --- > > Key: HDFS-14917 > URL: https://issues.apache.org/jira/browse/HDFS-14917 > Project: Hadoop HDFS > Issue Type: Improvement > Components: ui >Reporter: Xieming Li >Assignee: Xieming Li >Priority: Trivial > Attachments: HDFS-14917.patch, image-2019-10-21-17-49-10-635.png, > image-2019-10-21-17-49-58-759.png, image-2019-10-21-18-03-53-914.png, > image-2019-10-21-18-04-52-405.png, image-2019-10-21-18-05-19-160.png, > image-2019-10-21-18-13-01-884.png, image-2019-10-21-18-13-54-427.png > > > This is a really simple UI change proposal: > The icon of "Decommissioned & dead" datanode could be improved. It can be > changed from !image-2019-10-21-18-05-19-160.png|width=31,height=28! to > !image-2019-10-21-18-04-52-405.png|width=32,height=29! so that, > # icon " !image-2019-10-21-18-13-01-884.png|width=26,height=25! " can be > used for all status starts with "decommission" on dfshealth.html, > # icon " !image-2019-10-21-18-13-01-884.png|width=26,height=25! " can be > differentiated with icon " !image-2019-10-21-18-13-54-427.png! " on > federationhealth.html > |*DataNode Infomation Legend (now)* > dfshealth.html#tab-datanode > |!image-2019-10-21-17-49-10-635.png|width=516,height=55!| > |*DataNode* *Infomation* *Legend (proposed)* > dfshealth.html#tab-datanode > |!image-2019-10-21-18-03-53-914.png|width=589,height=60!| > |*NameService Legend* > > federationhealth.htm#tab-namenode|!image-2019-10-21-17-49-58-759.png|width=445,height=43!| -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14638) [Dynamometer] Fix scripts to refer to current build structure
[
https://issues.apache.org/jira/browse/HDFS-14638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takanobu Asanuma updated HDFS-14638:
Status: Patch Available (was: Open)
Created PR.
> [Dynamometer] Fix scripts to refer to current build structure
> -
>
> Key: HDFS-14638
> URL: https://issues.apache.org/jira/browse/HDFS-14638
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode, test
>Reporter: Erik Krogen
>Assignee: Takanobu Asanuma
>Priority: Major
>
> The scripts within the Dynamometer build dirs all refer to the old
> distribution structure with a single {{bin}} directory and a single {{lib}}
> directory. We need to update them to refer to the Hadoop-standard layout.
> Also as pointed out by [~pingsutw]:
> {quote}
> Due to dynamometer rename to hadoop-dynamometer in hadoop-tools
> but we still use old name of jar inside the scripts
> {code}
> "$hadoop_cmd" jar "${script_pwd}"/lib/dynamometer-infra-*.jar
> org.apache.hadoop.tools.dynamometer.Client "$@"
> {code}
> We should rename these jar inside the scripts
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14907) [Dynamometer] DataNode can't find junit jar when using Hadoop-3 binary
[
https://issues.apache.org/jira/browse/HDFS-14907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16958459#comment-16958459
]
Takanobu Asanuma commented on HDFS-14907:
-
Thanks for your comment and sorry for late response, [~xkrogen]
We also need to be careful of the junit path since it is different each hadoop
version.
{noformat}
$ find hadoop-2.9.2 -name "junit*.jar"
hadoop-2.9.2/share/hadoop/common/lib/junit-4.11.jar
hadoop-2.9.2/share/hadoop/mapreduce/lib/junit-4.11.jar
$ find hadoop-3.0.0 -name "junit*.jar"
hadoop-3.0.0/share/hadoop/common/lib/junit-4.11.jar
$ find hadoop-3.1.3 -name "junit*.jar"
hadoop-3.1.3/share/hadoop/mapreduce/lib/junit-4.11.jar
$ find hadoop-3.2.1 -name "junit*.jar"
hadoop-3.2.1/share/hadoop/mapreduce/lib/junit-4.11.jar
$ find hadoop-3.3.0-SNAPSHOT -name "junit*.jar"
hadoop-3.3.0-SNAPSHOT/share/hadoop/tools/lib/junit-4.12.jar
{noformat}
As I commented, we can avoid this issue by adding the jar to classpath in
hadoop-env.sh in the conf directory which is pass to the argument of
start-dynamometer-cluster.sh. If there is not other way, should we update
dynamometer document?
> [Dynamometer] DataNode can't find junit jar when using Hadoop-3 binary
> --
>
> Key: HDFS-14907
> URL: https://issues.apache.org/jira/browse/HDFS-14907
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Takanobu Asanuma
>Priority: Major
>
> When executing {{start-dynamometer-cluster.sh}} with Hadoop-3 binary,
> datanodes fail to run with the following log and
> {{start-dynamometer-cluster.sh}} fails.
> {noformat}
> LogType:stderr
> LogLastModifiedTime:Wed Oct 09 15:03:09 +0900 2019
> LogLength:1386
> LogContents:
> Exception in thread "main" java.lang.NoClassDefFoundError: org/junit/Assert
> at
> org.apache.hadoop.test.GenericTestUtils.assertExists(GenericTestUtils.java:299)
> at
> org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:243)
> at
> org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:252)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.getBaseDirectory(MiniDFSCluster.java:2982)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.determineDfsBaseDir(MiniDFSCluster.java:2972)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.formatDataNodeDirs(MiniDFSCluster.java:2834)
> at
> org.apache.hadoop.tools.dynamometer.SimulatedDataNodes.run(SimulatedDataNodes.java:123)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at
> org.apache.hadoop.tools.dynamometer.SimulatedDataNodes.main(SimulatedDataNodes.java:88)
> Caused by: java.lang.ClassNotFoundException: org.junit.Assert
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 9 more
> ./start-component.sh: line 317: kill: (2261) - No such process
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14916) RBF: line breaks are missing from the output of "hdfs dfsrouteradmin -ls"
[
https://issues.apache.org/jira/browse/HDFS-14916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955837#comment-16955837
]
Takanobu Asanuma commented on HDFS-14916:
-
+1 on [^HDFS-14916.001.patch], pending Jenkins.
> RBF: line breaks are missing from the output of "hdfs dfsrouteradmin -ls"
> -
>
> Key: HDFS-14916
> URL: https://issues.apache.org/jira/browse/HDFS-14916
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf, ui
>Reporter: Xieming Li
>Assignee: Xieming Li
>Priority: Minor
> Attachments: HDFS-14916.001.patch, HDFS-14916.patch
>
>
> line breaks seem to be missing from the output of "hdfs dfsrouteradmin -ls"
> e.g.:
> The output of "hdfs dfsrouteradmin -ls" now:
> {code:java}
> [sri@nn00070 ~]$ hdfs dfsrouteradmin -ls
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /testDir1 subCluster1->/user/user1 hdfs
> hdfs rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] /user2
> subCluster1->/tmp/user2user2 users
>rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] /user/user3
> subCluster1->/user/user3 user3 hadoop
> rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] [sri@nn00070 ~]$
> {code}
> This should be:
> {code:java}
> [sri@nn00070 ~]$ hdfs dfsrouteradmin -ls -d
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /testDir1subCluster1->/user/user1 hdfs
> hdfs rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
> /user2subCluster1->/tmp/user2user2
> users rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
> /user/user3 subCluster1->/user/user3 user3
> hadoop rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
> [sri@nn00070 ~]$
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14916) RBF: line breaks are missing from the output of "hdfs dfsrouteradmin -ls"
[
https://issues.apache.org/jira/browse/HDFS-14916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955824#comment-16955824
]
Takanobu Asanuma commented on HDFS-14916:
-
[~risyomei] Thanks for catching the issue.
IntelliJ warns the empty is unnecessary. We can use {{System.out.println()}}.
The Other looks good to me.
> RBF: line breaks are missing from the output of "hdfs dfsrouteradmin -ls"
> -
>
> Key: HDFS-14916
> URL: https://issues.apache.org/jira/browse/HDFS-14916
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf, ui
>Reporter: Xieming Li
>Assignee: Xieming Li
>Priority: Minor
> Attachments: HDFS-14916.patch
>
>
> line breaks seem to be missing from the output of "hdfs dfsrouteradmin -ls"
> e.g.:
> The output of "hdfs dfsrouteradmin -ls" now:
> {code:java}
> [sri@nn00070 ~]$ hdfs dfsrouteradmin -ls
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /testDir1 subCluster1->/user/user1 hdfs
> hdfs rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] /user2
> subCluster1->/tmp/user2user2 users
>rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] /user/user3
> subCluster1->/user/user3 user3 hadoop
> rwxr-xr-x [NsQuota: -/-, SsQuota: -/-] [sri@nn00070 ~]$
> {code}
> This should be:
> {code:java}
> [sri@nn00070 ~]$ hdfs dfsrouteradmin -ls -d
> Mount Table Entries:
> SourceDestinations Owner
> Group Mode Quota/Usage
> /testDir1subCluster1->/user/user1 hdfs
> hdfs rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
> /user2subCluster1->/tmp/user2user2
> users rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
> /user/user3 subCluster1->/user/user3 user3
> hadoop rwxr-xr-x [NsQuota: -/-, SsQuota: -/-]
> [sri@nn00070 ~]$
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14887) RBF: In Router Web UI, Observer Namenode Information displaying as Unavailable
[ https://issues.apache.org/jira/browse/HDFS-14887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16955147#comment-16955147 ] Takanobu Asanuma commented on HDFS-14887: - +1 on [^HDFS-14887.009.patch]. [~elgoiri] Could you review it once again? > RBF: In Router Web UI, Observer Namenode Information displaying as Unavailable > -- > > Key: HDFS-14887 > URL: https://issues.apache.org/jira/browse/HDFS-14887 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > Attachments: 14887.008.png, 14887.after.png, 14887.before.png, > HDFS-14887.001.patch, HDFS-14887.002.patch, HDFS-14887.003.patch, > HDFS-14887.004.patch, HDFS-14887.005.patch, HDFS-14887.006.patch, > HDFS-14887.007.patch, HDFS-14887.008.patch, HDFS-14887.009.patch > > > In Router Web UI, Observer Namenode Information displaying as Unavailable. > We should show a proper icon for them. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14887) RBF: In Router Web UI, Observer Namenode Information displaying as Unavailable
[
https://issues.apache.org/jira/browse/HDFS-14887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16953909#comment-16953909
]
Takanobu Asanuma commented on HDFS-14887:
-
Thanks for the discussion and updating the patch, [~hemanthboyina] and
[~elgoiri].
I tested with 008 patch and found that the observer icon is used in the
Nameservice Information when there are ActiveNN and ObserverNN in the same
nameservice. Please see [^14887.008.png].
* This is because the enum order in {{FederationNamenodeServiceState}} is used
for the comparator. I think the order should be ACTIVE > OBSERVER > STANDBY.
* I want to use the same order(active>observer>stadby) in
{{FederationNamenodeServiceState#getState()}} and {{augment_namenodes()}} in
federationhealth.js.
* We may also need to add the observer icon to
federationhealth-namenode-legend in Nameservice Information.
> RBF: In Router Web UI, Observer Namenode Information displaying as Unavailable
> --
>
> Key: HDFS-14887
> URL: https://issues.apache.org/jira/browse/HDFS-14887
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: hemanthboyina
>Assignee: hemanthboyina
>Priority: Major
> Attachments: 14887.008.png, 14887.after.png, 14887.before.png,
> HDFS-14887.001.patch, HDFS-14887.002.patch, HDFS-14887.003.patch,
> HDFS-14887.004.patch, HDFS-14887.005.patch, HDFS-14887.006.patch,
> HDFS-14887.007.patch, HDFS-14887.008.patch
>
>
> In Router Web UI, Observer Namenode Information displaying as Unavailable.
> We should show a proper icon for them.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14887) RBF: In Router Web UI, Observer Namenode Information displaying as Unavailable
[ https://issues.apache.org/jira/browse/HDFS-14887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HDFS-14887: Attachment: 14887.008.png > RBF: In Router Web UI, Observer Namenode Information displaying as Unavailable > -- > > Key: HDFS-14887 > URL: https://issues.apache.org/jira/browse/HDFS-14887 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > Attachments: 14887.008.png, 14887.after.png, 14887.before.png, > HDFS-14887.001.patch, HDFS-14887.002.patch, HDFS-14887.003.patch, > HDFS-14887.004.patch, HDFS-14887.005.patch, HDFS-14887.006.patch, > HDFS-14887.007.patch, HDFS-14887.008.patch > > > In Router Web UI, Observer Namenode Information displaying as Unavailable. > We should show a proper icon for them. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14902) RBF: NullPointer When Misconfigured
[
https://issues.apache.org/jira/browse/HDFS-14902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16951765#comment-16951765
]
Takanobu Asanuma commented on HDFS-14902:
-
Uploaded the 001 patch.
> RBF: NullPointer When Misconfigured
> ---
>
> Key: HDFS-14902
> URL: https://issues.apache.org/jira/browse/HDFS-14902
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: Takanobu Asanuma
>Priority: Minor
> Attachments: HDFS-14902.001.patch
>
>
> Admittedly the server was mis-configured, but this should be a bit more
> elegant.
> {code:none}
> 2019-10-08 11:19:52,505 ERROR router.NamenodeHeartbeatService: Unhandled
> exception updating NN registration for null:null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$NamenodeMembershipRecordProto$Builder.setServiceAddress(HdfsServerFederationProtos.java:3831)
> at
> org.apache.hadoop.hdfs.server.federation.store.records.impl.pb.MembershipStatePBImpl.setServiceAddress(MembershipStatePBImpl.java:119)
> at
> org.apache.hadoop.hdfs.server.federation.store.records.MembershipState.newInstance(MembershipState.java:108)
> at
> org.apache.hadoop.hdfs.server.federation.resolver.MembershipNamenodeResolver.registerNamenode(MembershipNamenodeResolver.java:259)
> at
> org.apache.hadoop.hdfs.server.federation.router.NamenodeHeartbeatService.updateState(NamenodeHeartbeatService.java:223)
> at
> org.apache.hadoop.hdfs.server.federation.router.NamenodeHeartbeatService.periodicInvoke(NamenodeHeartbeatService.java:159)
> at
> org.apache.hadoop.hdfs.server.federation.router.PeriodicService$1.run(PeriodicService.java:178)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14902) RBF: NullPointer When Misconfigured
[
https://issues.apache.org/jira/browse/HDFS-14902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takanobu Asanuma updated HDFS-14902:
Status: Patch Available (was: Open)
> RBF: NullPointer When Misconfigured
> ---
>
> Key: HDFS-14902
> URL: https://issues.apache.org/jira/browse/HDFS-14902
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: Takanobu Asanuma
>Priority: Minor
> Attachments: HDFS-14902.001.patch
>
>
> Admittedly the server was mis-configured, but this should be a bit more
> elegant.
> {code:none}
> 2019-10-08 11:19:52,505 ERROR router.NamenodeHeartbeatService: Unhandled
> exception updating NN registration for null:null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$NamenodeMembershipRecordProto$Builder.setServiceAddress(HdfsServerFederationProtos.java:3831)
> at
> org.apache.hadoop.hdfs.server.federation.store.records.impl.pb.MembershipStatePBImpl.setServiceAddress(MembershipStatePBImpl.java:119)
> at
> org.apache.hadoop.hdfs.server.federation.store.records.MembershipState.newInstance(MembershipState.java:108)
> at
> org.apache.hadoop.hdfs.server.federation.resolver.MembershipNamenodeResolver.registerNamenode(MembershipNamenodeResolver.java:259)
> at
> org.apache.hadoop.hdfs.server.federation.router.NamenodeHeartbeatService.updateState(NamenodeHeartbeatService.java:223)
> at
> org.apache.hadoop.hdfs.server.federation.router.NamenodeHeartbeatService.periodicInvoke(NamenodeHeartbeatService.java:159)
> at
> org.apache.hadoop.hdfs.server.federation.router.PeriodicService$1.run(PeriodicService.java:178)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14902) RBF: NullPointer When Misconfigured
[
https://issues.apache.org/jira/browse/HDFS-14902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takanobu Asanuma updated HDFS-14902:
Attachment: HDFS-14902.001.patch
> RBF: NullPointer When Misconfigured
> ---
>
> Key: HDFS-14902
> URL: https://issues.apache.org/jira/browse/HDFS-14902
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: Takanobu Asanuma
>Priority: Minor
> Attachments: HDFS-14902.001.patch
>
>
> Admittedly the server was mis-configured, but this should be a bit more
> elegant.
> {code:none}
> 2019-10-08 11:19:52,505 ERROR router.NamenodeHeartbeatService: Unhandled
> exception updating NN registration for null:null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$NamenodeMembershipRecordProto$Builder.setServiceAddress(HdfsServerFederationProtos.java:3831)
> at
> org.apache.hadoop.hdfs.server.federation.store.records.impl.pb.MembershipStatePBImpl.setServiceAddress(MembershipStatePBImpl.java:119)
> at
> org.apache.hadoop.hdfs.server.federation.store.records.MembershipState.newInstance(MembershipState.java:108)
> at
> org.apache.hadoop.hdfs.server.federation.resolver.MembershipNamenodeResolver.registerNamenode(MembershipNamenodeResolver.java:259)
> at
> org.apache.hadoop.hdfs.server.federation.router.NamenodeHeartbeatService.updateState(NamenodeHeartbeatService.java:223)
> at
> org.apache.hadoop.hdfs.server.federation.router.NamenodeHeartbeatService.periodicInvoke(NamenodeHeartbeatService.java:159)
> at
> org.apache.hadoop.hdfs.server.federation.router.PeriodicService$1.run(PeriodicService.java:178)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14902) RBF: NullPointer When Misconfigured
[
https://issues.apache.org/jira/browse/HDFS-14902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16951740#comment-16951740
]
Takanobu Asanuma commented on HDFS-14902:
-
[~ayushtkn] Thanks for letting me know the jira.
I read it and I think it is a little different. This jira focuses on cleaning
NamenodeHeartbeatService logging.
Hi [~xuzq_zander], I would appreciate if you could watch this jira.
> RBF: NullPointer When Misconfigured
> ---
>
> Key: HDFS-14902
> URL: https://issues.apache.org/jira/browse/HDFS-14902
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: Takanobu Asanuma
>Priority: Minor
>
> Admittedly the server was mis-configured, but this should be a bit more
> elegant.
> {code:none}
> 2019-10-08 11:19:52,505 ERROR router.NamenodeHeartbeatService: Unhandled
> exception updating NN registration for null:null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$NamenodeMembershipRecordProto$Builder.setServiceAddress(HdfsServerFederationProtos.java:3831)
> at
> org.apache.hadoop.hdfs.server.federation.store.records.impl.pb.MembershipStatePBImpl.setServiceAddress(MembershipStatePBImpl.java:119)
> at
> org.apache.hadoop.hdfs.server.federation.store.records.MembershipState.newInstance(MembershipState.java:108)
> at
> org.apache.hadoop.hdfs.server.federation.resolver.MembershipNamenodeResolver.registerNamenode(MembershipNamenodeResolver.java:259)
> at
> org.apache.hadoop.hdfs.server.federation.router.NamenodeHeartbeatService.updateState(NamenodeHeartbeatService.java:223)
> at
> org.apache.hadoop.hdfs.server.federation.router.NamenodeHeartbeatService.periodicInvoke(NamenodeHeartbeatService.java:159)
> at
> org.apache.hadoop.hdfs.server.federation.router.PeriodicService$1.run(PeriodicService.java:178)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14638) [Dynamometer] Fix scripts to refer to current build structure
[
https://issues.apache.org/jira/browse/HDFS-14638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16951593#comment-16951593
]
Takanobu Asanuma commented on HDFS-14638:
-
Thanks for filing it. I'd like to work on this jira.
> [Dynamometer] Fix scripts to refer to current build structure
> -
>
> Key: HDFS-14638
> URL: https://issues.apache.org/jira/browse/HDFS-14638
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode, test
>Reporter: Erik Krogen
>Priority: Major
>
> The scripts within the Dynamometer build dirs all refer to the old
> distribution structure with a single {{bin}} directory and a single {{lib}}
> directory. We need to update them to refer to the Hadoop-standard layout.
> Also as pointed out by [~pingsutw]:
> {quote}
> Due to dynamometer rename to hadoop-dynamometer in hadoop-tools
> but we still use old name of jar inside the scripts
> {code}
> "$hadoop_cmd" jar "${script_pwd}"/lib/dynamometer-infra-*.jar
> org.apache.hadoop.tools.dynamometer.Client "$@"
> {code}
> We should rename these jar inside the scripts
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-14638) [Dynamometer] Fix scripts to refer to current build structure
[
https://issues.apache.org/jira/browse/HDFS-14638?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takanobu Asanuma reassigned HDFS-14638:
---
Assignee: Takanobu Asanuma
> [Dynamometer] Fix scripts to refer to current build structure
> -
>
> Key: HDFS-14638
> URL: https://issues.apache.org/jira/browse/HDFS-14638
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode, test
>Reporter: Erik Krogen
>Assignee: Takanobu Asanuma
>Priority: Major
>
> The scripts within the Dynamometer build dirs all refer to the old
> distribution structure with a single {{bin}} directory and a single {{lib}}
> directory. We need to update them to refer to the Hadoop-standard layout.
> Also as pointed out by [~pingsutw]:
> {quote}
> Due to dynamometer rename to hadoop-dynamometer in hadoop-tools
> but we still use old name of jar inside the scripts
> {code}
> "$hadoop_cmd" jar "${script_pwd}"/lib/dynamometer-infra-*.jar
> org.apache.hadoop.tools.dynamometer.Client "$@"
> {code}
> We should rename these jar inside the scripts
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14907) [Dynamometer] DataNode can't find junit jar when using Hadoop-3 binary
[
https://issues.apache.org/jira/browse/HDFS-14907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16951584#comment-16951584
]
Takanobu Asanuma commented on HDFS-14907:
-
[~aajisaka] Thanks for your comments, I tried it but it didn't solve the
problem. It is no effect on datanode which is started by
start-dynamometer-cluster.sh.
The wrong jar name is issued by HDFS-14638.
> [Dynamometer] DataNode can't find junit jar when using Hadoop-3 binary
> --
>
> Key: HDFS-14907
> URL: https://issues.apache.org/jira/browse/HDFS-14907
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Takanobu Asanuma
>Priority: Major
>
> When executing {{start-dynamometer-cluster.sh}} with Hadoop-3 binary,
> datanodes fail to run with the following log and
> {{start-dynamometer-cluster.sh}} fails.
> {noformat}
> LogType:stderr
> LogLastModifiedTime:Wed Oct 09 15:03:09 +0900 2019
> LogLength:1386
> LogContents:
> Exception in thread "main" java.lang.NoClassDefFoundError: org/junit/Assert
> at
> org.apache.hadoop.test.GenericTestUtils.assertExists(GenericTestUtils.java:299)
> at
> org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:243)
> at
> org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:252)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.getBaseDirectory(MiniDFSCluster.java:2982)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.determineDfsBaseDir(MiniDFSCluster.java:2972)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.formatDataNodeDirs(MiniDFSCluster.java:2834)
> at
> org.apache.hadoop.tools.dynamometer.SimulatedDataNodes.run(SimulatedDataNodes.java:123)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at
> org.apache.hadoop.tools.dynamometer.SimulatedDataNodes.main(SimulatedDataNodes.java:88)
> Caused by: java.lang.ClassNotFoundException: org.junit.Assert
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 9 more
> ./start-component.sh: line 317: kill: (2261) - No such process
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14907) [Dynamometer] DataNode can't find junit jar when using Hadoop-3 binary
[
https://issues.apache.org/jira/browse/HDFS-14907?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16951459#comment-16951459
]
Takanobu Asanuma commented on HDFS-14907:
-
There is a workaround by adding the jar to classpath in hadoop-env.sh.
{code:bash}
export
HADOOP_CLASSPATH="${HADOOP_CLASSPATH}:${HADOOP_TOOLS_HOME}/${HADOOP_TOOLS_LIB_JARS_DIR}/junit-4.12.jar"
{code}
> [Dynamometer] DataNode can't find junit jar when using Hadoop-3 binary
> --
>
> Key: HDFS-14907
> URL: https://issues.apache.org/jira/browse/HDFS-14907
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Takanobu Asanuma
>Priority: Major
>
> When executing {{start-dynamometer-cluster.sh}} with Hadoop-3 binary,
> datanodes fail to run with the following log and
> {{start-dynamometer-cluster.sh}} fails.
> {noformat}
> LogType:stderr
> LogLastModifiedTime:Wed Oct 09 15:03:09 +0900 2019
> LogLength:1386
> LogContents:
> Exception in thread "main" java.lang.NoClassDefFoundError: org/junit/Assert
> at
> org.apache.hadoop.test.GenericTestUtils.assertExists(GenericTestUtils.java:299)
> at
> org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:243)
> at
> org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:252)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.getBaseDirectory(MiniDFSCluster.java:2982)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.determineDfsBaseDir(MiniDFSCluster.java:2972)
> at
> org.apache.hadoop.hdfs.MiniDFSCluster.formatDataNodeDirs(MiniDFSCluster.java:2834)
> at
> org.apache.hadoop.tools.dynamometer.SimulatedDataNodes.run(SimulatedDataNodes.java:123)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at
> org.apache.hadoop.tools.dynamometer.SimulatedDataNodes.main(SimulatedDataNodes.java:88)
> Caused by: java.lang.ClassNotFoundException: org.junit.Assert
> at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
> at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
> ... 9 more
> ./start-component.sh: line 317: kill: (2261) - No such process
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14907) [Dynamometer] DataNode can't find junit jar when using Hadoop-3 binary
Takanobu Asanuma created HDFS-14907:
---
Summary: [Dynamometer] DataNode can't find junit jar when using
Hadoop-3 binary
Key: HDFS-14907
URL: https://issues.apache.org/jira/browse/HDFS-14907
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Takanobu Asanuma
When executing {{start-dynamometer-cluster.sh}} with Hadoop-3 binary, datanodes
fail to run with the following log and {{start-dynamometer-cluster.sh}} fails.
{noformat}
LogType:stderr
LogLastModifiedTime:Wed Oct 09 15:03:09 +0900 2019
LogLength:1386
LogContents:
Exception in thread "main" java.lang.NoClassDefFoundError: org/junit/Assert
at
org.apache.hadoop.test.GenericTestUtils.assertExists(GenericTestUtils.java:299)
at
org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:243)
at
org.apache.hadoop.test.GenericTestUtils.getTestDir(GenericTestUtils.java:252)
at
org.apache.hadoop.hdfs.MiniDFSCluster.getBaseDirectory(MiniDFSCluster.java:2982)
at
org.apache.hadoop.hdfs.MiniDFSCluster.determineDfsBaseDir(MiniDFSCluster.java:2972)
at
org.apache.hadoop.hdfs.MiniDFSCluster.formatDataNodeDirs(MiniDFSCluster.java:2834)
at
org.apache.hadoop.tools.dynamometer.SimulatedDataNodes.run(SimulatedDataNodes.java:123)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at
org.apache.hadoop.tools.dynamometer.SimulatedDataNodes.main(SimulatedDataNodes.java:88)
Caused by: java.lang.ClassNotFoundException: org.junit.Assert
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:349)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 9 more
./start-component.sh: line 317: kill: (2261) - No such process
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-14902) RBF: NullPointer When Misconfigured
[
https://issues.apache.org/jira/browse/HDFS-14902?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takanobu Asanuma reassigned HDFS-14902:
---
Assignee: Takanobu Asanuma
> RBF: NullPointer When Misconfigured
> ---
>
> Key: HDFS-14902
> URL: https://issues.apache.org/jira/browse/HDFS-14902
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Assignee: Takanobu Asanuma
>Priority: Minor
>
> Admittedly the server was mis-configured, but this should be a bit more
> elegant.
> {code:none}
> 2019-10-08 11:19:52,505 ERROR router.NamenodeHeartbeatService: Unhandled
> exception updating NN registration for null:null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$NamenodeMembershipRecordProto$Builder.setServiceAddress(HdfsServerFederationProtos.java:3831)
> at
> org.apache.hadoop.hdfs.server.federation.store.records.impl.pb.MembershipStatePBImpl.setServiceAddress(MembershipStatePBImpl.java:119)
> at
> org.apache.hadoop.hdfs.server.federation.store.records.MembershipState.newInstance(MembershipState.java:108)
> at
> org.apache.hadoop.hdfs.server.federation.resolver.MembershipNamenodeResolver.registerNamenode(MembershipNamenodeResolver.java:259)
> at
> org.apache.hadoop.hdfs.server.federation.router.NamenodeHeartbeatService.updateState(NamenodeHeartbeatService.java:223)
> at
> org.apache.hadoop.hdfs.server.federation.router.NamenodeHeartbeatService.periodicInvoke(NamenodeHeartbeatService.java:159)
> at
> org.apache.hadoop.hdfs.server.federation.router.PeriodicService$1.run(PeriodicService.java:178)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14902) RBF: NullPointer When Misconfigured
[
https://issues.apache.org/jira/browse/HDFS-14902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16949142#comment-16949142
]
Takanobu Asanuma commented on HDFS-14902:
-
Thanks for reporting the issue, [~belugabehr] . I'd like to work on it.
> RBF: NullPointer When Misconfigured
> ---
>
> Key: HDFS-14902
> URL: https://issues.apache.org/jira/browse/HDFS-14902
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: rbf
>Affects Versions: 3.2.0
>Reporter: David Mollitor
>Priority: Minor
>
> Admittedly the server was mis-configured, but this should be a bit more
> elegant.
> {code:none}
> 2019-10-08 11:19:52,505 ERROR router.NamenodeHeartbeatService: Unhandled
> exception updating NN registration for null:null
> java.lang.NullPointerException
> at
> org.apache.hadoop.hdfs.federation.protocol.proto.HdfsServerFederationProtos$NamenodeMembershipRecordProto$Builder.setServiceAddress(HdfsServerFederationProtos.java:3831)
> at
> org.apache.hadoop.hdfs.server.federation.store.records.impl.pb.MembershipStatePBImpl.setServiceAddress(MembershipStatePBImpl.java:119)
> at
> org.apache.hadoop.hdfs.server.federation.store.records.MembershipState.newInstance(MembershipState.java:108)
> at
> org.apache.hadoop.hdfs.server.federation.resolver.MembershipNamenodeResolver.registerNamenode(MembershipNamenodeResolver.java:259)
> at
> org.apache.hadoop.hdfs.server.federation.router.NamenodeHeartbeatService.updateState(NamenodeHeartbeatService.java:223)
> at
> org.apache.hadoop.hdfs.server.federation.router.NamenodeHeartbeatService.periodicInvoke(NamenodeHeartbeatService.java:159)
> at
> org.apache.hadoop.hdfs.server.federation.router.PeriodicService$1.run(PeriodicService.java:178)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14891) RBF: namenode links in NameFederation Health page (federationhealth.html) cannot use https scheme
[
https://issues.apache.org/jira/browse/HDFS-14891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948358#comment-16948358
]
Takanobu Asanuma edited comment on HDFS-14891 at 10/10/19 9:56 AM:
---
Thanks for your detailed explanation, [~risyomei].
Seems it's not easy to get the exact scheme information from subclusters. As a
compromise, I think we can use the assumption that users should use the same
{{dfs.http.policy}} among router and its namenodes.
What do you think, [~elgoiri]?
was (Author: tasanuma0829):
Thanks for your detailed explanation, [~risyomei].
Seems it's not easy to get the exact scheme information from subclusters. As a
compromise, I think we can use the assumption that users should use the same
{{dfs.http.policy}} among router and its namenodes.
> RBF: namenode links in NameFederation Health page (federationhealth.html)
> cannot use https scheme
> --
>
> Key: HDFS-14891
> URL: https://issues.apache.org/jira/browse/HDFS-14891
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf, ui
>Reporter: Xieming Li
>Assignee: Xieming Li
>Priority: Major
> Attachments: HDFS-14891.001.patch, HDFS-14891.patch
>
>
> The scheme of links in federationhealth.html are hard coded as 'http'.
> It should be set to 'https' when dfs.http.policy is HTTPS_ONLY
> (HTTP_AND_HTTPS also, maybe)
>
> [https://github.com/apache/hadoop/blob/c99a12167ff9566012ef32104a3964887d62c899/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/webapps/router/federationhealth.html#L168-L169]
> [https://github.com/apache/hadoop/blob/c99a12167ff9566012ef32104a3964887d62c899/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/webapps/router/federationhealth.html#L236]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14891) RBF: namenode links in NameFederation Health page (federationhealth.html) cannot use https scheme
[
https://issues.apache.org/jira/browse/HDFS-14891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948358#comment-16948358
]
Takanobu Asanuma commented on HDFS-14891:
-
Thanks for your detailed explanation, [~risyomei].
Seems it's not easy to get the exact scheme information from subclusters. As a
compromise, I think we can use the assumption that users should use the same
{{dfs.http.policy}} among router and its namenodes.
> RBF: namenode links in NameFederation Health page (federationhealth.html)
> cannot use https scheme
> --
>
> Key: HDFS-14891
> URL: https://issues.apache.org/jira/browse/HDFS-14891
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf, ui
>Reporter: Xieming Li
>Assignee: Xieming Li
>Priority: Major
> Attachments: HDFS-14891.001.patch, HDFS-14891.patch
>
>
> The scheme of links in federationhealth.html are hard coded as 'http'.
> It should be set to 'https' when dfs.http.policy is HTTPS_ONLY
> (HTTP_AND_HTTPS also, maybe)
>
> [https://github.com/apache/hadoop/blob/c99a12167ff9566012ef32104a3964887d62c899/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/webapps/router/federationhealth.html#L168-L169]
> [https://github.com/apache/hadoop/blob/c99a12167ff9566012ef32104a3964887d62c899/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/webapps/router/federationhealth.html#L236]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14887) RBF: In Router Web UI, Observer Namenode Information displaying as Unavailable
[
https://issues.apache.org/jira/browse/HDFS-14887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948316#comment-16948316
]
Takanobu Asanuma commented on HDFS-14887:
-
{quote}BTW, can you share your experience with RBF and Observer NNs?
{quote}
In my experience, Router doesn't send any read requests to Observer NNs now
(even after this jira) . IIUC, HDFS-13522 is addressing it.
> RBF: In Router Web UI, Observer Namenode Information displaying as Unavailable
> --
>
> Key: HDFS-14887
> URL: https://issues.apache.org/jira/browse/HDFS-14887
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: hemanthboyina
>Assignee: hemanthboyina
>Priority: Major
> Attachments: 14887.after.png, 14887.before.png, HDFS-14887.001.patch
>
>
> In Router Web UI, Observer Namenode Information displaying as Unavailable.
> We should show a proper icon for them.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14887) RBF: In Router Web UI, Observer Namenode Information displaying as Unavailable
[
https://issues.apache.org/jira/browse/HDFS-14887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948311#comment-16948311
]
Takanobu Asanuma commented on HDFS-14887:
-
Sorry, I'd like to cancel my +1. We also need to fix
\{{FederationNamenodeServiceState}} in this jira. Could you update the patch,
[~hemanthboyina]?
> RBF: In Router Web UI, Observer Namenode Information displaying as Unavailable
> --
>
> Key: HDFS-14887
> URL: https://issues.apache.org/jira/browse/HDFS-14887
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: hemanthboyina
>Assignee: hemanthboyina
>Priority: Major
> Attachments: 14887.after.png, 14887.before.png, HDFS-14887.001.patch
>
>
> In Router Web UI, Observer Namenode Information displaying as Unavailable.
> We should show a proper icon for them.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14887) RBF: In Router Web UI, Observer Namenode Information displaying as Unavailable
[ https://issues.apache.org/jira/browse/HDFS-14887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16948274#comment-16948274 ] Takanobu Asanuma commented on HDFS-14887: - +1. Will commit it. > RBF: In Router Web UI, Observer Namenode Information displaying as Unavailable > -- > > Key: HDFS-14887 > URL: https://issues.apache.org/jira/browse/HDFS-14887 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > Attachments: 14887.after.png, 14887.before.png, HDFS-14887.001.patch > > > In Router Web UI, Observer Namenode Information displaying as Unavailable. > We should show a proper icon for them. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14891) RBF: namenode links in NameFederation Health page (federationhealth.html) cannot use https scheme
[
https://issues.apache.org/jira/browse/HDFS-14891?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16946613#comment-16946613
]
Takanobu Asanuma commented on HDFS-14891:
-
Thanks for finding the issue and submitting the patch, [~risyomei].
Most of the changes look good to me. Some minor comments:
* Could you remove the first blank line in {{MembershipStatePBImpl#getScheme}}?
* Is it possible to add the unit test that the scheme is https?
> RBF: namenode links in NameFederation Health page (federationhealth.html)
> cannot use https scheme
> --
>
> Key: HDFS-14891
> URL: https://issues.apache.org/jira/browse/HDFS-14891
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: rbf, ui
>Reporter: Xieming Li
>Assignee: Xieming Li
>Priority: Major
> Attachments: HDFS-14891.001.patch, HDFS-14891.patch
>
>
> The scheme of links in federationhealth.html are hard coded as 'http'.
> It should be set to 'https' when dfs.http.policy is HTTPS_ONLY
> (HTTP_AND_HTTPS also, maybe)
>
> [https://github.com/apache/hadoop/blob/c99a12167ff9566012ef32104a3964887d62c899/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/webapps/router/federationhealth.html#L168-L169]
> [https://github.com/apache/hadoop/blob/c99a12167ff9566012ef32104a3964887d62c899/hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/webapps/router/federationhealth.html#L236]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14879) Header was wrong in Snapshot web UI
[ https://issues.apache.org/jira/browse/HDFS-14879?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HDFS-14879: Fix Version/s: 3.2.2 3.1.4 3.3.0 Resolution: Fixed Status: Resolved (was: Patch Available) Committed to trunk, branch-3.2 and branch-3.1. Thanks for your contribution, [~hemanthboyina]. > Header was wrong in Snapshot web UI > --- > > Key: HDFS-14879 > URL: https://issues.apache.org/jira/browse/HDFS-14879 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > Fix For: 3.3.0, 3.1.4, 3.2.2 > > Attachments: HDFS-14879.001.patch, snapshotted.JPG > > > we are showing the list of snapshots but the header is Snapshotted Directories > attached screenshot of UI -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14879) Header was wrong in Snapshot web UI
[ https://issues.apache.org/jira/browse/HDFS-14879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16944293#comment-16944293 ] Takanobu Asanuma commented on HDFS-14879: - Seems the word of Snapshots is more suitable. +1. > Header was wrong in Snapshot web UI > --- > > Key: HDFS-14879 > URL: https://issues.apache.org/jira/browse/HDFS-14879 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Major > Attachments: HDFS-14879.001.patch, snapshotted.JPG > > > we are showing the list of snapshots but the header is Snapshotted Directories > attached screenshot of UI -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14832) RBF : Add Icon for ReadOnly False
[ https://issues.apache.org/jira/browse/HDFS-14832?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HDFS-14832: Fix Version/s: 3.3.0 Resolution: Fixed Status: Resolved (was: Patch Available) Committed to trunk. Thanks for your contribution, [~hemanthboyina]. Thanks for your review, [~elgoiri]. > RBF : Add Icon for ReadOnly False > - > > Key: HDFS-14832 > URL: https://issues.apache.org/jira/browse/HDFS-14832 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Minor > Fix For: 3.3.0 > > Attachments: HDFS-14832.001.patch, HDFS-14832.002.patch, > HDFS-14832.003.patch, HDFS-14832.after.JPG, ReadWrite.After.JPG, > ReadWrite.before.JPG, Screenshot from 2019-09-18 23-55-17.png > > > In Router Web UI for Mount Table information , add icon for read only state > false -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14832) RBF : Add Icon for ReadOnly False
[ https://issues.apache.org/jira/browse/HDFS-14832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16937801#comment-16937801 ] Takanobu Asanuma commented on HDFS-14832: - +1. Will commit it. > RBF : Add Icon for ReadOnly False > - > > Key: HDFS-14832 > URL: https://issues.apache.org/jira/browse/HDFS-14832 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Minor > Attachments: HDFS-14832.001.patch, HDFS-14832.002.patch, > HDFS-14832.003.patch, HDFS-14832.after.JPG, ReadWrite.After.JPG, > ReadWrite.before.JPG, Screenshot from 2019-09-18 23-55-17.png > > > In Router Web UI for Mount Table information , add icon for read only state > false -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14832) RBF : Add Icon for ReadOnly False
[ https://issues.apache.org/jira/browse/HDFS-14832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934218#comment-16934218 ] Takanobu Asanuma edited comment on HDFS-14832 at 9/20/19 9:12 AM: -- Thanks for uploading the 001 patch, [~hemanthboyina]. * Jenkins accidentally applied the image as a patch and failed. It would run successfully with the next patch. * I think "Mount Option" or "Writable" or something like that would be match for the column name. * Could you use the same upper/lower case? The icon status is "Read only" and federationhealth-mounttable-legend is "Read Only". was (Author: tasanuma0829): Thanks for uploading the 001 patch, [~hemanthboyina]. * Jenkins accidentally applied the image as a patch and failed. It would run successfully with the next patch. * I think "Mount Option" or "Writable" or something like that would be match for the column name. * Could you use the same upper/lower case? There are "Read only" and "Read Only". > RBF : Add Icon for ReadOnly False > - > > Key: HDFS-14832 > URL: https://issues.apache.org/jira/browse/HDFS-14832 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Minor > Attachments: HDFS-14832.001.patch, HDFS-14832.002.patch, > ReadWrite.After.JPG, ReadWrite.before.JPG, Screenshot from 2019-09-18 > 23-55-17.png > > > In Router Web UI for Mount Table information , add icon for read only state > false -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14832) RBF : Add Icon for ReadOnly False
[ https://issues.apache.org/jira/browse/HDFS-14832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16934218#comment-16934218 ] Takanobu Asanuma commented on HDFS-14832: - Thanks for uploading the 001 patch, [~hemanthboyina]. * Jenkins accidentally applied the image as a patch and failed. It would run successfully with the next patch. * I think "Mount Option" or "Writable" or something like that would be match for the column name. * Could you use the same upper/lower case? There are "Read only" and "Read Only". > RBF : Add Icon for ReadOnly False > - > > Key: HDFS-14832 > URL: https://issues.apache.org/jira/browse/HDFS-14832 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Minor > Attachments: HDFS-14832.001.patch, HDFS-14832.002.patch, > ReadWrite.After.JPG, ReadWrite.before.JPG, Screenshot from 2019-09-18 > 23-55-17.png > > > In Router Web UI for Mount Table information , add icon for read only state > false -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14832) RBF : Add Icon for ReadOnly False
[
https://issues.apache.org/jira/browse/HDFS-14832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16933104#comment-16933104
]
Takanobu Asanuma commented on HDFS-14832:
-
Thanks for sharing your thoughts, [~hemanthboyina] and [~elgoiri].
All right, Let's use icons with _federationhealth-mounttable-legend._
{quote}I think glyphicon edit matches for read-write scenario.
{quote}
That looks good to me too.
> RBF : Add Icon for ReadOnly False
> -
>
> Key: HDFS-14832
> URL: https://issues.apache.org/jira/browse/HDFS-14832
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: hemanthboyina
>Assignee: hemanthboyina
>Priority: Minor
> Attachments: Screenshot from 2019-09-18 23-55-17.png
>
>
> In Router Web UI for Mount Table information , add icon for read only state
> false
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14609) RBF: Security should use common AuthenticationFilter
[ https://issues.apache.org/jira/browse/HDFS-14609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16932191#comment-16932191 ] Takanobu Asanuma commented on HDFS-14609: - +1 on [^HDFS-14609.006.patch]. > RBF: Security should use common AuthenticationFilter > > > Key: HDFS-14609 > URL: https://issues.apache.org/jira/browse/HDFS-14609 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: CR Hota >Assignee: Chen Zhang >Priority: Major > Attachments: HDFS-14609.001.patch, HDFS-14609.002.patch, > HDFS-14609.003.patch, HDFS-14609.004.patch, HDFS-14609.005.patch, > HDFS-14609.006.patch > > > We worked on router based federation security as part of HDFS-13532. We kept > it compatible with the way namenode works. However with HADOOP-16314 and > HDFS-16354 in trunk, auth filters seems to have been changed causing tests to > fail. > Changes are needed appropriately in RBF, mainly fixing broken tests. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14609) RBF: Security should use common AuthenticationFilter
[ https://issues.apache.org/jira/browse/HDFS-14609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928791#comment-16928791 ] Takanobu Asanuma commented on HDFS-14609: - Thanks for letting me know the issue, [~crh]. The rest looks good to me. > RBF: Security should use common AuthenticationFilter > > > Key: HDFS-14609 > URL: https://issues.apache.org/jira/browse/HDFS-14609 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: CR Hota >Assignee: Chen Zhang >Priority: Major > Attachments: HDFS-14609.001.patch, HDFS-14609.002.patch, > HDFS-14609.003.patch, HDFS-14609.004.patch, HDFS-14609.005.patch, > HDFS-14609.006.patch > > > We worked on router based federation security as part of HDFS-13532. We kept > it compatible with the way namenode works. However with HADOOP-16314 and > HDFS-16354 in trunk, auth filters seems to have been changed causing tests to > fail. > Changes are needed appropriately in RBF, mainly fixing broken tests. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14609) RBF: Security should use common AuthenticationFilter
[
https://issues.apache.org/jira/browse/HDFS-14609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928518#comment-16928518
]
Takanobu Asanuma commented on HDFS-14609:
-
[~zhangchen] Thanks for your hard work on this issue.
I ran the unit tests several times and sometimes saw the following error. This
test may be flaky.
{noformat}
$ mvn test -Dtest=TestRouterHttpDelegationToken
...
[INFO] Running
org.apache.hadoop.hdfs.server.federation.security.TestRouterHttpDelegationToken
[ERROR] Tests run: 3, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 6.146 s
<<< FAILURE! - in
org.apache.hadoop.hdfs.server.federation.security.TestRouterHttpDelegationToken
[ERROR]
testGetDelegationToken(org.apache.hadoop.hdfs.server.federation.security.TestRouterHttpDelegationToken)
Time elapsed: 0.115 s <<< ERROR!
org.apache.hadoop.service.ServiceStateException:
org.apache.hadoop.security.KerberosAuthException: failure to login: for
principal: router/localh...@example.com from keytab
/.../hadoop/hadoop-hdfs-project/hadoop-hdfs-rbf/target/test/data/SecurityConfUtil/test.keytab
javax.security.auth.login.LoginException: Integrity check on decrypted field
failed (31) - PREAUTH_FAILED
at
org.apache.hadoop.service.ServiceStateException.convert(ServiceStateException.java:105)
at
org.apache.hadoop.service.AbstractService.init(AbstractService.java:173)
at
org.apache.hadoop.hdfs.server.federation.security.TestRouterHttpDelegationToken.setup(TestRouterHttpDelegationToken.java:132)
{noformat}
The patch seems no problem, but I think {{SecurityConfUtil#initSecurity()}} has
some problems.
> RBF: Security should use common AuthenticationFilter
>
>
> Key: HDFS-14609
> URL: https://issues.apache.org/jira/browse/HDFS-14609
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: CR Hota
>Assignee: Chen Zhang
>Priority: Major
> Attachments: HDFS-14609.001.patch, HDFS-14609.002.patch,
> HDFS-14609.003.patch, HDFS-14609.004.patch, HDFS-14609.005.patch,
> HDFS-14609.006.patch
>
>
> We worked on router based federation security as part of HDFS-13532. We kept
> it compatible with the way namenode works. However with HADOOP-16314 and
> HDFS-16354 in trunk, auth filters seems to have been changed causing tests to
> fail.
> Changes are needed appropriately in RBF, mainly fixing broken tests.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14832) RBF : Add Icon for ReadOnly False
[ https://issues.apache.org/jira/browse/HDFS-14832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928275#comment-16928275 ] Takanobu Asanuma commented on HDFS-14832: - Thanks for your comment, [~hemanthboyina]. * I feel lock icon means read-only. And judging based on the icon color may confuse admins. * If Read-only false (which means read-write) status shows there, I prefer to rename the column name from "Read Only" to "Mount Option". * I also listened to my colleagues' opinions. In summary, using icon for read-only may be a bit much. String texts may be clearer. [~hemanthboyina] [~elgoiri] How does look that? > RBF : Add Icon for ReadOnly False > - > > Key: HDFS-14832 > URL: https://issues.apache.org/jira/browse/HDFS-14832 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: hemanthboyina >Assignee: hemanthboyina >Priority: Minor > > In Router Web UI for Mount Table information , add icon for read only state > false -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14832) RBF : Add Icon for ReadOnly False
[
https://issues.apache.org/jira/browse/HDFS-14832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16928161#comment-16928161
]
Takanobu Asanuma commented on HDFS-14832:
-
Thanks for working on this, [~hemanthboyina], and thanks for pinging me,
[~elgoiri].
If we have both of read-only and read-write icons, we may need
{{federationhealth-mounttable-legend}} section like other pages.
> RBF : Add Icon for ReadOnly False
> -
>
> Key: HDFS-14832
> URL: https://issues.apache.org/jira/browse/HDFS-14832
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: hemanthboyina
>Assignee: hemanthboyina
>Priority: Minor
>
> In Router Web UI for Mount Table information , add icon for read only state
> false
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14838) RBF: Display RPC (instead of HTTP) Port Number in RBF web UI
[ https://issues.apache.org/jira/browse/HDFS-14838?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HDFS-14838: Fix Version/s: 3.1.4 3.2.1 3.3.0 Resolution: Fixed Status: Resolved (was: Patch Available) Committed to trunk, branch-3.2 and branch-3.1. Thanks for your contribution, [~risyomei]! > RBF: Display RPC (instead of HTTP) Port Number in RBF web UI > > > Key: HDFS-14838 > URL: https://issues.apache.org/jira/browse/HDFS-14838 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf, ui >Affects Versions: 3.1.2 >Reporter: Xieming Li >Assignee: Xieming Li >Priority: Minor > Labels: RBF, ui > Fix For: 3.3.0, 3.2.1, 3.1.4 > > Attachments: HDFS-14838-1.patch, HDFS-14838.001.patch, > HDFS-14838.patch, router-ui.jpg > > > Currently The WebUI of RBF is using : in its heading. > It should be changed to : as the WebUI of NameNode and > DataNode do. > > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14838) RBF: Display RPC (instead of HTTP) Port Number in RBF web UI
[ https://issues.apache.org/jira/browse/HDFS-14838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927348#comment-16927348 ] Takanobu Asanuma commented on HDFS-14838: - The failed tests are not related to the patch. Will commit it. > RBF: Display RPC (instead of HTTP) Port Number in RBF web UI > > > Key: HDFS-14838 > URL: https://issues.apache.org/jira/browse/HDFS-14838 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf, ui >Affects Versions: 3.1.2 >Reporter: Xieming Li >Assignee: Xieming Li >Priority: Minor > Labels: RBF, ui > Attachments: HDFS-14838-1.patch, HDFS-14838.001.patch, > HDFS-14838.patch, router-ui.jpg > > > Currently The WebUI of RBF is using : in its heading. > It should be changed to : as the WebUI of NameNode and > DataNode do. > > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14838) RBF: Display RPC (instead of HTTP) Port Number in RBF web UI
[ https://issues.apache.org/jira/browse/HDFS-14838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927248#comment-16927248 ] Takanobu Asanuma edited comment on HDFS-14838 at 9/11/19 5:22 AM: -- +1 for HDFS-14838.001.patch, pending Jenkins. was (Author: tasanuma0829): +1 for HDFS-14838-1.patch, pending Jenkins. > RBF: Display RPC (instead of HTTP) Port Number in RBF web UI > > > Key: HDFS-14838 > URL: https://issues.apache.org/jira/browse/HDFS-14838 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf, ui >Affects Versions: 3.1.2 >Reporter: Xieming Li >Assignee: Xieming Li >Priority: Minor > Labels: RBF, ui > Attachments: HDFS-14838-1.patch, HDFS-14838.001.patch, > HDFS-14838.patch, router-ui.jpg > > > Currently The WebUI of RBF is using : in its heading. > It should be changed to : as the WebUI of NameNode and > DataNode do. > > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14838) RBF: Display RPC (instead of HTTP) Port Number in RBF web UI
[ https://issues.apache.org/jira/browse/HDFS-14838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927248#comment-16927248 ] Takanobu Asanuma commented on HDFS-14838: - +1 for HDFS-14838-1.patch, pending Jenkins. > RBF: Display RPC (instead of HTTP) Port Number in RBF web UI > > > Key: HDFS-14838 > URL: https://issues.apache.org/jira/browse/HDFS-14838 > Project: Hadoop HDFS > Issue Type: Bug > Components: rbf, ui >Affects Versions: 3.1.2 >Reporter: Xieming Li >Assignee: Xieming Li >Priority: Minor > Labels: RBF, ui > Attachments: HDFS-14838-1.patch, HDFS-14838.patch, router-ui.jpg > > > Currently The WebUI of RBF is using : in its heading. > It should be changed to : as the WebUI of NameNode and > DataNode do. > > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14838) RBF: Display RPC (instead of HTTP) Port Number in RBF web UI
[
https://issues.apache.org/jira/browse/HDFS-14838?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16927192#comment-16927192
]
Takanobu Asanuma commented on HDFS-14838:
-
Thanks for working on this issue and submitting the patch, [~risyomei].
Since we want to keep the same interface between Router and NameNode, instead
of using {{RouterId}}, it would be better to fix
{{RBFMetrics#getHostAndPort()}}.
> RBF: Display RPC (instead of HTTP) Port Number in RBF web UI
>
>
> Key: HDFS-14838
> URL: https://issues.apache.org/jira/browse/HDFS-14838
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf, ui
>Affects Versions: 3.1.2
>Reporter: Xieming Li
>Assignee: Xieming Li
>Priority: Minor
> Labels: RBF, ui
> Attachments: HDFS-14838.patch, router-ui.jpg
>
>
> Currently The WebUI of RBF is using : in its heading.
> It should be changed to : as the WebUI of NameNode and
> DataNode do.
>
>
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14835) RBF: Secured Router should not run when it can't initialize DelegationTokenSecretManager
[
https://issues.apache.org/jira/browse/HDFS-14835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takanobu Asanuma updated HDFS-14835:
Fix Version/s: 3.3.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
Merged the PR to trunk. Thanks again for your reviews, [~crh] and [~elgoiri].
> RBF: Secured Router should not run when it can't initialize
> DelegationTokenSecretManager
>
>
> Key: HDFS-14835
> URL: https://issues.apache.org/jira/browse/HDFS-14835
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Labels: RBF
> Fix For: 3.3.0
>
>
> Currently, even if secured router fails to create
> DelegationTokenSecretManager, it can start and keeps running. This router
> can't handle requests with delegation tokens.
> {noformat}
> ERROR org.apache.hadoop.hdfs.server.federation.router.FederationUtil: Could
> not instantiate: ZKDelegationTokenSecretManagerImpl
> ...
> INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port
> {noformat}
> In this case, I think router should not start.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14835) RBF: Secured Router should not run when it can't initialize DelegationTokenSecretManager
[
https://issues.apache.org/jira/browse/HDFS-14835?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takanobu Asanuma updated HDFS-14835:
Status: Patch Available (was: Open)
> RBF: Secured Router should not run when it can't initialize
> DelegationTokenSecretManager
>
>
> Key: HDFS-14835
> URL: https://issues.apache.org/jira/browse/HDFS-14835
> Project: Hadoop HDFS
> Issue Type: Bug
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Labels: RBF
>
> Currently, even if secured router fails to create
> DelegationTokenSecretManager, it can start and keeps running. This router
> can't handle requests with delegation tokens.
> {noformat}
> ERROR org.apache.hadoop.hdfs.server.federation.router.FederationUtil: Could
> not instantiate: ZKDelegationTokenSecretManagerImpl
> ...
> INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port
> {noformat}
> In this case, I think router should not start.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14835) RBF: Secured Router should not run when it can't initialize DelegationTokenSecretManager
Takanobu Asanuma created HDFS-14835:
---
Summary: RBF: Secured Router should not run when it can't
initialize DelegationTokenSecretManager
Key: HDFS-14835
URL: https://issues.apache.org/jira/browse/HDFS-14835
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Takanobu Asanuma
Assignee: Takanobu Asanuma
Currently, even if secured router fails to create DelegationTokenSecretManager,
it can start and keeps running. This router can't handle requests with
delegation tokens.
{noformat}
ERROR org.apache.hadoop.hdfs.server.federation.router.FederationUtil: Could not
instantiate: ZKDelegationTokenSecretManagerImpl
...
INFO org.apache.hadoop.ipc.Server: Starting Socket Reader #1 for port
{noformat}
In this case, I think router should not start.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14772) RBF: hdfs-rbf-site.xml can't be loaded automatically
[
https://issues.apache.org/jira/browse/HDFS-14772?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takanobu Asanuma updated HDFS-14772:
Resolution: Fixed
Status: Resolved (was: Patch Available)
> RBF: hdfs-rbf-site.xml can't be loaded automatically
>
>
> Key: HDFS-14772
> URL: https://issues.apache.org/jira/browse/HDFS-14772
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf
>Reporter: Yuxuan Wang
>Assignee: Yuxuan Wang
>Priority: Major
> Labels: RBF
> Attachments: HDFS-14772.001.patch, HDFS-14772.002.patch,
> HDFS-14772.003.patch, HDFS-14772.004.patch
>
>
> ISSUE:
> hdfs-rbf-site.xml can't be loaded automatically
> WHY:
> Currently the code is
> {code:title=RBFConfigKeys.java|borderStyle=solid}
> static {
> Configuration.addDefaultResource(HDFS_RBF_SITE_XML);
> }
> {code}
> But it will never be executed unless we explicitly load the class.
> HOW TO FIX:
> Reference to class *HdfsConfiguration*, make a method
> {code:title=RBFConfigKeys.java|borderStyle=solid}
> public static void init() {
> }
> {code}
> and call it in other class.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14772) RBF: hdfs-rbf-site.xml can't be loaded automatically
[
https://issues.apache.org/jira/browse/HDFS-14772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16916392#comment-16916392
]
Takanobu Asanuma commented on HDFS-14772:
-
Committed to trunk. Thanks for your contribution, [~John Smith], and thanks for
your reviews and comments [~surendrasingh] and others!
> RBF: hdfs-rbf-site.xml can't be loaded automatically
>
>
> Key: HDFS-14772
> URL: https://issues.apache.org/jira/browse/HDFS-14772
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf
>Reporter: Yuxuan Wang
>Assignee: Yuxuan Wang
>Priority: Major
> Labels: RBF
> Attachments: HDFS-14772.001.patch, HDFS-14772.002.patch,
> HDFS-14772.003.patch, HDFS-14772.004.patch
>
>
> ISSUE:
> hdfs-rbf-site.xml can't be loaded automatically
> WHY:
> Currently the code is
> {code:title=RBFConfigKeys.java|borderStyle=solid}
> static {
> Configuration.addDefaultResource(HDFS_RBF_SITE_XML);
> }
> {code}
> But it will never be executed unless we explicitly load the class.
> HOW TO FIX:
> Reference to class *HdfsConfiguration*, make a method
> {code:title=RBFConfigKeys.java|borderStyle=solid}
> public static void init() {
> }
> {code}
> and call it in other class.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14772) RBF: hdfs-rbf-site.xml can't be loaded automatically
[
https://issues.apache.org/jira/browse/HDFS-14772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915776#comment-16915776
]
Takanobu Asanuma commented on HDFS-14772:
-
The failed tests are not related.
[~John Smith] Could you fix the checkstyle issue? Then, I'll commit it.
> RBF: hdfs-rbf-site.xml can't be loaded automatically
>
>
> Key: HDFS-14772
> URL: https://issues.apache.org/jira/browse/HDFS-14772
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf
>Reporter: Yuxuan Wang
>Assignee: Yuxuan Wang
>Priority: Major
> Labels: RBF
> Attachments: HDFS-14772.001.patch, HDFS-14772.002.patch,
> HDFS-14772.003.patch
>
>
> ISSUE:
> hdfs-rbf-site.xml can't be loaded automatically
> WHY:
> Currently the code is
> {code:title=RBFConfigKeys.java|borderStyle=solid}
> static {
> Configuration.addDefaultResource(HDFS_RBF_SITE_XML);
> }
> {code}
> But it will never be executed unless we explicitly load the class.
> HOW TO FIX:
> Reference to class *HdfsConfiguration*, make a method
> {code:title=RBFConfigKeys.java|borderStyle=solid}
> public static void init() {
> }
> {code}
> and call it in other class.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14772) RBF: hdfs-rbf-site.xml can't be loaded automatically
[
https://issues.apache.org/jira/browse/HDFS-14772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915682#comment-16915682
]
Takanobu Asanuma commented on HDFS-14772:
-
+1, pending Jenkins.
> RBF: hdfs-rbf-site.xml can't be loaded automatically
>
>
> Key: HDFS-14772
> URL: https://issues.apache.org/jira/browse/HDFS-14772
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf
>Reporter: Yuxuan Wang
>Assignee: Yuxuan Wang
>Priority: Major
> Labels: RBF
> Attachments: HDFS-14772.001.patch, HDFS-14772.002.patch,
> HDFS-14772.003.patch
>
>
> ISSUE:
> hdfs-rbf-site.xml can't be loaded automatically
> WHY:
> Currently the code is
> {code:title=RBFConfigKeys.java|borderStyle=solid}
> static {
> Configuration.addDefaultResource(HDFS_RBF_SITE_XML);
> }
> {code}
> But it will never be executed unless we explicitly load the class.
> HOW TO FIX:
> Reference to class *HdfsConfiguration*, make a method
> {code:title=RBFConfigKeys.java|borderStyle=solid}
> public static void init() {
> }
> {code}
> and call it in other class.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14772) RBF: hdfs-rbf-site.xml can't be loaded automatically
[
https://issues.apache.org/jira/browse/HDFS-14772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915649#comment-16915649
]
Takanobu Asanuma commented on HDFS-14772:
-
Thanks for updating the patch, [~John Smith].
bq. Do we need remove the code here?
Yes, I think we can remove it from RBFConfigKeys since \{{DFSRouter#main}} and
\{{RouterAdmin#main}} load HdfsConfiguration.
> RBF: hdfs-rbf-site.xml can't be loaded automatically
>
>
> Key: HDFS-14772
> URL: https://issues.apache.org/jira/browse/HDFS-14772
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf
>Reporter: Yuxuan Wang
>Assignee: Yuxuan Wang
>Priority: Major
> Labels: RBF
> Attachments: HDFS-14772.001.patch, HDFS-14772.002.patch
>
>
> ISSUE:
> hdfs-rbf-site.xml can't be loaded automatically
> WHY:
> Currently the code is
> {code:title=RBFConfigKeys.java|borderStyle=solid}
> static {
> Configuration.addDefaultResource(HDFS_RBF_SITE_XML);
> }
> {code}
> But it will never be executed unless we explicitly load the class.
> HOW TO FIX:
> Reference to class *HdfsConfiguration*, make a method
> {code:title=RBFConfigKeys.java|borderStyle=solid}
> public static void init() {
> }
> {code}
> and call it in other class.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14772) RBF: hdfs-rbf-site.xml can't be loaded automatically
[
https://issues.apache.org/jira/browse/HDFS-14772?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16914768#comment-16914768
]
Takanobu Asanuma commented on HDFS-14772:
-
Thanks [~John Smith].
I agree with [~surendrasingh] 's suggestion.
> RBF: hdfs-rbf-site.xml can't be loaded automatically
>
>
> Key: HDFS-14772
> URL: https://issues.apache.org/jira/browse/HDFS-14772
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf
>Reporter: Yuxuan Wang
>Assignee: Yuxuan Wang
>Priority: Major
> Labels: RBF
> Attachments: HDFS-14772.001.patch
>
>
> ISSUE:
> hdfs-rbf-site.xml can't be loaded automatically
> WHY:
> Currently the code is
> {code:title=RBFConfigKeys.java|borderStyle=solid}
> static {
> Configuration.addDefaultResource(HDFS_RBF_SITE_XML);
> }
> {code}
> But it will never be executed unless we explicitly load the class.
> HOW TO FIX:
> Reference to class *HdfsConfiguration*, make a method
> {code:title=RBFConfigKeys.java|borderStyle=solid}
> public static void init() {
> }
> {code}
> and call it in other class.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14755) [Dynamometer] Hadoop-2 DataNode fail to start
[
https://issues.apache.org/jira/browse/HDFS-14755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16913830#comment-16913830
]
Takanobu Asanuma commented on HDFS-14755:
-
Thanks for reviewing and committing it, [~xkrogen]!
> [Dynamometer] Hadoop-2 DataNode fail to start
> -
>
> Key: HDFS-14755
> URL: https://issues.apache.org/jira/browse/HDFS-14755
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: tools
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Fix For: 3.3.0
>
>
> When using a fsimage of Hadoop-2 with hadoop-dynamometer, datanodes fail to
> start with the following error.
> {noformat}
> Exception in thread "main" java.lang.IllegalAccessError: tried to access
> method
> org.apache.hadoop.hdfs.server.datanode.StorageLocation.getUri()Ljava/net/URI;
> from class org.apache.hadoop.tools.dynamometer.SimulatedDataNodes
> at
> org.apache.hadoop.tools.dynamometer.SimulatedDataNodes.run(SimulatedDataNodes.java:113)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at
> org.apache.hadoop.tools.dynamometer.SimulatedDataNodes.main(SimulatedDataNodes.java:88)
> ./start-component.sh: line 317: kill: (9876) - No such process
> {noformat}
> The cause of this error is an incompatibility of StorageLocation.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14763) Fix package name of audit log class in Dynamometer document
[ https://issues.apache.org/jira/browse/HDFS-14763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HDFS-14763: Fix Version/s: 3.3.0 Resolution: Fixed Status: Resolved (was: Patch Available) Committed to trunk. > Fix package name of audit log class in Dynamometer document > --- > > Key: HDFS-14763 > URL: https://issues.apache.org/jira/browse/HDFS-14763 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: documentation, tools >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Major > Fix For: 3.3.0 > > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14763) Fix package name of audit log class in Dynamometer document
[ https://issues.apache.org/jira/browse/HDFS-14763?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HDFS-14763: Status: Patch Available (was: Open) > Fix package name of audit log class in Dynamometer document > --- > > Key: HDFS-14763 > URL: https://issues.apache.org/jira/browse/HDFS-14763 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: documentation, tools >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Major > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14763) Fix package name of audit log class in Dynamometer document
Takanobu Asanuma created HDFS-14763: --- Summary: Fix package name of audit log class in Dynamometer document Key: HDFS-14763 URL: https://issues.apache.org/jira/browse/HDFS-14763 Project: Hadoop HDFS Issue Type: Sub-task Components: documentation, tools Reporter: Takanobu Asanuma Assignee: Takanobu Asanuma -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14755) [Dynamometer] Hadoop-2 DataNode fail to start
[
https://issues.apache.org/jira/browse/HDFS-14755?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16911480#comment-16911480
]
Takanobu Asanuma commented on HDFS-14755:
-
Hi [~xkrogen]. Thanks for your review and providing the information!
Actually, when using hadoop-dynamometer(latest trunk) with the patch, I have
confirmed that HDP-2.6(hadoop-2.7.3) cluster just runs successfully. Though I
haven't confirmed workload jobs yet.
> [Dynamometer] Hadoop-2 DataNode fail to start
> -
>
> Key: HDFS-14755
> URL: https://issues.apache.org/jira/browse/HDFS-14755
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: tools
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
>
> When using a fsimage of Hadoop-2 with hadoop-dynamometer, datanodes fail to
> start with the following error.
> {noformat}
> Exception in thread "main" java.lang.IllegalAccessError: tried to access
> method
> org.apache.hadoop.hdfs.server.datanode.StorageLocation.getUri()Ljava/net/URI;
> from class org.apache.hadoop.tools.dynamometer.SimulatedDataNodes
> at
> org.apache.hadoop.tools.dynamometer.SimulatedDataNodes.run(SimulatedDataNodes.java:113)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at
> org.apache.hadoop.tools.dynamometer.SimulatedDataNodes.main(SimulatedDataNodes.java:88)
> ./start-component.sh: line 317: kill: (9876) - No such process
> {noformat}
> The cause of this error is an incompatibility of StorageLocation.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14755) [Dynamometer] Hadoop-2 DataNode fail to start
[
https://issues.apache.org/jira/browse/HDFS-14755?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takanobu Asanuma updated HDFS-14755:
Status: Patch Available (was: Open)
> [Dynamometer] Hadoop-2 DataNode fail to start
> -
>
> Key: HDFS-14755
> URL: https://issues.apache.org/jira/browse/HDFS-14755
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: tools
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
>
> When using a fsimage of Hadoop-2 with hadoop-dynamometer, datanodes fail to
> start with the following error.
> {noformat}
> Exception in thread "main" java.lang.IllegalAccessError: tried to access
> method
> org.apache.hadoop.hdfs.server.datanode.StorageLocation.getUri()Ljava/net/URI;
> from class org.apache.hadoop.tools.dynamometer.SimulatedDataNodes
> at
> org.apache.hadoop.tools.dynamometer.SimulatedDataNodes.run(SimulatedDataNodes.java:113)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at
> org.apache.hadoop.tools.dynamometer.SimulatedDataNodes.main(SimulatedDataNodes.java:88)
> ./start-component.sh: line 317: kill: (9876) - No such process
> {noformat}
> The cause of this error is an incompatibility of StorageLocation.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14755) [Dynamometer] Hadoop-2 DataNode fail to start
Takanobu Asanuma created HDFS-14755:
---
Summary: [Dynamometer] Hadoop-2 DataNode fail to start
Key: HDFS-14755
URL: https://issues.apache.org/jira/browse/HDFS-14755
Project: Hadoop HDFS
Issue Type: Sub-task
Components: tools
Reporter: Takanobu Asanuma
Assignee: Takanobu Asanuma
When using a fsimage of Hadoop-2 with hadoop-dynamometer, datanodes fail to
start with the following error.
{noformat}
Exception in thread "main" java.lang.IllegalAccessError: tried to access method
org.apache.hadoop.hdfs.server.datanode.StorageLocation.getUri()Ljava/net/URI;
from class org.apache.hadoop.tools.dynamometer.SimulatedDataNodes
at
org.apache.hadoop.tools.dynamometer.SimulatedDataNodes.run(SimulatedDataNodes.java:113)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
at
org.apache.hadoop.tools.dynamometer.SimulatedDataNodes.main(SimulatedDataNodes.java:88)
./start-component.sh: line 317: kill: (9876) - No such process
{noformat}
The cause of this error is an incompatibility of StorageLocation.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14609) RBF: Security should use common AuthenticationFilter
[ https://issues.apache.org/jira/browse/HDFS-14609?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16906787#comment-16906787 ] Takanobu Asanuma commented on HDFS-14609: - Thanks for the comments and the investigation, Chen Zhang and CR Hota. I'm sorry that I coudn't work on this recently due to other tasks. This is the old revision of HDFS-13891. The faild tests are succeeded in this stage. Comparing this branch and trunk may help to find the problem. [https://github.com/tasanuma/hadoop-private] > RBF: Security should use common AuthenticationFilter > > > Key: HDFS-14609 > URL: https://issues.apache.org/jira/browse/HDFS-14609 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: CR Hota >Assignee: Chen Zhang >Priority: Major > > We worked on router based federation security as part of HDFS-13532. We kept > it compatible with the way namenode works. However with HADOOP-16314 and > HDFS-16354 in trunk, auth filters seems to have been changed causing tests to > fail. > Changes are needed appropriately in RBF, mainly fixing broken tests. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14656) RBF: NPE in RBFMetrics
[ https://issues.apache.org/jira/browse/HDFS-14656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16888724#comment-16888724 ] Takanobu Asanuma commented on HDFS-14656: - I also think it's useful that router can run without State Store for testing. But, TestRouterRpc should probably use State Store. > RBF: NPE in RBFMetrics > -- > > Key: HDFS-14656 > URL: https://issues.apache.org/jira/browse/HDFS-14656 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > > Caused by: java.lang.NullPointerException > at > org.apache.hadoop.hdfs.server.federation.metrics.RBFMetrics.getActiveNamenodeRegistrations(RBFMetrics.java:726) > at > org.apache.hadoop.hdfs.server.federation.metrics.RBFMetrics.getNameserviceAggregatedInt(RBFMetrics.java:688) > at > org.apache.hadoop.hdfs.server.federation.metrics.RBFMetrics.getNumInMaintenanceDeadDataNodes(RBFMetrics.java:467) > at > org.apache.hadoop.hdfs.server.federation.metrics.NamenodeBeanMetrics.getNumInMaintenanceDeadDataNodes(NamenodeBeanMetrics.java:693) > at sun.reflect.GeneratedMethodAccessor71.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at sun.reflect.misc.Trampoline.invoke(MethodUtil.java:71) > at sun.reflect.GeneratedMethodAccessor14.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:498) > at sun.reflect.misc.MethodUtil.invoke(MethodUtil.java:275) > at > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:112) > at > com.sun.jmx.mbeanserver.StandardMBeanIntrospector.invokeM2(StandardMBeanIntrospector.java:46) > at > com.sun.jmx.mbeanserver.MBeanIntrospector.invokeM(MBeanIntrospector.java:237) > at > com.sun.jmx.mbeanserver.PerInterface.getAttribute(PerInterface.java:83) > at > com.sun.jmx.mbeanserver.MBeanSupport.getAttribute(MBeanSupport.java:206) > at javax.management.StandardMBean.getAttribute(StandardMBean.java:372) > at > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:647) > ... 42 more > 2019-07-16 19:35:35,228 [qtp1811922029-78] ERROR jmx.JMXJsonServlet > (JMXJsonServlet.java:writeAttribute(345)) - getting attribute > NumEnteringMaintenanceDataNodes of > Hadoop:service=NameNode,name=FSNamesystem-3 threw an exception > javax.management.RuntimeMBeanException: java.lang.NullPointerException > at > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrow(DefaultMBeanServerInterceptor.java:839) > at > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.rethrowMaybeMBeanException(DefaultMBeanServerInterceptor.java:852) > at > com.sun.jmx.interceptor.DefaultMBeanServerInterceptor.getAttribute(DefaultMBeanServerInterceptor.java:651) > at > com.sun.jmx.mbeanserver.JmxMBeanServer.getAttribute(JmxMBeanServer.java:678) > at > org.apache.hadoop.jmx.JMXJsonServlet.writeAttribute(JMXJsonServlet.java:338) > at > org.apache.hadoop.jmx.JMXJsonServlet.listBeans(JMXJsonServlet.java:316) > at org.apache.hadoop.jmx.JMXJsonServlet.doGet(JMXJsonServlet.java:210) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:687) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:790) > at > org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:848) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1772) > at > org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:644) > at > org.apache.hadoop.security.authentication.server.ProxyUserAuthenticationFilter.doFilter(ProxyUserAuthenticationFilter.java:104) > at > org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:592) > at org.apache.hadoop.hdfs.web.AuthFilter.doFilter(AuthFilter.java:51) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) > at > org.apache.hadoop.http.lib.StaticUserWebFilter$StaticUserFilter.doFilter(StaticUserWebFilter.java:110) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) > at > org.apache.hadoop.http.HttpServer2$QuotingInputFilter.doFilter(HttpServer2.java:1604) > at > org.eclipse.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1759) > at
[jira] [Updated] (HDFS-14653) RBF: Correct the default value for dfs.federation.router.namenode.heartbeat.enable
[
https://issues.apache.org/jira/browse/HDFS-14653?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takanobu Asanuma updated HDFS-14653:
Resolution: Fixed
Fix Version/s: 3.3.0
Status: Resolved (was: Patch Available)
Committed to trunk. Thanks for your contribution, [~ayushtkn]!
> RBF: Correct the default value for
> dfs.federation.router.namenode.heartbeat.enable
> --
>
> Key: HDFS-14653
> URL: https://issues.apache.org/jira/browse/HDFS-14653
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HDFS-14653-01.patch, HDFS-14653-02.patch
>
>
> dfs.federation.router.namenode.heartbeat.enable is suppose to take the value
> of dfs.federation.router.heartbeat.enable when it isn't explicitly specified.
> by ::
> {noformat}
> boolean isRouterHeartbeatEnabled = conf.getBoolean(
> RBFConfigKeys.DFS_ROUTER_HEARTBEAT_ENABLE,
> RBFConfigKeys.DFS_ROUTER_HEARTBEAT_ENABLE_DEFAULT);
> boolean isNamenodeHeartbeatEnable = conf.getBoolean(
> RBFConfigKeys.DFS_ROUTER_NAMENODE_HEARTBEAT_ENABLE,
> isRouterHeartbeatEnabled);
> {noformat}
> But since now RBF-Defaults are added by default, this logic doesn't hold, and
> the value defaults to true.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14653) RBF: Correct the default value for dfs.federation.router.namenode.heartbeat.enable
[
https://issues.apache.org/jira/browse/HDFS-14653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16887526#comment-16887526
]
Takanobu Asanuma commented on HDFS-14653:
-
Surely, the default value should be empty to follow the description. +1.
Will commit it soon.
> RBF: Correct the default value for
> dfs.federation.router.namenode.heartbeat.enable
> --
>
> Key: HDFS-14653
> URL: https://issues.apache.org/jira/browse/HDFS-14653
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Ayush Saxena
>Assignee: Ayush Saxena
>Priority: Major
> Attachments: HDFS-14653-01.patch, HDFS-14653-02.patch
>
>
> dfs.federation.router.namenode.heartbeat.enable is suppose to take the value
> of dfs.federation.router.heartbeat.enable when it isn't explicitly specified.
> by ::
> {noformat}
> boolean isRouterHeartbeatEnabled = conf.getBoolean(
> RBFConfigKeys.DFS_ROUTER_HEARTBEAT_ENABLE,
> RBFConfigKeys.DFS_ROUTER_HEARTBEAT_ENABLE_DEFAULT);
> boolean isNamenodeHeartbeatEnable = conf.getBoolean(
> RBFConfigKeys.DFS_ROUTER_NAMENODE_HEARTBEAT_ENABLE,
> isRouterHeartbeatEnabled);
> {noformat}
> But since now RBF-Defaults are added by default, this logic doesn't hold, and
> the value defaults to true.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13647) Fix the description of storageType option for space quota
[
https://issues.apache.org/jira/browse/HDFS-13647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16887519#comment-16887519
]
Takanobu Asanuma commented on HDFS-13647:
-
Thanks for reviewing and committing it, [~ayushtkn]!
> Fix the description of storageType option for space quota
> -
>
> Key: HDFS-13647
> URL: https://issues.apache.org/jira/browse/HDFS-13647
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation, tools
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HDFS-13647.1.patch, HDFS-13647.2.patch
>
>
> This is the current description.
> {noformat}
> $ hdfs dfsadmin -help setSpaceQuota
> ...
> Available storageTypes are
> - RAM_DISK
> - DISK
> - SSD
> - ARCHIVE
> {noformat}
> Actually, space quota doesn't support {{RAM_DISK}} and support {{PROVIDED}}.
> Looks like the document has a same bug.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13647) Fix the description of storageType option for space quota
[
https://issues.apache.org/jira/browse/HDFS-13647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16886783#comment-16886783
]
Takanobu Asanuma commented on HDFS-13647:
-
Thanks for catching up and reviewing it, [~ayushtkn]!
Uploaded the 2nd patch addressing your comment.
> Fix the description of storageType option for space quota
> -
>
> Key: HDFS-13647
> URL: https://issues.apache.org/jira/browse/HDFS-13647
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation, tools
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Attachments: HDFS-13647.1.patch, HDFS-13647.2.patch
>
>
> This is the current description.
> {noformat}
> $ hdfs dfsadmin -help setSpaceQuota
> ...
> Available storageTypes are
> - RAM_DISK
> - DISK
> - SSD
> - ARCHIVE
> {noformat}
> Actually, space quota doesn't support {{RAM_DISK}} and support {{PROVIDED}}.
> Looks like the document has a same bug.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13647) Fix the description of storageType option for space quota
[
https://issues.apache.org/jira/browse/HDFS-13647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Takanobu Asanuma updated HDFS-13647:
Attachment: HDFS-13647.2.patch
> Fix the description of storageType option for space quota
> -
>
> Key: HDFS-13647
> URL: https://issues.apache.org/jira/browse/HDFS-13647
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: documentation, tools
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Attachments: HDFS-13647.1.patch, HDFS-13647.2.patch
>
>
> This is the current description.
> {noformat}
> $ hdfs dfsadmin -help setSpaceQuota
> ...
> Available storageTypes are
> - RAM_DISK
> - DISK
> - SSD
> - ARCHIVE
> {noformat}
> Actually, space quota doesn't support {{RAM_DISK}} and support {{PROVIDED}}.
> Looks like the document has a same bug.
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14593) RBF: Implement deletion feature for expired records in State Store
[ https://issues.apache.org/jira/browse/HDFS-14593?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16885752#comment-16885752 ] Takanobu Asanuma commented on HDFS-14593: - Created HDFS-14654 for the flaky test. > RBF: Implement deletion feature for expired records in State Store > -- > > Key: HDFS-14593 > URL: https://issues.apache.org/jira/browse/HDFS-14593 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: rbf >Reporter: Takanobu Asanuma >Assignee: Takanobu Asanuma >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-14593.001.patch, HDFS-14593.002.patch, > HDFS-14593.003.patch, HDFS-14593.004.patch, HDFS-14593.005.patch, > HDFS-14593.006.patch, HDFS-14593.007.patch, HDFS-14593.008.patch, > HDFS-14593.009.patch, HDFS-14593.010.patch, HDFS-14593.011.patch > > > Currently, any router seems to exist in the Router Information eternally. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14654) RBF: TestRouterRpc tests are flaky
[ https://issues.apache.org/jira/browse/HDFS-14654?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Takanobu Asanuma updated HDFS-14654: Attachment: error.log > RBF: TestRouterRpc tests are flaky > -- > > Key: HDFS-14654 > URL: https://issues.apache.org/jira/browse/HDFS-14654 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Takanobu Asanuma >Priority: Major > Attachments: error.log > > > They sometimes pass and sometimes fail. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14654) RBF: TestRouterRpc tests are flaky
[ https://issues.apache.org/jira/browse/HDFS-14654?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16885751#comment-16885751 ] Takanobu Asanuma commented on HDFS-14654: - Attached the error log. > RBF: TestRouterRpc tests are flaky > -- > > Key: HDFS-14654 > URL: https://issues.apache.org/jira/browse/HDFS-14654 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Takanobu Asanuma >Priority: Major > Attachments: error.log > > > They sometimes pass and sometimes fail. -- This message was sent by Atlassian JIRA (v7.6.14#76016) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
