[jira] [Updated] (HDFS-13995) RBF: Security documentation
[ https://issues.apache.org/jira/browse/HDFS-13995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] CR Hota updated HDFS-13995: --- Attachment: HDFS-13995-HDFS-13891.003.patch > RBF: Security documentation > --- > > Key: HDFS-13995 > URL: https://issues.apache.org/jira/browse/HDFS-13995 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: CR Hota >Assignee: CR Hota >Priority: Major > Attachments: HDFS-13995-HDFS-13891.001.patch, > HDFS-13995-HDFS-13891.002.patch, HDFS-13995-HDFS-13891.003.patch > > > Documentation for users under the section relating to security needs to be > updated once security work is completely. > [https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs-rbf/HDFSRouterFederation.html#Security] > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14460) DFSUtil#getNamenodeWebAddr should return HTTPS address based on policy configured
[ https://issues.apache.org/jira/browse/HDFS-14460?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] CR Hota updated HDFS-14460: --- Attachment: HDFS-14460.003.patch > DFSUtil#getNamenodeWebAddr should return HTTPS address based on policy > configured > - > > Key: HDFS-14460 > URL: https://issues.apache.org/jira/browse/HDFS-14460 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: CR Hota >Assignee: CR Hota >Priority: Major > Attachments: HDFS-14460.001.patch, HDFS-14460.002.patch, > HDFS-14460.003.patch > > > DFSUtil#getNamenodeWebAddr does a look-up of HTTP address irrespective of > policy configured. It should instead look at the policy configured and return > appropriate web address. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14245) Class cast error in GetGroups with ObserverReadProxyProvider
[
https://issues.apache.org/jira/browse/HDFS-14245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830854#comment-16830854
]
Hadoop QA commented on HDFS-14245:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
12s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
14s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
15s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
40s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 5s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
15s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
37s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
36s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 39s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
51s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 80m 35s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}145m 46s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeHotSwapVolumes |
| | hadoop.hdfs.web.TestWebHdfsTimeouts |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-14245 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12967546/HDFS-14245.003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 9b73c76b819a 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 4877f0a |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit |
[jira] [Commented] (HDFS-14440) RBF: Optimize the file write process in case of multiple destinations.
[ https://issues.apache.org/jira/browse/HDFS-14440?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830843#comment-16830843 ] Ayush Saxena commented on HDFS-14440: - [~elgoiri] can you pls give a check? > RBF: Optimize the file write process in case of multiple destinations. > -- > > Key: HDFS-14440 > URL: https://issues.apache.org/jira/browse/HDFS-14440 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Ayush Saxena >Assignee: Ayush Saxena >Priority: Major > Attachments: HDFS-14440-HDFS-13891-01.patch, > HDFS-14440-HDFS-13891-02.patch, HDFS-14440-HDFS-13891-03.patch > > > In case of multiple destinations, We need to check if the file already exists > in one of the subclusters for which we use the existing getBlockLocation() > API which is by default a sequential Call, > In an ideal scenario where the file needs to be created each subcluster shall > be checked sequentially, this can be done concurrently to save time. > In another case where the file is found and if the last block is null, we > need to do getFileInfo to all the locations to get the location where the > file exists. This also can be prevented by use of ConcurrentCall since we > shall be having the remoteLocation to where the getBlockLocation returned a > non null entry. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14454) RBF: getContentSummary() should allow non-existing folders
[ https://issues.apache.org/jira/browse/HDFS-14454?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830842#comment-16830842 ] Ayush Saxena commented on HDFS-14454: - Thanx [~elgoiri] for the patch. v006 LGTM +1 Will push once the branch is confirmed to be OK. > RBF: getContentSummary() should allow non-existing folders > -- > > Key: HDFS-14454 > URL: https://issues.apache.org/jira/browse/HDFS-14454 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-14454-HDFS-13891.000.patch, > HDFS-14454-HDFS-13891.001.patch, HDFS-14454-HDFS-13891.002.patch, > HDFS-14454-HDFS-13891.003.patch, HDFS-14454-HDFS-13891.004.patch, > HDFS-14454-HDFS-13891.005.patch, HDFS-14454-HDFS-13891.006.patch > > > We have a mount point with HASH_ALL and one of the subclusters does not > contain the folder. > In this case, getContentSummary() returns FileNotFoundException. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13955) RBF: Support secure Namenode in NamenodeHeartbeatService
[ https://issues.apache.org/jira/browse/HDFS-13955?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830838#comment-16830838 ] Ayush Saxena commented on HDFS-13955: - Hi [~brahmareddy] While I was checking to find HDFS-14457, Acc. to me only that was lost, I compared the present state with the old state. That too due to the failed re base activity. Can you too give a check and confirm, when the branch is safe to push. > RBF: Support secure Namenode in NamenodeHeartbeatService > > > Key: HDFS-13955 > URL: https://issues.apache.org/jira/browse/HDFS-13955 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: CR Hota >Priority: Major > Attachments: HDFS-13955-HDFS-13532.000.patch, > HDFS-13955-HDFS-13532.001.patch > > > Currently, the NamenodeHeartbeatService uses JMX to get the metrics from the > Namenodes. We should support HTTPs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14463) Add Log Level link under NameNode and DataNode Web UI Utilities dropdown
[ https://issues.apache.org/jira/browse/HDFS-14463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830825#comment-16830825 ] Wei-Chiu Chuang commented on HDFS-14463: +1 > Add Log Level link under NameNode and DataNode Web UI Utilities dropdown > > > Key: HDFS-14463 > URL: https://issues.apache.org/jira/browse/HDFS-14463 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs >Affects Versions: 3.2.0, 3.1.2 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Trivial > Attachments: HDFS-14463.001.patch, dn_postpatch.png, nn_postpatch.png > > > Add Log Level link under NameNode and DataNode Web UI Utilities dropdown: > !nn_postpatch.png! > !dn_postpatch.png! > CC [~arpitagarwal] [~jojochuang] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14312) Scale test KMS using kms audit log
[ https://issues.apache.org/jira/browse/HDFS-14312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830824#comment-16830824 ] Wei-Chiu Chuang commented on HDFS-14312: I've been working on this in the past week and got a fair good result. So assign it to me and will post the patch after I polish it a little more. > Scale test KMS using kms audit log > -- > > Key: HDFS-14312 > URL: https://issues.apache.org/jira/browse/HDFS-14312 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: kms >Affects Versions: 3.3.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > > It appears to me that Dynamometer's architecture allows KMS scale tests too. > I imagine there are two ways to scale test a KMS. > # Take KMS audit logs, and replay the logs against a KMS. > # Configure Dynamometer to start KMS in addition to NameNode. Assuming the > fsimage comes from an encrypted cluster, replaying HDFS audit log also tests > KMS. > It would be even more interesting to have a tool that converts uncrypted > cluster fsimage to an encrypted one. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14312) KMS-o-meter: Scale test KMS using kms audit log
[ https://issues.apache.org/jira/browse/HDFS-14312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-14312: --- Issue Type: New Feature (was: Sub-task) Parent: (was: HDFS-14281) > KMS-o-meter: Scale test KMS using kms audit log > --- > > Key: HDFS-14312 > URL: https://issues.apache.org/jira/browse/HDFS-14312 > Project: Hadoop HDFS > Issue Type: New Feature > Components: kms >Affects Versions: 3.3.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > > It appears to me that Dynamometer's architecture allows KMS scale tests too. > I imagine there are two ways to scale test a KMS. > # Take KMS audit logs, and replay the logs against a KMS. > # Configure Dynamometer to start KMS in addition to NameNode. Assuming the > fsimage comes from an encrypted cluster, replaying HDFS audit log also tests > KMS. > It would be even more interesting to have a tool that converts uncrypted > cluster fsimage to an encrypted one. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14312) KMS-o-meter: Scale test KMS using kms audit log
[ https://issues.apache.org/jira/browse/HDFS-14312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-14312: --- Target Version/s: 3.3.0 > KMS-o-meter: Scale test KMS using kms audit log > --- > > Key: HDFS-14312 > URL: https://issues.apache.org/jira/browse/HDFS-14312 > Project: Hadoop HDFS > Issue Type: New Feature > Components: kms >Affects Versions: 3.3.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > > It appears to me that Dynamometer's architecture allows KMS scale tests too. > I imagine there are two ways to scale test a KMS. > # Take KMS audit logs, and replay the logs against a KMS. > # Configure Dynamometer to start KMS in addition to NameNode. Assuming the > fsimage comes from an encrypted cluster, replaying HDFS audit log also tests > KMS. > It would be even more interesting to have a tool that converts uncrypted > cluster fsimage to an encrypted one. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14312) KMS-o-meter: Scale test KMS using kms audit log
[ https://issues.apache.org/jira/browse/HDFS-14312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang updated HDFS-14312: --- Summary: KMS-o-meter: Scale test KMS using kms audit log (was: Scale test KMS using kms audit log) > KMS-o-meter: Scale test KMS using kms audit log > --- > > Key: HDFS-14312 > URL: https://issues.apache.org/jira/browse/HDFS-14312 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: kms >Affects Versions: 3.3.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > > It appears to me that Dynamometer's architecture allows KMS scale tests too. > I imagine there are two ways to scale test a KMS. > # Take KMS audit logs, and replay the logs against a KMS. > # Configure Dynamometer to start KMS in addition to NameNode. Assuming the > fsimage comes from an encrypted cluster, replaying HDFS audit log also tests > KMS. > It would be even more interesting to have a tool that converts uncrypted > cluster fsimage to an encrypted one. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work started] (HDFS-14312) Scale test KMS using kms audit log
[ https://issues.apache.org/jira/browse/HDFS-14312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-14312 started by Wei-Chiu Chuang. -- > Scale test KMS using kms audit log > -- > > Key: HDFS-14312 > URL: https://issues.apache.org/jira/browse/HDFS-14312 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: kms >Affects Versions: 3.3.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > > It appears to me that Dynamometer's architecture allows KMS scale tests too. > I imagine there are two ways to scale test a KMS. > # Take KMS audit logs, and replay the logs against a KMS. > # Configure Dynamometer to start KMS in addition to NameNode. Assuming the > fsimage comes from an encrypted cluster, replaying HDFS audit log also tests > KMS. > It would be even more interesting to have a tool that converts uncrypted > cluster fsimage to an encrypted one. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-14312) Scale test KMS using kms audit log
[ https://issues.apache.org/jira/browse/HDFS-14312?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei-Chiu Chuang reassigned HDFS-14312: -- Assignee: Wei-Chiu Chuang > Scale test KMS using kms audit log > -- > > Key: HDFS-14312 > URL: https://issues.apache.org/jira/browse/HDFS-14312 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: kms >Affects Versions: 3.3.0 >Reporter: Wei-Chiu Chuang >Assignee: Wei-Chiu Chuang >Priority: Major > > It appears to me that Dynamometer's architecture allows KMS scale tests too. > I imagine there are two ways to scale test a KMS. > # Take KMS audit logs, and replay the logs against a KMS. > # Configure Dynamometer to start KMS in addition to NameNode. Assuming the > fsimage comes from an encrypted cluster, replaying HDFS audit log also tests > KMS. > It would be even more interesting to have a tool that converts uncrypted > cluster fsimage to an encrypted one. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14245) Class cast error in GetGroups with ObserverReadProxyProvider
[
https://issues.apache.org/jira/browse/HDFS-14245?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Krogen updated HDFS-14245:
---
Attachment: HDFS-14245.003.patch
> Class cast error in GetGroups with ObserverReadProxyProvider
>
>
> Key: HDFS-14245
> URL: https://issues.apache.org/jira/browse/HDFS-14245
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: HDFS-12943
>Reporter: Shen Yinjie
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HDFS-14245.000.patch, HDFS-14245.001.patch,
> HDFS-14245.002.patch, HDFS-14245.003.patch, HDFS-14245.patch
>
>
> Run "hdfs groups" with ObserverReadProxyProvider, Exception throws as :
> {code:java}
> Exception in thread "main" java.io.IOException: Couldn't create proxy
> provider class
> org.apache.hadoop.hdfs.server.namenode.ha.ObserverReadProxyProvider
> at
> org.apache.hadoop.hdfs.NameNodeProxiesClient.createFailoverProxyProvider(NameNodeProxiesClient.java:261)
> at
> org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:119)
> at
> org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:95)
> at org.apache.hadoop.hdfs.tools.GetGroups.getUgmProtocol(GetGroups.java:87)
> at org.apache.hadoop.tools.GetGroupsBase.run(GetGroupsBase.java:71)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.hdfs.tools.GetGroups.main(GetGroups.java:96)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hdfs.NameNodeProxiesClient.createFailoverProxyProvider(NameNodeProxiesClient.java:245)
> ... 7 more
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hdfs.server.namenode.ha.NameNodeHAProxyFactory cannot be
> cast to org.apache.hadoop.hdfs.server.namenode.ha.ClientHAProxyFactory
> at
> org.apache.hadoop.hdfs.server.namenode.ha.ObserverReadProxyProvider.(ObserverReadProxyProvider.java:123)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.ObserverReadProxyProvider.(ObserverReadProxyProvider.java:112)
> ... 12 more
> {code}
> similar with HDFS-14116, we did a simple fix.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14245) Class cast error in GetGroups with ObserverReadProxyProvider
[
https://issues.apache.org/jira/browse/HDFS-14245?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830818#comment-16830818
]
Erik Krogen commented on HDFS-14245:
Thanks for taking a look [~shv]!
{quote}It would be better if getProxyAsClientProtocol() was throwing
IOException rather than RuntimeException.
{quote}
I'm not sure I agree with this. If the proxy is, in fact, not a
{{ClientProtocol}}, no level of retry will fix it. An {{IOException}} may
trigger failover or retry logic, which will just continue to fail. Also, it
indicates a bug, so it seems to me that it would be better to surface it rather
than hiding it under an {{IOException}} which is more likely to get ignored
(since {{IOExceptions}} are common). I can probably be convinced if you have
some better reasoning than me or if there is precedence for your approach.
{quote}It looks that getHAServiceState() in current revision assumes STANDBY
state no matter what error. I think it should only assume STANDBY state when it
gets StandbyException, and re-throw if anything else.
{quote}
I don't agree with this. Throwing an exception from here will actually trigger
failover of the active proxy, which is definitely not what we want. Assuming
{{STANDBY}} state will achieve the desired effect of no longer contacting this
node. Though something like {{UNAVAILABLE}} or {{UNREACHABLE}} may be more
accurate, I don't think adding a new {{HAServiceState}} makes sense for this
use case, and I think {{STANDBY}} is more applicable than any of the other
states:
{code:java}
INITIALIZING("initializing"),
ACTIVE("active"),
STANDBY("standby"),
OBSERVER("observer"),
STOPPING("stopping");
{code}
{quote}Also LOG.error() rather than info().
{quote}
I think a WARN may be reasonable, but I really don't think it's an ERROR. It
doesn't indicate anything fatal or broken; e.g. if one of the NameNodes is down
temporarily for maintenance you will get an {{IOException}} here. This is
expected and the client will just continue to move on to the next NameNode. I
think that the explanations for when to use different log levels provided in
the answers
[here|https://stackoverflow.com/questions/2031163/when-to-use-the-different-log-levels]
are pretty good, and I think this solidly does not fit into the category of an
ERROR.
I'm attaching a v003 patch which changes the log level to a WARN.
> Class cast error in GetGroups with ObserverReadProxyProvider
>
>
> Key: HDFS-14245
> URL: https://issues.apache.org/jira/browse/HDFS-14245
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: HDFS-12943
>Reporter: Shen Yinjie
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HDFS-14245.000.patch, HDFS-14245.001.patch,
> HDFS-14245.002.patch, HDFS-14245.patch
>
>
> Run "hdfs groups" with ObserverReadProxyProvider, Exception throws as :
> {code:java}
> Exception in thread "main" java.io.IOException: Couldn't create proxy
> provider class
> org.apache.hadoop.hdfs.server.namenode.ha.ObserverReadProxyProvider
> at
> org.apache.hadoop.hdfs.NameNodeProxiesClient.createFailoverProxyProvider(NameNodeProxiesClient.java:261)
> at
> org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:119)
> at
> org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:95)
> at org.apache.hadoop.hdfs.tools.GetGroups.getUgmProtocol(GetGroups.java:87)
> at org.apache.hadoop.tools.GetGroupsBase.run(GetGroupsBase.java:71)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:76)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:90)
> at org.apache.hadoop.hdfs.tools.GetGroups.main(GetGroups.java:96)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at
> org.apache.hadoop.hdfs.NameNodeProxiesClient.createFailoverProxyProvider(NameNodeProxiesClient.java:245)
> ... 7 more
> Caused by: java.lang.ClassCastException:
> org.apache.hadoop.hdfs.server.namenode.ha.NameNodeHAProxyFactory cannot be
> cast to org.apache.hadoop.hdfs.server.namenode.ha.ClientHAProxyFactory
> at
> org.apache.hadoop.hdfs.server.namenode.ha.ObserverReadProxyProvider.(ObserverReadProxyProvider.java:123)
> at
> org.apache.hadoop.hdfs.server.namenode.ha.ObserverReadProxyProvider.(ObserverReadProxyProvider.java:112)
> ... 12 more
> {code}
> similar with HDFS-14116, we did a simple fix.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (HDFS-14463) Add Log Level link under NameNode and DataNode Web UI Utilities dropdown
[
https://issues.apache.org/jira/browse/HDFS-14463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830811#comment-16830811
]
Hadoop QA commented on HDFS-14463:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 17m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
28m 2s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 54s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 42m 17s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-14463 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12967541/HDFS-14463.001.patch |
| Optional Tests | dupname asflicense shadedclient |
| uname | Linux 3ba1378a208a 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 4877f0a |
| maven | version: Apache Maven 3.3.9 |
| Max. process+thread count | 442 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
hadoop-hdfs-project/hadoop-hdfs |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/26734/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Add Log Level link under NameNode and DataNode Web UI Utilities dropdown
>
>
> Key: HDFS-14463
> URL: https://issues.apache.org/jira/browse/HDFS-14463
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: webhdfs
>Affects Versions: 3.2.0, 3.1.2
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Trivial
> Attachments: HDFS-14463.001.patch, dn_postpatch.png, nn_postpatch.png
>
>
> Add Log Level link under NameNode and DataNode Web UI Utilities dropdown:
> !nn_postpatch.png!
> !dn_postpatch.png!
> CC [~arpitagarwal] [~jojochuang]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14463) Add Log Level link under NameNode and DataNode Web UI Utilities dropdown
[
https://issues.apache.org/jira/browse/HDFS-14463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830800#comment-16830800
]
Hadoop QA commented on HDFS-14463:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
0s{color} | {color:blue} Docker mode activated. {color} |
| {color:red}-1{color} | {color:red} patch {color} | {color:red} 0m 6s{color}
| {color:red} HDFS-14463 does not apply to trunk. Rebase required? Wrong
Branch? See https://wiki.apache.org/hadoop/HowToContribute for help. {color} |
\\
\\
|| Subsystem || Report/Notes ||
| JIRA Issue | HDFS-14463 |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/26735/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> Add Log Level link under NameNode and DataNode Web UI Utilities dropdown
>
>
> Key: HDFS-14463
> URL: https://issues.apache.org/jira/browse/HDFS-14463
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: webhdfs
>Affects Versions: 3.2.0, 3.1.2
>Reporter: Siyao Meng
>Assignee: Siyao Meng
>Priority: Trivial
> Attachments: HDFS-14463.001.patch, dn_postpatch.png, nn_postpatch.png
>
>
> Add Log Level link under NameNode and DataNode Web UI Utilities dropdown:
> !nn_postpatch.png!
> !dn_postpatch.png!
> CC [~arpitagarwal] [~jojochuang]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14463) Add Log Level link under NameNode and DataNode Web UI Utilities dropdown
[ https://issues.apache.org/jira/browse/HDFS-14463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-14463: -- Description: Add Log Level link under NameNode and DataNode Web UI Utilities dropdown: !nn_postpatch.png! !dn_postpatch.png! CC [~arpitagarwal] [~jojochuang] was:A > Add Log Level link under NameNode and DataNode Web UI Utilities dropdown > > > Key: HDFS-14463 > URL: https://issues.apache.org/jira/browse/HDFS-14463 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs >Affects Versions: 3.2.0, 3.1.2 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Trivial > Attachments: HDFS-14463.001.patch, dn_postpatch.png, nn_postpatch.png > > > Add Log Level link under NameNode and DataNode Web UI Utilities dropdown: > !nn_postpatch.png! > !dn_postpatch.png! > CC [~arpitagarwal] [~jojochuang] -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14463) Add Log Level link under NameNode and DataNode Web UI Utilities dropdown
[ https://issues.apache.org/jira/browse/HDFS-14463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-14463: -- Attachment: dn_postpatch.png > Add Log Level link under NameNode and DataNode Web UI Utilities dropdown > > > Key: HDFS-14463 > URL: https://issues.apache.org/jira/browse/HDFS-14463 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs >Affects Versions: 3.2.0, 3.1.2 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Trivial > Attachments: HDFS-14463.001.patch, dn_postpatch.png, nn_postpatch.png > > > A -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13995) RBF: Security documentation
[ https://issues.apache.org/jira/browse/HDFS-13995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830799#comment-16830799 ] Íñigo Goiri commented on HDFS-13995: Thanks [~crh] for the updated patch. In terms of content I think this is good. A couple minor style comments: * Use HDFS in capitals instead of hdfs. * Use the single quote to quote core-site.xml and hdfs-site.xml to mark them being files. > RBF: Security documentation > --- > > Key: HDFS-13995 > URL: https://issues.apache.org/jira/browse/HDFS-13995 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: CR Hota >Assignee: CR Hota >Priority: Major > Attachments: HDFS-13995-HDFS-13891.001.patch, > HDFS-13995-HDFS-13891.002.patch > > > Documentation for users under the section relating to security needs to be > updated once security work is completely. > [https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs-rbf/HDFSRouterFederation.html#Security] > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14463) Add Log Level link under NameNode and DataNode Web UI Utilities dropdown
[ https://issues.apache.org/jira/browse/HDFS-14463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-14463: -- Attachment: nn_postpatch.png > Add Log Level link under NameNode and DataNode Web UI Utilities dropdown > > > Key: HDFS-14463 > URL: https://issues.apache.org/jira/browse/HDFS-14463 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs >Affects Versions: 3.2.0, 3.1.2 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Trivial > Attachments: HDFS-14463.001.patch, nn_postpatch.png > > > A -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14463) Add Log Level link under NameNode and DataNode Web UI Utilities dropdown
[ https://issues.apache.org/jira/browse/HDFS-14463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-14463: -- Description: A > Add Log Level link under NameNode and DataNode Web UI Utilities dropdown > > > Key: HDFS-14463 > URL: https://issues.apache.org/jira/browse/HDFS-14463 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs >Affects Versions: 3.2.0, 3.1.2 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Trivial > Attachments: HDFS-14463.001.patch, nn_postpatch.png > > > A -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14460) DFSUtil#getNamenodeWebAddr should return HTTPS address based on policy configured
[
https://issues.apache.org/jira/browse/HDFS-14460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830797#comment-16830797
]
Íñigo Goiri commented on HDFS-14460:
My bad... I wrote the same.
{{getHttpPolicy()}} instead of {{getHttpClientScheme()}}.
The second just uses the first one.
I think it is better to just use the root one with the enum instead of the
filtered by string.
Potentially you can use a switch/case but up to you there.
> DFSUtil#getNamenodeWebAddr should return HTTPS address based on policy
> configured
> -
>
> Key: HDFS-14460
> URL: https://issues.apache.org/jira/browse/HDFS-14460
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: CR Hota
>Assignee: CR Hota
>Priority: Major
> Attachments: HDFS-14460.001.patch, HDFS-14460.002.patch
>
>
> DFSUtil#getNamenodeWebAddr does a look-up of HTTP address irrespective of
> policy configured. It should instead look at the policy configured and return
> appropriate web address.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14463) Add Log Level link under NameNode and DataNode Web UI Utilities dropdown
[ https://issues.apache.org/jira/browse/HDFS-14463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-14463: -- Attachment: HDFS-14463.001.patch Status: Patch Available (was: Open) Uploaded patch 001. Manually tested that the links work. > Add Log Level link under NameNode and DataNode Web UI Utilities dropdown > > > Key: HDFS-14463 > URL: https://issues.apache.org/jira/browse/HDFS-14463 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs >Affects Versions: 3.1.2, 3.2.0 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Trivial > Attachments: HDFS-14463.001.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14463) Add Log Level link under NameNode and DataNode Web UI Utilities dropdown
[ https://issues.apache.org/jira/browse/HDFS-14463?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Siyao Meng updated HDFS-14463: -- Component/s: webhdfs > Add Log Level link under NameNode and DataNode Web UI Utilities dropdown > > > Key: HDFS-14463 > URL: https://issues.apache.org/jira/browse/HDFS-14463 > Project: Hadoop HDFS > Issue Type: Improvement > Components: webhdfs >Affects Versions: 3.2.0, 3.1.2 >Reporter: Siyao Meng >Assignee: Siyao Meng >Priority: Trivial > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14463) Add Log Level link under NameNode and DataNode Web UI Utilities dropdown
Siyao Meng created HDFS-14463: - Summary: Add Log Level link under NameNode and DataNode Web UI Utilities dropdown Key: HDFS-14463 URL: https://issues.apache.org/jira/browse/HDFS-14463 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 3.1.2, 3.2.0 Reporter: Siyao Meng Assignee: Siyao Meng -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1473) DataNode ID file should be human readable
[ https://issues.apache.org/jira/browse/HDDS-1473?focusedWorklogId=235618=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-235618 ] ASF GitHub Bot logged work on HDDS-1473: Author: ASF GitHub Bot Created on: 01/May/19 00:38 Start Date: 01/May/19 00:38 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #781: HDDS-1473. DataNode ID file should be human readable. URL: https://github.com/apache/hadoop/pull/781#issuecomment-488163217 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 23 | Docker mode activated. | ||| _ Prechecks _ | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 74 | Maven dependency ordering for branch | | +1 | mvninstall | 1320 | trunk passed | | +1 | compile | 1412 | trunk passed | | +1 | checkstyle | 166 | trunk passed | | +1 | mvnsite | 141 | trunk passed | | +1 | shadedclient | 1189 | branch has no errors when building and testing our client artifacts. | | 0 | findbugs | 0 | Skipped patched modules with no Java source: hadoop-ozone/integration-test | | +1 | findbugs | 60 | trunk passed | | +1 | javadoc | 62 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 20 | Maven dependency ordering for patch | | +1 | mvninstall | 67 | the patch passed | | +1 | compile | 935 | the patch passed | | +1 | javac | 935 | the patch passed | | +1 | checkstyle | 185 | the patch passed | | +1 | mvnsite | 90 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 747 | patch has no errors when building and testing our client artifacts. | | 0 | findbugs | 0 | Skipped patched modules with no Java source: hadoop-ozone/integration-test | | +1 | findbugs | 65 | the patch passed | | +1 | javadoc | 61 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 72 | container-service in the patch failed. | | -1 | unit | 807 | integration-test in the patch failed. | | +1 | asflicense | 47 | The patch does not generate ASF License warnings. | | | | 7503 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.om.TestOmInit | | | hadoop.hdds.scm.container.TestContainerStateManagerIntegration | | | hadoop.ozone.client.rpc.TestBlockOutputStream | | | hadoop.ozone.client.rpc.TestBCSID | | | hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException | | | hadoop.hdds.scm.pipeline.TestNode2PipelineMap | | | hadoop.ozone.om.TestOzoneManagerRestInterface | | | hadoop.ozone.web.client.TestKeys | | | hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis | | | hadoop.ozone.container.common.statemachine.commandhandler.TestCloseContainerByPipeline | | | hadoop.ozone.om.TestOzoneManagerHA | | | hadoop.ozone.client.rpc.TestContainerStateMachineFailures | | | hadoop.hdds.scm.safemode.TestSCMSafeModeWithPipelineRules | | | hadoop.ozone.container.TestContainerReplication | | | hadoop.ozone.om.TestOMDbCheckpointServlet | | | hadoop.ozone.om.TestOmMetrics | | | hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient | | | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient | | | hadoop.ozone.client.rpc.TestCommitWatcher | | | hadoop.ozone.client.rpc.TestReadRetries | | | hadoop.ozone.container.ozoneimpl.TestOzoneContainer | | | hadoop.ozone.client.rpc.TestContainerStateMachine | | | hadoop.ozone.client.rpc.TestHybridPipelineOnDatanode | | | hadoop.ozone.scm.node.TestSCMNodeMetrics | | | hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures | | | hadoop.ozone.TestContainerOperations | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-781/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/781 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 19a3851ea1f3 4.4.0-141-generic #167~14.04.1-Ubuntu SMP Mon Dec 10 13:20:24 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 4877f0a | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-781/4/artifact/out/patch-unit-hadoop-hdds_container-service.txt | | unit |
[jira] [Work logged] (HDDS-1475) Fix OzoneContainer start method
[ https://issues.apache.org/jira/browse/HDDS-1475?focusedWorklogId=235607=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-235607 ] ASF GitHub Bot logged work on HDDS-1475: Author: ASF GitHub Bot Created on: 01/May/19 00:06 Start Date: 01/May/19 00:06 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #788: HDDS-1475 : Fix OzoneContainer start method. URL: https://github.com/apache/hadoop/pull/788#issuecomment-488158080 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 27 | Docker mode activated. | ||| _ Prechecks _ | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 25 | Maven dependency ordering for branch | | +1 | mvninstall | 1040 | trunk passed | | +1 | compile | 975 | trunk passed | | +1 | checkstyle | 141 | trunk passed | | +1 | mvnsite | 158 | trunk passed | | +1 | shadedclient | 1034 | branch has no errors when building and testing our client artifacts. | | 0 | findbugs | 0 | Skipped patched modules with no Java source: hadoop-ozone/integration-test | | +1 | findbugs | 61 | trunk passed | | +1 | javadoc | 72 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 24 | Maven dependency ordering for patch | | +1 | mvninstall | 69 | the patch passed | | +1 | compile | 921 | the patch passed | | +1 | javac | 921 | the patch passed | | +1 | checkstyle | 139 | the patch passed | | +1 | mvnsite | 91 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 695 | patch has no errors when building and testing our client artifacts. | | 0 | findbugs | 0 | Skipped patched modules with no Java source: hadoop-ozone/integration-test | | +1 | findbugs | 70 | the patch passed | | +1 | javadoc | 72 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 67 | container-service in the patch failed. | | -1 | unit | 615 | integration-test in the patch failed. | | +1 | asflicense | 55 | The patch does not generate ASF License warnings. | | | | 6330 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.om.TestOzoneManagerHA | | | hadoop.ozone.scm.TestContainerSmallFile | | | hadoop.hdds.scm.pipeline.TestNodeFailure | | | hadoop.hdds.scm.container.TestContainerStateManagerIntegration | | | hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis | | | hadoop.ozone.TestMiniOzoneCluster | | | hadoop.ozone.om.TestOMDbCheckpointServlet | | | hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient | | | hadoop.ozone.scm.node.TestSCMNodeMetrics | | | hadoop.ozone.TestStorageContainerManager | | | hadoop.ozone.web.client.TestBuckets | | | hadoop.ozone.om.TestOmMetrics | | | hadoop.ozone.web.client.TestKeysRatis | | | hadoop.hdds.scm.pipeline.TestPipelineClose | | | hadoop.ozone.scm.TestSCMMXBean | | | hadoop.hdds.scm.pipeline.TestSCMRestart | | | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.scm.TestAllocateContainer | | | hadoop.ozone.om.TestMultipleContainerReadWrite | | | hadoop.ozone.om.TestOmBlockVersioning | | | hadoop.ozone.client.rpc.TestBlockOutputStream | | | hadoop.ozone.web.TestOzoneWebAccess | | | hadoop.hdds.scm.pipeline.TestRatisPipelineUtils | | | hadoop.ozone.container.common.statemachine.commandhandler.TestCloseContainerByPipeline | | | hadoop.ozone.scm.pipeline.TestPipelineManagerMXBean | | | hadoop.ozone.scm.TestXceiverClientManager | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-788/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/788 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 3f12e314450c 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 4877f0a | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-788/2/artifact/out/patch-unit-hadoop-hdds_container-service.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-788/2/artifact/out/patch-unit-hadoop-ozone_integration-test.txt | | Test Results |
[jira] [Work logged] (HDDS-1473) DataNode ID file should be human readable
[
https://issues.apache.org/jira/browse/HDDS-1473?focusedWorklogId=235594=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-235594
]
ASF GitHub Bot logged work on HDDS-1473:
Author: ASF GitHub Bot
Created on: 30/Apr/19 23:56
Start Date: 30/Apr/19 23:56
Worklog Time Spent: 10m
Work Description: swagle commented on pull request #781: HDDS-1473.
DataNode ID file should be human readable.
URL: https://github.com/apache/hadoop/pull/781#discussion_r279978215
##
File path:
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/common/helpers/DatanodeIdYaml.java
##
@@ -0,0 +1,186 @@
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.hadoop.ozone.container.common.helpers;
+
+import java.io.File;
+import java.io.FileInputStream;
+import java.io.FileOutputStream;
+import java.io.IOException;
+import java.io.OutputStreamWriter;
+import java.io.Writer;
+import java.util.LinkedHashMap;
+import java.util.Map;
+
+import org.apache.commons.collections.CollectionUtils;
+import org.apache.commons.collections.MapUtils;
+import org.apache.hadoop.hdds.protocol.DatanodeDetails;
+import org.yaml.snakeyaml.DumperOptions;
+import org.yaml.snakeyaml.Yaml;
+
+import com.google.common.base.Preconditions;
+
+/**
+ * Class for creating datanode.id file in yaml format.
+ */
+public class DatanodeIdYaml {
+
+ private DatanodeIdYaml() {
+
+ }
+
+ /**
+ * Creates a yaml file using DatnodeDetails. This method expects the path
+ * validation to be performed by the caller.
+ *
+ * @param datanodeDetails {@link DatanodeDetails}
+ * @param pathPath to datnode.id file
+ */
+ public static void createDatanodeIdFile(DatanodeDetails datanodeDetails,
+ File path) throws IOException {
+Preconditions.checkNotNull(path);
+DumperOptions options = new DumperOptions();
+options.setPrettyFlow(true);
+options.setDefaultFlowStyle(DumperOptions.FlowStyle.FLOW);
+Yaml yaml = new Yaml(options);
+
+try (Writer writer = new OutputStreamWriter(
+new FileOutputStream(path), "UTF-8")) {
+ yaml.dump(getDatanodeDetailsYaml(datanodeDetails), writer);
+}
+ }
+
+ /**
+ * Read datanode.id from file.
+ */
+ public static DatanodeDetails readDatanodeIdFile(File path)
+ throws IOException {
+Preconditions.checkNotNull(path);
+DatanodeDetails datanodeDetails;
+try (FileInputStream inputFileStream = new FileInputStream(path)) {
+ Yaml yaml = new Yaml();
+ DatanodeDetailsYaml datanodeDetailsYaml;
+ try {
+datanodeDetailsYaml =
+yaml.loadAs(inputFileStream, DatanodeDetailsYaml.class);
+ } catch (Exception e) {
+throw new IOException("Unable to parse yaml file.", e);
+ }
+
+ DatanodeDetails.Builder builder = DatanodeDetails.newBuilder();
+ builder.setUuid(datanodeDetailsYaml.getUuid().toString())
+ .setIpAddress(datanodeDetailsYaml.getIpAddress())
Review comment:
Created HDDS-1480 to track.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 235594)
Time Spent: 2.5h (was: 2h 20m)
> DataNode ID file should be human readable
> -
>
> Key: HDDS-1473
> URL: https://issues.apache.org/jira/browse/HDDS-1473
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Datanode
>Reporter: Arpit Agarwal
>Assignee: Siddharth Wagle
>Priority: Major
> Labels: newbie, pull-request-available
> Time Spent: 2.5h
> Remaining Estimate: 0h
>
> The DataNode ID file should be human readable to make debugging easier. We
>
[jira] [Updated] (HDDS-1480) Ip address should not be a part of the DatanodeID since it can change
[
https://issues.apache.org/jira/browse/HDDS-1480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Wagle updated HDDS-1480:
--
Labels: newbie (was: )
> Ip address should not be a part of the DatanodeID since it can change
> -
>
> Key: HDDS-1480
> URL: https://issues.apache.org/jira/browse/HDDS-1480
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Datanode
>Affects Versions: 0.3.0
>Reporter: Siddharth Wagle
>Priority: Major
> Labels: newbie
>
> The DatanodeID identified by the DatanodeDetails object is persisted to disk
> and read back on restart. The following fields are currently being serialized
> and we should omit ip address from this set.
> {quote}
> UUID uuid;
> String ipAddress;
> String hostName;
> List ports;
> String certSerialId;
> {quote}
> cc: [~arpaga] this is follow-up from HDDS-1473
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1480) Ip address should not be a part of the DatanodeID since it can change
[
https://issues.apache.org/jira/browse/HDDS-1480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Wagle updated HDDS-1480:
--
Description:
The DatanodeID identified by the DatanodeDetails object is persisted to disk
and read back on restart. The following fields are currently being serialized
and we should omit ip address from this set.
{quote}
UUID uuid;
String ipAddress;
String hostName;
List ports;
String certSerialId;
{quote}
cc: [~arpaga] this is follow-up from HDDS-1473
was:
The DatanodeID identified by the DatanodeDetails object is persisted to disk
and read back on restart. The following fields are currently being serialized
and we should omit ip address from this set.
{quote}
UUID uuid;
String ipAddress;
String hostName;
List ports;
String certSerialId;
{quote}
cc: [~arpaga]
> Ip address should not be a part of the DatanodeID since it can change
> -
>
> Key: HDDS-1480
> URL: https://issues.apache.org/jira/browse/HDDS-1480
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Datanode
>Affects Versions: 0.3.0
>Reporter: Siddharth Wagle
>Priority: Major
>
> The DatanodeID identified by the DatanodeDetails object is persisted to disk
> and read back on restart. The following fields are currently being serialized
> and we should omit ip address from this set.
> {quote}
> UUID uuid;
> String ipAddress;
> String hostName;
> List ports;
> String certSerialId;
> {quote}
> cc: [~arpaga] this is follow-up from HDDS-1473
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1480) Ip address should not be a part of the DatanodeID since it can change
[
https://issues.apache.org/jira/browse/HDDS-1480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Siddharth Wagle updated HDDS-1480:
--
Description:
The DatanodeID identified by the DatanodeDetails object is persisted to disk
and read back on restart. The following fields are currently being serialized
and we should omit ip address from this set.
{quote}
UUID uuid;
String ipAddress;
String hostName;
List ports;
String certSerialId;
{quote}
cc: [~arpaga]
was:
The DatanodeID identified by the DatanodeDetails object is persisted to disk
and read back on restart. The following fields are currently being serialized
and we should omit ip address from this set.
{quote}
UUID uuid;
String ipAddress;
String hostName;
List ports;
String certSerialId;
{quote}
> Ip address should not be a part of the DatanodeID since it can change
> -
>
> Key: HDDS-1480
> URL: https://issues.apache.org/jira/browse/HDDS-1480
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Datanode
>Affects Versions: 0.3.0
>Reporter: Siddharth Wagle
>Priority: Major
>
> The DatanodeID identified by the DatanodeDetails object is persisted to disk
> and read back on restart. The following fields are currently being serialized
> and we should omit ip address from this set.
> {quote}
> UUID uuid;
> String ipAddress;
> String hostName;
> List ports;
> String certSerialId;
> {quote}
> cc: [~arpaga]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1480) Ip address should not be a part of the DatanodeID since it can change
Siddharth Wagle created HDDS-1480:
-
Summary: Ip address should not be a part of the DatanodeID since
it can change
Key: HDDS-1480
URL: https://issues.apache.org/jira/browse/HDDS-1480
Project: Hadoop Distributed Data Store
Issue Type: Improvement
Components: Ozone Datanode
Affects Versions: 0.3.0
Reporter: Siddharth Wagle
The DatanodeID identified by the DatanodeDetails object is persisted to disk
and read back on restart. The following fields are currently being serialized
and we should omit ip address from this set.
{quote}
UUID uuid;
String ipAddress;
String hostName;
List ports;
String certSerialId;
{quote}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1384) TestBlockOutputStreamWithFailures is failing
[
https://issues.apache.org/jira/browse/HDDS-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jitendra Nath Pandey updated HDDS-1384:
---
Fix Version/s: 0.5.0
> TestBlockOutputStreamWithFailures is failing
>
>
> Key: HDDS-1384
> URL: https://issues.apache.org/jira/browse/HDDS-1384
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Reporter: Nanda kumar
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Fix For: 0.5.0
>
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> TestBlockOutputStreamWithFailures is failing with the following error
> {noformat}
> 2019-04-04 18:52:43,240 INFO volume.ThrottledAsyncChecker
> (ThrottledAsyncChecker.java:schedule(140)) - Scheduling a check for
> org.apache.hadoop.ozone.container.common.volume.HddsVolume@1f6c0e8a
> 2019-04-04 18:52:43,240 INFO volume.HddsVolumeChecker
> (HddsVolumeChecker.java:checkAllVolumes(203)) - Scheduled health check for
> volume org.apache.hadoop.ozone.container.common.volume.HddsVolume@1f6c0e8a
> 2019-04-04 18:52:43,241 ERROR server.GrpcService
> (ExitUtils.java:terminate(133)) - Terminating with exit status 1: Failed to
> start Grpc server
> java.io.IOException: Failed to bind
> at
> org.apache.ratis.thirdparty.io.grpc.netty.NettyServer.start(NettyServer.java:253)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl.start(ServerImpl.java:166)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl.start(ServerImpl.java:81)
> at org.apache.ratis.grpc.server.GrpcService.startImpl(GrpcService.java:144)
> at org.apache.ratis.util.LifeCycle.startAndTransition(LifeCycle.java:202)
> at
> org.apache.ratis.server.impl.RaftServerRpcWithProxy.start(RaftServerRpcWithProxy.java:69)
> at
> org.apache.ratis.server.impl.RaftServerProxy.lambda$start$3(RaftServerProxy.java:300)
> at org.apache.ratis.util.LifeCycle.startAndTransition(LifeCycle.java:202)
> at
> org.apache.ratis.server.impl.RaftServerProxy.start(RaftServerProxy.java:298)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.start(XceiverServerRatis.java:419)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer.start(OzoneContainer.java:186)
> at
> org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.start(DatanodeStateMachine.java:169)
> at
> org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.lambda$startDaemon$0(DatanodeStateMachine.java:338)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:433)
> at sun.nio.ch.Net.bind(Net.java:425)
> at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
> at
> org.apache.ratis.thirdparty.io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:130)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:558)
> at
> org.apache.ratis.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1358)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:501)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:486)
> at
> org.apache.ratis.thirdparty.io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:1019)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel.bind(AbstractChannel.java:254)
> at
> org.apache.ratis.thirdparty.io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:366)
> at
> org.apache.ratis.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
> at
> org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
> at
> org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462)
> at
> org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
> at
> org.apache.ratis.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> ... 1 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1395) Key write fails with BlockOutputStream has been closed exception
[
https://issues.apache.org/jira/browse/HDDS-1395?focusedWorklogId=235583=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-235583
]
ASF GitHub Bot logged work on HDDS-1395:
Author: ASF GitHub Bot
Created on: 30/Apr/19 23:06
Start Date: 30/Apr/19 23:06
Worklog Time Spent: 10m
Work Description: jnp commented on issue #749: HDDS-1395. Key write fails
with BlockOutputStream has been closed exception
URL: https://github.com/apache/hadoop/pull/749#issuecomment-488146825
+1 for the patch, if the test failures are unrelated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 235583)
Time Spent: 4h (was: 3h 50m)
> Key write fails with BlockOutputStream has been closed exception
>
>
> Key: HDDS-1395
> URL: https://issues.apache.org/jira/browse/HDDS-1395
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Client
>Affects Versions: 0.4.0
>Reporter: Mukul Kumar Singh
>Assignee: Shashikant Banerjee
>Priority: Major
> Labels: MiniOzoneChaosCluster, pull-request-available
> Attachments: HDDS-1395.000.patch, HDDS-1395.001.patch
>
> Time Spent: 4h
> Remaining Estimate: 0h
>
> Key write fails with BlockOutputStream has been closed
> {code}
> 2019-04-05 11:24:47,770 ERROR ozone.MiniOzoneLoadGenerator
> (MiniOzoneLoadGenerator.java:load(102)) - LOADGEN: Create
> key:pool-431-thread-9-2092651262 failed with exception, but skipping
> java.io.IOException: BlockOutputStream has been closed.
> at
> org.apache.hadoop.hdds.scm.storage.BlockOutputStream.checkOpen(BlockOutputStream.java:662)
> at
> org.apache.hadoop.hdds.scm.storage.BlockOutputStream.write(BlockOutputStream.java:245)
> at
> org.apache.hadoop.ozone.client.io.BlockOutputStreamEntry.write(BlockOutputStreamEntry.java:131)
> at
> org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:325)
> at
> org.apache.hadoop.ozone.client.io.KeyOutputStream.write(KeyOutputStream.java:287)
> at
> org.apache.hadoop.ozone.client.io.OzoneOutputStream.write(OzoneOutputStream.java:49)
> at java.io.OutputStream.write(OutputStream.java:75)
> at
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.load(MiniOzoneLoadGenerator.java:100)
> at
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.lambda$startIO$0(MiniOzoneLoadGenerator.java:143)
> at
> java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1473) DataNode ID file should be human readable
[ https://issues.apache.org/jira/browse/HDDS-1473?focusedWorklogId=235575=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-235575 ] ASF GitHub Bot logged work on HDDS-1473: Author: ASF GitHub Bot Created on: 30/Apr/19 22:32 Start Date: 30/Apr/19 22:32 Worklog Time Spent: 10m Work Description: swagle commented on issue #781: HDDS-1473. DataNode ID file should be human readable. URL: https://github.com/apache/hadoop/pull/781#issuecomment-488139612 Thanks, @hanishakoneru for the review. Made checkstyle fixes in the last push. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 235575) Time Spent: 2h 20m (was: 2h 10m) > DataNode ID file should be human readable > - > > Key: HDDS-1473 > URL: https://issues.apache.org/jira/browse/HDDS-1473 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Reporter: Arpit Agarwal >Assignee: Siddharth Wagle >Priority: Major > Labels: newbie, pull-request-available > Time Spent: 2h 20m > Remaining Estimate: 0h > > The DataNode ID file should be human readable to make debugging easier. We > should use YAML as we have used it elsewhere for meta files. > Currently it is a binary file whose contents are protobuf encoded. This is a > tiny file read once on startup, so performance is not a concern. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1475) Fix OzoneContainer start method
[ https://issues.apache.org/jira/browse/HDDS-1475?focusedWorklogId=235572=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-235572 ] ASF GitHub Bot logged work on HDDS-1475: Author: ASF GitHub Bot Created on: 30/Apr/19 22:31 Start Date: 30/Apr/19 22:31 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #788: HDDS-1475 : Fix OzoneContainer start method. URL: https://github.com/apache/hadoop/pull/788#issuecomment-488139128 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 93 | Docker mode activated. | ||| _ Prechecks _ | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 66 | Maven dependency ordering for branch | | +1 | mvninstall | 1136 | trunk passed | | +1 | compile | 1295 | trunk passed | | +1 | checkstyle | 161 | trunk passed | | +1 | mvnsite | 97 | trunk passed | | +1 | shadedclient | 1009 | branch has no errors when building and testing our client artifacts. | | 0 | findbugs | 0 | Skipped patched modules with no Java source: hadoop-ozone/integration-test | | +1 | findbugs | 51 | trunk passed | | +1 | javadoc | 54 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 30 | Maven dependency ordering for patch | | +1 | mvninstall | 72 | the patch passed | | +1 | compile | 1077 | the patch passed | | +1 | javac | 1077 | the patch passed | | +1 | checkstyle | 139 | the patch passed | | +1 | mvnsite | 91 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 738 | patch has no errors when building and testing our client artifacts. | | 0 | findbugs | 0 | Skipped patched modules with no Java source: hadoop-ozone/integration-test | | +1 | findbugs | 75 | the patch passed | | +1 | javadoc | 82 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 97 | container-service in the patch failed. | | -1 | unit | 1540 | integration-test in the patch failed. | | +1 | asflicense | 58 | The patch does not generate ASF License warnings. | | | | 7988 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestOzoneRpcClient | | | hadoop.ozone.scm.node.TestQueryNode | | | hadoop.ozone.web.client.TestOzoneClient | | | hadoop.ozone.TestStorageContainerManager | | | hadoop.ozone.container.TestContainerReplication | | | hadoop.ozone.client.rpc.TestSecureOzoneRpcClient | | | hadoop.ozone.TestMiniChaosOzoneCluster | | | hadoop.ozone.om.TestContainerReportWithKeys | | | hadoop.ozone.om.TestMultipleContainerReadWrite | | | hadoop.hdds.scm.container.TestContainerStateManagerIntegration | | | hadoop.hdds.scm.pipeline.TestNode2PipelineMap | | | hadoop.ozone.TestContainerOperations | | | hadoop.ozone.scm.TestXceiverClientMetrics | | | hadoop.ozone.scm.TestAllocateContainer | | | hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis | | | hadoop.ozone.web.client.TestKeys | | | hadoop.ozone.scm.pipeline.TestSCMPipelineMetrics | | | hadoop.ozone.ozShell.TestOzoneDatanodeShell | | | hadoop.ozone.scm.TestGetCommittedBlockLengthAndPutKey | | | hadoop.hdds.scm.pipeline.TestRatisPipelineUtils | | | hadoop.ozone.om.TestOmAcls | | | hadoop.ozone.container.common.statemachine.commandhandler.TestCloseContainerByPipeline | | | hadoop.ozone.om.TestOmMetrics | | | hadoop.ozone.om.TestOMDbCheckpointServlet | | | hadoop.ozone.ozShell.TestS3Shell | | | hadoop.ozone.om.TestOzoneManagerConfiguration | | | hadoop.ozone.scm.TestXceiverClientManager | | | hadoop.hdds.scm.pipeline.TestPipelineClose | | | hadoop.ozone.web.TestOzoneRestWithMiniCluster | | | hadoop.ozone.scm.TestSCMMXBean | | | hadoop.ozone.om.TestScmSafeMode | | | hadoop.hdds.scm.safemode.TestSCMSafeModeWithPipelineRules | | | hadoop.ozone.ozShell.TestOzoneShell | | | hadoop.ozone.web.TestOzoneVolumes | | | hadoop.ozone.web.client.TestBuckets | | | hadoop.ozone.web.client.TestVolume | | | hadoop.ozone.client.rpc.TestCommitWatcher | | | hadoop.hdds.scm.pipeline.TestSCMRestart | | | hadoop.ozone.web.client.TestKeysRatis | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-788/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/788 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit
[jira] [Work logged] (HDDS-1475) Fix OzoneContainer start method
[
https://issues.apache.org/jira/browse/HDDS-1475?focusedWorklogId=235566=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-235566
]
ASF GitHub Bot logged work on HDDS-1475:
Author: ASF GitHub Bot
Created on: 30/Apr/19 22:20
Start Date: 30/Apr/19 22:20
Worklog Time Spent: 10m
Work Description: avijayanhwx commented on pull request #788: HDDS-1475 :
Fix OzoneContainer start method.
URL: https://github.com/apache/hadoop/pull/788#discussion_r279958400
##
File path:
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OzoneContainer.java
##
@@ -183,8 +183,12 @@ private void stopContainerScrub() {
public void start(String scmId) throws IOException {
LOG.info("Attempting to start container services.");
startContainerScrub();
Review comment:
Added.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 235566)
Time Spent: 50m (was: 40m)
> Fix OzoneContainer start method
> ---
>
> Key: HDDS-1475
> URL: https://issues.apache.org/jira/browse/HDDS-1475
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Reporter: Bharat Viswanadham
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: newbie, pull-request-available
> Time Spent: 50m
> Remaining Estimate: 0h
>
> In OzoneContainer start() we have
> {code:java}
> startContainerScrub();
> writeChannel.start();
> readChannel.start();
> hddsDispatcher.init();
> hddsDispatcher.setScmId(scmId);{code}
>
> Suppose here if readChannel.start() failed due to some reason, from
> VersionEndPointTask, we try to start OzoneContainer again. This can cause an
> issue for writeChannel.start() if it is already started.
>
> Fix the logic such a way that if service is started, don't attempt to start
> the service again. Similar changes needed to be done for stop().
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-3246) pRead equivalent for direct read path
[ https://issues.apache.org/jira/browse/HDFS-3246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830732#comment-16830732 ] Hudson commented on HDFS-3246: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16483 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16483/]) HDFS-3246: pRead equivalent for direct read path (#597) (todd: rev 4877f0aa518832c37a06e6d3bd2c9552fc3141dc) * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/TestCryptoStreams.java * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/test_libhdfs_ops.c * (add) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ByteBufferPositionedReadable.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/TestCryptoStreamsNormal.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/StreamCapabilities.java * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs/hdfs.c * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/crypto/CryptoInputStream.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/ByteBufferReadable.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/CryptoStreamsTestBase.java * (edit) hadoop-hdfs-project/hadoop-hdfs-client/src/main/java/org/apache/hadoop/hdfs/DFSInputStream.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/fs/FSDataInputStream.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/crypto/TestCryptoStreamsForLocalFS.java * (add) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestByteBufferPread.java * (edit) hadoop-hdfs-project/hadoop-hdfs-native-client/src/main/native/libhdfs-tests/hdfs_test.h > pRead equivalent for direct read path > - > > Key: HDFS-3246 > URL: https://issues.apache.org/jira/browse/HDFS-3246 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client, performance >Affects Versions: 3.0.0-alpha1 >Reporter: Henry Robinson >Assignee: Sahil Takiar >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-3246.001.patch, HDFS-3246.002.patch, > HDFS-3246.003.patch, HDFS-3246.004.patch, HDFS-3246.005.patch, > HDFS-3246.006.patch, HDFS-3246.007.patch > > > There is no pread equivalent in ByteBufferReadable. We should consider adding > one. It would be relatively easy to implement for the distributed case > (certainly compared to HDFS-2834), since DFSInputStream does most of the > heavy lifting. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1473) DataNode ID file should be human readable
[ https://issues.apache.org/jira/browse/HDDS-1473?focusedWorklogId=235562=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-235562 ] ASF GitHub Bot logged work on HDDS-1473: Author: ASF GitHub Bot Created on: 30/Apr/19 22:08 Start Date: 30/Apr/19 22:08 Worklog Time Spent: 10m Work Description: hanishakoneru commented on issue #781: HDDS-1473. DataNode ID file should be human readable. URL: https://github.com/apache/hadoop/pull/781#issuecomment-488133314 LGTM. +1 pending CI checks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 235562) Time Spent: 2h 10m (was: 2h) > DataNode ID file should be human readable > - > > Key: HDDS-1473 > URL: https://issues.apache.org/jira/browse/HDDS-1473 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: Ozone Datanode >Reporter: Arpit Agarwal >Assignee: Siddharth Wagle >Priority: Major > Labels: newbie, pull-request-available > Time Spent: 2h 10m > Remaining Estimate: 0h > > The DataNode ID file should be human readable to make debugging easier. We > should use YAML as we have used it elsewhere for meta files. > Currently it is a binary file whose contents are protobuf encoded. This is a > tiny file read once on startup, so performance is not a concern. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-3246) pRead equivalent for direct read path
[ https://issues.apache.org/jira/browse/HDFS-3246?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sahil Takiar updated HDFS-3246: --- Resolution: Fixed Fix Version/s: 3.3.0 Status: Resolved (was: Patch Available) > pRead equivalent for direct read path > - > > Key: HDFS-3246 > URL: https://issues.apache.org/jira/browse/HDFS-3246 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client, performance >Affects Versions: 3.0.0-alpha1 >Reporter: Henry Robinson >Assignee: Sahil Takiar >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-3246.001.patch, HDFS-3246.002.patch, > HDFS-3246.003.patch, HDFS-3246.004.patch, HDFS-3246.005.patch, > HDFS-3246.006.patch, HDFS-3246.007.patch > > > There is no pread equivalent in ByteBufferReadable. We should consider adding > one. It would be relatively easy to implement for the distributed case > (certainly compared to HDFS-2834), since DFSInputStream does most of the > heavy lifting. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13995) RBF: Security documentation
[
https://issues.apache.org/jira/browse/HDFS-13995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830693#comment-16830693
]
Hadoop QA commented on HDFS-13995:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
45s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
|| || || || {color:brown} HDFS-13891 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
32s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
43s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
37m 19s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
15m 13s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 56m 12s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-13995 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12967521/HDFS-13995-HDFS-13891.002.patch
|
| Optional Tests | dupname asflicense mvnsite |
| uname | Linux faa89fcfe2cd 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-13891 / aeb3b61 |
| maven | version: Apache Maven 3.3.9 |
| Max. process+thread count | 295 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/26733/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> RBF: Security documentation
> ---
>
> Key: HDFS-13995
> URL: https://issues.apache.org/jira/browse/HDFS-13995
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: CR Hota
>Assignee: CR Hota
>Priority: Major
> Attachments: HDFS-13995-HDFS-13891.001.patch,
> HDFS-13995-HDFS-13891.002.patch
>
>
> Documentation for users under the section relating to security needs to be
> updated once security work is completely.
> [https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs-rbf/HDFSRouterFederation.html#Security]
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14460) DFSUtil#getNamenodeWebAddr should return HTTPS address based on policy configured
[ https://issues.apache.org/jira/browse/HDFS-14460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830679#comment-16830679 ] CR Hota commented on HDFS-14460: [~elgoiri] Thanks for the review. Could you help understand the first point better? > DFSUtil#getNamenodeWebAddr should return HTTPS address based on policy > configured > - > > Key: HDFS-14460 > URL: https://issues.apache.org/jira/browse/HDFS-14460 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: CR Hota >Assignee: CR Hota >Priority: Major > Attachments: HDFS-14460.001.patch, HDFS-14460.002.patch > > > DFSUtil#getNamenodeWebAddr does a look-up of HTTP address irrespective of > policy configured. It should instead look at the policy configured and return > appropriate web address. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1475) Fix OzoneContainer start method
[
https://issues.apache.org/jira/browse/HDDS-1475?focusedWorklogId=235527=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-235527
]
ASF GitHub Bot logged work on HDDS-1475:
Author: ASF GitHub Bot
Created on: 30/Apr/19 20:22
Start Date: 30/Apr/19 20:22
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #788:
HDDS-1475 : Fix OzoneContainer start method.
URL: https://github.com/apache/hadoop/pull/788#discussion_r279920219
##
File path:
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OzoneContainer.java
##
@@ -183,8 +183,12 @@ private void stopContainerScrub() {
public void start(String scmId) throws IOException {
LOG.info("Attempting to start container services.");
startContainerScrub();
Review comment:
Do we need to do something similar for startContainerScrub also?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 235527)
Time Spent: 40m (was: 0.5h)
> Fix OzoneContainer start method
> ---
>
> Key: HDDS-1475
> URL: https://issues.apache.org/jira/browse/HDDS-1475
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Reporter: Bharat Viswanadham
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: newbie, pull-request-available
> Time Spent: 40m
> Remaining Estimate: 0h
>
> In OzoneContainer start() we have
> {code:java}
> startContainerScrub();
> writeChannel.start();
> readChannel.start();
> hddsDispatcher.init();
> hddsDispatcher.setScmId(scmId);{code}
>
> Suppose here if readChannel.start() failed due to some reason, from
> VersionEndPointTask, we try to start OzoneContainer again. This can cause an
> issue for writeChannel.start() if it is already started.
>
> Fix the logic such a way that if service is started, don't attempt to start
> the service again. Similar changes needed to be done for stop().
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1475) Fix OzoneContainer start method
[
https://issues.apache.org/jira/browse/HDDS-1475?focusedWorklogId=235526=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-235526
]
ASF GitHub Bot logged work on HDDS-1475:
Author: ASF GitHub Bot
Created on: 30/Apr/19 20:22
Start Date: 30/Apr/19 20:22
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #788:
HDDS-1475 : Fix OzoneContainer start method.
URL: https://github.com/apache/hadoop/pull/788#discussion_r279920219
##
File path:
hadoop-hdds/container-service/src/main/java/org/apache/hadoop/ozone/container/ozoneimpl/OzoneContainer.java
##
@@ -183,8 +183,12 @@ private void stopContainerScrub() {
public void start(String scmId) throws IOException {
LOG.info("Attempting to start container services.");
startContainerScrub();
Review comment:
Do we need to do somthing similar for startContainerScrub also?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 235526)
Time Spent: 0.5h (was: 20m)
> Fix OzoneContainer start method
> ---
>
> Key: HDDS-1475
> URL: https://issues.apache.org/jira/browse/HDDS-1475
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Reporter: Bharat Viswanadham
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: newbie, pull-request-available
> Time Spent: 0.5h
> Remaining Estimate: 0h
>
> In OzoneContainer start() we have
> {code:java}
> startContainerScrub();
> writeChannel.start();
> readChannel.start();
> hddsDispatcher.init();
> hddsDispatcher.setScmId(scmId);{code}
>
> Suppose here if readChannel.start() failed due to some reason, from
> VersionEndPointTask, we try to start OzoneContainer again. This can cause an
> issue for writeChannel.start() if it is already started.
>
> Fix the logic such a way that if service is started, don't attempt to start
> the service again. Similar changes needed to be done for stop().
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1475) Fix OzoneContainer start method
[
https://issues.apache.org/jira/browse/HDDS-1475?focusedWorklogId=235524=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-235524
]
ASF GitHub Bot logged work on HDDS-1475:
Author: ASF GitHub Bot
Created on: 30/Apr/19 20:17
Start Date: 30/Apr/19 20:17
Worklog Time Spent: 10m
Work Description: avijayanhwx commented on issue #788: HDDS-1475 : Fix
OzoneContainer start method.
URL: https://github.com/apache/hadoop/pull/788#issuecomment-488099389
/label ozone
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 235524)
Time Spent: 20m (was: 10m)
> Fix OzoneContainer start method
> ---
>
> Key: HDDS-1475
> URL: https://issues.apache.org/jira/browse/HDDS-1475
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Reporter: Bharat Viswanadham
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: newbie, pull-request-available
> Time Spent: 20m
> Remaining Estimate: 0h
>
> In OzoneContainer start() we have
> {code:java}
> startContainerScrub();
> writeChannel.start();
> readChannel.start();
> hddsDispatcher.init();
> hddsDispatcher.setScmId(scmId);{code}
>
> Suppose here if readChannel.start() failed due to some reason, from
> VersionEndPointTask, we try to start OzoneContainer again. This can cause an
> issue for writeChannel.start() if it is already started.
>
> Fix the logic such a way that if service is started, don't attempt to start
> the service again. Similar changes needed to be done for stop().
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1475) Fix OzoneContainer start method
[
https://issues.apache.org/jira/browse/HDDS-1475?focusedWorklogId=235522=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-235522
]
ASF GitHub Bot logged work on HDDS-1475:
Author: ASF GitHub Bot
Created on: 30/Apr/19 20:16
Start Date: 30/Apr/19 20:16
Worklog Time Spent: 10m
Work Description: avijayanhwx commented on pull request #788: HDDS-1475 :
Fix OzoneContainer start method.
URL: https://github.com/apache/hadoop/pull/788
In OzoneContainer start() we have
startContainerScrub();
writeChannel.start();
readChannel.start();
hddsDispatcher.init();
hddsDispatcher.setScmId(scmId);
Suppose here if the readChannel.start() failed due to some reason, from
VersionEndPointTask, we try to start OzoneContainer again. This can cause an
issue for writeChannel.start() if it is already started.
Fix the logic such a way that if service is started, don't attempt to start
the service again. Similar changes needed to be done for stop().
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 235522)
Time Spent: 10m
Remaining Estimate: 0h
> Fix OzoneContainer start method
> ---
>
> Key: HDDS-1475
> URL: https://issues.apache.org/jira/browse/HDDS-1475
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Reporter: Bharat Viswanadham
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: newbie, pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> In OzoneContainer start() we have
> {code:java}
> startContainerScrub();
> writeChannel.start();
> readChannel.start();
> hddsDispatcher.init();
> hddsDispatcher.setScmId(scmId);{code}
>
> Suppose here if readChannel.start() failed due to some reason, from
> VersionEndPointTask, we try to start OzoneContainer again. This can cause an
> issue for writeChannel.start() if it is already started.
>
> Fix the logic such a way that if service is started, don't attempt to start
> the service again. Similar changes needed to be done for stop().
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1475) Fix OzoneContainer start method
[
https://issues.apache.org/jira/browse/HDDS-1475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-1475:
-
Labels: newbie pull-request-available (was: newbie)
> Fix OzoneContainer start method
> ---
>
> Key: HDDS-1475
> URL: https://issues.apache.org/jira/browse/HDDS-1475
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Datanode
>Reporter: Bharat Viswanadham
>Assignee: Aravindan Vijayan
>Priority: Major
> Labels: newbie, pull-request-available
>
> In OzoneContainer start() we have
> {code:java}
> startContainerScrub();
> writeChannel.start();
> readChannel.start();
> hddsDispatcher.init();
> hddsDispatcher.setScmId(scmId);{code}
>
> Suppose here if readChannel.start() failed due to some reason, from
> VersionEndPointTask, we try to start OzoneContainer again. This can cause an
> issue for writeChannel.start() if it is already started.
>
> Fix the logic such a way that if service is started, don't attempt to start
> the service again. Similar changes needed to be done for stop().
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13995) RBF: Security documentation
[ https://issues.apache.org/jira/browse/HDFS-13995?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830640#comment-16830640 ] CR Hota commented on HDFS-13995: [~elgoiri] Thanks for the comments. Uploaded a new draft that highlights couple of important points wrt delegation tokens. > RBF: Security documentation > --- > > Key: HDFS-13995 > URL: https://issues.apache.org/jira/browse/HDFS-13995 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: CR Hota >Assignee: CR Hota >Priority: Major > Attachments: HDFS-13995-HDFS-13891.001.patch, > HDFS-13995-HDFS-13891.002.patch > > > Documentation for users under the section relating to security needs to be > updated once security work is completely. > [https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs-rbf/HDFSRouterFederation.html#Security] > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13995) RBF: Security documentation
[ https://issues.apache.org/jira/browse/HDFS-13995?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] CR Hota updated HDFS-13995: --- Attachment: HDFS-13995-HDFS-13891.002.patch > RBF: Security documentation > --- > > Key: HDFS-13995 > URL: https://issues.apache.org/jira/browse/HDFS-13995 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: CR Hota >Assignee: CR Hota >Priority: Major > Attachments: HDFS-13995-HDFS-13891.001.patch, > HDFS-13995-HDFS-13891.002.patch > > > Documentation for users under the section relating to security needs to be > updated once security work is completely. > [https://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-hdfs-rbf/HDFSRouterFederation.html#Security] > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13522) Support observer node from Router-Based Federation
[
https://issues.apache.org/jira/browse/HDFS-13522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830536#comment-16830536
]
Íñigo Goiri commented on HDFS-13522:
Thank you [~surendrasingh], I think the idea in [^HDFS-13522.001.patch] looks
good.
I think the counting of the RPC queries is a little out of place.
The metrics already track this kind of things (e.g., calls to the State Store
or calls to Standby NNs).
We should try to leverage the same.
For checking if it is a READ operation we also track that in OpCategory in
RouterRpcServer.
Not sure if we should leverage that or rely on the observer infra.
We should change the UI a little too to show the observer NN.
> Support observer node from Router-Based Federation
> --
>
> Key: HDFS-13522
> URL: https://issues.apache.org/jira/browse/HDFS-13522
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: federation, namenode
>Reporter: Erik Krogen
>Assignee: Chao Sun
>Priority: Major
> Attachments: HDFS-13522.001.patch
>
>
> Changes will need to occur to the router to support the new observer node.
> One such change will be to make the router understand the observer state,
> e.g. {{FederationNamenodeServiceState}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1468) Inject configuration values to Java objects
[ https://issues.apache.org/jira/browse/HDDS-1468?focusedWorklogId=235437=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-235437 ] ASF GitHub Bot logged work on HDDS-1468: Author: ASF GitHub Bot Created on: 30/Apr/19 17:52 Start Date: 30/Apr/19 17:52 Worklog Time Spent: 10m Work Description: anuengineer commented on issue #772: HDDS-1468. Inject configuration values to Java objects URL: https://github.com/apache/hadoop/pull/772#issuecomment-488050065 +1, Looks good. Thanks. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 235437) Time Spent: 40m (was: 0.5h) > Inject configuration values to Java objects > --- > > Key: HDDS-1468 > URL: https://issues.apache.org/jira/browse/HDDS-1468 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > According to the design doc in the parent issue we would like to support java > configuration objects which are simple POJO but the fields/setters are > annotated. As a first step we can introduce the > OzoneConfiguration.getConfigObject() api which can create the config object > and inject configuration. > Later we can improve it with annotation processor which can generate the > ozone-default.xml. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1469) Generate default configuration fragments based on annotations
[ https://issues.apache.org/jira/browse/HDDS-1469?focusedWorklogId=235438=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-235438 ] ASF GitHub Bot logged work on HDDS-1469: Author: ASF GitHub Bot Created on: 30/Apr/19 17:52 Start Date: 30/Apr/19 17:52 Worklog Time Spent: 10m Work Description: anuengineer commented on issue #773: HDDS-1469. Generate default configuration fragments based on annotations URL: https://github.com/apache/hadoop/pull/773#issuecomment-488050243 +1 , thanks LGTM. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 235438) Time Spent: 4h 40m (was: 4.5h) > Generate default configuration fragments based on annotations > - > > Key: HDDS-1469 > URL: https://issues.apache.org/jira/browse/HDDS-1469 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Time Spent: 4h 40m > Remaining Estimate: 0h > > See the design doc in the parent jira for more details. > In this jira I introduce a new annotation processor which can generate > ozone-default.xml fragments based on the annotations which are introduced by > HDDS-1468. > The ozone-default-generated.xml fragments can be used directly by the > OzoneConfiguration as I added a small code to the constructor to check ALL > the available ozone-default-generated.xml files and add them to the available > resources. > With this approach we don't need to edit ozone-default.xml as all the > configuration can be defined in java code. > As a side effect each service will see only the available configuration keys > and values based on the classpath. (If the ozone-default-generated.xml file > of OzoneManager is not on the classpath of the SCM, SCM doesn't see the > available configs.) -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12510) RBF: Add security to UI
[ https://issues.apache.org/jira/browse/HDFS-12510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830521#comment-16830521 ] CR Hota commented on HDFS-12510: [~brahmareddy] [~elgoiri] Thanks for your comments. Am fine either ways, but quite curious to know what the CORS issue is wrt Router since router UI doesn't do any redirects. [~raviprak] Gentle ping again to help guide us ! :) > RBF: Add security to UI > --- > > Key: HDFS-12510 > URL: https://issues.apache.org/jira/browse/HDFS-12510 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: CR Hota >Priority: Major > Labels: RBF > Attachments: HDFS-12510-HDFS-13891.001.patch > > > HDFS-12273 implemented the UI for Router Based Federation without security. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14353) Erasure Coding: metrics xmitsInProgress become to negative.
[ https://issues.apache.org/jira/browse/HDFS-14353?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830520#comment-16830520 ] Íñigo Goiri commented on HDFS-14353: Can we tyr somebody else involved in EC? Otherwise, let's just put a comment that more or less conveys the idea of having always 1. > Erasure Coding: metrics xmitsInProgress become to negative. > --- > > Key: HDFS-14353 > URL: https://issues.apache.org/jira/browse/HDFS-14353 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: datanode, erasure-coding >Affects Versions: 3.3.0 >Reporter: maobaolong >Assignee: maobaolong >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-14353.001.patch, HDFS-14353.002.patch, > HDFS-14353.003.patch, HDFS-14353.004.patch, HDFS-14353.005.patch, > screenshot-1.png > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14460) DFSUtil#getNamenodeWebAddr should return HTTPS address based on policy configured
[
https://issues.apache.org/jira/browse/HDFS-14460?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830517#comment-16830517
]
Íñigo Goiri commented on HDFS-14460:
* What if we use {{getHttpClientScheme()}} instead of the indirect
{{getHttpClientScheme()}}?
* Probably we can use something like setEnum in the test for the policy or at
least: {{HttpConfig.Policy.HTTP_ONLY.name()}}.
* A minor style comment: I would fix the spacing for the equals in
NS1_NN1_HTTPS_ADDR and NS1_NN1_HTTP_ADDR; they are not aligned to anything
right now.
* I would just change the code to leave it closer to the old one:
{code}
String webAddrBaseKey = DFSConfigKeys.DFS_NAMENODE_HTTP_ADDRESS_KEY;
String webAddrDefault = DFSConfigKeys.DFS_NAMENODE_HTTP_ADDRESS_DEFAULT
if (https) {
webAddrBaseKey = DFSConfigKeys.DFS_NAMENODE_HTTPS_ADDRESS_KEY;
webAddrDefault = DFSConfigKeys.DFS_NAMENODE_HTTPS_ADDRESS_DEFAULT;
}
String webAddrKey = DFSUtilClient.concatSuffixes(webAddrBaseKey, nsId, nnId);
String webAddr = conf.get(webAddrKey, webAddrDefault);
{code}
> DFSUtil#getNamenodeWebAddr should return HTTPS address based on policy
> configured
> -
>
> Key: HDFS-14460
> URL: https://issues.apache.org/jira/browse/HDFS-14460
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: CR Hota
>Assignee: CR Hota
>Priority: Major
> Attachments: HDFS-14460.001.patch, HDFS-14460.002.patch
>
>
> DFSUtil#getNamenodeWebAddr does a look-up of HTTP address irrespective of
> policy configured. It should instead look at the policy configured and return
> appropriate web address.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-12510) RBF: Add security to UI
[ https://issues.apache.org/jira/browse/HDFS-12510?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830500#comment-16830500 ] Íñigo Goiri commented on HDFS-12510: Two options: # We change the title, open a new JIRA for the CORS part and commit this. # We create a new JIRA for [^HDFS-12510-HDFS-13891.001.patch] and commit that. I would go with 2. > RBF: Add security to UI > --- > > Key: HDFS-12510 > URL: https://issues.apache.org/jira/browse/HDFS-12510 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: CR Hota >Priority: Major > Labels: RBF > Attachments: HDFS-12510-HDFS-13891.001.patch > > > HDFS-12273 implemented the UI for Router Based Federation without security. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-1458) Create a maven profile to run fault injection tests
[
https://issues.apache.org/jira/browse/HDDS-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830481#comment-16830481
]
Eric Yang edited comment on HDDS-1458 at 4/30/19 5:15 PM:
--
{quote}I don't think it's a strong limitation to make the docker run to
privileged.
{quote}
Cool, I will keep --privileged option in the patch.
{quote}But it may be useful to move out blockade tests from dist to a separated
project if you prefer it.
{quote}
I look at integration-test project and found that it was used to build mini
ozone cluster. I agree that blockade code belongs to a separate project from
dist and integration-test, and have done the refactoring for next patch.
{quote}I would use a dedicated profile. The standard mvn verify (or mvn
install) is expected to be used by new contributors. I would keep that one as
simple as possible.
{quote}
I will keep -Pit for triggering the fault injection tests.
{quote}One additional question about disk failure testing (If I understood well
that will be implemented as a next step). How they are connected with blockade
tests? Do you need any functionality from blockade? Can we use the robot test
based smoketests for the same?
{quote}
Disk tests are written using docker-compose with mountable data disks. They
are separate projects from network tests. I basically created a submodule
called fault-injection-test and having disk-tests and network-tests as
submodule. In disk-tests, there are read-write-test, read-only-test, and
corruption-test. Blockcade tests are stored in network-tests. Jenkinsfile
will be updated to activate docker mode, and include -Pit profile to trigger
the tests.
was (Author: eyang):
{quote}I don't think it's a strong limitation to make the docker run to
privileged.\{quote}
Cool, I will keep --privileged option in the patch.
{quote}But it may be useful to move out blockade tests from dist to a separated
project if you prefer it.\{quote}
I look at integration-test project and found that it was used to build
miniozone cluster. I agree that blockade code belongs to a separate project
from dist and integration-test, and have done the refactoring for next patch.
{quote}I would use a dedicated profile. The standard mvn verify (or mvn
install) is expected to be used by new contributors. I would keep that one as
simple as possible.\{quote}
I will keep -Pit for triggering the fault injection tests.
{quote}One additional question about disk failure testing (If I understood well
that will be implemented as a next step). How they are connected with blockade
tests? Do you need any functionality from blockade? Can we use the robot test
based smoketests for the same?\{quote}
Disk tests are written using docker-compose with mountable data disks. They
are separate projects from network tests. I basically created a submodule
called fault-injection-test and having disk-tests and network-tests as
submodule. In disk-tests, there are read-write-test, read-only-test, and
corruption-test. Blockcade tests are stored in network-tests. Jenkinsfile
will be updated to include -Pit profile to trigger the tests.
> Create a maven profile to run fault injection tests
> ---
>
> Key: HDDS-1458
> URL: https://issues.apache.org/jira/browse/HDDS-1458
> Project: Hadoop Distributed Data Store
> Issue Type: Test
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: HDDS-1458.001.patch
>
>
> Some fault injection tests have been written using blockade. It would be
> nice to have ability to start docker compose and exercise the blockade test
> cases against Ozone docker containers, and generate reports. This is
> optional integration tests to catch race conditions and fault tolerance
> defects.
> We can introduce a profile with id: it (short for integration tests). This
> will launch docker compose via maven-exec-plugin and run blockade to simulate
> container failures and timeout.
> Usage command:
> {code}
> mvn clean verify -Pit
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1458) Create a maven profile to run fault injection tests
[
https://issues.apache.org/jira/browse/HDDS-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830481#comment-16830481
]
Eric Yang commented on HDDS-1458:
-
{quote}I don't think it's a strong limitation to make the docker run to
privileged.\{quote}
Cool, I will keep --privileged option in the patch.
{quote}But it may be useful to move out blockade tests from dist to a separated
project if you prefer it.\{quote}
I look at integration-test project and found that it was used to build
miniozone cluster. I agree that blockade code belongs to a separate project
from dist and integration-test, and have done the refactoring for next patch.
{quote}I would use a dedicated profile. The standard mvn verify (or mvn
install) is expected to be used by new contributors. I would keep that one as
simple as possible.\{quote}
I will keep -Pit for triggering the fault injection tests.
{quote}One additional question about disk failure testing (If I understood well
that will be implemented as a next step). How they are connected with blockade
tests? Do you need any functionality from blockade? Can we use the robot test
based smoketests for the same?\{quote}
Disk tests are written using docker-compose with mountable data disks. They
are separate projects from network tests. I basically created a submodule
called fault-injection-test and having disk-tests and network-tests as
submodule. In disk-tests, there are read-write-test, read-only-test, and
corruption-test. Blockcade tests are stored in network-tests. Jenkinsfile
will be updated to include -Pit profile to trigger the tests.
> Create a maven profile to run fault injection tests
> ---
>
> Key: HDDS-1458
> URL: https://issues.apache.org/jira/browse/HDDS-1458
> Project: Hadoop Distributed Data Store
> Issue Type: Test
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: HDDS-1458.001.patch
>
>
> Some fault injection tests have been written using blockade. It would be
> nice to have ability to start docker compose and exercise the blockade test
> cases against Ozone docker containers, and generate reports. This is
> optional integration tests to catch race conditions and fault tolerance
> defects.
> We can introduce a profile with id: it (short for integration tests). This
> will launch docker compose via maven-exec-plugin and run blockade to simulate
> container failures and timeout.
> Usage command:
> {code}
> mvn clean verify -Pit
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14462) WebHDFS throws "Error writing request body to server" instead of NSQuotaExceededException
Erik Krogen created HDFS-14462:
--
Summary: WebHDFS throws "Error writing request body to server"
instead of NSQuotaExceededException
Key: HDFS-14462
URL: https://issues.apache.org/jira/browse/HDFS-14462
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Affects Versions: 3.1.2, 2.7.7, 2.8.5, 3.0.3, 2.9.2, 3.2.0
Reporter: Erik Krogen
We noticed recently in our environment that, when writing data to HDFS via
WebHDFS, a quota exception is returned to the client as:
{code}
java.io.IOException: Error writing request body to server
at
sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.checkError(HttpURLConnection.java:3536)
~[?:1.8.0_172]
at
sun.net.www.protocol.http.HttpURLConnection$StreamingOutputStream.write(HttpURLConnection.java:3519)
~[?:1.8.0_172]
at
java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
~[?:1.8.0_172]
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
~[?:1.8.0_172]
at java.io.FilterOutputStream.flush(FilterOutputStream.java:140)
~[?:1.8.0_172]
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
~[?:1.8.0_172]
{code}
It is entirely opaque to the user that this exception was caused because they
exceeded their quota. Yet in the DataNode logs:
{code}
2019-04-24 02:13:09,639 WARN org.apache.hadoop.hdfs.DFSClient: DataStreamer
Exception
org.apache.hadoop.hdfs.protocol.DSQuotaExceededException: The DiskSpace quota
of /foo/path/here is exceeded: quota = B = X TB but diskspace
consumed = B = X TB
at
org.apache.hadoop.hdfs.server.namenode.DirectoryWithQuotaFeature.verifyStoragespaceQuota(DirectoryWithQuotaFeature.java:211)
at
org.apache.hadoop.hdfs.server.namenode.DirectoryWithQuotaFeature.verifyQuota(DirectoryWithQuotaFeature.java:239)
{code}
This was on a 2.7.x cluster, but I verified that the same logic exists on
trunk. I believe we need to fix some of the logic within the
{{ExceptionHandler}} to add special handling for the quota exception.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDFS-14401) Refine the implementation for HDFS cache on SCM
[
https://issues.apache.org/jira/browse/HDFS-14401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830376#comment-16830376
]
Rakesh R edited comment on HDFS-14401 at 4/30/19 3:09 PM:
--
Overall the patch looks good and I think its nearing completion. Could you
please take care below comments:
# Rename PmemVolumeManager variable '{{i}}' to '{{nextIndex}}'.
# How about resetting {{nextIndex}} to avoid growing to infinity, probably can
refer below idea or you can explicitly reset {{nextIndex=0}}, {{if (nextIndex
== count)}}.
{code:java}
private byte nextIndex = 0;
..
..
while (k++ != count) {
nextIndex = (byte) (nextIndex % count);
byte index = nextIndex;
nextIndex++;
long availableBytes = usedBytesCounts.get(index).getAvailableBytes();
if (availableBytes >= bytesCount) {
return index;
}
if (availableBytes > maxAvailableSpace) {
maxAvailableSpace = availableBytes;
}
}
{code}
# Instead of {{memCacheStats.getCacheUsed()}}, it should be
{{cacheLoader.getCacheUsed()}}, right?
{code:java}
LOG.debug("Caching of {} was aborted. We are now caching only {} "
+ "bytes in total.", key, cacheLoader.getCacheUsed());
{code}
# Please double check the chances of any scenario where it adds
{{blockKeyToVolume.put(key, index);}} entry and then
{{usedBytesCounts.get(index).reserve(bytesCount);}} return -1?
was (Author: rakeshr):
Overall the patch looks good and I think its nearing completion. Could you
please take care below comments:
# Rename PmemVolumeManager variable '{{i}}' to '{{nextIndex}}'.
# How about resetting {{nextIndex}} to avoid growing to infinity, probably can
refer below idea or you can explicitly reset {{nextIndex=0}}, {{if (nextIndex
== count)}}.
{code:java}
private byte nextIndex = 0;
..
..
while (k++ != count) {
nextIndex = (byte) (nextIndex % count);
byte index = nextIndex;
nextIndex++;
long availableBytes = usedBytesCounts.get(index).getAvailableBytes();
if (availableBytes >= bytesCount) {
return index;
}
if (availableBytes > maxAvailableSpace) {
maxAvailableSpace = availableBytes;
}
}
{code}
# Instead of {{memCacheStats.getCacheUsed()}}, it should be
{{cacheLoader.getCacheUsed()}}, right?
{code:java}
LOG.debug("Caching of {} was aborted. We are now caching only {} "
+ "bytes in total.", key, cacheLoader.getCacheUsed());
{code}
# Please double check the chances of any scenario where it adds
{{blockKeyToVolume.put(key, index);}} entry and then
{{usedBytesCounts.get(index).reserve(bytesCount);}} return -1?
> Refine the implementation for HDFS cache on SCM
> ---
>
> Key: HDFS-14401
> URL: https://issues.apache.org/jira/browse/HDFS-14401
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: caching, datanode
>Reporter: Feilong He
>Assignee: Feilong He
>Priority: Major
> Attachments: HDFS-14401.000.patch, HDFS-14401.001.patch,
> HDFS-14401.002.patch, HDFS-14401.003.patch, HDFS-14401.004.patch,
> HDFS-14401.005.patch, HDFS-14401.006.patch
>
>
> In this Jira, we will refine the implementation for HDFS cache on SCM, such
> as: 1) Handle full pmem volume in VolumeManager; 2) Refine pmem volume
> selection impl; 3) Clean up MapppableBlockLoader interface; etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14401) Refine the implementation for HDFS cache on SCM
[
https://issues.apache.org/jira/browse/HDFS-14401?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830376#comment-16830376
]
Rakesh R commented on HDFS-14401:
-
Overall the patch looks good and I think its nearing completion. Could you
please take care below comments:
# Rename PmemVolumeManager variable '{{i}}' to '{{nextIndex}}'.
# How about resetting {{nextIndex}} to avoid growing to infinity, probably can
refer below idea or you can explicitly reset {{nextIndex=0}}, {{if (nextIndex
== count)}}.
{code:java}
private byte nextIndex = 0;
..
..
while (k++ != count) {
nextIndex = (byte) (nextIndex % count);
byte index = nextIndex;
nextIndex++;
long availableBytes = usedBytesCounts.get(index).getAvailableBytes();
if (availableBytes >= bytesCount) {
return index;
}
if (availableBytes > maxAvailableSpace) {
maxAvailableSpace = availableBytes;
}
}
{code}
# Instead of {{memCacheStats.getCacheUsed()}}, it should be
{{cacheLoader.getCacheUsed()}}, right?
{code:java}
LOG.debug("Caching of {} was aborted. We are now caching only {} "
+ "bytes in total.", key, cacheLoader.getCacheUsed());
{code}
# Please double check the chances of any scenario where it adds
{{blockKeyToVolume.put(key, index);}} entry and then
{{usedBytesCounts.get(index).reserve(bytesCount);}} return -1?
> Refine the implementation for HDFS cache on SCM
> ---
>
> Key: HDFS-14401
> URL: https://issues.apache.org/jira/browse/HDFS-14401
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: caching, datanode
>Reporter: Feilong He
>Assignee: Feilong He
>Priority: Major
> Attachments: HDFS-14401.000.patch, HDFS-14401.001.patch,
> HDFS-14401.002.patch, HDFS-14401.003.patch, HDFS-14401.004.patch,
> HDFS-14401.005.patch, HDFS-14401.006.patch
>
>
> In this Jira, we will refine the implementation for HDFS cache on SCM, such
> as: 1) Handle full pmem volume in VolumeManager; 2) Refine pmem volume
> selection impl; 3) Clean up MapppableBlockLoader interface; etc.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1478) Provide k8s resources files for prometheus and performance tests
[ https://issues.apache.org/jira/browse/HDDS-1478?focusedWorklogId=235333=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-235333 ] ASF GitHub Bot logged work on HDDS-1478: Author: ASF GitHub Bot Created on: 30/Apr/19 14:53 Start Date: 30/Apr/19 14:53 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #783: HDDS-1478. Provide k8s resources files for prometheus and performance tests URL: https://github.com/apache/hadoop/pull/783#issuecomment-487984251 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 20 | Docker mode activated. | ||| _ Prechecks _ | | 0 | yamllint | 1 | yamllint was not available. | | +1 | @author | 0 | The patch does not contain any @author tags. | | -1 | test4tests | 0 | The patch doesn't appear to include any new or modified tests. Please justify why no new tests are needed for this patch. Also please list what manual steps were performed to verify this patch. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 39 | Maven dependency ordering for branch | | +1 | mvninstall | 1182 | trunk passed | | +1 | compile | 118 | trunk passed | | +1 | mvnsite | 90 | trunk passed | | +1 | shadedclient | 721 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 63 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 14 | Maven dependency ordering for patch | | -1 | mvninstall | 22 | dist in the patch failed. | | +1 | compile | 105 | the patch passed | | +1 | javac | 105 | the patch passed | | +1 | hadolint | 1 | There were no new hadolint issues. | | +1 | mvnsite | 56 | the patch passed | | +1 | shellcheck | 2 | There were no new shellcheck issues. | | +1 | shelldocs | 16 | The patch generated 0 new + 104 unchanged - 132 fixed = 104 total (was 236) | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | xml | 3 | The patch has no ill-formed XML file. | | +1 | shadedclient | 823 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 57 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 39 | common in the patch passed. | | +1 | unit | 23 | dist in the patch passed. | | +1 | asflicense | 29 | The patch does not generate ASF License warnings. | | | | 3606 | | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-783/2/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/783 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient xml hadolint shellcheck shelldocs yamllint | | uname | Linux aabcb57ed666 4.4.0-143-generic #169~14.04.2-Ubuntu SMP Wed Feb 13 15:00:41 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / dead9b4 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | shellcheck | v0.4.6 | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-783/2/artifact/out/patch-mvninstall-hadoop-ozone_dist.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-783/2/testReport/ | | Max. process+thread count | 307 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/common hadoop-ozone/dist U: hadoop-ozone | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-783/2/console | | Powered by | Apache Yetus 0.9.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 235333) Time Spent: 0.5h (was: 20m) > Provide k8s resources files for prometheus and performance tests > > > Key: HDDS-1478 > URL: https://issues.apache.org/jira/browse/HDDS-1478 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > Similar to HDDS-1412 we can further improve the available k8s resources with > providing example resources to:
[jira] [Commented] (HDDS-1458) Create a maven profile to run fault injection tests
[
https://issues.apache.org/jira/browse/HDDS-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830311#comment-16830311
]
Elek, Marton commented on HDDS-1458:
One additional question about disk failure testing (If I understood well that
will be implemented as a next step). How they are connected with blockade
tests? Do you need any functionality from blockade? Can we use the robot test
based smoketests for the same?
> Create a maven profile to run fault injection tests
> ---
>
> Key: HDDS-1458
> URL: https://issues.apache.org/jira/browse/HDDS-1458
> Project: Hadoop Distributed Data Store
> Issue Type: Test
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: HDDS-1458.001.patch
>
>
> Some fault injection tests have been written using blockade. It would be
> nice to have ability to start docker compose and exercise the blockade test
> cases against Ozone docker containers, and generate reports. This is
> optional integration tests to catch race conditions and fault tolerance
> defects.
> We can introduce a profile with id: it (short for integration tests). This
> will launch docker compose via maven-exec-plugin and run blockade to simulate
> container failures and timeout.
> Usage command:
> {code}
> mvn clean verify -Pit
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1458) Create a maven profile to run fault injection tests
[
https://issues.apache.org/jira/browse/HDDS-1458?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830307#comment-16830307
]
Elek, Marton commented on HDDS-1458:
Thank you very much to work on this [~eyang]. It looks very promising.
Here is my personal opinion:
bq. Are we ok with docker in docker addition to start-build-env.sh because it
uses --privileged command to gain access to host level docker. In my opinion,
the existing setup already require user to have access to docker. This new
privileged flag gives more power to break out of container environment, but it
is necessary to simulate network or disk failures. I am fine without bundle
this in start-build-env.sh, but it is nicer without having to look for
developer dependencies to start development.
As of now we use '--privileged' on jenkins builds, so the usage of privileged
shouldn't be a problem. It's an other question to include it as a default
option in ./start-build.env.sh. That script is not used on jenkins and I have
no strong preferences.
As most of the user run build without any container isolation (just in the
local machine) I don't think it's a strong limitation to make the docker run to
privileged.
bq. Can we move hadoop-ozone/dist/src/main/blockade into integration-test
project? It seems a more logical choice to host fault injection test suites.
I have no strong preferences, I can live with both the options. Personally I
would prefer to keep it separated. For me the junit test based integration
tests and the blockade tests are different things. But it may be useful to move
out blockade tests from dist to a separated project if you prefer it.
bq. Do we want the test to run as a profile, or default "mvn verify" is good?
I would use a dedicated profile. The standard mvn verify (or mvn install) is
expected to be used by new contributors. I would keep that one as simple as
possible.
> Create a maven profile to run fault injection tests
> ---
>
> Key: HDDS-1458
> URL: https://issues.apache.org/jira/browse/HDDS-1458
> Project: Hadoop Distributed Data Store
> Issue Type: Test
>Reporter: Eric Yang
>Assignee: Eric Yang
>Priority: Major
> Attachments: HDDS-1458.001.patch
>
>
> Some fault injection tests have been written using blockade. It would be
> nice to have ability to start docker compose and exercise the blockade test
> cases against Ozone docker containers, and generate reports. This is
> optional integration tests to catch race conditions and fault tolerance
> defects.
> We can introduce a profile with id: it (short for integration tests). This
> will launch docker compose via maven-exec-plugin and run blockade to simulate
> container failures and timeout.
> Usage command:
> {code}
> mvn clean verify -Pit
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1384) TestBlockOutputStreamWithFailures is failing
[
https://issues.apache.org/jira/browse/HDDS-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830268#comment-16830268
]
Hudson commented on HDDS-1384:
--
SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16481 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/16481/])
HDDS-1384. TestBlockOutputStreamWithFailures is failing (elek: rev
dead9b4049484c31e0608956e53a9ef07a45819d)
* (edit)
hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/MiniOzoneClusterImpl.java
> TestBlockOutputStreamWithFailures is failing
>
>
> Key: HDDS-1384
> URL: https://issues.apache.org/jira/browse/HDDS-1384
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Reporter: Nanda kumar
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> TestBlockOutputStreamWithFailures is failing with the following error
> {noformat}
> 2019-04-04 18:52:43,240 INFO volume.ThrottledAsyncChecker
> (ThrottledAsyncChecker.java:schedule(140)) - Scheduling a check for
> org.apache.hadoop.ozone.container.common.volume.HddsVolume@1f6c0e8a
> 2019-04-04 18:52:43,240 INFO volume.HddsVolumeChecker
> (HddsVolumeChecker.java:checkAllVolumes(203)) - Scheduled health check for
> volume org.apache.hadoop.ozone.container.common.volume.HddsVolume@1f6c0e8a
> 2019-04-04 18:52:43,241 ERROR server.GrpcService
> (ExitUtils.java:terminate(133)) - Terminating with exit status 1: Failed to
> start Grpc server
> java.io.IOException: Failed to bind
> at
> org.apache.ratis.thirdparty.io.grpc.netty.NettyServer.start(NettyServer.java:253)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl.start(ServerImpl.java:166)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl.start(ServerImpl.java:81)
> at org.apache.ratis.grpc.server.GrpcService.startImpl(GrpcService.java:144)
> at org.apache.ratis.util.LifeCycle.startAndTransition(LifeCycle.java:202)
> at
> org.apache.ratis.server.impl.RaftServerRpcWithProxy.start(RaftServerRpcWithProxy.java:69)
> at
> org.apache.ratis.server.impl.RaftServerProxy.lambda$start$3(RaftServerProxy.java:300)
> at org.apache.ratis.util.LifeCycle.startAndTransition(LifeCycle.java:202)
> at
> org.apache.ratis.server.impl.RaftServerProxy.start(RaftServerProxy.java:298)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.start(XceiverServerRatis.java:419)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer.start(OzoneContainer.java:186)
> at
> org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.start(DatanodeStateMachine.java:169)
> at
> org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.lambda$startDaemon$0(DatanodeStateMachine.java:338)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:433)
> at sun.nio.ch.Net.bind(Net.java:425)
> at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
> at
> org.apache.ratis.thirdparty.io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:130)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:558)
> at
> org.apache.ratis.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1358)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:501)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:486)
> at
> org.apache.ratis.thirdparty.io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:1019)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel.bind(AbstractChannel.java:254)
> at
> org.apache.ratis.thirdparty.io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:366)
> at
> org.apache.ratis.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
> at
> org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
> at
> org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462)
> at
> org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
> at
> org.apache.ratis.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> ... 1 more
> {noformat}
--
This message
[jira] [Commented] (HDFS-13933) [JDK 11] SWebhdfsFileSystem related tests fail with hostname verification problems for "localhost"
[
https://issues.apache.org/jira/browse/HDFS-13933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830266#comment-16830266
]
Kitti Nanasi commented on HDFS-13933:
-
I was not correct in my previous comment, looking into it a bit more, these
tests fail because of a "javax.net.ssl.SSLPeerUnverifiedException: peer not
authenticated" exception from where sslSession.getPeerCertificates() is invoked
(it is used in 3 different places in our code).
I think it is because of the following bug in OpenJDK:
[https://bugs.openjdk.java.net/browse/JDK-8212885]
[https://bugs.openjdk.java.net/browse/JDK-8220723]
The issue affects OpenJDK 11.0.2 and it seems like it was backported to OpenJDK
11.0.3 and OpenJDK 12.0.1. I verified that these tests pass with OpenJDK 12.0.1.
> [JDK 11] SWebhdfsFileSystem related tests fail with hostname verification
> problems for "localhost"
> --
>
> Key: HDFS-13933
> URL: https://issues.apache.org/jira/browse/HDFS-13933
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: test
>Reporter: Andrew Purtell
>Priority: Minor
>
> Tests with issues:
> * TestHttpFSFWithSWebhdfsFileSystem
> * TestWebHdfsTokens
> * TestSWebHdfsFileContextMainOperations
> Possibly others. Failure looks like
> {noformat}
> java.io.IOException: localhost:50260: HTTPS hostname wrong: should be
>
> {noformat}
> These tests set up a trust store and use HTTPS connections, and with Java 11
> the client validation of the server name in the generated self-signed
> certificate is failing. Exceptions originate in the JRE's HTTP client
> library. How everything hooks together uses static initializers, static
> methods, JUnit MethodRules... There's a lot to unpack, not sure how to fix.
> This is Java 11+28
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1384) TestBlockOutputStreamWithFailures is failing
[
https://issues.apache.org/jira/browse/HDDS-1384?focusedWorklogId=235270=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-235270
]
ASF GitHub Bot logged work on HDDS-1384:
Author: ASF GitHub Bot
Created on: 30/Apr/19 12:41
Start Date: 30/Apr/19 12:41
Worklog Time Spent: 10m
Work Description: elek commented on pull request #750: HDDS-1384.
TestBlockOutputStreamWithFailures is failing
URL: https://github.com/apache/hadoop/pull/750
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 235270)
Time Spent: 1h 40m (was: 1.5h)
> TestBlockOutputStreamWithFailures is failing
>
>
> Key: HDDS-1384
> URL: https://issues.apache.org/jira/browse/HDDS-1384
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Reporter: Nanda kumar
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> TestBlockOutputStreamWithFailures is failing with the following error
> {noformat}
> 2019-04-04 18:52:43,240 INFO volume.ThrottledAsyncChecker
> (ThrottledAsyncChecker.java:schedule(140)) - Scheduling a check for
> org.apache.hadoop.ozone.container.common.volume.HddsVolume@1f6c0e8a
> 2019-04-04 18:52:43,240 INFO volume.HddsVolumeChecker
> (HddsVolumeChecker.java:checkAllVolumes(203)) - Scheduled health check for
> volume org.apache.hadoop.ozone.container.common.volume.HddsVolume@1f6c0e8a
> 2019-04-04 18:52:43,241 ERROR server.GrpcService
> (ExitUtils.java:terminate(133)) - Terminating with exit status 1: Failed to
> start Grpc server
> java.io.IOException: Failed to bind
> at
> org.apache.ratis.thirdparty.io.grpc.netty.NettyServer.start(NettyServer.java:253)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl.start(ServerImpl.java:166)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl.start(ServerImpl.java:81)
> at org.apache.ratis.grpc.server.GrpcService.startImpl(GrpcService.java:144)
> at org.apache.ratis.util.LifeCycle.startAndTransition(LifeCycle.java:202)
> at
> org.apache.ratis.server.impl.RaftServerRpcWithProxy.start(RaftServerRpcWithProxy.java:69)
> at
> org.apache.ratis.server.impl.RaftServerProxy.lambda$start$3(RaftServerProxy.java:300)
> at org.apache.ratis.util.LifeCycle.startAndTransition(LifeCycle.java:202)
> at
> org.apache.ratis.server.impl.RaftServerProxy.start(RaftServerProxy.java:298)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.start(XceiverServerRatis.java:419)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer.start(OzoneContainer.java:186)
> at
> org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.start(DatanodeStateMachine.java:169)
> at
> org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.lambda$startDaemon$0(DatanodeStateMachine.java:338)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:433)
> at sun.nio.ch.Net.bind(Net.java:425)
> at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
> at
> org.apache.ratis.thirdparty.io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:130)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:558)
> at
> org.apache.ratis.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1358)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:501)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:486)
> at
> org.apache.ratis.thirdparty.io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:1019)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel.bind(AbstractChannel.java:254)
> at
> org.apache.ratis.thirdparty.io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:366)
> at
> org.apache.ratis.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
> at
>
[jira] [Updated] (HDDS-1384) TestBlockOutputStreamWithFailures is failing
[
https://issues.apache.org/jira/browse/HDDS-1384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Elek, Marton updated HDDS-1384:
---
Resolution: Fixed
Status: Resolved (was: Patch Available)
> TestBlockOutputStreamWithFailures is failing
>
>
> Key: HDDS-1384
> URL: https://issues.apache.org/jira/browse/HDDS-1384
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Reporter: Nanda kumar
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> TestBlockOutputStreamWithFailures is failing with the following error
> {noformat}
> 2019-04-04 18:52:43,240 INFO volume.ThrottledAsyncChecker
> (ThrottledAsyncChecker.java:schedule(140)) - Scheduling a check for
> org.apache.hadoop.ozone.container.common.volume.HddsVolume@1f6c0e8a
> 2019-04-04 18:52:43,240 INFO volume.HddsVolumeChecker
> (HddsVolumeChecker.java:checkAllVolumes(203)) - Scheduled health check for
> volume org.apache.hadoop.ozone.container.common.volume.HddsVolume@1f6c0e8a
> 2019-04-04 18:52:43,241 ERROR server.GrpcService
> (ExitUtils.java:terminate(133)) - Terminating with exit status 1: Failed to
> start Grpc server
> java.io.IOException: Failed to bind
> at
> org.apache.ratis.thirdparty.io.grpc.netty.NettyServer.start(NettyServer.java:253)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl.start(ServerImpl.java:166)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl.start(ServerImpl.java:81)
> at org.apache.ratis.grpc.server.GrpcService.startImpl(GrpcService.java:144)
> at org.apache.ratis.util.LifeCycle.startAndTransition(LifeCycle.java:202)
> at
> org.apache.ratis.server.impl.RaftServerRpcWithProxy.start(RaftServerRpcWithProxy.java:69)
> at
> org.apache.ratis.server.impl.RaftServerProxy.lambda$start$3(RaftServerProxy.java:300)
> at org.apache.ratis.util.LifeCycle.startAndTransition(LifeCycle.java:202)
> at
> org.apache.ratis.server.impl.RaftServerProxy.start(RaftServerProxy.java:298)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.start(XceiverServerRatis.java:419)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer.start(OzoneContainer.java:186)
> at
> org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.start(DatanodeStateMachine.java:169)
> at
> org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.lambda$startDaemon$0(DatanodeStateMachine.java:338)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:433)
> at sun.nio.ch.Net.bind(Net.java:425)
> at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
> at
> org.apache.ratis.thirdparty.io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:130)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:558)
> at
> org.apache.ratis.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1358)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:501)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:486)
> at
> org.apache.ratis.thirdparty.io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:1019)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel.bind(AbstractChannel.java:254)
> at
> org.apache.ratis.thirdparty.io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:366)
> at
> org.apache.ratis.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
> at
> org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:404)
> at
> org.apache.ratis.thirdparty.io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:462)
> at
> org.apache.ratis.thirdparty.io.netty.util.concurrent.SingleThreadEventExecutor$5.run(SingleThreadEventExecutor.java:897)
> at
> org.apache.ratis.thirdparty.io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
> ... 1 more
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1384) TestBlockOutputStreamWithFailures is failing
[
https://issues.apache.org/jira/browse/HDDS-1384?focusedWorklogId=235245=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-235245
]
ASF GitHub Bot logged work on HDDS-1384:
Author: ASF GitHub Bot
Created on: 30/Apr/19 12:03
Start Date: 30/Apr/19 12:03
Worklog Time Spent: 10m
Work Description: elek commented on issue #750: HDDS-1384.
TestBlockOutputStreamWithFailures is failing
URL: https://github.com/apache/hadoop/pull/750#issuecomment-487925530
Thanks the review @bshashikant. I will merge it to trunk. Hopefully it will
help to avoid a few intermittent tests.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 235245)
Time Spent: 1.5h (was: 1h 20m)
> TestBlockOutputStreamWithFailures is failing
>
>
> Key: HDDS-1384
> URL: https://issues.apache.org/jira/browse/HDDS-1384
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Reporter: Nanda kumar
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> TestBlockOutputStreamWithFailures is failing with the following error
> {noformat}
> 2019-04-04 18:52:43,240 INFO volume.ThrottledAsyncChecker
> (ThrottledAsyncChecker.java:schedule(140)) - Scheduling a check for
> org.apache.hadoop.ozone.container.common.volume.HddsVolume@1f6c0e8a
> 2019-04-04 18:52:43,240 INFO volume.HddsVolumeChecker
> (HddsVolumeChecker.java:checkAllVolumes(203)) - Scheduled health check for
> volume org.apache.hadoop.ozone.container.common.volume.HddsVolume@1f6c0e8a
> 2019-04-04 18:52:43,241 ERROR server.GrpcService
> (ExitUtils.java:terminate(133)) - Terminating with exit status 1: Failed to
> start Grpc server
> java.io.IOException: Failed to bind
> at
> org.apache.ratis.thirdparty.io.grpc.netty.NettyServer.start(NettyServer.java:253)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl.start(ServerImpl.java:166)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl.start(ServerImpl.java:81)
> at org.apache.ratis.grpc.server.GrpcService.startImpl(GrpcService.java:144)
> at org.apache.ratis.util.LifeCycle.startAndTransition(LifeCycle.java:202)
> at
> org.apache.ratis.server.impl.RaftServerRpcWithProxy.start(RaftServerRpcWithProxy.java:69)
> at
> org.apache.ratis.server.impl.RaftServerProxy.lambda$start$3(RaftServerProxy.java:300)
> at org.apache.ratis.util.LifeCycle.startAndTransition(LifeCycle.java:202)
> at
> org.apache.ratis.server.impl.RaftServerProxy.start(RaftServerProxy.java:298)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.start(XceiverServerRatis.java:419)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer.start(OzoneContainer.java:186)
> at
> org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.start(DatanodeStateMachine.java:169)
> at
> org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.lambda$startDaemon$0(DatanodeStateMachine.java:338)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:433)
> at sun.nio.ch.Net.bind(Net.java:425)
> at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
> at
> org.apache.ratis.thirdparty.io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:130)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:558)
> at
> org.apache.ratis.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1358)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:501)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:486)
> at
> org.apache.ratis.thirdparty.io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:1019)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel.bind(AbstractChannel.java:254)
> at
> org.apache.ratis.thirdparty.io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:366)
> at
>
[jira] [Commented] (HDFS-13677) Dynamic refresh Disk configuration results in overwriting VolumeMap
[ https://issues.apache.org/jira/browse/HDFS-13677?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830191#comment-16830191 ] Stephen O'Donnell commented on HDFS-13677: -- [~arpitagarwal] I have uploaded a patch based on the 2.9 branch. Two changes were needed: 1. Remove Lamdas and replace with nested for loops. 2. Replace String.join with StringUtils.join as String.join does not exist in Java 7. I ran both the changed test classes locally and they all passed and the code compiles under Java 7. > Dynamic refresh Disk configuration results in overwriting VolumeMap > --- > > Key: HDFS-13677 > URL: https://issues.apache.org/jira/browse/HDFS-13677 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: xuzq >Assignee: xuzq >Priority: Blocker > Fix For: 3.3.0, 3.2.1, 3.1.3 > > Attachments: HDFS-13677-001.patch, HDFS-13677-002-2.9-branch.patch, > HDFS-13677-002.patch, image-2018-06-14-13-05-54-354.png, > image-2018-06-14-13-10-24-032.png > > > When I added a new disk by dynamically refreshing the configuration, an > exception "FileNotFound while finding block" was caused. > > The steps are as follows: > 1.Change the hdfs-site.xml of DataNode to add a new disk. > 2.Refresh the configuration by "./bin/hdfs dfsadmin -reconfig datanode > :50020 start" > > The error is like: > ``` > VolumeScannerThread(/media/disk5/hdfs/dn): FileNotFound while finding block > BP-233501496-*.*.*.*-1514185698256:blk_1620868560_547245090 on volume > /media/disk5/hdfs/dn > org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not > found for BP-1997955181-*.*.*.*-1514186468560:blk_1090885868_17145082 > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.getReplica(BlockSender.java:471) > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.(BlockSender.java:240) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:553) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:148) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:254) > at java.lang.Thread.run(Thread.java:748) > ``` > I added some logs for confirmation, as follows: > Log Code like: > !image-2018-06-14-13-05-54-354.png! > And the result is like: > !image-2018-06-14-13-10-24-032.png! > The Size of 'VolumeMap' has been reduced, and We found the 'VolumeMap' to be > overridden by the new Disk Block by the method 'ReplicaMap.addAll(ReplicaMap > other)'. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13677) Dynamic refresh Disk configuration results in overwriting VolumeMap
[ https://issues.apache.org/jira/browse/HDFS-13677?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Stephen O'Donnell updated HDFS-13677: - Attachment: HDFS-13677-002-2.9-branch.patch > Dynamic refresh Disk configuration results in overwriting VolumeMap > --- > > Key: HDFS-13677 > URL: https://issues.apache.org/jira/browse/HDFS-13677 > Project: Hadoop HDFS > Issue Type: Bug >Reporter: xuzq >Assignee: xuzq >Priority: Blocker > Fix For: 3.3.0, 3.2.1, 3.1.3 > > Attachments: HDFS-13677-001.patch, HDFS-13677-002-2.9-branch.patch, > HDFS-13677-002.patch, image-2018-06-14-13-05-54-354.png, > image-2018-06-14-13-10-24-032.png > > > When I added a new disk by dynamically refreshing the configuration, an > exception "FileNotFound while finding block" was caused. > > The steps are as follows: > 1.Change the hdfs-site.xml of DataNode to add a new disk. > 2.Refresh the configuration by "./bin/hdfs dfsadmin -reconfig datanode > :50020 start" > > The error is like: > ``` > VolumeScannerThread(/media/disk5/hdfs/dn): FileNotFound while finding block > BP-233501496-*.*.*.*-1514185698256:blk_1620868560_547245090 on volume > /media/disk5/hdfs/dn > org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not > found for BP-1997955181-*.*.*.*-1514186468560:blk_1090885868_17145082 > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.getReplica(BlockSender.java:471) > at > org.apache.hadoop.hdfs.server.datanode.BlockSender.(BlockSender.java:240) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:553) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:148) > at > org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:103) > at > org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:254) > at java.lang.Thread.run(Thread.java:748) > ``` > I added some logs for confirmation, as follows: > Log Code like: > !image-2018-06-14-13-05-54-354.png! > And the result is like: > !image-2018-06-14-13-10-24-032.png! > The Size of 'VolumeMap' has been reduced, and We found the 'VolumeMap' to be > overridden by the new Disk Block by the method 'ReplicaMap.addAll(ReplicaMap > other)'. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1384) TestBlockOutputStreamWithFailures is failing
[
https://issues.apache.org/jira/browse/HDDS-1384?focusedWorklogId=235208=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-235208
]
ASF GitHub Bot logged work on HDDS-1384:
Author: ASF GitHub Bot
Created on: 30/Apr/19 09:44
Start Date: 30/Apr/19 09:44
Worklog Time Spent: 10m
Work Description: bshashikant commented on issue #750: HDDS-1384.
TestBlockOutputStreamWithFailures is failing
URL: https://github.com/apache/hadoop/pull/750#issuecomment-487890532
Thanks @elek for working on this. The change looks good to me. +1.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 235208)
Time Spent: 1h 20m (was: 1h 10m)
> TestBlockOutputStreamWithFailures is failing
>
>
> Key: HDDS-1384
> URL: https://issues.apache.org/jira/browse/HDDS-1384
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Reporter: Nanda kumar
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 20m
> Remaining Estimate: 0h
>
> TestBlockOutputStreamWithFailures is failing with the following error
> {noformat}
> 2019-04-04 18:52:43,240 INFO volume.ThrottledAsyncChecker
> (ThrottledAsyncChecker.java:schedule(140)) - Scheduling a check for
> org.apache.hadoop.ozone.container.common.volume.HddsVolume@1f6c0e8a
> 2019-04-04 18:52:43,240 INFO volume.HddsVolumeChecker
> (HddsVolumeChecker.java:checkAllVolumes(203)) - Scheduled health check for
> volume org.apache.hadoop.ozone.container.common.volume.HddsVolume@1f6c0e8a
> 2019-04-04 18:52:43,241 ERROR server.GrpcService
> (ExitUtils.java:terminate(133)) - Terminating with exit status 1: Failed to
> start Grpc server
> java.io.IOException: Failed to bind
> at
> org.apache.ratis.thirdparty.io.grpc.netty.NettyServer.start(NettyServer.java:253)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl.start(ServerImpl.java:166)
> at
> org.apache.ratis.thirdparty.io.grpc.internal.ServerImpl.start(ServerImpl.java:81)
> at org.apache.ratis.grpc.server.GrpcService.startImpl(GrpcService.java:144)
> at org.apache.ratis.util.LifeCycle.startAndTransition(LifeCycle.java:202)
> at
> org.apache.ratis.server.impl.RaftServerRpcWithProxy.start(RaftServerRpcWithProxy.java:69)
> at
> org.apache.ratis.server.impl.RaftServerProxy.lambda$start$3(RaftServerProxy.java:300)
> at org.apache.ratis.util.LifeCycle.startAndTransition(LifeCycle.java:202)
> at
> org.apache.ratis.server.impl.RaftServerProxy.start(RaftServerProxy.java:298)
> at
> org.apache.hadoop.ozone.container.common.transport.server.ratis.XceiverServerRatis.start(XceiverServerRatis.java:419)
> at
> org.apache.hadoop.ozone.container.ozoneimpl.OzoneContainer.start(OzoneContainer.java:186)
> at
> org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.start(DatanodeStateMachine.java:169)
> at
> org.apache.hadoop.ozone.container.common.statemachine.DatanodeStateMachine.lambda$startDaemon$0(DatanodeStateMachine.java:338)
> at java.lang.Thread.run(Thread.java:748)
> Caused by: java.net.BindException: Address already in use
> at sun.nio.ch.Net.bind0(Native Method)
> at sun.nio.ch.Net.bind(Net.java:433)
> at sun.nio.ch.Net.bind(Net.java:425)
> at sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:223)
> at
> org.apache.ratis.thirdparty.io.netty.channel.socket.nio.NioServerSocketChannel.doBind(NioServerSocketChannel.java:130)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel$AbstractUnsafe.bind(AbstractChannel.java:558)
> at
> org.apache.ratis.thirdparty.io.netty.channel.DefaultChannelPipeline$HeadContext.bind(DefaultChannelPipeline.java:1358)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.invokeBind(AbstractChannelHandlerContext.java:501)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannelHandlerContext.bind(AbstractChannelHandlerContext.java:486)
> at
> org.apache.ratis.thirdparty.io.netty.channel.DefaultChannelPipeline.bind(DefaultChannelPipeline.java:1019)
> at
> org.apache.ratis.thirdparty.io.netty.channel.AbstractChannel.bind(AbstractChannel.java:254)
> at
> org.apache.ratis.thirdparty.io.netty.bootstrap.AbstractBootstrap$2.run(AbstractBootstrap.java:366)
> at
> org.apache.ratis.thirdparty.io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:163)
> at
>
[jira] [Resolved] (HDDS-1282) TestFailureHandlingByClient causes a jvm exit
[
https://issues.apache.org/jira/browse/HDDS-1282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shashikant Banerjee resolved HDDS-1282.
---
Resolution: Fixed
Fix Version/s: 0.5.0
As [~elek] explained, this issue does not exist any more and the other issue is
tracked by HDDS-1384. Resolving this.
> TestFailureHandlingByClient causes a jvm exit
> -
>
> Key: HDDS-1282
> URL: https://issues.apache.org/jira/browse/HDDS-1282
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: test
>Reporter: Mukul Kumar Singh
>Assignee: Shashikant Banerjee
>Priority: Major
> Fix For: 0.5.0
>
> Attachments: HDDS-1282.001.patch,
> org.apache.hadoop.ozone.client.rpc.TestFailureHandlingByClient-output.txt
>
>
> The test causes jvm exit because the test exits prematurely.
> {code}
> [ERROR] org.apache.hadoop.ozone.client.rpc.TestFailureHandlingByClient
> [ERROR] org.apache.maven.surefire.booter.SurefireBooterForkException:
> ExecutionException The forked VM terminated without properly saying goodbye.
> VM crash or System.exit called?
> [ERROR] Command was /bin/sh -c cd
> /Users/msingh/code/apache/ozone/oz_new1/hadoop-ozone/integration-test &&
> /Library/Java/JavaVirtualMachines/jdk1.8.0_171.jdk/Contents/Home/jre/bin/java
> -Xmx2048m -XX:+HeapDumpOnOutOfMemoryError -jar
> /Users/msingh/code/apache/ozone/oz_new1/hadoop-ozone/integration-test/target/surefire/surefirebooter5405606309417840457.jar
>
> /Users/msingh/code/apache/ozone/oz_new1/hadoop-ozone/integration-test/target/surefire
> 2019-03-13T23-31-09_018-jvmRun1 surefire5934599060460829594tmp
> surefire_1202723709650989744795tmp
> [ERROR] Error occurred in starting fork, check output in log
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-973) HDDS/Ozone fail to build on Windows
[ https://issues.apache.org/jira/browse/HDDS-973?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830117#comment-16830117 ] Elek, Marton commented on HDDS-973: --- It seems to be fixed by HDDS-1322, but the commit is missing from ozone-0.4. I will cherry-pick it to the ozone-0.4. > HDDS/Ozone fail to build on Windows > --- > > Key: HDDS-973 > URL: https://issues.apache.org/jira/browse/HDDS-973 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Sammi Chen >Assignee: Xiaoyu Yao >Priority: Major > Attachments: HDDS-973.001.patch > > > Thanks [~Sammi] for reporting the issue on building hdds/ozone with Windows > OS. I can repro it locally and will post a fix shortly. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1395) Key write fails with BlockOutputStream has been closed exception
[
https://issues.apache.org/jira/browse/HDDS-1395?focusedWorklogId=235205=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-235205
]
ASF GitHub Bot logged work on HDDS-1395:
Author: ASF GitHub Bot
Created on: 30/Apr/19 09:30
Start Date: 30/Apr/19 09:30
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on issue #749: HDDS-1395. Key
write fails with BlockOutputStream has been closed exception
URL: https://github.com/apache/hadoop/pull/749#issuecomment-487883662
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|::|--:|:|:|
| 0 | reexec | 0 | Docker mode activated. |
| -1 | patch | 6 | https://github.com/apache/hadoop/pull/749 does not apply
to trunk. Rebase required? Wrong Branch? See
https://wiki.apache.org/hadoop/HowToContribute for help. |
| Subsystem | Report/Notes |
|--:|:-|
| GITHUB PR | https://github.com/apache/hadoop/pull/749 |
| Console output |
https://builds.apache.org/job/hadoop-multibranch/job/PR-749/7/console |
| Powered by | Apache Yetus 0.9.0 http://yetus.apache.org |
This message was automatically generated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 235205)
Time Spent: 3h 50m (was: 3h 40m)
> Key write fails with BlockOutputStream has been closed exception
>
>
> Key: HDDS-1395
> URL: https://issues.apache.org/jira/browse/HDDS-1395
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
> Components: Ozone Client
>Affects Versions: 0.4.0
>Reporter: Mukul Kumar Singh
>Assignee: Shashikant Banerjee
>Priority: Major
> Labels: MiniOzoneChaosCluster, pull-request-available
> Attachments: HDDS-1395.000.patch, HDDS-1395.001.patch
>
> Time Spent: 3h 50m
> Remaining Estimate: 0h
>
> Key write fails with BlockOutputStream has been closed
> {code}
> 2019-04-05 11:24:47,770 ERROR ozone.MiniOzoneLoadGenerator
> (MiniOzoneLoadGenerator.java:load(102)) - LOADGEN: Create
> key:pool-431-thread-9-2092651262 failed with exception, but skipping
> java.io.IOException: BlockOutputStream has been closed.
> at
> org.apache.hadoop.hdds.scm.storage.BlockOutputStream.checkOpen(BlockOutputStream.java:662)
> at
> org.apache.hadoop.hdds.scm.storage.BlockOutputStream.write(BlockOutputStream.java:245)
> at
> org.apache.hadoop.ozone.client.io.BlockOutputStreamEntry.write(BlockOutputStreamEntry.java:131)
> at
> org.apache.hadoop.ozone.client.io.KeyOutputStream.handleWrite(KeyOutputStream.java:325)
> at
> org.apache.hadoop.ozone.client.io.KeyOutputStream.write(KeyOutputStream.java:287)
> at
> org.apache.hadoop.ozone.client.io.OzoneOutputStream.write(OzoneOutputStream.java:49)
> at java.io.OutputStream.write(OutputStream.java:75)
> at
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.load(MiniOzoneLoadGenerator.java:100)
> at
> org.apache.hadoop.ozone.MiniOzoneLoadGenerator.lambda$startIO$0(MiniOzoneLoadGenerator.java:143)
> at
> java.util.concurrent.CompletableFuture$AsyncRun.run(CompletableFuture.java:1626)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> at java.lang.Thread.run(Thread.java:748)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13522) Support observer node from Router-Based Federation
[
https://issues.apache.org/jira/browse/HDFS-13522?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830075#comment-16830075
]
Surendra Singh Lilhore commented on HDFS-13522:
---
Thank you all for discussion.
I tried observer read in router and attached initial patch for this. Patch do
following things
# Store observer state in state store.
# Keep the last transaction id for all the namespace in memory.
# Send the write direct to active NN.
# For read first do the msync() and send read call to observer NN.
# Added two property to support observer read in Router, a)
*dfs.federation.router.observer.read.enable*, b)
*dfs.federation.router.observer.auto-msync-period*.
# Now in client side no change is required.
This is initial patch, others can give their suggestion to improve this.
> Support observer node from Router-Based Federation
> --
>
> Key: HDFS-13522
> URL: https://issues.apache.org/jira/browse/HDFS-13522
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: federation, namenode
>Reporter: Erik Krogen
>Assignee: Chao Sun
>Priority: Major
> Attachments: HDFS-13522.001.patch
>
>
> Changes will need to occur to the router to support the new observer node.
> One such change will be to make the router understand the observer state,
> e.g. {{FederationNamenodeServiceState}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13522) Support observer node from Router-Based Federation
[
https://issues.apache.org/jira/browse/HDFS-13522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-13522:
--
Attachment: HDFS-13522.001.patch
> Support observer node from Router-Based Federation
> --
>
> Key: HDFS-13522
> URL: https://issues.apache.org/jira/browse/HDFS-13522
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: federation, namenode
>Reporter: Erik Krogen
>Assignee: Chao Sun
>Priority: Major
> Attachments: HDFS-13522.001.patch
>
>
> Changes will need to occur to the router to support the new observer node.
> One such change will be to make the router understand the observer state,
> e.g. {{FederationNamenodeServiceState}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13522) Support observer node from Router-Based Federation
[
https://issues.apache.org/jira/browse/HDFS-13522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-13522:
--
Attachment: (was: HDFS-13522.001.patch)
> Support observer node from Router-Based Federation
> --
>
> Key: HDFS-13522
> URL: https://issues.apache.org/jira/browse/HDFS-13522
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: federation, namenode
>Reporter: Erik Krogen
>Assignee: Chao Sun
>Priority: Major
>
> Changes will need to occur to the router to support the new observer node.
> One such change will be to make the router understand the observer state,
> e.g. {{FederationNamenodeServiceState}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-13522) Support observer node from Router-Based Federation
[
https://issues.apache.org/jira/browse/HDFS-13522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Surendra Singh Lilhore updated HDFS-13522:
--
Attachment: HDFS-13522.001.patch
> Support observer node from Router-Based Federation
> --
>
> Key: HDFS-13522
> URL: https://issues.apache.org/jira/browse/HDFS-13522
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: federation, namenode
>Reporter: Erik Krogen
>Assignee: Chao Sun
>Priority: Major
> Attachments: HDFS-13522.001.patch
>
>
> Changes will need to occur to the router to support the new observer node.
> One such change will be to make the router understand the observer state,
> e.g. {{FederationNamenodeServiceState}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1448) RatisPipelineProvider should only consider open pipeline while excluding dn for pipeline allocation
[ https://issues.apache.org/jira/browse/HDDS-1448?focusedWorklogId=235165=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-235165 ] ASF GitHub Bot logged work on HDDS-1448: Author: ASF GitHub Bot Created on: 30/Apr/19 08:27 Start Date: 30/Apr/19 08:27 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #786: HDDS-1448 : RatisPipelineProvider should only consider open pipeline … URL: https://github.com/apache/hadoop/pull/786#issuecomment-487860952 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 24 | Docker mode activated. | ||| _ Prechecks _ | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 2 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 61 | Maven dependency ordering for branch | | +1 | mvninstall | 1035 | trunk passed | | +1 | compile | 962 | trunk passed | | +1 | checkstyle | 135 | trunk passed | | +1 | mvnsite | 107 | trunk passed | | +1 | shadedclient | 976 | branch has no errors when building and testing our client artifacts. | | 0 | findbugs | 0 | Skipped patched modules with no Java source: hadoop-ozone/integration-test | | +1 | findbugs | 50 | trunk passed | | +1 | javadoc | 60 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 24 | Maven dependency ordering for patch | | +1 | mvninstall | 64 | the patch passed | | +1 | compile | 925 | the patch passed | | +1 | javac | 925 | the patch passed | | +1 | checkstyle | 139 | the patch passed | | +1 | mvnsite | 88 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 685 | patch has no errors when building and testing our client artifacts. | | 0 | findbugs | 0 | Skipped patched modules with no Java source: hadoop-ozone/integration-test | | +1 | findbugs | 58 | the patch passed | | +1 | javadoc | 66 | the patch passed | ||| _ Other Tests _ | | -1 | unit | 117 | server-scm in the patch failed. | | -1 | unit | 1289 | integration-test in the patch failed. | | +1 | asflicense | 54 | The patch does not generate ASF License warnings. | | | | 6953 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdds.scm.block.TestBlockManager | | | hadoop.ozone.client.rpc.TestOzoneClientRetriesOnException | | | hadoop.ozone.client.rpc.TestBlockOutputStreamWithFailures | | | hadoop.ozone.client.rpc.TestCommitWatcher | | | hadoop.ozone.client.rpc.TestOzoneRpcClientWithRatis | | | hadoop.hdds.scm.pipeline.TestSCMRestart | | | hadoop.hdds.scm.safemode.TestSCMSafeModeWithPipelineRules | | | hadoop.ozone.client.rpc.TestCloseContainerHandlingByClient | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-786/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/786 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux fb760c7a563b 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / 7fbaa7d | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | findbugs | v3.1.0-RC1 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-786/1/artifact/out/patch-unit-hadoop-hdds_server-scm.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-786/1/artifact/out/patch-unit-hadoop-ozone_integration-test.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-786/1/testReport/ | | Max. process+thread count | 5402 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/server-scm hadoop-ozone/integration-test U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-786/1/console | | Powered by | Apache Yetus 0.9.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 235165) Time Spent: 0.5h (was: 20m) > RatisPipelineProvider should only consider open pipeline while
[jira] [Commented] (HDFS-13189) Standby NameNode should roll active edit log when checkpointing
[
https://issues.apache.org/jira/browse/HDFS-13189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16830014#comment-16830014
]
xuzq commented on HDFS-13189:
-
I think the edit log roll and tailing edits need to do in separate threads in
Standby, because they all independency.
There is one other problem to solve. In current trunk, both Observers and
Standbys can rollEdit, it may produce too many small segments. And will
produce some negative effects, such as too many small files in each
JournalNode.
> Standby NameNode should roll active edit log when checkpointing
> ---
>
> Key: HDFS-13189
> URL: https://issues.apache.org/jira/browse/HDFS-13189
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: namenode
>Reporter: Chao Sun
>Priority: Minor
>
> When the SBN is doing checkpointing, it will hold the {{cpLock}}. In the
> current implementation of edit log tailer thread, it will first check and
> roll active edit log, and then tail and apply edits. In the case of
> checkpointing, it will be blocked on the {{cpLock}} and will not roll the
> edit log.
> It seems there is no dependency between the edit log roll and tailing edits,
> so a better may be to do these in separate threads. This will be helpful for
> people who uses the observer feature without in-progress edit log tailing.
> An alternative is to configure
> {{dfs.namenode.edit.log.autoroll.multiplier.threshold}} and
> {{dfs.namenode.edit.log.autoroll.check.interval.ms}} to let ANN roll its own
> log more frequently in case SBN is stuck on the lock.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1448) RatisPipelineProvider should only consider open pipeline while excluding dn for pipeline allocation
[ https://issues.apache.org/jira/browse/HDDS-1448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Aravindan Vijayan updated HDDS-1448: Status: Patch Available (was: Open) > RatisPipelineProvider should only consider open pipeline while excluding dn > for pipeline allocation > --- > > Key: HDDS-1448 > URL: https://issues.apache.org/jira/browse/HDDS-1448 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Affects Versions: 0.3.0 >Reporter: Mukul Kumar Singh >Assignee: Aravindan Vijayan >Priority: Major > Labels: MiniOzoneChaosCluster, pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > While allocation pipelines, Ratis pipeline provider considers all the > pipelines irrespective of the state of the pipeline. This can lead to case > where all the datanodes are up but the pipelines are in closing state in SCM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1448) RatisPipelineProvider should only consider open pipeline while excluding dn for pipeline allocation
[ https://issues.apache.org/jira/browse/HDDS-1448?focusedWorklogId=235089=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-235089 ] ASF GitHub Bot logged work on HDDS-1448: Author: ASF GitHub Bot Created on: 30/Apr/19 06:30 Start Date: 30/Apr/19 06:30 Worklog Time Spent: 10m Work Description: avijayanhwx commented on issue #786: HDDS-1448 : RatisPipelineProvider should only consider open pipeline … URL: https://github.com/apache/hadoop/pull/786#issuecomment-487834169 /label ozone This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 235089) Time Spent: 20m (was: 10m) > RatisPipelineProvider should only consider open pipeline while excluding dn > for pipeline allocation > --- > > Key: HDDS-1448 > URL: https://issues.apache.org/jira/browse/HDDS-1448 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Affects Versions: 0.3.0 >Reporter: Mukul Kumar Singh >Assignee: Aravindan Vijayan >Priority: Major > Labels: MiniOzoneChaosCluster, pull-request-available > Time Spent: 20m > Remaining Estimate: 0h > > While allocation pipelines, Ratis pipeline provider considers all the > pipelines irrespective of the state of the pipeline. This can lead to case > where all the datanodes are up but the pipelines are in closing state in SCM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1448) RatisPipelineProvider should only consider open pipeline while excluding dn for pipeline allocation
[ https://issues.apache.org/jira/browse/HDDS-1448?focusedWorklogId=235087=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-235087 ] ASF GitHub Bot logged work on HDDS-1448: Author: ASF GitHub Bot Created on: 30/Apr/19 06:29 Start Date: 30/Apr/19 06:29 Worklog Time Spent: 10m Work Description: avijayanhwx commented on pull request #786: HDDS-1448 : RatisPipelineProvider should only consider open pipeline … URL: https://github.com/apache/hadoop/pull/786 …while excluding dn for pipeline allocation. While allocation pipelines, Ratis pipeline provider considers all the pipelines irrespective of the state of the pipeline. This can lead to case where all the datanodes are up but the pipelines are in closing state in SCM. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 235087) Time Spent: 10m Remaining Estimate: 0h > RatisPipelineProvider should only consider open pipeline while excluding dn > for pipeline allocation > --- > > Key: HDDS-1448 > URL: https://issues.apache.org/jira/browse/HDDS-1448 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Affects Versions: 0.3.0 >Reporter: Mukul Kumar Singh >Assignee: Aravindan Vijayan >Priority: Major > Labels: MiniOzoneChaosCluster, pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > While allocation pipelines, Ratis pipeline provider considers all the > pipelines irrespective of the state of the pipeline. This can lead to case > where all the datanodes are up but the pipelines are in closing state in SCM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1448) RatisPipelineProvider should only consider open pipeline while excluding dn for pipeline allocation
[ https://issues.apache.org/jira/browse/HDDS-1448?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-1448: - Labels: MiniOzoneChaosCluster pull-request-available (was: MiniOzoneChaosCluster) > RatisPipelineProvider should only consider open pipeline while excluding dn > for pipeline allocation > --- > > Key: HDDS-1448 > URL: https://issues.apache.org/jira/browse/HDDS-1448 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Affects Versions: 0.3.0 >Reporter: Mukul Kumar Singh >Assignee: Aravindan Vijayan >Priority: Major > Labels: MiniOzoneChaosCluster, pull-request-available > > While allocation pipelines, Ratis pipeline provider considers all the > pipelines irrespective of the state of the pipeline. This can lead to case > where all the datanodes are up but the pipelines are in closing state in SCM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
