[jira] [Updated] (HDFS-14315) Add more detailed log message when decreasing replication factor < max in snapshots
[
https://issues.apache.org/jira/browse/HDFS-14315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daisuke Kobayashi updated HDFS-14315:
-
Attachment: HDFS-14315.001.patch
> Add more detailed log message when decreasing replication factor < max in
> snapshots
> ---
>
> Key: HDFS-14315
> URL: https://issues.apache.org/jira/browse/HDFS-14315
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.1.2
>Reporter: Daisuke Kobayashi
>Assignee: Daisuke Kobayashi
>Priority: Minor
> Attachments: HDFS-14315.000.patch, HDFS-14315.001.patch
>
>
> When changing replication factor for a given file, the following 3 types of
> logging appear in the namenode log, but more detailed message would be useful
> especially when the file is in snapshot(s).
> {noformat}
> Decreasing replication from X to Y for FILE
> Increasing replication from X to Y for FILE
> Replication remains unchanged at X for FILE
> {noformat}
> I have added additional log messages as well as further test scenarios to
> org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotReplication#testReplicationWithSnapshot.
> The test sequence is:
> 1) A file is created with replication factor 1 (there are 5 datanodes)
> 2) Take a snapshot and increase the factor by 1. Continue this loop up to 5.
> 3) Setrep back to 3, but the target replication is decided to 4, which is the
> maximum in snapshots.
> {noformat}
> 2019-02-25 17:17:26,800 [IPC Server handler 9 on default port 55726] INFO
> namenode.FSDirectory (FSDirAttrOp.java:unprotectedSetReplication(408)) -
> Decreasing replication from 5 to 4 for /TestSnapshot/sub1/file1. Requested
> value is 3, but 4 is the maximum in snapshots
> {noformat}
> 4) Setrep to 3 again, but it's unchanged as follows. Both 3) and 4) are
> expected.
> {noformat}
> 2019-02-25 17:17:26,804 [IPC Server handler 6 on default port 55726] INFO
> namenode.FSDirectory (FSDirAttrOp.java:unprotectedSetReplication(420)) -
> Replication remains unchanged at 4 for /TestSnapshot/sub1/file1 . Requested
> value is 3, but 4 is the maximum in snapshots.
> {noformat}
> 5) Make sure the number of replicas in datanodes remains 4.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14314) fullBlockReportLeaseId should be reset after registering to NN
[
https://issues.apache.org/jira/browse/HDFS-14314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
star updated HDFS-14314:
Attachment: HDFS-14314-trunk.001.patch
> fullBlockReportLeaseId should be reset after registering to NN
> --
>
> Key: HDFS-14314
> URL: https://issues.apache.org/jira/browse/HDFS-14314
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.8.4
> Environment:
>
>
>Reporter: star
>Priority: Critical
> Fix For: 2.8.4
>
> Attachments: HDFS-14314-trunk.001.patch, HDFS-14314.0.patch,
> HDFS-14314.2.patch, HDFS-14314.patch
>
>
> since HDFS-7923 ,to rate-limit DN block report, DN will ask for a full
> block lease id from active NN before sending full block to NN. Then DN will
> send full block report together with lease id. If the lease id is invalid, NN
> will reject the full block report and log "not in the pending set".
> In a case when DN is doing full block reporting while NN is restarted.
> It happens that DN will later send a full block report with lease id
> ,acquired from previous NN instance, which is invalid to the new NN instance.
> Though DN recognized the new NN instance by heartbeat and reregister itself,
> it did not reset the lease id from previous instance.
> The issuse may cause DNs to temporarily go dead, making it unsafe to
> restart NN especially in hadoop cluster which has large amount of DNs.
> HDFS-12914 reported the issue without any clues why it occurred and remain
> unsolved.
> To make it clear, look at code below. We take it from method
> offerService of class BPServiceActor. We eliminate some code to focus on
> current issue. fullBlockReportLeaseId is a local variable to hold lease id
> from NN. Exceptions will occur at blockReport call when NN restarting, which
> will be caught by catch block in while loop. Thus fullBlockReportLeaseId will
> not be set to 0. After NN restarted, DN will send full block report which
> will be rejected by the new NN instance. DN will never send full block report
> until the next full block report schedule, about an hour later.
> Solution is simple, just reset fullBlockReportLeaseId to 0 after any
> exception or after registering to NN. Thus it will ask for a valid
> fullBlockReportLeaseId from new NN instance.
> {code:java}
> private void offerService() throws Exception {
> long fullBlockReportLeaseId = 0;
> //
> // Now loop for a long time
> //
> while (shouldRun()) {
> try {
> final long startTime = scheduler.monotonicNow();
> //
> // Every so often, send heartbeat or block-report
> //
> final boolean sendHeartbeat = scheduler.isHeartbeatDue(startTime);
> HeartbeatResponse resp = null;
> if (sendHeartbeat) {
>
> boolean requestBlockReportLease = (fullBlockReportLeaseId == 0) &&
> scheduler.isBlockReportDue(startTime);
> scheduler.scheduleNextHeartbeat();
> if (!dn.areHeartbeatsDisabledForTests()) {
> resp = sendHeartBeat(requestBlockReportLease);
> assert resp != null;
> if (resp.getFullBlockReportLeaseId() != 0) {
> if (fullBlockReportLeaseId != 0) {
> LOG.warn(nnAddr + " sent back a full block report lease " +
> "ID of 0x" +
> Long.toHexString(resp.getFullBlockReportLeaseId()) +
> ", but we already have a lease ID of 0x" +
> Long.toHexString(fullBlockReportLeaseId) + ". " +
> "Overwriting old lease ID.");
> }
> fullBlockReportLeaseId = resp.getFullBlockReportLeaseId();
> }
>
> }
> }
>
>
> if ((fullBlockReportLeaseId != 0) || forceFullBr) {
> //Exception occurred here when NN restarting
> cmds = blockReport(fullBlockReportLeaseId);
> fullBlockReportLeaseId = 0;
> }
>
> } catch(RemoteException re) {
>
> } // while (shouldRun())
> } // offerService{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14315) Add more detailed log message when decreasing replication factor < max in snapshots
[
https://issues.apache.org/jira/browse/HDFS-14315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daisuke Kobayashi updated HDFS-14315:
-
Status: Patch Available (was: Open)
Submitting the second patch which addressed the points made by Masatake.
> Add more detailed log message when decreasing replication factor < max in
> snapshots
> ---
>
> Key: HDFS-14315
> URL: https://issues.apache.org/jira/browse/HDFS-14315
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.1.2
>Reporter: Daisuke Kobayashi
>Assignee: Daisuke Kobayashi
>Priority: Minor
> Attachments: HDFS-14315.000.patch, HDFS-14315.001.patch
>
>
> When changing replication factor for a given file, the following 3 types of
> logging appear in the namenode log, but more detailed message would be useful
> especially when the file is in snapshot(s).
> {noformat}
> Decreasing replication from X to Y for FILE
> Increasing replication from X to Y for FILE
> Replication remains unchanged at X for FILE
> {noformat}
> I have added additional log messages as well as further test scenarios to
> org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotReplication#testReplicationWithSnapshot.
> The test sequence is:
> 1) A file is created with replication factor 1 (there are 5 datanodes)
> 2) Take a snapshot and increase the factor by 1. Continue this loop up to 5.
> 3) Setrep back to 3, but the target replication is decided to 4, which is the
> maximum in snapshots.
> {noformat}
> 2019-02-25 17:17:26,800 [IPC Server handler 9 on default port 55726] INFO
> namenode.FSDirectory (FSDirAttrOp.java:unprotectedSetReplication(408)) -
> Decreasing replication from 5 to 4 for /TestSnapshot/sub1/file1. Requested
> value is 3, but 4 is the maximum in snapshots
> {noformat}
> 4) Setrep to 3 again, but it's unchanged as follows. Both 3) and 4) are
> expected.
> {noformat}
> 2019-02-25 17:17:26,804 [IPC Server handler 6 on default port 55726] INFO
> namenode.FSDirectory (FSDirAttrOp.java:unprotectedSetReplication(420)) -
> Replication remains unchanged at 4 for /TestSnapshot/sub1/file1 . Requested
> value is 3, but 4 is the maximum in snapshots.
> {noformat}
> 5) Make sure the number of replicas in datanodes remains 4.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14315) Add more detailed log message when decreasing replication factor < max in snapshots
[
https://issues.apache.org/jira/browse/HDFS-14315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Daisuke Kobayashi updated HDFS-14315:
-
Status: Open (was: Patch Available)
> Add more detailed log message when decreasing replication factor < max in
> snapshots
> ---
>
> Key: HDFS-14315
> URL: https://issues.apache.org/jira/browse/HDFS-14315
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.1.2
>Reporter: Daisuke Kobayashi
>Assignee: Daisuke Kobayashi
>Priority: Minor
> Attachments: HDFS-14315.000.patch, HDFS-14315.001.patch
>
>
> When changing replication factor for a given file, the following 3 types of
> logging appear in the namenode log, but more detailed message would be useful
> especially when the file is in snapshot(s).
> {noformat}
> Decreasing replication from X to Y for FILE
> Increasing replication from X to Y for FILE
> Replication remains unchanged at X for FILE
> {noformat}
> I have added additional log messages as well as further test scenarios to
> org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotReplication#testReplicationWithSnapshot.
> The test sequence is:
> 1) A file is created with replication factor 1 (there are 5 datanodes)
> 2) Take a snapshot and increase the factor by 1. Continue this loop up to 5.
> 3) Setrep back to 3, but the target replication is decided to 4, which is the
> maximum in snapshots.
> {noformat}
> 2019-02-25 17:17:26,800 [IPC Server handler 9 on default port 55726] INFO
> namenode.FSDirectory (FSDirAttrOp.java:unprotectedSetReplication(408)) -
> Decreasing replication from 5 to 4 for /TestSnapshot/sub1/file1. Requested
> value is 3, but 4 is the maximum in snapshots
> {noformat}
> 4) Setrep to 3 again, but it's unchanged as follows. Both 3) and 4) are
> expected.
> {noformat}
> 2019-02-25 17:17:26,804 [IPC Server handler 6 on default port 55726] INFO
> namenode.FSDirectory (FSDirAttrOp.java:unprotectedSetReplication(420)) -
> Replication remains unchanged at 4 for /TestSnapshot/sub1/file1 . Requested
> value is 3, but 4 is the maximum in snapshots.
> {noformat}
> 5) Make sure the number of replicas in datanodes remains 4.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14299) ViewFs: Correct error message for read only operations
[ https://issues.apache.org/jira/browse/HDFS-14299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Surendra Singh Lilhore updated HDFS-14299: -- Summary: ViewFs: Correct error message for read only operations (was: ViewFs: Error message when operation on mount point) > ViewFs: Correct error message for read only operations > -- > > Key: HDFS-14299 > URL: https://issues.apache.org/jira/browse/HDFS-14299 > Project: Hadoop HDFS > Issue Type: Bug > Components: federation >Affects Versions: 3.1.2 >Reporter: hu xiaodong >Assignee: hu xiaodong >Priority: Minor > Attachments: .PNG, HDFS-14299.001.patch, HDFS-14299.002.patch > > > when error occurred when operating mount point, the error message liks this: > !.PNG! > I think a separator should be included between "operation" and "Path" > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14318) dn cannot be recognized and must be restarted to recognize the Repaired disk
[ https://issues.apache.org/jira/browse/HDFS-14318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hunshenshi updated HDFS-14318: -- Description: dn detected that disk a has failed. After disk a is repaired, dn cannot be recognized and must be restarted to recognize I make a patch to dn for recognize the repaired disk without restart dn was:Dn detected that disk a has failed. After disk a is repaired, dn cannot be recognized and must be restarted to recognize > dn cannot be recognized and must be restarted to recognize the Repaired disk > > > Key: HDFS-14318 > URL: https://issues.apache.org/jira/browse/HDFS-14318 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: hunshenshi >Priority: Major > > dn detected that disk a has failed. After disk a is repaired, dn cannot be > recognized and must be restarted to recognize > > I make a patch to dn for recognize the repaired disk without restart dn -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14305) Serial number in BlockTokenSecretManager could overlap between different namenodes
[
https://issues.apache.org/jira/browse/HDFS-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777652#comment-16777652
]
He Xiaoqiao commented on HDFS-14305:
Hi [~csun],[~xkrogen],[~jojochuang], update [^HDFS-14305.004.patch] following
review comments. Please give another review if you have some time. Thanks.
> Serial number in BlockTokenSecretManager could overlap between different
> namenodes
> --
>
> Key: HDFS-14305
> URL: https://issues.apache.org/jira/browse/HDFS-14305
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Reporter: Chao Sun
>Assignee: He Xiaoqiao
>Priority: Major
> Attachments: HDFS-14305.001.patch, HDFS-14305.002.patch,
> HDFS-14305.003.patch, HDFS-14305.004.patch
>
>
> Currently, a {{BlockTokenSecretManager}} starts with a random integer as the
> initial serial number, and then use this formula to rotate it:
> {code:java}
> this.intRange = Integer.MAX_VALUE / numNNs;
> this.nnRangeStart = intRange * nnIndex;
> this.serialNo = (this.serialNo % intRange) + (nnRangeStart);
> {code}
> while {{numNNs}} is the total number of NameNodes in the cluster, and
> {{nnIndex}} is the index of the current NameNode specified in the
> configuration {{dfs.ha.namenodes.}}.
> However, with this approach, different NameNode could have overlapping ranges
> for serial number. For simplicity, let's assume {{Integer.MAX_VALUE}} is 100,
> and we have 2 NameNodes {{nn1}} and {{nn2}} in configuration. Then the ranges
> for these two are:
> {code}
> nn1 -> [-49, 49]
> nn2 -> [1, 99]
> {code}
> This is because the initial serial number could be any negative integer.
> Moreover, when the keys are updated, the serial number will again be updated
> with the formula:
> {code}
> this.serialNo = (this.serialNo % intRange) + (nnRangeStart);
> {code}
> which means the new serial number could be updated to a range that belongs to
> a different NameNode, thus increasing the chance of collision again.
> When the collision happens, DataNodes could overwrite an existing key which
> will cause clients to fail because of {{InvalidToken}} error.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14305) Serial number in BlockTokenSecretManager could overlap between different namenodes
[
https://issues.apache.org/jira/browse/HDFS-14305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
He Xiaoqiao updated HDFS-14305:
---
Attachment: HDFS-14305.004.patch
> Serial number in BlockTokenSecretManager could overlap between different
> namenodes
> --
>
> Key: HDFS-14305
> URL: https://issues.apache.org/jira/browse/HDFS-14305
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: security
>Reporter: Chao Sun
>Assignee: He Xiaoqiao
>Priority: Major
> Attachments: HDFS-14305.001.patch, HDFS-14305.002.patch,
> HDFS-14305.003.patch, HDFS-14305.004.patch
>
>
> Currently, a {{BlockTokenSecretManager}} starts with a random integer as the
> initial serial number, and then use this formula to rotate it:
> {code:java}
> this.intRange = Integer.MAX_VALUE / numNNs;
> this.nnRangeStart = intRange * nnIndex;
> this.serialNo = (this.serialNo % intRange) + (nnRangeStart);
> {code}
> while {{numNNs}} is the total number of NameNodes in the cluster, and
> {{nnIndex}} is the index of the current NameNode specified in the
> configuration {{dfs.ha.namenodes.}}.
> However, with this approach, different NameNode could have overlapping ranges
> for serial number. For simplicity, let's assume {{Integer.MAX_VALUE}} is 100,
> and we have 2 NameNodes {{nn1}} and {{nn2}} in configuration. Then the ranges
> for these two are:
> {code}
> nn1 -> [-49, 49]
> nn2 -> [1, 99]
> {code}
> This is because the initial serial number could be any negative integer.
> Moreover, when the keys are updated, the serial number will again be updated
> with the formula:
> {code}
> this.serialNo = (this.serialNo % intRange) + (nnRangeStart);
> {code}
> which means the new serial number could be updated to a range that belongs to
> a different NameNode, thus increasing the chance of collision again.
> When the collision happens, DataNodes could overwrite an existing key which
> will cause clients to fail because of {{InvalidToken}} error.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14318) dn cannot be recognized and must be restarted to recognize the Repaired disk
[ https://issues.apache.org/jira/browse/HDFS-14318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hunshenshi updated HDFS-14318: -- Summary: dn cannot be recognized and must be restarted to recognize the Repaired disk (was: dn cannot be recognized and must be restarted to recognize) > dn cannot be recognized and must be restarted to recognize the Repaired disk > > > Key: HDFS-14318 > URL: https://issues.apache.org/jira/browse/HDFS-14318 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: hunshenshi >Priority: Major > > Dn detected that disk a has failed. After disk a is repaired, dn cannot be > recognized and must be restarted to recognize -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14318) dn cannot be recognized and must be restarted to recognize
[ https://issues.apache.org/jira/browse/HDFS-14318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hunshenshi updated HDFS-14318: -- Summary: dn cannot be recognized and must be restarted to recognize (was: Dn detected that disk a has failed. After disk a is repaired, dn cannot be recognized and must be restarted to recognize) > dn cannot be recognized and must be restarted to recognize > -- > > Key: HDFS-14318 > URL: https://issues.apache.org/jira/browse/HDFS-14318 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: hunshenshi >Priority: Major > > Dn detected that disk a has failed. After disk a is repaired, dn cannot be > recognized and must be restarted to recognize -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14318) Dn detected that disk a has failed. After disk a is repaired, dn cannot be recognized and must be restarted to recognize
[ https://issues.apache.org/jira/browse/HDFS-14318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hunshenshi updated HDFS-14318: -- Description: Dn detected that disk a has failed. After disk a is repaired, dn cannot be recognized and must be restarted to recognize > Dn detected that disk a has failed. After disk a is repaired, dn cannot be > recognized and must be restarted to recognize > > > Key: HDFS-14318 > URL: https://issues.apache.org/jira/browse/HDFS-14318 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: hunshenshi >Priority: Major > > Dn detected that disk a has failed. After disk a is repaired, dn cannot be > recognized and must be restarted to recognize -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14318) Dn detected that disk a has failed. After disk a is repaired, dn cannot be recognized and must be restarted to recognize
[ https://issues.apache.org/jira/browse/HDFS-14318?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] hunshenshi updated HDFS-14318: -- Summary: Dn detected that disk a has failed. After disk a is repaired, dn cannot be recognized and must be restarted to recognize (was: When dn) > Dn detected that disk a has failed. After disk a is repaired, dn cannot be > recognized and must be restarted to recognize > > > Key: HDFS-14318 > URL: https://issues.apache.org/jira/browse/HDFS-14318 > Project: Hadoop HDFS > Issue Type: Improvement > Components: datanode >Reporter: hunshenshi >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14318) When dn
hunshenshi created HDFS-14318: - Summary: When dn Key: HDFS-14318 URL: https://issues.apache.org/jira/browse/HDFS-14318 Project: Hadoop HDFS Issue Type: Improvement Components: datanode Reporter: hunshenshi -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14314) fullBlockReportLeaseId should be reset after registering to NN
[
https://issues.apache.org/jira/browse/HDFS-14314?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777640#comment-16777640
]
He Xiaoqiao commented on HDFS-14314:
[~starphin], Thanks for your contribution,
some minor comment,
a. please follow community code style such as indented by 4 spaces, delete
extra blank line, and some requisite comment for new method, I thinks the
format problem is only in the unit test.
b.naming your patch following -..patch,
refer:
https://cwiki.apache.org/confluence/display/HADOOP/How+To+Contribute#HowToContribute-Namingyourpatch.
+1 after update.
> fullBlockReportLeaseId should be reset after registering to NN
> --
>
> Key: HDFS-14314
> URL: https://issues.apache.org/jira/browse/HDFS-14314
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.8.4
> Environment:
>
>
>Reporter: star
>Priority: Critical
> Fix For: 2.8.4
>
> Attachments: HDFS-14314.0.patch, HDFS-14314.2.patch, HDFS-14314.patch
>
>
> since HDFS-7923 ,to rate-limit DN block report, DN will ask for a full
> block lease id from active NN before sending full block to NN. Then DN will
> send full block report together with lease id. If the lease id is invalid, NN
> will reject the full block report and log "not in the pending set".
> In a case when DN is doing full block reporting while NN is restarted.
> It happens that DN will later send a full block report with lease id
> ,acquired from previous NN instance, which is invalid to the new NN instance.
> Though DN recognized the new NN instance by heartbeat and reregister itself,
> it did not reset the lease id from previous instance.
> The issuse may cause DNs to temporarily go dead, making it unsafe to
> restart NN especially in hadoop cluster which has large amount of DNs.
> HDFS-12914 reported the issue without any clues why it occurred and remain
> unsolved.
> To make it clear, look at code below. We take it from method
> offerService of class BPServiceActor. We eliminate some code to focus on
> current issue. fullBlockReportLeaseId is a local variable to hold lease id
> from NN. Exceptions will occur at blockReport call when NN restarting, which
> will be caught by catch block in while loop. Thus fullBlockReportLeaseId will
> not be set to 0. After NN restarted, DN will send full block report which
> will be rejected by the new NN instance. DN will never send full block report
> until the next full block report schedule, about an hour later.
> Solution is simple, just reset fullBlockReportLeaseId to 0 after any
> exception or after registering to NN. Thus it will ask for a valid
> fullBlockReportLeaseId from new NN instance.
> {code:java}
> private void offerService() throws Exception {
> long fullBlockReportLeaseId = 0;
> //
> // Now loop for a long time
> //
> while (shouldRun()) {
> try {
> final long startTime = scheduler.monotonicNow();
> //
> // Every so often, send heartbeat or block-report
> //
> final boolean sendHeartbeat = scheduler.isHeartbeatDue(startTime);
> HeartbeatResponse resp = null;
> if (sendHeartbeat) {
>
> boolean requestBlockReportLease = (fullBlockReportLeaseId == 0) &&
> scheduler.isBlockReportDue(startTime);
> scheduler.scheduleNextHeartbeat();
> if (!dn.areHeartbeatsDisabledForTests()) {
> resp = sendHeartBeat(requestBlockReportLease);
> assert resp != null;
> if (resp.getFullBlockReportLeaseId() != 0) {
> if (fullBlockReportLeaseId != 0) {
> LOG.warn(nnAddr + " sent back a full block report lease " +
> "ID of 0x" +
> Long.toHexString(resp.getFullBlockReportLeaseId()) +
> ", but we already have a lease ID of 0x" +
> Long.toHexString(fullBlockReportLeaseId) + ". " +
> "Overwriting old lease ID.");
> }
> fullBlockReportLeaseId = resp.getFullBlockReportLeaseId();
> }
>
> }
> }
>
>
> if ((fullBlockReportLeaseId != 0) || forceFullBr) {
> //Exception occurred here when NN restarting
> cmds = blockReport(fullBlockReportLeaseId);
> fullBlockReportLeaseId = 0;
> }
>
> } catch(RemoteException re) {
>
> } // while (shouldRun())
> } // offerService{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.
[jira] [Work logged] (HDDS-1178) Healthy pipeline Chill Mode Rule
[ https://issues.apache.org/jira/browse/HDDS-1178?focusedWorklogId=204101&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-204101 ] ASF GitHub Bot logged work on HDDS-1178: Author: ASF GitHub Bot Created on: 26/Feb/19 06:21 Start Date: 26/Feb/19 06:21 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #518: HDDS-1178. Healthy pipeline Chill Mode Rule. URL: https://github.com/apache/hadoop/pull/518#issuecomment-467311361 Thank You @anuengineer for the review. 1. Low 10% is, as this rule main purpose is once we are out of chill mode, we have atleast few pipelines for writes to succeed. (As other rules like container chill mode rule, pipeline rule with at least one datanode reported by the time these completed, we might have more pipelines, this rule is more like a conservative side.) Let me know if you want to change it to any other default value. 2. Thanks for catching it. done. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 204101) Time Spent: 40m (was: 0.5h) > Healthy pipeline Chill Mode Rule > > > Key: HDDS-1178 > URL: https://issues.apache.org/jira/browse/HDDS-1178 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 40m > Remaining Estimate: 0h > > This Jira is to implement one of the chill mode rule. > h2. Pipeline Rule with configurable percentage of open pipelines with all > datanodes reported: > h2. In this rule, we consider when at least 10% of the pipelines have all 3 > data node's reported (if it is a 3 node ratis ring), and one node reported if > it is a one node ratis ring. (The percentage is configurable via ozone > configuration parameter). > > This rule satisfies, once the SCM is restarted, and after exiting chill mode, > we have enough pipelines to give for clients for writes to succeed. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1061) DelegationToken: Add certificate serial id to Ozone Delegation Token Identifier
[
https://issues.apache.org/jira/browse/HDDS-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777627#comment-16777627
]
Hadoop QA commented on HDDS-1061:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 25m
0s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 19m
42s{color} | {color:red} root in trunk failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
55s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
7s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 11s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
11s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
57s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
25s{color} | {color:red} common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
19s{color} | {color:red} ozone-manager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 15m
44s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 15m 44s{color} |
{color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 15m 44s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
48s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
38s{color} | {color:red} common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
34s{color} | {color:red} ozone-manager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 30s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
31s{color} | {color:red} common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
32s{color} | {color:red} ozone-manager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 15s{color}
| {color:red} common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 37s{color}
| {color:red} common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 33s{color}
| {color:red} ozone-manager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
42s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}113m 57s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdds.security.x509.certificate.client.TestDefaultCertificateClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0
[jira] [Work logged] (HDDS-1178) Healthy pipeline Chill Mode Rule
[ https://issues.apache.org/jira/browse/HDDS-1178?focusedWorklogId=204102&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-204102 ] ASF GitHub Bot logged work on HDDS-1178: Author: ASF GitHub Bot Created on: 26/Feb/19 06:24 Start Date: 26/Feb/19 06:24 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on issue #518: HDDS-1178. Healthy pipeline Chill Mode Rule. URL: https://github.com/apache/hadoop/pull/518#issuecomment-467311361 Thank You @anuengineer for the review. 1. Low 10% is, as this rule main purpose is once we are out of chill mode, we have atleast few pipelines for writes to succeed. (As other rules like container chill mode rule, pipeline rule with at least one datanode reported by the time these completed, we might have more pipelines, this rule is more like a conservative side.) Let me know if you want to change it to any other default value or any other suggestion for the default value. 2. Thanks for catching it. done. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 204102) Time Spent: 50m (was: 40m) > Healthy pipeline Chill Mode Rule > > > Key: HDDS-1178 > URL: https://issues.apache.org/jira/browse/HDDS-1178 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 50m > Remaining Estimate: 0h > > This Jira is to implement one of the chill mode rule. > h2. Pipeline Rule with configurable percentage of open pipelines with all > datanodes reported: > h2. In this rule, we consider when at least 10% of the pipelines have all 3 > data node's reported (if it is a 3 node ratis ring), and one node reported if > it is a one node ratis ring. (The percentage is configurable via ozone > configuration parameter). > > This rule satisfies, once the SCM is restarted, and after exiting chill mode, > we have enough pipelines to give for clients for writes to succeed. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14201) Ability to disallow safemode NN to become active
[
https://issues.apache.org/jira/browse/HDFS-14201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777624#comment-16777624
]
He Xiaoqiao commented on HDFS-14201:
Thanks [~surmountian] to push forward this issue.
{quote} I think combining the logic in HDFS-14201.002.patch and
HDFS-14201.003.patch could be an option.
The same configuration item would be controlling these logic to be on/off.
{quote}
+1, it makes sense to me, Thanks again.
> Ability to disallow safemode NN to become active
>
>
> Key: HDFS-14201
> URL: https://issues.apache.org/jira/browse/HDFS-14201
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: auto-failover
>Affects Versions: 3.1.1, 2.9.2
>Reporter: Xiao Liang
>Assignee: Xiao Liang
>Priority: Major
> Attachments: HDFS-14201.001.patch, HDFS-14201.002.patch,
> HDFS-14201.003.patch
>
>
> Currently with HA, Namenode in safemode can be possibly selected as active,
> for availability of both read and write, Namenodes not in safemode are better
> choices to become active though.
> It can take tens of minutes for a cold started Namenode to get out of
> safemode, especially when there are large number of files and blocks in HDFS,
> that means if a Namenode in safemode become active, the cluster will be not
> fully functioning for quite a while, even if it can while there is some
> Namenode not in safemode.
> The proposal here is to add an option, to allow Namenode to report itself as
> UNHEALTHY to ZKFC, if it's in safemode, so as to only allow fully functioning
> Namenode to become active, improving the general availability of the cluster.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14316) RBF: Support unavailable subclusters for mount points with multiple destinations
[
https://issues.apache.org/jira/browse/HDFS-14316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777607#comment-16777607
]
Hadoop QA commented on HDFS-14316:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
38s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 6 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-13891 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
51s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
22s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
38s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 20s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
56s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
38s{color} | {color:green} HDFS-13891 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} cc {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 17s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs-rbf: The patch
generated 5 new + 6 unchanged - 0 fixed = 11 total (was 6) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 47s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
9s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
36s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 25m 28s{color}
| {color:red} hadoop-hdfs-rbf in the patch failed. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m
30s{color} | {color:red} The patch generated 258 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 81m 28s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdfs.server.federation.router.TestRouterAllResolver |
| | hadoop.hdfs.server.federation.router.TestRouterAdminCLI |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14316 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12960125/HDFS-14316-HDFS-13891.000.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle cc |
| uname | Linux 399af27de852 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-13891 / d8dccda |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/26325/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs-rbf.txt
|
| unit |
htt
[jira] [Updated] (HDFS-14317) Standby does not trigger edit log rolling when in-progress edit log tailing is enabled
[
https://issues.apache.org/jira/browse/HDFS-14317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ekanth Sethuramalingam updated HDFS-14317:
--
Description:
The standby uses the following method to check if it is time to trigger edit
log rolling on active.
{code}
/**
* @return true if the configured log roll period has elapsed.
*/
private boolean tooLongSinceLastLoad() {
return logRollPeriodMs >= 0 &&
(monotonicNow() - lastLoadTimeMs) > logRollPeriodMs ;
}
{code}
In doTailEdits(), lastLoadTimeMs is updated when standby is able to
successfully tail any edits
{code}
if (editsLoaded > 0) {
lastLoadTimeMs = monotonicNow();
}
{code}
The default configuration for {{dfs.ha.log-roll.period}} is 120 seconds and
{{dfs.ha.tail-edits.period}} is 60 seconds. With in-progress edit log tailing
enabled, tooLongSinceLastLoad() will almost never return true resulting in edit
logs not rolled for a long time until this configuration
{{dfs.namenode.edit.log.autoroll.multiplier.threshold}} takes effect.
[In our deployment, this resulted in in-progress edit logs getting deleted. The
sequence of events is that standby was able to checkpoint twice while the
in-progress edit log was growing on active. When the NNStorageRetentionManager
decided to cleanup old checkpoints and edit logs, it cleaned up the in-progress
edit log from active and QJM (as the txnid on in-progress edit log was older
than the 2 most recent checkpoints) resulting in irrecoverably losing a few
minutes worth of metadata].
was:
The standby uses the following method to check if it is time to trigger edit
log rolling on active.
{{/**}}
\{{ * @return true if the configured log roll period has elapsed.}}
\{{ */}}
{{private boolean tooLongSinceLastLoad() {}}
\{{ return logRollPeriodMs >= 0 && }}
{{ (monotonicNow() - lastLoadTimeMs) > logRollPeriodMs ;}}
{{}}}
In doTailEdits(), lastLoadTimeMs is updated when standby is able to
successfully tail any edits
{{if (editsLoaded > 0) {}}
{{ lastLoadTimeMs = monotonicNow();}}
{{}}}
The default configuration for {{dfs.ha.log-roll.period}} is 120 seconds and
{{dfs.ha.tail-edits.period}} is 60 seconds. With in-progress edit log tailing
enabled, tooLongSinceLastLoad() will almost never return true resulting in edit
logs not rolled for a long time until this configuration
{{dfs.namenode.edit.log.autoroll.multiplier.threshold}} takes effect.
[In our deployment, this resulted in in-progress edit logs getting deleted. The
sequence of events is that standby was able to checkpoint twice while the
in-progress edit log was growing on active. When the NNStorageRetentionManager
decided to cleanup old checkpoints and edit logs, it cleaned up the in-progress
edit log from active and QJM (as the txnid on in-progress edit log was older
than the 2 most recent checkpoints) resulting in irrecoverably losing a few
minutes worth of metadata].
> Standby does not trigger edit log rolling when in-progress edit log tailing
> is enabled
> --
>
> Key: HDFS-14317
> URL: https://issues.apache.org/jira/browse/HDFS-14317
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 2.9.0, 3.0.0
>Reporter: Ekanth Sethuramalingam
>Assignee: Ekanth Sethuramalingam
>Priority: Critical
>
> The standby uses the following method to check if it is time to trigger edit
> log rolling on active.
> {code}
> /**
>* @return true if the configured log roll period has elapsed.
>*/
> private boolean tooLongSinceLastLoad() {
> return logRollPeriodMs >= 0 &&
> (monotonicNow() - lastLoadTimeMs) > logRollPeriodMs ;
> }
> {code}
> In doTailEdits(), lastLoadTimeMs is updated when standby is able to
> successfully tail any edits
> {code}
> if (editsLoaded > 0) {
> lastLoadTimeMs = monotonicNow();
> }
> {code}
> The default configuration for {{dfs.ha.log-roll.period}} is 120 seconds and
> {{dfs.ha.tail-edits.period}} is 60 seconds. With in-progress edit log tailing
> enabled, tooLongSinceLastLoad() will almost never return true resulting in
> edit logs not rolled for a long time until this configuration
> {{dfs.namenode.edit.log.autoroll.multiplier.threshold}} takes effect.
> [In our deployment, this resulted in in-progress edit logs getting deleted.
> The sequence of events is that standby was able to checkpoint twice while the
> in-progress edit log was growing on active. When the
> NNStorageRetentionManager decided to cleanup old checkpoints and edit logs,
> it cleaned up the in-progress edit log from active and QJM (as the txnid on
> in-progress edit log was older than the 2 most recent checkpoints) resulting
> in irrecoverably losing a few minutes worth of metadata].
--
This messag
[jira] [Created] (HDFS-14317) Standby does not trigger edit log rolling when in-progress edit log tailing is enabled
Ekanth Sethuramalingam created HDFS-14317:
-
Summary: Standby does not trigger edit log rolling when
in-progress edit log tailing is enabled
Key: HDFS-14317
URL: https://issues.apache.org/jira/browse/HDFS-14317
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 3.0.0, 2.9.0
Reporter: Ekanth Sethuramalingam
Assignee: Ekanth Sethuramalingam
The standby uses the following method to check if it is time to trigger edit
log rolling on active.
{{/**}}
\{{ * @return true if the configured log roll period has elapsed.}}
\{{ */}}
{{private boolean tooLongSinceLastLoad() {}}
\{{ return logRollPeriodMs >= 0 && }}
{{ (monotonicNow() - lastLoadTimeMs) > logRollPeriodMs ;}}
{{}}}
In doTailEdits(), lastLoadTimeMs is updated when standby is able to
successfully tail any edits
{{if (editsLoaded > 0) {}}
{{ lastLoadTimeMs = monotonicNow();}}
{{}}}
The default configuration for {{dfs.ha.log-roll.period}} is 120 seconds and
{{dfs.ha.tail-edits.period}} is 60 seconds. With in-progress edit log tailing
enabled, tooLongSinceLastLoad() will almost never return true resulting in edit
logs not rolled for a long time until this configuration
{{dfs.namenode.edit.log.autoroll.multiplier.threshold}} takes effect.
[In our deployment, this resulted in in-progress edit logs getting deleted. The
sequence of events is that standby was able to checkpoint twice while the
in-progress edit log was growing on active. When the NNStorageRetentionManager
decided to cleanup old checkpoints and edit logs, it cleaned up the in-progress
edit log from active and QJM (as the txnid on in-progress edit log was older
than the 2 most recent checkpoints) resulting in irrecoverably losing a few
minutes worth of metadata].
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-3246) pRead equivalent for direct read path
[ https://issues.apache.org/jira/browse/HDFS-3246?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777582#comment-16777582 ] Anoop Sam John commented on HDFS-3246: -- bq.int read(long position, ByteBuffer buf) When calling this API with buf remaining size of n and the file is having data size > n after given position, is it guaranteed to read the whole n bytes into BB in one go? Just wanted to confirm. Thanks. > pRead equivalent for direct read path > - > > Key: HDFS-3246 > URL: https://issues.apache.org/jira/browse/HDFS-3246 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client, performance >Affects Versions: 3.0.0-alpha1 >Reporter: Henry Robinson >Assignee: Sahil Takiar >Priority: Major > Attachments: HDFS-3246.001.patch, HDFS-3246.002.patch, > HDFS-3246.003.patch, HDFS-3246.004.patch > > > There is no pread equivalent in ByteBufferReadable. We should consider adding > one. It would be relatively easy to implement for the distributed case > (certainly compared to HDFS-2834), since DFSInputStream does most of the > heavy lifting. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14176) Replace incorrect use of system property user.name
[
https://issues.apache.org/jira/browse/HDFS-14176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777580#comment-16777580
]
Wei-Chiu Chuang commented on HDFS-14176:
+1
> Replace incorrect use of system property user.name
> --
>
> Key: HDFS-14176
> URL: https://issues.apache.org/jira/browse/HDFS-14176
> Project: Hadoop HDFS
> Issue Type: Bug
>Affects Versions: 3.3.0
> Environment: Kerberized
>Reporter: Wei-Chiu Chuang
>Assignee: Dinesh Chitlangia
>Priority: Major
> Attachments: HDFS-14176.01.patch, HDFS-14176.02.patch,
> HDFS-14176.03.patch, HDFS-14176.04.patch
>
>
> Looking at the Hadoop source code, there are a few places where the code
> assumes the user name can be acquired from Java's system property
> {{user.name}}.
> For example,
> {code:java|title=FileSystem}
> /** Return the current user's home directory in this FileSystem.
>* The default implementation returns {@code "/user/$USER/"}.
>*/
> public Path getHomeDirectory() {
> return this.makeQualified(
> new Path(USER_HOME_PREFIX + "/" + System.getProperty("user.name")));
> }
> {code}
> This is incorrect, as in a Kerberized environment, a user may login as a user
> principal different from its system login account.
> It would be better to use
> {{UserGroupInformation.getCurrentUser().getShortUserName()}}, similar to
> HDFS-12485.
> Unfortunately, I am seeing this improper use in Yarn, HDFS federation
> SFTPFilesystem and Ozone code (tests are ignored)
> The impact should be small, since it only affects the case where system is
> Kerberized and that the user principal is different from system login account.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14315) Add more detailed log message when decreasing replication factor < max in snapshots
[
https://issues.apache.org/jira/browse/HDFS-14315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777565#comment-16777565
]
Daisuke Kobayashi commented on HDFS-14315:
--
Thanks for reviewing the patch [~iwasakims]. All the comments make sense. Will
post the second patch, later.
> Add more detailed log message when decreasing replication factor < max in
> snapshots
> ---
>
> Key: HDFS-14315
> URL: https://issues.apache.org/jira/browse/HDFS-14315
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.1.2
>Reporter: Daisuke Kobayashi
>Assignee: Daisuke Kobayashi
>Priority: Minor
> Attachments: HDFS-14315.000.patch
>
>
> When changing replication factor for a given file, the following 3 types of
> logging appear in the namenode log, but more detailed message would be useful
> especially when the file is in snapshot(s).
> {noformat}
> Decreasing replication from X to Y for FILE
> Increasing replication from X to Y for FILE
> Replication remains unchanged at X for FILE
> {noformat}
> I have added additional log messages as well as further test scenarios to
> org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotReplication#testReplicationWithSnapshot.
> The test sequence is:
> 1) A file is created with replication factor 1 (there are 5 datanodes)
> 2) Take a snapshot and increase the factor by 1. Continue this loop up to 5.
> 3) Setrep back to 3, but the target replication is decided to 4, which is the
> maximum in snapshots.
> {noformat}
> 2019-02-25 17:17:26,800 [IPC Server handler 9 on default port 55726] INFO
> namenode.FSDirectory (FSDirAttrOp.java:unprotectedSetReplication(408)) -
> Decreasing replication from 5 to 4 for /TestSnapshot/sub1/file1. Requested
> value is 3, but 4 is the maximum in snapshots
> {noformat}
> 4) Setrep to 3 again, but it's unchanged as follows. Both 3) and 4) are
> expected.
> {noformat}
> 2019-02-25 17:17:26,804 [IPC Server handler 6 on default port 55726] INFO
> namenode.FSDirectory (FSDirAttrOp.java:unprotectedSetReplication(420)) -
> Replication remains unchanged at 4 for /TestSnapshot/sub1/file1 . Requested
> value is 3, but 4 is the maximum in snapshots.
> {noformat}
> 5) Make sure the number of replicas in datanodes remains 4.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1179) Ozone dist build failed on Jenkins
[
https://issues.apache.org/jira/browse/HDDS-1179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777564#comment-16777564
]
Hadoop QA commented on HDDS-1179:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
20s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
35m 40s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
18s{color} | {color:red} dist in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
18s{color} | {color:red} dist in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 18s{color}
| {color:red} dist in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
19s{color} | {color:red} dist in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 0s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 20s{color}
| {color:red} dist in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 26s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDDS-1179 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12960122/HDDS-1179.002.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient xml |
| uname | Linux 054d99133e39 4.4.0-138-generic #164~14.04.1-Ubuntu SMP Fri Oct
5 08:56:16 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / c6ea28c |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HDDS-Build/2377/artifact/out/patch-mvninstall-hadoop-ozone_dist.txt
|
| compile |
https://builds.apache.org/job/PreCommit-HDDS-Build/2377/artifact/out/patch-compile-hadoop-ozone_dist.txt
|
| javac |
https://builds.apache.org/job/PreCommit-HDDS-Build/2377/artifact/out/patch-compile-hadoop-ozone_dist.txt
|
| mvnsite |
https://builds.apache.org/job/PreCommit-HDDS-Build/2377/artifact/out/patch-mvnsite-hadoop-ozone_dist.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDDS-Build/2377/artifact/out/patch-unit-hadoop-ozone_dist.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDDS-Build/2377/testReport/ |
| Max. process+thread count | 309 (vs. ulimit of 1) |
| modules | C: hadoop-ozone/dist U: hadoop-ozone/dist |
| Console output |
https://builds.apache.org/job/PreCommit-HDDS-Build/2377/console |
| Powered by | A
[jira] [Commented] (HDFS-14315) Add more detailed log message when decreasing replication factor < max in snapshots
[
https://issues.apache.org/jira/browse/HDFS-14315?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777553#comment-16777553
]
Masatake Iwasaki commented on HDFS-14315:
-
[~daisuke.kobayashi] thanks for working on this.
{code:java}
import org.apache.hadoop.fs.*;
import org.apache.hadoop.hdfs.*;
{code}
We are usually avoiding to use wildcard import.
{code:java}
//Wait for block replication
try { Thread.sleep(1); } catch (InterruptedException ie) {}
{code}
Just sleeping makes test flaky or unnecessary long.
DFSTestUtil#waitForReplication or similar should be used if you can.
{code:java}
//File is in snapshots. Decreasing replication factor, but not the
requested value
if (oldBR > targetReplication && targetReplication > replication &&
iip.getLatestSnapshotId() != Snapshot.CURRENT_STATE_ID) {
FSDirectory.LOG.info("Decreasing replication from {} to {} for {}" +
". Requested value is {}, but {} is the maximum in
snapshots",
oldBR, targetReplication, iip.getPath(),
replication, targetReplication);
{code}
This looks error prone for future change of
INodeFile#getPreferredBlockReplication. FSDirAttrOp should not be aware of
implementation details of snapshot as far as possible. Just adding requested
replication facter to existing log would work for you?
> Add more detailed log message when decreasing replication factor < max in
> snapshots
> ---
>
> Key: HDFS-14315
> URL: https://issues.apache.org/jira/browse/HDFS-14315
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: hdfs
>Affects Versions: 3.1.2
>Reporter: Daisuke Kobayashi
>Assignee: Daisuke Kobayashi
>Priority: Minor
> Attachments: HDFS-14315.000.patch
>
>
> When changing replication factor for a given file, the following 3 types of
> logging appear in the namenode log, but more detailed message would be useful
> especially when the file is in snapshot(s).
> {noformat}
> Decreasing replication from X to Y for FILE
> Increasing replication from X to Y for FILE
> Replication remains unchanged at X for FILE
> {noformat}
> I have added additional log messages as well as further test scenarios to
> org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshotReplication#testReplicationWithSnapshot.
> The test sequence is:
> 1) A file is created with replication factor 1 (there are 5 datanodes)
> 2) Take a snapshot and increase the factor by 1. Continue this loop up to 5.
> 3) Setrep back to 3, but the target replication is decided to 4, which is the
> maximum in snapshots.
> {noformat}
> 2019-02-25 17:17:26,800 [IPC Server handler 9 on default port 55726] INFO
> namenode.FSDirectory (FSDirAttrOp.java:unprotectedSetReplication(408)) -
> Decreasing replication from 5 to 4 for /TestSnapshot/sub1/file1. Requested
> value is 3, but 4 is the maximum in snapshots
> {noformat}
> 4) Setrep to 3 again, but it's unchanged as follows. Both 3) and 4) are
> expected.
> {noformat}
> 2019-02-25 17:17:26,804 [IPC Server handler 6 on default port 55726] INFO
> namenode.FSDirectory (FSDirAttrOp.java:unprotectedSetReplication(420)) -
> Replication remains unchanged at 4 for /TestSnapshot/sub1/file1 . Requested
> value is 3, but 4 is the maximum in snapshots.
> {noformat}
> 5) Make sure the number of replicas in datanodes remains 4.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14316) RBF: Support unavailable subclusters for mount points with multiple destinations
[ https://issues.apache.org/jira/browse/HDFS-14316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-14316: --- Status: Patch Available (was: Open) > RBF: Support unavailable subclusters for mount points with multiple > destinations > > > Key: HDFS-14316 > URL: https://issues.apache.org/jira/browse/HDFS-14316 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-14316-HDFS-13891.000.patch > > > Currently mount points with multiple destinations (e.g., HASH_ALL) fail > writes when the destination subcluster is down. We need an option to allow > writing in other subclusters when one is down. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14316) RBF: Support unavailable subclusters for mount points with multiple destinations
[
https://issues.apache.org/jira/browse/HDFS-14316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777544#comment-16777544
]
Íñigo Goiri commented on HDFS-14316:
I added [^HDFS-14316-HDFS-13891.000.patch] with an initial proposal.
The core of the functionality is in {{RouterClientProtocol}}.
The rest are tests and wiring to expose it.
> RBF: Support unavailable subclusters for mount points with multiple
> destinations
>
>
> Key: HDFS-14316
> URL: https://issues.apache.org/jira/browse/HDFS-14316
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: Íñigo Goiri
>Priority: Major
> Attachments: HDFS-14316-HDFS-13891.000.patch
>
>
> Currently mount points with multiple destinations (e.g., HASH_ALL) fail
> writes when the destination subcluster is down. We need an option to allow
> writing in other subclusters when one is down.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14316) RBF: Support unavailable subclusters for mount points with multiple destinations
[ https://issues.apache.org/jira/browse/HDFS-14316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Íñigo Goiri updated HDFS-14316: --- Attachment: HDFS-14316-HDFS-13891.000.patch > RBF: Support unavailable subclusters for mount points with multiple > destinations > > > Key: HDFS-14316 > URL: https://issues.apache.org/jira/browse/HDFS-14316 > Project: Hadoop HDFS > Issue Type: Sub-task >Reporter: Íñigo Goiri >Assignee: Íñigo Goiri >Priority: Major > Attachments: HDFS-14316-HDFS-13891.000.patch > > > Currently mount points with multiple destinations (e.g., HASH_ALL) fail > writes when the destination subcluster is down. We need an option to allow > writing in other subclusters when one is down. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1179) Ozone dist build failed on Jenkins
[
https://issues.apache.org/jira/browse/HDDS-1179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HDDS-1179:
-
Attachment: HDDS-1179.002.patch
> Ozone dist build failed on Jenkins
> --
>
> Key: HDDS-1179
> URL: https://issues.apache.org/jira/browse/HDDS-1179
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
>Priority: Major
> Attachments: HDDS-1179.001.patch, HDDS-1179.002.patch
>
>
> This is part of the Jenkins execution and was reported in several latest HDDS
> Jenkins run.
> I spend some today and found a simplified repro steps:
> {code}
> cd hadoop-ozone/dist
> mvn -Phdds -DskipTests -fae clean install -DskipTests=true
> -Dmaven.javadoc.skip=true -Dcheckstyle.skip=true -Dfindbugs.skip=true
> {code}
>
> The root cause is that the
> hadoop-ozone/dist/dev-support/bin/dist-layout-stitching need
> objectstore-sevice jar being build earlier but he dependency was not
> explicitly declared in pom. I will attach a fix shortly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1178) Healthy pipeline Chill Mode Rule
[ https://issues.apache.org/jira/browse/HDDS-1178?focusedWorklogId=204044&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-204044 ] ASF GitHub Bot logged work on HDDS-1178: Author: ASF GitHub Bot Created on: 26/Feb/19 03:19 Start Date: 26/Feb/19 03:19 Worklog Time Spent: 10m Work Description: anuengineer commented on issue #518: HDDS-1178. Healthy pipeline Chill Mode Rule. URL: https://github.com/apache/hadoop/pull/518#issuecomment-467277972 A couple of comments: 1. Why is it 10% isn't that too low? 2. I see that for each pipeline report arrival, we check the pipeline manager for the state -- to check if the pipeline is in healthy. Isn't there a race condition here? How do we guarantee that this check and the pipeline report update does not race each other? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 204044) Time Spent: 0.5h (was: 20m) > Healthy pipeline Chill Mode Rule > > > Key: HDDS-1178 > URL: https://issues.apache.org/jira/browse/HDDS-1178 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 0.5h > Remaining Estimate: 0h > > This Jira is to implement one of the chill mode rule. > h2. Pipeline Rule with configurable percentage of open pipelines with all > datanodes reported: > h2. In this rule, we consider when at least 10% of the pipelines have all 3 > data node's reported (if it is a 3 node ratis ring), and one node reported if > it is a one node ratis ring. (The percentage is configurable via ozone > configuration parameter). > > This rule satisfies, once the SCM is restarted, and after exiting chill mode, > we have enough pipelines to give for clients for writes to succeed. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1177) Add validation to AuthorizationHeaderV4
[
https://issues.apache.org/jira/browse/HDDS-1177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777535#comment-16777535
]
Hadoop QA commented on HDDS-1177:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
16s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 55s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
25s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 3s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
42s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
34s{color} | {color:green} s3gateway in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 15s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDDS-1177 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12960113/HDDS-1177.01.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 9ce75a3438a6 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / c6ea28c |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDDS-Build/2376/testReport/ |
| Max. process+thread count | 447 (vs. ulimit of 1) |
| modules | C: hadoop-ozone/s3gateway U: hadoop-ozone/s3gateway |
| Console output |
https://builds.apache.org/job/PreCommit-HDDS-Build/2376/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automatically generated.
> Add validation to AuthorizationHeaderV4
> ---
>
> Key: HDDS-1177
> URL: https://issues.apache.org/jira/browse/HDDS-1177
>
[jira] [Commented] (HDDS-699) Detect Ozone Network topology
[ https://issues.apache.org/jira/browse/HDDS-699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777522#comment-16777522 ] Tsz Wo Nicholas Sze commented on HDDS-699: -- [~Sammi] thanks for the update. Thanks for adding many tests. Using AtomicInteger and AtomicReference may not work since individual fields may be mutated during a computation of a method. For example, when calling sortByDistanceCost, all nodes are in the topology initially. Then, some of the nodes may be removed during the computation of sortByDistanceCost. sortByDistanceCost may return incorrect results. I just have realized that the patches here are mostly adding new code but not yet change the existing code to use the new NetworkTopology. Then, the 01 patch actually is better. We may make NetworkTopology pluggable and improve it later. How about you address the previous comments from the 01 patch (using a single RW-lock)? > Detect Ozone Network topology > - > > Key: HDDS-699 > URL: https://issues.apache.org/jira/browse/HDDS-699 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Xiaoyu Yao >Assignee: Sammi Chen >Priority: Major > Attachments: HDDS-699.00.patch, HDDS-699.01.patch, HDDS-699.02.patch > > > Traditionally this has been implemented in Hadoop via script or customizable > java class. One thing we want to add here is the flexible multi-level support > instead of fixed levels like DC/Rack/NG/Node. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14272) [SBN read] ObserverReadProxyProvider should sync with active txnID on startup
[
https://issues.apache.org/jira/browse/HDFS-14272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777518#comment-16777518
]
Hadoop QA commented on HDFS-14272:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
45s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
31s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 21m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
22s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
14m 8s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
18s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
11s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m
39s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
12s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 48s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
13s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
44s{color} | {color:green} hadoop-hdfs-client in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red}105m 7s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}182m 7s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.qjournal.server.TestJournalNodeSync |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14272 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12960094/HDFS-14272.002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux e7f149c97846 3.13.0-153-generic #203-Ubuntu SMP Thu Jun 14
08:52:28 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / a6ab371 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Buil
[jira] [Commented] (HDDS-1072) Implement RetryProxy and FailoverProxy for OM client
[
https://issues.apache.org/jira/browse/HDDS-1072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777520#comment-16777520
]
Hadoop QA commented on HDDS-1072:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
16s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 24m
10s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 20m
4s{color} | {color:red} root in trunk failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
58s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
37s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
19m 52s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
45s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
26s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
46s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 17m
34s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 17m 34s{color} |
{color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 17m 34s{color}
| {color:red} root in the patch failed. {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
3m 51s{color} | {color:orange} root: The patch generated 4 new + 0 unchanged -
0 fixed = 4 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 3m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 43s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m
16s{color} | {color:red} hadoop-ozone/common generated 2 new + 0 unchanged - 0
fixed = 2 total (was 0) {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
50s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 23s{color}
| {color:red} common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
44s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
36s{color} | {color:green} client in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
11s{color} | {color:green} ozone-manager in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 14m 36s{color}
| {color:red} integration-test in the patch fai
[jira] [Resolved] (HDDS-615) ozone-dist should depend on hadoop-ozone-file-system
[
https://issues.apache.org/jira/browse/HDDS-615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham resolved HDDS-615.
-
Resolution: Done
This is fixed as part of HDDS-922
> ozone-dist should depend on hadoop-ozone-file-system
>
>
> Key: HDDS-615
> URL: https://issues.apache.org/jira/browse/HDDS-615
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Attachments: HDDS-615.001.patch
>
>
> In the Yetus build of HDDS-523 the build of the dist project was failed:
> {code:java}
> Mon Oct 8 14:16:06 UTC 2018
> cd /testptch/hadoop/hadoop-ozone/dist
> /usr/bin/mvn -Phdds
> -Dmaven.repo.local=/home/jenkins/yetus-m2/hadoop-trunk-patch-1 -Ptest-patch
> -DskipTests -fae clean install -DskipTests=true -Dmaven.javadoc.skip=true
> -Dcheckstyle.skip=true -Dfindbugs.skip=true
> [INFO] Scanning for projects...
> [INFO]
>
> [INFO]
>
> [INFO] Building Apache Hadoop Ozone Distribution 0.3.0-SNAPSHOT
> [INFO]
>
> [INFO]
> [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hadoop-ozone-dist
> ---
> [INFO] Deleting /testptch/hadoop/hadoop-ozone/dist (includes =
> [dependency-reduced-pom.xml], excludes = [])
> [INFO]
> [INFO] --- maven-antrun-plugin:1.7:run (create-testdirs) @ hadoop-ozone-dist
> ---
> [INFO] Executing tasks
> main:
> [mkdir] Created dir: /testptch/hadoop/hadoop-ozone/dist/target/test-dir
> [INFO] Executed tasks
> [INFO]
> [INFO] --- maven-remote-resources-plugin:1.5:process (default) @
> hadoop-ozone-dist ---
> [INFO]
> [INFO] --- exec-maven-plugin:1.3.1:exec (dist) @ hadoop-ozone-dist ---
> cp: cannot stat
> '/testptch/hadoop/hadoop-ozone/ozonefs/target/hadoop-ozone-filesystem-0.3.0-SNAPSHOT.jar':
> No such file or directory
> Current directory /testptch/hadoop/hadoop-ozone/dist/target
> $ rm -rf ozone-0.3.0-SNAPSHOT
> $ mkdir ozone-0.3.0-SNAPSHOT
> $ cd ozone-0.3.0-SNAPSHOT
> $ cp -p /testptch/hadoop/LICENSE.txt .
> $ cp -p /testptch/hadoop/NOTICE.txt .
> $ cp -p /testptch/hadoop/README.txt .
> $ mkdir -p ./share/hadoop/mapreduce
> $ mkdir -p ./share/hadoop/ozone
> $ mkdir -p ./share/hadoop/hdds
> $ mkdir -p ./share/hadoop/yarn
> $ mkdir -p ./share/hadoop/hdfs
> $ mkdir -p ./share/hadoop/common
> $ mkdir -p ./share/ozone/web
> $ mkdir -p ./bin
> $ mkdir -p ./sbin
> $ mkdir -p ./etc
> $ mkdir -p ./libexec
> $ cp -r /testptch/hadoop/hadoop-common-project/hadoop-common/src/main/conf
> etc/hadoop
> $ cp
> /testptch/hadoop/hadoop-ozone/common/src/main/conf/om-audit-log4j2.properties
> etc/hadoop
> $ cp /testptch/hadoop/hadoop-common-project/hadoop-common/src/main/bin/hadoop
> bin/
> $ cp
> /testptch/hadoop/hadoop-common-project/hadoop-common/src/main/bin/hadoop.cmd
> bin/
> $ cp /testptch/hadoop/hadoop-ozone/common/src/main/bin/ozone bin/
> $ cp
> /testptch/hadoop/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
> libexec/
> $ cp
> /testptch/hadoop/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.cmd
> libexec/
> $ cp
> /testptch/hadoop/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
> libexec/
> $ cp /testptch/hadoop/hadoop-ozone/common/src/main/bin/ozone-config.sh
> libexec/
> $ cp -r /testptch/hadoop/hadoop-ozone/common/src/main/shellprofile.d libexec/
> $ cp
> /testptch/hadoop/hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemons.sh
> sbin/
> $ cp
> /testptch/hadoop/hadoop-common-project/hadoop-common/src/main/bin/workers.sh
> sbin/
> $ cp /testptch/hadoop/hadoop-ozone/common/src/main/bin/start-ozone.sh sbin/
> $ cp /testptch/hadoop/hadoop-ozone/common/src/main/bin/stop-ozone.sh sbin/
> $ mkdir -p ./share/hadoop/ozonefs
> $ cp
> /testptch/hadoop/hadoop-ozone/ozonefs/target/hadoop-ozone-filesystem-0.3.0-SNAPSHOT.jar
> ./share/hadoop/ozonefs/hadoop-ozone-filesystem-0.3.0-SNAPSHOT.jar
> Failed!
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 7.832 s
> [INFO] Finished at: 2018-10-08T14:16:16+00:00
> [INFO] Final Memory: 33M/625M
> [INFO]
>
> [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.1:exec
> (dist) on project hadoop-ozone-dist: Command execution failed. Process exited
> with an error: 1 (Exit value: 1) -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR]
[jira] [Updated] (HDDS-615) ozone-dist should depend on hadoop-ozone-file-system
[
https://issues.apache.org/jira/browse/HDDS-615?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham updated HDDS-615:
Fix Version/s: 0.4.0
> ozone-dist should depend on hadoop-ozone-file-system
>
>
> Key: HDDS-615
> URL: https://issues.apache.org/jira/browse/HDDS-615
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Fix For: 0.4.0
>
> Attachments: HDDS-615.001.patch
>
>
> In the Yetus build of HDDS-523 the build of the dist project was failed:
> {code:java}
> Mon Oct 8 14:16:06 UTC 2018
> cd /testptch/hadoop/hadoop-ozone/dist
> /usr/bin/mvn -Phdds
> -Dmaven.repo.local=/home/jenkins/yetus-m2/hadoop-trunk-patch-1 -Ptest-patch
> -DskipTests -fae clean install -DskipTests=true -Dmaven.javadoc.skip=true
> -Dcheckstyle.skip=true -Dfindbugs.skip=true
> [INFO] Scanning for projects...
> [INFO]
>
> [INFO]
>
> [INFO] Building Apache Hadoop Ozone Distribution 0.3.0-SNAPSHOT
> [INFO]
>
> [INFO]
> [INFO] --- maven-clean-plugin:2.5:clean (default-clean) @ hadoop-ozone-dist
> ---
> [INFO] Deleting /testptch/hadoop/hadoop-ozone/dist (includes =
> [dependency-reduced-pom.xml], excludes = [])
> [INFO]
> [INFO] --- maven-antrun-plugin:1.7:run (create-testdirs) @ hadoop-ozone-dist
> ---
> [INFO] Executing tasks
> main:
> [mkdir] Created dir: /testptch/hadoop/hadoop-ozone/dist/target/test-dir
> [INFO] Executed tasks
> [INFO]
> [INFO] --- maven-remote-resources-plugin:1.5:process (default) @
> hadoop-ozone-dist ---
> [INFO]
> [INFO] --- exec-maven-plugin:1.3.1:exec (dist) @ hadoop-ozone-dist ---
> cp: cannot stat
> '/testptch/hadoop/hadoop-ozone/ozonefs/target/hadoop-ozone-filesystem-0.3.0-SNAPSHOT.jar':
> No such file or directory
> Current directory /testptch/hadoop/hadoop-ozone/dist/target
> $ rm -rf ozone-0.3.0-SNAPSHOT
> $ mkdir ozone-0.3.0-SNAPSHOT
> $ cd ozone-0.3.0-SNAPSHOT
> $ cp -p /testptch/hadoop/LICENSE.txt .
> $ cp -p /testptch/hadoop/NOTICE.txt .
> $ cp -p /testptch/hadoop/README.txt .
> $ mkdir -p ./share/hadoop/mapreduce
> $ mkdir -p ./share/hadoop/ozone
> $ mkdir -p ./share/hadoop/hdds
> $ mkdir -p ./share/hadoop/yarn
> $ mkdir -p ./share/hadoop/hdfs
> $ mkdir -p ./share/hadoop/common
> $ mkdir -p ./share/ozone/web
> $ mkdir -p ./bin
> $ mkdir -p ./sbin
> $ mkdir -p ./etc
> $ mkdir -p ./libexec
> $ cp -r /testptch/hadoop/hadoop-common-project/hadoop-common/src/main/conf
> etc/hadoop
> $ cp
> /testptch/hadoop/hadoop-ozone/common/src/main/conf/om-audit-log4j2.properties
> etc/hadoop
> $ cp /testptch/hadoop/hadoop-common-project/hadoop-common/src/main/bin/hadoop
> bin/
> $ cp
> /testptch/hadoop/hadoop-common-project/hadoop-common/src/main/bin/hadoop.cmd
> bin/
> $ cp /testptch/hadoop/hadoop-ozone/common/src/main/bin/ozone bin/
> $ cp
> /testptch/hadoop/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.sh
> libexec/
> $ cp
> /testptch/hadoop/hadoop-common-project/hadoop-common/src/main/bin/hadoop-config.cmd
> libexec/
> $ cp
> /testptch/hadoop/hadoop-common-project/hadoop-common/src/main/bin/hadoop-functions.sh
> libexec/
> $ cp /testptch/hadoop/hadoop-ozone/common/src/main/bin/ozone-config.sh
> libexec/
> $ cp -r /testptch/hadoop/hadoop-ozone/common/src/main/shellprofile.d libexec/
> $ cp
> /testptch/hadoop/hadoop-common-project/hadoop-common/src/main/bin/hadoop-daemons.sh
> sbin/
> $ cp
> /testptch/hadoop/hadoop-common-project/hadoop-common/src/main/bin/workers.sh
> sbin/
> $ cp /testptch/hadoop/hadoop-ozone/common/src/main/bin/start-ozone.sh sbin/
> $ cp /testptch/hadoop/hadoop-ozone/common/src/main/bin/stop-ozone.sh sbin/
> $ mkdir -p ./share/hadoop/ozonefs
> $ cp
> /testptch/hadoop/hadoop-ozone/ozonefs/target/hadoop-ozone-filesystem-0.3.0-SNAPSHOT.jar
> ./share/hadoop/ozonefs/hadoop-ozone-filesystem-0.3.0-SNAPSHOT.jar
> Failed!
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 7.832 s
> [INFO] Finished at: 2018-10-08T14:16:16+00:00
> [INFO] Final Memory: 33M/625M
> [INFO]
>
> [ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.1:exec
> (dist) on project hadoop-ozone-dist: Command execution failed. Process exited
> with an error: 1 (Exit value: 1) -> [Help 1]
> [ERROR]
> [ERROR] To see the full stack trace of the errors, re-run Maven with the -e
> switch.
> [ERROR] Re
[jira] [Updated] (HDFS-14052) RBF: Use Router keytab for WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-14052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Brahma Reddy Battula updated HDFS-14052:
Resolution: Fixed
Hadoop Flags: Reviewed
Fix Version/s: HDFS-13891
Status: Resolved (was: Patch Available)
Committed to HDFS-13891 branch. [~crh] thanks for contribution and thanks to
[~elgoiri] for review.
> RBF: Use Router keytab for WebHDFS
> --
>
> Key: HDFS-14052
> URL: https://issues.apache.org/jira/browse/HDFS-14052
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: CR Hota
>Priority: Major
> Fix For: HDFS-13891
>
> Attachments: HDFS-14052-HDFS-13891.0.patch,
> HDFS-14052-HDFS-13891.1.patch, HDFS-14052-HDFS-13891.2.patch,
> HDFS-14052-HDFS-13891.3.patch, HDFS-14052-HDFS-13891.4.patch
>
>
> When the RouterHttpServer starts it does:
> {code}
> NameNodeHttpServer.initWebHdfs(conf, httpAddress.getHostName(),
> httpServer,
> RouterWebHdfsMethods.class.getPackage().getName());
> {code}
> This function is in the NN and is pretty generic.
> However, it then calls to NameNodeHttpServer#getAuthFilterParams, which does:
> {code}
> String httpKeytab = conf.get(DFSUtil.getSpnegoKeytabKey(conf,
> DFSConfigKeys.DFS_NAMENODE_KEYTAB_FILE_KEY));
> {code}
> In most cases, the regular web keytab will kick in, but we should make this a
> parameter and load the Router one just in case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1178) Healthy pipeline Chill Mode Rule
[ https://issues.apache.org/jira/browse/HDDS-1178?focusedWorklogId=204022&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-204022 ] ASF GitHub Bot logged work on HDDS-1178: Author: ASF GitHub Bot Created on: 26/Feb/19 02:16 Start Date: 26/Feb/19 02:16 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #518: HDDS-1178. Healthy pipeline Chill Mode Rule. URL: https://github.com/apache/hadoop/pull/518#issuecomment-467265319 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 25 | Docker mode activated. | ||| _ Prechecks _ | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 24 | Maven dependency ordering for branch | | +1 | mvninstall | 1008 | trunk passed | | +1 | compile | 75 | trunk passed | | -0 | checkstyle | 26 | The patch fails to run checkstyle in hadoop-hdds | | -1 | mvnsite | 20 | server-scm in trunk failed. | | +1 | shadedclient | 743 | branch has no errors when building and testing our client artifacts. | | -1 | findbugs | 18 | server-scm in trunk failed. | | -1 | javadoc | 18 | server-scm in trunk failed. | ||| _ Patch Compile Tests _ | | 0 | mvndep | 12 | Maven dependency ordering for patch | | -1 | mvninstall | 12 | server-scm in the patch failed. | | +1 | compile | 68 | the patch passed | | +1 | javac | 68 | the patch passed | | -0 | checkstyle | 18 | The patch fails to run checkstyle in hadoop-hdds | | -1 | mvnsite | 13 | server-scm in the patch failed. | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 741 | patch has no errors when building and testing our client artifacts. | | -1 | findbugs | 16 | server-scm in the patch failed. | | -1 | javadoc | 16 | server-scm in the patch failed. | ||| _ Other Tests _ | | -1 | unit | 73 | common in the patch failed. | | -1 | unit | 16 | server-scm in the patch failed. | | +1 | asflicense | 29 | The patch does not generate ASF License warnings. | | | | 3366 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.hdds.security.x509.certificate.client.TestDefaultCertificateClient | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=17.05.0-ce Server=17.05.0-ce base: https://builds.apache.org/job/hadoop-multibranch/job/PR-518/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/518 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 970154b15eab 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / a6ab371 | | maven | version: Apache Maven 3.3.9 | | Default Java | 1.8.0_191 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-518/1/artifact/out//testptch/patchprocess/maven-branch-checkstyle-hadoop-hdds.txt | | mvnsite | https://builds.apache.org/job/hadoop-multibranch/job/PR-518/1/artifact/out/branch-mvnsite-hadoop-hdds_server-scm.txt | | findbugs | v3.1.0-RC1 | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-518/1/artifact/out/branch-findbugs-hadoop-hdds_server-scm.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-518/1/artifact/out/branch-javadoc-hadoop-hdds_server-scm.txt | | mvninstall | https://builds.apache.org/job/hadoop-multibranch/job/PR-518/1/artifact/out/patch-mvninstall-hadoop-hdds_server-scm.txt | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-518/1/artifact/out//testptch/patchprocess/maven-patch-checkstyle-hadoop-hdds.txt | | mvnsite | https://builds.apache.org/job/hadoop-multibranch/job/PR-518/1/artifact/out/patch-mvnsite-hadoop-hdds_server-scm.txt | | findbugs | https://builds.apache.org/job/hadoop-multibranch/job/PR-518/1/artifact/out/patch-findbugs-hadoop-hdds_server-scm.txt | | javadoc | https://builds.apache.org/job/hadoop-multibranch/job/PR-518/1/artifact/out/patch-javadoc-hadoop-hdds_server-scm.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-518/1/artifact/out/patch-unit-hadoop-hdds_common.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-518/1/artifact/out/patch-unit-hadoop-hdds_server-scm.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-518/1/testReport/ | | Max. process+thread count | 436 (vs. ulimit of 5500) | | modules | C:
[jira] [Updated] (HDDS-1177) Add validation to AuthorizationHeaderV4
[ https://issues.apache.org/jira/browse/HDDS-1177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDDS-1177: - Attachment: HDDS-1177.01.patch > Add validation to AuthorizationHeaderV4 > --- > > Key: HDDS-1177 > URL: https://issues.apache.org/jira/browse/HDDS-1177 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDDS-1177.00.patch, HDDS-1177.01.patch > > > Add validation to AuthorizationHeaderV4. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1179) Ozone dist build failed on Jenkins
[
https://issues.apache.org/jira/browse/HDDS-1179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777489#comment-16777489
]
Hadoop QA commented on HDDS-1179:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
17s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
27s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
35m 19s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
21s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
17s{color} | {color:red} dist in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 0m
18s{color} | {color:red} dist in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 0m 18s{color}
| {color:red} dist in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
20s{color} | {color:red} dist in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
1s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 42s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
14s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 19s{color}
| {color:red} dist in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
30s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 51m 44s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDDS-1179 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12960101/HDDS-1179.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient xml |
| uname | Linux 8c36b6b613c5 4.4.0-139-generic #165~14.04.1-Ubuntu SMP Wed Oct
31 10:55:11 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / a6ab371 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| mvninstall |
https://builds.apache.org/job/PreCommit-HDDS-Build/2375/artifact/out/patch-mvninstall-hadoop-ozone_dist.txt
|
| compile |
https://builds.apache.org/job/PreCommit-HDDS-Build/2375/artifact/out/patch-compile-hadoop-ozone_dist.txt
|
| javac |
https://builds.apache.org/job/PreCommit-HDDS-Build/2375/artifact/out/patch-compile-hadoop-ozone_dist.txt
|
| mvnsite |
https://builds.apache.org/job/PreCommit-HDDS-Build/2375/artifact/out/patch-mvnsite-hadoop-ozone_dist.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDDS-Build/2375/artifact/out/patch-unit-hadoop-ozone_dist.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDDS-Build/2375/testReport/ |
| Max. process+thread count | 307 (vs. ulimit of 1) |
| modules | C: hadoop-ozone/dist U: hadoop-ozone/dist |
| Console output |
https://builds.apache.org/job/PreCommit-HDDS-Build/2375/console |
| Powered by |
[jira] [Commented] (HDDS-594) SCM CA: DN sends CSR and uses certificate issued by SCM
[
https://issues.apache.org/jira/browse/HDDS-594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777483#comment-16777483
]
Hadoop QA commented on HDDS-594:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
30s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
18s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
57s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
20s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
34s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
16s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 51s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
4s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
9s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
8s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
59s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 36s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
15s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
56s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 1s{color}
| {color:red} common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 1s{color}
| {color:red} container-service in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
25s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 60m 23s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdds.security.x509.certificate.client.TestDefaultCertificateClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDDS-594 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12960099/HDDS-594.03.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 42e522dc73b6 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / a6ab371 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDDS-Build/2374/ar
[jira] [Commented] (HDFS-14052) RBF: Use Router keytab for WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-14052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777480#comment-16777480
]
Brahma Reddy Battula commented on HDFS-14052:
-
+1, Going to commit shortly.
> RBF: Use Router keytab for WebHDFS
> --
>
> Key: HDFS-14052
> URL: https://issues.apache.org/jira/browse/HDFS-14052
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: CR Hota
>Priority: Major
> Attachments: HDFS-14052-HDFS-13891.0.patch,
> HDFS-14052-HDFS-13891.1.patch, HDFS-14052-HDFS-13891.2.patch,
> HDFS-14052-HDFS-13891.3.patch, HDFS-14052-HDFS-13891.4.patch
>
>
> When the RouterHttpServer starts it does:
> {code}
> NameNodeHttpServer.initWebHdfs(conf, httpAddress.getHostName(),
> httpServer,
> RouterWebHdfsMethods.class.getPackage().getName());
> {code}
> This function is in the NN and is pretty generic.
> However, it then calls to NameNodeHttpServer#getAuthFilterParams, which does:
> {code}
> String httpKeytab = conf.get(DFSUtil.getSpnegoKeytabKey(conf,
> DFSConfigKeys.DFS_NAMENODE_KEYTAB_FILE_KEY));
> {code}
> In most cases, the regular web keytab will kick in, but we should make this a
> parameter and load the Router one just in case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1179) Ozone dist build failed on Jenkins
[
https://issues.apache.org/jira/browse/HDDS-1179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777470#comment-16777470
]
Bharat Viswanadham commented on HDDS-1179:
--
I think we can add scope as provided, as this is mainly done for mvn reactor
ordering.
> Ozone dist build failed on Jenkins
> --
>
> Key: HDDS-1179
> URL: https://issues.apache.org/jira/browse/HDDS-1179
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
>Priority: Major
> Attachments: HDDS-1179.001.patch
>
>
> This is part of the Jenkins execution and was reported in several latest HDDS
> Jenkins run.
> I spend some today and found a simplified repro steps:
> {code}
> cd hadoop-ozone/dist
> mvn -Phdds -DskipTests -fae clean install -DskipTests=true
> -Dmaven.javadoc.skip=true -Dcheckstyle.skip=true -Dfindbugs.skip=true
> {code}
>
> The root cause is that the
> hadoop-ozone/dist/dev-support/bin/dist-layout-stitching need
> objectstore-sevice jar being build earlier but he dependency was not
> explicitly declared in pom. I will attach a fix shortly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1061) DelegationToken: Add certificate serial id to Ozone Delegation Token Identifier
[
https://issues.apache.org/jira/browse/HDDS-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777472#comment-16777472
]
Hadoop QA commented on HDDS-1061:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
24s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
12s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
47s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 19m
28s{color} | {color:red} root in trunk failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 6s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
25s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
56s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
28s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
29s{color} | {color:red} common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
22s{color} | {color:red} ozone-manager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 17m
31s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 17m 31s{color} |
{color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 17m 31s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
45s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
39s{color} | {color:red} common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
32s{color} | {color:red} ozone-manager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 31s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
32s{color} | {color:red} common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
31s{color} | {color:red} ozone-manager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 2m
4s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 25s{color}
| {color:red} common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 37s{color}
| {color:red} common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 32s{color}
| {color:red} ozone-manager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}114m 56s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdds.security.x509.certificate.client.TestDefaultCertificateClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0
[jira] [Commented] (HDDS-1177) Add validation to AuthorizationHeaderV4
[
https://issues.apache.org/jira/browse/HDDS-1177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777463#comment-16777463
]
Hadoop QA commented on HDDS-1177:
-
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
22s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
28s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
19s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
35s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 30s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
22s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
22s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 10s{color} | {color:orange} hadoop-ozone/s3gateway: The patch generated 3
new + 0 unchanged - 0 fixed = 3 total (was 0) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
23s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 53s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
40s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
15s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
34s{color} | {color:green} s3gateway in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
26s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 49s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDDS-1177 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12960096/HDDS-1177.00.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux e843d2e031a7 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / a6ab371 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDDS-Build/2373/artifact/out/diff-checkstyle-hadoop-ozone_s3gateway.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDDS-Build/2373/testReport/ |
| Max. process+thread count | 412 (vs. ulimit of 1) |
| modules | C: hadoop-ozone/s3gateway U: hadoop-ozone/s3gateway |
| Console output |
https://builds.apache.org/job/PreCommit-HDDS-Build/2373/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was automaticall
[jira] [Commented] (HDDS-1038) Support Service Level Authorization for OM, SCM and DN
[
https://issues.apache.org/jira/browse/HDDS-1038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777459#comment-16777459
]
Hadoop QA commented on HDDS-1038:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 4 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
35s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
54s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 17m
48s{color} | {color:red} root in trunk failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
25s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
42s{color} | {color:red} dist in trunk failed. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 59s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test hadoop-ozone/dist {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
42s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 5m
12s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 1m
7s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
20s{color} | {color:red} dist in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 16m
31s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 16m 31s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
24s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
38s{color} | {color:red} dist in the patch failed. {color} |
| {color:green}+1{color} | {color:green} shellcheck {color} | {color:green} 0m
1s{color} | {color:green} There were no new shellcheck issues. {color} |
| {color:green}+1{color} | {color:green} shelldocs {color} | {color:green} 0m
33s{color} | {color:green} There were no new shelldocs issues. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
3s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 45s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test hadoop-ozone/dist {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 5m
44s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 4m
3s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
19s{color} | {color:green} common in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 22s{color}
| {color:red} container-service in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 1m
42s{color} | {color:green} server-scm in the patch passed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
53s{color} | {color:green} common in the patch passed. {color} |
| {color:green}+
[jira] [Updated] (HDDS-1179) Ozone dist build failed on Jenkins
[
https://issues.apache.org/jira/browse/HDDS-1179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HDDS-1179:
-
Status: Patch Available (was: Open)
> Ozone dist build failed on Jenkins
> --
>
> Key: HDDS-1179
> URL: https://issues.apache.org/jira/browse/HDDS-1179
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
>Priority: Major
> Attachments: HDDS-1179.001.patch
>
>
> This is part of the Jenkins execution and was reported in several latest HDDS
> Jenkins run.
> I spend some today and found a simplified repro steps:
> {code}
> cd hadoop-ozone/dist
> mvn -Phdds -DskipTests -fae clean install -DskipTests=true
> -Dmaven.javadoc.skip=true -Dcheckstyle.skip=true -Dfindbugs.skip=true
> {code}
>
> The root cause is that the
> hadoop-ozone/dist/dev-support/bin/dist-layout-stitching need
> objectstore-sevice jar being build earlier but he dependency was not
> explicitly declared in pom. I will attach a fix shortly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1179) Ozone dist build failed on Jenkins
[
https://issues.apache.org/jira/browse/HDDS-1179?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777458#comment-16777458
]
Xiaoyu Yao commented on HDDS-1179:
--
Related Error message from the repro steps.
{code}
[INFO] Apache Hadoop Ozone Distribution 0.4.0-SNAPSHOT FAILURE [ 1.305 s]
[INFO]
[INFO] BUILD FAILURE
[INFO]
[INFO] Total time: 07:26 min
[INFO] Finished at: 2019-02-25T12:01:32-08:00
[INFO]
[ERROR] Failed to execute goal org.codehaus.mojo:exec-maven-plugin:1.3.1:exec
(dist) on project hadoop-ozone-dist: Command execution failed.: Process exited
with an error: 1 (Exit value: 1) -> [Help 1]
Seems one file copy failed:
$ mkdir -p ./share/hadoop/ozoneplugin
$ cp
/Users/xyao/git/hadoop/hadoop-ozone/objectstore-service/target/hadoop-ozone-objectstore-service-0.4.0-SNAPSHOT-plugin.jar
./share/hadoop/ozoneplugin/hadoop-ozone-datanode-plugin-0.4.0-SNAPSHOT.jar
Failed!
{code}
> Ozone dist build failed on Jenkins
> --
>
> Key: HDDS-1179
> URL: https://issues.apache.org/jira/browse/HDDS-1179
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
>Priority: Major
> Attachments: HDDS-1179.001.patch
>
>
> This is part of the Jenkins execution and was reported in several latest HDDS
> Jenkins run.
> I spend some today and found a simplified repro steps:
> {code}
> cd hadoop-ozone/dist
> mvn -Phdds -DskipTests -fae clean install -DskipTests=true
> -Dmaven.javadoc.skip=true -Dcheckstyle.skip=true -Dfindbugs.skip=true
> {code}
>
> The root cause is that the
> hadoop-ozone/dist/dev-support/bin/dist-layout-stitching need
> objectstore-sevice jar being build earlier but he dependency was not
> explicitly declared in pom. I will attach a fix shortly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1178) Healthy pipeline Chill Mode Rule
[ https://issues.apache.org/jira/browse/HDDS-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1178: - Status: Patch Available (was: Open) > Healthy pipeline Chill Mode Rule > > > Key: HDDS-1178 > URL: https://issues.apache.org/jira/browse/HDDS-1178 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > This Jira is to implement one of the chill mode rule. > h2. Pipeline Rule with configurable percentage of open pipelines with all > datanodes reported: > h2. In this rule, we consider when at least 10% of the pipelines have all 3 > data node's reported (if it is a 3 node ratis ring), and one node reported if > it is a one node ratis ring. (The percentage is configurable via ozone > configuration parameter). > > This rule satisfies, once the SCM is restarted, and after exiting chill mode, > we have enough pipelines to give for clients for writes to succeed. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1178) Healthy pipeline Chill Mode Rule
[ https://issues.apache.org/jira/browse/HDDS-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-1178: - Labels: pull-request-available (was: ) > Healthy pipeline Chill Mode Rule > > > Key: HDDS-1178 > URL: https://issues.apache.org/jira/browse/HDDS-1178 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > > This Jira is to implement one of the chill mode rule. > h2. Pipeline Rule with configurable percentage of open pipelines with all > datanodes reported: > h2. In this rule, we consider when at least 10% of the pipelines have all 3 > data node's reported (if it is a 3 node ratis ring), and one node reported if > it is a one node ratis ring. (The percentage is configurable via ozone > configuration parameter). > > This rule satisfies, once the SCM is restarted, and after exiting chill mode, > we have enough pipelines to give for clients for writes to succeed. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1178) Healthy pipeline Chill Mode Rule
[ https://issues.apache.org/jira/browse/HDDS-1178?focusedWorklogId=204007&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-204007 ] ASF GitHub Bot logged work on HDDS-1178: Author: ASF GitHub Bot Created on: 26/Feb/19 01:18 Start Date: 26/Feb/19 01:18 Worklog Time Spent: 10m Work Description: bharatviswa504 commented on pull request #518: HDDS-1178. Healthy pipeline Chill Mode Rule. URL: https://github.com/apache/hadoop/pull/518 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 204007) Time Spent: 10m Remaining Estimate: 0h > Healthy pipeline Chill Mode Rule > > > Key: HDDS-1178 > URL: https://issues.apache.org/jira/browse/HDDS-1178 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > This Jira is to implement one of the chill mode rule. > h2. Pipeline Rule with configurable percentage of open pipelines with all > datanodes reported: > h2. In this rule, we consider when at least 10% of the pipelines have all 3 > data node's reported (if it is a 3 node ratis ring), and one node reported if > it is a one node ratis ring. (The percentage is configurable via ozone > configuration parameter). > > This rule satisfies, once the SCM is restarted, and after exiting chill mode, > we have enough pipelines to give for clients for writes to succeed. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1179) Ozone dist build failed on Jenkins
[
https://issues.apache.org/jira/browse/HDDS-1179?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HDDS-1179:
-
Attachment: HDDS-1179.001.patch
> Ozone dist build failed on Jenkins
> --
>
> Key: HDDS-1179
> URL: https://issues.apache.org/jira/browse/HDDS-1179
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Xiaoyu Yao
>Assignee: Xiaoyu Yao
>Priority: Major
> Attachments: HDDS-1179.001.patch
>
>
> This is part of the Jenkins execution and was reported in several latest HDDS
> Jenkins run.
> I spend some today and found a simplified repro steps:
> {code}
> cd hadoop-ozone/dist
> mvn -Phdds -DskipTests -fae clean install -DskipTests=true
> -Dmaven.javadoc.skip=true -Dcheckstyle.skip=true -Dfindbugs.skip=true
> {code}
>
> The root cause is that the
> hadoop-ozone/dist/dev-support/bin/dist-layout-stitching need
> objectstore-sevice jar being build earlier but he dependency was not
> explicitly declared in pom. I will attach a fix shortly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1178) Healthy pipeline Chill Mode Rule
[ https://issues.apache.org/jira/browse/HDDS-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1178: - Summary: Healthy pipeline Chill Mode Rule (was: Healthy pipeline Chill Mode) > Healthy pipeline Chill Mode Rule > > > Key: HDDS-1178 > URL: https://issues.apache.org/jira/browse/HDDS-1178 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > > This Jira is to implement one of the chill mode rule. > h2. Pipeline Rule with configurable percentage of open pipelines with all > datanodes reported: > h2. In this rule, we consider when at least 10% of the pipelines have all 3 > data node's reported (if it is a 3 node ratis ring), and one node reported if > it is a one node ratis ring. (The percentage is configurable via ozone > configuration parameter). > > This rule satisfies, once the SCM is restarted, and after exiting chill mode, > we have enough pipelines to give for clients for writes to succeed. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1179) Ozone dist build failed on Jenkins
Xiaoyu Yao created HDDS-1179:
Summary: Ozone dist build failed on Jenkins
Key: HDDS-1179
URL: https://issues.apache.org/jira/browse/HDDS-1179
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Xiaoyu Yao
Assignee: Xiaoyu Yao
This is part of the Jenkins execution and was reported in several latest HDDS
Jenkins run.
I spend some today and found a simplified repro steps:
{code}
cd hadoop-ozone/dist
mvn -Phdds -DskipTests -fae clean install -DskipTests=true
-Dmaven.javadoc.skip=true -Dcheckstyle.skip=true -Dfindbugs.skip=true
{code}
The root cause is that the
hadoop-ozone/dist/dev-support/bin/dist-layout-stitching need objectstore-sevice
jar being build earlier but he dependency was not explicitly declared in pom. I
will attach a fix shortly.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-1178) Healthy pipeline Chill Mode
Bharat Viswanadham created HDDS-1178: Summary: Healthy pipeline Chill Mode Key: HDDS-1178 URL: https://issues.apache.org/jira/browse/HDDS-1178 Project: Hadoop Distributed Data Store Issue Type: Bug Reporter: Bharat Viswanadham Assignee: Bharat Viswanadham This Jira is to implement one of the chill mode rule. h2. Pipeline Rule with configurable percentage of open pipelines with all datanodes reported: h2. In this rule, we consider when at least 10% of the pipelines have all 3 data node's reported (if it is a 3 node ratis ring), and one node reported if it is a one node ratis ring. (The percentage is configurable via ozone configuration parameter). This rule satisfies, once the SCM is restarted, and after exiting chill mode, we have enough pipelines to give for clients for writes to succeed. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1178) Healthy pipeline Chill Mode
[ https://issues.apache.org/jira/browse/HDDS-1178?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Bharat Viswanadham updated HDDS-1178: - Component/s: SCM > Healthy pipeline Chill Mode > --- > > Key: HDDS-1178 > URL: https://issues.apache.org/jira/browse/HDDS-1178 > Project: Hadoop Distributed Data Store > Issue Type: Bug > Components: SCM >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > > This Jira is to implement one of the chill mode rule. > h2. Pipeline Rule with configurable percentage of open pipelines with all > datanodes reported: > h2. In this rule, we consider when at least 10% of the pipelines have all 3 > data node's reported (if it is a 3 node ratis ring), and one node reported if > it is a one node ratis ring. (The percentage is configurable via ozone > configuration parameter). > > This rule satisfies, once the SCM is restarted, and after exiting chill mode, > we have enough pipelines to give for clients for writes to succeed. > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-594) SCM CA: DN sends CSR and uses certificate issued by SCM
[ https://issues.apache.org/jira/browse/HDDS-594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777449#comment-16777449 ] Ajay Kumar commented on HDDS-594: - patch v4 to address checkstyle and findbug. > SCM CA: DN sends CSR and uses certificate issued by SCM > --- > > Key: HDDS-594 > URL: https://issues.apache.org/jira/browse/HDDS-594 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDDS-594.00.patch, HDDS-594.01.patch, HDDS-594.02.patch, > HDDS-594.03.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-594) SCM CA: DN sends CSR and uses certificate issued by SCM
[ https://issues.apache.org/jira/browse/HDDS-594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDDS-594: Attachment: HDDS-594.03.patch > SCM CA: DN sends CSR and uses certificate issued by SCM > --- > > Key: HDDS-594 > URL: https://issues.apache.org/jira/browse/HDDS-594 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDDS-594.00.patch, HDDS-594.01.patch, HDDS-594.02.patch, > HDDS-594.03.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14314) fullBlockReportLeaseId should be reset after registering to NN
[
https://issues.apache.org/jira/browse/HDFS-14314?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
star updated HDFS-14314:
Attachment: HDFS-14314.2.patch
> fullBlockReportLeaseId should be reset after registering to NN
> --
>
> Key: HDFS-14314
> URL: https://issues.apache.org/jira/browse/HDFS-14314
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: datanode
>Affects Versions: 2.8.4
> Environment:
>
>
>Reporter: star
>Priority: Critical
> Fix For: 2.8.4
>
> Attachments: HDFS-14314.0.patch, HDFS-14314.2.patch, HDFS-14314.patch
>
>
> since HDFS-7923 ,to rate-limit DN block report, DN will ask for a full
> block lease id from active NN before sending full block to NN. Then DN will
> send full block report together with lease id. If the lease id is invalid, NN
> will reject the full block report and log "not in the pending set".
> In a case when DN is doing full block reporting while NN is restarted.
> It happens that DN will later send a full block report with lease id
> ,acquired from previous NN instance, which is invalid to the new NN instance.
> Though DN recognized the new NN instance by heartbeat and reregister itself,
> it did not reset the lease id from previous instance.
> The issuse may cause DNs to temporarily go dead, making it unsafe to
> restart NN especially in hadoop cluster which has large amount of DNs.
> HDFS-12914 reported the issue without any clues why it occurred and remain
> unsolved.
> To make it clear, look at code below. We take it from method
> offerService of class BPServiceActor. We eliminate some code to focus on
> current issue. fullBlockReportLeaseId is a local variable to hold lease id
> from NN. Exceptions will occur at blockReport call when NN restarting, which
> will be caught by catch block in while loop. Thus fullBlockReportLeaseId will
> not be set to 0. After NN restarted, DN will send full block report which
> will be rejected by the new NN instance. DN will never send full block report
> until the next full block report schedule, about an hour later.
> Solution is simple, just reset fullBlockReportLeaseId to 0 after any
> exception or after registering to NN. Thus it will ask for a valid
> fullBlockReportLeaseId from new NN instance.
> {code:java}
> private void offerService() throws Exception {
> long fullBlockReportLeaseId = 0;
> //
> // Now loop for a long time
> //
> while (shouldRun()) {
> try {
> final long startTime = scheduler.monotonicNow();
> //
> // Every so often, send heartbeat or block-report
> //
> final boolean sendHeartbeat = scheduler.isHeartbeatDue(startTime);
> HeartbeatResponse resp = null;
> if (sendHeartbeat) {
>
> boolean requestBlockReportLease = (fullBlockReportLeaseId == 0) &&
> scheduler.isBlockReportDue(startTime);
> scheduler.scheduleNextHeartbeat();
> if (!dn.areHeartbeatsDisabledForTests()) {
> resp = sendHeartBeat(requestBlockReportLease);
> assert resp != null;
> if (resp.getFullBlockReportLeaseId() != 0) {
> if (fullBlockReportLeaseId != 0) {
> LOG.warn(nnAddr + " sent back a full block report lease " +
> "ID of 0x" +
> Long.toHexString(resp.getFullBlockReportLeaseId()) +
> ", but we already have a lease ID of 0x" +
> Long.toHexString(fullBlockReportLeaseId) + ". " +
> "Overwriting old lease ID.");
> }
> fullBlockReportLeaseId = resp.getFullBlockReportLeaseId();
> }
>
> }
> }
>
>
> if ((fullBlockReportLeaseId != 0) || forceFullBr) {
> //Exception occurred here when NN restarting
> cmds = blockReport(fullBlockReportLeaseId);
> fullBlockReportLeaseId = 0;
> }
>
> } catch(RemoteException re) {
>
> } // while (shouldRun())
> } // offerService{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Comment Edited] (HDDS-134) SCM CA: OM sends CSR and uses certificate issued by SCM
[
https://issues.apache.org/jira/browse/HDDS-134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777418#comment-16777418
]
Ajay Kumar edited comment on HDDS-134 at 2/26/19 12:39 AM:
---
Failure in {{TestSecureOzoneCluster}} is due to bug in SCM cert storage
functionality. Handled in [HDDS-1176].
was (Author: ajayydv):
Failure in {{TestSecureOzoneCluster}} is due to bug in SCM cert storage
functionality. Handled in [HDDS-1177].
> SCM CA: OM sends CSR and uses certificate issued by SCM
> ---
>
> Key: HDDS-134
> URL: https://issues.apache.org/jira/browse/HDDS-134
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
> Attachments: HDDS-134.00.patch, HDDS-134.01.patch, HDDS-134.02.patch,
> HDDS-134.03.patch, HDDS-134.04.patch, HDDS-134.05.patch
>
>
> Initialize OM keypair and get SCM signed certificate.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1072) Implement RetryProxy and FailoverProxy for OM client
[ https://issues.apache.org/jira/browse/HDDS-1072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777439#comment-16777439 ] Arpit Agarwal commented on HDDS-1072: - Thanks [~hanishakoneru]. I will take a look at this patch. > Implement RetryProxy and FailoverProxy for OM client > > > Key: HDDS-1072 > URL: https://issues.apache.org/jira/browse/HDDS-1072 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: OM >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > Attachments: HDDS-1072.001.patch, HDDS-1072.002.patch > > > RPC Client should implement a retry and failover proxy provider to failover > between OM Ratis clients. The failover should occur in two scenarios: > # When the client is unable to connect to the OM (either because of network > issues or because the OM is down). The client retry proxy provider should > failover to next OM in the cluster. > # When OM Ratis Client receives a response from the Ratis server for its > request, it also gets the LeaderId of server which processed this request > (the current Leader OM nodeId). This information should be propagated back to > the client. The Client failover Proxy provider should failover to the leader > OM node. This helps avoid an extra hop from Follower OM Ratis Client to > Leader OM Ratis server for every request. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1177) Add validation to AuthorizationHeaderV4
[ https://issues.apache.org/jira/browse/HDDS-1177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDDS-1177: - Attachment: HDDS-1177.00.patch > Add validation to AuthorizationHeaderV4 > --- > > Key: HDDS-1177 > URL: https://issues.apache.org/jira/browse/HDDS-1177 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDDS-1177.00.patch > > > Add validation to AuthorizationHeaderV4. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1177) Add validation to AuthorizationHeaderV4
[ https://issues.apache.org/jira/browse/HDDS-1177?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777440#comment-16777440 ] Ajay Kumar commented on HDDS-1177: -- cc: [~anu] > Add validation to AuthorizationHeaderV4 > --- > > Key: HDDS-1177 > URL: https://issues.apache.org/jira/browse/HDDS-1177 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDDS-1177.00.patch > > > Add validation to AuthorizationHeaderV4. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1177) Add validation to AuthorizationHeaderV4
[ https://issues.apache.org/jira/browse/HDDS-1177?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDDS-1177: - Status: Patch Available (was: Open) > Add validation to AuthorizationHeaderV4 > --- > > Key: HDDS-1177 > URL: https://issues.apache.org/jira/browse/HDDS-1177 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDDS-1177.00.patch > > > Add validation to AuthorizationHeaderV4. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1072) Implement RetryProxy and FailoverProxy for OM client
[ https://issues.apache.org/jira/browse/HDDS-1072?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777432#comment-16777432 ] Hanisha Koneru commented on HDDS-1072: -- Thank you [~linyiqun] for the review. I am working on the unit tests for maxRetries and maxFailovers. Addressed the other two comments in patch v02. > Implement RetryProxy and FailoverProxy for OM client > > > Key: HDDS-1072 > URL: https://issues.apache.org/jira/browse/HDDS-1072 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: OM >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > Attachments: HDDS-1072.001.patch, HDDS-1072.002.patch > > > RPC Client should implement a retry and failover proxy provider to failover > between OM Ratis clients. The failover should occur in two scenarios: > # When the client is unable to connect to the OM (either because of network > issues or because the OM is down). The client retry proxy provider should > failover to next OM in the cluster. > # When OM Ratis Client receives a response from the Ratis server for its > request, it also gets the LeaderId of server which processed this request > (the current Leader OM nodeId). This information should be propagated back to > the client. The Client failover Proxy provider should failover to the leader > OM node. This helps avoid an extra hop from Follower OM Ratis Client to > Leader OM Ratis server for every request. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1061) DelegationToken: Add certificate serial id to Ozone Delegation Token Identifier
[
https://issues.apache.org/jira/browse/HDDS-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777429#comment-16777429
]
Hadoop QA commented on HDDS-1061:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
13s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 3 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
32s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
18s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 17m
9s{color} | {color:red} root in trunk failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
49s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 2m
4s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
18m 14s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
54s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
22s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
26s{color} | {color:red} common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvninstall {color} | {color:red} 0m
19s{color} | {color:red} ozone-manager in the patch failed. {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 15m
9s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} cc {color} | {color:red} 15m 9s{color} |
{color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 15m 9s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
46s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
39s{color} | {color:red} common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} mvnsite {color} | {color:red} 0m
32s{color} | {color:red} ozone-manager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 55s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
34s{color} | {color:red} common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 0m
33s{color} | {color:red} ozone-manager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
57s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 25s{color}
| {color:red} common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 39s{color}
| {color:red} common in the patch failed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 0m 31s{color}
| {color:red} ozone-manager in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
41s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}108m 52s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdds.security.x509.certificate.client.TestDefaultCertificateClient |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0
[jira] [Updated] (HDDS-1072) Implement RetryProxy and FailoverProxy for OM client
[ https://issues.apache.org/jira/browse/HDDS-1072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hanisha Koneru updated HDDS-1072: - Status: Patch Available (was: Open) > Implement RetryProxy and FailoverProxy for OM client > > > Key: HDDS-1072 > URL: https://issues.apache.org/jira/browse/HDDS-1072 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: OM >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > Attachments: HDDS-1072.001.patch, HDDS-1072.002.patch > > > RPC Client should implement a retry and failover proxy provider to failover > between OM Ratis clients. The failover should occur in two scenarios: > # When the client is unable to connect to the OM (either because of network > issues or because the OM is down). The client retry proxy provider should > failover to next OM in the cluster. > # When OM Ratis Client receives a response from the Ratis server for its > request, it also gets the LeaderId of server which processed this request > (the current Leader OM nodeId). This information should be propagated back to > the client. The Client failover Proxy provider should failover to the leader > OM node. This helps avoid an extra hop from Follower OM Ratis Client to > Leader OM Ratis server for every request. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1072) Implement RetryProxy and FailoverProxy for OM client
[ https://issues.apache.org/jira/browse/HDDS-1072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hanisha Koneru updated HDDS-1072: - Attachment: HDDS-1072.002.patch > Implement RetryProxy and FailoverProxy for OM client > > > Key: HDDS-1072 > URL: https://issues.apache.org/jira/browse/HDDS-1072 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: OM >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > Attachments: HDDS-1072.001.patch, HDDS-1072.002.patch > > > RPC Client should implement a retry and failover proxy provider to failover > between OM Ratis clients. The failover should occur in two scenarios: > # When the client is unable to connect to the OM (either because of network > issues or because the OM is down). The client retry proxy provider should > failover to next OM in the cluster. > # When OM Ratis Client receives a response from the Ratis server for its > request, it also gets the LeaderId of server which processed this request > (the current Leader OM nodeId). This information should be propagated back to > the client. The Client failover Proxy provider should failover to the leader > OM node. This helps avoid an extra hop from Follower OM Ratis Client to > Leader OM Ratis server for every request. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1072) Implement RetryProxy and FailoverProxy for OM client
[ https://issues.apache.org/jira/browse/HDDS-1072?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Hanisha Koneru updated HDDS-1072: - Status: Open (was: Patch Available) > Implement RetryProxy and FailoverProxy for OM client > > > Key: HDDS-1072 > URL: https://issues.apache.org/jira/browse/HDDS-1072 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task > Components: OM >Reporter: Hanisha Koneru >Assignee: Hanisha Koneru >Priority: Major > Attachments: HDDS-1072.001.patch, HDDS-1072.002.patch > > > RPC Client should implement a retry and failover proxy provider to failover > between OM Ratis clients. The failover should occur in two scenarios: > # When the client is unable to connect to the OM (either because of network > issues or because the OM is down). The client retry proxy provider should > failover to next OM in the cluster. > # When OM Ratis Client receives a response from the Ratis server for its > request, it also gets the LeaderId of server which processed this request > (the current Leader OM nodeId). This information should be propagated back to > the client. The Client failover Proxy provider should failover to the leader > OM node. This helps avoid an extra hop from Follower OM Ratis Client to > Leader OM Ratis server for every request. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14201) Ability to disallow safemode NN to become active
[ https://issues.apache.org/jira/browse/HDFS-14201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777427#comment-16777427 ] Xiao Liang commented on HDFS-14201: --- Thanks [~hexiaoqiao] for pointing out the manual transition, it's a valid point. Actually I think combining the logic in [^HDFS-14201.002.patch] and [^HDFS-14201.003.patch] could be an option, so that when the switch for this feature is on: # in auto-failover mode, ZKFC choose a ready-to-serve NameNode to become active, as those in safemode ones report UNHEALTHY; # in manual mode, NameNode in safemode will not be able to transit to active; The same configuration item would be controlling these logic to be on/off. How do you think [~hexiaoqiao]? I would upload a new patch as proposed if you think it's a reasonable option. > Ability to disallow safemode NN to become active > > > Key: HDFS-14201 > URL: https://issues.apache.org/jira/browse/HDFS-14201 > Project: Hadoop HDFS > Issue Type: Improvement > Components: auto-failover >Affects Versions: 3.1.1, 2.9.2 >Reporter: Xiao Liang >Assignee: Xiao Liang >Priority: Major > Attachments: HDFS-14201.001.patch, HDFS-14201.002.patch, > HDFS-14201.003.patch > > > Currently with HA, Namenode in safemode can be possibly selected as active, > for availability of both read and write, Namenodes not in safemode are better > choices to become active though. > It can take tens of minutes for a cold started Namenode to get out of > safemode, especially when there are large number of files and blocks in HDFS, > that means if a Namenode in safemode become active, the cluster will be not > fully functioning for quite a while, even if it can while there is some > Namenode not in safemode. > The proposal here is to add an option, to allow Namenode to report itself as > UNHEALTHY to ZKFC, if it's in safemode, so as to only allow fully functioning > Namenode to become active, improving the general availability of the cluster. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-134) SCM CA: OM sends CSR and uses certificate issued by SCM
[
https://issues.apache.org/jira/browse/HDDS-134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777418#comment-16777418
]
Ajay Kumar commented on HDDS-134:
-
Failure in {{TestSecureOzoneCluster}} is due to bug in SCM cert storage
functionality. Handled in [HDDS-1177].
> SCM CA: OM sends CSR and uses certificate issued by SCM
> ---
>
> Key: HDDS-134
> URL: https://issues.apache.org/jira/browse/HDDS-134
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Ajay Kumar
>Assignee: Ajay Kumar
>Priority: Major
> Attachments: HDDS-134.00.patch, HDDS-134.01.patch, HDDS-134.02.patch,
> HDDS-134.03.patch, HDDS-134.04.patch, HDDS-134.05.patch
>
>
> Initialize OM keypair and get SCM signed certificate.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14272) [SBN read] ObserverReadProxyProvider should sync with active txnID on startup
[
https://issues.apache.org/jira/browse/HDFS-14272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Krogen updated HDFS-14272:
---
Attachment: (was: HDFS-14272.002.patch)
> [SBN read] ObserverReadProxyProvider should sync with active txnID on startup
> -
>
> Key: HDFS-14272
> URL: https://issues.apache.org/jira/browse/HDFS-14272
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: tools
> Environment: CDH6.1 (Hadoop 3.0.x) + Consistency Reads from Standby +
> SSL + Kerberos + RPC encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HDFS-14272.000.patch, HDFS-14272.001.patch,
> HDFS-14272.002.patch
>
>
> It is typical for integration tests to create some files and then check their
> existence. For example, like the following simple bash script:
> {code:java}
> # hdfs dfs -touchz /tmp/abc
> # hdfs dfs -ls /tmp/abc
> {code}
> The test executes HDFS bash command sequentially, but it may fail with
> Consistent Standby Read because the -ls does not find the file.
> Analysis: the second bash command, while launched sequentially after the
> first one, is not aware of the state id returned from the first bash command.
> So ObserverNode wouldn't wait for the the edits to get propagated, and thus
> fails.
> I've got a cluster where the Observer has tens of seconds of RPC latency, and
> this becomes very annoying. (I am still trying to figure out why this
> Observer has such a long RPC latency. But that's another story.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14272) [SBN read] ObserverReadProxyProvider should sync with active txnID on startup
[
https://issues.apache.org/jira/browse/HDFS-14272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Krogen updated HDFS-14272:
---
Attachment: HDFS-14272.002.patch
> [SBN read] ObserverReadProxyProvider should sync with active txnID on startup
> -
>
> Key: HDFS-14272
> URL: https://issues.apache.org/jira/browse/HDFS-14272
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: tools
> Environment: CDH6.1 (Hadoop 3.0.x) + Consistency Reads from Standby +
> SSL + Kerberos + RPC encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HDFS-14272.000.patch, HDFS-14272.001.patch,
> HDFS-14272.002.patch
>
>
> It is typical for integration tests to create some files and then check their
> existence. For example, like the following simple bash script:
> {code:java}
> # hdfs dfs -touchz /tmp/abc
> # hdfs dfs -ls /tmp/abc
> {code}
> The test executes HDFS bash command sequentially, but it may fail with
> Consistent Standby Read because the -ls does not find the file.
> Analysis: the second bash command, while launched sequentially after the
> first one, is not aware of the state id returned from the first bash command.
> So ObserverNode wouldn't wait for the the edits to get propagated, and thus
> fails.
> I've got a cluster where the Observer has tens of seconds of RPC latency, and
> this becomes very annoying. (I am still trying to figure out why this
> Observer has such a long RPC latency. But that's another story.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14272) [SBN read] ObserverReadProxyProvider should sync with active txnID on startup
[
https://issues.apache.org/jira/browse/HDFS-14272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777406#comment-16777406
]
Erik Krogen commented on HDFS-14272:
[~shv]:
{code}
Since proxies to individual NNs have not been created yet, changeProxy() will
create proxy for the corresponding NN, then call getHAServiceState() and cache
the state. Then it will proceed with the msync() call, which is not needed
because the client already knows the state of that NN.
{code}
Yes, the {{changeProxy()}} within ORPP will call {{getHAServiceState()}}. If an
{{msync()}} was subsequently sent over this proxy, I agree it would be somewhat
redundant. However, {{msync()}} is called on the {{failoverProxy}}, which is a
CFPP. The {{changeProxy()}} method on CFPP will _not_ call
{{getHAServiceState()}}; the {{msync()}} method is the first call that is
performed on the proxy. Thus the number of RPC calls is identical. Whether
those RPCs are calls to {{getHAServiceState()}} or {{msync()}} is not important
for performance, and re-using the {{msync()}} functionality is cleaner IMO.
[~csun]: thanks for the comments, I uploaded v002 addressing them.
> [SBN read] ObserverReadProxyProvider should sync with active txnID on startup
> -
>
> Key: HDFS-14272
> URL: https://issues.apache.org/jira/browse/HDFS-14272
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: tools
> Environment: CDH6.1 (Hadoop 3.0.x) + Consistency Reads from Standby +
> SSL + Kerberos + RPC encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HDFS-14272.000.patch, HDFS-14272.001.patch,
> HDFS-14272.002.patch
>
>
> It is typical for integration tests to create some files and then check their
> existence. For example, like the following simple bash script:
> {code:java}
> # hdfs dfs -touchz /tmp/abc
> # hdfs dfs -ls /tmp/abc
> {code}
> The test executes HDFS bash command sequentially, but it may fail with
> Consistent Standby Read because the -ls does not find the file.
> Analysis: the second bash command, while launched sequentially after the
> first one, is not aware of the state id returned from the first bash command.
> So ObserverNode wouldn't wait for the the edits to get propagated, and thus
> fails.
> I've got a cluster where the Observer has tens of seconds of RPC latency, and
> this becomes very annoying. (I am still trying to figure out why this
> Observer has such a long RPC latency. But that's another story.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14272) [SBN read] ObserverReadProxyProvider should sync with active txnID on startup
[
https://issues.apache.org/jira/browse/HDFS-14272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Erik Krogen updated HDFS-14272:
---
Attachment: HDFS-14272.002.patch
> [SBN read] ObserverReadProxyProvider should sync with active txnID on startup
> -
>
> Key: HDFS-14272
> URL: https://issues.apache.org/jira/browse/HDFS-14272
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: tools
> Environment: CDH6.1 (Hadoop 3.0.x) + Consistency Reads from Standby +
> SSL + Kerberos + RPC encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HDFS-14272.000.patch, HDFS-14272.001.patch,
> HDFS-14272.002.patch
>
>
> It is typical for integration tests to create some files and then check their
> existence. For example, like the following simple bash script:
> {code:java}
> # hdfs dfs -touchz /tmp/abc
> # hdfs dfs -ls /tmp/abc
> {code}
> The test executes HDFS bash command sequentially, but it may fail with
> Consistent Standby Read because the -ls does not find the file.
> Analysis: the second bash command, while launched sequentially after the
> first one, is not aware of the state id returned from the first bash command.
> So ObserverNode wouldn't wait for the the edits to get propagated, and thus
> fails.
> I've got a cluster where the Observer has tens of seconds of RPC latency, and
> this becomes very annoying. (I am still trying to figure out why this
> Observer has such a long RPC latency. But that's another story.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14052) RBF: Use Router keytab for WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-14052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777403#comment-16777403
]
CR Hota commented on HDFS-14052:
[~brahmareddy] Thanks for the review. Changed the code appropriately.
> RBF: Use Router keytab for WebHDFS
> --
>
> Key: HDFS-14052
> URL: https://issues.apache.org/jira/browse/HDFS-14052
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: CR Hota
>Priority: Major
> Attachments: HDFS-14052-HDFS-13891.0.patch,
> HDFS-14052-HDFS-13891.1.patch, HDFS-14052-HDFS-13891.2.patch,
> HDFS-14052-HDFS-13891.3.patch, HDFS-14052-HDFS-13891.4.patch
>
>
> When the RouterHttpServer starts it does:
> {code}
> NameNodeHttpServer.initWebHdfs(conf, httpAddress.getHostName(),
> httpServer,
> RouterWebHdfsMethods.class.getPackage().getName());
> {code}
> This function is in the NN and is pretty generic.
> However, it then calls to NameNodeHttpServer#getAuthFilterParams, which does:
> {code}
> String httpKeytab = conf.get(DFSUtil.getSpnegoKeytabKey(conf,
> DFSConfigKeys.DFS_NAMENODE_KEYTAB_FILE_KEY));
> {code}
> In most cases, the regular web keytab will kick in, but we should make this a
> parameter and load the Router one just in case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14052) RBF: Use Router keytab for WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-14052?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777381#comment-16777381
]
Hadoop QA commented on HDFS-14052:
--
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} HDFS-13891 Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 16m
12s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
30s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
21s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
34s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 56s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
55s{color} | {color:green} HDFS-13891 passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
37s{color} | {color:green} HDFS-13891 passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
17s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
28s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 23s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
58s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 21m
33s{color} | {color:green} hadoop-hdfs-rbf in the patch passed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
22s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 67m 22s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDFS-14052 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12960084/HDFS-14052-HDFS-13891.4.patch
|
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 2514be2d45de 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24
10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | HDFS-13891 / 0477b0b |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/26323/testReport/ |
| Max. process+thread count | 1364 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs-rbf U:
hadoop-hdfs-project/hadoop-hdfs-rbf |
| Console output |
https://builds.apache.org/job/PreCommit-HDFS-Build/26323/console |
| Powered by | Apache Yetus 0.8.0 http://yetus.apache.org |
This message was automatically generated.
> RBF: Use Router keytab for WebHDFS
> --
>
> Key: HDF
[jira] [Commented] (HDFS-14272) [SBN read] ObserverReadProxyProvider should sync with active txnID on startup
[
https://issues.apache.org/jira/browse/HDFS-14272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777382#comment-16777382
]
Konstantin Shvachko commented on HDFS-14272:
??If msync() is called on an observer or standby NN, the node will reject the
request.??
Exactly right, so {{msync()}} will try to connect to all NNs until it finds the
active node. More precisely, msync() exploits the failover mechanism to loop
over the available NameNodes, which is a correct way of doing things, but not
the most efficient.
What I suggest is to loop explicitly through available NNs using
{{getHAServiceState()}} on the first RPC call. It is equivalent to what msync()
will be doing in this case, but with half of RPC calls. Since proxies to
individual NNs have not been created yet, {{changeProxy()}} will create proxy
for the corresponding NN, then call {{getHAServiceState()}} and cache the
state. Then it will proceed with the msync() call, which is not needed because
the client already knows the state of that NN.
> [SBN read] ObserverReadProxyProvider should sync with active txnID on startup
> -
>
> Key: HDFS-14272
> URL: https://issues.apache.org/jira/browse/HDFS-14272
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: tools
> Environment: CDH6.1 (Hadoop 3.0.x) + Consistency Reads from Standby +
> SSL + Kerberos + RPC encryption
>Reporter: Wei-Chiu Chuang
>Assignee: Erik Krogen
>Priority: Major
> Attachments: HDFS-14272.000.patch, HDFS-14272.001.patch
>
>
> It is typical for integration tests to create some files and then check their
> existence. For example, like the following simple bash script:
> {code:java}
> # hdfs dfs -touchz /tmp/abc
> # hdfs dfs -ls /tmp/abc
> {code}
> The test executes HDFS bash command sequentially, but it may fail with
> Consistent Standby Read because the -ls does not find the file.
> Analysis: the second bash command, while launched sequentially after the
> first one, is not aware of the state id returned from the first bash command.
> So ObserverNode wouldn't wait for the the edits to get propagated, and thus
> fails.
> I've got a cluster where the Observer has tens of seconds of RPC latency, and
> this becomes very annoying. (I am still trying to figure out why this
> Observer has such a long RPC latency. But that's another story.)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14111) hdfsOpenFile on HDFS causes unnecessary IO from file offset 0
[
https://issues.apache.org/jira/browse/HDFS-14111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777375#comment-16777375
]
Todd Lipcon commented on HDFS-14111:
{quote}
jthr = newJavaStr(env, "in:readbytebuffer", &jCapabilityString);
jthr = invokeMethod(env, &jVal, INSTANCE, jFile, HADOOP_ISTRM,
"hasCapability", "(Ljava/lang/String;)Z", jCapabilityString);
{quote}
We probably need to check 'jthr' from the newJavaStr before moving along to
invokeMethod, even though it's highly unlikely to hit an issue. (it could OOM)
Otherwise I think this makes sense. [~ste...@apache.org] does the
StreamCapabiltiies change look good to you?
> hdfsOpenFile on HDFS causes unnecessary IO from file offset 0
> -
>
> Key: HDFS-14111
> URL: https://issues.apache.org/jira/browse/HDFS-14111
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: hdfs-client, libhdfs
>Affects Versions: 3.2.0
>Reporter: Todd Lipcon
>Assignee: Sahil Takiar
>Priority: Major
> Attachments: HDFS-14111.001.patch, HDFS-14111.002.patch
>
>
> hdfsOpenFile() calls readDirect() with a 0-length argument in order to check
> whether the underlying stream supports bytebuffer reads. With DFSInputStream,
> the read(0) isn't short circuited, and results in the DFSClient opening a
> block reader. In the case of a remote block, the block reader will actually
> issue a read of the whole block, causing the datanode to perform unnecessary
> IO and network transfers in order to fill up the client's TCP buffers. This
> causes performance degradation.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1038) Support Service Level Authorization for OM, SCM and DN
[
https://issues.apache.org/jira/browse/HDDS-1038?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777368#comment-16777368
]
Xiaoyu Yao commented on HDDS-1038:
--
Attach patch v10 that fix the findbugs and the unit test failure related.
> Support Service Level Authorization for OM, SCM and DN
> --
>
> Key: HDDS-1038
> URL: https://issues.apache.org/jira/browse/HDDS-1038
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Ajay Kumar
>Assignee: Xiaoyu Yao
>Priority: Major
> Labels: Security
> Fix For: 0.4.0
>
> Attachments: HDDS-1038.00.patch, HDDS-1038.01.patch,
> HDDS-1038.02.patch, HDDS-1038.03.patch, HDDS-1038.04.patch,
> HDDS-1038.05.patch, HDDS-1038.06.patch, HDDS-1038.07.patch,
> HDDS-1038.08.patch, HDDS-1038.09.patch, HDDS-1038.10.patch
>
>
> In a secure Ozone cluster. Datanodes fail to connect to SCM on
> {{StorageContainerDatanodeProtocol}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1038) Support Service Level Authorization for OM, SCM and DN
[
https://issues.apache.org/jira/browse/HDDS-1038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao updated HDDS-1038:
-
Attachment: HDDS-1038.10.patch
> Support Service Level Authorization for OM, SCM and DN
> --
>
> Key: HDDS-1038
> URL: https://issues.apache.org/jira/browse/HDDS-1038
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Ajay Kumar
>Assignee: Xiaoyu Yao
>Priority: Major
> Labels: Security
> Fix For: 0.4.0
>
> Attachments: HDDS-1038.00.patch, HDDS-1038.01.patch,
> HDDS-1038.02.patch, HDDS-1038.03.patch, HDDS-1038.04.patch,
> HDDS-1038.05.patch, HDDS-1038.06.patch, HDDS-1038.07.patch,
> HDDS-1038.08.patch, HDDS-1038.09.patch, HDDS-1038.10.patch
>
>
> In a secure Ozone cluster. Datanodes fail to connect to SCM on
> {{StorageContainerDatanodeProtocol}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Resolved] (HDFS-13781) Unit tests for standby reads.
[ https://issues.apache.org/jira/browse/HDFS-13781?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko resolved HDFS-13781. Resolution: Duplicate Fix Version/s: HDFS-12943 This was handled by HDFS-13523, HDFS-13961, HDFS-13925, and a few other issues. Resolving as duplicate. > Unit tests for standby reads. > - > > Key: HDFS-13781 > URL: https://issues.apache.org/jira/browse/HDFS-13781 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: test >Reporter: Konstantin Shvachko >Priority: Major > Fix For: HDFS-12943 > > > Create more unit tests supporting standby reads feature. Let's come up with a > list of tests that provide sufficient test coverage. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14130) Make ZKFC ObserverNode aware
[ https://issues.apache.org/jira/browse/HDFS-14130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777361#comment-16777361 ] Hudson commented on HDFS-14130: --- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16065 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16065/]) HDFS-14130. [SBN read] Make ZKFC ObserverNode aware. Contributed by (shv: rev a6ab37192a90e5ee868376b42be226e00cce31d8) * (edit) hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/tools/TestDFSZKFailoverController.java * (edit) hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/ha/TestZKFailoverController.java * (edit) hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/ha/ZKFailoverController.java > Make ZKFC ObserverNode aware > > > Key: HDFS-14130 > URL: https://issues.apache.org/jira/browse/HDFS-14130 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha >Affects Versions: HDFS-12943 >Reporter: Konstantin Shvachko >Assignee: xiangheng >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-14130-HDFS-12943.001.patch, > HDFS-14130-HDFS-12943.003.patch, HDFS-14130-HDFS-12943.004.patch, > HDFS-14130-HDFS-12943.005.patch, HDFS-14130-HDFS-12943.006.patch, > HDFS-14130-HDFS-12943.007.patch, HDFS-14130.008.patch, HDFS-14130.009.patch, > HDFS-14130.010.patch, HDFS-14130.011.patch > > > Need to fix automatic failover with ZKFC. Currently it does not know about > ObserverNodes trying to convert them to SBNs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-935) Avoid creating an already created container on a datanode in case of disk removal followed by datanode restart
[
https://issues.apache.org/jira/browse/HDDS-935?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777360#comment-16777360
]
Arpit Agarwal commented on HDDS-935:
Hi [~shashikant], just took a look at this patch. A few thoughts:
# Nitpick: Let's use try with resources for fileInputStream in loadSnapshot
# same with takeSnapshot - we should use try with resources for the output
stream.
# Dispatcher#init - Init methods are usually to be avoided. Sometimes it may
not be possible. In this case it looks like we are calling init twice on the
HddsDispatcher at startup. Any way we can improve this?
# Looks like createContainerSet is not additive. i.e. it only tracks
containers since the last restart. Should this include containers created
previously via the Ratis snapshot?
# Possibly dumb question: Why do we add the container set to
DispatcherContext? We just need to update this set once the container is
successfully created right?
# We can add a bit more detail to this log message e.g. the container has been
lost and cannot be recreated on this DataNode.
{code}
if (getMissingContainerSet().contains(containerID)) {
StorageContainerException sce = new StorageContainerException(
"ContainerID " + containerID + " is missing",
{code}
> Avoid creating an already created container on a datanode in case of disk
> removal followed by datanode restart
> --
>
> Key: HDDS-935
> URL: https://issues.apache.org/jira/browse/HDDS-935
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone Datanode
>Affects Versions: 0.4.0
>Reporter: Rakesh R
>Assignee: Shashikant Banerjee
>Priority: Major
> Attachments: HDDS-935.000.patch, HDDS-935.001.patch,
> HDDS-935.002.patch, HDDS-935.003.patch, HDDS-935.004.patch, HDDS-935.005.patch
>
>
> Currently, a container gets created when a writeChunk request comes to
> HddsDispatcher and if the container does not exist already. In case a disk on
> which a container exists gets removed and datanode restarts and now, if a
> writeChunkRequest comes , it might end up creating the same container again
> with an updated BCSID as it won't detect the disk is removed. This won't be
> detected by SCM as well as it will have the latest BCSID. This Jira aims to
> address this issue.
> The proposed fix would be to persist the all the containerIds existing in the
> containerSet when a ratis snapshot is taken in the snapshot file. If the disk
> is removed and dn gets restarted, the container set will be rebuild after
> scanning all the available disks and the the container list stored in the
> snapshot file will give all the containers created in the datanode. The diff
> between these two will give the exact list of containers which were created
> but were not detected after the restart. Any writeChunk request now should
> validate the container Id from the list of missing containers. Also, we need
> to ensure container creation does not happen as part of applyTransaction of
> writeChunk request in Ratis.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-134) SCM CA: OM sends CSR and uses certificate issued by SCM
[
https://issues.apache.org/jira/browse/HDDS-134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777359#comment-16777359
]
Hadoop QA commented on HDDS-134:
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
19s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
1s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 5 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 7m
17s{color} | {color:blue} Maven dependency ordering for branch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 23m
21s{color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 18m
42s{color} | {color:red} root in trunk failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
31s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
53s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
16m 49s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
38s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:blue}0{color} | {color:blue} mvndep {color} | {color:blue} 0m
27s{color} | {color:blue} Maven dependency ordering for patch {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
41s{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} compile {color} | {color:red} 17m
33s{color} | {color:red} root in the patch failed. {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 17m 33s{color}
| {color:red} root in the patch failed. {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 3m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
52s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
10m 16s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:blue}0{color} | {color:blue} findbugs {color} | {color:blue} 0m
0s{color} | {color:blue} Skipped patched modules with no Java source:
hadoop-ozone/integration-test {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
20s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 25s{color}
| {color:red} common in the patch failed. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 0m
47s{color} | {color:green} ozone-manager in the patch passed. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 8s{color}
| {color:red} integration-test in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
43s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}124m 59s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests |
hadoop.hdds.security.x509.certificate.client.TestDefaultCertificateClient |
| | hadoop.ozone.TestSecureOzoneCluster |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDDS-134 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/129600
[jira] [Updated] (HDFS-14130) Make ZKFC ObserverNode aware
[ https://issues.apache.org/jira/browse/HDFS-14130?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Konstantin Shvachko updated HDFS-14130: --- Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.3.0 Status: Resolved (was: Patch Available) I just committed this. Thank you [~xiangheng]. > Make ZKFC ObserverNode aware > > > Key: HDFS-14130 > URL: https://issues.apache.org/jira/browse/HDFS-14130 > Project: Hadoop HDFS > Issue Type: Sub-task > Components: ha >Affects Versions: HDFS-12943 >Reporter: Konstantin Shvachko >Assignee: xiangheng >Priority: Major > Fix For: 3.3.0 > > Attachments: HDFS-14130-HDFS-12943.001.patch, > HDFS-14130-HDFS-12943.003.patch, HDFS-14130-HDFS-12943.004.patch, > HDFS-14130-HDFS-12943.005.patch, HDFS-14130-HDFS-12943.006.patch, > HDFS-14130-HDFS-12943.007.patch, HDFS-14130.008.patch, HDFS-14130.009.patch, > HDFS-14130.010.patch, HDFS-14130.011.patch > > > Need to fix automatic failover with ZKFC. Currently it does not know about > ObserverNodes trying to convert them to SBNs. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDDS-1038) Support Service Level Authorization for OM, SCM and DN
[
https://issues.apache.org/jira/browse/HDDS-1038?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoyu Yao reassigned HDDS-1038:
Assignee: Xiaoyu Yao (was: Ajay Kumar)
> Support Service Level Authorization for OM, SCM and DN
> --
>
> Key: HDDS-1038
> URL: https://issues.apache.org/jira/browse/HDDS-1038
> Project: Hadoop Distributed Data Store
> Issue Type: Sub-task
>Reporter: Ajay Kumar
>Assignee: Xiaoyu Yao
>Priority: Major
> Labels: Security
> Fix For: 0.4.0
>
> Attachments: HDDS-1038.00.patch, HDDS-1038.01.patch,
> HDDS-1038.02.patch, HDDS-1038.03.patch, HDDS-1038.04.patch,
> HDDS-1038.05.patch, HDDS-1038.06.patch, HDDS-1038.07.patch,
> HDDS-1038.08.patch, HDDS-1038.09.patch
>
>
> In a secure Ozone cluster. Datanodes fail to connect to SCM on
> {{StorageContainerDatanodeProtocol}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1061) DelegationToken: Add certificate serial id to Ozone Delegation Token Identifier
[ https://issues.apache.org/jira/browse/HDDS-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777344#comment-16777344 ] Ajay Kumar commented on HDDS-1061: -- patch v4 to address 2 checkstyle issues. > DelegationToken: Add certificate serial id to Ozone Delegation Token > Identifier > > > Key: HDDS-1061 > URL: https://issues.apache.org/jira/browse/HDDS-1061 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDDS-1061.00.patch, HDDS-1061.01.patch, > HDDS-1061.02.patch, HDDS-1061.03.patch, HDDS-1061.04.patch > > > 1. Add certificate serial id to Ozone Delegation Token Identifier. Required > for OM HA support. > 2. Validate Ozone token based on public key from OM certificate -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1061) DelegationToken: Add certificate serial id to Ozone Delegation Token Identifier
[ https://issues.apache.org/jira/browse/HDDS-1061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ajay Kumar updated HDDS-1061: - Attachment: HDDS-1061.04.patch > DelegationToken: Add certificate serial id to Ozone Delegation Token > Identifier > > > Key: HDDS-1061 > URL: https://issues.apache.org/jira/browse/HDDS-1061 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Attachments: HDDS-1061.00.patch, HDDS-1061.01.patch, > HDDS-1061.02.patch, HDDS-1061.03.patch, HDDS-1061.04.patch > > > 1. Add certificate serial id to Ozone Delegation Token Identifier. Required > for OM HA support. > 2. Validate Ozone token based on public key from OM certificate -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1174) Freon tests are failing with null pointer exception
[
https://issues.apache.org/jira/browse/HDDS-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777339#comment-16777339
]
Hadoop QA commented on HDDS-1174:
-
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
15s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 20m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
32s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
18s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
36s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 0s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
29s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
30s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
24s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
13s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
26s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 10s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 0m
46s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
19s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 1m 6s{color}
| {color:red} tools in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
28s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black} 52m 41s{color} |
{color:black} {color} |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=17.05.0-ce Server=17.05.0-ce Image:yetus/hadoop:8f97d6f |
| JIRA Issue | HDDS-1174 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12960015/HDDS-1174.001.patch |
| Optional Tests | asflicense compile javac javadoc mvninstall mvnsite
unit shadedclient findbugs checkstyle |
| uname | Linux 8362e9c23ced 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / 0edb0c5 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_191 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDDS-Build/2368/artifact/out/patch-unit-hadoop-ozone_tools.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDDS-Build/2368/testReport/ |
| Max. process+thread count | 1861 (vs. ulimit of 1) |
| modules | C: hadoop-ozone/tools U: hadoop-ozone/tools |
| Console output |
https://builds.apache.org/job/PreCommit-HDDS-Build/2368/console |
| Powered by | Apache Yetus 0.8.0-SNAPSHOT http://yetus.apache.org |
This message was
[jira] [Updated] (HDDS-1043) Enable token based authentication for S3 api
[ https://issues.apache.org/jira/browse/HDDS-1043?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDDS-1043: - Status: Open (was: Patch Available) > Enable token based authentication for S3 api > > > Key: HDDS-1043 > URL: https://issues.apache.org/jira/browse/HDDS-1043 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Labels: security > Fix For: 0.4.0 > > Attachments: HDDS-1043.00.patch, HDDS-1043.01.patch, > HDDS-1043.02.patch, HDDS-1043.03.patch > > > Ozone has a S3 api and mechanism to create S3 like secrets for user. This > jira proposes hadoop compatible token based authentication for S3 api which > utilizes S3 secret stored in OM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1151) Propagate the tracing id in ScmBlockLocationProtocol
[ https://issues.apache.org/jira/browse/HDDS-1151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777329#comment-16777329 ] Hudson commented on HDDS-1151: -- SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #16064 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/16064/]) HDDS-1151. Propagate the tracing id in ScmBlockLocationProtocol. (aengineer: rev 9de34d299002d267f9e77c35fb5f910edd3ea5c5) * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocol/ScmBlockLocationProtocol.java * (edit) hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/ScmBlockLocationTestIngClient.java * (edit) hadoop-hdds/common/src/main/proto/ScmBlockLocationProtocol.proto * (edit) hadoop-hdds/server-scm/src/main/java/org/apache/hadoop/hdds/scm/server/SCMBlockProtocolServer.java * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/hdds/scm/protocolPB/ScmBlockLocationProtocolClientSideTranslatorPB.java * (edit) hadoop-hdds/common/src/main/java/org/apache/hadoop/ozone/protocolPB/ScmBlockLocationProtocolServerSideTranslatorPB.java * (edit) hadoop-ozone/objectstore-service/src/main/java/org/apache/hadoop/hdfs/server/datanode/ObjectStoreHandler.java * (edit) hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/OzoneManager.java > Propagate the tracing id in ScmBlockLocationProtocol > > > Key: HDDS-1151 > URL: https://issues.apache.org/jira/browse/HDDS-1151 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Fix For: 0.4.0 > > Time Spent: 20m > Remaining Estimate: 0h > > The tracing propagation is not yet enabled for the ScmBlockLocationProtocol. > We can't see the internal calls between OM and SCM. We need to propagate it > at least for the allocateBlock call. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14052) RBF: Use Router keytab for WebHDFS
[
https://issues.apache.org/jira/browse/HDFS-14052?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
CR Hota updated HDFS-14052:
---
Attachment: HDFS-14052-HDFS-13891.4.patch
> RBF: Use Router keytab for WebHDFS
> --
>
> Key: HDFS-14052
> URL: https://issues.apache.org/jira/browse/HDFS-14052
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Íñigo Goiri
>Assignee: CR Hota
>Priority: Major
> Attachments: HDFS-14052-HDFS-13891.0.patch,
> HDFS-14052-HDFS-13891.1.patch, HDFS-14052-HDFS-13891.2.patch,
> HDFS-14052-HDFS-13891.3.patch, HDFS-14052-HDFS-13891.4.patch
>
>
> When the RouterHttpServer starts it does:
> {code}
> NameNodeHttpServer.initWebHdfs(conf, httpAddress.getHostName(),
> httpServer,
> RouterWebHdfsMethods.class.getPackage().getName());
> {code}
> This function is in the NN and is pretty generic.
> However, it then calls to NameNodeHttpServer#getAuthFilterParams, which does:
> {code}
> String httpKeytab = conf.get(DFSUtil.getSpnegoKeytabKey(conf,
> DFSConfigKeys.DFS_NAMENODE_KEYTAB_FILE_KEY));
> {code}
> In most cases, the regular web keytab will kick in, but we should make this a
> parameter and load the Router one just in case.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-919) Enable prometheus endpoints for Ozone datanodes
[ https://issues.apache.org/jira/browse/HDDS-919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777325#comment-16777325 ] Anu Engineer commented on HDDS-919: --- [~elek] Can you please rebase this PR when you get a chance? Thanks in advance. > Enable prometheus endpoints for Ozone datanodes > --- > > Key: HDDS-919 > URL: https://issues.apache.org/jira/browse/HDDS-919 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: newbie, pull-request-available > Time Spent: 1h 10m > Remaining Estimate: 0h > > HDDS-846 provides a new metric endpoint which publishes the available Hadoop > metrics in prometheus friendly format with a new servlet. > Unfortunately it's enabled only on the scm/om side. It would be great to > enable it in the Ozone/HDDS datanodes on the web server of the HDDS Rest > endpoint. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1151) Propagate the tracing id in ScmBlockLocationProtocol
[ https://issues.apache.org/jira/browse/HDDS-1151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Anu Engineer updated HDDS-1151: --- Resolution: Fixed Fix Version/s: 0.4.0 Status: Resolved (was: Patch Available) [~shashikant] Thank you for the review. [~elek] Thanks for the contribution. I have committed this patch to the trunk. > Propagate the tracing id in ScmBlockLocationProtocol > > > Key: HDDS-1151 > URL: https://issues.apache.org/jira/browse/HDDS-1151 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Fix For: 0.4.0 > > Time Spent: 20m > Remaining Estimate: 0h > > The tracing propagation is not yet enabled for the ScmBlockLocationProtocol. > We can't see the internal calls between OM and SCM. We need to propagate it > at least for the allocateBlock call. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-13123) RBF: Add a balancer tool to move data across subcluster
[ https://issues.apache.org/jira/browse/HDFS-13123?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777308#comment-16777308 ] Ekanth Sethuramalingam commented on HDFS-13123: --- [~elgoiri], [~ywskycn] was working on an increment over your original PoC patch. However, we did not pursue that any further. Unfortunately, we don't have anything here :( > RBF: Add a balancer tool to move data across subcluster > > > Key: HDFS-13123 > URL: https://issues.apache.org/jira/browse/HDFS-13123 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Wei Yan >Assignee: Wei Yan >Priority: Major > Attachments: HDFS Router-Based Federation Rebalancer.pdf > > > Follow the discussion in HDFS-12615. This Jira is to track effort for > building a rebalancer tool, used by router-based federation to move data > among subclusters. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14292) Introduce Java ExecutorService to DataXceiverServer
[
https://issues.apache.org/jira/browse/HDFS-14292?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777303#comment-16777303
]
Kihwal Lee commented on HDFS-14292:
---
There was also an effort to make data transfer use http (by [~wheat9]?). The
rationale was that we could use existing potentially async engines and also
take advantage of more standard encryption (TLS). It didn't go far, but I
thought it needs to be revisited.
> Introduce Java ExecutorService to DataXceiverServer
> ---
>
> Key: HDFS-14292
> URL: https://issues.apache.org/jira/browse/HDFS-14292
> Project: Hadoop HDFS
> Issue Type: Improvement
> Components: datanode
>Affects Versions: 3.2.0
>Reporter: BELUGA BEHR
>Assignee: BELUGA BEHR
>Priority: Major
> Attachments: HDFS-14292.1.patch, HDFS-14292.2.patch,
> HDFS-14292.3.patch, HDFS-14292.4.patch, HDFS-14292.5.patch,
> HDFS-14292.6.patch, HDFS-14292.6.patch, HDFS-14292.7.patch
>
>
> I wanted to investigate {{dfs.datanode.max.transfer.threads}} from
> {{hdfs-site.xml}}. It is described as "Specifies the maximum number of
> threads to use for transferring data in and out of the DN." The default
> value is 4096. I found it interesting because 4096 threads sounds like a lot
> to me. I'm not sure how a system with 8-16 cores would react to this large a
> thread count. Intuitively, I would say that the overhead of context
> switching would be immense.
> During mt investigation, I discovered the
> [following|https://github.com/apache/hadoop/blob/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataXceiverServer.java#L203-L216]
> setup in the {{DataXceiverServer}} class:
> # A peer connects to a DataNode
> # A new thread is spun up to service this connection
> # The thread runs to completion
> # The tread dies
> It would perhaps be better if we used a thread pool to better manage the
> lifecycle of the service threads and to allow the DataNode to re-use existing
> threads, saving on the need to create and spin-up threads on demand.
> In this JIRA, I have added a couple of things:
> # Added a thread pool to {{DataXceiverServer}} class that, on demand, will
> create up to {{dfs.datanode.max.transfer.threads}}. A thread that has
> completed its prior duties will stay idle for up to 60 seconds
> (configurable), it will be retired if no new work has arrived.
> # Added new methods to the {{Peer}} Interface to allow for better logging and
> less code within each Thread ({{DataXceiver}}).
> # Updated the Thread code ({{DataXceiver}}) regarding its interactions with
> {{blockReceiver}} instance variable
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-1174) Freon tests are failing with null pointer exception
[
https://issues.apache.org/jira/browse/HDDS-1174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Anu Engineer updated HDDS-1174:
---
Status: Patch Available (was: Open)
> Freon tests are failing with null pointer exception
> ---
>
> Key: HDDS-1174
> URL: https://issues.apache.org/jira/browse/HDDS-1174
> Project: Hadoop Distributed Data Store
> Issue Type: Improvement
> Components: Ozone CLI
>Affects Versions: 0.4.0
> Environment: {code:java}
> {code}
>Reporter: Shashikant Banerjee
>Assignee: Shashikant Banerjee
>Priority: Major
> Fix For: 0.4.0
>
> Attachments: HDDS-1174.001.patch
>
>
> java.lang.NullPointerException at
> org.apache.hadoop.ozone.freon.RandomKeyGenerator.init(RandomKeyGenerator.java:218)
> at
> org.apache.hadoop.ozone.freon.RandomKeyGenerator.call(RandomKeyGenerator.java:224)
> at
> org.apache.hadoop.ozone.freon.TestDataValidate.ratisTestLargeKey(TestDataValidate.java:75)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1043) Enable token based authentication for S3 api
[ https://issues.apache.org/jira/browse/HDDS-1043?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16777284#comment-16777284 ] Ajay Kumar commented on HDDS-1043: -- [~anu] thanks for review and detailed comment. I think your main concern is lack of validation. This patch doesn't validate http request. Created [HDDS-1177] to handle it separately. Will update this jira once HDDS-1177 is committed. > Enable token based authentication for S3 api > > > Key: HDDS-1043 > URL: https://issues.apache.org/jira/browse/HDDS-1043 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Ajay Kumar >Assignee: Ajay Kumar >Priority: Major > Labels: security > Fix For: 0.4.0 > > Attachments: HDDS-1043.00.patch, HDDS-1043.01.patch, > HDDS-1043.02.patch, HDDS-1043.03.patch > > > Ozone has a S3 api and mechanism to create S3 like secrets for user. This > jira proposes hadoop compatible token based authentication for S3 api which > utilizes S3 secret stored in OM. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
