[jira] [Commented] (HDFS-14648) DeadNodeDetector basic model
[
https://issues.apache.org/jira/browse/HDFS-14648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915165#comment-16915165
]
Chen Zhang commented on HDFS-14648:
---
[~leosun08], since this is a big patch, I think it would be better to upload a
design doc with detailed description and flowchart or graphs for other
reviewers to understand the patch easily, which can accelerate the review
progress.
> DeadNodeDetector basic model
>
>
> Key: HDFS-14648
> URL: https://issues.apache.org/jira/browse/HDFS-14648
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HDFS-14648.001.patch, HDFS-14648.002.patch,
> HDFS-14648.003.patch, HDFS-14648.004.patch
>
>
> This Jira constructs DeadNodeDetector state machine model. The function it
> implements as follow:
> # When a DFSInputstream is opened, a BlockReader is opened. If some DataNode
> of the block is found to inaccessible, put the DataNode into
> DeadNodeDetector#deadnode.(HDFS-14649) will optimize this part. Because when
> DataNode is not accessible, it is likely that the replica has been removed
> from the DataNode.Therefore, it needs to be confirmed by re-probing and
> requires a higher priority processing.
> # DeadNodeDetector will periodically detect the Node in
> DeadNodeDetector#deadnode, If the access is successful, the Node will be
> moved from DeadNodeDetector#deadnode. Continuous detection of the dead node
> is necessary. The DataNode need rejoin the cluster due to a service
> restart/machine repair. The DataNode may be permanently excluded if there is
> no added probe mechanism.
> # DeadNodeDetector#dfsInputStreamNodes Record the DFSInputstream using
> DataNode. When the DFSInputstream is closed, it will be moved from
> DeadNodeDetector#dfsInputStreamNodes.
> # Every time get the global deanode, update the DeadNodeDetector#deadnode.
> The new DeadNodeDetector#deadnode Equals to the intersection of the old
> DeadNodeDetector#deadnode and the Datanodes are by
> DeadNodeDetector#dfsInputStreamNodes.
> # DeadNodeDetector has a switch that is turned off by default. When it is
> closed, each DFSInputstream still uses its own local deadnode.
> # This feature has been used in the XIAOMI production environment for a long
> time. Reduced hbase read stuck, due to node hangs.
> # Just open the DeadNodeDetector switch and you can use it directly. No
> other restrictions. Don't want to use DeadNodeDetector, just close it.
> {code:java}
> if (sharedDeadNodesEnabled && deadNodeDetector == null) {
> deadNodeDetector = new DeadNodeDetector(name);
> deadNodeDetectorThr = new Daemon(deadNodeDetector);
> deadNodeDetectorThr.start();
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14648) DeadNodeDetector basic model
[
https://issues.apache.org/jira/browse/HDFS-14648?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915143#comment-16915143
]
Lisheng Sun commented on HDFS-14648:
Thank [~jojochuang] for your good suggetions. I updated the descriptions and
introducd the function and its implementation in more detail.
I have shared this feature with the MeiTuan, JingDong HDFS Team. They are
looking forward to this feature and hope to use it sooner. Thank [~jojochuang]
again.
> DeadNodeDetector basic model
>
>
> Key: HDFS-14648
> URL: https://issues.apache.org/jira/browse/HDFS-14648
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HDFS-14648.001.patch, HDFS-14648.002.patch,
> HDFS-14648.003.patch, HDFS-14648.004.patch
>
>
> This Jira constructs DeadNodeDetector state machine model. The function it
> implements as follow:
> # When a DFSInputstream is opened, a BlockReader is opened. If some DataNode
> of the block is found to inaccessible, put the DataNode into
> DeadNodeDetector#deadnode.(HDFS-14649) will optimize this part. Because when
> DataNode is not accessible, it is likely that the replica has been removed
> from the DataNode.Therefore, it needs to be confirmed by re-probing and
> requires a higher priority processing.
> # DeadNodeDetector will periodically detect the Node in
> DeadNodeDetector#deadnode, If the access is successful, the Node will be
> moved from DeadNodeDetector#deadnode. Continuous detection of the dead node
> is necessary. The DataNode need rejoin the cluster due to a service
> restart/machine repair. The DataNode may be permanently excluded if there is
> no added probe mechanism.
> # DeadNodeDetector#dfsInputStreamNodes Record the DFSInputstream using
> DataNode. When the DFSInputstream is closed, it will be moved from
> DeadNodeDetector#dfsInputStreamNodes.
> # Every time get the global deanode, update the DeadNodeDetector#deadnode.
> The new DeadNodeDetector#deadnode Equals to the intersection of the old
> DeadNodeDetector#deadnode and the Datanodes are by
> DeadNodeDetector#dfsInputStreamNodes.
> # DeadNodeDetector has a switch that is turned off by default. When it is
> closed, each DFSInputstream still uses its own local deadnode.
> # This feature has been used in the XIAOMI production environment for a long
> time. Reduced hbase read stuck, due to node hangs.
> # Just open the DeadNodeDetector switch and you can use it directly. No
> other restrictions. Don't want to use DeadNodeDetector, just close it.
> {code:java}
> if (sharedDeadNodesEnabled && deadNodeDetector == null) {
> deadNodeDetector = new DeadNodeDetector(name);
> deadNodeDetectorThr = new Daemon(deadNodeDetector);
> deadNodeDetectorThr.start();
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14648) DeadNodeDetector basic model
[
https://issues.apache.org/jira/browse/HDFS-14648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lisheng Sun updated HDFS-14648:
---
Description:
This Jira constructs DeadNodeDetector state machine model. The function it
implements as follow:
# When a DFSInputstream is opened, a BlockReader is opened. If some DataNode
of the block is found to inaccessible, put the DataNode into
DeadNodeDetector#deadnode.(HDFS-14649) will optimize this part. Because when
DataNode is not accessible, it is likely that the replica has been removed from
the DataNode.Therefore, it needs to be confirmed by re-probing and requires a
higher priority processing.
# DeadNodeDetector will periodically detect the Node in
DeadNodeDetector#deadnode, If the access is successful, the Node will be moved
from DeadNodeDetector#deadnode. Continuous detection of the dead node is
necessary. The DataNode need rejoin the cluster due to a service
restart/machine repair. The DataNode may be permanently excluded if there is no
added probe mechanism.
# DeadNodeDetector#dfsInputStreamNodes Record the DFSInputstream using
DataNode. When the DFSInputstream is closed, it will be moved from
DeadNodeDetector#dfsInputStreamNodes.
# Every time get the global deanode, update the DeadNodeDetector#deadnode. The
new DeadNodeDetector#deadnode Equals to the intersection of the old
DeadNodeDetector#deadnode and the Datanodes are by
DeadNodeDetector#dfsInputStreamNodes.
# DeadNodeDetector has a switch that is turned off by default. When it is
closed, each DFSInputstream still uses its own local deadnode.
# This feature has been used in the XIAOMI production environment for a long
time. Reduced hbase read stuck, due to node hangs.
# Just open the DeadNodeDetector switch and you can use it directly. No other
restrictions. Don't want to use DeadNodeDetector, just close it.
{code:java}
if (sharedDeadNodesEnabled && deadNodeDetector == null) {
deadNodeDetector = new DeadNodeDetector(name);
deadNodeDetectorThr = new Daemon(deadNodeDetector);
deadNodeDetectorThr.start();
}
{code}
was:
This Jira constructs DeadNodeDetector state machine model. The function it
implements as follow:
# When a DFSInputstream is opened, a BlockReader is opened. If some DataNode
of the block is found to inaccessible, put the DataNode into
DeadNodeDetector#deadnode.(HDFS-14649) will optimize this part. Because when
DataNode is not accessible, it is likely that the replica has been removed from
the DataNode.Therefore, it needs to be confirmed by re-probing and requires a
higher priority processing.
# DeadNodeDetector will periodically detect the Node in
DeadNodeDetector#deadnode, If the access is successful, the Node will be moved
from DeadNodeDetector#deadnode. Continuous detection of the dead node is
necessary. The DataNode need rejoin the cluster due to a service
restart/machine repair. The DataNode may be permanently excluded if there is no
added probe mechanism.
# DeadNodeDetector#dfsInputStreamNodes Record the DFSInputstream using
DataNode. When the DFSInputstream is closed, it will be moved from
DeadNodeDetector#dfsInputStreamNodes.
# Every time get the global deanode, update the DeadNodeDetector#deadnode. The
new DeadNodeDetector#deadnode Equals to the intersection of the old
DeadNodeDetector#deadnode and the Datanodes are by
DeadNodeDetector#dfsInputStreamNodes.
# DeadNodeDetector has a switch that is turned off by default. When it is
closed, each DFSInputstream still uses its own local deadnode.
#
{code:java}
if (sharedDeadNodesEnabled && deadNodeDetector == null) {
deadNodeDetector = new DeadNodeDetector(name);
deadNodeDetectorThr = new Daemon(deadNodeDetector);
deadNodeDetectorThr.start();
}
{code}
> DeadNodeDetector basic model
>
>
> Key: HDFS-14648
> URL: https://issues.apache.org/jira/browse/HDFS-14648
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HDFS-14648.001.patch, HDFS-14648.002.patch,
> HDFS-14648.003.patch, HDFS-14648.004.patch
>
>
> This Jira constructs DeadNodeDetector state machine model. The function it
> implements as follow:
> # When a DFSInputstream is opened, a BlockReader is opened. If some DataNode
> of the block is found to inaccessible, put the DataNode into
> DeadNodeDetector#deadnode.(HDFS-14649) will optimize this part. Because when
> DataNode is not accessible, it is likely that the replica has been removed
> from the DataNode.Therefore, it needs to be confirmed by re-probing and
> requires a higher priority processing.
> # DeadNodeDetector will periodically detect the Node in
> DeadNodeDetector#deadnode, If the access is successful, the Node will be
> moved from DeadNodeDetector#deadnode. Continuous detection of the
[jira] [Updated] (HDFS-14648) DeadNodeDetector basic model
[
https://issues.apache.org/jira/browse/HDFS-14648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lisheng Sun updated HDFS-14648:
---
Description:
This Jira constructs DeadNodeDetector state machine model. The function it
implements as follow:
# When a DFSInputstream is opened, a BlockReader is opened. If some DataNode
of the block is found to inaccessible, put the DataNode into
DeadNodeDetector#deadnode. HDFS-14649 will optimize this part. Because when
DataNode is not accessible, it is likely that the replica has been removed from
the DataNode.Therefore, it needs to be confirmed by re-probing and requires a
higher priority processing.
# DeadNodeDetector will periodically detect the Node in
DeadNodeDetector#deadnode, If the access is successful, the Node will be moved
from DeadNodeDetector#deadnode. Continuous detection of the dead node is
necessary. The DataNode need rejoin the cluster due to a service
restart/machine repair. The DataNode may be permanently excluded if there is no
added probe mechanism.
# DeadNodeDetector#dfsInputStreamNodes Record the DFSInputstream using
DataNode. When the DFSInputstream is closed, it will be moved from
DeadNodeDetector#dfsInputStreamNodes.
# Every time get the global deanode, update the DeadNodeDetector#deadnode. The
new DeadNodeDetector#deadnode Equals to the intersection of the old
DeadNodeDetector#deadnode and the Datanodes are by
DeadNodeDetector#dfsInputStreamNodes.
# A DeadNodeDetector that belongs to a DFSClient has a switch that is turned
off by default. When it is closed, each DFSInputstream still uses its own local
deadnode.
# This feature has been used in the XIAOMI production environment for a long
time. Reduced hbase read stuck, due to node hangs.
# Just open the DeadNodeDetector switch and you can use it directly. No other
restrictions.
{code:java}
String DFS_CLIENT_DEAD_NODE_DETECT_ENABLED_KEY =
"dfs.client.deadnode.detect.enabled";
boolean DFS_CLIENT_DEAD_NODE_DETECT_ENABLED_DEFAULT = false;
if (sharedDeadNodesEnabled && deadNodeDetector == null) {
deadNodeDetector = new DeadNodeDetector(name);
deadNodeDetectorThr = new Daemon(deadNodeDetector);
deadNodeDetectorThr.start();
}
{code}
was:
This Jira constructs DeadNodeDetector state machine model. The function it
implements as follow:
# After DFSInputstream detects some DataNode die, it put in DeadNodeDetector
and share this information to others in the same DFSClient. The ohter
DFSInputstreams will not read this DataNode.
# DeadNodeDetector also have DFSInputstream reference relationships to each
DataNode. When DFSInputstream close, DeadNodeDetector also remove this
reference. If some DeadNode of DeadNodeDetector is not read by DFSInputstream,
it also is removed from DeadNodeDetector.
> DeadNodeDetector basic model
>
>
> Key: HDFS-14648
> URL: https://issues.apache.org/jira/browse/HDFS-14648
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HDFS-14648.001.patch, HDFS-14648.002.patch,
> HDFS-14648.003.patch, HDFS-14648.004.patch
>
>
> This Jira constructs DeadNodeDetector state machine model. The function it
> implements as follow:
> # When a DFSInputstream is opened, a BlockReader is opened. If some DataNode
> of the block is found to inaccessible, put the DataNode into
> DeadNodeDetector#deadnode. HDFS-14649 will optimize this part. Because when
> DataNode is not accessible, it is likely that the replica has been removed
> from the DataNode.Therefore, it needs to be confirmed by re-probing and
> requires a higher priority processing.
> # DeadNodeDetector will periodically detect the Node in
> DeadNodeDetector#deadnode, If the access is successful, the Node will be
> moved from DeadNodeDetector#deadnode. Continuous detection of the dead node
> is necessary. The DataNode need rejoin the cluster due to a service
> restart/machine repair. The DataNode may be permanently excluded if there is
> no added probe mechanism.
> # DeadNodeDetector#dfsInputStreamNodes Record the DFSInputstream using
> DataNode. When the DFSInputstream is closed, it will be moved from
> DeadNodeDetector#dfsInputStreamNodes.
> # Every time get the global deanode, update the DeadNodeDetector#deadnode.
> The new DeadNodeDetector#deadnode Equals to the intersection of the old
> DeadNodeDetector#deadnode and the Datanodes are by
> DeadNodeDetector#dfsInputStreamNodes.
> # A DeadNodeDetector that belongs to a DFSClient has a switch that is turned
> off by default. When it is closed, each DFSInputstream still uses its own
> local deadnode.
> # This feature has been used in the XIAOMI production environment for a long
> time. Reduced hbase read stuck, due to node hangs.
>
[jira] [Updated] (HDFS-14648) DeadNodeDetector basic model
[
https://issues.apache.org/jira/browse/HDFS-14648?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lisheng Sun updated HDFS-14648:
---
Description:
This Jira constructs DeadNodeDetector state machine model. The function it
implements as follow:
# When a DFSInputstream is opened, a BlockReader is opened. If some DataNode
of the block is found to inaccessible, put the DataNode into
DeadNodeDetector#deadnode.(HDFS-14649) will optimize this part. Because when
DataNode is not accessible, it is likely that the replica has been removed from
the DataNode.Therefore, it needs to be confirmed by re-probing and requires a
higher priority processing.
# DeadNodeDetector will periodically detect the Node in
DeadNodeDetector#deadnode, If the access is successful, the Node will be moved
from DeadNodeDetector#deadnode. Continuous detection of the dead node is
necessary. The DataNode need rejoin the cluster due to a service
restart/machine repair. The DataNode may be permanently excluded if there is no
added probe mechanism.
# DeadNodeDetector#dfsInputStreamNodes Record the DFSInputstream using
DataNode. When the DFSInputstream is closed, it will be moved from
DeadNodeDetector#dfsInputStreamNodes.
# Every time get the global deanode, update the DeadNodeDetector#deadnode. The
new DeadNodeDetector#deadnode Equals to the intersection of the old
DeadNodeDetector#deadnode and the Datanodes are by
DeadNodeDetector#dfsInputStreamNodes.
# DeadNodeDetector has a switch that is turned off by default. When it is
closed, each DFSInputstream still uses its own local deadnode.
#
{code:java}
if (sharedDeadNodesEnabled && deadNodeDetector == null) {
deadNodeDetector = new DeadNodeDetector(name);
deadNodeDetectorThr = new Daemon(deadNodeDetector);
deadNodeDetectorThr.start();
}
{code}
was:
This Jira constructs DeadNodeDetector state machine model. The function it
implements as follow:
# After DFSInputstream detects some DataNode die, it put in DeadNodeDetector
and share this information to others in the same DFSClient. The ohter
DFSInputstreams will not read this DataNode.
# DeadNodeDetector also have DFSInputstream reference relationships to each
DataNode. When DFSInputstream close, DeadNodeDetector also remove this
reference. If some DeadNode of DeadNodeDetector is not read by DFSInputstream,
it also is removed from DeadNodeDetector.
> DeadNodeDetector basic model
>
>
> Key: HDFS-14648
> URL: https://issues.apache.org/jira/browse/HDFS-14648
> Project: Hadoop HDFS
> Issue Type: Sub-task
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HDFS-14648.001.patch, HDFS-14648.002.patch,
> HDFS-14648.003.patch, HDFS-14648.004.patch
>
>
> This Jira constructs DeadNodeDetector state machine model. The function it
> implements as follow:
> # When a DFSInputstream is opened, a BlockReader is opened. If some DataNode
> of the block is found to inaccessible, put the DataNode into
> DeadNodeDetector#deadnode.(HDFS-14649) will optimize this part. Because when
> DataNode is not accessible, it is likely that the replica has been removed
> from the DataNode.Therefore, it needs to be confirmed by re-probing and
> requires a higher priority processing.
> # DeadNodeDetector will periodically detect the Node in
> DeadNodeDetector#deadnode, If the access is successful, the Node will be
> moved from DeadNodeDetector#deadnode. Continuous detection of the dead node
> is necessary. The DataNode need rejoin the cluster due to a service
> restart/machine repair. The DataNode may be permanently excluded if there is
> no added probe mechanism.
> # DeadNodeDetector#dfsInputStreamNodes Record the DFSInputstream using
> DataNode. When the DFSInputstream is closed, it will be moved from
> DeadNodeDetector#dfsInputStreamNodes.
> # Every time get the global deanode, update the DeadNodeDetector#deadnode.
> The new DeadNodeDetector#deadnode Equals to the intersection of the old
> DeadNodeDetector#deadnode and the Datanodes are by
> DeadNodeDetector#dfsInputStreamNodes.
> # DeadNodeDetector has a switch that is turned off by default. When it is
> closed, each DFSInputstream still uses its own local deadnode.
> #
> {code:java}
> if (sharedDeadNodesEnabled && deadNodeDetector == null) {
> deadNodeDetector = new DeadNodeDetector(name);
> deadNodeDetectorThr = new Daemon(deadNodeDetector);
> deadNodeDetectorThr.start();
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-6524) Cdecides retry times considering with block replica number
[ https://issues.apache.org/jira/browse/HDFS-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915127#comment-16915127 ] Lisheng Sun commented on HDFS-6524: --- hi [~elgoiri] [~xkrogen] Could you help take a review for this patch? Thank you. > Cdecides retry times considering with block replica number > -- > > Key: HDFS-6524 > URL: https://issues.apache.org/jira/browse/HDFS-6524 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 3.0.0-alpha1 >Reporter: Liang Xie >Assignee: Lisheng Sun >Priority: Minor > Labels: BB2015-05-TBR > Attachments: HDFS-6524.001.patch, HDFS-6524.txt > > > Currently the chooseDataNode() does retry with the setting: > dfsClientConf.maxBlockAcquireFailures, which by default is 3 > (DFS_CLIENT_MAX_BLOCK_ACQUIRE_FAILURES_DEFAULT = 3), it would be better > having another option, block replication factor. One cluster with only two > block replica setting, or using Reed-solomon encoding solution with one > replica factor. It helps to reduce the long tail latency. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-6524) Cdecides retry times considering with block replica number
[ https://issues.apache.org/jira/browse/HDFS-6524?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lisheng Sun updated HDFS-6524: -- Summary: Cdecides retry times considering with block replica number (was: chooseDataNode decides retry times considering with block replica number) > Cdecides retry times considering with block replica number > -- > > Key: HDFS-6524 > URL: https://issues.apache.org/jira/browse/HDFS-6524 > Project: Hadoop HDFS > Issue Type: Improvement > Components: hdfs-client >Affects Versions: 3.0.0-alpha1 >Reporter: Liang Xie >Assignee: Lisheng Sun >Priority: Minor > Labels: BB2015-05-TBR > Attachments: HDFS-6524.001.patch, HDFS-6524.txt > > > Currently the chooseDataNode() does retry with the setting: > dfsClientConf.maxBlockAcquireFailures, which by default is 3 > (DFS_CLIENT_MAX_BLOCK_ACQUIRE_FAILURES_DEFAULT = 3), it would be better > having another option, block replication factor. One cluster with only two > block replica setting, or using Reed-solomon encoding solution with one > replica factor. It helps to reduce the long tail latency. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Assigned] (HDFS-12904) Add DataTransferThrottler to the Datanode transfers
[
https://issues.apache.org/jira/browse/HDFS-12904?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lisheng Sun reassigned HDFS-12904:
--
Assignee: Lisheng Sun (was: Íñigo Goiri)
> Add DataTransferThrottler to the Datanode transfers
> ---
>
> Key: HDFS-12904
> URL: https://issues.apache.org/jira/browse/HDFS-12904
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: datanode
>Reporter: Íñigo Goiri
>Assignee: Lisheng Sun
>Priority: Minor
> Attachments: HDFS-12904.000.patch, HDFS-12904.001.patch,
> HDFS-12904.002.patch
>
>
> The {{DataXceiverServer}} already uses throttling for the balancing. The
> Datanode should also allow throttling the regular data transfers.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14769) Add throttler for DataNode#DataTransfer
[
https://issues.apache.org/jira/browse/HDFS-14769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915124#comment-16915124
]
Lisheng Sun commented on HDFS-14769:
[~elgoiri] I would like do as your suggestion.
marked this patch duplicated and asigned HDFS-12904 to me.
> Add throttler for DataNode#DataTransfer
> ---
>
> Key: HDFS-14769
> URL: https://issues.apache.org/jira/browse/HDFS-14769
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HDFS-14769.001.patch, HDFS-14769.002.patch,
> HDFS-14769.003.patch
>
>
> When a amount of block replicas are moving, the bandwidth will be full
> without throtter in DataNode#DataTransfer.
> This will affect normal reading and writing.
> {code:java}
> /**
> * Do the deed, write the bytes
> */
> @Override
> public void run() {
> // send data & checksum
> blockSender.sendBlock(out, unbufOut, null);
> }{code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14769) Add throttler for DataNode#DataTransfer
[
https://issues.apache.org/jira/browse/HDFS-14769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lisheng Sun updated HDFS-14769:
---
Resolution: Duplicate
Status: Resolved (was: Patch Available)
> Add throttler for DataNode#DataTransfer
> ---
>
> Key: HDFS-14769
> URL: https://issues.apache.org/jira/browse/HDFS-14769
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HDFS-14769.001.patch, HDFS-14769.002.patch,
> HDFS-14769.003.patch
>
>
> When a amount of block replicas are moving, the bandwidth will be full
> without throtter in DataNode#DataTransfer.
> This will affect normal reading and writing.
> {code:java}
> /**
> * Do the deed, write the bytes
> */
> @Override
> public void run() {
> // send data & checksum
> blockSender.sendBlock(out, unbufOut, null);
> }{code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1937) Acceptance tests fail if scm webui shows invalid json
[
https://issues.apache.org/jira/browse/HDDS-1937?focusedWorklogId=300771=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-300771
]
ASF GitHub Bot logged work on HDDS-1937:
Author: ASF GitHub Bot
Created on: 24/Aug/19 22:53
Start Date: 24/Aug/19 22:53
Worklog Time Spent: 10m
Work Description: hadoop-yetus commented on issue #1256: HDDS-1937.
Acceptance tests fail if scm webui shows invalid json
URL: https://github.com/apache/hadoop/pull/1256#issuecomment-524586378
:broken_heart: **-1 overall**
| Vote | Subsystem | Runtime | Comment |
|::|--:|:|:|
| 0 | reexec | 134 | Docker mode activated. |
||| _ Prechecks _ |
| +1 | dupname | 0 | No case conflicting files found. |
| 0 | shelldocs | 0 | Shelldocs was not available. |
| +1 | @author | 0 | The patch does not contain any @author tags. |
| -1 | test4tests | 0 | The patch doesn't appear to include any new or
modified tests. Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch. |
||| _ trunk Compile Tests _ |
| +1 | mvninstall | 847 | trunk passed |
| +1 | mvnsite | 0 | trunk passed |
| +1 | shadedclient | 1090 | branch has no errors when building and testing
our client artifacts. |
||| _ Patch Compile Tests _ |
| +1 | mvninstall | 598 | the patch passed |
| +1 | mvnsite | 0 | the patch passed |
| +1 | shellcheck | 1 | There were no new shellcheck issues. |
| +1 | whitespace | 0 | The patch has no whitespace issues. |
| +1 | shadedclient | 764 | patch has no errors when building and testing
our client artifacts. |
||| _ Other Tests _ |
| +1 | unit | 128 | hadoop-hdds in the patch passed. |
| +1 | unit | 327 | hadoop-ozone in the patch passed. |
| +1 | asflicense | 48 | The patch does not generate ASF License warnings. |
| | | 4171 | |
| Subsystem | Report/Notes |
|--:|:-|
| Docker | Client=19.03.1 Server=19.03.1 base:
https://builds.apache.org/job/hadoop-multibranch/job/PR-1256/6/artifact/out/Dockerfile
|
| GITHUB PR | https://github.com/apache/hadoop/pull/1256 |
| Optional Tests | dupname asflicense mvnsite unit shellcheck shelldocs |
| uname | Linux 9d874a11e5ed 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24
10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | personality/hadoop.sh |
| git revision | trunk / d2225c8 |
| Test Results |
https://builds.apache.org/job/hadoop-multibranch/job/PR-1256/6/testReport/ |
| Max. process+thread count | 305 (vs. ulimit of 5500) |
| modules | C: hadoop-ozone/dist U: hadoop-ozone/dist |
| Console output |
https://builds.apache.org/job/hadoop-multibranch/job/PR-1256/6/console |
| versions | git=2.7.4 maven=3.3.9 shellcheck=0.4.6 |
| Powered by | Apache Yetus 0.10.0 http://yetus.apache.org |
This message was automatically generated.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 300771)
Time Spent: 2h (was: 1h 50m)
> Acceptance tests fail if scm webui shows invalid json
> -
>
> Key: HDDS-1937
> URL: https://issues.apache.org/jira/browse/HDDS-1937
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 2h
> Remaining Estimate: 0h
>
> Acceptance test of a nightly build is failed with the following error:
> {code}
> Creating ozonesecure_datanode_3 ...
> [7A[2K
> Creating ozonesecure_kdc_1 ... [32mdone[0m
> [7B[6A[2K
> Creating ozonesecure_om_1 ... [32mdone[0m
> [6B[8A[2K
> Creating ozonesecure_scm_1 ... [32mdone[0m
> [8B[1A[2K
> Creating ozonesecure_datanode_3 ... [32mdone[0m
> [1B[5A[2K
> Creating ozonesecure_kms_1 ... [32mdone[0m
> [5B[4A[2K
> Creating ozonesecure_s3g_1 ... [32mdone[0m
> [4B[2A[2K
> Creating ozonesecure_datanode_2 ... [32mdone[0m
> [2B[3A[2K
> Creating ozonesecure_datanode_1 ... [32mdone[0m
> [3Bparse error: Invalid numeric literal at line 2, column 0
> {code}
> https://raw.githubusercontent.com/elek/ozone-ci/master/byscane/byscane-nightly-5b87q/acceptance/output.log
> The problem is in the script which checks the number of available datanodes.
> If the HTTP endpoint
[jira] [Updated] (HDDS-2030) Generate simplifed reports by the dev-support/checks/*.sh scripts
[ https://issues.apache.org/jira/browse/HDDS-2030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Elek, Marton updated HDDS-2030: --- Status: Patch Available (was: Open) > Generate simplifed reports by the dev-support/checks/*.sh scripts > - > > Key: HDDS-2030 > URL: https://issues.apache.org/jira/browse/HDDS-2030 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: build >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > hadoop-ozone/dev-support/checks directory contains shell scripts to execute > different type of code checks (findbugs, checkstyle, etc.) > Currently the contract is very simple. Every shell script executes one (and > only one) check and the shell response code is set according to the result > (non-zero code if failed). > To have better reporting in the github pr build, it would be great to improve > the scripts to generate simple summary files and save the relevant files for > archiving. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2030) Generate simplifed reports by the dev-support/checks/*.sh scripts
[ https://issues.apache.org/jira/browse/HDDS-2030?focusedWorklogId=300762=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-300762 ] ASF GitHub Bot logged work on HDDS-2030: Author: ASF GitHub Bot Created on: 24/Aug/19 21:58 Start Date: 24/Aug/19 21:58 Worklog Time Spent: 10m Work Description: elek commented on pull request #1348: HDDS-2030. Generate simplifed reports by the dev-support/checks/*.sh scripts URL: https://github.com/apache/hadoop/pull/1348 hadoop-ozone/dev-support/checks directory contains shell scripts to execute different type of code checks (findbugs, checkstyle, etc.) Currently the contract is very simple. Every shell script executes one (and only one) check and the shell response code is set according to the result (non-zero code if failed). To have better reporting in the github pr build, it would be great to improve the scripts to generate simple summary files and save the relevant files for archiving. See: https://issues.apache.org/jira/browse/HDDS-2030 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 300762) Remaining Estimate: 0h Time Spent: 10m > Generate simplifed reports by the dev-support/checks/*.sh scripts > - > > Key: HDDS-2030 > URL: https://issues.apache.org/jira/browse/HDDS-2030 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: build >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > Time Spent: 10m > Remaining Estimate: 0h > > hadoop-ozone/dev-support/checks directory contains shell scripts to execute > different type of code checks (findbugs, checkstyle, etc.) > Currently the contract is very simple. Every shell script executes one (and > only one) check and the shell response code is set according to the result > (non-zero code if failed). > To have better reporting in the github pr build, it would be great to improve > the scripts to generate simple summary files and save the relevant files for > archiving. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2030) Generate simplifed reports by the dev-support/checks/*.sh scripts
[ https://issues.apache.org/jira/browse/HDDS-2030?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] ASF GitHub Bot updated HDDS-2030: - Labels: pull-request-available (was: ) > Generate simplifed reports by the dev-support/checks/*.sh scripts > - > > Key: HDDS-2030 > URL: https://issues.apache.org/jira/browse/HDDS-2030 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: build >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > Labels: pull-request-available > > hadoop-ozone/dev-support/checks directory contains shell scripts to execute > different type of code checks (findbugs, checkstyle, etc.) > Currently the contract is very simple. Every shell script executes one (and > only one) check and the shell response code is set according to the result > (non-zero code if failed). > To have better reporting in the github pr build, it would be great to improve > the scripts to generate simple summary files and save the relevant files for > archiving. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2030) Generate simplifed reports by the dev-support/checks/*.sh scripts
Elek, Marton created HDDS-2030: -- Summary: Generate simplifed reports by the dev-support/checks/*.sh scripts Key: HDDS-2030 URL: https://issues.apache.org/jira/browse/HDDS-2030 Project: Hadoop Distributed Data Store Issue Type: Improvement Components: build Reporter: Elek, Marton Assignee: Elek, Marton hadoop-ozone/dev-support/checks directory contains shell scripts to execute different type of code checks (findbugs, checkstyle, etc.) Currently the contract is very simple. Every shell script executes one (and only one) check and the shell response code is set according to the result (non-zero code if failed). To have better reporting in the github pr build, it would be great to improve the scripts to generate simple summary files and save the relevant files for archiving. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14728) RBF: GetDatanodeReport causes a large GC pressure on the NameNodes
[ https://issues.apache.org/jira/browse/HDFS-14728?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915101#comment-16915101 ] Wei-Chiu Chuang commented on HDFS-14728: How often does *GetDatanodeReport* get called on each NameNode? The default cache timeout is 10 seconds. Is that sufficient? > RBF: GetDatanodeReport causes a large GC pressure on the NameNodes > -- > > Key: HDFS-14728 > URL: https://issues.apache.org/jira/browse/HDFS-14728 > Project: Hadoop HDFS > Issue Type: Improvement > Components: rbf >Reporter: xuzq >Assignee: xuzq >Priority: Major > Attachments: HDFS-14728-trunk-001.patch, HDFS-14728-trunk-002.patch, > HDFS-14728-trunk-003.patch, HDFS-14728-trunk-004.patch, > HDFS-14728-trunk-005.patch, HDFS-14728-trunk-006.patch, > HDFS-14728-trunk-007.patch, HDFS-14728-trunk-008.patch, > HDFS-14728-trunk-009.patch > > > When a cluster contains millions of DNs, *GetDatanodeReport* is pretty > expensive, and it will cause a large GC pressure on NameNode. > When multiple NSs share the millions DNs by federation and the router listens > to the NSs, the problem will be more serious. > All the NSs will be GC at the same time. > RBF should cache the datanode report informations and have an option to > disable the cache. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDFS-14774) Improve RouterWebhdfsMethods#chooseDatanode() error handling
Wei-Chiu Chuang created HDFS-14774:
--
Summary: Improve RouterWebhdfsMethods#chooseDatanode() error
handling
Key: HDFS-14774
URL: https://issues.apache.org/jira/browse/HDFS-14774
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Wei-Chiu Chuang
HDFS-13972 added the following code:
{code}
try {
dns = rpcServer.getDatanodeReport(DatanodeReportType.LIVE);
} catch (IOException e) {
LOG.error("Cannot get the datanodes from the RPC server", e);
} finally {
// Reset ugi to remote user for remaining operations.
RouterRpcServer.resetCurrentUser();
}
HashSet excludes = new HashSet();
if (excludeDatanodes != null) {
Collection collection =
getTrimmedStringCollection(excludeDatanodes);
for (DatanodeInfo dn : dns) {
if (collection.contains(dn.getName())) {
excludes.add(dn);
}
}
}
{code}
If {{rpcServer.getDatanodeReport()}} throws an exception, {{dns}} will become
null. This does't look like the best way to handle the exception. Should router
retry upon exception? Does it perform retry automatically under the hood?
[~crh] [~brahmareddy]
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1937) Acceptance tests fail if scm webui shows invalid json
[
https://issues.apache.org/jira/browse/HDDS-1937?focusedWorklogId=300755=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-300755
]
ASF GitHub Bot logged work on HDDS-1937:
Author: ASF GitHub Bot
Created on: 24/Aug/19 20:18
Start Date: 24/Aug/19 20:18
Worklog Time Spent: 10m
Work Description: elek commented on issue #1256: HDDS-1937. Acceptance
tests fail if scm webui shows invalid json
URL: https://github.com/apache/hadoop/pull/1256#issuecomment-524577702
/retest
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 300755)
Time Spent: 1h 50m (was: 1h 40m)
> Acceptance tests fail if scm webui shows invalid json
> -
>
> Key: HDDS-1937
> URL: https://issues.apache.org/jira/browse/HDDS-1937
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 50m
> Remaining Estimate: 0h
>
> Acceptance test of a nightly build is failed with the following error:
> {code}
> Creating ozonesecure_datanode_3 ...
> [7A[2K
> Creating ozonesecure_kdc_1 ... [32mdone[0m
> [7B[6A[2K
> Creating ozonesecure_om_1 ... [32mdone[0m
> [6B[8A[2K
> Creating ozonesecure_scm_1 ... [32mdone[0m
> [8B[1A[2K
> Creating ozonesecure_datanode_3 ... [32mdone[0m
> [1B[5A[2K
> Creating ozonesecure_kms_1 ... [32mdone[0m
> [5B[4A[2K
> Creating ozonesecure_s3g_1 ... [32mdone[0m
> [4B[2A[2K
> Creating ozonesecure_datanode_2 ... [32mdone[0m
> [2B[3A[2K
> Creating ozonesecure_datanode_1 ... [32mdone[0m
> [3Bparse error: Invalid numeric literal at line 2, column 0
> {code}
> https://raw.githubusercontent.com/elek/ozone-ci/master/byscane/byscane-nightly-5b87q/acceptance/output.log
> The problem is in the script which checks the number of available datanodes.
> If the HTTP endpoint of the SCM is already started BUT not ready yet it may
> return with a simple HTML error message instead of json. Which can not be
> parsed by jq:
> In testlib.sh:
> {code}
> 37 │ if [[ "${SECURITY_ENABLED}" == 'true' ]]; then
> 38 │ docker-compose -f "${compose_file}" exec -T scm bash -c "kinit
> -k HTTP/scm@EXAMPL
>│ E.COM -t /etc/security/keytabs/HTTP.keytab && curl --negotiate -u :
> -s '${jmx_url}'"
> 39 │ else
> 40 │ docker-compose -f "${compose_file}" exec -T scm curl -s
> "${jmx_url}"
> 41 │ fi \
> 42 │ | jq -r '.beans[0].NodeCount[] | select(.key=="HEALTHY") |
> .value'
> {code}
> One possible fix is to adjust the error handling (set +x / set -x) per method
> instead of using a generic set -x at the beginning. It would provide a more
> predictable behavior. In our case count_datanode should not fail evert (as
> the caller method: wait_for_datanodes can retry anyway).
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1937) Acceptance tests fail if scm webui shows invalid json
[
https://issues.apache.org/jira/browse/HDDS-1937?focusedWorklogId=300749=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-300749
]
ASF GitHub Bot logged work on HDDS-1937:
Author: ASF GitHub Bot
Created on: 24/Aug/19 20:02
Start Date: 24/Aug/19 20:02
Worklog Time Spent: 10m
Work Description: elek commented on issue #1256: HDDS-1937. Acceptance
tests fail if scm webui shows invalid json
URL: https://github.com/apache/hadoop/pull/1256#issuecomment-524576824
/retest
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 300749)
Time Spent: 1h 40m (was: 1.5h)
> Acceptance tests fail if scm webui shows invalid json
> -
>
> Key: HDDS-1937
> URL: https://issues.apache.org/jira/browse/HDDS-1937
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1h 40m
> Remaining Estimate: 0h
>
> Acceptance test of a nightly build is failed with the following error:
> {code}
> Creating ozonesecure_datanode_3 ...
> [7A[2K
> Creating ozonesecure_kdc_1 ... [32mdone[0m
> [7B[6A[2K
> Creating ozonesecure_om_1 ... [32mdone[0m
> [6B[8A[2K
> Creating ozonesecure_scm_1 ... [32mdone[0m
> [8B[1A[2K
> Creating ozonesecure_datanode_3 ... [32mdone[0m
> [1B[5A[2K
> Creating ozonesecure_kms_1 ... [32mdone[0m
> [5B[4A[2K
> Creating ozonesecure_s3g_1 ... [32mdone[0m
> [4B[2A[2K
> Creating ozonesecure_datanode_2 ... [32mdone[0m
> [2B[3A[2K
> Creating ozonesecure_datanode_1 ... [32mdone[0m
> [3Bparse error: Invalid numeric literal at line 2, column 0
> {code}
> https://raw.githubusercontent.com/elek/ozone-ci/master/byscane/byscane-nightly-5b87q/acceptance/output.log
> The problem is in the script which checks the number of available datanodes.
> If the HTTP endpoint of the SCM is already started BUT not ready yet it may
> return with a simple HTML error message instead of json. Which can not be
> parsed by jq:
> In testlib.sh:
> {code}
> 37 │ if [[ "${SECURITY_ENABLED}" == 'true' ]]; then
> 38 │ docker-compose -f "${compose_file}" exec -T scm bash -c "kinit
> -k HTTP/scm@EXAMPL
>│ E.COM -t /etc/security/keytabs/HTTP.keytab && curl --negotiate -u :
> -s '${jmx_url}'"
> 39 │ else
> 40 │ docker-compose -f "${compose_file}" exec -T scm curl -s
> "${jmx_url}"
> 41 │ fi \
> 42 │ | jq -r '.beans[0].NodeCount[] | select(.key=="HEALTHY") |
> .value'
> {code}
> One possible fix is to adjust the error handling (set +x / set -x) per method
> instead of using a generic set -x at the beginning. It would provide a more
> predictable behavior. In our case count_datanode should not fail evert (as
> the caller method: wait_for_datanodes can retry anyway).
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1937) Acceptance tests fail if scm webui shows invalid json
[
https://issues.apache.org/jira/browse/HDDS-1937?focusedWorklogId=300745=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-300745
]
ASF GitHub Bot logged work on HDDS-1937:
Author: ASF GitHub Bot
Created on: 24/Aug/19 19:20
Start Date: 24/Aug/19 19:20
Worklog Time Spent: 10m
Work Description: elek commented on issue #1256: HDDS-1937. Acceptance
tests fail if scm webui shows invalid json
URL: https://github.com/apache/hadoop/pull/1256#issuecomment-524574560
Ok. Let's fix this problem with the suggested `|| true` to quickly fix the
problem and discuss further how the `set -x` usage can be used safely...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 300745)
Time Spent: 1.5h (was: 1h 20m)
> Acceptance tests fail if scm webui shows invalid json
> -
>
> Key: HDDS-1937
> URL: https://issues.apache.org/jira/browse/HDDS-1937
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Elek, Marton
>Assignee: Elek, Marton
>Priority: Major
> Labels: pull-request-available
> Time Spent: 1.5h
> Remaining Estimate: 0h
>
> Acceptance test of a nightly build is failed with the following error:
> {code}
> Creating ozonesecure_datanode_3 ...
> [7A[2K
> Creating ozonesecure_kdc_1 ... [32mdone[0m
> [7B[6A[2K
> Creating ozonesecure_om_1 ... [32mdone[0m
> [6B[8A[2K
> Creating ozonesecure_scm_1 ... [32mdone[0m
> [8B[1A[2K
> Creating ozonesecure_datanode_3 ... [32mdone[0m
> [1B[5A[2K
> Creating ozonesecure_kms_1 ... [32mdone[0m
> [5B[4A[2K
> Creating ozonesecure_s3g_1 ... [32mdone[0m
> [4B[2A[2K
> Creating ozonesecure_datanode_2 ... [32mdone[0m
> [2B[3A[2K
> Creating ozonesecure_datanode_1 ... [32mdone[0m
> [3Bparse error: Invalid numeric literal at line 2, column 0
> {code}
> https://raw.githubusercontent.com/elek/ozone-ci/master/byscane/byscane-nightly-5b87q/acceptance/output.log
> The problem is in the script which checks the number of available datanodes.
> If the HTTP endpoint of the SCM is already started BUT not ready yet it may
> return with a simple HTML error message instead of json. Which can not be
> parsed by jq:
> In testlib.sh:
> {code}
> 37 │ if [[ "${SECURITY_ENABLED}" == 'true' ]]; then
> 38 │ docker-compose -f "${compose_file}" exec -T scm bash -c "kinit
> -k HTTP/scm@EXAMPL
>│ E.COM -t /etc/security/keytabs/HTTP.keytab && curl --negotiate -u :
> -s '${jmx_url}'"
> 39 │ else
> 40 │ docker-compose -f "${compose_file}" exec -T scm curl -s
> "${jmx_url}"
> 41 │ fi \
> 42 │ | jq -r '.beans[0].NodeCount[] | select(.key=="HEALTHY") |
> .value'
> {code}
> One possible fix is to adjust the error handling (set +x / set -x) per method
> instead of using a generic set -x at the beginning. It would provide a more
> predictable behavior. In our case count_datanode should not fail evert (as
> the caller method: wait_for_datanodes can retry anyway).
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14633) The StorageType quota and consume in QuotaFeature is not handled when rename and setStoragePolicy etc.
[
https://issues.apache.org/jira/browse/HDFS-14633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915071#comment-16915071
]
Hadoop QA commented on HDFS-14633:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
45s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 22m
12s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
10s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
45s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
9s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 46s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
33s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
55s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
16s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 1m
5s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
44s{color} | {color:green} hadoop-hdfs-project/hadoop-hdfs: The patch generated
0 new + 101 unchanged - 1 fixed = 101 total (was 102) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
10s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 24s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
27s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red}138m 25s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
39s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}201m 21s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.datanode.TestDataNodeUUID |
| | hadoop.hdfs.server.namenode.TestReconstructStripedBlocks |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapshotDiffReport |
| | hadoop.hdfs.server.namenode.ha.TestDFSUpgradeWithHA |
| | hadoop.hdfs.TestDFSInputStream |
| | hadoop.hdfs.TestRollingUpgrade |
| | hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots |
| | hadoop.hdfs.server.namenode.snapshot.TestSnapRootDescendantDiff |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-14633 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12978495/HDFS-14633.006.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 3402dd040c1b 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2
17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / d2225c8 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| unit |
[jira] [Commented] (HDFS-14769) Add throttler for DataNode#DataTransfer
[
https://issues.apache.org/jira/browse/HDFS-14769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16915045#comment-16915045
]
Íñigo Goiri commented on HDFS-14769:
The other JIRA has issues but can be fixed.
[~leosun08] do you mind marking this one as dupe and take over the other one?
> Add throttler for DataNode#DataTransfer
> ---
>
> Key: HDFS-14769
> URL: https://issues.apache.org/jira/browse/HDFS-14769
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HDFS-14769.001.patch, HDFS-14769.002.patch,
> HDFS-14769.003.patch
>
>
> When a amount of block replicas are moving, the bandwidth will be full
> without throtter in DataNode#DataTransfer.
> This will affect normal reading and writing.
> {code:java}
> /**
> * Do the deed, write the bytes
> */
> @Override
> public void run() {
> // send data & checksum
> blockSender.sendBlock(out, unbufOut, null);
> }{code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2029) Fix license issues on ozone-0.4.1
[ https://issues.apache.org/jira/browse/HDDS-2029?focusedWorklogId=300731=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-300731 ] ASF GitHub Bot logged work on HDDS-2029: Author: ASF GitHub Bot Created on: 24/Aug/19 16:18 Start Date: 24/Aug/19 16:18 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1346: HDDS-2029. Fix license issues on ozone-0.4.1. URL: https://github.com/apache/hadoop/pull/1346#issuecomment-524562963 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 46 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 4 new or modified test files. | ||| _ ozone-0.4.1 Compile Tests _ | | 0 | mvndep | 69 | Maven dependency ordering for branch | | +1 | mvninstall | 646 | ozone-0.4.1 passed | | +1 | compile | 370 | ozone-0.4.1 passed | | +1 | checkstyle | 74 | ozone-0.4.1 passed | | +1 | mvnsite | 0 | ozone-0.4.1 passed | | +1 | shadedclient | 871 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 168 | ozone-0.4.1 passed | | 0 | spotbugs | 459 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 686 | ozone-0.4.1 passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 29 | Maven dependency ordering for patch | | +1 | mvninstall | 616 | the patch passed | | +1 | compile | 403 | the patch passed | | +1 | javac | 403 | the patch passed | | +1 | checkstyle | 69 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 664 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 159 | the patch passed | | +1 | findbugs | 709 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 317 | hadoop-hdds in the patch passed. | | -1 | unit | 1818 | hadoop-ozone in the patch failed. | | +1 | asflicense | 45 | The patch does not generate ASF License warnings. | | | | 7923 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.client.rpc.TestOzoneRpcClientForAclAuditLog | | | hadoop.ozone.client.rpc.TestWatchForCommit | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1346/1/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1346 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 58f5722c335e 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | ozone-0.4.1 / 1df1a7c | | Default Java | 1.8.0_222 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1346/1/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1346/1/testReport/ | | Max. process+thread count | 4721 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-ozone/dist hadoop-ozone/integration-test hadoop-ozone/ozone-manager U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1346/1/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 300731) Time Spent: 0.5h (was: 20m) > Fix license issues on ozone-0.4.1 > - > > Key: HDDS-2029 > URL: https://issues.apache.org/jira/browse/HDDS-2029 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Nanda kumar >Assignee: Nanda kumar >Priority: Blocker > Labels: pull-request-available > Fix For: 0.4.1 > > Time Spent: 0.5h > Remaining Estimate: 0h > > There are files on ozone-0.4.1 branch which doesn't have apache license > header, they have to be fixed. >
[jira] [Work logged] (HDDS-2029) Fix license issues on ozone-0.4.1
[
https://issues.apache.org/jira/browse/HDDS-2029?focusedWorklogId=300724=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-300724
]
ASF GitHub Bot logged work on HDDS-2029:
Author: ASF GitHub Bot
Created on: 24/Aug/19 15:58
Start Date: 24/Aug/19 15:58
Worklog Time Spent: 10m
Work Description: bharatviswa504 commented on pull request #1346:
HDDS-2029. Fix license issues on ozone-0.4.1.
URL: https://github.com/apache/hadoop/pull/1346
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 300724)
Time Spent: 20m (was: 10m)
> Fix license issues on ozone-0.4.1
> -
>
> Key: HDDS-2029
> URL: https://issues.apache.org/jira/browse/HDDS-2029
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 0.4.1
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> There are files on ozone-0.4.1 branch which doesn't have apache license
> header, they have to be fixed.
> {noformat}
> hadoop/hadoop-ozone/dist/src/main/compose/ozones3-haproxy/haproxy-conf/haproxy.cfg
> hadoop/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientForAclAuditLog.java
>
> hadoop/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/response/s3/bucket/TestS3BucketDeleteResponse.java
> hadoop/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/response/s3/multipart/TestS3MultipartUploadAbortResponse.java
> hadoop/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/multipart/TestS3MultipartUploadAbortRequest.java
> hadoop/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/key/OMKeyPurgeResponse.java
> hadoop/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyPurgeRequest.java
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2029) Fix license issues on ozone-0.4.1
[
https://issues.apache.org/jira/browse/HDDS-2029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Bharat Viswanadham updated HDDS-2029:
-
Fix Version/s: 0.4.1
Resolution: Fixed
Status: Resolved (was: Patch Available)
> Fix license issues on ozone-0.4.1
> -
>
> Key: HDDS-2029
> URL: https://issues.apache.org/jira/browse/HDDS-2029
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Blocker
> Labels: pull-request-available
> Fix For: 0.4.1
>
> Time Spent: 20m
> Remaining Estimate: 0h
>
> There are files on ozone-0.4.1 branch which doesn't have apache license
> header, they have to be fixed.
> {noformat}
> hadoop/hadoop-ozone/dist/src/main/compose/ozones3-haproxy/haproxy-conf/haproxy.cfg
> hadoop/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientForAclAuditLog.java
>
> hadoop/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/response/s3/bucket/TestS3BucketDeleteResponse.java
> hadoop/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/response/s3/multipart/TestS3MultipartUploadAbortResponse.java
> hadoop/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/multipart/TestS3MultipartUploadAbortRequest.java
> hadoop/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/key/OMKeyPurgeResponse.java
> hadoop/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyPurgeRequest.java
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14699) Erasure Coding: Can NOT trigger the reconstruction when have the dup internal blocks and missing one internal block
[ https://issues.apache.org/jira/browse/HDFS-14699?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16914963#comment-16914963 ] Zhao Yi Ming commented on HDFS-14699: - [~ayushtkn] Thanks for your review and comments! I fixed the typo and submit the patch 03. Also closed the PR, will track the problem in Jira here. BTY: Before I think Jira only track the issue and discussion. git track the code review things. Now I understand we only choose one way to track the whole thing. Thanks again to point out the different! > Erasure Coding: Can NOT trigger the reconstruction when have the dup internal > blocks and missing one internal block > --- > > Key: HDFS-14699 > URL: https://issues.apache.org/jira/browse/HDFS-14699 > Project: Hadoop HDFS > Issue Type: Bug > Components: ec >Affects Versions: 3.2.0, 3.1.1, 3.3.0 >Reporter: Zhao Yi Ming >Assignee: Zhao Yi Ming >Priority: Critical > Labels: patch > Attachments: HDFS-14699.00.patch, HDFS-14699.01.patch, > HDFS-14699.02.patch, HDFS-14699.03.patch, image-2019-08-20-19-58-51-872.png > > > We are tried the EC function on 80 node cluster with hadoop 3.1.1, we hit the > same scenario as you said https://issues.apache.org/jira/browse/HDFS-8881. > Following are our testing steps, hope it can helpful.(following DNs have the > testing internal blocks) > # we customized a new 10-2-1024k policy and use it on a path, now we have 12 > internal block(12 live block) > # decommission one DN, after the decommission complete. now we have 13 > internal block(12 live block and 1 decommission block) > # then shutdown one DN which did not have the same block id as 1 > decommission block, now we have 12 internal block(11 live block and 1 > decommission block) > # after wait for about 600s (before the heart beat come) commission the > decommissioned DN again, now we have 12 internal block(11 live block and 1 > duplicate block) > # Then the EC is not reconstruct the missed block > We think this is a critical issue for using the EC function in a production > env. Could you help? Thanks a lot! -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14699) Erasure Coding: Can NOT trigger the reconstruction when have the dup internal blocks and missing one internal block
[ https://issues.apache.org/jira/browse/HDFS-14699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhao Yi Ming updated HDFS-14699: Attachment: HDFS-14699.03.patch Status: Patch Available (was: Open) Fix some typo > Erasure Coding: Can NOT trigger the reconstruction when have the dup internal > blocks and missing one internal block > --- > > Key: HDFS-14699 > URL: https://issues.apache.org/jira/browse/HDFS-14699 > Project: Hadoop HDFS > Issue Type: Bug > Components: ec >Affects Versions: 3.1.1, 3.2.0, 3.3.0 >Reporter: Zhao Yi Ming >Assignee: Zhao Yi Ming >Priority: Critical > Labels: patch > Attachments: HDFS-14699.00.patch, HDFS-14699.01.patch, > HDFS-14699.02.patch, HDFS-14699.03.patch, image-2019-08-20-19-58-51-872.png > > > We are tried the EC function on 80 node cluster with hadoop 3.1.1, we hit the > same scenario as you said https://issues.apache.org/jira/browse/HDFS-8881. > Following are our testing steps, hope it can helpful.(following DNs have the > testing internal blocks) > # we customized a new 10-2-1024k policy and use it on a path, now we have 12 > internal block(12 live block) > # decommission one DN, after the decommission complete. now we have 13 > internal block(12 live block and 1 decommission block) > # then shutdown one DN which did not have the same block id as 1 > decommission block, now we have 12 internal block(11 live block and 1 > decommission block) > # after wait for about 600s (before the heart beat come) commission the > decommissioned DN again, now we have 12 internal block(11 live block and 1 > duplicate block) > # Then the EC is not reconstruct the missed block > We think this is a critical issue for using the EC function in a production > env. Could you help? Thanks a lot! -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14699) Erasure Coding: Can NOT trigger the reconstruction when have the dup internal blocks and missing one internal block
[ https://issues.apache.org/jira/browse/HDFS-14699?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhao Yi Ming updated HDFS-14699: Status: Open (was: Patch Available) will fix some typo in 03 > Erasure Coding: Can NOT trigger the reconstruction when have the dup internal > blocks and missing one internal block > --- > > Key: HDFS-14699 > URL: https://issues.apache.org/jira/browse/HDFS-14699 > Project: Hadoop HDFS > Issue Type: Bug > Components: ec >Affects Versions: 3.1.1, 3.2.0, 3.3.0 >Reporter: Zhao Yi Ming >Assignee: Zhao Yi Ming >Priority: Critical > Labels: patch > Attachments: HDFS-14699.00.patch, HDFS-14699.01.patch, > HDFS-14699.02.patch, HDFS-14699.03.patch, image-2019-08-20-19-58-51-872.png > > > We are tried the EC function on 80 node cluster with hadoop 3.1.1, we hit the > same scenario as you said https://issues.apache.org/jira/browse/HDFS-8881. > Following are our testing steps, hope it can helpful.(following DNs have the > testing internal blocks) > # we customized a new 10-2-1024k policy and use it on a path, now we have 12 > internal block(12 live block) > # decommission one DN, after the decommission complete. now we have 13 > internal block(12 live block and 1 decommission block) > # then shutdown one DN which did not have the same block id as 1 > decommission block, now we have 12 internal block(11 live block and 1 > decommission block) > # after wait for about 600s (before the heart beat come) commission the > decommissioned DN again, now we have 12 internal block(11 live block and 1 > duplicate block) > # Then the EC is not reconstruct the missed block > We think this is a critical issue for using the EC function in a production > env. Could you help? Thanks a lot! -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14633) The StorageType quota and consume in QuotaFeature is not handled when rename and setStoragePolicy etc.
[ https://issues.apache.org/jira/browse/HDFS-14633?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jinglun updated HDFS-14633: --- Attachment: HDFS-14633.006.patch > The StorageType quota and consume in QuotaFeature is not handled when rename > and setStoragePolicy etc. > --- > > Key: HDFS-14633 > URL: https://issues.apache.org/jira/browse/HDFS-14633 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HDFS-14633-testcases-explanation, HDFS-14633.002.patch, > HDFS-14633.003.patch, HDFS-14633.004.patch, HDFS-14633.005.patch, > HDFS-14633.006.patch > > > The NameNode manages the global state of the cluster. We should always take > NameNode's records as the sole criterion because no matter what inconsistent > is the NameNode should finally make everything right based on it's records. > Let's call it rule NSC(NameNode is the Sole Criterion). That means when we do > all quota related rpcs, we do the quota check according to NameNode's records > regardless of any inconsistent situation, such as the replicas doesn't match > the storage policy of the file, or the replicas count doesn't match the > file's set replica. > The work SPS deals with the wrongly palced replicas. There is a thought > about putting off the consume update of the DirectoryQuota until all replicas > are re-placed by SPS. I can't agree that because if we do so we will abandon > letting the NameNode's records to be the sole criterion. The block > replication is a good example of the rule NSC. When we count the consume of a > file(CONTIGUOUS), we multiply the replication factor with the file's length, > no matter the blocks are under replicated or excessed. We should do the same > thing for the storage type quota. > Another concern is the change will let setStoragePolicy throw > QuotaByStorageTypeExceededException which it doesn't before. I don't think > it's a big problem since the setStoragePolicy already throws IOException. Or > we can wrap the QuotaByStorageTypeExceededException in an IOException, but I > won't recommend that because it's ugly. > To make storage type consume follow the rule NSC, we need change > rename(moving a file with storage policy inherited from it's parent) and > setStoragePolicy. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14633) The StorageType quota and consume in QuotaFeature is not handled when rename and setStoragePolicy etc.
[ https://issues.apache.org/jira/browse/HDFS-14633?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16914940#comment-16914940 ] Jinglun commented on HDFS-14633: Upload patch-006, fix checkstyle, verify quota for source and destination after a successful rename. > The StorageType quota and consume in QuotaFeature is not handled when rename > and setStoragePolicy etc. > --- > > Key: HDFS-14633 > URL: https://issues.apache.org/jira/browse/HDFS-14633 > Project: Hadoop HDFS > Issue Type: Improvement >Reporter: Jinglun >Assignee: Jinglun >Priority: Major > Attachments: HDFS-14633-testcases-explanation, HDFS-14633.002.patch, > HDFS-14633.003.patch, HDFS-14633.004.patch, HDFS-14633.005.patch, > HDFS-14633.006.patch > > > The NameNode manages the global state of the cluster. We should always take > NameNode's records as the sole criterion because no matter what inconsistent > is the NameNode should finally make everything right based on it's records. > Let's call it rule NSC(NameNode is the Sole Criterion). That means when we do > all quota related rpcs, we do the quota check according to NameNode's records > regardless of any inconsistent situation, such as the replicas doesn't match > the storage policy of the file, or the replicas count doesn't match the > file's set replica. > The work SPS deals with the wrongly palced replicas. There is a thought > about putting off the consume update of the DirectoryQuota until all replicas > are re-placed by SPS. I can't agree that because if we do so we will abandon > letting the NameNode's records to be the sole criterion. The block > replication is a good example of the rule NSC. When we count the consume of a > file(CONTIGUOUS), we multiply the replication factor with the file's length, > no matter the blocks are under replicated or excessed. We should do the same > thing for the storage type quota. > Another concern is the change will let setStoragePolicy throw > QuotaByStorageTypeExceededException which it doesn't before. I don't think > it's a big problem since the setStoragePolicy already throws IOException. Or > we can wrap the QuotaByStorageTypeExceededException in an IOException, but I > won't recommend that because it's ugly. > To make storage type consume follow the rule NSC, we need change > rename(moving a file with storage policy inherited from it's parent) and > setStoragePolicy. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14568) setStoragePolicy should check quota and update consume on storage type quota.
[
https://issues.apache.org/jira/browse/HDFS-14568?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16914938#comment-16914938
]
Jinglun commented on HDFS-14568:
Upload patch-003, fix checkstyle.
> setStoragePolicy should check quota and update consume on storage type quota.
> -
>
> Key: HDFS-14568
> URL: https://issues.apache.org/jira/browse/HDFS-14568
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.1.0
>Reporter: Jinglun
>Assignee: Jinglun
>Priority: Major
> Attachments: HDFS-14568-001.patch, HDFS-14568-unit-test.patch,
> HDFS-14568.002.patch, HDFS-14568.003.patch
>
>
> The quota and consume of the file's ancestors are not handled when the
> storage policy of the file is changed. For example:
> 1. Set quota StorageType.SSD fileSpace-1 to the parent dir;
> 2. Create a file size of fileSpace with storage policy \{DISK,DISK,DISK}
> under it;
> 3. Change the storage policy of the file to ALLSSD_STORAGE_POLICY_NAME and
> expect a QuotaByStorageTypeExceededException.
> Because the quota and consume is not handled, the expected exception is not
> threw out.
>
> There are 3 reasons why we should handle the consume and the quota.
> 1. Replication uses the new storage policy. Considering a file with BlockType
> CONTIGUOUS. It's replication factor is 3 and it's storage policy is "HOT".
> Now we change the policy to "ONE_SSD". If a DN goes down and the file needs
> replication, the NN will choose storages in policy "ONE_SSD" and replicate
> the block to a SSD storage.
> 2. We acturally have a cluster storaging both HOT and COLD data. We have a
> backgroud process searching all the files to find those that are not accessed
> for a period of time. Then we set them to COLD and start a mover to move the
> replicas. After moving, all the replicas are consistent with the storage
> policy.
> 3. The NameNode manages the global state of the cluster. If there is any
> inconsistent situation, such as the replicas doesn't match the storage policy
> of the file, we should take the NameNode as the standard and make the cluster
> to match the NameNode. The block replication is a good example of the rule.
> When we count the consume of a file(CONTIGUOUS), we multiply the replication
> factor with the file's length, no matter the file is under replicated or
> excessed. So does the storage type quota and consume.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14568) setStoragePolicy should check quota and update consume on storage type quota.
[
https://issues.apache.org/jira/browse/HDFS-14568?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jinglun updated HDFS-14568:
---
Attachment: HDFS-14568.003.patch
> setStoragePolicy should check quota and update consume on storage type quota.
> -
>
> Key: HDFS-14568
> URL: https://issues.apache.org/jira/browse/HDFS-14568
> Project: Hadoop HDFS
> Issue Type: Improvement
>Affects Versions: 3.1.0
>Reporter: Jinglun
>Assignee: Jinglun
>Priority: Major
> Attachments: HDFS-14568-001.patch, HDFS-14568-unit-test.patch,
> HDFS-14568.002.patch, HDFS-14568.003.patch
>
>
> The quota and consume of the file's ancestors are not handled when the
> storage policy of the file is changed. For example:
> 1. Set quota StorageType.SSD fileSpace-1 to the parent dir;
> 2. Create a file size of fileSpace with storage policy \{DISK,DISK,DISK}
> under it;
> 3. Change the storage policy of the file to ALLSSD_STORAGE_POLICY_NAME and
> expect a QuotaByStorageTypeExceededException.
> Because the quota and consume is not handled, the expected exception is not
> threw out.
>
> There are 3 reasons why we should handle the consume and the quota.
> 1. Replication uses the new storage policy. Considering a file with BlockType
> CONTIGUOUS. It's replication factor is 3 and it's storage policy is "HOT".
> Now we change the policy to "ONE_SSD". If a DN goes down and the file needs
> replication, the NN will choose storages in policy "ONE_SSD" and replicate
> the block to a SSD storage.
> 2. We acturally have a cluster storaging both HOT and COLD data. We have a
> backgroud process searching all the files to find those that are not accessed
> for a period of time. Then we set them to COLD and start a mover to move the
> replicas. After moving, all the replicas are consistent with the storage
> policy.
> 3. The NameNode manages the global state of the cluster. If there is any
> inconsistent situation, such as the replicas doesn't match the storage policy
> of the file, we should take the NameNode as the standard and make the cluster
> to match the NameNode. The block replication is a good example of the rule.
> When we count the consume of a file(CONTIGUOUS), we multiply the replication
> factor with the file's length, no matter the file is under replicated or
> excessed. So does the storage type quota and consume.
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDFS-14769) Add throttler for DataNode#DataTransfer
[
https://issues.apache.org/jira/browse/HDFS-14769?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lisheng Sun updated HDFS-14769:
---
Attachment: HDFS-14769.003.patch
> Add throttler for DataNode#DataTransfer
> ---
>
> Key: HDFS-14769
> URL: https://issues.apache.org/jira/browse/HDFS-14769
> Project: Hadoop HDFS
> Issue Type: Improvement
>Reporter: Lisheng Sun
>Assignee: Lisheng Sun
>Priority: Major
> Attachments: HDFS-14769.001.patch, HDFS-14769.002.patch,
> HDFS-14769.003.patch
>
>
> When a amount of block replicas are moving, the bandwidth will be full
> without throtter in DataNode#DataTransfer.
> This will affect normal reading and writing.
> {code:java}
> /**
> * Do the deed, write the bytes
> */
> @Override
> public void run() {
> // send data & checksum
> blockSender.sendBlock(out, unbufOut, null);
> }{code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14748) Make DataNodePeerMetrics#minOutlierDetectionSamples configurable
[ https://issues.apache.org/jira/browse/HDFS-14748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16914931#comment-16914931 ] Lisheng Sun commented on HDFS-14748: I confirmed the two UT are ok in my local. So the UT failures are unrelated to this patch. > Make DataNodePeerMetrics#minOutlierDetectionSamples configurable > > > Key: HDFS-14748 > URL: https://issues.apache.org/jira/browse/HDFS-14748 > Project: Hadoop HDFS > Issue Type: Improvement >Affects Versions: 2.9.0, 3.0.0-alpha4 >Reporter: Lisheng Sun >Assignee: Lisheng Sun >Priority: Major > Attachments: HDFS-14748.001.patch, HDFS-14748.002.patch, > HDFS-14748.003.patch > > > Slow node monitoring is to transfer 1000 packets between DataNodes within > three hours before they are eligible to calculate and upload transmission > delays to the namenode. > But if Write data is very small and number of packets is less than 1000, the > slow node will not be reported to NameNode, so make > DataNodePeerMetrics#minOutlierDetectionSamplesconfigurable. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2029) Fix license issues on ozone-0.4.1
[
https://issues.apache.org/jira/browse/HDDS-2029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nanda kumar updated HDDS-2029:
--
Status: Patch Available (was: Open)
> Fix license issues on ozone-0.4.1
> -
>
> Key: HDDS-2029
> URL: https://issues.apache.org/jira/browse/HDDS-2029
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> There are files on ozone-0.4.1 branch which doesn't have apache license
> header, they have to be fixed.
> {noformat}
> hadoop/hadoop-ozone/dist/src/main/compose/ozones3-haproxy/haproxy-conf/haproxy.cfg
> hadoop/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientForAclAuditLog.java
>
> hadoop/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/response/s3/bucket/TestS3BucketDeleteResponse.java
> hadoop/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/response/s3/multipart/TestS3MultipartUploadAbortResponse.java
> hadoop/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/multipart/TestS3MultipartUploadAbortRequest.java
> hadoop/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/key/OMKeyPurgeResponse.java
> hadoop/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyPurgeRequest.java
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-2029) Fix license issues on ozone-0.4.1
[
https://issues.apache.org/jira/browse/HDDS-2029?focusedWorklogId=300695=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-300695
]
ASF GitHub Bot logged work on HDDS-2029:
Author: ASF GitHub Bot
Created on: 24/Aug/19 11:15
Start Date: 24/Aug/19 11:15
Worklog Time Spent: 10m
Work Description: nandakumar131 commented on pull request #1346:
HDDS-2029. Fix license issues on ozone-0.4.1.
URL: https://github.com/apache/hadoop/pull/1346
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
Issue Time Tracking
---
Worklog Id: (was: 300695)
Remaining Estimate: 0h
Time Spent: 10m
> Fix license issues on ozone-0.4.1
> -
>
> Key: HDDS-2029
> URL: https://issues.apache.org/jira/browse/HDDS-2029
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Blocker
> Labels: pull-request-available
> Time Spent: 10m
> Remaining Estimate: 0h
>
> There are files on ozone-0.4.1 branch which doesn't have apache license
> header, they have to be fixed.
> {noformat}
> hadoop/hadoop-ozone/dist/src/main/compose/ozones3-haproxy/haproxy-conf/haproxy.cfg
> hadoop/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientForAclAuditLog.java
>
> hadoop/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/response/s3/bucket/TestS3BucketDeleteResponse.java
> hadoop/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/response/s3/multipart/TestS3MultipartUploadAbortResponse.java
> hadoop/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/multipart/TestS3MultipartUploadAbortRequest.java
> hadoop/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/key/OMKeyPurgeResponse.java
> hadoop/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyPurgeRequest.java
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Updated] (HDDS-2029) Fix license issues on ozone-0.4.1
[
https://issues.apache.org/jira/browse/HDDS-2029?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
ASF GitHub Bot updated HDDS-2029:
-
Labels: pull-request-available (was: )
> Fix license issues on ozone-0.4.1
> -
>
> Key: HDDS-2029
> URL: https://issues.apache.org/jira/browse/HDDS-2029
> Project: Hadoop Distributed Data Store
> Issue Type: Bug
>Reporter: Nanda kumar
>Assignee: Nanda kumar
>Priority: Blocker
> Labels: pull-request-available
>
> There are files on ozone-0.4.1 branch which doesn't have apache license
> header, they have to be fixed.
> {noformat}
> hadoop/hadoop-ozone/dist/src/main/compose/ozones3-haproxy/haproxy-conf/haproxy.cfg
> hadoop/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientForAclAuditLog.java
>
> hadoop/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/response/s3/bucket/TestS3BucketDeleteResponse.java
> hadoop/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/response/s3/multipart/TestS3MultipartUploadAbortResponse.java
> hadoop/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/multipart/TestS3MultipartUploadAbortRequest.java
> hadoop/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/key/OMKeyPurgeResponse.java
> hadoop/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyPurgeRequest.java
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14769) Add throttler for DataNode#DataTransfer
[
https://issues.apache.org/jira/browse/HDFS-14769?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16914913#comment-16914913
]
Hadoop QA commented on HDFS-14769:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
21s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:red}-1{color} | {color:red} test4tests {color} | {color:red} 0m
0s{color} | {color:red} The patch doesn't appear to include any new or modified
tests. Please justify why no new tests are needed for this patch. Also please
list what manual steps were performed to verify this patch. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
0s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
46s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
5s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
13m 4s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
13s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
50s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
1s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:orange}-0{color} | {color:orange} checkstyle {color} | {color:orange}
0m 42s{color} | {color:orange} hadoop-hdfs-project/hadoop-hdfs: The patch
generated 1 new + 154 unchanged - 0 fixed = 155 total (was 154) {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
6s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 24s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
11s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
51s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 85m 54s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
33s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}143m 52s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.namenode.ha.TestBootstrapAliasmap |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e53b4 |
| JIRA Issue | HDFS-14769 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12978471/HDFS-14769.002.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle |
| uname | Linux 3002826f6d4d 4.15.0-54-generic #58-Ubuntu SMP Mon Jun 24
10:55:24 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / d2225c8 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_222 |
| findbugs | v3.1.0-RC1 |
| checkstyle |
https://builds.apache.org/job/PreCommit-HDFS-Build/27660/artifact/out/diff-checkstyle-hadoop-hdfs-project_hadoop-hdfs.txt
|
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/27660/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
[jira] [Commented] (HDFS-14748) Make DataNodePeerMetrics#minOutlierDetectionSamples configurable
[
https://issues.apache.org/jira/browse/HDFS-14748?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16914912#comment-16914912
]
Hadoop QA commented on HDFS-14748:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m
23s{color} | {color:blue} Docker mode activated. {color} |
|| || || || {color:brown} Prechecks {color} ||
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m
0s{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s{color} | {color:green} The patch appears to include 2 new or modified test
files. {color} |
|| || || || {color:brown} trunk Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 19m
1s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 1m
2s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
48s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m
8s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
12m 8s{color} | {color:green} branch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 1m
56s{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} trunk passed {color} |
|| || || || {color:brown} Patch Compile Tests {color} ||
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 0m
54s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 0m
48s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
38s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 0m
53s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s{color} | {color:green} The patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} xml {color} | {color:green} 0m
2s{color} | {color:green} The patch has no ill-formed XML file. {color} |
| {color:green}+1{color} | {color:green} shadedclient {color} | {color:green}
11m 4s{color} | {color:green} patch has no errors when building and testing
our client artifacts. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
3s{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m
43s{color} | {color:green} the patch passed {color} |
|| || || || {color:brown} Other Tests {color} ||
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 79m 28s{color}
| {color:red} hadoop-hdfs in the patch failed. {color} |
| {color:green}+1{color} | {color:green} asflicense {color} | {color:green} 0m
35s{color} | {color:green} The patch does not generate ASF License warnings.
{color} |
| {color:black}{color} | {color:black} {color} | {color:black}133m 51s{color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| Failed junit tests | hadoop.hdfs.server.balancer.TestBalancerRPCDelay |
| | hadoop.hdfs.TestRollingUpgrade |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Client=19.03.1 Server=19.03.1 Image:yetus/hadoop:bdbca0e |
| JIRA Issue | HDFS-14748 |
| JIRA Patch URL |
https://issues.apache.org/jira/secure/attachment/12978470/HDFS-14748.003.patch |
| Optional Tests | dupname asflicense compile javac javadoc mvninstall
mvnsite unit shadedclient findbugs checkstyle xml |
| uname | Linux 114d830c6e13 4.4.0-157-generic #185-Ubuntu SMP Tue Jul 23
09:17:01 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux |
| Build tool | maven |
| Personality | /testptch/patchprocess/precommit/personality/provided.sh |
| git revision | trunk / d2225c8 |
| maven | version: Apache Maven 3.3.9 |
| Default Java | 1.8.0_212 |
| findbugs | v3.1.0-RC1 |
| unit |
https://builds.apache.org/job/PreCommit-HDFS-Build/27661/artifact/out/patch-unit-hadoop-hdfs-project_hadoop-hdfs.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-HDFS-Build/27661/testReport/ |
| Max. process+thread count | 3944 (vs. ulimit of 1) |
| modules | C: hadoop-hdfs-project/hadoop-hdfs U:
[jira] [Created] (HDDS-2029) Fix license issues on ozone-0.4.1
Nanda kumar created HDDS-2029:
-
Summary: Fix license issues on ozone-0.4.1
Key: HDDS-2029
URL: https://issues.apache.org/jira/browse/HDDS-2029
Project: Hadoop Distributed Data Store
Issue Type: Bug
Reporter: Nanda kumar
Assignee: Nanda kumar
There are files on ozone-0.4.1 branch which doesn't have apache license header,
they have to be fixed.
{noformat}
hadoop/hadoop-ozone/dist/src/main/compose/ozones3-haproxy/haproxy-conf/haproxy.cfg
hadoop/hadoop-ozone/integration-test/src/test/java/org/apache/hadoop/ozone/client/rpc/TestOzoneRpcClientForAclAuditLog.java
hadoop/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/response/s3/bucket/TestS3BucketDeleteResponse.java
hadoop/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/response/s3/multipart/TestS3MultipartUploadAbortResponse.java
hadoop/hadoop-ozone/ozone-manager/src/test/java/org/apache/hadoop/ozone/om/request/s3/multipart/TestS3MultipartUploadAbortRequest.java
hadoop/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/response/key/OMKeyPurgeResponse.java
hadoop/hadoop-ozone/ozone-manager/src/main/java/org/apache/hadoop/ozone/om/request/key/OMKeyPurgeRequest.java
{noformat}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Created] (HDDS-2028) Release Ozone 0.4.1
Nanda kumar created HDDS-2028: - Summary: Release Ozone 0.4.1 Key: HDDS-2028 URL: https://issues.apache.org/jira/browse/HDDS-2028 Project: Hadoop Distributed Data Store Issue Type: Test Reporter: Nanda kumar Assignee: Nanda kumar This jira is to track Ozone 0.4.1 release -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1753) Datanode unable to find chunk while replication data using ratis.
[ https://issues.apache.org/jira/browse/HDDS-1753?focusedWorklogId=300688=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-300688 ] ASF GitHub Bot logged work on HDDS-1753: Author: ASF GitHub Bot Created on: 24/Aug/19 09:54 Start Date: 24/Aug/19 09:54 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1318: HDDS-1753. Datanode unable to find chunk while replication data using ratis. URL: https://github.com/apache/hadoop/pull/1318#issuecomment-524537693 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 46 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 1 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 5 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 70 | Maven dependency ordering for branch | | +1 | mvninstall | 596 | trunk passed | | +1 | compile | 347 | trunk passed | | +1 | checkstyle | 62 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 802 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 149 | trunk passed | | 0 | spotbugs | 432 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 633 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 27 | Maven dependency ordering for patch | | +1 | mvninstall | 534 | the patch passed | | +1 | compile | 361 | the patch passed | | +1 | javac | 361 | the patch passed | | -0 | checkstyle | 31 | hadoop-hdds: The patch generated 8 new + 0 unchanged - 0 fixed = 8 total (was 0) | | -0 | checkstyle | 38 | hadoop-ozone: The patch generated 2 new + 0 unchanged - 0 fixed = 2 total (was 0) | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 638 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 155 | the patch passed | | +1 | findbugs | 663 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 290 | hadoop-hdds in the patch passed. | | -1 | unit | 2136 | hadoop-ozone in the patch failed. | | +1 | asflicense | 44 | The patch does not generate ASF License warnings. | | | | 7778 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.container.common.statemachine.commandhandler.TestBlockDeletion | | | hadoop.ozone.client.rpc.TestReadRetries | | | hadoop.ozone.container.server.TestSecureContainerServer | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1318/4/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1318 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux a32307802f67 4.4.0-139-generic #165-Ubuntu SMP Wed Oct 24 10:58:50 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / d2225c8 | | Default Java | 1.8.0_222 | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1318/4/artifact/out/diff-checkstyle-hadoop-hdds.txt | | checkstyle | https://builds.apache.org/job/hadoop-multibranch/job/PR-1318/4/artifact/out/diff-checkstyle-hadoop-ozone.txt | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1318/4/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1318/4/testReport/ | | Max. process+thread count | 4503 (vs. ulimit of 5500) | | modules | C: hadoop-hdds/common hadoop-hdds/container-service hadoop-ozone/integration-test U: . | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1318/4/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 300688) Time Spent: 3h 50m (was: 3h 40m) > Datanode unable to find chunk while replication data using ratis. >
[jira] [Updated] (HDDS-2002) Update documentation for 0.4.1 release
[ https://issues.apache.org/jira/browse/HDDS-2002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Nanda kumar updated HDDS-2002: -- Fix Version/s: 0.4.1 > Update documentation for 0.4.1 release > -- > > Key: HDDS-2002 > URL: https://issues.apache.org/jira/browse/HDDS-2002 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: documentation >Reporter: Nanda kumar >Assignee: Nanda kumar >Priority: Major > Labels: pull-request-available > Fix For: 0.4.1, 0.5.0 > > Time Spent: 4h > Remaining Estimate: 0h > > We have to update Ozone documentation based on the latest changes/features. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2025) Update the Dockerfile of the official apache/ozone image
[ https://issues.apache.org/jira/browse/HDDS-2025?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16914854#comment-16914854 ] Nanda kumar commented on HDDS-2025: --- Thanks [~elek] for the update. I agree that it is not a blocker for release as such, but I feel that the ozone image of 0.4.1 release should be based on ozone-runner as a base image instead of hadoop-runner, as all our tests (smoke/blockade) are done with ozone-runner base. Yes, it would be nice to have the wiki page also updated. It will make our life easy. :) > Update the Dockerfile of the official apache/ozone image > > > Key: HDDS-2025 > URL: https://issues.apache.org/jira/browse/HDDS-2025 > Project: Hadoop Distributed Data Store > Issue Type: Improvement >Reporter: Elek, Marton >Assignee: Elek, Marton >Priority: Major > > The hadoop-docker-ozone repository contains the definition of the > apache/ozone image. > https://github.com/apache/hadoop-docker-ozone/tree/ozone-latest > It creates a docker packaging for the voted and released artifact, therefore > it can be released after the final vote. > Since the latest release we did some modification in our Dockerfiles. We need > to apply the changes to the official image as well. Especially: > 1. use ozone-runner as a base image instead of hadoop-runner > 2. rename ozoneManager service to om as we did everywhere > 3. Adjust the starter location (the script is moved to the released tar file) > -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-1927) Consolidate add/remove Acl into OzoneAclUtil class
[ https://issues.apache.org/jira/browse/HDDS-1927?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16914853#comment-16914853 ] Nanda kumar commented on HDDS-1927: --- [~xyao], [~bharatviswa] This jira is targeted for 0.4.1, but this depends on HDDS-1849, HDDS-1856, HDDS-1884, HDDS-1895 and HDDS-1911. Do we need to back-port all of them? or do we need a separate patch for 0.4.1? > Consolidate add/remove Acl into OzoneAclUtil class > -- > > Key: HDDS-1927 > URL: https://issues.apache.org/jira/browse/HDDS-1927 > Project: Hadoop Distributed Data Store > Issue Type: Bug >Reporter: Bharat Viswanadham >Assignee: Xiaoyu Yao >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 9.5h > Remaining Estimate: 0h > > This Jira is created based on @xiaoyu comment on HDDS-1884 > Can we abstract these add/remove logic into common AclUtil class as we can > see similar logic in both bucket manager and key manager? For example, > public static boolean addAcl(List existingAcls, OzoneAcl newAcl) > public static boolean removeAcl(List existingAcls, OzoneAcl newAcl) > > But to do this, we need both OmKeyInfo and OMBucketInfo to use list of > OzoneAcl/OzoneAclInfo. > This Jira is to do that refactor, and also address above comment to move > common logic to AclUtils. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDDS-2002) Update documentation for 0.4.1 release
[ https://issues.apache.org/jira/browse/HDDS-2002?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16914845#comment-16914845 ] Hudson commented on HDDS-2002: -- FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17182 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/17182/]) HDDS-2002. Update documentation for 0.4.1 release. (aengineer: rev b661dcf563c0b3cb6fe6f22bb3a39f87e3ec1c57) * (edit) hadoop-hdds/docs/content/concept/Overview.md * (edit) hadoop-hdds/docs/content/concept/Hdds.md * (edit) hadoop-hdds/docs/content/start/Kubernetes.md * (edit) hadoop-hdds/docs/content/security/SecuringDatanodes.md * (edit) hadoop-hdds/docs/content/security/SecuringS3.md * (edit) hadoop-hdds/docs/content/shell/KeyCommands.md * (edit) hadoop-hdds/docs/content/security/SecurityAcls.md * (edit) hadoop-hdds/docs/content/beyond/DockerCheatSheet.md * (edit) hadoop-hdds/docs/content/recipe/SparkOzoneFSK8S.md * (edit) hadoop-hdds/docs/content/beyond/Containers.md * (edit) hadoop-hdds/docs/content/start/OnPrem.md * (edit) hadoop-hdds/docs/content/interface/S3.md * (edit) hadoop-hdds/docs/content/interface/JavaApi.md * (edit) hadoop-hdds/docs/content/interface/OzoneFS.md * (edit) hadoop-hdds/docs/content/concept/OzoneManager.md * (edit) hadoop-hdds/docs/content/shell/BucketCommands.md * (edit) hadoop-hdds/docs/content/shell/VolumeCommands.md * (edit) hadoop-hdds/docs/content/beyond/RunningWithHDFS.md * (edit) hadoop-hdds/docs/content/start/StartFromDockerHub.md * (edit) hadoop-hdds/docs/content/recipe/Prometheus.md * (edit) hadoop-hdds/docs/content/security/SecuityWithRanger.md * (edit) hadoop-hdds/docs/content/security/SecureOzone.md * (edit) hadoop-hdds/docs/content/concept/Datanodes.md * (edit) hadoop-hdds/docs/content/recipe/_index.md * (edit) hadoop-hdds/docs/content/security/SecuringTDE.md > Update documentation for 0.4.1 release > -- > > Key: HDDS-2002 > URL: https://issues.apache.org/jira/browse/HDDS-2002 > Project: Hadoop Distributed Data Store > Issue Type: Improvement > Components: documentation >Reporter: Nanda kumar >Assignee: Nanda kumar >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 4h > Remaining Estimate: 0h > > We have to update Ozone documentation based on the latest changes/features. -- This message was sent by Atlassian Jira (v8.3.2#803003) - To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Commented] (HDFS-14722) RBF: GetMountPointStatus should return mountTable information when getFileInfoAll throw IOException
[
https://issues.apache.org/jira/browse/HDFS-14722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16914846#comment-16914846
]
Hudson commented on HDFS-14722:
---
FAILURE: Integrated in Jenkins build Hadoop-trunk-Commit #17182 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/17182/])
HDFS-14722. RBF: GetMountPointStatus should return mountTable (ayushsaxena: rev
d2225c8ca8f9bdc5cef7266697518394d8763c88)
* (edit)
hadoop-hdfs-project/hadoop-hdfs-rbf/src/test/java/org/apache/hadoop/hdfs/server/federation/router/TestRouterMountTable.java
* (edit)
hadoop-hdfs-project/hadoop-hdfs-rbf/src/main/java/org/apache/hadoop/hdfs/server/federation/router/RouterClientProtocol.java
> RBF: GetMountPointStatus should return mountTable information when
> getFileInfoAll throw IOException
> ---
>
> Key: HDFS-14722
> URL: https://issues.apache.org/jira/browse/HDFS-14722
> Project: Hadoop HDFS
> Issue Type: Bug
> Components: rbf
>Reporter: xuzq
>Assignee: xuzq
>Priority: Major
> Fix For: 3.3.0
>
> Attachments: HDFS-14722-trunk-001.patch, HDFS-14722-trunk-002.patch,
> HDFS-14722-trunk-003.patch, HDFS-14722-trunk-004.patch,
> HDFS-14722-trunk-005.patch, HDFS-14722-trunk-006.patch,
> HDFS-14722-trunk-bug-discuss.patch
>
>
> When IOException in getFileInfoAll, we should return the mountTable
> informations instead of super information.
> Code like:
> {code:java}
> // RouterClientProtocol#getMountPointStatus
> try {
> String mName = name.startsWith("/") ? name : "/" + name;
> MountTableResolver mountTable = (MountTableResolver) subclusterResolver;
> MountTable entry = mountTable.getMountPoint(mName);
> if (entry != null) {
> RemoteMethod method = new RemoteMethod("getFileInfo",
> new Class[] {String.class}, new RemoteParam());
> HdfsFileStatus fInfo = getFileInfoAll(
> entry.getDestinations(), method, mountStatusTimeOut);
> if (fInfo != null) {
> permission = fInfo.getPermission();
> owner = fInfo.getOwner();
> group = fInfo.getGroup();
> childrenNum = fInfo.getChildrenNum();
> } else {
> permission = entry.getMode();
> owner = entry.getOwnerName();
> group = entry.getGroupName();
> }
> }
> } catch (IOException e) {
> LOG.error("Cannot get mount point: {}", e.getMessage());
> }
> {code}
--
This message was sent by Atlassian Jira
(v8.3.2#803003)
-
To unsubscribe, e-mail: hdfs-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-h...@hadoop.apache.org
[jira] [Work logged] (HDDS-1975) Implement default acls for bucket/volume/key for OM HA code
[ https://issues.apache.org/jira/browse/HDDS-1975?focusedWorklogId=300674=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-300674 ] ASF GitHub Bot logged work on HDDS-1975: Author: ASF GitHub Bot Created on: 24/Aug/19 06:06 Start Date: 24/Aug/19 06:06 Worklog Time Spent: 10m Work Description: hadoop-yetus commented on issue #1315: HDDS-1975. Implement default acls for bucket/volume/key for OM HA code. URL: https://github.com/apache/hadoop/pull/1315#issuecomment-524524077 :broken_heart: **-1 overall** | Vote | Subsystem | Runtime | Comment | |::|--:|:|:| | 0 | reexec | 43 | Docker mode activated. | ||| _ Prechecks _ | | +1 | dupname | 0 | No case conflicting files found. | | +1 | @author | 0 | The patch does not contain any @author tags. | | +1 | test4tests | 0 | The patch appears to include 1 new or modified test files. | ||| _ trunk Compile Tests _ | | 0 | mvndep | 29 | Maven dependency ordering for branch | | +1 | mvninstall | 618 | trunk passed | | +1 | compile | 376 | trunk passed | | +1 | checkstyle | 70 | trunk passed | | +1 | mvnsite | 0 | trunk passed | | +1 | shadedclient | 846 | branch has no errors when building and testing our client artifacts. | | +1 | javadoc | 162 | trunk passed | | 0 | spotbugs | 445 | Used deprecated FindBugs config; considering switching to SpotBugs. | | +1 | findbugs | 644 | trunk passed | ||| _ Patch Compile Tests _ | | 0 | mvndep | 20 | Maven dependency ordering for patch | | +1 | mvninstall | 574 | the patch passed | | +1 | compile | 368 | the patch passed | | +1 | javac | 368 | the patch passed | | +1 | checkstyle | 72 | the patch passed | | +1 | mvnsite | 0 | the patch passed | | +1 | whitespace | 0 | The patch has no whitespace issues. | | +1 | shadedclient | 664 | patch has no errors when building and testing our client artifacts. | | +1 | javadoc | 154 | the patch passed | | +1 | findbugs | 657 | the patch passed | ||| _ Other Tests _ | | +1 | unit | 289 | hadoop-hdds in the patch passed. | | -1 | unit | 1984 | hadoop-ozone in the patch failed. | | +1 | asflicense | 39 | The patch does not generate ASF License warnings. | | | | 7758 | | | Reason | Tests | |---:|:--| | Failed junit tests | hadoop.ozone.container.server.TestSecureContainerServer | | | hadoop.ozone.client.rpc.TestWatchForCommit | | Subsystem | Report/Notes | |--:|:-| | Docker | Client=19.03.1 Server=19.03.1 base: https://builds.apache.org/job/hadoop-multibranch/job/PR-1315/11/artifact/out/Dockerfile | | GITHUB PR | https://github.com/apache/hadoop/pull/1315 | | Optional Tests | dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient findbugs checkstyle | | uname | Linux 9b0c75acbd59 4.4.0-138-generic #164-Ubuntu SMP Tue Oct 2 17:16:02 UTC 2018 x86_64 x86_64 x86_64 GNU/Linux | | Build tool | maven | | Personality | personality/hadoop.sh | | git revision | trunk / d2225c8 | | Default Java | 1.8.0_212 | | unit | https://builds.apache.org/job/hadoop-multibranch/job/PR-1315/11/artifact/out/patch-unit-hadoop-ozone.txt | | Test Results | https://builds.apache.org/job/hadoop-multibranch/job/PR-1315/11/testReport/ | | Max. process+thread count | 5101 (vs. ulimit of 5500) | | modules | C: hadoop-ozone/ozone-manager hadoop-ozone/integration-test U: hadoop-ozone | | Console output | https://builds.apache.org/job/hadoop-multibranch/job/PR-1315/11/console | | versions | git=2.7.4 maven=3.3.9 findbugs=3.1.0-RC1 | | Powered by | Apache Yetus 0.10.0 http://yetus.apache.org | This message was automatically generated. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org Issue Time Tracking --- Worklog Id: (was: 300674) Time Spent: 3h 10m (was: 3h) > Implement default acls for bucket/volume/key for OM HA code > --- > > Key: HDDS-1975 > URL: https://issues.apache.org/jira/browse/HDDS-1975 > Project: Hadoop Distributed Data Store > Issue Type: Sub-task >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Labels: pull-request-available > Fix For: 0.5.0 > > Time Spent: 3h 10m > Remaining Estimate: 0h > > This Jira is to implement default ACLs for Ozone volume/bucket/key. -- This
