[jira] [Commented] (HDFS-6945) excessReplicateMap can increase infinitely
[
https://issues.apache.org/jira/browse/HDFS-6945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117122#comment-14117122
]
Hadoop QA commented on HDFS-6945:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12665699/HDFS-6945.2.patch
against trunk revision 258c7d0.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7864//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7864//console
This message is automatically generated.
excessReplicateMap can increase infinitely
--
Key: HDFS-6945
URL: https://issues.apache.org/jira/browse/HDFS-6945
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.5.0
Reporter: Akira AJISAKA
Assignee: Akira AJISAKA
Priority: Critical
Labels: metrics
Attachments: HDFS-6945.2.patch, HDFS-6945.patch
I'm seeing ExcessBlocks metric increases to more than 300K in some clusters,
however, there are no over-replicated blocks (confirmed by fsck).
After a further research, I noticed when deleting a block, BlockManager does
not remove the block from excessReplicateMap or decrement excessBlocksCount.
Usually the metric is decremented when processing block report, however, if
the block has been deleted, BlockManager does not remove the block from
excessReplicateMap or decrement the metric.
That way the metric and excessReplicateMap can increase infinitely (i.e.
memory leak can occur).
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6832) Fix the usage of 'hdfs namenode' command
[
https://issues.apache.org/jira/browse/HDFS-6832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117121#comment-14117121
]
Hadoop QA commented on HDFS-6832:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12665701/hdfs-6832_002.txt
against trunk revision 258c7d0.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.TestClusterId
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7865//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7865//console
This message is automatically generated.
Fix the usage of 'hdfs namenode' command
Key: HDFS-6832
URL: https://issues.apache.org/jira/browse/HDFS-6832
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.4.1
Reporter: Akira AJISAKA
Assignee: skrho
Priority: Minor
Labels: newbie
Attachments: hdfs-6832.txt, hdfs-6832_001.txt, hdfs-6832_002.txt
{code}
[root@trunk ~]# hdfs namenode -help
Usage: java NameNode [-backup] |
[-checkpoint] |
[-format [-clusterid cid ] [-force] [-nonInteractive] ] |
[-upgrade [-clusterid cid] [-renameReservedk-v pairs] ] |
[-upgradeOnly [-clusterid cid] [-renameReservedk-v pairs] ] |
[-rollback] |
[-rollingUpgrade downgrade|rollback ] |
[-finalize] |
[-importCheckpoint] |
[-initializeSharedEdits] |
[-bootstrapStandby] |
[-recover [ -force] ] |
[-metadataVersion ] ]
{code}
There're some issues in the usage to be fixed.
# Usage: java NameNode should be Usage: hdfs namenode
# -rollingUpgrade started option should be added
# The last ']' should be removed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-2975) Rename with overwrite flag true can make NameNode to stuck in safemode on NN (crash + restart).
[
https://issues.apache.org/jira/browse/HDFS-2975?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117198#comment-14117198
]
Yi Liu commented on HDFS-2975:
--
Thanks Uma for review and comment. Your consideration is reasonable.
After think more, we need to track blocks incrementally if HA enabled and in
safe-mode.
* While loading the editlog, for delete (or rename with overwrite) Op, we need
to track blocks incrementally and can call
{{removeBlocksAndUpdateSafemodeTotal}}.
* While HDFS client calls delete (or rename with overwrite) Op, NN is not in
safe-mode, and we don't need to track blocks incrementally. We can do
incremental deletion of blocks directly.
So current delete is correct. And for rename with overwrite, we have called
{{removeBlocksAndUpdateSafemodeTotal}} which loading editlog.
So the behavior in the patch is exact what we expect.
Rename with overwrite flag true can make NameNode to stuck in safemode on NN
(crash + restart).
---
Key: HDFS-2975
URL: https://issues.apache.org/jira/browse/HDFS-2975
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Reporter: Uma Maheswara Rao G
Assignee: Yi Liu
Attachments: HDFS-2975.001.patch
When we rename the file with overwrite flag as true, it will delete the
destination file blocks. After deleting the blocks, whenever it releases the
fsNameSystem lock, NN can give the invalidation work to corresponding DNs to
delete the blocks.
Parallaly it will sync the rename related edits to editlog file. At this step
before NN sync the edits if NN crashes, NN can stuck into safemode on
restart. This is because block already deleted from the DN as part of
invalidations. But dst file still exist as rename edits not persisted in log
file and no DN will report that blocks now.
This is similar to HDFS-2815
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-6606) Optimize HDFS Encrypted Transport performance
[
https://issues.apache.org/jira/browse/HDFS-6606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liu updated HDFS-6606:
-
Attachment: HDFS-6606.003.patch
Update the patch for all comments.
{quote}
why do we need 2 confs?, what was wrong with the original one?
{quote}
In the new patch, use {{DNConf}} to get {{Configuration}}
HADOOP-11040 will not affect this JIRA, so all tests are expected to pass.
Optimize HDFS Encrypted Transport performance
-
Key: HDFS-6606
URL: https://issues.apache.org/jira/browse/HDFS-6606
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode, hdfs-client, security
Reporter: Yi Liu
Assignee: Yi Liu
Attachments: HDFS-6606.001.patch, HDFS-6606.002.patch,
HDFS-6606.003.patch, OptimizeHdfsEncryptedTransportperformance.pdf
In HDFS-3637, [~atm] added support for encrypting the DataTransferProtocol,
it was a great work.
It utilizes SASL {{Digest-MD5}} mechanism (use Qop: auth-conf), it supports
three security strength:
* high 3des or rc4 (128bits)
* medium des or rc4(56bits)
* low rc4(40bits)
3des and rc4 are slow, only *tens of MB/s*,
http://www.javamex.com/tutorials/cryptography/ciphers.shtml
http://www.cs.wustl.edu/~jain/cse567-06/ftp/encryption_perf/
I will give more detailed performance data in future. Absolutely it’s
bottleneck and will vastly affect the end to end performance.
AES(Advanced Encryption Standard) is recommended as a replacement of DES,
it’s more secure; with AES-NI support, the throughput can reach nearly
*2GB/s*, it won’t be the bottleneck any more, AES and CryptoCodec work is
supported in HADOOP-10150, HADOOP-10603 and HADOOP-10693 (We may need to add
a new mode support for AES).
This JIRA will use AES with AES-NI support as encryption algorithm for
DataTransferProtocol.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-6950) Add Additional unit tests for HDFS-6581
[ https://issues.apache.org/jira/browse/HDFS-6950?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-6950: - Attachment: HDFS-6950.1.patch Update the patch. Add Additional unit tests for HDFS-6581 --- Key: HDFS-6950 URL: https://issues.apache.org/jira/browse/HDFS-6950 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Attachments: HDFS-6950.0.patch, HDFS-6950.1.patch Create additional unit tests for HDFS-6581 in addition to existing ones in HDFS-6927. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6961) Archival Storage: BlockPlacementPolicy#chooseTarget should check each valid storage type in each choosing round
[
https://issues.apache.org/jira/browse/HDFS-6961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117255#comment-14117255
]
Vinayakumar B commented on HDFS-6961:
-
One doubt I have in this part of the code in chooseLocalStorage()
{code}
+ for (IteratorMap.EntryStorageType, Integer iter = storageTypes
+ .entrySet().iterator(); iter.hasNext(); ) {
+Map.EntryStorageType, Integer entry = iter.next();
+StorageType type = entry.getKey();
+if (addIfIsGoodTarget(localStorage, excludedNodes, blocksize,
+maxNodesPerRack, false, results, avoidStaleNodes, type) = 0) {
+ int num = entry.getValue();
+ if (num == 1) {
+iter.remove();
+ } else {
+entry.setValue(num - 1);
+ }
+ return localStorage;
+}
{code}
If local node have multiple storages, ex: SSD, HDD, then SSD will be having the
preference, And if storage policy also says SSD, HDD to store first replica
on SSD, remaining on HDD.
Then above iterator may return HDD storageType first, then LocalStorage will be
chosen with HDD as storage type instead of SSD.
Will this be fine? Or Am I missing something here?
Archival Storage: BlockPlacementPolicy#chooseTarget should check each valid
storage type in each choosing round
---
Key: HDFS-6961
URL: https://issues.apache.org/jira/browse/HDFS-6961
Project: Hadoop HDFS
Issue Type: Sub-task
Components: balancer, namenode
Reporter: Jing Zhao
Assignee: Jing Zhao
Attachments: HDFS-6961.000.patch, HDFS-6961.001.patch
Currently in each choosing round of BlockPlacementPolicy#chooseTarget (e.g.,
round 1 for choosing local storage, round 2 for remote rack, etc.), we only
take into account the first storage type in the storage type list. This may
fail to choose the local machine or sometimes may even fail to choose enough
nodes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6606) Optimize HDFS Encrypted Transport performance
[
https://issues.apache.org/jira/browse/HDFS-6606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117349#comment-14117349
]
Hadoop QA commented on HDFS-6606:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12665737/HDFS-6606.003.patch
against trunk revision 258c7d0.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
org.apache.hadoop.hdfs.server.datanode.TestNNHandlesCombinedBlockReport
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7866//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7866//console
This message is automatically generated.
Optimize HDFS Encrypted Transport performance
-
Key: HDFS-6606
URL: https://issues.apache.org/jira/browse/HDFS-6606
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode, hdfs-client, security
Reporter: Yi Liu
Assignee: Yi Liu
Attachments: HDFS-6606.001.patch, HDFS-6606.002.patch,
HDFS-6606.003.patch, OptimizeHdfsEncryptedTransportperformance.pdf
In HDFS-3637, [~atm] added support for encrypting the DataTransferProtocol,
it was a great work.
It utilizes SASL {{Digest-MD5}} mechanism (use Qop: auth-conf), it supports
three security strength:
* high 3des or rc4 (128bits)
* medium des or rc4(56bits)
* low rc4(40bits)
3des and rc4 are slow, only *tens of MB/s*,
http://www.javamex.com/tutorials/cryptography/ciphers.shtml
http://www.cs.wustl.edu/~jain/cse567-06/ftp/encryption_perf/
I will give more detailed performance data in future. Absolutely it’s
bottleneck and will vastly affect the end to end performance.
AES(Advanced Encryption Standard) is recommended as a replacement of DES,
it’s more secure; with AES-NI support, the throughput can reach nearly
*2GB/s*, it won’t be the bottleneck any more, AES and CryptoCodec work is
supported in HADOOP-10150, HADOOP-10603 and HADOOP-10693 (We may need to add
a new mode support for AES).
This JIRA will use AES with AES-NI support as encryption algorithm for
DataTransferProtocol.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-6980) TestWebHdfsFileSystemContract fails in trunk
Akira AJISAKA created HDFS-6980: --- Summary: TestWebHdfsFileSystemContract fails in trunk Key: HDFS-6980 URL: https://issues.apache.org/jira/browse/HDFS-6980 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: Akira AJISAKA Many tests in TestWebHdfsFileSystemContract fail by too many open files error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6969) Archival Storage: INode#getStoragePolicyID should always return the latest storage policy
[
https://issues.apache.org/jira/browse/HDFS-6969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-6969:
Attachment: HDFS-6969.002.patch
Thanks for the review, Nicholas!
bq. i.e. use the single parameter convertStorageTypes(..) instead.
The type of {{p}} is {{ListStorageTypeProto}}, thus we can only call
{{convertStorageTypes(ListStorageTypeProto, int)}} here. But I think it will
better to use {{targets[i].length)}} instead of {{p.size()}}.
Archival Storage: INode#getStoragePolicyID should always return the latest
storage policy
-
Key: HDFS-6969
URL: https://issues.apache.org/jira/browse/HDFS-6969
Project: Hadoop HDFS
Issue Type: Sub-task
Components: balancer, namenode
Reporter: Jing Zhao
Assignee: Jing Zhao
Attachments: HDFS-6969.000.patch, HDFS-6969.001.patch,
HDFS-6969.002.patch
In general, every file should only provide exact one storage policy for the
Mover, no matter its snapshot states. Suppose a file /foo/bar, and it is
contained in snapshots s1 and s2 of the root. If /foo/bar,
/.snapshot/s1/foo/bar and /.snapshot/s2/foo/bar have different storage
policies, when running Mover, we have to select one of the storage policies,
among which the latest one should be the best. And if /foo/bar is deleted, we
should still use its storage policy before the deletion, since the file
deletion should not trigger data migration.
Thus maybe what we can do is:
1. For a file with policy directly specified on it, alway follow the latest
2. Otherwise follow its latest parental path to identify its storage policy
(simply following the parent link)
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6898) DN must reserve space for a full block when an RBW block is created
[ https://issues.apache.org/jira/browse/HDFS-6898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117730#comment-14117730 ] Arpit Agarwal commented on HDFS-6898: - [~cmccabe], did you get a chance to take a look at the updated patch? DN must reserve space for a full block when an RBW block is created --- Key: HDFS-6898 URL: https://issues.apache.org/jira/browse/HDFS-6898 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 2.5.0 Reporter: Gopal V Assignee: Arpit Agarwal Attachments: HDFS-6898.01.patch, HDFS-6898.03.patch, HDFS-6898.04.patch, HDFS-6898.05.patch DN will successfully create two RBW blocks on the same volume even if the free space is sufficient for just one full block. One or both block writers may subsequently get a DiskOutOfSpace exception. This can be avoided by allocating space up front. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6800) Support Datanode layout changes with rolling upgrade
[
https://issues.apache.org/jira/browse/HDFS-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-6800:
Fix Version/s: (was: 2.6.0)
Support Datanode layout changes with rolling upgrade
Key: HDFS-6800
URL: https://issues.apache.org/jira/browse/HDFS-6800
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 3.0.0
Reporter: Colin Patrick McCabe
Assignee: James Thomas
Fix For: 3.0.0
Attachments: HDFS-6800.2.patch, HDFS-6800.3.patch, HDFS-6800.4.patch,
HDFS-6800.5.patch, HDFS-6800.6.patch, HDFS-6800.7.patch, HDFS-6800.8.patch,
HDFS-6800.patch
We need to handle attempts to rolling-upgrade the DataNode to a new storage
directory layout.
One approach is to disallow such upgrades. If we choose this approach, we
should make sure that the system administrator gets a helpful error message
and a clean failure when trying to use rolling upgrade to a version that
doesn't support it. Based on the compatibility guarantees described in
HDFS-5535, this would mean that *any* future DataNode layout changes would
require a major version upgrade.
Another approach would be to support rolling upgrade from an old DN storage
layout to a new layout. This approach requires us to change our
documentation to explain to users that they should supply the {{\-rollback}}
command on the command-line when re-starting the DataNodes during rolling
rollback. Currently the documentation just says to restart the DataNode
normally.
Another issue here is that the DataNode's usage message describes rollback
options that no longer exist. The help text says that the DN supports
{{\-rollingupgrade rollback}}, but this option was removed by HDFS-6005.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-6800) Support Datanode layout changes with rolling upgrade
[
https://issues.apache.org/jira/browse/HDFS-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-6800:
Target Version/s: (was: 2.6.0)
Support Datanode layout changes with rolling upgrade
Key: HDFS-6800
URL: https://issues.apache.org/jira/browse/HDFS-6800
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 3.0.0
Reporter: Colin Patrick McCabe
Assignee: James Thomas
Fix For: 3.0.0
Attachments: HDFS-6800.2.patch, HDFS-6800.3.patch, HDFS-6800.4.patch,
HDFS-6800.5.patch, HDFS-6800.6.patch, HDFS-6800.7.patch, HDFS-6800.8.patch,
HDFS-6800.patch
We need to handle attempts to rolling-upgrade the DataNode to a new storage
directory layout.
One approach is to disallow such upgrades. If we choose this approach, we
should make sure that the system administrator gets a helpful error message
and a clean failure when trying to use rolling upgrade to a version that
doesn't support it. Based on the compatibility guarantees described in
HDFS-5535, this would mean that *any* future DataNode layout changes would
require a major version upgrade.
Another approach would be to support rolling upgrade from an old DN storage
layout to a new layout. This approach requires us to change our
documentation to explain to users that they should supply the {{\-rollback}}
command on the command-line when re-starting the DataNodes during rolling
rollback. Currently the documentation just says to restart the DataNode
normally.
Another issue here is that the DataNode's usage message describes rollback
options that no longer exist. The help text says that the DN supports
{{\-rollingupgrade rollback}}, but this option was removed by HDFS-6005.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-6800) Support Datanode layout changes with rolling upgrade
[
https://issues.apache.org/jira/browse/HDFS-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated HDFS-6800:
Affects Version/s: (was: 2.6.0)
3.0.0
Support Datanode layout changes with rolling upgrade
Key: HDFS-6800
URL: https://issues.apache.org/jira/browse/HDFS-6800
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 3.0.0
Reporter: Colin Patrick McCabe
Assignee: James Thomas
Fix For: 3.0.0
Attachments: HDFS-6800.2.patch, HDFS-6800.3.patch, HDFS-6800.4.patch,
HDFS-6800.5.patch, HDFS-6800.6.patch, HDFS-6800.7.patch, HDFS-6800.8.patch,
HDFS-6800.patch
We need to handle attempts to rolling-upgrade the DataNode to a new storage
directory layout.
One approach is to disallow such upgrades. If we choose this approach, we
should make sure that the system administrator gets a helpful error message
and a clean failure when trying to use rolling upgrade to a version that
doesn't support it. Based on the compatibility guarantees described in
HDFS-5535, this would mean that *any* future DataNode layout changes would
require a major version upgrade.
Another approach would be to support rolling upgrade from an old DN storage
layout to a new layout. This approach requires us to change our
documentation to explain to users that they should supply the {{\-rollback}}
command on the command-line when re-starting the DataNodes during rolling
rollback. Currently the documentation just says to restart the DataNode
normally.
Another issue here is that the DataNode's usage message describes rollback
options that no longer exist. The help text says that the DN supports
{{\-rollingupgrade rollback}}, but this option was removed by HDFS-6005.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-6482) Use block ID-based block layout on datanodes
[ https://issues.apache.org/jira/browse/HDFS-6482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-6482: Target Version/s: 3.0.0 (was: 2.6.0) Use block ID-based block layout on datanodes Key: HDFS-6482 URL: https://issues.apache.org/jira/browse/HDFS-6482 Project: Hadoop HDFS Issue Type: Improvement Components: datanode Affects Versions: 3.0.0 Reporter: James Thomas Assignee: James Thomas Fix For: 3.0.0 Attachments: 6482-design.doc, HDFS-6482.1.patch, HDFS-6482.2.patch, HDFS-6482.3.patch, HDFS-6482.4.patch, HDFS-6482.5.patch, HDFS-6482.6.patch, HDFS-6482.7.patch, HDFS-6482.8.patch, HDFS-6482.9.patch, HDFS-6482.patch, hadoop-24-datanode-dir.tgz Right now blocks are placed into directories that are split into many subdirectories when capacity is reached. Instead we can use a block's ID to determine the path it should go in. This eliminates the need for the LDir data structure that facilitates the splitting of directories when they reach capacity as well as fields in ReplicaInfo that keep track of a replica's location. An extension of the work in HDFS-3290. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6482) Use block ID-based block layout on datanodes
[ https://issues.apache.org/jira/browse/HDFS-6482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-6482: Affects Version/s: (was: 2.5.0) 3.0.0 Use block ID-based block layout on datanodes Key: HDFS-6482 URL: https://issues.apache.org/jira/browse/HDFS-6482 Project: Hadoop HDFS Issue Type: Improvement Components: datanode Affects Versions: 3.0.0 Reporter: James Thomas Assignee: James Thomas Fix For: 3.0.0 Attachments: 6482-design.doc, HDFS-6482.1.patch, HDFS-6482.2.patch, HDFS-6482.3.patch, HDFS-6482.4.patch, HDFS-6482.5.patch, HDFS-6482.6.patch, HDFS-6482.7.patch, HDFS-6482.8.patch, HDFS-6482.9.patch, HDFS-6482.patch, hadoop-24-datanode-dir.tgz Right now blocks are placed into directories that are split into many subdirectories when capacity is reached. Instead we can use a block's ID to determine the path it should go in. This eliminates the need for the LDir data structure that facilitates the splitting of directories when they reach capacity as well as fields in ReplicaInfo that keep track of a replica's location. An extension of the work in HDFS-3290. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-6981) DN upgrade with layout version change should not use trash
Arpit Agarwal created HDFS-6981: --- Summary: DN upgrade with layout version change should not use trash Key: HDFS-6981 URL: https://issues.apache.org/jira/browse/HDFS-6981 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 3.0.0 Reporter: James Thomas Post HDFS-6800, we can encounter the following scenario: # We start with DN software version -55 and initiate a rolling upgrade to version -56 # We delete some blocks, and they are moved to trash # We roll back to DN software version -55 using the -rollback flag – since we are running the old code (prior to this patch), we will restore the previous directory but will not delete the trash # We append to some of the blocks that were deleted in step 2 # We then restart a DN that contains blocks that were appended to – since the trash still exists, it will be restored at this point, the appended-to blocks will be overwritten, and we will lose the appended data So I think we need to avoid writing anything to the trash directory if we have a previous directory. Thanks to [~james.thomas] for reporting this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6800) Support Datanode layout changes with rolling upgrade
[
https://issues.apache.org/jira/browse/HDFS-6800?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117744#comment-14117744
]
Arpit Agarwal commented on HDFS-6800:
-
Thanks [~james.thomas]. I filed HDFS-6981 to investigate what you described.
My guess is if we might run into issues during step 3 itself.
Support Datanode layout changes with rolling upgrade
Key: HDFS-6800
URL: https://issues.apache.org/jira/browse/HDFS-6800
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode
Affects Versions: 3.0.0
Reporter: Colin Patrick McCabe
Assignee: James Thomas
Fix For: 3.0.0
Attachments: HDFS-6800.2.patch, HDFS-6800.3.patch, HDFS-6800.4.patch,
HDFS-6800.5.patch, HDFS-6800.6.patch, HDFS-6800.7.patch, HDFS-6800.8.patch,
HDFS-6800.patch
We need to handle attempts to rolling-upgrade the DataNode to a new storage
directory layout.
One approach is to disallow such upgrades. If we choose this approach, we
should make sure that the system administrator gets a helpful error message
and a clean failure when trying to use rolling upgrade to a version that
doesn't support it. Based on the compatibility guarantees described in
HDFS-5535, this would mean that *any* future DataNode layout changes would
require a major version upgrade.
Another approach would be to support rolling upgrade from an old DN storage
layout to a new layout. This approach requires us to change our
documentation to explain to users that they should supply the {{\-rollback}}
command on the command-line when re-starting the DataNodes during rolling
rollback. Currently the documentation just says to restart the DataNode
normally.
Another issue here is that the DataNode's usage message describes rollback
options that no longer exist. The help text says that the DN supports
{{\-rollingupgrade rollback}}, but this option was removed by HDFS-6005.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6981) DN upgrade with layout version change should not use trash
[ https://issues.apache.org/jira/browse/HDFS-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117782#comment-14117782 ] James Thomas commented on HDFS-6981: [~arpitagarwal], thanks for filing this. Shouldn't the fix version be 2.6.0? This is a non-issue if we leave HDFS-6482 and HDFS-6800 in trunk, as I noted in https://issues.apache.org/jira/browse/HDFS-6800?focusedCommentId=14116260page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14116260. If we fix this JIRA, we can put HDFS-6482 and HDFS-6800 in branch-2. But trunk is currently fine as it is. DN upgrade with layout version change should not use trash -- Key: HDFS-6981 URL: https://issues.apache.org/jira/browse/HDFS-6981 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 3.0.0 Reporter: James Thomas Post HDFS-6800, we can encounter the following scenario: # We start with DN software version -55 and initiate a rolling upgrade to version -56 # We delete some blocks, and they are moved to trash # We roll back to DN software version -55 using the -rollback flag – since we are running the old code (prior to this patch), we will restore the previous directory but will not delete the trash # We append to some of the blocks that were deleted in step 2 # We then restart a DN that contains blocks that were appended to – since the trash still exists, it will be restored at this point, the appended-to blocks will be overwritten, and we will lose the appended data So I think we need to avoid writing anything to the trash directory if we have a previous directory. Thanks to [~james.thomas] for reporting this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6969) Archival Storage: INode#getStoragePolicyID should always return the latest storage policy
[ https://issues.apache.org/jira/browse/HDFS-6969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117784#comment-14117784 ] Tsz Wo Nicholas Sze commented on HDFS-6969: --- Oops, I looked at convertStorageTypes(StorageType[] types, int startIdx) which was not the one the patch used. +1 patch looks good. Archival Storage: INode#getStoragePolicyID should always return the latest storage policy - Key: HDFS-6969 URL: https://issues.apache.org/jira/browse/HDFS-6969 Project: Hadoop HDFS Issue Type: Sub-task Components: balancer, namenode Reporter: Jing Zhao Assignee: Jing Zhao Attachments: HDFS-6969.000.patch, HDFS-6969.001.patch, HDFS-6969.002.patch In general, every file should only provide exact one storage policy for the Mover, no matter its snapshot states. Suppose a file /foo/bar, and it is contained in snapshots s1 and s2 of the root. If /foo/bar, /.snapshot/s1/foo/bar and /.snapshot/s2/foo/bar have different storage policies, when running Mover, we have to select one of the storage policies, among which the latest one should be the best. And if /foo/bar is deleted, we should still use its storage policy before the deletion, since the file deletion should not trigger data migration. Thus maybe what we can do is: 1. For a file with policy directly specified on it, alway follow the latest 2. Otherwise follow its latest parental path to identify its storage policy (simply following the parent link) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6981) DN upgrade with layout version change should not use trash
[ https://issues.apache.org/jira/browse/HDFS-6981?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117787#comment-14117787 ] Arpit Agarwal commented on HDFS-6981: - The fix version can't be 2.6.0 since we have no timeline yet on getting the changes to branch-2. DN upgrade with layout version change should not use trash -- Key: HDFS-6981 URL: https://issues.apache.org/jira/browse/HDFS-6981 Project: Hadoop HDFS Issue Type: Bug Components: datanode Affects Versions: 3.0.0 Reporter: James Thomas Post HDFS-6800, we can encounter the following scenario: # We start with DN software version -55 and initiate a rolling upgrade to version -56 # We delete some blocks, and they are moved to trash # We roll back to DN software version -55 using the -rollback flag – since we are running the old code (prior to this patch), we will restore the previous directory but will not delete the trash # We append to some of the blocks that were deleted in step 2 # We then restart a DN that contains blocks that were appended to – since the trash still exists, it will be restored at this point, the appended-to blocks will be overwritten, and we will lose the appended data So I think we need to avoid writing anything to the trash directory if we have a previous directory. Thanks to [~james.thomas] for reporting this. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6961) Archival Storage: BlockPlacementPolicy#chooseTarget should check each valid storage type in each choosing round
[
https://issues.apache.org/jira/browse/HDFS-6961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-6961:
Attachment: HDFS-6961.002.patch
Thanks for the comments, Nicholas and Vinay! Update the patch to address the
comments.
bq. if (!unavailableStorages.contains(type)) is not needed. boolean retry
is also not needed since storageTypes must be non-empty
Actually we may need the retry check here. This is because in {{chooseTarget}},
if the {{storageTypes}} is empty, another {{NotEnoughReplicasException}} will
be thrown and we can hit an infinite loop without the check in
{{unavailableStorages}} (the infinite loop can be produced by
TestReplicationPolicy#testChooseTargetWithMoreThanAvailableNodesWithStaleness):
{code}
try {
if ((numOfReplicas = requiredStorageTypes.size()) == 0) {
throw new NotEnoughReplicasException(
All required storage types are unavailable:
+ unavailableStorages= + unavailableStorages
+ , storagePolicy= + storagePolicy);
}
if (numOfResults == 0) {
{code}
bq. Then above iterator may return HDD storageType first, then LocalStorage
will be chosen with HDD as storage type instead of SSD.
I think you are right here, Vinay. For Archival Storage, what we can do here
may be switching the two for loops, since EnumMap.entrySet().iterator keeps the
order of the key, and DISK is before ARCHIVE in StorageType. To completely
solve the storage preference issue we may need better mechanism and maybe we
can do it in a separate jira?
Archival Storage: BlockPlacementPolicy#chooseTarget should check each valid
storage type in each choosing round
---
Key: HDFS-6961
URL: https://issues.apache.org/jira/browse/HDFS-6961
Project: Hadoop HDFS
Issue Type: Sub-task
Components: balancer, namenode
Reporter: Jing Zhao
Assignee: Jing Zhao
Attachments: HDFS-6961.000.patch, HDFS-6961.001.patch,
HDFS-6961.002.patch
Currently in each choosing round of BlockPlacementPolicy#chooseTarget (e.g.,
round 1 for choosing local storage, round 2 for remote rack, etc.), we only
take into account the first storage type in the storage type list. This may
fail to choose the local machine or sometimes may even fail to choose enough
nodes.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6606) Optimize HDFS Encrypted Transport performance
[
https://issues.apache.org/jira/browse/HDFS-6606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117810#comment-14117810
]
Yi Liu commented on HDFS-6606:
--
The tests failure are not related, I can run them successfully in local
environment.
Optimize HDFS Encrypted Transport performance
-
Key: HDFS-6606
URL: https://issues.apache.org/jira/browse/HDFS-6606
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode, hdfs-client, security
Reporter: Yi Liu
Assignee: Yi Liu
Attachments: HDFS-6606.001.patch, HDFS-6606.002.patch,
HDFS-6606.003.patch, OptimizeHdfsEncryptedTransportperformance.pdf
In HDFS-3637, [~atm] added support for encrypting the DataTransferProtocol,
it was a great work.
It utilizes SASL {{Digest-MD5}} mechanism (use Qop: auth-conf), it supports
three security strength:
* high 3des or rc4 (128bits)
* medium des or rc4(56bits)
* low rc4(40bits)
3des and rc4 are slow, only *tens of MB/s*,
http://www.javamex.com/tutorials/cryptography/ciphers.shtml
http://www.cs.wustl.edu/~jain/cse567-06/ftp/encryption_perf/
I will give more detailed performance data in future. Absolutely it’s
bottleneck and will vastly affect the end to end performance.
AES(Advanced Encryption Standard) is recommended as a replacement of DES,
it’s more secure; with AES-NI support, the throughput can reach nearly
*2GB/s*, it won’t be the bottleneck any more, AES and CryptoCodec work is
supported in HADOOP-10150, HADOOP-10603 and HADOOP-10693 (We may need to add
a new mode support for AES).
This JIRA will use AES with AES-NI support as encryption algorithm for
DataTransferProtocol.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-6969) Archival Storage: INode#getStoragePolicyID should always return the latest storage policy
[ https://issues.apache.org/jira/browse/HDFS-6969?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao resolved HDFS-6969. - Resolution: Fixed Hadoop Flags: Reviewed Thanks again, Nicholas! I've committed this. Archival Storage: INode#getStoragePolicyID should always return the latest storage policy - Key: HDFS-6969 URL: https://issues.apache.org/jira/browse/HDFS-6969 Project: Hadoop HDFS Issue Type: Sub-task Components: balancer, namenode Reporter: Jing Zhao Assignee: Jing Zhao Attachments: HDFS-6969.000.patch, HDFS-6969.001.patch, HDFS-6969.002.patch In general, every file should only provide exact one storage policy for the Mover, no matter its snapshot states. Suppose a file /foo/bar, and it is contained in snapshots s1 and s2 of the root. If /foo/bar, /.snapshot/s1/foo/bar and /.snapshot/s2/foo/bar have different storage policies, when running Mover, we have to select one of the storage policies, among which the latest one should be the best. And if /foo/bar is deleted, we should still use its storage policy before the deletion, since the file deletion should not trigger data migration. Thus maybe what we can do is: 1. For a file with policy directly specified on it, alway follow the latest 2. Otherwise follow its latest parental path to identify its storage policy (simply following the parent link) -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6961) Archival Storage: BlockPlacementPolicy#chooseTarget should check each valid storage type in each choosing round
[ https://issues.apache.org/jira/browse/HDFS-6961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-6961: Attachment: HDFS-6961.002.patch Rebase the patch since HDFS-6969 just got committed. Archival Storage: BlockPlacementPolicy#chooseTarget should check each valid storage type in each choosing round --- Key: HDFS-6961 URL: https://issues.apache.org/jira/browse/HDFS-6961 Project: Hadoop HDFS Issue Type: Sub-task Components: balancer, namenode Reporter: Jing Zhao Assignee: Jing Zhao Attachments: HDFS-6961.000.patch, HDFS-6961.001.patch, HDFS-6961.002.patch, HDFS-6961.002.patch Currently in each choosing round of BlockPlacementPolicy#chooseTarget (e.g., round 1 for choosing local storage, round 2 for remote rack, etc.), we only take into account the first storage type in the storage type list. This may fail to choose the local machine or sometimes may even fail to choose enough nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6886) Use single editlog record for creating file + overwrite.
[ https://issues.apache.org/jira/browse/HDFS-6886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Yi Liu updated HDFS-6886: - Attachment: HDFS-6886.003.patch Update patch for [~vinayrpet]'s comments and make a git patch Use single editlog record for creating file + overwrite. Key: HDFS-6886 URL: https://issues.apache.org/jira/browse/HDFS-6886 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Reporter: Yi Liu Assignee: Yi Liu Priority: Critical Attachments: HDFS-6886.001.patch, HDFS-6886.002.patch, HDFS-6886.003.patch, editsStored As discussed in HDFS-6871, as [~jingzhao] and [~cmccabe]'s suggestion, we could do further improvement to use one editlog record for creating file + overwrite in this JIRA. We could record the overwrite flag in editlog for creating file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (HDFS-6980) TestWebHdfsFileSystemContract fails in trunk
[ https://issues.apache.org/jira/browse/HDFS-6980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA reassigned HDFS-6980: Assignee: Tsuyoshi OZAWA TestWebHdfsFileSystemContract fails in trunk Key: HDFS-6980 URL: https://issues.apache.org/jira/browse/HDFS-6980 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: Akira AJISAKA Assignee: Tsuyoshi OZAWA Many tests in TestWebHdfsFileSystemContract fail by too many open files error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6980) TestWebHdfsFileSystemContract fails in trunk
[ https://issues.apache.org/jira/browse/HDFS-6980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated HDFS-6980: - Attachment: HDFS-6980.1.patch Thanks for reporting, Akira. I found that there are lots file descriptor leaks in the test cases. Attached a patch to fix the problem. TestWebHdfsFileSystemContract fails in trunk Key: HDFS-6980 URL: https://issues.apache.org/jira/browse/HDFS-6980 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: Akira AJISAKA Assignee: Tsuyoshi OZAWA Attachments: HDFS-6980.1.patch Many tests in TestWebHdfsFileSystemContract fail by too many open files error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6980) TestWebHdfsFileSystemContract fails in trunk
[ https://issues.apache.org/jira/browse/HDFS-6980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated HDFS-6980: - Status: Patch Available (was: Open) TestWebHdfsFileSystemContract fails in trunk Key: HDFS-6980 URL: https://issues.apache.org/jira/browse/HDFS-6980 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: Akira AJISAKA Assignee: Tsuyoshi OZAWA Attachments: HDFS-6980.1.patch Many tests in TestWebHdfsFileSystemContract fail by too many open files error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6980) TestWebHdfsFileSystemContract fails in trunk
[
https://issues.apache.org/jira/browse/HDFS-6980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117871#comment-14117871
]
Hadoop QA commented on HDFS-6980:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12665831/HDFS-6980.1.patch
against trunk revision 258c7d0.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The test build failed in
hadoop-hdfs-project/hadoop-hdfs
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7868//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7868//console
This message is automatically generated.
TestWebHdfsFileSystemContract fails in trunk
Key: HDFS-6980
URL: https://issues.apache.org/jira/browse/HDFS-6980
Project: Hadoop HDFS
Issue Type: Bug
Components: test
Reporter: Akira AJISAKA
Assignee: Tsuyoshi OZAWA
Attachments: HDFS-6980.1.patch
Many tests in TestWebHdfsFileSystemContract fail by too many open files
error.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6886) Use single editlog record for creating file + overwrite.
[
https://issues.apache.org/jira/browse/HDFS-6886?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117870#comment-14117870
]
Hadoop QA commented on HDFS-6886:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12665828/HDFS-6886.003.patch
against trunk revision 258c7d0.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithEncryptedTransfer
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7867//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7867//console
This message is automatically generated.
Use single editlog record for creating file + overwrite.
Key: HDFS-6886
URL: https://issues.apache.org/jira/browse/HDFS-6886
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Reporter: Yi Liu
Assignee: Yi Liu
Priority: Critical
Attachments: HDFS-6886.001.patch, HDFS-6886.002.patch,
HDFS-6886.003.patch, editsStored
As discussed in HDFS-6871, as [~jingzhao] and [~cmccabe]'s suggestion, we
could do further improvement to use one editlog record for creating file +
overwrite in this JIRA. We could record the overwrite flag in editlog for
creating file.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-6980) TestWebHdfsFileSystemContract fails in trunk
[ https://issues.apache.org/jira/browse/HDFS-6980?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsuyoshi OZAWA updated HDFS-6980: - Attachment: HDFS-6980.1-2.patch TestWebHdfsFileSystemContract fails in trunk Key: HDFS-6980 URL: https://issues.apache.org/jira/browse/HDFS-6980 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: Akira AJISAKA Assignee: Tsuyoshi OZAWA Attachments: HDFS-6980.1-2.patch, HDFS-6980.1.patch Many tests in TestWebHdfsFileSystemContract fail by too many open files error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6980) TestWebHdfsFileSystemContract fails in trunk
[ https://issues.apache.org/jira/browse/HDFS-6980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14117913#comment-14117913 ] Tsuyoshi OZAWA commented on HDFS-6980: -- The test failure looks related to TestPipelinesFailover, and not related to this patch. To confirm this, resubmitting a same again. TestWebHdfsFileSystemContract fails in trunk Key: HDFS-6980 URL: https://issues.apache.org/jira/browse/HDFS-6980 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: Akira AJISAKA Assignee: Tsuyoshi OZAWA Attachments: HDFS-6980.1-2.patch, HDFS-6980.1.patch Many tests in TestWebHdfsFileSystemContract fail by too many open files error. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
