[jira] [Updated] (HDFS-6606) Optimize HDFS Encrypted Transport performance
[
https://issues.apache.org/jira/browse/HDFS-6606?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liu updated HDFS-6606:
-
Attachment: HDFS-6606.007.patch
Update patch to address all comments.
Optimize HDFS Encrypted Transport performance
-
Key: HDFS-6606
URL: https://issues.apache.org/jira/browse/HDFS-6606
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode, hdfs-client, security
Reporter: Yi Liu
Assignee: Yi Liu
Attachments: HDFS-6606.001.patch, HDFS-6606.002.patch,
HDFS-6606.003.patch, HDFS-6606.004.patch, HDFS-6606.005.patch,
HDFS-6606.006.patch, HDFS-6606.007.patch,
OptimizeHdfsEncryptedTransportperformance.pdf
In HDFS-3637, [~atm] added support for encrypting the DataTransferProtocol,
it was a great work.
It utilizes SASL {{Digest-MD5}} mechanism (use Qop: auth-conf), it supports
three security strength:
* high 3des or rc4 (128bits)
* medium des or rc4(56bits)
* low rc4(40bits)
3des and rc4 are slow, only *tens of MB/s*,
http://www.javamex.com/tutorials/cryptography/ciphers.shtml
http://www.cs.wustl.edu/~jain/cse567-06/ftp/encryption_perf/
I will give more detailed performance data in future. Absolutely it’s
bottleneck and will vastly affect the end to end performance.
AES(Advanced Encryption Standard) is recommended as a replacement of DES,
it’s more secure; with AES-NI support, the throughput can reach nearly
*2GB/s*, it won’t be the bottleneck any more, AES and CryptoCodec work is
supported in HADOOP-10150, HADOOP-10603 and HADOOP-10693 (We may need to add
a new mode support for AES).
This JIRA will use AES with AES-NI support as encryption algorithm for
DataTransferProtocol.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7122) Very poor distribution of replication copies
[ https://issues.apache.org/jira/browse/HDFS-7122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14145982#comment-14145982 ] Jeff Buell commented on HDFS-7122: -- Yi, yes I ran it many times tweaking various things to try to get a good distribution. Although there was some variation, it was always much worse than before. Very poor distribution of replication copies Key: HDFS-7122 URL: https://issues.apache.org/jira/browse/HDFS-7122 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.3.0 Environment: medium-large environments with 100's to 1000's of DNs will be most affected, but potentially all environments. Reporter: Jeff Buell Assignee: Andrew Wang Priority: Blocker Labels: performance Summary: Since HDFS-6268, the distribution of replica block copies across the DataNodes (replicas 2,3,... as distinguished from the first primary replica) is extremely poor, to the point that TeraGen slows down by as much as 3X for certain configurations. This is almost certainly due to the introduction of Thread Local Random in HDFS-6268. The mechanism appears to be that this change causes all the random numbers in the threads to be correlated, thus preventing a truly random choice of DN for each replica copy. Testing details: 1 TB TeraGen on 638 slave nodes (virtual machines on 32 physical hosts), 256MB block size. This results in 6 primary blocks on each DN. With replication=3, there will be on average 12 more copies on each DN that are copies of blocks from other DNs. Because of the random selection of DNs, exactly 12 copies are not expected, but I found that about 160 DNs (1/4 of all DNs!) received absolutely no copies, while one DN received over 100 copies, and the elapsed time increased by about 3X from a pre-HDFS-6268 distro. There was no pattern to which DNs didn't receive copies, nor was the set of such DNs repeatable run-to-run. In addition to the performance problem, there could be capacity problems due to one or a few DNs running out of space. Testing was done on CDH 5.0.0 (before) and CDH 5.1.2 (after), but I don't see a significant difference from the Apache Hadoop source in this regard. The workaround to recover the previous behavior is to set dfs.namenode.handler.count=1 but of course this has scaling implications for large clusters. I recommend that the ThreadLocal Random part of HDFS-6268 be reverted until a better algorithm can be implemented and tested. Testing should include a case with many DNs and a small number of blocks on each. It should also be noted that even pre-HDFS-6268, the random choice of DN algorithm produces a rather non-uniform distribution of copies. This is not due to any bug, but purely a case of random distributions being much less uniform than one might intuitively expect. In the above case, pre-HDFS-6268 yields something like a range of 3 to 25 block copies on each DN. Surprisingly, the performance penalty of this non-uniformity is not as big as might be expected (maybe only 10-20%), but HDFS should do better, and in any case the capacity issue remains. Round-robin choice of DN? Better awareness of which DNs currently store fewer blocks? It's not sufficient that the total number of blocks is similar on each DN at the end, but that at each point in time no individual DN receives a disproportionate number of blocks at once (which could be a danger of a RR algorithm). Probably should limit this jira to tracking the ThreadLocal issue, and track the random choice issue in another one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6994) libhdfs3 - A native C/C++ HDFS client
[ https://issues.apache.org/jira/browse/HDFS-6994?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhanwei Wang updated HDFS-6994: --- Issue Type: New Feature (was: Task) libhdfs3 - A native C/C++ HDFS client - Key: HDFS-6994 URL: https://issues.apache.org/jira/browse/HDFS-6994 Project: Hadoop HDFS Issue Type: New Feature Components: hdfs-client Reporter: Zhanwei Wang Attachments: HDFS-6994-rpc-8.patch, HDFS-6994.patch Hi All I just got the permission to open source libhdfs3, which is a native C/C++ HDFS client based on Hadoop RPC protocol and HDFS Data Transfer Protocol. libhdfs3 provide the libhdfs style C interface and a C++ interface. Support both HADOOP RPC version 8 and 9. Support Namenode HA and Kerberos authentication. libhdfs3 is currently used by HAWQ of Pivotal I'd like to integrate libhdfs3 into HDFS source code to benefit others. You can find libhdfs3 code from github https://github.com/PivotalRD/libhdfs3 http://pivotalrd.github.io/libhdfs3/ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6534) Fix build on macosx: HDFS parts
[
https://issues.apache.org/jira/browse/HDFS-6534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14145999#comment-14145999
]
Hadoop QA commented on HDFS-6534:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12670893/HDFS-6534.v3.patch
against trunk revision ef784a2.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8177//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8177//console
This message is automatically generated.
Fix build on macosx: HDFS parts
---
Key: HDFS-6534
URL: https://issues.apache.org/jira/browse/HDFS-6534
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Binglin Chang
Assignee: Binglin Chang
Priority: Minor
Attachments: HDFS-6534.v1.patch, HDFS-6534.v2.patch,
HDFS-6534.v3.patch
When compiling native code on macosx using clang, compiler find more warning
and errors which gcc ignores, those should be fixed.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7122) Very poor distribution of replication copies
[ https://issues.apache.org/jira/browse/HDFS-7122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146009#comment-14146009 ] Tsz Wo Nicholas Sze commented on HDFS-7122: --- Hi [~hitliuyi], one problem is that we are not yet sure how to fix the bug. Why not reverting HDFS-6268 and redo it in a right way? Reverting a patch is not a big deal. It is a quick and effective way to fix a newly introduced bug. Very poor distribution of replication copies Key: HDFS-7122 URL: https://issues.apache.org/jira/browse/HDFS-7122 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.3.0 Environment: medium-large environments with 100's to 1000's of DNs will be most affected, but potentially all environments. Reporter: Jeff Buell Assignee: Andrew Wang Priority: Blocker Labels: performance Summary: Since HDFS-6268, the distribution of replica block copies across the DataNodes (replicas 2,3,... as distinguished from the first primary replica) is extremely poor, to the point that TeraGen slows down by as much as 3X for certain configurations. This is almost certainly due to the introduction of Thread Local Random in HDFS-6268. The mechanism appears to be that this change causes all the random numbers in the threads to be correlated, thus preventing a truly random choice of DN for each replica copy. Testing details: 1 TB TeraGen on 638 slave nodes (virtual machines on 32 physical hosts), 256MB block size. This results in 6 primary blocks on each DN. With replication=3, there will be on average 12 more copies on each DN that are copies of blocks from other DNs. Because of the random selection of DNs, exactly 12 copies are not expected, but I found that about 160 DNs (1/4 of all DNs!) received absolutely no copies, while one DN received over 100 copies, and the elapsed time increased by about 3X from a pre-HDFS-6268 distro. There was no pattern to which DNs didn't receive copies, nor was the set of such DNs repeatable run-to-run. In addition to the performance problem, there could be capacity problems due to one or a few DNs running out of space. Testing was done on CDH 5.0.0 (before) and CDH 5.1.2 (after), but I don't see a significant difference from the Apache Hadoop source in this regard. The workaround to recover the previous behavior is to set dfs.namenode.handler.count=1 but of course this has scaling implications for large clusters. I recommend that the ThreadLocal Random part of HDFS-6268 be reverted until a better algorithm can be implemented and tested. Testing should include a case with many DNs and a small number of blocks on each. It should also be noted that even pre-HDFS-6268, the random choice of DN algorithm produces a rather non-uniform distribution of copies. This is not due to any bug, but purely a case of random distributions being much less uniform than one might intuitively expect. In the above case, pre-HDFS-6268 yields something like a range of 3 to 25 block copies on each DN. Surprisingly, the performance penalty of this non-uniformity is not as big as might be expected (maybe only 10-20%), but HDFS should do better, and in any case the capacity issue remains. Round-robin choice of DN? Better awareness of which DNs currently store fewer blocks? It's not sufficient that the total number of blocks is similar on each DN at the end, but that at each point in time no individual DN receives a disproportionate number of blocks at once (which could be a danger of a RR algorithm). Probably should limit this jira to tracking the ThreadLocal issue, and track the random choice issue in another one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6606) Optimize HDFS Encrypted Transport performance
[
https://issues.apache.org/jira/browse/HDFS-6606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146062#comment-14146062
]
Hadoop QA commented on HDFS-6606:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12670913/HDFS-6606.007.patch
against trunk revision ef784a2.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.crypto.random.TestOsSecureRandom
org.apache.hadoop.security.token.delegation.web.TestWebDelegationToken
org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract
The following test timeouts occurred in
hadoop-common-project/hadoop-common hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestDecommission
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8180//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/8180//artifact/PreCommit-HADOOP-Build-patchprocess/newPatchFindbugsWarningshadoop-common.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8180//console
This message is automatically generated.
Optimize HDFS Encrypted Transport performance
-
Key: HDFS-6606
URL: https://issues.apache.org/jira/browse/HDFS-6606
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode, hdfs-client, security
Reporter: Yi Liu
Assignee: Yi Liu
Attachments: HDFS-6606.001.patch, HDFS-6606.002.patch,
HDFS-6606.003.patch, HDFS-6606.004.patch, HDFS-6606.005.patch,
HDFS-6606.006.patch, HDFS-6606.007.patch,
OptimizeHdfsEncryptedTransportperformance.pdf
In HDFS-3637, [~atm] added support for encrypting the DataTransferProtocol,
it was a great work.
It utilizes SASL {{Digest-MD5}} mechanism (use Qop: auth-conf), it supports
three security strength:
* high 3des or rc4 (128bits)
* medium des or rc4(56bits)
* low rc4(40bits)
3des and rc4 are slow, only *tens of MB/s*,
http://www.javamex.com/tutorials/cryptography/ciphers.shtml
http://www.cs.wustl.edu/~jain/cse567-06/ftp/encryption_perf/
I will give more detailed performance data in future. Absolutely it’s
bottleneck and will vastly affect the end to end performance.
AES(Advanced Encryption Standard) is recommended as a replacement of DES,
it’s more secure; with AES-NI support, the throughput can reach nearly
*2GB/s*, it won’t be the bottleneck any more, AES and CryptoCodec work is
supported in HADOOP-10150, HADOOP-10603 and HADOOP-10693 (We may need to add
a new mode support for AES).
This JIRA will use AES with AES-NI support as encryption algorithm for
DataTransferProtocol.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7081) Add new DistributedFileSystem API for getting all the existing storage policies
[
https://issues.apache.org/jira/browse/HDFS-7081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146061#comment-14146061
]
Hadoop QA commented on HDFS-7081:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12670902/HDFS-7081.004.patch
against trunk revision ef784a2.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract
org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
org.apache.hadoop.hdfs.TestDFSFinalize
The following test timeouts occurred in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestDecommission
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8179//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8179//console
This message is automatically generated.
Add new DistributedFileSystem API for getting all the existing storage
policies
---
Key: HDFS-7081
URL: https://issues.apache.org/jira/browse/HDFS-7081
Project: Hadoop HDFS
Issue Type: Sub-task
Components: balancer, namenode
Reporter: Jing Zhao
Assignee: Jing Zhao
Attachments: HDFS-7081.000.patch, HDFS-7081.001.patch,
HDFS-7081.002.patch, HDFS-7081.003.patch, HDFS-7081.004.patch
Instead of loading all the policies from a client side configuration file, it
may be better to provide Mover with a new RPC call for getting all the
storage policies from the namenode.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7122) Very poor distribution of replication copies
[ https://issues.apache.org/jira/browse/HDFS-7122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146064#comment-14146064 ] Yi Liu commented on HDFS-7122: -- Agree with your point for reverting it, thanks [~szetszwo]. Very poor distribution of replication copies Key: HDFS-7122 URL: https://issues.apache.org/jira/browse/HDFS-7122 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.3.0 Environment: medium-large environments with 100's to 1000's of DNs will be most affected, but potentially all environments. Reporter: Jeff Buell Assignee: Andrew Wang Priority: Blocker Labels: performance Summary: Since HDFS-6268, the distribution of replica block copies across the DataNodes (replicas 2,3,... as distinguished from the first primary replica) is extremely poor, to the point that TeraGen slows down by as much as 3X for certain configurations. This is almost certainly due to the introduction of Thread Local Random in HDFS-6268. The mechanism appears to be that this change causes all the random numbers in the threads to be correlated, thus preventing a truly random choice of DN for each replica copy. Testing details: 1 TB TeraGen on 638 slave nodes (virtual machines on 32 physical hosts), 256MB block size. This results in 6 primary blocks on each DN. With replication=3, there will be on average 12 more copies on each DN that are copies of blocks from other DNs. Because of the random selection of DNs, exactly 12 copies are not expected, but I found that about 160 DNs (1/4 of all DNs!) received absolutely no copies, while one DN received over 100 copies, and the elapsed time increased by about 3X from a pre-HDFS-6268 distro. There was no pattern to which DNs didn't receive copies, nor was the set of such DNs repeatable run-to-run. In addition to the performance problem, there could be capacity problems due to one or a few DNs running out of space. Testing was done on CDH 5.0.0 (before) and CDH 5.1.2 (after), but I don't see a significant difference from the Apache Hadoop source in this regard. The workaround to recover the previous behavior is to set dfs.namenode.handler.count=1 but of course this has scaling implications for large clusters. I recommend that the ThreadLocal Random part of HDFS-6268 be reverted until a better algorithm can be implemented and tested. Testing should include a case with many DNs and a small number of blocks on each. It should also be noted that even pre-HDFS-6268, the random choice of DN algorithm produces a rather non-uniform distribution of copies. This is not due to any bug, but purely a case of random distributions being much less uniform than one might intuitively expect. In the above case, pre-HDFS-6268 yields something like a range of 3 to 25 block copies on each DN. Surprisingly, the performance penalty of this non-uniformity is not as big as might be expected (maybe only 10-20%), but HDFS should do better, and in any case the capacity issue remains. Round-robin choice of DN? Better awareness of which DNs currently store fewer blocks? It's not sufficient that the total number of blocks is similar on each DN at the end, but that at each point in time no individual DN receives a disproportionate number of blocks at once (which could be a danger of a RR algorithm). Probably should limit this jira to tracking the ThreadLocal issue, and track the random choice issue in another one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7122) Very poor distribution of replication copies
[
https://issues.apache.org/jira/browse/HDFS-7122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146079#comment-14146079
]
Yi Liu commented on HDFS-7122:
--
Actually before [~jcb] confirms the blocks distributions are always bad, I have
concern if the issue existed and was random even before HDFS-6268 since
{{java.util.Random}} is not good enough.
Now agree to revert it since it seems an issue after HDFS-6268 and we fix it
later.
Very poor distribution of replication copies
Key: HDFS-7122
URL: https://issues.apache.org/jira/browse/HDFS-7122
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.3.0
Environment: medium-large environments with 100's to 1000's of DNs
will be most affected, but potentially all environments.
Reporter: Jeff Buell
Assignee: Andrew Wang
Priority: Blocker
Labels: performance
Summary:
Since HDFS-6268, the distribution of replica block copies across the
DataNodes (replicas 2,3,... as distinguished from the first primary
replica) is extremely poor, to the point that TeraGen slows down by as much
as 3X for certain configurations. This is almost certainly due to the
introduction of Thread Local Random in HDFS-6268. The mechanism appears to
be that this change causes all the random numbers in the threads to be
correlated, thus preventing a truly random choice of DN for each replica copy.
Testing details:
1 TB TeraGen on 638 slave nodes (virtual machines on 32 physical hosts),
256MB block size. This results in 6 primary blocks on each DN. With
replication=3, there will be on average 12 more copies on each DN that are
copies of blocks from other DNs. Because of the random selection of DNs,
exactly 12 copies are not expected, but I found that about 160 DNs (1/4 of
all DNs!) received absolutely no copies, while one DN received over 100
copies, and the elapsed time increased by about 3X from a pre-HDFS-6268
distro. There was no pattern to which DNs didn't receive copies, nor was the
set of such DNs repeatable run-to-run. In addition to the performance
problem, there could be capacity problems due to one or a few DNs running out
of space. Testing was done on CDH 5.0.0 (before) and CDH 5.1.2 (after), but I
don't see a significant difference from the Apache Hadoop source in this
regard. The workaround to recover the previous behavior is to set
dfs.namenode.handler.count=1 but of course this has scaling implications for
large clusters.
I recommend that the ThreadLocal Random part of HDFS-6268 be reverted until a
better algorithm can be implemented and tested. Testing should include a
case with many DNs and a small number of blocks on each.
It should also be noted that even pre-HDFS-6268, the random choice of DN
algorithm produces a rather non-uniform distribution of copies. This is not
due to any bug, but purely a case of random distributions being much less
uniform than one might intuitively expect. In the above case, pre-HDFS-6268
yields something like a range of 3 to 25 block copies on each DN.
Surprisingly, the performance penalty of this non-uniformity is not as big as
might be expected (maybe only 10-20%), but HDFS should do better, and in any
case the capacity issue remains. Round-robin choice of DN? Better awareness
of which DNs currently store fewer blocks? It's not sufficient that the total
number of blocks is similar on each DN at the end, but that at each point in
time no individual DN receives a disproportionate number of blocks at once
(which could be a danger of a RR algorithm).
Probably should limit this jira to tracking the ThreadLocal issue, and track
the random choice issue in another one.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7081) Add new DistributedFileSystem API for getting all the existing storage policies
[ https://issues.apache.org/jira/browse/HDFS-7081?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146108#comment-14146108 ] Tsz Wo Nicholas Sze commented on HDFS-7081: --- Th latest patch only changed the XAttr.NameSpace from TRUSTED to SYSTEM. The failed tests are not related. +1 Add new DistributedFileSystem API for getting all the existing storage policies --- Key: HDFS-7081 URL: https://issues.apache.org/jira/browse/HDFS-7081 Project: Hadoop HDFS Issue Type: Sub-task Components: balancer, namenode Reporter: Jing Zhao Assignee: Jing Zhao Attachments: HDFS-7081.000.patch, HDFS-7081.001.patch, HDFS-7081.002.patch, HDFS-7081.003.patch, HDFS-7081.004.patch Instead of loading all the policies from a client side configuration file, it may be better to provide Mover with a new RPC call for getting all the storage policies from the namenode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7130) TestDataTransferKeepalive fails intermittently on Windows.
[
https://issues.apache.org/jira/browse/HDFS-7130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146214#comment-14146214
]
Hudson commented on HDFS-7130:
--
FAILURE: Integrated in Hadoop-Yarn-trunk #690 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/690/])
HDFS-7130. TestDataTransferKeepalive fails intermittently on Windows.
Contributed by Chris Nauroth. (cnauroth: rev
b93d9603a25b6b93f67c69503130164eef047876)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDataTransferKeepalive.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
TestDataTransferKeepalive fails intermittently on Windows.
--
Key: HDFS-7130
URL: https://issues.apache.org/jira/browse/HDFS-7130
Project: Hadoop HDFS
Issue Type: Bug
Components: test
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Fix For: 2.6.0
Attachments: HDFS-7130.1.patch
{{TestDataTransferKeepalive}} has failed intermittently on Windows. These
tests rely on a 1 ms thread sleep to wait for a cache expiration. This is
likely too short on Windows, which has been observed to have a less granular
clock interrupt period compared to typical Linux machines.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7126) TestEncryptionZonesWithHA assumes Unix path separator for KMS key store path
[ https://issues.apache.org/jira/browse/HDFS-7126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146209#comment-14146209 ] Hudson commented on HDFS-7126: -- FAILURE: Integrated in Hadoop-Yarn-trunk #690 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/690/]) HDFS-7126. TestEncryptionZonesWithHA assumes Unix path separator for KMS key store path. Contributed by Xiaoyu Yao. (cnauroth: rev a1fd804a314481065c849cb8e1d7c75494e24660) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZonesWithHA.java TestEncryptionZonesWithHA assumes Unix path separator for KMS key store path Key: HDFS-7126 URL: https://issues.apache.org/jira/browse/HDFS-7126 Project: Hadoop HDFS Issue Type: Test Components: security, test Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Priority: Minor Fix For: 2.6.0 Attachments: HDFS-7126.0.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7132) hdfs namenode -metadataVersion command does not honor configured name dirs
[ https://issues.apache.org/jira/browse/HDFS-7132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146212#comment-14146212 ] Hudson commented on HDFS-7132: -- FAILURE: Integrated in Hadoop-Yarn-trunk #690 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/690/]) HDFS-7132. hdfs namenode -metadataVersion command does not honor configured name dirs. Contributed by Charles Lamb. (wang: rev f48686a1ad81823000534665a76264bba51182f4) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestMetadataVersionOutput.java * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java hdfs namenode -metadataVersion command does not honor configured name dirs -- Key: HDFS-7132 URL: https://issues.apache.org/jira/browse/HDFS-7132 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.6.0 Reporter: Charles Lamb Assignee: Charles Lamb Priority: Minor Fix For: 2.6.0 Attachments: HDFS-7132.001.patch The hdfs namenode -metadataVersion command does not honor dfs.namenode.name.dir.nameservice.namenode configuration parameters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7132) hdfs namenode -metadataVersion command does not honor configured name dirs
[ https://issues.apache.org/jira/browse/HDFS-7132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146351#comment-14146351 ] Hudson commented on HDFS-7132: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1881 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1881/]) HDFS-7132. hdfs namenode -metadataVersion command does not honor configured name dirs. Contributed by Charles Lamb. (wang: rev f48686a1ad81823000534665a76264bba51182f4) * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestMetadataVersionOutput.java hdfs namenode -metadataVersion command does not honor configured name dirs -- Key: HDFS-7132 URL: https://issues.apache.org/jira/browse/HDFS-7132 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.6.0 Reporter: Charles Lamb Assignee: Charles Lamb Priority: Minor Fix For: 2.6.0 Attachments: HDFS-7132.001.patch The hdfs namenode -metadataVersion command does not honor dfs.namenode.name.dir.nameservice.namenode configuration parameters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7130) TestDataTransferKeepalive fails intermittently on Windows.
[
https://issues.apache.org/jira/browse/HDFS-7130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146353#comment-14146353
]
Hudson commented on HDFS-7130:
--
SUCCESS: Integrated in Hadoop-Hdfs-trunk #1881 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1881/])
HDFS-7130. TestDataTransferKeepalive fails intermittently on Windows.
Contributed by Chris Nauroth. (cnauroth: rev
b93d9603a25b6b93f67c69503130164eef047876)
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDataTransferKeepalive.java
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
TestDataTransferKeepalive fails intermittently on Windows.
--
Key: HDFS-7130
URL: https://issues.apache.org/jira/browse/HDFS-7130
Project: Hadoop HDFS
Issue Type: Bug
Components: test
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Fix For: 2.6.0
Attachments: HDFS-7130.1.patch
{{TestDataTransferKeepalive}} has failed intermittently on Windows. These
tests rely on a 1 ms thread sleep to wait for a cache expiration. This is
likely too short on Windows, which has been observed to have a less granular
clock interrupt period compared to typical Linux machines.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7126) TestEncryptionZonesWithHA assumes Unix path separator for KMS key store path
[ https://issues.apache.org/jira/browse/HDFS-7126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146348#comment-14146348 ] Hudson commented on HDFS-7126: -- SUCCESS: Integrated in Hadoop-Hdfs-trunk #1881 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1881/]) HDFS-7126. TestEncryptionZonesWithHA assumes Unix path separator for KMS key store path. Contributed by Xiaoyu Yao. (cnauroth: rev a1fd804a314481065c849cb8e1d7c75494e24660) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZonesWithHA.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt TestEncryptionZonesWithHA assumes Unix path separator for KMS key store path Key: HDFS-7126 URL: https://issues.apache.org/jira/browse/HDFS-7126 Project: Hadoop HDFS Issue Type: Test Components: security, test Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Priority: Minor Fix For: 2.6.0 Attachments: HDFS-7126.0.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6826) Plugin interface to enable delegation of HDFS authorization assertions
[
https://issues.apache.org/jira/browse/HDFS-6826?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146399#comment-14146399
]
Daryn Sharp commented on HDFS-6826:
---
My main concern and opposition is the current approach so tightly couples the
plugin with the NN's current implementation that it will impede future
features. Too many implementation details are leaked or exposed for plugins
that ought not be. I have many changes related to combined path resolution,
locking, and permission checks that will be jeopardized.
Specific issues:
# It's all too tightly coupled to the current permission checker implementation
# The plugin will break every time a new namesystem method is added
# Block operations shouldn't be going through the plugin, should they?
# MAJOR: Splicing at the {{ClientNamenodeProtocolServerSideTranslatorPB}} level
allows webhdfs to sidestep the plugin
# Thread locals aren't always performant and they are used heavily primarily to
allow the plugin to avoid blowing up edit log processing because...
# Splicing into inodes is too low of a level. All authz checks should have
already been performed.
# Why splice into inode setters since checked exceptions cannot be thrown?
Silently ignoring the set will discombobulate the user. Throwing a runtime
exception may leave the namesystem in an inconsistent state, thus I don't see
the value add.
# Setters in {{DefaultAuthorizationProvider}} are too fragile. It's exposing
too many implementation details as to how inode information is encoded - today.
# Using a static for the provider allows one and only one custom authz
provider. What if the user wants to use _both_ sentry and argus?
Now let's look at the two use cases, correct me if I'm wrong:
* Argus: wants to augment the permission model for acls that cannot be
represented in hdfs acls
* Sentry: wants to return fake user/group/perms to avoid having other stack
components own their backing files
Based our offline discussion of Argus, it should be the easiest to implement
based on the explicit agreement it will only augment the permission model.
Just needs an additional callout.
The Sentry use case is a bit dubious but generally simple to add in a manner
that doesn't complicate future changes. It just needs to substitute inode
attributes prior to the standard permission checks. The file status change may
seem like a hack (today) but it will dovetail with fsdir changes to only
operate on pre-resolved inodes - not re-resolved paths. File status will
simply return the attributes of the last node which will conveniently be the
dummy values already substituted by the plugin.
Plugin interface to enable delegation of HDFS authorization assertions
--
Key: HDFS-6826
URL: https://issues.apache.org/jira/browse/HDFS-6826
Project: Hadoop HDFS
Issue Type: New Feature
Components: security
Affects Versions: 2.4.1
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Attachments: HDFS-6826-idea.patch, HDFS-6826-idea2.patch,
HDFS-6826-permchecker.patch, HDFS-6826v3.patch, HDFS-6826v4.patch,
HDFS-6826v5.patch, HDFS-6826v6.patch, HDFS-6826v7.1.patch,
HDFS-6826v7.2.patch, HDFS-6826v7.3.patch, HDFS-6826v7.4.patch,
HDFS-6826v7.5.patch, HDFS-6826v7.6.patch, HDFS-6826v7.patch,
HDFS-6826v8.patch, HDFS-6826v9.patch,
HDFSPluggableAuthorizationProposal-v2.pdf,
HDFSPluggableAuthorizationProposal.pdf
When Hbase data, HiveMetaStore data or Search data is accessed via services
(Hbase region servers, HiveServer2, Impala, Solr) the services can enforce
permissions on corresponding entities (databases, tables, views, columns,
search collections, documents). It is desirable, when the data is accessed
directly by users accessing the underlying data files (i.e. from a MapReduce
job), that the permission of the data files map to the permissions of the
corresponding data entity (i.e. table, column family or search collection).
To enable this we need to have the necessary hooks in place in the NameNode
to delegate authorization to an external system that can map HDFS
files/directories to data entities and resolve their permissions based on the
data entities permissions.
I’ll be posting a design proposal in the next few days.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (HDFS-7138) Fix hftp to work with encryption
Charles Lamb created HDFS-7138: -- Summary: Fix hftp to work with encryption Key: HDFS-7138 URL: https://issues.apache.org/jira/browse/HDFS-7138 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 2.6.0 Reporter: Charles Lamb Assignee: Charles Lamb Users cannot read files using hftp. bash-4.1$ hadoop fs -cat /enc/hello hello and goodbye bash-4.1$ hadoop fs -cat hftp://localhost:50070/enc/hello 14/09/16 10:56:52 WARN ssl.FileBasedKeyStoresFactory: The property 'ssl.client.truststore.location' has not been set, no TrustStore will be loaded P5/-?K?fbash-4.1$ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6534) Fix build on macosx: HDFS parts
[ https://issues.apache.org/jira/browse/HDFS-6534?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146421#comment-14146421 ] Allen Wittenauer commented on HDFS-6534: +1 LGTM. Committing to branch-2 and trunk. Thanks! Fix build on macosx: HDFS parts --- Key: HDFS-6534 URL: https://issues.apache.org/jira/browse/HDFS-6534 Project: Hadoop HDFS Issue Type: Bug Reporter: Binglin Chang Assignee: Binglin Chang Priority: Minor Fix For: 2.6.0 Attachments: HDFS-6534.v1.patch, HDFS-6534.v2.patch, HDFS-6534.v3.patch When compiling native code on macosx using clang, compiler find more warning and errors which gcc ignores, those should be fixed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6534) Fix build on macosx: HDFS parts
[ https://issues.apache.org/jira/browse/HDFS-6534?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer updated HDFS-6534: --- Resolution: Fixed Fix Version/s: 2.6.0 Status: Resolved (was: Patch Available) Fix build on macosx: HDFS parts --- Key: HDFS-6534 URL: https://issues.apache.org/jira/browse/HDFS-6534 Project: Hadoop HDFS Issue Type: Bug Reporter: Binglin Chang Assignee: Binglin Chang Priority: Minor Fix For: 2.6.0 Attachments: HDFS-6534.v1.patch, HDFS-6534.v2.patch, HDFS-6534.v3.patch When compiling native code on macosx using clang, compiler find more warning and errors which gcc ignores, those should be fixed. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7138) Fix hftp to work with encryption
[ https://issues.apache.org/jira/browse/HDFS-7138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HDFS-7138: --- Attachment: HDFS-7138.001.patch The attached patch changes namenode/StreamFile.java to call createWrappedInputStream during the open call. This causes hftp to be able to read encrypted files. The unit test for StreamFile was also changed to reflect the change from FSInputStream args to FSDataInputStream. Fix hftp to work with encryption Key: HDFS-7138 URL: https://issues.apache.org/jira/browse/HDFS-7138 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 2.6.0 Reporter: Charles Lamb Assignee: Charles Lamb Attachments: HDFS-7138.001.patch Users cannot read files using hftp. bash-4.1$ hadoop fs -cat /enc/hello hello and goodbye bash-4.1$ hadoop fs -cat hftp://localhost:50070/enc/hello 14/09/16 10:56:52 WARN ssl.FileBasedKeyStoresFactory: The property 'ssl.client.truststore.location' has not been set, no TrustStore will be loaded P5/-?K?fbash-4.1$ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7138) Fix hftp to work with encryption
[ https://issues.apache.org/jira/browse/HDFS-7138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HDFS-7138: --- Status: Patch Available (was: Open) Fix hftp to work with encryption Key: HDFS-7138 URL: https://issues.apache.org/jira/browse/HDFS-7138 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 2.6.0 Reporter: Charles Lamb Assignee: Charles Lamb Attachments: HDFS-7138.001.patch Users cannot read files using hftp. bash-4.1$ hadoop fs -cat /enc/hello hello and goodbye bash-4.1$ hadoop fs -cat hftp://localhost:50070/enc/hello 14/09/16 10:56:52 WARN ssl.FileBasedKeyStoresFactory: The property 'ssl.client.truststore.location' has not been set, no TrustStore will be loaded P5/-?K?fbash-4.1$ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7138) Fix hftp to work with encryption
[
https://issues.apache.org/jira/browse/HDFS-7138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146458#comment-14146458
]
Hadoop QA commented on HDFS-7138:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12670981/HDFS-7138.001.patch
against trunk revision 034df0e.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8181//console
This message is automatically generated.
Fix hftp to work with encryption
Key: HDFS-7138
URL: https://issues.apache.org/jira/browse/HDFS-7138
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Affects Versions: 2.6.0
Reporter: Charles Lamb
Assignee: Charles Lamb
Attachments: HDFS-7138.001.patch
Users cannot read files using hftp.
bash-4.1$ hadoop fs -cat /enc/hello
hello and goodbye
bash-4.1$ hadoop fs -cat hftp://localhost:50070/enc/hello
14/09/16 10:56:52 WARN ssl.FileBasedKeyStoresFactory: The property
'ssl.client.truststore.location' has not been set, no TrustStore will be
loaded
P5/-?K?fbash-4.1$
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7138) Fix hftp to work with encryption
[ https://issues.apache.org/jira/browse/HDFS-7138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146476#comment-14146476 ] Charles Lamb commented on HDFS-7138: I should have mentioned that the patch is for branch-2 and that's why it wouldn't apply for test-patch. I ran TestHftpFileSystem on my local machine (as well as TestStreamFile) and both worked fine with the patch. Fix hftp to work with encryption Key: HDFS-7138 URL: https://issues.apache.org/jira/browse/HDFS-7138 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 2.6.0 Reporter: Charles Lamb Assignee: Charles Lamb Attachments: HDFS-7138.001.patch Users cannot read files using hftp. bash-4.1$ hadoop fs -cat /enc/hello hello and goodbye bash-4.1$ hadoop fs -cat hftp://localhost:50070/enc/hello 14/09/16 10:56:52 WARN ssl.FileBasedKeyStoresFactory: The property 'ssl.client.truststore.location' has not been set, no TrustStore will be loaded P5/-?K?fbash-4.1$ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7130) TestDataTransferKeepalive fails intermittently on Windows.
[
https://issues.apache.org/jira/browse/HDFS-7130?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146514#comment-14146514
]
Hudson commented on HDFS-7130:
--
FAILURE: Integrated in Hadoop-Mapreduce-trunk #1906 (See
[https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1906/])
HDFS-7130. TestDataTransferKeepalive fails intermittently on Windows.
Contributed by Chris Nauroth. (cnauroth: rev
b93d9603a25b6b93f67c69503130164eef047876)
* hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
*
hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestDataTransferKeepalive.java
TestDataTransferKeepalive fails intermittently on Windows.
--
Key: HDFS-7130
URL: https://issues.apache.org/jira/browse/HDFS-7130
Project: Hadoop HDFS
Issue Type: Bug
Components: test
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Fix For: 2.6.0
Attachments: HDFS-7130.1.patch
{{TestDataTransferKeepalive}} has failed intermittently on Windows. These
tests rely on a 1 ms thread sleep to wait for a cache expiration. This is
likely too short on Windows, which has been observed to have a less granular
clock interrupt period compared to typical Linux machines.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7132) hdfs namenode -metadataVersion command does not honor configured name dirs
[ https://issues.apache.org/jira/browse/HDFS-7132?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146512#comment-14146512 ] Hudson commented on HDFS-7132: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1906 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1906/]) HDFS-7132. hdfs namenode -metadataVersion command does not honor configured name dirs. Contributed by Charles Lamb. (wang: rev f48686a1ad81823000534665a76264bba51182f4) * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestMetadataVersionOutput.java hdfs namenode -metadataVersion command does not honor configured name dirs -- Key: HDFS-7132 URL: https://issues.apache.org/jira/browse/HDFS-7132 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.6.0 Reporter: Charles Lamb Assignee: Charles Lamb Priority: Minor Fix For: 2.6.0 Attachments: HDFS-7132.001.patch The hdfs namenode -metadataVersion command does not honor dfs.namenode.name.dir.nameservice.namenode configuration parameters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7126) TestEncryptionZonesWithHA assumes Unix path separator for KMS key store path
[ https://issues.apache.org/jira/browse/HDFS-7126?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146509#comment-14146509 ] Hudson commented on HDFS-7126: -- FAILURE: Integrated in Hadoop-Mapreduce-trunk #1906 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1906/]) HDFS-7126. TestEncryptionZonesWithHA assumes Unix path separator for KMS key store path. Contributed by Xiaoyu Yao. (cnauroth: rev a1fd804a314481065c849cb8e1d7c75494e24660) * hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestEncryptionZonesWithHA.java * hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt TestEncryptionZonesWithHA assumes Unix path separator for KMS key store path Key: HDFS-7126 URL: https://issues.apache.org/jira/browse/HDFS-7126 Project: Hadoop HDFS Issue Type: Test Components: security, test Reporter: Xiaoyu Yao Assignee: Xiaoyu Yao Priority: Minor Fix For: 2.6.0 Attachments: HDFS-7126.0.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7081) Add new DistributedFileSystem API for getting all the existing storage policies
[ https://issues.apache.org/jira/browse/HDFS-7081?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-7081: Resolution: Fixed Fix Version/s: 3.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) I've committed this to trunk. Add new DistributedFileSystem API for getting all the existing storage policies --- Key: HDFS-7081 URL: https://issues.apache.org/jira/browse/HDFS-7081 Project: Hadoop HDFS Issue Type: Sub-task Components: balancer, namenode Reporter: Jing Zhao Assignee: Jing Zhao Fix For: 3.0.0 Attachments: HDFS-7081.000.patch, HDFS-7081.001.patch, HDFS-7081.002.patch, HDFS-7081.003.patch, HDFS-7081.004.patch Instead of loading all the policies from a client side configuration file, it may be better to provide Mover with a new RPC call for getting all the storage policies from the namenode. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7011) Implement basic utilities for libhdfs3
[ https://issues.apache.org/jira/browse/HDFS-7011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146548#comment-14146548 ] Colin Patrick McCabe commented on HDFS-7011: bq. As discussed before, libhdfs3 is not a good name and should be changed. libhdfs only provide C interface and do not have namespace issue. OK. I can put it in namespace hdfs for now. Please, let's put off the library name discussion until later or else we'll never finish :) bq. I think it is necessary to keep the copyright history since not like other code in HDFS, libhdfs3 was proprietary software before open source. Apache's policy is very clear here. See http://www.apache.org/legal/src-headers.html, the section labelled Source File Headers for Code Developed at the ASF bq. If the source file is submitted with a copyright notice included in it, the copyright owner (or owner's agent) must either: remove such notices, or move them to the NOTICE file associated with each applicable project release, or provide written permission for the ASF to make such removal or relocation of the notices. Each source file should include the following license header -- note that there should be no copyright notice in the header: \[apache 2.0 license header\] We also don't permit author tags (Jenkins specifically checks for and -1s any patches that contain them). bq. I block many signals in backend worker thread, these signals should only be handled in main thread. libhdfs3 will not raise SIGPIPE at all. I agree that SIGPIPE should not be raised if the proper precautions are taken during send, write, etc. But since we know we don't ever want SIGPIPE, why not block it? Is the reason you want to block the other signals in the worker thread that this will allow an application that wants to handle them specifically will be able to do so? That makes sense, and I will add back the other signal blocking. Implement basic utilities for libhdfs3 -- Key: HDFS-7011 URL: https://issues.apache.org/jira/browse/HDFS-7011 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs-client Reporter: Zhanwei Wang Assignee: Colin Patrick McCabe Attachments: HDFS-7011-pnative.001.patch, HDFS-7011.patch Implement basic utilities such as hash, exception handling, logger, configure parser, checksum calculate and so on. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6877) Avoid calling checkDisk and stop active BlockReceiver thread when an HDFS volume is removed during a write.
[ https://issues.apache.org/jira/browse/HDFS-6877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146575#comment-14146575 ] Colin Patrick McCabe commented on HDFS-6877: bq. I anticipated a behavior that the application is writing the block in very slow speed. For instance, a light traffic service opens a log file and only appends 10KB logs after waking up for every 30 seconds? Or some applications that dump immediate results (e.g., 1MB) after every 10 minutes? In such cases, the BlockReceiver thread can live for a long time, until the block is full and then be closed. I'm sorry, but your understanding is not correct. The BlockReceiver processes messages of about 64kb. If the DFSOutputStream doesn't have enough to send a message, no message is sent. It doesn't matter how slow the client is, the BlockReceiver will be fast. So please get rid of this interruption stuff. Avoid calling checkDisk and stop active BlockReceiver thread when an HDFS volume is removed during a write. --- Key: HDFS-6877 URL: https://issues.apache.org/jira/browse/HDFS-6877 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Affects Versions: 2.5.0 Reporter: Lei (Eddy) Xu Assignee: Lei (Eddy) Xu Attachments: HDFS-6877.000.consolidate.txt, HDFS-6877.000.delta-HDFS-6727.txt, HDFS-6877.001.combo.txt, HDFS-6877.001.patch, HDFS-6877.002.patch, HDFS-6877.003.patch, HDFS-6877.004.patch, HDFS-6877.005.patch Avoid calling checkDisk and stop active BlockReceiver thread when an HDFS volume is removed during a write. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7139) Unit test for creating encryption zone on root path
Zhe Zhang created HDFS-7139: --- Summary: Unit test for creating encryption zone on root path Key: HDFS-7139 URL: https://issues.apache.org/jira/browse/HDFS-7139 Project: Hadoop HDFS Issue Type: Improvement Reporter: Zhe Zhang Assignee: Zhe Zhang Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7139) Unit test for creating encryption zone on root path
[ https://issues.apache.org/jira/browse/HDFS-7139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-7139: Issue Type: Sub-task (was: Improvement) Parent: HDFS-6891 Unit test for creating encryption zone on root path --- Key: HDFS-7139 URL: https://issues.apache.org/jira/browse/HDFS-7139 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Zhe Zhang Assignee: Zhe Zhang Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6877) Avoid calling checkDisk and stop active BlockReceiver thread when an HDFS volume is removed during a write.
[ https://issues.apache.org/jira/browse/HDFS-6877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lei (Eddy) Xu updated HDFS-6877: Attachment: HDFS-6877.006.patch [~cmccabe] cool, thanks for pointing this out. I've reverted the changes and only addressed your last comments. Avoid calling checkDisk and stop active BlockReceiver thread when an HDFS volume is removed during a write. --- Key: HDFS-6877 URL: https://issues.apache.org/jira/browse/HDFS-6877 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Affects Versions: 2.5.0 Reporter: Lei (Eddy) Xu Assignee: Lei (Eddy) Xu Attachments: HDFS-6877.000.consolidate.txt, HDFS-6877.000.delta-HDFS-6727.txt, HDFS-6877.001.combo.txt, HDFS-6877.001.patch, HDFS-6877.002.patch, HDFS-6877.003.patch, HDFS-6877.004.patch, HDFS-6877.005.patch, HDFS-6877.006.patch Avoid calling checkDisk and stop active BlockReceiver thread when an HDFS volume is removed during a write. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7139) Unit test for creating encryption zone on root path
[ https://issues.apache.org/jira/browse/HDFS-7139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-7139: Status: Patch Available (was: Open) Unit test for creating encryption zone on root path --- Key: HDFS-7139 URL: https://issues.apache.org/jira/browse/HDFS-7139 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Zhe Zhang Assignee: Zhe Zhang Priority: Minor Attachments: HDFS-7139.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7139) Unit test for creating encryption zone on root path
[ https://issues.apache.org/jira/browse/HDFS-7139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-7139: Attachment: HDFS-7139.patch Unit test for creating encryption zone on root path --- Key: HDFS-7139 URL: https://issues.apache.org/jira/browse/HDFS-7139 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Zhe Zhang Assignee: Zhe Zhang Priority: Minor Attachments: HDFS-7139.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7139) Unit test for creating encryption zone on root path
[ https://issues.apache.org/jira/browse/HDFS-7139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146595#comment-14146595 ] Charles Lamb commented on HDFS-7139: Thanks for handling this Zhe. I think it might be a good idea to actually check the FileStatus.isEncrypted() bit on both the dir and the file. Also, it might be good to verify that the file and the /.reserved/raw version of the file are not equal. DFSTestUtils.verifyFilesNotEqual is useful for that. Unit test for creating encryption zone on root path --- Key: HDFS-7139 URL: https://issues.apache.org/jira/browse/HDFS-7139 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Zhe Zhang Assignee: Zhe Zhang Priority: Minor Attachments: HDFS-7139.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7122) Very poor distribution of replication copies
[ https://issues.apache.org/jira/browse/HDFS-7122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146596#comment-14146596 ] Andrew Wang commented on HDFS-7122: --- Can we hold off on the revert for now? There are a couple other JIRAs layered on top of HDFS-6268 right now, and as I said above, I'm working on this pretty actively and I think we can get it done by 2.6. Very poor distribution of replication copies Key: HDFS-7122 URL: https://issues.apache.org/jira/browse/HDFS-7122 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.3.0 Environment: medium-large environments with 100's to 1000's of DNs will be most affected, but potentially all environments. Reporter: Jeff Buell Assignee: Andrew Wang Priority: Blocker Labels: performance Summary: Since HDFS-6268, the distribution of replica block copies across the DataNodes (replicas 2,3,... as distinguished from the first primary replica) is extremely poor, to the point that TeraGen slows down by as much as 3X for certain configurations. This is almost certainly due to the introduction of Thread Local Random in HDFS-6268. The mechanism appears to be that this change causes all the random numbers in the threads to be correlated, thus preventing a truly random choice of DN for each replica copy. Testing details: 1 TB TeraGen on 638 slave nodes (virtual machines on 32 physical hosts), 256MB block size. This results in 6 primary blocks on each DN. With replication=3, there will be on average 12 more copies on each DN that are copies of blocks from other DNs. Because of the random selection of DNs, exactly 12 copies are not expected, but I found that about 160 DNs (1/4 of all DNs!) received absolutely no copies, while one DN received over 100 copies, and the elapsed time increased by about 3X from a pre-HDFS-6268 distro. There was no pattern to which DNs didn't receive copies, nor was the set of such DNs repeatable run-to-run. In addition to the performance problem, there could be capacity problems due to one or a few DNs running out of space. Testing was done on CDH 5.0.0 (before) and CDH 5.1.2 (after), but I don't see a significant difference from the Apache Hadoop source in this regard. The workaround to recover the previous behavior is to set dfs.namenode.handler.count=1 but of course this has scaling implications for large clusters. I recommend that the ThreadLocal Random part of HDFS-6268 be reverted until a better algorithm can be implemented and tested. Testing should include a case with many DNs and a small number of blocks on each. It should also be noted that even pre-HDFS-6268, the random choice of DN algorithm produces a rather non-uniform distribution of copies. This is not due to any bug, but purely a case of random distributions being much less uniform than one might intuitively expect. In the above case, pre-HDFS-6268 yields something like a range of 3 to 25 block copies on each DN. Surprisingly, the performance penalty of this non-uniformity is not as big as might be expected (maybe only 10-20%), but HDFS should do better, and in any case the capacity issue remains. Round-robin choice of DN? Better awareness of which DNs currently store fewer blocks? It's not sufficient that the total number of blocks is similar on each DN at the end, but that at each point in time no individual DN receives a disproportionate number of blocks at once (which could be a danger of a RR algorithm). Probably should limit this jira to tracking the ThreadLocal issue, and track the random choice issue in another one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6987) Move CipherSuite xattr information up to the encryption zone root
[ https://issues.apache.org/jira/browse/HDFS-6987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146607#comment-14146607 ] Andrew Wang commented on HDFS-6987: --- Hey Suresh, in terms of the must-dos before a release, I'd say this one, HDFS-7077 (almost ready), and HADOOP-3 (also actively being worked on). Charles is working on HFTP support and we've got a few other nice-to-haves, but I think that's basically it. Move CipherSuite xattr information up to the encryption zone root - Key: HDFS-6987 URL: https://issues.apache.org/jira/browse/HDFS-6987 Project: Hadoop HDFS Issue Type: Sub-task Components: encryption Reporter: Andrew Wang Assignee: Zhe Zhang Fix For: 3.0.0 Attachments: HDFS-6987-20140917-v1.patch, HDFS-6987-20140918-v1.patch, HDFS-6987-20140919-v1.patch, HDFS-6987-20140919-v2.patch, HDFS-6987-20140919-v3.patch, HDFS-6987-20140920-v1.patch, HDFS-6987-branch-2-v2.patch All files within a single EZ need to be encrypted with the same CipherSuite. Because of this, I think we can store the CipherSuite once in the EZ rather than on each file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-6877) Avoid calling checkDisk when an HDFS volume is removed during a write.
[ https://issues.apache.org/jira/browse/HDFS-6877?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lei (Eddy) Xu updated HDFS-6877: Summary: Avoid calling checkDisk when an HDFS volume is removed during a write. (was: Avoid calling checkDisk and stop active BlockReceiver thread when an HDFS volume is removed during a write.) Avoid calling checkDisk when an HDFS volume is removed during a write. -- Key: HDFS-6877 URL: https://issues.apache.org/jira/browse/HDFS-6877 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Affects Versions: 2.5.0 Reporter: Lei (Eddy) Xu Assignee: Lei (Eddy) Xu Attachments: HDFS-6877.000.consolidate.txt, HDFS-6877.000.delta-HDFS-6727.txt, HDFS-6877.001.combo.txt, HDFS-6877.001.patch, HDFS-6877.002.patch, HDFS-6877.003.patch, HDFS-6877.004.patch, HDFS-6877.005.patch, HDFS-6877.006.patch Avoid calling checkDisk and stop active BlockReceiver thread when an HDFS volume is removed during a write. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7122) Very poor distribution of replication copies
[ https://issues.apache.org/jira/browse/HDFS-7122?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146615#comment-14146615 ] Suresh Srinivas commented on HDFS-7122: --- [~andrew.wang], my preference is to revert the patch and commit the patch back with all the required fixes. But I will leave it up to you (reluctantly), if you think this can be fixed in a couple of days. Very poor distribution of replication copies Key: HDFS-7122 URL: https://issues.apache.org/jira/browse/HDFS-7122 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.3.0 Environment: medium-large environments with 100's to 1000's of DNs will be most affected, but potentially all environments. Reporter: Jeff Buell Assignee: Andrew Wang Priority: Blocker Labels: performance Summary: Since HDFS-6268, the distribution of replica block copies across the DataNodes (replicas 2,3,... as distinguished from the first primary replica) is extremely poor, to the point that TeraGen slows down by as much as 3X for certain configurations. This is almost certainly due to the introduction of Thread Local Random in HDFS-6268. The mechanism appears to be that this change causes all the random numbers in the threads to be correlated, thus preventing a truly random choice of DN for each replica copy. Testing details: 1 TB TeraGen on 638 slave nodes (virtual machines on 32 physical hosts), 256MB block size. This results in 6 primary blocks on each DN. With replication=3, there will be on average 12 more copies on each DN that are copies of blocks from other DNs. Because of the random selection of DNs, exactly 12 copies are not expected, but I found that about 160 DNs (1/4 of all DNs!) received absolutely no copies, while one DN received over 100 copies, and the elapsed time increased by about 3X from a pre-HDFS-6268 distro. There was no pattern to which DNs didn't receive copies, nor was the set of such DNs repeatable run-to-run. In addition to the performance problem, there could be capacity problems due to one or a few DNs running out of space. Testing was done on CDH 5.0.0 (before) and CDH 5.1.2 (after), but I don't see a significant difference from the Apache Hadoop source in this regard. The workaround to recover the previous behavior is to set dfs.namenode.handler.count=1 but of course this has scaling implications for large clusters. I recommend that the ThreadLocal Random part of HDFS-6268 be reverted until a better algorithm can be implemented and tested. Testing should include a case with many DNs and a small number of blocks on each. It should also be noted that even pre-HDFS-6268, the random choice of DN algorithm produces a rather non-uniform distribution of copies. This is not due to any bug, but purely a case of random distributions being much less uniform than one might intuitively expect. In the above case, pre-HDFS-6268 yields something like a range of 3 to 25 block copies on each DN. Surprisingly, the performance penalty of this non-uniformity is not as big as might be expected (maybe only 10-20%), but HDFS should do better, and in any case the capacity issue remains. Round-robin choice of DN? Better awareness of which DNs currently store fewer blocks? It's not sufficient that the total number of blocks is similar on each DN at the end, but that at each point in time no individual DN receives a disproportionate number of blocks at once (which could be a danger of a RR algorithm). Probably should limit this jira to tracking the ThreadLocal issue, and track the random choice issue in another one. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7139) Unit test for creating encryption zone on root path
[
https://issues.apache.org/jira/browse/HDFS-7139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146624#comment-14146624
]
Hadoop QA commented on HDFS-7139:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12670997/HDFS-7139.patch
against trunk revision 073bbd8.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8183//console
This message is automatically generated.
Unit test for creating encryption zone on root path
---
Key: HDFS-7139
URL: https://issues.apache.org/jira/browse/HDFS-7139
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Priority: Minor
Attachments: HDFS-7139.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7104) Fix and clarify INodeInPath getter functions
[
https://issues.apache.org/jira/browse/HDFS-7104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-7104:
Attachment: HDFS-7104-20140924-v1.patch
{{TestSnapshotPathINodes#testSnapshotPathINodes}} originally calls
{{getINodes()}} to trim the {{inodes}} array, and then verifies that
{{getLastINode}} returns the last _regular_ inode (not null inode introduced by
dot-snapshot paths). This is the wrong way of using {{getINodes}}, which is
only a getter.
Fix and clarify INodeInPath getter functions
Key: HDFS-7104
URL: https://issues.apache.org/jira/browse/HDFS-7104
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Priority: Minor
Attachments: HDFS-7104-20140923-v1.patch,
HDFS-7104-20140923-v3.patch, HDFS-7104-20140924-v1.patch
inodes is initialized with the number of patch components. After resolve, it
contains both non-null and null elements (introduced by dot-snapshot dirs).
When getINodes is called, an array is returned excluding all non elements,
which is the correct behavior. Meanwhile, the inodes array is trimmed too,
which shouldn't be done by a getter.
Because of the above, the behavior of getINodesInPath depends on whether
getINodes has been called, which is not correct.
The name of getLastINodeInPath is confusing – it actually returns the last
non-null inode in the path. Also, shouldn't the return type be a single INode?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6894) Add XDR parser method for each NFS response
[
https://issues.apache.org/jira/browse/HDFS-6894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146636#comment-14146636
]
Hadoop QA commented on HDFS-6894:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12670129/HDFS-6894.001.patch
against trunk revision 073bbd8.
{color:red}-1 patch{color}. Trunk compilation may be broken.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8184//console
This message is automatically generated.
Add XDR parser method for each NFS response
---
Key: HDFS-6894
URL: https://issues.apache.org/jira/browse/HDFS-6894
Project: Hadoop HDFS
Issue Type: Sub-task
Components: nfs
Reporter: Brandon Li
Assignee: Brandon Li
Attachments: HDFS-6894.001.patch
This can be an abstract method in NFS3Response to force the subclasses to
implement.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6581) Write to single replica in memory
[ https://issues.apache.org/jira/browse/HDFS-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146647#comment-14146647 ] Andrew Wang commented on HDFS-6581: --- Thanks for running these numbers Arpit. To convert into MB/s, it seems like we're going from 280 or 292 MB/s baseline to 455 or 487MBs with csums off. Are you aware of any further copies that can be eliminated in DTP, or via something like short-circuit writes? I'd be satisfied regarding the merge if we had a pretty good estimate of the perf after such improvements are made. Saying that we could go from e.g. 4 copies to 2 copies would suffice for this, better would be results from a prototype. Write to single replica in memory - Key: HDFS-6581 URL: https://issues.apache.org/jira/browse/HDFS-6581 Project: Hadoop HDFS Issue Type: Bug Components: datanode Reporter: Arpit Agarwal Assignee: Arpit Agarwal Attachments: HDFS-6581.merge.01.patch, HDFS-6581.merge.02.patch, HDFS-6581.merge.03.patch, HDFS-6581.merge.04.patch, HDFS-6581.merge.05.patch, HDFS-6581.merge.06.patch, HDFS-6581.merge.07.patch, HDFS-6581.merge.08.patch, HDFS-6581.merge.09.patch, HDFS-6581.merge.10.patch, HDFSWriteableReplicasInMemory.pdf, Test-Plan-for-HDFS-6581-Memory-Storage.pdf Per discussion with the community on HDFS-5851, we will implement writing to a single replica in DN memory via DataTransferProtocol. This avoids some of the issues with short-circuit writes, which we can revisit at a later time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7140) Add a tool to list all the existing block storage policies
Jing Zhao created HDFS-7140: --- Summary: Add a tool to list all the existing block storage policies Key: HDFS-7140 URL: https://issues.apache.org/jira/browse/HDFS-7140 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: 3.0.0 Reporter: Jing Zhao Assignee: Jing Zhao Priority: Minor After adding a new DistributedFileSystem API to allow normal users to get all the block storage policies, we should also provide a simple tool to list these policies. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7139) Unit test for creating encryption zone on root path
[ https://issues.apache.org/jira/browse/HDFS-7139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-7139: Attachment: HDFS-7139-v2.patch [~clamb] Thanks for the review. I've added those assertions. Unit test for creating encryption zone on root path --- Key: HDFS-7139 URL: https://issues.apache.org/jira/browse/HDFS-7139 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Zhe Zhang Assignee: Zhe Zhang Priority: Minor Attachments: HDFS-7139-v2.patch, HDFS-7139.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7140) Add a tool to list all the existing block storage policies
[ https://issues.apache.org/jira/browse/HDFS-7140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-7140: Attachment: HDFS-7140.001.patch HDFS-7140.000.patch patch 001 removes the changes on hdfs.cmd from patch 000 so that the Jenkins can pick it up. Add a tool to list all the existing block storage policies -- Key: HDFS-7140 URL: https://issues.apache.org/jira/browse/HDFS-7140 Project: Hadoop HDFS Issue Type: Sub-task Components: balancer, namenode Affects Versions: 3.0.0 Reporter: Jing Zhao Assignee: Jing Zhao Priority: Minor Attachments: HDFS-7140.000.patch, HDFS-7140.001.patch After adding a new DistributedFileSystem API to allow normal users to get all the block storage policies, we should also provide a simple tool to list these policies. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7139) Unit test for creating encryption zone on root path
[ https://issues.apache.org/jira/browse/HDFS-7139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146664#comment-14146664 ] Charles Lamb commented on HDFS-7139: +1 (non-binding) from me. Thanks again Zhe. Unit test for creating encryption zone on root path --- Key: HDFS-7139 URL: https://issues.apache.org/jira/browse/HDFS-7139 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Zhe Zhang Assignee: Zhe Zhang Priority: Minor Attachments: HDFS-7139-v2.patch, HDFS-7139.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6581) Write to single replica in memory
[ https://issues.apache.org/jira/browse/HDFS-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146704#comment-14146704 ] Arpit Agarwal commented on HDFS-6581: - I have not looked at the DataTransferProtocol well enough to understand how many copies we can eliminate. Reads get 1GB/s over the same protocol so intuitively we should be able to optimize writes to get a similar number. However improving the transfer protocol is a separate feature in itself and should not gate the current changes. Write to single replica in memory - Key: HDFS-6581 URL: https://issues.apache.org/jira/browse/HDFS-6581 Project: Hadoop HDFS Issue Type: Bug Components: datanode Reporter: Arpit Agarwal Assignee: Arpit Agarwal Attachments: HDFS-6581.merge.01.patch, HDFS-6581.merge.02.patch, HDFS-6581.merge.03.patch, HDFS-6581.merge.04.patch, HDFS-6581.merge.05.patch, HDFS-6581.merge.06.patch, HDFS-6581.merge.07.patch, HDFS-6581.merge.08.patch, HDFS-6581.merge.09.patch, HDFS-6581.merge.10.patch, HDFSWriteableReplicasInMemory.pdf, Test-Plan-for-HDFS-6581-Memory-Storage.pdf Per discussion with the community on HDFS-5851, we will implement writing to a single replica in DN memory via DataTransferProtocol. This avoids some of the issues with short-circuit writes, which we can revisit at a later time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6987) Move CipherSuite xattr information up to the encryption zone root
[ https://issues.apache.org/jira/browse/HDFS-6987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146734#comment-14146734 ] Andrew Wang commented on HDFS-6987: --- I'm +1 on this branch-2 patch, thanks Zhe. Will commit shortly. Move CipherSuite xattr information up to the encryption zone root - Key: HDFS-6987 URL: https://issues.apache.org/jira/browse/HDFS-6987 Project: Hadoop HDFS Issue Type: Sub-task Components: encryption Reporter: Andrew Wang Assignee: Zhe Zhang Fix For: 3.0.0 Attachments: HDFS-6987-20140917-v1.patch, HDFS-6987-20140918-v1.patch, HDFS-6987-20140919-v1.patch, HDFS-6987-20140919-v2.patch, HDFS-6987-20140919-v3.patch, HDFS-6987-20140920-v1.patch, HDFS-6987-branch-2-v2.patch All files within a single EZ need to be encrypted with the same CipherSuite. Because of this, I think we can store the CipherSuite once in the EZ rather than on each file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7076) Persist Block Storage Policy in FSImage and Edit Log
[ https://issues.apache.org/jira/browse/HDFS-7076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-7076: Status: Patch Available (was: Open) Persist Block Storage Policy in FSImage and Edit Log Key: HDFS-7076 URL: https://issues.apache.org/jira/browse/HDFS-7076 Project: Hadoop HDFS Issue Type: Sub-task Components: balancer, namenode Reporter: Jing Zhao Assignee: Jing Zhao Attachments: HDFS-7076.000.patch, HDFS-7076.001.patch, editsStored Currently block storage policies are loaded from a XML file. This may lead to policy inconsistency if the content of the XML gets modified or if the XML file is replaced by mistake. Thus it is better to persist policies in editlog/fsimage. Specifically we plan to do the following in this jira: # Add new DistributedFileSystem API for getting all the existing storage policies # Add new DistributedFileSystem/HdfsAdmin API for adding a new storage policy. The new storage policy is then persisted in editlog and later fsimage while checkpointing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7076) Persist Block Storage Policy in FSImage and Edit Log
[ https://issues.apache.org/jira/browse/HDFS-7076?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-7076: Attachment: HDFS-7076.001.patch editsStored Rebase the patch after HDFS-7081 got committed. Persist Block Storage Policy in FSImage and Edit Log Key: HDFS-7076 URL: https://issues.apache.org/jira/browse/HDFS-7076 Project: Hadoop HDFS Issue Type: Sub-task Components: balancer, namenode Reporter: Jing Zhao Assignee: Jing Zhao Attachments: HDFS-7076.000.patch, HDFS-7076.001.patch, editsStored Currently block storage policies are loaded from a XML file. This may lead to policy inconsistency if the content of the XML gets modified or if the XML file is replaced by mistake. Thus it is better to persist policies in editlog/fsimage. Specifically we plan to do the following in this jira: # Add new DistributedFileSystem API for getting all the existing storage policies # Add new DistributedFileSystem/HdfsAdmin API for adding a new storage policy. The new storage policy is then persisted in editlog and later fsimage while checkpointing. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7049) TestByteRangeInputStream.testPropagatedClose fails and throw NPE on branch-2
[
https://issues.apache.org/jira/browse/HDFS-7049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146740#comment-14146740
]
Haohui Mai commented on HDFS-7049:
--
+1
TestByteRangeInputStream.testPropagatedClose fails and throw NPE on branch-2
Key: HDFS-7049
URL: https://issues.apache.org/jira/browse/HDFS-7049
Project: Hadoop HDFS
Issue Type: Bug
Components: test
Affects Versions: 2.5.0
Reporter: Juan Yu
Assignee: Juan Yu
Priority: Minor
Attachments: HDFS-7049-branch-2.002.patch, HDFS-7049-branch-2.patch
On branch-2, TestByteRangeInputStream.testPropagatedClose throw NPE when
HftpFileSystem$RangeHeaderUrlOpener.connect
This is due to fix of HDFS-6143 WebHdfsFileSystem open should throw
FileNotFoundException for non-existing paths
public ByteRangeInputStream(URLOpener o, URLOpener r) throws IOException {
this.originalURL = o;
this.resolvedURL = r;
getInputStream();
}
the getInputStream() will be called in constructor now to verify if file
exists.
Since we just try to test if ByteRangeInputStream#close is called at proper
time, we could mock(ByteRangeInputStream.class, CALLS_REAL_METHODS) for
testing to avoid the NPE issue.
I believe the trunk version already does this, we just need to merge the test
from trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7049) TestByteRangeInputStream.testPropagatedClose fails and throw NPE on branch-2
[
https://issues.apache.org/jira/browse/HDFS-7049?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-7049:
-
Resolution: Fixed
Fix Version/s: 2.6.0
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
I've committed the patch to branch-2. Thanks [~j...@cloudera.com] for reporting
the issue and fixing it.
TestByteRangeInputStream.testPropagatedClose fails and throw NPE on branch-2
Key: HDFS-7049
URL: https://issues.apache.org/jira/browse/HDFS-7049
Project: Hadoop HDFS
Issue Type: Bug
Components: test
Affects Versions: 2.5.0
Reporter: Juan Yu
Assignee: Juan Yu
Priority: Minor
Fix For: 2.6.0
Attachments: HDFS-7049-branch-2.002.patch, HDFS-7049-branch-2.patch
On branch-2, TestByteRangeInputStream.testPropagatedClose throw NPE when
HftpFileSystem$RangeHeaderUrlOpener.connect
This is due to fix of HDFS-6143 WebHdfsFileSystem open should throw
FileNotFoundException for non-existing paths
public ByteRangeInputStream(URLOpener o, URLOpener r) throws IOException {
this.originalURL = o;
this.resolvedURL = r;
getInputStream();
}
the getInputStream() will be called in constructor now to verify if file
exists.
Since we just try to test if ByteRangeInputStream#close is called at proper
time, we could mock(ByteRangeInputStream.class, CALLS_REAL_METHODS) for
testing to avoid the NPE issue.
I believe the trunk version already does this, we just need to merge the test
from trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-6987) Move CipherSuite xattr information up to the encryption zone root
[ https://issues.apache.org/jira/browse/HDFS-6987?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-6987: -- Resolution: Fixed Fix Version/s: (was: 3.0.0) 2.6.0 Status: Resolved (was: Patch Available) Committed to branch-2 as well, thanks again Zhe. Move CipherSuite xattr information up to the encryption zone root - Key: HDFS-6987 URL: https://issues.apache.org/jira/browse/HDFS-6987 Project: Hadoop HDFS Issue Type: Sub-task Components: encryption Reporter: Andrew Wang Assignee: Zhe Zhang Fix For: 2.6.0 Attachments: HDFS-6987-20140917-v1.patch, HDFS-6987-20140918-v1.patch, HDFS-6987-20140919-v1.patch, HDFS-6987-20140919-v2.patch, HDFS-6987-20140919-v3.patch, HDFS-6987-20140920-v1.patch, HDFS-6987-branch-2-v2.patch All files within a single EZ need to be encrypted with the same CipherSuite. Because of this, I think we can store the CipherSuite once in the EZ rather than on each file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6987) Move CipherSuite xattr information up to the encryption zone root
[ https://issues.apache.org/jira/browse/HDFS-6987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146750#comment-14146750 ] Zhe Zhang commented on HDFS-6987: - [~andrew.wang] [~clamb] [~tucu00] Many thanks for the reviews and coding tips. Move CipherSuite xattr information up to the encryption zone root - Key: HDFS-6987 URL: https://issues.apache.org/jira/browse/HDFS-6987 Project: Hadoop HDFS Issue Type: Sub-task Components: encryption Reporter: Andrew Wang Assignee: Zhe Zhang Fix For: 2.6.0 Attachments: HDFS-6987-20140917-v1.patch, HDFS-6987-20140918-v1.patch, HDFS-6987-20140919-v1.patch, HDFS-6987-20140919-v2.patch, HDFS-6987-20140919-v3.patch, HDFS-6987-20140920-v1.patch, HDFS-6987-branch-2-v2.patch All files within a single EZ need to be encrypted with the same CipherSuite. Because of this, I think we can store the CipherSuite once in the EZ rather than on each file. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7049) TestByteRangeInputStream.testPropagatedClose fails and throw NPE on branch-2
[
https://issues.apache.org/jira/browse/HDFS-7049?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146758#comment-14146758
]
Juan Yu commented on HDFS-7049:
---
Thanks all for reviewing.
TestByteRangeInputStream.testPropagatedClose fails and throw NPE on branch-2
Key: HDFS-7049
URL: https://issues.apache.org/jira/browse/HDFS-7049
Project: Hadoop HDFS
Issue Type: Bug
Components: test
Affects Versions: 2.5.0
Reporter: Juan Yu
Assignee: Juan Yu
Priority: Minor
Fix For: 2.6.0
Attachments: HDFS-7049-branch-2.002.patch, HDFS-7049-branch-2.patch
On branch-2, TestByteRangeInputStream.testPropagatedClose throw NPE when
HftpFileSystem$RangeHeaderUrlOpener.connect
This is due to fix of HDFS-6143 WebHdfsFileSystem open should throw
FileNotFoundException for non-existing paths
public ByteRangeInputStream(URLOpener o, URLOpener r) throws IOException {
this.originalURL = o;
this.resolvedURL = r;
getInputStream();
}
the getInputStream() will be called in constructor now to verify if file
exists.
Since we just try to test if ByteRangeInputStream#close is called at proper
time, we could mock(ByteRangeInputStream.class, CALLS_REAL_METHODS) for
testing to avoid the NPE issue.
I believe the trunk version already does this, we just need to merge the test
from trunk.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7118) Improve diagnostics on storage directory rename operations by using NativeIO#renameTo in Storage#rename.
[
https://issues.apache.org/jira/browse/HDFS-7118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146767#comment-14146767
]
jay vyas commented on HDFS-7118:
Is this a hadoop common issue, or a HDFS specific issue?
Improve diagnostics on storage directory rename operations by using
NativeIO#renameTo in Storage#rename.
Key: HDFS-7118
URL: https://issues.apache.org/jira/browse/HDFS-7118
Project: Hadoop HDFS
Issue Type: Sub-task
Components: journal-node, namenode
Reporter: Chris Nauroth
Assignee: Chris Nauroth
If a file rename fails, {{NativeIO#renameTo}} includes more information about
the root cause than a plain {{java.io.File#renameTo}}. The native code can
throw an exception with a detailed error message and the {{errno}} on *nix or
the value of {{GetLastError}} on Windows. This issue proposes to use
{{NativeIO#renameTo}} inside or in place of {{Storage#rename}} to help
improve diagnostics. The method falls back to {{java.io.File#renameTo}} if
native code is not loaded, so this change would not introduce a compatibility
problem for deployments running without native code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6581) Write to single replica in memory
[ https://issues.apache.org/jira/browse/HDFS-6581?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146774#comment-14146774 ] Suresh Srinivas commented on HDFS-6581: --- bq. I'd be satisfied regarding the merge if we had a pretty good estimate of the perf after such improvements are made. [~andrew.wang], why does that gate this merge? If that is an important feature, certainly please consider creating a jira and making further improvements. But I am at a loss at all the issues we keep bringing up as requirements for the merge, despite a lot of work done to ensure valid concerns people are raising are addressed. Write to single replica in memory - Key: HDFS-6581 URL: https://issues.apache.org/jira/browse/HDFS-6581 Project: Hadoop HDFS Issue Type: Bug Components: datanode Reporter: Arpit Agarwal Assignee: Arpit Agarwal Attachments: HDFS-6581.merge.01.patch, HDFS-6581.merge.02.patch, HDFS-6581.merge.03.patch, HDFS-6581.merge.04.patch, HDFS-6581.merge.05.patch, HDFS-6581.merge.06.patch, HDFS-6581.merge.07.patch, HDFS-6581.merge.08.patch, HDFS-6581.merge.09.patch, HDFS-6581.merge.10.patch, HDFSWriteableReplicasInMemory.pdf, Test-Plan-for-HDFS-6581-Memory-Storage.pdf Per discussion with the community on HDFS-5851, we will implement writing to a single replica in DN memory via DataTransferProtocol. This avoids some of the issues with short-circuit writes, which we can revisit at a later time. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Work started] (HDFS-7011) Implement basic utilities for libhdfs3
[ https://issues.apache.org/jira/browse/HDFS-7011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Work on HDFS-7011 started by Colin Patrick McCabe. -- Implement basic utilities for libhdfs3 -- Key: HDFS-7011 URL: https://issues.apache.org/jira/browse/HDFS-7011 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs-client Reporter: Zhanwei Wang Assignee: Colin Patrick McCabe Attachments: HDFS-7011-pnative.001.patch, HDFS-7011-pnative.002.patch, HDFS-7011.patch Implement basic utilities such as hash, exception handling, logger, configure parser, checksum calculate and so on. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7011) Implement basic utilities for libhdfs3
[ https://issues.apache.org/jira/browse/HDFS-7011?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-7011: --- Attachment: HDFS-7011-pnative.002.patch Implement basic utilities for libhdfs3 -- Key: HDFS-7011 URL: https://issues.apache.org/jira/browse/HDFS-7011 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs-client Reporter: Zhanwei Wang Assignee: Colin Patrick McCabe Attachments: HDFS-7011-pnative.001.patch, HDFS-7011-pnative.002.patch, HDFS-7011.patch Implement basic utilities such as hash, exception handling, logger, configure parser, checksum calculate and so on. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7011) Implement basic utilities for libhdfs3
[
https://issues.apache.org/jira/browse/HDFS-7011?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146779#comment-14146779
]
Colin Patrick McCabe commented on HDFS-7011:
* add ifndef back to DEFAULT_STACK_PREFIX
* change namespace hdfs3 to namespace hdfs
* block more signals in {{ThreadBlockSignal}}
Implement basic utilities for libhdfs3
--
Key: HDFS-7011
URL: https://issues.apache.org/jira/browse/HDFS-7011
Project: Hadoop HDFS
Issue Type: Sub-task
Components: hdfs-client
Reporter: Zhanwei Wang
Assignee: Colin Patrick McCabe
Attachments: HDFS-7011-pnative.001.patch,
HDFS-7011-pnative.002.patch, HDFS-7011.patch
Implement basic utilities such as hash, exception handling, logger, configure
parser, checksum calculate and so on.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7118) Improve diagnostics on storage directory rename operations by using NativeIO#renameTo in Storage#rename.
[
https://issues.apache.org/jira/browse/HDFS-7118?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146784#comment-14146784
]
Chris Nauroth commented on HDFS-7118:
-
Hi, [~jayunit100]. In this case, it's an HDFS-specific issue. HDFS currently
calls {{java.io.File#renameTo}} in a few places, which doesn't propagate full
error details. We already implemented {{NativeIO#renameTo}} in hadoop-common a
while back to address similar issues. I don't anticipate any hadoop-common
changes as part of this. I think we'll just change HDFS code to call what
already exists in Common.
Improve diagnostics on storage directory rename operations by using
NativeIO#renameTo in Storage#rename.
Key: HDFS-7118
URL: https://issues.apache.org/jira/browse/HDFS-7118
Project: Hadoop HDFS
Issue Type: Sub-task
Components: journal-node, namenode
Reporter: Chris Nauroth
Assignee: Chris Nauroth
If a file rename fails, {{NativeIO#renameTo}} includes more information about
the root cause than a plain {{java.io.File#renameTo}}. The native code can
throw an exception with a detailed error message and the {{errno}} on *nix or
the value of {{GetLastError}} on Windows. This issue proposes to use
{{NativeIO#renameTo}} inside or in place of {{Storage#rename}} to help
improve diagnostics. The method falls back to {{java.io.File#renameTo}} if
native code is not loaded, so this change would not introduce a compatibility
problem for deployments running without native code.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7104) Fix and clarify INodeInPath getter functions
[
https://issues.apache.org/jira/browse/HDFS-7104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146794#comment-14146794
]
Hadoop QA commented on HDFS-7104:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12670999/HDFS-7104-20140924-v1.patch
against trunk revision 073bbd8.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1264 javac
compiler warnings (more than the trunk's current 1263 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS
The following test timeouts occurred in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestParallelUnixDomainRead
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8185//testReport/
Javac warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/8185//artifact/PreCommit-HADOOP-Build-patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8185//console
This message is automatically generated.
Fix and clarify INodeInPath getter functions
Key: HDFS-7104
URL: https://issues.apache.org/jira/browse/HDFS-7104
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Priority: Minor
Attachments: HDFS-7104-20140923-v1.patch,
HDFS-7104-20140923-v3.patch, HDFS-7104-20140924-v1.patch
inodes is initialized with the number of patch components. After resolve, it
contains both non-null and null elements (introduced by dot-snapshot dirs).
When getINodes is called, an array is returned excluding all non elements,
which is the correct behavior. Meanwhile, the inodes array is trimmed too,
which shouldn't be done by a getter.
Because of the above, the behavior of getINodesInPath depends on whether
getINodes has been called, which is not correct.
The name of getLastINodeInPath is confusing – it actually returns the last
non-null inode in the path. Also, shouldn't the return type be a single INode?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7139) Unit test for creating encryption zone on root path
[
https://issues.apache.org/jira/browse/HDFS-7139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146793#comment-14146793
]
Hadoop QA commented on HDFS-7139:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12671006/HDFS-7139-v2.patch
against trunk revision 073bbd8.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS
The following test timeouts occurred in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestParallelUnixDomainRead
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8186//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8186//console
This message is automatically generated.
Unit test for creating encryption zone on root path
---
Key: HDFS-7139
URL: https://issues.apache.org/jira/browse/HDFS-7139
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Priority: Minor
Attachments: HDFS-7139-v2.patch, HDFS-7139.patch
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Resolved] (HDFS-5722) Implement compression in the HTTP server of SNN / SBN instead of FSImage
[ https://issues.apache.org/jira/browse/HDFS-5722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai resolved HDFS-5722. -- Resolution: Invalid As the current PB-based fsimage format compresses each section, this jira is no longer valid. Implement compression in the HTTP server of SNN / SBN instead of FSImage Key: HDFS-5722 URL: https://issues.apache.org/jira/browse/HDFS-5722 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Haohui Mai The current FSImage format support compression, there is a field in the header which specifies the compression codec used to compress the data in the image. The main motivation was to reduce the number of bytes to be transferred between SNN / SBN / NN. The main disadvantage, however, is that it requires the client to access the FSImage in strictly sequential order. This might not fit well with the new design of FSImage. For example, serializing the data in protobuf allows the client to quickly skip data that it does not understand. The compression built-in the format, however, complicates the calculation of offsets and lengths. Recovering from a corrupted, compressed FSImage is also non-trivial as off-the-shelf tools like bzip2recover is inapplicable. This jira proposes to move the compression from the format of the FSImage to the transport layer, namely, the HTTP server of SNN / SBN. This design simplifies the format of FSImage, opens up the opportunity to quickly navigate through the FSImage, and eases the process of recovery. It also retains the benefits of reducing the number of bytes to be transferred across the wire since there are compression on the transport layer. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6914) Resolve huge memory consumption Issue with OIV processing PB-based fsimages
[ https://issues.apache.org/jira/browse/HDFS-6914?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146817#comment-14146817 ] Haohui Mai commented on HDFS-6914: -- [~haoch], do you mind if I take this jira and drive it forward? Resolve huge memory consumption Issue with OIV processing PB-based fsimages --- Key: HDFS-6914 URL: https://issues.apache.org/jira/browse/HDFS-6914 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.4.1 Reporter: Hao Chen Labels: hdfs Attachments: HDFS-6914.patch For better managing and supporting a lot of large hadoop clusters in production, we internally need to automatically export fsimage to delimited text files in LSR style and then analyse with hive or pig or build system metrics for real-time analyzing. However due to the internal layout changes introduced by the protobuf-based fsimage, OIV processing program consumes excessive amount of memory. For example, in order to export the fsimage in size of 8GB, it should have taken about 85GB memory which is really not reasonable and impacted performance of other services badly in the same server. To resolve above problem, I submit this patch which will reduce memory consumption of OIV LSR processing by 50%. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7104) Fix and clarify INodeInPath getter functions
[
https://issues.apache.org/jira/browse/HDFS-7104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146822#comment-14146822
]
Jing Zhao commented on HDFS-7104:
-
The patch looks good. Some minors:
# looks like now we can declare {{inodes}} as final.
# For the new javadoc added for {{getINodes}}, it will be better to also point
out that the new array's length will be equal to the value of {{capacity}}.
# Optional: there are also two javadoc errors in {{INodesInPath}} (for
{{snapshotRootIndex}} and {{getSnapshotRootIndex}}). Maybe we can use this
chance to fix them.
Fix and clarify INodeInPath getter functions
Key: HDFS-7104
URL: https://issues.apache.org/jira/browse/HDFS-7104
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Priority: Minor
Attachments: HDFS-7104-20140923-v1.patch,
HDFS-7104-20140923-v3.patch, HDFS-7104-20140924-v1.patch
inodes is initialized with the number of patch components. After resolve, it
contains both non-null and null elements (introduced by dot-snapshot dirs).
When getINodes is called, an array is returned excluding all non elements,
which is the correct behavior. Meanwhile, the inodes array is trimmed too,
which shouldn't be done by a getter.
Because of the above, the behavior of getINodesInPath depends on whether
getINodes has been called, which is not correct.
The name of getLastINodeInPath is confusing – it actually returns the last
non-null inode in the path. Also, shouldn't the return type be a single INode?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6321) Add a message on the old web UI that indicates the old UI is deprecated
[
https://issues.apache.org/jira/browse/HDFS-6321?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146823#comment-14146823
]
Haohui Mai commented on HDFS-6321:
--
Instead of adding a new file, the modification should be done in
{{dfshealth.jsp}} instead.
Add a message on the old web UI that indicates the old UI is deprecated
---
Key: HDFS-6321
URL: https://issues.apache.org/jira/browse/HDFS-6321
Project: Hadoop HDFS
Issue Type: Improvement
Reporter: Haohui Mai
Assignee: Tassapol Athiapinya
Attachments: HDFS-6321.000.patch, HDFS-6321.001.patch
HDFS-6252 has removed the jsp ui from trunk. We should add a message in the
old web ui to indicate that the ui has been deprecated and ask the user to
move towards the new web ui.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-6877) Avoid calling checkDisk when an HDFS volume is removed during a write.
[
https://issues.apache.org/jira/browse/HDFS-6877?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146839#comment-14146839
]
Hadoop QA commented on HDFS-6877:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12670996/HDFS-6877.006.patch
against trunk revision 073bbd8.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS
org.apache.hadoop.hdfs.web.TestWebHdfsFileSystemContract
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
The following test timeouts occurred in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.datanode.TestMultipleNNDataBlockScanner
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8182//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8182//console
This message is automatically generated.
Avoid calling checkDisk when an HDFS volume is removed during a write.
--
Key: HDFS-6877
URL: https://issues.apache.org/jira/browse/HDFS-6877
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode
Affects Versions: 2.5.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Attachments: HDFS-6877.000.consolidate.txt,
HDFS-6877.000.delta-HDFS-6727.txt, HDFS-6877.001.combo.txt,
HDFS-6877.001.patch, HDFS-6877.002.patch, HDFS-6877.003.patch,
HDFS-6877.004.patch, HDFS-6877.005.patch, HDFS-6877.006.patch
Avoid calling checkDisk and stop active BlockReceiver thread when an HDFS
volume is removed during a write.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7036) HDFS-6776 fix requires to upgrade insecure cluster, which means quite some user pain
[
https://issues.apache.org/jira/browse/HDFS-7036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146861#comment-14146861
]
Haohui Mai commented on HDFS-7036:
--
My position has not changed since HDFS-6776. Just to recap:
# This use case is so specific to distcp that it should be fixed distcp instead
of the underlying filesystem implementation.
# This is a hack for compatibility. There are many more users of
{{WebHdfsFileSystem}} compared to distcp. It is more reasonable to contain the
changes at higher layers (i.e. distcp) to avoid surprises to other applications.
I understand that hacking {{WebHdfsFileSystem}} is an easy enough fix, I also
understand that a hack might be needed either here or there, but parsing the
exception message in {{WebHdfsFileSystem}} does not seem the right solution
here.
If it turns out that this type of hack is unavoidable, I suggest doing it in
distcp.
HDFS-6776 fix requires to upgrade insecure cluster, which means quite some
user pain
Key: HDFS-7036
URL: https://issues.apache.org/jira/browse/HDFS-7036
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Affects Versions: 2.5.1
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
Attachments: HDFS-7036.001.patch
Issuing command
{code}
hadoop fs -lsr webhdfs://insecureCluster
{code}
at a secure cluster side fails with message Failed to get the token ...,
similar symptom as reported in HDFS-6776.
If the fix of HDFS-6776 is applied to only the secure cluster, doing
{code}
distcp webhdfs://insecureCluster secureCluster
{code}
would fail same way.
Basically running any application in secure cluster to access insecure
cluster via webhdfs would fail the same way, if the HDFS-6776 fix is not
applied to the insecure cluster.
This could be quite some user pain. Filing this jira for a solution to make
user's life easier.
One proposed solution was to add a msg-parsing mechanism in webhdfs, which is
a bit hacky. The other proposed solution is to do the same kind of hack at
application side, which means the same hack need to be applied in each
application.
Thanks [~daryn], [~wheat9], [~jingzhao], [~tucu00] and [~atm] for the
discussion in HDFS-6776.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7139) Unit test for creating encryption zone on root path
[ https://issues.apache.org/jira/browse/HDFS-7139?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146871#comment-14146871 ] Zhe Zhang commented on HDFS-7139: - The failed tests are unrelated since this patch only adds another unit test. [~andrew.wang] Could you take a look? Unit test for creating encryption zone on root path --- Key: HDFS-7139 URL: https://issues.apache.org/jira/browse/HDFS-7139 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Zhe Zhang Assignee: Zhe Zhang Priority: Minor Attachments: HDFS-7139-v2.patch, HDFS-7139.patch -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7141) Integrate HDFS-6581 memory usage with HDFS-4949
Colin Patrick McCabe created HDFS-7141: -- Summary: Integrate HDFS-6581 memory usage with HDFS-4949 Key: HDFS-7141 URL: https://issues.apache.org/jira/browse/HDFS-7141 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Blocker We need to be able to integrate HDFS-6581 memory usage with HDFS-4949, so that system administrators are managing a single pool of memory rather than multiple. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7142) Implement a 2Q eviction strategy for HDFS-6581
Colin Patrick McCabe created HDFS-7142: -- Summary: Implement a 2Q eviction strategy for HDFS-6581 Key: HDFS-7142 URL: https://issues.apache.org/jira/browse/HDFS-7142 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe We should implement a 2Q or approximate 2Q eviction strategy for HDFS-6581. It is well known that LRU is a poor fit for scanning workloads, which HDFS may often encounter. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7141) Integrate HDFS-6581 memory usage with HDFS-4949
[ https://issues.apache.org/jira/browse/HDFS-7141?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146907#comment-14146907 ] Arpit Agarwal commented on HDFS-7141: - HDFS-6919 exists to investigate this. Integrate HDFS-6581 memory usage with HDFS-4949 --- Key: HDFS-7141 URL: https://issues.apache.org/jira/browse/HDFS-7141 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Blocker We need to be able to integrate HDFS-6581 memory usage with HDFS-4949, so that system administrators are managing a single pool of memory rather than multiple. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HDFS-7141) Integrate HDFS-6581 memory usage with HDFS-4949
[ https://issues.apache.org/jira/browse/HDFS-7141?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal resolved HDFS-7141. - Resolution: Duplicate Integrate HDFS-6581 memory usage with HDFS-4949 --- Key: HDFS-7141 URL: https://issues.apache.org/jira/browse/HDFS-7141 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Reporter: Colin Patrick McCabe Assignee: Colin Patrick McCabe Priority: Blocker We need to be able to integrate HDFS-6581 memory usage with HDFS-4949, so that system administrators are managing a single pool of memory rather than multiple. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7104) Fix and clarify INodeInPath getter functions
[
https://issues.apache.org/jira/browse/HDFS-7104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146912#comment-14146912
]
Zhe Zhang commented on HDFS-7104:
-
[~jingzhao] Thanks for reviewing again.
bq. looks like now we can declare inodes as final.
{{fromINode()}} is still used in a few places which tries to set the {{inodes}}
array.
bq. Optional: there are also two javadoc errors in INodesInPath (for
snapshotRootIndex and getSnapshotRootIndex). Maybe we can use this chance to
fix them.
I agree. How about the below:
{code} index of the null inode introduced by the snapshot dir in {@link
#inodes}, -1 for non-snapshot paths. {code}
Fix and clarify INodeInPath getter functions
Key: HDFS-7104
URL: https://issues.apache.org/jira/browse/HDFS-7104
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Priority: Minor
Attachments: HDFS-7104-20140923-v1.patch,
HDFS-7104-20140923-v3.patch, HDFS-7104-20140924-v1.patch
inodes is initialized with the number of patch components. After resolve, it
contains both non-null and null elements (introduced by dot-snapshot dirs).
When getINodes is called, an array is returned excluding all non elements,
which is the correct behavior. Meanwhile, the inodes array is trimmed too,
which shouldn't be done by a getter.
Because of the above, the behavior of getINodesInPath depends on whether
getINodes has been called, which is not correct.
The name of getLastINodeInPath is confusing – it actually returns the last
non-null inode in the path. Also, shouldn't the return type be a single INode?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-6919) Enforce a single limit for RAM disk usage and replicas cached via locking
[ https://issues.apache.org/jira/browse/HDFS-6919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-6919: --- Priority: Blocker (was: Major) Summary: Enforce a single limit for RAM disk usage and replicas cached via locking (was: Consider enforcing single limit for RAM disk usage and replicas cached via locking) Enforce a single limit for RAM disk usage and replicas cached via locking - Key: HDFS-6919 URL: https://issues.apache.org/jira/browse/HDFS-6919 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Arpit Agarwal Priority: Blocker The DataNode can have a single limit for memory usage which applies to both replicas cached via CCM and replicas on RAM disk. See comments [1|https://issues.apache.org/jira/browse/HDFS-6581?focusedCommentId=14106025page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14106025], [2|https://issues.apache.org/jira/browse/HDFS-6581?focusedCommentId=14106245page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14106245] and [3|https://issues.apache.org/jira/browse/HDFS-6581?focusedCommentId=14106575page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14106575] for discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-6808) Add command line option to ask DataNode reload configuration.
[
https://issues.apache.org/jira/browse/HDFS-6808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146916#comment-14146916
]
Allen Wittenauer commented on HDFS-6808:
A quick vote around the office came out to be about a 5:1 ratio, with most
agreeing that refresh and reconfig are essentially the same operation.
That said, we can eventually make refresh point to reconfig after enough users
complain about it. So I'll go +0.
Add command line option to ask DataNode reload configuration.
-
Key: HDFS-6808
URL: https://issues.apache.org/jira/browse/HDFS-6808
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode
Affects Versions: 2.5.0
Reporter: Lei (Eddy) Xu
Assignee: Lei (Eddy) Xu
Attachments: HDFS-6808.000.combo.patch, HDFS-6808.000.patch,
HDFS-6808.001.combo.patch, HDFS-6808.001.patch, HDFS-6808.002.combo.patch,
HDFS-6808.002.patch, HDFS-6808.003.combo.txt, HDFS-6808.003.patch,
HDFS-6808.004.combo.patch, HDFS-6808.004.patch, HDFS-6808.005.combo.patch,
HDFS-6808.005.patch, HDFS-6808.006.combo.patch, HDFS-6808.006.patch,
HDFS-6808.007.combo.patch, HDFS-6808.007.patch, HDFS-6808.008.combo.patch,
HDFS-6808.008.patch, HDFS-6808.009.combo.patch, HDFS-6808.009.patch,
HDFS-6808.010.patch, HDFS-6808.011.patch
The workflow of dynamically changing data volumes on DataNode is
# Users manually changed {{dfs.datanode.data.dir}} in the configuration file
# User use command line to notify DN to reload configuration and updates its
volumes.
This work adds command line support to notify DN to reload configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7104) Fix and clarify INodeInPath getter functions
[
https://issues.apache.org/jira/browse/HDFS-7104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146928#comment-14146928
]
Jing Zhao commented on HDFS-7104:
-
bq. fromINode() is still used in a few places which tries to set the inodes
array.
Thanks for pointing this out! Then we just leave it unchanged.
bq. index of the null inode introduced by the snapshot dir in {@link #inodes},
-1 for non-snapshot paths.
Actually it's not null inode. For a snapshot path, it is the index of the
{{@link Snapshot.Root}} node in the {{inodes}} array.
Fix and clarify INodeInPath getter functions
Key: HDFS-7104
URL: https://issues.apache.org/jira/browse/HDFS-7104
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Priority: Minor
Attachments: HDFS-7104-20140923-v1.patch,
HDFS-7104-20140923-v3.patch, HDFS-7104-20140924-v1.patch
inodes is initialized with the number of patch components. After resolve, it
contains both non-null and null elements (introduced by dot-snapshot dirs).
When getINodes is called, an array is returned excluding all non elements,
which is the correct behavior. Meanwhile, the inodes array is trimmed too,
which shouldn't be done by a getter.
Because of the above, the behavior of getINodesInPath depends on whether
getINodes has been called, which is not correct.
The name of getLastINodeInPath is confusing – it actually returns the last
non-null inode in the path. Also, shouldn't the return type be a single INode?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-6932) Balancer and Mover tools should ignore replicas on RAM_DISK
[ https://issues.apache.org/jira/browse/HDFS-6932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-6932: - Attachment: HDFS-6932.2.patch Add Balancer test with RAM_DISK configuration and fix an array OOB issue in MiniDFSCluster#startDataNodes. Balancer and Mover tools should ignore replicas on RAM_DISK --- Key: HDFS-6932 URL: https://issues.apache.org/jira/browse/HDFS-6932 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Affects Versions: HDFS-6581 Reporter: Arpit Agarwal Assignee: Xiaoyu Yao Attachments: HDFS-6932.0.patch, HDFS-6932.1.patch, HDFS-6932.2.patch Per title, balancer and mover should just ignore replicas on RAM disk instead of attempting to move them to other nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-3107) HDFS truncate
[
https://issues.apache.org/jira/browse/HDFS-3107?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146941#comment-14146941
]
Aaron T. Myers commented on HDFS-3107:
--
I have not done any detailed review of the code or design, but in a quick look
at the current patch I do see one little problem:
{code}
+checkOperation(OperationCategory.WRITE);
+boolean res;
+writeLock();
+try {
+ if (isInSafeMode()) {
+throw new SafeModeException(Cannot truncate for + src, safeMode);
+ }
{code}
In the above, you should be checking for {{OperationCategory.WRITE}} both
before and after taking the write lock. Before to prevent clients from failing
over slowly when contacting the standby NN, after to ensure correctness.
HDFS truncate
-
Key: HDFS-3107
URL: https://issues.apache.org/jira/browse/HDFS-3107
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, namenode
Reporter: Lei Chang
Assignee: Plamen Jeliazkov
Attachments: HDFS-3107.patch, HDFS-3107.patch, HDFS_truncate.pdf,
HDFS_truncate_semantics_Mar15.pdf, HDFS_truncate_semantics_Mar21.pdf
Original Estimate: 1,344h
Remaining Estimate: 1,344h
Systems with transaction support often need to undo changes made to the
underlying storage when a transaction is aborted. Currently HDFS does not
support truncate (a standard Posix operation) which is a reverse operation of
append, which makes upper layer applications use ugly workarounds (such as
keeping track of the discarded byte range per file in a separate metadata
store, and periodically running a vacuum process to rewrite compacted files)
to overcome this limitation of HDFS.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7036) HDFS-6776 fix requires to upgrade insecure cluster, which means quite some user pain
[
https://issues.apache.org/jira/browse/HDFS-7036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146957#comment-14146957
]
Yongjun Zhang commented on HDFS-7036:
-
HI [~wheat9], Thanks for your comments.
Would you mind also comment on how we are going to deal with the similar
failure of issuing
{code}
hadoop fs -ls insecureCluster
{code}
from secure cluster side? There is the possibility that other applications too.
Given it's a hack, wonder if you would suggest to have the same hack to
different places?
Would you please give some *concrete* example about the potential damage it
would cause if we have the hack in webhdfs? I think that would help the
discussion most here.
To me, it's an easy and clean solution to have this hack in webhdfs, so all
applications are taken care of by the solution; and it's going to be easy to
take out this hack in the future when it's the time.
On the other hand, adding the hack in distcp and other applications, the
solution in each application is going to be more complicated than the webhdfs
one, and it has the potential to introduce more instability to the software
than the potential damage I can see with the webhdfs solution. So the
*concrete* example I hoped you could give can help the discussion here.
Thanks a lot.
HDFS-6776 fix requires to upgrade insecure cluster, which means quite some
user pain
Key: HDFS-7036
URL: https://issues.apache.org/jira/browse/HDFS-7036
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Affects Versions: 2.5.1
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
Attachments: HDFS-7036.001.patch
Issuing command
{code}
hadoop fs -lsr webhdfs://insecureCluster
{code}
at a secure cluster side fails with message Failed to get the token ...,
similar symptom as reported in HDFS-6776.
If the fix of HDFS-6776 is applied to only the secure cluster, doing
{code}
distcp webhdfs://insecureCluster secureCluster
{code}
would fail same way.
Basically running any application in secure cluster to access insecure
cluster via webhdfs would fail the same way, if the HDFS-6776 fix is not
applied to the insecure cluster.
This could be quite some user pain. Filing this jira for a solution to make
user's life easier.
One proposed solution was to add a msg-parsing mechanism in webhdfs, which is
a bit hacky. The other proposed solution is to do the same kind of hack at
application side, which means the same hack need to be applied in each
application.
Thanks [~daryn], [~wheat9], [~jingzhao], [~tucu00] and [~atm] for the
discussion in HDFS-6776.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7140) Add a tool to list all the existing block storage policies
[
https://issues.apache.org/jira/browse/HDFS-7140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jing Zhao updated HDFS-7140:
Attachment: HDFS-7140.002.patch
Update the patch to fix a test failure caused by inconsistent output message.
Also rename {{getStoragePolicySuite}} to {{getStoragePolicies}}.
Add a tool to list all the existing block storage policies
--
Key: HDFS-7140
URL: https://issues.apache.org/jira/browse/HDFS-7140
Project: Hadoop HDFS
Issue Type: Sub-task
Components: balancer, namenode
Affects Versions: 3.0.0
Reporter: Jing Zhao
Assignee: Jing Zhao
Priority: Minor
Attachments: HDFS-7140.000.patch, HDFS-7140.001.patch,
HDFS-7140.002.patch
After adding a new DistributedFileSystem API to allow normal users to get all
the block storage policies, we should also provide a simple tool to list
these policies.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7076) Persist Block Storage Policy in FSImage and Edit Log
[
https://issues.apache.org/jira/browse/HDFS-7076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146962#comment-14146962
]
Hadoop QA commented on HDFS-7076:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12671022/HDFS-7076.001.patch
against trunk revision 9fa5a89.
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 2.0.3) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.TestNamenodeRetryCache
org.apache.hadoop.hdfs.server.namenode.ha.TestRetryCacheWithHA
org.apache.hadoop.hdfs.server.namenode.ha.TestPipelinesFailover
org.apache.hadoop.hdfs.TestDFSInotifyEventInputStream
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
org.apache.hadoop.hdfs.TestEncryptionZonesWithKMS
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/8187//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/8187//console
This message is automatically generated.
Persist Block Storage Policy in FSImage and Edit Log
Key: HDFS-7076
URL: https://issues.apache.org/jira/browse/HDFS-7076
Project: Hadoop HDFS
Issue Type: Sub-task
Components: balancer, namenode
Reporter: Jing Zhao
Assignee: Jing Zhao
Attachments: HDFS-7076.000.patch, HDFS-7076.001.patch, editsStored
Currently block storage policies are loaded from a XML file. This may lead to
policy inconsistency if the content of the XML gets modified or if the XML
file is replaced by mistake. Thus it is better to persist policies in
editlog/fsimage.
Specifically we plan to do the following in this jira:
# Add new DistributedFileSystem API for getting all the existing storage
policies
# Add new DistributedFileSystem/HdfsAdmin API for adding a new storage
policy. The new storage policy is then persisted in editlog and later fsimage
while checkpointing.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-7104) Fix and clarify INodeInPath getter functions
[ https://issues.apache.org/jira/browse/HDFS-7104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhe Zhang updated HDFS-7104: Attachment: HDFS-7104-20140924-v2.patch Javadoc fixes and improvements. Fix and clarify INodeInPath getter functions Key: HDFS-7104 URL: https://issues.apache.org/jira/browse/HDFS-7104 Project: Hadoop HDFS Issue Type: Bug Reporter: Zhe Zhang Assignee: Zhe Zhang Priority: Minor Attachments: HDFS-7104-20140923-v1.patch, HDFS-7104-20140923-v3.patch, HDFS-7104-20140924-v1.patch, HDFS-7104-20140924-v2.patch inodes is initialized with the number of patch components. After resolve, it contains both non-null and null elements (introduced by dot-snapshot dirs). When getINodes is called, an array is returned excluding all non elements, which is the correct behavior. Meanwhile, the inodes array is trimmed too, which shouldn't be done by a getter. Because of the above, the behavior of getINodesInPath depends on whether getINodes has been called, which is not correct. The name of getLastINodeInPath is confusing – it actually returns the last non-null inode in the path. Also, shouldn't the return type be a single INode? -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7104) Fix and clarify INodeInPath getter functions
[
https://issues.apache.org/jira/browse/HDFS-7104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146988#comment-14146988
]
Jing Zhao commented on HDFS-7104:
-
Sorry I should mention this in my last comment:
{code}
E.g., for path /dir/.snapshot, an array of size 1 is returned, containing only
the inode of dir.
{code}
Here an array of size 2 is actually returned, since the root (/) node is also
included :)
We can change it to sth. like E.g., for path /dir/.snapshot with inodes
containing all the inodes involved in the path, an array of size 2 is returned,
containing both the root(/) and the inode of dir.
Fix and clarify INodeInPath getter functions
Key: HDFS-7104
URL: https://issues.apache.org/jira/browse/HDFS-7104
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Priority: Minor
Attachments: HDFS-7104-20140923-v1.patch,
HDFS-7104-20140923-v3.patch, HDFS-7104-20140924-v1.patch,
HDFS-7104-20140924-v2.patch
inodes is initialized with the number of patch components. After resolve, it
contains both non-null and null elements (introduced by dot-snapshot dirs).
When getINodes is called, an array is returned excluding all non elements,
which is the correct behavior. Meanwhile, the inodes array is trimmed too,
which shouldn't be done by a getter.
Because of the above, the behavior of getINodesInPath depends on whether
getINodes has been called, which is not correct.
The name of getLastINodeInPath is confusing – it actually returns the last
non-null inode in the path. Also, shouldn't the return type be a single INode?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7036) HDFS-6776 fix requires to upgrade insecure cluster, which means quite some user pain
[
https://issues.apache.org/jira/browse/HDFS-7036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14146989#comment-14146989
]
Haohui Mai commented on HDFS-7036:
--
bq. ...adding the hack in distcp and other applications...
Can you point out what the other applications are that need to access both
secure and insecure cluster at the same time? To my best knowledge, distcp is
the only use case. The title of this jira suggests it is focusing on distcp.
In contrast, YARN, MR and other applications that are built on top of them use
{{WebHdfsFileSystem}} extensively.
My message has been consistent since HDFS-6776 -- the hack should be contained
which result minimal damage in the codebase. I cannot +1 for the approach on
hacking {{WebHdfsFileSystem}} just for this issue.
If you still don't get it, it might be helpful to go through the distcp code
first.
HDFS-6776 fix requires to upgrade insecure cluster, which means quite some
user pain
Key: HDFS-7036
URL: https://issues.apache.org/jira/browse/HDFS-7036
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Affects Versions: 2.5.1
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
Attachments: HDFS-7036.001.patch
Issuing command
{code}
hadoop fs -lsr webhdfs://insecureCluster
{code}
at a secure cluster side fails with message Failed to get the token ...,
similar symptom as reported in HDFS-6776.
If the fix of HDFS-6776 is applied to only the secure cluster, doing
{code}
distcp webhdfs://insecureCluster secureCluster
{code}
would fail same way.
Basically running any application in secure cluster to access insecure
cluster via webhdfs would fail the same way, if the HDFS-6776 fix is not
applied to the insecure cluster.
This could be quite some user pain. Filing this jira for a solution to make
user's life easier.
One proposed solution was to add a msg-parsing mechanism in webhdfs, which is
a bit hacky. The other proposed solution is to do the same kind of hack at
application side, which means the same hack need to be applied in each
application.
Thanks [~daryn], [~wheat9], [~jingzhao], [~tucu00] and [~atm] for the
discussion in HDFS-6776.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Assigned] (HDFS-6919) Enforce a single limit for RAM disk usage and replicas cached via locking
[ https://issues.apache.org/jira/browse/HDFS-6919?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe reassigned HDFS-6919: -- Assignee: Colin Patrick McCabe Enforce a single limit for RAM disk usage and replicas cached via locking - Key: HDFS-6919 URL: https://issues.apache.org/jira/browse/HDFS-6919 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Arpit Agarwal Assignee: Colin Patrick McCabe Priority: Blocker The DataNode can have a single limit for memory usage which applies to both replicas cached via CCM and replicas on RAM disk. See comments [1|https://issues.apache.org/jira/browse/HDFS-6581?focusedCommentId=14106025page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14106025], [2|https://issues.apache.org/jira/browse/HDFS-6581?focusedCommentId=14106245page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14106245] and [3|https://issues.apache.org/jira/browse/HDFS-6581?focusedCommentId=14106575page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14106575] for discussion. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (HDFS-7104) Fix and clarify INodeInPath getter functions
[
https://issues.apache.org/jira/browse/HDFS-7104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14147018#comment-14147018
]
Zhe Zhang commented on HDFS-7104:
-
Good catch! How about the below:
{code:java}
/**
* @return a new array of inodes excluding the null elements introduced by
snapshot
* path elements. E.g., for path /dir/.snapshot, {@link #inodes} includes 3
elements,
* for /, dir, and .snapshot, while the returned array only contains
inodes of /
* and dir. Note the length of the returned array is always equal to
{@link #capacity}.
*/
{code}
Fix and clarify INodeInPath getter functions
Key: HDFS-7104
URL: https://issues.apache.org/jira/browse/HDFS-7104
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Priority: Minor
Attachments: HDFS-7104-20140923-v1.patch,
HDFS-7104-20140923-v3.patch, HDFS-7104-20140924-v1.patch,
HDFS-7104-20140924-v2.patch
inodes is initialized with the number of patch components. After resolve, it
contains both non-null and null elements (introduced by dot-snapshot dirs).
When getINodes is called, an array is returned excluding all non elements,
which is the correct behavior. Meanwhile, the inodes array is trimmed too,
which shouldn't be done by a getter.
Because of the above, the behavior of getINodesInPath depends on whether
getINodes has been called, which is not correct.
The name of getLastINodeInPath is confusing – it actually returns the last
non-null inode in the path. Also, shouldn't the return type be a single INode?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7104) Fix and clarify INodeInPath getter functions
[
https://issues.apache.org/jira/browse/HDFS-7104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14147035#comment-14147035
]
Jing Zhao commented on HDFS-7104:
-
Thanks for the fix!
bq. for path /dir/.snapshot, {@link #inodes} includes 3 elements, for /,
dir, and .snapshot
Maybe we can change it to After resolving the path /dir/.snapshot, {@link
#inodes} == {/, dir, null}. Other than this it looks good to me.
Fix and clarify INodeInPath getter functions
Key: HDFS-7104
URL: https://issues.apache.org/jira/browse/HDFS-7104
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Priority: Minor
Attachments: HDFS-7104-20140923-v1.patch,
HDFS-7104-20140923-v3.patch, HDFS-7104-20140924-v1.patch,
HDFS-7104-20140924-v2.patch
inodes is initialized with the number of patch components. After resolve, it
contains both non-null and null elements (introduced by dot-snapshot dirs).
When getINodes is called, an array is returned excluding all non elements,
which is the correct behavior. Meanwhile, the inodes array is trimmed too,
which shouldn't be done by a getter.
Because of the above, the behavior of getINodesInPath depends on whether
getINodes has been called, which is not correct.
The name of getLastINodeInPath is confusing – it actually returns the last
non-null inode in the path. Also, shouldn't the return type be a single INode?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7036) HDFS-6776 fix requires to upgrade insecure cluster, which means quite some user pain
[
https://issues.apache.org/jira/browse/HDFS-7036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14147038#comment-14147038
]
Yongjun Zhang commented on HDFS-7036:
-
Hi [~wheat9],
One example I described several times and in my previous comment is issuing
{{hadoop fs -ls insecureCluster}} from secure cluster side. It's broken for
the same reason. Do you suggest to hack the fsshell too or leave it as broken
as is? In addition, imagine user could write their own applications to access
data in both secure and insecure cluster.
Would you please provide some *concrete example of potential damage?
Thanks.
HDFS-6776 fix requires to upgrade insecure cluster, which means quite some
user pain
Key: HDFS-7036
URL: https://issues.apache.org/jira/browse/HDFS-7036
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Affects Versions: 2.5.1
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
Attachments: HDFS-7036.001.patch
Issuing command
{code}
hadoop fs -lsr webhdfs://insecureCluster
{code}
at a secure cluster side fails with message Failed to get the token ...,
similar symptom as reported in HDFS-6776.
If the fix of HDFS-6776 is applied to only the secure cluster, doing
{code}
distcp webhdfs://insecureCluster secureCluster
{code}
would fail same way.
Basically running any application in secure cluster to access insecure
cluster via webhdfs would fail the same way, if the HDFS-6776 fix is not
applied to the insecure cluster.
This could be quite some user pain. Filing this jira for a solution to make
user's life easier.
One proposed solution was to add a msg-parsing mechanism in webhdfs, which is
a bit hacky. The other proposed solution is to do the same kind of hack at
application side, which means the same hack need to be applied in each
application.
Thanks [~daryn], [~wheat9], [~jingzhao], [~tucu00] and [~atm] for the
discussion in HDFS-6776.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Comment Edited] (HDFS-7104) Fix and clarify INodeInPath getter functions
[
https://issues.apache.org/jira/browse/HDFS-7104?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14147035#comment-14147035
]
Jing Zhao edited comment on HDFS-7104 at 9/24/14 10:58 PM:
---
Thanks for the fix!
bq. for path /dir/.snapshot, inodes includes 3 elements, for /, dir, and
.snapshot.
Maybe we can change it to After resolving the path /dir/.snapshot,
{{inodes}} is {/, dir, null}. Other than this it looks good to me.
was (Author: jingzhao):
Thanks for the fix!
bq. for path /dir/.snapshot, {@link #inodes} includes 3 elements, for /,
dir, and .snapshot
Maybe we can change it to After resolving the path /dir/.snapshot, {@link
#inodes} == {/, dir, null}. Other than this it looks good to me.
Fix and clarify INodeInPath getter functions
Key: HDFS-7104
URL: https://issues.apache.org/jira/browse/HDFS-7104
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Priority: Minor
Attachments: HDFS-7104-20140923-v1.patch,
HDFS-7104-20140923-v3.patch, HDFS-7104-20140924-v1.patch,
HDFS-7104-20140924-v2.patch
inodes is initialized with the number of patch components. After resolve, it
contains both non-null and null elements (introduced by dot-snapshot dirs).
When getINodes is called, an array is returned excluding all non elements,
which is the correct behavior. Meanwhile, the inodes array is trimmed too,
which shouldn't be done by a getter.
Because of the above, the behavior of getINodesInPath depends on whether
getINodes has been called, which is not correct.
The name of getLastINodeInPath is confusing – it actually returns the last
non-null inode in the path. Also, shouldn't the return type be a single INode?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (HDFS-6932) Balancer and Mover tools should ignore replicas on RAM_DISK
[ https://issues.apache.org/jira/browse/HDFS-6932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Xiaoyu Yao updated HDFS-6932: - Attachment: HDFS-6932.3.patch Minor update to correct the timeout value for testBalancerWithRamDisk Balancer and Mover tools should ignore replicas on RAM_DISK --- Key: HDFS-6932 URL: https://issues.apache.org/jira/browse/HDFS-6932 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode Affects Versions: HDFS-6581 Reporter: Arpit Agarwal Assignee: Xiaoyu Yao Attachments: HDFS-6932.0.patch, HDFS-6932.1.patch, HDFS-6932.2.patch, HDFS-6932.3.patch Per title, balancer and mover should just ignore replicas on RAM disk instead of attempting to move them to other nodes. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7140) Add a tool to list all the existing block storage policies
[ https://issues.apache.org/jira/browse/HDFS-7140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-7140: -- Hadoop Flags: Reviewed +1 patch looks good. Thanks. Add a tool to list all the existing block storage policies -- Key: HDFS-7140 URL: https://issues.apache.org/jira/browse/HDFS-7140 Project: Hadoop HDFS Issue Type: Sub-task Components: balancer, namenode Affects Versions: 3.0.0 Reporter: Jing Zhao Assignee: Jing Zhao Priority: Minor Attachments: HDFS-7140.000.patch, HDFS-7140.001.patch, HDFS-7140.002.patch After adding a new DistributedFileSystem API to allow normal users to get all the block storage policies, we should also provide a simple tool to list these policies. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7140) Add a tool to list all the existing block storage policies
[ https://issues.apache.org/jira/browse/HDFS-7140?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-7140: -- Status: Patch Available (was: Open) Add a tool to list all the existing block storage policies -- Key: HDFS-7140 URL: https://issues.apache.org/jira/browse/HDFS-7140 Project: Hadoop HDFS Issue Type: Sub-task Components: balancer, namenode Affects Versions: 3.0.0 Reporter: Jing Zhao Assignee: Jing Zhao Priority: Minor Attachments: HDFS-7140.000.patch, HDFS-7140.001.patch, HDFS-7140.002.patch After adding a new DistributedFileSystem API to allow normal users to get all the block storage policies, we should also provide a simple tool to list these policies. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7077) Separate CipherSuite from crypto protocol version
[ https://issues.apache.org/jira/browse/HDFS-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-7077: -- Attachment: hdfs-7077.001.patch Patch attached. The main thing is using EncryptionProtocolVersion rather than trying to arbitrate based on the CipherSuite. Now, the CipherSuite is simply chosen based on the provided EZ key, and clients all need to use that CipherSuite. On write, the client provides its supported protocol versions. The NN chooses among these based on the encryption zone, throwing an exception if the client doesn't support the EZ's version. On read, the client checks the protocol version of the file and throws an exception if it doesn't support it. There are also a few other changes. I renamed the CipherSuite proto to CipherSuiteProto to make a static import possible. The CryptoCodec factory method was also throwing a runtime exception if the requested codec could not be created, which is counter to the javadoc and also an extreme response. Separate CipherSuite from crypto protocol version - Key: HDFS-7077 URL: https://issues.apache.org/jira/browse/HDFS-7077 Project: Hadoop HDFS Issue Type: Sub-task Components: encryption Affects Versions: 2.6.0 Reporter: Andrew Wang Assignee: Andrew Wang Attachments: hdfs-7077.001.patch Right now the CipherSuite is used for protocol version negotiation, which is wrong. We need to separate it out. An EZ should be locked to a certain CipherSuite and protocol version. A client reading and writing to the EZ then needs to negotiate based on both of these parameters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7077) Separate CipherSuite from crypto protocol version
[ https://issues.apache.org/jira/browse/HDFS-7077?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-7077: -- Status: Patch Available (was: Open) Separate CipherSuite from crypto protocol version - Key: HDFS-7077 URL: https://issues.apache.org/jira/browse/HDFS-7077 Project: Hadoop HDFS Issue Type: Sub-task Components: encryption Affects Versions: 2.6.0 Reporter: Andrew Wang Assignee: Andrew Wang Attachments: hdfs-7077.001.patch Right now the CipherSuite is used for protocol version negotiation, which is wrong. We need to separate it out. An EZ should be locked to a certain CipherSuite and protocol version. A client reading and writing to the EZ then needs to negotiate based on both of these parameters. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (HDFS-7104) Fix and clarify INodeInPath getter functions
[
https://issues.apache.org/jira/browse/HDFS-7104?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhe Zhang updated HDFS-7104:
Attachment: HDFS-7104-20140924-v3.patch
Fixing example in Javadoc of {{getINodes()}}
Fix and clarify INodeInPath getter functions
Key: HDFS-7104
URL: https://issues.apache.org/jira/browse/HDFS-7104
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Zhe Zhang
Assignee: Zhe Zhang
Priority: Minor
Attachments: HDFS-7104-20140923-v1.patch,
HDFS-7104-20140923-v3.patch, HDFS-7104-20140924-v1.patch,
HDFS-7104-20140924-v2.patch, HDFS-7104-20140924-v3.patch
inodes is initialized with the number of patch components. After resolve, it
contains both non-null and null elements (introduced by dot-snapshot dirs).
When getINodes is called, an array is returned excluding all non elements,
which is the correct behavior. Meanwhile, the inodes array is trimmed too,
which shouldn't be done by a getter.
Because of the above, the behavior of getINodesInPath depends on whether
getINodes has been called, which is not correct.
The name of getLastINodeInPath is confusing – it actually returns the last
non-null inode in the path. Also, shouldn't the return type be a single INode?
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (HDFS-7138) Fix hftp to work with encryption
[ https://issues.apache.org/jira/browse/HDFS-7138?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14147088#comment-14147088 ] Andrew Wang commented on HDFS-7138: --- Thanks Charles, LGTM. Could you quickly comment on manual testing? I noticed that you didn't add a unit test reading encrypted data via HFTP, but I'm okay with manual only since HFTP is deprecated. Fix hftp to work with encryption Key: HDFS-7138 URL: https://issues.apache.org/jira/browse/HDFS-7138 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 2.6.0 Reporter: Charles Lamb Assignee: Charles Lamb Attachments: HDFS-7138.001.patch Users cannot read files using hftp. bash-4.1$ hadoop fs -cat /enc/hello hello and goodbye bash-4.1$ hadoop fs -cat hftp://localhost:50070/enc/hello 14/09/16 10:56:52 WARN ssl.FileBasedKeyStoresFactory: The property 'ssl.client.truststore.location' has not been set, no TrustStore will be loaded P5/-?K?fbash-4.1$ -- This message was sent by Atlassian JIRA (v6.3.4#6332)
