[jira] [Commented] (HDFS-6577) Failure to create FinalizeCommand
[
https://issues.apache.org/jira/browse/HDFS-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039231#comment-14039231
]
Yongjun Zhang commented on HDFS-6577:
-
Thanks a lot Arpit! I will post a patch later today.
Failure to create FinalizeCommand
-
Key: HDFS-6577
URL: https://issues.apache.org/jira/browse/HDFS-6577
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.4.0
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
I visually see a bug in NameNodeRpcServer.java,
{code}
@Override // DatanodeProtocol
public DatanodeCommand blockReport(DatanodeRegistration nodeReg,
String poolId, StorageBlockReport[] reports) throws IOException {
verifyRequest(nodeReg);
if(blockStateChangeLog.isDebugEnabled()) {
blockStateChangeLog.debug(*BLOCK* NameNode.blockReport:
+ from + nodeReg + , reports.length= + reports.length);
}
final BlockManager bm = namesystem.getBlockManager();
boolean noStaleStorages = false;
for(StorageBlockReport r : reports) {
final BlockListAsLongs blocks = new BlockListAsLongs(r.getBlocks());
noStaleStorages = bm.processReport(nodeReg, r.getStorage(), poolId,
blocks);
metrics.incrStorageBlockReportOps();
}
if (nn.getFSImage().isUpgradeFinalized()
!nn.isStandbyState()
noStaleStorages) {
return new FinalizeCommand(poolId);
}
return null;
}
{code}
noStaleStorage is initialized to false, then in the for loop {{
for(StorageBlockReport r : reports) {}} it could be changed to true.
The problem is, since it's a loop of all reports, the value returned from the
last blockreport (the last iteration) will set it to the final value. Even if
it was set to true in the middle of a loop, the final iteration could
overwrite it to false. Similarly, if it was set to false in the middle, and
the final iter could overwrite it to true.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6579) TestSocketIOWithTimeout#testSocketIOWithTimeout fails on Power PC
[ https://issues.apache.org/jira/browse/HDFS-6579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated HDFS-6579: Fix Version/s: (was: 2.4.1) TestSocketIOWithTimeout#testSocketIOWithTimeout fails on Power PC -- Key: HDFS-6579 URL: https://issues.apache.org/jira/browse/HDFS-6579 Project: Hadoop HDFS Issue Type: Bug Components: test Affects Versions: 2.1.0-beta, 2.0.4-alpha, 2.2.0, 2.3.0, 2.4.0 Reporter: Jinghui Wang Attachments: HDFS-6579.patch SocketOutputStream closes its writer if it's partial written. But on PPC, after writing for some time, buf.capacity still equals buf.remaining. The reason might be what's written on PPC is buffered,so the buf.remaining will not change till a flush. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6222) Remove background token renewer from webhdfs
[ https://issues.apache.org/jira/browse/HDFS-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HDFS-6222: Hadoop Flags: Reviewed +1 for the patch. Rushabh, thank you for addressing the feedback. Can you please also provide an up-to-date branch-2 patch? I'd be happy to take care of committing this for you. Remove background token renewer from webhdfs Key: HDFS-6222 URL: https://issues.apache.org/jira/browse/HDFS-6222 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Attachments: HDFS-6222.branch-2-v2.patch, HDFS-6222.branch-2.patch, HDFS-6222.branch-2.patch, HDFS-6222.trunk-v2.patch, HDFS-6222.trunk-v2.patch, HDFS-6222.trunk-v3.patch, HDFS-6222.trunk.patch, HDFS-6222.trunk.patch The background token renewer is a source of problems for long-running daemons. Webhdfs should lazy fetch a new token when it receives an InvalidToken exception. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6222) Remove background token renewer from webhdfs
[ https://issues.apache.org/jira/browse/HDFS-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rushabh S Shah updated HDFS-6222: - Attachment: HDFS-6222.branch-2-v3.patch Thanks Chris for the review. Please find attached the revised patch for branch-2. Remove background token renewer from webhdfs Key: HDFS-6222 URL: https://issues.apache.org/jira/browse/HDFS-6222 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Attachments: HDFS-6222.branch-2-v2.patch, HDFS-6222.branch-2-v3.patch, HDFS-6222.branch-2.patch, HDFS-6222.branch-2.patch, HDFS-6222.trunk-v2.patch, HDFS-6222.trunk-v2.patch, HDFS-6222.trunk-v3.patch, HDFS-6222.trunk.patch, HDFS-6222.trunk.patch The background token renewer is a source of problems for long-running daemons. Webhdfs should lazy fetch a new token when it receives an InvalidToken exception. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6222) Remove background token renewer from webhdfs
[
https://issues.apache.org/jira/browse/HDFS-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039273#comment-14039273
]
Hadoop QA commented on HDFS-6222:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651716/HDFS-6222.branch-2-v3.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7191//console
This message is automatically generated.
Remove background token renewer from webhdfs
Key: HDFS-6222
URL: https://issues.apache.org/jira/browse/HDFS-6222
Project: Hadoop HDFS
Issue Type: Bug
Components: webhdfs
Affects Versions: 2.0.0-alpha, 3.0.0
Reporter: Daryn Sharp
Assignee: Daryn Sharp
Attachments: HDFS-6222.branch-2-v2.patch,
HDFS-6222.branch-2-v3.patch, HDFS-6222.branch-2.patch,
HDFS-6222.branch-2.patch, HDFS-6222.trunk-v2.patch, HDFS-6222.trunk-v2.patch,
HDFS-6222.trunk-v3.patch, HDFS-6222.trunk.patch, HDFS-6222.trunk.patch
The background token renewer is a source of problems for long-running
daemons. Webhdfs should lazy fetch a new token when it receives an
InvalidToken exception.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6475) WebHdfs clients fail without retry because incorrect handling of StandbyException
[
https://issues.apache.org/jira/browse/HDFS-6475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039327#comment-14039327
]
Daryn Sharp commented on HDFS-6475:
---
What I'm saying is I think the patch adds too much unnecessary code. Filing an
improvement to delete all but a few lines of the code changed in this patch
seems a bit odd. I think you just need to:
# Delete {{getTrueCause}} entirely instead of moving it elsewhere
# In {{saslProcess}}, just throw the exception instead of running it through
{{getTrueCause}} since it's not a InvalidToken wrapping another exception
anymore.
# Keep your 3-line change to unwrap {{SecurityException}} in {{toResponse}}
If it turns out to be a lot more complicated, then perhaps a followup jira is
ok, but since what should be the cleanest fix is 4 lines of change it would be
nice to try that first.
WebHdfs clients fail without retry because incorrect handling of
StandbyException
-
Key: HDFS-6475
URL: https://issues.apache.org/jira/browse/HDFS-6475
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, webhdfs
Affects Versions: 2.4.0
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
Attachments: HDFS-6475.001.patch, HDFS-6475.002.patch,
HDFS-6475.003.patch, HDFS-6475.003.patch, HDFS-6475.004.patch,
HDFS-6475.005.patch, HDFS-6475.006.patch
With WebHdfs clients connected to a HA HDFS service, the delegation token is
previously initialized with the active NN.
When clients try to issue request, the NN it contacts is stored in a map
returned by DFSUtil.getNNServiceRpcAddresses(conf). And the client contact
the NN based on the order, so likely the first one it runs into is StandbyNN.
If the StandbyNN doesn't have the updated client crediential, it will throw a
s SecurityException that wraps StandbyException.
The client is expected to retry another NN, but due to the insufficient
handling of SecurityException mentioned above, it failed.
Example message:
{code}
{RemoteException={message=Failed to obtain user group information:
org.apache.hadoop.security.token.SecretManager$InvalidToken:
StandbyException, javaCl
assName=java.lang.SecurityException, exception=SecurityException}}
org.apache.hadoop.ipc.RemoteException(java.lang.SecurityException): Failed to
obtain user group information:
org.apache.hadoop.security.token.SecretManager$InvalidToken: StandbyException
at
org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:159)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:325)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$700(WebHdfsFileSystem.java:107)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.getResponse(WebHdfsFileSystem.java:635)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:542)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:431)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:685)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:696)
at kclient1.kclient$1.run(kclient.java:64)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:356)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1528)
at kclient1.kclient.main(kclient.java:58)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6475) WebHdfs clients fail without retry because incorrect handling of StandbyException
[
https://issues.apache.org/jira/browse/HDFS-6475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039330#comment-14039330
]
Daryn Sharp commented on HDFS-6475:
---
I'm +0 and defer to atm if the fix is harder than what I just suggested.
WebHdfs clients fail without retry because incorrect handling of
StandbyException
-
Key: HDFS-6475
URL: https://issues.apache.org/jira/browse/HDFS-6475
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, webhdfs
Affects Versions: 2.4.0
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
Attachments: HDFS-6475.001.patch, HDFS-6475.002.patch,
HDFS-6475.003.patch, HDFS-6475.003.patch, HDFS-6475.004.patch,
HDFS-6475.005.patch, HDFS-6475.006.patch
With WebHdfs clients connected to a HA HDFS service, the delegation token is
previously initialized with the active NN.
When clients try to issue request, the NN it contacts is stored in a map
returned by DFSUtil.getNNServiceRpcAddresses(conf). And the client contact
the NN based on the order, so likely the first one it runs into is StandbyNN.
If the StandbyNN doesn't have the updated client crediential, it will throw a
s SecurityException that wraps StandbyException.
The client is expected to retry another NN, but due to the insufficient
handling of SecurityException mentioned above, it failed.
Example message:
{code}
{RemoteException={message=Failed to obtain user group information:
org.apache.hadoop.security.token.SecretManager$InvalidToken:
StandbyException, javaCl
assName=java.lang.SecurityException, exception=SecurityException}}
org.apache.hadoop.ipc.RemoteException(java.lang.SecurityException): Failed to
obtain user group information:
org.apache.hadoop.security.token.SecretManager$InvalidToken: StandbyException
at
org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:159)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:325)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$700(WebHdfsFileSystem.java:107)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.getResponse(WebHdfsFileSystem.java:635)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:542)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:431)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:685)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:696)
at kclient1.kclient$1.run(kclient.java:64)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:356)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1528)
at kclient1.kclient.main(kclient.java:58)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6475) WebHdfs clients fail without retry because incorrect handling of StandbyException
[
https://issues.apache.org/jira/browse/HDFS-6475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039361#comment-14039361
]
Yongjun Zhang commented on HDFS-6475:
-
Many thanks [~daryn], I will work on a new revision per your suggestion.
WebHdfs clients fail without retry because incorrect handling of
StandbyException
-
Key: HDFS-6475
URL: https://issues.apache.org/jira/browse/HDFS-6475
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, webhdfs
Affects Versions: 2.4.0
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
Attachments: HDFS-6475.001.patch, HDFS-6475.002.patch,
HDFS-6475.003.patch, HDFS-6475.003.patch, HDFS-6475.004.patch,
HDFS-6475.005.patch, HDFS-6475.006.patch
With WebHdfs clients connected to a HA HDFS service, the delegation token is
previously initialized with the active NN.
When clients try to issue request, the NN it contacts is stored in a map
returned by DFSUtil.getNNServiceRpcAddresses(conf). And the client contact
the NN based on the order, so likely the first one it runs into is StandbyNN.
If the StandbyNN doesn't have the updated client crediential, it will throw a
s SecurityException that wraps StandbyException.
The client is expected to retry another NN, but due to the insufficient
handling of SecurityException mentioned above, it failed.
Example message:
{code}
{RemoteException={message=Failed to obtain user group information:
org.apache.hadoop.security.token.SecretManager$InvalidToken:
StandbyException, javaCl
assName=java.lang.SecurityException, exception=SecurityException}}
org.apache.hadoop.ipc.RemoteException(java.lang.SecurityException): Failed to
obtain user group information:
org.apache.hadoop.security.token.SecretManager$InvalidToken: StandbyException
at
org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:159)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:325)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$700(WebHdfsFileSystem.java:107)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.getResponse(WebHdfsFileSystem.java:635)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:542)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:431)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:685)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:696)
at kclient1.kclient$1.run(kclient.java:64)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:356)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1528)
at kclient1.kclient.main(kclient.java:58)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6580) 2 internal APIs in FSNamesystem use raw dir.getFileInfo() to getAuditFileInfo()
[
https://issues.apache.org/jira/browse/HDFS-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039376#comment-14039376
]
Hadoop QA commented on HDFS-6580:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651704/patch_cadc3030359f6c317e46d43c9f7ef61c5576a872.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.namenode.TestFSImageWithSnapshot
org.apache.hadoop.hdfs.TestLease
org.apache.hadoop.hdfs.server.namenode.snapshot.TestRenameWithSnapshots
org.apache.hadoop.hdfs.server.namenode.snapshot.TestSnapshot
org.apache.hadoop.hdfs.server.namenode.snapshot.TestNestedSnapshots
org.apache.hadoop.hdfs.TestDFSClientRetries
org.apache.hadoop.hdfs.server.namenode.TestFsck
org.apache.hadoop.hdfs.server.namenode.TestAuditLogs
org.apache.hadoop.hdfs.server.namenode.ha.TestDNFencingWithReplication
org.apache.hadoop.hdfs.server.namenode.snapshot.TestINodeFileUnderConstructionWithSnapshot
org.apache.hadoop.hdfs.TestDatanodeBlockScanner
org.apache.hadoop.hdfs.server.blockmanagement.TestOverReplicatedBlocks
org.apache.hadoop.hdfs.TestFileAppend4
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithMultipleNameNodes
org.apache.hadoop.hdfs.server.namenode.web.resources.TestWebHdfsDataLocality
org.apache.hadoop.hdfs.web.TestWebHdfsWithMultipleNameNodes
org.apache.hadoop.hdfs.TestReplication
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7190//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7190//console
This message is automatically generated.
2 internal APIs in FSNamesystem use raw dir.getFileInfo() to
getAuditFileInfo()
---
Key: HDFS-6580
URL: https://issues.apache.org/jira/browse/HDFS-6580
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Reporter: Zhilei Xu
Assignee: Zhilei Xu
Labels: patch
Attachments: patch_cadc3030359f6c317e46d43c9f7ef61c5576a872.txt
In FSNamesystem.java, getAuditFileInfo() is the canonical way to get file
info for auditing purpose. getAuditFileInfo() returns null when auditing is
disabled, and calls dir.getFileInfo() when auditing is enabled.
2 internal APIs, startFileInt() and mkdirsInt() mistakenly use the raw
dir.getFileInfo() to get file info for auditing. Should change to
getAuditFileInfo().
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HDFS-6582) Missing null check in RpcProgramNfs3#read(XDR, SecurityHandler)
Ted Yu created HDFS-6582:
Summary: Missing null check in RpcProgramNfs3#read(XDR,
SecurityHandler)
Key: HDFS-6582
URL: https://issues.apache.org/jira/browse/HDFS-6582
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Ted Yu
Priority: Minor
Around line 691:
{code}
FSDataInputStream fis = clientCache.getDfsInputStream(userName,
Nfs3Utils.getFileIdPath(handle));
try {
readCount = fis.read(offset, readbuffer, 0, count);
{code}
fis may be null, leading to NullPointerException
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6572) Add an option to the NameNode that prints the software and on-disk image versions
[ https://issues.apache.org/jira/browse/HDFS-6572?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HDFS-6572: --- Attachment: HDFS-6572.002.patch New patch with unit test. Previous unit test failure was a spurious timeout. Add an option to the NameNode that prints the software and on-disk image versions - Key: HDFS-6572 URL: https://issues.apache.org/jira/browse/HDFS-6572 Project: Hadoop HDFS Issue Type: Bug Components: namenode Reporter: Charles Lamb Assignee: Charles Lamb Priority: Minor Attachments: HDFS-6572.001.patch, HDFS-6572.002.patch The HDFS namenode should have a startup option that prints the metadata versions of both the software and the on-disk version. This will be useful for debugging certain situations. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Reopened] (HDFS-6391) Get the Key/IV from the NameNode for encrypted files in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-6391?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang reopened HDFS-6391: --- Get the Key/IV from the NameNode for encrypted files in DFSClient - Key: HDFS-6391 URL: https://issues.apache.org/jira/browse/HDFS-6391 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode, security Reporter: Alejandro Abdelnur Assignee: Andrew Wang Attachments: HDFS-6391.1.patch When creating/opening and encrypted file, the DFSClient should get the encryption key material and the IV for the file in the create/open RPC call. HDFS admin users would never get key material/IV on encrypted files create/open. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6583) Remove clientNode in FileUnderConstructionFeature
Haohui Mai created HDFS-6583:
Summary: Remove clientNode in FileUnderConstructionFeature
Key: HDFS-6583
URL: https://issues.apache.org/jira/browse/HDFS-6583
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Haohui Mai
Attachments: HDFS-6583.000.patch
{{FileUnderConstructionFeature}} contains two fields {{clientMachine}} and
{{clientNode}}. {{clientNode}} keeps a reference of a {{DatanodeDescriptor}}.
The reference be recomputed by consulting {{DatanodeManager}}.
This jira proposes to remove {{clientNode}} in the
{{FileUnderConstructionFeature}} to simplify the code and to reduce overhead.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6583) Remove clientNode in FileUnderConstructionFeature
[
https://issues.apache.org/jira/browse/HDFS-6583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-6583:
-
Attachment: HDFS-6583.000.patch
Remove clientNode in FileUnderConstructionFeature
-
Key: HDFS-6583
URL: https://issues.apache.org/jira/browse/HDFS-6583
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Haohui Mai
Attachments: HDFS-6583.000.patch
{{FileUnderConstructionFeature}} contains two fields {{clientMachine}} and
{{clientNode}}. {{clientNode}} keeps a reference of a {{DatanodeDescriptor}}.
The reference be recomputed by consulting {{DatanodeManager}}.
This jira proposes to remove {{clientNode}} in the
{{FileUnderConstructionFeature}} to simplify the code and to reduce overhead.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6583) Remove clientNode in FileUnderConstructionFeature
[
https://issues.apache.org/jira/browse/HDFS-6583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-6583:
-
Status: Patch Available (was: Open)
Remove clientNode in FileUnderConstructionFeature
-
Key: HDFS-6583
URL: https://issues.apache.org/jira/browse/HDFS-6583
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Haohui Mai
Attachments: HDFS-6583.000.patch
{{FileUnderConstructionFeature}} contains two fields {{clientMachine}} and
{{clientNode}}. {{clientNode}} keeps a reference of a {{DatanodeDescriptor}}.
The reference be recomputed by consulting {{DatanodeManager}}.
This jira proposes to remove {{clientNode}} in the
{{FileUnderConstructionFeature}} to simplify the code and to reduce overhead.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6583) Remove clientNode in FileUnderConstructionFeature
[
https://issues.apache.org/jira/browse/HDFS-6583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039408#comment-14039408
]
Haohui Mai commented on HDFS-6583:
--
The v0 patch salvages most of the code from HDFS-5714.
Remove clientNode in FileUnderConstructionFeature
-
Key: HDFS-6583
URL: https://issues.apache.org/jira/browse/HDFS-6583
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Haohui Mai
Attachments: HDFS-6583.000.patch
{{FileUnderConstructionFeature}} contains two fields {{clientMachine}} and
{{clientNode}}. {{clientNode}} keeps a reference of a {{DatanodeDescriptor}}.
The reference be recomputed by consulting {{DatanodeManager}}.
This jira proposes to remove {{clientNode}} in the
{{FileUnderConstructionFeature}} to simplify the code and to reduce overhead.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6516) Persist and Restore the List of Encryption Zones
[ https://issues.apache.org/jira/browse/HDFS-6516?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HDFS-6516: --- Summary: Persist and Restore the List of Encryption Zones (was: Implement List Encryption Zones) Persist and Restore the List of Encryption Zones Key: HDFS-6516 URL: https://issues.apache.org/jira/browse/HDFS-6516 Project: Hadoop HDFS Issue Type: Sub-task Components: security Reporter: Charles Lamb Assignee: Charles Lamb Attachments: HDFS-6516.001.patch The list Encryption Zones command (CLI) and backend implementation (FSNamesystem) needs to be implemented. As part of this, the tests in TestEncryptionZonesAPI should be updated to use that to validate the results of the various CreateEZ and DeleteEZ tests. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6555) Specify file encryption attributes at create time
[ https://issues.apache.org/jira/browse/HDFS-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HDFS-6555: --- Summary: Specify file encryption attributes at create time (was: Create a crypto blob) Specify file encryption attributes at create time - Key: HDFS-6555 URL: https://issues.apache.org/jira/browse/HDFS-6555 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode, security Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134) Reporter: Charles Lamb Assignee: Charles Lamb We need to create a Crypto Blob for passing around crypto info. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6555) Specify file encryption attributes at create time
[ https://issues.apache.org/jira/browse/HDFS-6555?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039416#comment-14039416 ] Charles Lamb commented on HDFS-6555: This is important for distcp. Specify file encryption attributes at create time - Key: HDFS-6555 URL: https://issues.apache.org/jira/browse/HDFS-6555 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode, security Affects Versions: fs-encryption (HADOOP-10150 and HDFS-6134) Reporter: Charles Lamb Assignee: Charles Lamb We need to create a Crypto Blob for passing around crypto info. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6457) Maintain a list of all the Encryption Zones in the file system
[ https://issues.apache.org/jira/browse/HDFS-6457?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HDFS-6457: --- Issue Type: Bug (was: Sub-task) Parent: (was: HDFS-6134) Maintain a list of all the Encryption Zones in the file system -- Key: HDFS-6457 URL: https://issues.apache.org/jira/browse/HDFS-6457 Project: Hadoop HDFS Issue Type: Bug Components: namenode, security Reporter: Charles Lamb Assignee: Charles Lamb We need to maintain a list of all encryption zones in the file system so that we can ask questions about what EZ a path belongs to, if any, and let the admin know all the EZs in the system. [~andrew.wang] Why not just have a sorted structure with pointers to all the roots of the EZs? We can populate it during metadata loading on startup, and keep it updated during runtime. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6390) chown/chgrp users/groups blacklist for encrypted files
[ https://issues.apache.org/jira/browse/HDFS-6390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HDFS-6390: --- Issue Type: Bug (was: Sub-task) Parent: (was: HDFS-6134) chown/chgrp users/groups blacklist for encrypted files --- Key: HDFS-6390 URL: https://issues.apache.org/jira/browse/HDFS-6390 Project: Hadoop HDFS Issue Type: Bug Components: namenode, security Reporter: Alejandro Abdelnur Assignee: Charles Lamb A blacklist of users and groups that stops an admin from changing the owner/group of the file for encrypted files and directories. This blacklist would typically contain the regular users used by admins. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6393) User settable xAttr to stop HDFS admins from reading/chowning a file
[ https://issues.apache.org/jira/browse/HDFS-6393?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HDFS-6393: --- Issue Type: Bug (was: Sub-task) Parent: (was: HDFS-6134) User settable xAttr to stop HDFS admins from reading/chowning a file Key: HDFS-6393 URL: https://issues.apache.org/jira/browse/HDFS-6393 Project: Hadoop HDFS Issue Type: Bug Components: namenode, security Reporter: Alejandro Abdelnur Assignee: Charles Lamb A user should be able to set an xAttr on any file in HDFS to stop an HDFS admin user from reading the file. The blacklist for chown/chgrp would also enforced. This will stop an HDFS admin from gaining access to job token files and getting HDFS DelegationTokens that would allow him/her to read an encrypted file. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6475) WebHdfs clients fail without retry because incorrect handling of StandbyException
[
https://issues.apache.org/jira/browse/HDFS-6475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039425#comment-14039425
]
Yongjun Zhang commented on HDFS-6475:
-
HI [~daryn],
While I'm working on a revised version, I have a question. Your earlier
suggestion indicated that we should use
{{SecretManager#retriableRetrievePassword}} instead of
{{SecretManager#retrievePassword}}, does that mean client code has to be
modified? If so, is it a compatibility issue? thanks a lot.
WebHdfs clients fail without retry because incorrect handling of
StandbyException
-
Key: HDFS-6475
URL: https://issues.apache.org/jira/browse/HDFS-6475
Project: Hadoop HDFS
Issue Type: Bug
Components: ha, webhdfs
Affects Versions: 2.4.0
Reporter: Yongjun Zhang
Assignee: Yongjun Zhang
Attachments: HDFS-6475.001.patch, HDFS-6475.002.patch,
HDFS-6475.003.patch, HDFS-6475.003.patch, HDFS-6475.004.patch,
HDFS-6475.005.patch, HDFS-6475.006.patch
With WebHdfs clients connected to a HA HDFS service, the delegation token is
previously initialized with the active NN.
When clients try to issue request, the NN it contacts is stored in a map
returned by DFSUtil.getNNServiceRpcAddresses(conf). And the client contact
the NN based on the order, so likely the first one it runs into is StandbyNN.
If the StandbyNN doesn't have the updated client crediential, it will throw a
s SecurityException that wraps StandbyException.
The client is expected to retry another NN, but due to the insufficient
handling of SecurityException mentioned above, it failed.
Example message:
{code}
{RemoteException={message=Failed to obtain user group information:
org.apache.hadoop.security.token.SecretManager$InvalidToken:
StandbyException, javaCl
assName=java.lang.SecurityException, exception=SecurityException}}
org.apache.hadoop.ipc.RemoteException(java.lang.SecurityException): Failed to
obtain user group information:
org.apache.hadoop.security.token.SecretManager$InvalidToken: StandbyException
at
org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:159)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:325)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$700(WebHdfsFileSystem.java:107)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.getResponse(WebHdfsFileSystem.java:635)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:542)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:431)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:685)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:696)
at kclient1.kclient$1.run(kclient.java:64)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:356)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1528)
at kclient1.kclient.main(kclient.java:58)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6523) add unit tests for encryption zones api and cli
[ https://issues.apache.org/jira/browse/HDFS-6523?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HDFS-6523: --- Issue Type: Bug (was: Sub-task) Parent: (was: HDFS-6134) add unit tests for encryption zones api and cli --- Key: HDFS-6523 URL: https://issues.apache.org/jira/browse/HDFS-6523 Project: Hadoop HDFS Issue Type: Bug Components: namenode, security Reporter: Charles Lamb Assignee: Charles Lamb Add unit tests for EZ apis and CLI. This work will be done after HDFS-6386, HDFS-638, and HDFS-6516 are completed. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6508) Add an XAttr to specify the cipher mode
[ https://issues.apache.org/jira/browse/HDFS-6508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HDFS-6508: --- Issue Type: Bug (was: Sub-task) Parent: (was: HDFS-6134) Add an XAttr to specify the cipher mode --- Key: HDFS-6508 URL: https://issues.apache.org/jira/browse/HDFS-6508 Project: Hadoop HDFS Issue Type: Bug Components: namenode, security Reporter: Charles Lamb Assignee: Charles Lamb We should specify the cipher mode in the xattrs for compatibility sake. Crypto changes over time and we need to prepare for that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6562) Refactor rename() in FSDirectory
[
https://issues.apache.org/jira/browse/HDFS-6562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039426#comment-14039426
]
Daryn Sharp commented on HDFS-6562:
---
Are you sure this can't be a {{SnapshotAccessControlException}} (which seems
more appropriate) or tunneled out more cleanly?
{noformat}
+} catch (IOException e) {
+ // XXX: The client expects an exception in this case,
+ // and we can't subclass this exception due to backward compatibility
+ String msg = e.getMessage();
+ if (msg.contains( is snapshottable and already has snapshots)) {
{noformat}
Refactor rename() in FSDirectory
Key: HDFS-6562
URL: https://issues.apache.org/jira/browse/HDFS-6562
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-6562.000.patch, HDFS-6562.001.patch,
HDFS-6562.002.patch, HDFS-6562.003.patch
Currently there are two variants of {{rename()}} sitting in {{FSDirectory}}.
Both implementation shares quite a bit of common code.
This jira proposes to clean up these two variants and extract the common code.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6583) Remove clientNode in FileUnderConstructionFeature
[
https://issues.apache.org/jira/browse/HDFS-6583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039429#comment-14039429
]
Tsz Wo Nicholas Sze commented on HDFS-6583:
---
This is a good idea. Please as well remove
{code}
final DatanodeDescriptor clientNode =
blockManager.getDatanodeManager().getDatanodeByHost(clientMachine);
{code}
from appendFileInternal and startFileInternal in FSNamesystem.
Remove clientNode in FileUnderConstructionFeature
-
Key: HDFS-6583
URL: https://issues.apache.org/jira/browse/HDFS-6583
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Reporter: Haohui Mai
Attachments: HDFS-6583.000.patch
{{FileUnderConstructionFeature}} contains two fields {{clientMachine}} and
{{clientNode}}. {{clientNode}} keeps a reference of a {{DatanodeDescriptor}}.
The reference be recomputed by consulting {{DatanodeManager}}.
This jira proposes to remove {{clientNode}} in the
{{FileUnderConstructionFeature}} to simplify the code and to reduce overhead.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6583) Remove clientNode in FileUnderConstructionFeature
[
https://issues.apache.org/jira/browse/HDFS-6583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo Nicholas Sze updated HDFS-6583:
--
Component/s: namenode
Priority: Minor (was: Major)
Assignee: Haohui Mai
Remove clientNode in FileUnderConstructionFeature
-
Key: HDFS-6583
URL: https://issues.apache.org/jira/browse/HDFS-6583
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Reporter: Haohui Mai
Assignee: Haohui Mai
Priority: Minor
Attachments: HDFS-6583.000.patch
{{FileUnderConstructionFeature}} contains two fields {{clientMachine}} and
{{clientNode}}. {{clientNode}} keeps a reference of a {{DatanodeDescriptor}}.
The reference be recomputed by consulting {{DatanodeManager}}.
This jira proposes to remove {{clientNode}} in the
{{FileUnderConstructionFeature}} to simplify the code and to reduce overhead.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6583) Remove clientNode in FileUnderConstructionFeature
[
https://issues.apache.org/jira/browse/HDFS-6583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai updated HDFS-6583:
-
Attachment: HDFS-6583.001.patch
Remove clientNode in FileUnderConstructionFeature
-
Key: HDFS-6583
URL: https://issues.apache.org/jira/browse/HDFS-6583
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Reporter: Haohui Mai
Assignee: Haohui Mai
Priority: Minor
Attachments: HDFS-6583.000.patch, HDFS-6583.001.patch
{{FileUnderConstructionFeature}} contains two fields {{clientMachine}} and
{{clientNode}}. {{clientNode}} keeps a reference of a {{DatanodeDescriptor}}.
The reference be recomputed by consulting {{DatanodeManager}}.
This jira proposes to remove {{clientNode}} in the
{{FileUnderConstructionFeature}} to simplify the code and to reduce overhead.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6583) Remove clientNode in FileUnderConstructionFeature
[
https://issues.apache.org/jira/browse/HDFS-6583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039436#comment-14039436
]
Haohui Mai commented on HDFS-6583:
--
The v1 patch implements [~szetszwo]'s comments.
Remove clientNode in FileUnderConstructionFeature
-
Key: HDFS-6583
URL: https://issues.apache.org/jira/browse/HDFS-6583
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Reporter: Haohui Mai
Assignee: Haohui Mai
Priority: Minor
Attachments: HDFS-6583.000.patch, HDFS-6583.001.patch
{{FileUnderConstructionFeature}} contains two fields {{clientMachine}} and
{{clientNode}}. {{clientNode}} keeps a reference of a {{DatanodeDescriptor}}.
The reference be recomputed by consulting {{DatanodeManager}}.
This jira proposes to remove {{clientNode}} in the
{{FileUnderConstructionFeature}} to simplify the code and to reduce overhead.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Comment Edited] (HDFS-6562) Refactor rename() in FSDirectory
[
https://issues.apache.org/jira/browse/HDFS-6562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039444#comment-14039444
]
Jing Zhao edited comment on HDFS-6562 at 6/20/14 10:11 PM:
---
Thanks for the refactoring, Haohui! Just finished reviewing the changes for the
old rename. Some comments:
# Maybe we can let checkSnapshot throw a SnapshotException instead of the
current general IOException. In this way for the old version rename, we do not
need to check the exception msg when validating the rename source.
# Maybe we can rename RenameTransaction to RenameOperation?
# In RenameTransaction, the initialization of srcChildName, isSrcInSnapshot, and
scrChildIsReference can be moved into the constructor so that they can still
be declared as final.
# Since we no longer have INode replacement, we do not need to replace the last
inode of the srcIIP. Simply calling srcChild.recordModification should be good
enough.
{code}
if (isSrcInSnapshot) {
srcChild = srcChild.recordModification(srcIIP.getLatestSnapshotId());
srcIIP.setLastINode(srcChild);
}
{code}
# Similarly I guess we do not need the following code any more (this is in the
new rename):
{code}
if (dstParent.getParent() == null) {
// src and dst file/dir are in the same directory, and the dstParent has
// been replaced when we removed the src. Refresh the dstIIP and
// dstParent.
dstIIP = getINodesInPath4Write(dst, false);
}
{code}
And in removeLastINode(INodesInPath), we can remove the following code:
{code}
INodeDirectory newParent = last.getParent();
if (parent != newParent) {
iip.setINode(-2, newParent);
}
{code}
Will continue reviewing the new rename.
was (Author: jingzhao):
Thanks for the refactoring, Haohui! Just finished reviewing the changes for the
old rename. Some comments:
# Maybe we can let checkSnapshot throw a SnapshotException instead of the
current general IOException. In this way for the old version rename, we do not
need to check the exception msg when validating the rename source.
# Maybe we can rename RenameTransaction to RenameOperation?
# In RenameTransaction, the initialization of srcChildName, isSrcInSnapshot, and
scrChildIsReference can be moved into the constructor so that they can still
be declared as final.
# Since we no longer have INode replacement, we do not need to replace the last
inode of the srcIIP. Simply calling srcChild.recordModification should be good
enough.
{code}
if (isSrcInSnapshot) {
srcChild = srcChild.recordModification(srcIIP.getLatestSnapshotId());
srcIIP.setLastINode(srcChild);
}
{code}
# Similarly I guess we do not need the following code any more:
{code}
if (dstParent.getParent() == null) {
// src and dst file/dir are in the same directory, and the dstParent has
// been replaced when we removed the src. Refresh the dstIIP and
// dstParent.
dstIIP = getINodesInPath4Write(dst, false);
}
{code}
And in removeLastINode(INodesInPath), we can remove the following code:
{code}
INodeDirectory newParent = last.getParent();
if (parent != newParent) {
iip.setINode(-2, newParent);
}
{code}
Will continue reviewing the new rename.
Refactor rename() in FSDirectory
Key: HDFS-6562
URL: https://issues.apache.org/jira/browse/HDFS-6562
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-6562.000.patch, HDFS-6562.001.patch,
HDFS-6562.002.patch, HDFS-6562.003.patch
Currently there are two variants of {{rename()}} sitting in {{FSDirectory}}.
Both implementation shares quite a bit of common code.
This jira proposes to clean up these two variants and extract the common code.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6562) Refactor rename() in FSDirectory
[
https://issues.apache.org/jira/browse/HDFS-6562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039444#comment-14039444
]
Jing Zhao commented on HDFS-6562:
-
Thanks for the refactoring, Haohui! Just finished reviewing the changes for the
old rename. Some comments:
# Maybe we can let checkSnapshot throw a SnapshotException instead of the
current general IOException. In this way for the old version rename, we do not
need to check the exception msg when validating the rename source.
# Maybe we can rename RenameTransaction to RenameOperation?
# In RenameTransaction, the initialization of srcChildName, isSrcInSnapshot, and
scrChildIsReference can be moved into the constructor so that they can still
be declared as final.
# Since we no longer have INode replacement, we do not need to replace the last
inode of the srcIIP. Simply calling srcChild.recordModification should be good
enough.
{code}
if (isSrcInSnapshot) {
srcChild = srcChild.recordModification(srcIIP.getLatestSnapshotId());
srcIIP.setLastINode(srcChild);
}
{code}
# Similarly I guess we do not need the following code any more:
{code}
if (dstParent.getParent() == null) {
// src and dst file/dir are in the same directory, and the dstParent has
// been replaced when we removed the src. Refresh the dstIIP and
// dstParent.
dstIIP = getINodesInPath4Write(dst, false);
}
{code}
And in removeLastINode(INodesInPath), we can remove the following code:
{code}
INodeDirectory newParent = last.getParent();
if (parent != newParent) {
iip.setINode(-2, newParent);
}
{code}
Will continue reviewing the new rename.
Refactor rename() in FSDirectory
Key: HDFS-6562
URL: https://issues.apache.org/jira/browse/HDFS-6562
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-6562.000.patch, HDFS-6562.001.patch,
HDFS-6562.002.patch, HDFS-6562.003.patch
Currently there are two variants of {{rename()}} sitting in {{FSDirectory}}.
Both implementation shares quite a bit of common code.
This jira proposes to clean up these two variants and extract the common code.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-4667) Capture renamed files/directories in snapshot diff report
[ https://issues.apache.org/jira/browse/HDFS-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-4667: -- Component/s: (was: datanode) Assignee: Jing Zhao (was: Binglin Chang) Hadoop Flags: Reviewed +1 patch looks good. Thanks Jing. Capture renamed files/directories in snapshot diff report - Key: HDFS-4667 URL: https://issues.apache.org/jira/browse/HDFS-4667 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Jing Zhao Assignee: Jing Zhao Attachments: HDFS-4667.002.patch, HDFS-4667.002.patch, HDFS-4667.003.patch, HDFS-4667.004.patch, HDFS-4667.demo.patch, HDFS-4667.v1.patch, getfullname-snapshot-support.patch Currently in the diff report we only show file/dir creation, deletion and modification. After rename with snapshots is supported, renamed file/dir should also be captured in the diff report. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6391) Get the Key/IV from the NameNode for encrypted files in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-6391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039459#comment-14039459 ] Andrew Wang commented on HDFS-6391: --- I'm starting on this JIRA, and was wondering about how we should handle compatibility with regard to cipher types. I'm planning to add a FileEncryptionInfo class, which will incorporate a key, iv, and CipherType enum. Currently, our only CipherType is AES/CTR/NoPadding, but there were some plans in the works to use other CipherTypes as well. The question here relates to CryptoCodec. Right now, the factory is determined based on the client's conf, but for reading an encrypted file, it should be determined by the CipherType of the file. On the read-side, we also need to pass the CipherType from the client to the NN so it's stored correctly. [~hitliuyi], any thoughts on the best way of doing this? From my initial investigation, it looks like the strings used by javax.crypto.Cipher are the standard way of specifying a cipher type. If I provided a mapping from CipherType - javax.crypto.Cipher-style string, would that be sufficient? I'm also wondering about how the client and server will agree on which cipher to use. The NN needs to interpret the CipherType to know what KeyProvider operations are required. I think it could go something like this on the write-side: * Client specifies the list of CipherTypes it supports * NN filters the list by which it supports * Client chooses a CipherType, or errors out saying desired CipherType not supported * Client specifies its chosen CipherType with future commands. Read-side seems pretty trivial: * Client opens a file * NN returns the CipherType for the file * Client either has support for it and reads it, or can't Any thoughts here? Get the Key/IV from the NameNode for encrypted files in DFSClient - Key: HDFS-6391 URL: https://issues.apache.org/jira/browse/HDFS-6391 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode, security Reporter: Alejandro Abdelnur Assignee: Andrew Wang Attachments: HDFS-6391.1.patch When creating/opening and encrypted file, the DFSClient should get the encryption key material and the IV for the file in the create/open RPC call. HDFS admin users would never get key material/IV on encrypted files create/open. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6562) Refactor rename() in FSDirectory
[
https://issues.apache.org/jira/browse/HDFS-6562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039474#comment-14039474
]
Jing Zhao commented on HDFS-6562:
-
Comments for the remaining changes:
# I guess we can even put srcParent as a final field in RenameTransaction since
we will not replace the parent INode anymore.
# Can we have a validateOverwrite method to cover all the checks when rename
destination exists (the checks in the if (dstInode != null) section)?
Refactor rename() in FSDirectory
Key: HDFS-6562
URL: https://issues.apache.org/jira/browse/HDFS-6562
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-6562.000.patch, HDFS-6562.001.patch,
HDFS-6562.002.patch, HDFS-6562.003.patch
Currently there are two variants of {{rename()}} sitting in {{FSDirectory}}.
Both implementation shares quite a bit of common code.
This jira proposes to clean up these two variants and extract the common code.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6391) Get the Key/IV from the NameNode for encrypted files in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-6391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039479#comment-14039479 ] Mike Yoder commented on HDFS-6391: -- Regarding the write-side algo, the way TLS choose the CipherType is that the client sends the list of ciphers it supports, and the server side picks its favorite and returns it. Get the Key/IV from the NameNode for encrypted files in DFSClient - Key: HDFS-6391 URL: https://issues.apache.org/jira/browse/HDFS-6391 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode, security Reporter: Alejandro Abdelnur Assignee: Andrew Wang Attachments: HDFS-6391.1.patch When creating/opening and encrypted file, the DFSClient should get the encryption key material and the IV for the file in the create/open RPC call. HDFS admin users would never get key material/IV on encrypted files create/open. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6580) FSNamesystem.mkdirsInt uses raw dir.getFileInfo() to getAuditFileInfo()
[ https://issues.apache.org/jira/browse/HDFS-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhilei Xu updated HDFS-6580: Attachment: (was: patch_cadc3030359f6c317e46d43c9f7ef61c5576a872.txt) FSNamesystem.mkdirsInt uses raw dir.getFileInfo() to getAuditFileInfo() --- Key: HDFS-6580 URL: https://issues.apache.org/jira/browse/HDFS-6580 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Reporter: Zhilei Xu Assignee: Zhilei Xu Labels: patch In FSNamesystem.java, getAuditFileInfo() is the canonical way to get file info for auditing purpose. getAuditFileInfo() returns null when auditing is disabled, and calls dir.getFileInfo() when auditing is enabled. One internal APIs, mkdirsInt() mistakenly use the raw dir.getFileInfo() to get file info for auditing. Should change to getAuditFileInfo(). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6580) FSNamesystem.mkdirsInt uses raw dir.getFileInfo() to getAuditFileInfo()
[ https://issues.apache.org/jira/browse/HDFS-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhilei Xu updated HDFS-6580: Summary: FSNamesystem.mkdirsInt uses raw dir.getFileInfo() to getAuditFileInfo() (was: 2 internal APIs in FSNamesystem use raw dir.getFileInfo() to getAuditFileInfo()) FSNamesystem.mkdirsInt uses raw dir.getFileInfo() to getAuditFileInfo() --- Key: HDFS-6580 URL: https://issues.apache.org/jira/browse/HDFS-6580 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Reporter: Zhilei Xu Assignee: Zhilei Xu Labels: patch In FSNamesystem.java, getAuditFileInfo() is the canonical way to get file info for auditing purpose. getAuditFileInfo() returns null when auditing is disabled, and calls dir.getFileInfo() when auditing is enabled. 2 internal APIs, startFileInt() and mkdirsInt() mistakenly use the raw dir.getFileInfo() to get file info for auditing. Should change to getAuditFileInfo(). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6580) FSNamesystem.mkdirsInt uses raw dir.getFileInfo() to getAuditFileInfo()
[ https://issues.apache.org/jira/browse/HDFS-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhilei Xu updated HDFS-6580: Description: In FSNamesystem.java, getAuditFileInfo() is the canonical way to get file info for auditing purpose. getAuditFileInfo() returns null when auditing is disabled, and calls dir.getFileInfo() when auditing is enabled. One internal APIs, mkdirsInt() mistakenly use the raw dir.getFileInfo() to get file info for auditing. Should change to getAuditFileInfo(). was: In FSNamesystem.java, getAuditFileInfo() is the canonical way to get file info for auditing purpose. getAuditFileInfo() returns null when auditing is disabled, and calls dir.getFileInfo() when auditing is enabled. 2 internal APIs, startFileInt() and mkdirsInt() mistakenly use the raw dir.getFileInfo() to get file info for auditing. Should change to getAuditFileInfo(). FSNamesystem.mkdirsInt uses raw dir.getFileInfo() to getAuditFileInfo() --- Key: HDFS-6580 URL: https://issues.apache.org/jira/browse/HDFS-6580 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Reporter: Zhilei Xu Assignee: Zhilei Xu Labels: patch In FSNamesystem.java, getAuditFileInfo() is the canonical way to get file info for auditing purpose. getAuditFileInfo() returns null when auditing is disabled, and calls dir.getFileInfo() when auditing is enabled. One internal APIs, mkdirsInt() mistakenly use the raw dir.getFileInfo() to get file info for auditing. Should change to getAuditFileInfo(). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6584) Support archival storage
Tsz Wo Nicholas Sze created HDFS-6584: - Summary: Support archival storage Key: HDFS-6584 URL: https://issues.apache.org/jira/browse/HDFS-6584 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Reporter: Tsz Wo Nicholas Sze Assignee: Tsz Wo Nicholas Sze In most of the Hadoop clusters, as more and more data is stored for longer time, the demand for storage is outstripping the compute. Hadoop needs a cost effective and easy to manage solution to meet this demand for storage. Current solution is: - Delete the old unused data. This comes at operational cost of identifying unnecessary data and deleting them manually. - Add more nodes to the clusters. This adds along with storage capacity unnecessary compute capacity to the cluster. Hadoop needs a solution to decouple growing storage capacity from compute capacity. Nodes with higher density and less expensive storage with low compute power are becoming available and can be used as cold storage in the clusters. Based on policy the data from hot storage can be moved to cold storage. Adding more nodes to the cold storage can grow the storage independent of the compute capacity in the cluster. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-5682) Heterogeneous Storage phase 2 - APIs to expose Storage Types
[ https://issues.apache.org/jira/browse/HDFS-5682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tsz Wo Nicholas Sze updated HDFS-5682: -- Component/s: namenode datanode Issue Type: New Feature (was: Bug) Heterogeneous Storage phase 2 - APIs to expose Storage Types Key: HDFS-5682 URL: https://issues.apache.org/jira/browse/HDFS-5682 Project: Hadoop HDFS Issue Type: New Feature Components: datanode, namenode Affects Versions: 3.0.0 Reporter: Arpit Agarwal Assignee: Arpit Agarwal Attachments: 20140522-Heterogeneous-Storages-API.pdf Phase 1 (HDFS-2832) added support to present the DataNode as a collection of discrete storages of different types. This Jira is to track phase 2 of the Heterogeneous Storage work which involves exposing Storage Types to applications and adding Quota Management support for administrators. This phase will also include tools support for administrators/users. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6580) FSNamesystem.mkdirsInt uses raw dir.getFileInfo() to getAuditFileInfo()
[ https://issues.apache.org/jira/browse/HDFS-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhilei Xu updated HDFS-6580: Attachment: patch_c89bff2bb7a06bb2b0c66a85acbd5113db6b0526.txt FSNamesystem.mkdirsInt uses raw dir.getFileInfo() to getAuditFileInfo() --- Key: HDFS-6580 URL: https://issues.apache.org/jira/browse/HDFS-6580 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Reporter: Zhilei Xu Assignee: Zhilei Xu Labels: patch Attachments: patch_c89bff2bb7a06bb2b0c66a85acbd5113db6b0526.txt In FSNamesystem.java, getAuditFileInfo() is the canonical way to get file info for auditing purpose. getAuditFileInfo() returns null when auditing is disabled, and calls dir.getFileInfo() when auditing is enabled. One internal APIs, mkdirsInt() mistakenly use the raw dir.getFileInfo() to get file info for auditing. Should change to getAuditFileInfo(). -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6580) FSNamesystem.mkdirsInt uses raw dir.getFileInfo() to getAuditFileInfo()
[ https://issues.apache.org/jira/browse/HDFS-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhilei Xu updated HDFS-6580: Description: In FSNamesystem.java, getAuditFileInfo() is the canonical way to get file info for auditing purpose. getAuditFileInfo() returns null when auditing is disabled, and calls dir.getFileInfo() when auditing is enabled. One internal APIs, mkdirsInt() mistakenly use the raw dir.getFileInfo() to get file info for auditing. Should change to getAuditFileInfo(). Note that another internal API, startFileInt() uses dir.getFileInfo() correctly, because the returned file stat is returned out. was: In FSNamesystem.java, getAuditFileInfo() is the canonical way to get file info for auditing purpose. getAuditFileInfo() returns null when auditing is disabled, and calls dir.getFileInfo() when auditing is enabled. One internal APIs, mkdirsInt() mistakenly use the raw dir.getFileInfo() to get file info for auditing. Should change to getAuditFileInfo(). FSNamesystem.mkdirsInt uses raw dir.getFileInfo() to getAuditFileInfo() --- Key: HDFS-6580 URL: https://issues.apache.org/jira/browse/HDFS-6580 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Reporter: Zhilei Xu Assignee: Zhilei Xu Labels: patch Attachments: patch_c89bff2bb7a06bb2b0c66a85acbd5113db6b0526.txt In FSNamesystem.java, getAuditFileInfo() is the canonical way to get file info for auditing purpose. getAuditFileInfo() returns null when auditing is disabled, and calls dir.getFileInfo() when auditing is enabled. One internal APIs, mkdirsInt() mistakenly use the raw dir.getFileInfo() to get file info for auditing. Should change to getAuditFileInfo(). Note that another internal API, startFileInt() uses dir.getFileInfo() correctly, because the returned file stat is returned out. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-4667) Capture renamed files/directories in snapshot diff report
[ https://issues.apache.org/jira/browse/HDFS-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039491#comment-14039491 ] Jing Zhao commented on HDFS-4667: - Thanks for the review, Nicholas! [~decster], do you have any other comments? Capture renamed files/directories in snapshot diff report - Key: HDFS-4667 URL: https://issues.apache.org/jira/browse/HDFS-4667 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Jing Zhao Assignee: Jing Zhao Attachments: HDFS-4667.002.patch, HDFS-4667.002.patch, HDFS-4667.003.patch, HDFS-4667.004.patch, HDFS-4667.demo.patch, HDFS-4667.v1.patch, getfullname-snapshot-support.patch Currently in the diff report we only show file/dir creation, deletion and modification. After rename with snapshots is supported, renamed file/dir should also be captured in the diff report. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-6134: - Attachment: (was: HDFSDataAtRestEncryption.pdf) Transparent data at rest encryption --- Key: HDFS-6134 URL: https://issues.apache.org/jira/browse/HDFS-6134 Project: Hadoop HDFS Issue Type: New Feature Components: security Affects Versions: 2.3.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: ConceptualDesignProposal_2014-06-19.pdf Because of privacy and security regulations, for many industries, sensitive data at rest must be in encrypted form. For example: the healthÂcare industry (HIPAA regulations), the card payment industry (PCI DSS regulations) or the US government (FISMA regulations). This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can be used transparently by any application accessing HDFS via Hadoop Filesystem Java API, Hadoop libhdfs C library, or WebHDFS REST API. The resulting implementation should be able to be used in compliance with different regulation requirements. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HDFS-6134: --- Attachment: ConceptualDesignProposal_2014-06-20.pdf.pdf Revised one-pager. Transparent data at rest encryption --- Key: HDFS-6134 URL: https://issues.apache.org/jira/browse/HDFS-6134 Project: Hadoop HDFS Issue Type: New Feature Components: security Affects Versions: 2.3.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: ConceptualDesignProposal_2014-06-19.pdf, ConceptualDesignProposal_2014-06-20.pdf.pdf Because of privacy and security regulations, for many industries, sensitive data at rest must be in encrypted form. For example: the healthÂcare industry (HIPAA regulations), the card payment industry (PCI DSS regulations) or the US government (FISMA regulations). This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can be used transparently by any application accessing HDFS via Hadoop Filesystem Java API, Hadoop libhdfs C library, or WebHDFS REST API. The resulting implementation should be able to be used in compliance with different regulation requirements. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-6134: - Attachment: (was: ConceptualDesignProposal_2014-06-19.pdf) Transparent data at rest encryption --- Key: HDFS-6134 URL: https://issues.apache.org/jira/browse/HDFS-6134 Project: Hadoop HDFS Issue Type: New Feature Components: security Affects Versions: 2.3.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: ConceptualDesignProposal_2014-06-20.pdf.pdf Because of privacy and security regulations, for many industries, sensitive data at rest must be in encrypted form. For example: the healthÂcare industry (HIPAA regulations), the card payment industry (PCI DSS regulations) or the US government (FISMA regulations). This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can be used transparently by any application accessing HDFS via Hadoop Filesystem Java API, Hadoop libhdfs C library, or WebHDFS REST API. The resulting implementation should be able to be used in compliance with different regulation requirements. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb updated HDFS-6134: --- Attachment: ConceptualDesignProposal_2014-06-20.pdf Transparent data at rest encryption --- Key: HDFS-6134 URL: https://issues.apache.org/jira/browse/HDFS-6134 Project: Hadoop HDFS Issue Type: New Feature Components: security Affects Versions: 2.3.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: ConceptualDesignProposal_2014-06-20.pdf, HDFSDataatRestEncryptionProposal_obsolete.pdf Because of privacy and security regulations, for many industries, sensitive data at rest must be in encrypted form. For example: the healthÂcare industry (HIPAA regulations), the card payment industry (PCI DSS regulations) or the US government (FISMA regulations). This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can be used transparently by any application accessing HDFS via Hadoop Filesystem Java API, Hadoop libhdfs C library, or WebHDFS REST API. The resulting implementation should be able to be used in compliance with different regulation requirements. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-6134: - Attachment: (was: ConceptualDesignProposal_2014-06-20.pdf.pdf) Transparent data at rest encryption --- Key: HDFS-6134 URL: https://issues.apache.org/jira/browse/HDFS-6134 Project: Hadoop HDFS Issue Type: New Feature Components: security Affects Versions: 2.3.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: ConceptualDesignProposal_2014-06-20.pdf, HDFSDataatRestEncryptionProposal_obsolete.pdf Because of privacy and security regulations, for many industries, sensitive data at rest must be in encrypted form. For example: the healthÂcare industry (HIPAA regulations), the card payment industry (PCI DSS regulations) or the US government (FISMA regulations). This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can be used transparently by any application accessing HDFS via Hadoop Filesystem Java API, Hadoop libhdfs C library, or WebHDFS REST API. The resulting implementation should be able to be used in compliance with different regulation requirements. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-6134: - Attachment: HDFSDataatRestEncryptionProposal_obsolete.pdf Re-uploading the previous proposal as I've mistakenly deleted it. (using obsolete in the name to avoid confusion). Transparent data at rest encryption --- Key: HDFS-6134 URL: https://issues.apache.org/jira/browse/HDFS-6134 Project: Hadoop HDFS Issue Type: New Feature Components: security Affects Versions: 2.3.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: ConceptualDesignProposal_2014-06-20.pdf, HDFSDataatRestEncryptionProposal_obsolete.pdf Because of privacy and security regulations, for many industries, sensitive data at rest must be in encrypted form. For example: the healthÂcare industry (HIPAA regulations), the card payment industry (PCI DSS regulations) or the US government (FISMA regulations). This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can be used transparently by any application accessing HDFS via Hadoop Filesystem Java API, Hadoop libhdfs C library, or WebHDFS REST API. The resulting implementation should be able to be used in compliance with different regulation requirements. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-6134: - Attachment: (was: ConceptualDesignProposal_2014-06-20.pdf) Transparent data at rest encryption --- Key: HDFS-6134 URL: https://issues.apache.org/jira/browse/HDFS-6134 Project: Hadoop HDFS Issue Type: New Feature Components: security Affects Versions: 2.3.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HDFSDataatRestEncryptionProposal_obsolete.pdf Because of privacy and security regulations, for many industries, sensitive data at rest must be in encrypted form. For example: the healthÂcare industry (HIPAA regulations), the card payment industry (PCI DSS regulations) or the US government (FISMA regulations). This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can be used transparently by any application accessing HDFS via Hadoop Filesystem Java API, Hadoop libhdfs C library, or WebHDFS REST API. The resulting implementation should be able to be used in compliance with different regulation requirements. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Alejandro Abdelnur updated HDFS-6134: - Attachment: HDFSEncryptionConceptualDesignProposal-2014-06-20.pdf Transparent data at rest encryption --- Key: HDFS-6134 URL: https://issues.apache.org/jira/browse/HDFS-6134 Project: Hadoop HDFS Issue Type: New Feature Components: security Affects Versions: 2.3.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HDFSDataatRestEncryptionProposal_obsolete.pdf, HDFSEncryptionConceptualDesignProposal-2014-06-20.pdf Because of privacy and security regulations, for many industries, sensitive data at rest must be in encrypted form. For example: the healthÂcare industry (HIPAA regulations), the card payment industry (PCI DSS regulations) or the US government (FISMA regulations). This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can be used transparently by any application accessing HDFS via Hadoop Filesystem Java API, Hadoop libhdfs C library, or WebHDFS REST API. The resulting implementation should be able to be used in compliance with different regulation requirements. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-2856) Fix block protocol so that Datanodes don't require root or jsvc
[ https://issues.apache.org/jira/browse/HDFS-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HDFS-2856: Status: Open (was: Patch Available) Fix block protocol so that Datanodes don't require root or jsvc --- Key: HDFS-2856 URL: https://issues.apache.org/jira/browse/HDFS-2856 Project: Hadoop HDFS Issue Type: Improvement Components: datanode, security Affects Versions: 2.4.0, 3.0.0 Reporter: Owen O'Malley Assignee: Chris Nauroth Attachments: Datanode-Security-Design.pdf, Datanode-Security-Design.pdf, Datanode-Security-Design.pdf, HDFS-2856.1.patch, HDFS-2856.2.patch, HDFS-2856.prototype.patch Since we send the block tokens unencrypted to the datanode, we currently start the datanode as root using jsvc and get a secure ( 1024) port. If we have the datanode generate a nonce and send it on the connection and the sends an hmac of the nonce back instead of the block token it won't reveal any secrets. Thus, we wouldn't require a secure port and would not require root or jsvc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-2856) Fix block protocol so that Datanodes don't require root or jsvc
[
https://issues.apache.org/jira/browse/HDFS-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-2856:
Attachment: HDFS-2856.3.patch
I'm attaching patch v3 and notifying [~jnp], since I know he had started
reviewing the patch.
The new revision fixes a bug I found in that a client configured with SASL on
DataTransferProtocol would not have been able to communicate with an unsecured
cluster. This is a supported use case for things like distcp from a secured
source to an unsecured destination. The code change to fix this is in
{{SaslDataTransferClient#send}}. It only allows fallback to an unsecure
connection if configuration property
{{ipc.client.fallback-to-simple-auth-allowed}} has been set. This is
consistent with other RPC client code.
I also improved the tests in {{TestSaslDataTransfer}}.
Fix block protocol so that Datanodes don't require root or jsvc
---
Key: HDFS-2856
URL: https://issues.apache.org/jira/browse/HDFS-2856
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode, security
Affects Versions: 3.0.0, 2.4.0
Reporter: Owen O'Malley
Assignee: Chris Nauroth
Attachments: Datanode-Security-Design.pdf,
Datanode-Security-Design.pdf, Datanode-Security-Design.pdf,
HDFS-2856.1.patch, HDFS-2856.2.patch, HDFS-2856.3.patch,
HDFS-2856.prototype.patch
Since we send the block tokens unencrypted to the datanode, we currently
start the datanode as root using jsvc and get a secure ( 1024) port.
If we have the datanode generate a nonce and send it on the connection and
the sends an hmac of the nonce back instead of the block token it won't
reveal any secrets. Thus, we wouldn't require a secure port and would not
require root or jsvc.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-2856) Fix block protocol so that Datanodes don't require root or jsvc
[ https://issues.apache.org/jira/browse/HDFS-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HDFS-2856: Status: Patch Available (was: Open) Fix block protocol so that Datanodes don't require root or jsvc --- Key: HDFS-2856 URL: https://issues.apache.org/jira/browse/HDFS-2856 Project: Hadoop HDFS Issue Type: Improvement Components: datanode, security Affects Versions: 2.4.0, 3.0.0 Reporter: Owen O'Malley Assignee: Chris Nauroth Attachments: Datanode-Security-Design.pdf, Datanode-Security-Design.pdf, Datanode-Security-Design.pdf, HDFS-2856.1.patch, HDFS-2856.2.patch, HDFS-2856.3.patch, HDFS-2856.prototype.patch Since we send the block tokens unencrypted to the datanode, we currently start the datanode as root using jsvc and get a secure ( 1024) port. If we have the datanode generate a nonce and send it on the connection and the sends an hmac of the nonce back instead of the block token it won't reveal any secrets. Thus, we wouldn't require a secure port and would not require root or jsvc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6391) Get the Key/IV from the NameNode for encrypted files in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-6391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039524#comment-14039524 ] Andrew Wang commented on HDFS-6391: --- Ah, so the client would just initially only send the CipherTypes it'd be okay with. I was thinking it was better to let the client choose it's best CipherType (more flexible, since it's per-user rather than global setting), but this works too. The client can also always probe one-by-one if it really wants to choose. Get the Key/IV from the NameNode for encrypted files in DFSClient - Key: HDFS-6391 URL: https://issues.apache.org/jira/browse/HDFS-6391 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode, security Reporter: Alejandro Abdelnur Assignee: Andrew Wang Attachments: HDFS-6391.1.patch When creating/opening and encrypted file, the DFSClient should get the encryption key material and the IV for the file in the create/open RPC call. HDFS admin users would never get key material/IV on encrypted files create/open. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HDFS-6508) Add an XAttr to specify the cipher mode
[ https://issues.apache.org/jira/browse/HDFS-6508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Charles Lamb resolved HDFS-6508. Resolution: Not a Problem Add an XAttr to specify the cipher mode --- Key: HDFS-6508 URL: https://issues.apache.org/jira/browse/HDFS-6508 Project: Hadoop HDFS Issue Type: Bug Components: namenode, security Reporter: Charles Lamb Assignee: Charles Lamb We should specify the cipher mode in the xattrs for compatibility sake. Crypto changes over time and we need to prepare for that. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6134) Transparent data at rest encryption
[
https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039537#comment-14039537
]
Yi Liu commented on HDFS-6134:
--
[~yoderme], isn’t the same as I state?
Also the result
{code}
C1 xor C2 = P1 xor P2
{code}
isn’t {code} P2 = C2 xor (C1 xor P1){code} ?
My point is to guess P2, we should know P1. (Absolutely we know C1, C2)
CTR attack is not by finding existing two cipher-text using same encryption key
and same IV, it’s by ability of constructing file and make it encrypted by same
Data key and IV. If we can construct P1, then we know it.
Transparent data at rest encryption
---
Key: HDFS-6134
URL: https://issues.apache.org/jira/browse/HDFS-6134
Project: Hadoop HDFS
Issue Type: New Feature
Components: security
Affects Versions: 2.3.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Attachments: HDFSDataatRestEncryptionProposal_obsolete.pdf,
HDFSEncryptionConceptualDesignProposal-2014-06-20.pdf
Because of privacy and security regulations, for many industries, sensitive
data at rest must be in encrypted form. For example: the healthÂcare industry
(HIPAA regulations), the card payment industry (PCI DSS regulations) or the
US government (FISMA regulations).
This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can
be used transparently by any application accessing HDFS via Hadoop Filesystem
Java API, Hadoop libhdfs C library, or WebHDFS REST API.
The resulting implementation should be able to be used in compliance with
different regulation requirements.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6391) Get the Key/IV from the NameNode for encrypted files in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-6391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039540#comment-14039540 ] Mike Yoder commented on HDFS-6391: -- One further detail I forgot to mention - the client cipher list is _ordered_. So the client really does have a lot of control, but the server side has the final say depending on what it can support. Get the Key/IV from the NameNode for encrypted files in DFSClient - Key: HDFS-6391 URL: https://issues.apache.org/jira/browse/HDFS-6391 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode, security Reporter: Alejandro Abdelnur Assignee: Andrew Wang Attachments: HDFS-6391.1.patch When creating/opening and encrypted file, the DFSClient should get the encryption key material and the IV for the file in the create/open RPC call. HDFS admin users would never get key material/IV on encrypted files create/open. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6134) Transparent data at rest encryption
[ https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039546#comment-14039546 ] Mike Yoder commented on HDFS-6134: -- If you know P1 you can trivially get to P2, of course. My point was that we don't necessarily have to know P1 or P2 - if we only know (P1 xor P2), it's also generally easy to crack - much, much, much less work than AES encryption. Have a look at the wikipedia link above (my source of all knowledge :-) ). Transparent data at rest encryption --- Key: HDFS-6134 URL: https://issues.apache.org/jira/browse/HDFS-6134 Project: Hadoop HDFS Issue Type: New Feature Components: security Affects Versions: 2.3.0 Reporter: Alejandro Abdelnur Assignee: Alejandro Abdelnur Attachments: HDFSDataatRestEncryptionProposal_obsolete.pdf, HDFSEncryptionConceptualDesignProposal-2014-06-20.pdf Because of privacy and security regulations, for many industries, sensitive data at rest must be in encrypted form. For example: the healthÂcare industry (HIPAA regulations), the card payment industry (PCI DSS regulations) or the US government (FISMA regulations). This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can be used transparently by any application accessing HDFS via Hadoop Filesystem Java API, Hadoop libhdfs C library, or WebHDFS REST API. The resulting implementation should be able to be used in compliance with different regulation requirements. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6222) Remove background token renewer from webhdfs
[ https://issues.apache.org/jira/browse/HDFS-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Chris Nauroth updated HDFS-6222: Resolution: Fixed Fix Version/s: 2.5.0 3.0.0 Status: Resolved (was: Patch Available) I have committed this to trunk and branch-2. I co-credited the path to both [~shahrs87] and [~daryn]. Daryn, thank you for the initial patch. Rushabh, thank you for incorporating the code review feedback. Remove background token renewer from webhdfs Key: HDFS-6222 URL: https://issues.apache.org/jira/browse/HDFS-6222 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Fix For: 3.0.0, 2.5.0 Attachments: HDFS-6222.branch-2-v2.patch, HDFS-6222.branch-2-v3.patch, HDFS-6222.branch-2.patch, HDFS-6222.branch-2.patch, HDFS-6222.trunk-v2.patch, HDFS-6222.trunk-v2.patch, HDFS-6222.trunk-v3.patch, HDFS-6222.trunk.patch, HDFS-6222.trunk.patch The background token renewer is a source of problems for long-running daemons. Webhdfs should lazy fetch a new token when it receives an InvalidToken exception. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6391) Get the Key/IV from the NameNode for encrypted files in DFSClient
[ https://issues.apache.org/jira/browse/HDFS-6391?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039579#comment-14039579 ] Andrew Wang commented on HDFS-6391: --- Gotcha, that works for me. I might rename CipherType to CipherSuite too for parity with TLS, unless Java's Cipher says differently. Thanks for the info Mike! Get the Key/IV from the NameNode for encrypted files in DFSClient - Key: HDFS-6391 URL: https://issues.apache.org/jira/browse/HDFS-6391 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode, security Reporter: Alejandro Abdelnur Assignee: Andrew Wang Attachments: HDFS-6391.1.patch When creating/opening and encrypted file, the DFSClient should get the encryption key material and the IV for the file in the create/open RPC call. HDFS admin users would never get key material/IV on encrypted files create/open. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6572) Add an option to the NameNode that prints the software and on-disk image versions
[
https://issues.apache.org/jira/browse/HDFS-6572?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039584#comment-14039584
]
Hadoop QA commented on HDFS-6572:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651735/HDFS-6572.002.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7192//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7192//console
This message is automatically generated.
Add an option to the NameNode that prints the software and on-disk image
versions
-
Key: HDFS-6572
URL: https://issues.apache.org/jira/browse/HDFS-6572
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Reporter: Charles Lamb
Assignee: Charles Lamb
Priority: Minor
Attachments: HDFS-6572.001.patch, HDFS-6572.002.patch
The HDFS namenode should have a startup option that prints the metadata
versions of both the software and the on-disk version. This will be useful
for debugging certain situations.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6583) Remove clientNode in FileUnderConstructionFeature
[
https://issues.apache.org/jira/browse/HDFS-6583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039592#comment-14039592
]
Hadoop QA commented on HDFS-6583:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651743/HDFS-6583.000.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7193//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/7193//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7193//console
This message is automatically generated.
Remove clientNode in FileUnderConstructionFeature
-
Key: HDFS-6583
URL: https://issues.apache.org/jira/browse/HDFS-6583
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Reporter: Haohui Mai
Assignee: Haohui Mai
Priority: Minor
Attachments: HDFS-6583.000.patch, HDFS-6583.001.patch
{{FileUnderConstructionFeature}} contains two fields {{clientMachine}} and
{{clientNode}}. {{clientNode}} keeps a reference of a {{DatanodeDescriptor}}.
The reference be recomputed by consulting {{DatanodeManager}}.
This jira proposes to remove {{clientNode}} in the
{{FileUnderConstructionFeature}} to simplify the code and to reduce overhead.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6583) Remove clientNode in FileUnderConstructionFeature
[
https://issues.apache.org/jira/browse/HDFS-6583?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo Nicholas Sze updated HDFS-6583:
--
Hadoop Flags: Reviewed
+1 the new patch looks good.
Remove clientNode in FileUnderConstructionFeature
-
Key: HDFS-6583
URL: https://issues.apache.org/jira/browse/HDFS-6583
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Reporter: Haohui Mai
Assignee: Haohui Mai
Priority: Minor
Attachments: HDFS-6583.000.patch, HDFS-6583.001.patch
{{FileUnderConstructionFeature}} contains two fields {{clientMachine}} and
{{clientNode}}. {{clientNode}} keeps a reference of a {{DatanodeDescriptor}}.
The reference be recomputed by consulting {{DatanodeManager}}.
This jira proposes to remove {{clientNode}} in the
{{FileUnderConstructionFeature}} to simplify the code and to reduce overhead.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6134) Transparent data at rest encryption
[
https://issues.apache.org/jira/browse/HDFS-6134?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039608#comment-14039608
]
Yi Liu commented on HDFS-6134:
--
{quote}
My point was that we don't necessarily have to know P1 or P2 - if we only know
(P1 xor P2)
{quote}
The pictures have a point, and use an apple with white background. I agree if
we can get information for P2 from (P1 xor P2), then exposing (P1 xor P2) is
not good, just like the pictures.
Transparent data at rest encryption
---
Key: HDFS-6134
URL: https://issues.apache.org/jira/browse/HDFS-6134
Project: Hadoop HDFS
Issue Type: New Feature
Components: security
Affects Versions: 2.3.0
Reporter: Alejandro Abdelnur
Assignee: Alejandro Abdelnur
Attachments: HDFSDataatRestEncryptionProposal_obsolete.pdf,
HDFSEncryptionConceptualDesignProposal-2014-06-20.pdf
Because of privacy and security regulations, for many industries, sensitive
data at rest must be in encrypted form. For example: the healthÂcare industry
(HIPAA regulations), the card payment industry (PCI DSS regulations) or the
US government (FISMA regulations).
This JIRA aims to provide a mechanism to encrypt HDFS data at rest that can
be used transparently by any application accessing HDFS via Hadoop Filesystem
Java API, Hadoop libhdfs C library, or WebHDFS REST API.
The resulting implementation should be able to be used in compliance with
different regulation requirements.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6583) Remove clientNode in FileUnderConstructionFeature
[
https://issues.apache.org/jira/browse/HDFS-6583?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039620#comment-14039620
]
Hadoop QA commented on HDFS-6583:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651748/HDFS-6583.001.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 3 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7194//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7194//console
This message is automatically generated.
Remove clientNode in FileUnderConstructionFeature
-
Key: HDFS-6583
URL: https://issues.apache.org/jira/browse/HDFS-6583
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Reporter: Haohui Mai
Assignee: Haohui Mai
Priority: Minor
Attachments: HDFS-6583.000.patch, HDFS-6583.001.patch
{{FileUnderConstructionFeature}} contains two fields {{clientMachine}} and
{{clientNode}}. {{clientNode}} keeps a reference of a {{DatanodeDescriptor}}.
The reference be recomputed by consulting {{DatanodeManager}}.
This jira proposes to remove {{clientNode}} in the
{{FileUnderConstructionFeature}} to simplify the code and to reduce overhead.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6222) Remove background token renewer from webhdfs
[ https://issues.apache.org/jira/browse/HDFS-6222?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039624#comment-14039624 ] Hudson commented on HDFS-6222: -- SUCCESS: Integrated in Hadoop-trunk-Commit #5747 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/5747/]) HDFS-6222. Remove background token renewer from webhdfs. Contributed by Rushabh Shah and Daryn Sharp. (cnauroth: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1604300) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/security/token/delegation/DelegationTokenIdentifier.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/SWebHdfsFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/resources/META-INF/services/org.apache.hadoop.security.token.TokenIdentifier * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/web/TestWebHdfsTokens.java Remove background token renewer from webhdfs Key: HDFS-6222 URL: https://issues.apache.org/jira/browse/HDFS-6222 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 2.0.0-alpha, 3.0.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Fix For: 3.0.0, 2.5.0 Attachments: HDFS-6222.branch-2-v2.patch, HDFS-6222.branch-2-v3.patch, HDFS-6222.branch-2.patch, HDFS-6222.branch-2.patch, HDFS-6222.trunk-v2.patch, HDFS-6222.trunk-v2.patch, HDFS-6222.trunk-v3.patch, HDFS-6222.trunk.patch, HDFS-6222.trunk.patch The background token renewer is a source of problems for long-running daemons. Webhdfs should lazy fetch a new token when it receives an InvalidToken exception. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6580) FSNamesystem.mkdirsInt uses raw dir.getFileInfo() to getAuditFileInfo()
[
https://issues.apache.org/jira/browse/HDFS-6580?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039643#comment-14039643
]
Hadoop QA commented on HDFS-6580:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651757/patch_c89bff2bb7a06bb2b0c66a85acbd5113db6b0526.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.datanode.TestBPOfferService
org.apache.hadoop.hdfs.TestCrcCorruption
org.apache.hadoop.hdfs.TestFileAppend4
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7195//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7195//console
This message is automatically generated.
FSNamesystem.mkdirsInt uses raw dir.getFileInfo() to getAuditFileInfo()
---
Key: HDFS-6580
URL: https://issues.apache.org/jira/browse/HDFS-6580
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Reporter: Zhilei Xu
Assignee: Zhilei Xu
Labels: patch
Attachments: patch_c89bff2bb7a06bb2b0c66a85acbd5113db6b0526.txt
In FSNamesystem.java, getAuditFileInfo() is the canonical way to get file
info for auditing purpose. getAuditFileInfo() returns null when auditing is
disabled, and calls dir.getFileInfo() when auditing is enabled.
One internal APIs, mkdirsInt() mistakenly use the raw dir.getFileInfo() to
get file info for auditing. Should change to getAuditFileInfo().
Note that another internal API, startFileInt() uses dir.getFileInfo()
correctly, because the returned file stat is returned out.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HDFS-6585) INodesInPath.resolve is called multiple times in FSNamesystem.setPermission
Zhilei Xu created HDFS-6585: --- Summary: INodesInPath.resolve is called multiple times in FSNamesystem.setPermission Key: HDFS-6585 URL: https://issues.apache.org/jira/browse/HDFS-6585 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Reporter: Zhilei Xu Assignee: Zhilei Xu Attachments: patch_ab60af58e03b323dd4b18d32c4def1f008b98822.txt Most of the APIs (both internal and external) in FSNamesystem calls INodesInPath.resolve() to get the list of INodes corresponding to a file path. Usually one API will call resolve() multiple times and that's a waste of time. This issue particularly refers to FSNamesystem.setPermission, which calls resolve() twice indirectly: one from checkOwner(), another from dir.setPermission(). Should save the result of resolve(), and use it whenever possible throughout the lifetime of an API call, instead of making new resolve() calls. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6585) INodesInPath.resolve is called multiple times in FSNamesystem.setPermission
[ https://issues.apache.org/jira/browse/HDFS-6585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhilei Xu updated HDFS-6585: Attachment: patch_ab60af58e03b323dd4b18d32c4def1f008b98822.txt INodesInPath.resolve is called multiple times in FSNamesystem.setPermission --- Key: HDFS-6585 URL: https://issues.apache.org/jira/browse/HDFS-6585 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Reporter: Zhilei Xu Assignee: Zhilei Xu Attachments: patch_ab60af58e03b323dd4b18d32c4def1f008b98822.txt Most of the APIs (both internal and external) in FSNamesystem calls INodesInPath.resolve() to get the list of INodes corresponding to a file path. Usually one API will call resolve() multiple times and that's a waste of time. This issue particularly refers to FSNamesystem.setPermission, which calls resolve() twice indirectly: one from checkOwner(), another from dir.setPermission(). Should save the result of resolve(), and use it whenever possible throughout the lifetime of an API call, instead of making new resolve() calls. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6585) INodesInPath.resolve is called multiple times in FSNamesystem.setPermission
[ https://issues.apache.org/jira/browse/HDFS-6585?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhilei Xu updated HDFS-6585: Labels: patch (was: ) Status: Patch Available (was: Open) INodesInPath.resolve is called multiple times in FSNamesystem.setPermission --- Key: HDFS-6585 URL: https://issues.apache.org/jira/browse/HDFS-6585 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Reporter: Zhilei Xu Assignee: Zhilei Xu Labels: patch Attachments: patch_ab60af58e03b323dd4b18d32c4def1f008b98822.txt Most of the APIs (both internal and external) in FSNamesystem calls INodesInPath.resolve() to get the list of INodes corresponding to a file path. Usually one API will call resolve() multiple times and that's a waste of time. This issue particularly refers to FSNamesystem.setPermission, which calls resolve() twice indirectly: one from checkOwner(), another from dir.setPermission(). Should save the result of resolve(), and use it whenever possible throughout the lifetime of an API call, instead of making new resolve() calls. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-2856) Fix block protocol so that Datanodes don't require root or jsvc
[
https://issues.apache.org/jira/browse/HDFS-2856?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039649#comment-14039649
]
Hadoop QA commented on HDFS-2856:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651766/HDFS-2856.3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 5 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7196//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7196//console
This message is automatically generated.
Fix block protocol so that Datanodes don't require root or jsvc
---
Key: HDFS-2856
URL: https://issues.apache.org/jira/browse/HDFS-2856
Project: Hadoop HDFS
Issue Type: Improvement
Components: datanode, security
Affects Versions: 3.0.0, 2.4.0
Reporter: Owen O'Malley
Assignee: Chris Nauroth
Attachments: Datanode-Security-Design.pdf,
Datanode-Security-Design.pdf, Datanode-Security-Design.pdf,
HDFS-2856.1.patch, HDFS-2856.2.patch, HDFS-2856.3.patch,
HDFS-2856.prototype.patch
Since we send the block tokens unencrypted to the datanode, we currently
start the datanode as root using jsvc and get a secure ( 1024) port.
If we have the datanode generate a nonce and send it on the connection and
the sends an hmac of the nonce back instead of the block token it won't
reveal any secrets. Thus, we wouldn't require a secure port and would not
require root or jsvc.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-4667) Capture renamed files/directories in snapshot diff report
[ https://issues.apache.org/jira/browse/HDFS-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039667#comment-14039667 ] Binglin Chang commented on HDFS-4667: - bq. Binglin Chang, do you have any other comments? No, thanks for the patch, lgtm Capture renamed files/directories in snapshot diff report - Key: HDFS-4667 URL: https://issues.apache.org/jira/browse/HDFS-4667 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Jing Zhao Assignee: Jing Zhao Attachments: HDFS-4667.002.patch, HDFS-4667.002.patch, HDFS-4667.003.patch, HDFS-4667.004.patch, HDFS-4667.demo.patch, HDFS-4667.v1.patch, getfullname-snapshot-support.patch Currently in the diff report we only show file/dir creation, deletion and modification. After rename with snapshots is supported, renamed file/dir should also be captured in the diff report. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6475) WebHdfs clients fail without retry because incorrect handling of StandbyException
[
https://issues.apache.org/jira/browse/HDFS-6475?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039675#comment-14039675
]
Yongjun Zhang commented on HDFS-6475:
-
Hello Guys,
Thank you all for the review and discussion.
As a follow-up, the first thing I did was to check that
retriableRetrievePassword does give StandbyException.
Then I tried what [~daryn] suggested how we could fix this jira, and still run
into the same symptom (client side),:
{code}
org.apache.hadoop.security.token.SecretManager$InvalidToken: StandbyException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at
org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:106)
at
org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:95)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.toIOException(WebHdfsFileSystem.java:348)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$600(WebHdfsFileSystem.java:108)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.shouldRetry(WebHdfsFileSystem.java:580)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:546)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.run(WebHdfsFileSystem.java:431)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getHdfsFileStatus(WebHdfsFileSystem.java:685)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.getFileStatus(WebHdfsFileSystem.java:696)
at kclient1.kclient1$2.run(kclient1.java:80)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:356)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1528)
at kclient1.kclient1.main(kclient1.java:74)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by:
org.apache.hadoop.ipc.RemoteException(org.apache.hadoop.security.token.SecretManager$InvalidToken):
StandbyException
at
org.apache.hadoop.hdfs.web.JsonUtil.toRemoteException(JsonUtil.java:159)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.validateResponse(WebHdfsFileSystem.java:325)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem.access$700(WebHdfsFileSystem.java:108)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.getResponse(WebHdfsFileSystem.java:635)
at
org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:542)
{code}
I traced the server side and found the following:
UserProvider.getValue caught
{{org.apache.hadoop.security.token.SecretManager$InvalidToken:
StandbyException}}
with the following stack
{code}
UserProvider.getValue(HttpContext) line: 56
UserProvider.getValue(HttpContext) line: 41
InjectableValuesProvider.getInjectableValues(HttpContext) line: 46
AbstractResourceMethodDispatchProvider$ResponseOutInvoker(AbstractResourceMethodDispatchProvider$EntityParamInInvoker).getParams(HttpContext)
line: 153
AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(Object,
HttpContext) line: 203
AbstractResourceMethodDispatchProvider$ResponseOutInvoker(ResourceJavaMethodDispatcher).dispatch(Object,
HttpContext) line: 75
HttpMethodRule.accept(CharSequence, Object, UriRuleContext) line: 288
ResourceClassRule.accept(CharSequence, Object, UriRuleContext) line: 108
RightHandPathRule.accept(CharSequence, Object, UriRuleContext) line: 147
RootResourceClassesRule.accept(CharSequence, Object, UriRuleContext) line: 84
WebApplicationImpl._handleRequest(WebApplicationContext, ContainerRequest)
line: 1469
WebApplicationImpl._handleRequest(WebApplicationContext, ContainerRequest,
ContainerResponse) line: 1400
WebApplicationImpl.handleRequest(ContainerRequest, ContainerResponse) line: 1349
WebApplicationImpl.handleRequest(ContainerRequest, ContainerResponseWriter)
line: 1339
ServletContainer$InternalWebComponent(WebComponent).service(URI, URI,
HttpServletRequest, HttpServletResponse) line: 416
ServletContainer.service(URI, URI, HttpServletRequest, HttpServletResponse)
line: 537
ServletContainer.service(HttpServletRequest, HttpServletResponse) line: 699
ServletContainer(HttpServlet).service(ServletRequest, ServletResponse) line:
[jira] [Commented] (HDFS-6585) INodesInPath.resolve is called multiple times in FSNamesystem.setPermission
[
https://issues.apache.org/jira/browse/HDFS-6585?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14039694#comment-14039694
]
Hadoop QA commented on HDFS-6585:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651794/patch_ab60af58e03b323dd4b18d32c4def1f008b98822.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:red}-1 javac{color}. The applied patch generated 1262 javac
compiler warnings (more than the trunk's current 1259 warnings).
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 4 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7197//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/7197//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Javac warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/7197//artifact/trunk/patchprocess/diffJavacWarnings.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7197//console
This message is automatically generated.
INodesInPath.resolve is called multiple times in FSNamesystem.setPermission
---
Key: HDFS-6585
URL: https://issues.apache.org/jira/browse/HDFS-6585
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Reporter: Zhilei Xu
Assignee: Zhilei Xu
Labels: patch
Attachments: patch_ab60af58e03b323dd4b18d32c4def1f008b98822.txt
Most of the APIs (both internal and external) in FSNamesystem calls
INodesInPath.resolve() to get the list of INodes corresponding to a file
path. Usually one API will call resolve() multiple times and that's a waste
of time.
This issue particularly refers to FSNamesystem.setPermission, which calls
resolve() twice indirectly: one from checkOwner(), another from
dir.setPermission().
Should save the result of resolve(), and use it whenever possible throughout
the lifetime of an API call, instead of making new resolve() calls.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Updated] (HDFS-6527) Edit log corruption due to defered INode removal
[
https://issues.apache.org/jira/browse/HDFS-6527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arun C Murthy updated HDFS-6527:
Fix Version/s: (was: 2.5.0)
2.4.1
Edit log corruption due to defered INode removal
Key: HDFS-6527
URL: https://issues.apache.org/jira/browse/HDFS-6527
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.4.0
Reporter: Kihwal Lee
Assignee: Kihwal Lee
Priority: Blocker
Fix For: 2.4.1
Attachments: HDFS-6527.branch-2.4.patch, HDFS-6527.trunk.patch,
HDFS-6527.v2.patch, HDFS-6527.v3.patch, HDFS-6527.v4.patch, HDFS-6527.v5.patch
We have seen a SBN crashing with the following error:
{panel}
\[Edit log tailer\] ERROR namenode.FSEditLogLoader:
Encountered exception on operation AddBlockOp
[path=/xxx,
penultimateBlock=NULL, lastBlock=blk_111_111, RpcClientId=,
RpcCallId=-2]
java.io.FileNotFoundException: File does not exist: /xxx
{panel}
This was caused by the deferred removal of deleted inodes from the inode map.
Since getAdditionalBlock() acquires FSN read lock and then write lock, a
deletion can happen in between. Because of deferred inode removal outside FSN
write lock, getAdditionalBlock() can get the deleted inode from the inode map
with FSN write lock held. This allow addition of a block to a deleted file.
As a result, the edit log will contain OP_ADD, OP_DELETE, followed by
OP_ADD_BLOCK. This cannot be replayed by NN, so NN doesn't start up or SBN
crashes.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-4667) Capture renamed files/directories in snapshot diff report
[
https://issues.apache.org/jira/browse/HDFS-4667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14038585#comment-14038585
]
Hadoop QA commented on HDFS-4667:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651621/HDFS-4667.004.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7187//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7187//console
This message is automatically generated.
Capture renamed files/directories in snapshot diff report
-
Key: HDFS-4667
URL: https://issues.apache.org/jira/browse/HDFS-4667
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode, namenode
Reporter: Jing Zhao
Assignee: Binglin Chang
Attachments: HDFS-4667.002.patch, HDFS-4667.002.patch,
HDFS-4667.003.patch, HDFS-4667.004.patch, HDFS-4667.demo.patch,
HDFS-4667.v1.patch, getfullname-snapshot-support.patch
Currently in the diff report we only show file/dir creation, deletion and
modification. After rename with snapshots is supported, renamed file/dir
should also be captured in the diff report.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HDFS-6574) make sure addToDeadNodes() only be called when there's a connection issue occur
Liang Xie created HDFS-6574: --- Summary: make sure addToDeadNodes() only be called when there's a connection issue occur Key: HDFS-6574 URL: https://issues.apache.org/jira/browse/HDFS-6574 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Affects Versions: 3.0.0, 2.5.0 Reporter: Liang Xie Assignee: Liang Xie My colleague [~cuijianwei] found in a HBase testing scenario, once a bad disk occured, the local read will be skipped and lots of remote read be requested for a lengthy time, say, tens of minutes, then we had to trigger a compaction to help recovering the locality and read latency. It turned out relating with the addToDeadNodes(), imaging one disk in local node has something wrong, current impl will add the local node to the dead node list, then all other good disks in local node could not service any read request. So better chooses here to me, seems: 1) tell the detail IOException really is a connection related exception, then call addToDeadNodes(). 2) tell the IOException is related with bad block/disk, w/o call addToDeadNodes(); else call addToDeadNodes(). another thing need to consider is if we have got a disk exception from one node, should we refresh the locatedBlocks info from nn to clear all rotten caching for that bad disk of the node ? it'll be heavy somehow if it's a huge size file... I have a plan to make a patch for our internal hadoop branch, due to it'll degrade HBase read performance severely, also i'd like to contribute to community if you think this proposal is not too crazy... [~stack] -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6513) Add MirrorJournalManager to transfer edits from primary cluster to Mirror cluster
[ https://issues.apache.org/jira/browse/HDFS-6513?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Dian Fu updated HDFS-6513: -- Attachment: HDFS-6513.patch Add MirrorJournalManager to transfer edits from primary cluster to Mirror cluster - Key: HDFS-6513 URL: https://issues.apache.org/jira/browse/HDFS-6513 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Vinayakumar B Assignee: Dian Fu Attachments: HDFS-6513.patch, HDFS-6513.patch There is a need of separate JournalManager implementation to transfer the edits to mirror cluster in synchronous mode of data replication to mirror cluster. This Jira targets the implementation of the MirrorJournalManager, which will be used as another shared journal at the primary cluster's Active Namenode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6574) make sure addToDeadNodes() only be called when there's a connection issue occur
[ https://issues.apache.org/jira/browse/HDFS-6574?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Xie updated HDFS-6574: Description: My colleague [~cuijianwei] found in a HBase testing scenario, once a bad disk occured, the local read will be skipped and lots of remote reads be requested for a lengthy time, say, tens of minutes, then we had to trigger a compaction to help recovering the locality and read latency. It turned out relating with the addToDeadNodes(), imaging a disk in local node has something wrong, current impl will add the entity local node to the dead node list, then all other good disks in local node could not get read request any more. So better choices here to me, seems: 1) tell the detail IOException really is a connection related exception, then call addToDeadNodes(). or 2) tell the IOException is related with bad block/disk, w/o call addToDeadNodes(); else call addToDeadNodes(). another thing need to consider is if we have got a disk exception from one node, should we refresh the locatedBlocks info from nn to clear all rotten caching for that bad disk of the node ? it'll be heavy somehow if it's a huge size file... We have a plan to make a patch soon for our internal hadoop branch, due to it'll degrade HBase read performance severely once a sick disk ocurred, also we'd like to contribute to community if you think this is not too crazy... [~stack] was: My colleague [~cuijianwei] found in a HBase testing scenario, once a bad disk occured, the local read will be skipped and lots of remote read be requested for a lengthy time, say, tens of minutes, then we had to trigger a compaction to help recovering the locality and read latency. It turned out relating with the addToDeadNodes(), imaging one disk in local node has something wrong, current impl will add the local node to the dead node list, then all other good disks in local node could not service any read request. So better chooses here to me, seems: 1) tell the detail IOException really is a connection related exception, then call addToDeadNodes(). 2) tell the IOException is related with bad block/disk, w/o call addToDeadNodes(); else call addToDeadNodes(). another thing need to consider is if we have got a disk exception from one node, should we refresh the locatedBlocks info from nn to clear all rotten caching for that bad disk of the node ? it'll be heavy somehow if it's a huge size file... I have a plan to make a patch for our internal hadoop branch, due to it'll degrade HBase read performance severely, also i'd like to contribute to community if you think this proposal is not too crazy... [~stack] make sure addToDeadNodes() only be called when there's a connection issue occur --- Key: HDFS-6574 URL: https://issues.apache.org/jira/browse/HDFS-6574 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Affects Versions: 3.0.0, 2.5.0 Reporter: Liang Xie Assignee: Liang Xie My colleague [~cuijianwei] found in a HBase testing scenario, once a bad disk occured, the local read will be skipped and lots of remote reads be requested for a lengthy time, say, tens of minutes, then we had to trigger a compaction to help recovering the locality and read latency. It turned out relating with the addToDeadNodes(), imaging a disk in local node has something wrong, current impl will add the entity local node to the dead node list, then all other good disks in local node could not get read request any more. So better choices here to me, seems: 1) tell the detail IOException really is a connection related exception, then call addToDeadNodes(). or 2) tell the IOException is related with bad block/disk, w/o call addToDeadNodes(); else call addToDeadNodes(). another thing need to consider is if we have got a disk exception from one node, should we refresh the locatedBlocks info from nn to clear all rotten caching for that bad disk of the node ? it'll be heavy somehow if it's a huge size file... We have a plan to make a patch soon for our internal hadoop branch, due to it'll degrade HBase read performance severely once a sick disk ocurred, also we'd like to contribute to community if you think this is not too crazy... [~stack] -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6513) Add MirrorJournalManager to transfer edits from primary cluster to Mirror cluster
[
https://issues.apache.org/jira/browse/HDFS-6513?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14038589#comment-14038589
]
Dian Fu commented on HDFS-6513:
---
Thanks Vinay for the review.
{quote}
a) put editLogSyncProcessed = true; after long endTxId =
syncLogsFrom(lastCommittedTxId + 1);.
b) use Preconditions instead of assert in assert endTxId + 1 == txId;
2. There are some trailing whitespaces at the beginning of the line. Please
remove them
{quote}
Have update the patch according to the above comments.
{quote}
3. There are still some TODO items in MirrorJournalManager. Are you planning to
work in these later?
{quote}
They are generated by eclipse and have removed them in the updated patch.
Thanks again.
Add MirrorJournalManager to transfer edits from primary cluster to Mirror
cluster
-
Key: HDFS-6513
URL: https://issues.apache.org/jira/browse/HDFS-6513
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Reporter: Vinayakumar B
Assignee: Dian Fu
Attachments: HDFS-6513.patch, HDFS-6513.patch
There is a need of separate JournalManager implementation to transfer the
edits to mirror cluster in synchronous mode of data replication to mirror
cluster.
This Jira targets the implementation of the MirrorJournalManager, which will
be used as another shared journal at the primary cluster's Active Namenode.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Resolved] (HDFS-5184) BlockPlacementPolicyWithNodeGroup does not work correct when avoidStaleNodes is true
[
https://issues.apache.org/jira/browse/HDFS-5184?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Nikola Vujic resolved HDFS-5184.
Resolution: Done
This is fixed in HDP 2 with the new implementation of the block placement
policy with node group.
BlockPlacementPolicyWithNodeGroup does not work correct when avoidStaleNodes
is true
Key: HDFS-5184
URL: https://issues.apache.org/jira/browse/HDFS-5184
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Nikola Vujic
Priority: Minor
If avoidStaleNodes is true then choosing targets is potentially done in two
attempts. If we don't find enough targets to place replicas in the first
attempt then second attempt is invoked with the aim to use stale nodes in
order to find the remaining targets. This second attempt breaks node group
rule of not having two replicas in the same node group.
Invocation of the second attempt looks like this:
{code}
DatanodeDescriptor choseTarget(excludeNodes,...) {
oldExcludedNodes=new HashMapNode, Node(excludedNodes);
// first attempt
// if we don't find enough targets then
if (avoidStaleNodes) {
for (Node node : results) {
oldExcludedNodes.put(node, node);
}
numOfReplicas = totalReplicasExpected - results.size();
return chooseTarget(numOfReplicas, writer, oldExcludedNodes, blocksize,
maxNodesPerRack, results, false);
}
}
{code}
So, all excluded nodes from the first attempt which are neither in
oldExcludedNodes nor in results will be ignored and the second invocation of
chooseTarget will use an incomplete set of excluded nodes. For example, if we
have next topology:
dn1 - /d1/r1/n1
dn2 - /d1/r1/n1
dn3 - /d1/r1/n2
dn4 - /d1/r1/n2
and if we want to choose 3 targets with avoidStaleNodes=true then in the
first attempt we will choose 2 targets since we have only two node groups.
Let's say we choose dn1 and dn3. Then, we will add dn1 and dn3 in the
oldExcudedNodes and use that set of excluded nodes in the second attempt.
This set of excluded nodes is incomplete and allows us to select dn2 and dn4
in the second attempt which should not be selected due to node group
awareness but it is happening in the current code!
Repro:
- add
CONF.setBoolean(DFSConfigKeys.DFS_NAMENODE_AVOID_STALE_DATANODE_FOR_WRITE_KEY,
true); to TestReplicationPolicyWithNodeGroup.
- testChooseMoreTargetsThanNodeGroups() should fail.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HDFS-6575) HDFS quota is not updated when changing File replication
George Wong created HDFS-6575: - Summary: HDFS quota is not updated when changing File replication Key: HDFS-6575 URL: https://issues.apache.org/jira/browse/HDFS-6575 Project: Hadoop HDFS Issue Type: Bug Components: namenode Reporter: George Wong Assignee: George Wong When I am working on HD-6535, I find that when changing file replication factor, the quota of file is not updated. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6575) HDFS quota is not updated when changing File replication
[ https://issues.apache.org/jira/browse/HDFS-6575?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14038633#comment-14038633 ] George Wong commented on HDFS-6575: --- I am going to give a patch later. This issue is found in the trunk. HDFS quota is not updated when changing File replication Key: HDFS-6575 URL: https://issues.apache.org/jira/browse/HDFS-6575 Project: Hadoop HDFS Issue Type: Bug Components: namenode Reporter: George Wong Assignee: George Wong When I am working on HD-6535, I find that when changing file replication factor, the quota of file is not updated. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6535) HDFS quota update is wrong when file is appended
[ https://issues.apache.org/jira/browse/HDFS-6535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] George Wong updated HDFS-6535: -- Status: Patch Available (was: Open) HDFS quota update is wrong when file is appended Key: HDFS-6535 URL: https://issues.apache.org/jira/browse/HDFS-6535 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.4.0 Reporter: George Wong Assignee: George Wong Attachments: HDFS-6535.patch, HDFS-6535_v1.patch, TestHDFSQuota.java when a file in the directory with Quota feature is appended, the cached disk consumption should be updated. But currently, the update is wrong. Use the uploaded UT to reproduce this bug. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (HDFS-6535) HDFS quota update is wrong when file is appended
[ https://issues.apache.org/jira/browse/HDFS-6535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] George Wong updated HDFS-6535: -- Attachment: HDFS-6535_v1.patch change the patch to address Jing's comments. HDFS quota update is wrong when file is appended Key: HDFS-6535 URL: https://issues.apache.org/jira/browse/HDFS-6535 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.4.0 Reporter: George Wong Assignee: George Wong Attachments: HDFS-6535.patch, HDFS-6535_v1.patch, TestHDFSQuota.java when a file in the directory with Quota feature is appended, the cached disk consumption should be updated. But currently, the update is wrong. Use the uploaded UT to reproduce this bug. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6535) HDFS quota update is wrong when file is appended
[ https://issues.apache.org/jira/browse/HDFS-6535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14038640#comment-14038640 ] George Wong commented on HDFS-6535: --- [~jingzhao], when I am working on this issue, I find another issue about Quota update. Here is the filed jira https://issues.apache.org/jira/browse/HDFS-6575 I think I may give a patch later. HDFS quota update is wrong when file is appended Key: HDFS-6535 URL: https://issues.apache.org/jira/browse/HDFS-6535 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.4.0 Reporter: George Wong Assignee: George Wong Attachments: HDFS-6535.patch, HDFS-6535_v1.patch, TestHDFSQuota.java when a file in the directory with Quota feature is appended, the cached disk consumption should be updated. But currently, the update is wrong. Use the uploaded UT to reproduce this bug. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6535) HDFS quota update is wrong when file is appended
[ https://issues.apache.org/jira/browse/HDFS-6535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14038641#comment-14038641 ] George Wong commented on HDFS-6535: --- [~jingzhao], Could you please help me to review the updated patch? Thanks a lot. HDFS quota update is wrong when file is appended Key: HDFS-6535 URL: https://issues.apache.org/jira/browse/HDFS-6535 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.4.0 Reporter: George Wong Assignee: George Wong Attachments: HDFS-6535.patch, HDFS-6535_v1.patch, TestHDFSQuota.java when a file in the directory with Quota feature is appended, the cached disk consumption should be updated. But currently, the update is wrong. Use the uploaded UT to reproduce this bug. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6312) WebHdfs HA failover is broken on secure clusters
[ https://issues.apache.org/jira/browse/HDFS-6312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14038703#comment-14038703 ] Hudson commented on HDFS-6312: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #589 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/589/]) HDFS-6312. WebHdfs HA failover is broken on secure clusters. (daryn via tucu) (tucu: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1604045) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/WebHdfsFileSystem.java WebHdfs HA failover is broken on secure clusters Key: HDFS-6312 URL: https://issues.apache.org/jira/browse/HDFS-6312 Project: Hadoop HDFS Issue Type: Bug Components: webhdfs Affects Versions: 3.0.0, 2.4.0 Reporter: Daryn Sharp Assignee: Daryn Sharp Priority: Blocker Fix For: 2.5.0 Attachments: HDFS-6312.attempted.patch When webhdfs does a failover, it blanks out the delegation token. This will cause subsequent operations against the other NN to acquire a new token. Tasks cannot acquire a token (no kerberos credentials) so jobs will fail. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6403) Add metrics for log warnings reported by JVM pauses
[ https://issues.apache.org/jira/browse/HDFS-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14038706#comment-14038706 ] Hudson commented on HDFS-6403: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #589 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/589/]) HDFS-6403. Add metrics for log warnings reported by JVM pauses. Contributed by Yongjun Zhang. (atm: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1604074) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/source/JvmMetrics.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/source/JvmMetricsInfo.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/JvmPauseMonitor.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/site/apt/Metrics.apt.vm * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/source/TestJvmMetrics.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/NameNodeMetrics.java Add metrics for log warnings reported by JVM pauses --- Key: HDFS-6403 URL: https://issues.apache.org/jira/browse/HDFS-6403 Project: Hadoop HDFS Issue Type: Improvement Components: datanode, namenode Affects Versions: 2.4.0 Reporter: Yongjun Zhang Assignee: Yongjun Zhang Fix For: 2.5.0 Attachments: HDFS-6403.001.patch, HDFS-6403.002.patch, HDFS-6403.003.patch, HDFS-6403.004.patch, HDFS-6403.005.patch HADOOP-9618 logs warnings when there are long GC pauses. If this is exposed as a metric, then they can be monitored. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6549) Add support for accessing the NFS gateway from the AIX NFS client
[ https://issues.apache.org/jira/browse/HDFS-6549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14038698#comment-14038698 ] Hudson commented on HDFS-6549: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #589 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/589/]) HDFS-6549. Add support for accessing the NFS gateway from the AIX NFS client. Contributed by Aaron T. Myers. (atm: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1604022) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/conf/NfsConfigKeys.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtx.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/WriteManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java/org/apache/hadoop/hdfs/nfs/nfs3/TestWrites.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsNfsGateway.apt.vm Add support for accessing the NFS gateway from the AIX NFS client - Key: HDFS-6549 URL: https://issues.apache.org/jira/browse/HDFS-6549 Project: Hadoop HDFS Issue Type: Bug Components: nfs Affects Versions: 2.4.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Fix For: 2.5.0 Attachments: HDFS-6549.patch, HDFS-6549.patch, HDFS-6549.patch We've identified two issues when trying to access the HDFS NFS Gateway from an AIX NFS client: # In the case of COMMITs, the AIX NFS client will always send 4096, or a multiple of the page size, for the offset to be committed, even if fewer bytes than this have ever, or will ever, be written to the file. This will cause a write to a file from the AIX NFS client to hang on close unless the size of that file is a multiple of 4096. # In the case of READDIR and READDIRPLUS, the AIX NFS client will send the same cookie verifier for a given directory seemingly forever after that directory is first accessed over NFS, instead of getting a new cookie verifier for every set of incremental readdir calls. This means that if a directory's mtime ever changes, the FS must be unmounted/remounted before readdir calls on that dir from AIX will ever succeed again. From my interpretation of RFC-1813, the NFS Gateway is in fact doing the correct thing in both cases, but we can introduce simple changes on the NFS Gateway side to be able to optionally work around these incompatibilities. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-3848) A Bug in recoverLeaseInternal method of FSNameSystem class
[ https://issues.apache.org/jira/browse/HDFS-3848?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14038701#comment-14038701 ] Hudson commented on HDFS-3848: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #589 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/589/]) HDFS-3848. A Bug in recoverLeaseInternal method of FSNameSystem class. Contributed by Hooman Peiro Sajjad and Chen He. (kihwal: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1604011) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/TestLeaseRecovery2.java A Bug in recoverLeaseInternal method of FSNameSystem class -- Key: HDFS-3848 URL: https://issues.apache.org/jira/browse/HDFS-3848 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 0.23.1 Reporter: Hooman Peiro Sajjad Assignee: Chen He Labels: patch Fix For: 3.0.0, 2.5.0 Attachments: HDFS-3848-1.patch, HDFS-3848-v2.patch Original Estimate: 1h Remaining Estimate: 1h This is a bug in logic of the method recoverLeaseInternal. In line 1322 it checks if the owner of the file is trying to recreate the file. The condition of the if statement is (leaseFile != null leaseFile.equals(lease)) || lease.getHolder().equals(holder) As it can be seen, there are two operands (conditions) connected with an or operator. The first operand is straight and will be true only if the holder of the file is the new holder. But the problem is the second operand which will be always true since the lease object is the one found by the holder by calling Lease lease = leaseManager.getLease(holder); in line 1315. To fix this I think the if statement only should contain the following the condition: (leaseFile != null leaseFile.getHolder().equals(holder)) -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6492) Support create-time xattrs and atomically setting multiple xattrs
[ https://issues.apache.org/jira/browse/HDFS-6492?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14038700#comment-14038700 ] Hudson commented on HDFS-6492: -- SUCCESS: Integrated in Hadoop-Yarn-trunk #589 (See [https://builds.apache.org/job/Hadoop-Yarn-trunk/589/]) HDFS-6492. Support create-time xattrs and atomically setting multiple xattrs. (wang) (wang: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1603971) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/fs/XAttr.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/protocolPB/PBHelper.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLog.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogLoader.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSEditLogOp.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/xattr.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSDirectory.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/snapshot/TestXAttrWithSnapshot.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/resources/editsStored.xml Support create-time xattrs and atomically setting multiple xattrs - Key: HDFS-6492 URL: https://issues.apache.org/jira/browse/HDFS-6492 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 2.4.0 Reporter: Andrew Wang Assignee: Andrew Wang Fix For: 2.5.0 Attachments: HDFS-6492.001.patch, HDFS-6492.002.patch, HDFS-6492.003.patch Ongoing work in HDFS-6134 requires being able to set system namespace extended attributes at create and mkdir time, as well as being able to atomically set multiple xattrs at once. There's currently no need to expose this functionality in the client API, so let's not unless we have to. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6375) Listing extended attributes with the search permission
[
https://issues.apache.org/jira/browse/HDFS-6375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14038704#comment-14038704
]
Hudson commented on HDFS-6375:
--
SUCCESS: Integrated in Hadoop-Yarn-trunk #589 (See
[https://builds.apache.org/job/Hadoop-Yarn-trunk/589/])
Move HDFS-6375 down to 2.5 section in CHANGES.txt (wang:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1603974)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
Listing extended attributes with the search permission
--
Key: HDFS-6375
URL: https://issues.apache.org/jira/browse/HDFS-6375
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Affects Versions: 3.0.0
Reporter: Andrew Wang
Assignee: Charles Lamb
Fix For: 3.0.0, 2.5.0
Attachments: HDFS-6375.1.patch, HDFS-6375.10.patch,
HDFS-6375.11.patch, HDFS-6375.13.patch, HDFS-6375.2.patch, HDFS-6375.3.patch,
HDFS-6375.4.patch, HDFS-6375.5.patch, HDFS-6375.6.patch, HDFS-6375.7.patch,
HDFS-6375.8.patch, HDFS-6375.9.patch
From the attr(5) manpage:
{noformat}
Users with search access to a file or directory may retrieve a list of
attribute names defined for that file or directory.
{noformat}
This is like doing {{getfattr}} without the {{-d}} flag, which we currently
don't support.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6535) HDFS quota update is wrong when file is appended
[
https://issues.apache.org/jira/browse/HDFS-6535?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14038754#comment-14038754
]
Hadoop QA commented on HDFS-6535:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12651642/HDFS-6535_v1.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. There were no new javadoc warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/7188//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/7188//console
This message is automatically generated.
HDFS quota update is wrong when file is appended
Key: HDFS-6535
URL: https://issues.apache.org/jira/browse/HDFS-6535
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.4.0
Reporter: George Wong
Assignee: George Wong
Attachments: HDFS-6535.patch, HDFS-6535_v1.patch, TestHDFSQuota.java
when a file in the directory with Quota feature is appended, the cached disk
consumption should be updated.
But currently, the update is wrong.
Use the uploaded UT to reproduce this bug.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HDFS-6576) Datanode log is generating at root directory in security mode
surendra singh lilhore created HDFS-6576:
Summary: Datanode log is generating at root directory in security
mode
Key: HDFS-6576
URL: https://issues.apache.org/jira/browse/HDFS-6576
Project: Hadoop HDFS
Issue Type: Bug
Components: datanode
Affects Versions: 2.4.0
Reporter: surendra singh lilhore
Priority: Minor
In hadoop-env.sh script we are exporting HADOOP_SECURE_DN_LOG_DIR , but in
above line export statement for HADOOP_LOG_DIR is commented
If in user environment HADOOP_LOG_DIR is not exported then
HADOOP_SECURE_DN_LOG_DIR env variable will export with / value and DN will
logs in root directory.
{noformat}
# Where log files are stored. $HADOOP_HOME/logs by default.
#export HADOOP_LOG_DIR=${HADOOP_LOG_DIR}/$USER
# Where log files are stored in the secure data environment.
export HADOOP_SECURE_DN_LOG_DIR=${HADOOP_LOG_DIR}/${HADOOP_HDFS_USER}
{noformat}
I think we should comment this line.
In hadoop-daemon.sh already handled case if value of HADOOP_SECURE_DN_LOG_DIR
and HADOOP_LOG_DIR is empty.
In hadoop-daemon.sh we assigning value of HADOOP_SECURE_DN_LOG_DIR in
HADOOP_LOG_DIR and after that we are checking if HADOOP_LOG_DIR is empty then
HADOOP_LOG_DIR env variable export with $HADOOP_PREFIX/logs value
{noformat}
# Determine if we're starting a secure datanode, and if so, redefine
appropriate variables
if [ $command == datanode ] [ $EUID -eq 0 ] [ -n
$HADOOP_SECURE_DN_USER ]; then
export HADOOP_PID_DIR=$HADOOP_SECURE_DN_PID_DIR
export HADOOP_LOG_DIR=$HADOOP_SECURE_DN_LOG_DIR
export HADOOP_IDENT_STRING=$HADOOP_SECURE_DN_USER
starting_secure_dn=true
fi
if [ $HADOOP_IDENT_STRING = ]; then
export HADOOP_IDENT_STRING=$USER
fi
# get log directory
if [ $HADOOP_LOG_DIR = ]; then
export HADOOP_LOG_DIR=$HADOOP_PREFIX/logs
fi
{noformat}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6549) Add support for accessing the NFS gateway from the AIX NFS client
[ https://issues.apache.org/jira/browse/HDFS-6549?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14038812#comment-14038812 ] Hudson commented on HDFS-6549: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1780 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1780/]) HDFS-6549. Add support for accessing the NFS gateway from the AIX NFS client. Contributed by Aaron T. Myers. (atm: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1604022) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/conf/NfsConfigKeys.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/OpenFileCtx.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/RpcProgramNfs3.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/main/java/org/apache/hadoop/hdfs/nfs/nfs3/WriteManager.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs-nfs/src/test/java/org/apache/hadoop/hdfs/nfs/nfs3/TestWrites.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/site/apt/HdfsNfsGateway.apt.vm Add support for accessing the NFS gateway from the AIX NFS client - Key: HDFS-6549 URL: https://issues.apache.org/jira/browse/HDFS-6549 Project: Hadoop HDFS Issue Type: Bug Components: nfs Affects Versions: 2.4.0 Reporter: Aaron T. Myers Assignee: Aaron T. Myers Fix For: 2.5.0 Attachments: HDFS-6549.patch, HDFS-6549.patch, HDFS-6549.patch We've identified two issues when trying to access the HDFS NFS Gateway from an AIX NFS client: # In the case of COMMITs, the AIX NFS client will always send 4096, or a multiple of the page size, for the offset to be committed, even if fewer bytes than this have ever, or will ever, be written to the file. This will cause a write to a file from the AIX NFS client to hang on close unless the size of that file is a multiple of 4096. # In the case of READDIR and READDIRPLUS, the AIX NFS client will send the same cookie verifier for a given directory seemingly forever after that directory is first accessed over NFS, instead of getting a new cookie verifier for every set of incremental readdir calls. This means that if a directory's mtime ever changes, the FS must be unmounted/remounted before readdir calls on that dir from AIX will ever succeed again. From my interpretation of RFC-1813, the NFS Gateway is in fact doing the correct thing in both cases, but we can introduce simple changes on the NFS Gateway side to be able to optionally work around these incompatibilities. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (HDFS-6375) Listing extended attributes with the search permission
[
https://issues.apache.org/jira/browse/HDFS-6375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14038818#comment-14038818
]
Hudson commented on HDFS-6375:
--
FAILURE: Integrated in Hadoop-Hdfs-trunk #1780 (See
[https://builds.apache.org/job/Hadoop-Hdfs-trunk/1780/])
Move HDFS-6375 down to 2.5 section in CHANGES.txt (wang:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1603974)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
Listing extended attributes with the search permission
--
Key: HDFS-6375
URL: https://issues.apache.org/jira/browse/HDFS-6375
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Affects Versions: 3.0.0
Reporter: Andrew Wang
Assignee: Charles Lamb
Fix For: 3.0.0, 2.5.0
Attachments: HDFS-6375.1.patch, HDFS-6375.10.patch,
HDFS-6375.11.patch, HDFS-6375.13.patch, HDFS-6375.2.patch, HDFS-6375.3.patch,
HDFS-6375.4.patch, HDFS-6375.5.patch, HDFS-6375.6.patch, HDFS-6375.7.patch,
HDFS-6375.8.patch, HDFS-6375.9.patch
From the attr(5) manpage:
{noformat}
Users with search access to a file or directory may retrieve a list of
attribute names defined for that file or directory.
{noformat}
This is like doing {{getfattr}} without the {{-d}} flag, which we currently
don't support.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (HDFS-6403) Add metrics for log warnings reported by JVM pauses
[ https://issues.apache.org/jira/browse/HDFS-6403?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14038820#comment-14038820 ] Hudson commented on HDFS-6403: -- FAILURE: Integrated in Hadoop-Hdfs-trunk #1780 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1780/]) HDFS-6403. Add metrics for log warnings reported by JVM pauses. Contributed by Yongjun Zhang. (atm: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1604074) * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/source/JvmMetrics.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/metrics2/source/JvmMetricsInfo.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/main/java/org/apache/hadoop/util/JvmPauseMonitor.java * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/site/apt/Metrics.apt.vm * /hadoop/common/trunk/hadoop-common-project/hadoop-common/src/test/java/org/apache/hadoop/metrics2/source/TestJvmMetrics.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/metrics/DataNodeMetrics.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/metrics/NameNodeMetrics.java Add metrics for log warnings reported by JVM pauses --- Key: HDFS-6403 URL: https://issues.apache.org/jira/browse/HDFS-6403 Project: Hadoop HDFS Issue Type: Improvement Components: datanode, namenode Affects Versions: 2.4.0 Reporter: Yongjun Zhang Assignee: Yongjun Zhang Fix For: 2.5.0 Attachments: HDFS-6403.001.patch, HDFS-6403.002.patch, HDFS-6403.003.patch, HDFS-6403.004.patch, HDFS-6403.005.patch HADOOP-9618 logs warnings when there are long GC pauses. If this is exposed as a metric, then they can be monitored. -- This message was sent by Atlassian JIRA (v6.2#6252)
