[jira] [Commented] (HDFS-4983) Numeric usernames do not work with WebHDFS FS
[
https://issues.apache.org/jira/browse/HDFS-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843218#comment-13843218
]
Daryn Sharp commented on HDFS-4983:
---
I've questioned the necessity of this pattern before because it's required
multiple tweaks since being added. Like RPC, why not rely on the UGI to
delegate to the underlying OS for determining the validity of a username?
Numeric usernames do not work with WebHDFS FS
-
Key: HDFS-4983
URL: https://issues.apache.org/jira/browse/HDFS-4983
Project: Hadoop HDFS
Issue Type: Improvement
Components: webhdfs
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Assignee: Yongjun Zhang
Labels: patch
Fix For: 2.4.0
Attachments: HDFS-4983.001.patch, HDFS-4983.002.patch,
HDFS-4983.003.patch, HDFS-4983.004.patch, HDFS-4983.005.patch,
HDFS-4983.006.patch, HDFS-4983.006.patch
Per the file
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/UserParam.java,
the DOMAIN pattern is set to: {{^[A-Za-z_][A-Za-z0-9._-]*[$]?$}}.
Given this, using a username such as 123 seems to fail for some reason
(tried on insecure setup):
{code}
[123@host-1 ~]$ whoami
123
[123@host-1 ~]$ hadoop fs -fs webhdfs://host-2.domain.com -ls /
-ls: Invalid value: 123 does not belong to the domain
^[A-Za-z_][A-Za-z0-9._-]*[$]?$
Usage: hadoop fs [generic options] -ls [-d] [-h] [-R] [path ...]
{code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5639) rpc scheduler abstraction
[ https://issues.apache.org/jira/browse/HDFS-5639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843234#comment-13843234 ] Daryn Sharp commented on HDFS-5639: --- This patch seems excessively large to me - compounded by unnecessary changes such as using Thread.getCurrentThread when this is already a thread. Although conceptually different to some degree, it appears to overlap with HADOOP-9640. Hiding a scheduler behind a custom BlockingQueue implementation may be a bit less intrusive. Would you please work with Chris Li to see if there's enough similarity to combine these efforts (although still via separate jiras). rpc scheduler abstraction - Key: HDFS-5639 URL: https://issues.apache.org/jira/browse/HDFS-5639 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ming Ma Attachments: HDFS-5639-2.patch, HDFS-5639.patch We have run into various issues in namenode and hbase w.r.t. rpc handling in multi-tenant clusters. The examples are https://issues.apache.org/jira/i#browse/HADOOP-9640 https://issues.apache.org/jira/i#browse/HBASE-8836 There are different ideas on how to prioritize rpc requests. It could be based on user id, or whether it is read request or write request, or it could use specific rule like datanode's RPC is more important than client RPC. We want to enable people to implement and experiiment different rpc schedulers. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred
[ https://issues.apache.org/jira/browse/HDFS-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843240#comment-13843240 ] Liang Xie commented on HDFS-5637: - [~stack], seems no objection until now. try to refeatchToken while local read InvalidToken occurred --- Key: HDFS-5637 URL: https://issues.apache.org/jira/browse/HDFS-5637 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client, security Affects Versions: 2.0.5-alpha, 2.2.0 Reporter: Liang Xie Assignee: Liang Xie Attachments: HDFS-5637.txt we observed several warning logs like below from region server nodes: 2013-12-05,13:22:26,042 WARN org.apache.hadoop.hdfs.DFSClient: Failed to connect to /10.2.201.110:11402 for block, add to deadNodes and continue. org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with block_token_identifier (expiryDate=1386060141977, keyId=-333530248, userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, blockId=-190217754078101701, access modes=[READ]) is expired. at org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280) at org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88) at org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082) at org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033) at org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112) at org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687) org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with block_token_identifier (expiryDate=1386060141977, keyId=-333530248, userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, blockId=-190217754078101701, access modes=[READ]) is expired. at org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280) at org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88) at org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082) at org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033) at org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112) at org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90) at
[jira] [Updated] (HDFS-1605) Convert DFSInputStream synchronized sections to a ReadWrite lock
[ https://issues.apache.org/jira/browse/HDFS-1605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Xie updated HDFS-1605: Attachment: HDFS-1605.txt Attached is a patch against trunk. The getFileLength hotspot pattern like below: IPC Server handler 88 on 12600 daemon prio=10 tid=0x7f82fc241580 nid=0x4ddc waiting for monitor entry [0x7f821eefb000] java.lang.Thread.State: BLOCKED (on object monitor) at org.apache.hadoop.hdfs.DFSInputStream.getFileLength(DFSInputStream.java:242) - waiting to lock 0x0004404597a8 (a org.apache.hadoop.hdfs.DFSInputStream) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:982) at org.apache.hadoop.fs.FSDataInputStream.read(FSDataInputStream.java:73) at org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1393) at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1829) at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1673) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:341) at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:485) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:506) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seekAtOrAfter(StoreFileScanner.java:226) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.seek(StoreFileScanner.java:146) at org.apache.hadoop.hbase.regionserver.StoreFileScanner.enforceSeek(StoreFileScanner.java:354) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.pollRealKV(KeyValueHeap.java:385) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.generalizedSeek(KeyValueHeap.java:344) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.requestSeek(KeyValueHeap.java:304) at org.apache.hadoop.hbase.regionserver.StoreScanner.reseek(StoreScanner.java:584) - locked 0x0004285cb478 (a org.apache.hadoop.hbase.regionserver.StoreScanner) at org.apache.hadoop.hbase.regionserver.StoreScanner.next(StoreScanner.java:446) - locked 0x0004285cb478 (a org.apache.hadoop.hbase.regionserver.StoreScanner) at org.apache.hadoop.hbase.regionserver.KeyValueHeap.next(KeyValueHeap.java:164) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.nextInternal(HRegion.java:3658) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3590) - locked 0x0004285cb310 (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl) at org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl.next(HRegion.java:3615) - locked 0x0004285cb310 (a org.apache.hadoop.hbase.regionserver.HRegion$RegionScannerImpl) at org.apache.hadoop.hbase.regionserver.HRegionServer.next(HRegionServer.java:2513) at sun.reflect.GeneratedMethodAccessor28.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25) at java.lang.reflect.Method.invoke(Method.java:597) at org.apache.hadoop.hbase.ipc.SecureRpcEngine$Server.call(SecureRpcEngine.java:460) at org.apache.hadoop.hbase.ipc.HBaseServer$Handler.run(HBaseServer.java:1457) IPC Server handler 192 on 12600 daemon prio=10 tid=0x7f82fc2dd910 nid=0x4e44 runnable [0x7f821d493000] java.lang.Thread.State: RUNNABLE at java.io.FileInputStream.readBytes(Native Method) at java.io.FileInputStream.read(FileInputStream.java:220) at org.apache.hadoop.hdfs.BlockReaderLocal.read(BlockReaderLocal.java:568) - locked 0x0004287fb350 (a org.apache.hadoop.hdfs.BlockReaderLocal) at org.apache.hadoop.hdfs.DFSInputStream$ByteArrayStrategy.doRead(DFSInputStream.java:542) at org.apache.hadoop.hdfs.DFSInputStream.readBuffer(DFSInputStream.java:594) - locked 0x0004404597a8 (a org.apache.hadoop.hdfs.DFSInputStream) at org.apache.hadoop.hdfs.DFSInputStream.readWithStrategy(DFSInputStream.java:648) at org.apache.hadoop.hdfs.DFSInputStream.read(DFSInputStream.java:689) - locked 0x0004404597a8 (a org.apache.hadoop.hdfs.DFSInputStream) at java.io.DataInputStream.read(DataInputStream.java:132) at org.apache.hadoop.hbase.io.hfile.HFileBlock.readWithExtra(HFileBlock.java:614) at org.apache.hadoop.hbase.io.hfile.HFileBlock$AbstractFSReader.readAtOffset(HFileBlock.java:1384) at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockDataInternal(HFileBlock.java:1829) at org.apache.hadoop.hbase.io.hfile.HFileBlock$FSReaderV2.readBlockData(HFileBlock.java:1673) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2.readBlock(HFileReaderV2.java:341) at org.apache.hadoop.hbase.io.hfile.HFileBlockIndex$BlockIndexReader.loadDataBlockWithScanInfo(HFileBlockIndex.java:254) at org.apache.hadoop.hbase.io.hfile.HFileReaderV2$AbstractScannerV2.seekTo(HFileReaderV2.java:485)
[jira] [Commented] (HDFS-1605) Convert DFSInputStream synchronized sections to a ReadWrite lock
[ https://issues.apache.org/jira/browse/HDFS-1605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843244#comment-13843244 ] Liang Xie commented on HDFS-1605: - Considering risk, i changes most of synchronized section with writeLock. only few places(like getFileLengh, some simple getter methods) use readLock in current impl, i think it's ok since both i and [~dhruba] observed the very same hotspot: getFileLengh, in similar scenario: HBase read. Convert DFSInputStream synchronized sections to a ReadWrite lock Key: HDFS-1605 URL: https://issues.apache.org/jira/browse/HDFS-1605 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: DFSClientRWlock.1.txt, DFSClientRWlock.3.txt, HDFS-1605.txt Hbase does concurrent preads from multiple threads to different blocks of the same hdfs file. Each of these pread calls invoke DFSInputStream.getFileLength() and DFSInputStream.getBlockAt(). These methods are synchronized, thus causing all the concurrent threads to serialize. It would help performance to convert this to a Read/Write lock -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5318) Account for shared storage in NameNode replica counting algorithm
[
https://issues.apache.org/jira/browse/HDFS-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843254#comment-13843254
]
Eric Sirianni commented on HDFS-5318:
-
bq. Currently core Hadoop probably does not require this functionality. Should
we consider pluggable interface to enable this?
That seems reasonable. I'll look at what the right level of abstraction should
be for this and take a first cut at a patch.
Account for shared storage in NameNode replica counting algorithm
-
Key: HDFS-5318
URL: https://issues.apache.org/jira/browse/HDFS-5318
Project: Hadoop HDFS
Issue Type: Improvement
Components: namenode
Affects Versions: 2.4.0
Reporter: Eric Sirianni
There are several use cases for using shared-storage for datanode block
storage in an HDFS environment (storing cold blocks on a NAS device, Amazon
S3, etc.).
With shared-storage, there is a distinction between:
# a distinct physical copy of a block
# an access-path to that block via a datanode.
A single 'replication count' metric cannot accurately capture both aspects.
However, for most of the current uses of 'replication count' in the Namenode,
the number of physical copies aspect seems to be the appropriate semantic.
I propose altering the replication counting algorithm in the Namenode to
accurately infer distinct physical copies in a shared storage environment.
With HDFS-5115, a {{StorageID}} is a UUID. I propose associating some minor
additional semantics to the {{StorageID}} - namely that multiple datanodes
attaching to the same physical shared storage pool should report the same
{{StorageID}} for that pool. A minor modification would be required in the
DataNode to enable the generation of {{StorageID}} s to be pluggable behind
the {{FsDatasetSpi}} interface.
With those semantics in place, the number of physical copies of a block in a
shared storage environment can be calculated as the number of _distinct_
{{StorageID}} s associated with that block.
Consider the following combinations for two {{(DataNode ID, Storage ID)}}
pairs {{(DN_A, S_A) (DN_B, S_B)}} for a given block B:
* {{DN_A != DN_B S_A != S_B}} - *different* access paths to *different*
physical replicas (i.e. the traditional HDFS case with local disks)
** rarr; Block B has {{ReplicationCount == 2}}
* {{DN_A != DN_B S_A == S_B}} - *different* access paths to the *same*
physical replica (e.g. HDFS datanodes mounting the same NAS share)
** rarr; Block B has {{ReplicationCount == 1}}
For example, if block B has the following location tuples:
* {{DN_1, STORAGE_A}}
* {{DN_2, STORAGE_A}}
* {{DN_3, STORAGE_B}}
* {{DN_4, STORAGE_B}},
the effect of this proposed change would be to calculate the replication
factor in the namenode as *2* instead of *4*.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5023) TestSnapshotPathINodes.testAllowSnapshot is failing in branch-2
[ https://issues.apache.org/jira/browse/HDFS-5023?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843287#comment-13843287 ] Mit Desai commented on HDFS-5023: - Jing, I have tried making the changes according your suggestion. That would work but this will cause problem with the two tests i) testAllowSnapshot ii) testNonSnapshotPathINodes. -These two tests does not create snapshots, so moving the disallowSnapshot and deleteSnapshot lines to After method, will throw an exception after their completion. -As a fix to that, we will have to create an unnecessary snapshot after these tests. -Or another fix would be to place the disallowSnapshot and deleteSnapshot lines in a try block and we don't do anything in the catch block ( This is just an additional option. But it does not seem to be good choice to me ). But, I think that none of these solutions can be good to go. What do you think? TestSnapshotPathINodes.testAllowSnapshot is failing in branch-2 --- Key: HDFS-5023 URL: https://issues.apache.org/jira/browse/HDFS-5023 Project: Hadoop HDFS Issue Type: Bug Components: snapshots, test Affects Versions: 2.4.0 Reporter: Ravi Prakash Assignee: Mit Desai Labels: test Attachments: HDFS-5023.patch, HDFS-5023.patch, HDFS-5023.patch, TEST-org.apache.hadoop.hdfs.server.namenode.TestSnapshotPathINodes.xml, org.apache.hadoop.hdfs.server.namenode.TestSnapshotPathINodes-output.txt The assertion on line 91 is failing. I am using Fedora 19 + JDK7. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-4983) Numeric usernames do not work with WebHDFS FS
[
https://issues.apache.org/jira/browse/HDFS-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843292#comment-13843292
]
Yongjun Zhang commented on HDFS-4983:
-
Hi Daryn,
Please see the comment Alejandro Abdelnur made on 05/Dec/13 01:01 above.
Thanks.
--Yongjun
Numeric usernames do not work with WebHDFS FS
-
Key: HDFS-4983
URL: https://issues.apache.org/jira/browse/HDFS-4983
Project: Hadoop HDFS
Issue Type: Improvement
Components: webhdfs
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Assignee: Yongjun Zhang
Labels: patch
Fix For: 2.4.0
Attachments: HDFS-4983.001.patch, HDFS-4983.002.patch,
HDFS-4983.003.patch, HDFS-4983.004.patch, HDFS-4983.005.patch,
HDFS-4983.006.patch, HDFS-4983.006.patch
Per the file
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/UserParam.java,
the DOMAIN pattern is set to: {{^[A-Za-z_][A-Za-z0-9._-]*[$]?$}}.
Given this, using a username such as 123 seems to fail for some reason
(tried on insecure setup):
{code}
[123@host-1 ~]$ whoami
123
[123@host-1 ~]$ hadoop fs -fs webhdfs://host-2.domain.com -ls /
-ls: Invalid value: 123 does not belong to the domain
^[A-Za-z_][A-Za-z0-9._-]*[$]?$
Usage: hadoop fs [generic options] -ls [-d] [-h] [-R] [path ...]
{code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5642) libhdfs Windows compatibility.
[
https://issues.apache.org/jira/browse/HDFS-5642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843391#comment-13843391
]
Colin Patrick McCabe commented on HDFS-5642:
I would prefer putting the Windows or Linux specific parts in separate files,
rather than using #ifdefs, if possible.
I agree that we should probably get rid of the C99 stuff like declaring
variables in places other than the start of a block.
There was a suggestion to use a library like uthash rather than the glibc hash
table functions, since Windows doesn't have those. That could be a good idea.
Alternately, we could just create our own hash table, since that is just not a
very complex data structure. Having a statically sized hash table would
probably be ok, since we should know roughly how many entries it should have
based on looking at the code.
libhdfs Windows compatibility.
--
Key: HDFS-5642
URL: https://issues.apache.org/jira/browse/HDFS-5642
Project: Hadoop HDFS
Issue Type: Improvement
Components: libhdfs
Affects Versions: 3.0.0, 2.2.0
Reporter: Chris Nauroth
Currently, the libhdfs codebase does not compile on Windows due to use of
several Linux-specific functions, lack of build script support, and use of
C99 constructs. The scope of this issue includes converting those function
calls to cross-platform equivalents (or use {{#ifdef}} if necessary), setting
up build support, and some code clean-ups to follow the C89 rules.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5644) Reduce lock contention in libhdfs.
[
https://issues.apache.org/jira/browse/HDFS-5644?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843396#comment-13843396
]
Colin Patrick McCabe commented on HDFS-5644:
libhdfs doesn't actually use locking to get the JNI environment. It uses
thread-local data for that, since you need a different {{JNIEnv}} per thread.
The shared hash table stores global class references. Since this is
insert-only (we never remove elements), and we should not need to resize this
if we choose the initial size properly, it would be relatively easy to create a
concurrent version of this.
Reduce lock contention in libhdfs.
--
Key: HDFS-5644
URL: https://issues.apache.org/jira/browse/HDFS-5644
Project: Hadoop HDFS
Issue Type: Improvement
Components: libhdfs
Affects Versions: 3.0.0, 2.2.0
Reporter: Chris Nauroth
libhdfs uses locking internally for coordinating access to shared hash tables
and the JNI environment. The scope of this issue is to improve performance
of libhdfs by reducing lock contention.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Assigned] (HDFS-5632) Add Snapshot feature to INodeDirectory
[ https://issues.apache.org/jira/browse/HDFS-5632?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao reassigned HDFS-5632: --- Assignee: Jing Zhao Add Snapshot feature to INodeDirectory -- Key: HDFS-5632 URL: https://issues.apache.org/jira/browse/HDFS-5632 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Jing Zhao Assignee: Jing Zhao We will add snapshot feature to INodeDirectory and remove INodeDirectoryWithSnapshot in this jira. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-5644) Reduce lock contention in libhdfs.
[ https://issues.apache.org/jira/browse/HDFS-5644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-5644: --- Description: libhdfs has a hash table which contains class references. This hash table is shared between threads and currently uses a single global lock. The scope of this issue is to improve performance of libhdfs by removing this single global lock. (was: libhdfs uses locking internally for coordinating access to shared hash tables and the JNI environment. The scope of this issue is to improve performance of libhdfs by reducing lock contention.) Reduce lock contention in libhdfs. -- Key: HDFS-5644 URL: https://issues.apache.org/jira/browse/HDFS-5644 Project: Hadoop HDFS Issue Type: Improvement Components: libhdfs Affects Versions: 3.0.0, 2.2.0 Reporter: Chris Nauroth libhdfs has a hash table which contains class references. This hash table is shared between threads and currently uses a single global lock. The scope of this issue is to improve performance of libhdfs by removing this single global lock. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-5644) Reduce lock contention in libhdfs global class reference hash table
[ https://issues.apache.org/jira/browse/HDFS-5644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Colin Patrick McCabe updated HDFS-5644: --- Summary: Reduce lock contention in libhdfs global class reference hash table (was: Reduce lock contention in libhdfs.) Reduce lock contention in libhdfs global class reference hash table --- Key: HDFS-5644 URL: https://issues.apache.org/jira/browse/HDFS-5644 Project: Hadoop HDFS Issue Type: Improvement Components: libhdfs Affects Versions: 3.0.0, 2.2.0 Reporter: Chris Nauroth libhdfs has a hash table which contains class references. This hash table is shared between threads and currently uses a single global lock. The scope of this issue is to improve performance of libhdfs by removing this single global lock. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5642) libhdfs Windows compatibility.
[
https://issues.apache.org/jira/browse/HDFS-5642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843404#comment-13843404
]
Chris Nauroth commented on HDFS-5642:
-
bq. I would prefer putting the Windows or Linux specific parts in separate
files, rather than using #ifdefs, if possible.
I agree with this in principle, and I would have preferred this pattern in
libhadoop.so/hadoop.dll too. However, I seem to recall there was disagreement
on this a while ago. Does anyone else have an opinion?
bq. ...hash table...
Agreed that the details of this are open for consideration, rather than locked
down on choosing uthash. In fact, Stephen told me he encountered a compilation
error in uthash on HP-UX, so perhaps it's not as cross-platform as we thought.
libhdfs Windows compatibility.
--
Key: HDFS-5642
URL: https://issues.apache.org/jira/browse/HDFS-5642
Project: Hadoop HDFS
Issue Type: Improvement
Components: libhdfs
Affects Versions: 3.0.0, 2.2.0
Reporter: Chris Nauroth
Currently, the libhdfs codebase does not compile on Windows due to use of
several Linux-specific functions, lack of build script support, and use of
C99 constructs. The scope of this issue includes converting those function
calls to cross-platform equivalents (or use {{#ifdef}} if necessary), setting
up build support, and some code clean-ups to follow the C89 rules.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-4201) NPE in BPServiceActor#sendHeartBeat
[
https://issues.apache.org/jira/browse/HDFS-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843410#comment-13843410
]
Colin Patrick McCabe commented on HDFS-4201:
wouldn't it be easier to use a try+finally here to set bpNSInfo back to null if
initBlockPool fails? That way, the code could call the normal methods like
getNamespaceInfo. This may make it easier to merge this change with any other
changes to the initialization path that are coming up.
NPE in BPServiceActor#sendHeartBeat
---
Key: HDFS-4201
URL: https://issues.apache.org/jira/browse/HDFS-4201
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Reporter: Eli Collins
Assignee: Jimmy Xiang
Priority: Critical
Fix For: 3.0.0
Attachments: trunk-4201.patch, trunk-4201_v2.patch
Saw the following NPE in a log.
Think this is likely due to {{dn}} or {{dn.getFSDataset()}} being null, (not
{{bpRegistration}}) due to a configuration or local directory failure.
{code}
2012-09-25 04:33:20,782 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
For namenode svsrs00127/11.164.162.226:8020 using DELETEREPORT_INTERVAL of
30 msec BLOCKREPORT_INTERVAL of 2160msec Initial delay: 0msec;
heartBeatInterval=3000
2012-09-25 04:33:20,782 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in BPOfferService
for Block pool BP-1678908700-11.164.162.226-1342785481826 (storage id
DS-1031100678-11.164.162.251-5010-1341933415989) service to
svsrs00127/11.164.162.226:8020
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:434)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:520)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:673)
at java.lang.Thread.run(Thread.java:722)
{code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5594) FileSystem API for ACLs.
[
https://issues.apache.org/jira/browse/HDFS-5594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843417#comment-13843417
]
Haohui Mai commented on HDFS-5594:
--
The patch looks good to me overall. I have two minor comments:
# Why Acl#entries is a set? I'm wondering whether it should be ordered.
# It seems to me that guava's Objects.equal() and Objects.hashCode() could be
more efficient than Apache Common's EqualsBuilder() and HashCodeBuilder().
FileSystem API for ACLs.
Key: HDFS-5594
URL: https://issues.apache.org/jira/browse/HDFS-5594
Project: Hadoop HDFS
Issue Type: Sub-task
Components: hdfs-client
Affects Versions: HDFS ACLs (HDFS-4685)
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: HDFS-5594.1.patch, HDFS-5594.2.patch, HDFS-5594.3.patch
Add new methods to {{FileSystem}} for manipulating ACLs.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5594) FileSystem API for ACLs.
[
https://issues.apache.org/jira/browse/HDFS-5594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843436#comment-13843436
]
Haohui Mai commented on HDFS-5594:
--
BTW, it seems there're trailing whitespace in the patch. It would be nice if
you can fix them when you update the patch.
FileSystem API for ACLs.
Key: HDFS-5594
URL: https://issues.apache.org/jira/browse/HDFS-5594
Project: Hadoop HDFS
Issue Type: Sub-task
Components: hdfs-client
Affects Versions: HDFS ACLs (HDFS-4685)
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: HDFS-5594.1.patch, HDFS-5594.2.patch, HDFS-5594.3.patch
Add new methods to {{FileSystem}} for manipulating ACLs.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5594) FileSystem API for ACLs.
[
https://issues.apache.org/jira/browse/HDFS-5594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843445#comment-13843445
]
Chris Nauroth commented on HDFS-5594:
-
Thanks for taking a look, Haohui.
bq. Why Acl#entries is a set? I'm wondering whether it should be ordered.
I chose set rather than something else like list, because the entries are
unique. However, looking at this again, I think I've got it wrong. The
entries ought to be unique on scope+type+name and the ACL ought to disallow
associating that key with different permission settings. By including the
perms in {{AclEntry#equals}} and {{AclEntry#hashCode}}, I haven't achieved
that. I'm thinking of changing the set to something like {{MapAclEntry.Key,
FsAction}} to better represent the unique constraint. Let me know if you
agree, and I'll make the change.
The natural ordering for entries within an ACL is: 1) owner, 2) all named user
entries (internal ordering undefined), 3) owning group, 4) all named group
entries (internal ordering undefined), 5) other. This is the ordering that
will be required when we implement the enforcement logic. I was planning on
putting that ordering logic on the NameNode side, but I could also do it here
on the model side in a {{AclEntry#Key#compareTo}} method if you prefer. I'll
include that version in the next patch.
I can switch to the Guava equivalents for {{equals}} and {{hashCode}}.
FileSystem API for ACLs.
Key: HDFS-5594
URL: https://issues.apache.org/jira/browse/HDFS-5594
Project: Hadoop HDFS
Issue Type: Sub-task
Components: hdfs-client
Affects Versions: HDFS ACLs (HDFS-4685)
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: HDFS-5594.1.patch, HDFS-5594.2.patch, HDFS-5594.3.patch
Add new methods to {{FileSystem}} for manipulating ACLs.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-4201) NPE in BPServiceActor#sendHeartBeat
[
https://issues.apache.org/jira/browse/HDFS-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843458#comment-13843458
]
Jimmy Xiang commented on HDFS-4201:
---
That's another solution I considered. With try+finally, we need to catch all
known and unknown exceptions thrown by initBlockPool, then re-throw it, which
may not look very good. Is there any known initialization patch change coming
soon?
NPE in BPServiceActor#sendHeartBeat
---
Key: HDFS-4201
URL: https://issues.apache.org/jira/browse/HDFS-4201
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Reporter: Eli Collins
Assignee: Jimmy Xiang
Priority: Critical
Fix For: 3.0.0
Attachments: trunk-4201.patch, trunk-4201_v2.patch
Saw the following NPE in a log.
Think this is likely due to {{dn}} or {{dn.getFSDataset()}} being null, (not
{{bpRegistration}}) due to a configuration or local directory failure.
{code}
2012-09-25 04:33:20,782 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
For namenode svsrs00127/11.164.162.226:8020 using DELETEREPORT_INTERVAL of
30 msec BLOCKREPORT_INTERVAL of 2160msec Initial delay: 0msec;
heartBeatInterval=3000
2012-09-25 04:33:20,782 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in BPOfferService
for Block pool BP-1678908700-11.164.162.226-1342785481826 (storage id
DS-1031100678-11.164.162.251-5010-1341933415989) service to
svsrs00127/11.164.162.226:8020
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:434)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:520)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:673)
at java.lang.Thread.run(Thread.java:722)
{code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5594) FileSystem API for ACLs.
[
https://issues.apache.org/jira/browse/HDFS-5594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843457#comment-13843457
]
Haohui Mai commented on HDFS-5594:
--
bq. By including the perms in AclEntry#equals and AclEntry#hashCode, I haven't
achieved that. I'm thinking of changing the set to something like
MapAclEntry.Key, FsAction to better represent the unique constraint. Let me
know if you agree, and I'll make the change.
I'm wondering whether a compact representation, such as an arraylist might be
better fit. This is because that there're relatively few ACLs, and checking the
permissions is the dominant operation. Maybe it is a good idea is to spend the
effort in addAcl() / removeAcl() to maintain the optimal orders to check the
permissions efficiently.
FileSystem API for ACLs.
Key: HDFS-5594
URL: https://issues.apache.org/jira/browse/HDFS-5594
Project: Hadoop HDFS
Issue Type: Sub-task
Components: hdfs-client
Affects Versions: HDFS ACLs (HDFS-4685)
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: HDFS-5594.1.patch, HDFS-5594.2.patch, HDFS-5594.3.patch
Add new methods to {{FileSystem}} for manipulating ACLs.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Created] (HDFS-5646) Exceptions during HDFS failover
Nikhil Mulley created HDFS-5646: --- Summary: Exceptions during HDFS failover Key: HDFS-5646 URL: https://issues.apache.org/jira/browse/HDFS-5646 Project: Hadoop HDFS Issue Type: Bug Components: ha Reporter: Nikhil Mulley Hi, In our HDFS HA, I see the following excpetions when I try to failback. I have an auto failover mechanism enabled. Although the failback operation succeeds, the exceptions and the return status of 255 tend to worry me (because I cannot script this if I needed to) Please let me know if this is anything that is known and easily resolvable. I am using Cloudera Hadoop 4.4.0, if that helps.Please let me know if I need to open this ticket with CDH Jira, instead. Thanks. sudo -u hdfs hdfs haadmin -failover nn2 nn1 Operation failed: Unable to become active. Service became unhealthy while trying to failover. at org.apache.hadoop.ha.ZKFailoverController.doGracefulFailover(ZKFailoverController.java:652) at org.apache.hadoop.ha.ZKFailoverController.access$400(ZKFailoverController.java:58) at org.apache.hadoop.ha.ZKFailoverController$3.run(ZKFailoverController.java:591) at org.apache.hadoop.ha.ZKFailoverController$3.run(ZKFailoverController.java:588) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) at org.apache.hadoop.ha.ZKFailoverController.gracefulFailoverToYou(ZKFailoverController.java:588) at org.apache.hadoop.ha.ZKFCRpcServer.gracefulFailover(ZKFCRpcServer.java:94) at org.apache.hadoop.ha.protocolPB.ZKFCProtocolServerSideTranslatorPB.gracefulFailover(ZKFCProtocolServerSideTranslatorPB.java:61) at org.apache.hadoop.ha.proto.ZKFCProtocolProtos$ZKFCProtocolService$2.callBlockingMethod(ZKFCProtocolProtos.java:1351) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:1002) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1751) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1747) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1745) -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-4983) Numeric usernames do not work with WebHDFS FS
[
https://issues.apache.org/jira/browse/HDFS-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yongjun Zhang updated HDFS-4983:
Attachment: HDFS-4983-branch2.3.001.patch
Add modified patch for branch2.3. Thanks.
Numeric usernames do not work with WebHDFS FS
-
Key: HDFS-4983
URL: https://issues.apache.org/jira/browse/HDFS-4983
Project: Hadoop HDFS
Issue Type: Improvement
Components: webhdfs
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Assignee: Yongjun Zhang
Labels: patch
Fix For: 2.4.0
Attachments: HDFS-4983-branch2.3.001.patch, HDFS-4983.001.patch,
HDFS-4983.002.patch, HDFS-4983.003.patch, HDFS-4983.004.patch,
HDFS-4983.005.patch, HDFS-4983.006.patch, HDFS-4983.006.patch
Per the file
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/UserParam.java,
the DOMAIN pattern is set to: {{^[A-Za-z_][A-Za-z0-9._-]*[$]?$}}.
Given this, using a username such as 123 seems to fail for some reason
(tried on insecure setup):
{code}
[123@host-1 ~]$ whoami
123
[123@host-1 ~]$ hadoop fs -fs webhdfs://host-2.domain.com -ls /
-ls: Invalid value: 123 does not belong to the domain
^[A-Za-z_][A-Za-z0-9._-]*[$]?$
Usage: hadoop fs [generic options] -ls [-d] [-h] [-R] [path ...]
{code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5274) Add Tracing to HDFS
[
https://issues.apache.org/jira/browse/HDFS-5274?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843547#comment-13843547
]
stack commented on HDFS-5274:
-
This facility is excellent.
A few notes on the patch.
Long line and formatting seems off here:
{code}
+ if (header.hasTraceInfo() header.getTraceInfo() != null
header.getTraceInfo().hasTraceId()) {
+String traceDescription = rpcRequest.toString();
+
+// If the incoming RPC included tracing info, always continue the trace
+ TraceInfo parentSpan = new
TraceInfo(header.getTraceInfo().getTraceId(),
+ header.getTraceInfo().getParentId());
+ traceSpan = Trace.startSpan(traceDescription, parentSpan).detach();
+}
{code}
This is a javadoc fix?
- * @param buf - SASL wrapped request of one or more RPCs
+ * @param inBuf - SASL wrapped request of one or more RPCs
Are you breaking your pattern of putting trace string building behind a check
that tracing is enabled in the below?
+ TraceScope traceSpan = Trace.startSpan(Call + method.getName() + to
+
+ remoteId, CLIENT_TRACE_SAMPLER);
nit: declare and define all in the one line rather than do initialization in
constructor: +receivers = new HashSetSpanReceiver();
Any reason we take config on construction and in init for SpanReceiverHost?
SpanReceiverHost is on only when trace is enabled, right? If so, say so in
class comment.
Has to be a shutdown hook? ShutdownHookManager.get().addShutdownHook ? This is
fine unless we envision someone having to override it which I suppose should
never happen for an optionally enabled, rare, trace function?
HTraceConfiguration is for testing only? Should be @visiblefortesting only or
a comment at least?
nit: Do Classname.simple() here instead? +TraceScope scope =
Trace.startSpan(DFSInputStream.class.getSimpleName());
Should there be defines for a few of these? DFSInputStream.close seems
fine... only used once DFSInputStream.read?
Is there something off in the formatting here?
-
- private static class Packet {
-private static final long HEART_BEAT_SEQNO = -1L;
-long seqno; // sequencenumber of buffer in block
-final long offsetInBlock; // offset in block
-boolean syncBlock; // this packet forces the current block to disk
-int numChunks; // number of chunks currently in packet
-final int maxChunks; // max chunks in packet
+ private Span traceSpan = null;
+
+private static class Packet {
+ private static final long HEART_BEAT_SEQNO = -1L;
+ long seqno; // sequencenumber of buffer in block
+ final long offsetInBlock; // offset in block
+ boolean syncBlock; // this packet forces the current block to disk
+ int numChunks; // number of chunks currently in packet
+ final int maxChunks; // max chunks in packet
Or is it just git cleaning whitespace?
Gratuitous change?
-ClientOperationHeaderProto header =
+ClientOperationHeaderProto.Builder builder =
ClientOperationHeaderProto.newBuilder()
.setBaseHeader(buildBaseHeader(blk, blockToken))
-.setClientName(client)
-.build();
-return header;
+.setClientName(client);
+return builder.build();
Is an annotation of 'J' only intentional?
+if (traceSpan != null) {
+ traceSpan.addTimelineAnnotation(J);
Put inside a isTracing check? +Trace.addTimelineAnnotation(Connecting
to downstream + mirrorNode);
ditto: +Trace.addTimelineAnnotation(Waiting for connect ack from
downstream);
Add Tracing to HDFS
---
Key: HDFS-5274
URL: https://issues.apache.org/jira/browse/HDFS-5274
Project: Hadoop HDFS
Issue Type: New Feature
Components: datanode, namenode
Affects Versions: 2.1.1-beta
Reporter: Elliott Clark
Assignee: Elliott Clark
Attachments: HDFS-5274-0.patch, HDFS-5274-1.patch, HDFS-5274-2.patch,
HDFS-5274-3.patch, HDFS-5274-4.patch, HDFS-5274-5.patch, HDFS-5274-6.patch,
Zipkin Trace a06e941b0172ec73.png, Zipkin Trace d0f0d66b8a258a69.png
Since Google's Dapper paper has shown the benefits of tracing for a large
distributed system, it seems like a good time to add tracing to HDFS. HBase
has added tracing using HTrace. I propose that the same can be done within
HDFS.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-2832) Enable support for heterogeneous storages in HDFS
[
https://issues.apache.org/jira/browse/HDFS-2832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843549#comment-13843549
]
Arpit Agarwal commented on HDFS-2832:
-
{quote}
I bring them up again because, 4 months later, I was wondering if you had any
thoughts on potential solutions that could be added to the doc. It's fine if
automatic migration, open files, more elaborate resource management, and
additional storage types are all not in immediate scope, but I assume we'll
want them in the future.
{quote}
Andrew, automatic migration is not in scope for our design. wrt open files can
you describe a specific use case you think we should be handling that we have
not described? Maybe it will help me understand your concern better. If you are
concerned about reclaiming capacity for in use blocks, that is analogous to
asking If a process keeps a long-lived handle to a file what will the
operating system do to reclaim disk space used by the file? and the answer is
the same - nothing.
I don't want anyone reading your comments to get a false impression that the
feature is incompatible with SCR.
{quote}
Well, CDH supports rolling upgrades in some situations. ATM is working on
metadata upgrade with HA enabled (HDFS-5138) and I've seen some recent JIRAs
related to rolling upgrade (HDFS-5535), so it seems like a reasonable question.
At least at the protobuf level, everything so far looks compatible, so I
thought it might work as long as the handler code is compatible too.
{quote}
I am not familiar with how CDH does rolling upgrades so I cannot tell you
whether it will work. You recently bumped the layout version for caching so you
might recall that HDFS layout version checks prevent a DN registering with an
NN with a mismatched version. To my knowledge HDFS-5535 will not fix this
limitation either. That said, we have retained wire-protocol compatibility.
{quote}
Do you forsee heartbeats and block reports always being combined in realistic
scenarios? Or are there reasons to split it? Is there any additional overhead
from splitting? Can we save any complexity by not supporting split reports? I
see this on the test matrix.
{quote}
I thought I answered it, maybe if you describe your concerns I can give you a
better answer. When the test plan says 'split' I meant splitting the reports
across multiple requests. Reports will always be split by storage but we are
not splitting them across multiple messages for now. What kind of overhead are
you thinking of?
{quote}
b1. Have you put any thought about metrics and tooling to help users and admins
debug their quota usage and issues with migrating files to certain storage
types?
{quote}
We'll include it in the next design rev as we start phase 2.
Enable support for heterogeneous storages in HDFS
-
Key: HDFS-2832
URL: https://issues.apache.org/jira/browse/HDFS-2832
Project: Hadoop HDFS
Issue Type: New Feature
Affects Versions: 0.24.0
Reporter: Suresh Srinivas
Assignee: Suresh Srinivas
Attachments: 20130813-HeterogeneousStorage.pdf,
20131125-HeterogeneousStorage-TestPlan.pdf,
20131125-HeterogeneousStorage.pdf,
20131202-HeterogeneousStorage-TestPlan.pdf,
20131203-HeterogeneousStorage-TestPlan.pdf, H2832_20131107.patch,
editsStored, h2832_20131023.patch, h2832_20131023b.patch,
h2832_20131025.patch, h2832_20131028.patch, h2832_20131028b.patch,
h2832_20131029.patch, h2832_20131103.patch, h2832_20131104.patch,
h2832_20131105.patch, h2832_20131107b.patch, h2832_20131108.patch,
h2832_20131110.patch, h2832_20131110b.patch, h2832_2013.patch,
h2832_20131112.patch, h2832_20131112b.patch, h2832_20131114.patch,
h2832_20131118.patch, h2832_20131119.patch, h2832_20131119b.patch,
h2832_20131121.patch, h2832_20131122.patch, h2832_20131122b.patch,
h2832_20131123.patch, h2832_20131124.patch, h2832_20131202.patch,
h2832_20131203.patch
HDFS currently supports configuration where storages are a list of
directories. Typically each of these directories correspond to a volume with
its own file system. All these directories are homogeneous and therefore
identified as a single storage at the namenode. I propose, change to the
current model where Datanode * is a * storage, to Datanode * is a collection
* of strorages.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred
[ https://issues.apache.org/jira/browse/HDFS-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843550#comment-13843550 ] Colin Patrick McCabe commented on HDFS-5637: is there any reason why you didn't change it in fetchBlockByteRange as well? try to refeatchToken while local read InvalidToken occurred --- Key: HDFS-5637 URL: https://issues.apache.org/jira/browse/HDFS-5637 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client, security Affects Versions: 2.0.5-alpha, 2.2.0 Reporter: Liang Xie Assignee: Liang Xie Attachments: HDFS-5637.txt we observed several warning logs like below from region server nodes: 2013-12-05,13:22:26,042 WARN org.apache.hadoop.hdfs.DFSClient: Failed to connect to /10.2.201.110:11402 for block, add to deadNodes and continue. org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with block_token_identifier (expiryDate=1386060141977, keyId=-333530248, userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, blockId=-190217754078101701, access modes=[READ]) is expired. at org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280) at org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88) at org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082) at org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033) at org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112) at org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687) org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with block_token_identifier (expiryDate=1386060141977, keyId=-333530248, userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, blockId=-190217754078101701, access modes=[READ]) is expired. at org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280) at org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88) at org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082) at org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033) at org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112) at org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90) at
[jira] [Updated] (HDFS-5431) support cachepool-based quota management in path-based caching
[ https://issues.apache.org/jira/browse/HDFS-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Andrew Wang updated HDFS-5431: -- Attachment: hdfs-5431-2.patch support cachepool-based quota management in path-based caching -- Key: HDFS-5431 URL: https://issues.apache.org/jira/browse/HDFS-5431 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, namenode Affects Versions: 3.0.0 Reporter: Colin Patrick McCabe Assignee: Andrew Wang Attachments: hdfs-5431-1.patch, hdfs-5431-2.patch We should support cachepool-based quota management in path-based caching. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5629) Support HTTPS in JournalNode
[
https://issues.apache.org/jira/browse/HDFS-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843555#comment-13843555
]
Hadoop QA commented on HDFS-5629:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12617527/HDFS-5629.003.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 2 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5677//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5677//console
This message is automatically generated.
Support HTTPS in JournalNode
Key: HDFS-5629
URL: https://issues.apache.org/jira/browse/HDFS-5629
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-5629.000.patch, HDFS-5629.001.patch,
HDFS-5629.002.patch, HDFS-5629.003.patch
Currently JournalNode has only HTTP support only. This jira tracks the effort
to add HTTPS support into JournalNode.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5431) support cachepool-based quota management in path-based caching
[ https://issues.apache.org/jira/browse/HDFS-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843560#comment-13843560 ] Andrew Wang commented on HDFS-5431: --- Thanks Colin, new patch. Rundown of changes: * I re-did the FSImager serialization to use a flag int. I also adjusted CacheDirective to use an int instead of byte, and moved all that code to FSImageSerialization and cleaned it up a bit. I think it's better not to have these methods exposed in a public class. * Removed reservation and weight, le sigh * Renamed quota to limit, hope that's reasonable * Added pool stat for bytesOverlimit. I guess this could be in the cache directive too, but logically it's more of a pool setting. * Used parseShort I still like having a hard fail when you're at quota. This is how we force someone to go look at their pool stats and fix things. Accepting further directives in this situation feels like a silent failure, since the user might go off their merry way and only much later realize their data wasn't cached. I imagine monitoring tools will still be watching the statistics and throwing up their own errors too. It might also be worth noting that DN liveness won't affect limits since file sizes are kept on the NN, but it will be affected by adding more files to a directory or writing more data to a file. I agree that we could still use more debugging-type information here. If a user has to go to the logs in an expected scenario, we've failed. support cachepool-based quota management in path-based caching -- Key: HDFS-5431 URL: https://issues.apache.org/jira/browse/HDFS-5431 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, namenode Affects Versions: 3.0.0 Reporter: Colin Patrick McCabe Assignee: Andrew Wang Attachments: hdfs-5431-1.patch, hdfs-5431-2.patch We should support cachepool-based quota management in path-based caching. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Assigned] (HDFS-5596) DistributedFileSystem: implement getAcls and setAcl.
[
https://issues.apache.org/jira/browse/HDFS-5596?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Haohui Mai reassigned HDFS-5596:
Assignee: Haohui Mai
DistributedFileSystem: implement getAcls and setAcl.
Key: HDFS-5596
URL: https://issues.apache.org/jira/browse/HDFS-5596
Project: Hadoop HDFS
Issue Type: Sub-task
Components: hdfs-client, namenode
Affects Versions: HDFS ACLs (HDFS-4685)
Reporter: Chris Nauroth
Assignee: Haohui Mai
Implement and test {{getAcls}} and {{setAcl}} in {{DistributedFileSystem}}.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HDFS-5629) Support HTTPS in JournalNode and SecondaryNameNode
[ https://issues.apache.org/jira/browse/HDFS-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-5629: Summary: Support HTTPS in JournalNode and SecondaryNameNode (was: Support HTTPS in JournalNode and Secondary NameNode) Support HTTPS in JournalNode and SecondaryNameNode -- Key: HDFS-5629 URL: https://issues.apache.org/jira/browse/HDFS-5629 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Haohui Mai Assignee: Haohui Mai Attachments: HDFS-5629.000.patch, HDFS-5629.001.patch, HDFS-5629.002.patch, HDFS-5629.003.patch Currently JournalNode has only HTTP support only. This jira tracks the effort to add HTTPS support into JournalNode. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-5629) Support HTTPS in JournalNode and Secondary NameNode
[ https://issues.apache.org/jira/browse/HDFS-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-5629: Summary: Support HTTPS in JournalNode and Secondary NameNode (was: Support HTTPS in JournalNode) Support HTTPS in JournalNode and Secondary NameNode --- Key: HDFS-5629 URL: https://issues.apache.org/jira/browse/HDFS-5629 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Haohui Mai Assignee: Haohui Mai Attachments: HDFS-5629.000.patch, HDFS-5629.001.patch, HDFS-5629.002.patch, HDFS-5629.003.patch Currently JournalNode has only HTTP support only. This jira tracks the effort to add HTTPS support into JournalNode. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-5594) FileSystem API for ACLs.
[
https://issues.apache.org/jira/browse/HDFS-5594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-5594:
Attachment: HDFS-5594.4.patch
Here is version 4 of the patch with following changes:
# {{equals}} and {{hashCode}} methods have switched from Commons Lang to Guava.
# {{Acl#entries}} is converted from {{Set}} to {{List}}. Let's go with that,
and the NameNode implementation can still enforce the unique constraint that I
mentioned.
# I implemented {{AclEntry#compareTo}} and sorted the entries on the way out of
{{Acl#Builder}}. I added another test for this.
Haohui, the only trailing whitespace I saw was in the diff context for existing
code, not the new code. {{ViewFileSystem}} has a lot of trailing whitepsace
right now, and I'd prefer not to roll a full cleanup into this patch. If you
see trailing whitespace elsewhere in the new code, please let me know.
FileSystem API for ACLs.
Key: HDFS-5594
URL: https://issues.apache.org/jira/browse/HDFS-5594
Project: Hadoop HDFS
Issue Type: Sub-task
Components: hdfs-client
Affects Versions: HDFS ACLs (HDFS-4685)
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: HDFS-5594.1.patch, HDFS-5594.2.patch, HDFS-5594.3.patch,
HDFS-5594.4.patch
Add new methods to {{FileSystem}} for manipulating ACLs.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HDFS-5629) Support HTTPS in JournalNode and SecondaryNameNode
[ https://issues.apache.org/jira/browse/HDFS-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jing Zhao updated HDFS-5629: Resolution: Fixed Fix Version/s: 3.0.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) +1 for the 003 patch. I've committed this to trunk. Support HTTPS in JournalNode and SecondaryNameNode -- Key: HDFS-5629 URL: https://issues.apache.org/jira/browse/HDFS-5629 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Haohui Mai Assignee: Haohui Mai Fix For: 3.0.0 Attachments: HDFS-5629.000.patch, HDFS-5629.001.patch, HDFS-5629.002.patch, HDFS-5629.003.patch Currently JournalNode has only HTTP support only. This jira tracks the effort to add HTTPS support into JournalNode. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5594) FileSystem API for ACLs.
[
https://issues.apache.org/jira/browse/HDFS-5594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843599#comment-13843599
]
Haohui Mai commented on HDFS-5594:
--
btw, there're a couple whitespace in the new patch, all of which reside in the
comments (e.g., line 24, 38, 52).
The import statements below can be deleted:
{code}
+import org.apache.commons.lang.builder.EqualsBuilder;
+import org.apache.commons.lang.builder.HashCodeBuilder;
{code}
+1 on the new patch.
FileSystem API for ACLs.
Key: HDFS-5594
URL: https://issues.apache.org/jira/browse/HDFS-5594
Project: Hadoop HDFS
Issue Type: Sub-task
Components: hdfs-client
Affects Versions: HDFS ACLs (HDFS-4685)
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: HDFS-5594.1.patch, HDFS-5594.2.patch, HDFS-5594.3.patch,
HDFS-5594.4.patch
Add new methods to {{FileSystem}} for manipulating ACLs.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5639) rpc scheduler abstraction
[ https://issues.apache.org/jira/browse/HDFS-5639?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843606#comment-13843606 ] Chris Li commented on HDFS-5639: Something like this will be needed down the road if HADOOP-9640 is adopted; I'll open separate jiras for these enhancements when we're ready. rpc scheduler abstraction - Key: HDFS-5639 URL: https://issues.apache.org/jira/browse/HDFS-5639 Project: Hadoop HDFS Issue Type: Improvement Reporter: Ming Ma Attachments: HDFS-5639-2.patch, HDFS-5639.patch We have run into various issues in namenode and hbase w.r.t. rpc handling in multi-tenant clusters. The examples are https://issues.apache.org/jira/i#browse/HADOOP-9640 https://issues.apache.org/jira/i#browse/HBASE-8836 There are different ideas on how to prioritize rpc requests. It could be based on user id, or whether it is read request or write request, or it could use specific rule like datanode's RPC is more important than client RPC. We want to enable people to implement and experiiment different rpc schedulers. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5629) Support HTTPS in JournalNode and SecondaryNameNode
[ https://issues.apache.org/jira/browse/HDFS-5629?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843612#comment-13843612 ] Hudson commented on HDFS-5629: -- SUCCESS: Integrated in Hadoop-trunk-Commit #4855 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/4855/]) HDFS-5629. Support HTTPS in JournalNode and SecondaryNameNode. Contributed by Haohui Mai. (jing9: http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1549692) * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSConfigKeys.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/DFSUtil.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/client/IPCLoggerChannel.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/JournalNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/JournalNodeHttpServer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/qjournal/server/JournalNodeRpcServer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/DataNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/NameNodeHttpServer.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/SecondaryNameNode.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/main/proto/QJournalProtocol.proto * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/MiniJournalCluster.java * /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/qjournal/server/TestJournalNode.java Support HTTPS in JournalNode and SecondaryNameNode -- Key: HDFS-5629 URL: https://issues.apache.org/jira/browse/HDFS-5629 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Haohui Mai Assignee: Haohui Mai Fix For: 3.0.0 Attachments: HDFS-5629.000.patch, HDFS-5629.001.patch, HDFS-5629.002.patch, HDFS-5629.003.patch Currently JournalNode has only HTTP support only. This jira tracks the effort to add HTTPS support into JournalNode. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-4201) NPE in BPServiceActor#sendHeartBeat
[
https://issues.apache.org/jira/browse/HDFS-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843619#comment-13843619
]
Colin Patrick McCabe commented on HDFS-4201:
What about this paradigm?
{code}
boolean success = false;
try {
...
success = true
} finally {
if (!success) {
cleanup
}
}
{code}
we use this paradigm many places in hdfs to avoid having to enumerate all
possible exceptions.
NPE in BPServiceActor#sendHeartBeat
---
Key: HDFS-4201
URL: https://issues.apache.org/jira/browse/HDFS-4201
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Reporter: Eli Collins
Assignee: Jimmy Xiang
Priority: Critical
Fix For: 3.0.0
Attachments: trunk-4201.patch, trunk-4201_v2.patch
Saw the following NPE in a log.
Think this is likely due to {{dn}} or {{dn.getFSDataset()}} being null, (not
{{bpRegistration}}) due to a configuration or local directory failure.
{code}
2012-09-25 04:33:20,782 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
For namenode svsrs00127/11.164.162.226:8020 using DELETEREPORT_INTERVAL of
30 msec BLOCKREPORT_INTERVAL of 2160msec Initial delay: 0msec;
heartBeatInterval=3000
2012-09-25 04:33:20,782 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in BPOfferService
for Block pool BP-1678908700-11.164.162.226-1342785481826 (storage id
DS-1031100678-11.164.162.251-5010-1341933415989) service to
svsrs00127/11.164.162.226:8020
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:434)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:520)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:673)
at java.lang.Thread.run(Thread.java:722)
{code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5594) FileSystem API for ACLs.
[
https://issues.apache.org/jira/browse/HDFS-5594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843620#comment-13843620
]
Hadoop QA commented on HDFS-5594:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12617908/HDFS-5594.4.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5679//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5679//console
This message is automatically generated.
FileSystem API for ACLs.
Key: HDFS-5594
URL: https://issues.apache.org/jira/browse/HDFS-5594
Project: Hadoop HDFS
Issue Type: Sub-task
Components: hdfs-client
Affects Versions: HDFS ACLs (HDFS-4685)
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: HDFS-5594.1.patch, HDFS-5594.2.patch, HDFS-5594.3.patch,
HDFS-5594.4.patch
Add new methods to {{FileSystem}} for manipulating ACLs.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-4201) NPE in BPServiceActor#sendHeartBeat
[
https://issues.apache.org/jira/browse/HDFS-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843622#comment-13843622
]
Jimmy Xiang commented on HDFS-4201:
---
Sure, I will do as suggested so that we can minimize the changes. Thanks.
NPE in BPServiceActor#sendHeartBeat
---
Key: HDFS-4201
URL: https://issues.apache.org/jira/browse/HDFS-4201
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Reporter: Eli Collins
Assignee: Jimmy Xiang
Priority: Critical
Fix For: 3.0.0
Attachments: trunk-4201.patch, trunk-4201_v2.patch
Saw the following NPE in a log.
Think this is likely due to {{dn}} or {{dn.getFSDataset()}} being null, (not
{{bpRegistration}}) due to a configuration or local directory failure.
{code}
2012-09-25 04:33:20,782 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
For namenode svsrs00127/11.164.162.226:8020 using DELETEREPORT_INTERVAL of
30 msec BLOCKREPORT_INTERVAL of 2160msec Initial delay: 0msec;
heartBeatInterval=3000
2012-09-25 04:33:20,782 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in BPOfferService
for Block pool BP-1678908700-11.164.162.226-1342785481826 (storage id
DS-1031100678-11.164.162.251-5010-1341933415989) service to
svsrs00127/11.164.162.226:8020
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:434)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:520)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:673)
at java.lang.Thread.run(Thread.java:722)
{code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HDFS-5594) FileSystem API for ACLs.
[
https://issues.apache.org/jira/browse/HDFS-5594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-5594:
Attachment: HDFS-5594.5.patch
Ah, I hadn't checked for trailing whitespace on the comment lines. Here is
patch version 5 to clean that up.
I didn't see any remaining imports of the commons-lang stuff, so patch version
5 doesn't change anything related to that.
FileSystem API for ACLs.
Key: HDFS-5594
URL: https://issues.apache.org/jira/browse/HDFS-5594
Project: Hadoop HDFS
Issue Type: Sub-task
Components: hdfs-client
Affects Versions: HDFS ACLs (HDFS-4685)
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: HDFS-5594.1.patch, HDFS-5594.2.patch, HDFS-5594.3.patch,
HDFS-5594.4.patch, HDFS-5594.5.patch
Add new methods to {{FileSystem}} for manipulating ACLs.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5594) FileSystem API for ACLs.
[
https://issues.apache.org/jira/browse/HDFS-5594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843685#comment-13843685
]
Hadoop QA commented on HDFS-5594:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12617918/HDFS-5594.5.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5680//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5680//console
This message is automatically generated.
FileSystem API for ACLs.
Key: HDFS-5594
URL: https://issues.apache.org/jira/browse/HDFS-5594
Project: Hadoop HDFS
Issue Type: Sub-task
Components: hdfs-client
Affects Versions: HDFS ACLs (HDFS-4685)
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: HDFS-5594.1.patch, HDFS-5594.2.patch, HDFS-5594.3.patch,
HDFS-5594.4.patch, HDFS-5594.5.patch
Add new methods to {{FileSystem}} for manipulating ACLs.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5431) support cachepool-based quota management in path-based caching
[
https://issues.apache.org/jira/browse/HDFS-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843686#comment-13843686
]
Hadoop QA commented on HDFS-5431:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12617896/hdfs-5431-2.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 6 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:red}-1 findbugs{color}. The patch appears to introduce 1 new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5678//testReport/
Findbugs warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/5678//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-hdfs.html
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5678//console
This message is automatically generated.
support cachepool-based quota management in path-based caching
--
Key: HDFS-5431
URL: https://issues.apache.org/jira/browse/HDFS-5431
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode, namenode
Affects Versions: 3.0.0
Reporter: Colin Patrick McCabe
Assignee: Andrew Wang
Attachments: hdfs-5431-1.patch, hdfs-5431-2.patch
We should support cachepool-based quota management in path-based caching.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5431) support cachepool-based quota management in path-based caching
[
https://issues.apache.org/jira/browse/HDFS-5431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843691#comment-13843691
]
Andrew Wang commented on HDFS-5431:
---
I ran the OEV test locally, passes once you do the copy:
{noformat}
---
T E S T S
---
Running org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 12.455 sec - in
org.apache.hadoop.hdfs.tools.offlineEditsViewer.TestOfflineEditsViewer
Results :
Tests run: 3, Failures: 0, Errors: 0, Skipped: 0
[INFO]
[INFO] BUILD SUCCESS
[INFO]
[INFO] Total time: 30.613s
[INFO] Finished at: Mon Dec 09 15:31:47 PST 2013
[INFO] Final Memory: 32M/560M
[INFO]
{noformat}
support cachepool-based quota management in path-based caching
--
Key: HDFS-5431
URL: https://issues.apache.org/jira/browse/HDFS-5431
Project: Hadoop HDFS
Issue Type: Sub-task
Components: datanode, namenode
Affects Versions: 3.0.0
Reporter: Colin Patrick McCabe
Assignee: Andrew Wang
Attachments: hdfs-5431-1.patch, hdfs-5431-2.patch
We should support cachepool-based quota management in path-based caching.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HDFS-5647) Merge INodeDirectory.Feature and INodeFile.Feature
[ https://issues.apache.org/jira/browse/HDFS-5647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5647: - Summary: Merge INodeDirectory.Feature and INodeFile.Feature (was: Merge) Merge INodeDirectory.Feature and INodeFile.Feature -- Key: HDFS-5647 URL: https://issues.apache.org/jira/browse/HDFS-5647 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Haohui Mai -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Assigned] (HDFS-5647) Merge INodeDirectory.Feature and INodeFile.Feature
[ https://issues.apache.org/jira/browse/HDFS-5647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai reassigned HDFS-5647: Assignee: Haohui Mai Merge INodeDirectory.Feature and INodeFile.Feature -- Key: HDFS-5647 URL: https://issues.apache.org/jira/browse/HDFS-5647 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Haohui Mai Assignee: Haohui Mai -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Created] (HDFS-5647) Merge
Haohui Mai created HDFS-5647: Summary: Merge Key: HDFS-5647 URL: https://issues.apache.org/jira/browse/HDFS-5647 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Haohui Mai -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-5647) Merge INodeDirectory.Feature and INodeFile.Feature
[ https://issues.apache.org/jira/browse/HDFS-5647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5647: - Attachment: HDFS-5647.000.patch Merge INodeDirectory.Feature and INodeFile.Feature -- Key: HDFS-5647 URL: https://issues.apache.org/jira/browse/HDFS-5647 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Haohui Mai Assignee: Haohui Mai Attachments: HDFS-5647.000.patch HDFS-4685 implements ACLs for HDFS, which can benefit from the INode features introduced in HDFS-5284. The current code separates the INode feature of INodeFile and INodeDirectory into two different class hierarchies. This hinders the implementation of ACL since ACL is a concept that applies to both INodeFile and INodeDirectory. This jira proposes to merge the two class hierarchies (i.e., INodeDirectory.Feature and INodeFile.Feature) to simplify the implementation of ACLs. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-5647) Merge INodeDirectory.Feature and INodeFile.Feature
[ https://issues.apache.org/jira/browse/HDFS-5647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5647: - Description: HDFS-4685 implements ACLs for HDFS, which can benefit from the INode features introduced in HDFS-5284. The current code separates the INode feature of INodeFile and INodeDirectory into two different class hierarchies. This hinders the implementation of ACL since ACL is a concept that applies to both INodeFile and INodeDirectory. This jira proposes to merge the two class hierarchies (i.e., INodeDirectory.Feature and INodeFile.Feature) to simplify the implementation of ACLs. Merge INodeDirectory.Feature and INodeFile.Feature -- Key: HDFS-5647 URL: https://issues.apache.org/jira/browse/HDFS-5647 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Haohui Mai Assignee: Haohui Mai Attachments: HDFS-5647.000.patch HDFS-4685 implements ACLs for HDFS, which can benefit from the INode features introduced in HDFS-5284. The current code separates the INode feature of INodeFile and INodeDirectory into two different class hierarchies. This hinders the implementation of ACL since ACL is a concept that applies to both INodeFile and INodeDirectory. This jira proposes to merge the two class hierarchies (i.e., INodeDirectory.Feature and INodeFile.Feature) to simplify the implementation of ACLs. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-5647) Merge INodeDirectory.Feature and INodeFile.Feature
[ https://issues.apache.org/jira/browse/HDFS-5647?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Haohui Mai updated HDFS-5647: - Status: Patch Available (was: Open) Merge INodeDirectory.Feature and INodeFile.Feature -- Key: HDFS-5647 URL: https://issues.apache.org/jira/browse/HDFS-5647 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Haohui Mai Assignee: Haohui Mai Attachments: HDFS-5647.000.patch HDFS-4685 implements ACLs for HDFS, which can benefit from the INode features introduced in HDFS-5284. The current code separates the INode feature of INodeFile and INodeDirectory into two different class hierarchies. This hinders the implementation of ACL since ACL is a concept that applies to both INodeFile and INodeDirectory. This jira proposes to merge the two class hierarchies (i.e., INodeDirectory.Feature and INodeFile.Feature) to simplify the implementation of ACLs. -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Commented] (HDFS-5634) allow BlockReaderLocal to switch between checksumming and not
[
https://issues.apache.org/jira/browse/HDFS-5634?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843767#comment-13843767
]
Andrew Wang commented on HDFS-5634:
---
This is good stuff, thanks Colin. I have a few review comments:
BRL-related:
* Do we mean to setCachingStrategy in DFSInputStream#getBlockReader? Also, I
get that there are a zillion parameters for the BRL constructor, but builders
are for when there are optional arguments. Here, it looks like we want to set
all of them.
* We have both {{verifyChecksum}} and {{skipChecksum}} right now. Let's get rid
of one, seems error-prone to be flipping booleans.
* The {{skipChecksum || mlocked.get()}} idiom is used in a few places, maybe
should be a {{shouldSkipChecksum()}} method?
* IIUC, {{fillDataBuf}} fills the bounce buffer, and {{drainBounceBuffer}}
empties it. Rename {{fillDataBuf}} to {{fillBounceBuffer}} for parity?
* BlockReaderLocal:500, trailing whitespace
* I'm wondering what happens in the bounce buffer read paths when readahead is
turned off. It looks like they use readahead to determine how much to read,
regardless of the bytes needed, so what if it's zero?
* For the slow lane, {{fillDataBuf}} doesn't actually fill the returned buf, so
when we hit the EOF and break, it looks like we make the user read again to
flush out the bounce buffer. Can we save this?
* {{fillDataBuf}} doesn't fill just the data buf, it also fills the checksum
buf and verifies checksums via {{fillBuffer}}. Would be nice to javadoc this.
* I noticed there are two readahead config options too,
dfs.client.cache.readahead and dfs.datanode.readahead.bytes. Seems like we
should try to emulate the same behavior as remote reads which (according to
hdfs-default.xml) use the DN setting, and override with the client setting.
Right now it's just using the DN readahead in BRL, so the test that sets client
readahead to 0 isn't doing much.
* I don't quite understand why we check {{needed maxReadahead...}} for the
fast lane. Once we're checksum aligned via draining the bounce buffer, can't we
just stay in the fast lane? Seems like the slow vs. fast lane determination
should be based on read alignment, not bytes left.
Little stuff:
* It's a little weird to me that the readahead chunks is min'd with the buffer
size (default 1MB). I get why (memory consumption), but this linkage should be
documented somewhere.
* DirectBufferPool, would it be better to use a Deque's stack operations rather
than a Queue? Might give better cache locality to do LIFO rather than FIFO.
* TestEnhancedByteBufferAccess has an import only change
* Kinda unrelated, but should the dfs.client.read.shortcircuit.* keys be in
hdfs-default.xml? I also noticed that dfs.client.cache.readahead says this
setting causes the datanode to... so the readahead documentation might need to
be updated too.
allow BlockReaderLocal to switch between checksumming and not
-
Key: HDFS-5634
URL: https://issues.apache.org/jira/browse/HDFS-5634
Project: Hadoop HDFS
Issue Type: Sub-task
Components: hdfs-client
Affects Versions: 3.0.0
Reporter: Colin Patrick McCabe
Assignee: Colin Patrick McCabe
Attachments: HDFS-5634.001.patch, HDFS-5634.002.patch
BlockReaderLocal should be able to switch between checksumming and
non-checksumming, so that when we get notifications that something is mlocked
(see HDFS-5182), we can avoid checksumming when reading from that block.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-4710) SCR should honor dfs.client.read.shortcircuit.buffer.size even when checksums are off
[
https://issues.apache.org/jira/browse/HDFS-4710?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843780#comment-13843780
]
Hadoop QA commented on HDFS-4710:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12589940/HDFS-4710.002.patch
against trunk revision .
{color:red}-1 patch{color}. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5682//console
This message is automatically generated.
SCR should honor dfs.client.read.shortcircuit.buffer.size even when checksums
are off
-
Key: HDFS-4710
URL: https://issues.apache.org/jira/browse/HDFS-4710
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs-client
Affects Versions: 2.0.4-alpha
Environment: Centos (EC2) + short-circuit reads on
Reporter: Gopal V
Assignee: Colin Patrick McCabe
Priority: Minor
Labels: perfomance
Attachments: HDFS-4710.001.patch, HDFS-4710.002.patch
When short-circuit reads are on, HDFS client slows down when checksums are
turned off.
With checksums on, the query takes 45.341 seconds and with it turned off, it
takes 56.345 seconds. This is slower than the speeds observed when
short-circuiting is turned off.
The issue seems to be that FSDataInputStream.readByte() calls are directly
transferred to the disk fd when the checksums are turned off.
Even though all the columns are integers, the data being read will be read
via DataInputStream which does
{code}
public final int readInt() throws IOException {
int ch1 = in.read();
int ch2 = in.read();
int ch3 = in.read();
int ch4 = in.read();
{code}
To confirm, an strace of the Yarn container shows
{code}
26690 read(154, B, 1) = 1
26690 read(154, \250, 1) = 1
26690 read(154, ., 1) = 1
26690 read(154, \24, 1) = 1
{code}
To emulate this without the entirety of Hive code, I have written a simpler
test app
https://github.com/t3rmin4t0r/shortcircuit-reader
The jar will read a file in -bs n sized buffers. Running it with 1 byte
blocks gives similar results to the Hive test run.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5594) FileSystem API for ACLs.
[
https://issues.apache.org/jira/browse/HDFS-5594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843820#comment-13843820
]
Vinay commented on HDFS-5594:
-
Hi Chris, latest patch looks great. +1
I have some small concern, Not exactly about your patch.
Since you are touching FileSystem.java can clean following unused imports
{code}import java.util.Arrays;
import java.util.Collections;
import java.util.concurrent.atomic.AtomicInteger;{code}
One more thing, (general), why cant we check of trailing space in modified
lines using Hadoop QA? This is done in bookkeeper's QA.
FileSystem API for ACLs.
Key: HDFS-5594
URL: https://issues.apache.org/jira/browse/HDFS-5594
Project: Hadoop HDFS
Issue Type: Sub-task
Components: hdfs-client
Affects Versions: HDFS ACLs (HDFS-4685)
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: HDFS-5594.1.patch, HDFS-5594.2.patch, HDFS-5594.3.patch,
HDFS-5594.4.patch, HDFS-5594.5.patch
Add new methods to {{FileSystem}} for manipulating ACLs.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-4983) Numeric usernames do not work with WebHDFS FS
[
https://issues.apache.org/jira/browse/HDFS-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843836#comment-13843836
]
Andrew Wang commented on HDFS-4983:
---
+1 looks good to me, thanks Yongjun. Typically we don't put the CHANGES.txt in
the generated patch, but that's easy for me to fixup.
Will commit to branch-2.3 shortly and shuffle CHANGES.txt accordingly.
Numeric usernames do not work with WebHDFS FS
-
Key: HDFS-4983
URL: https://issues.apache.org/jira/browse/HDFS-4983
Project: Hadoop HDFS
Issue Type: Improvement
Components: webhdfs
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Assignee: Yongjun Zhang
Labels: patch
Fix For: 2.4.0
Attachments: HDFS-4983-branch2.3.001.patch, HDFS-4983.001.patch,
HDFS-4983.002.patch, HDFS-4983.003.patch, HDFS-4983.004.patch,
HDFS-4983.005.patch, HDFS-4983.006.patch, HDFS-4983.006.patch
Per the file
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/UserParam.java,
the DOMAIN pattern is set to: {{^[A-Za-z_][A-Za-z0-9._-]*[$]?$}}.
Given this, using a username such as 123 seems to fail for some reason
(tried on insecure setup):
{code}
[123@host-1 ~]$ whoami
123
[123@host-1 ~]$ hadoop fs -fs webhdfs://host-2.domain.com -ls /
-ls: Invalid value: 123 does not belong to the domain
^[A-Za-z_][A-Za-z0-9._-]*[$]?$
Usage: hadoop fs [generic options] -ls [-d] [-h] [-R] [path ...]
{code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5647) Merge INodeDirectory.Feature and INodeFile.Feature
[
https://issues.apache.org/jira/browse/HDFS-5647?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843859#comment-13843859
]
Hadoop QA commented on HDFS-5647:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12617937/HDFS-5647.000.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5681//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5681//console
This message is automatically generated.
Merge INodeDirectory.Feature and INodeFile.Feature
--
Key: HDFS-5647
URL: https://issues.apache.org/jira/browse/HDFS-5647
Project: Hadoop HDFS
Issue Type: Sub-task
Components: namenode
Reporter: Haohui Mai
Assignee: Haohui Mai
Attachments: HDFS-5647.000.patch
HDFS-4685 implements ACLs for HDFS, which can benefit from the INode features
introduced in HDFS-5284. The current code separates the INode feature of
INodeFile and INodeDirectory into two different class hierarchies. This
hinders the implementation of ACL since ACL is a concept that applies to both
INodeFile and INodeDirectory.
This jira proposes to merge the two class hierarchies (i.e.,
INodeDirectory.Feature and INodeFile.Feature) to simplify the implementation
of ACLs.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HDFS-1605) Convert DFSInputStream synchronized sections to a ReadWrite lock
[ https://issues.apache.org/jira/browse/HDFS-1605?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Xie updated HDFS-1605: Status: Patch Available (was: Open) Convert DFSInputStream synchronized sections to a ReadWrite lock Key: HDFS-1605 URL: https://issues.apache.org/jira/browse/HDFS-1605 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Reporter: dhruba borthakur Assignee: dhruba borthakur Attachments: DFSClientRWlock.1.txt, DFSClientRWlock.3.txt, HDFS-1605.txt Hbase does concurrent preads from multiple threads to different blocks of the same hdfs file. Each of these pread calls invoke DFSInputStream.getFileLength() and DFSInputStream.getBlockAt(). These methods are synchronized, thus causing all the concurrent threads to serialize. It would help performance to convert this to a Read/Write lock -- This message was sent by Atlassian JIRA (v6.1.4#6159)
[jira] [Updated] (HDFS-4983) Numeric usernames do not work with WebHDFS FS
[
https://issues.apache.org/jira/browse/HDFS-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Andrew Wang updated HDFS-4983:
--
Fix Version/s: (was: 2.4.0)
2.3.0
Committed to branch-2.3, fixed CHANGES.txt in branch-2 and trunk.
Numeric usernames do not work with WebHDFS FS
-
Key: HDFS-4983
URL: https://issues.apache.org/jira/browse/HDFS-4983
Project: Hadoop HDFS
Issue Type: Improvement
Components: webhdfs
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Assignee: Yongjun Zhang
Labels: patch
Fix For: 2.3.0
Attachments: HDFS-4983-branch2.3.001.patch, HDFS-4983.001.patch,
HDFS-4983.002.patch, HDFS-4983.003.patch, HDFS-4983.004.patch,
HDFS-4983.005.patch, HDFS-4983.006.patch, HDFS-4983.006.patch
Per the file
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/UserParam.java,
the DOMAIN pattern is set to: {{^[A-Za-z_][A-Za-z0-9._-]*[$]?$}}.
Given this, using a username such as 123 seems to fail for some reason
(tried on insecure setup):
{code}
[123@host-1 ~]$ whoami
123
[123@host-1 ~]$ hadoop fs -fs webhdfs://host-2.domain.com -ls /
-ls: Invalid value: 123 does not belong to the domain
^[A-Za-z_][A-Za-z0-9._-]*[$]?$
Usage: hadoop fs [generic options] -ls [-d] [-h] [-R] [path ...]
{code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-4201) NPE in BPServiceActor#sendHeartBeat
[
https://issues.apache.org/jira/browse/HDFS-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843867#comment-13843867
]
Colin Patrick McCabe commented on HDFS-4201:
thanks, Jimmy. I am +1 once this is addressed.
NPE in BPServiceActor#sendHeartBeat
---
Key: HDFS-4201
URL: https://issues.apache.org/jira/browse/HDFS-4201
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Reporter: Eli Collins
Assignee: Jimmy Xiang
Priority: Critical
Fix For: 3.0.0
Attachments: trunk-4201.patch, trunk-4201_v2.patch
Saw the following NPE in a log.
Think this is likely due to {{dn}} or {{dn.getFSDataset()}} being null, (not
{{bpRegistration}}) due to a configuration or local directory failure.
{code}
2012-09-25 04:33:20,782 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
For namenode svsrs00127/11.164.162.226:8020 using DELETEREPORT_INTERVAL of
30 msec BLOCKREPORT_INTERVAL of 2160msec Initial delay: 0msec;
heartBeatInterval=3000
2012-09-25 04:33:20,782 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in BPOfferService
for Block pool BP-1678908700-11.164.162.226-1342785481826 (storage id
DS-1031100678-11.164.162.251-5010-1341933415989) service to
svsrs00127/11.164.162.226:8020
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:434)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:520)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:673)
at java.lang.Thread.run(Thread.java:722)
{code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-4983) Numeric usernames do not work with WebHDFS FS
[
https://issues.apache.org/jira/browse/HDFS-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843873#comment-13843873
]
Hudson commented on HDFS-4983:
--
SUCCESS: Integrated in Hadoop-trunk-Commit #4857 (See
[https://builds.apache.org/job/Hadoop-trunk-Commit/4857/])
Move HDFS-4983 to 2.3 in CHANGES.txt (wang:
http://svn.apache.org/viewcvs.cgi/?root=Apache-SVNview=revrev=1549747)
* /hadoop/common/trunk/hadoop-hdfs-project/hadoop-hdfs/CHANGES.txt
Numeric usernames do not work with WebHDFS FS
-
Key: HDFS-4983
URL: https://issues.apache.org/jira/browse/HDFS-4983
Project: Hadoop HDFS
Issue Type: Improvement
Components: webhdfs
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Assignee: Yongjun Zhang
Labels: patch
Fix For: 2.3.0
Attachments: HDFS-4983-branch2.3.001.patch, HDFS-4983.001.patch,
HDFS-4983.002.patch, HDFS-4983.003.patch, HDFS-4983.004.patch,
HDFS-4983.005.patch, HDFS-4983.006.patch, HDFS-4983.006.patch
Per the file
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/UserParam.java,
the DOMAIN pattern is set to: {{^[A-Za-z_][A-Za-z0-9._-]*[$]?$}}.
Given this, using a username such as 123 seems to fail for some reason
(tried on insecure setup):
{code}
[123@host-1 ~]$ whoami
123
[123@host-1 ~]$ hadoop fs -fs webhdfs://host-2.domain.com -ls /
-ls: Invalid value: 123 does not belong to the domain
^[A-Za-z_][A-Za-z0-9._-]*[$]?$
Usage: hadoop fs [generic options] -ls [-d] [-h] [-R] [path ...]
{code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-4960) Unnecessary .meta seeks even when skip checksum is true
[
https://issues.apache.org/jira/browse/HDFS-4960?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843883#comment-13843883
]
Hadoop QA commented on HDFS-4960:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12591131/4960-trunk.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
-14 warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:red}-1 core tests{color}. The patch failed these unit tests in
hadoop-hdfs-project/hadoop-hdfs:
org.apache.hadoop.hdfs.server.balancer.TestBalancerWithNodeGroup
org.apache.hadoop.hdfs.web.TestWebHDFS
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5683//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/5683//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5683//console
This message is automatically generated.
Unnecessary .meta seeks even when skip checksum is true
---
Key: HDFS-4960
URL: https://issues.apache.org/jira/browse/HDFS-4960
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 3.0.0, 2.1.0-beta
Reporter: Varun Sharma
Assignee: Varun Sharma
Attachments: 4960-branch2.patch, 4960-trunk.patch
While attempting to benchmark an HBase + Hadoop 2.0 setup on SSDs, we found
unnecessary seeks into .meta files, each seek was a 7 byte read at the head
of the file - this attempts to validate the version #. Since the client is
requesting no-checksum, we should not be needing to touch the .meta file at
all.
Since the purpose of skip checksum is to also avoid the performance penalty
of the extra seek, we should not be seeking into .meta if skip checksum is
true
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HDFS-4201) NPE in BPServiceActor#sendHeartBeat
[
https://issues.apache.org/jira/browse/HDFS-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Jimmy Xiang updated HDFS-4201:
--
Attachment: trunk-4201_v3.patch
Attached v3 that isolated the changes as suggested.
NPE in BPServiceActor#sendHeartBeat
---
Key: HDFS-4201
URL: https://issues.apache.org/jira/browse/HDFS-4201
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Reporter: Eli Collins
Assignee: Jimmy Xiang
Priority: Critical
Fix For: 3.0.0
Attachments: trunk-4201.patch, trunk-4201_v2.patch,
trunk-4201_v3.patch
Saw the following NPE in a log.
Think this is likely due to {{dn}} or {{dn.getFSDataset()}} being null, (not
{{bpRegistration}}) due to a configuration or local directory failure.
{code}
2012-09-25 04:33:20,782 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
For namenode svsrs00127/11.164.162.226:8020 using DELETEREPORT_INTERVAL of
30 msec BLOCKREPORT_INTERVAL of 2160msec Initial delay: 0msec;
heartBeatInterval=3000
2012-09-25 04:33:20,782 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in BPOfferService
for Block pool BP-1678908700-11.164.162.226-1342785481826 (storage id
DS-1031100678-11.164.162.251-5010-1341933415989) service to
svsrs00127/11.164.162.226:8020
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:434)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:520)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:673)
at java.lang.Thread.run(Thread.java:722)
{code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred
[ https://issues.apache.org/jira/browse/HDFS-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843907#comment-13843907 ] Junping Du commented on HDFS-5637: -- Thanks [~xieliang007] for the patch! Agree with Colin that handling exception in getBlockReader() in fetchBlockByteRange should be the same. In addition, I would suggest to refactor the exception handling code here to be a method and get shared in places calling getBlockReader(). How it sounds? try to refeatchToken while local read InvalidToken occurred --- Key: HDFS-5637 URL: https://issues.apache.org/jira/browse/HDFS-5637 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client, security Affects Versions: 2.0.5-alpha, 2.2.0 Reporter: Liang Xie Assignee: Liang Xie Attachments: HDFS-5637.txt we observed several warning logs like below from region server nodes: 2013-12-05,13:22:26,042 WARN org.apache.hadoop.hdfs.DFSClient: Failed to connect to /10.2.201.110:11402 for block, add to deadNodes and continue. org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with block_token_identifier (expiryDate=1386060141977, keyId=-333530248, userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, blockId=-190217754078101701, access modes=[READ]) is expired. at org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280) at org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88) at org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082) at org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033) at org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112) at org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687) org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with block_token_identifier (expiryDate=1386060141977, keyId=-333530248, userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, blockId=-190217754078101701, access modes=[READ]) is expired. at org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280) at org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88) at org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082) at org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033) at org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112) at org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27)
[jira] [Commented] (HDFS-4983) Numeric usernames do not work with WebHDFS FS
[
https://issues.apache.org/jira/browse/HDFS-4983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843932#comment-13843932
]
Yongjun Zhang commented on HDFS-4983:
-
Thanks a lot Andrew!
Numeric usernames do not work with WebHDFS FS
-
Key: HDFS-4983
URL: https://issues.apache.org/jira/browse/HDFS-4983
Project: Hadoop HDFS
Issue Type: Improvement
Components: webhdfs
Affects Versions: 2.0.0-alpha
Reporter: Harsh J
Assignee: Yongjun Zhang
Labels: patch
Fix For: 2.3.0
Attachments: HDFS-4983-branch2.3.001.patch, HDFS-4983.001.patch,
HDFS-4983.002.patch, HDFS-4983.003.patch, HDFS-4983.004.patch,
HDFS-4983.005.patch, HDFS-4983.006.patch, HDFS-4983.006.patch
Per the file
hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/web/resources/UserParam.java,
the DOMAIN pattern is set to: {{^[A-Za-z_][A-Za-z0-9._-]*[$]?$}}.
Given this, using a username such as 123 seems to fail for some reason
(tried on insecure setup):
{code}
[123@host-1 ~]$ whoami
123
[123@host-1 ~]$ hadoop fs -fs webhdfs://host-2.domain.com -ls /
-ls: Invalid value: 123 does not belong to the domain
^[A-Za-z_][A-Za-z0-9._-]*[$]?$
Usage: hadoop fs [generic options] -ls [-d] [-h] [-R] [path ...]
{code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-1605) Convert DFSInputStream synchronized sections to a ReadWrite lock
[
https://issues.apache.org/jira/browse/HDFS-1605?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843948#comment-13843948
]
Hadoop QA commented on HDFS-1605:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12617846/HDFS-1605.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:red}-1 tests included{color}. The patch doesn't appear to include
any new or modified tests.
Please justify why no new tests are needed for this
patch.
Also please list what manual steps were performed to
verify this patch.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5684//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5684//console
This message is automatically generated.
Convert DFSInputStream synchronized sections to a ReadWrite lock
Key: HDFS-1605
URL: https://issues.apache.org/jira/browse/HDFS-1605
Project: Hadoop HDFS
Issue Type: Improvement
Components: hdfs-client
Reporter: dhruba borthakur
Assignee: dhruba borthakur
Attachments: DFSClientRWlock.1.txt, DFSClientRWlock.3.txt,
HDFS-1605.txt
Hbase does concurrent preads from multiple threads to different blocks of the
same hdfs file. Each of these pread calls invoke
DFSInputStream.getFileLength() and DFSInputStream.getBlockAt(). These methods
are synchronized, thus causing all the concurrent threads to serialize. It
would help performance to convert this to a Read/Write lock
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-4201) NPE in BPServiceActor#sendHeartBeat
[
https://issues.apache.org/jira/browse/HDFS-4201?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843980#comment-13843980
]
Hadoop QA commented on HDFS-4201:
-
{color:red}-1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12617973/trunk-4201_v3.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:red}-1 javadoc{color}. The javadoc tool appears to have generated
-12 warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:red}-1 release audit{color}. The applied patch generated 1
release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-hdfs-project/hadoop-hdfs.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5685//testReport/
Release audit warnings:
https://builds.apache.org/job/PreCommit-HDFS-Build/5685//artifact/trunk/patchprocess/patchReleaseAuditProblems.txt
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5685//console
This message is automatically generated.
NPE in BPServiceActor#sendHeartBeat
---
Key: HDFS-4201
URL: https://issues.apache.org/jira/browse/HDFS-4201
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Reporter: Eli Collins
Assignee: Jimmy Xiang
Priority: Critical
Fix For: 3.0.0
Attachments: trunk-4201.patch, trunk-4201_v2.patch,
trunk-4201_v3.patch
Saw the following NPE in a log.
Think this is likely due to {{dn}} or {{dn.getFSDataset()}} being null, (not
{{bpRegistration}}) due to a configuration or local directory failure.
{code}
2012-09-25 04:33:20,782 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
For namenode svsrs00127/11.164.162.226:8020 using DELETEREPORT_INTERVAL of
30 msec BLOCKREPORT_INTERVAL of 2160msec Initial delay: 0msec;
heartBeatInterval=3000
2012-09-25 04:33:20,782 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode: Exception in BPOfferService
for Block pool BP-1678908700-11.164.162.226-1342785481826 (storage id
DS-1031100678-11.164.162.251-5010-1341933415989) service to
svsrs00127/11.164.162.226:8020
java.lang.NullPointerException
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.sendHeartBeat(BPServiceActor.java:434)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.offerService(BPServiceActor.java:520)
at
org.apache.hadoop.hdfs.server.datanode.BPServiceActor.run(BPServiceActor.java:673)
at java.lang.Thread.run(Thread.java:722)
{code}
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred
[ https://issues.apache.org/jira/browse/HDFS-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Liang Xie updated HDFS-5637: Attachment: HDFS-5637-v2.txt try to refeatchToken while local read InvalidToken occurred --- Key: HDFS-5637 URL: https://issues.apache.org/jira/browse/HDFS-5637 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client, security Affects Versions: 2.0.5-alpha, 2.2.0 Reporter: Liang Xie Assignee: Liang Xie Attachments: HDFS-5637-v2.txt, HDFS-5637.txt we observed several warning logs like below from region server nodes: 2013-12-05,13:22:26,042 WARN org.apache.hadoop.hdfs.DFSClient: Failed to connect to /10.2.201.110:11402 for block, add to deadNodes and continue. org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with block_token_identifier (expiryDate=1386060141977, keyId=-333530248, userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, blockId=-190217754078101701, access modes=[READ]) is expired. at org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280) at org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88) at org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082) at org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033) at org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112) at org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687) org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with block_token_identifier (expiryDate=1386060141977, keyId=-333530248, userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, blockId=-190217754078101701, access modes=[READ]) is expired. at org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280) at org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88) at org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082) at org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033) at org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112) at org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at java.lang.reflect.Constructor.newInstance(Constructor.java:513) at org.apache.hadoop.ipc.RemoteException.instantiateException(RemoteException.java:90) at org.apache.hadoop.ipc.RemoteException.unwrapRemoteException(RemoteException.java:57) at
[jira] [Commented] (HDFS-5637) try to refeatchToken while local read InvalidToken occurred
[ https://issues.apache.org/jira/browse/HDFS-5637?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13843999#comment-13843999 ] Liang Xie commented on HDFS-5637: - Good catch, V2 should address comments from Colin and Junping, seems i am too lazy to do a grep before make a patch... For refactoring, it doesn't repeat more than twice and the code section is pretty short, to me it still ok for leaving it unchanged:) try to refeatchToken while local read InvalidToken occurred --- Key: HDFS-5637 URL: https://issues.apache.org/jira/browse/HDFS-5637 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client, security Affects Versions: 2.0.5-alpha, 2.2.0 Reporter: Liang Xie Assignee: Liang Xie Attachments: HDFS-5637-v2.txt, HDFS-5637.txt we observed several warning logs like below from region server nodes: 2013-12-05,13:22:26,042 WARN org.apache.hadoop.hdfs.DFSClient: Failed to connect to /10.2.201.110:11402 for block, add to deadNodes and continue. org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with block_token_identifier (expiryDate=1386060141977, keyId=-333530248, userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, blockId=-190217754078101701, access modes=[READ]) is expired. at org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280) at org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88) at org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082) at org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033) at org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112) at org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687) org.apache.hadoop.security.token.SecretManager$InvalidToken: Block token with block_token_identifier (expiryDate=1386060141977, keyId=-333530248, userId=hbase_srv, blockPoolId=BP-1310313570-10.101.10.66-1373527541386, blockId=-190217754078101701, access modes=[READ]) is expired. at org.apache.hadoop.hdfs.security.token.block.BlockTokenSecretManager.checkAccess(BlockTokenSecretManager.java:280) at org.apache.hadoop.hdfs.security.token.block.BlockPoolTokenSecretManager.checkAccess(BlockPoolTokenSecretManager.java:88) at org.apache.hadoop.hdfs.server.datanode.DataNode.checkBlockToken(DataNode.java:1082) at org.apache.hadoop.hdfs.server.datanode.DataNode.getBlockLocalPathInfo(DataNode.java:1033) at org.apache.hadoop.hdfs.protocolPB.ClientDatanodeProtocolServerSideTranslatorPB.getBlockLocalPathInfo(ClientDatanodeProtocolServerSideTranslatorPB.java:112) at org.apache.hadoop.hdfs.protocol.proto.ClientDatanodeProtocolProtos$ClientDatanodeProtocolService$2.callBlockingMethod(ClientDatanodeProtocolProtos.java:5104) at org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:453) at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:898) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1693) at org.apache.hadoop.ipc.Server$Handler$1.run(Server.java:1689) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1332) at org.apache.hadoop.ipc.Server$Handler.run(Server.java:1687) at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:39) at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:27) at
[jira] [Updated] (HDFS-5594) FileSystem API for ACLs.
[
https://issues.apache.org/jira/browse/HDFS-5594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-5594:
Attachment: HDFS-5594.6.patch
bq. Since you are touching FileSystem.java can clean following unused imports
Sure thing. Here is patch version 6 to make that change.
bq. One more thing, (general), why cant we check of trailing space in modified
lines using Hadoop QA? This is done in bookkeeper's QA.
I'm not sure if this is an enforced standard for the project, although it is
something I strive to do in my patches. I suggest filing a separate jira to do
this, and if anyone objects, then they can discuss there. It's definitely
feasible to add a check for this in test-patch.sh. It would need to be smart
enough to detect trailing whitespace only on new code, not diff context from
existing code. i.e. something like:
{code}
grep '^\+.*\s$'
{code}
FileSystem API for ACLs.
Key: HDFS-5594
URL: https://issues.apache.org/jira/browse/HDFS-5594
Project: Hadoop HDFS
Issue Type: Sub-task
Components: hdfs-client
Affects Versions: HDFS ACLs (HDFS-4685)
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: HDFS-5594.1.patch, HDFS-5594.2.patch, HDFS-5594.3.patch,
HDFS-5594.4.patch, HDFS-5594.5.patch, HDFS-5594.6.patch
Add new methods to {{FileSystem}} for manipulating ACLs.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HDFS-5594) FileSystem API for ACLs.
[
https://issues.apache.org/jira/browse/HDFS-5594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Tsz Wo (Nicholas), SZE updated HDFS-5594:
-
Hadoop Flags: Reviewed
+1 patch looks good. Minor question/typo:
- In FileSystem, should modifyAclEntries be called as addAclEntries? Both
add and remove are modifying the entries but modify here means add.
- Typo in Acl: associatd
FileSystem API for ACLs.
Key: HDFS-5594
URL: https://issues.apache.org/jira/browse/HDFS-5594
Project: Hadoop HDFS
Issue Type: Sub-task
Components: hdfs-client
Affects Versions: HDFS ACLs (HDFS-4685)
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: HDFS-5594.1.patch, HDFS-5594.2.patch, HDFS-5594.3.patch,
HDFS-5594.4.patch, HDFS-5594.5.patch, HDFS-5594.6.patch
Add new methods to {{FileSystem}} for manipulating ACLs.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Commented] (HDFS-5594) FileSystem API for ACLs.
[
https://issues.apache.org/jira/browse/HDFS-5594?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13844035#comment-13844035
]
Hadoop QA commented on HDFS-5594:
-
{color:green}+1 overall{color}. Here are the results of testing the latest
attachment
http://issues.apache.org/jira/secure/attachment/12618000/HDFS-5594.6.patch
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author
tags.
{color:green}+1 tests included{color}. The patch appears to include 4 new
or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the
total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any
warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with
eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new
Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase
the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in
hadoop-common-project/hadoop-common.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results:
https://builds.apache.org/job/PreCommit-HDFS-Build/5687//testReport/
Console output: https://builds.apache.org/job/PreCommit-HDFS-Build/5687//console
This message is automatically generated.
FileSystem API for ACLs.
Key: HDFS-5594
URL: https://issues.apache.org/jira/browse/HDFS-5594
Project: Hadoop HDFS
Issue Type: Sub-task
Components: hdfs-client
Affects Versions: HDFS ACLs (HDFS-4685)
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: HDFS-5594.1.patch, HDFS-5594.2.patch, HDFS-5594.3.patch,
HDFS-5594.4.patch, HDFS-5594.5.patch, HDFS-5594.6.patch
Add new methods to {{FileSystem}} for manipulating ACLs.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
[jira] [Updated] (HDFS-5594) FileSystem API for ACLs.
[
https://issues.apache.org/jira/browse/HDFS-5594?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth updated HDFS-5594:
Attachment: HDFS-5594.7.patch
Thanks, Nicholas. I'm uploading patch version 7. I plan to commit this
version to the HDFS-4685 branch tomorrow unless there is more feedback by then.
bq. In FileSystem, should modifyAclEntries be called as addAclEntries?
This method actually merges the requested modifications into the file's
existing ACL. The aclSpec can contain a mix of adding new ACL entries and
modifying the permissions associated with existing ACL entries. If the file
has existing ACL entries that are not mentioned in the aclSpec, then those
entries are retained. I think {{modifyAclEntries}} is the appropriate name,
but I also think my JavaDocs didn't explain this clearly enough. I've tried to
improve on this in the new patch.
bq. Typo in Acl: associatd
Thanks for catching this. I fixed it.
FileSystem API for ACLs.
Key: HDFS-5594
URL: https://issues.apache.org/jira/browse/HDFS-5594
Project: Hadoop HDFS
Issue Type: Sub-task
Components: hdfs-client
Affects Versions: HDFS ACLs (HDFS-4685)
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: HDFS-5594.1.patch, HDFS-5594.2.patch, HDFS-5594.3.patch,
HDFS-5594.4.patch, HDFS-5594.5.patch, HDFS-5594.6.patch, HDFS-5594.7.patch
Add new methods to {{FileSystem}} for manipulating ACLs.
--
This message was sent by Atlassian JIRA
(v6.1.4#6159)
