[jira] [Created] (HDFS-7568) Support immutability (Write-once-read-many) in HDFS
Suresh Srinivas created HDFS-7568: - Summary: Support immutability (Write-once-read-many) in HDFS Key: HDFS-7568 URL: https://issues.apache.org/jira/browse/HDFS-7568 Project: Hadoop HDFS Issue Type: New Feature Components: namenode Affects Versions: 2.7.0 Reporter: Suresh Srinivas Assignee: Suresh Srinivas Many regulatory compliance requires storage to support WORM functionality to protect sensitive data from being modified or deleted. This jira proposes adding that feature to HDFS. See the following comment for more description. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-7408) Add a counter in the log that shows the number of block reports processed
Suresh Srinivas created HDFS-7408: - Summary: Add a counter in the log that shows the number of block reports processed Key: HDFS-7408 URL: https://issues.apache.org/jira/browse/HDFS-7408 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Suresh Srinivas It would be great to have in the info log corresponding to block report processing, printing information on how many block reports have been processed. This can be useful to debug when namenode is unresponsive especially during startup time to understand if datanodes are sending block reports multiple times. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HDFS-6320) Namenode exits on exception without printing stack trace in AbstractDelegationTokenSecretManager
Suresh Srinivas created HDFS-6320: - Summary: Namenode exits on exception without printing stack trace in AbstractDelegationTokenSecretManager Key: HDFS-6320 URL: https://issues.apache.org/jira/browse/HDFS-6320 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 1.2.1, 2.0.0-alpha Reporter: Suresh Srinivas Not printing the stack trace makes debugging harder. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6274) Cleanup javadoc warnings in HDFS code
Suresh Srinivas created HDFS-6274: - Summary: Cleanup javadoc warnings in HDFS code Key: HDFS-6274 URL: https://issues.apache.org/jira/browse/HDFS-6274 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: 2.4.0 Reporter: Suresh Srinivas Attachments: HDFS-6274.patch Javadoc in HDFS code has many issues - The parameter names are not correct, parameters are not documented, exceptions declared as throws are not thrown, typos etc. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6275) Fix warnings - type arguments can be inferred and redudant local variable
Suresh Srinivas created HDFS-6275: - Summary: Fix warnings - type arguments can be inferred and redudant local variable Key: HDFS-6275 URL: https://issues.apache.org/jira/browse/HDFS-6275 Project: Hadoop HDFS Issue Type: Sub-task Affects Versions: 2.4.0 Reporter: Suresh Srinivas -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6276) Remove unnecessary conditions and null check
Suresh Srinivas created HDFS-6276: - Summary: Remove unnecessary conditions and null check Key: HDFS-6276 URL: https://issues.apache.org/jira/browse/HDFS-6276 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Suresh Srinivas Attachments: HDFS-6276.patch The code has many places where null check and other condition checks are unnecessary. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6271) Rename JournalProtocol to BackupNodeJournalProtocol
Suresh Srinivas created HDFS-6271: - Summary: Rename JournalProtocol to BackupNodeJournalProtocol Key: HDFS-6271 URL: https://issues.apache.org/jira/browse/HDFS-6271 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Suresh Srinivas [~shv], as indicated in earlier comments, JournalProtocol is used only for sending journal to backupnode .Renaming it would make it clear how the protocol is being used and will not conflict with QuorumJournalProtocol. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6185) HDFS operational and debuggability improvements
Suresh Srinivas created HDFS-6185: - Summary: HDFS operational and debuggability improvements Key: HDFS-6185 URL: https://issues.apache.org/jira/browse/HDFS-6185 Project: Hadoop HDFS Issue Type: Improvement Reporter: Suresh Srinivas Assignee: Suresh Srinivas This umbrella jira proposes improvements in HDFS to make operations simpler and the system more robust. The areas of improvements includes - ensuring configuration is correct and follows the know best practices, make HDFS robust against some of the failures observed due to cluster not being monitored correctly, better metadata management to avoid lengthy startup time etc. Subtasks will be filed with more details on individual improvements. The goal is simplify operations, improve debuggability and robustness. If you have ideas on improvements that fall under this umbrella, please file subtasks under this jira. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6186) Pause deletion of blocks when the namenode starts up
Suresh Srinivas created HDFS-6186: - Summary: Pause deletion of blocks when the namenode starts up Key: HDFS-6186 URL: https://issues.apache.org/jira/browse/HDFS-6186 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Suresh Srinivas HDFS namenode can delete blocks very quickly, given the deletion happens as a parallel operation spread across many datanodes. One of the frequent anxieties I see is that a lot of data can be deleted very quickly, when a cluster is brought up, especially when one of the storage directories has failed and namenode metadata was copied from another storage. Copying wrong metadata would results in some of the newer files (if old metadata was copied) being deleted along with their blocks. HDFS-5986 now captures the number of pending deletion block on namenode webUI and JMX. I propose pausing deletion of blocks for a configured period of time (default 1 hour?) after namenode comes out of safemode. This will give enough time for the administrator to notice large number of pending deletion blocks and take corrective action. Thoughts? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6162) Format strings should use platform independent line separator
Suresh Srinivas created HDFS-6162: - Summary: Format strings should use platform independent line separator Key: HDFS-6162 URL: https://issues.apache.org/jira/browse/HDFS-6162 Project: Hadoop HDFS Issue Type: Bug Reporter: Suresh Srinivas Assignee: Suresh Srinivas Priority: Minor In format strings, it is generally preferable better to use %n, which will result in platform-specific line separator. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6155) Fix Boxing/unboxing to parse a primitive findbugs warnings
Suresh Srinivas created HDFS-6155: - Summary: Fix Boxing/unboxing to parse a primitive findbugs warnings Key: HDFS-6155 URL: https://issues.apache.org/jira/browse/HDFS-6155 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Suresh Srinivas Assignee: Suresh Srinivas There are many instances of such findbugs warnings related to performance. See for details - http://findbugs.sourceforge.net/bugDescriptions.html#DM_BOXED_PRIMITIVE_FOR_PARSING -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6150) Add inode id information in the logs to make debugging easier
Suresh Srinivas created HDFS-6150: - Summary: Add inode id information in the logs to make debugging easier Key: HDFS-6150 URL: https://issues.apache.org/jira/browse/HDFS-6150 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Reporter: Suresh Srinivas Attachments: HDFS-6150.patch Inode information and path information are missing in the logs and exceptions. Adding this will help debug multi threading issues related to using incode INode ID information. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Reopened] (HDFS-5138) Support HDFS upgrade in HA
[ https://issues.apache.org/jira/browse/HDFS-5138?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas reopened HDFS-5138: --- Support HDFS upgrade in HA -- Key: HDFS-5138 URL: https://issues.apache.org/jira/browse/HDFS-5138 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.1.1-beta Reporter: Kihwal Lee Assignee: Aaron T. Myers Priority: Blocker Fix For: 3.0.0 Attachments: HDFS-5138.patch, HDFS-5138.patch, HDFS-5138.patch, HDFS-5138.patch, HDFS-5138.patch, HDFS-5138.patch, HDFS-5138.patch, HDFS-5138.patch, HDFS-5138.patch, HDFS-5138.patch, HDFS-5138.patch, hdfs-5138-branch-2.txt With HA enabled, NN wo't start with -upgrade. Since there has been a layout version change between 2.0.x and 2.1.x, starting NN in upgrade mode was necessary when deploying 2.1.x to an existing 2.0.x cluster. But the only way to get around this was to disable HA and upgrade. The NN and the cluster cannot be flipped back to HA until the upgrade is finalized. If HA is disabled only on NN for layout upgrade and HA is turned back on without involving DNs, things will work, but finaliizeUpgrade won't work (the NN is in HA and it cannot be in upgrade mode) and DN's upgrade snapshots won't get removed. We will need a different ways of doing layout upgrade and upgrade snapshot. I am marking this as a 2.1.1-beta blocker based on feedback from others. If there is a reasonable workaround that does not increase maintenance window greatly, we can lower its priority from blocker to critical. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6117) Print file path information in FileNotFoundException
Suresh Srinivas created HDFS-6117: - Summary: Print file path information in FileNotFoundException Key: HDFS-6117 URL: https://issues.apache.org/jira/browse/HDFS-6117 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.2.0 Reporter: Suresh Srinivas Assignee: Suresh Srinivas Print file path information in FileNotFoundException thrown from INodeId#checkId(). This helps debug issues related possible INode Id mismatches during namenode fail over. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6118) Code cleanup
Suresh Srinivas created HDFS-6118: - Summary: Code cleanup Key: HDFS-6118 URL: https://issues.apache.org/jira/browse/HDFS-6118 Project: Hadoop HDFS Issue Type: Bug Reporter: Suresh Srinivas Assignee: Suresh Srinivas HDFS code needs cleanup related to many typos, undocumented parameters, unused methods, unnecessary cast, imports and exceptions declared as thrown to name a few. I plan on working on cleaning this up as I get time. To keep code review manageable, I will create sub tasks and cleanup the code a few classes at a time. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6119) FSNamesystem code cleanup
Suresh Srinivas created HDFS-6119: - Summary: FSNamesystem code cleanup Key: HDFS-6119 URL: https://issues.apache.org/jira/browse/HDFS-6119 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Suresh Srinivas Assignee: Suresh Srinivas -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6124) Add final modifier to class members
Suresh Srinivas created HDFS-6124: - Summary: Add final modifier to class members Key: HDFS-6124 URL: https://issues.apache.org/jira/browse/HDFS-6124 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 2.4.0 Reporter: Suresh Srinivas Assignee: Suresh Srinivas Attachments: HDFS-6124.patch Many of the member variables declaration for classes are missing final modifier in HDFS. This jira adds final modifier where possible. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Reopened] (HDFS-6099) HDFS file system limits not enforced on renames.
[
https://issues.apache.org/jira/browse/HDFS-6099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suresh Srinivas reopened HDFS-6099:
---
Sorry, accidentally resolved this.
HDFS file system limits not enforced on renames.
Key: HDFS-6099
URL: https://issues.apache.org/jira/browse/HDFS-6099
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.3.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Fix For: 2.4.0
Attachments: HDFS-6099.1.patch
{{dfs.namenode.fs-limits.max-component-length}} and
{{dfs.namenode.fs-limits.max-directory-items}} are not enforced on the
destination path during rename operations. This means that it's still
possible to create files that violate these limits.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HDFS-6055) Change default configuration to limit file name length in HDFS
Suresh Srinivas created HDFS-6055: - Summary: Change default configuration to limit file name length in HDFS Key: HDFS-6055 URL: https://issues.apache.org/jira/browse/HDFS-6055 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Reporter: Suresh Srinivas Assignee: Suresh Srinivas Currently configuration dfs.namenode.fs-limits.max-component-length is set to 0. With this HDFS file names have no length limit. However, we see more users run into issues where they copy files from HDFS to another file system and the copy fails due to the file name length being too long. I propose changing the default configuration dfs.namenode.fs-limits.max-component-length to a reasonable value. This will be an incompatible change. However, user who need long file names can override this configuration to turn off length limit. What do folks think? -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Resolved] (HDFS-3225) Revist upgrade snapshots, roll back, finalize to enable rolling upgrades
[ https://issues.apache.org/jira/browse/HDFS-3225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HDFS-3225. --- Resolution: Done The rolling upgrade and the related issues have been address in the umbrella jiar, HDFS-5535. I do not think there is any work that needs to be done to address this. Resolving this as Done Revist upgrade snapshots, roll back, finalize to enable rolling upgrades Key: HDFS-3225 URL: https://issues.apache.org/jira/browse/HDFS-3225 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Sanjay Radia Assignee: Sanjay Radia With introduction of wire comaptibility, HDFS is ready for rolling upgrades. Various design choices such as version checks and when to do upgrade snaopshots, rollbacks, finalize need to be revisited. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HDFS-6005) Simplify Datanode rollback and downgrade
Suresh Srinivas created HDFS-6005: - Summary: Simplify Datanode rollback and downgrade Key: HDFS-6005 URL: https://issues.apache.org/jira/browse/HDFS-6005 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, namenode Reporter: Suresh Srinivas Assignee: Suresh Srinivas Problem: When rolling upgrade fails, the cluster can either be downgraded or rolled back. With the current functionality in this feature branch, it is possible to downgrade namenode, while datanode is incorrectly rolled back. This does not affect the cluster state. The old blocks that appear back on the datanode due to rollback will be deleted. Similarly it is also possible to rollback namenode, while datanode is not rolled back. This can cause problem where old blocks do not appear back on the datanode and can result in missing blocks. Solution: I propose making the following changes: During rollback or downgrade, the entire cluster must be restarted. The datanodes always restore the deleted blocks on restart and go back to trash disabled mode. There is no need for datanodes to be started up -rollingUpgrade -rollback, anymore. # On namenode downgrade, the restored blocks are deleted. # On namenode rollback, the restored blocks will be retained and any newly created blocks (since the start of rolling upgrade) are deleted. This is much simpler operationally and solves the problem described above. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Created] (HDFS-5986) Capture the number of blocks pending deletion on namenode webUI
Suresh Srinivas created HDFS-5986: - Summary: Capture the number of blocks pending deletion on namenode webUI Key: HDFS-5986 URL: https://issues.apache.org/jira/browse/HDFS-5986 Project: Hadoop HDFS Issue Type: Bug Components: namenode Reporter: Suresh Srinivas When a directory that has large number of directories and files are deleted, the namespace deletes the corresponding inodes immediately. However it is hard to to know when the invalidated blocks are actually deleted on the datanodes, which could take a while. I propose adding on namenode webUI, along with under replicated blocks, the number of blocks that are pending deletion. -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Resolved] (HDFS-5720) DataNode always READ_BLOCK or WRITE_BLOCK
[
https://issues.apache.org/jira/browse/HDFS-5720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suresh Srinivas resolved HDFS-5720.
---
Resolution: Not A Problem
DataNode always READ_BLOCK or WRITE_BLOCK
-
Key: HDFS-5720
URL: https://issues.apache.org/jira/browse/HDFS-5720
Project: Hadoop HDFS
Issue Type: Task
Components: datanode
Affects Versions: 2.2.0
Reporter: yangping wu
Priority: Trivial
Original Estimate: 480h
Remaining Estimate: 480h
I find many following errors on my datanode's log. What is the reason for
this and how can I resolved it?
{code}
2014-01-05 00:14:40,589 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode: date51:50010:DataXceiver
error processing WRITE_BLOCK operation src: /192.168.246.57:35455 dest:
/192.168.246.75:50010
java.io.IOException: Premature EOF from inputStream
at org.apache.hadoop.io.IOUtils.readFully(IOUtils.java:194)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doReadFully(PacketReceiver.java:213)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.doRead(PacketReceiver.java:134)
at
org.apache.hadoop.hdfs.protocol.datatransfer.PacketReceiver.receiveNextPacket(PacketReceiver.java:109)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receivePacket(BlockReceiver.java:435)
at
org.apache.hadoop.hdfs.server.datanode.BlockReceiver.receiveBlock(BlockReceiver.java:693)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.writeBlock(DataXceiver.java:569)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opWriteBlock(Receiver.java:115)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:68)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:221)
at java.lang.Thread.run(Thread.java:744)
2014-01-05 02:50:52,552 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode: date51:50010:DataXceiver
error processing READ_BLOCK operation src: /192.168.246.39:46693 dest:
/192.168.246.75:50010
org.apache.hadoop.hdfs.server.datanode.ReplicaNotFoundException: Replica not
found for
BP-1398136447-192.168.246.60-1386067202761:blk_1089253308_1099605945003
at
org.apache.hadoop.hdfs.server.datanode.BlockSender.getReplica(BlockSender.java:418)
at
org.apache.hadoop.hdfs.server.datanode.BlockSender.init(BlockSender.java:228)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.readBlock(DataXceiver.java:327)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.opReadBlock(Receiver.java:101)
at
org.apache.hadoop.hdfs.protocol.datatransfer.Receiver.processOp(Receiver.java:65)
at
org.apache.hadoop.hdfs.server.datanode.DataXceiver.run(DataXceiver.java:221)
at java.lang.Thread.run(Thread.java:744)
2014-01-05 07:39:30,939 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode:
DataNode{data=FSDataset{dirpath='[/data1/hadoop/dfs/current,
/data2/hadoop/dfs/current, /data3/hadoop/dfs/current,
/data4/hadoop/dfs/current, /data5/hadoop/dfs/current,
/data6/hadoop/dfs/current, /data7/hadoop/dfs/current,
/data8/hadoop/dfs/current, /data9/hadoop/dfs/current,
/data10/hadoop/dfs/current, /data11/hadoop/dfs/current,
/data12/hadoop/dfs/current]'}, localName='date51:50010',
storageID='DS-1312530819-192.168.246.75-50010-1388062043867',
xmitsInProgress=0}:Exception transfering block
BP-1398136447-192.168.246.60-1386067202761:blk_1089401766_1099606093471 to
mirror 192.168.242.51:50010: java.io.IOException: Connection reset by peer
2014-01-05 07:39:30,939 INFO org.apache.hadoop.hdfs.server.datanode.DataNode:
opWriteBlock
BP-1398136447-192.168.246.60-1386067202761:blk_1089401766_1099606093471
received exception java.io.IOException: Connection reset by peer
2014-01-05 07:39:30,939 ERROR
org.apache.hadoop.hdfs.server.datanode.DataNode: date51:50010:DataXceiver
error processing WRITE_BLOCK operation src: /192.168.246.75:44779 dest:
/192.168.246.75:50010
java.io.IOException: Connection reset by peer
at sun.nio.ch.FileDispatcherImpl.read0(Native Method)
at sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
at sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:223)
at sun.nio.ch.IOUtil.read(IOUtil.java:197)
at sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:379)
at
org.apache.hadoop.net.SocketInputStream$Reader.performIO(SocketInputStream.java:57)
at
org.apache.hadoop.net.SocketIOWithTimeout.doIO(SocketIOWithTimeout.java:142)
at
org.apache.hadoop.net.SocketInputStream.read(SocketInputStream.java:161)
at
[jira] [Created] (HDFS-5704) Change OP_UPDATE_BLOCKS with a new OP_ADD_BLOCK
Suresh Srinivas created HDFS-5704: - Summary: Change OP_UPDATE_BLOCKS with a new OP_ADD_BLOCK Key: HDFS-5704 URL: https://issues.apache.org/jira/browse/HDFS-5704 Project: Hadoop HDFS Issue Type: Bug Environment: Currently every time a block a allocated, the entire list of blocks are written in the editlog in OP_UPDATE_BLOCKS operation. This has n^2 growth issue. The total size of editlog records for a file with large number of blocks could be huge. The goal of this jira is discuss adding a different editlog record that only records allocation of block and not the entire block list, on every block allocation. Reporter: Suresh Srinivas Assignee: Suresh Srinivas -- This message was sent by Atlassian JIRA (v6.1.5#6160)
[jira] [Resolved] (HDFS-5367) Restoring namenode storage locks namenode due to unnecessary fsimage write
[ https://issues.apache.org/jira/browse/HDFS-5367?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HDFS-5367. --- Resolution: Fixed Fix Version/s: 1.3.0 Hadoop Flags: Reviewed I committed the patch to branch-1. Thank you John Zhao. Restoring namenode storage locks namenode due to unnecessary fsimage write -- Key: HDFS-5367 URL: https://issues.apache.org/jira/browse/HDFS-5367 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 1.2.1 Reporter: zhaoyunjiong Assignee: zhaoyunjiong Fix For: 1.3.0 Attachments: HDFS-5367-branch-1.2.patch Our cluster have 40G fsimage, we write one copy of edit log to NFS. After NFS temporary failed, when doing checkpoint, NameNode try to recover it, and it will save 40G fsimage to NFS, it takes some time ( 40G/128MB/s = 320 seconds) , and it locked FSNamesystem, and this bring down our cluster. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HDFS-5340) Add Hive to the list of projects using AbstractDelegationTokenSecretManager
Suresh Srinivas created HDFS-5340: - Summary: Add Hive to the list of projects using AbstractDelegationTokenSecretManager Key: HDFS-5340 URL: https://issues.apache.org/jira/browse/HDFS-5340 Project: Hadoop HDFS Issue Type: Bug Components: security Reporter: Suresh Srinivas org.apache.hadoop.hive.thrift.DelegationTokenSecretManager extends AbstractDelegationTokenSecretManager. This should be captured in the InterfaceAudience annotation of AbstractDelegationTokenSecretManager. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HDFS-5316) Namenode ignore the default https port
Suresh Srinivas created HDFS-5316:
-
Summary: Namenode ignore the default https port
Key: HDFS-5316
URL: https://issues.apache.org/jira/browse/HDFS-5316
Project: Hadoop HDFS
Issue Type: Sub-task
Reporter: Suresh Srinivas
When dfs.https.enable is true and dfs.https.port is not configured, namenode
does not pickup the default https port (50470), instead picks up random port.
See:
{code}
boolean needClientAuth = conf.getBoolean(dfs.https.need.client.auth, false);
InetSocketAddress secInfoSocAddr = NetUtils.createSocketAddr(infoHost +
: + conf.get(
DFSConfigKeys.DFS_NAMENODE_HTTPS_PORT_KEY, 0));
Configuration sslConf = new Configuration(false);
if (certSSL) {
sslConf.addResource(conf.get(DFSConfigKeys.DFS_SERVER_HTTPS_KEYSTORE_RESOURCE_KEY,
ssl-server.xml));
}
{code}
Unless https port is specifically configured as 0, we should not be picking
random port. This needs to be documented in hdfs-default.xml
--
This message was sent by Atlassian JIRA
(v6.1#6144)
[jira] [Created] (HDFS-5317) Go back to DFS Home link does not work on datanode webUI
Suresh Srinivas created HDFS-5317: - Summary: Go back to DFS Home link does not work on datanode webUI Key: HDFS-5317 URL: https://issues.apache.org/jira/browse/HDFS-5317 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Suresh Srinivas Priority: Critical Here is how to reproduce this problem: # Goto https://namenode:https_port/dfshealth.jsp # Click on Live Nodes link # On the Live Nodes page, click one of the links for the datanodes. This link has always hardcodes the http link instead of http/https based on the context of the webpage. On datanode webUI page, the link Go back to DFS home does not work. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Resolved] (HDFS-3538) TestBlocksWithNotEnoughRacks fails
[ https://issues.apache.org/jira/browse/HDFS-3538?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HDFS-3538. --- Resolution: Duplicate This is a duplicate of HDFS-3267. TestBlocksWithNotEnoughRacks fails -- Key: HDFS-3538 URL: https://issues.apache.org/jira/browse/HDFS-3538 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 0.24.0 Reporter: Brandon Li It failed for a few days in jenkins test. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HDFS-5310) Add http policy support for YARN daemons
Suresh Srinivas created HDFS-5310: - Summary: Add http policy support for YARN daemons Key: HDFS-5310 URL: https://issues.apache.org/jira/browse/HDFS-5310 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.0-alpha Reporter: Suresh Srinivas This YARN part of HADOOP-10022. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HDFS-5306) Datanode https port is not available in the namenode
Suresh Srinivas created HDFS-5306: - Summary: Datanode https port is not available in the namenode Key: HDFS-5306 URL: https://issues.apache.org/jira/browse/HDFS-5306 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, namenode Reporter: Suresh Srinivas Assignee: Suresh Srinivas To enable https access, the datanode http server https port is needed in namenode web pages and redirects from the namenode. This jira adds an additional optional field to DatanodeIDProto in hdfs.proto and the corresponding DatanodeID java class. -- This message was sent by Atlassian JIRA (v6.1#6144)
[jira] [Created] (HDFS-5187) Deletion of non-existing cluster succeeds
Suresh Srinivas created HDFS-5187:
-
Summary: Deletion of non-existing cluster succeeds
Key: HDFS-5187
URL: https://issues.apache.org/jira/browse/HDFS-5187
Project: Hadoop HDFS
Issue Type: Bug
Reporter: Suresh Srinivas
Following command succeeds even if the cluster of name non-existent has not
been added:
{noformat}
bin/falcon entity -delete -type cluster -name non-existent
falcon/abc(cluster) removed successfully
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4940) namenode OOMs under Bigtop's TestCLI
[ https://issues.apache.org/jira/browse/HDFS-4940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HDFS-4940. --- Resolution: Cannot Reproduce [~rvs], I have closed this. We can reopen this, if this problem occurs again. namenode OOMs under Bigtop's TestCLI Key: HDFS-4940 URL: https://issues.apache.org/jira/browse/HDFS-4940 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Roman Shaposhnik Assignee: Roman Shaposhnik Priority: Blocker Fix For: 2.3.0 Bigtop's TestCLI when executed against Hadoop 2.1.0 seems to make it OOM quite reliably regardless of the heap size settings. I'm attaching a heap dump URL. Alliteratively anybody can just take Bigtop's tests, compiled them against Hadoop 2.1.0 bits and try to reproduce it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4747) Convert snapshot user guide to APT from XDOC
[ https://issues.apache.org/jira/browse/HDFS-4747?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HDFS-4747. --- Resolution: Won't Fix Marking this as won't fix. Reopen if necessary. Convert snapshot user guide to APT from XDOC Key: HDFS-4747 URL: https://issues.apache.org/jira/browse/HDFS-4747 Project: Hadoop HDFS Issue Type: Sub-task Components: documentation Affects Versions: Snapshot (HDFS-2802) Reporter: Aaron T. Myers Assignee: Aaron T. Myers Attachments: HDFS-4747.patch, HdfsSnapshots.html To be consistent with the rest of the HDFS docs, the snapshots user guide should use APT instead of XDOC. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-5015) Error while running custom writable code - DFSClient_NONMAPREDUCE_549327626_1 doesnot have any open file
[
https://issues.apache.org/jira/browse/HDFS-5015?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suresh Srinivas resolved HDFS-5015.
---
Resolution: Invalid
Please use forums for help. Jira is for reporting bugs.
Error while running custom writable code -
DFSClient_NONMAPREDUCE_549327626_1 doesnot have any open file
-
Key: HDFS-5015
URL: https://issues.apache.org/jira/browse/HDFS-5015
Project: Hadoop HDFS
Issue Type: Bug
Components: hdfs-client
Environment: Cloudera 4.3
Reporter: Allan Gonsalves
hi ,
We are facing an below error while running custom writable code, our driver
code is using below code . logwritable is the name of custom writable
interface.
--Error message at the code execution is as follows
DFSClient_NONMAPREDUCE_549327626_1 doesnot have any open file
--Driver code -
job.setMapOutputKeyClass(logwritable.class);
job.setMapOutputValueClass(logwritable.class);
job.setNumReducerTasks(0);
--logwritable class implements Writable interface
--Definition of mapper is as follows
public class MAPPER extends MapperText,IntWritable,Logwritable, Logwritabe{
contaxt.write(new logwritable(), new logwritable());
}
Any help to resolve above issue will be great help
Thanks
Allan
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-5008) Make ClientProtocol#abandonBlock() idempotent
Suresh Srinivas created HDFS-5008: - Summary: Make ClientProtocol#abandonBlock() idempotent Key: HDFS-5008 URL: https://issues.apache.org/jira/browse/HDFS-5008 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Reporter: Suresh Srinivas Currently when abandonBlock is called, if the block does not exist, then the namenode throws an exception. I propose making it idempotent, by returning success if the block that is being abandoned does not exist. Thoughts? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4986) Namenode webUI changes to incorporate storage type
Suresh Srinivas created HDFS-4986: - Summary: Namenode webUI changes to incorporate storage type Key: HDFS-4986 URL: https://issues.apache.org/jira/browse/HDFS-4986 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Suresh Srinivas Namenode needs to change cluster storage summary to show storage information per storage type. Datanode list must also include this information. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4985) Add storage type to the protocol and expose it in block report and block locations
Suresh Srinivas created HDFS-4985: - Summary: Add storage type to the protocol and expose it in block report and block locations Key: HDFS-4985 URL: https://issues.apache.org/jira/browse/HDFS-4985 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, namenode Reporter: Suresh Srinivas With HDFS-2880 datanode now supports storage abstraction. This is to add storage type in to the protocol. Datanodes currently report blocks per storage. Storage would include storage type attribute. Namenode also exposes the storage type of a block in block locations. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4987) Namenode changes to track multiple storages
Suresh Srinivas created HDFS-4987: - Summary: Namenode changes to track multiple storages Key: HDFS-4987 URL: https://issues.apache.org/jira/browse/HDFS-4987 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Suresh Srinivas Currently namenode track in BlockInfo, the corresponding DataNodeDescriptor. I propose changing this with new abstraction StorageDescriptor. This change will also make DatanodeDescriptor a collection of StroageDescriptors. This will allow given a block, identify its storage and also Datanode. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4988) Datanode must support all the volumes as individual storages
Suresh Srinivas created HDFS-4988: - Summary: Datanode must support all the volumes as individual storages Key: HDFS-4988 URL: https://issues.apache.org/jira/browse/HDFS-4988 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Suresh Srinivas Currently all the volumes on datanode is reported as a single storage. This change proposes reporting them as individual storage. This requires: # A unique storage ID for each storage #* This needs to be generated during formatting # There should be an option to allow existing disks to be reported as single storage unit for backward compatibility. # A functionality is also needed to split the existing all volumes as single storage unit to to individual storage units. # Configuration must allow for each storage unit a storage type attribute. # Block reports must be sent on a per storage basis. In some cases (such memory tier) block reports may need to be sent more frequently. That means block reporting period must be on a per storage type basis. My proposal is for new clusters to configure volumes by default as separate storage unit. Lets discuss. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4989) Balancer needs to consider storage type in balancing
Suresh Srinivas created HDFS-4989: - Summary: Balancer needs to consider storage type in balancing Key: HDFS-4989 URL: https://issues.apache.org/jira/browse/HDFS-4989 Project: Hadoop HDFS Issue Type: Sub-task Components: balancer, datanode, namenode Reporter: Suresh Srinivas Balancer needs to balance with in a storage tier. Also needs an option to balance only a specific storage type. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4990) Block placement for storage types
Suresh Srinivas created HDFS-4990: - Summary: Block placement for storage types Key: HDFS-4990 URL: https://issues.apache.org/jira/browse/HDFS-4990 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Suresh Srinivas Currently block location for writes are made based on: # Datanode load (number of transceivers) # Space left on datanode # Topology With storage abstraction, namenode must choose a storage instead of a datanode for block placement. It also needs to consider storage type, load on the storage etc. As an additional benefit, currently HDFS support heterogeneous nodes (nodes with different number of spindles etc.) poorly. This work should help solve that issue as well. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4973) Move the Rpc request call ID generation to client side InvocationHandler
Suresh Srinivas created HDFS-4973: - Summary: Move the Rpc request call ID generation to client side InvocationHandler Key: HDFS-4973 URL: https://issues.apache.org/jira/browse/HDFS-4973 Project: Hadoop HDFS Issue Type: Bug Reporter: Suresh Srinivas Currently when RetryInvocationHandler is used to retry an RPC request, a new RPC request call ID is generated. This jira proposes moving call ID generation to InvocationHandler so that retried RPC requests retain the same call ID. This is needed for RetryCache functionality proposed in HDFS-4942. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4974) Analyze and add annotations to the Namenode and Datanode protocol methods to enable retry
Suresh Srinivas created HDFS-4974: - Summary: Analyze and add annotations to the Namenode and Datanode protocol methods to enable retry Key: HDFS-4974 URL: https://issues.apache.org/jira/browse/HDFS-4974 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Suresh Srinivas This jira is intended for: # Discussing current @Idempotent annotations in HDFS protocols and adding that annotation where it is missing. # Discuss how retry should be enabled for non-idempotent requests. I will post the analysis of current methods in subsequent comment. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4976) Rename Client#uuid to Client#clientId
Suresh Srinivas created HDFS-4976: - Summary: Rename Client#uuid to Client#clientId Key: HDFS-4976 URL: https://issues.apache.org/jira/browse/HDFS-4976 Project: Hadoop HDFS Issue Type: Sub-task Reporter: Suresh Srinivas To address the comment - https://issues.apache.org/jira/browse/HADOOP-9688?focusedCommentId=13705032page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13705032 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-3092) Enable journal protocol based editlog streaming for standby namenode
[ https://issues.apache.org/jira/browse/HDFS-3092?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HDFS-3092. --- Resolution: Won't Fix Enable journal protocol based editlog streaming for standby namenode Key: HDFS-3092 URL: https://issues.apache.org/jira/browse/HDFS-3092 Project: Hadoop HDFS Issue Type: Improvement Components: ha, namenode Affects Versions: 0.24.0, 0.23.3 Reporter: Suresh Srinivas Assignee: Suresh Srinivas Attachments: ComparisonofApproachesforHAJournals.pdf, JNStates.png, MultipleSharedJournals.pdf, MultipleSharedJournals.pdf, MultipleSharedJournals.pdf, Removingfilerdependency.pdf Currently standby namenode relies on reading shared editlogs to stay current with the active namenode, for namespace changes. BackupNode used streaming edits from active namenode for doing the same. This jira is to explore using journal protocol based editlog streams for the standby namenode. A daemon in standby will get the editlogs from the active and write it to local edits. To begin with, the existing standby mechanism of reading from a file, will continue to be used, instead of from shared edits, from the local edits. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4955) Use InodeID to improve snapshot diff
[ https://issues.apache.org/jira/browse/HDFS-4955?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HDFS-4955. --- Resolution: Duplicate Duplicate of HDFS-4667 Use InodeID to improve snapshot diff Key: HDFS-4955 URL: https://issues.apache.org/jira/browse/HDFS-4955 Project: Hadoop HDFS Issue Type: Bug Reporter: Binglin Chang Currently snapshot diff doesn't handle file/dir rename very well, so distcp can't do minimal data transfer in incremental backup. Using InodeID can uniquely identify a file, and can help diff search algorithm. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4956) DistributedFileSystem does not handle CreateFlag.APPEND in create call
Suresh Srinivas created HDFS-4956: - Summary: DistributedFileSystem does not handle CreateFlag.APPEND in create call Key: HDFS-4956 URL: https://issues.apache.org/jira/browse/HDFS-4956 Project: Hadoop HDFS Issue Type: Bug Components: namenode Reporter: Suresh Srinivas Currently DistributedFileSystem does not handle CreateFlag.APPEND in the implementation of FileSystem#create() method. It only support OVERWRITE CREATE or just CREATE. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4942) Add retry cache support in Namenode
Suresh Srinivas created HDFS-4942: - Summary: Add retry cache support in Namenode Key: HDFS-4942 URL: https://issues.apache.org/jira/browse/HDFS-4942 Project: Hadoop HDFS Issue Type: Improvement Components: ha, namenode Reporter: Suresh Srinivas Assignee: Suresh Srinivas In current HA mechanism with FailoverProxyProvider and non HA setups with RetryProxy retry a request from the RPC layer. If the retried request has already been processed at the namenode, the subsequent attempts fail for idempotent operations such as create, append, delete, rename etc. This will cause application failures during HA failover, network issues etc. This jira proposes adding retry cache at the namenode to handle these failures. More details in the comments. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4923) Save namespace when the namenode is stopped
Suresh Srinivas created HDFS-4923: - Summary: Save namespace when the namenode is stopped Key: HDFS-4923 URL: https://issues.apache.org/jira/browse/HDFS-4923 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 3.0.0 Reporter: Suresh Srinivas Assignee: Suresh Srinivas In rare instances the namenode fails to load editlog due to corruption during startup. This has more severe impact if editlog segment to be checkpointed has corruption, as checkpointing fails because the editlog with corruption cannot be consumed. If an administrator does not notice this and address it by saving the namespace, recovering the namespace would involve complex file editing, using previous backups or losing last set of modifications. The other issue that also happens frequently is, checkpointing fails and has not happened for a long time, resulting in long editlogs and even corrupt editlogs. To handle these issues, when namenode is stopped, we can put it in safemode and save the namespace, before the process is shutdown. As an added benefit, the namenode restart would be faster, given there is no editlog to consume. What do folks think? -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4921) Create a web page to display the namenode features and whether they are turned on or off
Suresh Srinivas created HDFS-4921: - Summary: Create a web page to display the namenode features and whether they are turned on or off Key: HDFS-4921 URL: https://issues.apache.org/jira/browse/HDFS-4921 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 3.0.0 Reporter: Suresh Srinivas The only way to determine a feature is turned on or off is to look at namenode logs or configuration. Some time when features change/modified (short circuit for example), the corresponding log message changes or goes from info to debug etc. Logs are not a reliable way to programmatically, for example in tests, to determine if a feature is turned on/off. Also when debugging issues, routinely I have to ask for configuration to determine if the feature is turned on or off and hope that the configuration provided is the correct one loaded in the namenode. While namenode does provide config it has loaded over http, some times either due to poor documentation or lack of understanding of configuration mechanism, people may not be able to determine if a state of the feature. I propose adding a web page that shows all that we print today in the logs, that feature status and the corresponding configuration values and feature status. This should also be made available over JMX and JMX http, just like all the other namenode webUI info, so that scripts and other programs can consume it. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4912) Cleanup FSNamesystem#startFileInternal
Suresh Srinivas created HDFS-4912: - Summary: Cleanup FSNamesystem#startFileInternal Key: HDFS-4912 URL: https://issues.apache.org/jira/browse/HDFS-4912 Project: Hadoop HDFS Issue Type: Bug Components: namenode Reporter: Suresh Srinivas Assignee: Suresh Srinivas FSNamesystem#startFileInternal is used by both create and append. This results in ugly if else conditions to consider append/create scenarios. This method can be refactored and the code can be cleaned up. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-2498) TestParallelRead times out consistently on Jenkins
[ https://issues.apache.org/jira/browse/HDFS-2498?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HDFS-2498. --- Resolution: Not A Problem TestParallelRead times out consistently on Jenkins -- Key: HDFS-2498 URL: https://issues.apache.org/jira/browse/HDFS-2498 Project: Hadoop HDFS Issue Type: Bug Components: test Affects Versions: 0.22.0 Reporter: Konstantin Shvachko Assignee: Jakob Homan Priority: Blocker Attachments: TestParallelRead.txt During last several Jenkins builds TestParallelRead consistently fails. See Hadoop-Hdfs-22-branch for logs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4903) Print trash configuration and trash emptier state in namenode log
Suresh Srinivas created HDFS-4903: - Summary: Print trash configuration and trash emptier state in namenode log Key: HDFS-4903 URL: https://issues.apache.org/jira/browse/HDFS-4903 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Reporter: Suresh Srinivas Priority: Minor Namenode should print an info level log that print trash interval and if trash emptier is started or not. Also fix a typo Cannot start tresh in Namenode.java. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4904) Remove JournalService
Suresh Srinivas created HDFS-4904: - Summary: Remove JournalService Key: HDFS-4904 URL: https://issues.apache.org/jira/browse/HDFS-4904 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.0.3-alpha Reporter: Suresh Srinivas JournalService class was added in HDFS-3099. Since it was not used in HDFS-3077, which has JournalNodeRpcServer instead, I propose deleting this dead code. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4896) dfs -command webhdfs:// gives IllegalStateException for secure cluster
[ https://issues.apache.org/jira/browse/HDFS-4896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HDFS-4896. --- Resolution: Duplicate Resolving this as duplicate of HDFS-4841. dfs -command webhdfs:// gives IllegalStateException for secure cluster Key: HDFS-4896 URL: https://issues.apache.org/jira/browse/HDFS-4896 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: yeshavora Fix For: 2.1.0-beta Running: hadoop dfs -copyToLocal webhdfs://node1:port1/File1 /tmp/File1 13/06/07 21:54:58 WARN util.ShutdownHookManager: ShutdownHook 'ClientFinalizer' failed, java.lang.IllegalStateException: Shutdown in progress, cannot add a shutdownHook java.lang.IllegalStateException: Shutdown in progress, cannot add a shutdownHook at org.apache.hadoop.util.ShutdownHookManager.addShutdownHook(ShutdownHookManager.java:152) at org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2401) at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2373) at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:352) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$DtRenewer.getWebHdfs(WebHdfsFileSystem.java:1003) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$DtRenewer.cancel(WebHdfsFileSystem.java:1015) at org.apache.hadoop.security.token.Token.cancel(Token.java:382) at org.apache.hadoop.fs.DelegationTokenRenewer$RenewAction.cancel(DelegationTokenRenewer.java:152) at org.apache.hadoop.fs.DelegationTokenRenewer$RenewAction.access$200(DelegationTokenRenewer.java:58) at org.apache.hadoop.fs.DelegationTokenRenewer.removeRenewAction(DelegationTokenRenewer.java:241) at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.close(WebHdfsFileSystem.java:824) at org.apache.hadoop.fs.FileSystem$Cache.closeAll(FileSystem.java:2447) at org.apache.hadoop.fs.FileSystem$Cache$ClientFinalizer.run(FileSystem.java:2464) at org.apache.hadoop.util.ShutdownHookManager$1.run(ShutdownHookManager.java:54) -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4892) Number of transceivers reported by the datanode is incorrect
Suresh Srinivas created HDFS-4892: - Summary: Number of transceivers reported by the datanode is incorrect Key: HDFS-4892 URL: https://issues.apache.org/jira/browse/HDFS-4892 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.0-alpha Reporter: Suresh Srinivas Currently a datanode reports the transceiver count to namenode. Namenode aggregates this as TotalLoad metrics and is used for monitoring cluster activity. It is also used for making block placement decision, to qualify if a datanode is a good target. Currently transceiver count is = 1 (for XceiverServer) + 1 * (number of readers) + 2 * (Number of writers in pipeline) + 1 * (number of datanode replications) + 1 * (number of recover blocks) Should the number of transceiver just reflect number of readers + writers, instead of reporting it as is currently done. Separately we should perhaps report readers and writers as separate count. This -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4893) Report number of readers and writes from Datanode to Namenode
Suresh Srinivas created HDFS-4893: - Summary: Report number of readers and writes from Datanode to Namenode Key: HDFS-4893 URL: https://issues.apache.org/jira/browse/HDFS-4893 Project: Hadoop HDFS Issue Type: Improvement Affects Versions: 2.0.0-alpha Reporter: Suresh Srinivas Currently datanode reports combined number of readers and writers as transceiver count to the namenode in heartbeats. This jira proposes reporting number of readers and writers as separate fields in heartbeats. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4889) passing -Dfs.trash.interval to command line is not respected.
[ https://issues.apache.org/jira/browse/HDFS-4889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HDFS-4889. --- Resolution: Not A Problem passing -Dfs.trash.interval to command line is not respected. - Key: HDFS-4889 URL: https://issues.apache.org/jira/browse/HDFS-4889 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.1.0-beta Reporter: Trupti Dhavle Fix For: 2.1.0-beta Ran hadoop dfs -Dfs.trash.interval=0 -rm /user/username/README DEPRECATED: Use of this script to execute hdfs command is deprecated. Instead use the hdfs command for it. Moved: 'hdfs://host:port/user/username/README' to trash at: hdfs://host:port/user/username/.Trash/Current Expected that the file doesnt go to Trash and gets deleted directly. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4808) Limit snapshot related methods to DistributedFileSystem
Suresh Srinivas created HDFS-4808: - Summary: Limit snapshot related methods to DistributedFileSystem Key: HDFS-4808 URL: https://issues.apache.org/jira/browse/HDFS-4808 Project: Hadoop HDFS Issue Type: Bug Reporter: Suresh Srinivas HDFS-2802 introduces snapshots related methods into FileSystem. This jira is for discussing whether these snapshots related methods are needed in FileSystem or should be limited to only DistributedFileSystem. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4434) Provide a mapping from INodeId to INode
[ https://issues.apache.org/jira/browse/HDFS-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HDFS-4434. --- Resolution: Fixed Fix Version/s: (was: 3.0.0) 2.0.5-beta Provide a mapping from INodeId to INode --- Key: HDFS-4434 URL: https://issues.apache.org/jira/browse/HDFS-4434 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 3.0.0 Reporter: Brandon Li Assignee: Suresh Srinivas Fix For: 2.0.5-beta Attachments: HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch This JIRA is to provide a way to access the INode via its id. The proposed solution is to have an in-memory mapping from INodeId to INode. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4785) Concat operation does not removed concatenated files from InodeMap
Suresh Srinivas created HDFS-4785: - Summary: Concat operation does not removed concatenated files from InodeMap Key: HDFS-4785 URL: https://issues.apache.org/jira/browse/HDFS-4785 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 3.0.0 Reporter: Suresh Srinivas Assignee: Suresh Srinivas Concat operations takes a set of src files under a directory and concatenates them to a single target file. The inodes corresponding to src files are not deleted from the InodeMap. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4610) Move to using common utils FileUtil#setReadable/Writable/Executable and FileUtil#canRead/Write/Execute
[ https://issues.apache.org/jira/browse/HDFS-4610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HDFS-4610. --- Resolution: Fixed Fix Version/s: 3.0.0 Hadoop Flags: Reviewed +1 for the patch. I committed it to trunk. Thank you Ivan. Thanks to Arpit and Chris for the reviews. Move to using common utils FileUtil#setReadable/Writable/Executable and FileUtil#canRead/Write/Execute -- Key: HDFS-4610 URL: https://issues.apache.org/jira/browse/HDFS-4610 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.0.0 Reporter: Ivan Mitic Assignee: Ivan Mitic Fix For: 3.0.0 Attachments: HDFS-4610.commonfileutils.2.patch, HDFS-4610.commonfileutils.3.patch, HDFS-4610.commonfileutils.patch Switch to using common utils described in HADOOP-9413 that work well cross-platform. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-4610) Move to using common utils FileUtil#setReadable/Writable/Executable and FileUtil#canRead/Write/Execute
[ https://issues.apache.org/jira/browse/HDFS-4610?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas reopened HDFS-4610: --- Move to using common utils FileUtil#setReadable/Writable/Executable and FileUtil#canRead/Write/Execute -- Key: HDFS-4610 URL: https://issues.apache.org/jira/browse/HDFS-4610 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.0.0 Reporter: Ivan Mitic Assignee: Ivan Mitic Fix For: 3.0.0 Attachments: HDFS-4610.commonfileutils.2.patch, HDFS-4610.commonfileutils.3.patch, HDFS-4610.commonfileutils.patch Switch to using common utils described in HADOOP-9413 that work well cross-platform. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4777) File creation code in namenode is incorrectly synchronized
Suresh Srinivas created HDFS-4777: - Summary: File creation code in namenode is incorrectly synchronized Key: HDFS-4777 URL: https://issues.apache.org/jira/browse/HDFS-4777 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.0.0-alpha, 0.23.0 Reporter: Suresh Srinivas Assignee: Suresh Srinivas Priority: Blocker FSNamesystem#startFileInternal calls delete. Delete method releases the write lock, making parts of startFileInternal code unintentionally executed without write lock being held. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-4489) Use InodeID as as an identifier of a file in HDFS protocols and APIs
[ https://issues.apache.org/jira/browse/HDFS-4489?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas reopened HDFS-4489: --- Use InodeID as as an identifier of a file in HDFS protocols and APIs Key: HDFS-4489 URL: https://issues.apache.org/jira/browse/HDFS-4489 Project: Hadoop HDFS Issue Type: Bug Components: namenode Reporter: Brandon Li Assignee: Brandon Li Fix For: 2.0.5-beta The benefit of using InodeID to uniquely identify a file can be multiple folds. Here are a few of them: 1. uniquely identify a file cross rename, related JIRAs include HDFS-4258, HDFS-4437. 2. modification checks in tools like distcp. Since a file could have been replaced or renamed to, the file name and size combination is no t reliable, but the combination of file id and size is unique. 3. id based protocol support (e.g., NFS) 4. to make the pluggable block placement policy use fileid instead of filename (HDFS-385). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Reopened] (HDFS-4434) Provide a mapping from INodeId to INode
[ https://issues.apache.org/jira/browse/HDFS-4434?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas reopened HDFS-4434: --- Provide a mapping from INodeId to INode --- Key: HDFS-4434 URL: https://issues.apache.org/jira/browse/HDFS-4434 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 3.0.0 Reporter: Brandon Li Assignee: Suresh Srinivas Fix For: 3.0.0, 2.0.5-beta Attachments: HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch, HDFS-4434.patch This JIRA is to provide a way to access the INode via its id. The proposed solution is to have an in-memory mapping from INodeId to INode. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4703) Permission checker should not expose resolved paths
Suresh Srinivas created HDFS-4703: - Summary: Permission checker should not expose resolved paths Key: HDFS-4703 URL: https://issues.apache.org/jira/browse/HDFS-4703 Project: Hadoop HDFS Issue Type: Improvement Reporter: Suresh Srinivas Attachments: HDFS-4703.patch Namenode currently prints inode information in AccessControlException. When path provided by user is different from the actual path is resolved to a different path in namenode (some examples as in snapshot paths, HDFS-4434), this might expose information about resolved path. In this jira, in permission checker, I propose adding a separate field for path to print in AccessControlException. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4413) Secondary namenode won't start if HDFS isn't the default file system
[ https://issues.apache.org/jira/browse/HDFS-4413?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HDFS-4413. --- Resolution: Fixed Fix Version/s: 1.2.0 Hadoop Flags: Reviewed I have committed this patch to branch-1, 1.2 and 1-win. Thank you Mostafa. Thank you Chris for reviewing the patch. Secondary namenode won't start if HDFS isn't the default file system Key: HDFS-4413 URL: https://issues.apache.org/jira/browse/HDFS-4413 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 1-win, 1.2.1 Reporter: Mostafa Elhemali Assignee: Mostafa Elhemali Fix For: 1.2.0 Attachments: HDFS-4413.branch-1.patch If HDFS is not the default file system (fs.default.name is something other than hdfs://...), then secondary namenode throws early on in its initialization. This is a needless check as far as I can tell, and blocks scenarios where HDFS services are up but HDFS is not the default file system. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4676) TestHDFSFileSystemContract should set MiniDFSCluster variable to null to free up memory
Suresh Srinivas created HDFS-4676: - Summary: TestHDFSFileSystemContract should set MiniDFSCluster variable to null to free up memory Key: HDFS-4676 URL: https://issues.apache.org/jira/browse/HDFS-4676 Project: Hadoop HDFS Issue Type: Bug Components: test Reporter: Suresh Srinivas Assignee: Suresh Srinivas Priority: Minor TestHDFSFileSystemContract should reset the cluster member to null in order to make garbage collection quickly collect large chunk of memory associated with MiniDFSCluster. This avoids OutOfMemory errors. See https://issues.apache.org/jira/browse/HDFS-4434?focusedCommentId=13624246page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13624246 and the next jenkins tests where the OOM was fixed. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4645) Move from randomly generated block ID to sequentially generated block ID
Suresh Srinivas created HDFS-4645: - Summary: Move from randomly generated block ID to sequentially generated block ID Key: HDFS-4645 URL: https://issues.apache.org/jira/browse/HDFS-4645 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Affects Versions: 3.0.0 Reporter: Suresh Srinivas Assignee: Suresh Srinivas Currently block IDs are randomly generated. This means there is no pattern to block ID generation and no guarantees such as uniqueness of block ID for the life time of the system can be made. I propose using SequentialNumber for block ID generation. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4635) Move BlockManager#computeCapacity to LightWeightGSet
Suresh Srinivas created HDFS-4635: - Summary: Move BlockManager#computeCapacity to LightWeightGSet Key: HDFS-4635 URL: https://issues.apache.org/jira/browse/HDFS-4635 Project: Hadoop HDFS Issue Type: Improvement Components: namenode Reporter: Suresh Srinivas Assignee: Suresh Srinivas The computeCapacity in BlockManager that calculates the LightWeightGSet capacity as the percentage of total JVM memory should be moved to LightWeightGSet. This helps in other maps that are based on the GSet to make use of the same functionality. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4454) Enable offline image viewers to show file Id
[ https://issues.apache.org/jira/browse/HDFS-4454?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HDFS-4454. --- Resolution: Duplicate This could be done as HDFS-4339. Enable offline image viewers to show file Id Key: HDFS-4454 URL: https://issues.apache.org/jira/browse/HDFS-4454 Project: Hadoop HDFS Issue Type: Sub-task Components: tools Reporter: Brandon Li Assignee: Brandon Li -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4469) Inculde file id in these ClientProtocol RPCs which were originally only using path to identify a file
[ https://issues.apache.org/jira/browse/HDFS-4469?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HDFS-4469. --- Resolution: Invalid Given the design from HDFS-4489, a variant of protocol methods to include INodeID to identify the inode is no longer required. Existing path will be used for carrying the inode information and will be part of HDFS-4434. Inculde file id in these ClientProtocol RPCs which were originally only using path to identify a file - Key: HDFS-4469 URL: https://issues.apache.org/jira/browse/HDFS-4469 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Affects Versions: 3.0.0 Reporter: Brandon Li Assignee: Brandon Li Like HDFS-4340(add fileId to addBlock), this JIRA is to track the change to add fileId to other RPC calls, such as append, complete and etc. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4398) Change WebHDFS to support file ID
[ https://issues.apache.org/jira/browse/HDFS-4398?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HDFS-4398. --- Resolution: Invalid This is no longer needed. Path will be used for carrying inode information as proposed in HDFS-4489. So no protocol change or WebHDFS changes required. I am also going make this a task so that it does not clutter the subtask list of HDFS-4489. Change WebHDFS to support file ID - Key: HDFS-4398 URL: https://issues.apache.org/jira/browse/HDFS-4398 Project: Hadoop HDFS Issue Type: Sub-task Components: webhdfs Affects Versions: 3.0.0 Reporter: Brandon Li Assignee: Brandon Li -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4630) Datanode is going OOM due to small files in hdfs
[ https://issues.apache.org/jira/browse/HDFS-4630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HDFS-4630. --- Resolution: Invalid Datanode is going OOM due to small files in hdfs Key: HDFS-4630 URL: https://issues.apache.org/jira/browse/HDFS-4630 Project: Hadoop HDFS Issue Type: Bug Components: datanode, namenode Affects Versions: 2.0.0-alpha Environment: Ubuntu, Java 1.6 Reporter: Ankush Bhatiya Priority: Blocker Hi, We have very small files(size ranging 10KB-1MB) in our hdfs and no of files are in tens of millions. Due to this namenode and datanode both going out of memory very frequently. When we analyse the head dump of datanode most of the memory was used by ReplicaMap. Can we use EhCache or other to not to store all the data in memory? Thanks Ankush -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4627) Fix FSImageFormat#Loader NPE and synchronization issues
[ https://issues.apache.org/jira/browse/HDFS-4627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HDFS-4627. --- Resolution: Fixed Fix Version/s: Snapshot (HDFS-2802) Hadoop Flags: Reviewed I committed the patch to HDFS-2802 branch. Thank you Jing. Fix FSImageFormat#Loader NPE and synchronization issues --- Key: HDFS-4627 URL: https://issues.apache.org/jira/browse/HDFS-4627 Project: Hadoop HDFS Issue Type: Sub-task Components: namenode Reporter: Jing Zhao Assignee: Jing Zhao Fix For: Snapshot (HDFS-2802) Attachments: HDFS-4627.001.patch, HDFS-4627.002.patch 1. Currently FSImageFormat#Loader#loadFilesUnderConstruction calls FSDirectory#unprotectedReplaceINodeFile without acquiring FSDirectory's lock. FSDirectory#replaceINodeFile should be called instead. 2. Currently FSImage#Loader#loadINode does not create block[] object (i.e., keep it as null) when the length of the block is 0. This will cause NPE when the loaded INodeFile is empty, and then an append operation on this inode is applied from edit log. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4622) Remove redundant synchronized from FSNamesystem#rollEditLog in branch-1
[ https://issues.apache.org/jira/browse/HDFS-4622?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HDFS-4622. --- Resolution: Fixed Fix Version/s: 1.2.1 Hadoop Flags: Reviewed +1 for the patch. I committed it to branch-1. Thank you Jing. Remove redundant synchronized from FSNamesystem#rollEditLog in branch-1 --- Key: HDFS-4622 URL: https://issues.apache.org/jira/browse/HDFS-4622 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 1.2.0 Reporter: Jing Zhao Assignee: Jing Zhao Priority: Trivial Fix For: 1.2.1 Attachments: HDFS-4622.b1.001.patch HDFS-4222 moves checkSuperuserPrivilege out of the namespace lock scope but missed the rollEditLog method. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4602) TestBookKeeperHACheckpoints fails
Suresh Srinivas created HDFS-4602: - Summary: TestBookKeeperHACheckpoints fails Key: HDFS-4602 URL: https://issues.apache.org/jira/browse/HDFS-4602 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.0.0 Reporter: Suresh Srinivas See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1344/ Failed tests: testStandbyExceptionThrownDuringCheckpoint(org.apache.hadoop.contrib.bkjournal.TestBookKeeperHACheckpoints): SBN should have still been checkpointing. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4144) Create test for all snapshot-related metrics
[ https://issues.apache.org/jira/browse/HDFS-4144?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HDFS-4144. --- Resolution: Fixed Fix Version/s: Snapshot (HDFS-2802) I committed the patch. Thank you Jing. Create test for all snapshot-related metrics Key: HDFS-4144 URL: https://issues.apache.org/jira/browse/HDFS-4144 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, namenode Reporter: Jing Zhao Assignee: Jing Zhao Fix For: Snapshot (HDFS-2802) Attachments: HDFS-4144.001.patch, HDFS-4144.002.patch, HDFS-4144.003.patch Add testcases for all snapshot-related metrics. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4595) When short circuit read is disallowed, DFSClient does disable short circuit and fall back to regular reads
Suresh Srinivas created HDFS-4595: - Summary: When short circuit read is disallowed, DFSClient does disable short circuit and fall back to regular reads Key: HDFS-4595 URL: https://issues.apache.org/jira/browse/HDFS-4595 Project: Hadoop HDFS Issue Type: Bug Components: hdfs-client Affects Versions: 2.0.0-alpha Reporter: Suresh Srinivas Assignee: Suresh Srinivas In 1.0, when short circuit is disallowed for a user, the DFSClient disables short circuit and falls back to the regular socket based reads over DataTransferProtocol. In release 2.0, this fallback functionality is not working. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4535) port HDFS-1457 to branch-1.1
[ https://issues.apache.org/jira/browse/HDFS-4535?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HDFS-4535. --- Resolution: Won't Fix port HDFS-1457 to branch-1.1 Key: HDFS-4535 URL: https://issues.apache.org/jira/browse/HDFS-4535 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 1.1.2 Reporter: Xiaobo Peng Assignee: Xiaobo Peng Priority: Minor Attachments: HDFS-4535-branch-1.1.patch port HDFS-1457 (configuration option to enable limiting the transfer rate used when sending the image and edits for checkpointing) to branch-1.1 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4579) Annotate snapshot tests
[ https://issues.apache.org/jira/browse/HDFS-4579?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HDFS-4579. --- Resolution: Fixed Hadoop Flags: Reviewed +1 for the patch. I committed it. Thank you Arpit. Annotate snapshot tests --- Key: HDFS-4579 URL: https://issues.apache.org/jira/browse/HDFS-4579 Project: Hadoop HDFS Issue Type: Sub-task Components: test Affects Versions: Snapshot (HDFS-2802) Reporter: Arpit Agarwal Assignee: Arpit Agarwal Fix For: Snapshot (HDFS-2802) Attachments: HDFS-4579.patch Add annotations to snapshot tests, required to merge into trunk. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-3578) missing hadoop.css
[ https://issues.apache.org/jira/browse/HDFS-3578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HDFS-3578. --- Resolution: Duplicate Fix Version/s: 2.0.2-alpha Duplicate of HDFS-1013. missing hadoop.css -- Key: HDFS-3578 URL: https://issues.apache.org/jira/browse/HDFS-3578 Project: Hadoop HDFS Issue Type: Bug Components: build Affects Versions: 2.0.0-alpha Reporter: Roman Shaposhnik Assignee: Roman Shaposhnik Priority: Minor Fix For: 2.0.2-alpha From a report by Konstantin Olchanski: Web interface of HDFS is missing the file hadoop.css and the web pages look funny. This is the fix: [root@ladd12 ~]# locate hadoop.css /usr/lib/hadoop-0.20-mapreduce/webapps/static/hadoop.css [root@ladd12 ~]# mkdir /usr/lib/hadoop-hdfs/webapps/static [root@ladd12 ~]# cp `locate hadoop.css` /usr/lib/hadoop-hdfs/webapps/static -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4222) NN is unresponsive and lose heartbeats of DNs when Hadoop is configured to use LDAP and LDAP has issues
[ https://issues.apache.org/jira/browse/HDFS-4222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HDFS-4222. --- Resolution: Fixed Fix Version/s: 1.2.0 Hadoop Flags: Reviewed I committed the patch to branch-1 as well. Thank you Xiaobo! NN is unresponsive and lose heartbeats of DNs when Hadoop is configured to use LDAP and LDAP has issues --- Key: HDFS-4222 URL: https://issues.apache.org/jira/browse/HDFS-4222 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 1.0.0, 0.23.3, 2.0.0-alpha Reporter: Xiaobo Peng Assignee: Xiaobo Peng Priority: Minor Fix For: 1.2.0, 0.23.7, 2.0.4-beta Attachments: HDFS-4222.23.patch, hdfs-4222-branch-0.23.3.patch, HDFS-4222-branch-1.patch, HDFS-4222.patch, HDFS-4222.patch, hdfs-4222-release-1.0.3.patch For Hadoop clusters configured to access directory information by LDAP, the FSNamesystem calls on behave of DFS clients might hang due to LDAP issues (including LDAP access issues caused by networking issues) while holding the single lock of FSNamesystem. That will result in the NN unresponsive and loss of the heartbeats from DNs. The places LDAP got accessed by FSNamesystem calls are the instantiation of FSPermissionChecker, which could be moved out of the lock scope since the instantiation does not need the FSNamesystem lock. After the move, a DFS client hang will not affect other threads by hogging the single lock. This is especially helpful when we use separate RPC servers for ClientProtocol and DatanodeProtocol since the calls for DatanodeProtocol do not need to access LDAP. So even if DFS clients hang due to LDAP issues, the NN will still be able to process the requests (including heartbeats) from DNs. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4479) logSync() with the FSNamesystem lock held in commitBlockSynchronization
[ https://issues.apache.org/jira/browse/HDFS-4479?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HDFS-4479. --- Resolution: Fixed Fix Version/s: 1.2.0 Hadoop Flags: Reviewed I committed the patch to branch-1. Thank you Jing. logSync() with the FSNamesystem lock held in commitBlockSynchronization --- Key: HDFS-4479 URL: https://issues.apache.org/jira/browse/HDFS-4479 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 1.2.0 Reporter: Jing Zhao Assignee: Jing Zhao Fix For: 1.2.0 Attachments: HDFS-4479.b1.001.patch, HDFS-4479.b1.002.patch In FSNamesystem#commitBlockSynchronization of branch-1, logSync() may be called when the FSNamesystem lock is held. Similar with HDFS-4186, this may cause some performance issue. Since logSync is called right after the synchronization section, we can simply remove the logSync call. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4340) Update addBlock() to inculde inode id as additional argument
[ https://issues.apache.org/jira/browse/HDFS-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HDFS-4340. --- Resolution: Fixed Marking this issue back as resolved. Update addBlock() to inculde inode id as additional argument Key: HDFS-4340 URL: https://issues.apache.org/jira/browse/HDFS-4340 Project: Hadoop HDFS Issue Type: Sub-task Components: hdfs-client, namenode Affects Versions: 3.0.0 Reporter: Brandon Li Assignee: Brandon Li Fix For: 3.0.0 Attachments: HDFS-4340.patch, HDFS-4340.patch, HDFS-4340.patch, HDFS-4340.patch, HDFS-4340.patch, HDFS-4340.patch, HDFS-4340.patch, HDFS-4340.patch, HDFS-4340.patch, HDFS-4340.patch, HDFS-4340.patch -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4466) Remove the deadlock from AbstractDelegationTokenSecretManager
[
https://issues.apache.org/jira/browse/HDFS-4466?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Suresh Srinivas resolved HDFS-4466.
---
Resolution: Fixed
Fix Version/s: 1.2.0
Hadoop Flags: Reviewed
I committed the patch to branch-1. Thank you Brandon!
Remove the deadlock from AbstractDelegationTokenSecretManager
-
Key: HDFS-4466
URL: https://issues.apache.org/jira/browse/HDFS-4466
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode, security
Affects Versions: 1.2.0
Reporter: Brandon Li
Assignee: Brandon Li
Fix For: 1.2.0
Attachments: HDFS-4466.branch-1.patch
In HDFS-3374, new synchronization in
AbstractDelegationTokenSecretManager.ExpiredTokenRemover was added to make
sure the ExpiredTokenRemover thread can be interrupted in time. Otherwise
TestDelegation fails intermittently because the MiniDFScluster thread could
be shut down before tokenRemover thread.
However, as Todd pointed out in HDFS-3374, a potential deadlock was
introduced by its patch:
{quote}
* FSNamesystem.saveNamespace (holding FSN lock) calls
DTSM.saveSecretManagerState (which takes DTSM lock)
* ExpiredTokenRemover.run (holding DTSM lock) calls rollMasterKey calls
updateCurrentKey calls logUpdateMasterKey which takes FSN lock
So if there is a concurrent saveNamespace at the same tie as the expired
token remover runs, it might make the NN deadlock. {quote}
This JIRA is to track the change of removing the possible deadlock from
AbstractDelegationTokenSecretManager.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4475) OutOfMemory by BPServiceActor.offerService() takes down DataNode
[ https://issues.apache.org/jira/browse/HDFS-4475?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HDFS-4475. --- Resolution: Invalid OutOfMemory by BPServiceActor.offerService() takes down DataNode Key: HDFS-4475 URL: https://issues.apache.org/jira/browse/HDFS-4475 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 3.0.0, 2.0.3-alpha Reporter: Plamen Jeliazkov Assignee: Plamen Jeliazkov Fix For: 3.0.0, 2.0.3-alpha In DataNode, there are catchs around BPServiceActor.offerService() call but no catch for OutOfMemory as there is for the DataXeiver as introduced in 0.22.0. The issue can be replicated like this: 1) Create a cluster of X DataNodes and 1 NameNode and low memory settings (-Xmx128M or something similar). 2) Flood HDFS with small file creations (any should work actually). 3) DataNodes will hit OoM, stop blockpool service, and shutdown. The resolution is to catch the OoMException and handle it properly when calling BPServiceActor.offerService() in DataNode.java; like as done in 0.22.0 of Hadoop. DataNodes should not shutdown or crash but remain in a sort of frozen state until memory issues are resolved by GC. LOG ERROR: 2013-02-04 11:46:01,854 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Unexpected exception in block pool Block pool BP-1105714849-10.10.10.110-1360005776467 (storage id DS-1952316202-10.10.10.112-50010-1360005820993) service to vmhost2-vm0/10.10.10.110:8020 java.lang.OutOfMemoryError: GC overhead limit exceeded 2013-02-04 11:46:01,854 WARN org.apache.hadoop.hdfs.server.datanode.DataNode: Ending block pool service for: Block pool BP-1105714849-10.10.10.110-1360005776467 (storage id DS-1952316202-10.10.10.112-50010-1360005820993) service to vmhost2-vm0/10.10.10.110:8020 -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4414) Add support for getting snapshot diff from DistributedFileSystem
[ https://issues.apache.org/jira/browse/HDFS-4414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HDFS-4414. --- Resolution: Fixed Fix Version/s: Snapshot (HDFS-2802) Hadoop Flags: Reviewed Committed the patch to HDFS-2802 branch. Thank you Jing! Add support for getting snapshot diff from DistributedFileSystem Key: HDFS-4414 URL: https://issues.apache.org/jira/browse/HDFS-4414 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, namenode Reporter: Jing Zhao Assignee: Jing Zhao Fix For: Snapshot (HDFS-2802) Attachments: HDFS-4414.001.patch, HDFS-4414.003.patch, HDFS-4414.004.patch, HDFS-4414.005.patch, HDFS-4414+4131.002.patch HDFS-4131 computes the difference between two snapshots (or between a snapshot and the current tree). In this jira we create a DiffReport class to represent the diff to end users. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4343) When storageID of dfs.data.dir of being inconsistent, restart datanode will be failure.
[ https://issues.apache.org/jira/browse/HDFS-4343?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HDFS-4343. --- Resolution: Invalid Closing this jira as Invalid. The behavior in the description is the expected behavior. When storage directories are in inconsistent state, invalid or stale directories are expected to be cleaned manually, before restarting a datanode. If you disagree, feel free to reopen the jira. Please justify why you think it is a valid jira, when you reopen the jira. When storageID of dfs.data.dir of being inconsistent, restart datanode will be failure. --- Key: HDFS-4343 URL: https://issues.apache.org/jira/browse/HDFS-4343 Project: Hadoop HDFS Issue Type: Bug Components: datanode Environment: namenode datanode Reporter: liuyang Attachments: hadoop-root-datanode-167-52-0-55.log, VERSION-1, VERSION-2 A datanode has multiple storage directories configured using dfs.data.dir. When the storageID in the VERSION files in these directories, the datanode fails to startup. Consider a scenario, when old data in a storage directory is not cleared, the storage ID from it will not match with storage ID of in other storage storage directories. In this situation, the DataNode will quit and restart fails. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4465) Optimize datanode ReplicasMap and ReplicaInfo
Suresh Srinivas created HDFS-4465: - Summary: Optimize datanode ReplicasMap and ReplicaInfo Key: HDFS-4465 URL: https://issues.apache.org/jira/browse/HDFS-4465 Project: Hadoop HDFS Issue Type: Improvement Components: datanode Reporter: Suresh Srinivas Assignee: Suresh Srinivas In Hadoop a lot of optimization has been done in namenode data structures to be memory efficient. Similar optimizations are necessary for Datanode process. With the growth in storage per datanode and number of blocks hosted on datanode, this jira intends to optimize long lived ReplicasMap and ReplicaInfo objects. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4131) Add capability to namenode to get snapshot diff
[ https://issues.apache.org/jira/browse/HDFS-4131?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HDFS-4131. --- Resolution: Fixed Fix Version/s: Snapshot (HDFS-2802) Hadoop Flags: Reviewed I committed the change to the branch. Thank you Jing! Add capability to namenode to get snapshot diff --- Key: HDFS-4131 URL: https://issues.apache.org/jira/browse/HDFS-4131 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, namenode Affects Versions: Snapshot (HDFS-2802) Reporter: Suresh Srinivas Assignee: Jing Zhao Fix For: Snapshot (HDFS-2802) Attachments: HDFS-4131.001.patch, HDFS-4131.002.patch, HDFS-4131.003.patch, HDFS-4131.004.patch, HDFS-4131.005.patch, HDFS-4131.006.patch This jira provides internal data structures and computation processes for calculating and representing the diff between two snapshots, or the diff between a snapshot and the current tree. Specifically, a new method getSnapshotDiffReport(Path, String, String) is added to FSNamesystem to compute the snapshot diff, which is then later represented as a SnapshotDiffReport. In later jiras we will add support to represent the SnapshotDiffReport to end users. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4432) Support INodeFileUnderConstructionWithSnapshot in FSImage saving/loading
[ https://issues.apache.org/jira/browse/HDFS-4432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HDFS-4432. --- Resolution: Fixed Fix Version/s: Snapshot (HDFS-2802) Hadoop Flags: Reviewed I committed the patch. Thank you Jing! Support INodeFileUnderConstructionWithSnapshot in FSImage saving/loading Key: HDFS-4432 URL: https://issues.apache.org/jira/browse/HDFS-4432 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, namenode Reporter: Jing Zhao Assignee: Jing Zhao Fix For: Snapshot (HDFS-2802) Attachments: HDFS-4432.001.patch, HDFS-4432.002.patch, HDFS-4432.003.patch 1. FSImage saver/loader need to be able to recognize INodeFileUnderConstruction, INodeFileUnderConstructionWithSnapshot, and INodeFileUnderConstructionSnapshot. 2. FSImage saver/loader should be able to form the correct circular linked list for INodeFileUnderConstruction(With)Snapshot. 3. Add new corresponding unit tests and support file appending in TestSnapshot#testSnapshot. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-4126) Add reading/writing snapshot information to FSImage
[ https://issues.apache.org/jira/browse/HDFS-4126?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Suresh Srinivas resolved HDFS-4126. --- Resolution: Fixed Fix Version/s: Snapshot (HDFS-2802) Hadoop Flags: Reviewed I committed the patch to HDFS-2802 branch. Thank you Jing! Add reading/writing snapshot information to FSImage --- Key: HDFS-4126 URL: https://issues.apache.org/jira/browse/HDFS-4126 Project: Hadoop HDFS Issue Type: Sub-task Components: datanode, namenode Affects Versions: Snapshot (HDFS-2802) Reporter: Suresh Srinivas Assignee: Jing Zhao Fix For: Snapshot (HDFS-2802) Attachments: HDFS-4126.001.patch, HDFS-4126.002.patch, HDFS-4126.002.patch, HDFS-4126.003.patch After the changes proposed in HDFS-4125 is completed, reading and writing snapshot related information from FSImage can be implemented. This jira tracks changes required for: # Loading snapshot information from FSImage # Loading snapshot related operations from editlog # Writing snapshot information in FSImage # Unit tests related to this functionality -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4375) Use token request messages defined in hadoop common
Suresh Srinivas created HDFS-4375: - Summary: Use token request messages defined in hadoop common Key: HDFS-4375 URL: https://issues.apache.org/jira/browse/HDFS-4375 Project: Hadoop HDFS Issue Type: Improvement Components: namenode, security Affects Versions: 2.0.2-alpha Reporter: Suresh Srinivas HDFS changes related to HADOOP-9192 to reuse the protobuf messages defined in common. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4367) GetDataEncryptionKeyResponseProto does not handle null response
Suresh Srinivas created HDFS-4367: - Summary: GetDataEncryptionKeyResponseProto does not handle null response Key: HDFS-4367 URL: https://issues.apache.org/jira/browse/HDFS-4367 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.0.2-alpha Reporter: Suresh Srinivas Assignee: Suresh Srinivas Priority: Blocker GetDataEncryptionKeyResponseProto member dataEncryptionKey should be optional to handle null response. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4369) GetBlockKeysResponseProto does not handle null response
Suresh Srinivas created HDFS-4369: - Summary: GetBlockKeysResponseProto does not handle null response Key: HDFS-4369 URL: https://issues.apache.org/jira/browse/HDFS-4369 Project: Hadoop HDFS Issue Type: Bug Components: namenode Affects Versions: 2.0.2-alpha Reporter: Suresh Srinivas Assignee: Suresh Srinivas Priority: Blocker GetBlockKeysResponseProto#keys should be optional to handle null response -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4371) Move token related request/response messages to common
Suresh Srinivas created HDFS-4371: - Summary: Move token related request/response messages to common Key: HDFS-4371 URL: https://issues.apache.org/jira/browse/HDFS-4371 Project: Hadoop HDFS Issue Type: Improvement Components: security Affects Versions: 2.0.2-alpha Reporter: Suresh Srinivas Assignee: Suresh Srinivas Get, Renew and Cancel delegation token requests and responses are repeated in HDFS, Yarn and MR. This jira proposes to move these messages into Security.proto in common. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (HDFS-4364) GetLinkTargetResponseProto does not handle null path
Suresh Srinivas created HDFS-4364: - Summary: GetLinkTargetResponseProto does not handle null path Key: HDFS-4364 URL: https://issues.apache.org/jira/browse/HDFS-4364 Project: Hadoop HDFS Issue Type: Bug Affects Versions: 2.0.2-alpha Reporter: Suresh Srinivas Assignee: Suresh Srinivas ClientProtocol#getLinkTarget() returns null targetPath. Hence protobuf definition GetLinkTargetResponseProto#targetPath should be optional. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
