Jenkins build is back to normal : Hadoop-Hdfs-trunk #1698
See https://builds.apache.org/job/Hadoop-Hdfs-trunk/1698/changes
Re: Thinking ahead to 2.4
Sorry, the previous link had a bug, the correct one is: http://s.apache.org/hadoop-2.4.0-blockers. We are currently down to 12 blockers; with several PA. thanks, Arun On Mar 6, 2014, at 1:40 PM, Arun C Murthy a...@hortonworks.com wrote: Gang, Most of the big-ticket items are already in, awesome! I'm thinking we could roll out a 2.4 RC in the next 2-3 weeks after we get through the list of blockers. Here is a handy link: http://s.apache.org/hadoop-2.4-blockers If you find more, please set Target Version to 2.4.0 and mark it a blocker. I'll try nudging people to start closing these soon, appreciate any help! thanks, Arun On Feb 20, 2014, at 3:45 PM, Arun C Murthy a...@hortonworks.com wrote: Thanks Azuryy Suresh. I've updated the roadmap wiki to reflect this. Arun On Feb 20, 2014, at 2:01 PM, Suresh Srinivas sur...@hortonworks.com wrote: Arun, Some of the previously 2.4 targeted features were made available in 2.3: - Heterogeneous storage support - Datanode cache The following are being targeted for 2.4: - Use protobuf for fsimge (already in) - ACLs (in trunk. In a week or so, this will be merged to branch-2.4) - Rolling upgrades (last bunch of jiras being worked in feature branch. Will be in 2.4 in around two weeks. Currently testing is in progress) So HDFS features should be ready in two weeks. On Sat, Feb 15, 2014 at 4:47 PM, Azuryy azury...@gmail.com wrote: Hi， I think you omit some key pieces in 2.4 Protobuf fsimage, rolling upgrade are also targeting 2.4 Sent from my iPhone5s On 2014年2月16日, at 6:59, Arun C Murthy a...@hortonworks.com wrote: Folks, With hadoop-2.3 nearly done, I think it's time to think ahead to hadoop-2.4. I think it was a good idea to expedite release of 2.3 while we finished up pieces that didn't make it in such as HDFS Caching Support for Heterogenous Storage. Now, most of the key pieces incl. Resource Manager Automatic Failover (YARN-149), Application History Server (YARN-321) Application Timeline Server (YARN-1530) are either complete or very close to done, and I think we will benefit with an extended test-cycle for 2.4 - similar to what happened with 2.2. To provide some context: 2.2 went through nearly 6 weeks of extended testing and it really helped us push out a very stable release. I think it will be good to create a 2.4 branch ASAP and start testing. As such, I plan to cut the branch early next week. With this, we should be good shape sometime to release 2.4 in mid-March. I've updated https://wiki.apache.org/hadoop/Roadmap to reflect this. Also, we should start thinking ahead to 2.5 and what folks would like to see in it. If we continue our 6-week cycles, we could shoot to get that out in April. Thoughts? thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- http://hortonworks.com/download/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[jira] [Created] (HDFS-6087) Unify HDFS write/append/truncate
Guo Ruijing created HDFS-6087: - Summary: Unify HDFS write/append/truncate Key: HDFS-6087 URL: https://issues.apache.org/jira/browse/HDFS-6087 Project: Hadoop HDFS Issue Type: Improvement Components: hdfs-client Reporter: Guo Ruijing Attachments: HDFS Design Proposal.pdf In existing implementation, HDFS file can be appended and HDFS block can be reopened for append. This design will introduce complexity including lease recovery. If we design HDFS block as immutable, it will be very simple for append truncate. The idea is that HDFS block is immutable if the block is committed to namenode. If the block is not committed to namenode, it is HDFS client’s responsibility to re-added with new block ID. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6088) Add configurable maximum block count for datanode
Kihwal Lee created HDFS-6088: Summary: Add configurable maximum block count for datanode Key: HDFS-6088 URL: https://issues.apache.org/jira/browse/HDFS-6088 Project: Hadoop HDFS Issue Type: Bug Reporter: Kihwal Lee Currently datanode resources are protected by the free space check and the balancer. But datanodes can run out of memory simply storing too many blocks. If the sizes of blocks are small, datanodes will appear to have plenty of space to put more blocks. I propose adding a configurable max block count to datanode. Since datanodes can have different heap configurations, it will make sense to make it datanode-level, rather than something enforced by namenode. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6089) Standby NN while transitioning to active throws a connection refused error when the prior active NN process is suspended
Arpit Gupta created HDFS-6089: - Summary: Standby NN while transitioning to active throws a connection refused error when the prior active NN process is suspended Key: HDFS-6089 URL: https://issues.apache.org/jira/browse/HDFS-6089 Project: Hadoop HDFS Issue Type: Bug Components: ha Affects Versions: 2.4.0 Reporter: Arpit Gupta Assignee: Jing Zhao The following scenario was tested: * Determine Active NN and suspend the process (kill -19) * Wait about 60s to let the standby transition to active * Get the service state for nn1 and nn2 and make sure nn2 has transitioned to active. What was noticed that some times the call to get the service state of nn2 got a socket time out connection. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6090) Use MiniDFSCluster.Builder instead of deprecated constructors
Akira AJISAKA created HDFS-6090:
---
Summary: Use MiniDFSCluster.Builder instead of deprecated
constructors
Key: HDFS-6090
URL: https://issues.apache.org/jira/browse/HDFS-6090
Project: Hadoop HDFS
Issue Type: Improvement
Components: test
Reporter: Akira AJISAKA
Priority: Minor
Some test classes are using deprecated constructors such as
{{MiniDFSCluster(Configuration, int, boolean, String[], String[])}} for
building a MiniDFSCluster.
These classes should use {{MiniDFSCluster.Builder}} to reduce javac warnings
and improve code readability.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HDFS-6091) dfs.journalnode.edits.dir should accept URI
Allen Wittenauer created HDFS-6091: -- Summary: dfs.journalnode.edits.dir should accept URI Key: HDFS-6091 URL: https://issues.apache.org/jira/browse/HDFS-6091 Project: Hadoop HDFS Issue Type: Bug Components: journal-node Affects Versions: 2.2.0 Reporter: Allen Wittenauer Priority: Minor Using a URI in dfs.journalnode.edits.dir (such as file:///foo) throws a Journal dir 'file:/foo' should be an absolute path'. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6092) DistributedFileSystem#getCanonicalServiceName() and DistributedFileSystem#getUri() may return inconsistent results w.r.t. port
Ted Yu created HDFS-6092:
Summary: DistributedFileSystem#getCanonicalServiceName() and
DistributedFileSystem#getUri() may return inconsistent results w.r.t. port
Key: HDFS-6092
URL: https://issues.apache.org/jira/browse/HDFS-6092
Project: Hadoop HDFS
Issue Type: Bug
Affects Versions: 2.3.0
Reporter: Ted Yu
I discovered this when working on HBASE-10717
Here is sample code to reproduce the problem:
{code}
Path desPath = new Path(hdfs://127.0.0.1/);
FileSystem desFs = desPath.getFileSystem(conf);
String s = desFs.getCanonicalServiceName();
URI uri = desFs.getUri();
{code}
Canonical name string contains the default port - 8020
But uri doesn't contain port.
This would result in the following exception:
{code}
testIsSameHdfs(org.apache.hadoop.hbase.util.TestFSHDFSUtils) Time elapsed:
0.001 sec ERROR!
java.lang.IllegalArgumentException: port out of range:-1
at java.net.InetSocketAddress.checkPort(InetSocketAddress.java:143)
at java.net.InetSocketAddress.init(InetSocketAddress.java:224)
at
org.apache.hadoop.hbase.util.FSHDFSUtils.getNNAddresses(FSHDFSUtils.java:88)
{code}
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Created] (HDFS-6093) Expose more caching information for debugging by users
Andrew Wang created HDFS-6093: - Summary: Expose more caching information for debugging by users Key: HDFS-6093 URL: https://issues.apache.org/jira/browse/HDFS-6093 Project: Hadoop HDFS Issue Type: Improvement Components: caching Affects Versions: 2.4.0 Reporter: Andrew Wang Assignee: Andrew Wang When users submit a new cache directive, it's unclear if the NN has recognized it and is actively trying to cache it, or if it's hung for some other reason. It'd be nice to expose a pending caching/uncaching count the same way we expose pending replication work. It'd also be nice to display the aggregate cache capacity and usage in dfsadmin -report, since we already have have it as a metric and expose it per-DN in report output. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Created] (HDFS-6094) The same block can be counted twice towards safe mode threshold
Arpit Agarwal created HDFS-6094:
---
Summary: The same block can be counted twice towards safe mode
threshold
Key: HDFS-6094
URL: https://issues.apache.org/jira/browse/HDFS-6094
Project: Hadoop HDFS
Issue Type: Bug
Components: namenode
Affects Versions: 2.4.0
Reporter: Arpit Agarwal
Assignee: Arpit Agarwal
{{BlockManager#addStoredBlock}} can cause the same block can be counted towards
safe mode threshold. We see this manifest via
{{TestHASafeMode#testBlocksAddedWhileStandbyIsDown}} failures on Ubuntu. More
details in a comment.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
