Protocol Buffers version
The current version of Protocol Buffers is 2.6.1 but the current version required by Hadoop is 2.5.0. Is there any reason for this, or should I log a JIRA to get it updated? -- Alan Burlison --
[jira] [Resolved] (HADOOP-11966) Variable cygwin is undefined in hadoop-config.sh when executed through hadoop-daemon.sh.
[
https://issues.apache.org/jira/browse/HADOOP-11966?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth resolved HADOOP-11966.
Resolution: Fixed
Variable cygwin is undefined in hadoop-config.sh when executed through
hadoop-daemon.sh.
Key: HADOOP-11966
URL: https://issues.apache.org/jira/browse/HADOOP-11966
Project: Hadoop Common
Issue Type: Bug
Components: scripts
Affects Versions: 2.7.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Priority: Critical
Fix For: 2.7.1
Attachments: HADOOP-11966-branch-2.001.patch
HADOOP-11464 reinstated support for running the bash scripts through Cygwin.
The logic involves setting a {{cygwin}} flag variable to indicate if the
script is executing through Cygwin. The flag is set in all of the
interactive scripts: {{hadoop}}, {{hdfs}}, {{yarn}} and {{mapred}}. The flag
is not set through hadoop-daemon.sh though. This can cause an erroneous
overwrite of {{HADOOP_HOME}} and {{JAVA_LIBRARY_PATH}} inside
hadoop-config.sh.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: Protocol Buffers version
On May 13, 2015, at 5:02 AM, Alan Burlison alan.burli...@oracle.com wrote: The current version of Protocol Buffers is 2.6.1 but the current version required by Hadoop is 2.5.0. Is there any reason for this, or should I log a JIRA to get it updated? The story of protocol buffers is part of a shameful past where Hadoop trusted Google. This was a terrible mistake, based upon the last time the project upgraded. 2.4-2.5 required some source level, non-backward compatible, and completely-avoidable-but-G-made-us-do-it-anyway surgery to make work. This also ended up being a flag day for every single developer who not only worked with Hadoop but all of the downstream projects as well. Big disaster. The fact that when Google shut down Google Code, they didn't even tag previous releases in the github source tree without significant amount of pressure from the open source community was just adding insult to injury. As a result, I believe the collective opinion is to just flat out avoid adding any more Google bits into the system. See also: guava, which suffers from the same shortsightedness. At some point, we'll either upgrade, switch to a different protocol serialization format, or fork protobuf.
Re: HADOOP-9279 still not fixed, needs reopening
Last time I checked, this was particular to mvn compile and would not occur for other kinds of builds from the root of the source tree like mvn test or mvn package. This is a Maven behavior that is unavoidable unless we pull hadoop-maven-plugins out into its own project separate from the Hadoop source tree, which is something the community already decided against. We can only document the behavior. Please feel free to file a new issue if you'd like to see this clarified in BUILDING.txt. We'd welcome a patch if you want to provide one. Thanks! --Chris Nauroth On 5/13/15, 4:14 AM, Alan Burlison alan.burli...@oracle.com wrote: https://issues.apache.org/jira/browse/HADOOP-9279 Document the need to build hadoop-maven-plugins for eclipse and separate project builds Although the BUILDING.txt mentions the need to to build hadoop-maven-plugins for Eclipse, it's actually needed for *all* builds, even from the top level, otherwise a top-level build will fail with: [ERROR] Failed to parse plugin descriptor for org.apache.hadoop:hadoop-maven-plugins:3.0.0-SNAPSHOT (/pool1/home/alanbur/bleaklow/bigdata/hadoop/hadoop-maven-plugins/target/c lasses): No plugin descriptor found at META-INF/maven/plugin.xml - [Help 1] Manually building hadoop-maven-plugins fixes the problem, but if I'm reading HADOOP-9279 right it shouldn't be required. What's the correct course of action here, reopen HADOOP-9279 or log a new bug? If it's the former I can't see any way of reopening it in JIRA, presumably because I'm not blessed enough to do so. Thanks, -- Alan Burlison --
Re: Protocol Buffers version
Some additional details... A few years ago, we moved from Protocol Buffers 2.4.1 to 2.5.0. There were some challenges with that upgrade, because 2.5.0 was not backwards-compatible with 2.4.1. We needed to coordinate carefully with projects downstream of Hadoop that receive our protobuf classes through transitive dependency. Here are a few issues with more background: https://issues.apache.org/jira/browse/HADOOP-9845 https://issues.apache.org/jira/browse/HBASE-8165 https://issues.apache.org/jira/browse/HIVE-5112 It was important to complete this upgrade before Hadoop 2.x came out of beta. After that, we committed to a policy of backwards-compatibility within the 2.x release line. I can't find a statement about whether or not Protocol Buffers 2.6.1 is backwards-compatible with 2.5.0 (both at compile time and on the wire). Do you know the answer? If it's backwards-incompatible, then we wouldn't be able to do this upgrade within Hadoop 2.x, though we could consider it for 3.x (trunk). In general, we upgrade dependencies when a new release offers a compelling benefit, not solely to keep up with the latest. In the case of 2.5.0, there was a performance benefit. Looking at the release notes for 2.6.0 and 2.6.1, I don't see anything particularly compelling. (That's just my opinion though, and others might disagree.) https://github.com/google/protobuf/blob/master/CHANGES.txt BTW, if anyone is curious, it's possible to try a custom build right now linked against 2.6.1. You'd pass -Dprotobuf.version=2.6.1 and -Dprotoc.path=path to protoc 2.6.1 binary when you run the mvn command. --Chris Nauroth On 5/13/15, 8:59 AM, Allen Wittenauer a...@altiscale.com wrote: On May 13, 2015, at 5:02 AM, Alan Burlison alan.burli...@oracle.com wrote: The current version of Protocol Buffers is 2.6.1 but the current version required by Hadoop is 2.5.0. Is there any reason for this, or should I log a JIRA to get it updated? The story of protocol buffers is part of a shameful past where Hadoop trusted Google. This was a terrible mistake, based upon the last time the project upgraded. 2.4-2.5 required some source level, non-backward compatible, and completely-avoidable-but-G-made-us-do-it-anyway surgery to make work. This also ended up being a flag day for every single developer who not only worked with Hadoop but all of the downstream projects as well. Big disaster. The fact that when Google shut down Google Code, they didn't even tag previous releases in the github source tree without significant amount of pressure from the open source community was just adding insult to injury. As a result, I believe the collective opinion is to just flat out avoid adding any more Google bits into the system. See also: guava, which suffers from the same shortsightedness. At some point, we'll either upgrade, switch to a different protocol serialization format, or fork protobuf.
[jira] [Resolved] (HADOOP-11908) Erasure coding: Should be able to encode part of parity blocks.
[
https://issues.apache.org/jira/browse/HADOOP-11908?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Yi Liu resolved HADOOP-11908.
-
Resolution: Duplicate
Erasure coding: Should be able to encode part of parity blocks.
---
Key: HADOOP-11908
URL: https://issues.apache.org/jira/browse/HADOOP-11908
Project: Hadoop Common
Issue Type: Sub-task
Components: io
Reporter: Yi Liu
Assignee: Kai Zheng
{code}
public void encode(ByteBuffer[] inputs, ByteBuffer[] outputs);
{code}
Currently when we do encode, the outputs are all parity blocks, we should be
able to encode part of parity blocks.
This is required when we do datanode striped block recovery, if one or more
parity blocks are missed, we need to do encode to recovery them. Only encode
part of parity blocks should be more efficient than all.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: HADOOP-9279 still not fixed, needs reopening
On 13/05/2015 17:38, Chris Nauroth wrote: I usually tell people starting fresh to do mvn clean install -Pnative -DskipTests as a first step. That's a good validation of your build environment, and it will put a SNAPSHOT version of each sub-module into your local Maven repository. After that, you can do smaller incremental builds by cd'ing into hadoop-hdfs or hadoop-yarn-server-nodemanager or anything else and running things like mvn test for just that sub-module. Thanks, I've logged https://issues.apache.org/jira/browse/HADOOP-11968 BUILDING.txt is still unclear about the need to build hadoop-maven-plugins first And assigned it to myself. -- Alan Burlison --
[jira] [Created] (HADOOP-11968) BUILDING.txt is still unclear about the need to build
Alan Burlison created HADOOP-11968: -- Summary: BUILDING.txt is still unclear about the need to build Key: HADOOP-11968 URL: https://issues.apache.org/jira/browse/HADOOP-11968 Project: Hadoop Common Issue Type: Improvement Components: build Affects Versions: 2.7.0 Environment: All Reporter: Alan Burlison Assignee: Alan Burlison Priority: Minor HADOOP-9279 attempted to address the issue of having to make sure that hadoop-maven-plugins is built first by modifying BUILDING.txt but it still isn't clear this is a requirement. The compile target doesn't do this, so if the first build is a mvn compile it will fail. BUILDING.txt should be modified to recommend either that an mvn compile is done first in hadoop-maven-plugins or that a mvn clean install -Pnative -DskipTests is done from the top-level. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: Planning Hadoop 2.6.1 release
Hi Akira, Can we also include YARN-3242? YARN-3242 fixed a critical ZKRMStateStore bug. It will work better with YARN-2992. thanks zhihai On Tue, May 12, 2015 at 10:38 PM, Akira AJISAKA ajisa...@oss.nttdata.co.jp wrote: Thanks all for collecting jiras for 2.6.1 release. In addition, I'd like to include the following: * HADOOP-11343. Overflow is not properly handled in calculating final iv for AES CTR * YARN-2874. Dead lock in DelegationTokenRenewer which blocks RM to execute any further apps * YARN-2992. ZKRMStateStore crashes due to session expiry * YARN-3013. AMRMClientImpl does not update AMRM token properly * YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to die * MAPREDUCE-6303. Read timeout when retrying a fetch error can be fatal to a reducer All of these are marked as blocker bug for 2.7.0 but not fixed in 2.6.0. Regards, Akira On 5/4/15 11:15, Brahma Reddy Battula wrote: Hello Vinod, I am thinking,can we include HADOOP-11491 also..? wihout this jira harfs will not be usable when cluster installed in HA mode and try to get filecontext like below.. Path path = new Path(har:///archivedLogs/application_1428917727658_0005-application_1428917727658_0008-1428927448352.har); FileSystem fs = path.getFileSystem(new Configuration()); path = fs.makeQualified(path); FileContext fc = FileContext.getFileContext(path.toUri(),new Configuration()); Thanks Regards Brahma Reddy Battula From: Chris Nauroth [cnaur...@hortonworks.com] Sent: Friday, May 01, 2015 4:32 AM To: mapreduce-...@hadoop.apache.org; common-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; hdfs-...@hadoop.apache.org Subject: Re: Planning Hadoop 2.6.1 release Thank you, Arpit. In addition, I suggest we include the following: HADOOP-11333. Fix deadlock in DomainSocketWatcher when the notification pipe is full HADOOP-11604. Prevent ConcurrentModificationException while closing domain sockets during shutdown of DomainSocketWatcher thread. HADOOP-11648. Set DomainSocketWatcher thread name explicitly HADOOP-11802. DomainSocketWatcher thread terminates sometimes after there is an I/O error during requestShortCircuitShm HADOOP-11604 and 11648 are not critical by themselves, but they are pre-requisites to getting a clean cherry-pick of 11802, which we believe finally fixes the root cause of this issue. --Chris Nauroth On 4/30/15, 3:55 PM, Arpit Agarwal aagar...@hortonworks.com wrote: HDFS candidates for back-porting to Hadoop 2.6.1. The first two were requested in [1]. HADOOP-11674. oneByteBuf in CryptoInputStream and CryptoOutputStream should be non static HADOOP-11710. Make CryptoOutputStream behave like DFSOutputStream wrt synchronization HDFS-7009. Active NN and standby NN have different live nodes. HDFS-7035. Make adding a new data directory to the DataNode an atomic and improve error handling HDFS-7425. NameNode block deletion logging uses incorrect appender. HDFS-7443. Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate block files are present in the same volume. HDFS-7489. Incorrect locking in FsVolumeList#checkDirs can hang datanodes HDFS-7503. Namenode restart after large deletions can cause slow processReport. HDFS-7575. Upgrade should generate a unique storage ID for each volume. HDFS-7579. Improve log reporting during block report rpc failure. HDFS-7587. Edit log corruption can happen if append fails with a quota violation. HDFS-7596. NameNode should prune dead storages from storageMap. HDFS-7611. deleteSnapshot and delete of a file can leave orphaned blocks in the blocksMap on NameNode restart. HDFS-7714. Simultaneous restart of HA NameNodes and DataNode can cause DataNode to register successfully with only one NameNode. HDFS-7733. NFS: readdir/readdirplus return null directory attribute on failure. HDFS-7831. Fix the starting index and end condition of the loop in FileDiffList.findEarlierSnapshotBlocks(). HDFS-7885. Datanode should not trust the generation stamp provided by client. HDFS-7960. The full block report should prune zombie storages even if they're not empty. HDFS-8072. Reserved RBW space is not released if client terminates while writing block. HDFS-8127. NameNode Failover during HA upgrade can cause DataNode to finalize upgrade. Arpit [1] Will Hadoop 2.6.1 be released soon? http://markmail.org/thread/zlsr6prejyogdyvh On 4/27/15, 11:47 AM, Vinod Kumar Vavilapalli vino...@apache.org wrote: There were several requests on the user lists [1] for a 2.6.1 release. I got many offline comments too. Planning to do a 2.6.1 release in a few weeks time. We already have a bunch of tickets committed to 2.7.1. I created a filter [2] to tracking pending tickets. We need to collectively come up with a list of critical issues. We can use the JIRA Target Version field for the same. I see some but not a whole lot of new work for this
[jira] [Reopened] (HADOOP-11566) Add tests and fix for erasure coders to recover erased parity units
[ https://issues.apache.org/jira/browse/HADOOP-11566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinayakumar B reopened HADOOP-11566: Add tests and fix for erasure coders to recover erased parity units Key: HADOOP-11566 URL: https://issues.apache.org/jira/browse/HADOOP-11566 Project: Hadoop Common Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Fix For: HDFS-7285 Attachments: HADOOP-11566-HDFS-7285-v2.patch, HADOOP-11566-HDFS-7285-v2.patch, HADOOP-11566-v1.patch Discussing with [~zhz] in HADOOP-11542: it's planned to have follow up a JIRA to enhance the tests for parity chunks as well. Like erasedDataIndexes, erasedParityIndexes will be added to specify which parity units are to be erased and recovered then. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HADOOP-11566) Add tests and fix for erasure coders to recover erased parity units
[ https://issues.apache.org/jira/browse/HADOOP-11566?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Vinayakumar B resolved HADOOP-11566. Resolution: Fixed Fix Version/s: HDFS-7285 Resolving as FIXED Add tests and fix for erasure coders to recover erased parity units Key: HADOOP-11566 URL: https://issues.apache.org/jira/browse/HADOOP-11566 Project: Hadoop Common Issue Type: Sub-task Reporter: Kai Zheng Assignee: Kai Zheng Fix For: HDFS-7285 Attachments: HADOOP-11566-HDFS-7285-v2.patch, HADOOP-11566-HDFS-7285-v2.patch, HADOOP-11566-v1.patch Discussing with [~zhz] in HADOOP-11542: it's planned to have follow up a JIRA to enhance the tests for parity chunks as well. Like erasedDataIndexes, erasedParityIndexes will be added to specify which parity units are to be erased and recovered then. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
