Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions
To close the loop on this one, I created a label for the candidate list of patches: https://issues.apache.org/jira/issues/?jql=labels%20%3D%202.6.1-candidatehttps://issues.apache.org/jira/issues/?jql=labels%20=%202.6.1-candidate, in order to make progress while the issue is still hot. The discussion is continuing on the 2.6.1 release planning thread. Thanks +Vinod On Jul 15, 2015, at 6:09 PM, Andrew Purtell apurt...@apache.orgmailto:apurt...@apache.org wrote: Inline On Wed, Jul 15, 2015 at 5:22 PM, Vinod Kumar Vavilapalli vino...@hortonworks.commailto:vino...@hortonworks.com wrote: I can understand these (sort of newish) questions from hbase-dev. We already have a well laid-out release-management process. If people want to learn more about how it works, please head over to http://hadoop.apache.org/bylaws.html. In terms of 2.6.1 release management, it wasn’t stuck for lack of volunteers for RM. I originally was driving it [1], and then Akira recently volunteered too in a separate thread [2] (which I totally missed participating in before), so we have enough help. Right, but in that thread it looks like you were stymied sorting through the potential set of updates to apply to the patch release. That was the motivation behind my sharing a bit about HBase branch RMs. Totally fine if that's not something Hadoop wants to explore. I only offered it as a data point where another project avoids that problem with a different kind of RM focus. And, so, that's that FWIW It is clearly acknowledged there is a demand for 2.6.1, we now have to figure out the mechanics, agreeing on a reasonable common list of fixes to start with. Thank you very much for this consideration! Tx for the feedback so far @hbase-dev! I agree with Karthik earlier - let’s continue the discussion in hadoop-dev in the release thread [1] for 2.6.1. Thanks +Vinod [1] Planning Hadoop 2.6.1 release http://markmail.org/message/sbykjn5xgnksh6wg [2] [DISCUSS] More Maintenance Releases http://s.apache.org/MMR On Jul 15, 2015, at 4:07 PM, Stack st...@duboce.netmailto: st...@duboce.net wrote: Is there anyone interested in volunteering to run a 2.6.1 release (Akira?)? You'd get some help (especially if the bar is set high and only critical bug fixes are allowed in: i.e. no features, no 'perf' fixes, no jar updates, and so on). St.Ack On Wed, Jul 15, 2015 at 3:07 PM, Chris Douglas cdoug...@apache.org mailto:cdoug...@apache.org wrote: On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey bus...@cloudera.commailto: bus...@cloudera.com wrote: Alternatively, why not appoint a Release Manager for the minor release line and then allow them to arbitrate when there's disagreement about inclusion? This has worked well in the HBase community. Release managers aren't appointed in Hadoop. Any committer can RM a release branch and encourage others to help with it. An RM can set the bar arbitrarily, but an RC only becomes a release when a majority of PMC votes approve it in a VOTE. -C On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey bus...@cloudera.commailto: bus...@cloudera.com wrote: Why not just include all backwards compatible bug fixes? Alternatively, why not appoint a Release Manager for the minor release line and then allow them to arbitrate when there's disagreement about inclusion? This has worked well in the HBase community. On Wed, Jul 15, 2015 at 3:49 PM, Karthik Kambatla ka...@cloudera.com mailto:ka...@cloudera.com wrote: As I proposed in the other thread, how about we adopting the following model: x.y.1 releases have all Blocker, Critical, Major bug fixes applied to the next minor release. x.y.2 releases have all Blocker, Critical bug fixes applied to the next minor release. x.y.3 releases have all Blocker bug fixes applied to next minor release. Here I am assuming there are no security-fix-only or other urgent releases. We could apply this approach for 2.7.x onwards, and do an adhoc 2.6 release. On Wed, Jul 15, 2015 at 12:59 PM, Vinod Kumar Vavilapalli vino...@hortonworks.commailto:vino...@hortonworks.com wrote: Yeah, I started a thread while back on this one ( http://markmail.org/message/sbykjn5xgnksh6wg) and had many offline discussions re 2.6.1. The biggest problem I found offline was about what bug-fixes are acceptable and what aren’t for everyone wishing to consume 2.6.1. Given the number of bug-fixes that went into 2.7.x and into branch-2.8, figuring out a set of patches that is acceptable for everyone is a huge challenge which kind of stalled my attempts. Thanks +Vinod On Jul 15, 2015, at 8:57 AM, Sangjin Lee sjl...@gmail.commailto: sjl...@gmail.com wrote: Strong +1 for having a 2.6.1 release. I understand Vinod has been trying to get that effort going but it's been stalled a little bit. It would be good to rekindle that effort. Companies with big hadoop 2.x deployments (including mine) have always tried to stabilize a 2.x release by testing/collecting/researching critical issues on the release. Each would come up
[jira] [Created] (HADOOP-12240) Fix tests requiring native library to be skipped in non-native profile
Masatake Iwasaki created HADOOP-12240: - Summary: Fix tests requiring native library to be skipped in non-native profile Key: HADOOP-12240 URL: https://issues.apache.org/jira/browse/HADOOP-12240 Project: Hadoop Common Issue Type: Bug Components: test Reporter: Masatake Iwasaki Assignee: Masatake Iwasaki Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions
Inline On Wed, Jul 15, 2015 at 5:22 PM, Vinod Kumar Vavilapalli vino...@hortonworks.com wrote: I can understand these (sort of newish) questions from hbase-dev. We already have a well laid-out release-management process. If people want to learn more about how it works, please head over to http://hadoop.apache.org/bylaws.html. In terms of 2.6.1 release management, it wasn’t stuck for lack of volunteers for RM. I originally was driving it [1], and then Akira recently volunteered too in a separate thread [2] (which I totally missed participating in before), so we have enough help. Right, but in that thread it looks like you were stymied sorting through the potential set of updates to apply to the patch release. That was the motivation behind my sharing a bit about HBase branch RMs. Totally fine if that's not something Hadoop wants to explore. I only offered it as a data point where another project avoids that problem with a different kind of RM focus. And, so, that's that FWIW It is clearly acknowledged there is a demand for 2.6.1, we now have to figure out the mechanics, agreeing on a reasonable common list of fixes to start with. Thank you very much for this consideration! Tx for the feedback so far @hbase-dev! I agree with Karthik earlier - let’s continue the discussion in hadoop-dev in the release thread [1] for 2.6.1. Thanks +Vinod [1] Planning Hadoop 2.6.1 release http://markmail.org/message/sbykjn5xgnksh6wg [2] [DISCUSS] More Maintenance Releases http://s.apache.org/MMR On Jul 15, 2015, at 4:07 PM, Stack st...@duboce.netmailto: st...@duboce.net wrote: Is there anyone interested in volunteering to run a 2.6.1 release (Akira?)? You'd get some help (especially if the bar is set high and only critical bug fixes are allowed in: i.e. no features, no 'perf' fixes, no jar updates, and so on). St.Ack On Wed, Jul 15, 2015 at 3:07 PM, Chris Douglas cdoug...@apache.org mailto:cdoug...@apache.org wrote: On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey bus...@cloudera.commailto: bus...@cloudera.com wrote: Alternatively, why not appoint a Release Manager for the minor release line and then allow them to arbitrate when there's disagreement about inclusion? This has worked well in the HBase community. Release managers aren't appointed in Hadoop. Any committer can RM a release branch and encourage others to help with it. An RM can set the bar arbitrarily, but an RC only becomes a release when a majority of PMC votes approve it in a VOTE. -C On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey bus...@cloudera.commailto: bus...@cloudera.com wrote: Why not just include all backwards compatible bug fixes? Alternatively, why not appoint a Release Manager for the minor release line and then allow them to arbitrate when there's disagreement about inclusion? This has worked well in the HBase community. On Wed, Jul 15, 2015 at 3:49 PM, Karthik Kambatla ka...@cloudera.com mailto:ka...@cloudera.com wrote: As I proposed in the other thread, how about we adopting the following model: x.y.1 releases have all Blocker, Critical, Major bug fixes applied to the next minor release. x.y.2 releases have all Blocker, Critical bug fixes applied to the next minor release. x.y.3 releases have all Blocker bug fixes applied to next minor release. Here I am assuming there are no security-fix-only or other urgent releases. We could apply this approach for 2.7.x onwards, and do an adhoc 2.6 release. On Wed, Jul 15, 2015 at 12:59 PM, Vinod Kumar Vavilapalli vino...@hortonworks.commailto:vino...@hortonworks.com wrote: Yeah, I started a thread while back on this one ( http://markmail.org/message/sbykjn5xgnksh6wg) and had many offline discussions re 2.6.1. The biggest problem I found offline was about what bug-fixes are acceptable and what aren’t for everyone wishing to consume 2.6.1. Given the number of bug-fixes that went into 2.7.x and into branch-2.8, figuring out a set of patches that is acceptable for everyone is a huge challenge which kind of stalled my attempts. Thanks +Vinod On Jul 15, 2015, at 8:57 AM, Sangjin Lee sjl...@gmail.commailto: sjl...@gmail.com wrote: Strong +1 for having a 2.6.1 release. I understand Vinod has been trying to get that effort going but it's been stalled a little bit. It would be good to rekindle that effort. Companies with big hadoop 2.x deployments (including mine) have always tried to stabilize a 2.x release by testing/collecting/researching critical issues on the release. Each would come up with its own set of fixes to backport. We would also communicate it via offline channels. During the hadoop summit, we thought it would be great if we all came together and create a public stability/bugfix release on top of 2.x (2.6.1 for 2.6 for example) with all the critical issues fixed. Thanks, Sangjin On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa oz...@apache.orgmailto:
[jira] [Created] (HADOOP-12235) hadoop-openstack junit mockito dependencies should be provided
Steve Loughran created HADOOP-12235: --- Summary: hadoop-openstack junit mockito dependencies should be provided Key: HADOOP-12235 URL: https://issues.apache.org/jira/browse/HADOOP-12235 Project: Hadoop Common Issue Type: Bug Components: build, fs/swift Affects Versions: 2.6.0 Reporter: Steve Loughran Priority: Minor The scope for JUnit mockito in hadoop-openstack is compile, which means it ends up on the downstream classpath unless excluded. it should be provided, which was the original intent -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (HADOOP-12239) StorageException complaining no lease ID when updating FolderLastModifiedTime in WASB
Duo Xu created HADOOP-12239:
---
Summary: StorageException complaining no lease ID when updating
FolderLastModifiedTime in WASB
Key: HADOOP-12239
URL: https://issues.apache.org/jira/browse/HADOOP-12239
Project: Hadoop Common
Issue Type: Bug
Components: tools
Affects Versions: 2.7.0
Reporter: Duo Xu
Assignee: Duo Xu
This is a similar issue as HADOOP-11523 and HADOOP-12089, which I found in a
customer's HBase cluster logs, but the piece of code is in a different place.
{code}
2015-07-09 13:38:57,388 INFO org.apache.hadoop.hbase.master.SplitLogManager:
dead splitlog workers
[workernode3.skypedataprdhbaeus00.b6.internal.cloudapp.net,60020,1436448555180]
2015-07-09 13:38:57,466 ERROR org.apache.hadoop.hbase.executor.EventHandler:
Caught throwable while processing event M_SERVER_SHUTDOWN
java.io.IOException: failed log splitting for
workernode12.skypedataprdhbaeus00.b6.internal.cloudapp.net,60020,1436448566374,
will retry
at
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.resubmit(ServerShutdownHandler.java:343)
at
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:211)
at
org.apache.hadoop.hbase.executor.EventHandler.run(EventHandler.java:128)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Unable to write RenamePending file for folder
rename from
hbase/WALs/workernode12.skypedataprdhbaeus00.b6.internal.cloudapp.net,60020,1436448566374
to
hbase/WALs/workernode12.skypedataprdhbaeus00.b6.internal.cloudapp.net,60020,1436448566374-splitting
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem$FolderRenamePending.writeFile(NativeAzureFileSystem.java:258)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.prepareAtomicFolderRename(NativeAzureFileSystem.java:2110)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.rename(NativeAzureFileSystem.java:1998)
at
org.apache.hadoop.hbase.master.MasterFileSystem.getLogDirs(MasterFileSystem.java:325)
at
org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:412)
at
org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:390)
at
org.apache.hadoop.hbase.master.MasterFileSystem.splitLog(MasterFileSystem.java:288)
at
org.apache.hadoop.hbase.master.handler.ServerShutdownHandler.process(ServerShutdownHandler.java:204)
... 4 more
Caused by: org.apache.hadoop.fs.azure.AzureException:
com.microsoft.azure.storage.StorageException: There is currently a lease on the
blob and no lease ID was specified in the request.
at
org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2598)
at
org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2609)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.create(NativeAzureFileSystem.java:1366)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem.create(NativeAzureFileSystem.java:1195)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:908)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:889)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:786)
at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:775)
at
org.apache.hadoop.fs.azure.NativeAzureFileSystem$FolderRenamePending.writeFile(NativeAzureFileSystem.java:255)
... 11 more
Caused by: com.microsoft.azure.storage.StorageException: There is currently a
lease on the blob and no lease ID was specified in the request.
at
com.microsoft.azure.storage.StorageException.translateException(StorageException.java:89)
at
com.microsoft.azure.storage.core.StorageRequest.materializeException(StorageRequest.java:307)
at
com.microsoft.azure.storage.core.ExecutionEngine.executeWithRetry(ExecutionEngine.java:182)
at
com.microsoft.azure.storage.blob.CloudBlob.uploadProperties(CloudBlob.java:2892)
at
org.apache.hadoop.fs.azure.StorageInterfaceImpl$CloudBlobWrapperImpl.uploadProperties(StorageInterfaceImpl.java:372)
at
org.apache.hadoop.fs.azure.AzureNativeFileSystemStore.updateFolderLastModifiedTime(AzureNativeFileSystemStore.java:2593)
... 19 more
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Jenkins build is back to normal : Hadoop-Common-trunk #1555
See https://builds.apache.org/job/Hadoop-Common-trunk/1555/changes
2.7.2 release plan
Hi all, Thanks everyone for the push on 2.7.1! Branch-2.7 is now open for commits to a 2.7.2 release. JIRA also now has a 2.7.2 version for all the sub-projects. Continuing the previous 2.7.1 thread on steady maintenance releases [1], we should follow up 2.7.1 with a 2.7.2 within 4 weeks. Earlier I tried a 2-3 week cycle for 2.7.1, but it seems to be impractical given the community size. So, I propose we target a release by the end for 4 weeks from now, starting the release close-down within 2-3 weeks. The focus obviously is to have blocker issues [2], bug-fixes and *no* features / improvements. I need help from all committers in automatically merging in any patch that fits the above criterion into 2.7.2 instead of only on trunk or 2.8. Thoughts? Thanks, +Vinod [1] A 2.7.1 release to follow up 2.7.0 http://markmail.org/message/zwzze6cqqgwq4rmw [2] 2.7.2 release blockers: https://issues.apache.org/jira/issues/?filter=12332867
Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions
Why not just have the discussion here? It seems integral to the matter of having more maintenance releases on those versions. On Wed, Jul 15, 2015 at 11:39 AM, Karthik Kambatla ka...@cloudera.com wrote: I believe there was general consensus to do more maintenance releases, as witnessed in the other thread. There have been discussions on what should go into 2.x.1, 2.x.2, etc., but I don't think we have a clear proposal. It would be nice to put that together, so committers know where all to commit patches to. Otherwise, release managers will have to look through branch-2 and pull in the fixes. Either approach is fine by me, but would be nice to start a [DISCUSS] thread on how to go about this. Any takers? On Wed, Jul 15, 2015 at 8:57 AM, Sangjin Lee sjl...@gmail.com wrote: Strong +1 for having a 2.6.1 release. I understand Vinod has been trying to get that effort going but it's been stalled a little bit. It would be good to rekindle that effort. Companies with big hadoop 2.x deployments (including mine) have always tried to stabilize a 2.x release by testing/collecting/researching critical issues on the release. Each would come up with its own set of fixes to backport. We would also communicate it via offline channels. During the hadoop summit, we thought it would be great if we all came together and create a public stability/bugfix release on top of 2.x (2.6.1 for 2.6 for example) with all the critical issues fixed. Thanks, Sangjin On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa oz...@apache.org wrote: Thank you for the notification. Trying to back port bug fixes. - Tsuyoshi On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey bus...@cloudera.com wrote: Hi Hadoopers! Over in HBase we've been discussing the impact of our dependencies on our downstream users. As our most fundamental dependency, Hadoop plays a big role in the operational cost of running an HBase instance. Currently the HBase 1.y release line supports Hadoop 2.4, 2.5, and 2.6[1]. We don't drop Hadoop minor release lines in minor releases so we are unlikely remove anything from this set until HBase 2.0, probably at the end of 2015 / start of 2016 (and currently we plan to continue supporting at least 2.4 for HBase 2.0 [2]). Lately we've been discussing updating our shipped binaries to Hadoop 2.6, following some stability testing by part of our community[3]. Unfortunately, 2.6.0 in particular has a couple of bugs that could destroy HBase clusters should users decide to turn on HDFS encryption[4]. Our installation instructions tell folks to replace these jars with the version of Hadoop they are actually running, but not all users follow those instructions so we want to minimize the pain for them. Regular maintenance releases are key to keeping operational burdens low for our downstream users; we don't want them to be forced to choose between living with broken systems and stomaching the risk of upgrades across minor/major version numbers. Looking back over the three aforementioned Hadoop versions, 2.6 hasn't had a patch release since 2.6.0 came out in Nov 2014, when 2.5 had its last patch release as well. Hadoop 2.4 looks to be a year without a release[5]. On our discussion of shipping Hadoop 2.6 binaries, one of your PMC members mentioned that with continued work on the 2.7 line y'all weren't planning any additional releases of the earlier minor versions[6]. The HBase community requests that Hadoop pick up making bug-fix-only patch releases again on a regular schedule[7]. Preferably on the 2.6 line and preferably monthly. We realize that given the time gap since 2.6.0 it will likely take a big to get 2.6.1 together, but after that it should take much less effort to continue. [1]: http://hbase.apache.org/book.html#hadoop [2]: http://s.apache.org/ReP [3]: HBASE-13339 [4]: HADOOP-11674 and HADOOP-11710 [5]: http://hadoop.apache.org/releases.html [6]: http://s.apache.org/MTY [7]: http://s.apache.org/ViP -- Sean -- Karthik Kambatla Software Engineer, Cloudera Inc. http://five.sentenc.es -- Sean
Re: Planning Hadoop 2.6.1 release
Got pinged on a recent thread on this one. As I mentioned there, I had many offline discussions re 2.6.1. The biggest problem I found offline was about what bug-fixes are acceptable and what aren’t for everyone wishing to consume 2.6.1. Given the number of bug-fixes that went into 2.7.x and into branch-2.8, figuring out a set of patches that is acceptable for everyone is a huge challenge which kind of stalled my attempts. Thanks +Vinod On Jul 1, 2015, at 12:41 PM, Sean Busbey bus...@cloudera.com wrote: Any update on a release plan for 2.6.1? On Wed, Jun 10, 2015 at 1:25 AM, Brahma Reddy Battula brahmareddy.batt...@huawei.com wrote: HI vinod any update on this..? are we planning to give 2.6.1 Or can we make 2.7.1 as stable give..? Thanks Regards Brahma Reddy Battula From: Zhihai Xu [z...@cloudera.com] Sent: Wednesday, May 13, 2015 12:04 PM To: mapreduce-...@hadoop.apache.org Cc: common-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; hdfs-...@hadoop.apache.org Subject: Re: Planning Hadoop 2.6.1 release Hi Akira, Can we also include YARN-3242? YARN-3242 fixed a critical ZKRMStateStore bug. It will work better with YARN-2992. thanks zhihai On Tue, May 12, 2015 at 10:38 PM, Akira AJISAKA ajisa...@oss.nttdata.co.jp wrote: Thanks all for collecting jiras for 2.6.1 release. In addition, I'd like to include the following: * HADOOP-11343. Overflow is not properly handled in calculating final iv for AES CTR * YARN-2874. Dead lock in DelegationTokenRenewer which blocks RM to execute any further apps * YARN-2992. ZKRMStateStore crashes due to session expiry * YARN-3013. AMRMClientImpl does not update AMRM token properly * YARN-3369. Missing NullPointer check in AppSchedulingInfo causes RM to die * MAPREDUCE-6303. Read timeout when retrying a fetch error can be fatal to a reducer All of these are marked as blocker bug for 2.7.0 but not fixed in 2.6.0. Regards, Akira On 5/4/15 11:15, Brahma Reddy Battula wrote: Hello Vinod, I am thinking,can we include HADOOP-11491 also..? wihout this jira harfs will not be usable when cluster installed in HA mode and try to get filecontext like below.. Path path = new Path(har:///archivedLogs/application_1428917727658_0005-application_1428917727658_0008-1428927448352.har); FileSystem fs = path.getFileSystem(new Configuration()); path = fs.makeQualified(path); FileContext fc = FileContext.getFileContext(path.toUri(),new Configuration()); Thanks Regards Brahma Reddy Battula From: Chris Nauroth [cnaur...@hortonworks.com] Sent: Friday, May 01, 2015 4:32 AM To: mapreduce-...@hadoop.apache.org; common-dev@hadoop.apache.org; yarn-...@hadoop.apache.org; hdfs-...@hadoop.apache.org Subject: Re: Planning Hadoop 2.6.1 release Thank you, Arpit. In addition, I suggest we include the following: HADOOP-11333. Fix deadlock in DomainSocketWatcher when the notification pipe is full HADOOP-11604. Prevent ConcurrentModificationException while closing domain sockets during shutdown of DomainSocketWatcher thread. HADOOP-11648. Set DomainSocketWatcher thread name explicitly HADOOP-11802. DomainSocketWatcher thread terminates sometimes after there is an I/O error during requestShortCircuitShm HADOOP-11604 and 11648 are not critical by themselves, but they are pre-requisites to getting a clean cherry-pick of 11802, which we believe finally fixes the root cause of this issue. --Chris Nauroth On 4/30/15, 3:55 PM, Arpit Agarwal aagar...@hortonworks.com wrote: HDFS candidates for back-porting to Hadoop 2.6.1. The first two were requested in [1]. HADOOP-11674. oneByteBuf in CryptoInputStream and CryptoOutputStream should be non static HADOOP-11710. Make CryptoOutputStream behave like DFSOutputStream wrt synchronization HDFS-7009. Active NN and standby NN have different live nodes. HDFS-7035. Make adding a new data directory to the DataNode an atomic and improve error handling HDFS-7425. NameNode block deletion logging uses incorrect appender. HDFS-7443. Datanode upgrade to BLOCKID_BASED_LAYOUT fails if duplicate block files are present in the same volume. HDFS-7489. Incorrect locking in FsVolumeList#checkDirs can hang datanodes HDFS-7503. Namenode restart after large deletions can cause slow processReport. HDFS-7575. Upgrade should generate a unique storage ID for each volume. HDFS-7579. Improve log reporting during block report rpc failure. HDFS-7587. Edit log corruption can happen if append fails with a quota violation. HDFS-7596. NameNode should prune dead storages from storageMap. HDFS-7611. deleteSnapshot and delete of a file can leave orphaned blocks in the blocksMap on NameNode restart. HDFS-7714. Simultaneous restart of HA NameNodes and DataNode can cause DataNode to register successfully with only one NameNode. HDFS-7733. NFS:
Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions
Yeah, I started a thread while back on this one (http://markmail.org/message/sbykjn5xgnksh6wg) and had many offline discussions re 2.6.1. The biggest problem I found offline was about what bug-fixes are acceptable and what aren’t for everyone wishing to consume 2.6.1. Given the number of bug-fixes that went into 2.7.x and into branch-2.8, figuring out a set of patches that is acceptable for everyone is a huge challenge which kind of stalled my attempts. Thanks +Vinod On Jul 15, 2015, at 8:57 AM, Sangjin Lee sjl...@gmail.com wrote: Strong +1 for having a 2.6.1 release. I understand Vinod has been trying to get that effort going but it's been stalled a little bit. It would be good to rekindle that effort. Companies with big hadoop 2.x deployments (including mine) have always tried to stabilize a 2.x release by testing/collecting/researching critical issues on the release. Each would come up with its own set of fixes to backport. We would also communicate it via offline channels. During the hadoop summit, we thought it would be great if we all came together and create a public stability/bugfix release on top of 2.x (2.6.1 for 2.6 for example) with all the critical issues fixed. Thanks, Sangjin On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa oz...@apache.org wrote: Thank you for the notification. Trying to back port bug fixes. - Tsuyoshi On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey bus...@cloudera.com wrote: Hi Hadoopers! Over in HBase we've been discussing the impact of our dependencies on our downstream users. As our most fundamental dependency, Hadoop plays a big role in the operational cost of running an HBase instance. Currently the HBase 1.y release line supports Hadoop 2.4, 2.5, and 2.6[1]. We don't drop Hadoop minor release lines in minor releases so we are unlikely remove anything from this set until HBase 2.0, probably at the end of 2015 / start of 2016 (and currently we plan to continue supporting at least 2.4 for HBase 2.0 [2]). Lately we've been discussing updating our shipped binaries to Hadoop 2.6, following some stability testing by part of our community[3]. Unfortunately, 2.6.0 in particular has a couple of bugs that could destroy HBase clusters should users decide to turn on HDFS encryption[4]. Our installation instructions tell folks to replace these jars with the version of Hadoop they are actually running, but not all users follow those instructions so we want to minimize the pain for them. Regular maintenance releases are key to keeping operational burdens low for our downstream users; we don't want them to be forced to choose between living with broken systems and stomaching the risk of upgrades across minor/major version numbers. Looking back over the three aforementioned Hadoop versions, 2.6 hasn't had a patch release since 2.6.0 came out in Nov 2014, when 2.5 had its last patch release as well. Hadoop 2.4 looks to be a year without a release[5]. On our discussion of shipping Hadoop 2.6 binaries, one of your PMC members mentioned that with continued work on the 2.7 line y'all weren't planning any additional releases of the earlier minor versions[6]. The HBase community requests that Hadoop pick up making bug-fix-only patch releases again on a regular schedule[7]. Preferably on the 2.6 line and preferably monthly. We realize that given the time gap since 2.6.0 it will likely take a big to get 2.6.1 together, but after that it should take much less effort to continue. [1]: http://hbase.apache.org/book.html#hadoop [2]: http://s.apache.org/ReP [3]: HBASE-13339 [4]: HADOOP-11674 and HADOOP-11710 [5]: http://hadoop.apache.org/releases.html [6]: http://s.apache.org/MTY [7]: http://s.apache.org/ViP -- Sean
Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions
Over on HBase, committers volunteer to be release runners for a whole release line. I wouldn't use the word 'appoint' necessarily because the arrangement is an informal communal practice, not written down anywhere as policy or codified into bylaws. If it is helpful to have a data point from another community, I am the RM for the 0.98.x line of HBase releases. I think it helps us that one committer and PMC is able to specialize on a particular release line. I know the back-porting nits in and out. Back port decisions are predicated on user demand. Sometimes users ask for changes to come back. Sometimes I will find something relevant - especially bug fixes, or solutions for operational warts - and proactively pick it back (I run 0.98 in production). Sometimes contributions are from users or devs running 0.98 and of course they need their fixes to ultimately arrive in the next 0.98 release. I help out with that process where possible. I may spend a couple of days every couple of weeks carefully picking back and porting changes. Every month I spin a new release candidate and ask our PMC to judge it worthy. We have other volunteer RMs for 1.0.x, 1.1.x, and 1.2.x. This of course doesn't infinitely scale. It could be argued as a good side effect we won't have many active release lines, only what we are capable of focusing on as a PMC. For example, should I want to throw my hat in the ring for 1.3.x, when the time arrives, I may ask for consensus on retiring 0.98 first. On Wed, Jul 15, 2015 at 3:07 PM, Chris Douglas cdoug...@apache.org wrote: On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey bus...@cloudera.com wrote: Alternatively, why not appoint a Release Manager for the minor release line and then allow them to arbitrate when there's disagreement about inclusion? This has worked well in the HBase community. Release managers aren't appointed in Hadoop. Any committer can RM a release branch and encourage others to help with it. An RM can set the bar arbitrarily, but an RC only becomes a release when a majority of PMC votes approve it in a VOTE. -C On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey bus...@cloudera.com wrote: Why not just include all backwards compatible bug fixes? Alternatively, why not appoint a Release Manager for the minor release line and then allow them to arbitrate when there's disagreement about inclusion? This has worked well in the HBase community. On Wed, Jul 15, 2015 at 3:49 PM, Karthik Kambatla ka...@cloudera.com wrote: As I proposed in the other thread, how about we adopting the following model: x.y.1 releases have all Blocker, Critical, Major bug fixes applied to the next minor release. x.y.2 releases have all Blocker, Critical bug fixes applied to the next minor release. x.y.3 releases have all Blocker bug fixes applied to next minor release. Here I am assuming there are no security-fix-only or other urgent releases. We could apply this approach for 2.7.x onwards, and do an adhoc 2.6 release. On Wed, Jul 15, 2015 at 12:59 PM, Vinod Kumar Vavilapalli vino...@hortonworks.com wrote: Yeah, I started a thread while back on this one ( http://markmail.org/message/sbykjn5xgnksh6wg) and had many offline discussions re 2.6.1. The biggest problem I found offline was about what bug-fixes are acceptable and what aren’t for everyone wishing to consume 2.6.1. Given the number of bug-fixes that went into 2.7.x and into branch-2.8, figuring out a set of patches that is acceptable for everyone is a huge challenge which kind of stalled my attempts. Thanks +Vinod On Jul 15, 2015, at 8:57 AM, Sangjin Lee sjl...@gmail.com wrote: Strong +1 for having a 2.6.1 release. I understand Vinod has been trying to get that effort going but it's been stalled a little bit. It would be good to rekindle that effort. Companies with big hadoop 2.x deployments (including mine) have always tried to stabilize a 2.x release by testing/collecting/researching critical issues on the release. Each would come up with its own set of fixes to backport. We would also communicate it via offline channels. During the hadoop summit, we thought it would be great if we all came together and create a public stability/bugfix release on top of 2.x (2.6.1 for 2.6 for example) with all the critical issues fixed. Thanks, Sangjin On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa oz...@apache.org wrote: Thank you for the notification. Trying to back port bug fixes. - Tsuyoshi On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey bus...@cloudera.com wrote: Hi Hadoopers! Over in HBase we've been discussing the impact of our dependencies on our downstream users. As our most fundamental dependency, Hadoop plays a big role in the operational cost of running an HBase instance. Currently
Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions
Is there anyone interested in volunteering to run a 2.6.1 release (Akira?)? You'd get some help (especially if the bar is set high and only critical bug fixes are allowed in: i.e. no features, no 'perf' fixes, no jar updates, and so on). St.Ack On Wed, Jul 15, 2015 at 3:07 PM, Chris Douglas cdoug...@apache.org wrote: On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey bus...@cloudera.com wrote: Alternatively, why not appoint a Release Manager for the minor release line and then allow them to arbitrate when there's disagreement about inclusion? This has worked well in the HBase community. Release managers aren't appointed in Hadoop. Any committer can RM a release branch and encourage others to help with it. An RM can set the bar arbitrarily, but an RC only becomes a release when a majority of PMC votes approve it in a VOTE. -C On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey bus...@cloudera.com wrote: Why not just include all backwards compatible bug fixes? Alternatively, why not appoint a Release Manager for the minor release line and then allow them to arbitrate when there's disagreement about inclusion? This has worked well in the HBase community. On Wed, Jul 15, 2015 at 3:49 PM, Karthik Kambatla ka...@cloudera.com wrote: As I proposed in the other thread, how about we adopting the following model: x.y.1 releases have all Blocker, Critical, Major bug fixes applied to the next minor release. x.y.2 releases have all Blocker, Critical bug fixes applied to the next minor release. x.y.3 releases have all Blocker bug fixes applied to next minor release. Here I am assuming there are no security-fix-only or other urgent releases. We could apply this approach for 2.7.x onwards, and do an adhoc 2.6 release. On Wed, Jul 15, 2015 at 12:59 PM, Vinod Kumar Vavilapalli vino...@hortonworks.com wrote: Yeah, I started a thread while back on this one ( http://markmail.org/message/sbykjn5xgnksh6wg) and had many offline discussions re 2.6.1. The biggest problem I found offline was about what bug-fixes are acceptable and what aren’t for everyone wishing to consume 2.6.1. Given the number of bug-fixes that went into 2.7.x and into branch-2.8, figuring out a set of patches that is acceptable for everyone is a huge challenge which kind of stalled my attempts. Thanks +Vinod On Jul 15, 2015, at 8:57 AM, Sangjin Lee sjl...@gmail.com wrote: Strong +1 for having a 2.6.1 release. I understand Vinod has been trying to get that effort going but it's been stalled a little bit. It would be good to rekindle that effort. Companies with big hadoop 2.x deployments (including mine) have always tried to stabilize a 2.x release by testing/collecting/researching critical issues on the release. Each would come up with its own set of fixes to backport. We would also communicate it via offline channels. During the hadoop summit, we thought it would be great if we all came together and create a public stability/bugfix release on top of 2.x (2.6.1 for 2.6 for example) with all the critical issues fixed. Thanks, Sangjin On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa oz...@apache.org wrote: Thank you for the notification. Trying to back port bug fixes. - Tsuyoshi On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey bus...@cloudera.com wrote: Hi Hadoopers! Over in HBase we've been discussing the impact of our dependencies on our downstream users. As our most fundamental dependency, Hadoop plays a big role in the operational cost of running an HBase instance. Currently the HBase 1.y release line supports Hadoop 2.4, 2.5, and 2.6[1]. We don't drop Hadoop minor release lines in minor releases so we are unlikely remove anything from this set until HBase 2.0, probably at the end of 2015 / start of 2016 (and currently we plan to continue supporting at least 2.4 for HBase 2.0 [2]). Lately we've been discussing updating our shipped binaries to Hadoop 2.6, following some stability testing by part of our community[3]. Unfortunately, 2.6.0 in particular has a couple of bugs that could destroy HBase clusters should users decide to turn on HDFS encryption[4]. Our installation instructions tell folks to replace these jars with the version of Hadoop they are actually running, but not all users follow those instructions so we want to minimize the pain for them. Regular maintenance releases are key to keeping operational burdens low for our downstream users; we don't want them to be forced to choose between living with broken systems and stomaching the risk of upgrades across minor/major version numbers. Looking back over the three aforementioned Hadoop versions, 2.6 hasn't had a patch
Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions
As I proposed in the other thread, how about we adopting the following model: x.y.1 releases have all Blocker, Critical, Major bug fixes applied to the next minor release. x.y.2 releases have all Blocker, Critical bug fixes applied to the next minor release. x.y.3 releases have all Blocker bug fixes applied to next minor release. Here I am assuming there are no security-fix-only or other urgent releases. We could apply this approach for 2.7.x onwards, and do an adhoc 2.6 release. On Wed, Jul 15, 2015 at 12:59 PM, Vinod Kumar Vavilapalli vino...@hortonworks.com wrote: Yeah, I started a thread while back on this one ( http://markmail.org/message/sbykjn5xgnksh6wg) and had many offline discussions re 2.6.1. The biggest problem I found offline was about what bug-fixes are acceptable and what aren’t for everyone wishing to consume 2.6.1. Given the number of bug-fixes that went into 2.7.x and into branch-2.8, figuring out a set of patches that is acceptable for everyone is a huge challenge which kind of stalled my attempts. Thanks +Vinod On Jul 15, 2015, at 8:57 AM, Sangjin Lee sjl...@gmail.com wrote: Strong +1 for having a 2.6.1 release. I understand Vinod has been trying to get that effort going but it's been stalled a little bit. It would be good to rekindle that effort. Companies with big hadoop 2.x deployments (including mine) have always tried to stabilize a 2.x release by testing/collecting/researching critical issues on the release. Each would come up with its own set of fixes to backport. We would also communicate it via offline channels. During the hadoop summit, we thought it would be great if we all came together and create a public stability/bugfix release on top of 2.x (2.6.1 for 2.6 for example) with all the critical issues fixed. Thanks, Sangjin On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa oz...@apache.org wrote: Thank you for the notification. Trying to back port bug fixes. - Tsuyoshi On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey bus...@cloudera.com wrote: Hi Hadoopers! Over in HBase we've been discussing the impact of our dependencies on our downstream users. As our most fundamental dependency, Hadoop plays a big role in the operational cost of running an HBase instance. Currently the HBase 1.y release line supports Hadoop 2.4, 2.5, and 2.6[1]. We don't drop Hadoop minor release lines in minor releases so we are unlikely remove anything from this set until HBase 2.0, probably at the end of 2015 / start of 2016 (and currently we plan to continue supporting at least 2.4 for HBase 2.0 [2]). Lately we've been discussing updating our shipped binaries to Hadoop 2.6, following some stability testing by part of our community[3]. Unfortunately, 2.6.0 in particular has a couple of bugs that could destroy HBase clusters should users decide to turn on HDFS encryption[4]. Our installation instructions tell folks to replace these jars with the version of Hadoop they are actually running, but not all users follow those instructions so we want to minimize the pain for them. Regular maintenance releases are key to keeping operational burdens low for our downstream users; we don't want them to be forced to choose between living with broken systems and stomaching the risk of upgrades across minor/major version numbers. Looking back over the three aforementioned Hadoop versions, 2.6 hasn't had a patch release since 2.6.0 came out in Nov 2014, when 2.5 had its last patch release as well. Hadoop 2.4 looks to be a year without a release[5]. On our discussion of shipping Hadoop 2.6 binaries, one of your PMC members mentioned that with continued work on the 2.7 line y'all weren't planning any additional releases of the earlier minor versions[6]. The HBase community requests that Hadoop pick up making bug-fix-only patch releases again on a regular schedule[7]. Preferably on the 2.6 line and preferably monthly. We realize that given the time gap since 2.6.0 it will likely take a big to get 2.6.1 together, but after that it should take much less effort to continue. [1]: http://hbase.apache.org/book.html#hadoop [2]: http://s.apache.org/ReP [3]: HBASE-13339 [4]: HADOOP-11674 and HADOOP-11710 [5]: http://hadoop.apache.org/releases.html [6]: http://s.apache.org/MTY [7]: http://s.apache.org/ViP -- Sean -- Karthik Kambatla Software Engineer, Cloudera Inc. http://five.sentenc.es
Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions
If people are concerned about regression then just don't install new versions, or install a vendor tested stable version. Giving users choices is a good thing for stability. On Wed, Jul 15, 2015 at 2:17 PM, Sangjin Lee sjl...@gmail.com wrote: I think the bar for making the maintenance releases should be set reasonably high, and the main reason is the concern for stability/regression. Unfortunately there have been cases where a seemingly innocuous bug fix introduced regressions, small or large. And that defeats the purpose of a maintenance release. On Wed, Jul 15, 2015 at 2:11 PM, Karthik Kambatla ka...@cloudera.com wrote: Every new patch potentially brings in new bugs. So, if we want to limit the kinds of potential bugs introduced in point releases, we might want to limit what gets in. Would be nice to make sure a point release is more stable than a previous point release in that line. On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey bus...@cloudera.com wrote: Why not just include all backwards compatible bug fixes? Alternatively, why not appoint a Release Manager for the minor release line and then allow them to arbitrate when there's disagreement about inclusion? This has worked well in the HBase community. On Wed, Jul 15, 2015 at 3:49 PM, Karthik Kambatla ka...@cloudera.com wrote: As I proposed in the other thread, how about we adopting the following model: x.y.1 releases have all Blocker, Critical, Major bug fixes applied to the next minor release. x.y.2 releases have all Blocker, Critical bug fixes applied to the next minor release. x.y.3 releases have all Blocker bug fixes applied to next minor release. Here I am assuming there are no security-fix-only or other urgent releases. We could apply this approach for 2.7.x onwards, and do an adhoc 2.6 release. On Wed, Jul 15, 2015 at 12:59 PM, Vinod Kumar Vavilapalli vino...@hortonworks.com wrote: Yeah, I started a thread while back on this one ( http://markmail.org/message/sbykjn5xgnksh6wg) and had many offline discussions re 2.6.1. The biggest problem I found offline was about what bug-fixes are acceptable and what aren’t for everyone wishing to consume 2.6.1. Given the number of bug-fixes that went into 2.7.x and into branch-2.8, figuring out a set of patches that is acceptable for everyone is a huge challenge which kind of stalled my attempts. Thanks +Vinod On Jul 15, 2015, at 8:57 AM, Sangjin Lee sjl...@gmail.com wrote: Strong +1 for having a 2.6.1 release. I understand Vinod has been trying to get that effort going but it's been stalled a little bit. It would be good to rekindle that effort. Companies with big hadoop 2.x deployments (including mine) have always tried to stabilize a 2.x release by testing/collecting/researching critical issues on the release. Each would come up with its own set of fixes to backport. We would also communicate it via offline channels. During the hadoop summit, we thought it would be great if we all came together and create a public stability/bugfix release on top of 2.x (2.6.1 for 2.6 for example) with all the critical issues fixed. Thanks, Sangjin On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa oz...@apache.org wrote: Thank you for the notification. Trying to back port bug fixes. - Tsuyoshi On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey bus...@cloudera.com wrote: Hi Hadoopers! Over in HBase we've been discussing the impact of our dependencies on our downstream users. As our most fundamental dependency, Hadoop plays a big role in the operational cost of running an HBase instance. Currently the HBase 1.y release line supports Hadoop 2.4, 2.5, and 2.6[1]. We don't drop Hadoop minor release lines in minor releases so we are unlikely remove anything from this set until HBase 2.0, probably at the end of 2015 / start of 2016 (and currently we plan to continue supporting at least 2.4 for HBase 2.0 [2]). Lately we've been discussing updating our shipped binaries to Hadoop 2.6, following some stability testing by part of our community[3]. Unfortunately, 2.6.0 in particular has a couple of bugs that could destroy HBase clusters should users decide to turn on HDFS encryption[4]. Our installation instructions tell folks to replace these jars with the version of Hadoop they are actually running, but not all users follow those instructions so we want to minimize the pain for them. Regular
Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions
Every new patch potentially brings in new bugs. So, if we want to limit the kinds of potential bugs introduced in point releases, we might want to limit what gets in. Would be nice to make sure a point release is more stable than a previous point release in that line. On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey bus...@cloudera.com wrote: Why not just include all backwards compatible bug fixes? Alternatively, why not appoint a Release Manager for the minor release line and then allow them to arbitrate when there's disagreement about inclusion? This has worked well in the HBase community. On Wed, Jul 15, 2015 at 3:49 PM, Karthik Kambatla ka...@cloudera.com wrote: As I proposed in the other thread, how about we adopting the following model: x.y.1 releases have all Blocker, Critical, Major bug fixes applied to the next minor release. x.y.2 releases have all Blocker, Critical bug fixes applied to the next minor release. x.y.3 releases have all Blocker bug fixes applied to next minor release. Here I am assuming there are no security-fix-only or other urgent releases. We could apply this approach for 2.7.x onwards, and do an adhoc 2.6 release. On Wed, Jul 15, 2015 at 12:59 PM, Vinod Kumar Vavilapalli vino...@hortonworks.com wrote: Yeah, I started a thread while back on this one ( http://markmail.org/message/sbykjn5xgnksh6wg) and had many offline discussions re 2.6.1. The biggest problem I found offline was about what bug-fixes are acceptable and what aren’t for everyone wishing to consume 2.6.1. Given the number of bug-fixes that went into 2.7.x and into branch-2.8, figuring out a set of patches that is acceptable for everyone is a huge challenge which kind of stalled my attempts. Thanks +Vinod On Jul 15, 2015, at 8:57 AM, Sangjin Lee sjl...@gmail.com wrote: Strong +1 for having a 2.6.1 release. I understand Vinod has been trying to get that effort going but it's been stalled a little bit. It would be good to rekindle that effort. Companies with big hadoop 2.x deployments (including mine) have always tried to stabilize a 2.x release by testing/collecting/researching critical issues on the release. Each would come up with its own set of fixes to backport. We would also communicate it via offline channels. During the hadoop summit, we thought it would be great if we all came together and create a public stability/bugfix release on top of 2.x (2.6.1 for 2.6 for example) with all the critical issues fixed. Thanks, Sangjin On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa oz...@apache.org wrote: Thank you for the notification. Trying to back port bug fixes. - Tsuyoshi On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey bus...@cloudera.com wrote: Hi Hadoopers! Over in HBase we've been discussing the impact of our dependencies on our downstream users. As our most fundamental dependency, Hadoop plays a big role in the operational cost of running an HBase instance. Currently the HBase 1.y release line supports Hadoop 2.4, 2.5, and 2.6[1]. We don't drop Hadoop minor release lines in minor releases so we are unlikely remove anything from this set until HBase 2.0, probably at the end of 2015 / start of 2016 (and currently we plan to continue supporting at least 2.4 for HBase 2.0 [2]). Lately we've been discussing updating our shipped binaries to Hadoop 2.6, following some stability testing by part of our community[3]. Unfortunately, 2.6.0 in particular has a couple of bugs that could destroy HBase clusters should users decide to turn on HDFS encryption[4]. Our installation instructions tell folks to replace these jars with the version of Hadoop they are actually running, but not all users follow those instructions so we want to minimize the pain for them. Regular maintenance releases are key to keeping operational burdens low for our downstream users; we don't want them to be forced to choose between living with broken systems and stomaching the risk of upgrades across minor/major version numbers. Looking back over the three aforementioned Hadoop versions, 2.6 hasn't had a patch release since 2.6.0 came out in Nov 2014, when 2.5 had its last patch release as well. Hadoop 2.4 looks to be a year without a release[5]. On our discussion of shipping Hadoop 2.6 binaries, one of your PMC members mentioned that with continued work on the 2.7 line y'all weren't planning any additional releases of the earlier minor versions[6]. The HBase community requests that Hadoop pick up making bug-fix-only patch releases again on a regular schedule[7]. Preferably on the 2.6 line and preferably monthly. We realize that given the
Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions
I think the bar for making the maintenance releases should be set reasonably high, and the main reason is the concern for stability/regression. Unfortunately there have been cases where a seemingly innocuous bug fix introduced regressions, small or large. And that defeats the purpose of a maintenance release. On Wed, Jul 15, 2015 at 2:11 PM, Karthik Kambatla ka...@cloudera.com wrote: Every new patch potentially brings in new bugs. So, if we want to limit the kinds of potential bugs introduced in point releases, we might want to limit what gets in. Would be nice to make sure a point release is more stable than a previous point release in that line. On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey bus...@cloudera.com wrote: Why not just include all backwards compatible bug fixes? Alternatively, why not appoint a Release Manager for the minor release line and then allow them to arbitrate when there's disagreement about inclusion? This has worked well in the HBase community. On Wed, Jul 15, 2015 at 3:49 PM, Karthik Kambatla ka...@cloudera.com wrote: As I proposed in the other thread, how about we adopting the following model: x.y.1 releases have all Blocker, Critical, Major bug fixes applied to the next minor release. x.y.2 releases have all Blocker, Critical bug fixes applied to the next minor release. x.y.3 releases have all Blocker bug fixes applied to next minor release. Here I am assuming there are no security-fix-only or other urgent releases. We could apply this approach for 2.7.x onwards, and do an adhoc 2.6 release. On Wed, Jul 15, 2015 at 12:59 PM, Vinod Kumar Vavilapalli vino...@hortonworks.com wrote: Yeah, I started a thread while back on this one ( http://markmail.org/message/sbykjn5xgnksh6wg) and had many offline discussions re 2.6.1. The biggest problem I found offline was about what bug-fixes are acceptable and what aren’t for everyone wishing to consume 2.6.1. Given the number of bug-fixes that went into 2.7.x and into branch-2.8, figuring out a set of patches that is acceptable for everyone is a huge challenge which kind of stalled my attempts. Thanks +Vinod On Jul 15, 2015, at 8:57 AM, Sangjin Lee sjl...@gmail.com wrote: Strong +1 for having a 2.6.1 release. I understand Vinod has been trying to get that effort going but it's been stalled a little bit. It would be good to rekindle that effort. Companies with big hadoop 2.x deployments (including mine) have always tried to stabilize a 2.x release by testing/collecting/researching critical issues on the release. Each would come up with its own set of fixes to backport. We would also communicate it via offline channels. During the hadoop summit, we thought it would be great if we all came together and create a public stability/bugfix release on top of 2.x (2.6.1 for 2.6 for example) with all the critical issues fixed. Thanks, Sangjin On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa oz...@apache.org wrote: Thank you for the notification. Trying to back port bug fixes. - Tsuyoshi On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey bus...@cloudera.com wrote: Hi Hadoopers! Over in HBase we've been discussing the impact of our dependencies on our downstream users. As our most fundamental dependency, Hadoop plays a big role in the operational cost of running an HBase instance. Currently the HBase 1.y release line supports Hadoop 2.4, 2.5, and 2.6[1]. We don't drop Hadoop minor release lines in minor releases so we are unlikely remove anything from this set until HBase 2.0, probably at the end of 2015 / start of 2016 (and currently we plan to continue supporting at least 2.4 for HBase 2.0 [2]). Lately we've been discussing updating our shipped binaries to Hadoop 2.6, following some stability testing by part of our community[3]. Unfortunately, 2.6.0 in particular has a couple of bugs that could destroy HBase clusters should users decide to turn on HDFS encryption[4]. Our installation instructions tell folks to replace these jars with the version of Hadoop they are actually running, but not all users follow those instructions so we want to minimize the pain for them. Regular maintenance releases are key to keeping operational burdens low for our downstream users; we don't want them to be forced to choose between living with broken systems and stomaching the risk of upgrades across minor/major version numbers. Looking back over the three aforementioned Hadoop versions, 2.6 hasn't had a patch release since 2.6.0 came out in Nov 2014,
Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions
Why not just include all backwards compatible bug fixes? Alternatively, why not appoint a Release Manager for the minor release line and then allow them to arbitrate when there's disagreement about inclusion? This has worked well in the HBase community. On Wed, Jul 15, 2015 at 3:49 PM, Karthik Kambatla ka...@cloudera.com wrote: As I proposed in the other thread, how about we adopting the following model: x.y.1 releases have all Blocker, Critical, Major bug fixes applied to the next minor release. x.y.2 releases have all Blocker, Critical bug fixes applied to the next minor release. x.y.3 releases have all Blocker bug fixes applied to next minor release. Here I am assuming there are no security-fix-only or other urgent releases. We could apply this approach for 2.7.x onwards, and do an adhoc 2.6 release. On Wed, Jul 15, 2015 at 12:59 PM, Vinod Kumar Vavilapalli vino...@hortonworks.com wrote: Yeah, I started a thread while back on this one ( http://markmail.org/message/sbykjn5xgnksh6wg) and had many offline discussions re 2.6.1. The biggest problem I found offline was about what bug-fixes are acceptable and what aren’t for everyone wishing to consume 2.6.1. Given the number of bug-fixes that went into 2.7.x and into branch-2.8, figuring out a set of patches that is acceptable for everyone is a huge challenge which kind of stalled my attempts. Thanks +Vinod On Jul 15, 2015, at 8:57 AM, Sangjin Lee sjl...@gmail.com wrote: Strong +1 for having a 2.6.1 release. I understand Vinod has been trying to get that effort going but it's been stalled a little bit. It would be good to rekindle that effort. Companies with big hadoop 2.x deployments (including mine) have always tried to stabilize a 2.x release by testing/collecting/researching critical issues on the release. Each would come up with its own set of fixes to backport. We would also communicate it via offline channels. During the hadoop summit, we thought it would be great if we all came together and create a public stability/bugfix release on top of 2.x (2.6.1 for 2.6 for example) with all the critical issues fixed. Thanks, Sangjin On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa oz...@apache.org wrote: Thank you for the notification. Trying to back port bug fixes. - Tsuyoshi On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey bus...@cloudera.com wrote: Hi Hadoopers! Over in HBase we've been discussing the impact of our dependencies on our downstream users. As our most fundamental dependency, Hadoop plays a big role in the operational cost of running an HBase instance. Currently the HBase 1.y release line supports Hadoop 2.4, 2.5, and 2.6[1]. We don't drop Hadoop minor release lines in minor releases so we are unlikely remove anything from this set until HBase 2.0, probably at the end of 2015 / start of 2016 (and currently we plan to continue supporting at least 2.4 for HBase 2.0 [2]). Lately we've been discussing updating our shipped binaries to Hadoop 2.6, following some stability testing by part of our community[3]. Unfortunately, 2.6.0 in particular has a couple of bugs that could destroy HBase clusters should users decide to turn on HDFS encryption[4]. Our installation instructions tell folks to replace these jars with the version of Hadoop they are actually running, but not all users follow those instructions so we want to minimize the pain for them. Regular maintenance releases are key to keeping operational burdens low for our downstream users; we don't want them to be forced to choose between living with broken systems and stomaching the risk of upgrades across minor/major version numbers. Looking back over the three aforementioned Hadoop versions, 2.6 hasn't had a patch release since 2.6.0 came out in Nov 2014, when 2.5 had its last patch release as well. Hadoop 2.4 looks to be a year without a release[5]. On our discussion of shipping Hadoop 2.6 binaries, one of your PMC members mentioned that with continued work on the 2.7 line y'all weren't planning any additional releases of the earlier minor versions[6]. The HBase community requests that Hadoop pick up making bug-fix-only patch releases again on a regular schedule[7]. Preferably on the 2.6 line and preferably monthly. We realize that given the time gap since 2.6.0 it will likely take a big to get 2.6.1 together, but after that it should take much less effort to continue. [1]: http://hbase.apache.org/book.html#hadoop [2]: http://s.apache.org/ReP [3]: HBASE-13339 [4]: HADOOP-11674 and HADOOP-11710 [5]: http://hadoop.apache.org/releases.html [6]: http://s.apache.org/MTY [7]: http://s.apache.org/ViP -- Sean -- Karthik Kambatla Software Engineer, Cloudera Inc.
Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions
I can understand these (sort of newish) questions from hbase-dev. We already have a well laid-out release-management process. If people want to learn more about how it works, please head over to http://hadoop.apache.org/bylaws.html. In terms of 2.6.1 release management, it wasn’t stuck for lack of volunteers for RM. I originally was driving it [1], and then Akira recently volunteered too in a separate thread [2] (which I totally missed participating in before), so we have enough help. It is clearly acknowledged there is a demand for 2.6.1, we now have to figure out the mechanics, agreeing on a reasonable common list of fixes to start with. Tx for the feedback so far @hbase-dev! I agree with Karthik earlier - let’s continue the discussion in hadoop-dev in the release thread [1] for 2.6.1. Thanks +Vinod [1] Planning Hadoop 2.6.1 release http://markmail.org/message/sbykjn5xgnksh6wg [2] [DISCUSS] More Maintenance Releases http://s.apache.org/MMR On Jul 15, 2015, at 4:07 PM, Stack st...@duboce.netmailto:st...@duboce.net wrote: Is there anyone interested in volunteering to run a 2.6.1 release (Akira?)? You'd get some help (especially if the bar is set high and only critical bug fixes are allowed in: i.e. no features, no 'perf' fixes, no jar updates, and so on). St.Ack On Wed, Jul 15, 2015 at 3:07 PM, Chris Douglas cdoug...@apache.orgmailto:cdoug...@apache.org wrote: On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey bus...@cloudera.commailto:bus...@cloudera.com wrote: Alternatively, why not appoint a Release Manager for the minor release line and then allow them to arbitrate when there's disagreement about inclusion? This has worked well in the HBase community. Release managers aren't appointed in Hadoop. Any committer can RM a release branch and encourage others to help with it. An RM can set the bar arbitrarily, but an RC only becomes a release when a majority of PMC votes approve it in a VOTE. -C On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey bus...@cloudera.commailto:bus...@cloudera.com wrote: Why not just include all backwards compatible bug fixes? Alternatively, why not appoint a Release Manager for the minor release line and then allow them to arbitrate when there's disagreement about inclusion? This has worked well in the HBase community. On Wed, Jul 15, 2015 at 3:49 PM, Karthik Kambatla ka...@cloudera.commailto:ka...@cloudera.com wrote: As I proposed in the other thread, how about we adopting the following model: x.y.1 releases have all Blocker, Critical, Major bug fixes applied to the next minor release. x.y.2 releases have all Blocker, Critical bug fixes applied to the next minor release. x.y.3 releases have all Blocker bug fixes applied to next minor release. Here I am assuming there are no security-fix-only or other urgent releases. We could apply this approach for 2.7.x onwards, and do an adhoc 2.6 release. On Wed, Jul 15, 2015 at 12:59 PM, Vinod Kumar Vavilapalli vino...@hortonworks.commailto:vino...@hortonworks.com wrote: Yeah, I started a thread while back on this one ( http://markmail.org/message/sbykjn5xgnksh6wg) and had many offline discussions re 2.6.1. The biggest problem I found offline was about what bug-fixes are acceptable and what aren’t for everyone wishing to consume 2.6.1. Given the number of bug-fixes that went into 2.7.x and into branch-2.8, figuring out a set of patches that is acceptable for everyone is a huge challenge which kind of stalled my attempts. Thanks +Vinod On Jul 15, 2015, at 8:57 AM, Sangjin Lee sjl...@gmail.commailto:sjl...@gmail.com wrote: Strong +1 for having a 2.6.1 release. I understand Vinod has been trying to get that effort going but it's been stalled a little bit. It would be good to rekindle that effort. Companies with big hadoop 2.x deployments (including mine) have always tried to stabilize a 2.x release by testing/collecting/researching critical issues on the release. Each would come up with its own set of fixes to backport. We would also communicate it via offline channels. During the hadoop summit, we thought it would be great if we all came together and create a public stability/bugfix release on top of 2.x (2.6.1 for 2.6 for example) with all the critical issues fixed. Thanks, Sangjin On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa oz...@apache.orgmailto:oz...@apache.org wrote: Thank you for the notification. Trying to back port bug fixes. - Tsuyoshi On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey bus...@cloudera.commailto:bus...@cloudera.com wrote: Hi Hadoopers! Over in HBase we've been discussing the impact of our dependencies on our downstream users. As our most fundamental dependency, Hadoop plays a big role in the operational cost of running an HBase instance. Currently the HBase 1.y release line supports Hadoop 2.4, 2.5, and 2.6[1]. We don't drop Hadoop minor release lines in minor releases so we are unlikely remove anything from this set until HBase 2.0, probably at the end of 2015 / start of 2016
Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions
On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey bus...@cloudera.com wrote: Alternatively, why not appoint a Release Manager for the minor release line and then allow them to arbitrate when there's disagreement about inclusion? This has worked well in the HBase community. Release managers aren't appointed in Hadoop. Any committer can RM a release branch and encourage others to help with it. An RM can set the bar arbitrarily, but an RC only becomes a release when a majority of PMC votes approve it in a VOTE. -C On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey bus...@cloudera.com wrote: Why not just include all backwards compatible bug fixes? Alternatively, why not appoint a Release Manager for the minor release line and then allow them to arbitrate when there's disagreement about inclusion? This has worked well in the HBase community. On Wed, Jul 15, 2015 at 3:49 PM, Karthik Kambatla ka...@cloudera.com wrote: As I proposed in the other thread, how about we adopting the following model: x.y.1 releases have all Blocker, Critical, Major bug fixes applied to the next minor release. x.y.2 releases have all Blocker, Critical bug fixes applied to the next minor release. x.y.3 releases have all Blocker bug fixes applied to next minor release. Here I am assuming there are no security-fix-only or other urgent releases. We could apply this approach for 2.7.x onwards, and do an adhoc 2.6 release. On Wed, Jul 15, 2015 at 12:59 PM, Vinod Kumar Vavilapalli vino...@hortonworks.com wrote: Yeah, I started a thread while back on this one ( http://markmail.org/message/sbykjn5xgnksh6wg) and had many offline discussions re 2.6.1. The biggest problem I found offline was about what bug-fixes are acceptable and what aren’t for everyone wishing to consume 2.6.1. Given the number of bug-fixes that went into 2.7.x and into branch-2.8, figuring out a set of patches that is acceptable for everyone is a huge challenge which kind of stalled my attempts. Thanks +Vinod On Jul 15, 2015, at 8:57 AM, Sangjin Lee sjl...@gmail.com wrote: Strong +1 for having a 2.6.1 release. I understand Vinod has been trying to get that effort going but it's been stalled a little bit. It would be good to rekindle that effort. Companies with big hadoop 2.x deployments (including mine) have always tried to stabilize a 2.x release by testing/collecting/researching critical issues on the release. Each would come up with its own set of fixes to backport. We would also communicate it via offline channels. During the hadoop summit, we thought it would be great if we all came together and create a public stability/bugfix release on top of 2.x (2.6.1 for 2.6 for example) with all the critical issues fixed. Thanks, Sangjin On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa oz...@apache.org wrote: Thank you for the notification. Trying to back port bug fixes. - Tsuyoshi On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey bus...@cloudera.com wrote: Hi Hadoopers! Over in HBase we've been discussing the impact of our dependencies on our downstream users. As our most fundamental dependency, Hadoop plays a big role in the operational cost of running an HBase instance. Currently the HBase 1.y release line supports Hadoop 2.4, 2.5, and 2.6[1]. We don't drop Hadoop minor release lines in minor releases so we are unlikely remove anything from this set until HBase 2.0, probably at the end of 2015 / start of 2016 (and currently we plan to continue supporting at least 2.4 for HBase 2.0 [2]). Lately we've been discussing updating our shipped binaries to Hadoop 2.6, following some stability testing by part of our community[3]. Unfortunately, 2.6.0 in particular has a couple of bugs that could destroy HBase clusters should users decide to turn on HDFS encryption[4]. Our installation instructions tell folks to replace these jars with the version of Hadoop they are actually running, but not all users follow those instructions so we want to minimize the pain for them. Regular maintenance releases are key to keeping operational burdens low for our downstream users; we don't want them to be forced to choose between living with broken systems and stomaching the risk of upgrades across minor/major version numbers. Looking back over the three aforementioned Hadoop versions, 2.6 hasn't had a patch release since 2.6.0 came out in Nov 2014, when 2.5 had its last patch release as well. Hadoop 2.4 looks to be a year without a release[5]. On our discussion of shipping Hadoop 2.6 binaries, one of your PMC members mentioned that with continued work on the 2.7 line y'all weren't planning any additional releases of the earlier minor versions[6]. The HBase community requests that Hadoop pick up making bug-fix-only patch releases again
Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions
+1 Chris is right. ++ Chris Mattmann, Ph.D. Chief Architect Instrument Software and Science Data Systems Section (398) NASA Jet Propulsion Laboratory Pasadena, CA 91109 USA Office: 168-519, Mailstop: 168-527 Email: chris.a.mattm...@nasa.gov WWW: http://sunset.usc.edu/~mattmann/ ++ Adjunct Associate Professor, Computer Science Department University of Southern California, Los Angeles, CA 90089 USA ++ -Original Message- From: Chris Douglas cdoug...@apache.org Reply-To: common-dev@hadoop.apache.org common-dev@hadoop.apache.org Date: Wednesday, July 15, 2015 at 3:07 PM To: common-dev@hadoop.apache.org common-dev@hadoop.apache.org Cc: dev d...@hbase.apache.org Subject: Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey bus...@cloudera.com wrote: Alternatively, why not appoint a Release Manager for the minor release line and then allow them to arbitrate when there's disagreement about inclusion? This has worked well in the HBase community. Release managers aren't appointed in Hadoop. Any committer can RM a release branch and encourage others to help with it. An RM can set the bar arbitrarily, but an RC only becomes a release when a majority of PMC votes approve it in a VOTE. -C On Wed, Jul 15, 2015 at 2:07 PM, Sean Busbey bus...@cloudera.com wrote: Why not just include all backwards compatible bug fixes? Alternatively, why not appoint a Release Manager for the minor release line and then allow them to arbitrate when there's disagreement about inclusion? This has worked well in the HBase community. On Wed, Jul 15, 2015 at 3:49 PM, Karthik Kambatla ka...@cloudera.com wrote: As I proposed in the other thread, how about we adopting the following model: x.y.1 releases have all Blocker, Critical, Major bug fixes applied to the next minor release. x.y.2 releases have all Blocker, Critical bug fixes applied to the next minor release. x.y.3 releases have all Blocker bug fixes applied to next minor release. Here I am assuming there are no security-fix-only or other urgent releases. We could apply this approach for 2.7.x onwards, and do an adhoc 2.6 release. On Wed, Jul 15, 2015 at 12:59 PM, Vinod Kumar Vavilapalli vino...@hortonworks.com wrote: Yeah, I started a thread while back on this one ( http://markmail.org/message/sbykjn5xgnksh6wg) and had many offline discussions re 2.6.1. The biggest problem I found offline was about what bug-fixes are acceptable and what aren’t for everyone wishing to consume 2.6.1. Given the number of bug-fixes that went into 2.7.x and into branch-2.8, figuring out a set of patches that is acceptable for everyone is a huge challenge which kind of stalled my attempts. Thanks +Vinod On Jul 15, 2015, at 8:57 AM, Sangjin Lee sjl...@gmail.com wrote: Strong +1 for having a 2.6.1 release. I understand Vinod has been trying to get that effort going but it's been stalled a little bit. It would be good to rekindle that effort. Companies with big hadoop 2.x deployments (including mine) have always tried to stabilize a 2.x release by testing/collecting/researching critical issues on the release. Each would come up with its own set of fixes to backport. We would also communicate it via offline channels. During the hadoop summit, we thought it would be great if we all came together and create a public stability/bugfix release on top of 2.x (2.6.1 for 2.6 for example) with all the critical issues fixed. Thanks, Sangjin On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa oz...@apache.org wrote: Thank you for the notification. Trying to back port bug fixes. - Tsuyoshi On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey bus...@cloudera.com wrote: Hi Hadoopers! Over in HBase we've been discussing the impact of our dependencies on our downstream users. As our most fundamental dependency, Hadoop plays a big role in the operational cost of running an HBase instance. Currently the HBase 1.y release line supports Hadoop 2.4, 2.5, and 2.6[1]. We don't drop Hadoop minor release lines in minor releases so we are unlikely remove anything from this set until HBase 2.0, probably at the end of 2015 / start of 2016 (and currently we plan to continue supporting at least 2.4 for HBase 2.0 [2]). Lately we've been discussing updating our shipped binaries to Hadoop 2.6, following some stability testing by part of our community[3]. Unfortunately, 2.6.0 in particular has a couple of bugs that could destroy HBase clusters should users decide to turn on HDFS encryption[4]. Our installation instructions tell folks to replace these jars with the version of
[jira] [Created] (HADOOP-12237) releasedocmaker.py doesn't work behind a proxy
Tsuyoshi Ozawa created HADOOP-12237:
---
Summary: releasedocmaker.py doesn't work behind a proxy
Key: HADOOP-12237
URL: https://issues.apache.org/jira/browse/HADOOP-12237
Project: Hadoop Common
Issue Type: Sub-task
Reporter: Tsuyoshi Ozawa
HADOOP-12236 for Yetus.
{quote}
releasedocmaker.py doesn't work behind a proxy because urllib.urlopen doesn't
care environment varialibes like $http_proxy or $https_proxy.
{quote}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions
Strong +1 for having a 2.6.1 release. I understand Vinod has been trying to get that effort going but it's been stalled a little bit. It would be good to rekindle that effort. Companies with big hadoop 2.x deployments (including mine) have always tried to stabilize a 2.x release by testing/collecting/researching critical issues on the release. Each would come up with its own set of fixes to backport. We would also communicate it via offline channels. During the hadoop summit, we thought it would be great if we all came together and create a public stability/bugfix release on top of 2.x (2.6.1 for 2.6 for example) with all the critical issues fixed. Thanks, Sangjin On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa oz...@apache.org wrote: Thank you for the notification. Trying to back port bug fixes. - Tsuyoshi On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey bus...@cloudera.com wrote: Hi Hadoopers! Over in HBase we've been discussing the impact of our dependencies on our downstream users. As our most fundamental dependency, Hadoop plays a big role in the operational cost of running an HBase instance. Currently the HBase 1.y release line supports Hadoop 2.4, 2.5, and 2.6[1]. We don't drop Hadoop minor release lines in minor releases so we are unlikely remove anything from this set until HBase 2.0, probably at the end of 2015 / start of 2016 (and currently we plan to continue supporting at least 2.4 for HBase 2.0 [2]). Lately we've been discussing updating our shipped binaries to Hadoop 2.6, following some stability testing by part of our community[3]. Unfortunately, 2.6.0 in particular has a couple of bugs that could destroy HBase clusters should users decide to turn on HDFS encryption[4]. Our installation instructions tell folks to replace these jars with the version of Hadoop they are actually running, but not all users follow those instructions so we want to minimize the pain for them. Regular maintenance releases are key to keeping operational burdens low for our downstream users; we don't want them to be forced to choose between living with broken systems and stomaching the risk of upgrades across minor/major version numbers. Looking back over the three aforementioned Hadoop versions, 2.6 hasn't had a patch release since 2.6.0 came out in Nov 2014, when 2.5 had its last patch release as well. Hadoop 2.4 looks to be a year without a release[5]. On our discussion of shipping Hadoop 2.6 binaries, one of your PMC members mentioned that with continued work on the 2.7 line y'all weren't planning any additional releases of the earlier minor versions[6]. The HBase community requests that Hadoop pick up making bug-fix-only patch releases again on a regular schedule[7]. Preferably on the 2.6 line and preferably monthly. We realize that given the time gap since 2.6.0 it will likely take a big to get 2.6.1 together, but after that it should take much less effort to continue. [1]: http://hbase.apache.org/book.html#hadoop [2]: http://s.apache.org/ReP [3]: HBASE-13339 [4]: HADOOP-11674 and HADOOP-11710 [5]: http://hadoop.apache.org/releases.html [6]: http://s.apache.org/MTY [7]: http://s.apache.org/ViP -- Sean
[jira] [Created] (HADOOP-12238) Passing project option to releasedocmaker.py for running mvn site -Preleasedocs
Tsuyoshi Ozawa created HADOOP-12238: --- Summary: Passing project option to releasedocmaker.py for running mvn site -Preleasedocs Key: HADOOP-12238 URL: https://issues.apache.org/jira/browse/HADOOP-12238 Project: Hadoop Common Issue Type: Sub-task Reporter: Tsuyoshi Ozawa Assignee: Tsuyoshi Ozawa Currently we cannot run mvn site -Preleasedocs on branch for HADOOP-12111. This patch fix it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (HADOOP-12238) Passing project option to releasedocmaker.py for running mvn site -Preleasedocs
[ https://issues.apache.org/jira/browse/HADOOP-12238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Allen Wittenauer resolved HADOOP-12238. --- Resolution: Duplicate There's already a JIRA covering moving trunk's releasedoc support to Yetus. Passing project option to releasedocmaker.py for running mvn site -Preleasedocs --- Key: HADOOP-12238 URL: https://issues.apache.org/jira/browse/HADOOP-12238 Project: Hadoop Common Issue Type: Sub-task Components: yetus Reporter: Tsuyoshi Ozawa Assignee: Tsuyoshi Ozawa Attachments: HADOOP-12238.001.patch Currently we cannot run mvn site -Preleasedocs on branch for HADOOP-12111. This patch fix it. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
Re: [DISCUSS] Additional maintenance releases for Hadoop 2.y versions
I believe there was general consensus to do more maintenance releases, as witnessed in the other thread. There have been discussions on what should go into 2.x.1, 2.x.2, etc., but I don't think we have a clear proposal. It would be nice to put that together, so committers know where all to commit patches to. Otherwise, release managers will have to look through branch-2 and pull in the fixes. Either approach is fine by me, but would be nice to start a [DISCUSS] thread on how to go about this. Any takers? On Wed, Jul 15, 2015 at 8:57 AM, Sangjin Lee sjl...@gmail.com wrote: Strong +1 for having a 2.6.1 release. I understand Vinod has been trying to get that effort going but it's been stalled a little bit. It would be good to rekindle that effort. Companies with big hadoop 2.x deployments (including mine) have always tried to stabilize a 2.x release by testing/collecting/researching critical issues on the release. Each would come up with its own set of fixes to backport. We would also communicate it via offline channels. During the hadoop summit, we thought it would be great if we all came together and create a public stability/bugfix release on top of 2.x (2.6.1 for 2.6 for example) with all the critical issues fixed. Thanks, Sangjin On Tue, Jul 14, 2015 at 10:42 PM, Tsuyoshi Ozawa oz...@apache.org wrote: Thank you for the notification. Trying to back port bug fixes. - Tsuyoshi On Wed, Jul 15, 2015 at 3:45 AM, Sean Busbey bus...@cloudera.com wrote: Hi Hadoopers! Over in HBase we've been discussing the impact of our dependencies on our downstream users. As our most fundamental dependency, Hadoop plays a big role in the operational cost of running an HBase instance. Currently the HBase 1.y release line supports Hadoop 2.4, 2.5, and 2.6[1]. We don't drop Hadoop minor release lines in minor releases so we are unlikely remove anything from this set until HBase 2.0, probably at the end of 2015 / start of 2016 (and currently we plan to continue supporting at least 2.4 for HBase 2.0 [2]). Lately we've been discussing updating our shipped binaries to Hadoop 2.6, following some stability testing by part of our community[3]. Unfortunately, 2.6.0 in particular has a couple of bugs that could destroy HBase clusters should users decide to turn on HDFS encryption[4]. Our installation instructions tell folks to replace these jars with the version of Hadoop they are actually running, but not all users follow those instructions so we want to minimize the pain for them. Regular maintenance releases are key to keeping operational burdens low for our downstream users; we don't want them to be forced to choose between living with broken systems and stomaching the risk of upgrades across minor/major version numbers. Looking back over the three aforementioned Hadoop versions, 2.6 hasn't had a patch release since 2.6.0 came out in Nov 2014, when 2.5 had its last patch release as well. Hadoop 2.4 looks to be a year without a release[5]. On our discussion of shipping Hadoop 2.6 binaries, one of your PMC members mentioned that with continued work on the 2.7 line y'all weren't planning any additional releases of the earlier minor versions[6]. The HBase community requests that Hadoop pick up making bug-fix-only patch releases again on a regular schedule[7]. Preferably on the 2.6 line and preferably monthly. We realize that given the time gap since 2.6.0 it will likely take a big to get 2.6.1 together, but after that it should take much less effort to continue. [1]: http://hbase.apache.org/book.html#hadoop [2]: http://s.apache.org/ReP [3]: HBASE-13339 [4]: HADOOP-11674 and HADOOP-11710 [5]: http://hadoop.apache.org/releases.html [6]: http://s.apache.org/MTY [7]: http://s.apache.org/ViP -- Sean -- Karthik Kambatla Software Engineer, Cloudera Inc. http://five.sentenc.es
