http://git-wip-us.apache.org/repos/asf/hadoop/blob/d759b4bd/hadoop-common-project/hadoop-common/src/site/markdown/release/0.20.1/RELEASENOTES.0.20.1.md ---------------------------------------------------------------------- diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/release/0.20.1/RELEASENOTES.0.20.1.md b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.20.1/RELEASENOTES.0.20.1.md new file mode 100644 index 0000000..afe74f1 --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.20.1/RELEASENOTES.0.20.1.md @@ -0,0 +1,218 @@ + +<!--- +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +--> +# Apache Hadoop 0.20.1 Release Notes + +These release notes cover new developer and user-facing incompatibilities, features, and major improvements. + + +--- + +* [HADOOP-6213](https://issues.apache.org/jira/browse/HADOOP-6213) | *Blocker* | **Remove commons dependency on commons-cli2** + +GenericOptionsParser in branch 0.20 depends on commons-cli2. This jira removes the dependency of branch 0.20 on commons-cli2 completely. The problem is seen after 'ant binary' where all the library files are copied to '$hadoop-home/lib' which already has commons-cli2. + + +--- + +* [HADOOP-6080](https://issues.apache.org/jira/browse/HADOOP-6080) | *Major* | **Handling of Trash with quota** + +Provide a new option to rm and rmr, -skipTrash, which will immediately delete the files specified, rather than moving them to the trash. + + +--- + +* [HADOOP-5924](https://issues.apache.org/jira/browse/HADOOP-5924) | *Major* | **JT fails to recover the jobs after restart after HADOOP:4372** + +Post HADOOP-4372, empty job history files caused NPE. This issues fixes that by creating new files if no old file is found. + + +--- + +* [HADOOP-5921](https://issues.apache.org/jira/browse/HADOOP-5921) | *Major* | **JobTracker does not come up because of NotReplicatedYetException** + +Jobtracker crashes if it fails to create jobtracker.info file (i.e if sufficient datanodes are not up). With this patch it keeps on retrying on IOExceptions assuming IOExceptions in jobtracker.info creation implies that the hdfs is not in *ready *state. + + +--- + +* [HADOOP-5920](https://issues.apache.org/jira/browse/HADOOP-5920) | *Major* | **TestJobHistory fails some times.** + +TestJobHistory fails as jobtracker is restarted very fast (within a minute) and history files from earlier testcases were not cleaned up. This patch cleans up the history-dir and mapred-system-dir after every test. + + +--- + +* [HADOOP-5884](https://issues.apache.org/jira/browse/HADOOP-5884) | *Major* | **Capacity scheduler should account high memory jobs as using more capacity of the queue** + +Fixes Capacity scheduler to account more capacity of a queue for a high memory job. Done by considering these jobs to +take more slots proportionally with respect to a slot's default memory size. + + +--- + +* [HADOOP-5881](https://issues.apache.org/jira/browse/HADOOP-5881) | *Major* | **Simplify configuration related to task-memory-monitoring and memory-based scheduling** + +**WARNING: No release note provided for this incompatible change.** + + +--- + +* [HADOOP-5746](https://issues.apache.org/jira/browse/HADOOP-5746) | *Major* | **Errors encountered in MROutputThread after the last map/reduce call can go undetected** + +If the child (streaming) process returns successfully and the MROutputThread throws an error, there was no way to detect that as all the IOExceptions was ignored. Such issues can occur when DFS clients were closed etc. Now a check for errors (in threads) is made before finishing off the task and an exception is thrown that fails he task. + + +--- + +* [HADOOP-5726](https://issues.apache.org/jira/browse/HADOOP-5726) | *Major* | **Remove pre-emption from the capacity scheduler code base** + +Removed pre-emption from capacity scheduler. The impact of this change is that capacities for queues can no longer be guaranteed within a given span of time. Also changed configuration variables to remove pre-emption related variables and better reflect the absence of guarantees. + + +--- + +* [HADOOP-5210](https://issues.apache.org/jira/browse/HADOOP-5210) | *Minor* | **Reduce Task Progress shows \> 100% when the total size of map outputs (for a single reducer) is high** + +This patch resets the variable totalBytesProcessed before the final merge sothat it will be used for calculating the progress of reducePhase(the 3rd phase of reduce task) correctly. + + +--- + +* [HADOOP-3315](https://issues.apache.org/jira/browse/HADOOP-3315) | *Major* | **New binary file format** + +Add a new, binary file format TFile. + + +--- + +* [MAPREDUCE-838](https://issues.apache.org/jira/browse/MAPREDUCE-838) | *Blocker* | **Task succeeds even when committer.commitTask fails with IOException** + +Fixed a bug in the way commit of task outputs happens. The bug was that if commit fails with IOException, the task would be declared as successful. + + +--- + +* [MAPREDUCE-834](https://issues.apache.org/jira/browse/MAPREDUCE-834) | *Major* | **When TaskTracker config use old memory management values its memory monitoring is diabled.** + +The tasktracker's startup code was modified to use deprecated memory management configuration variables, when specified, and enable memory monitoring of tasks. + + +--- + +* [MAPREDUCE-832](https://issues.apache.org/jira/browse/MAPREDUCE-832) | *Major* | **Too many WARN messages about deprecated memorty config variables in JobTacker log** + +Reduced the frequency of log messages printed when a deprecated memory management variable is found in configuration of a job. + + +--- + +* [MAPREDUCE-818](https://issues.apache.org/jira/browse/MAPREDUCE-818) | *Minor* | **org.apache.hadoop.mapreduce.Counters.getGroup returns null if the group name doesnt exist.** + +Fixed a bug in the new org.apache.hadoop.mapreduce.Counters.getGroup() method to return an empty group if group name doesn't exist, instead of null, thus making sure that it is in sync with the Javadoc. + + +--- + +* [MAPREDUCE-807](https://issues.apache.org/jira/browse/MAPREDUCE-807) | *Blocker* | **Stray user files in mapred.system.dir with permissions other than 777 can prevent the jobtracker from starting up.** + +The JobTracker tries to delete the mapred.system.dir when it is starting up (with the job recovery disabled). The fix provided by this jira is that JobTracker will fail (bail out) with AccessControlException if it fails to delete files/directories in mapred.system.dir due to access control issues. + + +--- + +* [MAPREDUCE-805](https://issues.apache.org/jira/browse/MAPREDUCE-805) | *Major* | **Deadlock in Jobtracker** + +Job initialization process was changed to not change (run) states during initialization. The reason is two fold +- this can lead to deadlock as state changes require circular locking (i.e JobInProgress requires JobTracker lock) +- events were not raised as these state changes were not informed/propogated back to the JobTracker + +Now the JobTracker takes care of initializing/failing/killing the job and raising appropriate events. The simple rule that was enforced was that "The JobTracker lock is *must* before changing the run-state of a job". + + +--- + +* [MAPREDUCE-796](https://issues.apache.org/jira/browse/MAPREDUCE-796) | *Major* | **Encountered "ClassCastException" on tasktracker while running wordcount with MultithreadedMapRunner** + +Multithreaded mapper was modified to create a new Runtime exception (object) from a throwable instead of casting a throwable into a RuntimeException, once the Multithreaded map encounters a fault. + + +--- + +* [MAPREDUCE-767](https://issues.apache.org/jira/browse/MAPREDUCE-767) | *Major* | **to remove mapreduce dependency on commons-cli2** + +Removes the dependency of hadoop-mapred from commons-cli2 and uses commons-cli1.2 for command-line parsing. + + +--- + +* [MAPREDUCE-745](https://issues.apache.org/jira/browse/MAPREDUCE-745) | *Major* | **TestRecoveryManager fails sometimes** + +JobTracker was changed to take an identifier as an argument. This helps in testcases where the jobtracker/mapred-cluster is (re)started in a short span of time and the chances of jobtracker identifier clashing are high. Also the RecoveryManager was modified to throw an exception if a job fails in init during the recovery process. The reason being that this event will trigger a job failure in the recovery process and will remove the failed job from further initialization and processing. + + +--- + +* [MAPREDUCE-657](https://issues.apache.org/jira/browse/MAPREDUCE-657) | *Major* | **CompletedJobStatusStore hardcodes filesystem to hdfs** + +CompletedJobStatusStore was hardcored to persist to hdfs. This patch allows to persist to local fs. Just qualify mapred.job.tracker.persist.jobstatus.dir with file:// + + +--- + +* [MAPREDUCE-430](https://issues.apache.org/jira/browse/MAPREDUCE-430) | *Major* | **Task stuck in cleanup with OutOfMemoryErrors** + +Various code paths in the framework caught Throwable and tried to do inline cleanup. In case of OOM errors, such inline-cleanups can result into hung jvms. With this fix, the TaskTracker provides a api to report fatal errors (any throwable other than FSErrror and Exceptions). On catching a Throwable, Mapper/Reducer tries to inform the TT. + + +--- + +* [MAPREDUCE-383](https://issues.apache.org/jira/browse/MAPREDUCE-383) | *Major* | **pipes combiner does not reset properly after a spill** + +Fixed a bug in Pipes combiner to reset the spilled bytes count after the spill. + + +--- + +* [MAPREDUCE-130](https://issues.apache.org/jira/browse/MAPREDUCE-130) | *Major* | **Delete the jobconf copy from the log directory of the JobTracker when the job is retired** + +When a job is initialized, it localizes the job conf to the logs dir. Without this patch I never gets deleted. Now when the job retires, the conf is deleted. This local copy is required to display on the webui. + + +--- + +* [MAPREDUCE-40](https://issues.apache.org/jira/browse/MAPREDUCE-40) | *Blocker* | **Memory management variables need a backwards compatibility option after HADOOP-5881** + +Fixed backwards compatibility by re-introducing and deprecating removed memory monitoring related configuration options. + + +--- + +* [MAPREDUCE-18](https://issues.apache.org/jira/browse/MAPREDUCE-18) | *Blocker* | **Under load the shuffle sometimes gets incorrect data** + +This patch adds the mapid and reduceid in the http header of mapoutput when being sent to reduce node. Also validates compressed length, decompressed length, mapid and reduceid from http header at reduce node. + + +--- + +* [MAPREDUCE-2](https://issues.apache.org/jira/browse/MAPREDUCE-2) | *Major* | **ArrayOutOfIndex error in KeyFieldBasedPartitioner on empty key** + +KeyFieldBasedPartitioner throws ArrayOutOfIndex when passed an empty key. This patch hashes empty key to 0 hashcode. + + +
http://git-wip-us.apache.org/repos/asf/hadoop/blob/d759b4bd/hadoop-common-project/hadoop-common/src/site/markdown/release/0.20.2/CHANGES.0.20.2.md ---------------------------------------------------------------------- diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/release/0.20.2/CHANGES.0.20.2.md b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.20.2/CHANGES.0.20.2.md new file mode 100644 index 0000000..fb21db9 --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.20.2/CHANGES.0.20.2.md @@ -0,0 +1,109 @@ + +<!--- +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +--> +# Apache Hadoop Changelog + +## Release 0.20.2 - 2010-02-16 + +### INCOMPATIBLE CHANGES: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | +| [HDFS-793](https://issues.apache.org/jira/browse/HDFS-793) | DataNode should first receive the whole packet ack message before it constructs and sends its own ack message for the packet | Blocker | datanode | Hairong Kuang | Hairong Kuang | + + +### NEW FEATURES: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | + + +### IMPROVEMENTS: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | +| [HADOOP-1849](https://issues.apache.org/jira/browse/HADOOP-1849) | IPC server max queue size should be configurable | Major | ipc | Raghu Angadi | Konstantin Shvachko | +| [MAPREDUCE-623](https://issues.apache.org/jira/browse/MAPREDUCE-623) | Resolve javac warnings in mapred | Major | build | Jothi Padmanabhan | Jothi Padmanabhan | + + +### BUG FIXES: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | +| [HADOOP-6576](https://issues.apache.org/jira/browse/HADOOP-6576) | TestStreamingStatus is failing on 0.20 branch | Major | . | Chris Douglas | Todd Lipcon | +| [HADOOP-6575](https://issues.apache.org/jira/browse/HADOOP-6575) | Tests do not run on 0.20 branch | Major | . | Chris Douglas | Chris Douglas | +| [HADOOP-6524](https://issues.apache.org/jira/browse/HADOOP-6524) | Contrib tests are failing Clover'ed build | Major | build | Konstantin Boudnik | Konstantin Boudnik | +| [HADOOP-6506](https://issues.apache.org/jira/browse/HADOOP-6506) | Failing tests prevent the rest of test targets from execution. | Major | build | Konstantin Boudnik | Konstantin Boudnik | +| [HADOOP-6498](https://issues.apache.org/jira/browse/HADOOP-6498) | IPC client bug may cause rpc call hang | Blocker | ipc | Ruyue Ma | Ruyue Ma | +| [HADOOP-6460](https://issues.apache.org/jira/browse/HADOOP-6460) | Namenode runs of out of memory due to memory leak in ipc Server | Blocker | . | Suresh Srinivas | Suresh Srinivas | +| [HADOOP-6428](https://issues.apache.org/jira/browse/HADOOP-6428) | HttpServer sleeps with negative values | Major | . | Tsz Wo Nicholas Sze | Konstantin Boudnik | +| [HADOOP-6315](https://issues.apache.org/jira/browse/HADOOP-6315) | GzipCodec should not represent BuiltInZlibInflater as decompressorType | Major | io | Aaron Kimball | Aaron Kimball | +| [HADOOP-6269](https://issues.apache.org/jira/browse/HADOOP-6269) | Missing synchronization for defaultResources in Configuration.addResource | Major | conf | Todd Lipcon | Sreekanth Ramakrishnan | +| [HADOOP-6231](https://issues.apache.org/jira/browse/HADOOP-6231) | Allow caching of filesystem instances to be disabled on a per-instance basis | Major | fs | Tom White | Ben Slusky | +| [HADOOP-6097](https://issues.apache.org/jira/browse/HADOOP-6097) | Multiple bugs w/ Hadoop archives | Major | fs | Ben Slusky | Ben Slusky | +| [HADOOP-5759](https://issues.apache.org/jira/browse/HADOOP-5759) | IllegalArgumentException when CombineFileInputFormat is used as job InputFormat | Major | . | Amareshwari Sriramadasu | Amareshwari Sriramadasu | +| [HADOOP-5623](https://issues.apache.org/jira/browse/HADOOP-5623) | Streaming: process provided status messages are overwritten every 10 seoncds | Major | . | Rick Cox | Rick Cox | +| [HADOOP-5612](https://issues.apache.org/jira/browse/HADOOP-5612) | Some c++ scripts are not chmodded before ant execution | Major | build | Todd Lipcon | Todd Lipcon | +| [HADOOP-5611](https://issues.apache.org/jira/browse/HADOOP-5611) | C++ libraries do not build on Debian Lenny | Critical | . | Todd Lipcon | Todd Lipcon | +| [HDFS-927](https://issues.apache.org/jira/browse/HDFS-927) | DFSInputStream retries too many times for new block locations | Critical | hdfs-client | Todd Lipcon | Todd Lipcon | +| [HDFS-872](https://issues.apache.org/jira/browse/HDFS-872) | DFSClient 0.20.1 is incompatible with HDFS 0.20.2 | Major | datanode, hdfs-client | Bassam Tabbara | Todd Lipcon | +| [HDFS-781](https://issues.apache.org/jira/browse/HDFS-781) | Metrics PendingDeletionBlocks is not decremented | Blocker | namenode | Suresh Srinivas | Suresh Srinivas | +| [HDFS-761](https://issues.apache.org/jira/browse/HDFS-761) | Failure to process rename operation from edits log due to quota verification | Major | namenode | Suresh Srinivas | Suresh Srinivas | +| [HDFS-745](https://issues.apache.org/jira/browse/HDFS-745) | TestFsck timeout on 0.20. | Major | . | Tsz Wo Nicholas Sze | Tsz Wo Nicholas Sze | +| [HDFS-732](https://issues.apache.org/jira/browse/HDFS-732) | HDFS files are ending up truncated | Blocker | hdfs-client | Christian Kunz | Tsz Wo Nicholas Sze | +| [HDFS-723](https://issues.apache.org/jira/browse/HDFS-723) | Deadlock in DFSClient#DFSOutputStream | Blocker | . | Hairong Kuang | Hairong Kuang | +| [HDFS-677](https://issues.apache.org/jira/browse/HDFS-677) | Rename failure due to quota results in deletion of src directory | Blocker | namenode | Suresh Srinivas | Suresh Srinivas | +| [HDFS-596](https://issues.apache.org/jira/browse/HDFS-596) | Memory leak in libhdfs: hdfsFreeFileInfo() in libhdfs does not free memory for mOwner and mGroup | Blocker | fuse-dfs | Zhang Bingjun | Zhang Bingjun | +| [HDFS-579](https://issues.apache.org/jira/browse/HDFS-579) | HADOOP-3792 update of DfsTask incomplete | Major | hdfs-client | Christian Kunz | Christian Kunz | +| [HDFS-187](https://issues.apache.org/jira/browse/HDFS-187) | TestStartup fails if hdfs is running in the same machine | Major | test | Tsz Wo Nicholas Sze | Todd Lipcon | +| [HDFS-185](https://issues.apache.org/jira/browse/HDFS-185) | Chown , chgrp , chmod operations allowed when namenode is in safemode . | Major | . | Ravi Phulari | Ravi Phulari | +| [HDFS-101](https://issues.apache.org/jira/browse/HDFS-101) | DFS write pipeline : DFSClient sometimes does not detect second datanode failure | Blocker | datanode | Raghu Angadi | Hairong Kuang | +| [MAPREDUCE-1251](https://issues.apache.org/jira/browse/MAPREDUCE-1251) | c++ utils doesn't compile | Major | . | Eli Collins | Eli Collins | +| [MAPREDUCE-1182](https://issues.apache.org/jira/browse/MAPREDUCE-1182) | Reducers fail with OutOfMemoryError while copying Map outputs | Blocker | . | Chandra Prakash Bhagtani | Chandra Prakash Bhagtani | +| [MAPREDUCE-1163](https://issues.apache.org/jira/browse/MAPREDUCE-1163) | hdfsJniHelper.h: Yahoo! specific paths are encoded | Trivial | . | Allen Wittenauer | Allen Wittenauer | +| [MAPREDUCE-1147](https://issues.apache.org/jira/browse/MAPREDUCE-1147) | Map output records counter missing for map-only jobs in new API | Blocker | . | Chris Douglas | Amar Kamat | +| [MAPREDUCE-1070](https://issues.apache.org/jira/browse/MAPREDUCE-1070) | Deadlock in FairSchedulerServlet | Major | . | Todd Lipcon | Todd Lipcon | +| [MAPREDUCE-1068](https://issues.apache.org/jira/browse/MAPREDUCE-1068) | In hadoop-0.20.0 streaming job do not throw proper verbose error message if file is not present | Major | contrib/streaming | Peeyush Bishnoi | Amareshwari Sriramadasu | +| [MAPREDUCE-1010](https://issues.apache.org/jira/browse/MAPREDUCE-1010) | Adding tests for changes in archives. | Minor | harchive | Mahadev konar | Mahadev konar | +| [MAPREDUCE-979](https://issues.apache.org/jira/browse/MAPREDUCE-979) | JobConf.getMemoryFor{Map\|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED\_MEMORY\_LIMIT of -1 | Blocker | jobtracker, tasktracker | Arun C Murthy | Sreekanth Ramakrishnan | +| [MAPREDUCE-826](https://issues.apache.org/jira/browse/MAPREDUCE-826) | harchive doesn't use ToolRunner / harchive returns 0 even if the job fails with exception | Trivial | harchive | Koji Noguchi | Koji Noguchi | +| [MAPREDUCE-617](https://issues.apache.org/jira/browse/MAPREDUCE-617) | Streaming should not throw java.lang.RuntimeException and ERROR while displaying help | Minor | contrib/streaming | Karam Singh | | +| [MAPREDUCE-433](https://issues.apache.org/jira/browse/MAPREDUCE-433) | TestReduceFetch failed. | Major | . | Tsz Wo Nicholas Sze | Chris Douglas | +| [MAPREDUCE-112](https://issues.apache.org/jira/browse/MAPREDUCE-112) | Reduce Input Records and Reduce Output Records counters are not being set when using the new Mapreduce reducer API | Blocker | . | Jothi Padmanabhan | Jothi Padmanabhan | + + +### TESTS: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | +| [HDFS-919](https://issues.apache.org/jira/browse/HDFS-919) | Create test to validate the BlocksVerified metric | Major | test | gary murry | | +| [HDFS-907](https://issues.apache.org/jira/browse/HDFS-907) | Add tests for getBlockLocations and totalLoad metrics. | Minor | namenode | Ravi Phulari | Ravi Phulari | + + +### SUB-TASKS: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | + + +### OTHER: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | + + http://git-wip-us.apache.org/repos/asf/hadoop/blob/d759b4bd/hadoop-common-project/hadoop-common/src/site/markdown/release/0.20.2/RELEASENOTES.0.20.2.md ---------------------------------------------------------------------- diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/release/0.20.2/RELEASENOTES.0.20.2.md b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.20.2/RELEASENOTES.0.20.2.md new file mode 100644 index 0000000..f363e0a --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.20.2/RELEASENOTES.0.20.2.md @@ -0,0 +1,150 @@ + +<!--- +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +--> +# Apache Hadoop 0.20.2 Release Notes + +These release notes cover new developer and user-facing incompatibilities, features, and major improvements. + + +--- + +* [HADOOP-6498](https://issues.apache.org/jira/browse/HADOOP-6498) | *Blocker* | **IPC client bug may cause rpc call hang** + +Correct synchronization error in IPC where handler thread could hang if request reader got an error. + + +--- + +* [HADOOP-6460](https://issues.apache.org/jira/browse/HADOOP-6460) | *Blocker* | **Namenode runs of out of memory due to memory leak in ipc Server** + +If an IPC server response buffer has grown to than 1MB, it is replaced by a smaller buffer to free up the Java heap that was used. This will improve the longevity of the name service. + + +--- + +* [HADOOP-6428](https://issues.apache.org/jira/browse/HADOOP-6428) | *Major* | **HttpServer sleeps with negative values** + +Corrected arithmetic error that made sleep times less than zero. + + +--- + +* [HADOOP-6231](https://issues.apache.org/jira/browse/HADOOP-6231) | *Major* | **Allow caching of filesystem instances to be disabled on a per-instance basis** + +Allow a general mechanism to disable the cache on a per filesystem basis by using property fs.\<schemename\>.impl.disable.cache. eg. fs.har.impl.disable.cache in core-default.xml + + +--- + +* [HADOOP-6097](https://issues.apache.org/jira/browse/HADOOP-6097) | *Major* | **Multiple bugs w/ Hadoop archives** + +Bugs fixed for Hadoop archives: character escaping in paths, LineReader and file system caching. + + +--- + +* [HDFS-793](https://issues.apache.org/jira/browse/HDFS-793) | *Blocker* | **DataNode should first receive the whole packet ack message before it constructs and sends its own ack message for the packet** + +**WARNING: No release note provided for this incompatible change.** + + +--- + +* [HDFS-781](https://issues.apache.org/jira/browse/HDFS-781) | *Blocker* | **Metrics PendingDeletionBlocks is not decremented** + +Correct PendingDeletionBlocks metric to properly decrement counts. + + +--- + +* [HDFS-761](https://issues.apache.org/jira/browse/HDFS-761) | *Major* | **Failure to process rename operation from edits log due to quota verification** + +Corrected an error when checking quota policy that resulted in a failure to read the edits log, stopping the primary/secondary name node. + + +--- + +* [HDFS-677](https://issues.apache.org/jira/browse/HDFS-677) | *Blocker* | **Rename failure due to quota results in deletion of src directory** + +Rename properly considers the case where both source and destination are over quota; operation will fail with error indication. + + +--- + +* [HDFS-596](https://issues.apache.org/jira/browse/HDFS-596) | *Blocker* | **Memory leak in libhdfs: hdfsFreeFileInfo() in libhdfs does not free memory for mOwner and mGroup** + +Memory leak in function hdfsFreeFileInfo in libhdfs. This bug affects fuse-dfs severely. + + +--- + +* [MAPREDUCE-1182](https://issues.apache.org/jira/browse/MAPREDUCE-1182) | *Blocker* | **Reducers fail with OutOfMemoryError while copying Map outputs** + +Modifies shuffle related memory parameters to use 'long' from 'int' so that sizes greater than maximum integer size are handled correctly + + +--- + +* [MAPREDUCE-1147](https://issues.apache.org/jira/browse/MAPREDUCE-1147) | *Blocker* | **Map output records counter missing for map-only jobs in new API** + +Adds a counter to track the number of records emitted by map writing directly to HDFS i.e map tasks of job with 0 reducers. + + +--- + +* [MAPREDUCE-1068](https://issues.apache.org/jira/browse/MAPREDUCE-1068) | *Major* | **In hadoop-0.20.0 streaming job do not throw proper verbose error message if file is not present** + +Fix streaming job to show proper message if file is is not present, for -file option. + + +--- + +* [MAPREDUCE-979](https://issues.apache.org/jira/browse/MAPREDUCE-979) | *Blocker* | **JobConf.getMemoryFor{Map\|Reduce}Task doesn't fallback to newer config knobs when mapred.taskmaxvmem is set to DISABLED\_MEMORY\_LIMIT of -1** + +Added support to fallback to new task memory configuration when deprecated memory configuration values are set to disabled. + + +--- + +* [MAPREDUCE-826](https://issues.apache.org/jira/browse/MAPREDUCE-826) | *Trivial* | **harchive doesn't use ToolRunner / harchive returns 0 even if the job fails with exception** + +Use ToolRunner for archives job and return non zero error code on failure. + + +--- + +* [MAPREDUCE-623](https://issues.apache.org/jira/browse/MAPREDUCE-623) | *Major* | **Resolve javac warnings in mapred** + +Removes javac warnings by either resolving them or suppressing them (wherever resolution is not possible) + + +--- + +* [MAPREDUCE-433](https://issues.apache.org/jira/browse/MAPREDUCE-433) | *Major* | **TestReduceFetch failed.** + +Resolves the test failure by modifying the test to base it on spill counters rather than on bytes read/written. It also introduces a new configuration parameter "mapred.job.shuffle.input.buffer.percent" to provide finer grained control on the memory limit to be used during shuffle. + + +--- + +* [MAPREDUCE-112](https://issues.apache.org/jira/browse/MAPREDUCE-112) | *Blocker* | **Reduce Input Records and Reduce Output Records counters are not being set when using the new Mapreduce reducer API** + +Updates of counters for reduce input and output records were added in the new API so they are available for jobs using the new API. + + + http://git-wip-us.apache.org/repos/asf/hadoop/blob/d759b4bd/hadoop-common-project/hadoop-common/src/site/markdown/release/0.20.203.0/CHANGES.0.20.203.0.md ---------------------------------------------------------------------- diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/release/0.20.203.0/CHANGES.0.20.203.0.md b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.20.203.0/CHANGES.0.20.203.0.md new file mode 100644 index 0000000..d951eef --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.20.203.0/CHANGES.0.20.203.0.md @@ -0,0 +1,95 @@ + +<!--- +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +--> +# Apache Hadoop Changelog + +## Release 0.20.203.0 - 2011-05-11 + +### INCOMPATIBLE CHANGES: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | + + +### NEW FEATURES: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | +| [HADOOP-7108](https://issues.apache.org/jira/browse/HADOOP-7108) | hadoop-0.20.100 | Major | . | Arun C Murthy | Arun C Murthy | +| [HADOOP-4343](https://issues.apache.org/jira/browse/HADOOP-4343) | Adding user and service-to-service authentication to Hadoop | Blocker | . | Kan Zhang | Kan Zhang | + + +### IMPROVEMENTS: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | +| [HADOOP-7247](https://issues.apache.org/jira/browse/HADOOP-7247) | Fix documentation to reflect new jar names | Major | . | Owen O'Malley | Owen O'Malley | +| [HDFS-1626](https://issues.apache.org/jira/browse/HDFS-1626) | Make BLOCK\_INVALIDATE\_LIMIT configurable | Minor | namenode | Arun C Murthy | Tsz Wo Nicholas Sze | +| [HDFS-457](https://issues.apache.org/jira/browse/HDFS-457) | better handling of volume failure in Data Node storage | Major | datanode | Boris Shkolnik | Boris Shkolnik | +| [MAPREDUCE-2355](https://issues.apache.org/jira/browse/MAPREDUCE-2355) | Add an out of band heartbeat damper | Major | jobtracker | Owen O'Malley | Arun C Murthy | +| [MAPREDUCE-2316](https://issues.apache.org/jira/browse/MAPREDUCE-2316) | Update docs for CapacityScheduler | Major | capacity-sched, documentation | Arun C Murthy | Arun C Murthy | +| [MAPREDUCE-1943](https://issues.apache.org/jira/browse/MAPREDUCE-1943) | Implement limits on per-job JobConf, Counters, StatusReport, Split-Sizes | Major | . | Mahadev konar | Mahadev konar | +| [MAPREDUCE-478](https://issues.apache.org/jira/browse/MAPREDUCE-478) | separate jvm param for mapper and reducer | Minor | . | Koji Noguchi | Arun C Murthy | + + +### BUG FIXES: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | +| [HADOOP-7259](https://issues.apache.org/jira/browse/HADOOP-7259) | contrib modules should include build.properties from parent. | Major | build | Owen O'Malley | Owen O'Malley | +| [HADOOP-7258](https://issues.apache.org/jira/browse/HADOOP-7258) | Gzip codec should not return null decompressors | Major | . | Owen O'Malley | Owen O'Malley | +| [HADOOP-7253](https://issues.apache.org/jira/browse/HADOOP-7253) | Fix default config | Major | . | Owen O'Malley | Owen O'Malley | +| [HADOOP-7246](https://issues.apache.org/jira/browse/HADOOP-7246) | The default log4j configuration causes warnings about EventCounter | Major | . | Owen O'Malley | Luke Lu | +| [HADOOP-7243](https://issues.apache.org/jira/browse/HADOOP-7243) | Fix contrib unit tests (fairshare, hdfsproxy, datajoin, streaming) | Major | . | Owen O'Malley | Owen O'Malley | +| [HADOOP-7232](https://issues.apache.org/jira/browse/HADOOP-7232) | Fix javadoc warnings | Blocker | documentation | Owen O'Malley | Owen O'Malley | +| [HADOOP-7215](https://issues.apache.org/jira/browse/HADOOP-7215) | RPC clients must connect over a network interface corresponding to the host name in the client's kerberos principal key | Blocker | security | Suresh Srinivas | Suresh Srinivas | +| [HADOOP-7190](https://issues.apache.org/jira/browse/HADOOP-7190) | Put metrics v1 back into the hadoop-20-security branch | Major | metrics | Owen O'Malley | Owen O'Malley | +| [HADOOP-7163](https://issues.apache.org/jira/browse/HADOOP-7163) | "java.net.SocketTimeoutException: 60000 millis timeout" happens a lot | Major | ipc | Owen O'Malley | Devaraj Das | +| [HADOOP-7143](https://issues.apache.org/jira/browse/HADOOP-7143) | Hive Hadoop20SShims depends on removed HadoopArchives | Major | fs | Joep Rottinghuis | Joep Rottinghuis | +| [HADOOP-7040](https://issues.apache.org/jira/browse/HADOOP-7040) | DiskChecker:mkdirsWithExistsCheck swallows FileNotFoundException. | Major | . | Boris Shkolnik | Boris Shkolnik | +| [HADOOP-6907](https://issues.apache.org/jira/browse/HADOOP-6907) | Rpc client doesn't use the per-connection conf to figure out server's Kerberos principal | Major | ipc, security | Kan Zhang | Kan Zhang | +| [HADOOP-5647](https://issues.apache.org/jira/browse/HADOOP-5647) | TestJobHistory fails if /tmp/\_logs is not writable to. Testcase should not depend on /tmp | Major | test | Ravi Gummadi | Ravi Gummadi | +| [HDFS-1822](https://issues.apache.org/jira/browse/HDFS-1822) | Editlog opcodes overlap between 20 security and later releases | Blocker | namenode | Suresh Srinivas | Suresh Srinivas | +| [HDFS-1022](https://issues.apache.org/jira/browse/HDFS-1022) | Merge under-10-min tests specs into one file | Major | test | Erik Steffl | Erik Steffl | +| [MAPREDUCE-2365](https://issues.apache.org/jira/browse/MAPREDUCE-2365) | Add counters for FileInputFormat (BYTES\_READ) and FileOutputFormat (BYTES\_WRITTEN) | Major | . | Owen O'Malley | Siddharth Seth | +| [MAPREDUCE-2278](https://issues.apache.org/jira/browse/MAPREDUCE-2278) | DistributedCache shouldn't hold a ref to JobConf | Major | distributed-cache, tasktracker | Arun C Murthy | Chris Douglas | +| [MAPREDUCE-1699](https://issues.apache.org/jira/browse/MAPREDUCE-1699) | JobHistory shouldn't be disabled for any reason | Major | jobtracker | Arun C Murthy | Krishna Ramachandran | +| [MAPREDUCE-1280](https://issues.apache.org/jira/browse/MAPREDUCE-1280) | Eclipse Plugin does not work with Eclipse Ganymede (3.4) | Major | . | Aaron Kimball | Alex Kozlov | +| [MAPREDUCE-1233](https://issues.apache.org/jira/browse/MAPREDUCE-1233) | Incorrect Waiting maps/reduces in Jobtracker metrics | Major | jobtracker | V.Karthikeyan | Luke Lu | +| [MAPREDUCE-1118](https://issues.apache.org/jira/browse/MAPREDUCE-1118) | Capacity Scheduler scheduling information is hard to read / should be tabular format | Major | capacity-sched | Allen Wittenauer | Krishna Ramachandran | +| [MAPREDUCE-323](https://issues.apache.org/jira/browse/MAPREDUCE-323) | Improve the way job history files are managed | Critical | jobtracker | Amar Kamat | Dick King | + + +### TESTS: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | + + +### SUB-TASKS: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | + + +### OTHER: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | + + http://git-wip-us.apache.org/repos/asf/hadoop/blob/d759b4bd/hadoop-common-project/hadoop-common/src/site/markdown/release/0.20.203.0/RELEASENOTES.0.20.203.0.md ---------------------------------------------------------------------- diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/release/0.20.203.0/RELEASENOTES.0.20.203.0.md b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.20.203.0/RELEASENOTES.0.20.203.0.md new file mode 100644 index 0000000..1c1dd4c --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.20.203.0/RELEASENOTES.0.20.203.0.md @@ -0,0 +1,117 @@ + +<!--- +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +--> +# Apache Hadoop 0.20.203.0 Release Notes + +These release notes cover new developer and user-facing incompatibilities, features, and major improvements. + + +--- + +* [HADOOP-5647](https://issues.apache.org/jira/browse/HADOOP-5647) | *Major* | **TestJobHistory fails if /tmp/\_logs is not writable to. Testcase should not depend on /tmp** + +Removed dependency of testcase on /tmp and made it to use test.build.data directory instead. + + +--- + +* [HDFS-1626](https://issues.apache.org/jira/browse/HDFS-1626) | *Minor* | **Make BLOCK\_INVALIDATE\_LIMIT configurable** + +Added a new configuration property dfs.block.invalidate.limit for FSNamesystem.blockInvalidateLimit. + + +--- + +* [HDFS-457](https://issues.apache.org/jira/browse/HDFS-457) | *Major* | **better handling of volume failure in Data Node storage** + +Datanode can continue if a volume for replica storage fails. Previously a datanode resigned if any volume failed. + + +--- + +* [MAPREDUCE-1118](https://issues.apache.org/jira/browse/MAPREDUCE-1118) | *Major* | **Capacity Scheduler scheduling information is hard to read / should be tabular format** + +Add CapacityScheduler servlet to enhance web UI for queue information. + + +--- + +* [MAPREDUCE-478](https://issues.apache.org/jira/browse/MAPREDUCE-478) | *Minor* | **separate jvm param for mapper and reducer** + +Allow map and reduce jvm parameters, environment variables and ulimit to be set separately. + +Configuration changes: + add mapred.map.child.java.opts + add mapred.reduce.child.java.opts + add mapred.map.child.env + add mapred.reduce.child.ulimit + add mapred.map.child.env + add mapred.reduce.child.ulimit + deprecated mapred.child.java.opts + deprecated mapred.child.env + deprecated mapred.child.ulimit + + +--- + +* [MAPREDUCE-323](https://issues.apache.org/jira/browse/MAPREDUCE-323) | *Critical* | **Improve the way job history files are managed** + +This patch does four things: + + * it changes the directory structure of the done directory that holds history logs for jobs that are completed, + * it builds toy databases for completed jobs, so we no longer have to scan 2N files on DFS to find out facts about the N jobs that have completed since the job tracker started [which can be hundreds of thousands of files in practical cases], + * it changes the job history browser to display more information and allow more filtering criteria, and + * it creates a new programmatic interface for finding files matching user-chosen criteria. This allows users to no longer be concerned with our methods of storing them, in turn allowing us to change those at will. + +The new API described above, which can be used to programmatically obtain history file PATHs given search criteria, is described below: + + package org.apache.hadoop.mapreduce.jobhistory; + ... + + // this interface is within O.A.H.mapreduce.jobhistory.JobHistory: + + // holds information about one job hostory log in the done + // job history logs + public static class JobHistoryJobRecord { + public Path getPath() { ... } + public String getJobIDString() { ... } + public long getSubmitTime() { ... } + public String getUserName() { ... } + public String getJobName() { ... } + } + + public class JobHistoryRecordRetriever implements Iterator\<JobHistoryJobRecord\> { + // usual Interface methods -- remove() throws UnsupportedOperationException + // returns the number of calls to next() that will succeed + public int numMatches() { ... } + } + + // returns a JobHistoryRecordRetriever that delivers all Path's of job matching job history files, + // in no particular order. Any criterion that is null or the empty string does not constrain. + // All criteria that are specified are applied conjunctively, except that if there's more than + // one date you retrieve all Path's matching ANY date. + // soughtUser and soughtJobid must match exactly. + // soughtJobName can match the entire job name or any substring. + // dates must be in the format exactly MM/DD/YYYY . + // Dates' leading digits must be 2's . We're incubating a Y3K problem. + public JobHistoryRecordRetriever getMatchingJob + (String soughtUser, String soughtJobName, String[] dateStrings, String soughtJobid) + throws IOException + + + http://git-wip-us.apache.org/repos/asf/hadoop/blob/d759b4bd/hadoop-common-project/hadoop-common/src/site/markdown/release/0.20.203.1/CHANGES.0.20.203.1.md ---------------------------------------------------------------------- diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/release/0.20.203.1/CHANGES.0.20.203.1.md b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.20.203.1/CHANGES.0.20.203.1.md new file mode 100644 index 0000000..def258c --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.20.203.1/CHANGES.0.20.203.1.md @@ -0,0 +1,66 @@ + +<!--- +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +--> +# Apache Hadoop Changelog + +## Release 0.20.203.1 - Unreleased + +### INCOMPATIBLE CHANGES: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | + + +### NEW FEATURES: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | + + +### IMPROVEMENTS: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | + + +### BUG FIXES: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | +| [HADOOP-7330](https://issues.apache.org/jira/browse/HADOOP-7330) | The metrics source mbean implementation should return the attribute value instead of the object | Major | metrics | Luke Lu | Luke Lu | +| [HADOOP-7297](https://issues.apache.org/jira/browse/HADOOP-7297) | Error in the documentation regarding Checkpoint/Backup Node | Trivial | documentation | arnaud p | Harsh J | + + +### TESTS: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | + + +### SUB-TASKS: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | + + +### OTHER: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | + + http://git-wip-us.apache.org/repos/asf/hadoop/blob/d759b4bd/hadoop-common-project/hadoop-common/src/site/markdown/release/0.20.203.1/RELEASENOTES.0.20.203.1.md ---------------------------------------------------------------------- diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/release/0.20.203.1/RELEASENOTES.0.20.203.1.md b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.20.203.1/RELEASENOTES.0.20.203.1.md new file mode 100644 index 0000000..5827567 --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.20.203.1/RELEASENOTES.0.20.203.1.md @@ -0,0 +1,24 @@ + +<!--- +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +--> +# Apache Hadoop 0.20.203.1 Release Notes + +These release notes cover new developer and user-facing incompatibilities, features, and major improvements. + + + http://git-wip-us.apache.org/repos/asf/hadoop/blob/d759b4bd/hadoop-common-project/hadoop-common/src/site/markdown/release/0.20.204.0/CHANGES.0.20.204.0.md ---------------------------------------------------------------------- diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/release/0.20.204.0/CHANGES.0.20.204.0.md b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.20.204.0/CHANGES.0.20.204.0.md new file mode 100644 index 0000000..b7eb2b9 --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.20.204.0/CHANGES.0.20.204.0.md @@ -0,0 +1,127 @@ + +<!--- +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +--> +# Apache Hadoop Changelog + +## Release 0.20.204.0 - 2011-09-02 + +### INCOMPATIBLE CHANGES: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | +| [HADOOP-6255](https://issues.apache.org/jira/browse/HADOOP-6255) | Create an rpm integration project | Major | . | Owen O'Malley | Eric Yang | + + +### NEW FEATURES: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | +| [HADOOP-7398](https://issues.apache.org/jira/browse/HADOOP-7398) | create a mechanism to suppress the HADOOP\_HOME deprecated warning | Major | . | Owen O'Malley | Owen O'Malley | +| [HADOOP-7144](https://issues.apache.org/jira/browse/HADOOP-7144) | Expose JMX with something like JMXProxyServlet | Major | . | Luke Lu | Robert Joseph Evans | +| [MAPREDUCE-2558](https://issues.apache.org/jira/browse/MAPREDUCE-2558) | Add queue-level metrics 0.20-security branch | Major | jobtracker | Jeffrey Naisbitt | Jeffrey Naisbitt | +| [MAPREDUCE-1938](https://issues.apache.org/jira/browse/MAPREDUCE-1938) | Ability for having user's classes take precedence over the system classes for tasks' classpath | Blocker | job submission, task, tasktracker | Devaraj Das | Krishna Ramachandran | + + +### IMPROVEMENTS: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | +| [HADOOP-7277](https://issues.apache.org/jira/browse/HADOOP-7277) | Add Eclipse launch tasks for the 0.20-security branch | Minor | build | Jeffrey Naisbitt | Jeffrey Naisbitt | +| [HADOOP-7248](https://issues.apache.org/jira/browse/HADOOP-7248) | Have a way to automatically update Eclipse .classpath file when new libs are added to the classpath through Ivy for 0.20-\* based sources | Minor | build | Konstantin Boudnik | Thomas Graves | +| [HDFS-1773](https://issues.apache.org/jira/browse/HDFS-1773) | Remove a datanode from cluster if include list is not empty and this datanode is removed from both include and exclude lists | Minor | namenode | Tanping Wang | Tanping Wang | +| [MAPREDUCE-2524](https://issues.apache.org/jira/browse/MAPREDUCE-2524) | Backport trunk heuristics for failing maps when we get fetch failures retrieving map output during shuffle | Minor | tasktracker | Thomas Graves | Thomas Graves | +| [MAPREDUCE-2495](https://issues.apache.org/jira/browse/MAPREDUCE-2495) | The distributed cache cleanup thread has no monitoring to check to see if it has died for some reason | Minor | distributed-cache | Robert Joseph Evans | Robert Joseph Evans | +| [MAPREDUCE-2490](https://issues.apache.org/jira/browse/MAPREDUCE-2490) | Log blacklist debug count | Trivial | jobtracker | Jonathan Eagles | Jonathan Eagles | +| [MAPREDUCE-2479](https://issues.apache.org/jira/browse/MAPREDUCE-2479) | Backport MAPREDUCE-1568 to hadoop security branch | Major | tasktracker | Robert Joseph Evans | Robert Joseph Evans | +| [MAPREDUCE-2456](https://issues.apache.org/jira/browse/MAPREDUCE-2456) | Show the reducer taskid and map/reduce tasktrackers for "Failed fetch notification #\_ for task attempt..." log messages | Trivial | jobtracker | Jeffrey Naisbitt | Jeffrey Naisbitt | + + +### BUG FIXES: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | +| [HADOOP-7475](https://issues.apache.org/jira/browse/HADOOP-7475) | hadoop-setup-single-node.sh is broken | Blocker | . | Eric Yang | Eric Yang | +| [HADOOP-7373](https://issues.apache.org/jira/browse/HADOOP-7373) | Tarball deployment doesn't work with {start,stop}-{dfs,mapred} | Major | . | Owen O'Malley | Owen O'Malley | +| [HADOOP-7364](https://issues.apache.org/jira/browse/HADOOP-7364) | TestMiniMRDFSCaching fails if test.build.dir is set to something other than build/test | Major | test | Thomas Graves | Thomas Graves | +| [HADOOP-7356](https://issues.apache.org/jira/browse/HADOOP-7356) | RPM packages broke bin/hadoop script for hadoop 0.20.205 | Blocker | . | Eric Yang | Eric Yang | +| [HADOOP-7330](https://issues.apache.org/jira/browse/HADOOP-7330) | The metrics source mbean implementation should return the attribute value instead of the object | Major | metrics | Luke Lu | Luke Lu | +| [HADOOP-7324](https://issues.apache.org/jira/browse/HADOOP-7324) | Ganglia plugins for metrics v2 | Blocker | metrics | Luke Lu | Priyo Mustafi | +| [HADOOP-7274](https://issues.apache.org/jira/browse/HADOOP-7274) | CLONE - IOUtils.readFully and IOUtils.skipFully have typo in exception creation's message | Minor | util | Jonathan Eagles | Jonathan Eagles | +| [HADOOP-7232](https://issues.apache.org/jira/browse/HADOOP-7232) | Fix javadoc warnings | Blocker | documentation | Owen O'Malley | Owen O'Malley | +| [HDFS-2057](https://issues.apache.org/jira/browse/HDFS-2057) | Wait time to terminate the threads causing unit tests to take longer time | Major | datanode | Bharath Mundlapudi | Bharath Mundlapudi | +| [HDFS-2023](https://issues.apache.org/jira/browse/HDFS-2023) | Backport of NPE for File.list and File.listFiles | Major | datanode | Bharath Mundlapudi | Bharath Mundlapudi | +| [HDFS-1878](https://issues.apache.org/jira/browse/HDFS-1878) | TestHDFSServerPorts unit test failure - race condition in FSNamesystem.close() causes NullPointerException without serious consequence | Minor | namenode | Matt Foley | Matt Foley | +| [HDFS-1822](https://issues.apache.org/jira/browse/HDFS-1822) | Editlog opcodes overlap between 20 security and later releases | Blocker | namenode | Suresh Srinivas | Suresh Srinivas | +| [HDFS-1758](https://issues.apache.org/jira/browse/HDFS-1758) | Web UI JSP pages thread safety issue | Minor | tools | Tanping Wang | Tanping Wang | +| [HDFS-1750](https://issues.apache.org/jira/browse/HDFS-1750) | fs -ls hftp://file not working | Major | . | Tsz Wo Nicholas Sze | Tsz Wo Nicholas Sze | +| [HDFS-1692](https://issues.apache.org/jira/browse/HDFS-1692) | In secure mode, Datanode process doesn't exit when disks fail. | Major | datanode | Bharath Mundlapudi | Bharath Mundlapudi | +| [HDFS-1592](https://issues.apache.org/jira/browse/HDFS-1592) | Datanode startup doesn't honor volumes.tolerated | Major | . | Bharath Mundlapudi | Bharath Mundlapudi | +| [HDFS-1377](https://issues.apache.org/jira/browse/HDFS-1377) | Quota bug for partial blocks allows quotas to be violated | Blocker | namenode | Eli Collins | Eli Collins | +| [HDFS-1258](https://issues.apache.org/jira/browse/HDFS-1258) | Clearing namespace quota on "/" corrupts FS image | Blocker | namenode | Aaron T. Myers | Aaron T. Myers | +| [HDFS-1189](https://issues.apache.org/jira/browse/HDFS-1189) | Quota counts missed between clear quota and set quota | Major | namenode | Kang Xiao | John George | +| [MAPREDUCE-2846](https://issues.apache.org/jira/browse/MAPREDUCE-2846) | a small % of all tasks fail with DefaultTaskController | Blocker | task, task-controller, tasktracker | Allen Wittenauer | Owen O'Malley | +| [MAPREDUCE-2804](https://issues.apache.org/jira/browse/MAPREDUCE-2804) | "Creation of symlink to attempt log dir failed." message is not useful | Blocker | . | Allen Wittenauer | Owen O'Malley | +| [MAPREDUCE-2651](https://issues.apache.org/jira/browse/MAPREDUCE-2651) | Race condition in Linux Task Controller for job log directory creation | Major | task-controller | Bharath Mundlapudi | Bharath Mundlapudi | +| [MAPREDUCE-2621](https://issues.apache.org/jira/browse/MAPREDUCE-2621) | TestCapacityScheduler fails with "Queue "q1" does not exist" | Minor | . | Sherry Chen | Sherry Chen | +| [MAPREDUCE-2555](https://issues.apache.org/jira/browse/MAPREDUCE-2555) | JvmInvalidate errors in the gridmix TT logs | Minor | tasktracker | Thomas Graves | Thomas Graves | +| [MAPREDUCE-2529](https://issues.apache.org/jira/browse/MAPREDUCE-2529) | Recognize Jetty bug 1342 and handle it | Major | tasktracker | Thomas Graves | Thomas Graves | +| [MAPREDUCE-2514](https://issues.apache.org/jira/browse/MAPREDUCE-2514) | ReinitTrackerAction class name misspelled RenitTrackerAction in task tracker log | Trivial | tasktracker | Jonathan Eagles | Jonathan Eagles | +| [MAPREDUCE-2451](https://issues.apache.org/jira/browse/MAPREDUCE-2451) | Log the reason string of healthcheck script | Trivial | jobtracker | Thomas Graves | Thomas Graves | +| [MAPREDUCE-2447](https://issues.apache.org/jira/browse/MAPREDUCE-2447) | Set JvmContext sooner for a task - MR2429 | Minor | . | Siddharth Seth | Siddharth Seth | +| [MAPREDUCE-2443](https://issues.apache.org/jira/browse/MAPREDUCE-2443) | Fix FI build - broken after MR-2429 | Minor | test | Siddharth Seth | Siddharth Seth | +| [MAPREDUCE-2429](https://issues.apache.org/jira/browse/MAPREDUCE-2429) | Check jvmid during task status report | Major | tasktracker | Arun C Murthy | Siddharth Seth | +| [MAPREDUCE-2418](https://issues.apache.org/jira/browse/MAPREDUCE-2418) | Errors not shown in the JobHistory servlet (specifically Counter Limit Exceeded) | Minor | . | Siddharth Seth | Siddharth Seth | +| [MAPREDUCE-2411](https://issues.apache.org/jira/browse/MAPREDUCE-2411) | When you submit a job to a queue with no ACLs you get an inscrutible NPE | Minor | . | Dick King | Dick King | +| [MAPREDUCE-2409](https://issues.apache.org/jira/browse/MAPREDUCE-2409) | Distributed Cache does not differentiate between file /archive for files with the same path | Major | distributed-cache | Siddharth Seth | Siddharth Seth | +| [MAPREDUCE-2366](https://issues.apache.org/jira/browse/MAPREDUCE-2366) | TaskTracker can't retrieve stdout and stderr from web UI | Major | tasktracker | Owen O'Malley | Dick King | +| [MAPREDUCE-2364](https://issues.apache.org/jira/browse/MAPREDUCE-2364) | Shouldn't hold lock on rjob while localizing resources. | Major | tasktracker | Owen O'Malley | Devaraj Das | +| [MAPREDUCE-2362](https://issues.apache.org/jira/browse/MAPREDUCE-2362) | Unit test failures: TestBadRecords and TestTaskTrackerMemoryManager | Major | test | Owen O'Malley | Greg Roelofs | +| [MAPREDUCE-2360](https://issues.apache.org/jira/browse/MAPREDUCE-2360) | Pig fails when using non-default FileSystem | Major | client | Owen O'Malley | | +| [MAPREDUCE-2359](https://issues.apache.org/jira/browse/MAPREDUCE-2359) | Distributed cache doesn't use non-default FileSystems correctly | Major | . | Owen O'Malley | Krishna Ramachandran | +| [MAPREDUCE-2358](https://issues.apache.org/jira/browse/MAPREDUCE-2358) | MapReduce assumes HDFS as the default filesystem | Major | . | Owen O'Malley | Krishna Ramachandran | +| [MAPREDUCE-2357](https://issues.apache.org/jira/browse/MAPREDUCE-2357) | When extending inputsplit (non-FileSplit), all exceptions are ignored | Major | task | Owen O'Malley | Luke Lu | +| [MAPREDUCE-2356](https://issues.apache.org/jira/browse/MAPREDUCE-2356) | A task succeeded even though there were errors on all attempts. | Major | . | Owen O'Malley | Luke Lu | +| [MAPREDUCE-517](https://issues.apache.org/jira/browse/MAPREDUCE-517) | The capacity-scheduler should assign multiple tasks per heartbeat | Critical | . | Arun C Murthy | Arun C Murthy | +| [MAPREDUCE-118](https://issues.apache.org/jira/browse/MAPREDUCE-118) | Job.getJobID() will always return null | Blocker | client | Amar Kamat | Amareshwari Sriramadasu | + + +### TESTS: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | +| [HDFS-2218](https://issues.apache.org/jira/browse/HDFS-2218) | Disable TestHdfsProxy.testHdfsProxyInterface in 0.20-security and branch-1 until HDFS-2217 is fixed | Blocker | contrib/hdfsproxy, test | Matt Foley | Matt Foley | +| [HDFS-2044](https://issues.apache.org/jira/browse/HDFS-2044) | TestQueueProcessingStatistics failing automatic test due to timing issues | Major | test | Matt Foley | Matt Foley | + + +### SUB-TASKS: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | +| [HDFS-1767](https://issues.apache.org/jira/browse/HDFS-1767) | Namenode should ignore non-initial block reports from datanodes when in safemode during startup | Major | datanode | Matt Foley | Matt Foley | +| [HDFS-1541](https://issues.apache.org/jira/browse/HDFS-1541) | Not marking datanodes dead When namenode in safemode | Major | namenode | Hairong Kuang | Hairong Kuang | +| [HDFS-1445](https://issues.apache.org/jira/browse/HDFS-1445) | Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file | Major | datanode | Matt Foley | Matt Foley | +| [MAPREDUCE-2415](https://issues.apache.org/jira/browse/MAPREDUCE-2415) | Distribute TaskTracker userlogs onto multiple disks | Major | task-controller, tasktracker | Bharath Mundlapudi | Bharath Mundlapudi | +| [MAPREDUCE-2413](https://issues.apache.org/jira/browse/MAPREDUCE-2413) | TaskTracker should handle disk failures at both startup and runtime | Major | task-controller, tasktracker | Bharath Mundlapudi | Ravi Gummadi | + + +### OTHER: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | + + http://git-wip-us.apache.org/repos/asf/hadoop/blob/d759b4bd/hadoop-common-project/hadoop-common/src/site/markdown/release/0.20.204.0/RELEASENOTES.0.20.204.0.md ---------------------------------------------------------------------- diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/release/0.20.204.0/RELEASENOTES.0.20.204.0.md b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.20.204.0/RELEASENOTES.0.20.204.0.md new file mode 100644 index 0000000..7a5f560 --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.20.204.0/RELEASENOTES.0.20.204.0.md @@ -0,0 +1,83 @@ + +<!--- +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +--> +# Apache Hadoop 0.20.204.0 Release Notes + +These release notes cover new developer and user-facing incompatibilities, features, and major improvements. + + +--- + +* [HADOOP-6255](https://issues.apache.org/jira/browse/HADOOP-6255) | *Major* | **Create an rpm integration project** + +Added RPM/DEB packages to build system. + + +--- + +* [HDFS-2218](https://issues.apache.org/jira/browse/HDFS-2218) | *Blocker* | **Disable TestHdfsProxy.testHdfsProxyInterface in 0.20-security and branch-1 until HDFS-2217 is fixed** + +Test case TestHdfsProxy.testHdfsProxyInterface has been temporarily disabled for this release, due to failure in the Hudson automated test environment. + + +--- + +* [HDFS-1445](https://issues.apache.org/jira/browse/HDFS-1445) | *Major* | **Batch the calls in DataStorage to FileUtil.createHardLink(), so we call it once per directory instead of once per file** + +Batch hardlinking during "upgrade" snapshots, cutting time from aprx 8 minutes per volume to aprx 8 seconds. Validated in both Linux and Windows. Depends on prior integration with patch for HADOOP-7133. + + +--- + +* [MAPREDUCE-2846](https://issues.apache.org/jira/browse/MAPREDUCE-2846) | *Blocker* | **a small % of all tasks fail with DefaultTaskController** + +Fixed a race condition in writing the log index file that caused tasks to 'fail'. + + +--- + +* [MAPREDUCE-2804](https://issues.apache.org/jira/browse/MAPREDUCE-2804) | *Blocker* | **"Creation of symlink to attempt log dir failed." message is not useful** + +Removed duplicate chmods of job log dir that were vulnerable to race conditions between tasks. Also improved the messages when the symlinks failed to be created. + + +--- + +* [MAPREDUCE-2529](https://issues.apache.org/jira/browse/MAPREDUCE-2529) | *Major* | **Recognize Jetty bug 1342 and handle it** + +Added 2 new config parameters: + +mapreduce.reduce.shuffle.catch.exception.stack.regex +mapreduce.reduce.shuffle.catch.exception.message.regex + + +--- + +* [MAPREDUCE-2524](https://issues.apache.org/jira/browse/MAPREDUCE-2524) | *Minor* | **Backport trunk heuristics for failing maps when we get fetch failures retrieving map output during shuffle** + +Added a new configuration option: mapreduce.reduce.shuffle.maxfetchfailures, and removed a no longer used option: mapred.reduce.copy.backoff. + + +--- + +* [MAPREDUCE-2479](https://issues.apache.org/jira/browse/MAPREDUCE-2479) | *Major* | **Backport MAPREDUCE-1568 to hadoop security branch** + +Added mapreduce.tasktracker.distributedcache.checkperiod to the task tracker that defined the period to wait while cleaning up the distributed cache. The default is 1 min. + + + http://git-wip-us.apache.org/repos/asf/hadoop/blob/d759b4bd/hadoop-common-project/hadoop-common/src/site/markdown/release/0.20.204.1/CHANGES.0.20.204.1.md ---------------------------------------------------------------------- diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/release/0.20.204.1/CHANGES.0.20.204.1.md b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.20.204.1/CHANGES.0.20.204.1.md new file mode 100644 index 0000000..56c3f85 --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.20.204.1/CHANGES.0.20.204.1.md @@ -0,0 +1,64 @@ + +<!--- +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +--> +# Apache Hadoop Changelog + +## Release 0.20.204.1 - Unreleased + +### INCOMPATIBLE CHANGES: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | + + +### NEW FEATURES: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | + + +### IMPROVEMENTS: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | + + +### BUG FIXES: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | + + +### TESTS: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | + + +### SUB-TASKS: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | + + +### OTHER: + +| JIRA | Summary | Priority | Component | Reporter | Contributor | +|:---- |:---- | :--- |:---- |:---- |:---- | + + http://git-wip-us.apache.org/repos/asf/hadoop/blob/d759b4bd/hadoop-common-project/hadoop-common/src/site/markdown/release/0.20.204.1/RELEASENOTES.0.20.204.1.md ---------------------------------------------------------------------- diff --git a/hadoop-common-project/hadoop-common/src/site/markdown/release/0.20.204.1/RELEASENOTES.0.20.204.1.md b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.20.204.1/RELEASENOTES.0.20.204.1.md new file mode 100644 index 0000000..db70b82 --- /dev/null +++ b/hadoop-common-project/hadoop-common/src/site/markdown/release/0.20.204.1/RELEASENOTES.0.20.204.1.md @@ -0,0 +1,24 @@ + +<!--- +# Licensed to the Apache Software Foundation (ASF) under one +# or more contributor license agreements. See the NOTICE file +# distributed with this work for additional information +# regarding copyright ownership. The ASF licenses this file +# to you under the Apache License, Version 2.0 (the +# "License"); you may not use this file except in compliance +# with the License. You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +--> +# Apache Hadoop 0.20.204.1 Release Notes + +These release notes cover new developer and user-facing incompatibilities, features, and major improvements. + + +
