Author: omalley
Date: Mon Jul 18 23:07:42 2011
New Revision: 1148088
URL: http://svn.apache.org/viewvc?rev=1148088&view=rev
Log:
HADOOP-7045. TestDU fails on systems with local file systems with
extended attributes. (eli)
Modified:
hadoop/common/branches/branch-0.20-security/ (props changed)
hadoop/common/branches/branch-0.20-security/CHANGES.txt (contents, props
changed)
hadoop/common/branches/branch-0.20-security/src/test/org/apache/hadoop/fs/TestDU.java
Propchange: hadoop/common/branches/branch-0.20-security/
------------------------------------------------------------------------------
--- svn:mergeinfo (original)
+++ svn:mergeinfo Mon Jul 18 23:07:42 2011
@@ -1,5 +1,5 @@
/hadoop/common/branches/branch-0.20:826138,826568,829987,831184,833001,880632,898713,909245,909723,960946,1044225
/hadoop/common/branches/branch-0.20-security-203:1096071,1097011,1097249,1097269,1097281,1097966,1098816,1098819,1098823,1098827,1098832,1098839,1098854,1098863,1099088,1099191,1099324,1099330,1099333,1102071,1128115
-/hadoop/common/branches/branch-0.20-security-204:1128390,1147228
+/hadoop/common/branches/branch-0.20-security-204:1128390,1147228,1148069
/hadoop/core/branches/branch-0.19:713112
/hadoop/core/trunk:727001,727117,727191,727212,727217,727228,727255,727869,728187,729052,729987,732385,732572,732613,732777,732838,732869,733887,734870,734916,736426,738328,738697,740077,740157,741703,741762,743745,743816,743892,744894,745180,746010,746206,746227,746233,746274,746338,746902-746903,746925,746944,746968,746970,747279,747289,747802,748084,748090,748783,749262,749318,749863,750533,752073,752609,752834,752836,752913,752932,753112-753113,753346,754645,754847,754927,755035,755226,755348,755370,755418,755426,755790,755905,755938,755960,755986,755998,756352,757448,757624,757849,758156,758180,759398,759932,760502,760783,761046,761482,761632,762216,762879,763107,763502,764967,765016,765809,765951,771607,771661,772844,772876,772884,772920,773889,776638,778962,778966,779893,781720,784661,785046,785569
Modified: hadoop/common/branches/branch-0.20-security/CHANGES.txt
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.20-security/CHANGES.txt?rev=1148088&r1=1148087&r2=1148088&view=diff
==============================================================================
--- hadoop/common/branches/branch-0.20-security/CHANGES.txt (original)
+++ hadoop/common/branches/branch-0.20-security/CHANGES.txt Mon Jul 18 23:07:42
2011
@@ -1,6 +1,6 @@
Hadoop Change Log
-Release 0.20.206.0 - unreleased
+Release 0.20.205.0 - unreleased
NEW FEATURES
@@ -17,10 +17,25 @@ Release 0.20.206.0 - unreleased
HDFS-1836. Thousand of CLOSE_WAIT socket. Contributed by Todd Lipcon,
ported to security branch by Bharath Mundlapudi. (via mattf)
-Release 0.20.205.0 - unreleased
+Release 0.20.204.0 - unreleased
+
+ NEW FEATURES
+
+ HADOOP-6255. Create RPM and Debian packages for common. Changes deployment
+ layout to be consistent across the binary tgz, rpm, and deb. Adds setup
+ scripts for easy one node cluster configuration and user creation.
+ (Eric Yang via omalley)
+
+ HADOOP-7324. Ganglia plugins for metrics v2. (Priyo Mustafi via llu)
BUG FIXES
+ HADOOP-7045. TestDU fails on systems with local file systems with
+ extended attributes. (eli)
+
+ MAPREDUCE-2495. exit() the TaskTracker when the distributed cache cleanup
+ thread dies. (Robert Joseph Evans via cdouglas)
+
HDFS-1878. TestHDFSServerPorts unit test failure - race condition
in FSNamesystem.close() causes NullPointerException without serious
consequence. (mattf)
@@ -28,89 +43,57 @@ Release 0.20.205.0 - unreleased
MAPREDUCE-2452. Moves the cancellation of delegation tokens to a separate
thread. (ddas)
- HDFS-1377. Quota bug for partial blocks allows quotas to be violated. (eli)
+ MAPREDUCE-2555. Avoid sprious logging from completedtasks. (Thomas Graves
+ via cdouglas)
MAPREDUCE-2451. Log the details from health check script at the
JobTracker. (Thomas Graves via cdouglas)
+ MAPREDUCE-2535. Fix NPE in JobClient caused by retirement. (Robert Joseph
+ Evans via cdouglas)
+
MAPREDUCE-2456. Log the reduce taskID and associated TaskTrackers with
failed fetch notifications in the JobTracker log.
(Jeffrey Naisbitt via cdouglas)
+ HDFS-2044. TestQueueProcessingStatistics failing automatic test due to
+ timing issues. (mattf)
+
HADOOP-7248. Update eclipse target to generate .classpath from ivy config.
(Thomas Graves and Tom White via cdouglas)
+ MAPREDUCE-2558. Add queue-level metrics 0.20-security branch (test fixes)
+ (Jeffrey Naisbitt via mahadev)
+
+ HADOOP-7364. TestMiniMRDFSCaching fails if test.build.dir is set to
+ something other than build/test. (Thomas Graves via mahadev)
+
HADOOP-7277. Add generation of run configurations to eclipse target.
(Jeffrey Naisbitt and Philip Zeyliger via cdouglas)
+ HADOOP-7373. Fix {start,stop}-{dfs,mapred} and hadoop-daemons.sh from
+ trying to use the wrong bin directory. (omalley)
+
HADOOP-7274. Fix typos in IOUtils. (Jonathan Eagles via cdouglas)
+ HADOOP-7369. Fix permissions in tarball for sbin/* and libexec/* (omalley)
+
MAPREDUCE-2479. Move distributed cache cleanup to a background task,
backporting MAPREDUCE-1568. (Robert Joseph Evans via cdouglas)
- HADOOP-7272. Remove unnecessary security related info logs. (suresh)
+ HADOOP-7356. Fix bin/hadoop scripts (eyang via omalley)
- HDFS-1906. Remove logging exception stack trace in client logs when one of
- the datanode targets to read from is not reachable. (suresh)
+ HADOOP-7272. Remove unnecessary security related info logs. (suresh)
MAPREDUCE-2514. Fix typo in TaskTracker ReinitTrackerAction log message.
(Jonathan Eagles via cdouglas)
+ HDFS-1906. Remove logging exception stack trace in client logs when one of
+ the datanode targets to read from is not reachable. (suresh)
+
MAPREDUCE-2490. Add logging to graylist and blacklist activity to aid
diagnosis of related issues. (Jonathan Eagles via cdouglas)
- MAPREDUCE-2495. exit() the TaskTracker when the distributed cache cleanup
- thread dies. (Robert Joseph Evans via cdouglas)
-
- MAPREDUCE-2555. Avoid sprious logging from completedtasks. (Thomas Graves
- via cdouglas)
-
- MAPREDUCE-2558. Add JobTracker metrics for scheduling queues. (Jeffrey
- Naisbitt via cdouglas)
-
- MAPREDUCE-2535. Fix NPE in JobClient caused by retirement. (Robert Joseph
- Evans via cdouglas)
-
- HADOOP-7144. Expose JMX metrics via JSON servlet. (Robert Joseph Evans via
- cdouglas)
-
- MAPREDUCE-2524. Port reduce failure reporting semantics from trunk, to
- fail faulty maps more aggressively. (Thomas Graves via cdouglas)
-
- MAPREDUCE-2529. Add support for regex-based shuffle metric counting
- exceptions. (Thomas Graves via cdouglas)
-
- HDFS-2044. TestQueueProcessingStatistics failing automatic test due to
- timing issues. (mattf)
-
- MAPREDUCE-2558. Add queue-level metrics 0.20-security branch (test fixes)
- (Jeffrey Naisbitt via mahadev)
-
- HADOOP-7364. TestMiniMRDFSCaching fails if test.build.dir is set to
- something other than build/test (Thomas Graves via mahadev)
-
-Release 0.20.204.0 - unreleased
-
- NEW FEATURES
-
- HADOOP-6255. Create RPM and Debian packages for common. Changes deployment
- layout to be consistent across the binary tgz, rpm, and deb. Adds setup
- scripts for easy one node cluster configuration and user creation.
- (Eric Yang via omalley)
-
- HADOOP-7324. Ganglia plugins for metrics v2. (Priyo Mustafi via llu)
-
- BUG FIXES
-
- HADOOP-7373. Fix {start,stop}-{dfs,mapred} and hadoop-daemons.sh from
- trying to use the wrong bin directory. (omalley)
-
- HADOOP-7369. Fix permissions in tarball for sbin/* and libexec/* (omalley)
-
- HADOOP-7356. Fix bin/hadoop scripts (eyang via omalley)
-
- MAPREDUCE-2316. Updated CapacityScheduler documentation. (acmurthy)
-
MAPREDUCE-2447. Fix Child.java to set Task.jvmContext sooner to avoid
corner cases in error handling. (Siddharth Seth via acmurthy)
@@ -147,9 +130,6 @@ Release 0.20.204.0 - unreleased
HDFS-1258. Clearing namespace quota on "/" corrupts fs image.
(Aaron T. Myers via szetszwo)
- HADOOP-7215. RPC clients must use network interface corresponding to
- the host in the client's kerberos principal key. (suresh)
-
HDFS-1189. Quota counts missed between clear quota and set quota.
(John George via szetszwo)
@@ -166,11 +146,22 @@ Release 0.20.204.0 - unreleased
by throwing an error to indicate the editlog needs to be empty.
(suresh)
+ HDFS-1377. Quota bug for partial blocks allows quotas to be violated. (eli)
+
HDFS-2057. Wait time to terminate the threads causes unit tests to
take longer time. (Bharath Mundlapudi via suresh)
IMPROVEMENTS
+ HADOOP-7144. Expose JMX metrics via JSON servlet. (Robert Joseph Evans via
+ cdouglas)
+
+ MAPREDUCE-2524. Port reduce failure reporting semantics from trunk, to
+ fail faulty maps more aggressively. (Thomas Graves via cdouglas)
+
+ MAPREDUCE-2529. Add support for regex-based shuffle metric counting
+ exceptions. (Thomas Graves via cdouglas)
+
HADOOP-7398. Add mechanism to suppress warnings about use of HADOOP_HOME.
(omalley)
@@ -217,6 +208,8 @@ Release 0.20.203.0 - 2011-5-11
HADOOP-7143. Restore HadoopArchives. (Joep Rottinghuis via omalley)
+ MAPREDUCE-2316. Updated CapacityScheduler documentation. (acmurthy)
+
HADOOP-7243. Fix contrib unit tests missing dependencies. (omalley)
MAPREDUCE-2355. Add a dampner to out-of-band heartbeats. (acmurthy)
@@ -271,6 +264,9 @@ Release 0.20.203.0 - 2011-5-11
HADOOP-6879. Provide SSH based (Jsch) remote execution API for system
tests. (cos)
+ HADOOP-7215. RPC clients must use network interface corresponding to
+ the host in the client's kerberos principal key. (suresh)
+
HADOOP-7232. Fix Javadoc warnings. (omalley)
HADOOP-7258. The Gzip codec should not return null decompressors. (omalley)
Propchange: hadoop/common/branches/branch-0.20-security/CHANGES.txt
------------------------------------------------------------------------------
--- svn:mergeinfo (original)
+++ svn:mergeinfo Mon Jul 18 23:07:42 2011
@@ -1,6 +1,6 @@
/hadoop/common/branches/branch-0.20/CHANGES.txt:826138,826568,829987,831184,833001,880632,898713,909245,909723,960946,1044225
/hadoop/common/branches/branch-0.20-security-203/CHANGES.txt:1096071,1097011,1097249,1097269,1097281,1097966,1098816,1098819,1098823,1098827,1098832,1098839,1098854,1098863,1099088,1099191,1099324,1099330,1099333,1102071,1128115
-/hadoop/common/branches/branch-0.20-security-204/CHANGES.txt:1128390,1147228
+/hadoop/common/branches/branch-0.20-security-204/CHANGES.txt:1128390,1147228,1148069
/hadoop/core/branches/branch-0.18/CHANGES.txt:727226
/hadoop/core/branches/branch-0.19/CHANGES.txt:713112
/hadoop/core/trunk/CHANGES.txt:727001,727117,727191,727212,727228,727255,727869,728187,729052,729987,732385,732572,732613,732777,732838,732869,733887,734870,734916,735082,736426,738602,738697,739416,740077,740157,741703,741762,743296,743745,743816,743892,744894,745180,745268,746010,746193,746206,746227,746233,746274,746902-746903,746925,746944,746968,746970,747279,747289,747802,748084,748090,748783,749262,749318,749863,750533,752073,752514,752555,752590,752609,752834,752836,752913,752932,753112-753113,753346,754645,754847,754927,755035,755226,755348,755370,755418,755426,755790,755905,755938,755986,755998,756352,757448,757624,757849,758156,758180,759398,759932,760502,760783,761046,761482,761632,762216,762879,763107,763502,764967,765016,765809,765951,771607,772844,772876,772884,772920,773889,776638,778962,778966,779893,781720,784661,785046,785569
Modified:
hadoop/common/branches/branch-0.20-security/src/test/org/apache/hadoop/fs/TestDU.java
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.20-security/src/test/org/apache/hadoop/fs/TestDU.java?rev=1148088&r1=1148087&r2=1148088&view=diff
==============================================================================
---
hadoop/common/branches/branch-0.20-security/src/test/org/apache/hadoop/fs/TestDU.java
(original)
+++
hadoop/common/branches/branch-0.20-security/src/test/org/apache/hadoop/fs/TestDU.java
Mon Jul 18 23:07:42 2011
@@ -65,7 +65,10 @@ public class TestDU extends TestCase {
* @throws InterruptedException
*/
public void testDU() throws IOException, InterruptedException {
- int writtenSize = 32*1024; // writing 32K
+ final int writtenSize = 32*1024; // writing 32K
+ // Allow for extra 4K on-disk slack for local file systems
+ // that may store additional file metadata (eg ext attrs).
+ final int slack = 4*1024;
File file = new File(DU_DIR, "data");
createFile(file, writtenSize);
@@ -76,7 +79,9 @@ public class TestDU extends TestCase {
long duSize = du.getUsed();
du.shutdown();
- assertEquals(writtenSize, duSize);
+ assertTrue("Invalid on-disk size",
+ duSize >= writtenSize &&
+ writtenSize <= (duSize + slack));
//test with 0 interval, will not launch thread
du = new DU(file, 0);
@@ -84,12 +89,16 @@ public class TestDU extends TestCase {
duSize = du.getUsed();
du.shutdown();
- assertEquals(writtenSize, duSize);
+ assertTrue("Invalid on-disk size",
+ duSize >= writtenSize &&
+ writtenSize <= (duSize + slack));
//test without launching thread
du = new DU(file, 10000);
duSize = du.getUsed();
-
- assertEquals(writtenSize, duSize);
+
+ assertTrue("Invalid on-disk size",
+ duSize >= writtenSize &&
+ writtenSize <= (duSize + slack));
}
}
