svn commit: r1468336 - /hadoop/common/branches/branch-1.2/src/docs/releasenotes.html
Author: mattf Date: Tue Apr 16 09:35:51 2013 New Revision: 1468336 URL: http://svn.apache.org/r1468336 Log: Release notes for 1.2.0 Modified: hadoop/common/branches/branch-1.2/src/docs/releasenotes.html Modified: hadoop/common/branches/branch-1.2/src/docs/releasenotes.html URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-1.2/src/docs/releasenotes.html?rev=1468336r1=1468335r2=1468336view=diff == --- hadoop/common/branches/branch-1.2/src/docs/releasenotes.html (original) +++ hadoop/common/branches/branch-1.2/src/docs/releasenotes.html Tue Apr 16 09:35:51 2013 @@ -194,6 +194,11 @@ To run secure Datanodes users must insta bAllow setting of end-of-record delimiter for TextInputFormat/bbr blockquoteThe patch for https://issues.apache.org/jira/browse/MAPREDUCE-2254 required minor changes to the LineReader class to allow extensions (see attached 2.patch). Description copied below:brbrIt will be useful to allow setting the end-of-record delimiter for TextInputFormat. The current implementation hardcodes apos;\napos;, apos;\rapos; or apos;\r\napos; as the only possible record delimiters. This is a problem if users have embedded newlines in their data fields (which is pretty common). This is also a problem for other .../blockquote/li +li a href=https://issues.apache.org/jira/browse/HADOOP-7101;HADOOP-7101/a. + Blocker bug reported by tlipcon and fixed by tlipcon (security)br + bUserGroupInformation.getCurrentUser() fails when called from non-Hadoop JAAS context/bbr + blockquoteIf a Hadoop client is run from inside a container like Tomcat, and the current AccessControlContext has a Subject associated with it that is not created by Hadoop, then UserGroupInformation.getCurrentUser() will throw NoSuchElementException, since it assumes that any Subject will have a hadoop User principal./blockquote/li + li a href=https://issues.apache.org/jira/browse/HADOOP-7688;HADOOP-7688/a. Major improvement reported by szetszwo and fixed by umamaheswararao br bWhen a servlet filter throws an exception in init(..), the Jetty server failed silently. /bbr @@ -369,6 +374,11 @@ To run secure Datanodes users must insta bTestSinkQueue.testConcurrentConsumers fails intermittently (Backports HADOOP-7292)/bbr blockquoteorg.apache.hadoop.metrics2.impl.TestSinkQueue.testConcurrentConsumersbr brbrError Messagebrbrshouldapos;ve thrownbrStacktracebrbrjunit.framework.AssertionFailedError: shouldapos;ve thrownbr at org.apache.hadoop.metrics2.impl.TestSinkQueue.shouldThrowCME(TestSinkQueue.java:229)br at org.apache.hadoop.metrics2.impl.TestSinkQueue.testConcurrentConsumers(TestSinkQueue.java:195)brStandard Outputbrbr2012-10-03 16:51:31,694 INFO impl.TestSinkQueue (TestSinkQueue.java:consume(243)) - sleepingbr/blockquote/li +li a href=https://issues.apache.org/jira/browse/HADOOP-9071;HADOOP-9071/a. + Major improvement reported by gkesavan and fixed by gkesavan (build)br + bconfigure ivy log levels for resolve/retrieve/bbr + blockquote/blockquote/li + li a href=https://issues.apache.org/jira/browse/HADOOP-9090;HADOOP-9090/a. Minor new feature reported by mostafae and fixed by mostafae (metrics)br bSupport on-demand publish of metrics/bbr @@ -439,6 +449,31 @@ To run secure Datanodes users must insta bPort HADOOP-7290 to branch-1 to fix TestUserGroupInformation failure/bbr blockquoteUnit test failure in TestUserGroupInformation.testGetServerSideGroups. port HADOOP-7290 to branch-1.1 /blockquote/li +li a href=https://issues.apache.org/jira/browse/HADOOP-9379;HADOOP-9379/a. + Trivial improvement reported by arpitgupta and fixed by arpitgupta br + bcapture the ulimit info after printing the log to the console/bbr + blockquoteBased on the discussions in HADOOP-9253 people prefer if we dont print the ulimit info to the console but still have it in the logs.brbrJust need to move the head statement to before the capture of ulimit code./blockquote/li + +li a href=https://issues.apache.org/jira/browse/HADOOP-9434;HADOOP-9434/a. + Minor improvement reported by carp84 and fixed by carp84 (bin)br + bBackport HADOOP-9267 to branch-1/bbr + blockquoteCurrently in hadoop 1.1.2, if user issue quot;bin/hadoop helpquot; in command line, it will throw below exception. We can improve this to print the usage message.br===brException in thread quot;mainquot; java.lang.NoClassDefFoundError: helpbr===brbrThis issue is already resolved in HADOOP-9267 in trunk, so we only need to backport it into branch-1/blockquote/li + +li a href=https://issues.apache.org/jira/browse/HADOOP-9451;HADOOP-9451/a. + Major bug reported by djp and fixed by djp (net)br + bNode with one topology layer should be handled as fault topology when
svn commit: r1468337 - /hadoop/common/branches/branch-1.2/src/docs/releasenotes.html
Author: mattf Date: Tue Apr 16 09:38:21 2013 New Revision: 1468337 URL: http://svn.apache.org/r1468337 Log: Release notes for 1.2.0, tweak Modified: hadoop/common/branches/branch-1.2/src/docs/releasenotes.html Modified: hadoop/common/branches/branch-1.2/src/docs/releasenotes.html URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-1.2/src/docs/releasenotes.html?rev=1468337r1=1468336r2=1468337view=diff == --- hadoop/common/branches/branch-1.2/src/docs/releasenotes.html (original) +++ hadoop/common/branches/branch-1.2/src/docs/releasenotes.html Tue Apr 16 09:38:21 2013 @@ -10,7 +10,7 @@ /STYLE /head body -h1Hadoop 1.2.0 Release Notes - Preliminary/h1 +h1Hadoop 1.2.0 Release Notes/h1 These release notes include new developer and user-facing incompatibilities, features, and major improvements. a name=changes/
svn commit: r1468339 - /hadoop/common/branches/branch-1.2/CHANGES.txt
Author: mattf Date: Tue Apr 16 09:39:48 2013 New Revision: 1468339 URL: http://svn.apache.org/r1468339 Log: prepare for 1.2.0 release build Modified: hadoop/common/branches/branch-1.2/CHANGES.txt Modified: hadoop/common/branches/branch-1.2/CHANGES.txt URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-1.2/CHANGES.txt?rev=1468339r1=1468338r2=1468339view=diff == --- hadoop/common/branches/branch-1.2/CHANGES.txt (original) +++ hadoop/common/branches/branch-1.2/CHANGES.txt Tue Apr 16 09:39:48 2013 @@ -1,6 +1,6 @@ Hadoop Change Log -Release 1.2.0 - unreleased +Release 1.2.0 - 2013.04.16 INCOMPATIBLE CHANGES
svn commit: r1468342 - /hadoop/common/tags/release-1.2.0-rc0/
Author: mattf Date: Tue Apr 16 09:44:16 2013 New Revision: 1468342 URL: http://svn.apache.org/r1468342 Log: Hadoop 1.2.0-rc0 release candidate. Added: hadoop/common/tags/release-1.2.0-rc0/ (props changed) - copied from r1468341, hadoop/common/branches/branch-1.2/ Propchange: hadoop/common/tags/release-1.2.0-rc0/ -- --- svn:ignore (added) +++ svn:ignore Tue Apr 16 09:44:16 2013 @@ -0,0 +1,9 @@ +build +build.properties +logs +.classpath +.git +.project +.settings +.launches +.externalToolBuilders Propchange: hadoop/common/tags/release-1.2.0-rc0/ -- --- svn:mergeinfo (added) +++ svn:mergeinfo Tue Apr 16 09:44:16 2013 @@ -0,0 +1,10 @@ +/hadoop/common/branches/branch-0.20:826138,826568,829987,831184,833001,880632,898713,909245,909723,960946,990003,1044225 +/hadoop/common/branches/branch-0.20-append:955380,955398,955448,956329 +/hadoop/common/branches/branch-0.20-security-203:1096071,1097011,1097249,1097269,1097281,1097966,1098816,1098819,1098823,1098827,1098832,1098839,1098854,1098863,1099088,1099191,1099324,1099330,1099333,1102071,1128115 +/hadoop/common/branches/branch-0.20-security-204:1128390,1147228,1148069,1149316,1154413 +/hadoop/common/branches/branch-0.20-security-205:1174370,1174917,1176042,1176248,1176638,1176645,1202378 +/hadoop/common/branches/branch-1:1455883,1461952 +/hadoop/common/branches/branch-1.0:1214410,1291091,1330040,1334877 +/hadoop/common/branches/branch-1.1:1347101,1358689,1408468 +/hadoop/core/branches/branch-0.19:713112 +/hadoop/core/trunk:727001,727117,727191,727212,727217,727228,727255,727869,728187,729052,729987,732385,732572,732613,732777,732838,732869,733887,734870,734916,736426,738328,738697,740077,740157,741703,741762,743745,743816,743892,744894,745180,746010,746206,746227,746233,746274,746338,746902-746903,746925,746944,746968,746970,747279,747289,747802,748084,748090,748783,749262,749318,749863,750533,752073,752609,752834,752836,752913,752932,753112-753113,753346,754645,754847,754927,755035,755226,755348,755370,755418,755426,755790,755905,755938,755960,755986,755998,756352,757448,757624,757849,758156,758180,759398,759932,760502,760783,761046,761482,761632,762216,762879,763107,763502,764967,765016,765809,765951,771607,771661,772844,772876,772884,772920,773889,776638,778962,778966,779893,781720,784661,785046,785569
svn commit: r1468629 - in /hadoop/common/trunk/hadoop-tools/hadoop-distcp/src: main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java test/java/org/apache/hadoop/tools/mapred/TestCopyM
Author: kihwal
Date: Tue Apr 16 22:01:18 2013
New Revision: 1468629
URL: http://svn.apache.org/r1468629
Log:
MAPREDUCE-5065. DistCp should skip checksum comparisons if block-sizes are
different on source/target. Contributed by Mithun Radhakrishnan.
Modified:
hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/mapred/TestCopyMapper.java
Modified:
hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
URL:
http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java?rev=1468629r1=1468628r2=1468629view=diff
==
---
hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
(original)
+++
hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
Tue Apr 16 22:01:18 2013
@@ -140,10 +140,17 @@ public class RetriableFileCopyCommand ex
private void compareCheckSums(FileSystem sourceFS, Path source,
FileSystem targetFS, Path target)
throws IOException {
-if (!DistCpUtils.checksumsAreEqual(sourceFS, source, targetFS, target))
- throw new IOException(Check-sum mismatch between
- + source + and + target);
-
+if (!DistCpUtils.checksumsAreEqual(sourceFS, source, targetFS, target)) {
+ StringBuilder errorMessage = new StringBuilder(Check-sum mismatch
between )
+ .append(source).append( and ).append(target).append(.);
+ if (sourceFS.getFileStatus(source).getBlockSize() !=
targetFS.getFileStatus(target).getBlockSize()) {
+errorMessage.append( Source and target differ in block-size.)
+.append( Use -pb to preserve block-sizes during copy.)
+.append( Alternatively, skip checksum-checks altogether, using
-skipCrc.)
+ .append( (NOTE: By skipping
checksums, one runs the risk of masking data-corruption during
file-transfer.));
+ }
+ throw new IOException(errorMessage.toString());
+}
}
//If target file exists and unable to delete target - fail
Modified:
hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/mapred/TestCopyMapper.java
URL:
http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/mapred/TestCopyMapper.java?rev=1468629r1=1468628r2=1468629view=diff
==
---
hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/mapred/TestCopyMapper.java
(original)
+++
hadoop/common/trunk/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/mapred/TestCopyMapper.java
Tue Apr 16 22:01:18 2013
@@ -53,7 +53,7 @@ public class TestCopyMapper {
private static final Log LOG = LogFactory.getLog(TestCopyMapper.class);
private static ListPath pathList = new ArrayListPath();
private static int nFiles = 0;
- private static final int FILE_SIZE = 1024;
+ private static final int DEFAULT_FILE_SIZE = 1024;
private static MiniDFSCluster cluster;
@@ -92,7 +92,7 @@ public class TestCopyMapper {
configuration.setBoolean(DistCpOptionSwitch.OVERWRITE.getConfigLabel(),
false);
configuration.setBoolean(DistCpOptionSwitch.SKIP_CRC.getConfigLabel(),
-true);
+false);
configuration.setBoolean(DistCpOptionSwitch.SYNC_FOLDERS.getConfigLabel(),
true);
configuration.set(DistCpOptionSwitch.PRESERVE_STATUS.getConfigLabel(),
@@ -112,6 +112,18 @@ public class TestCopyMapper {
touchFile(SOURCE_PATH + /7/8/9);
}
+ private static void createSourceDataWithDifferentBlockSize() throws
Exception {
+mkdirs(SOURCE_PATH + /1);
+mkdirs(SOURCE_PATH + /2);
+mkdirs(SOURCE_PATH + /2/3/4);
+mkdirs(SOURCE_PATH + /2/3);
+mkdirs(SOURCE_PATH + /5);
+touchFile(SOURCE_PATH + /5/6, true);
+mkdirs(SOURCE_PATH + /7);
+mkdirs(SOURCE_PATH + /7/8);
+touchFile(SOURCE_PATH + /7/8/9);
+ }
+
private static void mkdirs(String path) throws Exception {
FileSystem fileSystem = cluster.getFileSystem();
final Path qualifiedPath = new
Path(path).makeQualified(fileSystem.getUri(),
@@ -121,17 +133,31 @@ public class TestCopyMapper {
}
private static void touchFile(String path) throws Exception {
+touchFile(path, false);
+ }
+
+ private static void touchFile(String path, boolean createMultipleBlocks)
throws Exception {
+final long NON_DEFAULT_BLOCK_SIZE = 4096;
FileSystem fs;
svn commit: r1468631 - in /hadoop/common/branches/branch-2/hadoop-tools/hadoop-distcp/src: main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java test/java/org/apache/hadoop/tools/mapr
Author: kihwal
Date: Tue Apr 16 22:03:40 2013
New Revision: 1468631
URL: http://svn.apache.org/r1468631
Log:
svn merge -c 1468629 Merging from trunk to branch-2 to fix MR-5065.
Modified:
hadoop/common/branches/branch-2/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
hadoop/common/branches/branch-2/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/mapred/TestCopyMapper.java
Modified:
hadoop/common/branches/branch-2/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java?rev=1468631r1=1468630r2=1468631view=diff
==
---
hadoop/common/branches/branch-2/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
(original)
+++
hadoop/common/branches/branch-2/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
Tue Apr 16 22:03:40 2013
@@ -140,10 +140,17 @@ public class RetriableFileCopyCommand ex
private void compareCheckSums(FileSystem sourceFS, Path source,
FileSystem targetFS, Path target)
throws IOException {
-if (!DistCpUtils.checksumsAreEqual(sourceFS, source, targetFS, target))
- throw new IOException(Check-sum mismatch between
- + source + and + target);
-
+if (!DistCpUtils.checksumsAreEqual(sourceFS, source, targetFS, target)) {
+ StringBuilder errorMessage = new StringBuilder(Check-sum mismatch
between )
+ .append(source).append( and ).append(target).append(.);
+ if (sourceFS.getFileStatus(source).getBlockSize() !=
targetFS.getFileStatus(target).getBlockSize()) {
+errorMessage.append( Source and target differ in block-size.)
+.append( Use -pb to preserve block-sizes during copy.)
+.append( Alternatively, skip checksum-checks altogether, using
-skipCrc.)
+ .append( (NOTE: By skipping
checksums, one runs the risk of masking data-corruption during
file-transfer.));
+ }
+ throw new IOException(errorMessage.toString());
+}
}
//If target file exists and unable to delete target - fail
Modified:
hadoop/common/branches/branch-2/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/mapred/TestCopyMapper.java
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/branch-2/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/mapred/TestCopyMapper.java?rev=1468631r1=1468630r2=1468631view=diff
==
---
hadoop/common/branches/branch-2/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/mapred/TestCopyMapper.java
(original)
+++
hadoop/common/branches/branch-2/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/mapred/TestCopyMapper.java
Tue Apr 16 22:03:40 2013
@@ -53,7 +53,7 @@ public class TestCopyMapper {
private static final Log LOG = LogFactory.getLog(TestCopyMapper.class);
private static ListPath pathList = new ArrayListPath();
private static int nFiles = 0;
- private static final int FILE_SIZE = 1024;
+ private static final int DEFAULT_FILE_SIZE = 1024;
private static MiniDFSCluster cluster;
@@ -92,7 +92,7 @@ public class TestCopyMapper {
configuration.setBoolean(DistCpOptionSwitch.OVERWRITE.getConfigLabel(),
false);
configuration.setBoolean(DistCpOptionSwitch.SKIP_CRC.getConfigLabel(),
-true);
+false);
configuration.setBoolean(DistCpOptionSwitch.SYNC_FOLDERS.getConfigLabel(),
true);
configuration.set(DistCpOptionSwitch.PRESERVE_STATUS.getConfigLabel(),
@@ -112,6 +112,18 @@ public class TestCopyMapper {
touchFile(SOURCE_PATH + /7/8/9);
}
+ private static void createSourceDataWithDifferentBlockSize() throws
Exception {
+mkdirs(SOURCE_PATH + /1);
+mkdirs(SOURCE_PATH + /2);
+mkdirs(SOURCE_PATH + /2/3/4);
+mkdirs(SOURCE_PATH + /2/3);
+mkdirs(SOURCE_PATH + /5);
+touchFile(SOURCE_PATH + /5/6, true);
+mkdirs(SOURCE_PATH + /7);
+mkdirs(SOURCE_PATH + /7/8);
+touchFile(SOURCE_PATH + /7/8/9);
+ }
+
private static void mkdirs(String path) throws Exception {
FileSystem fileSystem = cluster.getFileSystem();
final Path qualifiedPath = new
Path(path).makeQualified(fileSystem.getUri(),
@@ -121,17 +133,31 @@ public class TestCopyMapper {
}
private static void touchFile(String path) throws Exception {
+touchFile(path, false);
+ }
+
+ private static void touchFile(String path, boolean createMultipleBlocks)
throws Exception {
+final long
svn commit: r1468636 - in /hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/src: main/java/org/apache/hadoop/tools/mapred/ test/java/org/apache/hadoop/tools/mapred/
Author: kihwal
Date: Tue Apr 16 22:07:14 2013
New Revision: 1468636
URL: http://svn.apache.org/r1468636
Log:
MAPREDUCE-5065. DistCp should skip checksum comparisons if block-sizes are
different on source/target. Contributed by Mithun Radhakrishnan.
Modified:
hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java
hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/src/test/java/org/apache/hadoop/tools/mapred/TestCopyMapper.java
Modified:
hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java?rev=1468636r1=1468635r2=1468636view=diff
==
---
hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java
(original)
+++
hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/CopyMapper.java
Tue Apr 16 22:07:14 2013
@@ -255,7 +255,7 @@ public class CopyMapper extends MapperT
long bytesCopied;
try {
- bytesCopied = (Long)new RetriableFileCopyCommand(description)
+ bytesCopied = (Long)new RetriableFileCopyCommand(skipCrc, description)
.execute(sourceFileStatus, target, context,
fileAttributes);
} catch (Exception e) {
context.setStatus(Copy Failure: + sourceFileStatus.getPath());
Modified:
hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java?rev=1468636r1=1468635r2=1468636view=diff
==
---
hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
(original)
+++
hadoop/common/branches/branch-0.23/hadoop-tools/hadoop-distcp/src/main/java/org/apache/hadoop/tools/mapred/RetriableFileCopyCommand.java
Tue Apr 16 22:07:14 2013
@@ -41,6 +41,7 @@ public class RetriableFileCopyCommand ex
private static Log LOG = LogFactory.getLog(RetriableFileCopyCommand.class);
private static int BUFFER_SIZE = 8 * 1024;
+ private boolean skipCrc = false;
/**
* Constructor, taking a description of the action.
@@ -51,6 +52,17 @@ public class RetriableFileCopyCommand ex
}
/**
+ * Create a RetriableFileCopyCommand.
+ *
+ * @param skipCrc Whether to skip the crc check.
+ * @param description A verbose description of the copy operation.
+ */
+ public RetriableFileCopyCommand(boolean skipCrc, String description) {
+this(description);
+this.skipCrc = skipCrc;
+ }
+
+ /**
* Implementation of RetriableCommand::doExecute().
* This is the actual copy-implementation.
* @param arguments Argument-list to the command.
@@ -92,7 +104,7 @@ public class RetriableFileCopyCommand ex
compareFileLengths(sourceFileStatus, tmpTargetPath, configuration,
bytesRead);
//At this point, srcdest lengths are same. if length==0, we skip
checksum
- if (bytesRead != 0) {
+ if (bytesRead != 0 !skipCrc) {
compareCheckSums(sourceFS, sourceFileStatus.getPath(), targetFS,
tmpTargetPath);
}
promoteTmpToTarget(tmpTargetPath, target, targetFS);
@@ -128,10 +140,17 @@ public class RetriableFileCopyCommand ex
private void compareCheckSums(FileSystem sourceFS, Path source,
FileSystem targetFS, Path target)
throws IOException {
-if (!DistCpUtils.checksumsAreEqual(sourceFS, source, targetFS, target))
- throw new IOException(Check-sum mismatch between
- + source + and + target);
-
+if (!DistCpUtils.checksumsAreEqual(sourceFS, source, targetFS, target)) {
+ StringBuilder errorMessage = new StringBuilder(Check-sum mismatch
between )
+.append(source).append( and
).append(target).append(.);
+ if (sourceFS.getFileStatus(source).getBlockSize() !=
targetFS.getFileStatus(target).getBlockSize()) {
+errorMessage.append( Source and target differ in block-size.)
+.append( Use -pb to preserve block-sizes during copy.)
+.append( Alternatively, skip checksum-checks altogether,
using -skipCrc.)
+.append( (NOTE: By skipping checksums, one runs the risk
of
svn commit: r1468698 - in /hadoop/common/branches/branch-1: CHANGES.txt src/hdfs/org/apache/hadoop/hdfs/server/namenode/BlocksMap.java src/hdfs/org/apache/hadoop/hdfs/util/LightWeightGSet.java src/tes
Author: suresh
Date: Wed Apr 17 00:23:22 2013
New Revision: 1468698
URL: http://svn.apache.org/r1468698
Log:
HDFS-4635. Move BlockManager#computeCapacity to LightWeightGSet. Contributed by
Suresh Srinivas.
Modified:
hadoop/common/branches/branch-1/CHANGES.txt
hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/server/namenode/BlocksMap.java
hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/util/LightWeightGSet.java
hadoop/common/branches/branch-1/src/test/org/apache/hadoop/hdfs/util/TestGSet.java
Modified: hadoop/common/branches/branch-1/CHANGES.txt
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/CHANGES.txt?rev=1468698r1=1468697r2=1468698view=diff
==
--- hadoop/common/branches/branch-1/CHANGES.txt (original)
+++ hadoop/common/branches/branch-1/CHANGES.txt Wed Apr 17 00:23:22 2013
@@ -236,6 +236,9 @@ Release 1.2.0 - unreleased
MAPREDUCE-5129. Allow tags to JobHistory for deeper analytics. (billie via
acmurthy)
+HDFS-4635. Move BlockManager#computeCapacity to LightWeightGSet.
+(suresh)
+
BUG FIXES
HADOOP-9467. Metrics2 record filter should check name as well as tags.
Modified:
hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/server/namenode/BlocksMap.java
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/server/namenode/BlocksMap.java?rev=1468698r1=1468697r2=1468698view=diff
==
---
hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/server/namenode/BlocksMap.java
(original)
+++
hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/server/namenode/BlocksMap.java
Wed Apr 17 00:23:22 2013
@@ -315,37 +315,11 @@ class BlocksMap {
private GSetBlock, BlockInfo blocks;
BlocksMap(int initialCapacity, float loadFactor) {
-this.capacity = computeCapacity();
+// Use 2% of total memory to size the GSet capacity
+this.capacity = LightWeightGSet.computeCapacity(2.0, BlocksMap);
this.blocks = new LightWeightGSetBlock, BlockInfo(capacity);
}
- /**
- * Let t = 2% of max memory.
- * Let e = round(log_2 t).
- * Then, we choose capacity = 2^e/(size of reference),
- * unless it is outside the close interval [1, 2^30].
- */
- private static int computeCapacity() {
-//VM detection
-//See http://java.sun.com/docs/hotspot/HotSpotFAQ.html#64bit_detection
-final String vmBit = System.getProperty(sun.arch.data.model);
-
-//2% of max memory
-final double twoPC = Runtime.getRuntime().maxMemory()/50.0;
-
-//compute capacity
-final int e1 = (int)(Math.log(twoPC)/Math.log(2.0) + 0.5);
-final int e2 = e1 - (32.equals(vmBit)? 2: 3);
-final int exponent = e2 0? 0: e2 30? 30: e2;
-final int c = 1 exponent;
-
-LightWeightGSet.LOG.info(VM type = + vmBit + -bit);
-LightWeightGSet.LOG.info(2% max memory = + twoPC/(1 20) + MB);
-LightWeightGSet.LOG.info(capacity = 2^ + exponent
-+ = + c + entries);
-return c;
- }
-
void close() {
blocks = null;
}
Modified:
hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/util/LightWeightGSet.java
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/util/LightWeightGSet.java?rev=1468698r1=1468697r2=1468698view=diff
==
---
hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/util/LightWeightGSet.java
(original)
+++
hadoop/common/branches/branch-1/src/hdfs/org/apache/hadoop/hdfs/util/LightWeightGSet.java
Wed Apr 17 00:23:22 2013
@@ -280,4 +280,53 @@ public class LightWeightGSetK, E extend
throw new UnsupportedOperationException(Remove is not supported.);
}
}
+
+ /**
+ * Let t = percentage of max memory.
+ * Let e = round(log_2 t).
+ * Then, we choose capacity = 2^e/(size of reference),
+ * unless it is outside the close interval [1, 2^30].
+ */
+ public static int computeCapacity(double percentage, String mapName) {
+return computeCapacity(Runtime.getRuntime().maxMemory(), percentage,
+mapName);
+ }
+
+ /** Visible for testing */
+ static int computeCapacity(long maxMemory, double percentage,
+ String mapName) {
+if (percentage 100.0 || percentage 0.0) {
+ throw new IllegalArgumentException(Percentage + percentage
+ + must be greater than or equal to 0
+ + and less than or equal to 100);
+}
+if (maxMemory 0) {
+ throw new IllegalArgumentException(Memory + maxMemory
+ + must be greater than or equal to 0);
+}
+if (percentage == 0.0 || maxMemory == 0) {
+ return 0;
+}
+//VM detection
+//See http://java.sun.com/docs/hotspot/HotSpotFAQ.html#64bit_detection
+final
svn commit: r1468699 - in /hadoop/common/branches/branch-1.2: CHANGES.txt src/hdfs/org/apache/hadoop/hdfs/server/namenode/BlocksMap.java src/hdfs/org/apache/hadoop/hdfs/util/LightWeightGSet.java src/t
Author: suresh
Date: Wed Apr 17 00:24:24 2013
New Revision: 1468699
URL: http://svn.apache.org/r1468699
Log:
HDFS-4635. Merge change r1468698 from trunk.
Modified:
hadoop/common/branches/branch-1.2/CHANGES.txt
hadoop/common/branches/branch-1.2/src/hdfs/org/apache/hadoop/hdfs/server/namenode/BlocksMap.java
hadoop/common/branches/branch-1.2/src/hdfs/org/apache/hadoop/hdfs/util/LightWeightGSet.java
hadoop/common/branches/branch-1.2/src/test/org/apache/hadoop/hdfs/util/TestGSet.java
Modified: hadoop/common/branches/branch-1.2/CHANGES.txt
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/branch-1.2/CHANGES.txt?rev=1468699r1=1468698r2=1468699view=diff
==
--- hadoop/common/branches/branch-1.2/CHANGES.txt (original)
+++ hadoop/common/branches/branch-1.2/CHANGES.txt Wed Apr 17 00:24:24 2013
@@ -202,6 +202,9 @@ Release 1.2.0 - 2013.04.16
MAPREDUCE-5129. Allow tags to JobHistory for deeper analytics. (billie via
acmurthy)
+HDFS-4635. Move BlockManager#computeCapacity to LightWeightGSet.
+(suresh)
+
BUG FIXES
HADOOP-9467. Metrics2 record filter should check name as well as tags.
Modified:
hadoop/common/branches/branch-1.2/src/hdfs/org/apache/hadoop/hdfs/server/namenode/BlocksMap.java
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/branch-1.2/src/hdfs/org/apache/hadoop/hdfs/server/namenode/BlocksMap.java?rev=1468699r1=1468698r2=1468699view=diff
==
---
hadoop/common/branches/branch-1.2/src/hdfs/org/apache/hadoop/hdfs/server/namenode/BlocksMap.java
(original)
+++
hadoop/common/branches/branch-1.2/src/hdfs/org/apache/hadoop/hdfs/server/namenode/BlocksMap.java
Wed Apr 17 00:24:24 2013
@@ -315,37 +315,11 @@ class BlocksMap {
private GSetBlock, BlockInfo blocks;
BlocksMap(int initialCapacity, float loadFactor) {
-this.capacity = computeCapacity();
+// Use 2% of total memory to size the GSet capacity
+this.capacity = LightWeightGSet.computeCapacity(2.0, BlocksMap);
this.blocks = new LightWeightGSetBlock, BlockInfo(capacity);
}
- /**
- * Let t = 2% of max memory.
- * Let e = round(log_2 t).
- * Then, we choose capacity = 2^e/(size of reference),
- * unless it is outside the close interval [1, 2^30].
- */
- private static int computeCapacity() {
-//VM detection
-//See http://java.sun.com/docs/hotspot/HotSpotFAQ.html#64bit_detection
-final String vmBit = System.getProperty(sun.arch.data.model);
-
-//2% of max memory
-final double twoPC = Runtime.getRuntime().maxMemory()/50.0;
-
-//compute capacity
-final int e1 = (int)(Math.log(twoPC)/Math.log(2.0) + 0.5);
-final int e2 = e1 - (32.equals(vmBit)? 2: 3);
-final int exponent = e2 0? 0: e2 30? 30: e2;
-final int c = 1 exponent;
-
-LightWeightGSet.LOG.info(VM type = + vmBit + -bit);
-LightWeightGSet.LOG.info(2% max memory = + twoPC/(1 20) + MB);
-LightWeightGSet.LOG.info(capacity = 2^ + exponent
-+ = + c + entries);
-return c;
- }
-
void close() {
blocks = null;
}
Modified:
hadoop/common/branches/branch-1.2/src/hdfs/org/apache/hadoop/hdfs/util/LightWeightGSet.java
URL:
http://svn.apache.org/viewvc/hadoop/common/branches/branch-1.2/src/hdfs/org/apache/hadoop/hdfs/util/LightWeightGSet.java?rev=1468699r1=1468698r2=1468699view=diff
==
---
hadoop/common/branches/branch-1.2/src/hdfs/org/apache/hadoop/hdfs/util/LightWeightGSet.java
(original)
+++
hadoop/common/branches/branch-1.2/src/hdfs/org/apache/hadoop/hdfs/util/LightWeightGSet.java
Wed Apr 17 00:24:24 2013
@@ -280,4 +280,53 @@ public class LightWeightGSetK, E extend
throw new UnsupportedOperationException(Remove is not supported.);
}
}
+
+ /**
+ * Let t = percentage of max memory.
+ * Let e = round(log_2 t).
+ * Then, we choose capacity = 2^e/(size of reference),
+ * unless it is outside the close interval [1, 2^30].
+ */
+ public static int computeCapacity(double percentage, String mapName) {
+return computeCapacity(Runtime.getRuntime().maxMemory(), percentage,
+mapName);
+ }
+
+ /** Visible for testing */
+ static int computeCapacity(long maxMemory, double percentage,
+ String mapName) {
+if (percentage 100.0 || percentage 0.0) {
+ throw new IllegalArgumentException(Percentage + percentage
+ + must be greater than or equal to 0
+ + and less than or equal to 100);
+}
+if (maxMemory 0) {
+ throw new IllegalArgumentException(Memory + maxMemory
+ + must be greater than or equal to 0);
+}
+if (percentage == 0.0 || maxMemory == 0) {
+ return 0;
+}
+//VM detection
+//See http://java.sun.com/docs/hotspot/HotSpotFAQ.html#64bit_detection
+final String vmBit =
svn commit: r1468731 - /hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt
Author: llu Date: Wed Apr 17 02:57:29 2013 New Revision: 1468731 URL: http://svn.apache.org/r1468731 Log: Fix HADOOP-9467 attribution. Modified: hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt Modified: hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt URL: http://svn.apache.org/viewvc/hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt?rev=1468731r1=1468730r2=1468731view=diff == --- hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt (original) +++ hadoop/common/trunk/hadoop-common-project/hadoop-common/CHANGES.txt Wed Apr 17 02:57:29 2013 @@ -641,7 +641,7 @@ Release 2.0.4-alpha - UNRELEASED BUG FIXES HADOOP-9467. Metrics2 record filter should check name as well as tags. -(Ganeshan Iyler via llu) +(Chris Nauroth and Ganeshan Iyler via llu) HADOOP-9406. hadoop-client leaks dependency on JDK tools jar. (tucu)
svn commit: r1468732 - /hadoop/common/branches/branch-1/CHANGES.txt
Author: llu Date: Wed Apr 17 03:00:31 2013 New Revision: 1468732 URL: http://svn.apache.org/r1468732 Log: Fix HADOOP-9467 attribution. Modified: hadoop/common/branches/branch-1/CHANGES.txt Modified: hadoop/common/branches/branch-1/CHANGES.txt URL: http://svn.apache.org/viewvc/hadoop/common/branches/branch-1/CHANGES.txt?rev=1468732r1=1468731r2=1468732view=diff == --- hadoop/common/branches/branch-1/CHANGES.txt (original) +++ hadoop/common/branches/branch-1/CHANGES.txt Wed Apr 17 03:00:31 2013 @@ -242,7 +242,7 @@ Release 1.2.0 - unreleased BUG FIXES HADOOP-9467. Metrics2 record filter should check name as well as tags. -(Ganeshan Iyler via llu) +(Chris Nauroth and Ganeshan Iyler via llu) HADOOP-9451. Fault single-layer config if node group topology is enabled. (Junping Du via llu)
