Re: [VOTE] Release Apache Hadoop 2.1.1-beta
On Mon, Sep 16, 2013 at 11:38 PM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (rc0) for hadoop-2.1.1-beta that I would like to get released - this release fixes a number of bugs on top of hadoop-2.1.0-beta as a result of significant amounts of testing. If things go well, this might be the last of the *beta* releases of hadoop-2.x. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.1.1-beta-rc0 The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.1-beta-rc0 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. Short of HDFS-5225 from the Bigtop perspective this RC gets a +1. All tests passed in both secure and unsecure modes in 4 nodes pseudo distributed cluster with all the members of Hadoop ecosystem running smoke tests. Thanks, Roman.
Hadoop-Mapreduce-trunk - Build # 1557 - Still Failing
See https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1557/ ### ## LAST 60 LINES OF THE CONSOLE ### [...truncated 30538 lines...] Running org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServicesTasks Tests run: 20, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 8.175 sec - in org.apache.hadoop.mapreduce.v2.app.webapp.TestAMWebServicesTasks Running org.apache.hadoop.mapreduce.v2.app.TestKill Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.229 sec - in org.apache.hadoop.mapreduce.v2.app.TestKill Running org.apache.hadoop.mapreduce.v2.app.TestStagingCleanup Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.869 sec - in org.apache.hadoop.mapreduce.v2.app.TestStagingCleanup Running org.apache.hadoop.mapreduce.v2.app.commit.TestCommitterEventHandler Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.289 sec - in org.apache.hadoop.mapreduce.v2.app.commit.TestCommitterEventHandler Running org.apache.hadoop.mapreduce.v2.app.launcher.TestContainerLauncherImpl Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.723 sec - in org.apache.hadoop.mapreduce.v2.app.launcher.TestContainerLauncherImpl Running org.apache.hadoop.mapreduce.v2.app.launcher.TestContainerLauncher Tests run: 3, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 13.694 sec - in org.apache.hadoop.mapreduce.v2.app.launcher.TestContainerLauncher Running org.apache.hadoop.mapreduce.jobhistory.TestJobHistoryEventHandler Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 10.33 sec - in org.apache.hadoop.mapreduce.jobhistory.TestJobHistoryEventHandler Running org.apache.hadoop.mapreduce.jobhistory.TestEvents Tests run: 4, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.714 sec - in org.apache.hadoop.mapreduce.jobhistory.TestEvents Results : Tests run: 236, Failures: 0, Errors: 0, Skipped: 0 [INFO] [INFO] Reactor Summary: [INFO] [INFO] hadoop-mapreduce-client ... SUCCESS [2.541s] [INFO] hadoop-mapreduce-client-core .. SUCCESS [38.559s] [INFO] hadoop-mapreduce-client-common SUCCESS [27.217s] [INFO] hadoop-mapreduce-client-shuffle ... SUCCESS [2.403s] [INFO] hadoop-mapreduce-client-app ... FAILURE [5:29.057s] [INFO] hadoop-mapreduce-client-hs SKIPPED [INFO] hadoop-mapreduce-client-jobclient . SKIPPED [INFO] hadoop-mapreduce-client-hs-plugins SKIPPED [INFO] Apache Hadoop MapReduce Examples .. SKIPPED [INFO] hadoop-mapreduce .. SKIPPED [INFO] [INFO] BUILD FAILURE [INFO] [INFO] Total time: 6:40.411s [INFO] Finished at: Mon Sep 23 13:24:19 UTC 2013 [INFO] Final Memory: 22M/182M [INFO] [ERROR] Failed to execute goal org.apache.maven.plugins:maven-surefire-plugin:2.16:test (default-test) on project hadoop-mapreduce-client-app: ExecutionException; nested exception is java.util.concurrent.ExecutionException: java.lang.RuntimeException: The forked VM terminated without saying properly goodbye. VM crash or System.exit called ? [ERROR] Command was/bin/sh -c cd /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app /home/jenkins/tools/java/jdk1.6.0_26/jre/bin/java -Xmx1024m -XX:+HeapDumpOnOutOfMemoryError -jar /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/target/surefire/surefirebooter2680430168702368923.jar /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/target/surefire/surefire326753898507750tmp /home/jenkins/jenkins-slave/workspace/Hadoop-Mapreduce-trunk/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/target/surefire/surefire_558639605743818416793tmp [ERROR] - [Help 1] [ERROR] [ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch. [ERROR] Re-run Maven using the -X switch to enable full debug logging. [ERROR] [ERROR] For more information about the errors and possible solutions, please read the following articles: [ERROR] [Help 1] http://cwiki.apache.org/confluence/display/MAVEN/MojoFailureException [ERROR] [ERROR] After correcting the problems, you can resume the build with the command [ERROR] mvn goals -rf :hadoop-mapreduce-client-app Build step 'Execute shell' marked build as failure
[jira] [Created] (MAPREDUCE-5527) Add CONTAINERS_MILLIS_MAPS|REDUCES counters
Sandy Ryza created MAPREDUCE-5527: - Summary: Add CONTAINERS_MILLIS_MAPS|REDUCES counters Key: MAPREDUCE-5527 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5527 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Sandy Ryza It would be helpful to have a counters which report the total wallclock time spent in all map/reduce tasks. This is what SLOTS_MILLIS_MAPS usually did in MR1. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [VOTE] Release Apache Hadoop 2.1.1-beta
Correct me if I am wrong, but FWIU, we already released a beta with the same symlink issues. Given 2.1.1 is just another beta, I believe we can go ahead with it and resolve the issues in the final GA release. Instead of resetting the testing done by everyone. It's a hard story to sell but beta phase is supposed to be only about bug-fixes, but incompatible changes that cannot be avoided, well cannot be avoided. Thanks, +Vinod On Sep 23, 2013, at 11:42 AM, Andrew Wang wrote: We still need to resolve some symlink issues; are we planning to spin a new RC? Leaving it as-is is not a good option. On Sun, Sep 22, 2013 at 11:23 PM, Roman Shaposhnik r...@apache.org wrote: On Mon, Sep 16, 2013 at 11:38 PM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (rc0) for hadoop-2.1.1-beta that I would like to get released - this release fixes a number of bugs on top of hadoop-2.1.0-beta as a result of significant amounts of testing. If things go well, this might be the last of the *beta* releases of hadoop-2.x. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.1.1-beta-rc0 The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.1-beta-rc0 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. Short of HDFS-5225 from the Bigtop perspective this RC gets a +1. All tests passed in both secure and unsecure modes in 4 nodes pseudo distributed cluster with all the members of Hadoop ecosystem running smoke tests. Thanks, Roman. -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. signature.asc Description: Message signed with OpenPGP using GPGMail
Re: [VOTE] Release Apache Hadoop 2.1.1-beta
Are we doing a new RC for 2.1.1-beta? On Mon, Sep 23, 2013 at 9:04 PM, Vinod Kumar Vavilapalli vino...@apache.org wrote: Correct me if I am wrong, but FWIU, we already released a beta with the same symlink issues. Given 2.1.1 is just another beta, I believe we can go ahead with it and resolve the issues in the final GA release. Instead of resetting the testing done by everyone. It's a hard story to sell but beta phase is supposed to be only about bug-fixes, but incompatible changes that cannot be avoided, well cannot be avoided. Thanks, +Vinod On Sep 23, 2013, at 11:42 AM, Andrew Wang wrote: We still need to resolve some symlink issues; are we planning to spin a new RC? Leaving it as-is is not a good option. On Sun, Sep 22, 2013 at 11:23 PM, Roman Shaposhnik r...@apache.org wrote: On Mon, Sep 16, 2013 at 11:38 PM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (rc0) for hadoop-2.1.1-beta that I would like to get released - this release fixes a number of bugs on top of hadoop-2.1.0-beta as a result of significant amounts of testing. If things go well, this might be the last of the *beta* releases of hadoop-2.x. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.1.1-beta-rc0 The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.1-beta-rc0 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. Short of HDFS-5225 from the Bigtop perspective this RC gets a +1. All tests passed in both secure and unsecure modes in 4 nodes pseudo distributed cluster with all the members of Hadoop ecosystem running smoke tests. Thanks, Roman. CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Alejandro
Re: [VOTE] Release Apache Hadoop 2.1.1-beta
+1 (binding) Verified the signatures and hashes for both src and binary tars. Built from the source, the binary distribution and the documentation. Started a single node cluster and tested the following: # Started HDFS cluster, verified the hdfs CLI commands such ls, copying data back and forth, verified namenode webUI etc. # Ran some tests such as sleep job, TestDFSIO, NNBench etc. On Mon, Sep 16, 2013 at 11:38 PM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (rc0) for hadoop-2.1.1-beta that I would like to get released - this release fixes a number of bugs on top of hadoop-2.1.0-beta as a result of significant amounts of testing. If things go well, this might be the last of the *beta* releases of hadoop-2.x. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.1.1-beta-rc0 The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.1-beta-rc0 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- http://hortonworks.com/download/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You.
[jira] [Created] (MAPREDUCE-5528) TeraSort fails with can't read paritions file - does not read partition file from distributed cache
Albert Chu created MAPREDUCE-5528:
-
Summary: TeraSort fails with can't read paritions file - does
not read partition file from distributed cache
Key: MAPREDUCE-5528
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5528
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: examples
Affects Versions: 3.0.0
Reporter: Albert Chu
Priority: Minor
I was trying to run TeraSort against a parallel networked file system,
setting things up via the 'file:// scheme. I always got the
following error when running terasort:
{noformat}
13/09/23 11:15:12 INFO mapreduce.Job: Task Id :
attempt_1379960046506_0001_m_80_1, Status : FAILED
Error: java.lang.IllegalArgumentException: can't read paritions file
at
org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.setConf(TeraSort.java:254)
at
org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:73)
at
org.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.init(MapTask.java:678)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:747)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:171)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:396)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1499)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:166)
Caused by: java.io.FileNotFoundException: File _partition.lst does not exist
at org.apache.hadoop.fs.Stat.parseExecResult(Stat.java:124)
at org.apache.hadoop.util.Shell.runCommand(Shell.java:486)
at org.apache.hadoop.util.Shell.run(Shell.java:417)
at org.apache.hadoop.fs.Stat.getFileStatus(Stat.java:74)
at
org.apache.hadoop.fs.RawLocalFileSystem.getNativeFileLinkStatus(RawLocalFileSystem.java:808)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileLinkStatusInternal(RawLocalFileSystem.java:740)
at
org.apache.hadoop.fs.RawLocalFileSystem.getFileStatus(RawLocalFileSystem.java:525)
at
org.apache.hadoop.fs.FilterFileSystem.getFileStatus(FilterFileSystem.java:398)
at
org.apache.hadoop.fs.ChecksumFileSystem$ChecksumFSInputChecker.init(ChecksumFileSystem.java:137)
at
org.apache.hadoop.fs.ChecksumFileSystem.open(ChecksumFileSystem.java:339)
at org.apache.hadoop.fs.FileSystem.open(FileSystem.java:763)
at
org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.readPartitions(TeraSort.java:161)
at
org.apache.hadoop.examples.terasort.TeraSort$TotalOrderPartitioner.setConf(TeraSort.java:246)
... 10 more
{noformat}
After digging into TeraSort, I noticed that the partitions file was
created in the output directory, then added into the distributed cache
{noformat}
Path outputDir = new Path(args[1]);
...
Path partitionFile = new Path(outputDir, TeraInputFormat.PARTITION_FILENAME);
...
job.addCacheFile(partitionUri);
{noformat}
but the partitions file doesn't seem to be read back from the output
directory or distributed cache:
{noformat}
FileSystem fs = FileSystem.getLocal(conf);
...
Path partFile = new Path(TeraInputFormat.PARTITION_FILENAME);
splitPoints = readPartitions(fs, partFile, conf);
{noformat}
It seems the file is being read from whatever the working directory is
for the filesystem returned from FileSystem.getLocal(conf).
Under HDFS this code works, the working directory seems to be the
distributed cache (I guess by default??).
But when I set things up with the networked file system and 'file://'
scheme, the working directory was the directory I was running my
Hadoop binaries out of.
The attached patch fixed things for me. It grabs the partition file from the
distributed cache all of the time, instead of trusting things underneath to
work out. It seems to be the right thing to do???
Apologies, I was unable to get this to reproduce under the TeraSort
example tests, such as TestTeraSort.java, so no test added. Not sure what the
subtle difference is in the setup. I tested under both HDFS 'file' scheme
and the patch worked under both.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-5529) Binary incompatibilities in mapred.lib.TotalOrderPartitioner between branch-1 and branch-2
Robert Kanter created MAPREDUCE-5529:
Summary: Binary incompatibilities in
mapred.lib.TotalOrderPartitioner between branch-1 and branch-2
Key: MAPREDUCE-5529
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5529
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv1, mrv2
Affects Versions: 2.1.1-beta
Reporter: Robert Kanter
Assignee: Robert Kanter
Priority: Blocker
{{mapred.lib.TotalPartitioner}} in branch-1 has these two methods:
{code:java}
public static String getPartitionFile(JobConf job)
public static void setPartitionFile(JobConf job, Path p)
{code}
In branch-2, {{mapred.lib.TotalPartitioner}} is now a subclass of
{{mapreduce.lib.TotalPartitioner}}, from which it inherits the similar methods:
{code:java}
public static String getPartitionFile(Configuration conf)
public static void setPartitionFile(Configuration conf, Path p)
{code}
This means that any code that does either of the following:
{code:java}
TotalOrderPartitioner.setPartitionFile(new JobConf(), new Path(/));
String str = TotalOrderPartitioner.getPartitionFile(new JobConf());
{code}
will not be binary compatible (that is, if compiled against branch-1, it will
throw a {{NoSuchMethodError}} if run against branch-2).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
Re: [VOTE] Release Apache Hadoop 2.1.1-beta
Vote for the 2.1.1-beta release is closing tonight, while we had quite a few +1s, it seems we need to address the following before doing a release: symlink discussion: get a concrete and explicit understanding on what we will do and in what release(s). Also, the following JIRAs seem nasty enough to require a new RC: https://issues.apache.org/jira/browse/HDFS-5225 (no patch avail) https://issues.apache.org/jira/browse/HDFS-5228 (patch avail) https://issues.apache.org/jira/browse/YARN-1089 (patch avail) https://issues.apache.org/jira/browse/MAPREDUCE-5529 (patch avail) I won't -1 the release but I'm un-casting my vote as I think we should address these things before. Thanks. Alejandro On Tue, Sep 24, 2013 at 1:49 AM, Suresh Srinivas sur...@hortonworks.comwrote: +1 (binding) Verified the signatures and hashes for both src and binary tars. Built from the source, the binary distribution and the documentation. Started a single node cluster and tested the following: # Started HDFS cluster, verified the hdfs CLI commands such ls, copying data back and forth, verified namenode webUI etc. # Ran some tests such as sleep job, TestDFSIO, NNBench etc. On Mon, Sep 16, 2013 at 11:38 PM, Arun C Murthy a...@hortonworks.com wrote: Folks, I've created a release candidate (rc0) for hadoop-2.1.1-beta that I would like to get released - this release fixes a number of bugs on top of hadoop-2.1.0-beta as a result of significant amounts of testing. If things go well, this might be the last of the *beta* releases of hadoop-2.x. The RC is available at: http://people.apache.org/~acmurthy/hadoop-2.1.1-beta-rc0 The RC tag in svn is here: http://svn.apache.org/repos/asf/hadoop/common/tags/release-2.1.1-beta-rc0 The maven artifacts are available via repository.apache.org. Please try the release and vote; the vote will run for the usual 7 days. thanks, Arun -- Arun C. Murthy Hortonworks Inc. http://hortonworks.com/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- http://hortonworks.com/download/ -- CONFIDENTIALITY NOTICE NOTICE: This message is intended for the use of the individual or entity to which it is addressed and may contain information that is confidential, privileged and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any printing, copying, dissemination, distribution, disclosure or forwarding of this communication is strictly prohibited. If you have received this communication in error, please contact the sender immediately and delete it from your system. Thank You. -- Alejandro
[jira] [Created] (MAPREDUCE-5530) Binary and source incompatibility in mapred.lib.CombineFileInputFormat between branch-1 and branch-2
Robert Kanter created MAPREDUCE-5530:
Summary: Binary and source incompatibility in
mapred.lib.CombineFileInputFormat between branch-1 and branch-2
Key: MAPREDUCE-5530
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5530
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv1, mrv2
Affects Versions: 2.1.1-beta
Reporter: Robert Kanter
Assignee: Robert Kanter
Priority: Blocker
{{mapred.lib.CombineFileInputFormat}} in branch-1 has this method:
{code:java}
protected boolean isSplitable(FileSystem fs, Path file)
{code}
In branch-2, {{mapred.lib.CombineFileInputFormat}} is now a subclass of
{{mapreduce.lib.input.CombineFileInputFormat}}, from which it inherits the
similar method:
{code:java}
protected boolean isSplitable(JobContext context, Path file)
{code}
This means that any code that subclasses {{mapred.lib.CombineFileInputFormat}}
and does not provide its own implementation of {{protected boolean
isSplitable(FileSystem fs, Path file)}} will not be binary or source compatible
if it tries to call {{isSplitable}} with a {{FileSystem}} argument anywhere
(that is, if compiled against branch-1, it will throw a {{NoSuchMethodError}}
if run against branch-2; also, it won't even compile against branch-2).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-5508) JobTracker memory leak caused by unreleased FileSystem objects in JobInProgress#cleanupJob
[
https://issues.apache.org/jira/browse/MAPREDUCE-5508?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Chris Nauroth resolved MAPREDUCE-5508.
--
Resolution: Fixed
Fix Version/s: 1.3.0
1-win
Target Version/s: 1-win, 1.3.0
Hadoop Flags: Reviewed
I have committed this to branch-1 and branch-1-win. Xi, thank you for
providing a patch for this tricky issue. Sandy, thank you for help with code
reviews.
JobTracker memory leak caused by unreleased FileSystem objects in
JobInProgress#cleanupJob
--
Key: MAPREDUCE-5508
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5508
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: jobtracker
Affects Versions: 1-win, 1.2.1
Reporter: Xi Fang
Assignee: Xi Fang
Priority: Critical
Fix For: 1-win, 1.3.0
Attachments: MAPREDUCE-5508.1.patch, MAPREDUCE-5508.2.patch,
MAPREDUCE-5508.3.patch, MAPREDUCE-5508.patch
MAPREDUCE-5351 fixed a memory leak problem but introducing another filesystem
object (see tempDirFs) that is not properly released.
{code} JobInProgress#cleanupJob()
void cleanupJob() {
...
tempDirFs = jobTempDirPath.getFileSystem(conf);
CleanupQueue.getInstance().addToQueue(
new PathDeletionContext(jobTempDirPath, conf, userUGI, jobId));
...
if (tempDirFs != fs) {
try {
fs.close();
} catch (IOException ie) {
...
}
{code}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
