[jira] [Updated] (MAPREDUCE-7298) Distcp doesn't close the job after the job is completed
[ https://issues.apache.org/jira/browse/MAPREDUCE-7298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated MAPREDUCE-7298: - Component/s: distcp > Distcp doesn't close the job after the job is completed > --- > > Key: MAPREDUCE-7298 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7298 > Project: Hadoop Map/Reduce > Issue Type: Task > Components: distcp >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Fix For: 3.4.0 > > Attachments: MAPREDUCE-7298.01.patch, MAPREDUCE-7298.02.patch > > > Distcp doesn't close the job after the job is completed. This leads to leaked > Truststore Reloader Threads. > The fix is to close the job once it is complete. job.close internally calls > yarnClient.close(), which then calls timelineConnector.serviceStop() . This > destroys the sslFactory cleaning up the ReloadingX509TrustManager. > Without the patch for each distcp job, a new ReloadingX509TrustManager is > created which creates a new thread. These threads are never killed and they > remain like that till HS2 is restarted. With the close, the thread will be > cleaned up once the job is completed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7298) Distcp doesn't close the job after the job is completed
[ https://issues.apache.org/jira/browse/MAPREDUCE-7298?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated MAPREDUCE-7298: - Fix Version/s: 3.4.0 Hadoop Flags: Reviewed Resolution: Fixed Status: Resolved (was: Patch Available) Committed to trunk. Thank you for the contribution [~aasha], and thanks [~wangda], [~shashikant] for reviewing. > Distcp doesn't close the job after the job is completed > --- > > Key: MAPREDUCE-7298 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7298 > Project: Hadoop Map/Reduce > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Fix For: 3.4.0 > > Attachments: MAPREDUCE-7298.01.patch, MAPREDUCE-7298.02.patch > > > Distcp doesn't close the job after the job is completed. This leads to leaked > Truststore Reloader Threads. > The fix is to close the job once it is complete. job.close internally calls > yarnClient.close(), which then calls timelineConnector.serviceStop() . This > destroys the sslFactory cleaning up the ReloadingX509TrustManager. > Without the patch for each distcp job, a new ReloadingX509TrustManager is > created which creates a new thread. These threads are never killed and they > remain like that till HS2 is restarted. With the close, the thread will be > cleaned up once the job is completed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7298) Distcp doesn't close the job after the job is completed
[ https://issues.apache.org/jira/browse/MAPREDUCE-7298?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17206257#comment-17206257 ] Arpit Agarwal commented on MAPREDUCE-7298: -- l commit this shortly based on Wangda's +1. The checkstyle issues are on lines that were not modified by this patch. > Distcp doesn't close the job after the job is completed > --- > > Key: MAPREDUCE-7298 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7298 > Project: Hadoop Map/Reduce > Issue Type: Task >Reporter: Aasha Medhi >Assignee: Aasha Medhi >Priority: Major > Attachments: MAPREDUCE-7298.01.patch, MAPREDUCE-7298.02.patch > > > Distcp doesn't close the job after the job is completed. This leads to leaked > Truststore Reloader Threads. > The fix is to close the job once it is complete. job.close internally calls > yarnClient.close(), which then calls timelineConnector.serviceStop() . This > destroys the sslFactory cleaning up the ReloadingX509TrustManager. > Without the patch for each distcp job, a new ReloadingX509TrustManager is > created which creates a new thread. These threads are never killed and they > remain like that till HS2 is restarted. With the close, the thread will be > cleaned up once the job is completed. -- This message was sent by Atlassian Jira (v8.3.4#803005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7066) TestQueue fails on Java9
[
https://issues.apache.org/jira/browse/MAPREDUCE-7066?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal updated MAPREDUCE-7066:
-
Fix Version/s: 3.1.1
> TestQueue fails on Java9
>
>
> Key: MAPREDUCE-7066
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7066
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: test
> Environment: Applied HADOOP-12760 and HDFS-11610
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Fix For: 3.2.0, 3.1.1
>
> Attachments: MAPREDUCE-7066.1.patch
>
>
> {noformat}
> [INFO] Running org.apache.hadoop.mapred.TestQueue
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.353
> s <<< FAILURE! - in org.apache.hadoop.mapred.TestQueue
> [ERROR] testQueue(org.apache.hadoop.mapred.TestQueue) Time elapsed: 1.186 s
> <<< FAILURE!
> org.junit.ComparisonFailure:
> expected:<...roperties":[{"key":"[capacity","value":"20"},{"key":"user-limit","value":"3]0"}],"children":[]}]...>
> but
> was:<...roperties":[{"key":"[user-limit","value":"30"},{"key":"capacity","value":"2]0"}],"children":[]}]...>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at org.apache.hadoop.mapred.TestQueue.testQueue(TestQueue.java:156)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:564)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-7066) TestQueue fails on Java9
[
https://issues.apache.org/jira/browse/MAPREDUCE-7066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16437631#comment-16437631
]
Arpit Agarwal commented on MAPREDUCE-7066:
--
Cherry-picked to branch-3.1.
> TestQueue fails on Java9
>
>
> Key: MAPREDUCE-7066
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7066
> Project: Hadoop Map/Reduce
> Issue Type: Test
> Components: test
> Environment: Applied HADOOP-12760 and HDFS-11610
>Reporter: Takanobu Asanuma
>Assignee: Takanobu Asanuma
>Priority: Major
> Fix For: 3.2.0, 3.1.1
>
> Attachments: MAPREDUCE-7066.1.patch
>
>
> {noformat}
> [INFO] Running org.apache.hadoop.mapred.TestQueue
> [ERROR] Tests run: 1, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 1.353
> s <<< FAILURE! - in org.apache.hadoop.mapred.TestQueue
> [ERROR] testQueue(org.apache.hadoop.mapred.TestQueue) Time elapsed: 1.186 s
> <<< FAILURE!
> org.junit.ComparisonFailure:
> expected:<...roperties":[{"key":"[capacity","value":"20"},{"key":"user-limit","value":"3]0"}],"children":[]}]...>
> but
> was:<...roperties":[{"key":"[user-limit","value":"30"},{"key":"capacity","value":"2]0"}],"children":[]}]...>
> at org.junit.Assert.assertEquals(Assert.java:115)
> at org.junit.Assert.assertEquals(Assert.java:144)
> at org.apache.hadoop.mapred.TestQueue.testQueue(TestQueue.java:156)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.base/java.lang.reflect.Method.invoke(Method.java:564)
> at
> org.junit.runners.model.FrameworkMethod$1.runReflectiveCall(FrameworkMethod.java:47)
> at
> org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
> at
> org.junit.runners.model.FrameworkMethod.invokeExplosively(FrameworkMethod.java:44)
> at
> org.junit.internal.runners.statements.InvokeMethod.evaluate(InvokeMethod.java:17)
> at
> org.junit.internal.runners.statements.FailOnTimeout$StatementThread.run(FailOnTimeout.java:74)
> {noformat}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7061) SingleCluster setup document needs to be updated
[ https://issues.apache.org/jira/browse/MAPREDUCE-7061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated MAPREDUCE-7061: - Resolution: Fixed Hadoop Flags: Reviewed Fix Version/s: 3.2.0 Status: Resolved (was: Patch Available) +1 I've committed this. Thanks [~bharatviswa]. > SingleCluster setup document needs to be updated > > > Key: MAPREDUCE-7061 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7061 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Fix For: 3.2.0 > > Attachments: HDFS-13160.00.patch > > > The following document needs an update: > https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html > We need to set mapreduce.application.classpath, without that we cannot launch > MR jobs using yarn. > CLASSPATH for MR applications. A comma-separated list of CLASSPATH entries. > If mapreduce.application.framework is set then this must specify the > appropriate classpath for that archive, and the name of the archive must be > present in the classpath. If mapreduce.app-submission.cross-platform is > false, platform-specific environment vairable expansion syntax would be used > to construct the default CLASSPATH entries. > > So, add the default value for the tarball download setup. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Moved] (MAPREDUCE-7061) SingleCluster setup document needs to be updated
[ https://issues.apache.org/jira/browse/MAPREDUCE-7061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal moved HDFS-13160 to MAPREDUCE-7061: - Key: MAPREDUCE-7061 (was: HDFS-13160) Project: Hadoop Map/Reduce (was: Hadoop HDFS) > SingleCluster setup document needs to be updated > > > Key: MAPREDUCE-7061 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-7061 > Project: Hadoop Map/Reduce > Issue Type: Improvement >Reporter: Bharat Viswanadham >Assignee: Bharat Viswanadham >Priority: Major > Attachments: HDFS-13160.00.patch > > > The following document needs an update: > https://hadoop.apache.org/docs/stable/hadoop-project-dist/hadoop-common/SingleCluster.html > We need to set mapreduce.application.classpath, without that we cannot launch > MR jobs using yarn. > CLASSPATH for MR applications. A comma-separated list of CLASSPATH entries. > If mapreduce.application.framework is set then this must specify the > appropriate classpath for that archive, and the name of the archive must be > present in the classpath. If mapreduce.app-submission.cross-platform is > false, platform-specific environment vairable expansion syntax would be used > to construct the default CLASSPATH entries. > > So, add the default value for the tarball download setup. -- This message was sent by Atlassian JIRA (v7.6.3#76005) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Assigned] (MAPREDUCE-7035) Fix javadoc issues in hadoop-mapreduce-client
[
https://issues.apache.org/jira/browse/MAPREDUCE-7035?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Arpit Agarwal reassigned MAPREDUCE-7035:
Assignee: Mukul Kumar Singh
> Fix javadoc issues in hadoop-mapreduce-client
> -
>
> Key: MAPREDUCE-7035
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7035
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: client
>Reporter: Mukul Kumar Singh
>Assignee: Mukul Kumar Singh
>Priority: Minor
> Attachments: MAPREDUCE-7035.001.patch
>
>
> Auto-generated code in map-reduce client throws many javadoc warnings.
> The warnings are thrown for the following auto-generated sources
> {code:java}
> hadoop-mapreduce-client-core/pom.xml
> target/generated-sources/avro/
> hadoop-mapreduce-client-common/pom.xml
> org.apache.hadoop.yarn.proto
> org.apache.hadoop.mapreduce.v2.proto
> org.apache.hadoop.mapreduce.v2.hs.proto
> hadoop-mapreduce-client-shuffle/pom.xml
> org.apache.hadoop.mapred.proto{code}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6791) remove unnecessary dependency from hadoop-mapreduce-client-jobclient to hadoop-mapreduce-client-shuffle
[ https://issues.apache.org/jira/browse/MAPREDUCE-6791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated MAPREDUCE-6791: - Hadoop Flags: Incompatible change,Reviewed (was: Reviewed) > remove unnecessary dependency from hadoop-mapreduce-client-jobclient to > hadoop-mapreduce-client-shuffle > --- > > Key: MAPREDUCE-6791 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6791 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 3.0.0-alpha1 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Minor > Labels: Incompatible > Fix For: 3.0.0-alpha2 > > Attachments: mapreduce6791.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-6791) remove unnecessary dependency from hadoop-mapreduce-client-jobclient to hadoop-mapreduce-client-shuffle
[ https://issues.apache.org/jira/browse/MAPREDUCE-6791?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated MAPREDUCE-6791: - Labels: Incompatible (was: ) > remove unnecessary dependency from hadoop-mapreduce-client-jobclient to > hadoop-mapreduce-client-shuffle > --- > > Key: MAPREDUCE-6791 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6791 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: mrv2 >Affects Versions: 3.0.0-alpha1 >Reporter: Haibo Chen >Assignee: Haibo Chen >Priority: Minor > Labels: Incompatible > Fix For: 3.0.0-alpha2 > > Attachments: mapreduce6791.001.patch > > -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-4522) DBOutputFormat Times out on large batch inserts
[ https://issues.apache.org/jira/browse/MAPREDUCE-4522?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated MAPREDUCE-4522: - Status: Patch Available (was: Open) Hi [~shyam_gav], submitting a patch just triggers a pre-commit build with unit tests. You don't need a code review to click 'submit patch'. I'll let a MapReduce/YARN dev take a look at the actual patch. > DBOutputFormat Times out on large batch inserts > --- > > Key: MAPREDUCE-4522 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4522 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: task-controller >Affects Versions: 0.20.205.0 >Reporter: Nathan Jarus >Assignee: Shyam Gavulla > Labels: newbie > Attachments: MAPREDUCE-4522.001.patch > > > In DBRecordWriter#close(), progress is never updated. In large batch inserts, > this can cause the reduce task to time out due to the amount of time it takes > the SQL engine to process that insert. > Potential solutions I can see: > Don't batch inserts; do the insert when DBRecordWriter#write() is called > (awful) > Spin up a thread in DBRecordWriter#close() and update progress in that. > (gross) > I can provide code for either if you're interested. -- This message was sent by Atlassian JIRA (v6.3.4#6332) - To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Commented] (MAPREDUCE-6563) Streaming documentation contains a stray '%' character.
[ https://issues.apache.org/jira/browse/MAPREDUCE-6563?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15122341#comment-15122341 ] Arpit Agarwal commented on MAPREDUCE-6563: -- +1 > Streaming documentation contains a stray '%' character. > --- > > Key: MAPREDUCE-6563 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6563 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: documentation >Reporter: Chris Nauroth >Assignee: Chris Nauroth >Priority: Trivial > Attachments: MAPREDUCE-6563.001.patch > > > We have an unneeded '%' character above the title in this page. > http://hadoop.apache.org/docs/r2.7.1/hadoop-streaming/HadoopStreaming.html -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6471) Document distcp incremental copy
Arpit Agarwal created MAPREDUCE-6471:
Summary: Document distcp incremental copy
Key: MAPREDUCE-6471
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6471
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distcp
Affects Versions: 2.7.1
Reporter: Arpit Agarwal
MAPREDUCE-5899 added distcp support for incremental copy with a new {{append}}
flag.
It should be documented.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5852) Prepare MapReduce codebase for JUnit 4.11.
[
https://issues.apache.org/jira/browse/MAPREDUCE-5852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13976158#comment-13976158
]
Arpit Agarwal commented on MAPREDUCE-5852:
--
+1 for the patch. Nice catch in {{TestMRApps}}.
Prepare MapReduce codebase for JUnit 4.11.
--
Key: MAPREDUCE-5852
URL: https://issues.apache.org/jira/browse/MAPREDUCE-5852
Project: Hadoop Map/Reduce
Issue Type: Test
Components: test
Affects Versions: 3.0.0, 2.4.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Priority: Minor
Attachments: MAPREDUCE-5852.1.patch
HADOOP-10503 upgrades the entire Hadoop repo to use JUnit 4.11. Some of the
MapReduce code needs some minor updates to fix deprecation warnings before
the upgrade.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
[jira] [Commented] (MAPREDUCE-5794) SliveMapper always uses default FileSystem.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13933877#comment-13933877 ] Arpit Agarwal commented on MAPREDUCE-5794: -- +1 for the patch, I will commit it shortly. SliveMapper always uses default FileSystem. --- Key: MAPREDUCE-5794 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5794 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Reporter: Tsz Wo Nicholas Sze Assignee: Tsz Wo Nicholas Sze Priority: Minor Attachments: m5794_20140311.patch Similar to MAPREDUCE-5780, SliveMapper should use the test path to get FileSystem. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Updated] (MAPREDUCE-5794) SliveMapper always uses default FileSystem.
[ https://issues.apache.org/jira/browse/MAPREDUCE-5794?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated MAPREDUCE-5794: - Resolution: Fixed Fix Version/s: 2.4.0 3.0.0 Target Version/s: 2.4.0 Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) Committed from trunk through branch-2.4. Thanks Nicholas! SliveMapper always uses default FileSystem. --- Key: MAPREDUCE-5794 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5794 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Reporter: Tsz Wo Nicholas Sze Assignee: Tsz Wo Nicholas Sze Priority: Minor Fix For: 3.0.0, 2.4.0 Attachments: m5794_20140311.patch Similar to MAPREDUCE-5780, SliveMapper should use the test path to get FileSystem. -- This message was sent by Atlassian JIRA (v6.2#6252)
[jira] [Commented] (MAPREDUCE-5095) TestShuffleExceptionCount#testCheckException fails occasionally with JDK7
[ https://issues.apache.org/jira/browse/MAPREDUCE-5095?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13662260#comment-13662260 ] Arpit Agarwal commented on MAPREDUCE-5095: -- Hi Hitesh, Thanks for reviewing! abortCalled cannot be a non-static since it is referenced from a static nested class. TestShuffleExceptionCount#testCheckException fails occasionally with JDK7 - Key: MAPREDUCE-5095 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5095 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 1.1.2 Environment: Open JDK7 Reporter: Arpit Agarwal Assignee: Arpit Agarwal Fix For: 1.3.0 Attachments: MAPREDUCE-5095.patch Original Estimate: 1h Time Spent: 1h Remaining Estimate: 0h The test fails due a test-order dependency that can be violated when running with JDK 7. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4987) TestMRJobs#testDistributedCache fails on Windows due to unexpected behavior of symlinks
[
https://issues.apache.org/jira/browse/MAPREDUCE-4987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13603718#comment-13603718
]
Arpit Agarwal commented on MAPREDUCE-4987:
--
+1
Chris explained to me offline about the change in
FileUtil#createJarWithClassPath. Quoting here since I found it helpful to
understand the change.
{quote}
In the method sanitizeEnv, you'll see that nodemanager does various things to
set up a new environment for the container to be launched. The final state of
this environment will be different from the environment of the currently
running process (the nodemanager itself).
The most glaring problem with this bug was the setting of PWD to the new
container work directory. There are various classpath entries for the
distributed cache files that are of the form $PWD/file on Mac or %PWD%/file on
Windows, and FileUtil#createJarWithClassPath needs to expand this to
container_dir/file. Without this change, the variable expansion would be
incorrect: nodemanager_working_dir/file on Mac or just /file on Windows
(since Windows doesn't intrinsically have %PWD% defined until nodemanager sets
it in sanitizeEnv).
{quote}
TestMRJobs#testDistributedCache fails on Windows due to unexpected behavior
of symlinks
---
Key: MAPREDUCE-4987
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4987
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: distributed-cache, nodemanager
Affects Versions: 3.0.0
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: MAPREDUCE-4987.1.patch
On Windows, {{TestMRJobs#testDistributedCache}} fails on an assertion while
checking the length of a symlink. It expects to see the length of the target
of the symlink, but Java 6 on Windows always reports that a symlink has
length 0.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4969) TestKeyValueTextInputFormat test fails with Open JDK 7
[ https://issues.apache.org/jira/browse/MAPREDUCE-4969?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13567847#comment-13567847 ] Arpit Agarwal commented on MAPREDUCE-4969: -- Thanks, Suresh and thanks for fixing the JIRA fields. TestKeyValueTextInputFormat test fails with Open JDK 7 -- Key: MAPREDUCE-4969 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4969 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 1.2.0, 3.0.0, 1-win, trunk-win Reporter: Arpit Agarwal Assignee: Arpit Agarwal Fix For: 1.2.0, 2.0.3-alpha Attachments: HADOOP-9176.branch-1.patch, HADOOP-9176.patch, HADOOP-9176.trunk.patch RawLocalFileSystem.delete fails on Windows even when the files are not expected to be in use. It does not reproduce with Sun JDK 6. -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4909) TestKeyValueTextInputFormat fails with Open JDK 7 on Windows
[ https://issues.apache.org/jira/browse/MAPREDUCE-4909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated MAPREDUCE-4909: - Component/s: test TestKeyValueTextInputFormat fails with Open JDK 7 on Windows Key: MAPREDUCE-4909 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4909 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 1-win Reporter: Arpit Agarwal Assignee: Arpit Agarwal Attachments: MAPREDUCE-4909.patch, MAPREDUCE-4909.patch, MAPREDUCE-4909.patch, MAPREDUCE-4909.patch TestKeyValueTextInputFormat.testFormat fails with Open JDK 7. The root cause appears to be a failure to delete in-use files via LocalFileSystem.delete (RawLocalFileSystem.delete). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4909) TestKeyValueTextInputFormat fails with Open JDK 7 on Windows
[ https://issues.apache.org/jira/browse/MAPREDUCE-4909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated MAPREDUCE-4909: - Attachment: MAPREDUCE-4909.patch Removed the comments altogether. TestKeyValueTextInputFormat fails with Open JDK 7 on Windows Key: MAPREDUCE-4909 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4909 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 1-win Reporter: Arpit Agarwal Assignee: Arpit Agarwal Attachments: MAPREDUCE-4909.patch, MAPREDUCE-4909.patch, MAPREDUCE-4909.patch, MAPREDUCE-4909.patch TestKeyValueTextInputFormat.testFormat fails with Open JDK 7. The root cause appears to be a failure to delete in-use files via LocalFileSystem.delete (RawLocalFileSystem.delete). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4909) TestKeyValueTextInputFormat fails with Open JDK 7 on Windows
[ https://issues.apache.org/jira/browse/MAPREDUCE-4909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated MAPREDUCE-4909: - Affects Version/s: (was: 1-win) 1.2.0 TestKeyValueTextInputFormat fails with Open JDK 7 on Windows Key: MAPREDUCE-4909 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4909 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 1.2.0 Reporter: Arpit Agarwal Assignee: Arpit Agarwal Attachments: MAPREDUCE-4909.patch, MAPREDUCE-4909.patch, MAPREDUCE-4909.patch, MAPREDUCE-4909.patch TestKeyValueTextInputFormat.testFormat fails with Open JDK 7. The root cause appears to be a failure to delete in-use files via LocalFileSystem.delete (RawLocalFileSystem.delete). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4909) TestKeyValueTextInputFormat fails with Open JDK 7 on Windows
[ https://issues.apache.org/jira/browse/MAPREDUCE-4909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated MAPREDUCE-4909: - Fix Version/s: 1.2.0 TestKeyValueTextInputFormat fails with Open JDK 7 on Windows Key: MAPREDUCE-4909 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4909 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 1.2.0 Reporter: Arpit Agarwal Assignee: Arpit Agarwal Fix For: 1.2.0 Attachments: MAPREDUCE-4909.patch, MAPREDUCE-4909.patch, MAPREDUCE-4909.patch, MAPREDUCE-4909.patch TestKeyValueTextInputFormat.testFormat fails with Open JDK 7. The root cause appears to be a failure to delete in-use files via LocalFileSystem.delete (RawLocalFileSystem.delete). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4909) TestKeyValueTextInputFormat fails with Open JDK 7 on Windows
[ https://issues.apache.org/jira/browse/MAPREDUCE-4909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated MAPREDUCE-4909: - Target Version/s: (was: 1-win) TestKeyValueTextInputFormat fails with Open JDK 7 on Windows Key: MAPREDUCE-4909 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4909 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 1.2.0 Reporter: Arpit Agarwal Assignee: Arpit Agarwal Fix For: 1.2.0 Attachments: MAPREDUCE-4909.patch, MAPREDUCE-4909.patch, MAPREDUCE-4909.patch, MAPREDUCE-4909.patch TestKeyValueTextInputFormat.testFormat fails with Open JDK 7. The root cause appears to be a failure to delete in-use files via LocalFileSystem.delete (RawLocalFileSystem.delete). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4909) TestKeyValueTextInputFormat fails with Open JDK 7 on Windows
[ https://issues.apache.org/jira/browse/MAPREDUCE-4909?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13543328#comment-13543328 ] Arpit Agarwal commented on MAPREDUCE-4909: -- Thanks Suresh! TestKeyValueTextInputFormat fails with Open JDK 7 on Windows Key: MAPREDUCE-4909 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4909 Project: Hadoop Map/Reduce Issue Type: Bug Components: test Affects Versions: 1.2.0 Reporter: Arpit Agarwal Assignee: Arpit Agarwal Fix For: 1.2.0 Attachments: MAPREDUCE-4909.patch, MAPREDUCE-4909.patch, MAPREDUCE-4909.patch, MAPREDUCE-4909.patch TestKeyValueTextInputFormat.testFormat fails with Open JDK 7. The root cause appears to be a failure to delete in-use files via LocalFileSystem.delete (RawLocalFileSystem.delete). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Created] (MAPREDUCE-4909) TestKeyValueTextInputFormat fails with Open JDK 7 on Windows
Arpit Agarwal created MAPREDUCE-4909: Summary: TestKeyValueTextInputFormat fails with Open JDK 7 on Windows Key: MAPREDUCE-4909 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4909 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 1-win Reporter: Arpit Agarwal Assignee: Arpit Agarwal TestKeyValueTextInputFormat.testFormat fails with Open JDK 7. The root cause appears to be a failure to delete in-use files on Windows via LocalFileSystem.delete (RawLocalFileSystem.delete). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4909) TestKeyValueTextInputFormat fails with Open JDK 7 on Windows
[ https://issues.apache.org/jira/browse/MAPREDUCE-4909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated MAPREDUCE-4909: - Attachment: MAPREDUCE-4909.patch Submitting a patch to work around the test failures. Filed HADOOP-9176 to address the root cause. TestKeyValueTextInputFormat fails with Open JDK 7 on Windows Key: MAPREDUCE-4909 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4909 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 1-win Reporter: Arpit Agarwal Assignee: Arpit Agarwal Attachments: MAPREDUCE-4909.patch TestKeyValueTextInputFormat.testFormat fails with Open JDK 7. The root cause appears to be a failure to delete in-use files on Windows via LocalFileSystem.delete (RawLocalFileSystem.delete). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4909) TestKeyValueTextInputFormat fails with Open JDK 7 on Windows
[ https://issues.apache.org/jira/browse/MAPREDUCE-4909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated MAPREDUCE-4909: - Attachment: MAPREDUCE-4909.patch Added TODO. TestKeyValueTextInputFormat fails with Open JDK 7 on Windows Key: MAPREDUCE-4909 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4909 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 1-win Reporter: Arpit Agarwal Assignee: Arpit Agarwal Attachments: MAPREDUCE-4909.patch, MAPREDUCE-4909.patch TestKeyValueTextInputFormat.testFormat fails with Open JDK 7. The root cause appears to be a failure to delete in-use files on Windows via LocalFileSystem.delete (RawLocalFileSystem.delete). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4909) TestKeyValueTextInputFormat fails with Open JDK 7 on Windows
[ https://issues.apache.org/jira/browse/MAPREDUCE-4909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated MAPREDUCE-4909: - Description: TestKeyValueTextInputFormat.testFormat fails with Open JDK 7. The root cause appears to be a failure to delete in-use files via LocalFileSystem.delete (RawLocalFileSystem.delete). (was: TestKeyValueTextInputFormat.testFormat fails with Open JDK 7. The root cause appears to be a failure to delete in-use files on Windows via LocalFileSystem.delete (RawLocalFileSystem.delete).) TestKeyValueTextInputFormat fails with Open JDK 7 on Windows Key: MAPREDUCE-4909 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4909 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 1-win Reporter: Arpit Agarwal Assignee: Arpit Agarwal Attachments: MAPREDUCE-4909.patch, MAPREDUCE-4909.patch TestKeyValueTextInputFormat.testFormat fails with Open JDK 7. The root cause appears to be a failure to delete in-use files via LocalFileSystem.delete (RawLocalFileSystem.delete). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4909) TestKeyValueTextInputFormat fails with Open JDK 7 on Windows
[ https://issues.apache.org/jira/browse/MAPREDUCE-4909?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Arpit Agarwal updated MAPREDUCE-4909: - Attachment: MAPREDUCE-4909.patch [~sureshms] Removed the Windows-specific comment. HADOOP-9176 was filed to address the root cause. Thanks! Arpit TestKeyValueTextInputFormat fails with Open JDK 7 on Windows Key: MAPREDUCE-4909 URL: https://issues.apache.org/jira/browse/MAPREDUCE-4909 Project: Hadoop Map/Reduce Issue Type: Bug Affects Versions: 1-win Reporter: Arpit Agarwal Assignee: Arpit Agarwal Attachments: MAPREDUCE-4909.patch, MAPREDUCE-4909.patch, MAPREDUCE-4909.patch TestKeyValueTextInputFormat.testFormat fails with Open JDK 7. The root cause appears to be a failure to delete in-use files via LocalFileSystem.delete (RawLocalFileSystem.delete). -- This message is automatically generated by JIRA. If you think it was sent incorrectly, please contact your JIRA administrators For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4869) TestMapReduceChildJVM fails in branch-trunk-win
[
https://issues.apache.org/jira/browse/MAPREDUCE-4869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13530349#comment-13530349
]
Arpit Agarwal commented on MAPREDUCE-4869:
--
I am not sure I understand how this test works.
On Windows should we substitute %CD% for %PWD%?
TestMapReduceChildJVM fails in branch-trunk-win
---
Key: MAPREDUCE-4869
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4869
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: trunk-win
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: MAPREDUCE-4869-branch-trunk-win.1.patch
The YARN-233 patch for getting YARN working on Windows forgot to include a
corresponding change in {{TestMapReduceChildJVM}}, so the test is failing now.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4869) TestMapReduceChildJVM fails in branch-trunk-win
[
https://issues.apache.org/jira/browse/MAPREDUCE-4869?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=13530493#comment-13530493
]
Arpit Agarwal commented on MAPREDUCE-4869:
--
Thanks for the explanation. That makes sense why it passes. It's fine to the
%CD% change in a separate jira.
+1
TestMapReduceChildJVM fails in branch-trunk-win
---
Key: MAPREDUCE-4869
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4869
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: test
Affects Versions: trunk-win
Reporter: Chris Nauroth
Assignee: Chris Nauroth
Attachments: MAPREDUCE-4869-branch-trunk-win.1.patch
The YARN-233 patch for getting YARN working on Windows forgot to include a
corresponding change in {{TestMapReduceChildJVM}}, so the test is failing now.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
