[
https://issues.apache.org/jira/browse/MAPREDUCE-7139?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Oleksandr Shevchenko updated MAPREDUCE-7139:
--------------------------------------------
Description:
TestShuffleProvider#testShuffleProviders fails with the following error:
{noformat}
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertNotNull(Assert.java:621)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TestShuffleProvider.testShuffleProviders(TestShuffleProvider.java:114)
{noformat}
This error occurs only when we run TestShuffleProvider#testShuffleProviders
test after some test which creates TaskAttemptImpl (For example after tests in
TestMapReduceChildJVM). !screenshot-1.png!
TestShuffleProvider#testShuffleProviders creates container launch context, set
additionals NM aux services and expects to get these services from container
launch context after creating.
But we have the following code in TaskAttemptImpl:
{code}
static ContainerLaunchContext createContainerLaunchContext(
Map<ApplicationAccessType, String> applicationACLs,
Configuration conf, Token<JobTokenIdentifier> jobToken, Task remoteTask,
final org.apache.hadoop.mapred.JobID oldJobId,
WrappedJvmID jvmID,
TaskAttemptListener taskAttemptListener,
Credentials credentials) {
synchronized (commonContainerSpecLock) {
if (commonContainerSpec == null) {
commonContainerSpec = createCommonContainerLaunchContext(
applicationACLs, conf, jobToken, oldJobId, credentials);
}
}
{code}
The property "commonContainerSpec" is static. As the result, we do not recreate
common container launch within JVM. This is normal for AM JVM since all
containers should be run with one context. But this leads to an error when we
run unit tests in one JVM and expect to recreate container launch context.
We can add a setter for "commonContainerSpec" with "VisibleForTesting"
annotation and reset CLC in a test if we need it.
Also, we can cleanup CLC after running tests in @After fixture which create CLC
but we have too many tests which use TaskAttemptImpl and do not need cleanup.
was:
TestShuffleProvider#testShuffleProviders fails with the following error:
{noformat}
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.assertTrue(Assert.java:41)
at org.junit.Assert.assertNotNull(Assert.java:621)
at
org.apache.hadoop.mapreduce.v2.app.job.impl.TestShuffleProvider.testShuffleProviders(TestShuffleProvider.java:114)
{noformat}
This error occurs only when we run TestShuffleProvider#testShuffleProviders
test after some test which creates TaskAttemptImpl (For example after tests in
TestMapReduceChildJVM). TestShuffleProvider#testShuffleProviders creates
container launch context, set additionals NM aux services and expects to get
these services from container launch context after creating.
But we have the following code in TaskAttemptImpl:
{code}
static ContainerLaunchContext createContainerLaunchContext(
Map<ApplicationAccessType, String> applicationACLs,
Configuration conf, Token<JobTokenIdentifier> jobToken, Task remoteTask,
final org.apache.hadoop.mapred.JobID oldJobId,
WrappedJvmID jvmID,
TaskAttemptListener taskAttemptListener,
Credentials credentials) {
synchronized (commonContainerSpecLock) {
if (commonContainerSpec == null) {
commonContainerSpec = createCommonContainerLaunchContext(
applicationACLs, conf, jobToken, oldJobId, credentials);
}
}
{code}
The property "commonContainerSpec" is static. As the result, we do not recreate
common container launch within JVM. This is normal for AM JVM since all
containers should be run with one context. But this leads to an error when we
run unit tests in one JVM and expect to recreate container launch context.
We can add a setter for "commonContainerSpec" with "VisibleForTesting"
annotation and reset CLC in a test if we need it.
Also, we can cleanup CLC after running tests in @After fixture which create CLC
but we have too many tests which use TaskAttemptImpl and do not need cleanup.
> TestShuffleProvider#testShuffleProviders fails since a static common
> container launch context does't recreate
> -------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-7139
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7139
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: test
> Reporter: Oleksandr Shevchenko
> Assignee: Oleksandr Shevchenko
> Priority: Minor
> Attachments: MAPREDUCE-7139.001.patch, screenshot-1.png
>
>
> TestShuffleProvider#testShuffleProviders fails with the following error:
> {noformat}
> at org.junit.Assert.fail(Assert.java:88)
> at org.junit.Assert.assertTrue(Assert.java:41)
> at org.junit.Assert.assertNotNull(Assert.java:621)
> at
> org.apache.hadoop.mapreduce.v2.app.job.impl.TestShuffleProvider.testShuffleProviders(TestShuffleProvider.java:114)
> {noformat}
> This error occurs only when we run TestShuffleProvider#testShuffleProviders
> test after some test which creates TaskAttemptImpl (For example after tests
> in TestMapReduceChildJVM). !screenshot-1.png!
> TestShuffleProvider#testShuffleProviders creates container launch context,
> set additionals NM aux services and expects to get these services from
> container launch context after creating.
> But we have the following code in TaskAttemptImpl:
> {code}
> static ContainerLaunchContext createContainerLaunchContext(
> Map<ApplicationAccessType, String> applicationACLs,
> Configuration conf, Token<JobTokenIdentifier> jobToken, Task remoteTask,
> final org.apache.hadoop.mapred.JobID oldJobId,
> WrappedJvmID jvmID,
> TaskAttemptListener taskAttemptListener,
> Credentials credentials) {
> synchronized (commonContainerSpecLock) {
> if (commonContainerSpec == null) {
> commonContainerSpec = createCommonContainerLaunchContext(
> applicationACLs, conf, jobToken, oldJobId, credentials);
> }
> }
> {code}
> The property "commonContainerSpec" is static. As the result, we do not
> recreate common container launch within JVM. This is normal for AM JVM since
> all containers should be run with one context. But this leads to an error
> when we run unit tests in one JVM and expect to recreate container launch
> context.
> We can add a setter for "commonContainerSpec" with "VisibleForTesting"
> annotation and reset CLC in a test if we need it.
> Also, we can cleanup CLC after running tests in @After fixture which create
> CLC but we have too many tests which use TaskAttemptImpl and do not need
> cleanup.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
