[jira] [Updated] (MAPREDUCE-7079) JobHistory#ServiceStop implementation is incorrect
[
https://issues.apache.org/jira/browse/MAPREDUCE-7079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Eric Payne updated MAPREDUCE-7079:
--
Fix Version/s: 2.10.1
3.2.2
3.1.4
3.3.0
Resolution: Fixed
Status: Resolved (was: Patch Available)
Committed to trunk, branch-3.2, branch-3.1, and branch-2.10.
Thanks [~ahussein].
> JobHistory#ServiceStop implementation is incorrect
> --
>
> Key: MAPREDUCE-7079
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7079
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Jason Darrell Lowe
>Assignee: Ahmed Hussein
>Priority: Major
> Fix For: 3.3.0, 3.1.4, 3.2.2, 2.10.1
>
> Attachments: 2020-01-10-MRApp-stack-dump.txt,
> 2020-01-10-org.apache.hadoop.mapred.TestMRIntermediateDataEncryption-version-14.txt,
> MAPREDUCE-7079.001.patch, MAPREDUCE-7079.002.patch,
> MAPREDUCE-7079.003.patch, MAPREDUCE-7079.004.patch, MAPREDUCE-7079.005.patch,
> MAPREDUCE-7079.006.patch, MAPREDUCE-7079.007.patch, MAPREDUCE-7079.008.patch,
> MAPREDUCE-7079.009.patch, MAPREDUCE-7079.010.patch
>
>
> {{JobHistory.serviceStop}} skips waiting for the thread pool to terminate.
> The problem is due to incorrect while condition that will evaluate to false
> on the iteration of the loop.
> {code:java}
> scheduledExecutor.shutdown();
> boolean interrupted = false;
> long currentTime = System.currentTimeMillis();
> while (!scheduledExecutor.isShutdown()
> && System.currentTimeMillis() > currentTime + 1000l &&
> !interrupted) {
> try {
> Thread.sleep(20);
> } catch (InterruptedException e) {
> interrupted = true;
> }
> }
> {code}
> The expression "{{System.currentTimeMillis() > currentTime + 1000L}}" is
> false because currentTime was just initialized with
> {{System.currentTimeMillis()}}. As a result the the thread won't wait until
> the executor is terminated. Instead, it will force a shutdown immediately.
> *TestMRIntermediateDataEncryption is failing in precommit builds*
> TestMRIntermediateDataEncryption is either timing out or tearing down the JVM
> which causes the unit tests in jobclient to not pass cleanly during precommit
> builds. From sample precommit console output, note the lack of a test results
> line when the test is run:
> {noformat}
> [INFO] Running org.apache.hadoop.mapred.TestSequenceFileInputFormat
> [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.976
> s - in org.apache.hadoop.mapred.TestSequenceFileInputFormat
> [INFO] Running org.apache.hadoop.mapred.TestMRIntermediateDataEncryption
> [INFO] Running org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
> [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.659
> s - in org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
> [...]
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 02:14 h
> [INFO] Finished at: 2018-04-12T04:27:06+00:00
> [INFO] Final Memory: 24M/594M
> [INFO]
>
> [WARNING] The requested profile "parallel-tests" could not be activated
> because it does not exist.
> [WARNING] The requested profile "native" could not be activated because it
> does not exist.
> [WARNING] The requested profile "yarn-ui" could not be activated because it
> does not exist.
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.21.0:test (default-test) on
> project hadoop-mapreduce-client-jobclient: There was a timeout or other error
> in the fork -> [Help 1]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7079) JobHistory#ServiceStop implementation is incorrect
[
https://issues.apache.org/jira/browse/MAPREDUCE-7079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7079:
-
Attachment: MAPREDUCE-7079.010.patch
> JobHistory#ServiceStop implementation is incorrect
> --
>
> Key: MAPREDUCE-7079
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7079
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Jason Darrell Lowe
>Assignee: Ahmed Hussein
>Priority: Major
> Attachments: 2020-01-10-MRApp-stack-dump.txt,
> 2020-01-10-org.apache.hadoop.mapred.TestMRIntermediateDataEncryption-version-14.txt,
> MAPREDUCE-7079.001.patch, MAPREDUCE-7079.002.patch,
> MAPREDUCE-7079.003.patch, MAPREDUCE-7079.004.patch, MAPREDUCE-7079.005.patch,
> MAPREDUCE-7079.006.patch, MAPREDUCE-7079.007.patch, MAPREDUCE-7079.008.patch,
> MAPREDUCE-7079.009.patch, MAPREDUCE-7079.010.patch
>
>
> {{JobHistory.serviceStop}} skips waiting for the thread pool to terminate.
> The problem is due to incorrect while condition that will evaluate to false
> on the iteration of the loop.
> {code:java}
> scheduledExecutor.shutdown();
> boolean interrupted = false;
> long currentTime = System.currentTimeMillis();
> while (!scheduledExecutor.isShutdown()
> && System.currentTimeMillis() > currentTime + 1000l &&
> !interrupted) {
> try {
> Thread.sleep(20);
> } catch (InterruptedException e) {
> interrupted = true;
> }
> }
> {code}
> The expression "{{System.currentTimeMillis() > currentTime + 1000L}}" is
> false because currentTime was just initialized with
> {{System.currentTimeMillis()}}. As a result the the thread won't wait until
> the executor is terminated. Instead, it will force a shutdown immediately.
> *TestMRIntermediateDataEncryption is failing in precommit builds*
> TestMRIntermediateDataEncryption is either timing out or tearing down the JVM
> which causes the unit tests in jobclient to not pass cleanly during precommit
> builds. From sample precommit console output, note the lack of a test results
> line when the test is run:
> {noformat}
> [INFO] Running org.apache.hadoop.mapred.TestSequenceFileInputFormat
> [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.976
> s - in org.apache.hadoop.mapred.TestSequenceFileInputFormat
> [INFO] Running org.apache.hadoop.mapred.TestMRIntermediateDataEncryption
> [INFO] Running org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
> [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.659
> s - in org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
> [...]
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 02:14 h
> [INFO] Finished at: 2018-04-12T04:27:06+00:00
> [INFO] Final Memory: 24M/594M
> [INFO]
>
> [WARNING] The requested profile "parallel-tests" could not be activated
> because it does not exist.
> [WARNING] The requested profile "native" could not be activated because it
> does not exist.
> [WARNING] The requested profile "yarn-ui" could not be activated because it
> does not exist.
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.21.0:test (default-test) on
> project hadoop-mapreduce-client-jobclient: There was a timeout or other error
> in the fork -> [Help 1]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7079) JobHistory#ServiceStop implementation is incorrect
[
https://issues.apache.org/jira/browse/MAPREDUCE-7079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7079:
-
Attachment: MAPREDUCE-7079.009.patch
> JobHistory#ServiceStop implementation is incorrect
> --
>
> Key: MAPREDUCE-7079
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7079
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Jason Darrell Lowe
>Assignee: Ahmed Hussein
>Priority: Major
> Attachments: 2020-01-10-MRApp-stack-dump.txt,
> 2020-01-10-org.apache.hadoop.mapred.TestMRIntermediateDataEncryption-version-14.txt,
> MAPREDUCE-7079.001.patch, MAPREDUCE-7079.002.patch,
> MAPREDUCE-7079.003.patch, MAPREDUCE-7079.004.patch, MAPREDUCE-7079.005.patch,
> MAPREDUCE-7079.006.patch, MAPREDUCE-7079.007.patch, MAPREDUCE-7079.008.patch,
> MAPREDUCE-7079.009.patch
>
>
> {{JobHistory.serviceStop}} skips waiting for the thread pool to terminate.
> The problem is due to incorrect while condition that will evaluate to false
> on the iteration of the loop.
> {code:java}
> scheduledExecutor.shutdown();
> boolean interrupted = false;
> long currentTime = System.currentTimeMillis();
> while (!scheduledExecutor.isShutdown()
> && System.currentTimeMillis() > currentTime + 1000l &&
> !interrupted) {
> try {
> Thread.sleep(20);
> } catch (InterruptedException e) {
> interrupted = true;
> }
> }
> {code}
> The expression "{{System.currentTimeMillis() > currentTime + 1000L}}" is
> false because currentTime was just initialized with
> {{System.currentTimeMillis()}}. As a result the the thread won't wait until
> the executor is terminated. Instead, it will force a shutdown immediately.
> *TestMRIntermediateDataEncryption is failing in precommit builds*
> TestMRIntermediateDataEncryption is either timing out or tearing down the JVM
> which causes the unit tests in jobclient to not pass cleanly during precommit
> builds. From sample precommit console output, note the lack of a test results
> line when the test is run:
> {noformat}
> [INFO] Running org.apache.hadoop.mapred.TestSequenceFileInputFormat
> [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.976
> s - in org.apache.hadoop.mapred.TestSequenceFileInputFormat
> [INFO] Running org.apache.hadoop.mapred.TestMRIntermediateDataEncryption
> [INFO] Running org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
> [INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.659
> s - in org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
> [...]
> [INFO]
>
> [INFO] BUILD FAILURE
> [INFO]
>
> [INFO] Total time: 02:14 h
> [INFO] Finished at: 2018-04-12T04:27:06+00:00
> [INFO] Final Memory: 24M/594M
> [INFO]
>
> [WARNING] The requested profile "parallel-tests" could not be activated
> because it does not exist.
> [WARNING] The requested profile "native" could not be activated because it
> does not exist.
> [WARNING] The requested profile "yarn-ui" could not be activated because it
> does not exist.
> [ERROR] Failed to execute goal
> org.apache.maven.plugins:maven-surefire-plugin:2.21.0:test (default-test) on
> project hadoop-mapreduce-client-jobclient: There was a timeout or other error
> in the fork -> [Help 1]
> {noformat}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
-
To unsubscribe, e-mail: mapreduce-issues-unsubscr...@hadoop.apache.org
For additional commands, e-mail: mapreduce-issues-h...@hadoop.apache.org
[jira] [Updated] (MAPREDUCE-7079) JobHistory#ServiceStop implementation is incorrect
[
https://issues.apache.org/jira/browse/MAPREDUCE-7079?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ahmed Hussein updated MAPREDUCE-7079:
-
Description:
{{JobHistory.serviceStop}} skips waiting for the thread pool to terminate. The
problem is due to incorrect while condition that will evaluate to false on the
iteration of the loop.
{code:java}
scheduledExecutor.shutdown();
boolean interrupted = false;
long currentTime = System.currentTimeMillis();
while (!scheduledExecutor.isShutdown()
&& System.currentTimeMillis() > currentTime + 1000l && !interrupted) {
try {
Thread.sleep(20);
} catch (InterruptedException e) {
interrupted = true;
}
}
{code}
The expression "{{System.currentTimeMillis() > currentTime + 1000L}}" is false
because currentTime was just initialized with {{System.currentTimeMillis()}}.
As a result the the thread won't wait until the executor is terminated.
Instead, it will force a shutdown immediately.
*TestMRIntermediateDataEncryption is failing in precommit builds*
TestMRIntermediateDataEncryption is either timing out or tearing down the JVM
which causes the unit tests in jobclient to not pass cleanly during precommit
builds. From sample precommit console output, note the lack of a test results
line when the test is run:
{noformat}
[INFO] Running org.apache.hadoop.mapred.TestSequenceFileInputFormat
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.976 s
- in org.apache.hadoop.mapred.TestSequenceFileInputFormat
[INFO] Running org.apache.hadoop.mapred.TestMRIntermediateDataEncryption
[INFO] Running org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.659 s
- in org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
[...]
[INFO]
[INFO] BUILD FAILURE
[INFO]
[INFO] Total time: 02:14 h
[INFO] Finished at: 2018-04-12T04:27:06+00:00
[INFO] Final Memory: 24M/594M
[INFO]
[WARNING] The requested profile "parallel-tests" could not be activated because
it does not exist.
[WARNING] The requested profile "native" could not be activated because it does
not exist.
[WARNING] The requested profile "yarn-ui" could not be activated because it
does not exist.
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:2.21.0:test (default-test) on
project hadoop-mapreduce-client-jobclient: There was a timeout or other error
in the fork -> [Help 1]
{noformat}
was:
TestMRIntermediateDataEncryption is either timing out or tearing down the JVM
which causes the unit tests in jobclient to not pass cleanly during precommit
builds. From sample precommit console output, note the lack of a test results
line when the test is run:
{noformat}
[INFO] Running org.apache.hadoop.mapred.TestSequenceFileInputFormat
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.976 s
- in org.apache.hadoop.mapred.TestSequenceFileInputFormat
[INFO] Running org.apache.hadoop.mapred.TestMRIntermediateDataEncryption
[INFO] Running org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
[INFO] Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 16.659 s
- in org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
[...]
[INFO]
[INFO] BUILD FAILURE
[INFO]
[INFO] Total time: 02:14 h
[INFO] Finished at: 2018-04-12T04:27:06+00:00
[INFO] Final Memory: 24M/594M
[INFO]
[WARNING] The requested profile "parallel-tests" could not be activated because
it does not exist.
[WARNING] The requested profile "native" could not be activated because it does
not exist.
[WARNING] The requested profile "yarn-ui" could not be activated because it
does not exist.
[ERROR] Failed to execute goal
org.apache.maven.plugins:maven-surefire-plugin:2.21.0:test (default-test) on
project hadoop-mapreduce-client-jobclient: There was a timeout or other error
in the fork -> [Help 1]
{noformat}
Summary: JobHistory#ServiceStop implementation is incorrect (was:
TestMRIntermediateDataEncryption is failing in precommit builds)
> JobHistory#ServiceStop implementation is incorrect
> --
>
> Key: MAPREDUCE-7079
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-7079
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Jason Darrell Lowe
>Assignee: Ahmed Hussein
>
