[jira] [Commented] (MAPREDUCE-6455) Unable to use surefire 2.18
[
https://issues.apache.org/jira/browse/MAPREDUCE-6455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14716662#comment-14716662
]
Charlie Helin commented on MAPREDUCE-6455:
--
Yeah the fixes to the POM appears to do the trick.
Ahh, yes fixing!
Unable to use surefire 2.18
-
Key: MAPREDUCE-6455
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6455
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.7.1
Reporter: Charlie Helin
Assignee: Charlie Helin
Fix For: 3.0.0
Attachments: mr-6455.1.patch, mr-6455.2.patch, mr-6455.2.patch,
mr-6455.3.patch, mr-6455.4.patch
There are some compelling features in later version of surefire which lets
one exclude/include tests based the content of a file, re-running of test
case etc.
However introduced in Surefire 2.18 is also
https://issues.apache.org/jira/browse/SUREFIRE-649. Which changed the
convention of null properties to empty string values (). This only applies
to forked tests such as the MapReduce tests and cause a couple of them to
fail because of functionality that is directly or indirectly dependent on the
value being null. One such example is Configuration.substituteVars() and
TaskLog.getBaseLogDir().
substituteVars() shows the issue when the getProperty returns empty String,
skipping the getRaw(var) expression. One way to work around this could be
{code}
if (val == null || val.isEmpty()) {
String raw = getRaw(var);
if (raw != null) {
// raw contains a value, otherwise default to whatever
System.getProperty returned
// since it could be an empty string
val = raw;
}
}
{code}
getBaseLogDir, similarly when returns an empty string the schematics of
java.io.File differs dependent on whether parent is null or . A null value
is interpreted as new File(file); whereas will be interpreted as new
File(defaultParent /* / */, file);
This could simply be addressed with
{code}
static String getBaseLogDir() {
String logDir = System.getProperty(hadoop.log.dir);
// there is a difference how null and is treated as a parent
// directory when creating a file
return logDir == null || logDir.isEmpty() ? null : logDir;
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6455) Unable to use surefire 2.18
[
https://issues.apache.org/jira/browse/MAPREDUCE-6455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Charlie Helin updated MAPREDUCE-6455:
-
Status: Open (was: Patch Available)
Unable to use surefire 2.18
-
Key: MAPREDUCE-6455
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6455
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.7.1
Reporter: Charlie Helin
Assignee: Charlie Helin
Fix For: 3.0.0
Attachments: mr-6455.1.patch, mr-6455.2.patch, mr-6455.2.patch,
mr-6455.3.patch, mr-6455.4.patch
There are some compelling features in later version of surefire which lets
one exclude/include tests based the content of a file, re-running of test
case etc.
However introduced in Surefire 2.18 is also
https://issues.apache.org/jira/browse/SUREFIRE-649. Which changed the
convention of null properties to empty string values (). This only applies
to forked tests such as the MapReduce tests and cause a couple of them to
fail because of functionality that is directly or indirectly dependent on the
value being null. One such example is Configuration.substituteVars() and
TaskLog.getBaseLogDir().
substituteVars() shows the issue when the getProperty returns empty String,
skipping the getRaw(var) expression. One way to work around this could be
{code}
if (val == null || val.isEmpty()) {
String raw = getRaw(var);
if (raw != null) {
// raw contains a value, otherwise default to whatever
System.getProperty returned
// since it could be an empty string
val = raw;
}
}
{code}
getBaseLogDir, similarly when returns an empty string the schematics of
java.io.File differs dependent on whether parent is null or . A null value
is interpreted as new File(file); whereas will be interpreted as new
File(defaultParent /* / */, file);
This could simply be addressed with
{code}
static String getBaseLogDir() {
String logDir = System.getProperty(hadoop.log.dir);
// there is a difference how null and is treated as a parent
// directory when creating a file
return logDir == null || logDir.isEmpty() ? null : logDir;
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6455) Unable to use surefire 2.18
[
https://issues.apache.org/jira/browse/MAPREDUCE-6455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14716735#comment-14716735
]
Hadoop QA commented on MAPREDUCE-6455:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 15m 18s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 42s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 49s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 29s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | yarn tests | 0m 58s | Tests passed in
hadoop-yarn-registry. |
| | | 36m 15s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752746/mr-6455.4.patch |
| Optional Tests | javadoc javac unit |
| git revision | trunk / 0bf2854 |
| hadoop-yarn-registry test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5958/artifact/patchprocess/testrun_hadoop-yarn-registry.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5958/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5958/console |
This message was automatically generated.
Unable to use surefire 2.18
-
Key: MAPREDUCE-6455
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6455
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.7.1
Reporter: Charlie Helin
Assignee: Charlie Helin
Fix For: 3.0.0
Attachments: mr-6455.1.patch, mr-6455.2.patch, mr-6455.2.patch,
mr-6455.3.patch, mr-6455.4.patch
There are some compelling features in later version of surefire which lets
one exclude/include tests based the content of a file, re-running of test
case etc.
However introduced in Surefire 2.18 is also
https://issues.apache.org/jira/browse/SUREFIRE-649. Which changed the
convention of null properties to empty string values (). This only applies
to forked tests such as the MapReduce tests and cause a couple of them to
fail because of functionality that is directly or indirectly dependent on the
value being null. One such example is Configuration.substituteVars() and
TaskLog.getBaseLogDir().
substituteVars() shows the issue when the getProperty returns empty String,
skipping the getRaw(var) expression. One way to work around this could be
{code}
if (val == null || val.isEmpty()) {
String raw = getRaw(var);
if (raw != null) {
// raw contains a value, otherwise default to whatever
System.getProperty returned
// since it could be an empty string
val = raw;
}
}
{code}
getBaseLogDir, similarly when returns an empty string the schematics of
java.io.File differs dependent on whether parent is null or . A null value
is interpreted as new File(file); whereas will be interpreted as new
File(defaultParent /* / */, file);
This could simply be addressed with
{code}
static String getBaseLogDir() {
String logDir = System.getProperty(hadoop.log.dir);
// there is a difference how null and is treated as a parent
// directory when creating a file
return logDir == null || logDir.isEmpty() ? null : logDir;
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6455) Unable to use surefire 2.18
[
https://issues.apache.org/jira/browse/MAPREDUCE-6455?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Charlie Helin updated MAPREDUCE-6455:
-
Status: Patch Available (was: Open)
Unable to use surefire 2.18
-
Key: MAPREDUCE-6455
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6455
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.7.1
Reporter: Charlie Helin
Assignee: Charlie Helin
Fix For: 3.0.0
Attachments: mr-6455.1.patch, mr-6455.2.patch, mr-6455.2.patch,
mr-6455.3.patch, mr-6455.4.patch
There are some compelling features in later version of surefire which lets
one exclude/include tests based the content of a file, re-running of test
case etc.
However introduced in Surefire 2.18 is also
https://issues.apache.org/jira/browse/SUREFIRE-649. Which changed the
convention of null properties to empty string values (). This only applies
to forked tests such as the MapReduce tests and cause a couple of them to
fail because of functionality that is directly or indirectly dependent on the
value being null. One such example is Configuration.substituteVars() and
TaskLog.getBaseLogDir().
substituteVars() shows the issue when the getProperty returns empty String,
skipping the getRaw(var) expression. One way to work around this could be
{code}
if (val == null || val.isEmpty()) {
String raw = getRaw(var);
if (raw != null) {
// raw contains a value, otherwise default to whatever
System.getProperty returned
// since it could be an empty string
val = raw;
}
}
{code}
getBaseLogDir, similarly when returns an empty string the schematics of
java.io.File differs dependent on whether parent is null or . A null value
is interpreted as new File(file); whereas will be interpreted as new
File(defaultParent /* / */, file);
This could simply be addressed with
{code}
static String getBaseLogDir() {
String logDir = System.getProperty(hadoop.log.dir);
// there is a difference how null and is treated as a parent
// directory when creating a file
return logDir == null || logDir.isEmpty() ? null : logDir;
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6455) Unable to use surefire 2.18
[
https://issues.apache.org/jira/browse/MAPREDUCE-6455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14717082#comment-14717082
]
Andrew Wang commented on MAPREDUCE-6455:
LGTM! Thanks Charlie, I'll revert the original and commit this one down.
Unable to use surefire 2.18
-
Key: MAPREDUCE-6455
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6455
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.7.1
Reporter: Charlie Helin
Assignee: Charlie Helin
Fix For: 3.0.0
Attachments: mr-6455.1.patch, mr-6455.2.patch, mr-6455.2.patch,
mr-6455.3.patch, mr-6455.4.patch
There are some compelling features in later version of surefire which lets
one exclude/include tests based the content of a file, re-running of test
case etc.
However introduced in Surefire 2.18 is also
https://issues.apache.org/jira/browse/SUREFIRE-649. Which changed the
convention of null properties to empty string values (). This only applies
to forked tests such as the MapReduce tests and cause a couple of them to
fail because of functionality that is directly or indirectly dependent on the
value being null. One such example is Configuration.substituteVars() and
TaskLog.getBaseLogDir().
substituteVars() shows the issue when the getProperty returns empty String,
skipping the getRaw(var) expression. One way to work around this could be
{code}
if (val == null || val.isEmpty()) {
String raw = getRaw(var);
if (raw != null) {
// raw contains a value, otherwise default to whatever
System.getProperty returned
// since it could be an empty string
val = raw;
}
}
{code}
getBaseLogDir, similarly when returns an empty string the schematics of
java.io.File differs dependent on whether parent is null or . A null value
is interpreted as new File(file); whereas will be interpreted as new
File(defaultParent /* / */, file);
This could simply be addressed with
{code}
static String getBaseLogDir() {
String logDir = System.getProperty(hadoop.log.dir);
// there is a difference how null and is treated as a parent
// directory when creating a file
return logDir == null || logDir.isEmpty() ? null : logDir;
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6455) Unable to use surefire 2.18
[
https://issues.apache.org/jira/browse/MAPREDUCE-6455?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14716984#comment-14716984
]
Charlie Helin commented on MAPREDUCE-6455:
--
[~andrew.wang] How about now?
Unable to use surefire 2.18
-
Key: MAPREDUCE-6455
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6455
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.7.1
Reporter: Charlie Helin
Assignee: Charlie Helin
Fix For: 3.0.0
Attachments: mr-6455.1.patch, mr-6455.2.patch, mr-6455.2.patch,
mr-6455.3.patch, mr-6455.4.patch
There are some compelling features in later version of surefire which lets
one exclude/include tests based the content of a file, re-running of test
case etc.
However introduced in Surefire 2.18 is also
https://issues.apache.org/jira/browse/SUREFIRE-649. Which changed the
convention of null properties to empty string values (). This only applies
to forked tests such as the MapReduce tests and cause a couple of them to
fail because of functionality that is directly or indirectly dependent on the
value being null. One such example is Configuration.substituteVars() and
TaskLog.getBaseLogDir().
substituteVars() shows the issue when the getProperty returns empty String,
skipping the getRaw(var) expression. One way to work around this could be
{code}
if (val == null || val.isEmpty()) {
String raw = getRaw(var);
if (raw != null) {
// raw contains a value, otherwise default to whatever
System.getProperty returned
// since it could be an empty string
val = raw;
}
}
{code}
getBaseLogDir, similarly when returns an empty string the schematics of
java.io.File differs dependent on whether parent is null or . A null value
is interpreted as new File(file); whereas will be interpreted as new
File(defaultParent /* / */, file);
This could simply be addressed with
{code}
static String getBaseLogDir() {
String logDir = System.getProperty(hadoop.log.dir);
// there is a difference how null and is treated as a parent
// directory when creating a file
return logDir == null || logDir.isEmpty() ? null : logDir;
}
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6415) Create a tool to combine aggregated logs into HAR files
[ https://issues.apache.org/jira/browse/MAPREDUCE-6415?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14717391#comment-14717391 ] Robert Kanter commented on MAPREDUCE-6415: -- [~jlowe], can you take a look at this? Create a tool to combine aggregated logs into HAR files --- Key: MAPREDUCE-6415 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6415 Project: Hadoop Map/Reduce Issue Type: New Feature Affects Versions: 2.8.0 Reporter: Robert Kanter Assignee: Robert Kanter Attachments: HAR-ableAggregatedLogs_v1.pdf, MAPREDUCE-6415.001.patch, MAPREDUCE-6415_branch-2.001.patch, MAPREDUCE-6415_branch-2_prelim_001.patch, MAPREDUCE-6415_branch-2_prelim_002.patch, MAPREDUCE-6415_prelim_001.patch, MAPREDUCE-6415_prelim_002.patch While we wait for YARN-2942 to become viable, it would still be great to improve the aggregated logs problem. We can write a tool that combines aggregated log files into a single HAR file per application, which should solve the too many files and too many blocks problems. See the design document for details. See YARN-2942 for more context. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6434) Add support for PartialFileOutputCommiter when checkpointing is an option during preemption
[ https://issues.apache.org/jira/browse/MAPREDUCE-6434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14717617#comment-14717617 ] Augusto Souza commented on MAPREDUCE-6434: -- What is the next step in order for this patch to be accepted? Add support for PartialFileOutputCommiter when checkpointing is an option during preemption --- Key: MAPREDUCE-6434 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6434 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Augusto Souza Assignee: Augusto Souza Attachments: MAPREDUCE-6434.001.patch, MAPREDUCE-6434.002.patch, MAPREDUCE-6434.003.patch, MAPREDUCE-6434.004.patch, MAPREDUCE-6434.005.patch, MAPREDUCE-6434.006.patch Finish up some renaming work related to the annotation @Preemptable (it should be @Checkpointable now) and help in the splitting of patch in MAPREDUCE-5269 that is too large for being reviewed or accepted by Jenkins CI scripts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Assigned] (MAPREDUCE-6185) YARN M/R may return NaN for reducer progress after Job completion
[ https://issues.apache.org/jira/browse/MAPREDUCE-6185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang reassigned MAPREDUCE-6185: -- Assignee: Weiwei Yang YARN M/R may return NaN for reducer progress after Job completion - Key: MAPREDUCE-6185 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6185 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 2.4.1 Reporter: sam liu Assignee: Weiwei Yang Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6185) YARN M/R may return NaN for reducer progress after Job completion
[ https://issues.apache.org/jira/browse/MAPREDUCE-6185?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14718076#comment-14718076 ] Weiwei Yang commented on MAPREDUCE-6185: I tried on 2.7.1 version, I did not see this problem. Job : TeraGen Map Status : 0.0 Job : TeraGen Reduce Status : 0.0 Job : TeraGen Map Status : 0.0 Job : TeraGen Reduce Status : 0.0 Job : TeraGen Map Status : 0.0 Job : TeraGen Reduce Status : 0.0 Job : TeraGen Map Status : 0.0 Job : TeraGen Reduce Status : 0.0 Job : TeraGen Map Status : 0.0 Job : TeraGen Reduce Status : 0.0 Job : TeraGen Map Status : 0.0 Job : TeraGen Reduce Status : 0.0 Job : TeraGen Map Status : 0.10577558 Job : TeraGen Reduce Status : 0.0 Job : TeraGen Map Status : 0.1667912 Job : TeraGen Reduce Status : 0.0 Job : TeraGen Map Status : 0.21577513 Job : TeraGen Reduce Status : 0.0 Job : TeraGen Map Status : 0.28504142 Job : TeraGen Reduce Status : 0.0 Job : TeraGen Map Status : 0.3270764 Job : TeraGen Reduce Status : 0.0 Job : TeraGen Map Status : 0.39930692 Job : TeraGen Reduce Status : 0.0 Job : TeraGen Map Status : 0.45963144 Job : TeraGen Reduce Status : 0.0 Job : TeraGen Map Status : 0.5045679 Job : TeraGen Reduce Status : 0.0 Job : TeraGen Map Status : 0.57796395 Job : TeraGen Reduce Status : 0.0 Job : TeraGen Map Status : 0.65182614 Job : TeraGen Reduce Status : 0.0 Job : TeraGen Map Status : 0.695593 Job : TeraGen Reduce Status : 0.0 Job : TeraGen Map Status : 0.7389177 Job : TeraGen Reduce Status : 0.0 Job : TeraGen Map Status : 0.7799666 Job : TeraGen Reduce Status : 0.0 Job : TeraGen Map Status : 0.8276253 Job : TeraGen Reduce Status : 0.0 15/08/27 20:24:08 INFO mapred.ClientServiceDelegate: Application state is completed. FinalApplicationStatus=SUCCEEDED. Redirecting to job history server Job : TeraGen Map Status : 1.0 Job : TeraGen Reduce Status : 1.0 Job : TeraGen Map Status : 1.0 Job : TeraGen Reduce Status : 1.0 Job : TeraGen Map Status : 1.0 Job : TeraGen Reduce Status : 1.0 Job : TeraGen Map Status : 1.0 Job : TeraGen Reduce Status : 1.0 Job : TeraGen Map Status : 1.0 Job : TeraGen Reduce Status : 1.0 Job : TeraGen Map Status : 1.0 Job : TeraGen Reduce Status : 1.0 Job : TeraGen Map Status : 1.0 Job : TeraGen Reduce Status : 1.0 Job : TeraGen Map Status : 1.0 Job : TeraGen Reduce Status : 1.0 Job : TeraGen Map Status : 1.0 Job : TeraGen Reduce Status : 1.0 When job finished, the reduce progress is 1.0, which indicates the job is done. YARN M/R may return NaN for reducer progress after Job completion - Key: MAPREDUCE-6185 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6185 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 2.4.1 Reporter: sam liu Assignee: Weiwei Yang Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Resolved] (MAPREDUCE-6185) YARN M/R may return NaN for reducer progress after Job completion
[ https://issues.apache.org/jira/browse/MAPREDUCE-6185?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Weiwei Yang resolved MAPREDUCE-6185. Resolution: Cannot Reproduce YARN M/R may return NaN for reducer progress after Job completion - Key: MAPREDUCE-6185 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6185 Project: Hadoop Map/Reduce Issue Type: Bug Components: jobhistoryserver Affects Versions: 2.4.1 Reporter: sam liu Assignee: Weiwei Yang Priority: Minor -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6444) Add a checkpointable version of shuffle and reduce context supported by a checkpoint manager which uses the HDFS
[
https://issues.apache.org/jira/browse/MAPREDUCE-6444?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14718093#comment-14718093
]
Hadoop QA commented on MAPREDUCE-6444:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 19m 34s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 8 new or modified test files. |
| {color:red}-1{color} | javac | 7m 54s | The applied patch generated 1
additional warning messages. |
| {color:green}+1{color} | javadoc | 10m 6s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 2m 16s | The applied patch generated
64 new checkstyle issues (total was 945, now 991). |
| {color:red}-1{color} | whitespace | 0m 31s | The patch has 1 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 30s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 4m 26s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | mapreduce tests | 9m 15s | Tests passed in
hadoop-mapreduce-client-app. |
| {color:green}+1{color} | mapreduce tests | 0m 47s | Tests passed in
hadoop-mapreduce-client-common. |
| {color:green}+1{color} | mapreduce tests | 1m 45s | Tests passed in
hadoop-mapreduce-client-core. |
| {color:red}-1{color} | mapreduce tests | 165m 30s | Tests failed in
hadoop-mapreduce-client-jobclient. |
| | | 224m 53s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.mapreduce.TestMRJobClient |
| | hadoop.conf.TestNoDefaultsJobConf |
| | hadoop.mapred.TestLocalJobSubmission |
| | hadoop.mapred.TestReporter |
| | hadoop.mapred.TestJobName |
| | hadoop.mapreduce.security.ssl.TestEncryptedShuffle |
| | hadoop.mapred.TestReduceTask |
| | hadoop.mapreduce.TestValueIterReset |
| | hadoop.mapred.TestMerge |
| | hadoop.mapred.TestClusterMapReduceTestCase |
| | hadoop.mapreduce.TestMapCollection |
| | hadoop.mapreduce.security.TestBinaryTokenFile |
| | hadoop.mapreduce.v2.TestMiniMRProxyUser |
| | hadoop.mapred.TestMapRed |
| | hadoop.mapred.TestSpecialCharactersInOutputPath |
| | hadoop.fs.TestDFSIO |
| | hadoop.mapred.TestLazyOutput |
| | hadoop.mapreduce.v2.TestMRJobs |
| | hadoop.mapred.TestReduceFetch |
| | hadoop.mapreduce.security.TestMRCredentials |
| | hadoop.mapreduce.TestMapReduceLazyOutput |
| | hadoop.mapreduce.v2.TestUberAM |
| | hadoop.mapred.TestReduceFetchFromPartialMem |
| Timed out tests |
org.apache.hadoop.mapreduce.lib.jobcontrol.TestMapReduceJobControl |
| | org.apache.hadoop.mapred.TestMRIntermediateDataEncryption |
| | org.apache.hadoop.mapred.TestJobSysDirWithDFS |
| | org.apache.hadoop.mapred.TestMiniMRWithDFSWithDistinctUsers |
| | org.apache.hadoop.mapred.TestMiniMRClasspath |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752907/MAPREDUCE-6444.002.patch
|
| Optional Tests | javac unit findbugs checkstyle javadoc |
| git revision | trunk / 035ed26 |
| javac |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5959/artifact/patchprocess/diffJavacWarnings.txt
|
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5959/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5959/artifact/patchprocess/whitespace.txt
|
| hadoop-mapreduce-client-app test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5959/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
|
| hadoop-mapreduce-client-common test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5959/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5959/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| hadoop-mapreduce-client-jobclient test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5959/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5959/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf907.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
[jira] [Commented] (MAPREDUCE-6452) NPE when intermediate encrypt enabled for LocalRunner
[
https://issues.apache.org/jira/browse/MAPREDUCE-6452?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14717953#comment-14717953
]
zhihai xu commented on MAPREDUCE-6452:
--
thanks for the review [~asuresh], [~ajithshetty] and [~mohdshahidkhan], All
these test failures are not related to the patch. All these test passed at my
local build.
{code}
---
T E S T S
---
Running org.apache.hadoop.mapreduce.security.TestUmbilicalProtocolWithJobToken
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.091 sec - in
org.apache.hadoop.mapreduce.security.TestUmbilicalProtocolWithJobToken
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
---
T E S T S
---
Running org.apache.hadoop.mapreduce.security.TestJHSSecurity
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 27.402 sec - in
org.apache.hadoop.mapreduce.security.TestJHSSecurity\
Results :
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0
---
T E S T S
---
Running org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 75.584 sec - in
org.apache.hadoop.mapreduce.security.TestBinaryTokenFile
Results :
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
{code}
If no objection, will commit it tomorrow.
NPE when intermediate encrypt enabled for LocalRunner
-
Key: MAPREDUCE-6452
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6452
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Bibin A Chundatt
Assignee: zhihai xu
Attachments: MAPREDUCE-6452.002.patch, MAPREDUCE-6452.1.patch,
TestLocalJobSubmission.java
Enable the below properties try running mapreduce job
mapreduce.framework.name=local
mapreduce.job.encrypted-intermediate-data=true
{code}
2015-08-14 16:27:25,248 WARN [Thread-21] mapred.LocalJobRunner
(LocalJobRunner.java:run(561)) - job_local473843898_0001
java.lang.Exception: java.lang.NullPointerException
at
org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:463)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:523)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.crypto.CryptoOutputStream.init(CryptoOutputStream.java:92)
at
org.apache.hadoop.fs.crypto.CryptoFSDataOutputStream.init(CryptoFSDataOutputStream.java:31)
at
org.apache.hadoop.mapreduce.CryptoUtils.wrapIfNecessary(CryptoUtils.java:112)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1611)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1492)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:244)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
Jobs are failing always
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6452) NPE when intermediate encrypt enabled for LocalRunner
[
https://issues.apache.org/jira/browse/MAPREDUCE-6452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhihai xu updated MAPREDUCE-6452:
-
Hadoop Flags: Reviewed
NPE when intermediate encrypt enabled for LocalRunner
-
Key: MAPREDUCE-6452
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6452
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Bibin A Chundatt
Assignee: zhihai xu
Attachments: MAPREDUCE-6452.002.patch, MAPREDUCE-6452.1.patch,
TestLocalJobSubmission.java
Enable the below properties try running mapreduce job
mapreduce.framework.name=local
mapreduce.job.encrypted-intermediate-data=true
{code}
2015-08-14 16:27:25,248 WARN [Thread-21] mapred.LocalJobRunner
(LocalJobRunner.java:run(561)) - job_local473843898_0001
java.lang.Exception: java.lang.NullPointerException
at
org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:463)
at
org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:523)
Caused by: java.lang.NullPointerException
at
org.apache.hadoop.crypto.CryptoOutputStream.init(CryptoOutputStream.java:92)
at
org.apache.hadoop.fs.crypto.CryptoFSDataOutputStream.init(CryptoFSDataOutputStream.java:31)
at
org.apache.hadoop.mapreduce.CryptoUtils.wrapIfNecessary(CryptoUtils.java:112)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1611)
at
org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1492)
at
org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at
org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:244)
at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
{code}
Jobs are failing always
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6434) Add support for PartialFileOutputCommiter when checkpointing is an option during preemption
[ https://issues.apache.org/jira/browse/MAPREDUCE-6434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14716496#comment-14716496 ] Augusto Souza commented on MAPREDUCE-6434: -- [~chris.douglas] agreed and submitted a new version (006), but the Jenkins script didn't run to validate it, I don't know why. Also, performed tests with the map-only job I described above (with zero reducers). Is this patch acceptable now? Add support for PartialFileOutputCommiter when checkpointing is an option during preemption --- Key: MAPREDUCE-6434 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6434 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Augusto Souza Assignee: Augusto Souza Attachments: MAPREDUCE-6434.001.patch, MAPREDUCE-6434.002.patch, MAPREDUCE-6434.003.patch, MAPREDUCE-6434.004.patch, MAPREDUCE-6434.005.patch, MAPREDUCE-6434.006.patch Finish up some renaming work related to the annotation @Preemptable (it should be @Checkpointable now) and help in the splitting of patch in MAPREDUCE-5269 that is too large for being reviewed or accepted by Jenkins CI scripts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6434) Add support for PartialFileOutputCommiter when checkpointing is an option during preemption
[
https://issues.apache.org/jira/browse/MAPREDUCE-6434?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanelfocusedCommentId=14716552#comment-14716552
]
Hadoop QA commented on MAPREDUCE-6434:
--
\\
\\
| (/) *{color:green}+1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 16m 38s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:green}+1{color} | tests included | 0m 0s | The patch appears to
include 2 new or modified test files. |
| {color:green}+1{color} | javac | 7m 58s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 10m 4s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 47s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 27s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 35s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 26s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | mapreduce tests | 1m 45s | Tests passed in
hadoop-mapreduce-client-core. |
| | | 41m 8s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12751789/MAPREDUCE-6434.006.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / 0bf2854 |
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5957/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5957/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5957/console |
This message was automatically generated.
Add support for PartialFileOutputCommiter when checkpointing is an option
during preemption
---
Key: MAPREDUCE-6434
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6434
Project: Hadoop Map/Reduce
Issue Type: Improvement
Reporter: Augusto Souza
Assignee: Augusto Souza
Attachments: MAPREDUCE-6434.001.patch, MAPREDUCE-6434.002.patch,
MAPREDUCE-6434.003.patch, MAPREDUCE-6434.004.patch, MAPREDUCE-6434.005.patch,
MAPREDUCE-6434.006.patch
Finish up some renaming work related to the annotation @Preemptable (it
should be @Checkpointable now) and help in the splitting of patch in
MAPREDUCE-5269 that is too large for being reviewed or accepted by Jenkins CI
scripts.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6444) Add a checkpointable version of shuffle and reduce context supported by a checkpoint manager which uses the HDFS
[ https://issues.apache.org/jira/browse/MAPREDUCE-6444?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Augusto Souza updated MAPREDUCE-6444: - Attachment: MAPREDUCE-6444.002.patch Add a checkpointable version of shuffle and reduce context supported by a checkpoint manager which uses the HDFS Key: MAPREDUCE-6444 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6444 Project: Hadoop Map/Reduce Issue Type: Improvement Reporter: Augusto Souza Assignee: Augusto Souza Attachments: MAPREDUCE-6444.001.patch, MAPREDUCE-6444.002.patch Patch to add classes to provide a checkpointable version of shuffle and reduce context. Also, add a classe which makes possible to save a checkpoint of a task in HDFS. Those classes are respectively: CheckpointableShuffle, CheckpointableReduceContextImpl and TaskCheckpointManager. This improvement also helps in the splitting of patch in MAPREDUCE-5269 that is too large for being reviewed or accepted by Jenkins CI scripts. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
