[jira] [Commented] (MAPREDUCE-6569) ApplicationMaster Stuck 10 min And RM kill the AM
[ https://issues.apache.org/jira/browse/MAPREDUCE-6569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063641#comment-15063641 ] zhangyubiao commented on MAPREDUCE-6569: [~templedf] [~djp], Could you please take a look ? Thanks > ApplicationMaster Stuck 10 min And RM kill the AM > -- > > Key: MAPREDUCE-6569 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6569 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster >Affects Versions: 2.2.0 > Environment: CentOS6.5 Hadoop-2.2.0 DefaultContainer >Reporter: zhangyubiao > Attachments: 297137_1.log, 297586_0001.log, > yarn-nodemanager-dumpam.log > > > ApplicationMaster Stuck 10 min > and printLogs > 2015-12-09 02:45:04,160 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: > Memory usage of ProcessTree 30634 for container- > id container_1449229056278_297586_01_01: 1.1 GB of 3 GB physical memory > used; 3.5 GB of 9.3 GB virtual memory used > 2015-12-09 02:45:07,527 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: > Memory usage of ProcessTree 30634 for container- > id container_1449229056278_297586_01_01: 1.1 GB of 3 GB physical memory > used; 3.5 GB of 9.3 GB virtual memory used > 2015-12-09 02:45:10,888 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: > Memory usage of ProcessTree 30634 for container- > id container_1449229056278_297586_01_01: 1.1 GB of 3 GB physical memory > used; 3.5 GB of 9.3 GB virtual memory used > 2015-12-09 02:45:14,274 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: > Memory usage of ProcessTree 30634 for container- > id container_1449229056278_297586_01_01: 1.1 GB of 3 GB physical memory > used; 3.5 GB of 9.3 GB virtual memory used > 2015-12-09 02:45:17,625 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: > Memory usage of ProcessTree 30634 for container- > id container_1449229056278_297586_01_01: 1.1 GB of 3 GB physical memory > used; 3.5 GB of 9.3 GB virtual memory used > And RM kill the AM -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6570) logs in MRAppMasterShutdownHook can not be output, beacuse Hook of Tasklog will be called to shutdown LogManager previously
[ https://issues.apache.org/jira/browse/MAPREDUCE-6570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lachisis updated MAPREDUCE-6570: Status: Patch Available (was: Reopened) > logs in MRAppMasterShutdownHook can not be output, beacuse Hook of Tasklog > will be called to shutdown LogManager previously > > > Key: MAPREDUCE-6570 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6570 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster >Affects Versions: 2.7.0, 2.6.0, 2.5.0 >Reporter: lachisis >Assignee: lachisis > Attachments: MAPREDUCE-6570-1.patch, MAPREDUCE-6570-2.patch, > MAPREDUCE-6570-3.patch, MAPREDUCE-6570-4.patch > > > priority of MRAppMasterShutdownHook is 30, while priority of TaskLogHook is > 50. > when a applicationMaster received a signal to shutdown, actually TaskLogHook > will be called first. Then when MRAppMasterShutdownHook be called next, no > logs can be output. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6570) logs in MRAppMasterShutdownHook can not be output, beacuse Hook of Tasklog will be called to shutdown LogManager previously
[ https://issues.apache.org/jira/browse/MAPREDUCE-6570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lachisis updated MAPREDUCE-6570: Attachment: MAPREDUCE-6570-4.patch > logs in MRAppMasterShutdownHook can not be output, beacuse Hook of Tasklog > will be called to shutdown LogManager previously > > > Key: MAPREDUCE-6570 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6570 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster >Affects Versions: 2.5.0, 2.6.0, 2.7.0 >Reporter: lachisis >Assignee: lachisis > Attachments: MAPREDUCE-6570-1.patch, MAPREDUCE-6570-2.patch, > MAPREDUCE-6570-3.patch, MAPREDUCE-6570-4.patch > > > priority of MRAppMasterShutdownHook is 30, while priority of TaskLogHook is > 50. > when a applicationMaster received a signal to shutdown, actually TaskLogHook > will be called first. Then when MRAppMasterShutdownHook be called next, no > logs can be output. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Reopened] (MAPREDUCE-6570) logs in MRAppMasterShutdownHook can not be output, beacuse Hook of Tasklog will be called to shutdown LogManager previously
[ https://issues.apache.org/jira/browse/MAPREDUCE-6570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lachisis reopened MAPREDUCE-6570: - > logs in MRAppMasterShutdownHook can not be output, beacuse Hook of Tasklog > will be called to shutdown LogManager previously > > > Key: MAPREDUCE-6570 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6570 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster >Affects Versions: 2.5.0, 2.6.0, 2.7.0 >Reporter: lachisis >Assignee: lachisis > Attachments: MAPREDUCE-6570-1.patch, MAPREDUCE-6570-2.patch, > MAPREDUCE-6570-3.patch > > > priority of MRAppMasterShutdownHook is 30, while priority of TaskLogHook is > 50. > when a applicationMaster received a signal to shutdown, actually TaskLogHook > will be called first. Then when MRAppMasterShutdownHook be called next, no > logs can be output. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6569) ApplicationMaster Stuck 10 min And RM kill the AM
[ https://issues.apache.org/jira/browse/MAPREDUCE-6569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063633#comment-15063633 ] zhangyubiao commented on MAPREDUCE-6569: [~mohdshahidkhan], We get the thread dump of AM. And Find that RMContainerAllocator.handleEvent and heartbeat BLOCKED > ApplicationMaster Stuck 10 min And RM kill the AM > -- > > Key: MAPREDUCE-6569 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6569 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster >Affects Versions: 2.2.0 > Environment: CentOS6.5 Hadoop-2.2.0 DefaultContainer >Reporter: zhangyubiao > Attachments: 297137_1.log, 297586_0001.log, > yarn-nodemanager-dumpam.log > > > ApplicationMaster Stuck 10 min > and printLogs > 2015-12-09 02:45:04,160 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: > Memory usage of ProcessTree 30634 for container- > id container_1449229056278_297586_01_01: 1.1 GB of 3 GB physical memory > used; 3.5 GB of 9.3 GB virtual memory used > 2015-12-09 02:45:07,527 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: > Memory usage of ProcessTree 30634 for container- > id container_1449229056278_297586_01_01: 1.1 GB of 3 GB physical memory > used; 3.5 GB of 9.3 GB virtual memory used > 2015-12-09 02:45:10,888 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: > Memory usage of ProcessTree 30634 for container- > id container_1449229056278_297586_01_01: 1.1 GB of 3 GB physical memory > used; 3.5 GB of 9.3 GB virtual memory used > 2015-12-09 02:45:14,274 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: > Memory usage of ProcessTree 30634 for container- > id container_1449229056278_297586_01_01: 1.1 GB of 3 GB physical memory > used; 3.5 GB of 9.3 GB virtual memory used > 2015-12-09 02:45:17,625 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: > Memory usage of ProcessTree 30634 for container- > id container_1449229056278_297586_01_01: 1.1 GB of 3 GB physical memory > used; 3.5 GB of 9.3 GB virtual memory used > And RM kill the AM -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6542) HistoryViewer use SimpleDateFormat,But SimpleDateFormat is not threadsafe
[
https://issues.apache.org/jira/browse/MAPREDUCE-6542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063604#comment-15063604
]
Hadoop QA commented on MAPREDUCE-6542:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 9m
16s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 12m 4s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
49s {color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
10s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 58s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
32s {color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 28s
{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
in trunk has 2 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 40s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 45s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 19s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m 19s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 9m 52s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 9m 52s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
2s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 41s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 3m
29s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 23s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 36s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 8m 6s {color} |
{color:red} hadoop-common in the patch failed with JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 5s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with
JDK v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 8m 28s
{color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_91.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 24s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with
JDK v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 23s
{color} | {color:red} Patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 97m 3s {color} |
{color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests |
hadoop.security.ssl.TestReloadingX509TrustManager |
\\
\\
|| Subsystem || Report/Notes ||
| Docker | Image:yetus/hadoop:0ca8df7 |
| JIRA Patch URL |
htt
[jira] [Created] (MAPREDUCE-6580) Test failure : TestMRJobsWithProfiler
Rohith Sharma K S created MAPREDUCE-6580:
Summary: Test failure : TestMRJobsWithProfiler
Key: MAPREDUCE-6580
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6580
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Rohith Sharma K S
>From
>[https://builds.apache.org/job/PreCommit-YARN-Build/9976/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt]
> TestMRJobsWithProfiler fails intermittently
{code}
Running org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler
Tests run: 2, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 212.973 sec <<<
FAILURE! - in org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler
testDifferentProfilers(org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler)
Time elapsed: 133.116 sec <<< FAILURE!
java.lang.AssertionError: expected:<4> but was:<1>
at org.junit.Assert.fail(Assert.java:88)
at org.junit.Assert.failNotEquals(Assert.java:743)
at org.junit.Assert.assertEquals(Assert.java:118)
at org.junit.Assert.assertEquals(Assert.java:555)
at org.junit.Assert.assertEquals(Assert.java:542)
at
org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler.testProfilerInternal(TestMRJobsWithProfiler.java:212)
at
org.apache.hadoop.mapreduce.v2.TestMRJobsWithProfiler.testDifferentProfilers(TestMRJobsWithProfiler.java:117)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6570) logs in MRAppMasterShutdownHook can not be output, beacuse Hook of Tasklog will be called to shutdown LogManager previously
[ https://issues.apache.org/jira/browse/MAPREDUCE-6570?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] lachisis updated MAPREDUCE-6570: Resolution: Fixed Hadoop Flags: Reviewed Status: Resolved (was: Patch Available) > logs in MRAppMasterShutdownHook can not be output, beacuse Hook of Tasklog > will be called to shutdown LogManager previously > > > Key: MAPREDUCE-6570 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6570 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster >Affects Versions: 2.5.0, 2.6.0, 2.7.0 >Reporter: lachisis >Assignee: lachisis > Attachments: MAPREDUCE-6570-1.patch, MAPREDUCE-6570-2.patch, > MAPREDUCE-6570-3.patch > > > priority of MRAppMasterShutdownHook is 30, while priority of TaskLogHook is > 50. > when a applicationMaster received a signal to shutdown, actually TaskLogHook > will be called first. Then when MRAppMasterShutdownHook be called next, no > logs can be output. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6579) Test failure : TestNetworkedJob
Rohith Sharma K S created MAPREDUCE-6579:
Summary: Test failure : TestNetworkedJob
Key: MAPREDUCE-6579
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6579
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Rohith Sharma K S
>From
>[https://builds.apache.org/job/PreCommit-YARN-Build/9976/artifact/patchprocess/patch-unit-hadoop-mapreduce-project_hadoop-mapreduce-client_hadoop-mapreduce-client-jobclient-jdk1.8.0_66.txt]
> TestNetworkedJob are failed intermittently.
{code}
Running org.apache.hadoop.mapred.TestNetworkedJob
Tests run: 5, Failures: 1, Errors: 0, Skipped: 0, Time elapsed: 81.131 sec <<<
FAILURE! - in org.apache.hadoop.mapred.TestNetworkedJob
testNetworkedJob(org.apache.hadoop.mapred.TestNetworkedJob) Time elapsed:
30.55 sec <<< FAILURE!
org.junit.ComparisonFailure: expected:<[[Tue Dec 15 14:02:45 + 2015]
Application is Activated, waiting for resources to be assigned for AM. Details
: AM Partition = ; Partition Resource = ; Queue's Absolute capacity = 100.0 % ; Queue's Absolute used
capacity = 0.0 % ; Queue's Absolute max capacity = 100.0 % ; ]> but was:<[]>
at org.junit.Assert.assertEquals(Assert.java:115)
at org.junit.Assert.assertEquals(Assert.java:144)
at
org.apache.hadoop.mapred.TestNetworkedJob.testNetworkedJob(TestNetworkedJob.java:174)
{code}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6569) ApplicationMaster Stuck 10 min And RM kill the AM
[ https://issues.apache.org/jira/browse/MAPREDUCE-6569?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] zhangyubiao updated MAPREDUCE-6569: --- Attachment: yarn-nodemanager-dumpam.log 297137_1.log > ApplicationMaster Stuck 10 min And RM kill the AM > -- > > Key: MAPREDUCE-6569 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6569 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster >Affects Versions: 2.2.0 > Environment: CentOS6.5 Hadoop-2.2.0 DefaultContainer >Reporter: zhangyubiao > Attachments: 297137_1.log, 297586_0001.log, > yarn-nodemanager-dumpam.log > > > ApplicationMaster Stuck 10 min > and printLogs > 2015-12-09 02:45:04,160 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: > Memory usage of ProcessTree 30634 for container- > id container_1449229056278_297586_01_01: 1.1 GB of 3 GB physical memory > used; 3.5 GB of 9.3 GB virtual memory used > 2015-12-09 02:45:07,527 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: > Memory usage of ProcessTree 30634 for container- > id container_1449229056278_297586_01_01: 1.1 GB of 3 GB physical memory > used; 3.5 GB of 9.3 GB virtual memory used > 2015-12-09 02:45:10,888 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: > Memory usage of ProcessTree 30634 for container- > id container_1449229056278_297586_01_01: 1.1 GB of 3 GB physical memory > used; 3.5 GB of 9.3 GB virtual memory used > 2015-12-09 02:45:14,274 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: > Memory usage of ProcessTree 30634 for container- > id container_1449229056278_297586_01_01: 1.1 GB of 3 GB physical memory > used; 3.5 GB of 9.3 GB virtual memory used > 2015-12-09 02:45:17,625 INFO > org.apache.hadoop.yarn.server.nodemanager.containermanager.monitor.ContainersMonitorImpl: > Memory usage of ProcessTree 30634 for container- > id container_1449229056278_297586_01_01: 1.1 GB of 3 GB physical memory > used; 3.5 GB of 9.3 GB virtual memory used > And RM kill the AM -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6542) HistoryViewer use SimpleDateFormat,But SimpleDateFormat is not threadsafe
[
https://issues.apache.org/jira/browse/MAPREDUCE-6542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063477#comment-15063477
]
zhangyubiao commented on MAPREDUCE-6542:
MAPREDUCE-6542-v9.patch for review.
> HistoryViewer use SimpleDateFormat,But SimpleDateFormat is not threadsafe
> -
>
> Key: MAPREDUCE-6542
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6542
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Affects Versions: 2.2.0, 2.7.1
> Environment: CentOS6.5 Hadoop
>Reporter: zhangyubiao
>Assignee: zhangyubiao
> Attachments: MAPREDUCE-6542-v2.patch, MAPREDUCE-6542-v3.patch,
> MAPREDUCE-6542-v4.patch, MAPREDUCE-6542-v5.patch, MAPREDUCE-6542-v6.patch,
> MAPREDUCE-6542-v7.patch, MAPREDUCE-6542-v8.patch, MAPREDUCE-6542-v9.patch,
> MAPREDUCE-6542.patch
>
>
> I use SimpleDateFormat to Parse the JobHistory File before
> {code}
> private static final SimpleDateFormat dateFormat =
> new SimpleDateFormat("-MM-dd HH:mm:ss");
> public static String getJobDetail(JobInfo job) {
> StringBuffer jobDetails = new StringBuffer("");
> SummarizedJob ts = new SummarizedJob(job);
> jobDetails.append(job.getJobId().toString().trim()).append("\t");
> jobDetails.append(job.getUsername()).append("\t");
> jobDetails.append(job.getJobname().replaceAll("\\n",
> "")).append("\t");
> jobDetails.append(job.getJobQueueName()).append("\t");
> jobDetails.append(job.getPriority()).append("\t");
> jobDetails.append(job.getJobConfPath()).append("\t");
> jobDetails.append(job.getUberized()).append("\t");
>
> jobDetails.append(dateFormat.format(job.getSubmitTime())).append("\t");
>
> jobDetails.append(dateFormat.format(job.getLaunchTime())).append("\t");
>
> jobDetails.append(dateFormat.format(job.getFinishTime())).append("\t");
>return jobDetails.toString();
> }
> {code}
> But I find I query the SubmitTime and LaunchTime in hive and compare
> JobHistory File time , I find that the submitTime and launchTime was wrong.
> Finally,I change to use the FastDateFormat to parse the time format and the
> time become right
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6542) HistoryViewer use SimpleDateFormat,But SimpleDateFormat is not threadsafe
[
https://issues.apache.org/jira/browse/MAPREDUCE-6542?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhangyubiao updated MAPREDUCE-6542:
---
Attachment: MAPREDUCE-6542-v9.patch
[~templedf], It's my mistake. Thanks for remind
> HistoryViewer use SimpleDateFormat,But SimpleDateFormat is not threadsafe
> -
>
> Key: MAPREDUCE-6542
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6542
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Affects Versions: 2.2.0, 2.7.1
> Environment: CentOS6.5 Hadoop
>Reporter: zhangyubiao
>Assignee: zhangyubiao
> Attachments: MAPREDUCE-6542-v2.patch, MAPREDUCE-6542-v3.patch,
> MAPREDUCE-6542-v4.patch, MAPREDUCE-6542-v5.patch, MAPREDUCE-6542-v6.patch,
> MAPREDUCE-6542-v7.patch, MAPREDUCE-6542-v8.patch, MAPREDUCE-6542-v9.patch,
> MAPREDUCE-6542.patch
>
>
> I use SimpleDateFormat to Parse the JobHistory File before
> {code}
> private static final SimpleDateFormat dateFormat =
> new SimpleDateFormat("-MM-dd HH:mm:ss");
> public static String getJobDetail(JobInfo job) {
> StringBuffer jobDetails = new StringBuffer("");
> SummarizedJob ts = new SummarizedJob(job);
> jobDetails.append(job.getJobId().toString().trim()).append("\t");
> jobDetails.append(job.getUsername()).append("\t");
> jobDetails.append(job.getJobname().replaceAll("\\n",
> "")).append("\t");
> jobDetails.append(job.getJobQueueName()).append("\t");
> jobDetails.append(job.getPriority()).append("\t");
> jobDetails.append(job.getJobConfPath()).append("\t");
> jobDetails.append(job.getUberized()).append("\t");
>
> jobDetails.append(dateFormat.format(job.getSubmitTime())).append("\t");
>
> jobDetails.append(dateFormat.format(job.getLaunchTime())).append("\t");
>
> jobDetails.append(dateFormat.format(job.getFinishTime())).append("\t");
>return jobDetails.toString();
> }
> {code}
> But I find I query the SubmitTime and LaunchTime in hive and compare
> JobHistory File time , I find that the submitTime and launchTime was wrong.
> Finally,I change to use the FastDateFormat to parse the time format and the
> time become right
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
[
https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063442#comment-15063442
]
Sangjin Lee commented on MAPREDUCE-6577:
All the issues that are flagged by jenkins (findbugs, javac, checkstyle, and
unit tests) appear to be existing issues that are unrelated to the patch.
I'd appreciate your review on the patch. Thanks!
> MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
>
>
> Key: MAPREDUCE-6577
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am
>Affects Versions: 2.6.0
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Critical
> Attachments: MAPREDUCE-6577.01.patch
>
>
> If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not
> configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library:
> {noformat}
> 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader:
> Unable to load native-hadoop library for your platform... using builtin-java
> classes where applicable
> {noformat}
> As a result, any code that needs the hadoop native library in the MR AM will
> fail. For example, an uber-AM with lz4 compression for the mapper task will
> fail:
> {noformat}
> 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner]
> org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local
> (uberized) 'child' : java.lang.RuntimeException: native lz4 library not
> available
> at
> org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125)
> at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
> at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
> at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114)
> at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
[
https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063219#comment-15063219
]
Hadoop QA commented on MAPREDUCE-6577:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 8m
9s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 20s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 19s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
20s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 34s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
43s {color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 16s
{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
in trunk has 2 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 54s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 59s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 1m
21s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 6s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 6m 51s {color}
| {color:red} hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.8.0_66 with
JDK v1.8.0_66 generated 1 new issues (was 360, now 360). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 6s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 3m 18s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} javac {color} | {color:red} 10m 9s {color}
| {color:red} hadoop-mapreduce-project_hadoop-mapreduce-client-jdk1.7.0_91 with
JDK v1.7.0_91 generated 2 new issues (was 365, now 365). {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 3m 18s
{color} | {color:green} the patch passed {color} |
| {color:red}-1{color} | {color:red} checkstyle {color} | {color:red} 0m 21s
{color} | {color:red} Patch generated 5 new checkstyle issues in
hadoop-mapreduce-project/hadoop-mapreduce-client (total was 794, now 796).
{color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 29s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
41s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 2m
52s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 52s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 0m 58s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 10s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with
JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 9m 42s {color}
| {color:red} hadoop-mapreduce-client-app in the patch failed with JDK
v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 108m 6s {color}
| {color:red} hadoop-mapreduce-client-jobclient in the patch failed with JDK
v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 34s
[jira] [Commented] (MAPREDUCE-6542) HistoryViewer use SimpleDateFormat,But SimpleDateFormat is not threadsafe
[
https://issues.apache.org/jira/browse/MAPREDUCE-6542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15063082#comment-15063082
]
Hadoop QA commented on MAPREDUCE-6542:
--
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | {color:blue} reexec {color} | {color:blue} 0m 0s
{color} | {color:blue} Docker mode activated. {color} |
| {color:green}+1{color} | {color:green} @author {color} | {color:green} 0m 0s
{color} | {color:green} The patch does not contain any @author tags. {color} |
| {color:green}+1{color} | {color:green} test4tests {color} | {color:green} 0m
0s {color} | {color:green} The patch appears to include 1 new or modified test
files. {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 7m
28s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 7m 56s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 47s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 0m
58s {color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 34s
{color} | {color:green} trunk passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
28s {color} | {color:green} trunk passed {color} |
| {color:red}-1{color} | {color:red} findbugs {color} | {color:red} 1m 10s
{color} | {color:red}
hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core
in trunk has 2 extant Findbugs warnings. {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 14s
{color} | {color:green} trunk passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 28s
{color} | {color:green} trunk passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} mvninstall {color} | {color:green} 2m
6s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 8m 28s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 8m 28s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} compile {color} | {color:green} 10m
27s {color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:green}+1{color} | {color:green} javac {color} | {color:green} 10m 27s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} checkstyle {color} | {color:green} 1m
7s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvnsite {color} | {color:green} 1m 43s
{color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} mvneclipse {color} | {color:green} 0m
30s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} whitespace {color} | {color:green} 0m
0s {color} | {color:green} Patch has no whitespace issues. {color} |
| {color:green}+1{color} | {color:green} findbugs {color} | {color:green} 4m
24s {color} | {color:green} the patch passed {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 22s
{color} | {color:green} the patch passed with JDK v1.8.0_66 {color} |
| {color:green}+1{color} | {color:green} javadoc {color} | {color:green} 1m 35s
{color} | {color:green} the patch passed with JDK v1.7.0_91 {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 7m 18s {color}
| {color:red} hadoop-common in the patch failed with JDK v1.8.0_66. {color} |
| {color:red}-1{color} | {color:red} unit {color} | {color:red} 2m 13s {color}
| {color:red} hadoop-mapreduce-client-core in the patch failed with JDK
v1.8.0_66. {color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 7m 48s
{color} | {color:green} hadoop-common in the patch passed with JDK v1.7.0_91.
{color} |
| {color:green}+1{color} | {color:green} unit {color} | {color:green} 2m 34s
{color} | {color:green} hadoop-mapreduce-client-core in the patch passed with
JDK v1.7.0_91. {color} |
| {color:red}-1{color} | {color:red} asflicense {color} | {color:red} 0m 25s
{color} | {color:red} Patch generated 1 ASF License warnings. {color} |
| {color:black}{color} | {color:black} {color} | {color:black} 86m 16s {color}
| {color:black} {color} |
\\
\\
|| Reason || Tests ||
| JDK v1.8.0_66 Failed junit tests | hadoop.test.TestTimedOutTestsListener |
| | hadoop.security.ssl.TestReloadingX509TrustManager |
| | hadoop.mapreduce.tools.TestCLI |
\\
\\
|| Subsystem || R
[jira] [Commented] (MAPREDUCE-6542) HistoryViewer use SimpleDateFormat,But SimpleDateFormat is not threadsafe
[
https://issues.apache.org/jira/browse/MAPREDUCE-6542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15062914#comment-15062914
]
Daniel Templeton commented on MAPREDUCE-6542:
-
[~piaoyu zhang], one last change:
* Please replace the {{import java.util.concurrent.*}} with the individual
imports.
+1 pending that change.
> HistoryViewer use SimpleDateFormat,But SimpleDateFormat is not threadsafe
> -
>
> Key: MAPREDUCE-6542
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6542
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: jobhistoryserver
>Affects Versions: 2.2.0, 2.7.1
> Environment: CentOS6.5 Hadoop
>Reporter: zhangyubiao
>Assignee: zhangyubiao
> Attachments: MAPREDUCE-6542-v2.patch, MAPREDUCE-6542-v3.patch,
> MAPREDUCE-6542-v4.patch, MAPREDUCE-6542-v5.patch, MAPREDUCE-6542-v6.patch,
> MAPREDUCE-6542-v7.patch, MAPREDUCE-6542-v8.patch, MAPREDUCE-6542.patch
>
>
> I use SimpleDateFormat to Parse the JobHistory File before
> {code}
> private static final SimpleDateFormat dateFormat =
> new SimpleDateFormat("-MM-dd HH:mm:ss");
> public static String getJobDetail(JobInfo job) {
> StringBuffer jobDetails = new StringBuffer("");
> SummarizedJob ts = new SummarizedJob(job);
> jobDetails.append(job.getJobId().toString().trim()).append("\t");
> jobDetails.append(job.getUsername()).append("\t");
> jobDetails.append(job.getJobname().replaceAll("\\n",
> "")).append("\t");
> jobDetails.append(job.getJobQueueName()).append("\t");
> jobDetails.append(job.getPriority()).append("\t");
> jobDetails.append(job.getJobConfPath()).append("\t");
> jobDetails.append(job.getUberized()).append("\t");
>
> jobDetails.append(dateFormat.format(job.getSubmitTime())).append("\t");
>
> jobDetails.append(dateFormat.format(job.getLaunchTime())).append("\t");
>
> jobDetails.append(dateFormat.format(job.getFinishTime())).append("\t");
>return jobDetails.toString();
> }
> {code}
> But I find I query the SubmitTime and LaunchTime in hive and compare
> JobHistory File time , I find that the submitTime and launchTime was wrong.
> Finally,I change to use the FastDateFormat to parse the time format and the
> time become right
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
[
https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangjin Lee updated MAPREDUCE-6577:
---
Status: Patch Available (was: Open)
Posted patch v.1.
Added the default for the AM admin user env (which has the same value as the
non-AM admin user env).
Also did a small amount of yak-shaving. I suspect that the lib path handling on
Windows is not quite right (PWD was not being added for Windows). I tried to
make it a little more cross-platform there.
Folks who use Windows ([~cnauroth]?), I'd greatly appreciate it if you could
review the patch from that perspective.
> MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
>
>
> Key: MAPREDUCE-6577
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am
>Affects Versions: 2.6.0
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Critical
> Attachments: MAPREDUCE-6577.01.patch
>
>
> If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not
> configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library:
> {noformat}
> 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader:
> Unable to load native-hadoop library for your platform... using builtin-java
> classes where applicable
> {noformat}
> As a result, any code that needs the hadoop native library in the MR AM will
> fail. For example, an uber-AM with lz4 compression for the mapper task will
> fail:
> {noformat}
> 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner]
> org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local
> (uberized) 'child' : java.lang.RuntimeException: native lz4 library not
> available
> at
> org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125)
> at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
> at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
> at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114)
> at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
[
https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Sangjin Lee updated MAPREDUCE-6577:
---
Attachment: MAPREDUCE-6577.01.patch
> MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
>
>
> Key: MAPREDUCE-6577
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am
>Affects Versions: 2.6.0
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Critical
> Attachments: MAPREDUCE-6577.01.patch
>
>
> If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not
> configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library:
> {noformat}
> 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader:
> Unable to load native-hadoop library for your platform... using builtin-java
> classes where applicable
> {noformat}
> As a result, any code that needs the hadoop native library in the MR AM will
> fail. For example, an uber-AM with lz4 compression for the mapper task will
> fail:
> {noformat}
> 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner]
> org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local
> (uberized) 'child' : java.lang.RuntimeException: native lz4 library not
> available
> at
> org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125)
> at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
> at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
> at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114)
> at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6577) MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
[
https://issues.apache.org/jira/browse/MAPREDUCE-6577?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15062330#comment-15062330
]
Jason Lowe commented on MAPREDUCE-6577:
---
Thanks for the report and analysis, Sangjin! I agree there should be a default
to match the behavior with tasks -- sorry this was missed in MAPREDUCE-6021.
We didn't catch it on our clusters because we set the admin properties in order
to use a MapReduce framework via HDFS. We explicitly do not want to rely on
any Hadoop bits from the nodes to make rolling upgrades more robust. Otherwise
we could mix versions of Java and JNI bits and lead to weird behaviors.
> MR AM unable to load native library without MR_AM_ADMIN_USER_ENV set
>
>
> Key: MAPREDUCE-6577
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6577
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mr-am
>Affects Versions: 2.6.0
>Reporter: Sangjin Lee
>Assignee: Sangjin Lee
>Priority: Critical
>
> If yarn.app.mapreduce.am.admin.user.env (or yarn.app.mapreduce.am.env) is not
> configured to set LD_LIBRARY_PATH, MR AM will fail to load the native library:
> {noformat}
> 2015-12-15 21:29:22,473 WARN [main] org.apache.hadoop.util.NativeCodeLoader:
> Unable to load native-hadoop library for your platform... using builtin-java
> classes where applicable
> {noformat}
> As a result, any code that needs the hadoop native library in the MR AM will
> fail. For example, an uber-AM with lz4 compression for the mapper task will
> fail:
> {noformat}
> 2015-12-15 21:30:17,575 WARN [uber-SubtaskRunner]
> org.apache.hadoop.mapred.LocalContainerLauncher: Exception running local
> (uberized) 'child' : java.lang.RuntimeException: native lz4 library not
> available
> at
> org.apache.hadoop.io.compress.Lz4Codec.getCompressorType(Lz4Codec.java:125)
> at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:148)
> at
> org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.java:163)
> at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:114)
> at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:97)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1602)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:457)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:343)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runSubtask(LocalContainerLauncher.java:391)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.runTask(LocalContainerLauncher.java:309)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler.access$200(LocalContainerLauncher.java:195)
> at
> org.apache.hadoop.mapred.LocalContainerLauncher$EventHandler$1.run(LocalContainerLauncher.java:238)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:471)
> at java.util.concurrent.FutureTask.run(FutureTask.java:262)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6554) MRAppMaster servicestart failing with NPE in MRAppMaster#parsePreviousJobHistory
[
https://issues.apache.org/jira/browse/MAPREDUCE-6554?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15062280#comment-15062280
]
Bibin A Chundatt commented on MAPREDUCE-6554:
-
[~jlowe]
Sorry for late response.
Instead of {{this.schema = parser.parse(in.readLine());}} before passing to
parser.parse() we can check for null and the return IOException.
For IOException {{org.apache.hadoop.tools.rumen.CurrentJHParser#canParse}}
returns false .
Please do correct me if i am wrong.
> MRAppMaster servicestart failing with NPE in
> MRAppMaster#parsePreviousJobHistory
> -
>
> Key: MAPREDUCE-6554
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6554
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: Bibin A Chundatt
>Priority: Critical
> Attachments: 0001-MAPREDUCE-6554.patch
>
>
> Create scenario so that MR app master gets preempted.
> On next MRAppMaster launch tried to recover previous job history file
> {{MRAppMaster#parsePreviousJobHistory}}
> {noformat}
> 2015-11-21 13:52:27,722 INFO [main]
> org.apache.hadoop.service.AbstractService: Service
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state STARTED;
> cause: java.lang.NullPointerException
> java.lang.NullPointerException
> at java.io.StringReader.(StringReader.java:50)
> at org.apache.avro.Schema$Parser.parse(Schema.java:917)
> at org.apache.avro.Schema.parse(Schema.java:966)
> at
> org.apache.hadoop.mapreduce.jobhistory.EventReader.(EventReader.java:75)
> at
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryParser.parse(JobHistoryParser.java:139)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.parsePreviousJobHistory(MRAppMaster.java:1256)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.processRecovery(MRAppMaster.java:1225)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceStart(MRAppMaster.java:1087)
> at
> org.apache.hadoop.service.AbstractService.start(AbstractService.java:193)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1570)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1673)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1566)
> at
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1499)
> 2015-11-21 13:52:27,725 INFO [main]
> org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler: Stopping
> JobHistoryEventHandler. Size of the outstanding queue size is 0
> {noformat}
> EventReader(EventReader stream)
> {noformat}
> this.version = in.readLine();
> ...
> Schema myschema = new
> SpecificData(Event.class.getClassLoader()).getSchema(Event.class);
> this.schema = Schema.parse(in.readLine());
> {noformat}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6578) Add support for HDFS heterogeneous storage testing to TestDFSIO
[ https://issues.apache.org/jira/browse/MAPREDUCE-6578?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Wei Zhou updated MAPREDUCE-6578: Attachment: MAPREDUCE-6578.00.patch Add a new option "-storagePolicy" to specify the storage policy. If not specified default policy is used as previous. Thanks! > Add support for HDFS heterogeneous storage testing to TestDFSIO > --- > > Key: MAPREDUCE-6578 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6578 > Project: Hadoop Map/Reduce > Issue Type: New Feature >Reporter: Wei Zhou >Assignee: Wei Zhou > Attachments: MAPREDUCE-6578.00.patch > > > HDFS heterogeneous storage allows user to store data blocks to different > storage medias according to predefined storage policies. Only 'Default' > policy is supported in current TestDFSIO implementation. This is going to add > an new option to enable tests of other storage polices. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6578) Add support for HDFS heterogeneous storage testing to TestDFSIO
Wei Zhou created MAPREDUCE-6578: --- Summary: Add support for HDFS heterogeneous storage testing to TestDFSIO Key: MAPREDUCE-6578 URL: https://issues.apache.org/jira/browse/MAPREDUCE-6578 Project: Hadoop Map/Reduce Issue Type: New Feature Reporter: Wei Zhou Assignee: Wei Zhou HDFS heterogeneous storage allows user to store data blocks to different storage medias according to predefined storage policies. Only 'Default' policy is supported in current TestDFSIO implementation. This is going to add an new option to enable tests of other storage polices. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-4788) Job are marking as FAILED even if there are no failed tasks in it
[ https://issues.apache.org/jira/browse/MAPREDUCE-4788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15061879#comment-15061879 ] Devaraj K commented on MAPREDUCE-4788: -- I faced this issue long time back during the version 2.0.4-alpha. I don't think it is still an issue with the changes happened over the time. bq. Is the job history not correct on this state as well? Yes, the job history was also showing the job state as FAILED and the task state as KILLED but we could see the attempt for the task as succeeded. I don't have logs for this issue since it happened some time ago. [~brahmareddy], Can you give details or share the log for the issue you are facing if you have anything? > Job are marking as FAILED even if there are no failed tasks in it > - > > Key: MAPREDUCE-4788 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-4788 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: applicationmaster >Affects Versions: 2.6.0 >Reporter: Devaraj K > Attachments: MAPREDUCE-4788.patch > > > Sometimes Jobs are marking as FAILED and some the tasks are marking as KILLED > in it. > In MRAppMaster, JobFinishEvent is triggering and waiting for the 5000 millis. > If any tasks final state is unknown by this time those tasks are marking as > KILLED and Job state is marking as FAILED. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
