[jira] [Updated] (MAPREDUCE-6464) mapred queue is throwing the NPE in case of the local framework
[
https://issues.apache.org/jira/browse/MAPREDUCE-6464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mohammad Shahid Khan updated MAPREDUCE-6464:
Attachment: MAPREDUCE-6464-1.patch
> mapred queue is throwing the NPE in case of the local framework
> --
>
> Key: MAPREDUCE-6464
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6464
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Affects Versions: 3.0.0, 2.7.1
> Environment: Linux
>Reporter: Mohammad Shahid Khan
>Assignee: Mohammad Shahid Khan
>Priority: Minor
> Attachments: MAPREDUCE-6464-1.patch
>
>
> mapred queue is throwing the NPE in case of the local framework.
> local framework does not support the same, so instead of throwing the NPE
> better we can give a meaning message.
> stack trace
> {CODE}
> /mapred queue -showacls
> 15/08/26 09:41:53 INFO Configuration.deprecation: session.id is deprecated.
> Instead, use dfs.metrics.session-id
> 15/08/26 09:41:53 INFO jvm.JvmMetrics: Initializing JVM Metrics with
> processName=JobTracker, sessionId=
> Exception in thread "main" java.lang.NullPointerException
> at
> org.apache.hadoop.mapred.JobClient.getQueueAclsForCurrentUser(JobClient.java:1210)
> at
> org.apache.hadoop.mapred.JobQueueClient.displayQueueAclsInfoForCurrentUser(JobQueueClient.java:201)
> at
> org.apache.hadoop.mapred.JobQueueClient.run(JobQueueClient.java:110)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at
> org.apache.hadoop.mapred.JobQueueClient.main(JobQueueClient.java:240)
> {CODE}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6464) mapred queue is throwing the NPE in case of the local framework
Mohammad Shahid Khan created MAPREDUCE-6464:
---
Summary: mapred queue is throwing the NPE in case of the
local framework
Key: MAPREDUCE-6464
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6464
Project: Hadoop Map/Reduce
Issue Type: Bug
Affects Versions: 2.7.1, 3.0.0
Environment: Linux
Reporter: Mohammad Shahid Khan
Assignee: Mohammad Shahid Khan
Priority: Minor
mapred queue is throwing the NPE in case of the local framework.
local framework does not support the same, so instead of throwing the NPE
better we can give a meaning message.
stack trace
{CODE}
/mapred queue -showacls
15/08/26 09:41:53 INFO Configuration.deprecation: session.id is deprecated.
Instead, use dfs.metrics.session-id
15/08/26 09:41:53 INFO jvm.JvmMetrics: Initializing JVM Metrics with
processName=JobTracker, sessionId=
Exception in thread "main" java.lang.NullPointerException
at
org.apache.hadoop.mapred.JobClient.getQueueAclsForCurrentUser(JobClient.java:1210)
at
org.apache.hadoop.mapred.JobQueueClient.displayQueueAclsInfoForCurrentUser(JobQueueClient.java:201)
at org.apache.hadoop.mapred.JobQueueClient.run(JobQueueClient.java:110)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at org.apache.hadoop.mapred.JobQueueClient.main(JobQueueClient.java:240)
{CODE}
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6452) NPE when intermediate encrypt enabled for LocalRunner
[
https://issues.apache.org/jira/browse/MAPREDUCE-6452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhihai xu updated MAPREDUCE-6452:
-
Attachment: MAPREDUCE-6452.002.patch
> NPE when intermediate encrypt enabled for LocalRunner
> -
>
> Key: MAPREDUCE-6452
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6452
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: zhihai xu
> Attachments: MAPREDUCE-6452.002.patch, MAPREDUCE-6452.1.patch,
> TestLocalJobSubmission.java
>
>
> Enable the below properties try running mapreduce job
> mapreduce.framework.name=local
> mapreduce.job.encrypted-intermediate-data=true
> {code}
> 2015-08-14 16:27:25,248 WARN [Thread-21] mapred.LocalJobRunner
> (LocalJobRunner.java:run(561)) - job_local473843898_0001
> java.lang.Exception: java.lang.NullPointerException
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:463)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:523)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.crypto.CryptoOutputStream.(CryptoOutputStream.java:92)
> at
> org.apache.hadoop.fs.crypto.CryptoFSDataOutputStream.(CryptoFSDataOutputStream.java:31)
> at
> org.apache.hadoop.mapreduce.CryptoUtils.wrapIfNecessary(CryptoUtils.java:112)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1611)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1492)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:244)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> Jobs are failing always
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-6452) NPE when intermediate encrypt enabled for LocalRunner
[
https://issues.apache.org/jira/browse/MAPREDUCE-6452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
zhihai xu updated MAPREDUCE-6452:
-
Attachment: (was: MAPREDUCE-6452.000.patch)
> NPE when intermediate encrypt enabled for LocalRunner
> -
>
> Key: MAPREDUCE-6452
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6452
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Bibin A Chundatt
>Assignee: zhihai xu
> Attachments: MAPREDUCE-6452.1.patch, TestLocalJobSubmission.java
>
>
> Enable the below properties try running mapreduce job
> mapreduce.framework.name=local
> mapreduce.job.encrypted-intermediate-data=true
> {code}
> 2015-08-14 16:27:25,248 WARN [Thread-21] mapred.LocalJobRunner
> (LocalJobRunner.java:run(561)) - job_local473843898_0001
> java.lang.Exception: java.lang.NullPointerException
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.runTasks(LocalJobRunner.java:463)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:523)
> Caused by: java.lang.NullPointerException
> at
> org.apache.hadoop.crypto.CryptoOutputStream.(CryptoOutputStream.java:92)
> at
> org.apache.hadoop.fs.crypto.CryptoFSDataOutputStream.(CryptoFSDataOutputStream.java:31)
> at
> org.apache.hadoop.mapreduce.CryptoUtils.wrapIfNecessary(CryptoUtils.java:112)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java:1611)
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1492)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:723)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:793)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at
> org.apache.hadoop.mapred.LocalJobRunner$Job$MapTaskRunnable.run(LocalJobRunner.java:244)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> Jobs are failing always
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6463) AM should register RM using IP address instead of hostname
[
https://issues.apache.org/jira/browse/MAPREDUCE-6463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14711449#comment-14711449
]
Jun Gong commented on MAPREDUCE-6463:
-
The failed test case passed in my laptop, not sure why it failed.
> AM should register RM using IP address instead of hostname
> --
>
> Key: MAPREDUCE-6463
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6463
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Jun Gong
>Assignee: Jun Gong
> Attachments: MAPREDUCE-6463.001.patch
>
>
> I copied hadoop direcotry to a host which is not in cluster, and ran a
> streaming job on it. I encountered following error:
> {noformat}
> 15/08/25 18:48:07 INFO impl.YarnClientImpl: Submitted application
> application_1440039112410_1578
> 15/08/25 18:48:07 INFO mapreduce.JobSubmitter: JobID:job_1440039112410_1578
> ClientSubmitJob time:692
> 15/08/25 18:48:07 INFO mapreduce.Job: The url to track the job:
> http://10.222.7.210:8080/proxy/application_1440039112410_1578/
> 15/08/25 18:48:07 INFO mapreduce.Job: Running job: job_1440039112410_1578
> 15/08/25 18:48:13 INFO mapred.ClientServiceDelegate: Connecting to
> szsk-ad-serving-10-222-7-204:41486
> 15/08/25 18:48:13 INFO mapred.ClientServiceDelegate: Failed to contact
> AM/History for job job_1440039112410_1578 retrying..
> java.net.UnknownHostException: Invalid host name: local host is: (unknown);
> destination host is: "szsk-ad-serving-10-222-7-204":41486;
> java.net.UnknownHostException; For more details see:
> http://wiki.apache.org/hadoop/UnknownHost
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783)
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:742)
> at org.apache.hadoop.ipc.Client$Connection.(Client.java:400)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1452)
> at org.apache.hadoop.ipc.Client.call(Client.java:1381)
> at org.apache.hadoop.ipc.Client.call(Client.java:1363)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.proxy.$Proxy16.getJobReport(Unknown Source)
> at
> org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getJobReport(MRClientProtocolPBClientImpl.java:133)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:350)
> at
> org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:517)
> at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:536)
> at org.apache.hadoop.mapreduce.Job$1.run(Job.java:318)
> at org.apache.hadoop.mapreduce.Job$1.run(Job.java:315)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:415)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1667)
> at org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:315)
> at org.apache.hadoop.mapreduce.Job.isComplete(Job.java:604)
> at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1337)
> at
> org.apache.hadoop.mapred.JobClient$NetworkedJob.monitorAndPrintJob(JobClient.java:409)
> at
> org.apache.hadoop.mapred.JobClient.monitorAndPrintJob(JobClient.java:858)
> at
> org.apache.hadoop.streaming.StreamJob.submitAndMonitorJob(StreamJob.java:1018)
> at org.apache.hadoop.streaming.StreamJob.run(StreamJob.java:135)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at
> org.apache.hadoop.streaming.HadoopStreaming.main(HadoopStreaming.java:50)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:606)
> at org.apache.hadoop.ut
[jira] [Commented] (MAPREDUCE-6463) AM should register RM using IP address instead of hostname
[
https://issues.apache.org/jira/browse/MAPREDUCE-6463?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14711398#comment-14711398
]
Hadoop QA commented on MAPREDUCE-6463:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 16m 4s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 44s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 47s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 23s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | checkstyle | 0m 38s | There were no new checkstyle
issues. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 28s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 33s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 1m 7s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:red}-1{color} | mapreduce tests | 9m 12s | Tests failed in
hadoop-mapreduce-client-app. |
| | | 47m 1s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.mapreduce.v2.app.rm.TestRMContainerAllocator |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752238/MAPREDUCE-6463.001.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / eee0d45 |
| hadoop-mapreduce-client-app test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5953/artifact/patchprocess/testrun_hadoop-mapreduce-client-app.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5953/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf905.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5953/console |
This message was automatically generated.
> AM should register RM using IP address instead of hostname
> --
>
> Key: MAPREDUCE-6463
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6463
> Project: Hadoop Map/Reduce
> Issue Type: Bug
>Reporter: Jun Gong
>Assignee: Jun Gong
> Attachments: MAPREDUCE-6463.001.patch
>
>
> I copied hadoop direcotry to a host which is not in cluster, and ran a
> streaming job on it. I encountered following error:
> {noformat}
> 15/08/25 18:48:07 INFO impl.YarnClientImpl: Submitted application
> application_1440039112410_1578
> 15/08/25 18:48:07 INFO mapreduce.JobSubmitter: JobID:job_1440039112410_1578
> ClientSubmitJob time:692
> 15/08/25 18:48:07 INFO mapreduce.Job: The url to track the job:
> http://10.222.7.210:8080/proxy/application_1440039112410_1578/
> 15/08/25 18:48:07 INFO mapreduce.Job: Running job: job_1440039112410_1578
> 15/08/25 18:48:13 INFO mapred.ClientServiceDelegate: Connecting to
> szsk-ad-serving-10-222-7-204:41486
> 15/08/25 18:48:13 INFO mapred.ClientServiceDelegate: Failed to contact
> AM/History for job job_1440039112410_1578 retrying..
> java.net.UnknownHostException: Invalid host name: local host is: (unknown);
> destination host is: "szsk-ad-serving-10-222-7-204":41486;
> java.net.UnknownHostException; For more details see:
> http://wiki.apache.org/hadoop/UnknownHost
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783)
> at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:742)
> at org.apache.hadoop.ipc.Client$Connection.(Client.java:400)
> at org.apache.hadoop.ipc.Client.getConnection(Client.java:1452)
> at org.apache.hadoop.ipc.Client.call(Client.java:1381)
> at org.apache.hadoop.ipc.Client.call(Client.java:1363)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
> at com.sun.p
[jira] [Commented] (MAPREDUCE-6461) An implementation of RecordReader/FileInputFormat for generic binary data
[ https://issues.apache.org/jira/browse/MAPREDUCE-6461?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14711377#comment-14711377 ] Ted Malaska commented on MAPREDUCE-6461: I made this a long time ago and I know it works, because I've seen it work at a number of companies. Let me know if you want me to add this to the MapReduce project. https://github.com/tmalaska/CopybookInputFormat :) > An implementation of RecordReader/FileInputFormat for generic binary data > - > > Key: MAPREDUCE-6461 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6461 > Project: Hadoop Map/Reduce > Issue Type: Wish > Components: mrv2 >Reporter: Dustin Cote >Priority: Minor > > It would be nice to have a RecordReader implementation designed for binary > data for ETL processing (especially for mainframe encodings like EBCDIC). > Currently, we have: > http://hadoop.apache.org/docs/current/api/org/apache/hadoop/mapred/FixedLengthInputFormat.html > This is great for fix length fields, but it would be even better if we could > delimit records by a byte string dynamically. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[
https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14711373#comment-14711373
]
Hadoop QA commented on MAPREDUCE-5870:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 17m 59s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 40s | There were no new javac warning
messages. |
| {color:green}+1{color} | javadoc | 9m 46s | There were no new javadoc
warning messages. |
| {color:green}+1{color} | release audit | 0m 24s | The applied patch does
not increase the total number of release audit warnings. |
| {color:red}-1{color} | checkstyle | 1m 31s | The applied patch generated
26 new checkstyle issues (total was 255, now 277). |
| {color:red}-1{color} | whitespace | 0m 1s | The patch has 2 line(s) that
end in whitespace. Use git apply --whitespace=fix. |
| {color:green}+1{color} | install | 1m 31s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 34s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | findbugs | 3m 21s | The patch does not introduce
any new Findbugs (version 3.0.0) warnings. |
| {color:green}+1{color} | mapreduce tests | 0m 47s | Tests passed in
hadoop-mapreduce-client-common. |
| {color:red}-1{color} | mapreduce tests | 1m 41s | Tests failed in
hadoop-mapreduce-client-core. |
| {color:red}-1{color} | mapreduce tests | 108m 1s | Tests failed in
hadoop-mapreduce-client-jobclient. |
| | | 153m 27s | |
\\
\\
|| Reason || Tests ||
| Failed unit tests | hadoop.mapred.TestJobConf |
| | hadoop.mapreduce.TestMRJobClient |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752221/0004-MAPREDUCE-5870.patch
|
| Optional Tests | javadoc javac unit findbugs checkstyle |
| git revision | trunk / eee0d45 |
| checkstyle |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5952/artifact/patchprocess/diffcheckstylehadoop-mapreduce-client-core.txt
|
| whitespace |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5952/artifact/patchprocess/whitespace.txt
|
| hadoop-mapreduce-client-common test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5952/artifact/patchprocess/testrun_hadoop-mapreduce-client-common.txt
|
| hadoop-mapreduce-client-core test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5952/artifact/patchprocess/testrun_hadoop-mapreduce-client-core.txt
|
| hadoop-mapreduce-client-jobclient test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5952/artifact/patchprocess/testrun_hadoop-mapreduce-client-jobclient.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5952/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf903.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5952/console |
This message was automatically generated.
> Support for passing Job priority through Application Submission Context in
> Mapreduce Side
> -
>
> Key: MAPREDUCE-5870
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: client
>Reporter: Sunil G
>Assignee: Sunil G
> Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch,
> 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch, Yarn-2002.1.patch
>
>
> Job Prioirty can be set from client side as below [Configuration and api].
> a. JobConf.getJobPriority() and
> Job.setPriority(JobPriority priority)
> b. We can also use configuration
> "mapreduce.job.priority".
> Now this Job priority can be passed in Application Submission
> context from Client side.
> Here we can reuse the MRJobConfig.PRIORITY configuration.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6451) DistCp has incorrect chunkFilePath for multiple jobs when strategy is dynamic
[ https://issues.apache.org/jira/browse/MAPREDUCE-6451?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14711273#comment-14711273 ] Kuhu Shukla commented on MAPREDUCE-6451: [~szetszwo]Requesting you to please review. Thank you. > DistCp has incorrect chunkFilePath for multiple jobs when strategy is dynamic > - > > Key: MAPREDUCE-6451 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6451 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: distcp >Affects Versions: 2.6.0 >Reporter: Kuhu Shukla >Assignee: Kuhu Shukla > Attachments: MAPREDUCE-6451-v1.patch > > > DistCp when used with dynamic strategy does not update the chunkFilePath and > other static variables any time other than for the first job. This is seen > when DistCp::run() is used. > A single copy succeeds but multiple jobs finish successfully without any real > copying. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Updated] (MAPREDUCE-5870) Support for passing Job priority through Application Submission Context in Mapreduce Side
[ https://issues.apache.org/jira/browse/MAPREDUCE-5870?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Sunil G updated MAPREDUCE-5870: --- Attachment: 0004-MAPREDUCE-5870.patch Updating a new version of patch addressing the comments. Kindly help to check the same. > Support for passing Job priority through Application Submission Context in > Mapreduce Side > - > > Key: MAPREDUCE-5870 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-5870 > Project: Hadoop Map/Reduce > Issue Type: Improvement > Components: client >Reporter: Sunil G >Assignee: Sunil G > Attachments: 0001-MAPREDUCE-5870.patch, 0002-MAPREDUCE-5870.patch, > 0003-MAPREDUCE-5870.patch, 0004-MAPREDUCE-5870.patch, Yarn-2002.1.patch > > > Job Prioirty can be set from client side as below [Configuration and api]. > a. JobConf.getJobPriority() and > Job.setPriority(JobPriority priority) > b. We can also use configuration > "mapreduce.job.priority". > Now this Job priority can be passed in Application Submission > context from Client side. > Here we can reuse the MRJobConfig.PRIORITY configuration. -- This message was sent by Atlassian JIRA (v6.3.4#6332)
[jira] [Created] (MAPREDUCE-6463) AM should register RM using IP address instead of hostname
Jun Gong created MAPREDUCE-6463:
---
Summary: AM should register RM using IP address instead of hostname
Key: MAPREDUCE-6463
URL: https://issues.apache.org/jira/browse/MAPREDUCE-6463
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Jun Gong
Assignee: Jun Gong
I copied hadoop direcotry to a host which is not in cluster, and ran a
streaming job on it. I encountered following error:
{noformat}
15/08/25 18:48:07 INFO impl.YarnClientImpl: Submitted application
application_1440039112410_1578
15/08/25 18:48:07 INFO mapreduce.JobSubmitter: JobID:job_1440039112410_1578
ClientSubmitJob time:692
15/08/25 18:48:07 INFO mapreduce.Job: The url to track the job:
http://10.222.7.210:8080/proxy/application_1440039112410_1578/
15/08/25 18:48:07 INFO mapreduce.Job: Running job: job_1440039112410_1578
15/08/25 18:48:13 INFO mapred.ClientServiceDelegate: Connecting to
szsk-ad-serving-10-222-7-204:41486
15/08/25 18:48:13 INFO mapred.ClientServiceDelegate: Failed to contact
AM/History for job job_1440039112410_1578 retrying..
java.net.UnknownHostException: Invalid host name: local host is: (unknown);
destination host is: "szsk-ad-serving-10-222-7-204":41486;
java.net.UnknownHostException; For more details see:
http://wiki.apache.org/hadoop/UnknownHost
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at
sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
at org.apache.hadoop.net.NetUtils.wrapWithMessage(NetUtils.java:783)
at org.apache.hadoop.net.NetUtils.wrapException(NetUtils.java:742)
at org.apache.hadoop.ipc.Client$Connection.(Client.java:400)
at org.apache.hadoop.ipc.Client.getConnection(Client.java:1452)
at org.apache.hadoop.ipc.Client.call(Client.java:1381)
at org.apache.hadoop.ipc.Client.call(Client.java:1363)
at
org.apache.hadoop.ipc.ProtobufRpcEngine$Invoker.invoke(ProtobufRpcEngine.java:206)
at com.sun.proxy.$Proxy16.getJobReport(Unknown Source)
at
org.apache.hadoop.mapreduce.v2.api.impl.pb.client.MRClientProtocolPBClientImpl.getJobReport(MRClientProtocolPBClientImpl.java:133)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at
org.apache.hadoop.mapred.ClientServiceDelegate.invoke(ClientServiceDelegate.java:350)
at
org.apache.hadoop.mapred.ClientServiceDelegate.getJobStatus(ClientServiceDelegate.java:517)
at org.apache.hadoop.mapred.YARNRunner.getJobStatus(YARNRunner.java:536)
at org.apache.hadoop.mapreduce.Job$1.run(Job.java:318)
at org.apache.hadoop.mapreduce.Job$1.run(Job.java:315)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1667)
at org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:315)
at org.apache.hadoop.mapreduce.Job.isComplete(Job.java:604)
at org.apache.hadoop.mapreduce.Job.monitorAndPrintJob(Job.java:1337)
at
org.apache.hadoop.mapred.JobClient$NetworkedJob.monitorAndPrintJob(JobClient.java:409)
at
org.apache.hadoop.mapred.JobClient.monitorAndPrintJob(JobClient.java:858)
at
org.apache.hadoop.streaming.StreamJob.submitAndMonitorJob(StreamJob.java:1018)
at org.apache.hadoop.streaming.StreamJob.run(StreamJob.java:135)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at
org.apache.hadoop.streaming.HadoopStreaming.main(HadoopStreaming.java:50)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:606)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.net.UnknownHostException
... 34 more
{noformat}
The exception is because AM register RM with hostname
'szsk-ad-serving-10-222-7-204' and client could not resolve it.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
[jira] [Commented] (MAPREDUCE-6241) Native compilation fails for Checksum.cc due to an incompatibility of assembler register constraint for PowerPC
[
https://issues.apache.org/jira/browse/MAPREDUCE-6241?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14710995#comment-14710995
]
Hadoop QA commented on MAPREDUCE-6241:
--
\\
\\
| (x) *{color:red}-1 overall{color}* |
\\
\\
|| Vote || Subsystem || Runtime || Comment ||
| {color:blue}0{color} | pre-patch | 5m 22s | Pre-patch trunk compilation is
healthy. |
| {color:green}+1{color} | @author | 0m 0s | The patch does not contain any
@author tags. |
| {color:red}-1{color} | tests included | 0m 0s | The patch doesn't appear
to include any new or modified tests. Please justify why no new tests are
needed for this patch. Also please list what manual steps were performed to
verify this patch. |
| {color:green}+1{color} | javac | 7m 54s | There were no new javac warning
messages. |
| {color:green}+1{color} | release audit | 0m 20s | The applied patch does
not increase the total number of release audit warnings. |
| {color:green}+1{color} | whitespace | 0m 0s | The patch has no lines that
end in whitespace. |
| {color:green}+1{color} | install | 1m 28s | mvn install still works. |
| {color:green}+1{color} | eclipse:eclipse | 0m 35s | The patch built with
eclipse:eclipse. |
| {color:green}+1{color} | mapreduce tests | 10m 33s | Tests passed in
hadoop-mapreduce-client-nativetask. |
| | | 26m 16s | |
\\
\\
|| Subsystem || Report/Notes ||
| Patch URL |
http://issues.apache.org/jira/secure/attachment/12752187/MAPREDUCE-6241.003.patch
|
| Optional Tests | javac unit |
| git revision | trunk / eee0d45 |
| hadoop-mapreduce-client-nativetask test log |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5951/artifact/patchprocess/testrun_hadoop-mapreduce-client-nativetask.txt
|
| Test Results |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5951/testReport/ |
| Java | 1.7.0_55 |
| uname | Linux asf908.gq1.ygridcore.net 3.13.0-36-lowlatency #63-Ubuntu SMP
PREEMPT Wed Sep 3 21:56:12 UTC 2014 x86_64 x86_64 x86_64 GNU/Linux |
| Console output |
https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/5951/console |
This message was automatically generated.
> Native compilation fails for Checksum.cc due to an incompatibility of
> assembler register constraint for PowerPC
>
>
> Key: MAPREDUCE-6241
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-6241
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: build
>Affects Versions: 3.0.0, 2.6.0
> Environment: Debian/Jessie, kernel 3.18.5, ppc64 GNU/Linux
> gcc (Debian 4.9.1-19)
> protobuf 2.6.1
> OpenJDK Runtime Environment (IcedTea 2.5.3) (7u71-2.5.3-2)
> OpenJDK Zero VM (build 24.65-b04, interpreted mode)
> source was cloned (and updated) from Apache-Hadoop's git repository
>Reporter: Stephan Drescher
>Assignee: Binglin Chang
>Priority: Minor
> Labels: BB2015-05-TBR, features
> Attachments: MAPREDUCE-6241.001.patch, MAPREDUCE-6241.002.patch,
> MAPREDUCE-6241.003.patch
>
>
> Issue when using assembler code for performance optimization on the powerpc
> platform (compiled for 32bit)
> mvn compile -Pnative -DskipTests
> [exec] /usr/bin/c++ -Dnativetask_EXPORTS -m32 -DSIMPLE_MEMCPY
> -fno-strict-aliasing -Wall -Wno-sign-compare -g -O2 -DNDEBUG -fPIC
> -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native/javah
>
> -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src
>
> -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util
>
> -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib
>
> -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test
>
> -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src
>
> -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native
> -I/home/hadoop/Java/java7/include -I/home/hadoop/Java/java7/include/linux
> -isystem
> /home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/gtest/include
> -o CMakeFiles/nativetask.dir/main/native/src/util/Checksum.cc.o -c
> /home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc
> [exec] CMakeFiles/n
[jira] [Updated] (MAPREDUCE-6241) Native compilation fails for Checksum.cc due to an incompatibility of assembler register constraint for PowerPC
[ https://issues.apache.org/jira/browse/MAPREDUCE-6241?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ayappan updated MAPREDUCE-6241: --- Attachment: MAPREDUCE-6241.003.patch Fixed the whitespace in the new patch. Since this issue happens while building the code itself, no new testcases are required > Native compilation fails for Checksum.cc due to an incompatibility of > assembler register constraint for PowerPC > > > Key: MAPREDUCE-6241 > URL: https://issues.apache.org/jira/browse/MAPREDUCE-6241 > Project: Hadoop Map/Reduce > Issue Type: Bug > Components: build >Affects Versions: 3.0.0, 2.6.0 > Environment: Debian/Jessie, kernel 3.18.5, ppc64 GNU/Linux > gcc (Debian 4.9.1-19) > protobuf 2.6.1 > OpenJDK Runtime Environment (IcedTea 2.5.3) (7u71-2.5.3-2) > OpenJDK Zero VM (build 24.65-b04, interpreted mode) > source was cloned (and updated) from Apache-Hadoop's git repository >Reporter: Stephan Drescher >Assignee: Binglin Chang >Priority: Minor > Labels: BB2015-05-TBR, features > Attachments: MAPREDUCE-6241.001.patch, MAPREDUCE-6241.002.patch, > MAPREDUCE-6241.003.patch > > > Issue when using assembler code for performance optimization on the powerpc > platform (compiled for 32bit) > mvn compile -Pnative -DskipTests > [exec] /usr/bin/c++ -Dnativetask_EXPORTS -m32 -DSIMPLE_MEMCPY > -fno-strict-aliasing -Wall -Wno-sign-compare -g -O2 -DNDEBUG -fPIC > -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native/javah > > -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src > > -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util > > -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/lib > > -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/test > > -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src > > -I/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native > -I/home/hadoop/Java/java7/include -I/home/hadoop/Java/java7/include/linux > -isystem > /home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/gtest/include > -o CMakeFiles/nativetask.dir/main/native/src/util/Checksum.cc.o -c > /home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc > [exec] CMakeFiles/nativetask.dir/build.make:744: recipe for target > 'CMakeFiles/nativetask.dir/main/native/src/util/Checksum.cc.o' failed > [exec] make[2]: Leaving directory > '/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native' > [exec] CMakeFiles/Makefile2:95: recipe for target > 'CMakeFiles/nativetask.dir/all' failed > [exec] make[1]: Leaving directory > '/home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/target/native' > [exec] Makefile:76: recipe for target 'all' failed > [exec] > /home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc: > In function ‘void NativeTask::init_cpu_support_flag()’: > /home/hadoop/Developer/hadoop/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-nativetask/src/main/native/src/util/Checksum.cc:611:14: > error: impossible register constraint in ‘asm’ > --> > "popl %%ebx" : "=a" (eax), [ebx] "=r"(ebx), "=c"(ecx), "=d"(edx) : "a" > (eax_in) : "cc"); > <-- -- This message was sent by Atlassian JIRA (v6.3.4#6332)
