[
https://issues.apache.org/jira/browse/HIVE-20253?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
gehaijiang updated HIVE-20253:
------------------------------
Description:
hadoop 3.0.3, Support nativetask.
mapred-site.xml:
<property>
<name>mapreduce.job.map.output.collector.class</name>
<value>org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator</value>
</property>
hive sql:
set
mapreduce.job.map.output.collector.class=org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator;
select count(*) from test_cold; --test_cold (orcfile table)
URL:
[http://0.0.0.0:8088/taskdetails.jsp?jobid=job_1532646043398_0019&tipid=task_1532646043398_0019_m_000000]
----
Diagnostic Messages for this Task:
Error: java.io.IOException: Initialization of all the collectors failed. Error
in last collector was:java.io.IOException: Cannot find serializer for
org.apache.hadoop.hive.ql.io.HiveKey
at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:423)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:454)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1686)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
Caused by: java.io.IOException: Cannot find serializer for
org.apache.hadoop.hive.ql.io.HiveKey
at
org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator.init(NativeMapOutputCollectorDelegator.java:127)
at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:408)
... 7 more
2018-07-27 10:08:25,391 ERROR operation.Operation (SQLOperation.java:run(209))
- Error running hive query:
org.apache.hive.service.cli.HiveSQLException: Error while processing
statement: FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.mr.MapRedTask
at
org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:316)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:156)
at
org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:71)
at
org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:206)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
was:
hadoop 3.0.3, Support nativetask.
mapred-site.xml:
<property>
<name>mapreduce.job.map.output.collector.class</name>
<value>org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator</value>
</property>
hive sql:
set
mapreduce.job.map.output.collector.class=org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator;
select count(*) from test_cold;
URL:
http://0.0.0.0:8088/taskdetails.jsp?jobid=job_1532646043398_0019&tipid=task_1532646043398_0019_m_000000
-----
Diagnostic Messages for this Task:
Error: java.io.IOException: Initialization of all the collectors failed. Error
in last collector was:java.io.IOException: Cannot find serializer for
org.apache.hadoop.hive.ql.io.HiveKey
at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:423)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:454)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1686)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
Caused by: java.io.IOException: Cannot find serializer for
org.apache.hadoop.hive.ql.io.HiveKey
at
org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator.init(NativeMapOutputCollectorDelegator.java:127)
at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:408)
... 7 more
2018-07-27 10:08:25,391 ERROR operation.Operation (SQLOperation.java:run(209))
- Error running hive query:
org.apache.hive.service.cli.HiveSQLException: Error while processing statement:
FAILED: Execution Error, return code 2 from
org.apache.hadoop.hive.ql.exec.mr.MapRedTask
at
org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:316)
at
org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:156)
at
org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:71)
at
org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:206)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
at
org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:218)
at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
> nativetask cann't working in hive
> ----------------------------------
>
> Key: HIVE-20253
> URL: https://issues.apache.org/jira/browse/HIVE-20253
> Project: Hive
> Issue Type: Wish
> Affects Versions: 1.2.1
> Environment: hadoop 3.0.3
> hive sql:
> set
> mapreduce.job.map.output.collector.class=org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator;
> select count(*) from test_cold;
> Reporter: gehaijiang
> Priority: Major
>
> hadoop 3.0.3, Support nativetask.
> mapred-site.xml:
> <property>
> <name>mapreduce.job.map.output.collector.class</name>
>
> <value>org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator</value>
> </property>
>
> hive sql:
> set
> mapreduce.job.map.output.collector.class=org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator;
> select count(*) from test_cold; --test_cold (orcfile table)
>
> URL:
>
> [http://0.0.0.0:8088/taskdetails.jsp?jobid=job_1532646043398_0019&tipid=task_1532646043398_0019_m_000000]
> ----
> Diagnostic Messages for this Task:
> Error: java.io.IOException: Initialization of all the collectors failed.
> Error in last collector was:java.io.IOException: Cannot find serializer for
> org.apache.hadoop.hive.ql.io.HiveKey
> at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:423)
> at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:454)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:349)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:174)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1686)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:168)
> Caused by: java.io.IOException: Cannot find serializer for
> org.apache.hadoop.hive.ql.io.HiveKey
> at
> org.apache.hadoop.mapred.nativetask.NativeMapOutputCollectorDelegator.init(NativeMapOutputCollectorDelegator.java:127)
> at org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:408)
> ... 7 more
>
> 2018-07-27 10:08:25,391 ERROR operation.Operation
> (SQLOperation.java:run(209)) - Error running hive query:
> org.apache.hive.service.cli.HiveSQLException: Error while processing
> statement: FAILED: Execution Error, return code 2 from
> org.apache.hadoop.hive.ql.exec.mr.MapRedTask
> at
> org.apache.hive.service.cli.operation.Operation.toSQLException(Operation.java:316)
> at
> org.apache.hive.service.cli.operation.SQLOperation.runQuery(SQLOperation.java:156)
> at
> org.apache.hive.service.cli.operation.SQLOperation.access$100(SQLOperation.java:71)
> at
> org.apache.hive.service.cli.operation.SQLOperation$1$1.run(SQLOperation.java:206)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1657)
> at
> org.apache.hive.service.cli.operation.SQLOperation$1.run(SQLOperation.java:218)
> at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
