vincentzhao created HIVE-20643:
----------------------------------
Summary: hive job was hang when execute aggregate calculation such
as max()、count()
Key: HIVE-20643
URL: https://issues.apache.org/jira/browse/HIVE-20643
Project: Hive
Issue Type: Bug
Affects Versions: 2.3.0
Reporter: vincentzhao
if i run "select * from table limit 10" in the hive, it can return successful.
but when i was run "select max() from table" ,it is hang.
*the output massage :-----------------------------------*
Logging initialized using configuration in
file:/app/apache-hive-2.3.0-bin/conf/hive-log4j2.properties Async: true
WARNING: Hive-on-MR is deprecated in Hive 2 and may not be available in the
future versions. Consider using a different execution engine (i.e. spark, tez)
or using Hive 1.X releases.
Query ID = root_20180927085541_71b0e652-ad29-4a2f-ac53-1e9a4f34cec1
Total jobs = 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
set hive.exec.reducers.bytes.per.reducer=<number>
In order to limit the maximum number of reducers:
set hive.exec.reducers.max=<number>
In order to set a constant number of reducers:
set mapreduce.job.reduces=<number>
Cannot run job locally: Input Size (= 562011785) is larger than
hive.exec.mode.local.auto.inputbytes.max (= 256000000)。
*The hive.log
:---------------------------------------------------------------------*
========"new HiveConf()"========
2018-09-27T08:55:46,192 INFO [5cf99469-4f8e-49b1-9108-b94b3928a2df main]:
lockmgr.DbLockManager (DbLockManager.java:lock(104)) - Response to
queryId=root_20180927085541_71b0e652-ad29-4a2f-ac53-1e9a4f34cec1
LockResponse(lockid:19039, state:ACQUIRED)
2018-09-27T08:55:46,193 INFO [5cf99469-4f8e-49b1-9108-b94b3928a2df main]:
lockmgr.DbTxnManager (DbTxnManager.java:startHeartbeat(507)) - Started
heartbeat with delay/interval = 150000/150000 MILLISECONDS for query:
root_20180927085541_71b0e652-ad29-4a2f-ac53-1e9a4f34cec1
2018-09-27T08:55:46,193 INFO [5cf99469-4f8e-49b1-9108-b94b3928a2df main]:
ql.Driver (Driver.java:execute(1734)) - Executing
command(queryId=root_20180927085541_71b0e652-ad29-4a2f-ac53-1e9a4f34cec1):
select count(*) from tick2
2018-09-27T08:55:46,195 WARN [5cf99469-4f8e-49b1-9108-b94b3928a2df main]:
ql.Driver (Driver.java:logMrWarning(2094)) - Hive-on-MR is deprecated in Hive 2
and may not be available in the future versions. Consider using a different
execution engine (i.e. spark, tez) or using Hive 1.X releases.
2018-09-27T08:55:46,195 INFO [5cf99469-4f8e-49b1-9108-b94b3928a2df main]:
ql.Driver (SessionState.java:printInfo(1111)) - WARNING: Hive-on-MR is
deprecated in Hive 2 and may not be available in the future versions. Consider
using a different execution engine (i.e. spark, tez) or using Hive 1.X releases.
2018-09-27T08:55:46,197 INFO [5cf99469-4f8e-49b1-9108-b94b3928a2df main]:
ql.Driver (SessionState.java:printInfo(1111)) - Query ID =
root_20180927085541_71b0e652-ad29-4a2f-ac53-1e9a4f34cec1
2018-09-27T08:55:46,197 INFO [5cf99469-4f8e-49b1-9108-b94b3928a2df main]:
ql.Driver (SessionState.java:printInfo(1111)) - Total jobs = 1
2018-09-27T08:55:46,209 INFO [5cf99469-4f8e-49b1-9108-b94b3928a2df main]:
ql.Driver (SessionState.java:printInfo(1111)) - Launching Job 1 out of 1
2018-09-27T08:55:46,211 INFO [5cf99469-4f8e-49b1-9108-b94b3928a2df main]:
ql.Driver (Driver.java:launchTask(2174)) - Starting task [Stage-1:MAPRED] in
parallel
2018-09-27T08:55:46,212 INFO [Thread-6]: exec.Task
(SessionState.java:printInfo(1111)) - Number of reduce tasks determined at
compile time: 1
2018-09-27T08:55:46,212 INFO [Thread-6]: exec.Task
(SessionState.java:printInfo(1111)) - In order to change the average load for a
reducer (in bytes):
2018-09-27T08:55:46,212 INFO [Thread-6]: exec.Task
(SessionState.java:printInfo(1111)) - set
hive.exec.reducers.bytes.per.reducer=<number>
2018-09-27T08:55:46,212 INFO [Thread-6]: exec.Task
(SessionState.java:printInfo(1111)) - In order to limit the maximum number of
reducers:
2018-09-27T08:55:46,212 INFO [Thread-6]: exec.Task
(SessionState.java:printInfo(1111)) - set hive.exec.reducers.max=<number>
2018-09-27T08:55:46,212 INFO [Thread-6]: exec.Task
(SessionState.java:printInfo(1111)) - In order to set a constant number of
reducers:
2018-09-27T08:55:46,212 INFO [Thread-6]: exec.Task
(SessionState.java:printInfo(1111)) - set mapreduce.job.reduces=<number>
2018-09-27T08:55:46,225 INFO [Thread-6]: exec.Task
(SessionState.java:printInfo(1111)) - Cannot run job locally: Input Size (=
562011785) is larger than hive.exec.mode.local.auto.inputbytes.max (= 256000000)
2018-09-27T08:55:46,231 INFO [Thread-6]: ql.Context
(Context.java:getMRScratchDir(454)) - New scratch dir is
hdfs://cluster/app/apache-hive-2.3.0-bin/tmp/hive-root/root/5cf99469-4f8e-49b1-9108-b94b3928a2df/hive_2018-09-27_08-55-41_865_3811548964141108768-2
2018-09-27T08:55:46,237 INFO [Thread-6]: mr.ExecDriver
(ExecDriver.java:execute(287)) - Using
org.apache.hadoop.hive.ql.io.CombineHiveInputFormat
2018-09-27T08:55:46,241 INFO [Thread-6]: exec.Utilities
(Utilities.java:getInputPaths(3023)) - Processing alias tick2
2018-09-27T08:55:46,241 INFO [Thread-6]: exec.Utilities
(Utilities.java:getInputPaths(3052)) - Adding 1 inputs; the first input is
hdfs://cluster/usr/hive/warehouse/tick2
2018-09-27T08:55:46,242 INFO [Thread-6]: exec.Utilities
(Utilities.java:isEmptyPath(2332)) - Content Summary
hdfs://cluster/usr/hive/warehouse/tick2length: 562011785 num files: 1 num
directories: 1
2018-09-27T08:55:46,242 INFO [Thread-6]: ql.Context
(Context.java:getMRScratchDir(454)) - New scratch dir is
hdfs://cluster/app/apache-hive-2.3.0-bin/tmp/hive-root/root/5cf99469-4f8e-49b1-9108-b94b3928a2df/hive_2018-09-27_08-55-41_865_3811548964141108768-2
2018-09-27T08:55:46,331 INFO [Thread-6]: exec.SerializationUtilities
(SerializationUtilities.java:serializePlan(450)) - Serializing MapWork using
kryo
2018-09-27T08:55:46,471 INFO [Thread-6]: Configuration.deprecation
(Configuration.java:warnOnceIfDeprecated(1173)) - mapred.submit.replication is
deprecated. Instead, use mapreduce.client.submit.file.replication
2018-09-27T08:55:46,476 INFO [Thread-6]: exec.Utilities
(Utilities.java:setBaseWork(616)) - Serialized plan (via FILE) - name: null
size: 6.48KB
2018-09-27T08:55:46,479 INFO [Thread-6]: exec.SerializationUtilities
(SerializationUtilities.java:serializePlan(450)) - Serializing ReduceWork using
kryo
2018-09-27T08:55:46,499 INFO [Thread-6]: exec.Utilities
(Utilities.java:setBaseWork(616)) - Serialized plan (via FILE) - name: null
size: 5.67KB
2018-09-27T08:55:46,736 INFO [Thread-6]: exec.Utilities
(Utilities.java:getBaseWork(407)) - PLAN PATH =
hdfs://cluster/app/apache-hive-2.3.0-bin/tmp/hive-root/root/5cf99469-4f8e-49b1-9108-b94b3928a2df/hive_2018-09-27_08-55-41_865_3811548964141108768-2/-mr-10005/0be9de35-7cf8-4896-8496-964bbeebc9c8/map.xml
2018-09-27T08:55:46,736 INFO [Thread-6]: exec.Utilities
(Utilities.java:getBaseWork(407)) - PLAN PATH =
hdfs://cluster/app/apache-hive-2.3.0-bin/tmp/hive-root/root/5cf99469-4f8e-49b1-9108-b94b3928a2df/hive_2018-09-27_08-55-41_865_3811548964141108768-2/-mr-10005/0be9de35-7cf8-4896-8496-964bbeebc9c8/reduce.xml
2018-09-27T08:58:16,195 INFO [Heartbeater-0]: lockmgr.DbTxnManager
(DbTxnManager.java:heartbeat(455)) - Sending heartbeat for txnid:0 and
lockid:19039 queryId=root_20180927085541_71b0e652-ad29-4a2f-ac53-1e9a4f34cec1
txnid:0
2018-09-27T09:00:46,194 INFO [Heartbeater-0]: lockmgr.DbTxnManager
(DbTxnManager.java:heartbeat(455)) - Sending heartbeat for txnid:0 and
lockid:19039 queryId=root_20180927085541_71b0e652-ad29-4a2f-ac53-1e9a4f34cec1
txnid:0
2018-09-27T09:03:16,194 INFO [Heartbeater-1]: lockmgr.DbTxnManager
(DbTxnManager.java:heartbeat(455)) - Sending heartbeat for txnid:0 and
lockid:19039 queryId=root_20180927085541_71b0e652-ad29-4a2f-ac53-1e9a4f34cec1
txnid:0
2018-09-27T09:05:46,193 INFO [Heartbeater-0]: lockmgr.DbTxnManager
(DbTxnManager.java:heartbeat(455)) - Sending heartbeat for txnid:0 and
lockid:19039 queryId=root_20180927085541_71b0e652-ad29-4a2f-ac53-1e9a4f34cec1
txnid:0
2018-09-27T09:08:16,194 INFO [Heartbeater-2]: lockmgr.DbTxnManager
(DbTxnManager.java:heartbeat(455)) - Sending heartbeat for txnid:0 and
lockid:19039 queryId=root_20180927085541_71b0e652-ad29-4a2f-ac53-1e9a4f34cec1
txnid:0
2018-09-27T09:10:46,198 INFO [Heartbeater-2]: lockmgr.DbTxnManager
(DbTxnManager.java:heartbeat(455)) - Sending heartbeat for txnid:0 and
lockid:19039 queryId=root_20180927085541_71b0e652-ad29-4a2f-ac53-1e9a4f34cec1
txnid:0
2018-09-27T09:13:16,193 INFO [Heartbeater-2]: lockmgr.DbTxnManager
(DbTxnManager.java:heartbeat(455)) - Sending heartbeat for txnid:0 and
lockid:19039 queryId=root_20180927085541_71b0e652-ad29-4a2f-ac53-1e9a4f34cec1
txnid:0
2018-09-27T09:15:46,194 INFO [Heartbeater-2]: lockmgr.DbTxnManager
(DbTxnManager.java:heartbeat(455)) - Sending heartbeat for txnid:0 and
lockid:19039 queryId=root_20180927085541_71b0e652-ad29-4a2f-ac53-1e9a4f34cec1
txnid:0
2018-09-27T09:18:16,193 INFO [Heartbeater-2]: lockmgr.DbTxnManager
(DbTxnManager.java:heartbeat(455)) - Sending heartbeat for txnid:0 and
lockid:19039 queryId=root_20180927085541_71b0e652-ad29-4a2f-ac53-1e9a4f34cec1
txnid:0
2018-09-27T09:20:46,194 INFO [Heartbeater-2]: lockmgr.DbTxnManager
(DbTxnManager.java:heartbeat(455)) - Sending heartbeat for txnid:0 and
lockid:19039 queryId=root_20180927085541_71b0e652-ad29-4a2f-ac53-1e9a4f34cec1
txnid:0
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
