Deepesh Khandelwal created HIVE-6521:
----------------------------------------
Summary: WebHCat cannot fetch correct percentComplete for Hive jobs
Key: HIVE-6521
URL: https://issues.apache.org/jira/browse/HIVE-6521
Project: Hive
Issue Type: Bug
Components: WebHCat
Affects Versions: 0.13.0
Reporter: Deepesh Khandelwal
Assignee: Deepesh Khandelwal
WebHCat E2E test TestHive_7 failed because percentComplete wasn't returned as
expected.
{noformat}
check_job_percent_complete failed. got percentComplete "map 0% reduce 0%",
expected "map 100% reduce 100%"
{noformat}
So, there are two problems here.
# The log parsing is broken for status of percentComplete. In the stderr of the
job we see:
{noformat}
Launching Job 1 out of 1
Number of reduce tasks is set to 0 since there's no reduce operator
Starting Job = job_1393486488858_0691, Tracking URL =
http://ambari-sec-1393480847-others-2-4.cs1cloud.internal:8088/proxy/application_1393486488858_0691/
Kill Command = /usr/lib/hadoop/bin/hadoop job -kill job_1393486488858_0691
Hadoop job information for Stage-1: number of mappers: 1; number of reducers: 0
2014-02-27 18:40:50,166 Stage-1 map = 0%, reduce = 0%
2014-02-27 18:40:56,599 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.87
sec
2014-02-27 18:40:57,656 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.87
sec
2014-02-27 18:40:58,706 Stage-1 map = 100%, reduce = 0%, Cumulative CPU 0.87
sec
MapReduce Total cumulative CPU time: 870 msec
Ended Job = job_1393486488858_0691
MapReduce Jobs Launched:
Job 0: Map: 1 Cumulative CPU: 0.87 sec HDFS Read: 305 HDFS Write: 0 SUCCESS
Total MapReduce CPU Time Spent: 870 msec
{noformat}
The assumption in the code is that the line containing the percent status will
end after "reduce = \d+%" but that fails with the above.
# The last status from Hive job is "map = 100%, reduce = 0%" instead of
expected "map = 100%, reduce = 100%".
--
This message was sent by Atlassian JIRA
(v6.1.5#6160)
