Re: common.HadoopJobStatusChecker:58 : error check status
7-07-14 10:41:42,544 DEBUG [Job 14691c4a-64d2-4b1d-ace5-
> d2d6ad9618d0-297] dao.ExecutableDao:217 : updating
> job output, id: 14691c4a-64d2-4b1d-ace5-d2d6ad9618d0-02
> 2017-07-14 10:41:53,548 INFO [Job 14691c4a-64d2-4b1d-ace5-
> d2d6ad9618d0-297] ipc.Client:867 : Retrying connect to server: dn1/
> 10.50.229.209:51098. Already tried 0 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=
> 1000 MILLISECONDS)
> 2017-07-14 10:41:54,549 INFO [Job 14691c4a-64d2-4b1d-ace5-
> d2d6ad9618d0-297] ipc.Client:867 : Retrying connect to server: dn1/
> 10.50.229.209:51098. Already tried 1 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=
> 1000 MILLISECONDS)
> 2017-07-14 10:41:55,549 INFO [Job 14691c4a-64d2-4b1d-ace5-
> d2d6ad9618d0-297] ipc.Client:867 : Retrying connect to server: dn1/
> 10.50.229.209:51098. Already tried 2 time(s); retry policy is
> RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=
> 1000 MILLISECONDS)
> 2017-07-14 10:41:55,663 INFO [Job 14691c4a-64d2-4b1d-ace5-
> d2d6ad9618d0-297] mapred.ClientServiceDelegate:277 : Application state is
> completed. FinalApplicationStatus=FAILED. Redirecting to job history
> server
> 2017-07-14 10:41:55,686 ERROR [Job 14691c4a-64d2-4b1d-ace5-
> d2d6ad9618d0-297] common.HadoopJobStatusChecker:58 : error check status
> java.io.IOException: Job status not available
> at org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:334)
> at org.apache.hadoop.mapreduce.Job.getStatus(Job.java:341)
> at org.apache.kylin.engine.mr.common.HadoopJobStatusChecker.checkStatus(
> HadoopJobStatusChecker.java:38)
> at org.apache.kylin.engine.mr.common.MapReduceExecutable.
> doWork(MapReduceExecutable.java:153)
> at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:124)
> at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(
> DefaultChainedExecutable.java:64)
> at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:124)
> at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(
> DefaultScheduler.java:142)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> 2017-07-14 10:41:55,687 ERROR [Job 14691c4a-64d2-4b1d-ace5-
> d2d6ad9618d0-297] common.MapReduceExecutable:197 : error execute
> MapReduceExecutable{id=14691c4a-64d2-4b1d-ace5-
> d2d6ad9618d0-02, name=Extract Fact Table Distinct Columns, state=RUNNING}
> java.lang.NullPointerException
> at org.apache.hadoop.mapreduce.Job.getTrackingURL(Job.java:380)
> at org.apache.kylin.engine.mr.common.HadoopCmdOutput.
> getInfo(HadoopCmdOutput.java:61)
> at org.apache.kylin.engine.mr.common.MapReduceExecutable.
> doWork(MapReduceExecutable.java:162)
> at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:124)
> at org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(
> DefaultChainedExecutable.java:64)
> at org.apache.kylin.job.execution.AbstractExecutable.
> execute(AbstractExecutable.java:124)
> at org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(
> DefaultScheduler.java:142)
> at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1142)
> at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:748)
> 2017-07-14 10:41:55,687 DEBUG [Job 14691c4a-64d2-4b1d-ace5-
> d2d6ad9618d0-297] dao.ExecutableDao:217 : updating
> job output, id: 14691c4a-64d2-4b1d-ace5-d2d6ad9618d0-02
> 2017-07-14 10:41:55,697 DEBUG [Job 14691c4a-64d2-4b1d-ace5-
> d2d6ad9618d0-297] dao.ExecutableDao:217 : updating
> job output, id: 14691c4a-64d2-4b1d-ace5-d2d6ad9618d0-02
> 2017-07-14 10:41:55,703 INFO [Job 14691c4a-64d2-4b1d-ace5-
> d2d6ad9618d0-297] execution.ExecutableManager:389 : job
> id:14691c4a-64d2-4b1d-ace5-d2d6ad9618d0-02 from RUNNING to ERROR
> 2017-07-14 10:41:55,713 DEBUG [Job 14691c4a-64d2-4b1d-ace5-
> d2d6ad9618d0-297] dao.ExecutableDao:217 : updating
> job output, id: 14691c4a-64d2-4b1d-ace5-d2d6ad9618d0
> 2017-07-14 10:41:55,728 DEBUG [Job 14691c4a-64d2-4b1d-ace5-
> d2d6ad9618d0-297] dao.ExecutableDao:217 : updating
> job output, id: 14691c4a-64d2-4b1d-ace5-d2d6ad9618d0
> 2017-07-14 10:41:55,731 DEBUG [Job 14691c4a-64d2-4b1d-ace5-
> d2d6ad9618d0-297] dao.ExecutableDao:217 : updating
> job output, id: 14691c4a-64d2-4b1d-ace5-d2d6ad9618d0
> 2017-07-14 10:41:55,734 INFO [Job 14691c4a-64d2-4b1d-ace5-
> d2d6ad9618d0-297] execution.ExecutableManager:389 : job
> id:14691c4a-64d2-4b1d-ace5-d2d6ad9618d0 from RUNNING t
回复：common.HadoopJobStatusChecker:58 : error check status
51098. Already
tried 2 time(s); retry policy is
RetryUpToMaximumCountWithFixedSleep(maxRetries=3, sleepTime=1000 MILLISECONDS)
2017-07-14 10:41:55,663 INFO [Job 14691c4a-64d2-4b1d-ace5-d2d6ad9618d0-297]
mapred.ClientServiceDelegate:277 : Application state is completed.
FinalApplicationStatus=FAILED. Redirecting to job history server
2017-07-14 10:41:55,686 ERROR [Job 14691c4a-64d2-4b1d-ace5-d2d6ad9618d0-297]
common.HadoopJobStatusChecker:58 : error check status
java.io.IOException: Job status not available
at org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:334)
at org.apache.hadoop.mapreduce.Job.getStatus(Job.java:341)
at
org.apache.kylin.engine.mr.common.HadoopJobStatusChecker.checkStatus(HadoopJobStatusChecker.java:38)
at
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:153)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:64)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:142)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
2017-07-14 10:41:55,687 ERROR [Job 14691c4a-64d2-4b1d-ace5-d2d6ad9618d0-297]
common.MapReduceExecutable:197 : error execute
MapReduceExecutable{id=14691c4a-64d2-4b1d-ace5-d2d6ad9618d0-02, name=Extract
Fact Table Distinct Columns, state=RUNNING}
java.lang.NullPointerException
at org.apache.hadoop.mapreduce.Job.getTrackingURL(Job.java:380)
at
org.apache.kylin.engine.mr.common.HadoopCmdOutput.getInfo(HadoopCmdOutput.java:61)
at
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:162)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:64)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:142)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
2017-07-14 10:41:55,687 DEBUG [Job 14691c4a-64d2-4b1d-ace5-d2d6ad9618d0-297]
dao.ExecutableDao:217 : updating job output, id:
14691c4a-64d2-4b1d-ace5-d2d6ad9618d0-02
2017-07-14 10:41:55,697 DEBUG [Job 14691c4a-64d2-4b1d-ace5-d2d6ad9618d0-297]
dao.ExecutableDao:217 : updating job output, id:
14691c4a-64d2-4b1d-ace5-d2d6ad9618d0-02
2017-07-14 10:41:55,703 INFO [Job 14691c4a-64d2-4b1d-ace5-d2d6ad9618d0-297]
execution.ExecutableManager:389 : job
id:14691c4a-64d2-4b1d-ace5-d2d6ad9618d0-02 from RUNNING to ERROR
2017-07-14 10:41:55,713 DEBUG [Job 14691c4a-64d2-4b1d-ace5-d2d6ad9618d0-297]
dao.ExecutableDao:217 : updating job output, id:
14691c4a-64d2-4b1d-ace5-d2d6ad9618d0
2017-07-14 10:41:55,728 DEBUG [Job 14691c4a-64d2-4b1d-ace5-d2d6ad9618d0-297]
dao.ExecutableDao:217 : updating job output, id:
14691c4a-64d2-4b1d-ace5-d2d6ad9618d0
2017-07-14 10:41:55,731 DEBUG [Job 14691c4a-64d2-4b1d-ace5-d2d6ad9618d0-297]
dao.ExecutableDao:217 : updating job output, id:
14691c4a-64d2-4b1d-ace5-d2d6ad9618d0
2017-07-14 10:41:55,734 INFO [Job 14691c4a-64d2-4b1d-ace5-d2d6ad9618d0-297]
execution.ExecutableManager:389 : job id:14691c4a-64d2-4b1d-ace5-d2d6ad9618d0
from RUNNING to ERROR
2017-07-14 10:41:55,734 WARN [Job 14691c4a-64d2-4b1d-ace5-d2d6ad9618d0-297]
execution.AbstractExecutable:258 : no need to send email, user list is empty
2017-07-14 10:41:55,745 INFO [pool-9-thread-1] threadpool.DefaultScheduler:124
: Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 10
already succeed, 9 error, 6 discarded, 0 others
2017-07-14 10:42:10,431 INFO [pool-9-thread-1] threadpool.DefaultScheduler:124
: Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 10
already succeed, 9 error, 6 discarded, 0 others
2017-07-14 10:43:10,432 INFO [pool-9-thread-1] threadpool.DefaultScheduler:124
: Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 10
already succeed, 9 error, 6 discarded, 0 others
2017-07-14 10:44:10,431 INFO [pool-9-thread-1] threadpool.DefaultScheduler:124
: Job Fetcher: 0 should running, 0 actual running, 0 stopped, 0 ready, 10
already succeed, 9 error, 6 discarded, 0 others
--发件人：crossme
<cros...@aliyun.com>发送时间：2017年7月13日(星期四) 19:51收件人：user <user@kylin.apache.org>主
题：common.HadoopJobStatusChecker:58 : error check status
Hi All The Cube build error on Step 3 Extract Fact Tab
common.HadoopJobStatusChecker:58 : error check status
Hi All The Cube build error on Step 3 Extract Fact Table Distinct Columns.
Here is the error message. Any help please.
Explain： This created 4 Cube test, only one of which all processes run,
Cube successfully constructed, can query, the rest of the Cube in the third
step error, the error log is below, do not achieve the status of the Job.
production environment:CDH-5.9 Kylin-2.0
2017-07-13 14:16:39,835 INFO [Job 3895f42c-8ee4-4eee-a0fc-9b511f9c0be4-437]
mapred.ClientServiceDelegate:277 : Application state is completed.
FinalApplicationStatus=SUCCEEDED. Redirecting to job history server
2017-07-13 14:16:39,849 ERROR [Job 3895f42c-8ee4-4eee-a0fc-9b511f9c0be4-437]
common.HadoopJobStatusChecker:58 : error check status
java.io.IOException: Job status not available
at org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:334)
at org.apache.hadoop.mapreduce.Job.getStatus(Job.java:341)
at
org.apache.kylin.engine.mr.common.HadoopJobStatusChecker.checkStatus(HadoopJobStatusChecker.java:38)
at
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:153)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:64)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:142)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
2017-07-13 14:16:39,850 ERROR [Job 3895f42c-8ee4-4eee-a0fc-9b511f9c0be4-437]
common.MapReduceExecutable:197 : error execute
MapReduceExecutable{id=3895f42c-8ee4-4eee-a0fc-9b511f9c0be4-07, name=Convert
Cuboid Data to HFile, state=RUNNING}
java.lang.NullPointerException
at org.apache.hadoop.mapreduce.Job.getTrackingURL(Job.java:380)
at
org.apache.kylin.engine.mr.common.HadoopCmdOutput.getInfo(HadoopCmdOutput.java:61)
at
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:162)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:64)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:124)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:142)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:748)
yarn-site.xml
yarn.resourcemanager.webapp.address
dn1:8088
