User can specify "kylin.job.yarn.app.rest.check.status.url" in
kylin.properties to overwrite the default setting in hadoop job
configuration;
You can search key word "yarn.resourcemanager.webapp.address" in
tomcat/logs/kylin.log to find the RM address value, manually replace the
{job_id} with your job's id, and then verify whether it is working.
Here is a sample of this property in kylin.properties, if you're going to
specify:
kylin.job.yarn.app.rest.check.status.url=https://
<your_rm_server>:<port>/ws/v1/cluster/apps/${job_id}?anonymous=true
2016-01-08 8:42 GMT+08:00 michael jones <mjones94...@gmail.com>:
> Am testing out Kyline with a very basic hive table
>
> Kyline - apache-kylin-1.3-HBase-1.1-SNAPSHOT-bin
> Hbase - 1.1.2
> Hadoop - 2.7.1
> Hive 1.2.1
>
> Sandbox all local setup.
>
>
>
> Build Cube
> Step#1 works well
> Step#2 fails
> #2 Step Name: Extract Fact Table Distinct Columns
> -conf /opt/kylin/conf/kylin_job_conf.xml -cubename sample -output
>
> /kylin/kylin_metadata/kylin-6493142b-66a9-4601-bc2f-03324fee1f6d/sample/fact_distinct_columns
> -jobname Kylin_Fact_Distinct_Columns_sample_Step -tablename
>
> default.kylin_intermediate_sample_19700101000000_2922789940817071255_6493142b_66a9_4601_bc2f_03324fee1f6d
>
> Exception: null
>
> I dont see any job being submitted by kylin on my hadoop cluster either.
> http://localhost:8088/cluster/apps
>
>
> I suspect i am missing some param in kylin properties to point kylin to the
> hadoop server/ resource manager ?
>
> kylin.log
>
> 2016-01-07 16:18:38,078 INFO [pool-20-thread-1]
> output.FileOutputCommitter: Saved output of task
> 'attempt_local1628663594_0002_r_000000_0' to
>
> hdfs://HOSTXXXX:9000/kylin/kylin_metadata/kylin-6493142b-66a9-4601-bc2f-03324fee1f6d/sample/fact_distinct_columns/_temporary/0/task_local1628663594_0002_r_000000
> 2016-01-07 16:18:38,079 INFO [pool-20-thread-1] mapred.LocalJobRunner:
> reduce > reduce
> 2016-01-07 16:18:38,079 INFO [pool-20-thread-1] mapred.Task: Task
> 'attempt_local1628663594_0002_r_000000_0' done.
> 2016-01-07 16:18:38,080 INFO [pool-20-thread-1] mapred.LocalJobRunner:
> Finishing task: attempt_local1628663594_0002_r_000000_0
> [pool-7-thread-6]:[2016-01-07
>
> 16:18:38,080][WARN][org.apache.commons.httpclient.HttpMethodBase.getResponseBody(HttpMethodBase.java:682)]
> - Going to buffer response body of large or unknown size. Using
> getResponseBodyAsStream instead is recommended.
> [pool-7-thread-6]:[2016-01-07
>
> 16:18:38,081][DEBUG][org.apache.kylin.job.tools.HadoopStatusGetter.getHttpResponse(HadoopStatusGetter.java:110)]
> - Job job_local1628663594_0002 get status check result.
>
> 2016-01-07 16:18:38,082 INFO [Thread-42] mapred.LocalJobRunner: reduce
> task executor complete.
> *[pool-7-thread-6]:[2016-01-07
>
> 16:18:38,098][ERROR][org.apache.kylin.job.tools.HadoopStatusChecker.checkStatus(HadoopStatusChecker.java:91)]
> - error check status*
> *java.lang.NullPointerException*
> at
>
> org.apache.kylin.job.tools.HadoopStatusGetter.get(HadoopStatusGetter.java:58)
> at
>
> org.apache.kylin.job.tools.HadoopStatusChecker.checkStatus(HadoopStatusChecker.java:56)
> at
>
> org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:144)
> at
>
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
> at
>
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)
> at
>
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
> at
>
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)
> at
>
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
>
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> [pool-7-thread-6]:[2016-01-07
>
> 16:18:38,100][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:200)]
> - Saving resource /execute_output/6493142b-66a9-4601-bc2f-03324fee1f6d-01
> (Store /opt/apache-kylin-1.3-HBase-1.1-SNAPSHOT-bin/meta)
> [pool-7-thread-6]:[2016-01-07
>
> 16:18:38,108][DEBUG][org.apache.kylin.job.common.HadoopCmdOutput.updateJobCounter(HadoopCmdOutput.java:94)]
> - Counters: 38
> File System Counters
> FILE: Number of bytes read=1309954
> FILE: Number of bytes written=3045158
> FILE: Number of read operations=0
> FILE: Number of large read operations=0
> FILE: Number of write operations=0
> HDFS: Number of bytes read=488
> HDFS: Number of bytes written=150
> HDFS: Number of read operations=61
> HDFS: Number of large read operations=0
> HDFS: Number of write operations=17
> Map-Reduce Framework
> Map input records=3
> Map output records=3
> Map output bytes=20
> Map output materialized bytes=42
> Input split bytes=1640
> Combine input records=3
> Combine output records=3
> Reduce input groups=1
> Reduce shuffle bytes=42
> Reduce input records=3
> Reduce output records=0
> Spilled Records=6
> Shuffled Maps =1
> Failed Shuffles=0
> Merged Map outputs=1
> GC time elapsed (ms)=0
> CPU time spent (ms)=0
> Physical memory (bytes) snapshot=0
> Virtual memory (bytes) snapshot=0
> Total committed heap usage (bytes)=2008023040
> Shuffle Errors
> BAD_ID=0
> CONNECTION=0
> IO_ERROR=0
> WRONG_LENGTH=0
> WRONG_MAP=0
> WRONG_REDUCE=0
> File Input Format Counters
> Bytes Read=0
> File Output Format Counters
> Bytes Written=128
> [pool-7-thread-6]:[2016-01-07
>
> 16:18:38,110][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:200)]
> - Saving resource /execute_output/6493142b-66a9-4601-bc2f-03324fee1f6d-01
> (Store /opt/apache-kylin-1.3-HBase-1.1-SNAPSHOT-bin/meta)
> [pool-7-thread-6]:[2016-01-07
>
> 16:18:38,114][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:200)]
> - Saving resource /execute_output/6493142b-66a9-4601-bc2f-03324fee1f6d-01
> (Store /opt/apache-kylin-1.3-HBase-1.1-SNAPSHOT-bin/meta)
> [pool-7-thread-6]:[2016-01-07
>
> 16:18:38,120][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:200)]
> - Saving resource /execute_output/6493142b-66a9-4601-bc2f-03324fee1f6d-01
> (Store /opt/apache-kylin-1.3-HBase-1.1-SNAPSHOT-bin/meta)
> [pool-7-thread-6]:[2016-01-07
>
> 16:18:38,121][INFO][org.apache.kylin.job.manager.ExecutableManager.updateJobOutput(ExecutableManager.java:241)]
> - job id:6493142b-66a9-4601-bc2f-03324fee1f6d-01 from RUNNING to ERROR
> [pool-7-thread-6]:[2016-01-07
>
> 16:18:38,124][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:200)]
> - Saving resource /execute_output/6493142b-66a9-4601-bc2f-03324fee1f6d
> (Store /opt/apache-kylin-1.3-HBase-1.1-SNAPSHOT-bin/meta)
> [pool-7-thread-6]:[2016-01-07
>
> 16:18:38,128][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:200)]
> - Saving resource /execute_output/6493142b-66a9-4601-bc2f-03324fee1f6d
> (Store /opt/apache-kylin-1.3-HBase-1.1-SNAPSHOT-bin/meta)
> [pool-7-thread-6]:[2016-01-07
>
> 16:18:38,131][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:200)]
> - Saving resource /execute_output/6493142b-66a9-4601-bc2f-03324fee1f6d
> (Store /opt/apache-kylin-1.3-HBase-1.1-SNAPSHOT-bin/meta)
> [pool-7-thread-6]:[2016-01-07
>
> 16:18:38,132][INFO][org.apache.kylin.job.manager.ExecutableManager.updateJobOutput(ExecutableManager.java:241)]
> - job id:6493142b-66a9-4601-bc2f-03324fee1f6d from RUNNING to ERROR
> [pool-6-thread-1]:[2016-01-07
>
> 16:18:38,136][INFO][org.apache.kylin.job.impl.threadpool.DefaultScheduler$FetcherRunner.run(DefaultScheduler.java:112)]
> - Job Fetcher: 0 running, 0 actual running, 0 ready, 3 others
>
--
Best regards,
Shaofeng Shi
