Thanks ShaoFeng.
This worked !
However the logging dont seem to show the params and the value for the
address.
On Thu, Jan 7, 2016 at 5:50 PM, ShaoFeng Shi <shaofeng...@apache.org> wrote:
> User can specify "kylin.job.yarn.app.rest.check.status.url" in
> kylin.properties to overwrite the default setting in hadoop job
> configuration;
>
> Added to the kylin.property and restarted.
> You can search key word "yarn.resourcemanager.webapp.address" in
> tomcat/logs/kylin.log to find the RM address value, manually replace the
> {job_id} with your job's id, and then verify whether it is working.
>
> I dint see "yarn.resourcemanager.webapp.address" anywhere in
@KYLIN_HOME/tomcat/logs/* including the kylin.log
my log level is at DEBUG.
> Here is a sample of this property in kylin.properties, if you're going to
> specify:
> kylin.job.yarn.app.rest.check.status.url=https://
> <your_rm_server>:<port>/ws/v1/cluster/apps/${job_id}?anonymous=true
>
> 2016-01-08 8:42 GMT+08:00 michael jones <mjones94...@gmail.com>:
>
> > Am testing out Kyline with a very basic hive table
> >
> > Kyline - apache-kylin-1.3-HBase-1.1-SNAPSHOT-bin
> > Hbase - 1.1.2
> > Hadoop - 2.7.1
> > Hive 1.2.1
> >
> > Sandbox all local setup.
> >
> >
> >
> > Build Cube
> > Step#1 works well
> > Step#2 fails
> > #2 Step Name: Extract Fact Table Distinct Columns
> > -conf /opt/kylin/conf/kylin_job_conf.xml -cubename sample -output
> >
> >
> /kylin/kylin_metadata/kylin-6493142b-66a9-4601-bc2f-03324fee1f6d/sample/fact_distinct_columns
> > -jobname Kylin_Fact_Distinct_Columns_sample_Step -tablename
> >
> >
> default.kylin_intermediate_sample_19700101000000_2922789940817071255_6493142b_66a9_4601_bc2f_03324fee1f6d
> >
> > Exception: null
> >
> > I dont see any job being submitted by kylin on my hadoop cluster either.
> > http://localhost:8088/cluster/apps
> >
> >
> > I suspect i am missing some param in kylin properties to point kylin to
> the
> > hadoop server/ resource manager ?
> >
> > kylin.log
> >
> > 2016-01-07 16:18:38,078 INFO [pool-20-thread-1]
> > output.FileOutputCommitter: Saved output of task
> > 'attempt_local1628663594_0002_r_000000_0' to
> >
> >
> hdfs://HOSTXXXX:9000/kylin/kylin_metadata/kylin-6493142b-66a9-4601-bc2f-03324fee1f6d/sample/fact_distinct_columns/_temporary/0/task_local1628663594_0002_r_000000
> > 2016-01-07 16:18:38,079 INFO [pool-20-thread-1] mapred.LocalJobRunner:
> > reduce > reduce
> > 2016-01-07 16:18:38,079 INFO [pool-20-thread-1] mapred.Task: Task
> > 'attempt_local1628663594_0002_r_000000_0' done.
> > 2016-01-07 16:18:38,080 INFO [pool-20-thread-1] mapred.LocalJobRunner:
> > Finishing task: attempt_local1628663594_0002_r_000000_0
> > [pool-7-thread-6]:[2016-01-07
> >
> >
> 16:18:38,080][WARN][org.apache.commons.httpclient.HttpMethodBase.getResponseBody(HttpMethodBase.java:682)]
> > - Going to buffer response body of large or unknown size. Using
> > getResponseBodyAsStream instead is recommended.
> > [pool-7-thread-6]:[2016-01-07
> >
> >
> 16:18:38,081][DEBUG][org.apache.kylin.job.tools.HadoopStatusGetter.getHttpResponse(HadoopStatusGetter.java:110)]
> > - Job job_local1628663594_0002 get status check result.
> >
> > 2016-01-07 16:18:38,082 INFO [Thread-42] mapred.LocalJobRunner: reduce
> > task executor complete.
> > *[pool-7-thread-6]:[2016-01-07
> >
> >
> 16:18:38,098][ERROR][org.apache.kylin.job.tools.HadoopStatusChecker.checkStatus(HadoopStatusChecker.java:91)]
> > - error check status*
> > *java.lang.NullPointerException*
> > at
> >
> >
> org.apache.kylin.job.tools.HadoopStatusGetter.get(HadoopStatusGetter.java:58)
> > at
> >
> >
> org.apache.kylin.job.tools.HadoopStatusChecker.checkStatus(HadoopStatusChecker.java:56)
> > at
> >
> >
> org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:144)
> > at
> >
> >
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
> > at
> >
> >
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)
> > at
> >
> >
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
> > at
> >
> >
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> > at
> >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> > at java.lang.Thread.run(Thread.java:745)
> > [pool-7-thread-6]:[2016-01-07
> >
> >
> 16:18:38,100][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:200)]
> > - Saving resource /execute_output/6493142b-66a9-4601-bc2f-03324fee1f6d-01
> > (Store /opt/apache-kylin-1.3-HBase-1.1-SNAPSHOT-bin/meta)
> > [pool-7-thread-6]:[2016-01-07
> >
> >
> 16:18:38,108][DEBUG][org.apache.kylin.job.common.HadoopCmdOutput.updateJobCounter(HadoopCmdOutput.java:94)]
> > - Counters: 38
> > File System Counters
> > FILE: Number of bytes read=1309954
> > FILE: Number of bytes written=3045158
> > FILE: Number of read operations=0
> > FILE: Number of large read operations=0
> > FILE: Number of write operations=0
> > HDFS: Number of bytes read=488
> > HDFS: Number of bytes written=150
> > HDFS: Number of read operations=61
> > HDFS: Number of large read operations=0
> > HDFS: Number of write operations=17
> > Map-Reduce Framework
> > Map input records=3
> > Map output records=3
> > Map output bytes=20
> > Map output materialized bytes=42
> > Input split bytes=1640
> > Combine input records=3
> > Combine output records=3
> > Reduce input groups=1
> > Reduce shuffle bytes=42
> > Reduce input records=3
> > Reduce output records=0
> > Spilled Records=6
> > Shuffled Maps =1
> > Failed Shuffles=0
> > Merged Map outputs=1
> > GC time elapsed (ms)=0
> > CPU time spent (ms)=0
> > Physical memory (bytes) snapshot=0
> > Virtual memory (bytes) snapshot=0
> > Total committed heap usage (bytes)=2008023040
> > Shuffle Errors
> > BAD_ID=0
> > CONNECTION=0
> > IO_ERROR=0
> > WRONG_LENGTH=0
> > WRONG_MAP=0
> > WRONG_REDUCE=0
> > File Input Format Counters
> > Bytes Read=0
> > File Output Format Counters
> > Bytes Written=128
> > [pool-7-thread-6]:[2016-01-07
> >
> >
> 16:18:38,110][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:200)]
> > - Saving resource /execute_output/6493142b-66a9-4601-bc2f-03324fee1f6d-01
> > (Store /opt/apache-kylin-1.3-HBase-1.1-SNAPSHOT-bin/meta)
> > [pool-7-thread-6]:[2016-01-07
> >
> >
> 16:18:38,114][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:200)]
> > - Saving resource /execute_output/6493142b-66a9-4601-bc2f-03324fee1f6d-01
> > (Store /opt/apache-kylin-1.3-HBase-1.1-SNAPSHOT-bin/meta)
> > [pool-7-thread-6]:[2016-01-07
> >
> >
> 16:18:38,120][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:200)]
> > - Saving resource /execute_output/6493142b-66a9-4601-bc2f-03324fee1f6d-01
> > (Store /opt/apache-kylin-1.3-HBase-1.1-SNAPSHOT-bin/meta)
> > [pool-7-thread-6]:[2016-01-07
> >
> >
> 16:18:38,121][INFO][org.apache.kylin.job.manager.ExecutableManager.updateJobOutput(ExecutableManager.java:241)]
> > - job id:6493142b-66a9-4601-bc2f-03324fee1f6d-01 from RUNNING to ERROR
> > [pool-7-thread-6]:[2016-01-07
> >
> >
> 16:18:38,124][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:200)]
> > - Saving resource /execute_output/6493142b-66a9-4601-bc2f-03324fee1f6d
> > (Store /opt/apache-kylin-1.3-HBase-1.1-SNAPSHOT-bin/meta)
> > [pool-7-thread-6]:[2016-01-07
> >
> >
> 16:18:38,128][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:200)]
> > - Saving resource /execute_output/6493142b-66a9-4601-bc2f-03324fee1f6d
> > (Store /opt/apache-kylin-1.3-HBase-1.1-SNAPSHOT-bin/meta)
> > [pool-7-thread-6]:[2016-01-07
> >
> >
> 16:18:38,131][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:200)]
> > - Saving resource /execute_output/6493142b-66a9-4601-bc2f-03324fee1f6d
> > (Store /opt/apache-kylin-1.3-HBase-1.1-SNAPSHOT-bin/meta)
> > [pool-7-thread-6]:[2016-01-07
> >
> >
> 16:18:38,132][INFO][org.apache.kylin.job.manager.ExecutableManager.updateJobOutput(ExecutableManager.java:241)]
> > - job id:6493142b-66a9-4601-bc2f-03324fee1f6d from RUNNING to ERROR
> > [pool-6-thread-1]:[2016-01-07
> >
> >
> 16:18:38,136][INFO][org.apache.kylin.job.impl.threadpool.DefaultScheduler$FetcherRunner.run(DefaultScheduler.java:112)]
> > - Job Fetcher: 0 running, 0 actual running, 0 ready, 3 others
> >
>
>
>
> --
> Best regards,
>
> Shaofeng Shi
>
