Kylin depends on a hadoop config "yarn.resourcemanager.webapp.address" to find resource manager and come up a ws URL to query job status.
You can check if the config presents in site xmls on Kylin classpath. Or if troubleshooting Hadoop is too painful, a simpler way is adding a line in conf/kylin.properties. kylin.job.yarn.app.rest.check.status.url= https://YOUR_RM_AND_PORT/ws/v1/cluster/apps/${job_id}?anonymous=true On Tue, Dec 8, 2015 at 11:46 AM, Mehak Mehta <[email protected]> wrote: > Hi, > > I am getting the following error while building my cube using browser > console on my Hivetable. > I am using default architecture > > hadoop-2.6.0 (cluster of 10 nodes) > apache-hive-0.14.0 (using default metastore) > apache-kylin-1.1.1 (on single node) > hbase-0.98.16 (on single node as pseudo cluster) > > Note I have recompiled hadoop (with Snappy library) for my system (Centos > 6.5 64 bit). > It starts building the cube but fails while checking its status with the > following *Kylin logs,* can you please help me: > > 015-12-05 03:34:35,779 INFO [Thread-36] mapred.LocalJobRunner: > OutputCommitter is > org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter > 2015-12-05 03:34:35,848 INFO [Thread-36] mapred.LocalJobRunner: Waiting > for map tasks > 2015-12-05 03:34:35,849 INFO [LocalJobRunner Map Task Executor #0] > mapred.LocalJobRunner: Starting task: > attempt_local1931418873_0001_m_000000_0 > 2015-12-05 03:34:35,890 INFO [LocalJobRunner Map Task Executor #0] > mapred.Task: Using ResourceCalculatorProcessTree : [ ] > 2015-12-05 03:34:35,897 INFO [LocalJobRunner Map Task Executor #0] > mapred.MapTask: Processing split: > org.apache.hive.hcatalog.mapreduce.HCatSplit@7d6447d > 2015-12-05 03:34:35,921 INFO [LocalJobRunner Map Task Executor #0] > mapred.MapTask: Map output collector class = > org.apache.hadoop.mapred.MapTask$MapOutputBuffer > 2015-12-05 03:34:35,963 INFO [LocalJobRunner Map Task Executor #0] > mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) > 2015-12-05 03:34:35,963 INFO [LocalJobRunner Map Task Executor #0] > mapred.MapTask: mapreduce.task.io.sort.mb: 100 > 2015-12-05 03:34:35,963 INFO [LocalJobRunner Map Task Executor #0] > mapred.MapTask: soft limit at 83886080 > 2015-12-05 03:34:35,963 INFO [LocalJobRunner Map Task Executor #0] > mapred.MapTask: bufstart = 0; bufvoid = 104857600 > 2015-12-05 03:34:35,963 INFO [LocalJobRunner Map Task Executor #0] > mapred.MapTask: kvstart = 26214396; length = 6553600 > [pool-4-thread-2]:[2015-12-05 > 03:34:35,966][WARN][org.apache.commons.httpclient.HttpMethodBase.getResponseBody(HttpMethodBase.java:682)] > - Going to buffer response body of large or unknown size. Using > getResponseBodyAsStream instead is recommended. > [pool-4-thread-2]:[2015-12-05 > 03:34:35,967][DEBUG][org.apache.kylin.job.tools.HadoopStatusGetter.getHttpResponse(HadoopStatusGetter.java:92)] > - Job job_local1931418873_0001 get status check result. > > [pool-4-thread-2]:[2015-12-05 > 03:34:35,994][ERROR][org.apache.kylin.job.tools.HadoopStatusChecker.checkStatus(HadoopStatusChecker.java:91)] > - error check status > *java.lang.NullPointerException* > * at > org.apache.kylin.job.tools.HadoopStatusGetter.get(HadoopStatusGetter.java:56)* > * at > org.apache.kylin.job.tools.HadoopStatusChecker.checkStatus(HadoopStatusChecker.java:56)* > * at > org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:137)* > * at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)* > * at > org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)* > * at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)* > * at > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)* > * at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)* > * at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)* > * at java.lang.Thread.run(Thread.java:745)* > [pool-4-thread-2]:[2015-12-05 > 03:34:35,998][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:195)] > - Saving resource /execute_output/d167963c-0bd0-422a-8abf-5c288d4f5131-01 > (Store kylin_metadata@hbase) > [pool-4-thread-2]:[2015-12-05 > 03:34:36,003][DEBUG][org.apache.kylin.job.common.HadoopCmdOutput.updateJobCounter(HadoopCmdOutput.java:94)] > - Counters: 0 > 2015-12-05 03:34:36,010 WARN [pool-4-thread-2] mapreduce.Counters: Group > FileSystemCounters is deprecated. Use > org.apache.hadoop.mapreduce.FileSystemCounter instead > [pool-4-thread-2]:[2015-12-05 > 03:34:36,013][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:195)] > - Saving resource /execute_output/d167963c-0bd0-422a-8abf-5c288d4f5131-01 > (Store kylin_metadata@hbase) > [pool-4-thread-2]:[2015-12-05 > 03:34:36,018][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:195)] > - Saving resource /execute_output/d167963c-0bd0-422a-8abf-5c288d4f5131-01 > (Store kylin_metadata@hbase) > [pool-4-thread-2]:[2015-12-05 > 03:34:36,023][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:195)] > - Saving resource /execute_output/d167963c-0bd0-422a-8abf-5c288d4f5131-01 > (Store kylin_metadata@hbase) > > > > I couldn't see any error in my Hive logs or Hadoop logs, but there were > some logs related to this in *Hbase Logs:* > > 2015-12-05 03:29:43,208 DEBUG [region-location-2] > regionserver.HRegionFileSystem: No StoreFiles for: > file:/data/shared/mehak/hadoop/hbase/hfiles/data/default/kylin_metadata_user/57de3063b02c8224546d5e67e5814455/a > *2015-12-05 03:29:43,208 DEBUG [region-location-2] > regionserver.HRegionFileSystem: No StoreFiles for: > file:/data/shared/mehak/hadoop/hbase/hfiles/data/default/kylin_metadata_user/57de3063b02c8224546d5e67e5814455/q* > *2015-12-05 03:30:01,382 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181 > <http://0.0.0.0/0.0.0.0:2181>] server.NIOServerCnxnFactory: Accepted socket > connection from /127.0.0.1:36859 <http://127.0.0.1:36859/>* > *2015-12-05 03:30:01,387 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181 > <http://0.0.0.0/0.0.0.0:2181>] server.ZooKeeperServer: Client attempting to > establish new session at /127.0.0.1:36859 <http://127.0.0.1:36859/>* > *2015-12-05 03:30:01,411 INFO [SyncThread:0] server.ZooKeeperServer: > Established session 0x15171398d090008 with negotiated timeout 40000 for > client /127.0.0.1:36859 <http://127.0.0.1:36859/>* > *2015-12-05 03:30:03,761 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181 > <http://0.0.0.0/0.0.0.0:2181>] server.NIOServerCnxnFactory: Accepted socket > connection from /127.0.0.1:36862 <http://127.0.0.1:36862/>* > *2015-12-05 03:30:03,761 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181 > <http://0.0.0.0/0.0.0.0:2181>] server.ZooKeeperServer: Client attempting to > establish new session at /127.0.0.1:36862 <http://127.0.0.1:36862/>* > *2015-12-05 03:30:03,763 INFO [SyncThread:0] server.ZooKeeperServer: > Established session 0x15171398d090009 with negotiated timeout 40000 for > client /127.0.0.1:36862 <http://127.0.0.1:36862/>* > *2015-12-05 03:30:03,816 INFO [ProcessThread(sid:0 cport:-1):] > server.PrepRequestProcessor: Got user-level KeeperException when processing > sessionid:0x15171398d090009 type:create cxid:0x1 zxid:0x52d txntype:-1 > reqpath:n/a Error Path:/kylin/job_engine/lock/kylin_metadata > Error:KeeperErrorCode = NoNode for /kylin/job_engine/lock/kylin_metadata* > *2015-12-05 03:30:43,210 DEBUG [region-location-3] > regionserver.HRegionFileSystem: No StoreFiles for: > file:/data/shared/mehak/hadoop/hbase/hfiles/data/default/kylin_metadata_user/57de3063b02c8224546d5e67e5814455/a* > *2015-12-05 03:30:43,211 DEBUG [region-location-3] > regionserver.HRegionFileSystem: No StoreFiles for: > file:/data/shared/mehak/hadoop/hbase/hfiles/data/default/kylin_metadata_user/57de3063b02c8224546d5e67e5814455/q* > *2015-12-05 03:31:43,210 DEBUG [region-location-4] > regionserver.HRegionFileSystem: No StoreFiles for: > file:/data/shared/mehak/hadoop/hbase/hfiles/data/default/kylin_metadata_user/57de3063b02c8224546d5e67e5814455/a* > *2015-12-05 03:31:43,210 DEBUG [region-location-4] > regionserver.HRegionFileSystem: No StoreFiles for: > file:/data/shared/mehak/hadoop/hbase/hfiles/data/default/kylin_metadata_user/57de3063b02c8224546d5e67e5814455/q* > > But I couldn't figure out the cause of the problem, can anybody please > help me in resolving this. > I have been struggling with this for sometime now. > > -- > Thanks & Regards > Mehak Mehta > Masters - Computer Science >
