Pls check https://issues.apache.org/jira/browse/KYLIN-953
On Fri, Dec 11, 2015 at 8:25 PM, Mehak Mehta <[email protected]> wrote: > Hi, > > Thanks Li Yang for your reply. > > I solved that previous issue. > Now my cube is building but it always fails at 70% with the following > error: > > 2015-12-11 07:12:37,021 INFO [pool-4-thread-1] > mapreduce.HFileOutputFormat2: bulkload locality sensitive enabled > 2015-12-11 07:12:37,021 INFO [pool-4-thread-1] > mapreduce.HFileOutputFormat2: Looking up current regions for table > KYLIN_23ZG714AUU > 2015-12-11 07:12:37,029 INFO [pool-4-thread-1] > mapreduce.HFileOutputFormat2: Configuring 1 reduce partitions to match > current region count > [pool-4-thread-1]:[2015-12-11 > 07:12:37,030][ERROR]*[org.apache.kylin.job.hadoop.cube.CubeHFileJob.run(CubeHFileJob.java:95)] > - error in CubeHFileJob* > *java.lang.IllegalArgumentException: Can not create a Path from a null > string* > * at org.apache.hadoop.fs.Path.checkPathArg(Path.java:123)* > * at org.apache.hadoop.fs.Path.<init>(Path.java:135)* > * at org.apache.hadoop.fs.Path.<init>(Path.java:89)* > * at > org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.configurePartitioner(HFileOutputFormat2.java:615)* > * at > org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.configureIncrementalLoad(HFileOutputFormat2.java:464)* > * at > org.apache.hadoop.hbase.mapreduce.HFileOutputFormat.configureIncrementalLoad(HFileOutputFormat.java:88)* > * at > org.apache.kylin.job.hadoop.cube.CubeHFileJob.run(CubeHFileJob.java:86)* > * at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)* > * at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)* > * at > org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:113)* > * at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)* > * at > org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)* > * at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)* > * at > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)* > * at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)* > * at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)* > * at java.lang.Thread.run(Thread.java:745)* > *usage: CubeHFileJob* > * -cubename <name> Cube name. For exmaple, flat_item_cube* > * -htablename <htable name> HTable name* > * -input <path> Input path* > * -jobname <name> Job name. For exmaple,* > * Kylin_Cuboid_Builder-clsfd_v2_Step_22-D)* > -output <path> Output path > [pool-4-thread-1]:[2015-12-11 > 07:12:37,034][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob.cleanupTempConfFile(AbstractHadoopJob.java:340)] > - tempMetaFileString is : > file:///data/shared/mehak/hadoop/kylin/./bin/../tomcat/temp/kylin_job_meta4467707123004651766/meta > *[pool-4-thread-1]:[2015-12-11 > 07:12:37,040][ERROR][org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:116)] > - error execute > MapReduceExecutable{id=dbf394d0-5a18-4299-b191-c7bc0dc9b941-09, > name=Convert Cuboid Data to HFile, state=RUNNING}* > *java.lang.IllegalArgumentException: Can not create a Path from a null > string* > * at org.apache.hadoop.fs.Path.checkPathArg(Path.java:123)* > * at org.apache.hadoop.fs.Path.<init>(Path.java:135)* > * at org.apache.hadoop.fs.Path.<init>(Path.java:89)* > * at > org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.configurePartitioner(HFileOutputFormat2.java:615)* > * at > org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.configureIncrementalLoad(HFileOutputFormat2.java:464)* > * at > org.apache.hadoop.hbase.mapreduce.HFileOutputFormat.configureIncrementalLoad(HFileOutputFormat.java:88)* > * at > org.apache.kylin.job.hadoop.cube.CubeHFileJob.run(CubeHFileJob.java:86)* > * at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)* > * at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)* > * at > org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:113)* > * at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)* > * at > org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)* > * at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)* > * at > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)* > * at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)* > * at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)* > * at java.lang.Thread.run(Thread.java:745)* > [pool-4-thread-1]:[2015-12-11 > 07:12:37,045][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:195)] > - Saving resource /execute_output/dbf394d0-5a18-4299-b191-c7bc0dc9b941-09 > (Store kylin_metadata@hbase) > [pool-4-thread-1]:[2015-12-11 > 07:12:37,051][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:195)] > - Saving resource /execute_output/dbf394d0-5a18-4299-b191-c7bc0dc9b941-09 > (Store kylin_metadata@hbase) > [pool-4-thread-1]:[2015-12-11 > 07:12:37,053][INFO][org.apache.kylin.job.manager.ExecutableManager.updateJobOutput(ExecutableManager.java:241)] > - job id:dbf394d0-5a18-4299-b191-c7bc0dc9b941-09 from RUNNING to ERROR > > I cannot see any errors in hadoop, hive or hbase logs. It seems to be some > configuration issue. > Kindly help, I have tried a lot configurations but it doesn't work. > > Thanks & Regards, > Mehak Mehta > > > > On Thu, Dec 10, 2015 at 4:42 AM, Li Yang <[email protected]> wrote: > >> Kylin depends on a hadoop config "yarn.resourcemanager.webapp.address" to >> find resource manager and come up a ws URL to query job status. >> >> You can check if the config presents in site xmls on Kylin classpath. >> >> Or if troubleshooting Hadoop is too painful, a simpler way is adding a >> line in conf/kylin.properties. >> >> kylin.job.yarn.app.rest.check.status.url= >> https://YOUR_RM_AND_PORT/ws/v1/cluster/apps/${job_id}?anonymous=true >> >> On Tue, Dec 8, 2015 at 11:46 AM, Mehak Mehta <[email protected]> >> wrote: >> >>> Hi, >>> >>> I am getting the following error while building my cube using browser >>> console on my Hivetable. >>> I am using default architecture >>> >>> hadoop-2.6.0 (cluster of 10 nodes) >>> apache-hive-0.14.0 (using default metastore) >>> apache-kylin-1.1.1 (on single node) >>> hbase-0.98.16 (on single node as pseudo cluster) >>> >>> Note I have recompiled hadoop (with Snappy library) for my system >>> (Centos 6.5 64 bit). >>> It starts building the cube but fails while checking its status with the >>> following *Kylin logs,* can you please help me: >>> >>> 015-12-05 03:34:35,779 INFO [Thread-36] mapred.LocalJobRunner: >>> OutputCommitter is >>> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter >>> 2015-12-05 03:34:35,848 INFO [Thread-36] mapred.LocalJobRunner: Waiting >>> for map tasks >>> 2015-12-05 03:34:35,849 INFO [LocalJobRunner Map Task Executor #0] >>> mapred.LocalJobRunner: Starting task: >>> attempt_local1931418873_0001_m_000000_0 >>> 2015-12-05 03:34:35,890 INFO [LocalJobRunner Map Task Executor #0] >>> mapred.Task: Using ResourceCalculatorProcessTree : [ ] >>> 2015-12-05 03:34:35,897 INFO [LocalJobRunner Map Task Executor #0] >>> mapred.MapTask: Processing split: >>> org.apache.hive.hcatalog.mapreduce.HCatSplit@7d6447d >>> 2015-12-05 03:34:35,921 INFO [LocalJobRunner Map Task Executor #0] >>> mapred.MapTask: Map output collector class = >>> org.apache.hadoop.mapred.MapTask$MapOutputBuffer >>> 2015-12-05 03:34:35,963 INFO [LocalJobRunner Map Task Executor #0] >>> mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584) >>> 2015-12-05 03:34:35,963 INFO [LocalJobRunner Map Task Executor #0] >>> mapred.MapTask: mapreduce.task.io.sort.mb: 100 >>> 2015-12-05 03:34:35,963 INFO [LocalJobRunner Map Task Executor #0] >>> mapred.MapTask: soft limit at 83886080 >>> 2015-12-05 03:34:35,963 INFO [LocalJobRunner Map Task Executor #0] >>> mapred.MapTask: bufstart = 0; bufvoid = 104857600 >>> 2015-12-05 03:34:35,963 INFO [LocalJobRunner Map Task Executor #0] >>> mapred.MapTask: kvstart = 26214396; length = 6553600 >>> [pool-4-thread-2]:[2015-12-05 >>> 03:34:35,966][WARN][org.apache.commons.httpclient.HttpMethodBase.getResponseBody(HttpMethodBase.java:682)] >>> - Going to buffer response body of large or unknown size. Using >>> getResponseBodyAsStream instead is recommended. >>> [pool-4-thread-2]:[2015-12-05 >>> 03:34:35,967][DEBUG][org.apache.kylin.job.tools.HadoopStatusGetter.getHttpResponse(HadoopStatusGetter.java:92)] >>> - Job job_local1931418873_0001 get status check result. >>> >>> [pool-4-thread-2]:[2015-12-05 >>> 03:34:35,994][ERROR][org.apache.kylin.job.tools.HadoopStatusChecker.checkStatus(HadoopStatusChecker.java:91)] >>> - error check status >>> *java.lang.NullPointerException* >>> * at >>> org.apache.kylin.job.tools.HadoopStatusGetter.get(HadoopStatusGetter.java:56)* >>> * at >>> org.apache.kylin.job.tools.HadoopStatusChecker.checkStatus(HadoopStatusChecker.java:56)* >>> * at >>> org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:137)* >>> * at >>> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)* >>> * at >>> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)* >>> * at >>> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)* >>> * at >>> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)* >>> * at >>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)* >>> * at >>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)* >>> * at java.lang.Thread.run(Thread.java:745)* >>> [pool-4-thread-2]:[2015-12-05 >>> 03:34:35,998][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:195)] >>> - Saving resource /execute_output/d167963c-0bd0-422a-8abf-5c288d4f5131-01 >>> (Store kylin_metadata@hbase) >>> [pool-4-thread-2]:[2015-12-05 >>> 03:34:36,003][DEBUG][org.apache.kylin.job.common.HadoopCmdOutput.updateJobCounter(HadoopCmdOutput.java:94)] >>> - Counters: 0 >>> 2015-12-05 03:34:36,010 WARN [pool-4-thread-2] mapreduce.Counters: >>> Group FileSystemCounters is deprecated. Use >>> org.apache.hadoop.mapreduce.FileSystemCounter instead >>> [pool-4-thread-2]:[2015-12-05 >>> 03:34:36,013][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:195)] >>> - Saving resource /execute_output/d167963c-0bd0-422a-8abf-5c288d4f5131-01 >>> (Store kylin_metadata@hbase) >>> [pool-4-thread-2]:[2015-12-05 >>> 03:34:36,018][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:195)] >>> - Saving resource /execute_output/d167963c-0bd0-422a-8abf-5c288d4f5131-01 >>> (Store kylin_metadata@hbase) >>> [pool-4-thread-2]:[2015-12-05 >>> 03:34:36,023][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:195)] >>> - Saving resource /execute_output/d167963c-0bd0-422a-8abf-5c288d4f5131-01 >>> (Store kylin_metadata@hbase) >>> >>> >>> >>> I couldn't see any error in my Hive logs or Hadoop logs, but there were >>> some logs related to this in *Hbase Logs:* >>> >>> 2015-12-05 03:29:43,208 DEBUG [region-location-2] >>> regionserver.HRegionFileSystem: No StoreFiles for: >>> file:/data/shared/mehak/hadoop/hbase/hfiles/data/default/kylin_metadata_user/57de3063b02c8224546d5e67e5814455/a >>> *2015-12-05 03:29:43,208 DEBUG [region-location-2] >>> regionserver.HRegionFileSystem: No StoreFiles for: >>> file:/data/shared/mehak/hadoop/hbase/hfiles/data/default/kylin_metadata_user/57de3063b02c8224546d5e67e5814455/q* >>> *2015-12-05 03:30:01,382 INFO >>> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181 <http://0.0.0.0/0.0.0.0:2181>] >>> server.NIOServerCnxnFactory: Accepted socket connection from >>> /127.0.0.1:36859 <http://127.0.0.1:36859/>* >>> *2015-12-05 03:30:01,387 INFO >>> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181 <http://0.0.0.0/0.0.0.0:2181>] >>> server.ZooKeeperServer: Client attempting to establish new session at >>> /127.0.0.1:36859 <http://127.0.0.1:36859/>* >>> *2015-12-05 03:30:01,411 INFO [SyncThread:0] server.ZooKeeperServer: >>> Established session 0x15171398d090008 with negotiated timeout 40000 for >>> client /127.0.0.1:36859 <http://127.0.0.1:36859/>* >>> *2015-12-05 03:30:03,761 INFO >>> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181 <http://0.0.0.0/0.0.0.0:2181>] >>> server.NIOServerCnxnFactory: Accepted socket connection from >>> /127.0.0.1:36862 <http://127.0.0.1:36862/>* >>> *2015-12-05 03:30:03,761 INFO >>> [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181 <http://0.0.0.0/0.0.0.0:2181>] >>> server.ZooKeeperServer: Client attempting to establish new session at >>> /127.0.0.1:36862 <http://127.0.0.1:36862/>* >>> *2015-12-05 03:30:03,763 INFO [SyncThread:0] server.ZooKeeperServer: >>> Established session 0x15171398d090009 with negotiated timeout 40000 for >>> client /127.0.0.1:36862 <http://127.0.0.1:36862/>* >>> *2015-12-05 03:30:03,816 INFO [ProcessThread(sid:0 cport:-1):] >>> server.PrepRequestProcessor: Got user-level KeeperException when processing >>> sessionid:0x15171398d090009 type:create cxid:0x1 zxid:0x52d txntype:-1 >>> reqpath:n/a Error Path:/kylin/job_engine/lock/kylin_metadata >>> Error:KeeperErrorCode = NoNode for /kylin/job_engine/lock/kylin_metadata* >>> *2015-12-05 03:30:43,210 DEBUG [region-location-3] >>> regionserver.HRegionFileSystem: No StoreFiles for: >>> file:/data/shared/mehak/hadoop/hbase/hfiles/data/default/kylin_metadata_user/57de3063b02c8224546d5e67e5814455/a* >>> *2015-12-05 03:30:43,211 DEBUG [region-location-3] >>> regionserver.HRegionFileSystem: No StoreFiles for: >>> file:/data/shared/mehak/hadoop/hbase/hfiles/data/default/kylin_metadata_user/57de3063b02c8224546d5e67e5814455/q* >>> *2015-12-05 03:31:43,210 DEBUG [region-location-4] >>> regionserver.HRegionFileSystem: No StoreFiles for: >>> file:/data/shared/mehak/hadoop/hbase/hfiles/data/default/kylin_metadata_user/57de3063b02c8224546d5e67e5814455/a* >>> *2015-12-05 03:31:43,210 DEBUG [region-location-4] >>> regionserver.HRegionFileSystem: No StoreFiles for: >>> file:/data/shared/mehak/hadoop/hbase/hfiles/data/default/kylin_metadata_user/57de3063b02c8224546d5e67e5814455/q* >>> >>> But I couldn't figure out the cause of the problem, can anybody please >>> help me in resolving this. >>> I have been struggling with this for sometime now. >>> >>> -- >>> Thanks & Regards >>> Mehak Mehta >>> Masters - Computer Science >>> >> >> > > > -- > Thanks & Regards > Mehak Mehta > Masters - Computer Science >
