Hi,
Thanks Li Yang for your reply.
I solved that previous issue.
Now my cube is building but it always fails at 70% with the following error:
2015-12-11 07:12:37,021 INFO [pool-4-thread-1]
mapreduce.HFileOutputFormat2: bulkload locality sensitive enabled
2015-12-11 07:12:37,021 INFO [pool-4-thread-1]
mapreduce.HFileOutputFormat2: Looking up current regions for table
KYLIN_23ZG714AUU
2015-12-11 07:12:37,029 INFO [pool-4-thread-1]
mapreduce.HFileOutputFormat2: Configuring 1 reduce partitions to match
current region count
[pool-4-thread-1]:[2015-12-11
07:12:37,030][ERROR]*[org.apache.kylin.job.hadoop.cube.CubeHFileJob.run(CubeHFileJob.java:95)]
- error in CubeHFileJob*
*java.lang.IllegalArgumentException: Can not create a Path from a null
string*
* at org.apache.hadoop.fs.Path.checkPathArg(Path.java:123)*
* at org.apache.hadoop.fs.Path.<init>(Path.java:135)*
* at org.apache.hadoop.fs.Path.<init>(Path.java:89)*
* at
org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.configurePartitioner(HFileOutputFormat2.java:615)*
* at
org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.configureIncrementalLoad(HFileOutputFormat2.java:464)*
* at
org.apache.hadoop.hbase.mapreduce.HFileOutputFormat.configureIncrementalLoad(HFileOutputFormat.java:88)*
* at
org.apache.kylin.job.hadoop.cube.CubeHFileJob.run(CubeHFileJob.java:86)*
* at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)*
* at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)*
* at
org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:113)*
* at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)*
* at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)*
* at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)*
* at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)*
* at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)*
* at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)*
* at java.lang.Thread.run(Thread.java:745)*
*usage: CubeHFileJob*
* -cubename <name> Cube name. For exmaple, flat_item_cube*
* -htablename <htable name> HTable name*
* -input <path> Input path*
* -jobname <name> Job name. For exmaple,*
* Kylin_Cuboid_Builder-clsfd_v2_Step_22-D)*
-output <path> Output path
[pool-4-thread-1]:[2015-12-11
07:12:37,034][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob.cleanupTempConfFile(AbstractHadoopJob.java:340)]
- tempMetaFileString is :
file:///data/shared/mehak/hadoop/kylin/./bin/../tomcat/temp/kylin_job_meta4467707123004651766/meta
*[pool-4-thread-1]:[2015-12-11
07:12:37,040][ERROR][org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:116)]
- error execute
MapReduceExecutable{id=dbf394d0-5a18-4299-b191-c7bc0dc9b941-09,
name=Convert Cuboid Data to HFile, state=RUNNING}*
*java.lang.IllegalArgumentException: Can not create a Path from a null
string*
* at org.apache.hadoop.fs.Path.checkPathArg(Path.java:123)*
* at org.apache.hadoop.fs.Path.<init>(Path.java:135)*
* at org.apache.hadoop.fs.Path.<init>(Path.java:89)*
* at
org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.configurePartitioner(HFileOutputFormat2.java:615)*
* at
org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2.configureIncrementalLoad(HFileOutputFormat2.java:464)*
* at
org.apache.hadoop.hbase.mapreduce.HFileOutputFormat.configureIncrementalLoad(HFileOutputFormat.java:88)*
* at
org.apache.kylin.job.hadoop.cube.CubeHFileJob.run(CubeHFileJob.java:86)*
* at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)*
* at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)*
* at
org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:113)*
* at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)*
* at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)*
* at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)*
* at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)*
* at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)*
* at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)*
* at java.lang.Thread.run(Thread.java:745)*
[pool-4-thread-1]:[2015-12-11
07:12:37,045][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:195)]
- Saving resource /execute_output/dbf394d0-5a18-4299-b191-c7bc0dc9b941-09
(Store kylin_metadata@hbase)
[pool-4-thread-1]:[2015-12-11
07:12:37,051][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:195)]
- Saving resource /execute_output/dbf394d0-5a18-4299-b191-c7bc0dc9b941-09
(Store kylin_metadata@hbase)
[pool-4-thread-1]:[2015-12-11
07:12:37,053][INFO][org.apache.kylin.job.manager.ExecutableManager.updateJobOutput(ExecutableManager.java:241)]
- job id:dbf394d0-5a18-4299-b191-c7bc0dc9b941-09 from RUNNING to ERROR
I cannot see any errors in hadoop, hive or hbase logs. It seems to be some
configuration issue.
Kindly help, I have tried a lot configurations but it doesn't work.
Thanks & Regards,
Mehak Mehta
On Thu, Dec 10, 2015 at 4:42 AM, Li Yang <[email protected]> wrote:
> Kylin depends on a hadoop config "yarn.resourcemanager.webapp.address" to
> find resource manager and come up a ws URL to query job status.
>
> You can check if the config presents in site xmls on Kylin classpath.
>
> Or if troubleshooting Hadoop is too painful, a simpler way is adding a
> line in conf/kylin.properties.
>
> kylin.job.yarn.app.rest.check.status.url=
> https://YOUR_RM_AND_PORT/ws/v1/cluster/apps/${job_id}?anonymous=true
>
> On Tue, Dec 8, 2015 at 11:46 AM, Mehak Mehta <[email protected]>
> wrote:
>
>> Hi,
>>
>> I am getting the following error while building my cube using browser
>> console on my Hivetable.
>> I am using default architecture
>>
>> hadoop-2.6.0 (cluster of 10 nodes)
>> apache-hive-0.14.0 (using default metastore)
>> apache-kylin-1.1.1 (on single node)
>> hbase-0.98.16 (on single node as pseudo cluster)
>>
>> Note I have recompiled hadoop (with Snappy library) for my system (Centos
>> 6.5 64 bit).
>> It starts building the cube but fails while checking its status with the
>> following *Kylin logs,* can you please help me:
>>
>> 015-12-05 03:34:35,779 INFO [Thread-36] mapred.LocalJobRunner:
>> OutputCommitter is
>> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter
>> 2015-12-05 03:34:35,848 INFO [Thread-36] mapred.LocalJobRunner: Waiting
>> for map tasks
>> 2015-12-05 03:34:35,849 INFO [LocalJobRunner Map Task Executor #0]
>> mapred.LocalJobRunner: Starting task:
>> attempt_local1931418873_0001_m_000000_0
>> 2015-12-05 03:34:35,890 INFO [LocalJobRunner Map Task Executor #0]
>> mapred.Task: Using ResourceCalculatorProcessTree : [ ]
>> 2015-12-05 03:34:35,897 INFO [LocalJobRunner Map Task Executor #0]
>> mapred.MapTask: Processing split:
>> org.apache.hive.hcatalog.mapreduce.HCatSplit@7d6447d
>> 2015-12-05 03:34:35,921 INFO [LocalJobRunner Map Task Executor #0]
>> mapred.MapTask: Map output collector class =
>> org.apache.hadoop.mapred.MapTask$MapOutputBuffer
>> 2015-12-05 03:34:35,963 INFO [LocalJobRunner Map Task Executor #0]
>> mapred.MapTask: (EQUATOR) 0 kvi 26214396(104857584)
>> 2015-12-05 03:34:35,963 INFO [LocalJobRunner Map Task Executor #0]
>> mapred.MapTask: mapreduce.task.io.sort.mb: 100
>> 2015-12-05 03:34:35,963 INFO [LocalJobRunner Map Task Executor #0]
>> mapred.MapTask: soft limit at 83886080
>> 2015-12-05 03:34:35,963 INFO [LocalJobRunner Map Task Executor #0]
>> mapred.MapTask: bufstart = 0; bufvoid = 104857600
>> 2015-12-05 03:34:35,963 INFO [LocalJobRunner Map Task Executor #0]
>> mapred.MapTask: kvstart = 26214396; length = 6553600
>> [pool-4-thread-2]:[2015-12-05
>> 03:34:35,966][WARN][org.apache.commons.httpclient.HttpMethodBase.getResponseBody(HttpMethodBase.java:682)]
>> - Going to buffer response body of large or unknown size. Using
>> getResponseBodyAsStream instead is recommended.
>> [pool-4-thread-2]:[2015-12-05
>> 03:34:35,967][DEBUG][org.apache.kylin.job.tools.HadoopStatusGetter.getHttpResponse(HadoopStatusGetter.java:92)]
>> - Job job_local1931418873_0001 get status check result.
>>
>> [pool-4-thread-2]:[2015-12-05
>> 03:34:35,994][ERROR][org.apache.kylin.job.tools.HadoopStatusChecker.checkStatus(HadoopStatusChecker.java:91)]
>> - error check status
>> *java.lang.NullPointerException*
>> * at
>> org.apache.kylin.job.tools.HadoopStatusGetter.get(HadoopStatusGetter.java:56)*
>> * at
>> org.apache.kylin.job.tools.HadoopStatusChecker.checkStatus(HadoopStatusChecker.java:56)*
>> * at
>> org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:137)*
>> * at
>> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)*
>> * at
>> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:51)*
>> * at
>> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)*
>> * at
>> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:130)*
>> * at
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)*
>> * at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)*
>> * at java.lang.Thread.run(Thread.java:745)*
>> [pool-4-thread-2]:[2015-12-05
>> 03:34:35,998][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:195)]
>> - Saving resource /execute_output/d167963c-0bd0-422a-8abf-5c288d4f5131-01
>> (Store kylin_metadata@hbase)
>> [pool-4-thread-2]:[2015-12-05
>> 03:34:36,003][DEBUG][org.apache.kylin.job.common.HadoopCmdOutput.updateJobCounter(HadoopCmdOutput.java:94)]
>> - Counters: 0
>> 2015-12-05 03:34:36,010 WARN [pool-4-thread-2] mapreduce.Counters: Group
>> FileSystemCounters is deprecated. Use
>> org.apache.hadoop.mapreduce.FileSystemCounter instead
>> [pool-4-thread-2]:[2015-12-05
>> 03:34:36,013][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:195)]
>> - Saving resource /execute_output/d167963c-0bd0-422a-8abf-5c288d4f5131-01
>> (Store kylin_metadata@hbase)
>> [pool-4-thread-2]:[2015-12-05
>> 03:34:36,018][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:195)]
>> - Saving resource /execute_output/d167963c-0bd0-422a-8abf-5c288d4f5131-01
>> (Store kylin_metadata@hbase)
>> [pool-4-thread-2]:[2015-12-05
>> 03:34:36,023][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:195)]
>> - Saving resource /execute_output/d167963c-0bd0-422a-8abf-5c288d4f5131-01
>> (Store kylin_metadata@hbase)
>>
>>
>>
>> I couldn't see any error in my Hive logs or Hadoop logs, but there were
>> some logs related to this in *Hbase Logs:*
>>
>> 2015-12-05 03:29:43,208 DEBUG [region-location-2]
>> regionserver.HRegionFileSystem: No StoreFiles for:
>> file:/data/shared/mehak/hadoop/hbase/hfiles/data/default/kylin_metadata_user/57de3063b02c8224546d5e67e5814455/a
>> *2015-12-05 03:29:43,208 DEBUG [region-location-2]
>> regionserver.HRegionFileSystem: No StoreFiles for:
>> file:/data/shared/mehak/hadoop/hbase/hfiles/data/default/kylin_metadata_user/57de3063b02c8224546d5e67e5814455/q*
>> *2015-12-05 03:30:01,382 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181
>> <http://0.0.0.0/0.0.0.0:2181>] server.NIOServerCnxnFactory: Accepted socket
>> connection from /127.0.0.1:36859 <http://127.0.0.1:36859/>*
>> *2015-12-05 03:30:01,387 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181
>> <http://0.0.0.0/0.0.0.0:2181>] server.ZooKeeperServer: Client attempting to
>> establish new session at /127.0.0.1:36859 <http://127.0.0.1:36859/>*
>> *2015-12-05 03:30:01,411 INFO [SyncThread:0] server.ZooKeeperServer:
>> Established session 0x15171398d090008 with negotiated timeout 40000 for
>> client /127.0.0.1:36859 <http://127.0.0.1:36859/>*
>> *2015-12-05 03:30:03,761 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181
>> <http://0.0.0.0/0.0.0.0:2181>] server.NIOServerCnxnFactory: Accepted socket
>> connection from /127.0.0.1:36862 <http://127.0.0.1:36862/>*
>> *2015-12-05 03:30:03,761 INFO [NIOServerCxn.Factory:0.0.0.0/0.0.0.0:2181
>> <http://0.0.0.0/0.0.0.0:2181>] server.ZooKeeperServer: Client attempting to
>> establish new session at /127.0.0.1:36862 <http://127.0.0.1:36862/>*
>> *2015-12-05 03:30:03,763 INFO [SyncThread:0] server.ZooKeeperServer:
>> Established session 0x15171398d090009 with negotiated timeout 40000 for
>> client /127.0.0.1:36862 <http://127.0.0.1:36862/>*
>> *2015-12-05 03:30:03,816 INFO [ProcessThread(sid:0 cport:-1):]
>> server.PrepRequestProcessor: Got user-level KeeperException when processing
>> sessionid:0x15171398d090009 type:create cxid:0x1 zxid:0x52d txntype:-1
>> reqpath:n/a Error Path:/kylin/job_engine/lock/kylin_metadata
>> Error:KeeperErrorCode = NoNode for /kylin/job_engine/lock/kylin_metadata*
>> *2015-12-05 03:30:43,210 DEBUG [region-location-3]
>> regionserver.HRegionFileSystem: No StoreFiles for:
>> file:/data/shared/mehak/hadoop/hbase/hfiles/data/default/kylin_metadata_user/57de3063b02c8224546d5e67e5814455/a*
>> *2015-12-05 03:30:43,211 DEBUG [region-location-3]
>> regionserver.HRegionFileSystem: No StoreFiles for:
>> file:/data/shared/mehak/hadoop/hbase/hfiles/data/default/kylin_metadata_user/57de3063b02c8224546d5e67e5814455/q*
>> *2015-12-05 03:31:43,210 DEBUG [region-location-4]
>> regionserver.HRegionFileSystem: No StoreFiles for:
>> file:/data/shared/mehak/hadoop/hbase/hfiles/data/default/kylin_metadata_user/57de3063b02c8224546d5e67e5814455/a*
>> *2015-12-05 03:31:43,210 DEBUG [region-location-4]
>> regionserver.HRegionFileSystem: No StoreFiles for:
>> file:/data/shared/mehak/hadoop/hbase/hfiles/data/default/kylin_metadata_user/57de3063b02c8224546d5e67e5814455/q*
>>
>> But I couldn't figure out the cause of the problem, can anybody please
>> help me in resolving this.
>> I have been struggling with this for sometime now.
>>
>> --
>> Thanks & Regards
>> Mehak Mehta
>> Masters - Computer Science
>>
>
>
--
Thanks & Regards
Mehak Mehta
Masters - Computer Science
