Re: Kylie Cube Building,Merging and Refreshing getting stucked at Create HFile Step

[kdcool6932]

Thanks,Found out the issue, actually it it was not related to Kylin but known 
Hbase issue for the version I am using.The Kylin cleanup storage job actually 
stuck , it was not able to drop a table due to hbase permission issues, and 
thus all other  subsequent requests for creating hbase table handlers started 
queueing.Got the issue while looking at hbase dump.

Regards,Ketan


Sent from my Samsung Galaxy smartphone.
 Original message From: ShaoFeng Shi  
Date: 16/02/2017  07:39  (GMT+05:30) To: dev  Subject: 
Re: Kylie Cube Building,Merging and Refreshing getting stucked at Create HFile 
Step 
hi ketan, which step do you get stuck on, "Create HTable" or "Convert to
HFile"? What's the output in log? If it is "Convert to HFile" step, please
also check the mapreduce job's output.

2017-02-14 17:20 GMT+08:00 ketan dikshit :

> Hi Guys,
> We have a cube which is getting stucked for all type of jobs:
> Build,Refresh and Merge on the “Create Hfile” step.
>
> Any idea, how can I get back to normalcy.
> We are using Kylin 1.5.4.1 and the cube is being built incrementally on an
> hourly basis.
>
>
> Thanks,
> kdcool6932




-- 
Best regards,

Shaofeng Shi 史少锋

Re: Problem while build cube：File not found for hdfs://...

[排骨瘦肉丁]

It seems that the problem is still in the Hadoop environment。No wonder the
official proposal to use the Enterprise Edition，Most of them are
environmental problems。。

--
View this message in context: 
http://apache-kylin.74782.x6.nabble.com/Problem-while-build-cube-File-not-found-for-hdfs-tp7139p7215.html
Sent from the Apache Kylin mailing list archive at Nabble.com.

Fwd: 我在跑kylin的demo的时候遇到了问题，恳请您帮助一下是否遇见过这样的错误

[Luke Han]

Forward to user mailing list for further support.

Thanks.


-- Forwarded message --
From: ⒐o、cc 
Date: Tue, Feb 14, 2017 at 7:32 PM
Subject: 我在跑kylin的demo的时候遇到了问题，恳请您帮助一下是否遇见过这样的错误
To: lukehan 


一、实验环境
由于Apache kylin用到了hive及hbase，那么hadoop集群是必须的了，所以先要保证hadoop集群稳定正常，
然后确保hive和hbase安装了切稳定正常。

测试环境如下：

三台 VMware上的Linux虚机，每台120G（非预先分配），内存1G。
IP与主机名：
192.168.65.61 kylin1
192.168.65.62 kylin2
192.168.65.63 kylin3

主机规划：
192.168.65.61做master，运行NameNode和ResourceManager进程。这三台主机都做slave，
运行DataNode和NodeManager进程。
操作系统：CentOS release 6.5
java版本："1.8.0_121"
hadoop版本：hadoop-2.7.3
zookeeper版本：zookeeper-3.4.9
hbase版本：hbase-1.2.4
hive版本：hive-2.1.1
kylin版本：apache-kylin-1.6.0-hbase1.x-bin.tar.gz

【问题】：
每次跑kylin自带的demo的时候都会在#17 Step Name: Convert Cuboid Data to HFile这一个步骤报错

日志信息为：
Log Type: syslog
Log Upload Time: 8-二月-2017 16:34:36
Log Length: 93072
2017-02-08 16:34:14,664 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster:
Created MRAppMaster for application appattempt_1486537920487_0014_01
2017-02-08 16:34:15,109 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster:
Executing with tokens:
2017-02-08 16:34:15,272 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster:
Kind: YARN_AM_RM_TOKEN, Service: , Ident: (appAttemptId { application_id {
id: 14 cluster_timestamp: 1486537920487 } attemptId: 1 } keyId: 909018579)
2017-02-08 16:34:15,293 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster:
Using mapred newApiCommitter.
2017-02-08 16:34:16,191 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster:
OutputCommitter set in config null
2017-02-08 16:34:16,268 INFO [main]
org.apache.hadoop.mapreduce.v2.app.MRAppMaster:
OutputCommitter is org.apache.hadoop.mapreduce.
lib.output.FileOutputCommitter
2017-02-08 16:34:16,302 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class org.apache.hadoop.mapreduce.jobhistory.EventType for
class org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler
2017-02-08 16:34:16,303 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class org.apache.hadoop.mapreduce.v2.app.job.event.JobEventType
for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$JobEventDispatcher
2017-02-08 16:34:16,304 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class org.apache.hadoop.mapreduce.v2.app.job.event.TaskEventType
for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$TaskEventDispatcher
2017-02-08 16:34:16,306 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.TaskAttemptEventType
for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$
TaskAttemptEventDispatcher
2017-02-08 16:34:16,306 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventType
for class org.apache.hadoop.mapreduce.v2.app.commit.CommitterEventHandler
2017-02-08 16:34:16,315 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.mapreduce.v2.app.speculate.Speculator$EventType
for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$
SpeculatorEventDispatcher
2017-02-08 16:34:16,316 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.mapreduce.v2.app.rm.ContainerAllocator$EventType
for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$
ContainerAllocatorRouter
2017-02-08 16:34:16,317 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.mapreduce.v2.app.launcher.ContainerLauncher$EventType
for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$
ContainerLauncherRouter
2017-02-08 16:34:16,437 INFO [main]
org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils:
Default file system [hdfs://cluster:8020]
2017-02-08 16:34:16,472 INFO [main]
org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils:
Default file system [hdfs://cluster:8020]
2017-02-08 16:34:16,510 INFO [main]
org.apache.hadoop.mapreduce.v2.jobhistory.JobHistoryUtils:
Default file system [hdfs://cluster:8020]
2017-02-08 16:34:16,531 INFO [main]
org.apache.hadoop.mapreduce.jobhistory.JobHistoryEventHandler:
Emitting job history data to the timeline server is not enabled
2017-02-08 16:34:16,591 INFO [main]
org.apache.hadoop.yarn.event.AsyncDispatcher:
Registering class
org.apache.hadoop.mapreduce.v2.app.job.event.JobFinishEvent$Type
for class org.apache.hadoop.mapreduce.v2.app.MRAppMaster$
JobFinishEventHandler
2017-02-08 16:34:16,882 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsConfig:
loaded properties from hadoop-metrics2.properties
2017-02-08 16:34:16,983 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl:
Scheduled snapshot period at 10 second(s).
2017-02-08 16:34:16,983 INFO [main]
org.apache.hadoop.metrics2.impl.MetricsSystemImpl:
MRAppMaster metrics system started
2017-02-08 16:34:16,995 INFO [main]

Re: Problem while build cube：File not found for hdfs://...

[排骨瘦肉丁]

if you build a new cube,the error will reappear..

--
View this message in context: 
http://apache-kylin.74782.x6.nabble.com/Problem-while-build-cube-File-not-found-for-hdfs-tp7139p7176.html
Sent from the Apache Kylin mailing list archive at Nabble.com.

Help needed - Issue when build the cube

[xingyejun]

Hi, 

Does anyone met the following issues  when building a cube? 

java.lang.IllegalStateException
at
org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:98)
at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:88)
at
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:120)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:57)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:136)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

result code:2

--
View this message in context: 
http://apache-kylin.74782.x6.nabble.com/Help-needed-Issue-when-build-the-cube-tp7191.html
Sent from the Apache Kylin mailing list archive at Nabble.com.

Trouble with building cube at step 3！！cry！cry！

[排骨瘦肉丁]

kylin build cube at step 3，why ？configuration error？ where？
java.io.FileNotFoundException: File does not exist:
hdfs://localhost:9000/home/root/hadoop/tmp/mapred/staging/root398825514/.staging/job_local398825514_0001/libjars/hive-exec-1.2.1.jar
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1072)
at
org.apache.hadoop.hdfs.DistributedFileSystem$17.doCall(DistributedFileSystem.java:1064)
at
org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
at
org.apache.hadoop.hdfs.DistributedFileSystem.getFileStatus(DistributedFileSystem.java:1064)
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:288)
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.getFileStatus(ClientDistributedCacheManager.java:224)
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestamps(ClientDistributedCacheManager.java:99)
at
org.apache.hadoop.mapreduce.filecache.ClientDistributedCacheManager.determineTimestampsAndCacheVisibilities(ClientDistributedCacheManager.java:57)
at
org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:265)
at
org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:301)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:389)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1285)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1282)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:422)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1614)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1282)
at
org.apache.kylin.engine.mr.common.AbstractHadoopJob.waitForCompletion(AbstractHadoopJob.java:149)
at
org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:108)
at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:92)
at
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:120)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:57)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:136)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)

result code:2

--
View this message in context: 
http://apache-kylin.74782.x6.nabble.com/Trouble-with-building-cube-at-step-3-cry-cry-tp7172.html
Sent from the Apache Kylin mailing list archive at Nabble.com.

Re: Kylie Cube Building,Merging and Refreshing getting stucked at Create HFile Step

[ShaoFeng Shi]

hi ketan, which step do you get stuck on, "Create HTable" or "Convert to
HFile"? What's the output in log? If it is "Convert to HFile" step, please
also check the mapreduce job's output.

2017-02-14 17:20 GMT+08:00 ketan dikshit :

> Hi Guys,
> We have a cube which is getting stucked for all type of jobs:
> Build,Refresh and Merge on the “Create Hfile” step.
>
> Any idea, how can I get back to normalcy.
> We are using Kylin 1.5.4.1 and the cube is being built incrementally on an
> hourly basis.
>
>
> Thanks,
> kdcool6932




-- 
Best regards,

Shaofeng Shi 史少锋

Re: Problem while build cube：File not found for hdfs://...

[bertram]

i agree with you, maybe you need a hadoop expert

--
View this message in context: 
http://apache-kylin.74782.x6.nabble.com/Problem-while-build-cube-File-not-found-for-hdfs-tp7139p7217.html
Sent from the Apache Kylin mailing list archive at Nabble.com.

Re: Kylin with Node JS

[rahulsingh]

i am facing this error when i am trying to connect kylin to nodejs.


{ [Error: write EPROTO 140375442388864:error:140770FC:SSL
routines:SSL23_GET_SERVER_HELLO:unknown
protocol:../deps/openssl/openssl/ssl/s23_clnt.c:794:
] code: 'EPROTO', errno: 'EPROTO', syscall: 'write' }

can anyone help.

Thank You.

--
View this message in context: 
http://apache-kylin.74782.x6.nabble.com/Kylin-with-Node-JS-tp7180p7216.html
Sent from the Apache Kylin mailing list archive at Nabble.com.

Re: Problem while build cube：File not found for hdfs://...

[bertram]

i have build sample succeed when use another hadoop cluster 

--
View this message in context: 
http://apache-kylin.74782.x6.nabble.com/Problem-while-build-cube-File-not-found-for-hdfs-tp7139p7214.html
Sent from the Apache Kylin mailing list archive at Nabble.com.