Hi, I'm not able to build cube. I'm getting the below error. I have added the nameservices property as well. Attached is hdfs-site.xml. Please let me know what is the issue? Here are the logs
pool-7-thread-1]:[2015-05-26
18:13:58,159][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:171)]
- Saving resource /execute_output/6031ba67-e270-4295-9fa5-cfe3126de275-00
(Store kylin_metadata@hbase)
[pool-7-thread-1]:[2015-05-26
18:14:01,572][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:171)]
- Saving resource /execute_output/6031ba67-e270-4295-9fa5-cfe3126de275-00
(Store kylin_metadata@hbase)
[pool-7-thread-1]:[2015-05-26
18:14:01,854][INFO][org.apache.kylin.job.manager.ExecutableManager.updateJobOutput(ExecutableManager.java:222)]
- job id:6031ba67-e270-4295-9fa5-cfe3126de275-00 from RUNNING to SUCCEED
[pool-7-thread-1]:[2015-05-26
18:14:06,601][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:171)]
- Saving resource /execute_output/6031ba67-e270-4295-9fa5-cfe3126de275 (Store
kylin_metadata@hbase)
[pool-7-thread-1]:[2015-05-26
18:14:26,176][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:171)]
- Saving resource /execute_output/6031ba67-e270-4295-9fa5-cfe3126de275 (Store
kylin_metadata@hbase)
[pool-7-thread-1]:[2015-05-26
18:14:26,442][INFO][org.apache.kylin.job.manager.ExecutableManager.updateJobOutput(ExecutableManager.java:222)]
- job id:6031ba67-e270-4295-9fa5-cfe3126de275 from RUNNING to READY
[pool-6-thread-1]:[2015-05-26
18:14:43,478][INFO][org.apache.kylin.job.impl.threadpool.DefaultScheduler$FetcherRunner.run(DefaultScheduler.java:107)]
- CubingJob{id=6031ba67-e270-4295-9fa5-cfe3126de275, name=new_star -
19700101000000_2922789940817071255 - BUILD - PDT 2015-05-26 05:39:23,
state=READY} prepare to schedule
[pool-6-thread-1]:[2015-05-26
18:14:43,478][INFO][org.apache.kylin.job.impl.threadpool.DefaultScheduler$FetcherRunner.run(DefaultScheduler.java:111)]
- CubingJob{id=6031ba67-e270-4295-9fa5-cfe3126de275, name=new_star -
19700101000000_2922789940817071255 - BUILD - PDT 2015-05-26 05:39:23,
state=READY} scheduled
[pool-7-thread-2]:[2015-05-26
18:14:46,633][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:171)]
- Saving resource /execute_output/6031ba67-e270-4295-9fa5-cfe3126de275 (Store
kylin_metadata@hbase)
[pool-7-thread-2]:[2015-05-26
18:14:46,899][INFO][org.apache.kylin.job.manager.ExecutableManager.updateJobOutput(ExecutableManager.java:222)]
- job id:6031ba67-e270-4295-9fa5-cfe3126de275 from READY to RUNNING
[pool-7-thread-2]:[2015-05-26
18:14:53,354][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:171)]
- Saving resource /execute_output/6031ba67-e270-4295-9fa5-cfe3126de275-01
(Store kylin_metadata@hbase)
[pool-7-thread-2]:[2015-05-26
18:14:53,620][INFO][org.apache.kylin.job.manager.ExecutableManager.updateJobOutput(ExecutableManager.java:222)]
- job id:6031ba67-e270-4295-9fa5-cfe3126de275-01 from READY to RUNNING
[pool-7-thread-2]:[2015-05-26
18:14:55,197][INFO][org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:107)]
- parameters of the MapReduceExecutable:
[pool-7-thread-2]:[2015-05-26
18:14:55,197][INFO][org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:108)]
- -conf
C:/Users/mobhatna/Kylin/server/../examples/test_case_data/sandbox/kylin_job_conf.xml
-cubename new_star -output
/tmp/kylin-6031ba67-e270-4295-9fa5-cfe3126de275/new_star/fact_distinct_columns
-jobname Kylin_Fact_Distinct_Columns_new_star_Step -tablename
kylin_intermediate_new_star_19700101000000_2922789940817071255_6031ba67_e270_4295_9fa5_cfe3126de275
[pool-7-thread-2]:[2015-05-26
18:14:55,245][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob.setJobClasspath(AbstractHadoopJob.java:149)]
- append job jar:
C:\Users\mobhatna\Kylin\lib\kylin-job-0.7.1-incubating-SNAPSHOT.jar
[pool-7-thread-2]:[2015-05-26
18:14:55,245][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob.setJobClasspath(AbstractHadoopJob.java:155)]
- append kylin.hive.dependency: null to mapreduce.application.classpath
[pool-7-thread-2]:[2015-05-26
18:14:55,245][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob.setJobClasspath(AbstractHadoopJob.java:170)]
- Hadoop job classpath is:
$HADOOP_CONF_DIR,/usr/hdp/current/hive-client/conf/,/usr/hdp/2.2.0.0-2041/hive/lib/hive-metastore.jar,/usr/hdp/2.2.0.0-2041/hive/lib/hive-exec.jar,/usr/hdp/2.2.0.0-2041/hive-hcatalog/share/hcatalog/*,$PWD/mr-framework/hadoop/share/hadoop/mapreduce/*,$PWD/mr-framework/hadoop/share/hadoop/mapreduce/lib/*,$PWD/mr-framework/hadoop/share/hadoop/common/*,$PWD/mr-framework/hadoop/share/hadoop/common/lib/*,$PWD/mr-framework/hadoop/share/hadoop/yarn/*,$PWD/mr-framework/hadoop/share/hadoop/yarn/lib/*,$PWD/mr-framework/hadoop/share/hadoop/hdfs/*,$PWD/mr-framework/hadoop/share/hadoop/hdfs/lib/*,/usr/hdp/2.2.0.0-2041/hadoop/lib/hadoop-lzo-0.6.0.2.2.0.0-2041.jar,/etc/hadoop/conf/secure
[pool-7-thread-2]:[2015-05-26
18:14:55,386][WARN][org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:2472)]
- HiveConf of name hive.optimize.mapjoin.mapreduce does not exist
[pool-7-thread-2]:[2015-05-26
18:14:55,386][WARN][org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:2472)]
- HiveConf of name hive.heapsize does not exist
[pool-7-thread-2]:[2015-05-26
18:14:55,386][WARN][org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:2472)]
- HiveConf of name hive.server2.enable.impersonation does not exist
[pool-7-thread-2]:[2015-05-26
18:14:55,386][WARN][org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:2472)]
- HiveConf of name hive.auto.convert.sortmerge.join.noconditionaltask does not
exist
[pool-7-thread-2]:[2015-05-26
18:14:55,432][INFO][org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:319)]
- Trying to connect to metastore with URI
thrift://sj1dra050.corp.adobe.com:9083
[pool-7-thread-2]:[2015-05-26
18:14:55,713][WARN][org.apache.hadoop.security.UserGroupInformation.getGroupNames(UserGroupInformation.java:1492)]
- No groups available for user mobhatna
[pool-7-thread-2]:[2015-05-26
18:14:55,978][INFO][org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:410)]
- Connected to metastore.
[pool-6-thread-1]:[2015-05-26
18:14:56,246][INFO][org.apache.kylin.job.impl.threadpool.DefaultScheduler$FetcherRunner.run(DefaultScheduler.java:117)]
- Job Fetcher: 0 running, 1 actual running, 1 ready, 12 others
[pool-7-thread-2]:[2015-05-26
18:14:56,621][WARN][org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:2472)]
- HiveConf of name hive.optimize.mapjoin.mapreduce does not exist
[pool-7-thread-2]:[2015-05-26
18:14:56,621][WARN][org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:2472)]
- HiveConf of name hive.heapsize does not exist
[pool-7-thread-2]:[2015-05-26
18:14:56,621][WARN][org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:2472)]
- HiveConf of name hive.server2.enable.impersonation does not exist
[pool-7-thread-2]:[2015-05-26
18:14:56,621][WARN][org.apache.hadoop.hive.conf.HiveConf.initialize(HiveConf.java:2472)]
- HiveConf of name hive.auto.convert.sortmerge.join.noconditionaltask does not
exist
[pool-7-thread-2]:[2015-05-26
18:14:57,245][INFO][org.apache.kylin.common.KylinConfig.createInstanceFromUri(KylinConfig.java:197)]
- The URI
C:\Users\mobhatna\AppData\Local\Temp\kylin_job_meta2280199537325480270\meta is
recognized as LOCAL_FOLDER
[pool-7-thread-2]:[2015-05-26
18:14:57,245][INFO][org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:67)]
- Using metadata url
C:\Users\mobhatna\AppData\Local\Temp\kylin_job_meta2280199537325480270\meta for
resource store
[pool-7-thread-2]:[2015-05-26
18:14:59,355][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:160)]
- Saving resource /cube/new_star.json (Store
C:\Users\mobhatna\AppData\Local\Temp\kylin_job_meta2280199537325480270\meta)
[pool-7-thread-2]:[2015-05-26
18:15:00,948][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:160)]
- Saving resource /model_desc/new_star.json (Store
C:\Users\mobhatna\AppData\Local\Temp\kylin_job_meta2280199537325480270\meta)
[pool-7-thread-2]:[2015-05-26
18:15:02,557][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:160)]
- Saving resource /cube_desc/new_star.json (Store
C:\Users\mobhatna\AppData\Local\Temp\kylin_job_meta2280199537325480270\meta)
[pool-7-thread-2]:[2015-05-26
18:15:04,734][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:160)]
- Saving resource /table/CC_STAR.SESSION.json (Store
C:\Users\mobhatna\AppData\Local\Temp\kylin_job_meta2280199537325480270\meta)
[pool-7-thread-2]:[2015-05-26
18:15:05,033][INFO][org.apache.hadoop.yarn.client.api.impl.TimelineClientImpl.serviceInit(TimelineClientImpl.java:285)]
- Timeline service address:
http://sj1dra051.corp.adobe.com:8188/ws/v1/timeline/
[pool-7-thread-2]:[2015-05-26
18:15:05,142][INFO][org.apache.hadoop.yarn.client.RMProxy.createRMProxy(RMProxy.java:98)]
- Connecting to ResourceManager at sj1dra051.corp.adobe.com/10.5.132.174:8021
[pool-7-thread-2]:[2015-05-26
18:15:30,638][INFO][org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:545)]
- Cleaning up the staging area /user/mobhatna/.staging/job_1431648296030_22357
[pool-7-thread-2]:[2015-05-26
18:15:30,904][ERROR][org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:83)]
- error in FactDistinctColumnsJob
java.lang.IllegalArgumentException: java.net.UnknownHostException: nameservice1
at
org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:374)
at
org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:312)
at
org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:178)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:665)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:601)
at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:148)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2596)
at
org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
at
org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
at
org.apache.hive.hcatalog.mapreduce.HCatBaseInputFormat.setInputPath(HCatBaseInputFormat.java:318)
at
org.apache.hive.hcatalog.mapreduce.HCatBaseInputFormat.getSplits(HCatBaseInputFormat.java:125)
at
org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:597)
at
org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:614)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:492)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293)
at
org.apache.kylin.job.hadoop.AbstractHadoopJob.waitForCompletion(AbstractHadoopJob.java:123)
at
org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:80)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at
org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:112)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.UnknownHostException: nameservice1
... 35 more
[pool-7-thread-2]:[2015-05-26
18:15:32,481][ERROR][org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:115)]
- error execute
MapReduceExecutable{id=6031ba67-e270-4295-9fa5-cfe3126de275-01, name=Extract
Fact Table Distinct Columns, state=RUNNING}
java.lang.IllegalArgumentException: java.net.UnknownHostException: nameservice1
at
org.apache.hadoop.security.SecurityUtil.buildTokenService(SecurityUtil.java:374)
at
org.apache.hadoop.hdfs.NameNodeProxies.createNonHAProxy(NameNodeProxies.java:312)
at
org.apache.hadoop.hdfs.NameNodeProxies.createProxy(NameNodeProxies.java:178)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:665)
at org.apache.hadoop.hdfs.DFSClient.<init>(DFSClient.java:601)
at
org.apache.hadoop.hdfs.DistributedFileSystem.initialize(DistributedFileSystem.java:148)
at
org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:2596)
at
org.apache.hadoop.fs.FileSystem.access$200(FileSystem.java:91)
at
org.apache.hadoop.fs.FileSystem$Cache.getInternal(FileSystem.java:2630)
at
org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:2612)
at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:370)
at org.apache.hadoop.fs.Path.getFileSystem(Path.java:296)
at
org.apache.hive.hcatalog.mapreduce.HCatBaseInputFormat.setInputPath(HCatBaseInputFormat.java:318)
at
org.apache.hive.hcatalog.mapreduce.HCatBaseInputFormat.getSplits(HCatBaseInputFormat.java:125)
at
org.apache.hadoop.mapreduce.JobSubmitter.writeNewSplits(JobSubmitter.java:597)
at
org.apache.hadoop.mapreduce.JobSubmitter.writeSplits(JobSubmitter.java:614)
at
org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:492)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1296)
at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1293)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at
org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
at org.apache.hadoop.mapreduce.Job.submit(Job.java:1293)
at
org.apache.kylin.job.hadoop.AbstractHadoopJob.waitForCompletion(AbstractHadoopJob.java:123)
at
org.apache.kylin.job.hadoop.cube.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:80)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
at
org.apache.kylin.job.common.MapReduceExecutable.doWork(MapReduceExecutable.java:112)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:50)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:107)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.UnknownHostException: nameservice1
... 35 more
[pool-7-thread-2]:[2015-05-26
18:15:34,077][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:171)]
- Saving resource /execute_output/6031ba67-e270-4295-9fa5-cfe3126de275-01
(Store kylin_metadata@hbase)
[pool-7-thread-2]:[2015-05-26
18:15:37,500][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:171)]
- Saving resource /execute_output/6031ba67-e270-4295-9fa5-cfe3126de275-01
(Store kylin_metadata@hbase)
[pool-7-thread-2]:[2015-05-26
18:15:37,766][INFO][org.apache.kylin.job.manager.ExecutableManager.updateJobOutput(ExecutableManager.java:222)]
- job id:6031ba67-e270-4295-9fa5-cfe3126de275-01 from RUNNING to ERROR
[pool-7-thread-2]:[2015-05-26
18:15:42,516][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:171)]
- Saving resource /execute_output/6031ba67-e270-4295-9fa5-cfe3126de275 (Store
kylin_metadata@hbase)
[pool-7-thread-2]:[2015-05-26
18:15:45,925][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:171)]
- Saving resource /execute_output/6031ba67-e270-4295-9fa5-cfe3126de275 (Store
kylin_metadata@hbase)
[pool-7-thread-2]:[2015-05-26
18:15:47,783][DEBUG][org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:171)]
- Saving resource /execute_output/6031ba67-e270-4295-9fa5-cfe3126de275 (Store
kylin_metadata@hbase)
[pool-7-thread-2]:[2015-05-26
18:15:48,049][INFO][org.apache.kylin.job.manager.ExecutableManager.updateJobOutput(ExecutableManager.java:222)]
- job id:6031ba67-e270-4295-9fa5-cfe3126de275 from RUNNING to ERROR
[pool-6-thread-1]:[2015-05-26
18:15:50,542][INFO][org.apache.kylin.job.impl.threadpool.DefaultScheduler$FetcherRunner.run(DefaultScheduler.java:117)]
- Job Fetcher: 1 running, 1 actual running, 0 ready, 12 others
[pool-7-thread-2]:[2015-05-26
18:15:57,503][ERROR][org.apache.kylin.job.execution.AbstractExecutable.notifyUserStatusChange(AbstractExecutable.java:221)]
-
java.lang.NullPointerException
at java.util.regex.Matcher.appendReplacement(Matcher.java:758)
at java.util.regex.Matcher.replaceAll(Matcher.java:906)
at java.lang.String.replaceAll(String.java:2162)
at
org.apache.kylin.job.cube.CubingJob.formatNotifications(CubingJob.java:92)
at
org.apache.kylin.job.execution.AbstractExecutable.notifyUserStatusChange(AbstractExecutable.java:211)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.onExecuteFinished(DefaultChainedExecutable.java:100)
at
org.apache.kylin.job.cube.CubingJob.onExecuteFinished(CubingJob.java:118)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:132)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)
Thanks,
Mohit
hdfs-site.xml
Description: hdfs-site.xml
