[jira] [Commented] (KYLIN-3607) can't build cube with spark in v2.5.0

2019-01-09 Thread ASF subversion and git services (JIRA) 


[ 
https://issues.apache.org/jira/browse/KYLIN-3607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16738870#comment-16738870
 ] 

ASF subversion and git services commented on KYLIN-3607:


Commit 8087b4b691257b5e859ddf80d1987943fa354f9b in kylin's branch 
refs/heads/document from Shao Feng Shi
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=8087b4b ]

Update cube_spark document with KYLIN-3607


> can't build cube with spark in v2.5.0
> -
>
> Key: KYLIN-3607
> URL: https://issues.apache.org/jira/browse/KYLIN-3607
> Project: Kylin
>  Issue Type: Bug
>  Components: Storage - HBase
>Affects Versions: v2.5.0
>Reporter: anil kumar
>Assignee: Lijun Cao
>Priority: Major
> Fix For: v2.6.0
>
>
> in Kylin v2.5.0, can't be built cube at step 8 Convert Cuboid Data to HFile, 
> the following is the related exception:
>  
> ERROR yarn.ApplicationMaster: User class threw exception: 
> java.lang.RuntimeException: error execute 
> org.apache.kylin.storage.hbase.steps.SparkCubeHFile. Root cause: Job aborted 
> due to stage failure: Task 0 in stage 1.0 failed 4 times, 
> java.lang.ExceptionInInitializerError
>  at 
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.getNewWriter(HFileOutputFormat2.java:247)
>  at 
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:194)
>  at 
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:152)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply$mcV$sp(PairRDDFunctions.scala:1125)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123)
>  at 
> org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1353)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1131)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1102)
>  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
>  at org.apache.spark.scheduler.Task.run(Task.scala:99)
>  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325)
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Could not create interface 
> org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactory Is the 
> hadoop compatibility jar on the classpath?
>  at 
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:73)
>  at org.apache.hadoop.hbase.io.MetricsIO.(MetricsIO.java:31)
>  at org.apache.hadoop.hbase.io.hfile.HFile.(HFile.java:192)
>  ... 15 more
> Caused by: java.util.NoSuchElementException
>  at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:365)
>  at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
>  at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
>  at 
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:59)
>  ... 17 more



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3607) can't build cube with spark in v2.5.0

2018-12-16 Thread ASF subversion and git services (JIRA) 


[ 
https://issues.apache.org/jira/browse/KYLIN-3607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16722633#comment-16722633
 ] 

ASF subversion and git services commented on KYLIN-3607:


Commit 4b65975a2c007bd7b87f0b4b90e9136d5070fb7b in kylin's branch 
refs/heads/2.5.x from [~caolijun1166]
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=4b65975 ]

KYLIN-3607 add hbase-hadoop*-compat*.jar


> can't build cube with spark in v2.5.0
> -
>
> Key: KYLIN-3607
> URL: https://issues.apache.org/jira/browse/KYLIN-3607
> Project: Kylin
>  Issue Type: Bug
>  Components: Storage - HBase
>Affects Versions: v2.5.0
>Reporter: ANIL KUMAR
>Assignee: Lijun Cao
>Priority: Major
> Fix For: v2.6.0
>
>
> in Kylin v2.5.0, can't be built cube at step 8 Convert Cuboid Data to HFile, 
> the following is the related exception:
>  
> ERROR yarn.ApplicationMaster: User class threw exception: 
> java.lang.RuntimeException: error execute 
> org.apache.kylin.storage.hbase.steps.SparkCubeHFile. Root cause: Job aborted 
> due to stage failure: Task 0 in stage 1.0 failed 4 times, 
> java.lang.ExceptionInInitializerError
>  at 
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.getNewWriter(HFileOutputFormat2.java:247)
>  at 
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:194)
>  at 
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:152)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply$mcV$sp(PairRDDFunctions.scala:1125)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123)
>  at 
> org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1353)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1131)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1102)
>  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
>  at org.apache.spark.scheduler.Task.run(Task.scala:99)
>  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325)
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Could not create interface 
> org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactory Is the 
> hadoop compatibility jar on the classpath?
>  at 
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:73)
>  at org.apache.hadoop.hbase.io.MetricsIO.(MetricsIO.java:31)
>  at org.apache.hadoop.hbase.io.hfile.HFile.(HFile.java:192)
>  ... 15 more
> Caused by: java.util.NoSuchElementException
>  at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:365)
>  at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
>  at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
>  at 
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:59)
>  ... 17 more



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3607) can't build cube with spark in v2.5.0

2018-12-15 Thread ASF subversion and git services (JIRA) 


[ 
https://issues.apache.org/jira/browse/KYLIN-3607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16722407#comment-16722407
 ] 

ASF subversion and git services commented on KYLIN-3607:


Commit dd0943a74bdba838a21342a21f32e8be0012fa13 in kylin's branch 
refs/heads/master from [~caolijun1166]
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=dd0943a ]

KYLIN-3607 add hbase-hadoop*-compat*.jar


> can't build cube with spark in v2.5.0
> -
>
> Key: KYLIN-3607
> URL: https://issues.apache.org/jira/browse/KYLIN-3607
> Project: Kylin
>  Issue Type: Bug
>  Components: Storage - HBase
>Affects Versions: v2.5.0
>Reporter: ANIL KUMAR
>Assignee: Lijun Cao
>Priority: Major
> Fix For: v2.6.0
>
>
> in Kylin v2.5.0, can't be built cube at step 8 Convert Cuboid Data to HFile, 
> the following is the related exception:
>  
> ERROR yarn.ApplicationMaster: User class threw exception: 
> java.lang.RuntimeException: error execute 
> org.apache.kylin.storage.hbase.steps.SparkCubeHFile. Root cause: Job aborted 
> due to stage failure: Task 0 in stage 1.0 failed 4 times, 
> java.lang.ExceptionInInitializerError
>  at 
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.getNewWriter(HFileOutputFormat2.java:247)
>  at 
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:194)
>  at 
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:152)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply$mcV$sp(PairRDDFunctions.scala:1125)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123)
>  at 
> org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1353)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1131)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1102)
>  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
>  at org.apache.spark.scheduler.Task.run(Task.scala:99)
>  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325)
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Could not create interface 
> org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactory Is the 
> hadoop compatibility jar on the classpath?
>  at 
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:73)
>  at org.apache.hadoop.hbase.io.MetricsIO.(MetricsIO.java:31)
>  at org.apache.hadoop.hbase.io.hfile.HFile.(HFile.java:192)
>  ... 15 more
> Caused by: java.util.NoSuchElementException
>  at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:365)
>  at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
>  at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
>  at 
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:59)
>  ... 17 more



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3607) can't build cube with spark in v2.5.0

2018-12-15 Thread ASF GitHub Bot (JIRA) 


[ 
https://issues.apache.org/jira/browse/KYLIN-3607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16722406#comment-16722406
 ] 

ASF GitHub Bot commented on KYLIN-3607:
---

shaofengshi closed pull request #395: KYLIN-3607 add hbase-hadoop*-compat*.jar
URL: https://github.com/apache/kylin/pull/395
 
 
   

This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:

As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):

diff --git 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseSparkSteps.java
 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseSparkSteps.java
index ccab22f878..86ad0fbe5d 100644
--- 
a/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseSparkSteps.java
+++ 
b/storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/HBaseSparkSteps.java
@@ -66,6 +66,11 @@ public AbstractExecutable 
createConvertCuboidToHfileStep(String jobId) {
 StringUtil.appendWithSeparator(jars, 
ClassUtil.findContainingJar("org.apache.htrace.Trace", null)); // 
htrace-core.jar
 StringUtil.appendWithSeparator(jars,
 
ClassUtil.findContainingJar("com.yammer.metrics.core.MetricsRegistry", null)); 
// metrics-core.jar
+//KYLIN-3607
+StringUtil.appendWithSeparator(jars,
+
ClassUtil.findContainingJar("org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactory",
 null));//hbase-hadoop-compat-1.1.1.jar
+StringUtil.appendWithSeparator(jars,
+
ClassUtil.findContainingJar("org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactoryImpl",
 null));//hbase-hadoop2-compat-1.1.1.jar
 
 StringUtil.appendWithSeparator(jars, 
seg.getConfig().getSparkAdditionalJars());
 sparkExecutable.setJars(jars.toString());


 


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> can't build cube with spark in v2.5.0
> -
>
> Key: KYLIN-3607
> URL: https://issues.apache.org/jira/browse/KYLIN-3607
> Project: Kylin
>  Issue Type: Bug
>  Components: Storage - HBase
>Affects Versions: v2.5.0
>Reporter: ANIL KUMAR
>Assignee: Lijun Cao
>Priority: Major
> Fix For: v2.6.0
>
>
> in Kylin v2.5.0, can't be built cube at step 8 Convert Cuboid Data to HFile, 
> the following is the related exception:
>  
> ERROR yarn.ApplicationMaster: User class threw exception: 
> java.lang.RuntimeException: error execute 
> org.apache.kylin.storage.hbase.steps.SparkCubeHFile. Root cause: Job aborted 
> due to stage failure: Task 0 in stage 1.0 failed 4 times, 
> java.lang.ExceptionInInitializerError
>  at 
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.getNewWriter(HFileOutputFormat2.java:247)
>  at 
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:194)
>  at 
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:152)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply$mcV$sp(PairRDDFunctions.scala:1125)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123)
>  at 
> org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1353)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1131)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1102)
>  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
>  at org.apache.spark.scheduler.Task.run(Task.scala:99)
>  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325)
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Could not create interface 
> org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactory Is the 
> hadoop compatibility jar on the classpath?
>  at 
>

[jira] [Commented] (KYLIN-3607) can't build cube with spark in v2.5.0

2018-12-15 Thread ASF GitHub Bot (JIRA) 


[ 
https://issues.apache.org/jira/browse/KYLIN-3607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16722355#comment-16722355
 ] 

ASF GitHub Bot commented on KYLIN-3607:
---

caolijun1166 opened a new pull request #395: KYLIN-3607 add 
hbase-hadoop*-compat*.jar
URL: https://github.com/apache/kylin/pull/395
 
 
   


This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org


> can't build cube with spark in v2.5.0
> -
>
> Key: KYLIN-3607
> URL: https://issues.apache.org/jira/browse/KYLIN-3607
> Project: Kylin
>  Issue Type: Bug
>  Components: Storage - HBase
>Affects Versions: v2.5.0
>Reporter: ANIL KUMAR
>Assignee: Lijun Cao
>Priority: Major
> Fix For: v2.6.0
>
>
> in Kylin v2.5.0, can't be built cube at step 8 Convert Cuboid Data to HFile, 
> the following is the related exception:
>  
> ERROR yarn.ApplicationMaster: User class threw exception: 
> java.lang.RuntimeException: error execute 
> org.apache.kylin.storage.hbase.steps.SparkCubeHFile. Root cause: Job aborted 
> due to stage failure: Task 0 in stage 1.0 failed 4 times, 
> java.lang.ExceptionInInitializerError
>  at 
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.getNewWriter(HFileOutputFormat2.java:247)
>  at 
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:194)
>  at 
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:152)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply$mcV$sp(PairRDDFunctions.scala:1125)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123)
>  at 
> org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1353)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1131)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1102)
>  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
>  at org.apache.spark.scheduler.Task.run(Task.scala:99)
>  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325)
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Could not create interface 
> org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactory Is the 
> hadoop compatibility jar on the classpath?
>  at 
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:73)
>  at org.apache.hadoop.hbase.io.MetricsIO.(MetricsIO.java:31)
>  at org.apache.hadoop.hbase.io.hfile.HFile.(HFile.java:192)
>  ... 15 more
> Caused by: java.util.NoSuchElementException
>  at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:365)
>  at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
>  at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
>  at 
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:59)
>  ... 17 more



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3607) can't build cube with spark in v2.5.0

2018-12-10 Thread Shaofeng SHI (JIRA) 


[ 
https://issues.apache.org/jira/browse/KYLIN-3607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16714403#comment-16714403
 ] 

Shaofeng SHI commented on KYLIN-3607:
-

[~caolijun1166] Excellent; Could you please create a PR for this? Kylin can 
automatically append the jars to the spark applications "--jars" parameter.

> can't build cube with spark in v2.5.0
> -
>
> Key: KYLIN-3607
> URL: https://issues.apache.org/jira/browse/KYLIN-3607
> Project: Kylin
>  Issue Type: Bug
>Reporter: ANIL KUMAR
>Assignee: Lijun Cao
>Priority: Major
>
> in Kylin v2.5.0, can't be built cube at step 8 Convert Cuboid Data to HFile, 
> the following is the related exception:
>  
> ERROR yarn.ApplicationMaster: User class threw exception: 
> java.lang.RuntimeException: error execute 
> org.apache.kylin.storage.hbase.steps.SparkCubeHFile. Root cause: Job aborted 
> due to stage failure: Task 0 in stage 1.0 failed 4 times, 
> java.lang.ExceptionInInitializerError
>  at 
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.getNewWriter(HFileOutputFormat2.java:247)
>  at 
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:194)
>  at 
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:152)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply$mcV$sp(PairRDDFunctions.scala:1125)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123)
>  at 
> org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1353)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1131)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1102)
>  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
>  at org.apache.spark.scheduler.Task.run(Task.scala:99)
>  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325)
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Could not create interface 
> org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactory Is the 
> hadoop compatibility jar on the classpath?
>  at 
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:73)
>  at org.apache.hadoop.hbase.io.MetricsIO.(MetricsIO.java:31)
>  at org.apache.hadoop.hbase.io.hfile.HFile.(HFile.java:192)
>  ... 15 more
> Caused by: java.util.NoSuchElementException
>  at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:365)
>  at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
>  at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
>  at 
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:59)
>  ... 17 more



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3607) can't build cube with spark in v2.5.0

2018-12-09 Thread Lijun Cao (JIRA) 


[ 
https://issues.apache.org/jira/browse/KYLIN-3607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16713977#comment-16713977
 ] 

Lijun Cao commented on KYLIN-3607:
--

The problem is caused by the hbase jar is not in the classpath according to 
http://mail-archives.apache.org/mod_mbox/hbase-user/201504.mbox/%3c36b43d99e0f285478e40cdfab7896507bac...@pacdcexmb22.cable.comcast.com%3E.

After I added hbase-hadoop2-compat-*.jar and hbase-hadoop-compat-*.jar into 
$KYLIN_HOME/spark/jars, then it worked.

> can't build cube with spark in v2.5.0
> -
>
> Key: KYLIN-3607
> URL: https://issues.apache.org/jira/browse/KYLIN-3607
> Project: Kylin
>  Issue Type: Bug
>Reporter: ANIL KUMAR
>Assignee: Lijun Cao
>Priority: Major
>
> in Kylin v2.5.0, can't be built cube at step 8 Convert Cuboid Data to HFile, 
> the following is the related exception:
>  
> ERROR yarn.ApplicationMaster: User class threw exception: 
> java.lang.RuntimeException: error execute 
> org.apache.kylin.storage.hbase.steps.SparkCubeHFile. Root cause: Job aborted 
> due to stage failure: Task 0 in stage 1.0 failed 4 times, 
> java.lang.ExceptionInInitializerError
>  at 
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.getNewWriter(HFileOutputFormat2.java:247)
>  at 
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:194)
>  at 
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:152)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply$mcV$sp(PairRDDFunctions.scala:1125)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123)
>  at 
> org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1353)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1131)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1102)
>  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
>  at org.apache.spark.scheduler.Task.run(Task.scala:99)
>  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325)
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Could not create interface 
> org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactory Is the 
> hadoop compatibility jar on the classpath?
>  at 
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:73)
>  at org.apache.hadoop.hbase.io.MetricsIO.(MetricsIO.java:31)
>  at org.apache.hadoop.hbase.io.hfile.HFile.(HFile.java:192)
>  ... 15 more
> Caused by: java.util.NoSuchElementException
>  at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:365)
>  at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
>  at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
>  at 
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:59)
>  ... 17 more



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3607) can't build cube with spark in v2.5.0

2018-12-06 Thread Lijun Cao (JIRA) 


[ 
https://issues.apache.org/jira/browse/KYLIN-3607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16712442#comment-16712442
 ] 

Lijun Cao commented on KYLIN-3607:
--

I also meet the same problem on HDInsight (ClusterType: HBase 1.1.2(HDI 3.6))

18/12/07 07:16:05 INFO scheduler.DAGScheduler: ShuffleMapStage 0 (flatMapToPair 
at SparkCubeHFile.java:206) finished in 4.199 s
18/12/07 07:16:05 INFO scheduler.DAGScheduler: looking for newly runnable stages
18/12/07 07:16:05 INFO scheduler.DAGScheduler: running: Set()
18/12/07 07:16:05 INFO scheduler.DAGScheduler: waiting: Set(ResultStage 1)
18/12/07 07:16:05 INFO scheduler.DAGScheduler: failed: Set()
18/12/07 07:16:05 INFO scheduler.DAGScheduler: Submitting ResultStage 1 
(MapPartitionsRDD[3] at mapToPair at SparkCubeHFile.java:229), which has no 
missing parents
18/12/07 07:16:05 INFO memory.MemoryStore: Block broadcast_2 stored as values 
in memory (estimated size 230.5 KB, free 246.2 MB)
18/12/07 07:16:05 INFO memory.MemoryStore: Block broadcast_2_piece0 stored as 
bytes in memory (estimated size 53.7 KB, free 246.2 MB)
18/12/07 07:16:05 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in 
memory on 10.0.0.9:35365 (size: 53.7 KB, free: 246.8 MB)
18/12/07 07:16:05 INFO spark.SparkContext: Created broadcast 2 from broadcast 
at DAGScheduler.scala:996
18/12/07 07:16:05 INFO scheduler.DAGScheduler: Submitting 1 missing tasks from 
ResultStage 1 (MapPartitionsRDD[3] at mapToPair at SparkCubeHFile.java:229)
18/12/07 07:16:05 INFO cluster.YarnClusterScheduler: Adding task set 1.0 with 1 
tasks
18/12/07 07:16:05 INFO scheduler.TaskSetManager: Starting task 0.0 in stage 1.0 
(TID 11, wn2-script.sgyy1peopn1uhmeaiuqkxk20sg.bx.internal.chinacloudapp.cn, 
executor 2, partition 0, NODE_LOCAL, 5784 bytes)
18/12/07 07:16:05 INFO storage.BlockManagerInfo: Added broadcast_2_piece0 in 
memory on 
wn2-script.sgyy1peopn1uhmeaiuqkxk20sg.bx.internal.chinacloudapp.cn:41591 (size: 
53.7 KB, free: 229.7 MB)
18/12/07 07:16:05 INFO spark.MapOutputTrackerMasterEndpoint: Asked to send map 
output locations for shuffle 0 to 10.0.0.6:42074
18/12/07 07:16:05 INFO spark.MapOutputTrackerMaster: Size of output statuses 
for shuffle 0 is 283 bytes
18/12/07 07:16:06 WARN scheduler.TaskSetManager: Lost task 0.0 in stage 1.0 
(TID 11, wn2-script.sgyy1peopn1uhmeaiuqkxk20sg.bx.internal.chinacloudapp.cn, 
executor 2): java.lang.ExceptionInInitializerError
at 
org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.getNewWriter(HFileOutputFormat2.java:247)
at 
org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:194)
at 
org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:152)
at 
org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply$mcV$sp(PairRDDFunctions.scala:1125)
at 
org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123)
at 
org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123)
at 
org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1353)
at 
org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1131)
at 
org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1102)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
at org.apache.spark.scheduler.Task.run(Task.scala:99)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
Caused by: java.lang.RuntimeException: Could not create  interface 
org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactory Is the 
hadoop compatibility jar on the classpath?
at 
org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:73)
at org.apache.hadoop.hbase.io.MetricsIO.(MetricsIO.java:31)
at org.apache.hadoop.hbase.io.hfile.HFile.(HFile.java:192)
... 15 more
Caused by: java.util.NoSuchElementException
at 
java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:365)
at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
at 
org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:59)
... 17 more

18/12/07 07:16:06 INFO scheduler.TaskSetManager: Starting

[jira] [Commented] (KYLIN-3607) can't build cube with spark in v2.5.0

2018-12-05 Thread jon shoberg (JIRA) 


[ 
https://issues.apache.org/jira/browse/KYLIN-3607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16710462#comment-16710462
 ] 

jon shoberg commented on KYLIN-3607:


I'm getting the same issue on Kylin-2.5.2 with other services deployed from TAR 
(non-HDP)

HBase 1.4.8 is the version currently being used and made sure there are no 
other HBase versioned jars which would conflict this.


{code:java}
export HADOOP_CONF_DIR=/opt/kylin/hadoop-conf && /opt/spark/bin/spark-submit 
--class org.apache.kylin.common.util.SparkEntry  --conf 
spark.executor.instances=40  --conf spark.yarn.queue=default  --conf 
spark.history.fs.logDirectory=hdfs:///kylin/spark-history  --conf 
spark.master=yarn  --conf spark.hadoop.yarn.timeline-service.enabled=false  
--conf spark.executor.memory=4G  --conf spark.eventLog.enabled=true  --conf 
spark.eventLog.dir=hdfs:///kylin/spark-history  --conf 
spark.yarn.executor.memoryOverhead=1024  --conf spark.driver.memory=2G  --conf 
spark.shuffle.service.enabled=true --jars 
/opt/hbase/lib/hbase-common-1.4.8.jar,/opt/hbase/lib/hbase-server-1.4.8.jar,/opt/hbase/lib/hbase-client-1.4.8.jar,/opt/hbase/lib/hbase-protocol-1.4.8.jar,/opt/hbase/lib/hbase-hadoop-compat-1.4.8.jar,/opt/hbase/lib/htrace-core-3.1.0-incubating.jar,/opt/hbase/lib/metrics-core-2.2.0.jar,
 /opt/kylin/lib/kylin-job-2.5.2.jar -className 
org.apache.kylin.storage.hbase.steps.SparkCubeHFile -partitions 
hdfs://192.168.1.20:9000/kylin/kylin_metadata/kylin-26d69d94-07e2-d3b2-6898-1f96ea65bc50/HoldingNodeCube/rowkey_stats/part-r-0_hfile
 -counterOutput 
hdfs://192.168.1.20:9000/kylin/kylin_metadata/kylin-26d69d94-07e2-d3b2-6898-1f96ea65bc50/HoldingNodeCube/counter
 -cubename HoldingNodeCube -output 
hdfs://192.168.1.20:9000/kylin/kylin_metadata/kylin-26d69d94-07e2-d3b2-6898-1f96ea65bc50/HoldingNodeCube/hfile
 -input 
hdfs://192.168.1.20:9000/kylin/kylin_metadata/kylin-26d69d94-07e2-d3b2-6898-1f96ea65bc50/HoldingNodeCube/cuboid/
 -segmentId 37ef5ffa-5894-980f-4e20-33ec301e6ecf -metaUrl 
kylin_metadata@hdfs,path=hdfs://192.168.1.20:9000/kylin/kylin_metadata/kylin-26d69d94-07e2-d3b2-6898-1f96ea65bc50/HoldingNodeCube/metadata
 -hbaseConfPath 
hdfs://192.168.1.20:9000/kylin/kylin_metadata/kylin-26d69d94-07e2-d3b2-6898-1f96ea65bc50/hbase-conf.xml
{code}

> can't build cube with spark in v2.5.0
> -
>
> Key: KYLIN-3607
> URL: https://issues.apache.org/jira/browse/KYLIN-3607
> Project: Kylin
>  Issue Type: Bug
>Reporter: ANIL KUMAR
>Priority: Major
>
> in Kylin v2.5.0, can't be built cube at step 8 Convert Cuboid Data to HFile, 
> the following is the related exception:
>  
> ERROR yarn.ApplicationMaster: User class threw exception: 
> java.lang.RuntimeException: error execute 
> org.apache.kylin.storage.hbase.steps.SparkCubeHFile. Root cause: Job aborted 
> due to stage failure: Task 0 in stage 1.0 failed 4 times, 
> java.lang.ExceptionInInitializerError
>  at 
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.getNewWriter(HFileOutputFormat2.java:247)
>  at 
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:194)
>  at 
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:152)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply$mcV$sp(PairRDDFunctions.scala:1125)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123)
>  at 
> org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1353)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1131)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1102)
>  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
>  at org.apache.spark.scheduler.Task.run(Task.scala:99)
>  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325)
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Could not create interface 
> org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactory Is the 
> hadoop compatibility jar on the classpath?
>  at 
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:73)
>  at org.apache.hadoop.hbase.io.MetricsIO.(MetricsIO.java:31)
>  at

[jira] [Commented] (KYLIN-3607) can't build cube with spark in v2.5.0

2018-12-03 Thread Shaofeng SHI (JIRA) 


[ 
https://issues.apache.org/jira/browse/KYLIN-3607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16708040#comment-16708040
 ] 

Shaofeng SHI commented on KYLIN-3607:
-

Interesting, I see your HBase version is 1.1.2, but in HBase source code, there 
is no "MetricsIO.java", where comes it? Can you double check there is no other 
version of HBase jar files on your cluster?

> can't build cube with spark in v2.5.0
> -
>
> Key: KYLIN-3607
> URL: https://issues.apache.org/jira/browse/KYLIN-3607
> Project: Kylin
>  Issue Type: Bug
>Reporter: ANIL KUMAR
>Priority: Major
>
> in Kylin v2.5.0, can't be built cube at step 8 Convert Cuboid Data to HFile, 
> the following is the related exception:
>  
> ERROR yarn.ApplicationMaster: User class threw exception: 
> java.lang.RuntimeException: error execute 
> org.apache.kylin.storage.hbase.steps.SparkCubeHFile. Root cause: Job aborted 
> due to stage failure: Task 0 in stage 1.0 failed 4 times, 
> java.lang.ExceptionInInitializerError
>  at 
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.getNewWriter(HFileOutputFormat2.java:247)
>  at 
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:194)
>  at 
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:152)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply$mcV$sp(PairRDDFunctions.scala:1125)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123)
>  at 
> org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1353)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1131)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1102)
>  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
>  at org.apache.spark.scheduler.Task.run(Task.scala:99)
>  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325)
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Could not create interface 
> org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactory Is the 
> hadoop compatibility jar on the classpath?
>  at 
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:73)
>  at org.apache.hadoop.hbase.io.MetricsIO.(MetricsIO.java:31)
>  at org.apache.hadoop.hbase.io.hfile.HFile.(HFile.java:192)
>  ... 15 more
> Caused by: java.util.NoSuchElementException
>  at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:365)
>  at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
>  at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
>  at 
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:59)
>  ... 17 more



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3607) can't build cube with spark in v2.5.0

2018-11-26 Thread Yiping Wei (JIRA) 


[ 
https://issues.apache.org/jira/browse/KYLIN-3607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16698748#comment-16698748
 ] 

Yiping Wei commented on KYLIN-3607:
---

I am facing the same problem on Kylin-2.5.1 with HDP-2.6.5.0-292.
I am sure the hadoop compatibility jar is on the classpath.This is the command 
in my build #7 step log:

{code:bash}
export HADOOP_CONF_DIR=/usr/hdp/2.6.5.0-292/hadoop/conf && 
/usr/hdp/current/spark2-client/bin/spark-submit --class 
org.apache.kylin.common.util.SparkEntry  --conf spark.executor.instances=3  
--conf spark.yarn.archive=hdfs://slave1:8020/kylin/spark/spark-libs.jar  --conf 
spark.yarn.queue=default  --conf 
spark.yarn.am.extraJavaOptions=-Dhdp.version=current  --conf 
spark.history.fs.logDirectory=hdfs:///kylin/spark-history  --conf 
spark.driver.extraJavaOptions=-Dhdp.version=current  --conf 
spark.io.compression.codec=org.apache.spark.io.SnappyCompressionCodec  --conf 
spark.master=yarn  --conf spark.executor.extraJavaOptions=-Dhdp.version=current 
 --conf spark.hadoop.yarn.timeline-service.enabled=false  --conf 
spark.executor.memory=2G  --conf spark.eventLog.enabled=false  --conf 
spark.eventLog.dir=hdfs:///kylin/spark-history  --conf 
spark.yarn.executor.memoryOverhead=1024  --conf spark.driver.memory=1G  --conf 
spark.submit.deployMode=client  --conf spark.shuffle.service.enabled=true 
--jars 
/usr/hdp/2.6.5.0-292/hbase/lib/hbase-common-1.1.2.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/hbase/lib/hbase-server-1.1.2.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/hbase/lib/hbase-client-1.1.2.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/hbase/lib/hbase-protocol-1.1.2.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/hbase/lib/hbase-hadoop-compat-1.1.2.2.6.5.0-292.jar,/usr/hdp/2.6.5.0-292/hbase/lib/htrace-core-3.1.0-incubating.jar,/usr/hdp/2.6.5.0-292/hbase/lib/metrics-core-2.2.0.jar,
 /home/kylin/apache-kylin-2.5.1-bin-hbase1x/lib/kylin-job-2.5.1.jar -className 
org.apache.kylin.storage.hbase.steps.SparkCubeHFile -partitions 
hdfs://slave1:8020/kylin/kylin_metadata/kylin-b44e4af9-1a47-f78d-6111-2c8ecce6ce99/INJECTION_MOLDING_COUNT/rowkey_stats/part-r-0_hfile
 -counterOutput 
hdfs://slave1:8020/kylin/kylin_metadata/kylin-b44e4af9-1a47-f78d-6111-2c8ecce6ce99/INJECTION_MOLDING_COUNT/counter
 -cubename INJECTION_MOLDING_COUNT -output 
hdfs://slave1:8020/kylin/kylin_metadata/kylin-b44e4af9-1a47-f78d-6111-2c8ecce6ce99/INJECTION_MOLDING_COUNT/hfile
 -input 
hdfs://slave1:8020/kylin/kylin_metadata/kylin-b44e4af9-1a47-f78d-6111-2c8ecce6ce99/INJECTION_MOLDING_COUNT/cuboid/
 -segmentId ab60256a-430d-d3c7-f2e3-98daed21717b -metaUrl 
kylin_metadata@hdfs,path=hdfs://slave1:8020/kylin/kylin_metadata/kylin-b44e4af9-1a47-f78d-6111-2c8ecce6ce99/INJECTION_MOLDING_COUNT/metadata
 -hbaseConfPath 
hdfs://slave1:8020/kylin/kylin_metadata/kylin-b44e4af9-1a47-f78d-6111-2c8ecce6ce99/hbase-conf.xml
{code}


> can't build cube with spark in v2.5.0
> -
>
> Key: KYLIN-3607
> URL: https://issues.apache.org/jira/browse/KYLIN-3607
> Project: Kylin
>  Issue Type: Bug
>Reporter: ANIL KUMAR
>Priority: Major
>
> in Kylin v2.5.0, can't be built cube at step 8 Convert Cuboid Data to HFile, 
> the following is the related exception:
>  
> ERROR yarn.ApplicationMaster: User class threw exception: 
> java.lang.RuntimeException: error execute 
> org.apache.kylin.storage.hbase.steps.SparkCubeHFile. Root cause: Job aborted 
> due to stage failure: Task 0 in stage 1.0 failed 4 times, 
> java.lang.ExceptionInInitializerError
>  at 
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.getNewWriter(HFileOutputFormat2.java:247)
>  at 
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:194)
>  at 
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:152)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply$mcV$sp(PairRDDFunctions.scala:1125)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123)
>  at 
> org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1353)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1131)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1102)
>  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
>  at org.apache.spark.scheduler.Task.run(Task.scala:99)
>  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325)
>  at 
>

[jira] [Commented] (KYLIN-3607) can't build cube with spark in v2.5.0

2018-10-11 Thread Shaofeng SHI (JIRA) 


[ 
https://issues.apache.org/jira/browse/KYLIN-3607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16647518#comment-16647518
 ] 

Shaofeng SHI commented on KYLIN-3607:
-

Please check this error "java.lang.RuntimeException: Could not create interface 
org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactory Is the 
hadoop compatibility jar on the classpath".

 

Besides, not sure whether you're using the Spark built in Kylin. If you have 
"SPARK_HOME" environment variable pointing to other installations, please make 
it point to KYLIN_HOME/spark.

> can't build cube with spark in v2.5.0
> -
>
> Key: KYLIN-3607
> URL: https://issues.apache.org/jira/browse/KYLIN-3607
> Project: Kylin
>  Issue Type: Bug
>Reporter: ANIL KUMAR
>Priority: Major
>
> in Kylin v2.5.0, can't be built cube at step 8 Convert Cuboid Data to HFile, 
> the following is the related exception:
>  
> ERROR yarn.ApplicationMaster: User class threw exception: 
> java.lang.RuntimeException: error execute 
> org.apache.kylin.storage.hbase.steps.SparkCubeHFile. Root cause: Job aborted 
> due to stage failure: Task 0 in stage 1.0 failed 4 times, 
> java.lang.ExceptionInInitializerError
>  at 
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.getNewWriter(HFileOutputFormat2.java:247)
>  at 
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:194)
>  at 
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:152)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply$mcV$sp(PairRDDFunctions.scala:1125)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123)
>  at 
> org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1353)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1131)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1102)
>  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
>  at org.apache.spark.scheduler.Task.run(Task.scala:99)
>  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325)
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Could not create interface 
> org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactory Is the 
> hadoop compatibility jar on the classpath?
>  at 
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:73)
>  at org.apache.hadoop.hbase.io.MetricsIO.(MetricsIO.java:31)
>  at org.apache.hadoop.hbase.io.hfile.HFile.(HFile.java:192)
>  ... 15 more
> Caused by: java.util.NoSuchElementException
>  at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:365)
>  at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
>  at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
>  at 
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:59)
>  ... 17 more



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3607) can't build cube with spark in v2.5.0

2018-10-01 Thread ANIL KUMAR (JIRA) 


[ 
https://issues.apache.org/jira/browse/KYLIN-3607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634875#comment-16634875
 ] 

ANIL KUMAR commented on KYLIN-3607:
---

Hadoop 2.6.4.25-1

HBase 1.1.2.2.6.4.25-1

> can't build cube with spark in v2.5.0
> -
>
> Key: KYLIN-3607
> URL: https://issues.apache.org/jira/browse/KYLIN-3607
> Project: Kylin
>  Issue Type: Bug
>Reporter: ANIL KUMAR
>Priority: Major
>
> in Kylin v2.5.0, can't be built cube at step 8 Convert Cuboid Data to HFile, 
> the following is the related exception:
>  
> ERROR yarn.ApplicationMaster: User class threw exception: 
> java.lang.RuntimeException: error execute 
> org.apache.kylin.storage.hbase.steps.SparkCubeHFile. Root cause: Job aborted 
> due to stage failure: Task 0 in stage 1.0 failed 4 times, 
> java.lang.ExceptionInInitializerError
>  at 
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.getNewWriter(HFileOutputFormat2.java:247)
>  at 
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:194)
>  at 
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:152)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply$mcV$sp(PairRDDFunctions.scala:1125)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123)
>  at 
> org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1353)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1131)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1102)
>  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
>  at org.apache.spark.scheduler.Task.run(Task.scala:99)
>  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325)
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Could not create interface 
> org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactory Is the 
> hadoop compatibility jar on the classpath?
>  at 
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:73)
>  at org.apache.hadoop.hbase.io.MetricsIO.(MetricsIO.java:31)
>  at org.apache.hadoop.hbase.io.hfile.HFile.(HFile.java:192)
>  ... 15 more
> Caused by: java.util.NoSuchElementException
>  at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:365)
>  at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
>  at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
>  at 
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:59)
>  ... 17 more



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Commented] (KYLIN-3607) can't build cube with spark in v2.5.0

2018-10-01 Thread Shaofeng SHI (JIRA) 


[ 
https://issues.apache.org/jira/browse/KYLIN-3607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16634872#comment-16634872
 ] 

Shaofeng SHI commented on KYLIN-3607:
-

What's the hadoop and hbase version?

> can't build cube with spark in v2.5.0
> -
>
> Key: KYLIN-3607
> URL: https://issues.apache.org/jira/browse/KYLIN-3607
> Project: Kylin
>  Issue Type: Bug
>Reporter: ANIL KUMAR
>Priority: Major
>
> in Kylin v2.5.0, can't be built cube at step 8 Convert Cuboid Data to HFile, 
> the following is the related exception:
>  
> ERROR yarn.ApplicationMaster: User class threw exception: 
> java.lang.RuntimeException: error execute 
> org.apache.kylin.storage.hbase.steps.SparkCubeHFile. Root cause: Job aborted 
> due to stage failure: Task 0 in stage 1.0 failed 4 times, 
> java.lang.ExceptionInInitializerError
>  at 
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.getNewWriter(HFileOutputFormat2.java:247)
>  at 
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:194)
>  at 
> org.apache.hadoop.hbase.mapreduce.HFileOutputFormat2$1.write(HFileOutputFormat2.java:152)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply$mcV$sp(PairRDDFunctions.scala:1125)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12$$anonfun$apply$4.apply(PairRDDFunctions.scala:1123)
>  at 
> org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1353)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1131)
>  at 
> org.apache.spark.rdd.PairRDDFunctions$$anonfun$saveAsNewAPIHadoopDataset$1$$anonfun$12.apply(PairRDDFunctions.scala:1102)
>  at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:87)
>  at org.apache.spark.scheduler.Task.run(Task.scala:99)
>  at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:325)
>  at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
>  at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
>  at java.lang.Thread.run(Thread.java:745)
> Caused by: java.lang.RuntimeException: Could not create interface 
> org.apache.hadoop.hbase.regionserver.MetricsRegionServerSourceFactory Is the 
> hadoop compatibility jar on the classpath?
>  at 
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:73)
>  at org.apache.hadoop.hbase.io.MetricsIO.(MetricsIO.java:31)
>  at org.apache.hadoop.hbase.io.hfile.HFile.(HFile.java:192)
>  ... 15 more
> Caused by: java.util.NoSuchElementException
>  at java.util.ServiceLoader$LazyIterator.nextService(ServiceLoader.java:365)
>  at java.util.ServiceLoader$LazyIterator.next(ServiceLoader.java:404)
>  at java.util.ServiceLoader$1.next(ServiceLoader.java:480)
>  at 
> org.apache.hadoop.hbase.CompatibilitySingletonFactory.getInstance(CompatibilitySingletonFactory.java:59)
>  ... 17 more



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)