Re: Rest Api “Load Hive Tables ”Return 405

[Chao Long]

Hi Johnson,
   You should place those parameters in the url, like this:
http://localhost:7070/kylin/api/tables/kylin_table/kylin_project

On Mon, Sep 30, 2019 at 4:59 PM Johnson  wrote:

> hi，all
>
>-
>
>I call the API “Load Hive Tables” according to the official document
>and return 405
>
> result：
>
>
>
>

Re: Real-time OLAP - NPE in stream receiver when processing event

[Chao Long]

Hi Andras,
  Real-time does not support count-distinct with non-integer datatype
column yet, is there a count-distinct measure with non-integer datatype
column?

On Tue, Sep 3, 2019 at 10:38 PM Andras Nagy 
wrote:

> Dear All,
>
> We seem to have run into an issue with real-time streaming (in the
> stream-receiver). We get a NPE in the StreamingConsumerChannel thread when
> an event is received from Kafka:
>
> 2019-09-03 13:50:59,341 ERROR [object_movements_channel]
> consumer.StreamingConsumerChannel:119 : error happens when save
> event:StreamingMessage{data=[1567492920948,
> 502a3dcb-547f-44bd-9f2b-e1e929ce42b0, d56ecaed-d5dd-460f-9e91-ea3fadf6f369,
> e6a1760b-d023-4c92-aab5-e848067bca8c, 0f86f96e-2eb6-4271-b7a1-31a24a0b6e8e,
> , , 40, 30, 6, 6, 0, 3, 9, 2019, e16a689e-f893-47a6-ab74-9881010333ec,
> 2019-09-03, device, c1332a9b-0160-456a-8e43-0a2319efcaa1,
> F3DBDC26-B993-4470-9EB7-22E9961B2195, 11.143630707875111],
> sourcePosition=org.apache.kylin.stream.source.kafka.KafkaPosition$KafkaPartitionPosition@26b00da,
> timestamp=1567518120948, params={}}
> java.lang.NullPointerException
> at
> org.apache.kylin.measure.bitmap.BitmapMeasureType$1.valueOf(BitmapMeasureType.java:114)
> at
> org.apache.kylin.measure.bitmap.BitmapMeasureType$1.valueOf(BitmapMeasureType.java:98)
> at
> org.apache.kylin.stream.core.storage.columnar.SegmentMemoryStore.buildValueOf(SegmentMemoryStore.java:166)
> at
> org.apache.kylin.stream.core.storage.columnar.SegmentMemoryStore.buildValue(SegmentMemoryStore.java:138)
> at
> org.apache.kylin.stream.core.storage.columnar.SegmentMemoryStore.index(SegmentMemoryStore.java:101)
> at
> org.apache.kylin.stream.core.storage.columnar.ColumnarSegmentStore.addEvent(ColumnarSegmentStore.java:134)
> at
> org.apache.kylin.stream.core.storage.StreamingCubeSegment.addEvent(StreamingCubeSegment.java:152)
> at
> org.apache.kylin.stream.core.storage.StreamingSegmentManager.addEvent(StreamingSegmentManager.java:173)
> at
> org.apache.kylin.stream.core.consumer.StreamingConsumerChannel.run(StreamingConsumerChannel.java:102)
> at java.lang.Thread.run(Thread.java:748)
>
>
> This is on our snapshot build from master (unchanged) but as I see the
> https://github.com/apache/kylin/blob/master/core-metadata/src/main/java/org/apache/kylin/measure/bitmap/BitmapMeasureType.java
> class has last changed on 8th April so is still the same.
>
> Does anyone have an idea what can be the issue or how we could debug it
> further?
>
> Thank you,
> Andras
>

Re: Caused by: java.lang.NoClassDefFoundError: org/apache/spark/api/java/function/Function

[Chao Long]

这个参数配置了么，*kylin.engine.spark-conf.spark.yarn.archive，*
是否配置为自己集群所在的hdfs目录，而不是文档上的hdfs://sandbox.hortonworks.com:8020

On Wed, Aug 14, 2019 at 2:37 PM 邵志鹏  wrote:

> 2019-08-14T14:31:09,188 INFO [http-bio-7070-exec-7]
> org.apache.kylin.cube.CubeManager - Updating cube instance
> 'kylin_sales_cube_clone_spark'
> 2019-08-14T14:31:09,188 INFO [http-bio-7070-exec-7]
> org.apache.kylin.cube.CubeManager - Remove segment
> kylin_sales_cube_clone_spark[2012010100_20140201235500]
>
> Action - Delete Segement can Remove segment , and then rebuild it.
>
> Main problem is
> Caused by: java.lang.ClassNotFoundException:
> org.apache.spark.api.java.function.Function。
>
> 不确定是否是jar报冲突，spark下面所有的jar都一起打包放到hdfs了，里面有spark-core。
> spark版本：2.3.2
> spark-core_2.11-2.3.2
>
>
> 在 2019-08-14 14:08:03，"邵志鹏"  写道：
>
> Hi kylin team:
>
> kylin-2.6.3
> 使用MapReduce已经成功构建smaple cube。
> 然后clone了一份kylin_sales_cube修改为Spark，build 报错。
>
> Spark是使用脚本下载的，配置参考了
> http://kylin.apache.org/cn/docs/tutorial/cube_spark.html。
>
>
> $KYLIN_HOME/bin/download-spark.sh
> jar cv0f spark-libs.jar -C $KYLIN_HOME/spark/jars/ .
> hadoop fs -mkdir -p /kylin/spark/
> hadoop fs -put spark-libs.jar /kylin/spark/
> ...
>
> 报错日志：
> Caused by: java.lang.NoClassDefFoundError:
> org/apache/spark/api/java/function/Function
> at
> org.apache.kylin.engine.spark.SparkBatchCubingJobBuilder2.(SparkBatchCubingJobBuilder2.java:53)
> ~[kylin-engine-spark-2.6.3.jar:2.6.3]
> at
> org.apache.kylin.engine.spark.SparkBatchCubingEngine2.createBatchCubingJob(SparkBatchCubingEngine2.java:44)
> ~[kylin-engine-spark-2.6.3.jar:2.6.3]
> at
> org.apache.kylin.engine.EngineFactory.createBatchCubingJob(EngineFactory.java:60)
> ~[kylin-core-job-2.6.3.jar:2.6.3]
> at
> org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:234)
> ~[kylin-server-base-2.6.3.jar:2.6.3]
> at
> org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202)
> ~[kylin-server-base-2.6.3.jar:2.6.3]
> at
> org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:395)
> ~[kylin-server-base-2.6.3.jar:2.6.3]
> ... 77 more
> Caused by: java.lang.ClassNotFoundException:
> org.apache.spark.api.java.function.Function
> at
> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1928)
> ~[catalina.jar:7.0.91]
> at
> org.apache.catalina.loader.WebappClassLoaderBase.loadClass(WebappClassLoaderBase.java:1771)
> ~[catalina.jar:7.0.91]
> at
> org.apache.kylin.engine.spark.SparkBatchCubingJobBuilder2.(SparkBatchCubingJobBuilder2.java:53)
> ~[kylin-engine-spark-2.6.3.jar:2.6.3]
> at
> org.apache.kylin.engine.spark.SparkBatchCubingEngine2.createBatchCubingJob(SparkBatchCubingEngine2.java:44)
> ~[kylin-engine-spark-2.6.3.jar:2.6.3]
> at
> org.apache.kylin.engine.EngineFactory.createBatchCubingJob(EngineFactory.java:60)
> ~[kylin-core-job-2.6.3.jar:2.6.3]
> at
> org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:234)
> ~[kylin-server-base-2.6.3.jar:2.6.3]
> at
> org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202)
> ~[kylin-server-base-2.6.3.jar:2.6.3]
> at
> org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:395)
> ~[kylin-server-base-2.6.3.jar:2.6.3]
>
>
> 报错之后，再去提交build，又报新的错：
> Caused by: org.apache.kylin.rest.exception.BadRequestException: The cube
> kylin_sales_cube_clone_spark has segments
> [kylin_sales_cube_clone_spark[2012010100_20140201235500]], but none of
> them is READY. It's not allowed for parallel building
> at
> org.apache.kylin.rest.service.JobService.checkAllowParallelBuilding(JobService.java:422)
> ~[kylin-server-base-2.6.3.jar:2.6.3]
> at
> org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:221)
> ~[kylin-server-base-2.6.3.jar:2.6.3]
> at
> org.apache.kylin.rest.service.JobService.submitJob(JobService.java:202)
> ~[kylin-server-base-2.6.3.jar:2.6.3]
> at
> org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:395)
> ~[kylin-server-base-2.6.3.jar:2.6.3]
> ... 77 more
>
>
> Monitor里面是没有新的构建任务的。
>
> Thanks.
>
> By 邵志鹏
>
>
>
>
>
>
>
>
>

Re: about the differences from kylin 1.6 and 2.6

[Chao Long]

Hi,
   The cube may optimized by cube planner, which will reduce the amount of
cuboids.

On Fri, Jul 5, 2019 at 11:21 AM 邓道远  wrote:

> HI all,
>
> I am upgrading my kylin from 1.6 to 2.6,
>
> there is a cube, its size in 1.6 is about 300MB, but only 100MB in 2.6,
> however, its configurations and data counts are the same,
>
> so, I`m not sure if there`s something wrong in my cube, or its just the
> new features in higher version?
>
> Any suggestion is welcome,
>
> thanks very much!
>
>
>
>

Re: Problem with Cube

[Chao Long]

Hi Cinto Sunny,
   You can try to set "kylin.engine.mr.uhc-reducer-count" a bigger value,
default is 1.

On Fri, Jun 21, 2019 at 2:44 PM Cinto Sunny  wrote:

> Hi All,
>
> I am building a cube with 10 dimensions and two measures. The total input
> size is 100 GB.
> I am trying to build using Roaring BitMap. One of the fact is user and has
> ~1.8B userids.
>
> The build is getting stuck at stage - Extract Fact Table Distinct Columns.
> One executor is stuck and is processing over 800M lines.
>
> I am using version - 2.6.
>
> Any pointers would be appreciated. Let me know is any further information
> is required.
>
> - Cinto
>

Re: 分发中间表数据倾斜

[Chao Long]

Hi wang,
   "DISTRIBUTE BY RAND()" may cause data inconsistency, so we changed it to
distribute by the first few columns of  the rowkey, and the default is the
first 3 columns. You can see the following issues for more details. If you
don't have a data skew problem, you can just disable the "redistribute"
step by setting "kylin.source.hive.redistribute-flat-table" to false.
https://issues.apache.org/jira/browse/KYLIN-3388
https://issues.apache.org/jira/browse/KYLIN-3457

On Thu, Jun 13, 2019 at 5:03 PM ning.w...@ymm56.com 
wrote:

> 按照文档说法，重新分发中间表的时候是随机方式DISTRIBUTE BY
> RAND()，我的cube里没有指定分片字段，但是不是按照随机方式分发的，而是取的维度字段里的前3个字段，由于cube里的维度没有高基维度导致数据倾斜，
> 怎么设置才能随机分发呢，或者有什么好的建议
>
> --
> ning.w...@ymm56.com
>

Re: Extract Fact Table Distinct Columns - Not running on Yarn ???

[Chao Long]

Hi J,
Step "Extract Fact Table Distinct Columns"  should be run on yarn as a
mr job or a spark job, you can click this icon and it will link to the job
on yarn.
[image: image.png]

On Tue, Jun 11, 2019 at 6:59 AM Jon Shoberg  wrote:

> It appears "Extract Fact Table Distinct Columns" is running as a local
> job? Is this correct? (screen shot below)
>
> This is on a Kylin 2.6.2 cluster recently upgraded.
>
> My concern is that I cannot see the job running on Yarn and most of the
> cluster is idle while this step is taking a long time (longer than expected)
>
> Any guidance is appreciated!
>
> Thanks! J
>
>
>
> [image: image.png]
>

Re: Error on startup - Using Hortonworks Sandbox for evaluation

[Chao Long]

Hi Yves Arsenault,
   Where is the $SPARK_HOME?
   There is a similar issue caused by wrong $SPARK_HOME.
https://issues.apache.org/jira/browse/KYLIN-3872
   If you are in the same situation, you can use the spark directory under
$KYLIN_HOME as the $SPARK_HOME to resolve this problem.

On Sun, May 19, 2019 at 7:58 PM Na Zhai  wrote:

> Hi, Yves Arsenault.
>
>
>
> It seems that the environment lacks the commons-configuration-.jar
>
>
>
>
>
> 发送自 Windows 10 版邮件 应用
>
>
> --
> *发件人:* Yves Arsenault 
> *发送时间:* Saturday, May 18, 2019 1:55:28 AM
> *收件人:* user@kylin.apache.org
> *主题:* Error on startup - Using Hortonworks Sandbox for evaluation
>
>
> Hello,
>
>
>
> I’ve installed Kylin on the Hortonworks sandbox for some
> testing/evaluation.
>
>
>
> The Web UI doesn’t work as there an error happening with the
> springframework… been trying to find the solution for a little bit, I
> thought I would reach out to see if there is something else going on that
> I’m not seeing
>
>
>
> When  starting Kylin, I get this in the kylin.log :
>
> 2019-05-17 17:41:41,692 INFO  [main] util.log:192 : Logging initialized
> @3180ms
>
> 2019-05-17 17:41:43,051 WARN  [main] http11.Http11Protocol:137 : The HTTP
> BIO connector has been removed in Tomcat 8.5.x onwards. The HTTP BIO
> connector configuration has been automatically switched to use the HTTP NIO
> connector instead.
>
> 2019-05-17 17:41:43,293 INFO  [main] core.AprLifecycleListener:119 : The
> APR based Apache Tomcat Native library which allows optimal performance in
> production environments was not found on the java.library.path:
> [/usr/hdp/current/hadoop-client/lib/native]
>
> 2019-05-17 17:41:45,096 INFO  [main] http11.Http11NioProtocol:119 :
> Initializing ProtocolHandler ["http-nio-7070"]
>
> 2019-05-17 17:41:45,160 INFO  [main] net.NioSelectorPool:119 : Using a
> shared selector for servlet write/read
>
> 2019-05-17 17:41:45,194 INFO  [main] http11.Http11NioProtocol:119 :
> Initializing ProtocolHandler ["https-jsse-nio-7443"]
>
> 2019-05-17 17:41:45,723 INFO  [main] net.NioSelectorPool:119 : Using a
> shared selector for servlet write/read
>
> 2019-05-17 17:41:45,744 INFO  [main] ajp.AjpNioProtocol:119 : Initializing
> ProtocolHandler ["ajp-nio-9009"]
>
> 2019-05-17 17:41:45,767 INFO  [main] net.NioSelectorPool:119 : Using a
> shared selector for servlet write/read
>
> 2019-05-17 17:41:45,770 INFO  [main] startup.Catalina:119 : Initialization
> processed in 3909 ms
>
> 2019-05-17 17:41:45,962 INFO  [main] core.StandardService:119 : Starting
> service [Catalina]
>
> 2019-05-17 17:41:45,963 INFO  [main] core.StandardEngine:119 : Starting
> Servlet Engine: Apache Tomcat/8.5.33
>
> 2019-05-17 17:41:46,021 INFO  [localhost-startStop-1]
> startup.HostConfig:119 : Deploying web application archive
> [/usr/local/apache-kylin-2.6.1-bin-hadoop3/tomcat/webapps/kylin.war]
>
> 2019-05-17 17:41:46,117 WARN  [localhost-startStop-1]
> digester.Digester:137 : [SetContextPropertiesRule]{Context} Setting
> property 'allowLinking' to 'true' did not find a matching property.
>
> 2019-05-17 17:44:07,287 INFO  [localhost-startStop-1]
> servlet.TldScanner:119 : At least one JAR was scanned for TLDs yet
> contained no TLDs. Enable debug logging for this logger for a complete list
> of JARs that were scanned but no TLDs were found in them. Skipping unneeded
> JARs during scanning can improve startup time and JSP compilation time.
>
> 2019-05-17 17:44:11,455 INFO  [localhost-startStop-1]
> servlet.TldScanner:119 : At least one JAR was scanned for TLDs yet
> contained no TLDs. Enable debug logging for this logger for a complete list
> of JARs that were scanned but no TLDs were found in them. Skipping unneeded
> JARs during scanning can improve startup time and JSP compilation time.
>
> 2019-05-17 17:44:11,509 INFO  [localhost-startStop-1]
> [localhost].[/kylin]:119 : No Spring WebApplicationInitializer types
> detected on classpath
>
> 2019-05-17 17:44:11,614 INFO  [localhost-startStop-1]
> common.KylinConfig:101 : Loading kylin-defaults.properties from
> file:/usr/local/apache-kylin-2.6.1-bin-hadoop3/tomcat/webapps/kylin/WEB-INF/lib/kylin-core-common-2.6.1.jar!/kylin-defaults.properties
>
> 2019-05-17 17:44:11,662 DEBUG [localhost-startStop-1]
> common.KylinConfig:328 : KYLIN_CONF property was not set, will seek
> KYLIN_HOME env variable
>
> 2019-05-17 17:44:11,663 INFO  [localhost-startStop-1]
> common.KylinConfig:334 : Use
> KYLIN_HOME=/usr/local/apache-kylin-2.6.1-bin-hadoop3
>
> 2019-05-17 17:44:11,666 INFO  [localhost-startStop-1]
> common.KylinConfig:136 : Initialized a new KylinConfig from
> getInstanceFromEnv : 260179257
>
> 2019-05-17 17:44:11,741 INFO  [localhost-startStop-1]
> [localhost].[/kylin]:119 : Set web app root system property: 'webapp.root'
> = [/usr/local/apache-kylin-2.6.1-bin-hadoop3/tomcat/webapps/kylin/]
>
> 2019-05-17 17:44:11,779 INFO  [localhost-startStop-1]
> [localhost].[/kylin]:119 : 

Re: Re: why kylin job failed when use beeline with zookeeper

[Chao Long]

Hi lk,
   I had a test on using beeline to connect hive, and I can pass the
"Redistribute Flat Hive Table" step successfully.
   Bellow is my configuration:

>kylin.source.hive.client=beeline
>kylin.source.hive.beeline-shell=beeline
>kylin.source.hive.beeline-params=-n root --hiveconf
> hive.security.authorization.sqlstd.confwhitelist.append='mapreduce.job.*|dfs.*'
> -u 'jdbc:hive2://cdh1.cloudera.com:2181,cdh2.cloudera.com:2181,
> cdh3.cloudera.com:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
> '



On Thu, Apr 18, 2019 at 3:19 PM lk_hadoop  wrote:

> hi, Chao Long ,thanks for your reply, the first setp I can see logs about :
>
>
> EOL
> beeline -n hive -p hiveadmin --hiveconf 
> hive.security.authorization.sqlstd.confwhitelist.append='mapreduce.job.*|dfs.*'
>  -u 
> jdbc:hive2://"bdp-scm-04:2181,bdp-scm-03:2181,bdp-scm-05:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2"
>  --hiveconf hive.merge.mapredfiles=false --hiveconf 
> hive.auto.convert.join=true --hiveconf dfs.replication=2 --hiveconf 
> hive.exec.compress.output=true --hiveconf 
> hive.auto.convert.join.noconditionaltask=true --hiveconf 
> mapreduce.job.split.metainfo.maxsize=-1 --hiveconf hive.merge.mapfiles=false 
> --hiveconf hive.auto.convert.join.noconditionaltask.size=1 --hiveconf 
> hive.stats.autogather=true -f 
> /tmp/cfadac57-d586-446b-a798-96a9c37e34b2.hql;ret_code=$?;rm -f 
> /tmp/cfadac57-d586-446b-a798-96a9c37e34b2.hql;exit $ret_code
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in 
> [jar:file:/home/devuser/bdp/env/hbase-1.2.0-cdh5.14.0/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in 
> [jar:file:/home/devuser/bdp/env/hadoop-2.6.0-cdh5.14.0/share/hadoop/common/lib/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an 
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> scan complete in 2ms
> Connecting to 
> jdbc:hive2://bdp-scm-04:2181,bdp-scm-03:2181,bdp-scm-05:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2
> 19/04/18 10:54:45 [main]: INFO jdbc.HiveConnection: Connected to 
> bdp-scm-06:1
> Connected to: Apache Hive (version 1.1.0-cdh5.14.0)
> Driver: Hive JDBC (version 1.1.0-cdh5.14.0)
> Transaction isolation: TRANSACTION_REPEATABLE_READ
> 0: jdbc:hive2://bdp-scm-04:2181,bdp-scm-03:21> USE mykylin;
> INFO  : Compiling 
> command(queryId=hive_20190418105454_f4e97998-7295-4c33-925f-654624f67c6c): 
> USE mykylin
> INFO  : Semantic Analysis Completed
> INFO  : Returning Hive schema: Schema(fieldSchemas:null, properties:null)
> INFO  : Completed compiling 
> command(queryId=hive_20190418105454_f4e97998-7295-4c33-925f-654624f67c6c); 
> Time taken: 0.074 seconds
> INFO  : Executing 
> command(queryId=hive_20190418105454_f4e97998-7295-4c33-925f-654624f67c6c): 
> USE mykylin
> INFO  : Starting task [Stage-0:DDL] in serial mode
> INFO  : Completed executing 
> command(queryId=hive_20190418105454_f4e97998-7295-4c33-925f-654624f67c6c); 
> Time taken: 0.008 seconds
> INFO  : OK
> No rows affected (0.122 seconds)
>
>
> So, it's that mean first step also use beeline ? And I have tested that on
> the SSH client , if not enclosed in double quotes ,hive can not parse the
> URL right.
>
>
> 2019-04-18
> --
> lk_hadoop
> --
>
> *发件人：*Chao Long 
> *发送时间：*2019-04-18 15:08
> *主题：*Re: why kylin job failed when use beeline with zookeeper
> *收件人：*"user"
> *抄送：*
>
> Hi lk,
>   First step use SSHClient to run "Create Hive Table" command, so I think
> it will not use beeline to connect hive.
>   "Redistribute Flat Hive Table" step need to compute row count of flat
> table, so it will use beeline to connect if you configured.
>And I see the zookeeper connect string are enclosed in double quotes,
> is that a right way?
>
> On Thu, Apr 18, 2019 at 11:04 AM lk_hadoop  wrote:
>
>> hi,all:
>>I'm using kylin-2.6.1-bin-cdh57 , when I connect to hive with
>> beelin :
>>  kylin.source.hive.beeline-params=-n hive -p hiveadmin --hiveconf
>> hive.security.authorization.sqlstd.confwhitelist.append='mapreduce.job.*|dfs.*'
>> -u
>> jdbc:hive2://"bdp-scm-04:2181,bdp-scm-03:2181,bdp-scm-05:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2"
>>I got error at step : Redistribute Flat Hive Table
>>
>>
>> java.lang.IllegalArgumentException: Illegal character in path at index 86: 
>> hive2://dummyhost:

Re: why kylin job failed when use beeline with zookeeper

[Chao Long]

Hi lk,
  First step use SSHClient to run "Create Hive Table" command, so I think
it will not use beeline to connect hive.
  "Redistribute Flat Hive Table" step need to compute row count of flat
table, so it will use beeline to connect if you configured.
   And I see the zookeeper connect string are enclosed in double quotes, is
that a right way?

On Thu, Apr 18, 2019 at 11:04 AM lk_hadoop  wrote:

> hi,all:
>I'm using kylin-2.6.1-bin-cdh57 , when I connect to hive with
> beelin :
>  kylin.source.hive.beeline-params=-n hive -p hiveadmin --hiveconf
> hive.security.authorization.sqlstd.confwhitelist.append='mapreduce.job.*|dfs.*'
> -u
> jdbc:hive2://"bdp-scm-04:2181,bdp-scm-03:2181,bdp-scm-05:2181/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2"
>I got error at step : Redistribute Flat Hive Table
>
>
> java.lang.IllegalArgumentException: Illegal character in path at index 86: 
> hive2://dummyhost:0/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2"
>   at java.net.URI.create(URI.java:852)
>   at org.apache.hive.jdbc.Utils.parseURL(Utils.java:302)
>   at org.apache.hive.jdbc.HiveConnection.(HiveConnection.java:122)
>   at org.apache.hive.jdbc.HiveDriver.connect(HiveDriver.java:105)
>   at java.sql.DriverManager.getConnection(DriverManager.java:664)
>   at java.sql.DriverManager.getConnection(DriverManager.java:208)
>   at 
> org.apache.kylin.source.hive.BeelineHiveClient.init(BeelineHiveClient.java:72)
>   at 
> org.apache.kylin.source.hive.BeelineHiveClient.(BeelineHiveClient.java:66)
>   at 
> org.apache.kylin.source.hive.HiveClientFactory.getHiveClient(HiveClientFactory.java:29)
>   at 
> org.apache.kylin.source.hive.RedistributeFlatHiveTableStep.computeRowCount(RedistributeFlatHiveTableStep.java:40)
>   at 
> org.apache.kylin.source.hive.RedistributeFlatHiveTableStep.doWork(RedistributeFlatHiveTableStep.java:91)
>   at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
>   at 
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
>   at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
>   at 
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
> Caused by: java.net.URISyntaxException: Illegal character in path at index 
> 86: 
> hive2://dummyhost:0/;serviceDiscoveryMode=zooKeeper;zooKeeperNamespace=hiveserver2"
>   at java.net.URI$Parser.fail(URI.java:2848)
>
>I don't know why , because I can pass the first step which also use
> the same JDBC URL.
>
> 2019-04-18
> --
> lk_hadoop
>

Re: Re: FactDistinctColumnsJob.run throw exception：IllegalStateException

[Chao Long]

You can take a look at this commit:
https://github.com/apache/kylin/commit/b780759c6641f0dfbd57f91d4f3d568b8dbb46f6

On Sat, Apr 13, 2019 at 5:58 PM bubugao0809  wrote:

> The bypassing seems a little tricky, especially in a auto building
> production env, in which one can not manually reload meta every time job
> failed.
> The issue can severely damege the robustness of kylin cluster.
> Is there any clue to fix the issue "It seems caused by the broadcaster
> doesn't broadcast metadata to the job engine in time." ?
> I found the snippet from source code :
>
> CubeSegment segment = cube.getSegmentById(segmentID);
> if (segment == null) {
> logger.error("Failed to find {} in cube {}", segmentID, cube);
> for (CubeSegment s : cube.getSegments()) {
> logger.error(s.getName() + " with status " + s.getStatus());
> }
> throw new IllegalStateException();
> }
>
>
>
>
> At 2019-04-13 12:42:08, "Chao Long"  wrote:
>
> It seems caused by the broadcaster doesn't broadcast metadata to the job
> engine in time. There is a workaround you can try: reload the metadata and
> resume the failed job.
>
> There are two related issues:
> https://issues.apache.org/jira/browse/KYLIN-3740
> https://issues.apache.org/jira/browse/KYLIN-3752
>
> On Fri, Apr 12, 2019 at 11:56 PM venkatesh Mora <
> venkatesh.m...@scanbuy.com> wrote:
>
>> I am running into the same. Can someone respond on this. I opened this a
>> couple of days ago.
>>
>> On Apr 11, 2019, at 9:31 PM, bubugao0809  wrote:
>>
>>
>> Hi all，
>>  I am using kylin-2.2.0, and some of my job constantly failed at the
>> third stage of "Extract Fact Table Distinct Columns", with exception
>> below:
>>
>> java.lang.IllegalStateException
>>  at 
>> org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:116)
>>  at 
>> org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:129)
>>  at 
>> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125)
>>  at 
>> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:64)
>>  at 
>> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125)
>>  at 
>> org.apache.kylin.job.impl.threadpool.DistributedScheduler$JobRunner.run(DistributedScheduler.java:171)
>>  at 
>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>>  at 
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>>  at java.lang.Thread.run(Thread.java:748)
>>
>> result code:2
>>
>>
>>

Re: FactDistinctColumnsJob.run throw exception：IllegalStateException

[Chao Long]

It seems caused by the broadcaster doesn't broadcast metadata to the job
engine in time. There is a workaround you can try: reload the metadata and
resume the failed job.

There are two related issues:
https://issues.apache.org/jira/browse/KYLIN-3740
https://issues.apache.org/jira/browse/KYLIN-3752

On Fri, Apr 12, 2019 at 11:56 PM venkatesh Mora 
wrote:

> I am running into the same. Can someone respond on this. I opened this a
> couple of days ago.
>
> On Apr 11, 2019, at 9:31 PM, bubugao0809  wrote:
>
>
> Hi all，
>  I am using kylin-2.2.0, and some of my job constantly failed at the
> third stage of "Extract Fact Table Distinct Columns", with exception
> below:
>
> java.lang.IllegalStateException
>   at 
> org.apache.kylin.engine.mr.steps.FactDistinctColumnsJob.run(FactDistinctColumnsJob.java:116)
>   at 
> org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:129)
>   at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125)
>   at 
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:64)
>   at 
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125)
>   at 
> org.apache.kylin.job.impl.threadpool.DistributedScheduler$JobRunner.run(DistributedScheduler.java:171)
>   at 
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
>   at 
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
>   at java.lang.Thread.run(Thread.java:748)
>
> result code:2
>
>
>

Re: Not able to start Apache Kylin

[Chao Long]

Hi Yogita,
   You can try to remove "jetty-runner" jar from the classpath by excluding it 
in "$KYLIN_HOME/bin/find-hive-dependency.sh" and see if the problem still 
exists.


   find line: hive_lib=`find -L ${hive_lib_dir} -name '*.jar' ! -name '*druid*' 
! -name '*slf4j*' ! -name '*avatica*' ! -name '*calcite*' !...
   add: ! -name '*jetty-runner*'
--
Best Regards,
Chao Long


-- Original --
From:  "Yogita Mokate";
Date:  Fri, Mar 29, 2019 06:23 PM
To:  "user@kylin.apache.org";

Subject:  Not able to start Apache Kylin



  
Hi All,
 
 
 
We are setting up Apache Kylin to be integrated with one of our product. 
However, the environment is not going up and running with below error message:
 
 
 
INFO: Deploying web application archive /opt/kylin2.5/tomcat/webapps/kylin.war
 
Mar 28, 2019 5:46:21 PM org.apache.catalina.core.ContainerBase addChildInternal
 
SEVERE: ContainerBase.addChild: start: 
 
org.apache.catalina.LifecycleException: Failed to start component 
[StandardEngine[Catalina].StandardHost[localhost].StandardContext[/kylin]]
 
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:162)
 
at 
org.apache.catalina.core.ContainerBase.addChildInternal(ContainerBase.java:1017)
 
at 
org.apache.catalina.core.ContainerBase.addChild(ContainerBase.java:993)
 
at org.apache.catalina.core.StandardHost.addChild(StandardHost.java:652)
 
at 
org.apache.catalina.startup.HostConfig.deployWAR(HostConfig.java:1127)
 
at 
org.apache.catalina.startup.HostConfig$DeployWar.run(HostConfig.java:2021)
 
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:522)
 
at java.util.concurrent.FutureTask.run(FutureTask.java:277)
 
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1160)
 
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
 
at java.lang.Thread.run(Thread.java:812)
 
Caused by: java.lang.NoSuchMethodError: 
org/apache/tomcat/JarScanner.scan(Lorg/apache/tomcat/JarScanType;Ljavax/servlet/ServletContext;Lorg/apache/tomcat/JarScannerCallback;)V
 (loaded from file:/opt/kylin2.5/tomcat/lib/tomcat-api.jar by 
sun.misc.Launcher$AppClassLoader@5c85b446)  called from class 
org.apache.jasper.servlet.TldScanner (loaded from 
file:/opt/hive/lib/jetty-runner-9.3.20.v20170531.jar by 
sun.misc.Launcher$AppClassLoader@5c85b446).
 
at org.apache.jasper.servlet.TldScanner.scanJars(TldScanner.java:262)
 
at org.apache.jasper.servlet.TldScanner.scan(TldScanner.java:106)
 
at 
org.apache.jasper.servlet.JasperInitializer.onStartup(JasperInitializer.java:103)
 
at 
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5669)
 
at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
 
... 10 more
 
 
 
 
 
Below are the component successfully setup as a supporting environment for 
Kylin:
 
 
  
Hadoop cluster with three nodes.(Hadoop version 2.8.5)

Hbase (Version 1.4.8)

Hive (Version 3.1.1)

Spark 2.4.0
 
 
 
Kylin versions tried:
  
Kylin 2.6.1

Kylin 2.5.0
 
 
 
Installation Guide followed from here  http://kylin.apache.org/docs/install/
 
 
 
We thought of the runtime library difference between tomcat-api.jar and jetty 
runner, so tried upgrading both the jars, however, it didn??t produce any 
positive results.
 
 
 
Regards,
 
Yogita Mokate
 
 
 
 
 
 DISCLAIMER
==
This e-mail may contain privileged and  confidential information which is the 
property of Persistent Systems  Ltd. It is intended only for the use of the 
individual or entity to  which it is addressed. If you are not the intended 
recipient, you are  not authorized to read, retain, copy, print, distribute or 
use this  message. If you have received this communication in error, please 
notify  the sender and delete all copies of this message. Persistent Systems  
Ltd. does not accept any liability for virus infected mails.

??????CubeBuildingCLI not work well

[Chao Long]

Hi,
   CubeBuildingCLI will not call method in CubeController, the stack trace show 
some exceptions throw from method CubeController.build(), which will be called 
by rest api. So you may check if there are other running cube-building crontab 
tasks.



--
Best Regards,
Chao Long


--  --
??: "lk_hadoop";
: 2019??3??25??(??) 5:20
??: "user";

: CubeBuildingCLI not work well



  hi,all:
 I'm using kylin 2.6.1 with CDH57, When calling  CubeBuildingCLI the job 
will not be scheduled,only when I click "reload  metadata" on kylin's web , The 
job can be scheduled.
 and if CubeBuildingCLI was called from   crontab  , I can see error 
message from the log :
  
 2019-03-25 16:20:02,325 ERROR [http-bio-7070-exec-2]  
controller.BasicController:63 :  
org.apache.kylin.rest.exception.InternalErrorException
 at  
org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:398)
 at  
org.apache.kylin.rest.controller.CubeController.rebuild(CubeController.java:354)
 at  
org.apache.kylin.rest.controller.CubeController.build(CubeController.java:343)
 at  sun.reflect.GeneratedMethodAccessor210.invoke(Unknown Source)
 at  
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at  java.lang.reflect.Method.invoke(Method.java:498)
 at  
org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
 at  
org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
 at  
org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
 at  
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
 at  
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
 at  
org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
 at  
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
 at  
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
 at  
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
 at  
org.springframework.web.servlet.FrameworkServlet.doPut(FrameworkServlet.java:883)
 at  javax.servlet.http.HttpServlet.service(HttpServlet.java:653)
 at  
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
 at  javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
 at  
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
 at  
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
 at  org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
 at  
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
 at  
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
 at  
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
 at  
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
 at  
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
 at  
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
 at  
org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
 at  
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
 at  
org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
 at  
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
 at  
org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
 at  
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
 at  
org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)
 at  
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
 at  
org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
 at  
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFi

??????CubeBuildingCLI not work well

[Chao Long]

Hi,
   CubeBuildingCLI will not call method in CubeController, the stack trace show 
some exceptions throw from method CubeController.build(), which will be called 
by rest api. So you may check if there are other running cube-building crontab 
tasks.


--
Best Regards,
Chao Long


--  --
??: "lk_hadoop";
: 2019??3??25??(??) 5:20
??: "user";

: CubeBuildingCLI not work well



  hi,all:
 I'm using kylin 2.6.1 with CDH57, When calling  CubeBuildingCLI the job 
will not be scheduled,only when I click "reload  metadata" on kylin's web , The 
job can be scheduled.
 and if CubeBuildingCLI was called from   crontab  , I can see error 
message from the log :
  
 2019-03-25 16:20:02,325 ERROR [http-bio-7070-exec-2]  
controller.BasicController:63 :  
org.apache.kylin.rest.exception.InternalErrorException
 at  
org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:398)
 at  
org.apache.kylin.rest.controller.CubeController.rebuild(CubeController.java:354)
 at  
org.apache.kylin.rest.controller.CubeController.build(CubeController.java:343)
 at  sun.reflect.GeneratedMethodAccessor210.invoke(Unknown Source)
 at  
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
 at  java.lang.reflect.Method.invoke(Method.java:498)
 at  
org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
 at  
org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
 at  
org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
 at  
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
 at  
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
 at  
org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
 at  
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
 at  
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
 at  
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
 at  
org.springframework.web.servlet.FrameworkServlet.doPut(FrameworkServlet.java:883)
 at  javax.servlet.http.HttpServlet.service(HttpServlet.java:653)
 at  
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
 at  javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
 at  
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
 at  
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
 at  org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
 at  
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
 at  
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
 at  
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
 at  
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
 at  
org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
 at  
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
 at  
org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
 at  
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
 at  
org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
 at  
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
 at  
org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
 at  
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
 at  
org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)
 at  
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
 at  
org.springframework.security.web.savedrequest.RequestCacheAwareFilter.doFilter(RequestCacheAwareFilter.java:63)
 at  
org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFi

Re: Question related to kylin jdbc source

[Chao Long]

Hi, Dong
   Kylin datasource-sdk has a workaround for this issue, you can see the 
details from the comments under 
jira[https://issues.apache.org/jira/browse/KYLIN-3779]. Btw, datasource-sdk was 
realeased in Kylin v2.6.0, if you want to use it, you need update your Kylin to 
v2.6.0 or above.
--
Best Regards,
Chao Long


-- Original --
From:  "Dong"<472795...@qq.com>;
Date:  Mon, Mar 18, 2019 11:03 AM
To:  "user";

Subject:  Question related to  kylin jdbc source



Hi
   I configured the mysql jdbc data source,the table name loaded in kylin is 
uppercase, but the table name in mysql is 


lowercase.The following exception occurred during the creation of the cube.





Business needs, mysql can not re-create the uppercase table name.



I am using kylin version 2.4.

Re: Please help me ! About kylin2.5 version dashboard and account management.

[Chao Long]

Hi yangxc,
  1, kylin.web.dashboard-enabled has no default value, you just need to add 
"kylin.web.dashboard-enabled=true" in kylin.properties. Btw, before enabling 
dashboard, please make sure you have setup system 
cube[http://kylin.apache.org/docs/tutorial/setup_systemcube.html]


 2, MODELER,ANALYST,ADMIN is the default user of role type, and the password of 
user MODELER,ANALYST is itself.
   role MODELER means a user can design/build cube, role ANALYST means a user 
can query the cube and do analysis, and role ADMIN means the system 
administrator


3, For user management, you can enable LDAP and 
SSO[http://kylin.apache.org/docs/howto/howto_ldap_and_sso.html].
For data access management, you can enable Project Level 
ACL[http://kylin.apache.org/docs/tutorial/project_level_acl.html]
--
Best Regards,
Chao Long


-- Original --
From:  "yan...@staryea.com";
Date:  Wed, Mar 13, 2019 02:52 PM
To:  "user";

Subject:  Please help me ! About kylin2.5 version dashboard and account 
management.



 Hello:
 The company recently researched the Kylin project, and I have several 
questions after deploying kylin:
 1. Version 2.5 of the dashboard I refer to the official documentation, 
did not find the kylin.web.dashboard-enabled configuration in the 
kylin.properties configuration file?
 2. Regarding kylin's account management, I checked the information. 
The default login and account is ADMIN/KYLIN. There are also two accounts 
(MODELER, ANALYST). The purpose is not very clear. How to get the passwords of 
these two accounts? The passwords in the files are over-encrypted. How to 
decrypt them?
 3. Regarding the addition and authorization management of new users, I 
want to create a new user to view the report data. Can I add different users 
for this piece, the authorization is different, and the data I see is 
different? How to configure?

 


 yan...@staryea.com

??????Dose joint dim constrained by Max Dimension Combination

[Chao Long]

Hi,
I check the code and find joint dim is constrained by Max Dimension 
Combination, so if you set Max Dimension Combination=3, the cuboid result will 
not contain the 4-dim joint. The cuboid contain more then 3-dim will be 
filtered.

   The final cuobid result contain:
  1, BaseCuboid.
  2, cuboid whose dim-count is less than Max Dimension Combination.
  3, fill padding cuboids, which contain the parent of the cuboid in 
2.(dim-count can more than Max Dimension Combination)



--
Best Regards,
Chao Long


--  --
??: "S*ST"<85318...@qq.com>;
: 2019??3??4??(??) 4:25
??: "user";

: Dose joint dim constrained by Max Dimension Combination




hi all: 

If i have a agg with joint dim has 4-dim,  and i set the agg  Max Dimension 
Combination=3.  So will the result cuboid contain the 4-dim joint? 

I find when i set the Max Dimension Combination with 3.  the result cuboid 
contain some dim>3. 

| Cuboid 111, est row: 
34884166, est MB: 848.34  | Cuboid 
101001100010011, est row: 31152743, est MB: 
512.49, shrink: 89.3%  | Cuboid 
101001100010011, est row: 31305812, est MB: 
485.15, shrink: 100.49%  | Cuboid 
101001100010011, est row: 26002267, est MB: 
371.97, shrink: 83.06%  | Cuboid  
100011100010011, est row: 18314746,  est 
MB: 240.16, shrink: 70.44%  | Cuboid  
100011100010011, est row: 17152263,  est 
MB: 204.47, shrink: 93.65%  | Cuboid  
100101100010011, est row: 11100040,  est 
MB: 116.44, shrink: 64.71%  | Cuboid  
100011100010011, est row: 143320,  est MB: 
1.26, shrink: 1.29%  | Cuboid  
111, est row: 281, est  MB: 0, 
shrink: 0.2%  | Cuboid  
101, est row: 22, est  MB: 0, 
shrink: 7.83%  | Cuboid  
110, est row: 30, est  MB: 0, 
shrink: 10.68%

??????cube ????????

[Chao Long]

Hi shen June,


Which build engine do you use, Mapreduce or Spark? Is there any useful 
information of "Merge Cuboid Data" job on yarn?
--
Best Regards,
Chao Long


--  --
??: "shen june";
: 2019??1??2??(??) 7:57
??: "ShaoFeng Shi";
: "user@kylin.apache.org"; 
: cube 



  
Hi, 
 
??cubejob33.3%Merge Cuboid 
DatajobKylin  
2.5.0??centos7??
 
 
 
?? Windows 10 ??

?????? ??????Kylin w/ Spark - Build 626min - Steps 1/2/3 455min - Steps 4-8 - 171min

[Chao Long]

An even distribution means there is not a skew distribution. If data skew 
happen, there may some task's execution time are very larger then average time. 
And the RedistributeFlatHiveTableStep is to avoid data skew as far as possible, 
for more details you can see 
https://issues.apache.org/jira/browse/KYLIN-1656
https://issues.apache.org/jira/browse/KYLIN-1677


And the parameter "kylin.engine.mr.uhc-reducer-count" work for Mapreduce and 
Spark. In Spark, a larger value means allocate more tasks. About what value 
should it be, I think you can see the task execution state of "Extract Fact 
Table Distinct Columns" job in Spark UI and identify the most time consuming 
task and give this parameter a suitable value. And about what exactly it is, I 
don't know.



--
Best Regards,
Chao Long


--  --
??: "Jon Shoberg";
: 2018??12??21??(??) 10:34
??: "user";

: Re: ??Kylin w/ Spark - Build 626min - Steps 1/2/3 455min - Steps 4-8 
- 171min



That??s great to know about step 2!

How would you define or determine an even distribution? This is a four node 
Hdfs cluster and the bz2 files as the data source (external table) have a dfs 
distribution of 2. I??d imagine the distribution would not be horrible on a 
small cluster. 


On the reducer could this is a spark setup. So on yarn I see this step running 
as a spark job. Does a mar reduce setting such as this apply? If so what is a 
larger value. I think the default here is 1 ... should it be 2,5,10,or 100? 
It??s a 4 node cluster with 10 cpus and ~550gb ram. 

Sent from my iPhoneX

On Dec 20, 2018, at 7:24 PM, Chao Long  wrote:


Hi,
  If the data have an even distribution, you can set 
"kylin.source.hive.redistribute-flat-table=false" to skip Step 2. And about 
Step 3, if you have many UHC dimension, you can set 
"kylin.engine.mr.uhc-reducer-count" a larger value to use more reducer to 
handle dict.


--
Best Regards,
Chao Long


--  --
??: "Jon Shoberg";
: 2018??12??20??(??) 10:20
??: "user";

: Kylin w/ Spark - Build 626min - Steps 1/2/3 455min - Steps 4-8 - 171min



Question ...

  Is there a way to optimize the first three steps of a Kylin build?


  Total build time of a development cube is 626 minutes and a break down by 
steps:

87  min - Create Intermediate Flat Hive Table

207 min -  Redistribute Flat Hive Table

248 min -  Extract Fact Table Distinct Columns

0   min

0   min

62  min -  Build Cube with Spark

19  min -  Convert Cuboid Data to HFile

0   min

0   min

0   min

0   min
   The data set is summary files (~35M records) and detail files (~4B records - 
40GB compressed).


   There is a join needed for the final data which is handled in a view within 
hive.  So I do expect a performance cost there.


   However, staging the data other ways (loading to sequence/org file vs 
external table to bz2 files) there is no net-gain.


   This means, pre-processing the data externally can make Kylin run a little 
faster but the overall time from absolute start to finish is still ~600min.


   Steps 1/2 seem to be a redundancy given how my data is structured; the 
hsql/sql commands Kylin sends to Hive could be done before the build process.


   Is it possible to optimize steps 1/2/3? Is it possible to skip steps 1/2 and 
jump to step 3 if the data was staged as-needed/correctly beforehand?


   My guess is there are mostly 'no' answers where (which is fine) but thought 
I'd ask.


   (The test lab is getting doubled in size today so I'm not ultimately worried 
but I'm seeking other improvements vs. only adding hardware and networking)


Thanks! J

?????? Spark tuning within Kylin? Article? Resource?

[Chao Long]

Hi J,
There is a slide about Spark tunning in Apache Kylin(author shaofengshi)
https://www.slideshare.net/ShiShaoFeng1/spark-tunning-in-apache-kylin 


About Step 3 (Extract Fact Table Distinct Columns) OOM, you can try to set this 
parameter "kylin.engine.mr.uhc-reducer-count" a larger value(default 1).



--
Best Regards,
Chao Long




--  --
??: "Jon Shoberg";
: 2018??12??18??(??) 11:16
??: "user";

: Re: Spark tuning within Kylin? Article? Resource?



Greatly appreciate the response.

I started there but after OOM errors I started to work on the settings for my 
test lab. After minimal success thought to ask if there was something more 
in-depth for tuning with other Kylin users found successful.


Right now I've gone to very basic configuration with dynamic allocation and see 
if I can avoid the late-stage OOM errors.


J


On Mon, Dec 17, 2018 at 7:44 PM JiaTao Tao  wrote:

Hope this may help: http://kylin.apache.org/docs/tutorial/cube_spark.html


Jon Shoberg  ??2018??12??18?? 2:34??

Is there a good/favorite article for tuning spark settings within Kylin?

I finally have Spark (2.1.3 as distributed with Kylin 2.5.2) running on my 
systems.


My small data set (35M records) runs well the default settings.


My medium data set (4B records, 40GB compressed source file, 5 measures, 6 
dimensions with low carnality) often dies at Step 3 (Extract Fact Table 
Distinct Columns) with out of memory errors.


After using exceptionally large memory settings the job completed but I'm 
trying to see if there is an optimization possible.


Any suggestions or ideas?  I've searched/read on spark tuning in general but 
otherwise feeling I'm not making too much progress on optimizing with the 
settings I've tried.


Thanks!J

 



-- 




Regards!

Aron Tao

?????? Spark tuning within Kylin? Article? Resource?

[Chao Long]

Hi J,
There is a slide about Spark tunning in Apache Kylin(author shaofengshi)
https://www.slideshare.net/ShiShaoFeng1/spark-tunning-in-apache-kylin


About Step 3 (Extract Fact Table Distinct Columns) OOM, you can try to set this 
parameter "kylin.engine.mr.uhc-reducer-count" a larger value(default 1).




--
Best Regards,
Chao Long


--  --
??: "Jon Shoberg";
: 2018??12??18??(??) 11:16
??: "user";

: Re: Spark tuning within Kylin? Article? Resource?



Greatly appreciate the response.

I started there but after OOM errors I started to work on the settings for my 
test lab. After minimal success thought to ask if there was something more 
in-depth for tuning with other Kylin users found successful.


Right now I've gone to very basic configuration with dynamic allocation and see 
if I can avoid the late-stage OOM errors.


J


On Mon, Dec 17, 2018 at 7:44 PM JiaTao Tao  wrote:

Hope this may help: http://kylin.apache.org/docs/tutorial/cube_spark.html


Jon Shoberg  ??2018??12??18?? 2:34??

Is there a good/favorite article for tuning spark settings within Kylin?

I finally have Spark (2.1.3 as distributed with Kylin 2.5.2) running on my 
systems.


My small data set (35M records) runs well the default settings.


My medium data set (4B records, 40GB compressed source file, 5 measures, 6 
dimensions with low carnality) often dies at Step 3 (Extract Fact Table 
Distinct Columns) with out of memory errors.


After using exceptionally large memory settings the job completed but I'm 
trying to see if there is an optimization possible.


Any suggestions or ideas?  I've searched/read on spark tuning in general but 
otherwise feeling I'm not making too much progress on optimizing with the 
settings I've tried.


Thanks!J

 



-- 




Regards!

Aron Tao

??????Evaluate Kylin on Parquet

[Chao Long]

In this PoC, we verified Kylin On Parquet is viable, but the query performance 
still have room to improve. We can improve it from the following aspects:


 1, Minimize result set serialization time
 Since Kylin need Object[] data to process, we convert Dataset to RDD, and then 
convert the "Row" type to Object[], so Spark need to serialize Object[] before 
return it to driver. Those time need to be avoided.


 2, Query without dictionary
 In this PoC, for less storage use, we keep dict encode value in Parquet file 
for dict-encode dimensions, so Kylin must load dictionary to convert dict value 
for query. If we keep original value for dict-encode dimension, dictionary is 
unnecessary. And we don't hava to worry about the storage use, because Parquet 
will encode it. We should remove dictionary from query.


 3, Remove query single-point issue
 In this PoC, we use Spark to read and process Cube data, which is distributed, 
but kylin alse need to process result data the Spark returned in single jvm. We 
can try to make it distributed too.


 4, Upgrade Parquet to 1.11 for page index
 In this PoC, Parquet don't have page index, we get a poor filter performance. 
We need to upgrade Parquet to version 1.11 which has page index to improve 
filter performance.



--
Best Regards,
Chao Long


 
--  --
??: "ShaoFeng Shi";
: 2018??12??14??(??) 4:39
??: "dev";"user";

: Evaluate Kylin on Parquet



Hello Kylin users,


The first version of Kylin on Parquet [1] feature has been staged in Kylin code 
repository for public review and evaluation. You can check out the 
"kylin-on-parquet" branch [2] to read the code, and also can make a binary 
build to run an example. When creating a cube, you can select "Parquet" as the 
storage in the "Advanced setting" page. Both MapReduce and Spark engines 
support this new storage. A tech blog is under drafting for the design and 
implementation.



Thanks so much to the engineers' hard work: Chao Long and Yichen Zhou!


This is not the final version; there is room to improve in many aspects, 
parquet, spark, and Kylin. It can be used for PoC at this moment. Your comments 
are welcomed. Let's improve it together.


[1] https://issues.apache.org/jira/browse/KYLIN-3621
[2] https://github.com/apache/kylin/tree/kylin-on-parquet

Best regards,

Shaofeng Shi ??
Apache Kylin PMC
Work email: shaofeng@kyligence.io

Kyligence Inc: https://kyligence.io/


Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscr...@kylin.apache.org
Join Kylin dev mail group: dev-subscr...@kylin.apache.org

Re: when create htable ,occur error

[Chao Long]

Hi, 
  What's your Kylin version and which build engine you use?


--
Best Regards,
Chao Long


 
-- Original --
From:  "??";
Date:  Thu, Dec 13, 2018 09:10 PM
To:  "user";

Subject:  when  create htable ,occur error



hi,all


when Create HTable ,i get this errors. 
resume this cube, do not  work.
how can i resolve the problem .



2018-12-13 20:49:50,439 INFO  [Scheduler 190313534 Job 
feed250d-4bbc-242f-ab6f-aa8317aacd87-871] common.HadoopShellExecutable:59 : 
parameters of the HadoopShellExecutable:  -cubename 
kuaidui_offline_action_crm_cube -segmentid e26a43d4-879a-6219-7edf-8b2528851207 
-partitions 
hdfs://ns1/kylin/kylin_metadata/kylin-feed250d-4bbc-242f-ab6f-aa8317aacd87/kuaidui_offline_action_crm_cube/rowkey_stats/part-r-0
 -cuboidMode CURRENT -hbaseConfPath 
hdfs://ns1/kylin/kylin_metadata/kylin-feed250d-4bbc-242f-ab6f-aa8317aacd87/hbase-conf.xml
2018-12-13 20:49:50,533 ERROR [Scheduler 190313534 Job 
feed250d-4bbc-242f-ab6f-aa8317aacd87-871] common.HadoopShellExecutable:65 : 
error execute HadoopShellExecutable{id=feed250d-4bbc-242f-ab6f-aa8317aacd87-05, 
name=Create HTable, state=RUNNING}
java.lang.NullPointerException
at 
org.apache.kylin.engine.mr.common.CubeStatsReader.getCuboidSizeMapFromRowCount(CubeStatsReader.java:207)
at 
org.apache.kylin.engine.mr.common.CubeStatsReader.getCuboidSizeMap(CubeStatsReader.java:165)
at 
org.apache.kylin.storage.hbase.steps.CreateHTableJob.run(CreateHTableJob.java:100)
at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:92)
at 
org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
at 
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:163)
at 
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:69)
at 
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:163)
at 
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:113)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2018-12-13 20:49:50,550 INFO  [Scheduler 190313534 Job 
feed250d-4bbc-242f-ab6f-aa8317aacd87-871] execution.ExecutableManager:434 : job 
id:feed250d-4bbc-242f-ab6f-aa8317aacd87-05 from RUNNING to ERROR
2018-12-13 20:49:50,552 ERROR [Scheduler 190313534 Job 
feed250d-4bbc-242f-ab6f-aa8317aacd87-871] execution.AbstractExecutable:165 : 
error running Executable: CubingJob{id=feed250d-4bbc-242f-ab6f-aa8317aacd87, 
name=BUILD CUBE - kuaidui_offline_action_crm_cube - 
2018120100_2018120200 - GMT+08:00 2018-12-13 20:25:57, state=RUNNING}
2018-12-13 20:49:50,561 DEBUG [pool-12-thread-1] cachesync.Broadcaster:113 : 
Servers in the cluster: [localhost:7070]
2018-12-13 20:49:50,562 DEBUG [pool-12-thread-1] cachesync.Broadcaster:123 : 
Announcing new broadcast to all: BroadcastEvent{entity=execute_output, 
event=update, cacheKey=feed250d-4bbc-242f-ab6f-aa8317aacd87}
2018-12-13 20:49:50,566 DEBUG [http-bio-7070-exec-4] cachesync.Broadcaster:247 
: Broadcasting UPDATE, execute_output, feed250d-4bbc-242f-ab6f-aa8317aacd87
2018-12-13 20:49:50,569 INFO  [Scheduler 190313534 Job 
feed250d-4bbc-242f-ab6f-aa8317aacd87-871] execution.ExecutableManager:434 : job 
id:feed250d-4bbc-242f-ab6f-aa8317aacd87 from RUNNING to ERROR
2018-12-13 20:49:50,569 DEBUG [http-bio-7070-exec-4] cachesync.Broadcaster:281 
: Done broadcasting UPDATE, execute_output, feed250d-4bbc-242f-ab6f-aa8317aacd87
2018-12-13 20:49:50,569 DEBUG [pool-12-thread-1] cachesync.Broadcaster:113 : 
Servers in the cluster: [localhost:7070]
2018-12-13 20:49:50,569 DEBUG [pool-12-thread-1] cachesync.Broadcaster:123 : 
Announcing new broadcast to all: BroadcastEvent{entity=execute_output, 
event=update, cacheKey=feed250d-4bbc-242f-ab6f-aa8317aacd87}
2018-12-13 20:49:50,575 DEBUG [http-bio-7070-exec-4] cachesync.Broadcaster:247 
: Broadcasting UPDATE, execute_output, feed250d-4bbc-242f-ab6f-aa8317aacd87
2018-12-13 20:49:50,578 DEBUG [http-bio-7070-exec-4] cachesync.Broadcaster:281 
: Done broadcasting UPDATE, execute_output, feed250d-4bbc-242f-ab6f-aa8317aacd87
2018-12-13 20:49:50,613 INFO  [Scheduler 190313534 Job 
feed250d-4bbc-242f-ab6f-aa8317aacd87-871] execution.AbstractExecutable:343 : 
prepare to send email to:[meizhen...@zuoyebang.com, liwenji...@zuoyebang.com]
2018-12-13 20:49:50,614 INFO  [Scheduler 190313534 Job 
feed250d-4bbc-242f-ab6f-aa8317aacd87-871] execution.AbstractExecutable:344 : 
job name:BUILD CUBE - kuaidui_offline_action_crm_cube - 
2018120100_2018120200 - GMT+08:00 2018-12-13 20:25:57
2018-12-13 20:49:50,614 INFO  [Scheduler 190313534 Job 
feed250d-4bbc-24

??????problem encountered when upgrade kylin from 2.0 to 2.2

[Chao Long]

Hi,
   I see your directory still Kylin-2.0.0, is that correct?



-
Chao Long


 




--  --
??: "bubugao0809";
: 2018??11??28??(??) 9:14
??: "user";

: Re:problem encountered when  upgrade kylin from 2.0 to 2.2



Pls take a look   







At 2018-11-26 19:43:08, "bubugao0809"  wrote:
 Our cluster is using kylin 2.0 currently, and I am trying to upgrade to 2.2.
I follow the step here http://kylin.apache.org/docs/howto/howto_upgrade.html 
(Including migrate, and update coprocessor, but I did not delete the table in 
hbase in case any accident)
However I still get the following error:


2018-11-26 19:17:59,913 DEBUG [localhost-startStop-1] hbase.HBaseConnection:279 
: HTable 'kylin_metadata' already exists
2018-11-26 19:18:00,941 WARN  [localhost-startStop-1] 
support.XmlWebApplicationContext:551 : Exception encountered during context 
initialization - cancelling refresh attempt: 
org.springframework.beans.factory.UnsatisfiedDependencyException: Error 
creating bean with name 'cubeController': Unsatisfied dependency expressed 
through field 'cubeService'; nested exception is 
org.springframework.beans.factory.UnsatisfiedDependencyException: Error 
creating bean with name 'cubeMgmtService': Unsatisfied dependency expressed 
through field 'jobService'; nested exception is 
org.springframework.beans.factory.BeanCreationException: Error creating bean 
with name 'jobService' defined in URL 
[jar:file:/home/kylin/kylin-2.0.0/tomcat/webapps/kylin/WEB-INF/lib/kylin-server-base-2.2.0.jar!/org/apache/kylin/rest/service/JobService.class]:
 Invocation of init method failed; nested exception is 
java.lang.RuntimeException: java.lang.ClassNotFoundException: 
org.apache.kylin.storage.hbase.util.ZookeeperDistributedJobLock
2018-11-26 19:18:00,945 ERROR [localhost-startStop-1] context.ContextLoader:350 
: Context initialization failed
org.springframework.beans.factory.UnsatisfiedDependencyException: Error 
creating bean with name 'cubeController': Unsatisfied dependency expressed 
through field 'cubeService'; nested exception is 
org.springframework.beans.factory.UnsatisfiedDependencyException: Error 
creating bean with name 'cubeMgmtService': Unsatisfied dependency expressed 
through field 'jobService'; nested exception is 
org.springframework.beans.factory.BeanCreationException: Error creating bean 
with name 'jobService' defined in URL 
[jar:file:/home/kylin/kylin-2.0.0/tomcat/webapps/kylin/WEB-INF/lib/kylin-server-base-2.2.0.jar!/org/apache/kylin/rest/service/JobService.class]:
 Invocation of init method failed; nested exception is 
java.lang.RuntimeException: java.lang.ClassNotFoundException: 
org.apache.kylin.storage.hbase.util.ZookeeperDistributedJobLock
at 
org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:588)
at 
org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88)
at 
org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:366)
at 
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1264)
at 
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
at 
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
at 
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at 
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at 
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at 
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at 
org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)
at 
org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
at 
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543)
at 
org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:443)
at 
org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:325)
at 
org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107)
at 
org.apache.catalina.core.StandardContext.listenerS

??????problem encountered when upgrade kylin from 2.0 to 2.2

[Chao Long]

Hi,
   I see your directory still Kylin-2.0.0, is that correct?




-
Chao Long


 




--  --
??: "bubugao0809";
: 2018??11??28??(??) 9:14
??: "user";

: Re:problem encountered when  upgrade kylin from 2.0 to 2.2



Pls take a look   







At 2018-11-26 19:43:08, "bubugao0809"  wrote:
 Our cluster is using kylin 2.0 currently, and I am trying to upgrade to 2.2.
I follow the step here http://kylin.apache.org/docs/howto/howto_upgrade.html 
(Including migrate, and update coprocessor, but I did not delete the table in 
hbase in case any accident)
However I still get the following error:


2018-11-26 19:17:59,913 DEBUG [localhost-startStop-1] hbase.HBaseConnection:279 
: HTable 'kylin_metadata' already exists
2018-11-26 19:18:00,941 WARN  [localhost-startStop-1] 
support.XmlWebApplicationContext:551 : Exception encountered during context 
initialization - cancelling refresh attempt: 
org.springframework.beans.factory.UnsatisfiedDependencyException: Error 
creating bean with name 'cubeController': Unsatisfied dependency expressed 
through field 'cubeService'; nested exception is 
org.springframework.beans.factory.UnsatisfiedDependencyException: Error 
creating bean with name 'cubeMgmtService': Unsatisfied dependency expressed 
through field 'jobService'; nested exception is 
org.springframework.beans.factory.BeanCreationException: Error creating bean 
with name 'jobService' defined in URL 
[jar:file:/home/kylin/kylin-2.0.0/tomcat/webapps/kylin/WEB-INF/lib/kylin-server-base-2.2.0.jar!/org/apache/kylin/rest/service/JobService.class]:
 Invocation of init method failed; nested exception is 
java.lang.RuntimeException: java.lang.ClassNotFoundException: 
org.apache.kylin.storage.hbase.util.ZookeeperDistributedJobLock
2018-11-26 19:18:00,945 ERROR [localhost-startStop-1] context.ContextLoader:350 
: Context initialization failed
org.springframework.beans.factory.UnsatisfiedDependencyException: Error 
creating bean with name 'cubeController': Unsatisfied dependency expressed 
through field 'cubeService'; nested exception is 
org.springframework.beans.factory.UnsatisfiedDependencyException: Error 
creating bean with name 'cubeMgmtService': Unsatisfied dependency expressed 
through field 'jobService'; nested exception is 
org.springframework.beans.factory.BeanCreationException: Error creating bean 
with name 'jobService' defined in URL 
[jar:file:/home/kylin/kylin-2.0.0/tomcat/webapps/kylin/WEB-INF/lib/kylin-server-base-2.2.0.jar!/org/apache/kylin/rest/service/JobService.class]:
 Invocation of init method failed; nested exception is 
java.lang.RuntimeException: java.lang.ClassNotFoundException: 
org.apache.kylin.storage.hbase.util.ZookeeperDistributedJobLock
at 
org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor$AutowiredFieldElement.inject(AutowiredAnnotationBeanPostProcessor.java:588)
at 
org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:88)
at 
org.springframework.beans.factory.annotation.AutowiredAnnotationBeanPostProcessor.postProcessPropertyValues(AutowiredAnnotationBeanPostProcessor.java:366)
at 
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.populateBean(AbstractAutowireCapableBeanFactory.java:1264)
at 
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:553)
at 
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
at 
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
at 
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
at 
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
at 
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)
at 
org.springframework.beans.factory.support.DefaultListableBeanFactory.preInstantiateSingletons(DefaultListableBeanFactory.java:761)
at 
org.springframework.context.support.AbstractApplicationContext.finishBeanFactoryInitialization(AbstractApplicationContext.java:867)
at 
org.springframework.context.support.AbstractApplicationContext.refresh(AbstractApplicationContext.java:543)
at 
org.springframework.web.context.ContextLoader.configureAndRefreshWebApplicationContext(ContextLoader.java:443)
at 
org.springframework.web.context.ContextLoader.initWebApplicationContext(ContextLoader.java:325)
at 
org.springframework.web.context.ContextLoaderListener.contextInitialized(ContextLoaderListener.java:107)
at 
org.apache.catalina.core.StandardContext.listenerS

??????[Announce] Welcome new Apache Kylin committer: ChunEn Ni ??????????

[Chao Long]

Congratulations! chunen


-
Chao Long


 




--  --
??: "ShaoFeng Shi";
: 2018??11??27??(??) 3:59
??: "dev";"user";"Apache 
Kylin PMC";

: [Announce] Welcome new Apache Kylin committer: ChunEn Ni ??



The Project Management Committee (PMC) for Apache Kylin

has invited ChunEn Ni?? to become a committer and we are pleased
to announce that he has accepted.

Congratulations and welcome, ChunEn!

Shaofeng Shi


On behalf of the Apache Kylin PMC

?????? get org.apache.kylin.rest.exception.InternalErrorException when querying

[Chao Long]

Can you provide the error log of the query with "group by"??




--  --
??: "Pengfei Xue";
: 2018??10??25??(??) 9:01
??: "user";

: Re: get org.apache.kylin.rest.exception.InternalErrorException when 
querying



find this link https://issues.apache.org/jira/browse/CALCITE-1952, kylin is 
uisng Calcite 1.13.0-kylin-r4, will this causes the query problem?


On Thu, Oct 25, 2018 at 8:29 PM Pengfei Xue  wrote:

hmm, updated my query with `group by`, the exception is still there, how to 
track the root cause?

On Thu, Oct 25, 2018 at 4:28 PM Chao Long  wrote:

This question has been added to the FAQ page, please check:


https://kylin.apache.org/docs/gettingstarted/faq.html





--  --
??: "Pengfei Xue";
: 2018??10??25??(??) 4:21
??: "user";

: get org.apache.kylin.rest.exception.InternalErrorException when querying




hi there, 

we are trying to use kylin as the query engine, now, we can successfully build 
the cube, but when we querying, get this error. following is the exception 
stack:


kylin version:2.4.0
hbase: 1.2.0


2018-10-25 15:52:53,510 INFO  [Scheduler 1740251910 FetcherRunner 
1417466356-45] common.KylinConfig:428 : Creating new manager instance of class 
org.apache.kylin.job.execution.ExecutableManager
2018-10-25 15:52:53,510 INFO  [Scheduler 1740251910 FetcherRunner 
1417466356-45] execution.ExecutableManager:68 : Using metadata url: 
kylin_metadata@hbase
2018-10-25 15:52:53,510 INFO  [Scheduler 1740251910 FetcherRunner 
1417466356-45] common.KylinConfig:428 : Creating new manager instance of class 
org.apache.kylin.job.dao.ExecutableDao
2018-10-25 15:52:53,510 INFO  [Scheduler 1740251910 FetcherRunner 
1417466356-45] dao.ExecutableDao:59 : Using metadata url: kylin_metadata@hbase
2018-10-25 15:52:53,531 INFO  [Scheduler 1740251910 FetcherRunner 
1417466356-45] threadpool.DefaultScheduler:275 : Job Fetcher: 0 should running, 
0 actual running, 0 stopped, 0 ready, 7 already succeed, 0 error, 0 discarded, 
0 others
2018-10-25 15:53:00,310 DEBUG [http-bio-7070-exec-4] 
servlet.DispatcherServlet:869 : DispatcherServlet with name 'kylin' processing 
POST request for [/kylin/api/query]
2018-10-25 15:53:00,310 DEBUG [http-bio-7070-exec-4] 
annotation.RequestMappingHandlerMapping:310 : Looking up handler method for 
path /query
2018-10-25 15:53:00,311 DEBUG [http-bio-7070-exec-4] 
annotation.RequestMappingHandlerMapping:317 : Returning handler method [public 
org.apache.kylin.rest.response.SQLResponse 
org.apache.kylin.rest.controller.QueryController.query(org.apache.kylin.rest.request.PrepareSqlRequest)]
2018-10-25 15:53:00,311 DEBUG [http-bio-7070-exec-4] 
support.DefaultListableBeanFactory:251 : Returning cached instance of singleton 
bean 'queryController'
2018-10-25 15:53:00,312 DEBUG [http-bio-7070-exec-4] 
cors.DefaultCorsProcessor:77 : Skip CORS processing: request is from same origin
2018-10-25 15:53:00,312 DEBUG [http-bio-7070-exec-4] 
annotation.RequestResponseBodyMethodProcessor:197 : Read [class 
org.apache.kylin.rest.request.PrepareSqlRequest] as 
"application/json;charset=UTF-8" with 
[org.springframework.http.converter.json.MappingJackson2HttpMessageConverter@11d4ebd3]
2018-10-25 15:53:00,313 INFO  [http-bio-7070-exec-4] service.QueryService:324 : 
Check query permission in 1 ms.
2018-10-25 15:53:00,313 INFO  [Query 734f5be5-e0f0-4af1-92aa-c37b50b044f7-70] 
service.QueryService:355 : Using project: test1
2018-10-25 15:53:00,313 INFO  [Query 734f5be5-e0f0-4af1-92aa-c37b50b044f7-70] 
service.QueryService:356 : The original query:  select * from ML_DEVICE_CHANNEL 
limit 100;
2018-10-25 15:53:00,314 DEBUG [Query 734f5be5-e0f0-4af1-92aa-c37b50b044f7-70] 
project.ProjectL2Cache:195 : Loading L2 project cache for test1
2018-10-25 15:53:00,314 INFO  [Query 734f5be5-e0f0-4af1-92aa-c37b50b044f7-70] 
common.KylinConfig:428 : Creating new manager instance of class 
org.apache.kylin.metadata.TableMetadataManager
2018-10-25 15:53:00,314 DEBUG [Query 734f5be5-e0f0-4af1-92aa-c37b50b044f7-70] 
cachesync.CachedCrudAssist:118 : Reloading TableDesc from 
kylin_metadata(key='/table')@kylin_metadata@hbase
2018-10-25 15:53:00,336 DEBUG [Query 734f5be5-e0f0-4af1-92aa-c37b50b044f7-70] 
cachesync.CachedCrudAssist:127 : Loaded 3 TableDesc(s) out of 3 resource
2018-10-25 15:53:00,336 DEBUG [Query 734f5be5-e0f0-4af1-92aa-c37b50b044f7-70] 
cachesync.CachedCrudAssist:118 : Reloading TableExtDesc from 
kylin_metadata(key='/table_exd')@kylin_metadata@hbase
2018-10-25 15:53:00,359 DEBUG [Query 734f5be5-e0f0-4af1-92aa-c37b50b044f7-70] 
cachesync.CachedCrudAssist:127 : Loaded 3 TableExtDesc(s) out of 3 resource
2018-10-25 15:53:00,359 DEBUG [Query 734f5be5-e0f0-4af1-92aa-c37b50b044f7-70] 
cachesync.CachedCrudAssist:118 : Reloading ExternalFilterDesc from 
kylin_metadata(key='/ext_filter')@kylin_metadata@hbase
2018-10-25 15:53

??????get org.apache.kylin.rest.exception.InternalErrorException when querying

[Chao Long]

This question has been added to the FAQ page, please check:


https://kylin.apache.org/docs/gettingstarted/faq.html





--  --
??: "Pengfei Xue";
: 2018??10??25??(??) 4:21
??: "user";

: get org.apache.kylin.rest.exception.InternalErrorException when querying



hi there, 

we are trying to use kylin as the query engine, now, we can successfully build 
the cube, but when we querying, get this error. following is the exception 
stack:


kylin version:2.4.0
hbase: 1.2.0


2018-10-25 15:52:53,510 INFO  [Scheduler 1740251910 FetcherRunner 
1417466356-45] common.KylinConfig:428 : Creating new manager instance of class 
org.apache.kylin.job.execution.ExecutableManager
2018-10-25 15:52:53,510 INFO  [Scheduler 1740251910 FetcherRunner 
1417466356-45] execution.ExecutableManager:68 : Using metadata url: 
kylin_metadata@hbase
2018-10-25 15:52:53,510 INFO  [Scheduler 1740251910 FetcherRunner 
1417466356-45] common.KylinConfig:428 : Creating new manager instance of class 
org.apache.kylin.job.dao.ExecutableDao
2018-10-25 15:52:53,510 INFO  [Scheduler 1740251910 FetcherRunner 
1417466356-45] dao.ExecutableDao:59 : Using metadata url: kylin_metadata@hbase
2018-10-25 15:52:53,531 INFO  [Scheduler 1740251910 FetcherRunner 
1417466356-45] threadpool.DefaultScheduler:275 : Job Fetcher: 0 should running, 
0 actual running, 0 stopped, 0 ready, 7 already succeed, 0 error, 0 discarded, 
0 others
2018-10-25 15:53:00,310 DEBUG [http-bio-7070-exec-4] 
servlet.DispatcherServlet:869 : DispatcherServlet with name 'kylin' processing 
POST request for [/kylin/api/query]
2018-10-25 15:53:00,310 DEBUG [http-bio-7070-exec-4] 
annotation.RequestMappingHandlerMapping:310 : Looking up handler method for 
path /query
2018-10-25 15:53:00,311 DEBUG [http-bio-7070-exec-4] 
annotation.RequestMappingHandlerMapping:317 : Returning handler method [public 
org.apache.kylin.rest.response.SQLResponse 
org.apache.kylin.rest.controller.QueryController.query(org.apache.kylin.rest.request.PrepareSqlRequest)]
2018-10-25 15:53:00,311 DEBUG [http-bio-7070-exec-4] 
support.DefaultListableBeanFactory:251 : Returning cached instance of singleton 
bean 'queryController'
2018-10-25 15:53:00,312 DEBUG [http-bio-7070-exec-4] 
cors.DefaultCorsProcessor:77 : Skip CORS processing: request is from same origin
2018-10-25 15:53:00,312 DEBUG [http-bio-7070-exec-4] 
annotation.RequestResponseBodyMethodProcessor:197 : Read [class 
org.apache.kylin.rest.request.PrepareSqlRequest] as 
"application/json;charset=UTF-8" with 
[org.springframework.http.converter.json.MappingJackson2HttpMessageConverter@11d4ebd3]
2018-10-25 15:53:00,313 INFO  [http-bio-7070-exec-4] service.QueryService:324 : 
Check query permission in 1 ms.
2018-10-25 15:53:00,313 INFO  [Query 734f5be5-e0f0-4af1-92aa-c37b50b044f7-70] 
service.QueryService:355 : Using project: test1
2018-10-25 15:53:00,313 INFO  [Query 734f5be5-e0f0-4af1-92aa-c37b50b044f7-70] 
service.QueryService:356 : The original query:  select * from ML_DEVICE_CHANNEL 
limit 100;
2018-10-25 15:53:00,314 DEBUG [Query 734f5be5-e0f0-4af1-92aa-c37b50b044f7-70] 
project.ProjectL2Cache:195 : Loading L2 project cache for test1
2018-10-25 15:53:00,314 INFO  [Query 734f5be5-e0f0-4af1-92aa-c37b50b044f7-70] 
common.KylinConfig:428 : Creating new manager instance of class 
org.apache.kylin.metadata.TableMetadataManager
2018-10-25 15:53:00,314 DEBUG [Query 734f5be5-e0f0-4af1-92aa-c37b50b044f7-70] 
cachesync.CachedCrudAssist:118 : Reloading TableDesc from 
kylin_metadata(key='/table')@kylin_metadata@hbase
2018-10-25 15:53:00,336 DEBUG [Query 734f5be5-e0f0-4af1-92aa-c37b50b044f7-70] 
cachesync.CachedCrudAssist:127 : Loaded 3 TableDesc(s) out of 3 resource
2018-10-25 15:53:00,336 DEBUG [Query 734f5be5-e0f0-4af1-92aa-c37b50b044f7-70] 
cachesync.CachedCrudAssist:118 : Reloading TableExtDesc from 
kylin_metadata(key='/table_exd')@kylin_metadata@hbase
2018-10-25 15:53:00,359 DEBUG [Query 734f5be5-e0f0-4af1-92aa-c37b50b044f7-70] 
cachesync.CachedCrudAssist:127 : Loaded 3 TableExtDesc(s) out of 3 resource
2018-10-25 15:53:00,359 DEBUG [Query 734f5be5-e0f0-4af1-92aa-c37b50b044f7-70] 
cachesync.CachedCrudAssist:118 : Reloading ExternalFilterDesc from 
kylin_metadata(key='/ext_filter')@kylin_metadata@hbase
2018-10-25 15:53:00,363 DEBUG [Query 734f5be5-e0f0-4af1-92aa-c37b50b044f7-70] 
cachesync.CachedCrudAssist:127 : Loaded 0 ExternalFilterDesc(s) out of 0 
resource
2018-10-25 15:53:00,364 INFO  [Query 734f5be5-e0f0-4af1-92aa-c37b50b044f7-70] 
common.KylinConfig:428 : Creating new manager instance of class 
org.apache.kylin.metadata.realization.RealizationRegistry
2018-10-25 15:53:00,364 INFO  [Query 734f5be5-e0f0-4af1-92aa-c37b50b044f7-70] 
realization.RealizationRegistry:54 : Initializing RealizationRegistry with 
metadata url kylin_metadata@hbase
2018-10-25 15:53:00,364 INFO  [Query 734f5be5-e0f0-4af1-92aa-c37b50b044f7-70] 
common.KylinConfig:428 : Creating new manager instance of class 

Re: How to build necessary cuboids rather than building all cuboids whenadding or removing one dimension

[Chao Long]

Hybrid is to consolidate history cube and new cube. And once a cube is built, 
its structure couldn??t be modified. To add/remove dimension, you need to clone 
a new cube, then add/remove in the new cube. And if you can accept the absence 
of new dimensions for historical data, you can build the new cube since the end 
time of the old cube. And then create a hybrid model over the old and new cube.




-- Original --
From:  "you Zhuang";
Date:  Tue, Sep 25, 2018 10:39 AM
To:  "user";

Subject:  How to build necessary cuboids rather than building all cuboids 
whenadding or removing one dimension 



Hybrid cube does not affect all segments, and not cover removing situation. So 
how to build the related adding or removing dimension cuboids only?