Re: Kyin HDFS Encryption - Cube Build Job Failed

[Yaqian Zhang]

Hi Singh:

Kylin has not been tested and supported in hadoop environment with KMS. 
According to the error report, you may need to modify and debug the source code 
to run kylin normally in your environment.

> 在 2022年3月18日，上午10:18，Singh Sonu  写道：
> 
> Hi Experts, We need your help with below query:
> 
> We enabled HDFS encryption using, hadoop KMS ( Hadoop 3.1.0 )..
> hdfs-site.xml and core-site.xml is moved to $KYLIN_HOME/spark/conf with the
> details of the kms keyprovider...
> 
> (Kylin 3.1.2) After this the spark cube build is failing in the job step :
> Convert Cuboid Data to HFile,
> 
> this is because Cuboid files which should be generated in the  Build Cube
> with Spark step is not created.. and in yarn logs, we could see the below
> error..
> 
> This is from build cuboid step
> --
> 2022-03-16 07:32:45,140 INFO spark.SparkCubingByLayer: RDD input path:
> hdfs://dev-enc:8020/apps/kylin/kylin/kylin-b122587f-6391-7943-7dfa-84f4898c94b1/qubz_inte
> rmediate_qubz_hive_metrics_query_rpc_qa_589ee76a_201b_932f_d691_5ce8714ff503
> 2022-03-16 07:32:45,140 INFO spark.SparkCubingByLayer: RDD Output path:
> hdfs://dev-enc:8020/apps/kylin/kylin/kylin-b122587f-6391-7943-7dfa-84f4898c94b1/QUBZ_HIV
> E_METRICS_QUERY_RPC_QA/cuboid/
> 2022-03-16 07:32:45,206 INFO compress.CodecPool: Got brand-new decompressor
> [.deflate]
> 2022-03-16 07:32:45,240 INFO spark.SparkCubingByLayer: All measure are
> normal (agg on all cuboids) ? : true
> 2022-03-16 07:32:45,257 WARN util.HadoopUtil: Read sequence file
> .hive-staging_hive_2022-03-16_07-29-50_524_5412701170850172132-1 failed.
> java.io.FileNotFoundException: Path is not a file:
> /apps/kylin/kylin/kylin-b122587f-6391-7943-7dfa-84f4898c94b1/qubz_intermediate_qubz_hive_metrics_query_rpc_qa
> _589ee76a_201b_932f_d691_5ce8714ff503/.hive-staging_hive_2022-03-16_07-29-50_524_5412701170850172132-1
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:90)
> at
> org.apache.hadoop.hdfs.server.namenode.INodeFile.valueOf(INodeFile.java:76)
> at
> org.apache.hadoop.hdfs.server.namenode.FSDirStatAndListingOp.getBlockLocations(FSDirStatAndListingOp.java:153)
> at
> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.getBlockLocations(FSNamesystem.java:1927)
> at
> org.apache.hadoop.hdfs.server.namenode.NameNodeRpcServer.getBlockLocations(NameNodeRpcServer.java:738)
> at
> org.apache.hadoop.hdfs.protocolPB.ClientNamenodeProtocolServerSideTranslatorPB.getBlockLocations(ClientNamenodeProtocolServerSideTranslatorPB.java
> :424)
> at
> org.apache.hadoop.hdfs.protocol.proto.ClientNamenodeProtocolProtos$ClientNamenodeProtocol$2.callBlockingMethod(ClientNamenodeProtocolProtos.java)
> at
> org.apache.hadoop.ipc.ProtobufRpcEngine$Server$ProtoBufRpcInvoker.call(ProtobufRpcEngine.java:523)
> at org.apache.hadoop.ipc.RPC$Server.call(RPC.java:991)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:869)
> at org.apache.hadoop.ipc.Server$RpcCall.run(Server.java:815)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1682)
> at org.apache.hadoop.ipc.Server$Handler.run(Server.java:2675)
> -
> This is from build cuboid step
> ---
> 2022-03-16 15:30:14,343 ERROR spark.SparkCubingByLayer: Exception
> java.lang.RuntimeException: java.io.IOException:
> java.lang.ClassCastException: org.apache.hadoop.fs.FsUrlConnection cannot
> be cast to java.net.HttpURLConnection
> at
> org.apache.kylin.dict.DictionaryManager.getDictionaryInfo(DictionaryManager.java:103)
> at
> org.apache.kylin.cube.CubeManager$DictionaryAssist.getDictionary(CubeManager.java:1210)
> at org.apache.kylin.cube.CubeManager.getDictionary(CubeManager.java:1132)
> at org.apache.kylin.cube.CubeSegment.getDictionary(CubeSegment.java:382)
> at
> org.apache.kylin.cube.kv.CubeDimEncMap.getDictionary(CubeDimEncMap.java:79)
> at org.apache.kylin.cube.kv.CubeDimEncMap.get(CubeDimEncMap.java:58)
> at
> org.apache.kylin.engine.mr.common.CubeStatsReader.getCuboidSizeMapFromRowCount(CubeStatsReader.java:213)
> at
> org.apache.kylin.engine.mr.common.CubeStatsReader.getCuboidSizeMap(CubeStatsReader.java:172)
> at
> org.apache.kylin.engine.mr.common.CubeStatsReader.getCuboidSizeMap(CubeStatsReader.java:168)
> at
> org.apache.kylin.engine.mr.common.CubeStatsReader.estimateLayerSize(CubeStatsReader.java:397)
> at
> org.apache.kylin.engine.spark.SparkUtil.estimateLayerPartitionNum(SparkUtil.java:104)
> at
> org.apache.kylin.engine.spark.SparkCubingByLayer.execute(SparkCubingByLayer.java:179)
> at
> org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:29)
> at org.apache.kylin.common.util.SparkEntry.main(SparkEntry.java:36)
> at 

Re: spark-3.1.2 以及 kylin4.0.0问题请教

[Yaqian Zhang]

Hi：

Is there actually a column named “first_level_category” in your source 
table?Have you added this column to the dimension of cube? Where does the 
"array" mentioned in the question come from?

> 在 2022年3月15日，上午9:52，Math  写道：
> 
> that not a problem, you just download kylin's shell, website is 
> *
> 
> 
> 
> 
> --原始邮件--
> 发件人:  
>   "dev"   
>  
>  发送时间:2022年3月14日(星期一) 晚上7:33
> 收件人:"dev" 
> 主题:spark-3.1.2 以及 kylin4.0.0问题请教
> 
> 
> 
> 您好，
>   我们公司在使用spark-3.1.2 以及 
> kylin4.0.0版本时，再kylin4创建model页面，setting选项卡中，添加了filter条件。但是构建cube失败，提示不能解析 
> filter 条件的属性。详细问题如下
> 
> filter如下图
> 
> 
> 
> 报错提示如下：
> 
> 2022-03-14 18:59:05,037 ERROR [Thread-1] application.SparkApplication : The 
> spark job execute failed!
> 
> org.apache.spark.sql.AnalysisException: cannot resolve 
> '`first_level_category`' given input columns: 
> 
> 以下是 columns 内容：其中与first_level_category相关的就是 
> RZP_INDUSTRY_0_DOT_0_FIRST_LEVEL_CATEGORY ！
> 
> 
> [DATE_DIMENSION_TABLE_0_DOT_0_DT_DATE, DATE_DIMENSION_TABLE_0_DOT_0_DT_DAY, 
> DATE_DIMENSION_TABLE_0_DOT_0_DT_MONTH, 
> DATE_DIMENSION_TABLE_0_DOT_0_DT_SEASON, DATE_DIMENSION_TABLE_0_DOT_0_DT_YEAR, 
> RZP_INDUSTRY_0_DOT_0_BOX_QUANTITY, RZP_INDUSTRY_0_DOT_0_BRAND, 
> RZP_INDUSTRY_0_DOT_0_COLLECTION_NUM, RZP_INDUSTRY_0_DOT_0_DATA_MONTH, 
> RZP_INDUSTRY_0_DOT_0_FEATURE_CALCIUM, 
> RZP_INDUSTRY_0_DOT_0_FEATURE_CUSTOMER_GROUP, 
> RZP_INDUSTRY_0_DOT_0_FEATURE_FAT, RZP_INDUSTRY_0_DOT_0_FEATURE_IMPORTED, 
> RZP_INDUSTRY_0_DOT_0_FEATURE_IRON_ZINC, RZP_INDUSTRY_0_DOT_0_FEATURE_ORGANIC, 
> RZP_INDUSTRY_0_DOT_0_FEATURE_PLACE_OF_PRODUCTION, 
> RZP_INDUSTRY_0_DOT_0_FEATURE_PREBIOTICS, 
> RZP_INDUSTRY_0_DOT_0_FEATURE_PROTEIN, RZP_INDUSTRY_0_DOT_0_FEATURE_SUGAR, 
> RZP_INDUSTRY_0_DOT_0_FIRST_LEVEL_CATEGORY, 
> RZP_INDUSTRY_0_DOT_0_INNER_PACKAGE, RZP_INDUSTRY_0_DOT_0_ONLINE_CHANEL, 
> RZP_INDUSTRY_0_DOT_0_OUTER_PACKAGE, RZP_INDUSTRY_0_DOT_0_PACKING_QUANTITY, 
> RZP_INDUSTRY_0_DOT_0_PRICE_SEGMENT, 
> RZP_INDUSTRY_0_DOT_0_PRODUCT_FIRST_PICTURE, 
> RZP_INDUSTRY_0_DOT_0_PRODUCT_LINK, RZP_INDUSTRY_0_DOT_0_PRODUCT_TITLE, 
> RZP_INDUSTRY_0_DOT_0_SALES_VALUE, RZP_INDUSTRY_0_DOT_0_SALES_VOLUME, 
> RZP_INDUSTRY_0_DOT_0_SALES_VOLUME_PRODUCT, 
> RZP_INDUSTRY_0_DOT_0_SECOND_LEVEL_CATEGORY, RZP_INDUSTRY_0_DOT_0_SERIES, 
> RZP_INDUSTRY_0_DOT_0_SHOP_FANS_NUM, RZP_INDUSTRY_0_DOT_0_SHOP_LINK, 
> RZP_INDUSTRY_0_DOT_0_SHOP_NAME, RZP_INDUSTRY_0_DOT_0_SHOP_TYPE, 
> RZP_INDUSTRY_0_DOT_0_SINGLE_PACKAGE_CAPACITY, 
> RZP_INDUSTRY_0_DOT_0_SINGLE_PACKAGE_PRICE, RZP_INDUSTRY_0_DOT_0_SKU, 
> RZP_INDUSTRY_0_DOT_0_SKU_CURRENT_PRICE, 
> RZP_INDUSTRY_0_DOT_0_SKU_ORIGINAL_PRICE, 
> RZP_INDUSTRY_0_DOT_0_SPECIFICATION_UNIT_PRICE, 
> RZP_INDUSTRY_0_DOT_0_SPEC_FLAVOUR]; line 1 pos 0;
> 'Filter ('first_level_category = 奶粉)
> 如果从以上数组中确实是找不到first_level_category字段。
> 
> 期待kylin团队，帮助我们解决该问题。感谢！
> 
> zhouzhih...@bxtdata.com

[jira] [Created] (KYLIN-5166) Use SQL hint to specify the cube that the user wants to query

[Yaqian Zhang (Jira)]

Yaqian Zhang created KYLIN-5166:
---

 Summary: Use SQL hint to specify the cube that the user wants to 
query
 Key: KYLIN-5166
 URL: https://issues.apache.org/jira/browse/KYLIN-5166
 Project: Kylin
  Issue Type: Improvement
Reporter: Yaqian Zhang
Assignee: Yaqian Zhang






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

[jira] [Created] (KYLIN-5162) Fix non-select sql pushdown

[Yaqian Zhang (Jira)]

Yaqian Zhang created KYLIN-5162:
---

 Summary: Fix non-select sql pushdown
 Key: KYLIN-5162
 URL: https://issues.apache.org/jira/browse/KYLIN-5162
 Project: Kylin
  Issue Type: Bug
Reporter: Yaqian Zhang
Assignee: Yaqian Zhang






--
This message was sent by Atlassian Jira
(v8.20.1#820001)

Re: How to install Kylin3 on independent Hbase2

[Yaqian Zhang]

Hi,

I want to confirm that you have tried to put the `hive-site.xm`l of HDP1 under 
directory `/etc/hive/conf`  in HDP2, and still encounter this error?

If you put the `hive-site.xml` of HDP1 in directory `/etc/hive/conf` in HDP2, 
Kylin will read the `hive-site.xml` of HDP1 when reading the Hadoop 
configuration, and the `LOCATION` here will use the HDFS directory configured 
in the `hive-site.xml` of HDP1.

> 在 2022年1月11日，下午4:55，Meng, Lingsheng  写道：
> 
> Hi,
> 
> I’m trying to create 2 Kylin clusters (primary + secondary).
> The 2 clusters use the same data source, but for the secondary cluster, I 
> want to user another Hbase installation.
> 
> I did the following changes compared with the primary:
> 
>  *   in .bashrc: changed the HBASE_HOME
>  *   in kylin.properties: define ` 
> kylin.storage.hbase.cluster-fs=hdfs://newhbase-cluster:8080`
> 
> After starting the kylin in secondary cluster, from the start up log, it 
> always trys to connect to the zookeeper of new Hbase.
> 
> Then I tried to build a cube, the log shows the hive command like
> “””
> ……
> 
> CREATE EXTERNAL TABLE IF NOT EXISTS 
> kylin_intermediate_test_cart_add_cube_0ed9de6e_80eb_1d90_dd3f_9ef85ddd8941
> 
> (
> 
> \`CART_ADD_USER_KEY\` string
> 
> )
> 
> STORED AS SEQUENCEFILE
> 
> LOCATION 
> 'hdfs://new_hbase_cluster_ns/tmp/aipl/kylin/aipl-kylin_metadata/kylin-6db3c2d5-49f0-dee4-adee-45f689d05460/kylin_intermediate_test_cart_add_cube_0ed9de6e_80eb_1d90_dd3f_9ef85ddd8941'
> 
> ;
> ……
> ”””
> Obviously, I got the error
> 
> Error: Error while compiling statement: FAILED: SemanticException 
> java.lang.IllegalArgumentException: java.net.UnknownHostException: 
> new_hbase_cluster_ns (state=42000,code=4)
> 
> My Env :
> Kylin: apache-kylin-3.1.2-bin-hadoop3
> Hadoop: 3.1
> Hbase: 2.0
> 
> Could you help me solve this issue?
> 
> Thank you very much!
> 
> 
> Best regards,
> 
> Lingsheng Meng | Forest | CNTD
> 
> 

Re: How to install Kylin3 on independent Hbase2

[Yaqian Zhang]

Hi lingsheng.meng:

You can refer to this blog for deploy Kylin with standalone HBase cluster : 
https://kylin.apache.org/blog/2016/06/10/standalone-hbase-cluster/

> 在 2022年1月11日，下午4:55，Meng, Lingsheng  写道：
> 
> Hi,
> 
> I’m trying to create 2 Kylin clusters (primary + secondary).
> The 2 clusters use the same data source, but for the secondary cluster, I 
> want to user another Hbase installation.
> 
> I did the following changes compared with the primary:
> 
>  *   in .bashrc: changed the HBASE_HOME
>  *   in kylin.properties: define ` 
> kylin.storage.hbase.cluster-fs=hdfs://newhbase-cluster:8080`
> 
> After starting the kylin in secondary cluster, from the start up log, it 
> always trys to connect to the zookeeper of new Hbase.
> 
> Then I tried to build a cube, the log shows the hive command like
> “””
> ……
> 
> CREATE EXTERNAL TABLE IF NOT EXISTS 
> kylin_intermediate_test_cart_add_cube_0ed9de6e_80eb_1d90_dd3f_9ef85ddd8941
> 
> (
> 
> \`CART_ADD_USER_KEY\` string
> 
> )
> 
> STORED AS SEQUENCEFILE
> 
> LOCATION 
> 'hdfs://new_hbase_cluster_ns/tmp/aipl/kylin/aipl-kylin_metadata/kylin-6db3c2d5-49f0-dee4-adee-45f689d05460/kylin_intermediate_test_cart_add_cube_0ed9de6e_80eb_1d90_dd3f_9ef85ddd8941'
> 
> ;
> ……
> ”””
> Obviously, I got the error
> 
> Error: Error while compiling statement: FAILED: SemanticException 
> java.lang.IllegalArgumentException: java.net.UnknownHostException: 
> new_hbase_cluster_ns (state=42000,code=4)
> 
> My Env :
> Kylin: apache-kylin-3.1.2-bin-hadoop3
> Hadoop: 3.1
> Hbase: 2.0
> 
> Could you help me solve this issue?
> 
> Thank you very much!
> 
> 
> Best regards,
> 
> Lingsheng Meng | Forest | CNTD
> 
> 

Re: [VOTE] Release apache-kylin-3.1.3 (RC1)

[Yaqian Zhang]

+1.

mvn clean test passed.

Happy path passed on cdh5.7, cdh6.3 and hdp2.4.

> 在 2021年12月30日，下午6:04，Xiaoxiang Yu  写道：
> 
> Hi all, I have created a build for Apache Kylin 3.1.3, release candidate 1.
> 
> 
> Changes highlights:
> KYLIN-4947 Implement spark engine for cube optimization jobs
> KYLIN-4948 Provide an API to allow users to adjust cuboids manually
> KYLIN-4971 Add new measure bitmap_map for count distinct measure in UI
> Thanks to everyone who has contributed to this release. Here are the release 
> notes:
> 
> 
> 
> The commit to being voted upon:
> 
> Its hash is fb07e6b1bf0f716090f3b73004c5afe0096b9be.
> 
> 
> The artifacts to be voted on, including the source package and four 
> pre-compiled binary packages are located here:
> 
> 
> 
> The hash of the artifacts are as follows:
> 
> 
> apache-kylin-3.1.3-source-release.zip.sha256
> 5da0cde33ae99899a40ead30cfd890315e8f5619b9ed4dae9239f1991c968d9d
> 
> 
> apache-kylin-3.1.3-bin-cdh57.tar.gz.sha256
> e8186eedda200db9420a7e439e74193e7a87f30a9c7938ec0250aab96ff5c048
> 
> 
> apache-kylin-3.1.3-bin-cdh60.tar.gz.sha256
> ad1544d1dbdd3637394376ce8a975a50eb78aef8e25f0e325a0570c40acb4347
> 
> 
> apache-kylin-3.1.3-bin-hadoop3.tar.gz.sha256
> 22ec20b232b50b379c01570bea08f3c02287bbbe666ece982fe5dd75c3c8bcde
> apache-kylin-3.1.3-bin-hbase1x.tar.gz.sha256
> 6f893cb3634461f856dd12bb174b3215bb8ee73eed4ea6efc697130479453182
> 
> 
> 
> 
> A staged Maven repository is available for review at:
> 
> 
> 
> Release artifacts are signed with the following key:
> 
> 
> 
> 
> 
> Please vote on releasing this package as Apache Kylin 3.1.3.
> 
> 
> The vote is open for the next 72 hours and passes if a majority of at least 
> three +1 binding votes are cast.
> [ ] +1 Release this package as Apache Kylin 3.1.3
> [ ] 0 I don't feel strongly about it, but I'm okay with the release
> [ ] -1 Do not release this package because...
> Here is my vote:
> +1 (binding)
> 
> --
> 
> Best wishes to you ! 
> From ：Xiaoxiang Yu

Re: [VOTE] Release apache-kylin-4.0.1 (RC1)

[Yaqian Zhang]

+1.

mvn clean test passed.

Happy path passed on CDH5.7 and CDH6.3.

> 在 2021年12月30日，下午5:17，Xiaoxiang Yu  写道：
> 
> Hi all, I have created a build for Apache Kylin 4.0.1, release candidate 1.
> 
> 
> Changes highlights:
> KYLIN-5067 CubeBuildJob build unnecessary snapshot
> KYLIN-5069 Remove find-hive-dependency.sh in Kylin 4
> KYLIN-5111 Record the time spent for each stage of query in kylin4's log
> KYLIN-5117 Support percentile function after aggregate sub query
> 
> 
> Thanks to everyone who has contributed to this release. Here are the release 
> notes:
> 
> 
> 
> The commit to being voted upon:
> 
> Its hash is 337026c97d47d0325c3eeefe9fd2f2bebdb1eb18.
> 
> 
> 
> 
> The artifacts to be voted on, including the source package and two
> pre-compiled binary packages are located here:
> 
> 
> 
> 
> 
> The hash of the artifacts are as follows:
> 
> 
> apache-kylin-4.0.1-source-release.zip.sha256
> 6b9671dbab337f7000768da552bb62473ebf919d1f9d12c5db02273eea2a9130
> 
> 
> apache-kylin-4.0.1-bin-spark2.tar.gz.sha256
> d680630ebc8aba6f6db8b06cab210ff7b1b79212554613cd0279077245b81c95
> 
> 
> apache-kylin-4.0.1-bin-spark3.tar.gz.sha256
> 94a897d52b6b556263d33dfb8b09db4556109b0a6c908c38b158283ac8110656
> 
> 
> 
> 
> A staged Maven repository is available for review at:
> 
> 
> 
> Release artifacts are signed with the following key:
> 
> 
> 
> 
> 
> 
> 
> Please vote on releasing this package as Apache Kylin 4.0.1.
> The vote is open for the next 72 hours and passes if a majority of at least 
> three +1 binding votes are cast.
> [ ] +1 Release this package as Apache Kylin 4.0.1
> [ ] 0 I don't feel strongly about it, but I'm okay with the release
> [ ] -1 Do not release this package because...
> Here is my vote:
> +1 (binding)
> 
> 
> 
> --
> 
> Best wishes to you ! 
> From ：Xiaoxiang Yu

Re: Apache Kylin - 3.1.3 Release - Stop/Cancel Query is not working for long queries.

[Yaqian Zhang]

Hi Sonu:

Thank you for your suggestion!

I think "upgrade this stop query functionality” is a good idea，those queries 
that run too long should be detected and stopped automatically.

Can you open a JIRA issue to track this? And you can write your thoughts on the 
description of the issue.

> 在 2021年12月23日，下午12:27，Singh Sonu  写道：
> 
> Hi Zhang,
> 
> I am calling this API using swagger and yes after the API call, I can see the 
> related logs got printed.
> 
> I am just thinking can we upgrade this stop query functionality? so we can 
> kill the running query thread because sometimes these queries are running in 
> HBase for 100-300 secs.
> 
> Thanks for your suggestions, I will surely going to try these all using the 
> property file.
> 
> 
>  You can reach me out at
>  Email- sonusingh.javat...@gmail.com <mailto:sonusingh.javat...@gmail.com>
> 
>  with regards,
>  Sonu Kumar Singh
>  
> 
> 
> 
> On Tue, Dec 21, 2021 at 12:21 PM Yaqian Zhang  <mailto:yaqian_zh...@126.com>> wrote:
> Hi Sonu Kumar Singh:
> 
> Can you describe in detail how you call the Stop Query API?  And after 
> calling the API,  whether relevant logs are output in 
> $KYLIN_HOME/logs/kylin.log? such as "the query: {} is not existing" or 
> "{quaryid} stopped by {user}".
> 
> By the way, you can set some query related parameters to prevent HBase 
> cluster from downtime due to excessive query execution. such as 
> kylin.query.max-scan-bytes, kylin.storage.partition.max-scan-bytes, etc.
> For details: 
> https://kylin.apache.org/cn/docs31/install/configuration.html#kylin-query 
> <https://kylin.apache.org/cn/docs31/install/configuration.html#kylin-query>.
> 
>> 在 2021年12月20日，下午10:33，Singh Sonu > <mailto:sonusingh.javat...@gmail.com>> 写道：
>> 
>> Hi Experts,
>> 
>> I am trying to kill long-running bad queries from Stop Query API and it's
>> not working.
>> Also, long-running bad queries are killing my HBasea and Kylin is down
>> frequently. I want to understand and need help from you, how we can kill
>> these kind of bad queries?
>> 
>> 
>> You can reach me out at
>> Email- sonusingh.javat...@gmail.com <mailto:sonusingh.javat...@gmail.com>
>> 
>> with regards,
>> Sonu Kumar Singh
> 

Re: Apache Kylin - 3.1.3 Release - Stop/Cancel Query is not working for long queries.

[Yaqian Zhang]

Hi Sonu Kumar Singh:

Can you describe in detail how you call the Stop Query API?  And after calling 
the API,  whether relevant logs are output in $KYLIN_HOME/logs/kylin.log? such 
as "the query: {} is not existing" or "{quaryid} stopped by {user}".

By the way, you can set some query related parameters to prevent HBase cluster 
from downtime due to excessive query execution. such as 
kylin.query.max-scan-bytes, kylin.storage.partition.max-scan-bytes, etc.
For details: 
https://kylin.apache.org/cn/docs31/install/configuration.html#kylin-query 
.

> 在 2021年12月20日，下午10:33，Singh Sonu  写道：
> 
> Hi Experts,
> 
> I am trying to kill long-running bad queries from Stop Query API and it's
> not working.
> Also, long-running bad queries are killing my HBasea and Kylin is down
> frequently. I want to understand and need help from you, how we can kill
> these kind of bad queries?
> 
> 
> You can reach me out at
> Email- sonusingh.javat...@gmail.com
> 
> with regards,
> Sonu Kumar Singh

[Kylin Security Notice] Impact analysis of Apache Log4j2 Remote Code Execution Vulnerability

[Yaqian Zhang]

Hi all:

This is a security notice about the impact analysis of Apache Log4j2 Remote 
Code Execution Vulnerability on Apache Kylin.
Background

Apache Log4j2 is a Java based logging tool, which is widely used in the 
industry. The recently discovered Remote Code Execution Vulnerability of Apache 
Log4j2 makes it possible for the program that introduces Apache Log4j2 to be 
triggered Remote Code Execution by an attacker who construct a special request.
Scope of influence

The version range of Log4j2 with security vulnerabilities is: Apache Log4j 2.x 
<= 2.14.1.
The currently released versions of Apache Kylin (Kylin 2.x, Kylin 3.x, Kylin 
4.x) use log4j version 1.2.17 by default. However, considering that kylin's 
startup script will load jars from Hadoop environment, including Hadoop, Spark, 
HBase, Hive and other components, the log4j version used in Hadoop3 environment 
is generally Apache Log4j2, so if your Hadoop is above version 3.0, it is 
recommended to upgrade the Log4j2  of Hadoop cluster, to avoid the possibility 
of polluting kylin services.
Solution 

If the Hadoop component used by kylin user's environment uses Log4j2, the user 
needs to comprehensively upgrade Log4j2 to the latest 2.15.0-rc2 to prevent 
Kylin from loading the jar of Log4j2 with security risks into Kylin's classpath 
through scripts.
After the log4j2 environment is fully upgraded, users can execute jinfo `cat 
pid` under $KYLIN_HOME to check whether the jar packages such as 
log4j-core-2.x.x.jar introduced by Kylin's classpath are the latest secure 
Log4j2 versions.


Best Regards!

Apache Kylin Team

Re: kylin v3.1.2启动失败

[Yaqian Zhang]

Hi:

What is your Hadoop environment version? It seems to be an environmental 
problem. You can first check whether your HBase service is normal.

> 在 2021年11月23日，下午2:55，qzhhz3  写道：
> 
> 你好！
>  kylinV3.1.2启动异常，排查未能解决，请教下大家的解决办法，问题如下：
> 2021-11-23 14:31:56,068 INFO  
> [ReadOnlyZKClient-localhost:2181@0x68ad99fe-SendThread(localhost:2181)] 
> zookeeper.ClientCnxn:1299 : Session establishment complete on server 
> localhost/0:0:0:0:0:0:0:1:2181, sessionid = 0x17d4b291b03, negotiated 
> timeout = 4
> 
> Exception in thread "main" java.lang.IllegalArgumentException: Failed to find 
> metadata store by url: kylin_metadata@hbase
> 
>at 
> org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:101)
> 
>at 
> org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:113)
> 
>at 
> org.apache.kylin.rest.service.AclTableMigrationTool.checkIfNeedMigrate(AclTableMigrationTool.java:99)
> 
>at 
> org.apache.kylin.tool.AclTableMigrationCLI.main(AclTableMigrationCLI.java:43)
> 
> Caused by: java.lang.reflect.InvocationTargetException
> 
>at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native 
> Method)
> 
>at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> 
>at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> 
>at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> 
>at 
> org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:94)
> 
>... 3 more
> 
> Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed 
> after attempts=4, exceptions:
> 
> Tue Nov 23 14:31:57 CST 2021, 
> RpcRetryingCaller{globalStartTime=1637649116370, pause=100, maxAttempts=4}, 
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after 
> attempts=4, exceptions:
> 
> Tue Nov 23 14:31:56 CST 2021, 
> RpcRetryingCaller{globalStartTime=1637649116414, pause=100, maxAttempts=4}, 
> org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException:
>  Message missing required fields: server
> 
> Tue Nov 23 14:31:56 CST 2021, 
> RpcRetryingCaller{globalStartTime=1637649116414, pause=100, maxAttempts=4}, 
> org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException:
>  Message missing required fields: server
> 
> Tue Nov 23 14:31:56 CST 2021, 
> RpcRetryingCaller{globalStartTime=1637649116414, pause=100, maxAttempts=4}, 
> org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException:
>  Message missing required fields: server
> 
> Tue Nov 23 14:31:57 CST 2021, 
> RpcRetryingCaller{globalStartTime=1637649116414, pause=100, maxAttempts=4}, 
> org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException:
>  Message missing required fields: server
> 
> 
> 
> 
> 
> 
> Tue Nov 23 14:31:59 CST 2021, 
> RpcRetryingCaller{globalStartTime=1637649118973, pause=100, maxAttempts=4}, 
> org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException:
>  Message missing required fields: server
> 
> 
> 
> 
>at 
> org.apache.hadoop.hbase.client.RpcRetryingCallerImpl.callWithRetries(RpcRetryingCallerImpl.java:146)
>at 
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3084)
>at 
> org.apache.hadoop.hbase.client.HBaseAdmin.executeCallable(HBaseAdmin.java:3076)
>at 
> org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:442)
>at 
> org.apache.kylin.storage.hbase.HBaseConnection.tableExists(HBaseConnection.java:291)
>at 
> org.apache.kylin.storage.hbase.HBaseConnection.createHTableIfNeeded(HBaseConnection.java:316)
>at 
> org.apache.kylin.storage.hbase.HBaseResourceStore.createHTableIfNeeded(HBaseResourceStore.java:121)
>at 
> org.apache.kylin.storage.hbase.HBaseResourceStore.(HBaseResourceStore.java:89)
>... 8 more
> Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed 
> after attempts=4, exceptions:
> Tue Nov 23 14:31:58 CST 2021, 
> RpcRetryingCaller{globalStartTime=1637649118973, pause=100, maxAttempts=4}, 
> org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException:
>  Message missing required fields: server
> Tue Nov 23 14:31:59 CST 2021, 
> RpcRetryingCaller{globalStartTime=1637649118973, pause=100, maxAttempts=4}, 
> org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException:
>  Message missing required fields: server
> Tue Nov 23 14:31:59 CST 2021, 
> RpcRetryingCaller{globalStartTime=1637649118973, pause=100, maxAttempts=4}, 
> org.apache.hbase.thirdparty.com.google.protobuf.InvalidProtocolBufferException:
>  Message missing required fields: server
> Tue Nov 23 14:31:59 CST 2021, 
> RpcRetryingCaller{globalStartTime=1637649118973, pause=100, maxAttempts=4}, 
> 

Re: An error is reported when connecting kylin (get schemas) through JDBC

[Yaqian Zhang]

Thank you for your feedback!

If there is any follow-up progress, welcome to discuss with community.

> 在 2021年11月2日，下午4:21，赵尉余  写道：
> 
> Hi，
> After many attempts, the picture still fails to be sent. This problem is 
> ignored temporarily. Thank you for your support
> 
> 
> --Original--
> From: "赵尉余" Date: Tue, Nov 2, 2021 04:18 PM
> To: "dev" 
> Subject: Re: An error is reported when connecting kylin (get schemas) 
> through JDBC
> 
> 
> 
> Add picture.
> 
> 
> -- Original --
> From: "赵尉余" Date: Tue, Nov 2, 2021 04:15 PM
> To: "dev" 
> Subject: Re: An error is reported when connecting kylin (get schemas) 
> through JDBC
> 
> 
> 
> Add
> 
> 
> 
> 
> 
> -- Original --
> From: "Yaqian Zhang" Date: Tue, Nov 2, 2021 03:40 PM
> To: "dev" 
> Subject: Re: An error is reported when connecting kylin (get schemas) 
> through JDBC
> 
> 
> 
> Hi：
> 
> I can't see your picture. It doesn't seem to display normally.
> 
> I don't know what action was taken during the test. Maybe it finished the 
> test without using project.
> 
>  在 2021年11月2日，下午3:20，赵尉余   
>  Hi，
>  My URL is filled with the project name, but it is a wrong project. At 
> present, the schame can be obtained by modifying the correct project name.
>  Thank you very much for your help！
>  
>  A new problem, the wrong project name can also pass the test?
>  
>  
>  
>  -- Original --
>  From: "Yaqian Zhang"  Date: Tue, Nov 2, 2021 02:36 PM
>  To: "dev"  Subject: Re: An error is reported when connecting kylin (get 
> schemas) through JDBC
>  
>  Hi:
>  
>  I checked the source code and found that the NullPointer may be caused 
> by kylin's project name not being set correctly.
>  
>  I wonder if you provided the 'project name' when connecting to 
> kylin through smartBI?
>  
>   在 2021年11月2日，上午10:39，zhaowe...@smartbi.com.cn 写道：
>   
>   Hi，
> I reported an error when I 
> connected kylin (get schemas) through JDBC during the test. What is the 
> reason?
> kylin：
> 
>   
>   
> error message
> Failed to get the list of 
> modes
>   at 
> smartbi.freequery.basicdata.MetaDataServiceImpl.getSchemas(MetaDataServiceImpl.java:525)
>   at ...(...)
>   at 
> smartbi.freequery.basicdata.SmartbiKYLINDialect.getSchemas(SmartbiKYLINDialect.java:32)
>   Caused by: java.lang.RuntimeException: java.io.IOException: GET 
> failed, error code 500 and response: 
> {"code":"999","data":null,"msg":null,"stacktrace":"java.lang.NullPointerException\n\tat
>  
> org.apache.kylin.query.schema.OLAPSchemaFactory.createTempOLAPJson(OLAPSchemaFactory.java:67)\n\tat
>  
> org.apache.kylin.query.QueryConnection.getConnection(QueryConnection.java:49)\n\tat
>  
> org.apache.kylin.rest.service.QueryService.getMetadata(QueryService.java:775)\n\tat
>  
> org.apache.kylin.rest.service.QueryService.getMetadata(QueryService.java:215)\n\tat
>  
> org.apache.kylin.rest.service.QueryService.getMetadataFilterByUser(QueryService.java:211)\n\tat
>  
> org.apache.kylin.rest.controller.QueryController.getMetadata(QueryController.java:193)\n\tat
>  sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat
>  
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat
>  java.lang.reflect.Method.invoke(Method.java:498)\n\tat 
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)\n\tat
>  
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)\n\tat
>  
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)\n\tat
>  
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:849)\n\tat
>  
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:760)\n\tat
>  
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)\n\tat
>  
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)\n\tat
>  
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)

Re: An error is reported when connecting kylin (get schemas) through JDBC

[Yaqian Zhang]

Hi：

I can't see your picture. It doesn't seem to display normally.

I don't know what action was taken during the test. Maybe it finished the test 
without using project.

> 在 2021年11月2日，下午3:20，赵尉余  写道：
> 
> Hi，
> My URL is filled with the project name, but it is a wrong project. At 
> present, the schame can be obtained by modifying the correct project name.
> Thank you very much for your help！
> 
> A new problem, the wrong project name can also pass the test?
> 
>  
>  
> -- Original --
> From:  "Yaqian Zhang";
> Date:  Tue, Nov 2, 2021 02:36 PM
> To:  "dev";
> Subject:  Re: An error is reported when connecting kylin (get schemas) 
> through JDBC
>  
> Hi:
> 
> I checked the source code and found that the NullPointer may be caused by 
> kylin's project name not being set correctly.
> 
>  I wonder if you provided the 'project name' when connecting to kylin through 
> smartBI?
> 
> > 在 2021年11月2日，上午10:39，zhaowe...@smartbi.com.cn 写道：
> > 
> > Hi，
> > I reported an error when I connected kylin (get schemas) through 
> > JDBC during the test. What is the reason?
> > kylin：
> > 
> > 
> > 
> > error message
> > Failed to get the list of modes
> > at 
> > smartbi.freequery.basicdata.MetaDataServiceImpl.getSchemas(MetaDataServiceImpl.java:525)
> > at ...(...)
> > at 
> > smartbi.freequery.basicdata.SmartbiKYLINDialect.getSchemas(SmartbiKYLINDialect.java:32)
> > Caused by: java.lang.RuntimeException: java.io.IOException: GET failed, 
> > error code 500 and response: 
> > {"code":"999","data":null,"msg":null,"stacktrace":"java.lang.NullPointerException\n\tat
> >  
> > org.apache.kylin.query.schema.OLAPSchemaFactory.createTempOLAPJson(OLAPSchemaFactory.java:67)\n\tat
> >  
> > org.apache.kylin.query.QueryConnection.getConnection(QueryConnection.java:49)\n\tat
> >  
> > org.apache.kylin.rest.service.QueryService.getMetadata(QueryService.java:775)\n\tat
> >  
> > org.apache.kylin.rest.service.QueryService.getMetadata(QueryService.java:215)\n\tat
> >  
> > org.apache.kylin.rest.service.QueryService.getMetadataFilterByUser(QueryService.java:211)\n\tat
> >  
> > org.apache.kylin.rest.controller.QueryController.getMetadata(QueryController.java:193)\n\tat
> >  sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat 
> > sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat
> >  
> > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat
> >  java.lang.reflect.Method.invoke(Method.java:498)\n\tat 
> > org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)\n\tat
> >  
> > org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)\n\tat
> >  
> > org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)\n\tat
> >  
> > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:849)\n\tat
> >  
> > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:760)\n\tat
> >  
> > org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)\n\tat
> >  
> > org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)\n\tat
> >  
> > org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)\n\tat
> >  
> > org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)\n\tat
> >  
> > org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)\n\tat
> >  javax.servlet.http.HttpServlet.service(HttpServlet.java:634)\n\tat 
> > org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)\n\tat
> >  javax.servlet.http.HttpServlet.service(HttpServlet.java:741)\n\tat 
> > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)\n\tat
> >  
> > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n\tat
> >  
> > org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)\n\tat
> >  
> > org.apache.catalina.core.ApplicationFilterChain.internalDo

Re: An error is reported when connecting kylin (get schemas) through JDBC

[Yaqian Zhang]

Hi:

I checked the source code and found that the NullPointer may be caused by 
kylin's project name not being set correctly.

 I wonder if you provided the 'project name' when connecting to kylin through 
smartBI?

> 在 2021年11月2日，上午10:39，zhaowe...@smartbi.com.cn 写道：
> 
> Hi，
> I reported an error when I connected kylin (get schemas) through JDBC 
> during the test. What is the reason?
> kylin：
> 
> 
> 
> error message
> Failed to get the list of modes
>   at 
> smartbi.freequery.basicdata.MetaDataServiceImpl.getSchemas(MetaDataServiceImpl.java:525)
>   at ...(...)
>   at 
> smartbi.freequery.basicdata.SmartbiKYLINDialect.getSchemas(SmartbiKYLINDialect.java:32)
> Caused by: java.lang.RuntimeException: java.io.IOException: GET failed, error 
> code 500 and response: 
> {"code":"999","data":null,"msg":null,"stacktrace":"java.lang.NullPointerException\n\tat
>  
> org.apache.kylin.query.schema.OLAPSchemaFactory.createTempOLAPJson(OLAPSchemaFactory.java:67)\n\tat
>  
> org.apache.kylin.query.QueryConnection.getConnection(QueryConnection.java:49)\n\tat
>  
> org.apache.kylin.rest.service.QueryService.getMetadata(QueryService.java:775)\n\tat
>  
> org.apache.kylin.rest.service.QueryService.getMetadata(QueryService.java:215)\n\tat
>  
> org.apache.kylin.rest.service.QueryService.getMetadataFilterByUser(QueryService.java:211)\n\tat
>  
> org.apache.kylin.rest.controller.QueryController.getMetadata(QueryController.java:193)\n\tat
>  sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat
>  
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat
>  java.lang.reflect.Method.invoke(Method.java:498)\n\tat 
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)\n\tat
>  
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)\n\tat
>  
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)\n\tat
>  
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:849)\n\tat
>  
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:760)\n\tat
>  
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)\n\tat
>  
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)\n\tat
>  
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)\n\tat
>  
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)\n\tat
>  
> org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:861)\n\tat
>  javax.servlet.http.HttpServlet.service(HttpServlet.java:634)\n\tat 
> org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)\n\tat
>  javax.servlet.http.HttpServlet.service(HttpServlet.java:741)\n\tat 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)\n\tat
>  
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n\tat
>  org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)\n\tat 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)\n\tat
>  
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n\tat
>  
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)\n\tat
>  
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)\n\tat
>  
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)\n\tat
>  
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)\n\tat
>  
> org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)\n\tat
>  
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)\n\tat
>  
> org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)\n\tat
>  
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)\n\tat
>  
> org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)\n\tat
>  
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)\n\tat
>  
> 

[jira] [Created] (KYLIN-5112) Use 'return' instead of 'exit' in script prepare_hadoop_dependency.sh

[Yaqian Zhang (Jira)]

Yaqian Zhang created KYLIN-5112:
---

 Summary: Use 'return' instead of 'exit' in script 
prepare_hadoop_dependency.sh
 Key: KYLIN-5112
 URL: https://issues.apache.org/jira/browse/KYLIN-5112
 Project: Kylin
  Issue Type: Bug
Reporter: Yaqian Zhang






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KYLIN-5111) Record the time spent for each stage of query in kylin4's log

[Yaqian Zhang (Jira)]

Yaqian Zhang created KYLIN-5111:
---

 Summary: Record the time spent for each stage of query in kylin4's 
log
 Key: KYLIN-5111
 URL: https://issues.apache.org/jira/browse/KYLIN-5111
 Project: Kylin
  Issue Type: Improvement
  Components: Metrics
Affects Versions: v4.0.0
Reporter: Yaqian Zhang
Assignee: Yaqian Zhang
 Fix For: v4.0.1






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[New Blog] Performance optimization of Kylin 4.0 in cloud: Local cache and soft affinity scheduling

[Yaqian Zhang]

Hi,

We have released a new blog on Apache Kylin’s official website: Performance 
optimization of Kylin 4.0 in cloud -- local cache and soft affinity scheduling, 
which is shared by Yaqian Zhang, R engineer from Kyligence and Apache Kylin 
committer.

This blog introduces how the kylin development team uses local cache and soft 
affinity scheduling to improve the performance of Kylin4.0 using object storage 
in the cloud, and shows the performance test results. Interested users please 
check: 
https://kylin.apache.org/blog/2021/10/21/Local-Cache-and-Soft-Affinity-Scheduling/
 
<https://kylin.apache.org/blog/2021/10/21/Local-Cache-and-Soft-Affinity-Scheduling/>


Best Regards！

[jira] [Created] (KYLIN-5090) The error message was not printed in class JobWoker

[Yaqian Zhang (Jira)]

Yaqian Zhang created KYLIN-5090:
---

 Summary: The error message was not printed in class JobWoker
 Key: KYLIN-5090
 URL: https://issues.apache.org/jira/browse/KYLIN-5090
 Project: Kylin
  Issue Type: Improvement
Affects Versions: v4.0.0
Reporter: Yaqian Zhang
Assignee: Yaqian Zhang
 Fix For: v4.0.1


In the execute method in JobWorker, after the error is thrown, the error 
details are not printed, but called the method for handling the error. When the 
method for handling the error is executed, other errors are generated, 
resulting in the real error information being hidden.

{code:java}
private def execute(): Unit = {
pool.execute(new Runnable {
  override def run(): Unit = {
try {
  logInfo("Start running job.")
  application.execute(args)
  eventLoop.post(JobSucceeded())
} catch {
  case exception: NoRetryException => 
eventLoop.post(UnknownThrowable(exception))
  case throwable: Throwable => eventLoop.post(ResourceLack(throwable))
}
  }
})
  }
{code}




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Use Beeline for Hive on Apache Kylin 4.0.0 ?

[Yaqian Zhang]

Hi:

Kylin4 use spark engine to build cube, and spark application are submitted 
through spark-submit, which has nothing to do with hive cli or beeline.. 

Therefore, this configuration of  “kylin.source.hive.client"  has no effect in 
kylin 4.

> 在 2021年9月16日，下午4:19，MAMERI Jamal - externe  
> 写道：
> 
> Hello,
> 
> I hope my email finds you well.
> 
> I have installed the Apache Kylin 4.0.0 version.
> 
> I want to use Beeline as Hive client but the following properties aren't 
> existing in the file : KYLIN_HOME/conf/kylin.properties :
> 
> 
> ## Hive client, valid value [cli, beeline]
> kylin.source.hive.client=beeline
> #
> ## Absolute path to beeline shell, can be set to spark beeline instead of the 
> default hive beeline on PATH
> kylin.source.hive.beeline-shell=/apache/hive/bin/beeline
> #
> ## Parameters for beeline client, only necessary if hive client is beeline
> kylin.source.hive.beeline-params=
> 
> 
> Is it just an oversight?
> Can we use the Beeline client for Hive with the new version 4.0.0 ?
> Or only Hive client (cli) is available for this version ?
> 
> Thanks,
> 
> Best Regards,
> 
> Jamal Mameri

Re: java.lang.ClassNotFoundException: org.apache.hadoop.hive.metastore.api.NoSuchObjectException

[Yaqian Zhang]

Hi：
You can test with the Kylin binary for Hadoop 3.1 + HBase 2.0, and the version 
seems to be compatible. 
However, we have only tested on Hadoop platforms such as HDP or CDH. We 
recommend using HDP or CDH to deploy Hadoop.

> 在 2021年9月10日，上午10:11，赵晓亮 <18664561...@163.com> 写道：
> 
> yes，I use spark3 to do the build. I will lower the spark version to try.Can 
> you help me see if there is a problem with other component versions except 
> spark? Thanks！
> 
> 
> kylin-3.1.2-bin-hadoop3
> jdk- 1.8.0_201
> hadoop-3.1.4
> hbase-2.4.5
> scala-2.12.12
> spark-3.0.1
> hive-3.1.2
> zookeeper-3.6.2
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 在 2021-09-07 10:49:06，"Yaqian Zhang"  写道：
>> Hi:
>> Does the build use spark engine? 
>> We only tested the spark engine of kylin3 on spark 2.4. There is a big gap 
>> between spark 3 and spark 2, and there may be a version compatibility problem
>> 
>>> 在 2021年9月7日，上午10:36，赵晓亮 <18664561...@163.com> 写道：
>>> 
>>> When I build Kylin environment, I met "java.lang.ClassNotFoundException: 
>>> org.apache.hadoop.hive.metastore.api.NoSuchObjectException" error. The 
>>> components versions are as follows：
>>> | Component | Version |
>>> |
>>> jdk
>>> |
>>> 1.8.0_201
>>> |
>>> |
>>> hadoop
>>> |
>>> 3.1.4
>>> |
>>> |
>>> hbase
>>> |
>>> 2.4.5
>>> |
>>> |
>>> scala
>>> |
>>> 2.12.12
>>> |
>>> |
>>> spark
>>> |
>>> 3.0.1
>>> |
>>> |
>>> hive
>>> |
>>> 3.1.2
>>> |
>>> |
>>> zookeeper
>>> |
>>> 3.6.2
>>> |
>>> |
>>> kylin
>>> |
>>> 3.1.2-bin-hadoop3
>>> |
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> Is it a version matching problem？ Does anyone known why？
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 
>>> 

Re: java.lang.ClassNotFoundException: org.apache.hadoop.hive.metastore.api.NoSuchObjectException

[Yaqian Zhang]

Hi:
Does the build use spark engine? 
We only tested the spark engine of kylin3 on spark 2.4. There is a big gap 
between spark 3 and spark 2, and there may be a version compatibility problem

> 在 2021年9月7日，上午10:36，赵晓亮 <18664561...@163.com> 写道：
> 
> When I build Kylin environment, I met "java.lang.ClassNotFoundException: 
> org.apache.hadoop.hive.metastore.api.NoSuchObjectException" error. The 
> components versions are as follows：
> | Component | Version |
> |
> jdk
> |
> 1.8.0_201
> |
> |
> hadoop
> |
> 3.1.4
> |
> |
> hbase
> |
> 2.4.5
> |
> |
> scala
> |
> 2.12.12
> |
> |
> spark
> |
> 3.0.1
> |
> |
> hive
> |
> 3.1.2
> |
> |
> zookeeper
> |
> 3.6.2
> |
> |
> kylin
> |
> 3.1.2-bin-hadoop3
> |
> 
> 
> 
> 
> 
> 
> Is it a version matching problem？ Does anyone known why？
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 
> 

[jira] [Created] (KYLIN-5076) Missing tracking URL in spark standalone mode in kylin4

[Yaqian Zhang (Jira)]

Yaqian Zhang created KYLIN-5076:
---

 Summary: Missing tracking URL in spark standalone mode in kylin4
 Key: KYLIN-5076
 URL: https://issues.apache.org/jira/browse/KYLIN-5076
 Project: Kylin
  Issue Type: Improvement
Affects Versions: v4.0.0
Reporter: Yaqian Zhang
Assignee: Yaqian Zhang
 Fix For: v4.0.1






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Release apache-kylin-4.0.0 (RC1)

[Yaqian Zhang]

+1
mvn clean test -skipRunIT=false passed
Happy path passed on CDH5.7

> 在 2021年8月25日，上午9:58，Zhichao Zhang  写道：
> 
> +1 from my side.
> 
> Best regards,
> Zhichao Zhang
> 
> On 2021/08/24 11:54:46, "Xiaoxiang Yu"  wrote: 
>> Hi all,
>> 
>> I have created a build for Apache Kylin 4.0.0, release candidate 1.
>> 
>> Changes highlights:
>> 
>> 
>> 
>> 
>> KYLIN-4903 cache parent datasource to accelerate next layer's cuboid building
>> 
>> KYLIN-4905 Support limit .. offset ... in spark query engine
>> 
>> KYLIN-4925 Use Spark 3 as build and query engine for Kylin 4
>> 
>> KYLIN-5011 Detect and scatter skewed data in dict encoding step
>> 
>> KYLIN-4948 Provide an API to allow users to adjust cuboids manually
>> 
>> KYLIN-5027 Add the config of whether to build base cuboid in kylin4
>> 
>> KYLIN-5019 Avoid building global dictionary from all data of fact table each 
>> time
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> Thanks to everyone who has contributed to this release.
>> 
>> Here are the release notes:
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> The commit to being voted upon:
>> 
>> 
>> 
>> Its hash is aa6d582c0a96fdf31e85791f86850cfc00a7644f.
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> The artifacts to be voted on, including the source package and two
>> 
>> pre-compiled binary packages are located here:
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> The hash of the artifacts are as follows:
>> 
>> 
>> 
>> 
>> apache-kylin-4.0.0-source-release.zip.sha256
>> 
>> 788d0402756dad6e735d6b435b94e3c67f2b3c730f38f8c9b4d71bc66dd71703
>> 
>> 
>> 
>> 
>> apache-kylin-4.0.0-bin-spark2.tar.gz.sha256
>> 
>> 748c4916489e1e014eae81fe7e0cc72ef564573eb5882969b53feb2ac7ab9379
>> 
>> 
>> 
>> 
>> apache-kylin-4.0.0-bin-spark3.tar.gz.sha256
>> 
>> 2378c066ff990b70f9a5f4ee32f56ed4a4d86afc88f5986dfbb05fd8f9cc5946
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> A staged Maven repository is available for review at:
>> 
>> 
>> 
>> 
>> 
>> 
>> Release artifacts are signed with the following key:
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>> Please vote on releasing this package as Apache Kylin 4.0.0 .
>> 
>> 
>> 
>> 
>> The vote is open for the next 72 hours and passes if a majority of
>> 
>> at least three +1 binding votes are cast.
>> 
>> 
>> 
>> 
>> [ ] +1 Release this package as Apache Kylin 3.1.2
>> 
>> [ ]  0 I don't feel strongly about it, but I'm okay with the release
>> 
>> [ ] -1 Do not release this package because...
>> 
>> 
>> 
>> 
>> Here is my vote:
>> 
>> +1 (binding)
>> 
>> --
>> 
>> Best wishes to you ! 
>> From ：Xiaoxiang Yu

Re: Using direct Query on Apache Kylin

[Yaqian Zhang]

Hi:

The error message does not looks like be reported by kylin. You can check 
kylin.log to see if there is any error message, or confirm whether the 
connection between kylin and powerbi is normal.

> 在 2021年8月20日，下午4:39，EL-KARMA Zakaria - externe 
>  写道：
> 
> 

[New Blog] How Meituan Dominates Online Shopping with Apache Kylin

[Yaqian Zhang]

Hi，

We have released a new blog on Apache kylin's official website: How Meituan 
Dominates Online Shopping with Apache Kylin, which is shared by Wang Yueqing,  
R engineer from Meituan.

This blog summarizes some experiences in using Apache kylin as an OLAP engine 
for users' reference:
https://kylin.apache.org/blog/2021/08/03/How-Meituan-Dominates-Online-Shopping-with-Apache-Kylin/
 
<https://kylin.apache.org/blog/2021/08/03/How-Meituan-Dominates-Online-Shopping-with-Apache-Kylin/>


Thanks to Wang Yueqing!

Best Regards!
Yaqian Zhang 张亚倩
Apache Kylin committer

[New Blog] Apache Kylin4 — A new storage and compute architecture

[Yaqian Zhang]

Hi，

We have released a new blog on Apache kylin's official website —— Apache Kylin4 
— A new storage and compute architecture, which is shared by Tao Jiatao.

For users interested in kylin4's new architecture:  
https://kylin.apache.org/blog/2021/07/02/Apache-Kylin4-A-new-storage-and-compute-architecture/

Thanks to Tao jiatao!

Best Regards!
Yaqian Zhang 张亚倩
Apache Kylin committer

Re: help me thank u

[Yaqian Zhang]

Maybe you can cherry-pick the follow commit to your source code:
https://github.com/apache/kylin/commit/c94374569a5534a92ab305898698a3943c89c7ef

> 在 2021年8月3日，上午10:15，Lihong,Fu  写道：
> 
> version： kylin-3.0.x-hadoop3.1]
> build command：build/script/package.sh -P cdh60
> 
> when down
> https://github.com/johannestroeger/bower-angular-unstable.git#1.1.5
> need input username and password. but not my name and password.
> 
> could you tar your 3.0 and 4.0-b version's  webapp/node_modules  directory 
> for me? thank you.
> 
> 
> -- 原始邮件 --
> 发件人: "dev" <598593...@qq.com.INVALID>;
> 发送时间: 2021年8月2日(星期一) 晚上11:32
> 收件人: "dev";
> 主题: Re:回复： help me thank u
> 
> 您好：您使用源代码的哪个分支来编译？

Re: help me thank u

[Yaqian Zhang]

Hi:

You need to use “git config” to config your git user info.

> 在 2021年8月2日，下午3:19，Lihong,Fu  写道：
> 
> when compile kylin version v3.1 or v4 ,need input user name and password ,I 
> can not passed.
> Could you help me ? thank u . hints as following:
> 
> 
> bower validate   1.2.15 against 
> https://github.com/angular/bower-angular.git#1.2.15
> Username for 'https://github.com':

[jira] [Created] (KYLIN-5040) Cuboid should not be exactly matched when there is no group by time partition column and there are multiple segments in the query

[Yaqian Zhang (Jira)]

Yaqian Zhang created KYLIN-5040:
---

 Summary: Cuboid should not be exactly matched when there is no 
group by time partition column and there are multiple segments in the query
 Key: KYLIN-5040
 URL: https://issues.apache.org/jira/browse/KYLIN-5040
 Project: Kylin
  Issue Type: Bug
  Components: Query Engine
Reporter: Yaqian Zhang
Assignee: Yaqian Zhang
 Fix For: v4.0.0


When there is no group by time partition column and there are multiple segments 
in the query， data of multiple segments need to be aggregated.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KYLIN-5036) The hive dependency of directory type is filtered out by mistake

[Yaqian Zhang (Jira)]

Yaqian Zhang created KYLIN-5036:
---

 Summary: The hive dependency of directory type is filtered out by 
mistake
 Key: KYLIN-5036
 URL: https://issues.apache.org/jira/browse/KYLIN-5036
 Project: Kylin
  Issue Type: Bug
Reporter: Yaqian Zhang
Assignee: Yaqian Zhang
 Fix For: v3.1.3






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KYLIN-5027) Add the config of whether to build base cuboid in kylin4

[Yaqian Zhang (Jira)]

Yaqian Zhang created KYLIN-5027:
---

 Summary: Add the config of whether to build base cuboid in kylin4
 Key: KYLIN-5027
 URL: https://issues.apache.org/jira/browse/KYLIN-5027
 Project: Kylin
  Issue Type: Improvement
Reporter: Yaqian Zhang
Assignee: Yaqian Zhang
 Fix For: v4.0.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KYLIN-5014) Spark driver log is abnormal in yarn cluster mode

[Yaqian Zhang (Jira)]

Yaqian Zhang created KYLIN-5014:
---

 Summary: Spark driver log is abnormal in yarn cluster mode
 Key: KYLIN-5014
 URL: https://issues.apache.org/jira/browse/KYLIN-5014
 Project: Kylin
  Issue Type: Bug
Reporter: Yaqian Zhang
Assignee: Yaqian Zhang
 Fix For: v4.0.0






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KYLIN-5005) Optimize segment is not cleaned up when OPTIMIZE CHECKPOINT job be discarded

[Yaqian Zhang (Jira)]

Yaqian Zhang created KYLIN-5005:
---

 Summary: Optimize segment is not cleaned up when OPTIMIZE 
CHECKPOINT job be discarded
 Key: KYLIN-5005
 URL: https://issues.apache.org/jira/browse/KYLIN-5005
 Project: Kylin
  Issue Type: Bug
  Components: Others
Reporter: Yaqian Zhang
Assignee: Yaqian Zhang






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KYLIN-4996) Merge cuboid statistics in merge job for kylin4

[Yaqian Zhang (Jira)]

Yaqian Zhang created KYLIN-4996:
---

 Summary: Merge cuboid statistics in merge job for kylin4
 Key: KYLIN-4996
 URL: https://issues.apache.org/jira/browse/KYLIN-4996
 Project: Kylin
  Issue Type: Task
  Components: Job Engine
Reporter: Yaqian Zhang
Assignee: Yaqian Zhang






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: cannot Get HiveTableMeta - GUI Error when importing Hive internal tables

[Yaqian Zhang]

Hi:

You can confirm that you are using the correct kylin binary package. For HDP3 
environment, the kylin package is:apache-kylin-3.1.1-bin-hadoop3.tar.gz 


> 在 2021年5月17日，22:12，Ottaviano, Francesco  写道：
> 
> I get "cannot Get HiveTableMeta" error in GUI when I load Hive internal 
> tables into my Kylin project. External tables are correctly imported.
> 
> I configured Kylin for HDP environment and I created a Kylin project. But 
> when I try to load Hive tables from UI (Model -> Data Source -> Load Table) I 
> get pop-up window with "cannot Get HiveTableMeta" error. How can I import 
> Hive internal tables in Kylin project?
> Environment versions:
> ·   HDP-3.1.0.277 (Hive-3.0.0.3.1; HBase-2.0.0.3.1; Zookeeper-3.4.9.3.1)
> ·   Kylin-3.1.1
>  
>  
> 2021-05-17 07:58:54,174 ERROR [http-nio-7070-exec-2] 
> controller.TableController:132 : Failed to load Hive Table 
> java.lang.RuntimeException: cannot get HiveTableMeta at 
> org.apache.kylin.source.hive.HiveMetadataExplorer.loadTableMetadata(HiveMetadataExplorer.java:68)
>  at 
> org.apache.kylin.rest.service.TableService.extractHiveTableMeta(TableService.java:260)
>  at 
> org.apache.kylin.rest.service.TableService.loadHiveTablesToProject(TableService.java:186)
>  at 
> org.apache.kylin.rest.controller.TableController.loadHiveTables(TableController.java:116)
>  at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at 
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) 
> at 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
>  at java.lang.reflect.Method.invoke(Method.java:498) at 
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
>  at 
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
>  at 
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
>  at 
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:849)
>  at 
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:760)
>  at 
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
>  at 
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
>  at 
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
>  at 
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
>  at 
> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
>  at javax.servlet.http.HttpServlet.service(HttpServlet.java:660) at 
> org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
>  at javax.servlet.http.HttpServlet.service(HttpServlet.java:741) at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
>  at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
>  at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52) at 
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
>  at 
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
>  at 
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)
>  at 
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)
>  at 
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)
>  at 
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>  at 
> org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
>  at 
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>  at 
> org.springframework.security.web.session.SessionManagementFilter.doFilter(SessionManagementFilter.java:137)
>  at 
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>  at 
> org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
>  at 
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
>  at 
> org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:170)
>  at 
> 

Re: Error while creating a cube on Spark engine

[Yaqian Zhang]

Hi：

It seems that your picture didn't load successfully. You can provide the error 
log as text.

> 在 2021年4月22日，18:23，Gourav Gupta  写道：
> 
> Hi Team,
>  
> Hope you are all doing great in this tough time!!
>  
> We are approaching here for a seek of help regarding the Kylin. My team has 
> been stuck in one of the issues related to Kerberos security, although, we 
> are not incorporating any services related to security like a Kerberos but 
> still getting the issue when I create the cube in Spark mode. The MapReduce 
> mode is working fine, through that we were able to create the cube. The error 
> is mentioned below :
>  
>  
> 
>  
> Looking for a positive response from your side.
>  
> Regards,
> Gourav

Re: Active cube run set to pending and a new cube request is run without finishing

[Yaqian Zhang]

Hi:

In my opinion, what you are describing is a normal phenomenon.

When executing the cube build job, kylin will add the task of each step in the 
job into a map. The FetcherRunner will periodically fetch the tasks that can be 
executed from the map and add them to the JobPool. If it is found that the 
number of jobs in JobPool has reached the configuration value 
("kylin.job.max-concurrent-jobs"), then this fetcher will return directly and 
wait for the fetcher in the next cycle.

That is to say, the task fetched by the FetcherRunner is a step task of a cube 
build job, which may belong to the previous job submitted by you, or the latter 
job submitted by you, so these build jobs may be executed alternately.

For related knowledge, you can refer to kylin's job scheduler related articles: 
https://blog.bcmeng.com/post/kylin-job.html 




> 在 2021年4月23日，12:50，xatax  写道：
> 
> I have obeserved that when the "kylin.job.max-concurrent-jobs" properties is
> used for a cluster configuration and set to for e.g. lets say value 1. Then
> for a given cube building job cluster, during a cube build if another cube
> build request is submitted the currently running job is set to pending and
> Kylin starts running the new job on the same cluster. Then again it might
> pause the running job and any step and then return to the execution of the
> first job. This is usually observed after the first step but can be random.
> 
> Is this expected behavior? 
> 
> The good thing is ultimately both jobs run to success.
> 
> Please share any knowledge on this.
> 
> thanks!
> 
> --
> Sent from: http://apache-kylin.74782.x6.nabble.com/

Re: [VOTE] Release apache-kylin-3.1.2 (RC1)

[Yaqian Zhang]

+1
mvn clean test passed
Happy path passed on CDH5.7 and CDH6.2

> 在 2021年4月21日，09:08，Zhichao Zhang <441586...@qq.com> 写道：
> 
> + 1 from myside , tested passed on HDP 2.
> 
> Best regards,
> Zhichao Zhang

[jira] [Created] (KYLIN-4979) Fix flink shaded jar version error in download-flink.sh

[Yaqian Zhang (Jira)]

Yaqian Zhang created KYLIN-4979:
---

 Summary: Fix flink shaded jar version error in download-flink.sh
 Key: KYLIN-4979
 URL: https://issues.apache.org/jira/browse/KYLIN-4979
 Project: Kylin
  Issue Type: Bug
Affects Versions: v3.1.1
Reporter: Yaqian Zhang
Assignee: Yaqian Zhang
 Fix For: v3.1.2






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KYLIN-4966) Refresh the existing segment according to the new cuboid list in kylin4

[Yaqian Zhang (Jira)]

Yaqian Zhang created KYLIN-4966:
---

 Summary: Refresh the existing segment according to the new cuboid 
list in kylin4
 Key: KYLIN-4966
 URL: https://issues.apache.org/jira/browse/KYLIN-4966
 Project: Kylin
  Issue Type: New Feature
  Components: Job Engine
Affects Versions: v4.0.0-beta
Reporter: Yaqian Zhang
Assignee: Yaqian Zhang
 Fix For: v4.0.0-GA


When the user wants to update the cuboid list of a cube, the existing segment 
needs to be refreshed.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: lambda streaming cube question

[Yaqian Zhang]

Hi:
Do you find only streaming data based on the query results? 
You can provide some related log, if only the phenomenon description is 
difficult to analyze this problem.

> 在 2021年3月9日，11:41，344955119 <344955...@qq.com> 写道：
> 
> Hi,I just followed thelambda streaming cube at 
> http://kylin.apache.org/docs30/tutorial/lambda_mode_and_timezone_realtime_olap.html
>  website.And I finally did not get history data merged with streaming.It is 
> like just streaming cube without lambda now.May you give me some suggestion?

Re: Kylin Benchmarking

[Yaqian Zhang]

Hi:
Here is the project of kylin benchmark using SSB data set, for 
reference:https://github.com/Kyligence/ssb-kylin 
.

> 在 2021年2月25日，20:05，Vishal Chavhan  写道：
> 
> Hi,
> 
> I am trying to build the cube on 450 millions of source data which has 19
> dimensions and 5 meansures. cube buidling is taking long time and never
> ends so just wanted to check  if there is any benchmarking doc available
> around this?
> 
> thanks

Re: Question about normal cube merge streaming cube

[Yaqian Zhang]

There is no Chinese version of this document at present. 
You should only be able to read the English version in a short time. hahaha

> 在 2021年2月26日，09:53，344955119 <344955...@qq.com> 写道：
> 
> Got that,I'll try it later.
> By theway,I user to always browser the document in chinese at kylin's office 
> site which didn't write about the section you just told.
> Maybe chinese part will not upgrade anymone?HaHaHa.
> Thanks for suggesting!
> 
> 
> 
> 
> 
> --原始邮件--
> 发件人:  
>   "dev"   
>  
>  发送时间:2021年2月25日(星期四) 上午10:29
> 收件人:"dev" 
> 主题:Re: Question about normal cube merge streaming cube
> 
> 
> 
> Hi：
> Maybe you can try lambda mode in kylin real-time, which supports adding 
> history table and real-time data source in a model. 
> Reference 
> documents：http://kylin.apache.org/docs/tutorial/lambda_mode_and_timezone_realtime_olap.html
>  
>  
>  在 2021年2月23日，16:52，344955119 <344955...@qq.com 写道：
>  
>  Hi,
>  
>  
>  My group is looking for a way to concat a streaming cube (realtime data) 
> with a normal cube (history data).May be it's about the 
> buttonnbsp;"merge"nbsp;in visable interface ,but I found it cann't 
> work as "merge" only supports cubes in the same model and they need to be 
> from the same source like table "table" basicly.
>  In the other side, someone suggect to usenbsp;system 
> withnbsp;flinknbsp;to make it complie.
>  So may you give me a suggest,whether there is a way to solve this 
> problem in kylin or I should turn to system withnbsp;flink?
>  nbsp;
>  Thanks,
>  William

[jira] [Created] (KYLIN-4917) Fix some problem of logger system in kylin4

[Yaqian Zhang (Jira)]

Yaqian Zhang created KYLIN-4917:
---

 Summary: Fix some problem of logger system in kylin4
 Key: KYLIN-4917
 URL: https://issues.apache.org/jira/browse/KYLIN-4917
 Project: Kylin
  Issue Type: Improvement
  Components: Others
Affects Versions: v4.0.0-beta
Reporter: Yaqian Zhang
Assignee: Yaqian Zhang
 Fix For: v4.0.0-GA






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Question about normal cube merge streaming cube

[Yaqian Zhang]

Hi：
Maybe you can try lambda mode in kylin real-time, which supports adding history 
table and real-time data source in a model. 
Reference 
documents：http://kylin.apache.org/docs/tutorial/lambda_mode_and_timezone_realtime_olap.html
 


> 在 2021年2月23日，16:52，344955119 <344955...@qq.com> 写道：
> 
> Hi,
> 
> 
> My group is looking for a way to concat a streaming cube (realtime data) with 
> a normal cube (history data).May be it's about the 
> button"merge"in visable interface ,but I found it cann't work as 
> "merge" only supports cubes in the same model and they need to be from the 
> same source like table "table" basicly.
> In the other side, someone suggect to usesystem withflinkto 
> make it complie.
> So may you give me a suggest,whether there is a way to solve this problem in 
> kylin or I should turn to system withflink?
> 
> Thanks,
> William

Re: Cube migration across hadoop clusters

[Yaqian Zhang]

Hi:
There are three ways of cube migration in the document of cube migration. Which 
one do you use? And what's your kylin version? Is the Hadoop version used by QA 
cluster and dev cluster consistent? Are there any errors reported during 
migration?

> 在 2021年2月25日，03:04，xatax  写道：
> 
> Per: http://kylin.apache.org/docs/tutorial/cube_migration.html
> 
> It says: "Support to migrate cube data (segments data on HBase) from QA env
> to PROD env."
> 
> I tried the migration but it only migrates the cube metadata without the
> data. The source kylin instance has the cube buiilt and in ready state.
> After I executed the command it said it was successful. But when I checked
> the destination kylin instance it only got the cube definition and not the
> data.
> 
> Can you please clarify if migrating the data is possible or not?
> 
> Or is the migration of the data a separate activity which involves the hbase
> snapshot import export.
> 
> thanks in advance
> 
> --
> Sent from: http://apache-kylin.74782.x6.nabble.com/

Re: Cluster mode - High availability. Metadata storage options other than MySQL?

[Yaqian Zhang]

Hi:
At present, kylin only supports MySQL and HBase as metadata storage.

> 在 2021年2月22日，01:19，xatax  写道：
> 
> thanks!
> What about something like dynamo db?
> 
> --
> Sent from: http://apache-kylin.74782.x6.nabble.com/

Re: Cluster mode kylin.metadata.url and kylin.server.cluster-servers examples for AWS EMR

[Yaqian Zhang]

Hi:
kylin.metadata.url  can be configured as table_name@hbase ,table_name is the 
name of a hbase table. If the table does not exist in HBase, kylin will create 
it.

kylin.server.cluster-server is a comma separated list of all kylin nodes in the 
cluster, such as node1:7070,node2:7070.

In fact, http://kylin.apache.org/ can provide you with more information.

> 在 2021年2月3日，21:53，xatax  写道：
> 
> Hi 
> I am trying to install Kylin in cluster mode on AWS EMRs using the
> documentation: http://kylin.apache.org/docs/install/kylin_cluster.html
> 
> Can you please help me with example values of kylin.metadata.url and
> kylin.server.cluster-servers so I know the format.
> 
> thanks!
> 
> --
> Sent from: http://apache-kylin.74782.x6.nabble.com/

[jira] [Created] (KYLIN-4898) Add automated test cases

[Yaqian Zhang (Jira)]

Yaqian Zhang created KYLIN-4898:
---

 Summary: Add automated test cases
 Key: KYLIN-4898
 URL: https://issues.apache.org/jira/browse/KYLIN-4898
 Project: Kylin
  Issue Type: Improvement
  Components: Others
Affects Versions: v4.0.0-alpha
Reporter: Yaqian Zhang
Assignee: Yaqian Zhang
 Fix For: Future






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Release apache-kylin-4.0.0-beta (RC2)

[Yaqian Zhang]

+1
Mvn clean test passed.
Happy path on EMR5.x passed.

> 在 2021年2月3日，21:31，Xiaoxiang Yu  写道：
> 
> Hi all,
> 
> 
> 
> 
> I have created a build for Apache Kylin 4.0.0-beta, release candidate 2. 
> Please note that this release is built on kylin-on-parquet-v2 branch.
> 
> 
> 
> 
> Changes highlights:
> 
> 
> 
> 
> [KYLIN-4857] - Refactor system cube for kylin4
> 
> [KYLIN-4842] - Supports grouping sets function for Kylin 4
> 
> [KYLIN-4829] - Support to use thread-level SparkSession to execute query
> 
> [KYLIN-4813] - Refine spark logger for Kylin 4 build engine
> 
> [KYLIN-4858] - Support Kylin4 deployment on CDH 6.X
> 
> [KYLIN-4818] - Calculate cuboid statistics in Kylin 4
> 
> [KYLIN-4817] - Refine Cube Migration Tool for Kylin4
> 
> 
> 
> 
> Thanks to everyone who has contributed to this release.
> 
> Here are the release notes:
> 
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316121=12348723
> 
> 
> 
> 
> The commit to being voted upon:
> 
> https://github.com/apache/kylin/commit/546381392c55136310f591969e5a3f3db6074988
> 
> Its hash is 546381392c55136310f591969e5a3f3db6074988.
> 
> 
> 
> 
> The artifacts to be voted on, including the source package and one 
> pre-compiled binary packages are located here:
> 
> https://dist.apache.org/repos/dist/dev/kylin/apache-kylin-4.0.0-beta-rc2/
> 
> 
> 
> 
> The hash of the artifacts are as follows:
> 
> apache-kylin-4.0.0-beta-source-release.zip.sha256
> 
> d5837ed106c4e10f71a6b7a6324520afdd536c10f5afd86bf6c98d734a3e64a8
> 
> 
> 
> 
> apache-kylin-4.0.0-beta-bin.tar.gz.sha256
> 
> e86ab855533104c2eac8fae53dc067d425ab263cebf558c666260a3ceb336b16
> 
> 
> 
> 
> 
> 
> 
> A staged Maven repository is available for review at:
> 
> https://repository.apache.org/content/repositories/orgapachekylin-1087/
> 
> 
> 
> 
> 
> 
> 
> Release artifacts are signed with the following key:
> 
> https://people.apache.org/keys/committer/xxyu.asc
> 
> 
> 
> 
> Please vote on releasing this package as Apache Kylin 4.0.0-beta.
> 
> 
> 
> 
> The vote is open for the next 72 hours and passes if a majority of
> 
> at least three +1 binding votes are cast.
> 
> 
> 
> 
> [ ] +1 Release this package as Apache Kylin 4.0.0-beta
> 
> [ ]  0 I don't feel strongly about it, but I'm okay with the release
> 
> [ ] -1 Do not release this package because...
> 
> 
> 
> 
> Here is my vote:
> 
> +1 (binding)
> --
> Best wishes to you ! 
> From ：Xiaoxiang Yu

[jira] [Created] (KYLIN-4897) Add table snapshot and global dictionary cleaning in StorageCleanupJob

[Yaqian Zhang (Jira)]

Yaqian Zhang created KYLIN-4897:
---

 Summary: Add table snapshot and global dictionary cleaning in 
StorageCleanupJob
 Key: KYLIN-4897
 URL: https://issues.apache.org/jira/browse/KYLIN-4897
 Project: Kylin
  Issue Type: Improvement
  Components: Client - CLI
Affects Versions: v4.0.0-alpha, v4.0.0-beta
Reporter: Yaqian Zhang
Assignee: Yaqian Zhang
 Fix For: v4.0.0-GA






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Kylin4.0-beta

[Yaqian Zhang]

Hi:
The release of kylin 4.0 beta is in progress  and is expected to be released in 
about a week.

> 在 2021年2月3日，18:29，wujunxi <462329...@qq.com> 写道：
> 
> 你好，请问kylin4.0的beta版本什么时候发布？很急希望可以得到回复

Re: Kylin4.0 HDP3.1

[Yaqian Zhang]

Hi:
We haven't tested kylin4 in the environment of hdp3.1, but we have tested 
kylin4 in other Hadoop3 environments such as CDH6 and EMR6 and formed 
documents, which can provide some references for you. 

You can check 
https://cwiki.apache.org/confluence/display/KYLIN/Deploy+Kylin+4+on+AWS+EMR 
here.

You can try and verify in your own hdp3.1 environment according to the similar 
idea of installing EMR6 in the document.

If you have any progress, welcome to come back here to update your progress and 
share it with other users.  Thanks!

> 在 2021年2月2日，16:01，wujunxi <462329...@qq.com> 写道：
> 
> Hi kylin developers,
> I am going to compile kylin4.0 for HDP3.1,but I met many conflicts,how 
> to deal with and run in HDP3.1

Re: [VOTE] Release apache-kylin-4.0.0-beta (RC1)

[Yaqian Zhang]

+1
Mvn clean test passed.
Happy path passed on CDH6, EMR5, EMR6.


> 在 2021年1月31日，10:03，Zhichao Zhang <441586...@qq.com> 写道：
> 
> +1 from my side.
> 
> Test passed on HDP2, CDH 5.
> 
> 
> Best regards,
> Zhichao Zhang

Re: cube can't building in flink 1.11.3 / 1.10.0

[Yaqian Zhang]

Hi:

Kylin version 3.1.1 supports Flink 1.9.x.

If you want to use Flink 1.10.0 or 1.11, someone in the Kylin community 
recently proposed the PR  https://github.com/apache/kylin/pull/1462 to upgrade 
Flink.

You can type this patch and try again.

> 在 2021年1月25日，16:14，WSJian <526165...@qq.com> 写道：
> 
> ENV: Kylin 3.1.1 Flink 1.10.0 / 1.11.3 CDH 5.7
> 
> /etc/profile -- FLINK_HOME、KYLIN_HOME、HADOOP_CONF_DIR all down.
> 
> 
> Only Spark Env no pass(Because there isn't) when I start kylin.sh. Then build 
> `kylin_sales_cube` cube. But throw ERROR during Build Cube With Flink 
> process. 
> 
> 
> java.lang.NoSuchFieldError: MANAGED_MEMORY_PRE_ALLOCATE
>at 
> org.apache.kylin.engine.flink.FlinkOnYarnConfigMapping...
> 
> 
> I looked at the source code. Found 
> ```java
>ConfigOption TaskManagerOptions.MANAGED_MEMORY_PRE_ALLOCATE;
> ```
> So I think flink not start in Yarn success, and after start flink in Yarn.
> Throw new Exception.
> 
> 
> DefaultClassNotFound: FlinkOnYarnConfigMapping
> 
> 
> This is mean the class not load? Or classpath error. Need moving dependent 
> JAR to where?

[jira] [Created] (KYLIN-4874) Fix CubeMigrationCLI bug for kylin4

[Yaqian Zhang (Jira)]

Yaqian Zhang created KYLIN-4874:
---

 Summary: Fix CubeMigrationCLI bug for kylin4
 Key: KYLIN-4874
 URL: https://issues.apache.org/jira/browse/KYLIN-4874
 Project: Kylin
  Issue Type: Bug
  Components: Client - CLI
Reporter: Yaqian Zhang
Assignee: Yaqian Zhang
 Fix For: v4.0.0-beta


Solve the problem that the source cluster and the target cluster model are not 
in the same project and the project information of the model is not modified.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KYLIN-4857) Refactor system cube for kylin4

[Yaqian Zhang (Jira)]

Yaqian Zhang created KYLIN-4857:
---

 Summary: Refactor system cube for kylin4
 Key: KYLIN-4857
 URL: https://issues.apache.org/jira/browse/KYLIN-4857
 Project: Kylin
  Issue Type: Improvement
  Components: Metrics
Affects Versions: v4.0.0-alpha
Reporter: Yaqian Zhang
Assignee: Yaqian Zhang
 Fix For: v4.0.0-beta


With the change of query engine and storage, the query related metrics 
collected by system cube in kylin3 are no longer applicable to kylin4, so it 
needs to be refactor to adapt to the spark query engine in kylin4.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KYLIN-4845) Fix NFilePruningTest report dup key error

[Yaqian Zhang (Jira)]

Yaqian Zhang created KYLIN-4845:
---

 Summary: Fix NFilePruningTest report dup key error
 Key: KYLIN-4845
 URL: https://issues.apache.org/jira/browse/KYLIN-4845
 Project: Kylin
  Issue Type: Test
Reporter: Yaqian Zhang
Assignee: Yaqian Zhang
 Fix For: v4.0.0-beta






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KYLIN-4844) Add lookup table duplicate key check when building job

[Yaqian Zhang (Jira)]

Yaqian Zhang created KYLIN-4844:
---

 Summary: Add lookup table duplicate key check when building job
 Key: KYLIN-4844
 URL: https://issues.apache.org/jira/browse/KYLIN-4844
 Project: Kylin
  Issue Type: Improvement
Affects Versions: v4.0.0-alpha
Reporter: Yaqian Zhang
Assignee: Yaqian Zhang
 Fix For: v4.0.0-beta






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Apache Kylin - Tableau connectivity with MacOS

[Yaqian Zhang]

Hi Priyanshu Kumar:

Kylin ODBC driver does not support Mac. I don't think you can use Kyligence 
ODBC driver to replace it, because Apache kylin and kyligence Enterprise are 
different. 

If you want to use Tableau to connect to kylin on  Mac, you can use kylin JDBC 
driver. Kylin 3.0.0 supports use JDBC connection to tableau.
Here is a reference link to the related mail: 
http://apache-kylin.74782.x6.nabble.com/Kylin-and-Tableau-JDBC-td13471.html

> 在 2020年11月25日，02:30，Kumar, Priyanshu  写道：
> 
> Hi Team,
> 
> We are trying to connect to Kylin instance using Tableau installed on MacOS 
> and facing below error. Has anyone seen or faced this earlier?
> "An error occurred while communicating with Other Databases (ODBC)
> Unable to connect to the server. Check that the server is running and that 
> you have access privileges to the requested database.
> Error Code: 06DA155A
> [Simba][ODBC] (10380) Unable to establish connection with data source. 
> Missing settings: {[PROJECT]}
> Generic ODBC requires additional configuration. The driver and DSN (data 
> source name) must be installed and configured to match the connection.
> Unable to connect to the server "http://xx.xxx.xx.xx; using the driver 
> "KyligenceODBCDriver". Check that the server is running and that you have 
> access privileges to the requested database."
> 
> Note: Since Kylin ODBC driver was not available for Mac so we downloaded and 
> used Kyligence Mac ODBC driver.
> 
> Thanks & regards,
> Priyanshu Kumar
> This is a confidential email. Tesco may monitor and record all emails. The 
> views expressed in this email are those of the sender and not Tesco. Tesco 
> Stores Limited Company Number: 519500 Registered in England Registered 
> Office: Tesco House, Shire Park, Kestrel Way, Welwyn Garden City, AL7 1GA VAT 
> Registration Number: GB 220 4302 31

[jira] [Created] (KYLIN-4825) Add spark job tracking url in step details page

[Yaqian Zhang (Jira)]

Yaqian Zhang created KYLIN-4825:
---

 Summary: Add spark job tracking url in step details page
 Key: KYLIN-4825
 URL: https://issues.apache.org/jira/browse/KYLIN-4825
 Project: Kylin
  Issue Type: Improvement
Affects Versions: Future
Reporter: Yaqian Zhang
Assignee: Yaqian Zhang
 Fix For: v4.0.0-beta


KYLIN-4813 separate spark log from kylin log, therefore kylin can't get spark 
job tracking url from Kylin.log.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KYLIN-4808) Auto copy hive-site.xml to hadoop_conf_dir in kylin4

[Yaqian Zhang (Jira)]

Yaqian Zhang created KYLIN-4808:
---

 Summary: Auto copy hive-site.xml to hadoop_conf_dir in kylin4
 Key: KYLIN-4808
 URL: https://issues.apache.org/jira/browse/KYLIN-4808
 Project: Kylin
  Issue Type: Improvement
Affects Versions: v4.0.0-alpha
Reporter: Yaqian Zhang
Assignee: Yaqian Zhang
 Fix For: v4.0.0-beta


Kylin4 need hive-site.xml in hadoop_conf_dir when build cube.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [ANNOUNCE] new committer: Xiao Chu

[Yaqian Zhang]

Congratulations!

> 在 2020年11月5日，08:35，Zhichao Zhang <441586...@qq.com> 写道：
> 
> Congratulations

[jira] [Created] (KYLIN-4801) Automatic testing framework based on gauge

[Yaqian Zhang (Jira)]

Yaqian Zhang created KYLIN-4801:
---

 Summary: Automatic testing framework based on gauge
 Key: KYLIN-4801
 URL: https://issues.apache.org/jira/browse/KYLIN-4801
 Project: Kylin
  Issue Type: Sub-task
Affects Versions: all
Reporter: Yaqian Zhang
Assignee: Yaqian Zhang






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KYLIN-4800) Add canary tool for spader-context

[Yaqian Zhang (Jira)]

Yaqian Zhang created KYLIN-4800:
---

 Summary: Add canary tool for spader-context
 Key: KYLIN-4800
 URL: https://issues.apache.org/jira/browse/KYLIN-4800
 Project: Kylin
  Issue Type: Improvement
  Components: Query Engine
Affects Versions: v4.0.0-alpha
Reporter: Yaqian Zhang
Assignee: Yaqian Zhang
 Fix For: v4.0.0-beta






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KYLIN-4793) In some Hadoop versions, kylin calcite reported error because there is no guava14

[Yaqian Zhang (Jira)]

Yaqian Zhang created KYLIN-4793:
---

 Summary: In some Hadoop versions, kylin calcite reported error 
because there is no guava14
 Key: KYLIN-4793
 URL: https://issues.apache.org/jira/browse/KYLIN-4793
 Project: Kylin
  Issue Type: Bug
  Components: Query Engine
Affects Versions: Future
Reporter: Yaqian Zhang
Assignee: Yaqian Zhang
 Fix For: v4.0.0-beta


Commit 
https://github.com/apache/kylin/commit/e1fd472d00eea438d26c26f032e819db56e33486 
remove guava14 from Kylin.war，which causes kylin calcite reported error because 
there is no guava14 in some Hadoop versions.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KYLIN-4792) Verify whether several attribute values of segment are correct in the build/merge test

[Yaqian Zhang (Jira)]

Yaqian Zhang created KYLIN-4792:
---

 Summary: Verify whether several attribute values of segment are 
correct in the build/merge test
 Key: KYLIN-4792
 URL: https://issues.apache.org/jira/browse/KYLIN-4792
 Project: Kylin
  Issue Type: Improvement
  Components: Tools, Build and Test
Affects Versions: v4.0.0-alpha
Reporter: Yaqian Zhang
Assignee: Yaqian Zhang
 Fix For: v4.0.0-beta






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KYLIN-4790) Automaticly copy required jars to SPARK_HOME/jars for HDI3.6

[Yaqian Zhang (Jira)]

Yaqian Zhang created KYLIN-4790:
---

 Summary: Automaticly copy required jars to SPARK_HOME/jars for 
HDI3.6
 Key: KYLIN-4790
 URL: https://issues.apache.org/jira/browse/KYLIN-4790
 Project: Kylin
  Issue Type: Improvement
  Components: Others
Affects Versions: v4.0.0-alpha
Reporter: Yaqian Zhang
Assignee: Yaqian Zhang
 Fix For: v4.0.0-beta






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Release apache-kylin-3.1.1 (RC1)

[Yaqian Zhang]

+1
Happy path passed on CDH57.

> 在 2020年10月14日，23:21，Zhichao Zhang <441586...@qq.com> 写道：
> 
> +1 from my side.
> Happy path passed on HDP3.
> 
> Best regards,
> Zhichao Zhang

Re: new committer: Rupeng Wang

[Yaqian Zhang]

Contratulations!

> 在 2020年10月15日，08:11，Xiaoxiang Yu  写道：
> 
> Congrats to rupeng.
> 
> 
> | |
> 敏丞
> 邮箱：hit_la...@126.com
> |
> 
> 签名由 网易邮箱大师 定制
> 
> On 10/14/2020 22:25, ShaoFeng Shi wrote:
> The Project Management Committee (PMC) for Apache Kylin has invited Rupeng
> Wang (王汝鹏, wangrup...@apache.org) to become a committer and we are pleased
> to announce that he has accepted.
> 
> Being a committer enables easier contribution to the project since there is
> no need to go via the patch submission process. This should enable better
> productivity.
> 
> Congratulations, Rupeng!
> 
> Best regards,
> 
> Shaofeng Shi 史少锋
> Apache Kylin PMC
> Email: shaofeng...@apache.org
> 
> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> Join Kylin user mail group: user-subscr...@kylin.apache.org
> Join Kylin dev mail group: dev-subscr...@kylin.apache.org

[jira] [Created] (KYLIN-4787) The script sample.sh cannot automatically switch to the hive database set by the user to create sample hive tables

[Yaqian Zhang (Jira)]

Yaqian Zhang created KYLIN-4787:
---

 Summary: The script sample.sh cannot automatically switch to the 
hive database set by the user to create sample hive tables
 Key: KYLIN-4787
 URL: https://issues.apache.org/jira/browse/KYLIN-4787
 Project: Kylin
  Issue Type: Bug
  Components: Client - CLI
Affects Versions: v3.1.0, v3.1.1
 Environment: HDP3
Reporter: Yaqian Zhang
Assignee: Yaqian Zhang
 Fix For: v3.1.2


The script sample.sh use --database to specifies create_sample_ tables.sql 
execute in which database, but this parameter has been cancelled in hive3.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: New PMC: Xiaoxiang Yu

[Yaqian Zhang]

Congratulations to Xiaoxiang!

> 在 2020年10月9日，18:30，JiaTao Tao  写道：
> 
> Congratulations!
> 
> 
> Regards!
> 
> Aron Tao
> 
> 
> ShaoFeng Shi  于2020年10月9日周五 下午3:35写道：
> 
>> The Project Management Committee (PMC) for Apache Kylin
>> has invited Xiaoxiang Yu to become a PMC and we are pleased
>> to announce that he has accepted.
>> 
>> Xiaoxiang becomes the Kylin committer in late 2019; Since then, he plays a
>> leadership role in the Kylin team, and actively coaches other contributors,
>> including answer the questions, reviewing PRs, making the release,
>> maintaining documents, etc. He also gave many sharings in the community.
>> 
>> Being a PMC member enables assistance with the management
>> and to guide the direction of the project.
>> 
>> Congratulations to Xiaoxiang!
>> 
>> Best regards,
>> 
>> Shaofeng Shi 史少锋
>> Apache Kylin PMC
>> Email: shaofeng...@apache.org
>> 
>> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
>> Join Kylin user mail group: user-subscr...@kylin.apache.org
>> Join Kylin dev mail group: dev-subscr...@kylin.apache.org
>> 

[jira] [Created] (KYLIN-4766) After the job is discarded, the temporary file and segment file are not deleted

[Yaqian Zhang (Jira)]

Yaqian Zhang created KYLIN-4766:
---

 Summary: After the job is discarded, the temporary file and 
segment file are not deleted
 Key: KYLIN-4766
 URL: https://issues.apache.org/jira/browse/KYLIN-4766
 Project: Kylin
  Issue Type: Improvement
Affects Versions: v4.0.0-alpha
Reporter: Yaqian Zhang
Assignee: Yaqian Zhang
 Fix For: v4.0.0-beta






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Error while launching kylin v4.0.0-alpha

[Yaqian Zhang]

Great, it's my pleasure to see your steps in deploying kylin4.0 in AWS! I think 
many users would like to see these steps. 

Maybe you can write it on the Apache kylin 
wiki<https://cwiki.apache.org/confluence/display/KYLIN/Apache+Kylin+Home>. You 
can create a wiki account, and then we will give you permission to edit the 
wiki or add new pages. 

Of course, you can share it in any other way you like!

> 在 2020年9月11日，21:47，ShaoFeng Shi  写道：
> 
> Awesome! Glad to see the first evaluation feedback of Kylin 4!
> 
> Best regards,
> 
> Shaofeng Shi 史少锋
> Apache Kylin PMC
> Email: shaofeng...@apache.org
> 
> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> Join Kylin user mail group: user-subscr...@kylin.apache.org
> Join Kylin dev mail group: dev-subscr...@kylin.apache.org
> 
> 
> 
> 
> hubert stefani  于2020年9月11日周五 下午9:20写道：
> 
>> Thanks for your advice. We succeeded in getting The Web UI  running.
>> 
>> We have gathered the different configuration steps in a documentation we
>> will share soon if you want.
>> It may help to overcome the specific deployment of Kylin on aws
>> .Regards,hubert
>> 
>>Le vendredi 11 septembre 2020 à 10:44:26 UTC, Yaqian Zhang <
>> yaqian_zh...@126.com> a écrit :
>> 
>> If you wan’t to create a user and database and grant the roles for kylin
>> metadata. You can follow the steps below:
>> 
>> 1、Use root user login mysql: mysql -u root -pPassword
>> 2、create a user named Kylin and password is kylin: CREATE USER 'kylin'@'%'
>> IDENTIFIED BY 'kylin’;
>> 3、create database kylin：create database kylin;
>> 4、grant roles：GRANT ALL ON kylin.* TO 'kylin'@'%’;
>> 
>> Then you can use 
>> 'kylin_metadata_test@jdbc,url=jdbc:mysql://master:3306/kylin,username=kylin,password=kylin,maxActive=10,maxIdle=10
>> '
>> 
>> as kylin.metadata.url.
>> 
>>> 在 2020年9月11日，17:59，Yaqian Zhang  写道：
>>> 
>>> It seems that mysql have some problem, I find some related webpage
>> provide follow method to resolve this error:
>>> 
>>> Add below content in /etc/mysql/my.cnf:
>>> 
>>> wait_timeout=31536000
>>> interactive_timeout=31536000
>>> 
>>> Then restart mysql.
>>> 
>>> I’m not sure it works, you can try.
>>> 
>>>> 在 2020年9月11日，17:37，hubert stefani  写道：
>>>> 
>>>> we did so (we found out the  jar was actually missing), and then we got
>> the  error below. (additional information : we try to deploy on aws emr
>> 5.30)
>>>> 
>>>> com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications
>> link failure
>>>> 
>>>> The last packet sent successfully to the server was 0 milliseconds ago.
>> The driver has not received any packets from the server.
>>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>>>> at
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>>>> at
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>>>> at com.mysql.jdbc.Util.handleNewInstance(Util.java:408)
>>>> at
>> com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1137)
>>>> at com.mysql.jdbc.MysqlIO.(MysqlIO.java:356)
>>>> at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2504)
>>>> 
>>>> is there  a step to create the database, create users and grant roles ?
>>>> Hubert
>>>> Le vendredi 11 septembre 2020 à 09:30:27 UTC, Yaqian Zhang <
>> yaqian_zh...@126.com> a écrit :
>>>> 
>>>> Hi Hubert:
>>>> 
>>>> Have you put  mysql-connector-java-xxx.jar into $KYLIN_HOME/ext ?
>>>> 
>>>> You can download the mysql JDBC jar from
>> https://downloads.mysql.com/archives/c-j/ <
>> https://downloads.mysql.com/archives/c-j/>.
>>>> 
>>>>> 在 2020年9月11日，17:17，hubert stefani 
>> 写道：
>>>>> 
>>>>> Hi,
>>>>> we encounter the following error when we start the Web UI. it seems
>> that mysql-driver is missing. Have we made a misconfiguration  ? Is there a
>> detailed configuration ?
>>>>> 
>>>>> 
>>>>> Exception in thread "main" java.lang.IllegalArgumentException: Failed
>> to find metadata store by url: kylin_metadata@jdbc
>> ,url=jdbc:mysql://lo

Re: Error while launching kylin v4.0.0-alpha

[Yaqian Zhang]

If you wan’t to create a user and database and grant the roles for kylin 
metadata. You can follow the steps below:

1、Use root user login mysql: mysql -u root -pPassword
2、create a user named Kylin and password is kylin: CREATE USER 'kylin'@'%' 
IDENTIFIED BY 'kylin’;
3、create database kylin：create database kylin;
4、grant roles：GRANT ALL ON kylin.* TO 'kylin'@'%’;

Then you can use 
'kylin_metadata_test@jdbc,url=jdbc:mysql://master:3306/kylin,username=kylin,password=kylin,maxActive=10,maxIdle=10
 
'
 

 as kylin.metadata.url.

> 在 2020年9月11日，17:59，Yaqian Zhang  写道：
> 
> It seems that mysql have some problem, I find some related webpage provide 
> follow method to resolve this error:
> 
> Add below content in /etc/mysql/my.cnf:
> 
> wait_timeout=31536000
> interactive_timeout=31536000
> 
> Then restart mysql.
> 
> I’m not sure it works, you can try.
> 
>> 在 2020年9月11日，17:37，hubert stefani  写道：
>> 
>> we did so (we found out the  jar was actually missing), and then we got the  
>> error below. (additional information : we try to deploy on aws emr 5.30)
>> 
>> com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link 
>> failure
>> 
>> The last packet sent successfully to the server was 0 milliseconds ago. The 
>> driver has not received any packets from the server.
>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>> at 
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>> at 
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>> at com.mysql.jdbc.Util.handleNewInstance(Util.java:408)
>> at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1137)
>> at com.mysql.jdbc.MysqlIO.(MysqlIO.java:356)
>> at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2504)
>> 
>> is there  a step to create the database, create users and grant roles ? 
>> Hubert
>>   Le vendredi 11 septembre 2020 à 09:30:27 UTC, Yaqian Zhang 
>>  a écrit :  
>> 
>> Hi Hubert:
>> 
>> Have you put  mysql-connector-java-xxx.jar into $KYLIN_HOME/ext ? 
>> 
>> You can download the mysql JDBC jar from 
>> https://downloads.mysql.com/archives/c-j/ 
>> <https://downloads.mysql.com/archives/c-j/>.
>> 
>>> 在 2020年9月11日，17:17，hubert stefani  写道：
>>> 
>>> Hi,
>>> we encounter the following error when we start the Web UI. it seems that 
>>> mysql-driver is missing. Have we made a misconfiguration  ? Is there a 
>>> detailed configuration ? 
>>> 
>>> 
>>> Exception in thread "main" java.lang.IllegalArgumentException: Failed to 
>>> find metadata store by url: 
>>> kylin_metadata@jdbc,url=jdbc:mysql://localhost:3306/kylin,username=,password=XX,maxActive=10,maxIdle=10
>>> at 
>>> org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:101)
>>> at 
>>> org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:113)
>>> at 
>>> org.apache.kylin.common.persistence.ResourceTool.copyParallel(ResourceTool.java:153)
>>> at 
>>> org.apache.kylin.common.persistence.ResourceTool.main(ResourceTool.java:102)
>>> Caused by: java.lang.reflect.InvocationTargetException
>>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>>> at 
>>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>>> at 
>>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>>> at 
>>> org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:94)
>>> ... 3 more
>>> Caused by: java.lang.IllegalArgumentException: 
>>> org.apache.commons.dbcp.SQLNestedException: Cannot load JDBC driver class 
>>> 'com.mysql.jdbc.Driver'
>>> at 
>>> org.apache.kylin.common.persistence.JDBCConnectionManager.(JDBCConnectionManager.java:68)
>>> at 
>>> org.apache.kylin.common.persistence.JDBCConnectionManager.getConnectionManager(JDBCConnectionManager.java:48)
>>> 
>>> Regards,Hubert
> 

Re: Error while launching kylin v4.0.0-alpha

[Yaqian Zhang]

It seems that mysql have some problem, I find some related webpage provide 
follow method to resolve this error:

Add below content in /etc/mysql/my.cnf:

wait_timeout=31536000
interactive_timeout=31536000

Then restart mysql.

I’m not sure it works, you can try.

> 在 2020年9月11日，17:37，hubert stefani  写道：
> 
> we did so (we found out the  jar was actually missing), and then we got the  
> error below. (additional information : we try to deploy on aws emr 5.30)
> 
> com.mysql.jdbc.exceptions.jdbc4.CommunicationsException: Communications link 
> failure
> 
> The last packet sent successfully to the server was 0 milliseconds ago. The 
> driver has not received any packets from the server.
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at com.mysql.jdbc.Util.handleNewInstance(Util.java:408)
> at com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1137)
> at com.mysql.jdbc.MysqlIO.(MysqlIO.java:356)
> at com.mysql.jdbc.ConnectionImpl.coreConnect(ConnectionImpl.java:2504)
> 
> is there  a step to create the database, create users and grant roles ? 
> Hubert
>Le vendredi 11 septembre 2020 à 09:30:27 UTC, Yaqian Zhang 
>  a écrit :  
> 
> Hi Hubert:
> 
> Have you put  mysql-connector-java-xxx.jar into $KYLIN_HOME/ext ? 
> 
> You can download the mysql JDBC jar from 
> https://downloads.mysql.com/archives/c-j/ 
> <https://downloads.mysql.com/archives/c-j/>.
> 
>> 在 2020年9月11日，17:17，hubert stefani  写道：
>> 
>> Hi,
>> we encounter the following error when we start the Web UI. it seems that 
>> mysql-driver is missing. Have we made a misconfiguration  ? Is there a 
>> detailed configuration ? 
>> 
>> 
>> Exception in thread "main" java.lang.IllegalArgumentException: Failed to 
>> find metadata store by url: 
>> kylin_metadata@jdbc,url=jdbc:mysql://localhost:3306/kylin,username=,password=XX,maxActive=10,maxIdle=10
>> at 
>> org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:101)
>> at 
>> org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:113)
>> at 
>> org.apache.kylin.common.persistence.ResourceTool.copyParallel(ResourceTool.java:153)
>> at 
>> org.apache.kylin.common.persistence.ResourceTool.main(ResourceTool.java:102)
>> Caused by: java.lang.reflect.InvocationTargetException
>> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
>> at 
>> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
>> at 
>> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
>> at 
>> org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:94)
>> ... 3 more
>> Caused by: java.lang.IllegalArgumentException: 
>> org.apache.commons.dbcp.SQLNestedException: Cannot load JDBC driver class 
>> 'com.mysql.jdbc.Driver'
>> at 
>> org.apache.kylin.common.persistence.JDBCConnectionManager.(JDBCConnectionManager.java:68)
>> at 
>> org.apache.kylin.common.persistence.JDBCConnectionManager.getConnectionManager(JDBCConnectionManager.java:48)
>> 
>> Regards,Hubert

Re: Error while launching kylin v4.0.0-alpha

[Yaqian Zhang]

Hi Hubert:

Have you put  mysql-connector-java-xxx.jar into $KYLIN_HOME/ext ? 

You can download the mysql JDBC jar from 
https://downloads.mysql.com/archives/c-j/ 
.

> 在 2020年9月11日，17:17，hubert stefani  写道：
> 
> Hi,
> we encounter the following error when we start the Web UI. it seems that 
> mysql-driver is missing. Have we made a misconfiguration  ? Is there a 
> detailed configuration ? 
> 
> 
> Exception in thread "main" java.lang.IllegalArgumentException: Failed to find 
> metadata store by url: 
> kylin_metadata@jdbc,url=jdbc:mysql://localhost:3306/kylin,username=,password=XX,maxActive=10,maxIdle=10
> at 
> org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:101)
> at 
> org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:113)
> at 
> org.apache.kylin.common.persistence.ResourceTool.copyParallel(ResourceTool.java:153)
> at 
> org.apache.kylin.common.persistence.ResourceTool.main(ResourceTool.java:102)
> Caused by: java.lang.reflect.InvocationTargetException
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
> at 
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at 
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423)
> at 
> org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:94)
> ... 3 more
> Caused by: java.lang.IllegalArgumentException: 
> org.apache.commons.dbcp.SQLNestedException: Cannot load JDBC driver class 
> 'com.mysql.jdbc.Driver'
> at 
> org.apache.kylin.common.persistence.JDBCConnectionManager.(JDBCConnectionManager.java:68)
> at 
> org.apache.kylin.common.persistence.JDBCConnectionManager.getConnectionManager(JDBCConnectionManager.java:48)
> 
> Regards,Hubert

[jira] [Created] (KYLIN-4754) The Cleanup tool cannot clean the the parquet file of the deleted cube and project

[Yaqian Zhang (Jira)]

Yaqian Zhang created KYLIN-4754:
---

 Summary: The Cleanup tool cannot clean the the parquet file of the 
deleted cube and project 
 Key: KYLIN-4754
 URL: https://issues.apache.org/jira/browse/KYLIN-4754
 Project: Kylin
  Issue Type: Improvement
Affects Versions: v4.0.0-alpha
Reporter: Yaqian Zhang
Assignee: Yaqian Zhang
 Fix For: v4.0.0-beta


If a cube is not purged but is dropped directly, the cleanup tool cannot find 
the cube path in hdfs, so the parquet file of this cube can't be cleanup.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [VOTE] Release apache-kylin-4.0.0-alpha (RC1)

[Yaqian Zhang]

+1
mvn clean test passed

> 在 2020年9月9日，11:50，Rupeng Wang  写道：
> 
> +1 (non-binding)
> mvn test passed on my machine;
> happy path passed on HDP2.4 && CDH 5.7.
> 
> ---
> Best wishes,
> Rupeng Wang
> 
> 
> 在 2020/9/9 11:32，“Xiaoxiang Yu” 写入:
>+1 
>mvn test passed on my machine; 
>happy path passed on CDH 5.7.
> 
>--
>Best wishes to you ! 
>From ：Xiaoxiang Yu
> 
>在 2020-09-09 08:21:57，"恩爸" <441586...@qq.com> 写道：
>> +1 from my side. (non-binding)
>> 
>> Best regards,
>> Zhichao Zhang
>> --原始邮件--
>> 发件人: 
>>"dev" 
>>
>> > 发送时间:2020年9月8日(星期二) 晚上11:49
>> 收件人:"dev"> 
>> 主题:[VOTE] Release apache-kylin-4.0.0-alpha (RC1)
>> 
>> 
>> 
>> Hi all,
>> 
>> I have created a build for Apache Kylin 4.0.0-alpha, release candidate
>> 1. Please note, this release is built on kylin-on-parquet-v2 branch.
>> 
>> 
>> Changes highlights:
>> [KYLIN-4213] - The new build engine with Spark-SQL
>> [KYLIN-4450] - Add the feature that adjusting spark driver memory adaptively
>> [KYLIN-4458] - FilePruner prune shards
>> [KYLIN-4475] - Support intersect count for Kylin on Parquet
>> [KYLIN-4462] - Support Count Distinct,TopN and Percentile by kylin on Parquet
>> [KYLIN-4713] - Support use diff spark schedule pool for diff query
>> [KYLIN-4468] - Support Percentile by kylin on Parquet
>> [KYLIN-4662] - Migrate from third-party Spark to offical Apache Spark
>> [KYLIN-4701] - Upgrade front-end from HBase Storage to Parquet Storage
>> [KYLIN-4644] - New tool to clean up intermediate files for Kylin 4.0
>> [KYLIN-4744] - Add tracking URL for build spark job on yarn
>> 
>> Thanks to everyone who has contributed to this release.
>> 
>> Here are the release
>> notes:https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316121version=12348093
>> 
>> 
>> The commit to being voted
>> upon:https://github.com/apache/kylin/commit/a285f9a5b84affc36c5466ce5a1b2fcdb4348b37
>> 
>> Its hash is a285f9a5b84affc36c5466ce5a1b2fcdb4348b37.
>> 
>> The artifacts to be voted on, including the source package and two
>> pre-compiled binary packages are located
>> here:https://dist.apache.org/repos/dist/dev/kylin/apache-kylin-4.0.0-alpha-rc1/
>> 
>> The hash of the artifacts are as follows:
>> apache-kylin-4.0.0-alpha-source-release.zip.sha256
>> f98da070a9839251c8cf3806c274c1191bb11d0b251288b4d2586e034f6ac291
>> apache-kylin-4.0.0-alpha-bin-hadoop2.tar.gz.sha256
>> 8075af2608b62177f04bc5a528194c555959775ed69ae80e5ccaf9a37ec1bf74
>> apache-kylin-4.0.0-alpha-bin-cdh57.tar.gz.sha256
>> f98da070a9839251c8cf3806c274c1191bb11d0b251288b4d2586e034f6ac291
>> 
>> 
>> A staged Maven repository is available for review
>> at:https://repository.apache.org/content/repositories/orgapachekylin-1081/
>> 
>> Release artifacts are signed with the following
>> key:https://people.apache.org/keys/committer/nic.asc
>> 
>> 
>> Please vote on releasing this package as Apache Kylin 4.0.0-alpha.
>> 
>> 
>> The vote is open for the next 72 hours and passes if a majority of at
>> least three +1 binding votes are cast.
>> 
>> 
>> [ ] +1 Release this package as Apache Kylin 4.0.0-alpha
>> [ ] 0 I don't feel strongly about it, but I'm okay with the release
>> [ ] -1 Do not release this package because...
>> 
>> 
>> Here is my vote:
>> 
>> +1 (binding)
>> 
>> -
>> 
>> Best regards,
>> Ni Chunen / George
> 

[jira] [Created] (KYLIN-4744) Add tracking URL for build spark job on yarn

[Yaqian Zhang (Jira)]

Yaqian Zhang created KYLIN-4744:
---

 Summary: Add tracking URL for build spark job on yarn
 Key: KYLIN-4744
 URL: https://issues.apache.org/jira/browse/KYLIN-4744
 Project: Kylin
  Issue Type: Improvement
  Components: Job Engine
Reporter: Yaqian Zhang
Assignee: Yaqian Zhang
 Fix For: v4.0.0-alpha






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KYLIN-4713) Support use diff spark schedule pool for diff query

[Yaqian Zhang (Jira)]

Yaqian Zhang created KYLIN-4713:
---

 Summary: Support use diff spark schedule pool for diff query
 Key: KYLIN-4713
 URL: https://issues.apache.org/jira/browse/KYLIN-4713
 Project: Kylin
  Issue Type: Improvement
  Components: Query Engine
Reporter: Yaqian Zhang
Assignee: Yaqian Zhang
 Fix For: v4.0.0-beta






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KYLIN-4712) Optimize CubeMetaIngester.java CLI

[Yaqian Zhang (Jira)]

Yaqian Zhang created KYLIN-4712:
---

 Summary: Optimize CubeMetaIngester.java CLI
 Key: KYLIN-4712
 URL: https://issues.apache.org/jira/browse/KYLIN-4712
 Project: Kylin
  Issue Type: Improvement
  Components: Client - CLI
Affects Versions: v3.1.0
Reporter: Yaqian Zhang
Assignee: Yaqian Zhang
 Fix For: v4.0.0-beta, v3.1.1






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Tableau reads data in Kylin

[Yaqian Zhang]

Hi:

Tableau reads data by sending queries to kylin, just like you get when you 
execute queries directly on kylin web. 
It can get all the data you have built. If there are duplicate cubes, it will 
select the latest data to answer query.

> 在 2020年8月13日，04:47，Fish  写道：
> 
> Hi,
> 
> I am wondering how Tableau reads data in Kylin. If I built several cubes,
> how Tableau reads the data from specified cube? or Tableau only reads the
> first cube ?
> 
> Thank you
> 
> --
> Sent from: http://apache-kylin.74782.x6.nabble.com/

Re: new committer: Vino Yang (杨华)

[Yaqian Zhang]

Congratulations！

> 在 2020年8月7日，13:55，George Ni  写道：
> 
> Congratulations, Vino.
> 
> ShaoFeng Shi  于2020年8月7日周五 上午10:36写道：
> 
>> The Project Management Committee (PMC) for Apache Kylin has invited Vino
>> Yang (杨华, vinoy...@apache.org) to become a committer and we are pleased to
>> announce that he has accepted.
>> 
>> Being a committer enables easier contribution to the project since there is
>> no need to go via the patch submission process. This should enable better
>> productivity.
>> 
>> Best regards,
>> 
>> Shaofeng Shi 史少锋
>> Apache Kylin PMC
>> Email: shaofeng...@apache.org
>> 
>> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
>> Join Kylin user mail group: user-subscr...@kylin.apache.org
>> Join Kylin dev mail group: dev-subscr...@kylin.apache.org
>> 
> 
> 
> -- 
> 
> -
> 
> Best regards,
> 
> 
> 
> Ni Chunen / George

Re: question of building cube with spark stay 18.1....

[Yaqian Zhang]

Hi：

Have you check log in spark history server? Is there some useful info?

You can try to manually submit the task of this step and observe the running 
result to determine whether it’s Kylin’s problem.

> 在 2020年7月30日，15:23，zzm_user <18817843...@163.com> 写道：
> 
> no logs with error .kylin cannot start spark ,and build cube stay pecrcent
> 18.1%  
> 
> 
> 2020-07-30 15:22:51,123 INFO  [pool-13-thread-1] spark.SparkExecutable:38 :
> 20/07/30 15:22:51 INFO yarn.Client: Application report for
> application_1596091427110_0004 (state: RUNNING)
> 2020-07-30 15:22:52,124 INFO  [pool-13-thread-1] spark.SparkExecutable:38 :
> 20/07/30 15:22:52 INFO yarn.Client: Application report for
> application_1596091427110_0004 (state: RUNNING)
> 2020-07-30 15:22:53,126 INFO  [pool-13-thread-1] spark.SparkExecutable:38 :
> 20/07/30 15:22:53 INFO yarn.Client: Application report for
> application_1596091427110_0004 (state: RUNNING)
> 2020-07-30 15:22:54,127 INFO  [pool-13-thread-1] spark.SparkExecutable:38 :
> 20/07/30 15:22:54 INFO yarn.Client: Application report for
> application_1596091427110_0004 (state: RUNNING)
> 2020-07-30 15:22:55,128 INFO  [pool-13-thread-1] spark.SparkExecutable:38 :
> 20/07/30 15:22:55 INFO yarn.Client: Application report for
> application_1596091427110_0004 (state: RUNNING)
> 2020-07-30 15:22:56,130 INFO  [pool-13-thread-1] spark.SparkExecutable:38 :
> 20/07/30 15:22:56 INFO yarn.Client: Application report for
> application_1596091427110_0004 (state: RUNNING)
> 2020-07-30 15:22:57,131 INFO  [pool-13-thread-1] spark.SparkExecutable:38 :
> 20/07/30 15:22:57 INFO yarn.Client: Application report for
> application_1596091427110_0004 (state: RUNNING)
> 
> --
> Sent from: http://apache-kylin.74782.x6.nabble.com/

Re: 关于 kylin-ODBC 驱动

[Yaqian Zhang]

Hi Terry：

You can download Kylin ODBC Driver from hell: 
https://kyligence.io/resources/kyligence-odbc-driver-for-apache-kylin-2/ 


> 在 2020年7月29日，15:21，Terry  写道：
> 
> 你好从昨天到今天，我下载过VS2019，尝试过编译。我太菜了，没有成功。是否能直接发送一份kylin-ODBC给我呀？

Re: Recommendation of loading the cube

[Yaqian Zhang]

Hi:

You can refer to this document to optimize the cube design and speed up cube 
building process:

 http://kylin.apache.org/docs/tutorial/cube_build_performance.html 
 

> 在 2020年7月20日，21:31，Fish  写道：
> 
> Hi,
> 
> I had a datafile which contains over 10,000,000 rows of data. The Kylin took
> a long time (about 4-5 hours) to finish building the full cube. Is there any
> recommendation which can speed up cube building process?
> 
> Thank you
> 
> --
> Sent from: http://apache-kylin.74782.x6.nabble.com/

[jira] [Created] (KYLIN-4646) Kylin on parquet: Merge segment error

[Yaqian Zhang (Jira)]

Yaqian Zhang created KYLIN-4646:
---

 Summary: Kylin on parquet: Merge segment error
 Key: KYLIN-4646
 URL: https://issues.apache.org/jira/browse/KYLIN-4646
 Project: Kylin
  Issue Type: Bug
  Components: Job Engine
Affects Versions: v4.0.0-beta
Reporter: Yaqian Zhang
Assignee: Yaqian Zhang
 Attachments: Snipaste_2020-07-17_17-23-29.png

When 'spark.driver.memory' is not set, the value will be auto adjust according 
to cuboid num, but Merge job is not be set  cuboid num. "spark.driver.memory" 
will be set to 0 in Merge Job.




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Kylin Build cube errors

[Yaqian Zhang]

Hi Carrie:

It looks like HBase regionServer failed for some reason. 

You can check the log of regionServer node and fix the problem, then restart it.

> 在 2020年7月17日，00:34，Carrie Yu  写道：
> 
> Hi,
> 
> I tried to build the cube in Apache Kylin and I got this error. I am using
> EMR 5.30.1 and Kylin 3.1.0
> 
> client.ZooKeeperRegistry:107 : ClusterId read in ZooKeeper is null
> Exception in thread "main" java.lang.IllegalArgumentException: Failed to
> find metadata store by url: kylin_metadata@hbase at
> org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:101)
> at
> org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:113)
> at
> org.apache.kylin.rest.service.AclTableMigrationTool.checkIfNeedMigrate(AclTableMigrationTool.java:99)
> at
> org.apache.kylin.tool.AclTableMigrationCLI.main(AclTableMigrationCLI.java:43)
> Caused by: java.lang.reflect.InvocationTargetException at
> sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method) at
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
> at
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> at java.lang.reflect.Constructor.newInstance(Constructor.java:423) at
> org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:94)
> ... 3 more Caused by:
> org.apache.hadoop.hbase.client.RetriesExhaustedException: Can't get the
> location for replica 0 at
> org.apache.hadoop.hbase.client.RpcRetryingCallerWithReadReplicas.getRegionLocations(RpcRetryingCallerWithReadReplicas.java:372)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:153)
> at
> org.apache.hadoop.hbase.client.ScannerCallableWithReplicas.call(ScannerCallableWithReplicas.java:58)
> at
> org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithoutRetries(RpcRetryingCaller.java:219)
> at
> org.apache.hadoop.hbase.client.ClientScanner.call(ClientScanner.java:275)
> at
> org.apache.hadoop.hbase.client.ClientScanner.loadCache(ClientScanner.java:436)
> at
> org.apache.hadoop.hbase.client.ClientScanner.next(ClientScanner.java:310)
> at
> org.apache.hadoop.hbase.MetaTableAccessor.fullScan(MetaTableAccessor.java:640)
> at
> org.apache.hadoop.hbase.MetaTableAccessor.tableExists(MetaTableAccessor.java:367)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.tableExists(HBaseAdmin.java:411)
> at
> org.apache.kylin.storage.hbase.HBaseConnection.tableExists(HBaseConnection.java:290)
> at
> org.apache.kylin.storage.hbase.HBaseConnection.createHTableIfNeeded(HBaseConnection.java:315)
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.createHTableIfNeeded(HBaseResourceStore.java:119)
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.(HBaseResourceStore.java:89)
> ... 8 more 2020-07-16 16:14:10,927 INFO [close-hbase-conn]
> hbase.HBaseConnection:137 : Closing HBase connections... 2020-07-16
> 16:14:10,927 INFO [close-hbase-conn]
> client.ConnectionManager$HConnectionImplementation:1767 : Closing zookeeper
> sessionid=0x1026ac60a64 2020-07-16 16:14:10,933 INFO [close-hbase-conn]
> zookeeper.ZooKeeper:693 : Session: 0x1026ac60a64 closed 2020-07-16
> 16:14:10,933 INFO [main-EventThread] zookeeper.ClientCnxn:522 : EventThread
> shut down for session: 0x1026ac60a64
> 
> Any help would be appreciated! Thank you

[jira] [Created] (KYLIN-4643) Metrics data can't flush

[Yaqian Zhang (Jira)]

Yaqian Zhang created KYLIN-4643:
---

 Summary: Metrics data can't flush
 Key: KYLIN-4643
 URL: https://issues.apache.org/jira/browse/KYLIN-4643
 Project: Kylin
  Issue Type: Improvement
Reporter: Yaqian Zhang






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KYLIN-4633) IT failed for can't detect the default value of config kylin.source.hive.databasedir

[Yaqian Zhang (Jira)]

Yaqian Zhang created KYLIN-4633:
---

 Summary: IT failed for can't detect the default value of config 
kylin.source.hive.databasedir
 Key: KYLIN-4633
 URL: https://issues.apache.org/jira/browse/KYLIN-4633
 Project: Kylin
  Issue Type: Improvement
  Components: Tools, Build and Test
Reporter: Yaqian Zhang
Assignee: Yaqian Zhang


KYLIN-4616 introduce the method of auto detect the default value of config 
kylin.source.hive.databasedir in find-hive-dependency, but IT doesn't execute 
this script so `kylin.source.hive.databasedir` is null, this will lead to 
BuildCubeWithEngine failed.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: [Announce] Apache Kylin 3.1.0 released

[Yaqian Zhang]

Hi ECOR:

You can refer to http://kylin.apache.org/cn/docs/install/index.html 
 to use kylin 3.1.

> 在 2020年7月6日，09:39，ECOR、one  写道：
> 
> i want to know how to useKylin3.1.0 anyone help me 
> ?
> 
> 
> 
> --原始邮件--
> 发件人: "Han Luke" 发送时间: 2020年7月4日(星期六) 晚上6:46
> 收件人: "u...@kylin.apache.org" 抄送: "dev" 主题: Re: [Announce] Apache Kylin 3.1.0 released
> 
> 
> 
> That’s cool, thanks everyone.
> 
> 
> From: ShaoFeng Shi  Sent: Thursday, July 2, 2020 10:11:37 PM
> To: user  Cc: dev  Subject: Re: [Announce] Apache Kylin 3.1.0 released
> 
> Great! Thanks to everyone who contributed to this release!
> 
> We encourage Kylin 2.x users to upgrade to Kylin 3, which has been verified 
> by several early and big users. In the future, we will focus more on 3.1 and 
> 4.0 development.
> 
> Best regards,
> 
> Shaofeng Shi 史少锋
> Apache Kylin PMC
> Email: shaofeng...@apache.org 
> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> Join Kylin user mail group: 
> user-subscr...@kylin.apache.org Join Kylin dev mail group: 
> dev-subscr...@kylin.apache.org 
> 
> 
> 
> George Ni mailto:n...@apache.org; 于2020年7月2日周四 
> 下午9:44写道：
> The Apache Kylin team is pleased to announce the immediate availability of 
> the 3.1.0 release.
> 
> This is a major release after 3.0.0, with 10 new features and 142 bug fixes 
> and enhancement. All of the changes in this release can be found in: 
> https://kylin.apache.org/docs/release_notes.html
> 
> You can download the source release and binary packages from Apache Kylin's 
> download page: https://kylin.apache.org/download/
> 
> Apache Kylin is an open-source Distributed Analytical Data Warehouse for Big 
> Data; it was designed to provide OLAP (Online Analytical Processing) 
> capability in the big data era. By renovating the multi-dimensional cube and 
> precalculation technology on Hadoop and Spark, Kylin is able to achieve 
> near-constant query speed regardless of the ever-growing data volume. 
> Reducing query latency from minutes to sub-second, Kylin brings online 
> analytics back to big data.
> 
> Apache Kylin lets you query billions of rows at sub-second latency in 3 steps:
> 1. Identify a Star/Snowflake Schema on Hadoop.
> 2. Build Cube from the identified tables.
> 3. Query using ANSI-SQL and get results in sub-second, via ODBC, JDBC or 
> RESTful API.
> 
> Thanks to everyone who has contributed to this release.
> 
> We welcome your help and feedback. For more information on how to report 
> problems, and to get involved, visit the project website at 
> https://kylin.apache.org/
> 
> --
> 
> -
> 
> Best regards,
> 
> 
> 
> Ni Chunen / George

Re: [VOTE] Release apache-kylin-3.1.0 (RC1)

[Yaqian Zhang]

+1
Mvn clean test passed

> 在 2020年6月29日，10:43，ShaoFeng Shi  写道：
> 
> +1 (binding)
> 
> I checked:
> 
>   -  Compile successfully with JDK 8 on MacOS;
>   -  "mvn test" success;
>   -  The sha 256 hash is correct;
>   -  The gpg signature is by n...@apache.org;
>   -  The NOTICE and LICENSE files;
> 
> 
> 
> Best regards,
> 
> Shaofeng Shi 史少锋
> Apache Kylin PMC
> Email: shaofeng...@apache.org
> 
> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> Join Kylin user mail group: user-subscr...@kylin.apache.org
> Join Kylin dev mail group: dev-subscr...@kylin.apache.org
> 
> 
> 
> 
> Rupeng Wang  于2020年6月29日周一 上午9:51写道：
> 
>> +1
>> 
>> mvn test passed
>> Binary package test passed on CDH6.0
>> 
>> 
>> ---
>> Best wishes,
>> Rupeng Wang
>> 
>> 
>> 
>> 在 2020/6/29 09:33，“Zhong, Yanghong” 写入:
>> 
>>+1 (binding)
>> 
>>mvn test passed.
>> 
>>Best regards,
>>Yanghong Zhong
>> 
>>On 2020/6/28, 11:13 PM, "恩爸" <441586...@qq.com> wrote:
>> 
>>External Email
>> 
>>
>>+1 from my side.
>>Tested passed on CDH5 and HDP3.
>> 
>> 
>> 
>> 
>> 
>> 
>> 
>>Best regards,
>>Zhichao Zhang
>> 
>> 
>> 
>> 
>>
>> 
>> 
>> 
>> 
>>--原始邮件--
>>发件人:"George Ni">发送时间:2020年6月28日(星期天) 晚上7:51
>>收件人:"dev"> 
>>主题:[VOTE] Release apache-kylin-3.1.0 (RC1)
>> 
>> 
>> 
>>Hi all,
>> 
>>I have created a build for Apache Kylin 3.1.0, release candidate 1.
>> 
>>Changes highlights:
>>[KYLIN-4342] - Build Global Dict by MR/Hive New Version
>>[KYLIN-3758] - Flink Cube Build Engine
>>[KYLIN-3832] - Kylin pushdown to support PostgreSQL
>>[KYLIN-4445] - Provide a Kylin on Kubernetes solution
>>[KYLIN-4485] - Create a self-service interface for cube migration
>>[KYLIN-4425] - Refactor Diagnosis Tool
>> 
>>Thanks to everyone who has contributed to this release.
>>Here are the release notes:
>> 
>> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fsecure%2FReleaseNote.jspa%3FprojectId%3D12316121%26amp%3Bversion%3D12346593data=02%7C01%7Cyangzhong%40ebay.com%7C3237b3771e5541ab35a408d81b75c6f3%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C637289539944798530sdata=ncIL3k82VXOGSPTCoo1HXD0Srmb9ti%2Bcvd%2FJbJQaqOo%3Dreserved=0
>> 
>> 
>>The commit to being voted upon:
>> 
>> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fkylin%2Fcommit%2F2e056748bf61ef89d421735bb7be41f4266d7b94data=02%7C01%7Cyangzhong%40ebay.com%7C3237b3771e5541ab35a408d81b75c6f3%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C637289539944798530sdata=lCJYAjlP1zM7RzkwO2gI1BRqWHWyagn9jaHfAUdg0hw%3Dreserved=0
>> 
>>Its hash is 2e056748bf61ef89d421735bb7be41f4266d7b94.
>> 
>>The artifacts to be voted on, including the source package and four
>>pre-compiled binary packages are located here:
>> 
>> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fkylin%2Fapache-kylin-3.1.0-rc1%2Fdata=02%7C01%7Cyangzhong%40ebay.com%7C3237b3771e5541ab35a408d81b75c6f3%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C637289539944798530sdata=%2FkLpG5SuRmROChmme2Q5l76AqEhHQd1xh%2FkfSbG7rr4%3Dreserved=0
>> 
>>The hash of the artifacts are as follows:
>>apache-kylin-3.1.0-source-release.zip.sha256
>>1efa8a60003ea8adaa39de7c020be2bd70ed045de736995153707c1ca08353e1
>>apache-kylin-3.1.0-bin-hbase1x.tar.gz.sha256
>>78921a27acba781adc41729aa65316833d2aec5d32047d6ffc195bfaf156cd5d
>>apache-kylin-3.1.0-bin-cdh57.tar.gz.sha256
>>bdc5a790880f1b6e1d894454e3512c54515f3b52d55f4d118363514c5007ba89
>>apache-kylin-3.1.0-bin-hadoop3.tar.gz.sha256
>>0dd5beafd97b258a1af2ea597b9b610a01a5c63d65b150d7c5a9ada9f0ab6697
>>apache-kylin-3.1.0-bin-cdh60.tar.gz.sha256
>>8e662950f331cdf9af877225d2b8d63b2797b0bb4453e21b80e9528e908414f2
>> 
>> 
>>A staged Maven repository is available for review at:
>> 
>> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Frepository.apache.org%2Fcontent%2Frepositories%2Forgapachekylin-1080%2Fdata=02%7C01%7Cyangzhong%40ebay.com%7C3237b3771e5541ab35a408d81b75c6f3%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C637289539944808522sdata=qZGAbjfOvVvIUqAb7B8lZciAp4rBxClkcxIjrw0nMsw%3Dreserved=0
>> 
>>Release artifacts are signed with the following key:
>> 
>> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fpeople.apache.org%2Fkeys%2Fcommitter%2Fnic.ascdata=02%7C01%7Cyangzhong%40ebay.com%7C3237b3771e5541ab35a408d81b75c6f3%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C637289539944808522sdata=R%2B9H3gchug6G1SrM4i3d2S5xyx8X5CM5ygbVV%2FtjT2g%3Dreserved=0
>> 
>> 
>>Please vote on releasing this package as Apache Kylin 3.1.0.
>> 
>>The vote is open for the next 72 

Re: Can Kylin run on HDP3.1.4?

[Yaqian Zhang]

Hi：

Kylin 3.1.0 is in the process of platform compatibility test, and has not been 
officially released yet

Please follow the release voting and announcement. 

After the official release, the binary package of Hadoop 3 will be provided. 

You can test it in your environment.

> 在 2020年6月9日，11:02，ShaoFeng Shi  写道：
> 
> Hi,
> 
> You're correct that there is conflict in the guava version on HDP 3.1.
> We're going to fix it in Kylin 3.1, which is on the way (plan to release in
> this month).
> 
> Best regards,
> 
> Shaofeng Shi 史少锋
> Apache Kylin PMC
> Email: shaofeng...@apache.org
> 
> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> Join Kylin user mail group: user-subscr...@kylin.apache.org
> Join Kylin dev mail group: dev-subscr...@kylin.apache.org
> 
> 
> 
> 
> xu lx  于2020年6月9日周二 上午10:49写道：
> 
>> Hello,
>> I find guava version is 14.0 in kylin, while it is 28.0 in HDP3.1.4. So
>> kylin failed to start.
>> Is there method to solve this problem?
>> Thanks.
>> 
>> 发送自 Windows 10 版邮件应用
>> 
>> 

[jira] [Created] (KYLIN-4581) Add spark and flink job for release test

[Yaqian Zhang (Jira)]

Yaqian Zhang created KYLIN-4581:
---

 Summary: Add spark and flink job for release test
 Key: KYLIN-4581
 URL: https://issues.apache.org/jira/browse/KYLIN-4581
 Project: Kylin
  Issue Type: Improvement
  Components: Tools, Build and Test
Reporter: Yaqian Zhang
Assignee: Yaqian Zhang
 Fix For: v3.1.1






--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: Rgarding connectivity with PowerBI

[Yaqian Zhang]

Hi sir:

Can you describe in detail the problems you have encountered? Such as Kylin 
Version? 

When you query with PowerBI, does $KYLIN_HOME/logs/kylin.log output 
corresponding logs or error messages?

> 在 2020年6月16日，15:17，Sanower Abedin Tamjit  
> 写道：
> 
> Hello Dear,
> First of all, I am thanking all developers of Kylin.I am trying to integrate 
> Kylin in our system.
> During integration with Power BI, I am facing some problem regarding Cube 
> Measurement.
> I am getting result when I query on Insight section.
> But in PowerBI when I put Kylin ODBC just come to all tables no measurement 
> shows and direct query also not worked. 
> How can I achieve this? If  have any solution in Apache Kylin please reply to 
> me.
> Thanks again.
> 
> 
> Thanks & Regards 
> | Sanower Abedin Tamjit
> |
> | Software Engineer 
> PRAN-RFL  Group (Industrial Automation Team)
> PRAN Center, 105 Maddha Badda, Dhaka 1212.
> Phone: +8801635872701
> Skype: sanower.tamjit
> Linkedin:  sanower-tamjit
> Github: SanowerTamjit |
> | 
> |
> 
> 
> 

[jira] [Created] (KYLIN-4542) The startup script did not properly handle the judgment of SPARK_HOME

[Yaqian Zhang (Jira)]

Yaqian Zhang created KYLIN-4542:
---

 Summary: The startup script did not properly handle the judgment 
of SPARK_HOME
 Key: KYLIN-4542
 URL: https://issues.apache.org/jira/browse/KYLIN-4542
 Project: Kylin
  Issue Type: Improvement
  Components: Storage - Parquet
Reporter: Yaqian Zhang
Assignee: Yaqian Zhang
 Fix For: v4.0.0-beta


1、When SPARK_HOME is not set, kylin.sh does not output an error message on the 
command line to remind the user, and the display starts normally. But the web 
page cannot be opened, and  error just reported in kylin.out.
2、After downloading spark with bin/download-spark.sh , still need set 
SPARK_HOME manually .



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KYLIN-4541) Kylin.log output error information during build job

[Yaqian Zhang (Jira)]

Yaqian Zhang created KYLIN-4541:
---

 Summary: Kylin.log output error information during build job
 Key: KYLIN-4541
 URL: https://issues.apache.org/jira/browse/KYLIN-4541
 Project: Kylin
  Issue Type: Improvement
  Components: Storage - Parquet
Reporter: Yaqian Zhang
Assignee: Yaqian Zhang
 Fix For: v4.0.0-beta
 Attachments: image-2020-06-01-14-53-01-996.png

Kylin.log always output some error information during build job, and the job is 
normal.
 !image-2020-06-01-14-53-01-996.png! 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

[jira] [Created] (KYLIN-4540) Some field values will become null after UNION ALL

[Yaqian Zhang (Jira)]

Yaqian Zhang created KYLIN-4540:
---

 Summary: Some field values will become null after UNION ALL
 Key: KYLIN-4540
 URL: https://issues.apache.org/jira/browse/KYLIN-4540
 Project: Kylin
  Issue Type: Bug
  Components: Query Engine
Affects Versions: v3.0.2
Reporter: Yaqian Zhang
Assignee: Yaqian Zhang
 Fix For: v3.1.0


It can be reproduced in learn_kylin.
see: 
https://lists.apache.org/thread.html/r5b17ec62b08dbfb82e1db597e957a51ace210d95a6e1ba2b217496df%40%3Cuser.kylin.apache.org%3E



--
This message was sent by Atlassian Jira
(v8.3.4#803005)