Re: Re: [VOTE] Release apache-kylin-3.1.0 (RC1)

[Chao Long]

+1
mvn test passed

On Mon, Jun 29, 2020 at 1:09 PM Ma Gang  wrote:

> +1 (binding)mvn test pass, my test env is JDK8, mac.
>
>
>
>
> --
>
> Thanks,
> Gang Ma
>
>
>
>
>
> 在 2020-06-29 10:56:50，"Yaqian Zhang"  写道：
> >+1
> >Mvn clean test passed
> >
> >> 在 2020年6月29日，10:43，ShaoFeng Shi  写道：
> >>
> >> +1 (binding)
> >>
> >> I checked:
> >>
> >>   -  Compile successfully with JDK 8 on MacOS;
> >>   -  "mvn test" success;
> >>   -  The sha 256 hash is correct;
> >>   -  The gpg signature is by n...@apache.org;
> >>   -  The NOTICE and LICENSE files;
> >>
> >>
> >>
> >> Best regards,
> >>
> >> Shaofeng Shi 史少锋
> >> Apache Kylin PMC
> >> Email: shaofeng...@apache.org
> >>
> >> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> >> Join Kylin user mail group: user-subscr...@kylin.apache.org
> >> Join Kylin dev mail group: dev-subscr...@kylin.apache.org
> >>
> >>
> >>
> >>
> >> Rupeng Wang  于2020年6月29日周一 上午9:51写道：
> >>
> >>> +1
> >>>
> >>> mvn test passed
> >>> Binary package test passed on CDH6.0
> >>>
> >>>
> >>> ---
> >>> Best wishes,
> >>> Rupeng Wang
> >>>
> >>>
> >>>
> >>> 在 2020/6/29 09:33，“Zhong, Yanghong” 写入:
> >>>
> >>>+1 (binding)
> >>>
> >>>mvn test passed.
> >>>
> >>>Best regards,
> >>>Yanghong Zhong
> >>>
> >>>On 2020/6/28, 11:13 PM, "恩爸" <441586...@qq.com> wrote:
> >>>
> >>>External Email
> >>>
> >>>
> >>>+1 from my side.
> >>>Tested passed on CDH5 and HDP3.
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>Best regards,
> >>>Zhichao Zhang
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>--原始邮件--
> >>>发件人:"George Ni" >>>发送时间:2020年6月28日(星期天) 晚上7:51
> >>>收件人:"dev" >>>
> >>>主题:[VOTE] Release apache-kylin-3.1.0 (RC1)
> >>>
> >>>
> >>>
> >>>Hi all,
> >>>
> >>>I have created a build for Apache Kylin 3.1.0, release
> candidate 1.
> >>>
> >>>Changes highlights:
> >>>[KYLIN-4342] - Build Global Dict by MR/Hive New Version
> >>>[KYLIN-3758] - Flink Cube Build Engine
> >>>[KYLIN-3832] - Kylin pushdown to support PostgreSQL
> >>>[KYLIN-4445] - Provide a Kylin on Kubernetes solution
> >>>[KYLIN-4485] - Create a self-service interface for cube
> migration
> >>>[KYLIN-4425] - Refactor Diagnosis Tool
> >>>
> >>>Thanks to everyone who has contributed to this release.
> >>>Here are the release notes:
> >>>
> >>>
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fsecure%2FReleaseNote.jspa%3FprojectId%3D12316121%26amp%3Bversion%3D12346593data=02%7C01%7Cyangzhong%40ebay.com%7C3237b3771e5541ab35a408d81b75c6f3%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C637289539944798530sdata=ncIL3k82VXOGSPTCoo1HXD0Srmb9ti%2Bcvd%2FJbJQaqOo%3Dreserved=0
> >>>
> >>>
> >>>The commit to being voted upon:
> >>>
> >>>
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fkylin%2Fcommit%2F2e056748bf61ef89d421735bb7be41f4266d7b94data=02%7C01%7Cyangzhong%40ebay.com%7C3237b3771e5541ab35a408d81b75c6f3%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C637289539944798530sdata=lCJYAjlP1zM7RzkwO2gI1BRqWHWyagn9jaHfAUdg0hw%3Dreserved=0
> >>>
> >>>Its hash is 2e056748bf61ef89d421735bb7be41f4266d7b94.
> >>>
> >>>The artifacts to be voted on, including the source package and
> four
> >>>pre-compiled binary packages are located here:
> >>>
> >>>
> https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.apache.org%2Frepos%2Fdist%2Fdev%2Fkylin%2Fapache-kylin-3.1.0-rc1%2Fdata=02%7C01%7Cyangzhong%40ebay.com%7C3237b3771e5541ab35a408d81b75c6f3%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C637289539944798530sdata=%2FkLpG5SuRmROChmme2Q5l76AqEhHQd1xh%2FkfSbG7rr4%3Dreserved=0
> >>>
> >>>The hash of the artifacts are as follows:
> >>>apache-kylin-3.1.0-source-release.zip.sha256
> >>>1efa8a60003ea8adaa39de7c020be2bd70ed045de736995153707c1ca08353e1
> >>>apache-kylin-3.1.0-bin-hbase1x.tar.gz.sha256
> >>>78921a27acba781adc41729aa65316833d2aec5d32047d6ffc195bfaf156cd5d
> >>>apache-kylin-3.1.0-bin-cdh57.tar.gz.sha256
> >>>bdc5a790880f1b6e1d894454e3512c54515f3b52d55f4d118363514c5007ba89
> >>>apache-kylin-3.1.0-bin-hadoop3.tar.gz.sha256
> >>>0dd5beafd97b258a1af2ea597b9b610a01a5c63d65b150d7c5a9ada9f0ab6697
> >>>apache-kylin-3.1.0-bin-cdh60.tar.gz.sha256
> >>>8e662950f331cdf9af877225d2b8d63b2797b0bb4453e21b80e9528e908414f2
> >>>
> >>>
> >>>A staged Maven repository is available for review at:
> >>>
> >>>
> 

[jira] [Created] (KYLIN-4497) PreparedStatement query do not support "limit ? offset ?"

[Chao Long (Jira)]

Chao Long created KYLIN-4497:


 Summary: PreparedStatement query do not support "limit ? offset ?"
 Key: KYLIN-4497
 URL: https://issues.apache.org/jira/browse/KYLIN-4497
 Project: Kylin
  Issue Type: Bug
  Components: Driver - JDBC
Reporter: Chao Long
Assignee: Chao Long
 Fix For: v3.1.0


Query
{code:java}
String sql = "select part_dt, sum(price) from KYLIN_SALES group by part_dt 
order by part_dt limit ? offset ?";

conn = getConnection();
statement = conn.prepareStatement(sql);
statement.setInt(1, 10);
statement.setInt(2, 0);
{code}
Error log
{code:java}
2020-05-15 09:32:36,062 ERROR [Query b87327f4-7299-32f2-4a2e-c19795a42288-44] 
service.QueryService:542 : Exception while executing query
java.sql.SQLException: Error while preparing statement [select part_dt, 
sum(price) from KYLIN_SALES group by part_dt order by part_dt limit ? offset ?]
at org.apache.calcite.avatica.Helper.createException(Helper.java:56)
at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
at 
org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement_(CalciteConnectionImpl.java:210)
at 
org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:192)
at 
org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:89)
at 
org.apache.calcite.avatica.AvaticaConnection.prepareStatement(AvaticaConnection.java:175)
at 
org.apache.kylin.rest.service.QueryService.createPreparedContext(QueryService.java:1262)
at 
org.apache.kylin.rest.service.QueryService.access$600(QueryService.java:141)
at 
org.apache.kylin.rest.service.QueryService$PreparedContextFactory.create(QueryService.java:1301)
at 
org.apache.kylin.rest.service.QueryService$PreparedContextFactory.create(QueryService.java:1296)
at 
org.apache.commons.pool2.BaseKeyedPooledObjectFactory.makeObject(BaseKeyedPooledObjectFactory.java:60)
at 
org.apache.commons.pool2.impl.GenericKeyedObjectPool.create(GenericKeyedObjectPool.java:1064)
at 
org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:358)
at 
org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:281)
at 
org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:680)
at 
org.apache.kylin.rest.service.QueryService.query(QueryService.java:219)
at 
org.apache.kylin.rest.service.QueryService.queryAndUpdateCache(QueryService.java:480)
at 
org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:440)
at 
org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:378)
at 
org.apache.kylin.rest.controller.QueryController.query(QueryController.java:93)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at 
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at 
org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at 
com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:209)
at 
com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:244)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at 
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
at 
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
at 
org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:492)
at 
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:165)
at 
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
at 
org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:1025)
at 
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at 
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:452)
at 
org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1195)
at 
org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(Abstr

Re: [VOTE] Release apache-kylin-2.6.6 (RC2)

[Chao Long]

+1
mvn test passed

On Fri, May 15, 2020 at 8:47 AM ShaoFeng Shi  wrote:

> +1 (binding)
>
> Checked the signature and sha hash;
> 'Mvn test' build successfully on Java 8;
>
> Best regards,
>
> Shaofeng Shi 史少锋
> Apache Kylin PMC
> Email: shaofeng...@apache.org
>
> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> Join Kylin user mail group: user-subscr...@kylin.apache.org
> Join Kylin dev mail group: dev-subscr...@kylin.apache.org
>
>
>
>
> Xiaoxiang Yu  于2020年5月14日周四 下午11:23写道：
>
> > +1 .
> > Maven test passed and happy path passed in CDH5.7.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > --
> >
> > Best wishes to you !
> > From ：Xiaoxiang Yu
> >
> >
> >
> >
> >
> > At 2020-05-14 17:06:55, "George Ni"  wrote:
> > >Hi all,
> > >
> > >
> > >
> > >I have created a build for Apache Kylin 2.6.6, release candidate 2.
> > >
> > >
> > >
> > >Changes highlights:
> > >
> > >[KYLIN-4390] - Update tomcat to 7.0.100
> > >
> > >[KYLIN-4426] - Refine CliCommandExecutor
> > >
> > >[KYLIN-4206] - Support Glue as Hive Metatdata
> > >
> > >
> > >
> > >Thanks to everyone who has contributed to this release.
> > >
> > >Here are the release notes:
> > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12346976==12316121
> > >
> > >
> > >
> > >The commit to being voted upon:
> > >
> > >
> >
> https://github.com/apache/kylin/commit/ddd5f8ecd4157b8f889b047e421dd9cfae7e1142
> > >
> > >
> > >Its hash is ddd5f8ecd4157b8f889b047e421dd9cfae7e1142.
> > >
> > >
> > >
> > >The artifacts to be voted on, including the source package and four
> > >
> > >pre-compiled binary packages are located here:
> > >
> > >https://dist.apache.org/repos/dist/dev/kylin/apache-kylin-2.6.6-rc2/
> > >
> > >
> > >
> > >The hash of the artifacts are as follows:
> > >
> > >apache-kylin-2.6.6-source-release.zip.sha256
> > >6d38671f494e3d5f2bb26dfb94d996a5ceb0c00c2a17b9c181ad853639198d3a
> > >
> > >apache-kylin-2.6.6-bin-hbase1x.tar.gz.sha256
> > >6a42962efbce5a51e2ce4bf8db0b8fa7341ef0b30e4f02e876a5c2fb0500944b
> > >
> > >apache-kylin-2.6.6-bin-cdh57.tar.gz.sha256
> > >85cb22e7d6d9adad214854f9ab285b7d47e874eb9f9df1c5bd01882877171762
> > >
> > >apache-kylin-2.6.6-bin-hadoop3.tar.gz.sha256
> > >f060f8e16f909ae74d9e3c188bb071fcfa87e0a21fd7581fc968e1bcf00e5121
> > >
> > >apache-kylin-2.6.6-bin-cdh60.tar.gz.sha256
> > >8d85a3036d312b47030e3b309af526afe4484720be156f7e3f05e626c02bf531
> > >
> > >
> > >
> > >A staged Maven repository is available for review at:
> > >
> > >https://repository.apache.org/content/repositories/orgapachekylin-1077/
> > >
> > >
> > >
> > >Release artifacts are signed with the following key:
> > >
> > >https://people.apache.org/keys/committer/nic.asc
> > >
> > >
> > >
> > >Please vote on releasing this package as Apache Kylin 2.6.6.
> > >
> > >
> > >
> > >The vote is open for the next 72 hours and passes if a majority of
> > >
> > >at least three +1 PMC votes are cast.
> > >
> > >
> > >
> > >[ ] +1 Release this package as Apache Kylin 2.6.6
> > >
> > >[ ]  0 I don't feel strongly about it, but I'm okay with the release
> > >
> > >[ ] -1 Do not release this package because...
> > >
> > >
> > >
> > >
> > >
> > >Here is my vote:
> > >
> > >
> > >
> > >+1 (binding)
> > >
> > >--
> > >
> > >-
> > >
> > >Best regards,
> > >
> > >
> > >
> > >Ni Chunen / George
> >
>

Re: [VOTE] Release apache-kylin-3.0.2 (RC2)

[Chao Long]

+1
mvn test passed

On Fri, May 15, 2020 at 8:43 AM ShaoFeng Shi  wrote:

> +1 (binding)
>
> Checked the signature and sha hash;
> 'Mvn test' build successfully on Java 8;
>
> Best regards,
>
> Shaofeng Shi 史少锋
> Apache Kylin PMC
> Email: shaofeng...@apache.org
>
> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> Join Kylin user mail group: user-subscr...@kylin.apache.org
> Join Kylin dev mail group: dev-subscr...@kylin.apache.org
>
>
>
>
> Xiaoxiang Yu  于2020年5月14日周四 下午11:21写道：
>
> > +1 .
> > Maven test passed and happy path passed in CDH5.7.
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> >
> > --
> >
> > Best wishes to you !
> > From ：Xiaoxiang Yu
> >
> >
> >
> >
> >
> > At 2020-05-14 17:11:28, "George Ni"  wrote:
> > >Hi all,
> > >
> > >
> > >
> > >I have created a build for Apache Kylin 3.0.2, release candidate 2.
> > >
> > >
> > >
> > >Changes highlights:
> > >
> > >[KYLIN-4390] - Update tomcat to 7.0.100
> > >
> > >[KYLIN-4426] - Refine CliCommandExecutor
> > >
> > >[KYLIN-4206] - Support Glue as Hive Metatdata
> > >
> > >
> > >
> > >Thanks to everyone who has contributed to this release.
> > >
> > >Here are the release notes:
> > >
> > >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12346975==12316121
> > >
> > >
> > >
> > >The commit to being voted upon:
> > >
> > >
> >
> https://github.com/apache/kylin/commit/57090efe4bdc079ccfde4f9c8729d69ba3a90624
> > >
> > >
> > >Its hash is 57090efe4bdc079ccfde4f9c8729d69ba3a90624.
> > >
> > >
> > >
> > >The artifacts to be voted on, including the source package and four
> > >
> > >pre-compiled binary packages are located here:
> > >
> > >https://dist.apache.org/repos/dist/dev/kylin/apache-kylin-3.0.2-rc2/
> > >
> > >
> > >
> > >The hash of the artifacts are as follows:
> > >
> > >apache-kylin-3.0.2-source-release.zip.sha256
> > >1add5892bd1d5994e0e467846e9a844758420f14819ceef63370c07a6aa0b8af
> > >
> > >apache-kylin-3.0.2-bin-hbase1x.tar.gz.sha256
> > >086397d9ecbccf80517977a4b65b660b8e1496ad097d890226bd78a34a9fe190
> > >
> > >apache-kylin-3.0.2-bin-cdh57.tar.gz.sha256
> > >181929fcd35a63a81b6dc097137a3dd1e129fd1f81400e09f64019dcb7ac8a21
> > >
> > >apache-kylin-3.0.2-bin-hadoop3.tar.gz.sha256
> > >c2250734fed971f32d242036a55ba955bcf8de91e0e73704e07cfb09124d9899
> > >
> > >apache-kylin-3.0.2-bin-cdh60.tar.gz.sha256
> > >83a68d2aec32e634475c490434981ebc91e8680dbb6388edc4ed919687ad1dac
> > >
> > >
> > >
> > >A staged Maven repository is available for review at:
> > >
> > >https://repository.apache.org/content/repositories/orgapachekylin-1078/
> > >
> > >
> > >
> > >Release artifacts are signed with the following key:
> > >
> > >https://people.apache.org/keys/committer/nic.asc
> > >
> > >
> > >
> > >Please vote on releasing this package as Apache Kylin 3.0.2.
> > >
> > >
> > >
> > >The vote is open for the next 72 hours and passes if a majority of
> > >
> > >at least three +1 PMC votes are cast.
> > >
> > >
> > >
> > >[ ] +1 Release this package as Apache Kylin 3.0.2
> > >
> > >[ ]  0 I don't feel strongly about it, but I'm okay with the release
> > >
> > >[ ] -1 Do not release this package because...
> > >
> > >
> > >
> > >
> > >
> > >Here is my vote:
> > >
> > >
> > >
> > >+1 (binding)
> > >
> > >--
> > >
> > >-
> > >
> > >Best regards,
> > >
> > >
> > >
> > >Ni Chunen / George
> >
>

[jira] [Created] (KYLIN-4464) Query ... row_number over(order by c1) ... order by c2 ... get wrong order result

[Chao Long (Jira)]

Chao Long created KYLIN-4464:


 Summary: Query ... row_number over(order by c1) ... order by c2 
... get wrong order result
 Key: KYLIN-4464
 URL: https://issues.apache.org/jira/browse/KYLIN-4464
 Project: Kylin
  Issue Type: Bug
  Components: Query Engine
Reporter: Chao Long
Assignee: Chao Long
 Fix For: Future
 Attachments: image-2020-04-20-18-19-40-925.png, 
image-2020-04-20-18-22-34-765.png

*SQL*
{code:java}
select t.*, row_number() over (order by t.lstg_format_name) as row_num from (
select lstg_format_name, sum(price) as GMV from
KYLIN_SALES
  where lstg_format_name is not null
   group by lstg_format_name
) as t
order by t.GMV desc limit 5{quote}
{code}
 

*Kylin result (order by lstg_format_name)*
 !image-2020-04-20-18-19-40-925.png!
  
 *Calcite execution plan*
{code:java}
EXECUTION PLAN BEFORE REWRITE
OLAPToEnumerableConverter
  OLAPWindowRel(window#0=[window(partition {} order by [0 DESC] rows between 
UNBOUNDED PRECEDING and CURRENT ROW aggs [ROW_NUMBER()])], ctx=[], 
groups=[[window(partition {} order by [0 DESC] rows between UNBOUNDED PRECEDING 
and CURRENT ROW aggs [ROW_NUMBER()])]])
OLAPLimitRel(ctx=[], fetch=[5])
  OLAPSortRel(sort0=[$1], dir0=[DESC], ctx=[])
OLAPAggregateRel(group=[{0}], GMV=[SUM($1)], ctx=[])
  OLAPProjectRel(LSTG_FORMAT_NAME=[$2], PRICE=[$5], ctx=[])
OLAPFilterRel(condition=[IS NOT NULL($2)], ctx=[])
  OLAPTableScan(table=[[DEFAULT, KYLIN_SALES]], ctx=[], fields=[[0, 
1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]])
{code}
 

*Hive result(order by GMV)*
!image-2020-04-20-18-22-34-765.png!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: New committer: Xiaoxiang Yu

[Chao Long]

Congratulations xiaoxiang

On Mon, Dec 30, 2019 at 9:46 AM Yaqian Zhang  wrote:

> Congratulations!
>
> > 在 2019年12月30日，09:40，Nikhil Jain  写道：
> >
> > Congratulations Xiaoxiang!
> >
> > Best Regards,
> > Nikhil Jain
> >
> > On Dec 29, 2019, at 5:18 PM, ShaoFeng Shi  > wrote:
> >
> > Hi folks,
> >
> > The Project Management Committee (PMC) for Apache Kylin
> > has invited Xiaoxiang Yu to become a committer and we are pleased to
> > announce that he has accepted.
> >
> > Xiaoxiang Yu (俞霄翔, email hit_la...@126.com)
> is one of the big data
> > engineers from Kyligence; He started to work on the Kylin project since
> the
> > middle of 2018. In the past time, he fixed many issues, investigated and
> > verified many new features (especially the v3.0 real-time streaming),
> > enhancements and bug fixes. Thank you and congratulations, Xiaoxiang!
> >
> > Let's warmly welcome Xiaoxiang as the Kylin committer!
> >
> > Best regards,
> >
> > Shaofeng Shi 史少锋
> > Apache Kylin PMC
> > Email: shaofeng...@apache.org
> >
> > Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> > Join Kylin user mail group: user-subscr...@kylin.apache.org user-subscr...@kylin.apache.org>
> > Join Kylin dev mail group: dev-subscr...@kylin.apache.org dev-subscr...@kylin.apache.org>
>
>

Re: [Announce] Apache Kylin 3.0.0 released

[Chao Long]

Congratulations! 


On Tue, Dec 24, 2019 at 7:10 PM Xiaoxiang Yu 
wrote:

> May new version of Kylin be even more powerful and stable!
>
> 
> Best wishes,
> Xiaoxiang Yu
>
>
> 在 2019/12/24 13:53，“Wang rupeng” 写入:
>
> Congratulations!
>
> ---
> Best wishes,
> Rupeng Wang
>
>
>
> 在 2019/12/24 12:08，“Xiaoyuan Gu” 写入:
>
> Big congrats! Looking forward to seeing Kylin embraces more
> state-of-the-art features. Kudos to all contributors!
>
>
> Bests,
> Xiaoyuan
>
>
>
> At 2019-12-20 20:45:16, "ShaoFeng Shi" 
> wrote:
> >The Apache Kylin team is pleased to announce the immediate
> availability of
> >the 3.0.0 release.
> >
> >This is the GA release of Kylin’s next generation after 2.x, with
> the new
> >real-time OLAP feature, Kylin can query streaming data with
> sub-second
> >latency. All of the
> > changes in this release can be found in:
> >https://kylin.apache.org/docs/release_notes.html
> >
> >
> >You can download the source release and binary packages from
> Apache Kylin's
> >download page:https://kylin.apache.org/download/
> >
> >
> >Apache Kylin is an open-source Distributed Analytics Engine
> designed to
> >provide SQL interface and multi-dimensional analysis (OLAP) on
> Apache
> >Hadoop, supporting extremely
> > large datasets.
> >
> >
> >Apache Kylin lets you query massive dataset at sub-second latency
> in 3
> >steps:
> >1. Identify a star schema or snowflake schema data set on Hadoop.
> >2. Build Cube on Hadoop.
> >3. Query data with ANSI-SQL and get results in sub-second, via
> ODBC, JDBC
> >or RESTful API.
> >
> >
> >Thanks to everyone who has contributed to this release.
> >
> >
> >We welcome your help and feedback. For more information on how to
> report
> >problems, and to get involved, visit the project website at
> >https://kylin.apache.org/
> >
> >Best regards,
> >
> >Shaofeng Shi 史少锋
> >Apache Kylin PMC
> >Email: shaofeng...@apache.org
> >
> >Apache Kylin FAQ:
> https://kylin.apache.org/docs/gettingstarted/faq.html
> >Join Kylin user mail group: user-subscr...@kylin.apache.org
> >Join Kylin dev mail group: dev-subscr...@kylin.apache.org
>
>
>
>
>
>

Re: [VOTE] Release apache-kylin-3.0.0 (RC1)

[Chao Long]

+1 real-time OLAP is coming!

mvn test passed

On Tue, Dec 10, 2019 at 3:26 PM yuzhang  wrote:

> expect it  +1
>
>
> | |
> yuzhang
> |
> |
> Email:shifengdefan...@163.com
> |
>
> Signature is customized by Netease Mail Master
>
> On 12/10/2019 14:07, ShaoFeng Shi wrote:
> Hi all,
>
> I have created a build for Apache Kylin 3.0.0, release candidate 1.
>
> Changes highlights:
> [KYLIN-4258] - Real-time OLAP may return an incorrect result for some case
> [KYLIN-4167] - Refactor streaming coordinator
> [KYLIN-4273] - Make cube planner works for real-time streaming job
> [KYLIN-4187] - Building dimension dictionary using spark
> [KYLIN-4098] - Add cube auto-merge API
>
> Thanks to everyone who has contributed to this release.
> Here are the release notes:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?version=12345005==12316121
>
> The commit to being voted upon:
>
> https://github.com/apache/kylin/commit/c75242a9b55fd57a3a58d92a2dfa9f21cfe4eebc
>
> Its hash is c75242a9b55fd57a3a58d92a2dfa9f21cfe4eebc.
>
> The artifacts to be voted on, including the source package and two
> pre-compiled binary packages are located here:
> https://dist.apache.org/repos/dist/dev/kylin/apache-kylin-3.0.0-rc1/
>
> The hash of the artifacts are as follows:
> apache-kylin-3.0.0-source-release.zip.sha256
> 9224742a87750b8d127c5031c03f3716e3af732c9805a6d0c64871605704f6c0
> apache-kylin-3.0.0-bin-hbase1x.tar.gz.sha256
> bdeddee3eb453c139eabaa2ce7ebd5d14f72d5ac48e5a64636aba2ed7357dda9
> apache-kylin-3.0.0-bin-cdh57.tar.gz.sha256
> c2ae9498f61edbacb6dae5fc32e2c4ea14539ef6d906d53194492e042c80185f
> apache-kylin-3.0.0-bin-hadoop3.tar.gz.sha256
> 116ba002d794058bd34bd05989da2c3a7ff87cf67d3647d2f1cc5b5717d445f6
> apache-kylin-3.0.0-bin-cdh60.tar.gz.sha256
> 22a0701b5a03a8d40c8b1be4fe4acb1ff2550a18c52d509b592d59ef5a094f7e
>
> A staged Maven repository is available for review at:
> https://repository.apache.org/content/repositories/orgapachekylin-1070/
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/shaofengshi.asc
>
> Please vote on releasing this package as Apache Kylin 3.0.0.
>
> The vote is open for the next 72 hours and passes if a majority of
> at least three +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Kylin 3.0.0
> [ ]  0 I don't feel strongly about it, but I'm okay with the release
> [ ] -1 Do not release this package because...
>
>
> Here is my vote:
>
> +1 (binding)
>
> Best regards,
>
> Shaofeng Shi 史少锋
> Apache Kylin PMC
> Email: shaofeng...@apache.org
>
> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> Join Kylin user mail group: user-subscr...@kylin.apache.org
> Join Kylin dev mail group: dev-subscr...@kylin.apache.org
>

Re: [ANNOUNCE] Please welcome Chunen Ni to the Apache Kylin PMC

[Chao Long]

Congratulations, Chunen.

On Mon, Dec 2, 2019 at 2:52 PM Guangxu Cheng  wrote:

> Congratulations, chunen!!!
>
> JiaTao Tao  于2019年12月2日周一 上午10:58写道：
>
> > Congratulations!
> >
> > --
> >
> > Regards!
> >
> > Aron Tao
> >
> >
> > ShaoFeng Shi  于2019年12月1日周日 上午10:47写道：
> >
> > > On behalf of the Apache Kylin PMC, I am pleased to announce that Chunen
> > Ni
> > > has accepted our invitation to become a PMC member on the Kylin
> project.
> > We
> > > appreciate Chunen stepping up to take more responsibility in the Kylin
> > > project.
> > >
> > > Please join me in welcoming Chunen to the Kylin PMC!
> > >
> > > Best regards,
> > >
> > > Shaofeng Shi 史少锋
> > > Apache Kylin PMC
> > > Email: shaofeng...@apache.org
> > >
> > > Apache Kylin FAQ:
> https://kylin.apache.org/docs/gettingstarted/faq.html
> > > Join Kylin user mail group: user-subscr...@kylin.apache.org
> > > Join Kylin dev mail group: dev-subscr...@kylin.apache.org
> > >
> > >
> > >
> >
>

[jira] [Created] (KYLIN-4262) pid in GC filename inconsistent with real pid

[Chao Long (Jira)]

Chao Long created KYLIN-4262:


 Summary: pid in GC filename inconsistent with real pid
 Key: KYLIN-4262
 URL: https://issues.apache.org/jira/browse/KYLIN-4262
 Project: Kylin
  Issue Type: Bug
Reporter: Chao Long
Assignee: Chao Long
 Attachments: image-2019-11-18-17-19-49-059.png, 
image-2019-11-18-17-19-56-990.png

pid in GC filename
!image-2019-11-18-17-19-49-059.png!
 
real pid
!image-2019-11-18-17-19-56-990.png!



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Re: New committer: Temple Zhou

[Chao Long]

Congratulations!

On Mon, Nov 18, 2019 at 10:17 AM Xiaoxiang Yu 
wrote:

> Temple Zhou , congratulations!
>
> 
> Best wishes,
> Xiaoxiang Yu
>
>
> 在 2019/11/17 22:57，“Temple Zhou” 写入:
>
> Sorry for late reply, thank you everyone.
>
> Kylin community is a very open and friendly community. I'm very
> honored to
> become a Kylin committer.
>
> I will make Kylin more reliable and excellent in every way I can. The
> more
> people join us, the better the community will be.
>
> On Sun, Nov 17, 2019, 09:15 Yaqian Zhang  wrote:
>
> > Congratulations!
> >
> > > 在 2019年11月16日，14:27，codingfor...@126.com 写道：
> > >
> > > Congratulations!
> > >
> > >
> > >> 在 2019年11月16日，13:57，nichunen  写道：
> > >>
> > >> Congratulations!
> > >
> >
> >
>
>
>

Re: [VOTE] Release apache-kylin-3.0.0-beta (RC2)

[Chao Long]

+1
mvn test passed.

On Thu, Oct 17, 2019 at 5:21 PM Li Yang  wrote:

> +1
>
> Cheers
>
> On Tue, Oct 15, 2019 at 5:00 PM ShaoFeng Shi 
> wrote:
>
> > Hi all,
> >
> > I have created a build for Apache Kylin 3.0.0-beta, release candidate 2.
> >
> > Changes highlights:
> > [KYLIN-4122] - Add Kylin user and group management modules
> > [KYLIN-4167] - Refactor streaming coordinator
> > [KYLIN-4114] - Provided a self-contained docker image for Kylin
> > [KYLIN-4137] - Accelerate metadata reloading
> >
> > Thanks to everyone who has contributed to this release.
> > Here’s the release notes:
> >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316121=12345686
> >
> > The commit to being voted upon:
> >
> >
> https://github.com/apache/kylin/commit/9d523862f29920292ab195b31c231f2f57f18594
> >
> > Its hash is 9d523862f29920292ab195b31c231f2f57f18594.
> >
> > The artifacts to be voted on, including the source package and two
> > pre-compiled binary packages are located here:
> >
> https://dist.apache.org/repos/dist/dev/kylin/apache-kylin-3.0.0-beta-rc2/
> >
> > The hash of the artifacts are as follows:
> > apache-kylin-3.0.0-beta-source-release.zip.sha256
> > 56c60b585a2033dd3867430fd053438e0c4ba3001e9b94a235118472a35e6fcf
> > apache-kylin-3.0.0-beta-bin-hbase1x.tar.gz.sha256
> > 2418466be24cd344c96c9bbbd86b08375134a68f9aabad83f8138fcf8136b879
> > apache-kylin-3.0.0-beta-bin-cdh57.tar.gz.sha256
> > 7a5b3d09229e559b27a5da45f6e3fbc6ee2d27277f55eed35c7c1f02907da328
> > apache-kylin-3.0.0-beta-bin-hadoop3.tar.gz.sha256
> > 4d16b8e7c3a8f47364b6c5946c817cbc855489cbd10c1f2f502997c887f8ca01
> > apache-kylin-3.0.0-beta-bin-cdh60.tar.gz.sha256
> > 0b8c1ff107ecae592812344e4a4fbb328628a3e51e74ee67109f918ba685b4c5
> >
> > A staged Maven repository is available for review at:
> > https://repository.apache.org/content/repositories/orgapachekylin-1069/
> >
> > Release artifacts are signed with the following key:
> > https://people.apache.org/keys/committer/shaofengshi.asc
> >
> > Please vote on releasing this package as Apache Kylin 3.0.0-beta.
> >
> > The vote is open for the next 72 hours and passes if a majority of
> > at least three +1 PMC votes are cast.
> >
> > [ ] +1 Release this package as Apache Kylin 3.0.0-beta
> > [ ]  0 I don't feel strongly about it, but I'm okay with the release
> > [ ] -1 Do not release this package because...
> >
> > Here is my vote:
> >
> > +1 (binding)
> >
> > Best regards,
> >
> > Shaofeng Shi 史少锋
> > Apache Kylin PMC
> > Email: shaofeng...@apache.org
> >
> > Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> > Join Kylin user mail group: user-subscr...@kylin.apache.org
> > Join Kylin dev mail group: dev-subscr...@kylin.apache.org
> >
>

Re: New committer: Chao Long

[Chao Long]

Thanks.

On Tue, Oct 8, 2019 at 2:24 PM Xiaoxiang Yu 
wrote:

> Chao, Congratulations!!
>
> 
> Best wishes,
> Xiaoxiang Yu
>
>
> 在 2019/10/7 10:01，“zjsy...@163.com 代表 nichunen” n...@apache.org> 写入:
>
> Congratulations!
>
>
>
>
> Best regards,
>
>
>
> Ni Chunen / George
>
>
>
> On 10/7/2019 09:22，Yichen Zhou wrote：
> Congratulations, Chao!!!
>
> Best,
> Yichen
>
> On Sun, Oct 6, 2019 at 6:19 PM ShaoFeng Shi 
> wrote:
>
> The Project Management Committee (PMC) for Apache Kylin
> has invited Chao Long to become a committer and we are pleased to
> announce
> that he has accepted.
>
> Chao Long (龙超，email: wayn...@qq.com) has started to contribute to
> Kylin
> since last year. Till today, he has made 81 commits on the master
> branch,
> resolved 71 JIRAs. His contribution includes: making fact distinct job
> in
> Spark, merging dictionary on Yarn, improving cube planner,  parquet
> storage
> PoC, and many bug fixes. Besides, he also answered many questions on
> the
> mailing lists.
>
> Congratulations, Chao!
>
> Best regards,
>
> Shaofeng Shi 史少锋
> Apache Kylin PMC
> Email: shaofeng...@apache.org
>
> Apache Kylin FAQ:
> https://kylin.apache.org/docs/gettingstarted/faq.html
> Join Kylin user mail group: user-subscr...@kylin.apache.org
> Join Kylin dev mail group: dev-subscr...@kylin.apache.org
>
>
>
>

Re: [VOTE] Release apache-kylin-2.6.4 (RC1)

[Chao Long]

+1
mvn test passed.

On Tue, Oct 8, 2019 at 5:01 PM ShaoFeng Shi  wrote:

> Hi all,
>
>
>
> I have created a build for Apache Kylin 2.6.4, release candidate 1.
>
>
>
> Changes highlights:
>
> [KYLIN-3628] - Lookup table queries always use the latest snapshot
>
> [KYLIN-3797] - Avoid out-of-memory error in Kylin server when flatting
> query filters with too many OR conditions
>
> [KYLIN-4121] - Cleanup hive view intermediate tables after job finished
>
> [KYLIN-1856] - Fix the issue that Kylin jobs show outdated output
> information when recovering from failures
>
> [KYLIN-4034] - Fix the issue that Insight page shows tables to which users
> have no access
>
> [KYLIN-4066] - Allow users who are not ROLE_ADMIN to access Planner pages
>
> [KYLIN-4131] - Fix memory leak issue within Broadcaster
>
> [KYLIN-4153] - Fix inconsistency within the transaction of metadata
> pushdown
>
> [KYLIN-4157] - Fix InternalErrorException throwing issue if
> users’PrepareStatement
> queries contain functions within WHERE clause
>
> [KYLIN-4158] - Fix wrong results caused by pushing down LIMIT condition
> when queries have expressions of columns within GROUP BY clause
>
>
>
> Thanks to everyone who has contributed to this release.
>
> Here are the release notes:
>
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316121=12345948
>
>
>
> The commit to being voted upon:
>
>
> https://github.com/apache/kylin/commit/7a8639f92f87b70dc712c6089d17120706fba87a
>
>
>
> its hash is 7a8639f92f87b70dc712c6089d17120706fba87a
>
>
>
> The artifacts to be voted on, including the source package and two
>
> pre-compiled binary packages are located here:
>
> https://dist.apache.org/repos/dist/dev/kylin/apache-kylin-2.6.4-rc1/
>
>
>
> The hash of the artifacts are as follows:
>
> apache-kylin-2.6.4-bin-cdh57.tar.gz.sha256
>
> 52c4083bd459e20e5b64672eaa23d00ccaa5a03ca7ee04f0d07d7c672b8e1974
>
> apache-kylin-2.6.4-bin-cdh60.tar.gz.sha256
>
> 4b369073fb6aff2257d6c62abf1415917a46ab6ae6014889d037f4b52603dbf5
>
> apache-kylin-2.6.4-bin-hadoop3.tar.gz.sha256
>
> b3939e78bd11830b792a05b035e3027f8ea13f84178cfd718b9ec83204958f5f
>
> apache-kylin-2.6.4-bin-hbase1x.tar.gz.sha256
>
> adb763495f3ba132045842c1d2f8d99def8f6a8cf3aadff60087e8dc110ed674
>
> apache-kylin-2.6.4-source-release.zip.sha256
>
> 3faad44f24830efff6e1f799caf09db70409e9ed3d3501afcf68c96f237f978c
>
>
>
> A staged Maven repository is available for review at:
>
> https://repository.apache.org/content/repositories/orgapachekylin-1068/
>
>
>
> Release artifacts are signed with the following key:
>
> https://people.apache.org/keys/committer/shaofengshi.asc
>
>
>
> Please vote on releasing this package as Apache Kylin 2.6.4.
>
>
>
> The vote is open for the next 72 hours and passes if a majority of
>
> at least three +1 PMC votes are cast.
>
>
>
> [ ] +1 Release this package as Apache Kylin 2.6.4
>
> [ ]  0 I don't feel strongly about it, but I'm okay with the release
>
> [ ] -1 Do not release this package because...
>
>
>
> Here is my vote:
>
>
>
> +1 (binding)
>
>
> Best regards,
>
> Shaofeng Shi 史少锋
> Apache Kylin PMC
> Email: shaofeng...@apache.org
>
> Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> Join Kylin user mail group: user-subscr...@kylin.apache.org
> Join Kylin dev mail group: dev-subscr...@kylin.apache.org
>

Re: [VOTE] Release apache-kylin-3.0.0-beta (RC1)

[Chao Long]

+1
mvn test passed

On Thu, Sep 26, 2019 at 8:44 PM Yaqian Zhang  wrote:

> +1
> mvn test passed
>
> > 在 2019年9月26日，20:10，nichunen  写道：
> >
> > +1
> >
> >
> >
> > Best regards,
> >
> >
> >
> > Ni Chunen / George
> >
> >
> >
> > On 09/26/2019 16:41，ShaoFeng Shi wrote：
> > Hi all,
> >
> > I have created a build for Apache Kylin 3.0.0-beta, release candidate 1.
> >
> > Changes highlights:
> > [KYLIN-4122] - Add Kylin user and group management modules
> > [KYLIN-4167] - Refactor streaming coordinator
> > [KYLIN-4114] - Provided a self-contained docker image for Kylin
> > [KYLIN-4137] - Accelerate metadata reloading
> >
> > Thanks to everyone who has contributed to this release.
> > Here’s the release notes:
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316121=12345686
> >
> > The commit to being voted upon:
> >
> https://github.com/apache/kylin/commit/721be80866223fecad9a6231fa2427a847bc8f48
> >
> > Its hash is 721be80866223fecad9a6231fa2427a847bc8f48.
> >
> > The artifacts to be voted on, including the source package and two
> > pre-compiled binary packages, are located here:
> >
> https://dist.apache.org/repos/dist/dev/kylin/apache-kylin-3.0.0-beta-rc1/
> >
> > The hash of the artifacts are as follows:
> > apache-kylin-3.0.0-beta-source-release.zip.sha256
> > 53547e8a94eb74cdcd329777ff03f1c79209020016c2f9a62351e8c73ac8e0bd
> > apache-kylin-3.0.0-beta-bin-hbase1x.tar.gz.sha256
> > 1d50348660899baa9005b78cf45243e0eb2495fa0403d6250b3439ff50bf1731
> > apache-kylin-3.0.0-beta-bin-cdh57.tar.gz.sha256
> > bc9e303154901d4061dbac3876157cb4be25f23307f4c709d083da70aa18524b
> > apache-kylin-3.0.0-beta-bin-hadoop3.tar.gz.sha256
> > 681452450248f56ebe107d278e3ccb1478e42137875a2dded953db8c03488f9a
> > apache-kylin-3.0.0-beta-bin-cdh60.tar.gz.sha256
> > 2f66497ed39d7d78ea5a634a8796ab408586dce369edc97ed9374ba90a88b03d
> >
> > A staged Maven repository is available for review at:
> > https://repository.apache.org/content/repositories/orgapachekylin-1066/
> >
> > Release artifacts are signed with the following key:
> > https://people.apache.org/keys/committer/shaofengshi.asc
> >
> > Please vote on releasing this package as Apache Kylin 3.0.0-beta.
> >
> > The vote is open for the next 72 hours and passes if a majority of
> > at least three +1 PMC votes are cast.
> >
> > [ ] +1 Release this package as Apache Kylin 3.0.0-beta
> > [ ]  0 I don't feel strongly about it, but I'm okay with the release
> > [ ] -1 Do not release this package because...
> >
> > Best regards,
> >
> > Shaofeng Shi 史少锋
> > Apache Kylin PMC
> > Email: shaofeng...@apache.org
> >
> > Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> > Join Kylin user mail group: user-subscr...@kylin.apache.org
> > Join Kylin dev mail group: dev-subscr...@kylin.apache.org
>
>

[jira] [Created] (KYLIN-4154) Metadata inconsistency between multi Kylin server caused by Broadcaster closing

[Chao Long (Jira)]

Chao Long created KYLIN-4154:


 Summary: Metadata inconsistency between multi Kylin server caused 
by Broadcaster closing 
 Key: KYLIN-4154
 URL: https://issues.apache.org/jira/browse/KYLIN-4154
 Project: Kylin
  Issue Type: Bug
  Components: Metadata
Reporter: Chao Long
Assignee: Chao Long
 Fix For: v3.0.0


To avoid Broadcaster memory leak, KYLIN-4131 close the metadata sync thread 
after receiving "Sync All" event, but there may be some events in the waiting 
queue, which haven't been synced yet. So we should sync all events before 
closing sync thread.



--
This message was sent by Atlassian Jira
(v8.3.2#803003)

Re: kylin 写入数据

[Chao Long]

Hi wangying,
What do you mean by "synchronously"?

1. A user write data from the front-end, for example creating a Cube, other
users can see the new data after refreshing the front-end page if they has
the permission to see it. Even if there are more than one Kylin servers,
one Kylin server can see the new data write by other server.

2. If the new record has been built in Cube, the other users can see the
new aggregation result.

On Mon, Sep 2, 2019 at 10:54 AM wangying <18611156...@163.com> wrote:

> Hi,
> I see the introduction of kylin from the website and are very interested
> in it. Thank you for your contribution.
> I plan to apply kylin to our product but still have some questions.It
> would be greatly appreciated if you can kindly give me some feedback.
>
> Firstly, if a user write data from the front end system, will other users
> be able to see the new data synchronously?
>
> Secondly, if a user write data in the lowest level,for example,"china" is
> the parebt dimension, if I add a new record"beijing,200", will other users
> be able to see the new aggregation result of  "China" increased by 200
> syschronously?
>
> Sincerely,
> Arya Wang
>
>
> | |
> 对外经贸-王影
> |
> |
> 邮箱：18611156...@163.com
> |
>
> 签名由 网易邮箱大师 定制

Re: Not able to start KYLIN

[Chao Long]

Hi Moulali Shaik,
 What's your Kylin version, and hbase version?

On Thu, Aug 8, 2019 at 6:06 AM Moulali Shaik 
wrote:

> Hi,
>
> I am unable to start the KYLIN in hortonworks sandbox, it is throwing below
> error
>
> It would be great if you could find solution for this or please let me know
> if I am missing something
>
>
> 2019-08-06 16:39:46,532 DEBUG [main] hbase.HBaseConnection:181 : Using the
> working dir FS for HBase: hdfs://XXX:8020
>
> 2019-08-06 16:39:46,532 INFO  [main] hbase.HBaseConnection:258 : connection
> is null or closed, creating a new one
>
> 2019-08-06 16:39:46,533 ERROR [main] hbase.HBaseConnection:271 : Error when
> open connection
> kylin_metadata@hbase
> ,hbase.client.scanner.timeout.period=1,hbase.rpc.timeout=5000,hbase.client.retries.number=1
>
> java.lang.NoClassDefFoundError:
> org/apache/hadoop/hbase/client/ConnectionFactory
>
> at
>
> org.apache.kylin.storage.hbase.HBaseConnection.get(HBaseConnection.java:259)
>
> at
>
> org.apache.kylin.storage.hbase.HBaseResourceStore.getConnection(HBaseResourceStore.java:85)
>
> at
>
> org.apache.kylin.storage.hbase.HBaseResourceStore.createHTableIfNeeded(HBaseResourceStore.java:111)
>
> at
>
> org.apache.kylin.storage.hbase.HBaseResourceStore.(HBaseResourceStore.java:92)
>
> at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>
> at
>
> sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:57)
>
> at
>
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
>
> at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>
> at
>
> org.apache.kylin.common.persistence.ResourceStore.createResourceStore(ResourceStore.java:90)
>
> at
>
> org.apache.kylin.common.persistence.ResourceStore.getStore(ResourceStore.java:108)
>
> at
>
> org.apache.kylin.rest.service.AclTableMigrationTool.checkIfNeedMigrate(AclTableMigrationTool.java:94)
>
> at
>
> org.apache.kylin.tool.AclTableMigrationCLI.main(AclTableMigrationCLI.java:41)
>
> Caused by: java.lang.ClassNotFoundException:
> org.apache.hadoop.hbase.client.ConnectionFactory
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:366)
>
> at java.net.URLClassLoader$1.run(URLClassLoader.java:355)
>
> at java.security.AccessController.doPrivileged(Native Method)
>
> at java.net.URLClassLoader.findClass(URLClassLoader.java:354)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:425)
>
> at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308)
>
> at java.lang.ClassLoader.loadClass(ClassLoader.java:358)
>
> ... 12 more
>
> Exception in thread "main" java.lang.IllegalArgumentException: Failed to
> find metadata store by url: kylin_metadata@hbase
>
> Regards,
> Moulali
>
> --
> You Can't Judge A Book By It's Cover
>

[jira] [Created] (KYLIN-4131) Broadcaster memory leak

[Chao Long (JIRA)]

Chao Long created KYLIN-4131:


 Summary: Broadcaster memory leak
 Key: KYLIN-4131
 URL: https://issues.apache.org/jira/browse/KYLIN-4131
 Project: Kylin
  Issue Type: Bug
  Components: Metadata
Reporter: Chao Long
Assignee: Chao Long
 Fix For: v3.0.0
 Attachments: image-2019-08-12-16-39-46-262.png, 
image-2019-08-12-16-39-52-327.png, image-2019-08-12-17-16-05-532.png, 
image-2019-08-12-17-22-15-783.png, image-2019-08-12-17-22-23-477.png

h3. *Test evidence*

Use MAT(Memory Analyzer) to analyze this issue.
 # start  Kylin, dump a jvm heap info snapshot and import into MAT. We can see 
there is only one Broadcaster and ProjectManager object.
!image-2019-08-12-17-22-15-783.png!

 # Call 'list cube' api and 'reload metadata' api many times. We can see there 
are many BroadCaster and Manager objects in the memory, which should be 
recycled by GC, but they are not.
!image-2019-08-12-17-22-23-477.png!

 

*Root cause*

Broadcaster hold the ref of Listener object defined in Manager, and the 
Listener object hold the ref of Manager, because the impl class of Listener is 
an inner class of Manager.

Broadcaster.java
{code:java}
private Map> listenerMap = Maps.newConcurrentMap();
{code}
 ProjectManager.java -> ProjectSyncListener
{code:java}
private class ProjectSyncListener extends Broadcaster.Listener {

@Override
public void onEntityChange(Broadcaster broadcaster, String entity, Event event, 
String cacheKey)
throws IOException {
String project = cacheKey;

if (event == Event.DROP) {
removeProjectLocal(project);
return;
}

reloadProjectQuietly(project);
broadcaster.notifyProjectSchemaUpdate(project);
broadcaster.notifyProjectDataUpdate(project);
}
}

{code}
And Broadcaster can't be recycled by GC because it holds a running thread, 
which cause the Manager object also not being recycled.

*After Fix*

 The Broadcaster objects and Manager objects are recycled by GC.

!image-2019-08-12-17-16-05-532.png!

 



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (KYLIN-4130) Coordinator->StreamingBuildJobStatusChecker thread always hold a old CubeManager

[Chao Long (JIRA)]

Chao Long created KYLIN-4130:


 Summary: Coordinator->StreamingBuildJobStatusChecker thread always 
hold a old CubeManager
 Key: KYLIN-4130
 URL: https://issues.apache.org/jira/browse/KYLIN-4130
 Project: Kylin
  Issue Type: Improvement
  Components: Real-time Streaming
Reporter: Chao Long
Assignee: Chao Long
 Fix For: v3.0.0


{code}
private class StreamingBuildJobStatusChecker implements Runnable {
private int maxJobTryCnt = 5;
private CubeManager cubeManager = 
CubeManager.getInstance(KylinConfig.getInstanceFromEnv());
private ConcurrentMap> segmentBuildJobMap = Maps
.newConcurrentMap();
private CopyOnWriteArrayList pendingCubeName = 
Lists.newCopyOnWriteArrayList();
{code}



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: [VOTE] Release apache-kylin-3.0.0-alpha2 (RC1)

[Chao Long]

+1
mvn test passed

On Fri, Jul 26, 2019 at 10:38 AM Temple Zhou  wrote:

> +1
>
> On Fri, Jul 26, 2019 at 10:00 AM nichunen  wrote:
>
> > +1
> >
> >
> >
> > Best regards,
> >
> >
> >
> > Ni Chunen / George
> >
> >
> >
> > On 07/26/2019 09:48，JiaTao Tao wrote：
> > +1
> >
> > --
> >
> > Regards!
> >
> > Aron Tao
> >
> > ShaoFeng Shi  于2019年7月25日周四 下午2:46写道：
> >
> > Hi all,
> >
> >
> >
> > I have created a build for Apache Kylin 3.0.0-alpha2, release candidate
> 1.
> >
> >
> >
> > Changes highlights:
> >
> > [KYLIN-3942] - Rea-time OLAP supports multi-level json event
> >
> > [KYLIN-4086] - Support connect Kylin with Tableau by JDBC
> >
> > [KYLIN-3841] - Build Global Dict by MR/Hive
> >
> > [KYLIN-4017] - Fix building engine failed to get zk lock and leads to the
> > building engine doesn't work
> >
> > [KYLIN-3843] - List kylin instances with their server mode on web
> >
> > [KYLIN-3997] - Add a health check job of Kylin
> >
> > [KYLIN-4028] - Speed up startup progress using cached dependency
> >
> > [KYLIN-4035] - Calculate column cardinality by using spark engine
> >
> >
> >
> > Thanks to everyone who has contributed to this release.
> >
> > Here’s the release notes:
> >
> >
> >
> >
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316121=12345840
> >
> >
> >
> > The commit to being voted upon:
> >
> >
> >
> >
> >
> >
> https://github.com/apache/kylin/commit/06f441cd04a98a988197f7b1d750608d6ee51cd8
> >
> >
> >
> > Its hash is 06f441cd04a98a988197f7b1d750608d6ee51cd8.
> >
> >
> >
> > The artifacts to be voted on, including the source package and two
> > pre-compiled
> > binary packages, are located here:
> >
> >
> https://dist.apache.org/repos/dist/dev/kylin/apache-kylin-3.0.0-alpha2-rc1/
> >
> >
> >
> > The hash of the artifacts are as follows:
> >
> > apache-kylin-3.0.0-alpha2-source-release.zip.sha256
> >
> > dc206ab0527703271e4cc435368a4f0980247465947a199396567dfc776bf7f8
> >
> > apache-kylin-3.0.0-alpha2-bin-hbase1x.tar.gz.sha256
> >
> > 406f2d7ea318df230d1e65600f4ffccb22accfeee47a7e2f85ee4f70bbc6bbdc
> >
> > apache-kylin-3.0.0-alpha2-bin-cdh57.tar.gz.sha256
> >
> > 1dadf9910d07ed5af233477a92013a8b6090ae18b3f8d9b48c7ee915acbafe25
> >
> > apache-kylin-3.0.0-alpha2-bin-hadoop3.tar.gz.sha256
> >
> > f0d075b0e2bc3c0953bf17c7220ddae03ae611fae03456fad215b36f9393eb28
> >
> > apache-kylin-3.0.0-alpha2-bin-cdh60.tar.gz.sha256
> >
> > 47faa1810f21fa8c4c109b82af9eb09d44c2f0d14d245a94655aaf783780a143
> >
> >
> >
> >
> >
> > A staged Maven repository is available for review at:
> >
> > https://repository.apache.org/content/repositories/orgapachekylin-1065/
> >
> >
> >
> > Release artifacts are signed with the following key:
> >
> > https://people.apache.org/keys/committer/shaofengshi.asc
> >
> >
> >
> > Please vote on releasing this package as Apache Kylin 3.0.0-alpha2.
> >
> >
> >
> > The vote is open for the next 72 hours and passes if a majority of
> >
> > at least three +1 PMC votes are cast.
> >
> >
> >
> > [ ] +1 Release this package as Apache Kylin 3.0.0-alpha2
> >
> > [ ]  0 I don't feel strongly about it, but I'm okay with the release
> >
> > [ ] -1 Do not release this package because...
> >
> >
> >
> >
> >
> > Here is my vote:
> >
> >
> >
> > +1 (binding)
> >
> >
> > Best regards,
> >
> > Shaofeng Shi 史少锋
> > Apache Kylin PMC
> > Email: shaofeng...@apache.org
> >
> > Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
> > Join Kylin user mail group: user-subscr...@kylin.apache.org
> > Join Kylin dev mail group: dev-subscr...@kylin.apache.org
> >
> >
>

[jira] [Created] (KYLIN-4108) Show slow query hit cube in slow query page

[Chao Long (JIRA)]

Chao Long created KYLIN-4108:


 Summary: Show slow query hit cube in slow query page
 Key: KYLIN-4108
 URL: https://issues.apache.org/jira/browse/KYLIN-4108
 Project: Kylin
  Issue Type: Improvement
  Components: Metrics
Reporter: Chao Long
Assignee: Chao Long
 Fix For: v3.0.0






--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (KYLIN-4089) Integration test failed with JDBCMetastore

[Chao Long (JIRA)]

Chao Long created KYLIN-4089:


 Summary: Integration test failed with JDBCMetastore
 Key: KYLIN-4089
 URL: https://issues.apache.org/jira/browse/KYLIN-4089
 Project: Kylin
  Issue Type: Bug
  Components: Integration, Tools, Build and Test
Reporter: Chao Long
Assignee: Chao Long
 Fix For: v3.0.0-alpha2


[ERROR] Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.822 s 
<<< FAILURE! - in 
org.apache.kylin.storage.hbase.ITAclTableMigrationToolTest[ERROR] 
testBasic(org.apache.kylin.storage.hbase.ITAclTableMigrationToolTest)  Time 
elapsed: 0.812 s  <<< ERROR!java.lang.NullPointerException
at 
org.apache.kylin.storage.hbase.ITAclTableMigrationToolTest.testBasic(ITAclTableMigrationToolTest.java:95)



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

[jira] [Created] (KYLIN-4085) Segment parallel building may cause segment not found

[Chao Long (JIRA)]

Chao Long created KYLIN-4085:


 Summary: Segment parallel building may cause segment not found
 Key: KYLIN-4085
 URL: https://issues.apache.org/jira/browse/KYLIN-4085
 Project: Kylin
  Issue Type: Bug
  Components: Metadata
Reporter: Chao Long
Assignee: Chao Long






--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

Re: Re: cannot get HiveTableMeta

[Chao Long]

This is a known issue and have been fixed in
https://issues.apache.org/jira/browse/KYLIN-3515. You can upgrade your
Kylin to v2.6.0 or above.

On Wed, Jul 10, 2019 at 4:39 PM 曼  wrote:

> The Kylin version is 1.6
>
>
> At 2019-07-10 10:36:03, "Chao Long"  wrote:
> >Hi
> >  What's your Kylin version?
> >
> >On Tue, Jul 9, 2019 at 6:42 PM 曼  wrote:
> >
> >> There are two cubes using the same dimension table, and when both cubes
> >> are built at the same time, both will do delete and create operations,
> >> causing one of them to fail, Please see the log
> >> pool-9-thread-10 and pool-9-thread-9 are two cube builds that are
> executed
> >> separately, pool-9-thread-10 failed
> >>
> >>
> >> 2019-07-09 15:49:03,870 INFO [pool-9-thread-10]
> >> common.ShellExecutable:52 : executing:hive -e "USE kylin_flat_db;
> >> SET dfs.replication=2;
> >> SET hive.exec.compress.output=true;
> >> SET hive.auto.convert.join.noconditionaltask=true;
> >> SET hive.auto.convert.join.noconditionaltask.size=1;
> >> SET mapreduce.output.fileoutputformat.compress.type=BLOCK;
> >> SET mapreduce.job.split.metainfo.maxsize=-1;
> >>
> >>
> >> DROP TABLE IF EXISTS
> >> kylin_intermediate_EDW_D_EDW_HOTEL_PROFILE_INFO_FIN_ADD_INDEX_VIEW;
> >> CREATE TABLE IF NOT EXISTS
> >> kylin_intermediate_EDW_D_EDW_HOTEL_PROFILE_INFO_FIN_ADD_INDEX_VIEW
> >> LOCATION
> >>
> '/kylin/kylin_metadata/kylin-88e0f1f2-c55e-44a2-88a9-37be8e07f445/kylin_intermediate_EDW_D_EDW_HOTEL_PROFILE_INFO_FIN_ADD_INDEX_VIEW'
> >> AS SELECT * FROM EDW.D_EDW_HOTEL_PROFILE_INFO_FIN_ADD_INDEX_VIEW;
> >>
> >>
> >> "
> >>
> >>
> >>
> >>
> >>
> >>
> >> 2019-07-09 15:49:53,997 INFO [pool-9-thread-9]
> >> common.ShellExecutable:52 : executing:hive -e "USE kylin_flat_db;
> >> SET dfs.replication=2;
> >> SET hive.exec.compress.output=true;
> >> SET hive.auto.convert.join.noconditionaltask=true;
> >> SET hive.auto.convert.join.noconditionaltask.size=1;
> >> SET mapreduce.output.fileoutputformat.compress.type=BLOCK;
> >> SET mapreduce.job.split.metainfo.maxsize=-1;
> >>
> >>
> >> DROP TABLE IF EXISTS
> >> kylin_intermediate_EDW_D_EDW_HOTEL_PROFILE_INFO_FIN_ADD_INDEX_VIEW;
> >> CREATE TABLE IF NOT EXISTS
> >> kylin_intermediate_EDW_D_EDW_HOTEL_PROFILE_INFO_FIN_ADD_INDEX_VIEW
> >> LOCATION
> >>
> '/kylin/kylin_metadata/kylin-feb1f455-3fa1-4d14-ba7c-b5d5fb9969b4/kylin_intermediate_EDW_D_EDW_HOTEL_PROFILE_INFO_FIN_ADD_INDEX_VIEW'
> >> AS SELECT * FROM EDW.D_EDW_HOTEL_PROFILE_INFO_FIN_ADD_INDEX_VIEW;
> >>
> >>
> >> "
> >>
> >>
> >> 2019-07-09 15:50:07,991 ERROR [pool-9-thread-10]
> >> common.HadoopShellExecutable:65 : error execute
> >> HadoopShellExecutable{id=88e0f1f2-c55e-44a2-88a9-37be8e07f445-04,
> >> name=Build Dimension Dicti
> >> onary, state=RUNNING}
> >> java.lang.RuntimeException: cannot get HiveTableMeta
> >> at
> org.apache.kylin.source.hive.HiveTable.(HiveTable.java:49)
> >> at
> >>
> org.apache.kylin.source.hive.HiveSource.createReadableTable(HiveSource.java:49)
> >> at
> >>
> org.apache.kylin.source.SourceFactory.createReadableTable(SourceFactory.java:42)
> >> at
> >>
> org.apache.kylin.cube.CubeManager.buildSnapshotTable(CubeManager.java:269)
> >> at
> >>
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:58)
> >> at
> >>
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:41)
> >> at org.apache.kylin.engine.mr
> >> .steps.CreateDictionaryJob.run(CreateDictionaryJob.java:54)
> >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> >> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> >> at org.apache.kylin.engine.mr
> >> .common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
> >> at
> >>
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
> >> at
> >>
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:57)
> >> at
> >>
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113

Re: cannot get HiveTableMeta

[Chao Long]

Hi
  What's your Kylin version?

On Tue, Jul 9, 2019 at 6:42 PM 曼  wrote:

> There are two cubes using the same dimension table, and when both cubes
> are built at the same time, both will do delete and create operations,
> causing one of them to fail, Please see the log
> pool-9-thread-10 and pool-9-thread-9 are two cube builds that are executed
> separately, pool-9-thread-10 failed
>
>
> 2019-07-09 15:49:03,870 INFO [pool-9-thread-10]
> common.ShellExecutable:52 : executing:hive -e "USE kylin_flat_db;
> SET dfs.replication=2;
> SET hive.exec.compress.output=true;
> SET hive.auto.convert.join.noconditionaltask=true;
> SET hive.auto.convert.join.noconditionaltask.size=1;
> SET mapreduce.output.fileoutputformat.compress.type=BLOCK;
> SET mapreduce.job.split.metainfo.maxsize=-1;
>
>
> DROP TABLE IF EXISTS
> kylin_intermediate_EDW_D_EDW_HOTEL_PROFILE_INFO_FIN_ADD_INDEX_VIEW;
> CREATE TABLE IF NOT EXISTS
> kylin_intermediate_EDW_D_EDW_HOTEL_PROFILE_INFO_FIN_ADD_INDEX_VIEW
> LOCATION
> '/kylin/kylin_metadata/kylin-88e0f1f2-c55e-44a2-88a9-37be8e07f445/kylin_intermediate_EDW_D_EDW_HOTEL_PROFILE_INFO_FIN_ADD_INDEX_VIEW'
> AS SELECT * FROM EDW.D_EDW_HOTEL_PROFILE_INFO_FIN_ADD_INDEX_VIEW;
>
>
> "
>
>
>
>
>
>
> 2019-07-09 15:49:53,997 INFO [pool-9-thread-9]
> common.ShellExecutable:52 : executing:hive -e "USE kylin_flat_db;
> SET dfs.replication=2;
> SET hive.exec.compress.output=true;
> SET hive.auto.convert.join.noconditionaltask=true;
> SET hive.auto.convert.join.noconditionaltask.size=1;
> SET mapreduce.output.fileoutputformat.compress.type=BLOCK;
> SET mapreduce.job.split.metainfo.maxsize=-1;
>
>
> DROP TABLE IF EXISTS
> kylin_intermediate_EDW_D_EDW_HOTEL_PROFILE_INFO_FIN_ADD_INDEX_VIEW;
> CREATE TABLE IF NOT EXISTS
> kylin_intermediate_EDW_D_EDW_HOTEL_PROFILE_INFO_FIN_ADD_INDEX_VIEW
> LOCATION
> '/kylin/kylin_metadata/kylin-feb1f455-3fa1-4d14-ba7c-b5d5fb9969b4/kylin_intermediate_EDW_D_EDW_HOTEL_PROFILE_INFO_FIN_ADD_INDEX_VIEW'
> AS SELECT * FROM EDW.D_EDW_HOTEL_PROFILE_INFO_FIN_ADD_INDEX_VIEW;
>
>
> "
>
>
> 2019-07-09 15:50:07,991 ERROR [pool-9-thread-10]
> common.HadoopShellExecutable:65 : error execute
> HadoopShellExecutable{id=88e0f1f2-c55e-44a2-88a9-37be8e07f445-04,
> name=Build Dimension Dicti
> onary, state=RUNNING}
> java.lang.RuntimeException: cannot get HiveTableMeta
> at org.apache.kylin.source.hive.HiveTable.(HiveTable.java:49)
> at
> org.apache.kylin.source.hive.HiveSource.createReadableTable(HiveSource.java:49)
> at
> org.apache.kylin.source.SourceFactory.createReadableTable(SourceFactory.java:42)
> at
> org.apache.kylin.cube.CubeManager.buildSnapshotTable(CubeManager.java:269)
> at
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:58)
> at
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:41)
> at org.apache.kylin.engine.mr
> .steps.CreateDictionaryJob.run(CreateDictionaryJob.java:54)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at org.apache.kylin.engine.mr
> .common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:57)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:113)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:136)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
> at java.lang.Thread.run(Thread.java:745)
> Caused by:
> NoSuchObjectException(message:KYLIN_FLAT_DB.kylin_intermediate_EDW_D_EDW_HOTEL_PROFILE_INFO_FIN_ADD_INDEX_VIEW
> table not found)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_table_result$get_table_resultStandardScheme.read(ThriftHiveMetastore.java:35066)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_table_result$get_table_resultStandardScheme.read(ThriftHiveMetastore.java:35034)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$get_table_result.read(ThriftHiveMetastore.java:34965)
> at
> org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:78)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_get_table(ThriftHiveMetastore.java:1218)
> at
> org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.get_table(ThriftHiveMetastore.java:1204)
> at
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTable(HiveMetaStoreClient.java:1209)
> at
> 

[jira] [Created] (KYLIN-4064) parameter 'engineType' is not working when running integration test

[Chao Long (JIRA)]

Chao Long created KYLIN-4064:


 Summary: parameter 'engineType' is not working when running 
integration test
 Key: KYLIN-4064
 URL: https://issues.apache.org/jira/browse/KYLIN-4064
 Project: Kylin
  Issue Type: Bug
  Components: Integration
Reporter: Chao Long
Assignee: Chao Long
 Fix For: v3.0.0-beta






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [VOTE] Release apache-kylin-2.6.3 (RC1)

[Chao Long]

+1
mvn test passed

On Mon, Jul 1, 2019 at 2:09 PM Cheng wang  wrote:

> ＋1(binding)
>
> Best regards,
> Cheng Wang
>
>
> On 7/1/19, 9:27 AM, "ShaoFeng Shi"  wrote:
>
> Hi all,
>
> I have created a build for Apache Kylin 2.6.3, release candidate 1.
>
> Changes highlights:
> - [KYLIN-4024] - Support pushdown to Presto
> - [KYLIN-3977] - Avoid mistaken deleting dicts by storage cleanup while
> building jobs are running
> - [KYLIN-4015] – Fix build cube error at the "Build UHC Dictionary"
> step
> - [KYLIN-4022] - Error with message "Unrecognized column type:
> DECIMAL(xx,xx)" happens when doing query pushdown
>
> Thanks to everyone who has contributed to this release.
> Here’s release notes:
>
> https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316121=12345582
>
> The commit to being voted upon:
>
>
> https://github.com/apache/kylin/commit/0d5f85b0a40c301134122de927204a0d17ad65fa
>
> Its hash is 0d5f85b0a40c301134122de927204a0d17ad65fa.
>
> The artifacts to be voted on are located here:
> https://dist.apache.org/repos/dist/dev/kylin/apache-kylin-2.6.3-rc1/
>
> The hash of the artifact is as follows:
> apache-kylin-2.6.3-source-release.zip.sha256
> 50d1cad423f1a15a5e25f1c3c68748c7ce10e0116fd67fa9e38c1470a11d389c
>
> A staged Maven repository is available for review at:
>
> https://repository.apache.org/content/repositories/orgapachekylin-1063/
>
> Release artifacts are signed with the following key:
> https://people.apache.org/keys/committer/shaofengshi.asc
>
> Please vote on releasing this package as Apache Kylin 2.6.3.
>
> The vote is open for the next 72 hours and passes if a majority of
> at least three +1 PMC votes are cast.
>
> [ ] +1 Release this package as Apache Kylin 2.6.3
> [ ]  0 I don't feel strongly about it, but I'm okay with the release
> [ ] -1 Do not release this package because...
>
>
> Here is my vote:
>
> +1 (binding)
>
> Best regards,
>
> Shaofeng Shi 史少锋
> Apache Kylin PMC
> Email: shaofeng...@apache.org
>
> Apache Kylin FAQ:
> https://kylin.apache.org/docs/gettingstarted/faq.html
> Join Kylin user mail group: user-subscr...@kylin.apache.org
> Join Kylin dev mail group: dev-subscr...@kylin.apache.org
>
>
>

[jira] [Created] (KYLIN-4049) Refresh segment job will always delete old segment storage

[Chao Long (JIRA)]

Chao Long created KYLIN-4049:


 Summary: Refresh segment job will always delete old segment storage
 Key: KYLIN-4049
 URL: https://issues.apache.org/jira/browse/KYLIN-4049
 Project: Kylin
  Issue Type: Bug
  Components: Job Engine
Reporter: Chao Long
Assignee: Chao Long
 Fix For: v3.0.0, v2.6.3






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KYLIN-4045) ParquetMROutput miss megering cuboid data step

[Chao Long (JIRA)]

Chao Long created KYLIN-4045:


 Summary: ParquetMROutput miss megering cuboid data step
 Key: KYLIN-4045
 URL: https://issues.apache.org/jira/browse/KYLIN-4045
 Project: Kylin
  Issue Type: Bug
Reporter: Chao Long
Assignee: Chao Long
 Fix For: Future






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: [ANNOUNCE] New Committer: Jiatao Tao

[Chao Long]

Congratulations！

On Thu, Jun 13, 2019 at 11:00 AM Billy Liu  wrote:

> Congrats
>
> With Warm regards
>
> Billy Liu
>
> PENG Zhengshuai  于2019年6月13日周四 上午10:48写道：
> >
> > Congrats to Jiatao!
> >
> > Best Regards
> > PENG Zhengshuai
> >
> > > On Jun 13, 2019, at 10:46 AM, ShaoFeng Shi 
> wrote:
> > >
> > > The Project Management Committee (PMC) for Apache Kylin
> > > has invited Jiatao Tao to become a committer and we are pleased
> > > to announce that he has accepted.
> > >
> > > Thanks for all your hard work Jiatao; we look forward to more
> > > contributions!
> > >
> > > Please join me in extending congratulations to Jiatao!
> > >
> > > Best regards,
> > >
> > > Shaofeng Shi 史少锋
> > > Apache Kylin PMC
> > > Email: shaofeng...@apache.org
> > >
> > > Apache Kylin FAQ:
> https://kylin.apache.org/docs/gettingstarted/faq.html
> > > Join Kylin user mail group: user-subscr...@kylin.apache.org
> > > Join Kylin dev mail group: dev-subscr...@kylin.apache.org
> >
>

Re: questions about kylin on parquet

[Chao Long]

Hi,
Yes, you are right, but only ParquetMROutput miss the merging cuboid data
step, because it will been done in SparkBatchMergeJobBuilder2 when using
spark engine.
Would you like to create a jira and fix it? thank you.

On Tue, Jun 11, 2019 at 6:22 PM pannenggang <181594...@qq.com> wrote:

> Hi
>
> I found a branch named “kylin-on-parquet” here:
> https://github.com/apache/kylin/tree/kylin-on-parquet. I am working on a
> little poc  based on it. I have a question about the function
> “getBatchMergeOutputSide” in class org.apache.kylin.storage.parquet.steps.
> ParquetMROutput(similarly ParquetSparkOutput), basically, I think there is
> no actually merge steps here, why is that? No need to do a merge step here?
>
> Best Regards
>

Re: How to build Kylin for HDP 3.

[Chao Long]

Hi  Александр Сидорчук
  You are right, the script sync_hbase_cdh_branches.sh do the similar thing
as step1 and step2,
but there may be some conflicts which should be resolved manually when
rebasing or applying patch. So I think those steps should be done manually.

On Wed, May 15, 2019 at 6:52 PM alex.sidorchuk85 
wrote:

> Hello Long Chao,
>
> Thank you for unswer.
>
> i'm doing next:
> 1. Check out master-hadoop3.1
> 2. Rebase HEAD to ... and choose master branch.
> 3. run *build/script/package.sh* to build package for HDP 3.0 (as you said)
>
> One more question
>
> There are script in dev-support/sync_hbase_cdh_branches.sh
> Is this script must be rewritten to use master-hadoop3.1 instead of
> master-hbase0.98 in he same manner, and this script will do automatically
> step 1 and 2 for me?
> am i right?
>
> I can actualize it, if it is true.
>
> --
> Sent from: http://apache-kylin.74782.x6.nabble.com/
>

Re: [VOTE] Release apache-kylin-2.6.2 (RC1)

[Chao Long]

+1
mvn test passed

On Tue, May 14, 2019 at 12:05 PM 李 栋  wrote:

> +1 Binding
>
> mvn test passed
> gpg, sha256 verified
>
> Environment:
> Apache Maven 3.6.0 (97c98ec64a1fdfee7767ce5ffb20918da4f719f3;
> 2018-10-25T02:41:47+08:00)
> Maven home: /opt/tools/apache-maven-3.6.0
> Java version: 1.8.0_201, vendor: Oracle Corporation, runtime:
> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.201.b09-2.el7_6.x86_64/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "3.10.0-957.10.1.el7.x86_64", arch: "amd64",
> family: "unix"
>
> Thanks,
> Dong Li
>
> -Original Message-
> From: Na Zhai 
> Sent: Tuesday, May 14, 2019 11:16 AM
> To: dev@kylin.apache.org
> Subject: 答复: [VOTE] Release apache-kylin-2.6.2 (RC1)
>
> +1
>
> mvn test passed
>
>
>
> 发送自 Windows 10 版邮件应用
>
>
>
> 
> 发件人: JiaTao Tao 
> 发送时间: Tuesday, May 14, 2019 10:16:16 AM
> 收件人: dev
> 主题: Re: [VOTE] Release apache-kylin-2.6.2 (RC1)
>
> +1
>
> --
>
>
> Regards!
>
> Aron Tao
>
> ShaoFeng Shi  于2019年5月14日周二 上午1:10写道：
>
> > Hi all,
> >
> > I have created a build for Apache Kylin 2.6.2, release candidate 1.
> >
> > Changes highlights:
> > [KYLIN-3892] - Set cubing job priority [KYLIN-3839] - Storage clean up
> > after refreshing or deleting a segment [KYLIN-3873] - Fix
> > inappropriate use of memory in SparkFactDistinct.java [KYLIN-3905] -
> > Enable shrunken dictionary default [KYLIN-3922] - Fail to update
> > coprocessor when run DeployCoprocessorCLI [KYLIN-3936] - MR/Spark task
> > will still run after the job is stopped.
> >
> >
> > Thanks to everyone who has contributed to this release.
> > Here’s release notes:
> >
> > https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316
> > 121=12345051
> >
> > The commit to being voted upon:
> >
> >
> > https://github.com/apache/kylin/commit/c507ae29fa64bc7234efd6a002dcfe9
> > 90969ad35
> >
> > Its hash is c507ae29fa64bc7234efd6a002dcfe990969ad35.
> >
> > The artifacts to be voted on are located here:
> > https://dist.apache.org/repos/dist/dev/kylin/apache-kylin-2.6.2-rc1/
> >
> > The hash of the artifact is as follows:
> > apache-kylin-2.6.2-source-release.zip.sha256
> > db2ab59d3e66d635462e9c9ef49fd7ca29342f07ff4eea0730e52777287e2ebf
> >
> > A staged Maven repository is available for review at:
> > https://repository.apache.org/content/repositories/orgapachekylin-1062
> > /
> >
> > Release artifacts are signed with the following key:
> > https://people.apache.org/keys/committer/shaofengshi.asc
> >
> > Please vote on releasing this package as Apache Kylin 2.6.2.
> >
> > The vote is open for the next 72 hours and passes if a majority of at
> > least three +1 PMC votes are cast.
> >
> > [ ] +1 Release this package as Apache Kylin 2.6.2 [ ]  0 I don't feel
> > strongly about it, but I'm okay with the release [ ] -1 Do not release
> > this package because...
> >
> >
> > Here is my vote:
> >
> > +1 (binding)
> >
> > Best regards,
> >
> > Shaofeng Shi 史少锋
> > Apache Kylin PMC
> > Email: shaofeng...@apache.org
> >
> > Apache Kylin FAQ:
> > https://kylin.apache.org/docs/gettingstarted/faq.html
> > Join Kylin user mail group: user-subscr...@kylin.apache.org Join Kylin
> > dev mail group: dev-subscr...@kylin.apache.org
> >
>

[jira] [Created] (KYLIN-3998) Make "bpus-min-benefit-ratio" configurable in cube planner phase 1

[Chao Long (JIRA)]

Chao Long created KYLIN-3998:


 Summary: Make "bpus-min-benefit-ratio" configurable in cube 
planner phase 1
 Key: KYLIN-3998
 URL: https://issues.apache.org/jira/browse/KYLIN-3998
 Project: Kylin
  Issue Type: Improvement
Reporter: Chao Long
Assignee: Chao Long
 Fix For: Future






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: How to build Kylin for HDP 3.

[Chao Long]

Hi,
Firstly, you need to switch the branch to master-hadoop3.1, which has the
code for hadoop3.
Secondly, rebase the master branch, because this branch is older than
master branch.

run *build/script/package.sh* to build package for HDP 3.0
run *build/script/package.sh -P cdh60* to build package for CDH 6.0

On Mon, May 6, 2019 at 5:54 PM Александр Сидорчук <
alex.sidorchu...@gmail.com> wrote:

> Hello,
>
> I can't find how to build Kylin to this package:
> for Hadoop 3.1 + HBase 2.0 (includes Hortonworks HDP 3.0)
>
> Please, help, there are that i find in documentation is only
> *mvn clean install -DskipTests*
> and
> Build Package for HBase 1.x
>
> cd kylin
> build/script/package.sh
>
> Build Package for CDH 5.7
>
> cd kylin
> build/script/package.sh -P cdh5.7
>
>
> But there is no information about HDP 3.0 build...
>

[jira] [Created] (KYLIN-3987) Give more reducer for UHC column in fact distinct job

[Chao Long (JIRA)]

Chao Long created KYLIN-3987:


 Summary: Give more reducer for UHC column in fact distinct job
 Key: KYLIN-3987
 URL: https://issues.apache.org/jira/browse/KYLIN-3987
 Project: Kylin
  Issue Type: Bug
Reporter: Chao Long
Assignee: Chao Long
 Fix For: v2.6.2






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KYLIN-3957) Query system_cube get exception Cannot cast "java.math.BigDecimal" to "java.lang.Double"

[Chao Long (JIRA)]

Chao Long created KYLIN-3957:


 Summary: Query system_cube get exception Cannot cast 
"java.math.BigDecimal" to "java.lang.Double"
 Key: KYLIN-3957
 URL: https://issues.apache.org/jira/browse/KYLIN-3957
 Project: Kylin
  Issue Type: Bug
  Components: Query Engine
    Reporter: Chao Long
    Assignee: Chao Long






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KYLIN-3950) Cube planner optimize job only use inmem algorithm

[Chao Long (JIRA)]

Chao Long created KYLIN-3950:


 Summary: Cube planner optimize job only use inmem algorithm
 Key: KYLIN-3950
 URL: https://issues.apache.org/jira/browse/KYLIN-3950
 Project: Kylin
  Issue Type: Bug
Reporter: Chao Long
Assignee: Chao Long
 Fix For: v2.6.2






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KYLIN-3946) No cube for AVG measure after include count column

[Chao Long (JIRA)]

Chao Long created KYLIN-3946:


 Summary: No cube for AVG measure after include count column
 Key: KYLIN-3946
 URL: https://issues.apache.org/jira/browse/KYLIN-3946
 Project: Kylin
  Issue Type: Bug
  Components: Query Engine
Reporter: Chao Long
Assignee: Chao Long
 Fix For: v2.6.2






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KYLIN-3945) Debug in local get error "Invalid 'log4jConfigLocation' parameter"

[Chao Long (JIRA)]

Chao Long created KYLIN-3945:


 Summary: Debug in local get error "Invalid 'log4jConfigLocation' 
parameter"
 Key: KYLIN-3945
 URL: https://issues.apache.org/jira/browse/KYLIN-3945
 Project: Kylin
  Issue Type: Bug
Affects Versions: v2.6.2
Reporter: Chao Long
Assignee: Chao Long


严重: Exception sending context initialized event to listener instance of class 
org.apache.kylin.rest.util.Log4jConfigListener
java.lang.IllegalArgumentException: Invalid 'log4jConfigLocation' parameter: 
class path resource [kylin-server-log4j.properties] cannot be resolved to URL 
because it does not exist
 at 
org.springframework.web.util.Log4jWebConfigurer.initLogging(Log4jWebConfigurer.java:158)
 at 
org.springframework.web.util.Log4jConfigListener.contextInitialized(Log4jConfigListener.java:49)
 at 
org.apache.kylin.rest.util.Log4jConfigListener.contextInitialized(Log4jConfigListener.java:36)
 at 
org.apache.catalina.core.StandardContext.listenerStart(StandardContext.java:5157)
 at 
org.apache.catalina.core.StandardContext.startInternal(StandardContext.java:5680)
 at org.apache.catalina.util.LifecycleBase.start(LifecycleBase.java:145)
 at 
org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1707)
 at 
org.apache.catalina.core.ContainerBase$StartChild.call(ContainerBase.java:1697)
 at java.util.concurrent.FutureTask.run$$$capture(FutureTask.java:266)
 at java.util.concurrent.FutureTask.run(FutureTask.java)
 at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KYLIN-3943) system-cube.sh has some problems

[Chao Long (JIRA)]

Chao Long created KYLIN-3943:


 Summary: system-cube.sh has some problems
 Key: KYLIN-3943
 URL: https://issues.apache.org/jira/browse/KYLIN-3943
 Project: Kylin
  Issue Type: Bug
Reporter: Chao Long
Assignee: Chao Long
 Fix For: v2.6.2






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: After starting kylin, Getting error while loading web UI

[Chao Long]

Hi rsanadhya,
   Do you use spark v2.4 as the SPARK_HOME? 


   If so, you can run "$KYLIN_HOME/bin/download-spark.sh" to download spark 
binary package of v2.3.2 and set the SPARK_HOME to "$KYLIN_HOME/spark". 


   There is a related issue, https://issues.apache.org/jira/browse/KYLIN-3872.
--
Best Regards,
Chao Long


-- Original --
From:  "rsanad...@gmail.com";
Date:  Fri, Apr 5, 2019 06:36 PM
To:  "dev";

Subject:  After starting kylin, Getting error while loading web UI



Hi , 
We have below services installed  in cluster--
HDFS 3.1
Hadoop 3.1 
Hbase 2.0.2
Hortonworks HDP 3.1

When we install Apache Kylin 2.6.1 it got installed and Kylin got started
with below error.
We are not able to login through web UI, Does apache Kylin2.6.1 supports
above configuration? 



Error-
2019-04-05 13:20:06,006 INFO  [localhost-startStop-1]
metrics.MetricsManager:136 : Kylin metrics monitor is not enabled
2019-04-05 13:20:06,066 WARN  [localhost-startStop-1]
support.XmlWebApplicationContext:551 : Exception encountered during context
initialization - cancelling refresh attempt:
org.springframework.beans.factory.BeanCreationException: Error creating bean
with name
'org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping':
Invocation of init method failed; nested exception is
java.lang.NoClassDefFoundError:
org/apache/commons/configuration/ConfigurationException
2019-04-05 13:20:06,071 ERROR [localhost-startStop-1]
context.ContextLoader:350 : Context initialization failed
org.springframework.beans.factory.BeanCreationException: Error creating bean
with name
'org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerMapping':
Invocation of init method failed; nested exception is
java.lang.NoClassDefFoundError:
org/apache/commons/configuration/ConfigurationException
   at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1628)
   at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.doCreateBean(AbstractAutowireCapableBeanFactory.java:555)
   at
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.createBean(AbstractAutowireCapableBeanFactory.java:483)
   at
org.springframework.beans.factory.support.AbstractBeanFactory$1.getObject(AbstractBeanFactory.java:306)
   at
org.springframework.beans.factory.support.DefaultSingletonBeanRegistry.getSingleton(DefaultSingletonBeanRegistry.java:230)
   at
org.springframework.beans.factory.support.AbstractBeanFactory.doGetBean(AbstractBeanFactory.java:302)
   at
org.springframework.beans.factory.support.AbstractBeanFactory.getBean(AbstractBeanFactory.java:197)

--
Sent from: http://apache-kylin.74782.x6.nabble.com/

??????Error when query with a filter on timestamp

[Chao Long]

Hi ??
   Can you provide more error stack trace about this query to locate the root 
cause? Because the error message you paste is about a common exception, we 
can't locate the specific exception.


--
Best Regards,
Chao Long


--  --
??: "??";
: 2019??4??3??(??) 6:16
??: "dev";

: Error when query with a filter on timestamp 



Hi??
We are running Kylin 2.6 with CDH and successfully build Cube on it.
The error occured only when timestamp in  range 1553011200-1553011223??and 
other queries are normal.

Here is the log:


Query Id: ed0e4398-2e5c-3b69-a61a-14457eb30286
SQL: SELECT
COUNT( "c"."DAY")  
FROM   "MD_SURVEY"AS  "a"
LEFT JOIN   "MD_DATE_TABLE"AS  "c" ON  (  "c"."STRDATE"   =   
"a"."SURVEYDATE" )
WHERE   "c"."TIMESTAMP" <=  1553011200
GROUP BY  "c"."DAY"
User: ADMIN
Success: false
Duration: 0.202
Project: MD_TEST
Realization Names: [CUBE[name=MD_FIX_all]]
Cuboid Ids: [288230376151711752]
Total scan count: 242
Total scan bytes: 77006
Result row count: 0
Accept Partial: true
Is Partial Result: false
Hit Exception Cache: false
Storage cache used: false
Is Query Push-Down: false
Is Prepare: false
Trace URL: null
Message: For input string: "2666829184000"
while executing SQL: "SELECT COUNT( "c"."DAY")   FROM   "MD_SURVEY"AS  
"a" LEFT JOIN   "MD_DATE_TABLE"AS  "c" ON  (  "c"."STRDATE"   =   
"a"."SURVEYDATE" ) WHERE   "c"."TIMESTAMP" <=  1553011200 GROUP BY  "c"."DAY" 
LIMIT 5"
==[QUERY]===

2019-04-02 16:43:47,597 ERROR [http-bio-7070-exec-22] 
controller.BasicController:63 :
org.apache.kylin.rest.exception.InternalErrorException: For input string: 
"2666829184000"
while executing SQL: "SELECT COUNT( "c"."DAY")   FROM   "MD_SURVEY"AS  
"a" LEFT JOIN   "MD_DATE_TABLE"AS  "c" ON  (  "c"."STRDATE"   =   
"a"."SURVEYDATE" ) WHERE   "c"."TIMESTAMP" <=  1553011200 GROUP BY  "c"."DAY" 
LIMIT 5"
at 
org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:441)
at 
org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:367)
at 
org.apache.kylin.rest.controller.QueryController.query(QueryController.java:87)
at sun.reflect.GeneratedMethodAccessor219.invoke(Unknown Source)
at 
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at 
org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at 
org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
at 
org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
at 
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
at 
org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
at 
org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
at 
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)
at 
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)
at 
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)
at 
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:650)
at 
org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at 
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at 
org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at 
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.j

[jira] [Created] (KYLIN-3933) Now replica set related operation need refresh current front-end page

[Chao Long (JIRA)]

Chao Long created KYLIN-3933:


 Summary: Now replica set related operation need refresh current 
front-end page
 Key: KYLIN-3933
 URL: https://issues.apache.org/jira/browse/KYLIN-3933
 Project: Kylin
  Issue Type: Bug
  Components: Real-time Streaming, Web 
Reporter: Chao Long






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KYLIN-3929) Check satisfaction before execute cubeplanner algorithm

[Chao Long (JIRA)]

Chao Long created KYLIN-3929:


 Summary: Check satisfaction before execute cubeplanner algorithm
 Key: KYLIN-3929
 URL: https://issues.apache.org/jira/browse/KYLIN-3929
 Project: Kylin
  Issue Type: Improvement
Reporter: Chao Long
Assignee: Chao Long
 Fix For: v2.6.2






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KYLIN-3923) UT GeneralColumnDataTest fail

[Chao Long (JIRA)]

Chao Long created KYLIN-3923:


 Summary: UT GeneralColumnDataTest fail
 Key: KYLIN-3923
 URL: https://issues.apache.org/jira/browse/KYLIN-3923
 Project: Kylin
  Issue Type: Bug
  Components: Real-time Streaming
Reporter: Chao Long
Assignee: Chao Long
 Fix For: v3.0.0-alpha






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Debug kylin2.6X with CDH5.15.It didn't build cube.

[Chao Long]

Hi Lio_Messi,
We usually debug Kylin with hdp sandbox, which is also officially 
recommended[http://kylin.apache.org/cn/development/dev_env.html]. Debug with 
cdh is not supported yet, maybe you can make it work.
--
Best Regards,
Chao Long


-- Original --
From:  "Lio_Messi";
Date:  Thu, Mar 28, 2019 08:11 PM
To:  "dev";

Subject:  Debug kylin2.6X with CDH5.15.It didn't build cube.



I want to debug kylin by runing DebugTomcat with CDH5.15.When I built a
cube,the following error appears:

2019-03-28 19:07:25,957 INFO  [BadQueryDetector]
service.BadQueryDetector:147 : Detect bad query.
2019-03-28 19:07:28,935 INFO  [FetcherRunner 1204410668-166]
threadpool.DefaultFetcherRunner:94 : Job Fetcher: 1 should running, 1 actual
running, 0 stopped, 0 ready, 3826 already succeed, 6 error, 4 discarded, 0
others
2019-03-28 19:07:34,140 INFO  [Scheduler 154250424 Job
e3e62e0e-daf8-e0bb-762d-81bcbddc55ae-202] mapred.ClientServiceDelegate:276 :
Application state is completed. FinalApplicationStatus=FAILED. Redirecting
to job history server
2019-03-28 19:07:34,163 ERROR [Scheduler 154250424 Job
e3e62e0e-daf8-e0bb-762d-81bcbddc55ae-202] common.HadoopJobStatusChecker:58 :
error check status
java.io.IOException: Job status not available 
at org.apache.hadoop.mapreduce.Job.updateStatus(Job.java:334)
at org.apache.hadoop.mapreduce.Job.getStatus(Job.java:341)
at
org.apache.kylin.engine.mr.common.HadoopJobStatusChecker.checkStatus(HadoopJobStatusChecker.java:38)
at
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:153)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2019-03-28 19:07:34,166 ERROR [Scheduler 154250424 Job
e3e62e0e-daf8-e0bb-762d-81bcbddc55ae-202] common.MapReduceExecutable:198 :
error execute
MapReduceExecutable{id=e3e62e0e-daf8-e0bb-762d-81bcbddc55ae-01, name=Extract
Fact Table Distinct Columns, state=RUNNING}
java.lang.NullPointerException
at org.apache.hadoop.mapreduce.Job.getTrackingURL(Job.java:380)
at
org.apache.kylin.engine.mr.common.HadoopCmdOutput.getInfo(HadoopCmdOutput.java:66)
at
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:163)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:166)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)
2019-03-28 19:07:34,176 INFO  [Scheduler 154250424 Job
e3e62e0e-daf8-e0bb-762d-81bcbddc55ae-202] execution.ExecutableManager:453 :
job id:e3e62e0e-daf8-e0bb-762d-81bcbddc55ae-01 from RUNNING to ERROR
2019-03-28 19:07:34,178 ERROR [Scheduler 154250424 Job
e3e62e0e-daf8-e0bb-762d-81bcbddc55ae-202] execution.AbstractExecutable:168 :
error running Executable: CubingJob{id=e3e62e0e-daf8-e0bb-762d-81bcbddc55ae,
name=BUILD CUBE - first_cube - 2015010100_2015060100 - GMT+08:00
2019-03-27 19:35:43, state=RUNNING}
2019-03-28 19:07:34,181 DEBUG [pool-6-thread-1] cachesync.Broadcaster:116 :
Servers in the cluster: [localhost:7070]
2019-03-28 19:07:34,181 DEBUG [pool-6-thread-1] cachesync.Broadcaster:126 :
Announcing new broadcast to all: BroadcastEvent{entity=execute_output,
event=update, cacheKey=e3e62e0e-daf8-e0bb-762d-81bcbddc55ae}
2019-03-28 19:07:34,185 DEBUG [http-bio-7070-exec-8]
cachesync.Broadcaster:246 : Broadcasting UPDATE, execute_output,
e3e62e0e-daf8-e0bb-762d-81bcbddc55ae
2019-03-28 19:07:34,185 DEBUG [pool-6-thread-1] cachesync.Broadcaster:116 :
Servers in the cluster: [localhost:7070]
2019-03-28 19:07:34,185 INFO  [Scheduler 154250424 Job
e3e62e0e-daf8-e0bb-762d-81bcbddc55ae-202] execution.ExecutableManager:453 :
job id:e3e62e0e-daf8-e0bb-762d-81bcbddc55ae from RUNNING to ERROR
2019-03-28 19:07:34,185 DEBUG [pool-6-thread-1] cachesync.Broadcaster:126 :
Announcing new broadcast to all: BroadcastEvent{entity=execute_output,
event=update, cacheKey=e3e

??????[DISCUSS] Kylin 3.0 alpha and beta release before GA

[Chao Long]

Good feature, looking forward to it!
--
Best Regards,
Chao Long


--  --
??: "ShaoFeng Shi";
: 2019??3??25??(??) 9:24
??: "Apache Kylin 
PMC";"dev";"user";

: [DISCUSS] Kylin 3.0 alpha and beta release before GA



Hello,

About two months ago, we raised the "[Discuss] Moving toward Apache Kylin
3.0" in the developer group, all agree to use 3.0 as the next major release
version when the Real-Time feature released. Now we're merging the code
from the RT feature branch into the master branch.

Although this feature has been in production in certain early users, it has
not been widely evaluated by the community. I would like to propose
releasing the alpha and beta before the GA release, just like what we did
in Kylin v2.0. This is to give our users enough time to evaluate; On the
other side, it gives the developers the time to hear feedback, to improve
the stability/performance, catch up the documentation and others.

A rough plan is:
- April, 3.0 alpha release
- June, 3.0 beta release
- July to Aug, 3.0 GA release

Before 3.0 GA, the v2.6 branch will roll out bug fix releases at a steady
pace; Usually, 1 version every 1-2 months, depends on the severity of the
reported issues.

We warmly welcome the community users to join the 3.0 alpha and beta.
Please share your comments here. Thank you for the support to Apache Kylin!

Best regards,

Shaofeng Shi ??
Apache Kylin PMC
Email: shaofeng...@apache.org

Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscr...@kylin.apache.org
Join Kylin dev mail group: dev-subscr...@kylin.apache.org

[jira] [Created] (KYLIN-3903) Fix dependence issue when debug in local

[Chao Long (JIRA)]

Chao Long created KYLIN-3903:


 Summary: Fix dependence issue when debug in local
 Key: KYLIN-3903
 URL: https://issues.apache.org/jira/browse/KYLIN-3903
 Project: Kylin
  Issue Type: Sub-task
Reporter: Chao Long
Assignee: Chao Long






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

??????????????cube????????????????dict??

[Chao Long]

Hi,
   You can run command "$KYLIN_HOME/bin/metastore.sh fetch /dict" and check 
weather the dict file is exists.
--
Best Regards,
Chao Long


--  --
??: "chouc_mei";
: 2019??3??18??(??) 6:36
??: "dev";

: cubedict??



??
  cube ?? Build Dimension 
Dictionary
?? 
34segment2-3 
cube??cube ??
??3segment ??


org.apache.kylin.engine.mr.exception.HadoopShellException:
java.lang.RuntimeException: java.io.IOException: Failed to read big resource
/dict/AD_V.IMPRESSION_LOG/SCHEDULEID/cdfa5043-e699-fbd7-e8db-b3646d927a5f.dict
at
org.apache.kylin.dict.DictionaryManager.getDictionaryInfo(DictionaryManager.java:108)
at
org.apache.kylin.dict.DictionaryManager.checkDupByContent(DictionaryManager.java:173)
at
org.apache.kylin.dict.DictionaryManager.trySaveNewDict(DictionaryManager.java:151)
at
org.apache.kylin.dict.DictionaryManager.saveDictionary(DictionaryManager.java:320)
at
org.apache.kylin.cube.CubeManager$DictionaryAssist.saveDictionary(CubeManager.java:1117)
at 
org.apache.kylin.cube.CubeManager.saveDictionary(CubeManager.java:1079)
at
org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:74)
at
org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:55)
at
org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:73)
at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:93)
at
org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165)
at
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:70)
at
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:165)
at
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.io.IOException: Failed to read big resource
/dict/AD_V.IMPRESSION_LOG/SCHEDULEID/cdfa5043-e699-fbd7-e8db-b3646d927a5f.dict
at
org.apache.kylin.common.persistence.PushdownResourceStore.openPushdown(PushdownResourceStore.java:176)
at
org.apache.kylin.storage.hbase.HBaseResourceStore.getInputStream(HBaseResourceStore.java:256)
at
org.apache.kylin.storage.hbase.HBaseResourceStore.rawResource(HBaseResourceStore.java:226)
at
org.apache.kylin.storage.hbase.HBaseResourceStore.getResourceImpl(HBaseResourceStore.java:276)
at
org.apache.kylin.common.persistence.ResourceStore$3.call(ResourceStore.java:291)
at
org.apache.kylin.common.persistence.ResourceStore$3.call(ResourceStore.java:288)
at
org.apache.kylin.common.persistence.ExponentialBackoffRetry.doWithRetry(ExponentialBackoffRetry.java:52)
at
org.apache.kylin.common.persistence.ResourceStore.getResourceWithRetry(ResourceStore.java:288)
at
org.apache.kylin.common.persistence.ResourceStore.getResource(ResourceStore.java:265)
at
org.apache.kylin.common.persistence.ResourceStore.getResource(ResourceStore.java:256)
at 
org.apache.kylin.dict.DictionaryManager.load(DictionaryManager.java:397)
at
org.apache.kylin.dict.DictionaryManager$1.load(DictionaryManager.java:80)
at
org.apache.kylin.dict.DictionaryManager$1.load(DictionaryManager.java:77)
at
com.google.common.cache.LocalCache$LoadingValueReference.loadFuture(LocalCache.java:3599)
at
com.google.common.cache.LocalCache$Segment.loadSync(LocalCache.java:2379)
at
com.google.common.cache.LocalCache$Segment.lockedGetOrLoad(LocalCache.java:2342)
at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2257)
at com.google.common.cache.LocalCache.get(LocalCache.java:4000)
at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:4004)
at
com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4874)
at
org.apache.kylin.dict.DictionaryManager.getDictionaryInfo(DictionaryManager.java:101)
... 17 more
Caused by: java.io.FileNotFoundException:
/kylin/kylin_metadata/resources/dict/AD_V.IMPRESSION_LOG/SCHEDULEID/cdfa5043-e699-fbd7-e8db-b3646d927a5f.dict
 
(FS: DFS[DFSClient[clientName=DFSClient_NONMAPREDUCE_1042273900_27, ugi=root
(auth:SIMPLE)]])
at
org.apache.kylin.common.persistence.PushdownResourceStore.openPushdown(

??????kylin top-n query

[Chao Long]

Hi ??
  There is a blog abort the principle of TopN 
query[http://kylin.apache.org/blog/2016/03/19/approximate-topn-measure/]. I 
hope that would be useful for you.
--
Best Regards,
Chao Long


--  --
??: "??";
: 2019??3??18??(??) 8:06
??: "dev";

: kylin top-n query



someone has  documents for   top-n query in kylin ?

回复： [Discussion] Enable shrunken dictionary by default

[Chao Long]

+1
--
Best Regards,
Chao Long


-- 原始邮件 --
发件人: "Zhong, Yanghong";
发送时间: 2019年3月18日(星期一) 上午10:30
收件人: "dev@kylin.apache.org";
抄送: "Xiaoxiang Yu"; 
主题: Re: [Discussion] Enable shrunken dictionary by default



+1.

Best regards,
Yanghong Zhong

On 2019/3/18, 10:27 AM, "Xiaoxiang Yu"  wrote:

Dear all,
I suggest enable "kylin.dictionary.shrunken-from-global-enabled" by 
default(it is disabled by default), because I found enable it will speed up 
cube build process when cube have count distinct(bitmap) on a large cardinality 
column. This feature is contributed in KYLIN-3491.

When using count distinct(bitmap) measure on a large cardinality 
column(this require global dictionary), build base cuboid step need frequent 
cache swap so it cannot finished within a reasonable period. KYLIN-3491 add a 
new step to build separated dictionary for each InputSplit before 
BuildBaseCuboid step. So mapper of BuildBaseCuboid step only has to fetch a 
smaller dictionary for itself(without unused value), instead of a larger global 
dictionary. It will reduce cache swap and make BuildBaseCuboid step run as 
quick as possible.

In my test env, my hadoop cluster is a CDH cluster with 56 vcore and 110GB 
Memory. I create a model with a fact table (153326740 rows) and three dimension 
tables, there are three count distinct(bitmap) measure which the largest 
cardinality of single column is 55200325. With ShrunkenDict disabled, the 
BuildBaseCuboid cannot completed in 22 hours. Comparatively, with ShrunkenDict 
enabled, build process completed in a reasonable duration(Extra Dictionary cost 
5 minutes, Build Base Cuboid costs 5 minutes).


https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fuser-images.githubusercontent.com%2F14030549%2F54363305-ad25e200-46a5-11e9-8bc7-fe2c385c0278.pngdata=02%7C01%7Cyangzhong%40ebay.com%7C5f549f14059d4731d7a808d6ab4954ef%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C636884728786178583sdata=KuUcbcerY42oG4J11G1jlEcIs4v%2BPPVt40B9G9fqa80%3Dreserved=0

If you want know more, please check 
https://nam01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FKYLIN-3491data=02%7C01%7Cyangzhong%40ebay.com%7C5f549f14059d4731d7a808d6ab4954ef%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C636884728786178583sdata=T1P1rCA1munwUedC0PC4qttqbFqiDkda%2FZ%2BgqgkQn%2BE%3Dreserved=0.
 If you have any suggestion, please let me know.


Best wishes,
Xiaoxiang Yu

[jira] [Created] (KYLIN-3873) Improve SparkFactDistinct performance

[Chao Long (JIRA)]

Chao Long created KYLIN-3873:


 Summary: Improve SparkFactDistinct performance
 Key: KYLIN-3873
 URL: https://issues.apache.org/jira/browse/KYLIN-3873
 Project: Kylin
  Issue Type: Improvement
Reporter: Chao Long
Assignee: Chao Long
 Fix For: v2.6.2






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

??????[VOTE] Release apache-kylin-2.6.1 (RC1)

[Chao Long]

+1
mvn test passed
--
Best Regards,
Chao Long


--  --
??: "ShaoFeng Shi";
: 2019??3??4??(??) 6:34
??: "dev";

: [VOTE] Release apache-kylin-2.6.1 (RC1)



Hi all,

I have created a build for Apache Kylin 2.6.1, release candidate 1.

Changes highlights:
[KYLIN-3494] - Build cube with spark reports ArrayIndexOutOfBoundsException
[KYLIN-3537] - Use Spark to build Cube on Yarn failed at Setp8 on HDP3.
[KYLIN-3815] - Unexpected behavior when joining the streaming table and
hive table
[KYLIN-3828] - ArrayIndexOutOfBoundsException thrown when building a
streaming cube with empty data in its first dimension
[KYLIN-3833] - Potential OOM in Spark Extract Fact Table Distinct Columns
step
[KYLIN-3826] - MergeCuboidJob only uploads necessary segment's dictionary

Thanks to everyone who has contributed to this release.
Here??s the release notes:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316121=12344845

The commit to being voted upon:

https://github.com/apache/kylin/commit/270cfe68ecc94c66141b29e2ccf20b9ec25e23dd

Its hash is 270cfe68ecc94c66141b29e2ccf20b9ec25e23dd.

The artifacts to be voted on are located here:
https://dist.apache.org/repos/dist/dev/kylin/apache-kylin-2.6.1-rc1/

The hash of the artifact is as follows:
apache-kylin-2.6.1-source-release.zip.sha256
961b8c8d0e781fe7936efb7f33cebb9661b4fbf83082669769a41b47cea19001

A staged Maven repository is available for review at:
https://repository.apache.org/content/repositories/orgapachekylin-1060/

Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/shaofengshi.asc

Please vote on releasing this package as Apache Kylin 2.6.1.

The vote is open for the next 72 hours and passes if a majority of
at least three +1 PMC votes are cast.

[ ] +1 Release this package as Apache Kylin 2.6.1
[ ]  0 I don't feel strongly about it, but I'm okay with the release
[ ] -1 Do not release this package because...


Here is my vote:

+1 (binding)

Best regards,

Shaofeng Shi ??
Apache Kylin PMC
Work email: shaofeng@kyligence.io
Kyligence Inc: https://kyligence.io/

Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscr...@kylin.apache.org
Join Kylin dev mail group: dev-subscr...@kylin.apache.org

[jira] [Created] (KYLIN-3840) Auto schedule metadata clean up job

[Chao Long (JIRA)]

Chao Long created KYLIN-3840:


 Summary: Auto schedule metadata clean up job
 Key: KYLIN-3840
 URL: https://issues.apache.org/jira/browse/KYLIN-3840
 Project: Kylin
  Issue Type: Improvement
  Components: Others
Reporter: Chao Long
Assignee: Chao Long






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KYLIN-3839) Strorage clean up after refreshing and deleting segment

[Chao Long (JIRA)]

Chao Long created KYLIN-3839:


 Summary: Strorage clean up after refreshing and deleting segment
 Key: KYLIN-3839
 URL: https://issues.apache.org/jira/browse/KYLIN-3839
 Project: Kylin
  Issue Type: Improvement
  Components: Others
Reporter: Chao Long
Assignee: Chao Long






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Retry mechanism is invalid when build with spark

[Chao Long]

Hi mailpig,
  I check the code and reproduce this problem. I create a jira issue for 
it[https://issues.apache.org/jira/browse/KYLIN-3838]. And If you have already 
fixed it, a pr is welcome.
--
Best Regards,
Chao Long


-- Original --
From:  "mailpig";
Date:  Mon, Feb 18, 2019 05:40 PM
To:  "dev";

Subject:  Retry mechanism is invalid when build with spark



In kylin-2.5.2, retry mechanism is invalid when build with spark.
In SparkExecutable, when spark app runing failed, it while return a result
with -1. Then, the function dowork will return a ExecuteResult without a
Throwable object, it's null. source code is:
@SuppressWarnings("checkstyle:methodlength")
@Override
protected ExecuteResult doWork(ExecutableContext context) throws
ExecuteException {
ExecutableManager mgr = getManager();
Map extra = mgr.getOutput(getId()).getExtra();
if (extra.containsKey(ExecutableConstants.SPARK_JOB_ID)) {
return onResumed(extra.get(ExecutableConstants.SPARK_JOB_ID),
mgr);
} else {
String cubeName =
this.getParam(SparkCubingByLayer.OPTION_CUBE_NAME.getOpt());
CubeInstance cube =
CubeManager.getInstance(context.getConfig()).getCube(cubeName);
final KylinConfig config = cube.getConfig();

setAlgorithmLayer();

if (KylinConfig.getSparkHome() == null) {
throw new NullPointerException();
}
if (config.getKylinJobJarPath() == null) {
throw new NullPointerException();
}
String jars = this.getParam(JARS);

//hadoop conf dir
String hadoopConf = null;
hadoopConf = System.getProperty("kylin.hadoop.conf.dir");

if (StringUtils.isEmpty(hadoopConf)) {
throw new RuntimeException(
"kylin_hadoop_conf_dir is empty, check if there's
error in the output of 'kylin.sh start'");
}

logger.info("Using " + hadoopConf + " as HADOOP_CONF_DIR");

String jobJar = config.getKylinJobJarPath();
if (StringUtils.isEmpty(jars)) {
jars = jobJar;
}

String segmentID =
this.getParam(SparkCubingByLayer.OPTION_SEGMENT_ID.getOpt());
CubeSegment segment = cube.getSegmentById(segmentID);
Segments mergingSeg =
cube.getMergingSegments(segment);
dumpMetadata(segment, mergingSeg);

StringBuilder stringBuilder = new StringBuilder();
if (Shell.osType == Shell.OSType.OS_TYPE_WIN) {
stringBuilder.append(
"set HADOOP_CONF_DIR=%s && %s/bin/spark-submit
--class org.apache.kylin.common.util.SparkEntry ");
} else {
stringBuilder.append(
"export HADOOP_CONF_DIR=%s && %s/bin/spark-submit
--class org.apache.kylin.common.util.SparkEntry ");
}

Map sparkConfs =
config.getSparkConfigOverride();

String sparkConfigName = getSparkConfigName();
if (sparkConfigName != null) {
Map sparkSpecificConfs =
config.getSparkConfigOverrideWithSpecificName(sparkConfigName);
sparkConfs.putAll(sparkSpecificConfs);
}

for (Map.Entry entry : sparkConfs.entrySet()) {
stringBuilder.append(" --conf
").append(entry.getKey()).append("=").append(entry.getValue())
.append(" ");
}

stringBuilder.append("--jars %s %s %s");
final String cmd = String.format(Locale.ROOT,
stringBuilder.toString(), hadoopConf,
KylinConfig.getSparkHome(), jars, jobJar, formatArgs());
logger.info("cmd: " + cmd);
final ExecutorService executorService =
Executors.newSingleThreadExecutor();
final CliCommandExecutor exec = new CliCommandExecutor();
final PatternedLogger patternedLogger = new
PatternedLogger(logger, new PatternedLogger.ILogListener() {
@Override
public void onLogEvent(String infoKey, Map
info) {
// only care three properties here
if (ExecutableConstants.SPARK_JOB_ID.equals(infoKey)
||
ExecutableConstants.YARN_APP_ID.equals(infoKey)
||
ExecutableConstants.YARN_APP_URL.equals(infoKey)) {
getManager().addJobInfo(getId(), info);
}
}
});
Callable callable = new Callable>() {
@Override
public Pair call() throws Exception {
Pair result;
try {
result = exec.execute(cmd,

[jira] [Created] (KYLIN-3838) retry mechanism is invalid when build with spark

[Chao Long (JIRA)]

Chao Long created KYLIN-3838:


 Summary: retry mechanism is invalid when build with spark
 Key: KYLIN-3838
 URL: https://issues.apache.org/jira/browse/KYLIN-3838
 Project: Kylin
  Issue Type: Bug
  Components: Spark Engine
Reporter: Chao Long
Assignee: Chao Long


description：http://mail-archives.apache.org/mod_mbox/kylin-dev/201902.mbox/%3C1550482835832-0.post%40n6.nabble.com%3E



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KYLIN-3836) Kylin StringUtil.join() may cause NPE if iterator is empty

[Chao Long (JIRA)]

Chao Long created KYLIN-3836:


 Summary: Kylin StringUtil.join() may cause NPE if iterator is empty
 Key: KYLIN-3836
 URL: https://issues.apache.org/jira/browse/KYLIN-3836
 Project: Kylin
  Issue Type: Bug
Reporter: Chao Long
Assignee: Chao Long
 Fix For: v2.6.1


org.apache.kylin.job.exception.ExecuteException: 
org.apache.kylin.job.exception.ExecuteException: java.lang.NullPointerException
 at 
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:183)
 at 
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:114)
 at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
 at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
 at java.lang.Thread.run(Thread.java:748)
Caused by: org.apache.kylin.job.exception.ExecuteException: 
java.lang.NullPointerException
 at 
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:183)
 at 
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:71)
 at 
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:168)
 ... 4 more
Caused by: java.lang.NullPointerException
 at com.google.common.base.Preconditions.checkNotNull(Preconditions.java:191)
 at com.google.common.base.Splitter.split(Splitter.java:383)
 at org.apache.kylin.common.util.StringUtil.splitAndTrim(StringUtil.java:181)
 at 
org.apache.kylin.source.hive.GarbageCollectionStep.getIntermediateTables(GarbageCollectionStep.java:95)
 at 
org.apache.kylin.source.hive.GarbageCollectionStep.cleanUpIntermediateFlatTable(GarbageCollectionStep.java:62)
 at 
org.apache.kylin.source.hive.GarbageCollectionStep.doWork(GarbageCollectionStep.java:48)
 at 
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:168)
 ... 6 more



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KYLIN-3820) Add a curator-based scheduler

[Chao Long (JIRA)]

Chao Long created KYLIN-3820:


 Summary: Add a curator-based scheduler
 Key: KYLIN-3820
 URL: https://issues.apache.org/jira/browse/KYLIN-3820
 Project: Kylin
  Issue Type: Improvement
Reporter: Chao Long
Assignee: Chao Long
 Fix For: v2.6.1






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KYLIN-3816) Current CI doesn

[Chao Long (JIRA)]

Chao Long created KYLIN-3816:


 Summary: Current CI doesn
 Key: KYLIN-3816
 URL: https://issues.apache.org/jira/browse/KYLIN-3816
 Project: Kylin
  Issue Type: Improvement
Reporter: Chao Long






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Unexpected behavior when joinning streaming table and hive table

[Chao Long]

Hi lifan,
  I reproduced this problem with sample streaming cube and I created a jira 
issue for it[https://issues.apache.org/jira/browse/KYLIN-3815]. If you have 
already fixed it, a pr is welcome.
--
Best Regards,
Chao Long


-- Original --
From:  "lifan.su";
Date:  Wed, Feb 13, 2019 04:30 PM
To:  "dev";

Subject:  Unexpected behavior when joinning streaming table and hive table



Hello, I am evaluating Kylin and tried to join streaming table and hive
table, but now got unexpected behavior.

All the scripts can be found in
https://gist.github.com/OstCollector/a4ac396e3169aa42a416d96db3021195
(may need to modify some script to match the environments)

Environment: 
Centos 7
Hadoop on CDH-5.8
dedicated Kafka-2.1 (not included in CDH)

How to reproduce this problem:

1. run gen_station.pl to generate dim table data
2. run import-data.sh to build dim table in Hive
3. run factdata.pl and pipe its output into kafka
4. create tables TEST_WEATHER.STATION_INFO(hive)
TEST_WEATHER.WEATHER(streaming) in Kylin
5. create model and cube in Kylin, join WEATHER.SATION_ID = STATION.ID
6. build the cube

Expected behavior:
The cube is built correctly and I can get data when search.

Actual behavior:
On apache-kylin-2.6.0-bin-cdh57: build failed at step #2 (Create
Intermediate Flat Hive Table)
On apache-kylin-2.5.2-bin-cdh57: got empty cube

I also tried with this case without streaming, with the format of timestamp
column changed to "%Y-%m-%d %H:%M:%S", and an additional table to store the
mapping of timestamp and {hour,day,month,year}_start.
In this case, the cube is built as expected. 


In both failed cases, the intermediate fact table on Hive built in step #2
seems to have wrong column order.
e.g. on version 2.5.2-cdh57, the schema and content of temp table are shown
below:

CREATE EXTERNAL TABLE IF NOT EXISTS
kylin_intermediate_weather_f32241e6_53c6_2949_b737_d9a88a4618df_fact
(
DAY_START date
,YEAR_START date
,STATION_ID string
,QUARTER_START date
,MONTH_START date
,TEMPERATURE bigint
,HOUR_START timestamp
)
STORED AS SEQUENCEFILE
LOCATION
'hdfs://hz-dev-hdfs-service/user/admin/kylin-2/kylin_metadata/kylin-5dbe40eb-55ba-2245-c0b5-1e9efcb67937/kylin_intermediate_weather_f32241e6_53c6_2949_b737_d9a88a4618df_fact';
ALTER TABLE
kylin_intermediate_weather_f32241e6_53c6_2949_b737_d9a88a4618df_fact SET
TBLPROPERTIES('auto.purge'='true');

hive> select * from
kylin_intermediate_weather_f32241e6_53c6_2949_b737_d9a88a4618df_fact limit
10;
OK
NULL2010-01-01  2010-01-01  2010-01-01  2010-01-01  NULL   
NULL
NULL2009-01-01  2009-10-01  2009-12-01  2009-12-31  NULL   
NULL
NULL2009-01-01  2009-10-01  2009-12-01  2009-12-31  NULL   
NULL
NULL2009-01-01  2009-10-01  2009-12-01  2009-12-31  NULL   
NULL
NULL2009-01-01  2009-10-01  2009-12-01  2009-12-31  NULL   
NULL
NULL2010-01-01  2010-01-01  2010-01-01  2010-01-01  NULL   
NULL
NULL2010-01-01  2010-01-01  2010-01-01  2010-01-01  NULL   
NULL
NULL2009-01-01  2009-10-01  2009-12-01  2009-12-31  NULL   
NULL
NULL2009-01-01  2009-10-01  2009-12-01  2009-12-31  NULL   
NULL
NULL2010-01-01  2010-01-01  2010-01-01  2010-01-01  NULL   
NULL
Time taken: 0.421 seconds, Fetched: 10 row(s)

While the the content of temp file is:
# hdfs dfs -text
hdfs://hz-dev-hdfs-service/user/admin/kylin-2/kylin_metadata/kylin-5dbe40eb-55ba-2245-c0b5-1e9efcb67937/kylin_intermediate_weather_f32241e6_53c6_2949_b737_d9a88a4618df_fact/part-m-1
| head -n 10
19/02/13 11:44:12 INFO zlib.ZlibFactory: Successfully loaded & initialized
native-zlib library
19/02/13 11:44:12 INFO compress.CodecPool: Got brand-new decompressor
[.deflate]
19/02/13 11:44:12 INFO compress.CodecPool: Got brand-new decompressor
[.deflate]
19/02/13 11:44:12 INFO compress.CodecPool: Got brand-new decompressor
[.deflate]
19/02/13 11:44:12 INFO compress.CodecPool: Got brand-new decompressor
[.deflate]
0030322010-01-012010-01-012010-01-012010-01-012010-01-01
07:00:001706
0075762010-01-012010-01-012010-01-012010-01-012010-01-01
07:00:002605
0113882010-01-012010-01-012010-01-012010-01-012010-01-01
07:00:002963
0214922010-01-012010-01-012010-01-012010-01-012010-01-01
07:00:001769
0303062010-01-012010-01-012010-01-012010-01-012010-01-01 07:00:00432
0377712010-01-012010-01-012010-01-012010-01-012010-01-01 07:00:00808
0443462010-01-012010-01-012010-01-012010-01-012010-01-01
07:00:001400
0500512010-01-012010-01-012010-01-012010-01-012010-01-01 07:00:00342
0537982010-01-012010-01-012010-01-012010-01-012010-01-01
07:00:001587
0597122010-01-012010-01-012010-01-012010-01-012010-01-01
07:00:00-1309
(the '\x01' character is not correctly copied)

So what am I doing wrong?

--
Sent from: http://apache-kylin.74782.x6.nabble.com/

[jira] [Created] (KYLIN-3815) Unexpected behavior when joinning streaming table and hive table

[Chao Long (JIRA)]

Chao Long created KYLIN-3815:


 Summary: Unexpected behavior when joinning streaming table and 
hive table
 Key: KYLIN-3815
 URL: https://issues.apache.org/jira/browse/KYLIN-3815
 Project: Kylin
  Issue Type: Bug
  Components: NRT Streaming
Reporter: Chao Long
Assignee: Chao Long
 Fix For: v2.6.1


http://apache-kylin.74782.x6.nabble.com/Unexpected-behavior-when-joinning-streaming-table-and-hive-table-td12766.html



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

??????????????????????

[Chao Long]

Hi ??
 I can't see the pictures you pasted in the email, can you please resend it?
--
Best Regards,
Chao Long


--  --
??: "??"<18125...@bjtu.edu.cn>;
: 2019??2??18??(??) 11:44
??: "dev";

: 



??  kylinHDP Sandbox 
kylin

1??Apache Kylin??sandox on a 
VM??






2http://192.168.244.128:4200,??root??SSH??root??screen??admin??

3http://192.168.244.128:1080,??admin??Hbase











4http://192.168.244.128:4200,??root??apache-kylin-2.6.0-bin-hadoop3
5??$KYLIN_HOME/bin/kylin.sh start






??sandbox??kylinhadoop??kylin??hive





??

??????????:????kylin??????????????????????

[Chao Long]

Hi Chen,
   Kylin provides a way to backup 
metadata[http://kylin.apache.org/cn/docs/howto/howto_backup_metadata.html].
you can recover data from backup metadata.
   If you want to migrate cube from a Kylin environment to another, you can use 
cube migration tool[http://kylin.apache.org/cn/docs/howto/howto_use_cli.html 
#CubeMigrationCLI.java]. (Note that the different Kylin environments should 
share the same Hadoop cluster, including HDFS, HBase and HIVE)
--
Best Regards,
Chao Long


--  --
??: "chen snowlake";
: 2019??2??14??(??) 10:43
??: "dev@kylin.apache.org";

: :kylin??



Hi Chao Long



Kylin??

   ??



SnowLake



??8??5114??

Email??che...@outlook.com



____
??: Chao Long 
: Wednesday, February 13, 2019 6:57:41 PM
??: dev
: ??[Kylin] 
/kylin/kylin_metadata/kylin-${jobid}/${cubename}/cuboid??

Hi Chen
   sequence??cubesegment 
mergemerge??merge 
hfilemerge
--
Best Regards,
Chao Long


--  --
??: "chen snowlake";
: 2019??2??13??(??) 6:11
??: "dev@kylin.apache.org";

: [Kylin] 
/kylin/kylin_metadata/kylin-${jobid}/${cubename}/cuboid??



Dear All??
??kylin
Hdfs://${HAname}/kylin/kylin_metadata/kylin-${jobid}/${cubename}/cuboid??Kylin??Hbasesegment


Cube build??
   ??
>>  cuboid data
>> convert cuboid data To Hfile  
>> Hdfs://${HAname}/kylin/kylin_metadata/kylin-${jobid}/${cubename}/hfilehfile
>> Hfile load To Hbase hile
??cuboid data  
??build??hfile

SnowLake

??????[Kylin] /kylin/kylin_metadata/kylin-${jobid}/${cubename}/cuboid??????????????

[Chao Long]

Hi Chen
   sequence??cubesegment 
mergemerge??merge 
hfilemerge
--
Best Regards,
Chao Long


--  --
??: "chen snowlake";
: 2019??2??13??(??) 6:11
??: "dev@kylin.apache.org";

: [Kylin] 
/kylin/kylin_metadata/kylin-${jobid}/${cubename}/cuboid??



Dear All??
??kylin
Hdfs://${HAname}/kylin/kylin_metadata/kylin-${jobid}/${cubename}/cuboid??Kylin??Hbasesegment


Cube build??
   ??
>>  cuboid data
>> convert cuboid data To Hfile  
>> Hdfs://${HAname}/kylin/kylin_metadata/kylin-${jobid}/${cubename}/hfilehfile
>> Hfile load To Hbase hile
??cuboid data  
??build??hfile

SnowLake

[jira] [Created] (KYLIN-3800) Count distinct result is incorrect

[Chao Long (JIRA)]

Chao Long created KYLIN-3800:


 Summary: Count distinct result is incorrect
 Key: KYLIN-3800
 URL: https://issues.apache.org/jira/browse/KYLIN-3800
 Project: Kylin
  Issue Type: Bug
  Components: Real-time Streaming
Reporter: Chao Long
Assignee: Chao Long






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

?????? ??????????

[Chao Long]

Hi liwei??
  Kylin v2.3.2 can not run on CDH6.0 directly and Kylin provides the binary 
package for CDH6.0 since v2.5.0. Maybe you can upgrade your Kylin to v2.5.2.
--
Best Regards,
Chao Long


--  --
??: "??";
: 2019??1??31??(??) 1:26
??: 
"dev@kylin.apache.org";"dev@kylin.apache.org";

: Re: ??



   Cdh6.0.1kylin2.3.2??

 Outlook for iOS<https://aka.ms/o0ukef>


??: penphi than 
: ??,  31, 2019 10:32
??: dev@kylin.apache.org
: Re: ??

??hivevarchartrim??


 11,4362,2012-01-01, 
ABIN,175750,0,12,1096,218.14,668,
TEST129
SELECT count(*) from test_kylin_fact
where lstg_format_name = ' ABIN' ??0??

 11,4362,2012-01-01,a 
ABIN,175750,0,12,1096,218.14,668,
TEST129
SELECT count(*) from test_kylin_fact
where lstg_format_name = 'a ABIN' ??1??

trim??
trim

On Thu, Jan 31, 2019 at 9:40 AM ??  wrote:

> hive?? hive??select?? 
> trim
>
>  Outlook for iOS<https://aka.ms/o0ukef>
>
> 
> ??: penphi than 
> : ??,  31, 2019 09:33
> ??: dev@kylin.apache.org
> : Re: ??
>
> ??SQL
> ??SQL
> ??
>
> 
> ??DBA
>
> ??trim??
>
> On Thu, Jan 31, 2019 at 9:14 AM ??  wrote:
>
> >  ??trim trim  
> >
> >  Outlook for iOS<https://aka.ms/o0ukef>
> >
> > 
> > ??: penphi than 
> > : ??,  31, 2019 09:00
> > ??: dev@kylin.apache.org
> > : Re: ??
> >
> > trim(HUOBI_YUNYING_DW_KYLIN_USER_INDEX_DAILY.EXCHANGE_NAME)
> >
> > ??
> >
> >  Outlook for Android<https://aka.ms/ghei36>
> >
> > 
> > From: ?? 
> > Sent: Thursday, January 31, 2019 8:47:02 AM
> > To: dev@kylin.apache.org; dev@kylin.apache.org
> > Subject: Re: ??
> >
> > like?? ??0 
> >
> >  Outlook for iOS<https://aka.ms/o0ukef>
> >
> > 
> > ??: penphi than 
> > : ??,  31, 2019 01:01
> > ??: dev@kylin.apache.org
> > : Re: ??
> >
> > ?? like `%b11%` b11
> >
> > 
> >
> >  Outlook for Android<https://aka.ms/ghei36>
> >
> > 
> > From: Billy Liu 
> > Sent: Wednesday, January 30, 2019 8:56:38 PM
> > To: dev
> > Subject: Re: ??
> >
> > select
> > EXCHANGE_NAME, count(*)
> > from HUOBI_GLOBAL.HUOBI_YUNYING_DW_KYLIN_USER_INDEX_DAILY
> > where HUOBI_YUNYING_DW_KYLIN_USER_INDEX_DAILY.EXCHANGE_NAME = 'b11'
> > group by EXCHANGE_NAME
> >
> > With Warm regards
> >
> > Billy Liu
> >
> > Yifei Wu  ??2019??1??30?? 3:10??
> > >
> > > Hi, liwei,
> > > It sounds like a Kylin Query bug, could you open a jira to describe
> this
> > question and how to reproduce this question in detail, for it be very
> > helpful for
> > > Locating the root cause.
> > >
> > > BR
> > > Yifei Wu
> > >
> > >
> > > ?? 2019/1/30 2:49 :
> > >
> > > select
> > >
> > > count(*),EXCHANGE_NAME
> > >
> > > from HUOBI_GLOBAL.HUOBI_YUNYING_DW_KYLIN_USER_INDEX_DAILY
> > >
> > > group by EXCHANGE_NAME
> > >
> > > order by EXCHANGE_NAME
> > > ?? ??b11 where
> > >
> > >  Outlook for iOS<https://aka.ms/o0ukef>
> > >
> > > 
> > > ??: Li Feng 
> > > : ??,  30, 2019 13:15
> > > ??: dev@kylin.apache.org
> > > : Re: ??
> > >
> 

??????Kylin+Spark = NoClassDefFoundError

[Chao Long]

Hi Kamil,
  Do you use "$KYLIN_HOME/spark" as the SPARK_HOME? If not, you can try it.
--
Best Regards,
Chao Long


--  --
??: "Kamil";
: 2019??1??31??(??) 0:19
??: "dev";

: Kylin+Spark = NoClassDefFoundError



Hello All,

I'm new Kylin user. I successfully managed to get everything work with 
"Sample Cube" (http://kylin.apache.org/docs/tutorial/kylin_sample.html)

Now I wanted to make it work with Spark 
(http://kylin.apache.org/docs/tutorial/cube_spark.html) but I'm 
struggling with one problem:

When I run "build", I got this exception:
kylin | Exception in thread "main" 
java.lang.NoClassDefFoundError: org/slf4j/Logger
kylin | at 
java.lang.Class.getDeclaredMethods0(Native Method)
kylin | at 
java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
kylin | at 
java.lang.Class.privateGetMethodRecursive(Class.java:3048)
kylin | at 
java.lang.Class.getMethod0(Class.java:3018)
kylin | at 
java.lang.Class.getMethod(Class.java:1784)
kylin | at 
sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
kylin | at 
sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
kylin | Caused by: java.lang.ClassNotFoundException: 
org.slf4j.Logger
kylin | at 
java.net.URLClassLoader.findClass(URLClassLoader.java:381)
kylin | at 
java.lang.ClassLoader.loadClass(ClassLoader.java:424)
kylin | at 
sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:335)
kylin | at 
java.lang.ClassLoader.loadClass(ClassLoader.java:357)
kylin | ... 7 more
kylin | The command is:
kylin | export HADOOP_CONF_DIR=/etc/hadoop && 
/opt/spark-2.3.2-bin-without-hadoop/bin/spark-submit --class 
org.apache.kylin.common.util.SparkEnt
ry  --conf spark.executor.instances=40  --conf 
spark.network.timeout=600  --conf spark.yarn.queue=default  --conf 
spark.history.fs.logDirectory=hdfs://namenode:
8020/kylin/spark-history  --conf 
spark.io.compression.codec=org.apache.spark.io.SnappyCompressionCodec 
--conf spark.dynamicAllocation.enabled=true  --conf spar
k.master=yarn  --conf spark.dynamicAllocation.executorIdleTimeout=300  
--conf spark.hadoop.yarn.timeline-service.enabled=false  --conf 
spark.executor.memory=4G
  --conf spark.eventLog.enabled=true  --conf 
spark.eventLog.dir=hdfs://namenode:8020/kylin/spark-history --conf 
spark.dynamicAllocation.minExecutors=1  --conf s
park.executor.cores=1  --conf 
spark.hadoop.mapreduce.output.fileoutputformat.compress=true --conf 
spark.yarn.executor.memoryOverhead=1024  --conf spark.hadoop.
dfs.replication=2  --conf spark.dynamicAllocation.maxExecutors=1000  
--conf 
spark.hadoop.mapreduce.output.fileoutputformat.compress.codec=org.apache.hadoop.io.c
ompress.DefaultCodec  --conf spark.driver.memory=2G  --conf 
spark.submit.deployMode=cluster  --conf 
spark.shuffle.service.enabled=true --jars /opt/apache-kylin-
2.6.0-bin/lib/kylin-job-2.6.0.jar 
/opt/apache-kylin-2.6.0-bin/lib/kylin-job-2.6.0.jar -className 
org.apache.kylin.engine.spark.SparkFactDistinct -counterOutput
hdfs://namenode:8020/kylin/kylin_metadata/kylin-50b3e245-c00f-0136-1ec8-d5c5c472a311/kylin_sales_cube/counter
 
-statisticssamplingpercent 100 -cubename kylin_sal
es_cube -hiveTable 
default.kylin_intermediate_kylin_sales_cube_a2c3dfb4_900c_f8eb_5086_8bbee7e5c60a
 
-output hdfs://namenode:8020/kylin/kylin_metadata/kylin-50b3
e245-c00f-0136-1ec8-d5c5c472a311/kylin_sales_cube/fact_distinct_columns 
-input 
hdfs://namenode:8020/kylin/kylin_metadata/kylin-50b3e245-c00f-0136-1ec8-d5c5c472a
311/kylin_intermediate_kylin_sales_cube_a2c3dfb4_900c_f8eb_5086_8bbee7e5c60a 
-segmentId a2c3dfb4-900c-f8eb-5086-8bbee7e5c60a -metaUrl 
kylin_metadata@hdfs,path=h
dfs://namenode:8020/kylin/kylin_metadata/kylin-50b3e245-c00f-0136-1ec8-d5c5c472a311/kylin_sales_cube/metadata
kylin | at 
org.apache.kylin.common.util.CliCommandExecutor.execute(CliCommandExecutor.java:96)
kylin | at 
org.apache.kylin.engine.spark.SparkExecutable$2.call(SparkExecutable.java:281)
kylin | at 
org.apache.kylin.engine.spark.SparkExecutable$2.call(SparkExecutable.java:276)
kylin | at 
java.util.concurrent.FutureTask.run(FutureTask.java:266)
kylin | at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
kylin | at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
kylin | at java.lang.Thread.run(Thread.java:748)
kylin | 2019-01

??????kylin cube statistics ????

[Chao Long]

Hi ??
  (1) Cube merging will merge the files under ??cube_statistics?? directory of 
the merging segments.
  (2) ??cube_statistics?? save the cube's estimated information(like cuboid 
rowcount, cuboid size), and it is used in many places, like spliting hbase 
region, estimating the partition num in spark cubing. So you'd better not 
delete it. 



--
Best Regards,
Chao Long


--  --
??: ""<1059790...@qq.com>;
: 2019??1??29??(??) 3:19
??: "dev";

: kylin cube statistics 



--  --
??: ""<1059790...@qq.com>;
: 2019??1??28??(??) 11:42
??: "dev";

: kylin cube statistics 



??  cube merge cube_statistics  ??cube_statistics 

[jira] [Created] (KYLIN-3786) Add integration test for real-time streaming

[Chao Long (JIRA)]

Chao Long created KYLIN-3786:


 Summary: Add integration test for real-time streaming
 Key: KYLIN-3786
 URL: https://issues.apache.org/jira/browse/KYLIN-3786
 Project: Kylin
  Issue Type: Improvement
  Components: Real-time Streaming
Reporter: Chao Long
Assignee: Chao Long






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

??????[Discuss] Moving toward Apache Kylin 3.0

[Chao Long]

+1


For parquet storage, I recently completed the development of removing dict from 
the query side, and the query performance has been inproved a lot. But there is 
still a lot of work to do, like removing query single-point issue, upgrading 
parquet to 1.11 for page index and etc. So I think release it in Kylin 3.0 is a 
great choice.
--
Best Regards,
Chao Long


--  --
??: "ShaoFeng Shi";
: 2019??1??23??(??) 3:56
??: "dev";

: [Discuss] Moving toward Apache Kylin 3.0



Hi Kylin developers,

In last week, Kylin released v2.6.0, with the enhanced & distributed query
cache and JDBC data source SDK. After this release, the next batch
candidate features include real-time streaming, parquet storage, and druid
storage. These features were developed in the past 1-2 years by different
Kylin players and were open sourced in the past 6 months. They have already
been staged in separate branches and are under evaluation by the community.
We have received much feedback from the community.

These candidate features are big supplements to as-is Kylin functions; For
example, the real-time streaming feature will bring Kylin from batch &
historical analytics into real-time analytics. The parquet storage will
make the deployment more flexible and more cloud-friendly. Of course,
stabilizing and improving these features need additional time and effort.

So, when we merging and releasing them, we'd better give it a new version
number so that user can clearly know the difference with current 2.x
versions. I discussed this with several developers offline, we think it is
time to move toward Kylin 3.0. So, if one of the above features is merged,
the version will be 3.0. The current 2.6 will be maintained until 3.x is
ready for production use.

Your comments, ideas, and suggestions are welcomed!

Best regards,

Shaofeng Shi ??
Apache Kylin PMC
Work email: shaofeng@kyligence.io
Kyligence Inc: https://kyligence.io/

Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscr...@kylin.apache.org
Join Kylin dev mail group: dev-subscr...@kylin.apache.org

??????[Discuss] Moving toward Apache Kylin 3.0

[Chao Long]

+1


For parquet storage, I recently completed the development of removing dict from 
the query side, and the query performance has been inproved a lot. But there is 
still a lot of work to do, like removing query single-point issue, upgrading 
parquet to 1.11 for page index and etc. So I think release it in Kylin 3.0 is a 
great choice.
--
Best Regards,
Chao Long


--  --
??: "ShaoFeng Shi";
: 2019??1??23??(??) 3:56
??: "dev";

: [Discuss] Moving toward Apache Kylin 3.0



Hi Kylin developers,

In last week, Kylin released v2.6.0, with the enhanced & distributed query
cache and JDBC data source SDK. After this release, the next batch
candidate features include real-time streaming, parquet storage, and druid
storage. These features were developed in the past 1-2 years by different
Kylin players and were open sourced in the past 6 months. They have already
been staged in separate branches and are under evaluation by the community.
We have received much feedback from the community.

These candidate features are big supplements to as-is Kylin functions; For
example, the real-time streaming feature will bring Kylin from batch &
historical analytics into real-time analytics. The parquet storage will
make the deployment more flexible and more cloud-friendly. Of course,
stabilizing and improving these features need additional time and effort.

So, when we merging and releasing them, we'd better give it a new version
number so that user can clearly know the difference with current 2.x
versions. I discussed this with several developers offline, we think it is
time to move toward Kylin 3.0. So, if one of the above features is merged,
the version will be 3.0. The current 2.6 will be maintained until 3.x is
ready for production use.

Your comments, ideas, and suggestions are welcomed!

Best regards,

Shaofeng Shi ??
Apache Kylin PMC
Work email: shaofeng@kyligence.io
Kyligence Inc: https://kyligence.io/

Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscr...@kylin.apache.org
Join Kylin dev mail group: dev-subscr...@kylin.apache.org

????????????cube????????????

[Chao Long]

Hi,
  The cube only has aggregated data, so your queries should include "group by" 
clause.
  You can check the faq: http://kylin.apache.org/docs/gettingstarted/faq.html(# 
Why I got an error when running a ??select * ?? query)


--
Best Regards,
Chao Long


--  --
??: ""<3513797...@qq.com>;
: 2019??1??22??(??) 11:06
??: "dev";

: ??cube



Hi,


??hivehivecube
 
?? select * from ICSTOCKBILL_1W where goodsid = '643'??
ICSTOCKBILL_1W ??
 
hive
cube


??goodsid 


env:
ubuntu 16.04 hadoop 3?? 
apache-kylin-2.5.2-bin-hadoop3  or apache-kylin-2.6.0-bin-hadoop3 ??kylin ??
hive:2.3.4
hbase:1.3.3
zookeeper:3.4.13 



??


Best regards

[jira] [Created] (KYLIN-3777) Remove dict from query

[Chao Long (JIRA)]

Chao Long created KYLIN-3777:


 Summary: Remove dict from query
 Key: KYLIN-3777
 URL: https://issues.apache.org/jira/browse/KYLIN-3777
 Project: Kylin
  Issue Type: Sub-task
Reporter: Chao Long
Assignee: Chao Long






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

??????[VOTE] Release apache-kylin-2.6.0 (RC1)

[Chao Long]

+1
mvn test passed


--
Best Regards,
Chao Long


--  --
??: "Yanghong Zhong";
: 2019??1??9??(??) 10:46
??: "dev";

: [VOTE] Release apache-kylin-2.6.0 (RC1)



Hi all,

I have created a build for Apache Kylin 2.6.0, release candidate 1.

Changes highlights:
[KYLIN-2895] - Refine query cache by changing the query cache expiration
strategy by signature checking and introducing memcached as distributed
cache
[KYLIN-2932] - Simplify the thread model for in-memory cubing
[KYLIN-3021] - Check MapReduce job failed reason and include the
diagnostics into email notification
[KYLIN-3272] - Upgrade Spark dependency to 2.3.2
[KYLIN-3540] - Improve Mandatory Cuboid Recommendation Algorithm
[KYLIN-3552] - Data Source SDK to ingest data from different JDBC sources
[KYLIN-3611] - Upgrade Tomcat to 7.0.91, 8.5.34 or later
[KYLIN-3656] - Improve HLLCounter performance
[KYLIN-3700] - Quote sql identities when creating flat table
[KYLIN-3729] - CLUSTER BY CAST(field AS STRING) will accelerate base cuboid
build with UHC global dict

Thanks to everyone who has contributed to this release.
Here??s release notes:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316121=12344003

The commit to be voted upon:

https://github.com/apache/kylin/commit/8737bc1f555a2789a67462c8f8420b6ab3be97ce

Its hash is 8737bc1f555a2789a67462c8f8420b6ab3be97ce.

The artifacts to be voted on are located here:
https://dist.apache.org/repos/dist/dev/kylin/apache-kylin-2.6.0-rc1/

The hash of the artifact is as follows:
apache-kylin-2.6.0-source-release.zip.sha256

3621750945823ff4f0c4124b6d5b5c7164d9b08686729352ea22b2f486958d2a

A staged Maven repository is available for review at:
https://repository.apache.org/content/repositories/orgapachekylin-1059/

Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/nju_yaho.asc

Please vote on releasing this package as Apache Kylin 2.6.0.

The vote is open for the next 72 hours and passes if a majority of
at least three +1 PMC votes are cast.

[ ] +1 Release this package as Apache Kylin 2.6.0
[ ]  0 I don't feel strongly about it, but I'm okay with the release
[ ] -1 Do not release this package because...


Here is my vote:

+1 (binding)

Best regards,
Yanghong Zhong
eBay Inc.

??????Please share a public maven setting.xml configuration

[Chao Long]

Hi,
  Which jars can not be found? I can build successfully using maven-3.5.4 with 
default settings. I think there may be a network problem.
--
Best Regards,
Chao Long


--  --
??: "_";
: 2018??12??29??(??) 4:44
??: "dev";

: Please share a public maven setting.xml configuration



Hi all:
I want to compile the kylin-on-parquet source code, but I find that some 
jars can not found depend on my maven configuration, so can anybody share an 
effective setting.xml? many thanks!

[jira] [Created] (KYLIN-3734) UT Failed:Invalid path string "/kylin/../examples/test_metadata/job_engine/global_job_engine_lock"

[Chao Long (JIRA)]

Chao Long created KYLIN-3734:


 Summary: UT Failed:Invalid path string 
"/kylin/../examples/test_metadata/job_engine/global_job_engine_lock"
 Key: KYLIN-3734
 URL: https://issues.apache.org/jira/browse/KYLIN-3734
 Project: Kylin
  Issue Type: Bug
  Components: Tools, Build and Test
Reporter: Chao Long
Assignee: Chao Long


After merge pr of issue 
[KYLIN-1819]|https://issues.apache.org/jira/browse/KYLIN-1819], I get a UT 
error about ZookeeperDistributedLock.

Error message:
Caused by: java.lang.IllegalStateException: Error while 
19...@sandbox.hortonworks.com trying to lock 
/kylin/../examples/test_metadata/job_engine/global_job_engine_lock
at 
org.apache.kylin.storage.hbase.util.ZookeeperDistributedLock.lock(ZookeeperDistributedLock.java:151)
at 
org.apache.kylin.storage.hbase.util.ZookeeperDistributedLock.lock(ZookeeperDistributedLock.java:168)
at 
org.apache.kylin.storage.hbase.util.ZookeeperDistributedLock.lockJobEngine(ZookeeperDistributedLock.java:308)
at 
org.apache.kylin.storage.hbase.util.ZookeeperJobLock.lockJobEngine(ZookeeperJobLock.java:81)
at 
org.apache.kylin.job.impl.threadpool.DefaultScheduler.init(DefaultScheduler.java:152)
at 
org.apache.kylin.rest.service.JobService.afterPropertiesSet(JobService.java:126)
at 
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.invokeInitMethods(AbstractAutowireCapableBeanFactory.java:1687)
at 
org.springframework.beans.factory.support.AbstractAutowireCapableBeanFactory.initializeBean(AbstractAutowireCapableBeanFactory.java:1624)
... 70 more
Caused by: java.lang.IllegalArgumentException: Invalid path string 
"/kylin/../examples/test_metadata/job_engine/global_job_engine_lock" caused by 
relative paths not allowed @8
at org.apache.curator.utils.PathUtils.validatePath(PathUtils.java:102)
at org.apache.curator.utils.PathUtils.validatePath(PathUtils.java:37)
at org.apache.curator.utils.ZKPaths.fixForNamespace(ZKPaths.java:105)
at 
org.apache.curator.framework.imps.NamespaceImpl.fixForNamespace(NamespaceImpl.java:104)
at 
org.apache.curator.framework.imps.CuratorFrameworkImpl.fixForNamespace(CuratorFrameworkImpl.java:599)
at 
org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:465)
at 
org.apache.curator.framework.imps.CreateBuilderImpl.forPath(CreateBuilderImpl.java:44)
at 
org.apache.kylin.storage.hbase.util.ZookeeperDistributedLock.lock(ZookeeperDistributedLock.java:147)
... 77 more
It throw this exception before, but because the scheduler is running in a new 
thread, the exception thrown by the thread will be ignored.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Re: Kylin will not delete old hbase table when refresh the segment

[Chao Long]

Hi,
You can use storage cleanup tool.
http://kylin.apache.org/docs/howto/howto_cleanup_storage.html


--
Best Regards,
Chao Long


-- Original --
From:  "mailpig";
Date:  Mon, Dec 17, 2018 05:16 PM
To:  "dev";

Subject:  Kylin will not delete old hbase table when refresh the segment



Hi, my kylin version is 2.1.0. It has run one year. Howerver, there has too
many tables in hbase. 
I found that when I refresh the segment, kylin will not delete the old hbase
table. Is there has a tool to delete the old hbase table?  

--
Sent from: http://apache-kylin.74782.x6.nabble.com/

??????Evaluate Kylin on Parquet

[Chao Long]

In this PoC, we verified Kylin On Parquet is viable, but the query performance 
still have room to improve. We can improve it from the following aspects:


 1, Minimize result set serialization time
 Since Kylin need Object[] data to process, we convert Dataset to RDD, and then 
convert the "Row" type to Object[], so Spark need to serialize Object[] before 
return it to driver. Those time need to be avoided.


 2, Query without dictionary
 In this PoC, for less storage use, we keep dict encode value in Parquet file 
for dict-encode dimensions, so Kylin must load dictionary to convert dict value 
for query. If we keep original value for dict-encode dimension, dictionary is 
unnecessary. And we don't hava to worry about the storage use, because Parquet 
will encode it. We should remove dictionary from query.


 3, Remove query single-point issue
 In this PoC, we use Spark to read and process Cube data, which is distributed, 
but kylin alse need to process result data the Spark returned in single jvm. We 
can try to make it distributed too.


 4, Upgrade Parquet to 1.11 for page index
 In this PoC, Parquet don't have page index, we get a poor filter performance. 
We need to upgrade Parquet to version 1.11 which has page index to improve 
filter performance.



--
Best Regards,
Chao Long


 
--  --
??: "ShaoFeng Shi";
: 2018??12??14??(??) 4:39
??: "dev";"user";

: Evaluate Kylin on Parquet



Hello Kylin users,

The first version of Kylin on Parquet [1] feature has been staged in Kylin
code repository for public review and evaluation. You can check out the
"kylin-on-parquet" branch [2] to read the code, and also can make a binary
build to run an example. When creating a cube, you can select "Parquet" as
the storage in the "Advanced setting" page. Both MapReduce and Spark
engines support this new storage. A tech blog is under drafting for the
design and implementation.

Thanks so much to the engineers' hard work: Chao Long and Yichen Zhou!

This is not the final version; there is room to improve in many aspects,
parquet, spark, and Kylin. It can be used for PoC at this moment. Your
comments are welcomed. Let's improve it together.

[1] https://issues.apache.org/jira/browse/KYLIN-3621
[2] https://github.com/apache/kylin/tree/kylin-on-parquet

Best regards,

Shaofeng Shi ??
Apache Kylin PMC
Work email: shaofeng@kyligence.io
Kyligence Inc: https://kyligence.io/

Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscr...@kylin.apache.org
Join Kylin dev mail group: dev-subscr...@kylin.apache.org

[jira] [Created] (KYLIN-3710) Spark cubing failed with JDBC data source

[Chao Long (JIRA)]

Chao Long created KYLIN-3710:


 Summary: Spark cubing failed with JDBC data source
 Key: KYLIN-3710
 URL: https://issues.apache.org/jira/browse/KYLIN-3710
 Project: Kylin
  Issue Type: Bug
  Components: RDBMS Source
Reporter: Chao Long
Assignee: Chao Long
 Fix For: v2.6.0


Caused by: java.lang.RuntimeException: Cannot adapt to interface 
org.apache.kylin.engine.spark.ISparkInput
 at 
org.apache.kylin.source.jdbc.JdbcSource.adaptToBuildEngine(JdbcSource.java:49)
 at 
org.apache.kylin.source.SourceManager.createEngineAdapter(SourceManager.java:148)
 at 
org.apache.kylin.engine.spark.SparkUtil.getBatchCubingInputSide(SparkUtil.java:57)
 at 
org.apache.kylin.engine.spark.SparkBatchCubingJobBuilder2.(SparkBatchCubingJobBuilder2.java:49)
 at 
org.apache.kylin.engine.spark.SparkBatchCubingEngine2.createBatchCubingJob(SparkBatchCubingEngine2.java:44)
 at 
org.apache.kylin.engine.EngineFactory.createBatchCubingJob(EngineFactory.java:56)
 at 
org.apache.kylin.rest.service.JobService.submitJobInternal(JobService.java:245)
 at org.apache.kylin.rest.service.JobService.submitJob(JobService.java:213)
 at 
org.apache.kylin.rest.controller.CubeController.buildInternal(CubeController.java:398)
 ... 77 more



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

??????Hybrid model????bitmap????

[Chao Long]

From my perspective, if the column of count distinct measure(bitmap) using 
GlobalDictionary, there is no limitation on query must with date dimension. Of 
cause, you can have a try on this scenario to confirm this conclusion.


--
Best Regards,
Chao Long




--  --
??: "chen snowlake";
: 2018??11??30??(??) 12:57
??: "dev@kylin.apache.org";

: Hybrid modelbitmap



Dear all??
Hybrid model??
*??, Hybrid model ?? ??bitmap?? ?? count distinct ?? cube 
??. *
??

1. 
??

2. Bitmap??count distinct

3. count distinc??cube
??


SnowLake


??8??5114??
Email??che...@outlook.com

回复： [DISCUSS] Stop inserting git diffs to JIRA ticket

[Chao Long]

+1


--
Best Regards,
Chao Long


 




-- 原始邮件 --
发件人: "Yichen Zhou";
发送时间: 2018年12月3日(星期一) 上午10:25
收件人: "dev";

主题: Re: [DISCUSS] Stop inserting git diffs to JIRA ticket



+1

Regards,
Yichen

On Mon, Dec 3, 2018 at 10:00 AM Zhong, Yanghong 
wrote:

> +1.
>
> Best regards,
> Yanghong Zhong
>
> On 12/3/18, 9:59 AM, "Billy Liu"  wrote:
>
> +1.
>
> With Warm regards
>
> Billy Liu
>
> ShaoFeng Shi  于2018年12月3日周一 上午9:46写道：
> >
> > Hello Kylin developers,
> >
> > After we enable the git box for Kylin code repository, when there is
> a PR
> > merged, the "ASF Github Bot" will insert the git diff to the
> associated
> > JIRA. We noticed this function will make the JIRA very long when the
> code
> > change is big. Besides, when cherry-picking the change to another
> branch,
> > it will append again. This makes it is too hard for a human to read
> the
> > JIRA, the important message may be overlooked.
> >
> > A typical sample is this:
> >
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissues.apache.org%2Fjira%2Fbrowse%2FKYLIN-3187data=02%7C01%7Cyangzhong%40ebay.com%7C18f5152fce4841b09ebe08d658c2f778%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C636793991734125047sdata=nVN71DWTfDcfnVUABdyIvJWJMqyJkUajlOdgxBTN3VA%3Dreserved=0
> >
> > My proposal is, stopping sync the code change from GitHub to JIRA;
> Only
> > keep necessary notifications like "A PR is created/closed" etc. For
> the
> > code change, people should go to GitHub code history, not JIRA.
> >
> > Please express your ideas; If no objection in the next couple of
> days, we
> > will raise a change request to the infrastructure team.
> >
> > Thanks for your input!
> >
> > Best regards,
> >
> > Shaofeng Shi 史少锋
> > Apache Kylin PMC
> > Work email: shaofeng@kyligence.io
> > Kyligence Inc:
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fkyligence.io%2Fdata=02%7C01%7Cyangzhong%40ebay.com%7C18f5152fce4841b09ebe08d658c2f778%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C636793991734125047sdata=u2IdyQRH82jqdN7m3KPM4hRzkXGDdMxWvilw3u%2FzqPs%3Dreserved=0
> >
> > Apache Kylin FAQ:
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fkylin.apache.org%2Fdocs%2Fgettingstarted%2Ffaq.htmldata=02%7C01%7Cyangzhong%40ebay.com%7C18f5152fce4841b09ebe08d658c2f778%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C636793991734125047sdata=ugb4KeubkMASYJxJx0MfKRqO%2FPlJheClZ7rKUTDAZxw%3Dreserved=0
> > Join Kylin user mail group: user-subscr...@kylin.apache.org
> > Join Kylin dev mail group: dev-subscr...@kylin.apache.org
>
>
>

??????[VOTE] Release apache-kylin-2.5.2 (RC2)

[Chao Long]

+1
mvn test passed


-
Chao Long


 
--  --
??: "ShaoFeng Shi";
: 2018??11??30??(??) 9:57
??: "dev";

: [VOTE] Release apache-kylin-2.5.2 (RC2)



Hi all,

I have created a build for Apache Kylin 2.5.2, release candidate 2.

Changes:
[KYLIN-3187] - JDK APIs using the default locale, time zone or character
set should be avoided
[KYLIN-3636] - Wrong "storage_type" in CubeDesc causing cube building error
[KYLIN-3666] - Mege cube step 2: Update dictionary throws
IllegalStateException
[KYLIN-3672] - Performance is poor when multiple queries occur in a short
period
[KYLIN-3676] - Update to custom calcite and remove the "atopcalcite"
[KYLIN-3678] - CacheStateChecker may remove a cache file that under a
building
[KYLIN-3683] - Package org.apache.commons.lang3 not exists
[KYLIN-3689] - When the startTime is equal to the endTime in build request,
the segment will build all data.
[KYLIN-3693] - TopN, Count distinct incorrect in Spark engine
[KYLIN-3705] - Segment Pruner mis-functions when the source data has
Chinese characters
Thanks to everyone who has contributed to this release.

Here are release notes:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316121=12344466

The commit to being voted upon:

https://github.com/apache/kylin/commit/0e519d859e217fbfadd534313376e532d2c647fa

Its hash is 0e519d859e217fbfadd534313376e532d2c647fa.

The artifacts to be voted on are located here:
https://dist.apache.org/repos/dist/dev/kylin/apache-kylin-2.5.2-rc2/

The hashe of the artifact is as follows:
apache-kylin-2.5.2-source-release.zip.sha256
fca5688cf64442ea595e07c2a4a4b2b549836d268ce8f10f3d559f05c22b61d0

A staged Maven repository is available for review at:
https://repository.apache.org/content/repositories/orgapachekylin-1058/

Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/shaofengshi.asc

Please vote on releasing this package as Apache Kylin 2.5.2.

The vote is open for the next 72 hours and passes if a majority of
at least three +1 PPMC votes are cast.

[ ] +1 Release this package as Apache Kylin 2.5.2
[ ]  0 I don't feel strongly about it, but I'm okay with the release
[ ] -1 Do not release this package because...


Here is my vote:

+1 (binding)

Best regards,

Shaofeng Shi ??
Apache Kylin PMC
Work email: shaofeng@kyligence.io
Kyligence Inc: https://kyligence.io/

Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscr...@kylin.apache.org
Join Kylin dev mail group: dev-subscr...@kylin.apache.org

??????[VOTE] Release apache-kylin-2.5.2 (RC1)

[Chao Long]

+1
mvn test pass


--
-
Chao Long


 




--  --
??: "ShaoFeng Shi";
: 2018??11??27??(??) 6:32
??: "dev";

: [VOTE] Release apache-kylin-2.5.2 (RC1)



Hi all,

I have created a build for Apache Kylin 2.5.2, release candidate 1.

Changes:
[KYLIN-3636] - Wrong "storage_type" in CubeDesc causing cube building error
[KYLIN-3666] - Mege cube step 2: Update dictionary throws
IllegalStateException
[KYLIN-3672] - Performance is poor when multiple queries occur in a short
period
[KYLIN-3676] - Update to custom calcite and remove the "atopcalcite"
[KYLIN-3678] - CacheStateChecker may remove a cache file that under a
building
[KYLIN-3683] - Package org.apache.commons.lang3 not exists
[KYLIN-3689] - When the startTime is equal to the endTime in build request,
the segment will build all data.
[KYLIN-3693] - TopN, Count distinct incorrect in Spark engine

Thanks to everyone who has contributed to this release.

Here are release notes:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316121=12344466

The commit to being voted upon:

https://github.com/apache/kylin/commit/481933a35fffb44f3e7c529ad24754afadae3f47

Its hash is 481933a35fffb44f3e7c529ad24754afadae3f47.

The artifacts to be voted on are located here:
https://dist.apache.org/repos/dist/dev/kylin/apache-kylin-2.5.2-rc1/

The hashe of the artifact is as follows:
apache-kylin-2.5.2-source-release.zip.sha256
7577b3353a1663b51ba3d927e3fe6762a8752825e675e7aba1a28ac861e90007

A staged Maven repository is available for review at:
https://repository.apache.org/content/repositories/orgapachekylin-1057/

Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/shaofengshi.asc

Please vote on releasing this package as Apache Kylin 2.5.2.

The vote is open for the next 72 hours and passes if a majority of
at least three +1 PPMC votes are cast.

[ ] +1 Release this package as Apache Kylin 2.5.2
[ ]  0 I don't feel strongly about it, but I'm okay with the release
[ ] -1 Do not release this package because...


Here is my vote:

+1 (binding)

Best regards,

Shaofeng Shi ??
Apache Kylin PMC
Work email: shaofeng@kyligence.io
Kyligence Inc: https://kyligence.io/

Apache Kylin FAQ: https://kylin.apache.org/docs/gettingstarted/faq.html
Join Kylin user mail group: user-subscr...@kylin.apache.org
Join Kylin dev mail group: dev-subscr...@kylin.apache.org

??????How to increase split number for Fact distinct columns when usingspark engine?

[Chao Long]

Hi ,
  Something goes wrong with the picture you paste. Can you send it again or 
share by a link?




--  --
??: "??chenxi07??-";
: 2018??11??5??(??) 11:55
??: "dev@kylin.apache.org";

: How to increase split number for Fact distinct columns when usingspark 
engine?



  
Hi, ALL:
 
   I??m using spark engine to build cube. 
 
Now I found the bottleneck of build time lies in the #3 Step Name: Extract Fact 
Table Distinct Columns.
 
When I look into the spark application, I found there is only two splits 
regardless of how large the input sequence file is.
 
I wonder how to increase the number of split for this step?
 
I??m new to spark and any help will be great thanks!
 
 
 
P.S. Spark job of #3 Step Name: Extract Fact Table Distinct Columns.
 
 
 

 

 

 

 
 
 
 
 
--
 
Best regards,
 
 
 
Xi Chen

[jira] [Created] (KYLIN-3665) Partition time column may never be added

[Chao Long (JIRA)]

Chao Long created KYLIN-3665:


 Summary: Partition time column may never be added
 Key: KYLIN-3665
 URL: https://issues.apache.org/jira/browse/KYLIN-3665
 Project: Kylin
  Issue Type: Bug
Reporter: Chao Long


The partition time column will never be added to the WHERE clause if the 
partition date column is type of int or bigint and the date format is 
"MMdd", "MMddHH", "MMddHHmm" or "MMddHHmmss".



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

?????? Redistribute intermediate table default not by rand()

[Chao Long]

Hi zhixin,
   As I remember  If you set "shard by" column in cube design page, Kylin will 
use this column as the condition of  "distribute by", rather than the first 
three field of rowkey.




--  --
??: "liuzhixin";
: 2018??11??2??(??) ????3:11
??: "dev";
: "Chao Long"; 
: Re: Redistribute intermediate table default not by rand()



Hi Chao Long??

Thank you for the answer.
#
Step1: Create Intermediate Flat Hive Table
Step2: Redistribute intermediate table
#
Perhaps, Kylin can insert one rand column in the intermediate hive table  for 
the next shard, (as default).
At the same time,  Kylin should support the custom column for shard. (has 
provided)

Best Wishes.

> ?? 2018??11??21:38??Chao Long  ??
> 
> Hi zhixin,
> Data may become not correct if use "distribute by rand()".
> https://issues.apache.org/jira/browse/KYLIN-3388
> 
> 
> 
> 
> --  --
> ??: "liuzhixin";
> : 2018??11??2??(??) 12:53
> ??: "dev";
> : "ShaoFeng Shi"; 
> : Re: Redistribute intermediate table default not by rand()
> 
> 
> 
> Hi kylin team:
> 
> Step: Redistribute intermediate table
> #
> ??DISTRIBUTE BYDISTRIBUTE BY 
> RAND()
> 
> 
> Best Regards??
> 
>> ?? 2018??11??212:03??liuzhixin  ??
>> 
>> Hi kylin team:
>> 
>> Version: Kylin2.5-hadoop3.1 for hdp3.0
>> #
>> Step: Redistribute intermediate table
>> #
>> DISTRIBUTE BY is that:
>> INSERT OVERWRITE TABLE table_intermediate SELECT * FROM table_intermediate 
>> DISTRIBUTE BY Field1, Field2, Field3;
>> #
>> Not DISTRIBUTE BY RAND()
>> #
>> Is this default DISTRIBUTE BY Field1, Field2, Field3? how to DISTRIBUTE BY 
>> RAND()?
>> 
>> Best wishes.

??????[VOTE] Release apache-kylin-2.5.1 (RC1)

[Chao Long]

+1




--  --
??: "ShaoFeng Shi";
: 2018??11??2??(??) 2:09
??: "dev";

: [VOTE] Release apache-kylin-2.5.1 (RC1)



Hi all,

I have created a build for Apache Kylin 2.5.1, release candidate 1.

Changes highlights:

[KYLIN-3531] - Login failed with case-insensitive username
[KYLIN-3604] - Can't build cube with spark in HBase standalone mode
[KYLIN-3613] - Kylin with Standalone HBase Cluster could not find the main
cluster namespace at "Create HTable" step
[KYLIN-3634] - When the filter column has null value may cause incorrect
query result
[KYLIN-3635] - Percentile calculation on Spark engine is wrong
[KYLIN-3644] - NumberFormatExcetion on null values when building cube with
Spark
[KYLIN-3599] - Bulk Add Measures

Thanks to everyone who has contributed to this release.
Here??s release notes:
https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12316121=12344108

The commit to be voted upon:

https://github.com/apache/kylin/commit/24e2452309a450ec4ef62339b003343eabe23016

Its hash is 24e2452309a450ec4ef62339b003343eabe23016.

The artifacts to be voted on are located here:
https://dist.apache.org/repos/dist/dev/kylin/apache-kylin-2.5.1-rc1/

The hashe of the artifact is as follows:
apache-kylin-2.5.1-source-release.zip.sha256
21db5dab4d3900a49237b9083b5d270c8471d1882a5427cddf1cc74873df42f2

A staged Maven repository is available for review at:
https://repository.apache.org/content/repositories/orgapachekylin-1056/

Release artifacts are signed with the following key:
https://people.apache.org/keys/committer/shaofengshi.asc

Please vote on releasing this package as Apache Kylin 2.5.1.

The vote is open for the next 72 hours and passes if a majority of
at least three +1 PPMC votes are cast.

[ ] +1 Release this package as Apache Kylin 2.5.1
[ ]  0 I don't feel strongly about it, but I'm okay with the release
[ ] -1 Do not release this package because...


Here is my vote:

+1 (binding)

-- 
Best regards,

Shaofeng Shi ??

?????? Redistribute intermediate table default not by rand()

[Chao Long]

Hi zhixin,
 Data may become not correct if use "distribute by rand()".
 https://issues.apache.org/jira/browse/KYLIN-3388




--  --
??: "liuzhixin";
: 2018??11??2??(??) 12:53
??: "dev";
: "ShaoFeng Shi"; 
: Re: Redistribute intermediate table default not by rand()



Hi kylin team:

Step: Redistribute intermediate table
#
??DISTRIBUTE BYDISTRIBUTE BY RAND()


Best Regards??

> ?? 2018??11??212:03??liuzhixin  ??
> 
> Hi kylin team:
> 
> Version: Kylin2.5-hadoop3.1 for hdp3.0
> #
> Step: Redistribute intermediate table
> #
> DISTRIBUTE BY is that:
> INSERT OVERWRITE TABLE table_intermediate SELECT * FROM table_intermediate 
> DISTRIBUTE BY Field1, Field2, Field3;
> #
> Not DISTRIBUTE BY RAND()
> #
> Is this default DISTRIBUTE BY Field1, Field2, Field3? how to DISTRIBUTE BY 
> RAND()?
> 
> Best wishes.
>

??????Merge Job in inconsistent state

[Chao Long]

Hi Ketan,
   As this merge job is an automatically triggered job, so it start again when 
you discard it. If you don't want this job to be triggered again, you can 
remove the "Auto Merge" related configuration on the cube design page until the 
problem is resolved or fixed(if it's a bug).


   This is the merging Segment[2018100508_2018101217]. And the 
exception occurred during the merging job, so it's in an incorrect state.
   Segment: 2018100508_2018101217 //**This segment Table was deleted 
from Hbase (somehow, we don??t have the reason)**//
   Start Time: 2018-10-05 08:00:00
   End Time: 2018-10-12 17:00:00
   Source Count: 0
   HBase Table: KYLIN_CFLY2CKMCU
   Region Count: 3
   Size: less than 1 MB


   To identify the root cause, you may provide more log around the error 
message.





--  --
??: "ketan dikshit";
: 2018??10??28??(??) 0:25
??: "dev";

: Merge Job in inconsistent state



Hi Team, 
We are using Kylin 2.3.1, And in the merge Job(which gets automatically 
triggered), we are getting this error in Merge Cuboid Data Step;

java.io.IOException: No input paths specified in job
at 
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.listStatus(FileInputFormat.java:239)
at 
org.apache.hadoop.mapreduce.lib.input.SequenceFileInputFormat.listStatus(SequenceFileInputFormat.java:59)
at 
org.apache.hadoop.mapreduce.lib.input.FileInputFormat.getSplits(FileInputFormat.java:387)
at 
org.apache.kylin.engine.mr.common.AbstractHadoopJob.getTotalMapInputMB(AbstractHadoopJob.java:622)
at 
org.apache.kylin.storage.hbase.steps.HBaseMROutput2Transition$HBaseMergeMROutputFormat.configureJobOutput(HBaseMROutput2Transition.java:166)
at 
org.apache.kylin.engine.mr.steps.MergeCuboidJob.run(MergeCuboidJob.java:82)
at 
org.apache.kylin.engine.mr.common.MapReduceExecutable.doWork(MapReduceExecutable.java:130)
at 
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:162)
at 
org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:67)
at 
org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:162)
at 
org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:300)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
at java.lang.Thread.run(Thread.java:745)

result code:2

As it is a merge job, whenever I discard this job,, it again starts 
automatically.
So we need a way(if there is one), of permanently discarding this job somehow. 

Also in case this can be done by changing the metadata for the cube segment, do 
let me know how it can be done(even if I have to delete/remove the segments for 
these below days)
We are open for all options (except dropping the cube as it has more than 1 
year data, and is crucial for organisation). We are stuck on this for some days 
now, help would really be appreciated.

Merge Thresholds:
8 (Hours)
1 (Days)
7 (Days)
15 (Days)

Merge Job, (start and end):
MERGE CUBE -  - 2018100508_2018101217 

Here are list of segments;
Segment: 2018100508_2018100608
Start Time: 2018-10-05 08:00:00
End Time: 2018-10-06 08:00:00
Source Count: 14899048
HBase Table: KYLIN_R1MUK56K71
Region Count: 1
Size: 860 MB

Segment: 2018100508_2018101217 //**This segment Table was deleted from 
Hbase (somehow, we don??t have the reason)**//
Start Time: 2018-10-05 08:00:00
End Time: 2018-10-12 17:00:00
Source Count: 0
HBase Table: KYLIN_CFLY2CKMCU
Region Count: 3
Size: less than 1 MB

Segment: 2018100608_2018100800
Start Time: 2018-10-06 08:00:00
End Time: 2018-10-08 00:00:00
Source Count: 24455686
HBase Table: KYLIN_0KH6PHTEM2
Region Count: 1
Size: 1.0498 GB

Segment: 2018100800_2018100900
Start Time: 2018-10-08 00:00:00
End Time: 2018-10-09 00:00:00
Source Count: 14882090
HBase Table: KYLIN_V1CC4LDSIR
Region Count: 1
Size: 598 MB

Segment: 2018100900_2018101000
Start Time: 2018-10-09 00:00:00
End Time: 2018-10-10 00:00:00
Source Count: 16245847
HBase Table: KYLIN_4A44K2VJEU
Region Count: 1
Size: 628 MB

Segment: 2018101000_2018101008
Start Time: 2018-10-10 00:00:00
End Time: 2018-10-10 08:00:00
Source Count: 5213022
HBase Table: KYLIN_EHO316VC7M
Region Count: 1
Size: 397 MB

Segment: 2018101008_2018101009
Start Time: 2018-10-10 08:00:00
End Time: 2018-10-10 09:00:00
Source Count: 865722
HBase Table: KYLIN_I9LEJ2JDZ8
Region Count: 1
Size: 181 MB

Segment: 2018101009_2018101010
Start Time: 2018-10-10 09:00:00
End Time: 2018-10-10 10:00:00
Source Count: 859127
HBase Table: KYLIN_9IBX3W4UNL
Region Count: 1
Size: 180 MB

Segment: 2018101010_2018101011
Start Time: 2018-10-10 10:00:00
End Time: 2018-10-10 

??????join column derived vs normal

[Chao Long]

I hope this may help you.
http://kylin.apache.org/docs/howto/howto_optimize_cubes.html





--  --
??: "Sachin Aggarwal";
: 2018??10??23??(??) 7:25
??: "user";"dev";

: join column derived vs normal



Hi Team,

while creating cube what are recommendations across join column between
fact and dimension.
should we mark it as derived or normal while selecting it from dimension
column.
what factors play a role in making this decision.


-- 

Thanks & Regards

Sachin Aggarwal

回复： [DISCUSS] Not sending Github PR notifications to dev@kylin

[Chao Long]

+1




-- 原始邮件 --
发件人: "Xiaoxiang Yu";
发送时间: 2018年10月4日(星期四) 晚上7:56
收件人: "dev@kylin.apache.org";

主题: Re: [DISCUSS] Not sending Github PR notifications to dev@kylin



+1

Xiaoxiang Yu 
xiaoxiang...@kyligence.io
 

On [DATE], "[NAME]" <[ADDRESS]> wrote:

Hello, Kylin dev subscribers,

Recently I received several complaints saying that there are many emails
sent to the "dev@kylin.apache.org" from the github.com pull request since
we enabled the Gitbox service for Kylin.

Today most patches and code reviews are performed on GitHub. Each pull
request action (even add a comment) will emit an email to dev@kylin,
instead of the individual contributor or reviewer; This generates many
spams and causes the emails from people are left in the basket.

Now I plan to change the Gitbox email notifications rule: removing 
dev@kylin,
use author and reviewer instead, as follows:


*For Github issues, please notify iss...@kylin.apache.org
 ;For Github PR, please notify the author,
reviewer and iss...@kylin.apache.org *

The related JIRA to Apache Infra is
https://issues.apache.org/jira/browse/INFRA-17073

Please +1 if you agree with the new rule, or -1 if you want to keep as
today. If no objection, we will move on with the new rule.

-- 
Best regards,

Shaofeng Shi 史少锋

回复： [VOTE] Release apache-kylin-2.5.0 (RC2)

[Chao Long]

+1

mvn test passed




-- 原始邮件 --
发件人: "Lijun Cao"<641507...@qq.com>;
发送时间: 2018年9月14日(星期五) 下午5:56
收件人: "dev";

主题: Re: [VOTE] Release apache-kylin-2.5.0 (RC2)



+1

> 在 2018年9月14日，16:20，Na Zhai  写道：
> 
> +1
> 
> mvn test passed
> 
> 
> 
> 发送自 Windows 10 版邮件应用
> 
> 
> 
> 
> 发件人: 李 栋 
> 发送时间: Friday, September 14, 2018 2:28:05 PM
> 收件人: dev@kylin.apache.org
> 主题: RE: [VOTE] Release apache-kylin-2.5.0 (RC2)
> 
> +1 binding
> 
> mvn test passed
> 
> Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe; 
> 2018-06-17T18:33:14Z)
> Maven home: /opt/apache-maven-3.5.4
> Java version: 1.8.0_161, vendor: Oracle Corporation, runtime: 
> /usr/lib/jvm/java-1.8.0-openjdk-1.8.0.161-3.b14.el6_9.x86_64/jre
> Default locale: en_US, platform encoding: UTF-8
> OS name: "linux", version: "2.6.32-573.18.1.el6.x86_64", arch: "amd64", 
> family: "unix"
> 
> Thanks,
> Dong Li
> 
> 
> -Original Message-
> From: Luke Han 
> Sent: Friday, September 14, 2018 11:53 AM
> To: dev 
> Subject: Re: [VOTE] Release apache-kylin-2.5.0 (RC2)
> 
> +1 binding
> 
> maven test passed
> 
> Best Regards!
> -
> 
> Luke Han
> 
> 
> On Thu, Sep 13, 2018 at 11:22 AM Zhong, Yanghong  wrote:
> 
>> mvn clean test, passed
>> 
>> +1 binding
>> 
>> My environment:
>> Apache Maven 3.5.4 (1edded0938998edf8bf061f1ceb3cfdeccf443fe;
>> 2018-06-18T02:33:14+08:00)
>> Maven home: /Users/yangzhong/Software/apache-maven-3.5.4
>> Java version: 1.8.0_66, vendor: Oracle Corporation, runtime:
>> /Library/Java/JavaVirtualMachines/jdk1.8.0_66.jdk/Contents/Home/jre
>> Default locale: en_US, platform encoding: UTF-8 OS name: "mac os x",
>> version: "10.11.6", arch: "x86_64", family: "mac"
>> 
>> Best regards,
>> Yanghong Zhong
>> 
>> On 9/13/18, 1:53 PM, "ShaoFeng Shi"  wrote:
>> 
>>Hi all,
>> 
>>I have created a build for Apache Kylin 2.5.0, release candidate 2. The
>>difference between RC1 and RC2 is just changing the scala-maven-plugin
>>version from 3.4.2 to 3.4.1.
>> 
>>Changes highlights:
>> 
>>[KYLIN-2565] - Support Hadoop 3.0
>>[KYLIN-3033] - Support HBase 2.0
>>[KYLIN-2998] - Kill spark app when cube job was discarded
>>[KYLIN-3370] - Enhanced segment pruning
>>[KYLIN-3488] - Support MySQL as Kylin metadata storage
>>[KYLIN-3418] - User interface for Hybrid model
>>[KYLIN-3419] - Upgrade to Java 8
>>[KYLIN-3421] - Improve job scheduler fetch performance
>>[KYLIN-3427] - Convert to HFile in Spark
>>[KYLIN-3441] - Merge cube segments in Spark
>>[KYLIN-3442] - Fact distinct columns in Spark
>>[KYLIN-3453] - Improve cube size estimation for TOPN, COUNT DISTINCT
>>[KYLIN-3457] - Distribute by multi columns if not set distribute column
>>[KYLIN-3471] - Merge dictionary and statistics on Yarn
>>[KYLIN-3480] - Change the conformance of calcite from default to
>> lenient
>>[KYLIN-3491] - Improve the cube building process when using global
>>dictionary
>>[KYLIN-3521] - Enable Cube Planner by default
>> 
>>Thanks to everyone who has contributed to this release.
>>Here are release notes:
>> 
>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fissue
>> s.apache.org%2Fjira%2Fsecure%2FReleaseNote.jspa%3FprojectId%3D12316121
>> %26version%3D12343540data=02%7C01%7Cyangzhong%40ebay.com%7Ca764ee
>> fe81584b1bf37108d6193d3a24%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%
>> 7C636724148150769018sdata=jfiVgq93KvjoHS3vPOvQ0zwNKvQ0zra%2BLSA3U
>> XETTFs%3Dreserved=0
>> 
>>The commit to being voted upon:
>> 
>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithu
>> b.com%2Fapache%2Fkylin%2Fcommit%2F158f8768debe99746c66e516e4596707a476
>> d7d6data=02%7C01%7Cyangzhong%40ebay.com%7Ca764eefe81584b1bf37108d
>> 6193d3a24%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C636724148150769
>> 018sdata=KEswnV3B6xlNosSIjKs7%2BVj1pX0DkucFbtapf63PPTQ%3Dres
>> erved=0
>> 
>>Its hash is 158f8768debe99746c66e516e4596707a476d7d6.
>> 
>>The artifacts to be voted on are located here:
>> 
>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fdist.
>> apache.org%2Frepos%2Fdist%2Fdev%2Fkylin%2Fapache-kylin-2.5.0-rc2%2F
>> p;data=02%7C01%7Cyangzhong%40ebay.com%7Ca764eefe81584b1bf37108d6193d3a
>> 24%7C46326bff992841a0baca17c16c94ea99%7C0%7C0%7C636724148150769018
>> ;sdata=MYMl8ARJfGgj9HLijudmnXrBmzrhG9cq3mkB64t9Eeg%3Dreserved=0
>> 
>>The hashes of the artifacts are as follows:
>>apache-kylin-2.5.0-source-release.zip.sha256
>>7035bf753b0e3de86f1cd663ff3b44381be5452a44de6cc1089cabf73da6bee4
>> 
>>A staged Maven repository is available for review at:
>> 
>> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Frepos
>> itory.apache.org%2Fcontent%2Frepositories%2Forgapachekylin-1055%2F
>> ;data=02%7C01%7Cyangzhong%40ebay.com%7Ca764eefe81584b1bf37108d6193d3a2
>> 

[jira] [Created] (KYLIN-3539) Hybrid segment overlap not cover some case

[Chao Long (JIRA)]

Chao Long created KYLIN-3539:


 Summary: Hybrid segment overlap not cover some case
 Key: KYLIN-3539
 URL: https://issues.apache.org/jira/browse/KYLIN-3539
 Project: Kylin
  Issue Type: Improvement
Reporter: Chao Long
Assignee: Chao Long
 Fix For: v2.5.0






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KYLIN-3509) Allocate more memory for "Merge dictionary on yarn" step

[Chao Long (JIRA)]

Chao Long created KYLIN-3509:


 Summary: Allocate more memory for "Merge dictionary on yarn" step
 Key: KYLIN-3509
 URL: https://issues.apache.org/jira/browse/KYLIN-3509
 Project: Kylin
  Issue Type: Improvement
Reporter: Chao Long
Assignee: Chao Long


"Merge dictionary on yarn" step has been finished in KYLIN-3471,  but this step 
will consume a great deal of memory, we should allocate more memory for it.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KYLIN-3471) Merge dictionary and statistics on yarn

[Chao Long (JIRA)]

Chao Long created KYLIN-3471:


 Summary: Merge dictionary and statistics on yarn
 Key: KYLIN-3471
 URL: https://issues.apache.org/jira/browse/KYLIN-3471
 Project: Kylin
  Issue Type: Improvement
Reporter: Chao Long
Assignee: Chao Long






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KYLIN-3457) Distribute by multi column if not set distribute column during the redistribute step

[Chao Long (JIRA)]

Chao Long created KYLIN-3457:


 Summary: Distribute by multi column if not set distribute column 
during the redistribute step
 Key: KYLIN-3457
 URL: https://issues.apache.org/jira/browse/KYLIN-3457
 Project: Kylin
  Issue Type: Improvement
Reporter: Chao Long






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

[jira] [Created] (KYLIN-3453) Improve cube size estimation for TOPN, COUNT DISTINCT

[Chao Long (JIRA)]

Chao Long created KYLIN-3453:


 Summary: Improve cube size estimation for TOPN, COUNT DISTINCT
 Key: KYLIN-3453
 URL: https://issues.apache.org/jira/browse/KYLIN-3453
 Project: Kylin
  Issue Type: Improvement
Reporter: Chao Long
Assignee: Chao Long






--
This message was sent by Atlassian JIRA
(v7.6.3#76005)