[jira] [Commented] (KYLIN-4036) Can't find kylin-2.5.0 mirror in http://mirror.bit.edu.cn/apache/kylin/
[ https://issues.apache.org/jira/browse/KYLIN-4036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16859767#comment-16859767 ] Billy Liu commented on KYLIN-4036: -- The website is clear, if you want to download the earlier versions, please download from https://archive.apache.org/dist/kylin/ > Can't find kylin-2.5.0 mirror in http://mirror.bit.edu.cn/apache/kylin/ > --- > > Key: KYLIN-4036 > URL: https://issues.apache.org/jira/browse/KYLIN-4036 > Project: Kylin > Issue Type: Bug > Components: Documentation >Affects Versions: document >Reporter: 朱天赐 >Priority: Minor > Labels: documentation, newbie > Fix For: document > > Original Estimate: 168h > Remaining Estimate: 168h > > I followed instructions in the official site and found something unexpected > in [http://kylin.apache.org/docs/install/index.html] > When I downloaded kylin-2.5.0 through using wget > [http://mirror.bit.edu.cn/apache/kylin/apache-kylin-2.5.0/apache-kylin-2.5.0-bin-hbase1x.tar.gz], > I found it couldn't be downloaded and given me a 404 Not Found. Then I went > to the website [http://mirror.bit.edu.cn/apache/kylin/] and found the > kylin-2.5.0 been removed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-4036) Can't find kylin-2.5.0 mirror in http://mirror.bit.edu.cn/apache/kylin/
[ https://issues.apache.org/jira/browse/KYLIN-4036?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu resolved KYLIN-4036. -- Resolution: Invalid > Can't find kylin-2.5.0 mirror in http://mirror.bit.edu.cn/apache/kylin/ > --- > > Key: KYLIN-4036 > URL: https://issues.apache.org/jira/browse/KYLIN-4036 > Project: Kylin > Issue Type: Bug > Components: Documentation >Affects Versions: document >Reporter: 朱天赐 >Priority: Minor > Labels: documentation, newbie > Fix For: document > > Original Estimate: 168h > Remaining Estimate: 168h > > I followed instructions in the official site and found something unexpected > in [http://kylin.apache.org/docs/install/index.html] > When I downloaded kylin-2.5.0 through using wget > [http://mirror.bit.edu.cn/apache/kylin/apache-kylin-2.5.0/apache-kylin-2.5.0-bin-hbase1x.tar.gz], > I found it couldn't be downloaded and given me a 404 Not Found. Then I went > to the website [http://mirror.bit.edu.cn/apache/kylin/] and found the > kylin-2.5.0 been removed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-4036) Can't find kylin-2.5.0 mirror in http://mirror.bit.edu.cn/apache/kylin/
[ https://issues.apache.org/jira/browse/KYLIN-4036?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16859642#comment-16859642 ] Billy Liu commented on KYLIN-4036: -- Since the limited storage resource on mirror servers, only the least 3 versions are kept. So you are suggested downloading Apache Kylin 2.6.x > Can't find kylin-2.5.0 mirror in http://mirror.bit.edu.cn/apache/kylin/ > --- > > Key: KYLIN-4036 > URL: https://issues.apache.org/jira/browse/KYLIN-4036 > Project: Kylin > Issue Type: Bug > Components: Documentation >Affects Versions: document >Reporter: 朱天赐 >Priority: Minor > Labels: documentation, newbie > Fix For: document > > Original Estimate: 168h > Remaining Estimate: 168h > > I followed instructions in the official site and found something unexpected > in [http://kylin.apache.org/docs/install/index.html] > When I downloaded kylin-2.5.0 through using wget > [http://mirror.bit.edu.cn/apache/kylin/apache-kylin-2.5.0/apache-kylin-2.5.0-bin-hbase1x.tar.gz], > I found it couldn't be downloaded and given me a 404 Not Found. Then I went > to the website [http://mirror.bit.edu.cn/apache/kylin/] and found the > kylin-2.5.0 been removed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-4018) apache kylin 连接powerbi出错
[ https://issues.apache.org/jira/browse/KYLIN-4018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850702#comment-16850702 ] Billy Liu commented on KYLIN-4018: -- How about this one: https://www.jianshu.com/p/eb59a8f174d1 > apache kylin 连接powerbi出错 > > > Key: KYLIN-4018 > URL: https://issues.apache.org/jira/browse/KYLIN-4018 > Project: Kylin > Issue Type: Wish > Components: Driver - ODBC > Environment: apache kylin，powerbi desktop >Reporter: jamie_lee >Priority: Major > > 我在将apache kylin的数据源连接到powebi > desktop时使用了KylinODBCDriver的驱动，好像只支持导入模式，不支持directquery模式，以至于虽然连接成功了，但是导入的数据好像格式有些问题(有些类型为bigint的字段显示为0)，所以现在apache > kylin和powerbi之间的odbc驱动不支持directquery模式了么？或者是不是我现在用的驱动不对？我应该用哪个驱动呢？求解答，谢谢！ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-4018) apache kylin 连接powerbi出错
[ https://issues.apache.org/jira/browse/KYLIN-4018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu updated KYLIN-4018: - Issue Type: Wish (was: Bug) > apache kylin 连接powerbi出错 > > > Key: KYLIN-4018 > URL: https://issues.apache.org/jira/browse/KYLIN-4018 > Project: Kylin > Issue Type: Wish > Components: Driver - ODBC > Environment: apache kylin，powerbi desktop >Reporter: jamie_lee >Priority: Major > > 我在将apache kylin的数据源连接到powebi > desktop时使用了KylinODBCDriver的驱动，好像只支持导入模式，不支持directquery模式，以至于虽然连接成功了，但是导入的数据好像格式有些问题(有些类型为bigint的字段显示为0)，所以现在apache > kylin和powerbi之间的odbc驱动不支持directquery模式了么？或者是不是我现在用的驱动不对？我应该用哪个驱动呢？求解答，谢谢！ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-4018) apache kylin 连接powerbi出错
[ https://issues.apache.org/jira/browse/KYLIN-4018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16850317#comment-16850317 ] Billy Liu commented on KYLIN-4018: -- As far as I know, Kylin ODBC Driver could not support DirectQuery mode yet. But some 3rd ODBC Driver provider for Kylin announced the support for DirectQuery between PowerBI and Kylin Query Engine. You may search for that. > apache kylin 连接powerbi出错 > > > Key: KYLIN-4018 > URL: https://issues.apache.org/jira/browse/KYLIN-4018 > Project: Kylin > Issue Type: Bug > Components: Driver - ODBC > Environment: apache kylin，powerbi desktop >Reporter: jamie_lee >Priority: Major > > 我在将apache kylin的数据源连接到powebi > desktop时使用了KylinODBCDriver的驱动，好像只支持导入模式，不支持directquery模式，以至于虽然连接成功了，但是导入的数据好像格式有些问题(有些类型为bigint的字段显示为0)，所以现在apache > kylin和powerbi之间的odbc驱动不支持directquery模式了么？或者是不是我现在用的驱动不对？我应该用哪个驱动呢？求解答，谢谢！ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3832) Kylin Pushdown query not support postgresql
[ https://issues.apache.org/jira/browse/KYLIN-3832?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778049#comment-16778049 ] Billy Liu commented on KYLIN-3832: -- Postgresql as pushdown engine is not supported by default. The contribution on postgresql data source adapter is welcomed. To learn more about data source sdk: http://kylin.apache.org/development/datasource_sdk.html > Kylin Pushdown query not support postgresql > --- > > Key: KYLIN-3832 > URL: https://issues.apache.org/jira/browse/KYLIN-3832 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v2.5.2 >Reporter: hailin.huang >Priority: Major > Fix For: Future > > > when I run pushdown to postgresql in my env, I encount the below exception. > it seems that kylin need support more JDBC Driver, > PushDownRunnerJdbcImpl.class should be more general. > 2019-02-26 16:12:53,168 ERROR [Query 207dcf77-7c14-8078-ea8b-79644a0c576d-48] > service.QueryService:989 : pushdown engine failed current query too > java.sql.SQLException: Unrecognized column type: int8 > at > org.apache.kylin.query.adhoc.PushDownRunnerJdbcImpl.toSqlType(PushDownRunnerJdbcImpl.java:260) > at > org.apache.kylin.query.adhoc.PushDownRunnerJdbcImpl.extractColumnMeta(PushDownRunnerJdbcImpl.java:192) > at > org.apache.kylin.query.adhoc.PushDownRunnerJdbcImpl.executeQuery(PushDownRunnerJdbcImpl.java:68) > at > org.apache.kylin.query.util.PushDownUtil.tryPushDownQuery(PushDownUtil.java:122) > at > org.apache.kylin.query.util.PushDownUtil.tryPushDownSelectQuery(PushDownUtil.java:69) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3831) 唯独超过62生成cuboid 错误
[ https://issues.apache.org/jira/browse/KYLIN-3831?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16778043#comment-16778043 ] Billy Liu commented on KYLIN-3831: -- [~photogamrun] The limitation is by design. Kylin use the _long-length_ bits to define the dimensions. > 唯独超过62生成cuboid 错误 > - > > Key: KYLIN-3831 > URL: https://issues.apache.org/jira/browse/KYLIN-3831 > Project: Kylin > Issue Type: Wish > Components: Others >Affects Versions: v2.3.1 >Reporter: zhangwei >Assignee: zhangwei >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3804) Advanced Snapshot Table save not friendly
[ https://issues.apache.org/jira/browse/KYLIN-3804?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16758834#comment-16758834 ] Billy Liu commented on KYLIN-3804: -- +1 > Advanced Snapshot Table save not friendly > - > > Key: KYLIN-3804 > URL: https://issues.apache.org/jira/browse/KYLIN-3804 > Project: Kylin > Issue Type: Improvement > Components: Web >Reporter: Pan, Julian >Assignee: Pan, Julian >Priority: Critical > Attachments: new implementation-edit.png, new > implementation-saved.png, not save.png, save.png > > > The configuration for "Advanced Snapshot Table" in "Advance Setting" may > confuse customer. > User need click "+" to save the snapshot info. > Change the the implementation similar with "Advanced Dictionaries". -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-2924) Utilize error-prone to discover common coding mistakes
[ https://issues.apache.org/jira/browse/KYLIN-2924?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16728249#comment-16728249 ] Billy Liu commented on KYLIN-2924: -- [~yaho] I think [~Shaofengshi] has disabled this feature, to avoid too much log. > Utilize error-prone to discover common coding mistakes > -- > > Key: KYLIN-2924 > URL: https://issues.apache.org/jira/browse/KYLIN-2924 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Billy Liu >Priority: Major > Fix For: v2.6.0 > > > http://errorprone.info/ is a tool which detects common coding mistakes. > We should incorporate into Kylin build. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3672) Performance is poor when multiple queries occur in short period
[ https://issues.apache.org/jira/browse/KYLIN-3672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16679387#comment-16679387 ] Billy Liu commented on KYLIN-3672: -- Impressive. [~zonli] Could you share more info about the root cause? > Performance is poor when multiple queries occur in short period > --- > > Key: KYLIN-3672 > URL: https://issues.apache.org/jira/browse/KYLIN-3672 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v2.5.0 > Environment: CentOS 6.7, HBase 1.2.0+cdh5.14.2+456 >Reporter: Zongwei Li >Assignee: Zongwei Li >Priority: Critical > Labels: patch, performance > Attachments: TrendChartBeforeFix.png > > > Hi, Kylin Team > We found one Kylin performance bug during performance tuning for our BI > report integrate with Kylin. > > +Background+ > Our BI report show customer usage report to enterprise customers, provide 15 > usage charts in report page. > Each chart need send API request to Kylin with different SQLs. So it means > for one user, it will trigger 15 API calls(by JDBC) to Kylin. > For our product scale, we need support at least 20 users to review the report > at same time for each Kylin query node. > So it means each Kylin node should be able to handle 15 * 20 = 300 queries > per second. > > +Performance Report+ > To reduce the network impact. We built up Kylin cluster and testing machine > in the same network with Hadoop system. > We use gatling and Jmeter tools to do several round testing, result as follow. > > |Thread|Handled Queries (in 60 seconds)|Handled Queries (per second)|Mean > Response Time > (ms)| > |1|773|13|77| > |15|3245|54|279| > |25|3844|64|390| > |50|4912|82|612| > |75|5405|90|841| > |100|5436|91|1108| > |150|5434|91|1688| > > And draw the trend chart as follow: > !TrendChartBeforeFix.png! > > +Conclusion+ > From the trend, when the thread count reach 75, the handled queries per > second reaches peak data 90, and cannot improved by increase the thread count. > Each Kylin query engine can handle 90 queries per second, it means only > support 90/15 = 6 users to review report page at same time. > Even we setup 3 query nodes, can extend to 18 users at same time, this > performance capacity cannot meet our business requirement. > > +Analyze+ > From test result, response for one thread is fast, but as the thread > increase, throughput of Kylin not increased as we expected. > We have full code review for Kylin query engine, and use Jstack and JProfile > to do analyze, found the root cause for this performance bottleneck. > This is one regression bug introduced by new feature involved one year before. > With bug fixing, one Kylin node can handle 350+ queries per second. Submit > this bug for contribute patch to Kylin. > > +Kylin Server Info+ > |*Role*|*vCPU*|*Memory(GB)*|*Volume(GB)*| > |Query Engine|16 (2.4G)|128|1024| > > +Kylin Package+ > apache-kylin-2.5.0-bin-cdh57.tar.gz (release package) > > +Query SQL+ > SQL with PreparedStatement cache enabled. (New feature in Kylin 2.5.0. If no > PreparedStatement cache, performance will be more worse). Filter will hit all > 6 segments. > > +Cube Info+ > Segment Number: 6 Total Size: 47 MB > > Segment: 2018010100_2018101100 > Start Time: 2018-01-01 00:00:00 > End Time: 2018-10-11 00:00:00 > Source Count: 351934019 > HBase Table: KYLIN_69Q9A850DZ > Region Count: 1 > Size: 47 MB > > Segment: 2018101100_2018101200 > Start Time: 2018-10-11 00:00:00 > End Time: 2018-10-12 00:00:00 > Source Count: 7085485 > HBase Table: KYLIN_ZCT39S8FUA > Region Count: 1 > Size: less than 1 MB > > > Segment: 2018101200_2018101300 > Start Time: 2018-10-12 00:00:00 > End Time: 2018-10-13 00:00:00 > Source Count: 5534968 > HBase Table: KYLIN_RKRRLA958T > Region Count: 1 > Size: less than 1 MB > > Segment: 2018101300_2018101400 > Start Time: 2018-10-13 00:00:00 > End Time: 2018-10-14 00:00:00 > Source Count: 242856 > HBase Table: KYLIN_Q6DKCONN81 > Region Count: 1 > Size: less than 1 MB > > Segment: 2018101400_2018101500 > Start Time: 2018-10-14 00:00:00 > End Time: 2018-10-15 00:00:00 > Source Count: 236122 > HBase Table: KYLIN_JY4WQD2MJH > Region Count: 1 > Size: less than 1 MB > > Segment: 2018101500_2018101600 > Start Time: 2018-10-15 00:00:00 > End Time: 2018-10-16 00:00:00 > Source Count: 6172353 > HBase Table: KYLIN_E2ELLINV22 > Region Count: 1 > Size: less than 1 MB > > +HBase Region Server+ > Count: 6 > hbase.regionserver.handler.count: 120. > Not blocked in CoProcessor RPC call. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3650) support for hive table partitioned on separate columns year,month and day
[ https://issues.apache.org/jira/browse/KYLIN-3650?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16674041#comment-16674041 ] Billy Liu commented on KYLIN-3650: -- [~dipesh.maurya] the patch is welcomed. Thank you for your contribution. > support for hive table partitioned on separate columns year,month and day > - > > Key: KYLIN-3650 > URL: https://issues.apache.org/jira/browse/KYLIN-3650 > Project: Kylin > Issue Type: Improvement > Components: Metadata >Reporter: dipesh >Priority: Major > > partition support on hive table having different partition columns year , > month and day -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3656) Improve HLLCounter performance
[ https://issues.apache.org/jira/browse/KYLIN-3656?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16674040#comment-16674040 ] Billy Liu commented on KYLIN-3656: -- Hi . [~baibaichen], could you share some performance improve data from your environment? > Improve HLLCounter performance > -- > > Key: KYLIN-3656 > URL: https://issues.apache.org/jira/browse/KYLIN-3656 > Project: Kylin > Issue Type: Improvement >Affects Versions: all >Reporter: Chang chen >Assignee: Chang chen >Priority: Major > Fix For: v2.6.0 > > Attachments: 0001-KYLIN-3656-Improve-HLLCounter-performance.patch > > > The current HLLCounter implementation has some room to improve performance, > as we find in our product environment. > # Create HLLCounter from another HLLCounter, we can copy register instead > of merge > # To compute harmonic mean in the HLLCSnapshot, we could > ## using table to cache all 1/2^r without computing on the fly > ## remove floating addition by using integer addition in the bigger loop > ## remove branch, e.g. needn't checking whether registers[i] is zero or not, > although this is minor improvement. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-2892) Make Kylin compile with Java 9
[
https://issues.apache.org/jira/browse/KYLIN-2892?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16633000#comment-16633000
]
Billy Liu commented on KYLIN-2892:
--
Since Hadoop 2.7.x is not compatible with Java 9+, Kylin has dependency with
Hadoop 2.7.x. So have to postpone the JDK upgrade to 9.
> Make Kylin compile with Java 9
> --
>
> Key: KYLIN-2892
> URL: https://issues.apache.org/jira/browse/KYLIN-2892
> Project: Kylin
> Issue Type: Bug
>Reporter: Ted Yu
>Priority: Major
>
> When I attempted building with Java 9, I got:
> {code}
> [ERROR] Failed to execute goal on project kylin-engine-mr: Could not resolve
> dependencies for project org.apache.kylin:kylin-engine-mr:jar:2.2.0-SNAPSHOT:
> Could not find artifact jdk.tools:jdk.tools:jar:1.7 at specified path
> /jdk-9/../lib/tools.jar -> [Help 1]
> {code}
> The dependency seems to come from kylin-engine-mr module.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3334) Prepare for Java 10
[
https://issues.apache.org/jira/browse/KYLIN-3334?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16632980#comment-16632980
]
Billy Liu commented on KYLIN-3334:
--
The compile fails since JDK 9, the same error message.
> Prepare for Java 10
> ---
>
> Key: KYLIN-3334
> URL: https://issues.apache.org/jira/browse/KYLIN-3334
> Project: Kylin
> Issue Type: Task
>Reporter: Ted Yu
>Priority: Major
>
> When compiling with Java 10 , MapReduce Engine module fails with
> {code}
> [ERROR] Failed to execute goal on project kylin-engine-mr: Could not resolve
> dependencies for project org.apache.kylin:kylin-engine-mr:jar:2.4.0-SNAPSHOT:
> Could not find artifact jdk.tools:jdk.tools:jar:1.7 at specified path
> /a/jdk-10/../lib/tools.jar -> [Help 1]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-2924) Utilize error-prone to discover common coding mistakes
[ https://issues.apache.org/jira/browse/KYLIN-2924?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu updated KYLIN-2924: - Fix Version/s: (was: Future) v2.6.0 > Utilize error-prone to discover common coding mistakes > -- > > Key: KYLIN-2924 > URL: https://issues.apache.org/jira/browse/KYLIN-2924 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Assignee: Billy Liu >Priority: Major > Fix For: v2.6.0 > > > http://errorprone.info/ is a tool which detects common coding mistakes. > We should incorporate into Kylin build. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3602) Enable more checkstyle rules
[ https://issues.apache.org/jira/browse/KYLIN-3602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16631917#comment-16631917 ] Billy Liu commented on KYLIN-3602: -- I think we could enable the Google error-prone also. > Enable more checkstyle rules > > > Key: KYLIN-3602 > URL: https://issues.apache.org/jira/browse/KYLIN-3602 > Project: Kylin > Issue Type: Improvement > Components: Others >Reporter: Yichen Zhou >Priority: Major > Fix For: v2.6.0 > > Attachments: checkstyle-aggregate.html > > > The checkstyle rules of kylin is too weak. We need to reinfore them to > achieve better code quality. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3552) Data Source SDK to ingest data from different JDBC sources
[ https://issues.apache.org/jira/browse/KYLIN-3552?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16610748#comment-16610748 ] Billy Liu commented on KYLIN-3552: -- +1. Besides the SDK and API, could you consider one full-featured data source adapter for the early adoption. > Data Source SDK to ingest data from different JDBC sources > -- > > Key: KYLIN-3552 > URL: https://issues.apache.org/jira/browse/KYLIN-3552 > Project: Kylin > Issue Type: New Feature > Components: Query Engine, RDBMS Source >Reporter: youcheng.zhang >Assignee: youcheng.zhang >Priority: Major > Fix For: v2.6.0 > > > Kylin already supports JDBC source, but it takes much efforts to develop an > implementation to a new source engine, like supporting metadata sync, cube > build and query pushdown. It’s mainly because the SQL dialects and jdbc > implementations between source engines are quite different. > We propose to develop a data source sdk, which provides APIs to help > developers handle these dialect differences and easily implement an adaptor > for one source engine. With this SDK, users can achieve followings from a > JDBC source: > # synchronize metadata and data from jdbc source. > # Build cube from jdbc source. > # query pushdown to jdbc source engine when cube is unmatched. > The SDK will define some interfaces of adaptors. To implement a new source, > developers only need to implement an adaptor for the source. > To keep compatibly, we will keep the existing JDBC source module, and > implement a new JDBC source module as the caller of the SDK. Besides, we will > also provide a pushdown implementation class as the caller of SDK for > pushdown scenarios. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3552) Data Source SDK to ingest data from different JDBC sources
[ https://issues.apache.org/jira/browse/KYLIN-3552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu updated KYLIN-3552: - Summary: Data Source SDK to ingest data from different JDBC sources (was: Provide a sdk to support different data sources) > Data Source SDK to ingest data from different JDBC sources > -- > > Key: KYLIN-3552 > URL: https://issues.apache.org/jira/browse/KYLIN-3552 > Project: Kylin > Issue Type: New Feature > Components: Query Engine, RDBMS Source >Reporter: youcheng.zhang >Assignee: youcheng.zhang >Priority: Minor > Fix For: v2.6.0 > > > Kylin already supports JDBC source, but it takes much efforts to develop an > implementation to a new source engine, like supporting metadata sync, cube > build and query pushdown. It’s mainly because the SQL dialects and jdbc > implementations between source engines are quite different. > We propose to develop a data source sdk, which provides APIs to help > developers handle these dialect differences and easily implement an adaptor > for one source engine. With this SDK, users can achieve followings from a > JDBC source: > # synchronize metadata and data from jdbc source. > # Build cube from jdbc source. > # query pushdown to jdbc source engine when cube is unmatched. > The SDK will define some interfaces of adaptors. To implement a new source, > developers only need to implement an adaptor for the source. > To keep compatibly, we will keep the existing JDBC source module, and > implement a new JDBC source module as the caller of the SDK. Besides, we will > also provide a pushdown implementation class as the caller of SDK for > pushdown scenarios. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3552) Data Source SDK to ingest data from different JDBC sources
[ https://issues.apache.org/jira/browse/KYLIN-3552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu updated KYLIN-3552: - Priority: Major (was: Minor) > Data Source SDK to ingest data from different JDBC sources > -- > > Key: KYLIN-3552 > URL: https://issues.apache.org/jira/browse/KYLIN-3552 > Project: Kylin > Issue Type: New Feature > Components: Query Engine, RDBMS Source >Reporter: youcheng.zhang >Assignee: youcheng.zhang >Priority: Major > Fix For: v2.6.0 > > > Kylin already supports JDBC source, but it takes much efforts to develop an > implementation to a new source engine, like supporting metadata sync, cube > build and query pushdown. It’s mainly because the SQL dialects and jdbc > implementations between source engines are quite different. > We propose to develop a data source sdk, which provides APIs to help > developers handle these dialect differences and easily implement an adaptor > for one source engine. With this SDK, users can achieve followings from a > JDBC source: > # synchronize metadata and data from jdbc source. > # Build cube from jdbc source. > # query pushdown to jdbc source engine when cube is unmatched. > The SDK will define some interfaces of adaptors. To implement a new source, > developers only need to implement an adaptor for the source. > To keep compatibly, we will keep the existing JDBC source module, and > implement a new JDBC source module as the caller of the SDK. Besides, we will > also provide a pushdown implementation class as the caller of SDK for > pushdown scenarios. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3521) Enable Cube Planner by default
[ https://issues.apache.org/jira/browse/KYLIN-3521?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16598383#comment-16598383 ] Billy Liu commented on KYLIN-3521: -- +1 > Enable Cube Planner by default > -- > > Key: KYLIN-3521 > URL: https://issues.apache.org/jira/browse/KYLIN-3521 > Project: Kylin > Issue Type: Improvement >Affects Versions: v2.5.0 >Reporter: Shaofeng SHI >Priority: Minor > > Cube planner can significantly reduce the cuboid number that to build. As it > wasn't enabled by default in 2.3 and 2.4, many users don't know that. > > To let more user to start using it, I suggest to enable it by default. As > Cube planner only works when build the first segment, it only affect the > cuboid scheduler of a new Cube. Old cubes will not be affected. > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3497) Make JDBC Module more testable
[ https://issues.apache.org/jira/browse/KYLIN-3497?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16579134#comment-16579134 ] Billy Liu commented on KYLIN-3497: -- +1. If you need any help, please let the community know. > Make JDBC Module more testable > -- > > Key: KYLIN-3497 > URL: https://issues.apache.org/jira/browse/KYLIN-3497 > Project: Kylin > Issue Type: Improvement > Components: Driver - JDBC >Reporter: Ian Hu >Assignee: Ian Hu >Priority: Minor > > While I am trying my work about KYLIN-3496, I found it is difficult to test. > I would offer a work to make it more testable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3393) Building streaming cube do not need to set “KAFKA_HOME” environment variable
[ https://issues.apache.org/jira/browse/KYLIN-3393?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16498889#comment-16498889 ] Billy Liu commented on KYLIN-3393: -- [~wuyingjun], Kylin does not depend on the Kafka Server, but the Kafka Client. So you do not need to install Kafka into kylin node, but copy the necessary client library. That's the same thing we are doing with Hadoop/Hive/HBase as Shaofeng said. Kylin keeps the dependency libraries out of the package to make it more compatible with the environment. If Kylin ships the fixed-version client libraries in binary, you have to recompile everything if the client library has conflicts with the server side. > Building streaming cube do not need to set “KAFKA_HOME” environment variable > - > > Key: KYLIN-3393 > URL: https://issues.apache.org/jira/browse/KYLIN-3393 > Project: Kylin > Issue Type: Improvement >Reporter: wuyingjun >Assignee: wuyingjun >Priority: Major > Attachments: 1.png, 2.png, KYLIN-3393.patch > > > You need hava KAFKA environment and set “KAFKA_HOME” environment variable > when building streaming cube, or else you can not submit a streaming cube > building job, like pic 1； > After we change the kafka dependency scope mode described in the > KYLIN-3393.patch , we can build streaming cube without a kafka environment in > the kylin service node successfully， like pic 2； -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3395) 无法删除model的问题。
[ https://issues.apache.org/jira/browse/KYLIN-3395?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16498883#comment-16498883 ] Billy Liu commented on KYLIN-3395: -- [~chenfeng_0829] First, please do not ask questions in JIRA. The Kylin user mailer is the right place to ask for help. JIRA is for confirmed bug or feature discussion. Second, if you think this is the repeatable bug, please repeat it against the sample project, and list all necessary steps and each step result. Then others will help. > 无法删除model的问题。 > - > > Key: KYLIN-3395 > URL: https://issues.apache.org/jira/browse/KYLIN-3395 > Project: Kylin > Issue Type: Bug > Components: REST Service >Affects Versions: v2.3.0 >Reporter: chenchen >Priority: Major > > org.apache.kylin.rest.exception.BadRequestException: Model is referenced by > Cube '***analysis_trd_detail_d_kq_1' , could not dropped > at > org.apache.kylin.rest.service.ModelService.dropModel(ModelService.java:154) > at > org.apache.kylin.rest.controller.ModelController.deleteModel(ModelController.java:173) > at sun.reflect.GeneratedMethodAccessor254.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) > at > org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) > at > org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827) > at > org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738) > at > org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85) > at > org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967) > at > org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901) > at > org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970) > at > org.springframework.web.servlet.FrameworkServlet.doDelete(FrameworkServlet.java:894) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:656) > at > org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846) > at javax.servlet.http.HttpServlet.service(HttpServlet.java:731) > at > org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303) > at > org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208) > cube已经被删除了，但是在删除model的时候还是提示关联的cube存在？ > 我的model表是一个视图。不知道是不是这个原因？ -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3351) Cube Planner not working in apche kylin 2.3.0(open Source)
[ https://issues.apache.org/jira/browse/KYLIN-3351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16459344#comment-16459344 ] Billy Liu commented on KYLIN-3351: -- Hello [~praveenece], could you join the Kylin user mailer and send your questions there? By the way, 2.3.0 is not suggested, please upgrade to 2.3.1 > Cube Planner not working in apche kylin 2.3.0(open Source) > -- > > Key: KYLIN-3351 > URL: https://issues.apache.org/jira/browse/KYLIN-3351 > Project: Kylin > Issue Type: Task >Reporter: praveenece >Priority: Major > > Hi Team > i want test Cube planner in apache-kylin(2.3.0),So i created cube with > segment, and i hit query to cube many times more than thousands but the cube > planner there is no change in cuboid level like color changing and not get > COUNT in exactly row count and other items.can u please guide me. > Note: > Could u tell me? > Cube Planner Working only old Cube (like 3 month before) or new Cube also. > Configuration(kylin.Properties) > kylin.cube.cubeplanner.enabled=true > Kylin Version=2.3.0(SINGLE NODE) > MyCube creation =10days before > I referred: http://kylin.apache.org/docs23/howto/howto_use_cube_planner.html > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3277) Kylin should override hiveconf settings when connecting to hive using jdbc
[ https://issues.apache.org/jira/browse/KYLIN-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu resolved KYLIN-3277. -- Resolution: Fixed > Kylin should override hiveconf settings when connecting to hive using jdbc > -- > > Key: KYLIN-3277 > URL: https://issues.apache.org/jira/browse/KYLIN-3277 > Project: Kylin > Issue Type: Bug >Reporter: Chuqian Yu >Assignee: Chuqian Yu >Priority: Major > Labels: patch > Fix For: v2.4.0 > > Attachments: 0001-KYLIN-3277.patch > > > Hi, kylin developers. My cube building procedure failing at Step 2 > "Redistribute Flat Hive Table" because Kylin always trying to submit a mr job > to the default yarn queue. > > I have overrided the mapred.job.queue.name property in both > kylin_hive_conf.xml and kylin.properties but it doesn't work. > > kylin.properties > ``` > kylin.source.hive.beeline-params=-n hive -p hive --hiveconf > mapred.job.queue.name=myQueue -u > "jdbc:hive2://myZk:2181/;serviceDiscoveryMode=zooKeeper;" > ``` > > kylin_hive_conf.xml > ``` > > mapred.job.queue.name > myQueue > > ``` > > After digging into the source code ,I found that kylin try to get the row > count of hive table before redistributing it. But it dose not override the > hive configuration when using jdbc to connect to hive server. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3311) Segments overlap error (refactor write conflict exception)
[ https://issues.apache.org/jira/browse/KYLIN-3311?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu resolved KYLIN-3311. -- Resolution: Fixed Assignee: liyang Fix Version/s: v2.4.0 > Segments overlap error (refactor write conflict exception) > -- > > Key: KYLIN-3311 > URL: https://issues.apache.org/jira/browse/KYLIN-3311 > Project: Kylin > Issue Type: Bug > Components: Metadata >Affects Versions: v2.3.0 >Reporter: xujing >Assignee: liyang >Priority: Major > Labels: build > Fix For: v2.4.0 > > Attachments: Segments_Overlap_ErrorLog.txt > > > when "updateCubeWithRetry" method be called at first time , > line newSegs.validate();was passed . > then > cube = crud.save(cube);seem with error throw exception > write conflict to update cube at try 0 ,will retry... > while retry "updateCubeWithRetry" start > line newSegs.validate();was not passed . > throw exception > Segments overlap: [2018031800_2018031900] and > sales_order_channel[2018031800_201803 > 1900] -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3315) allow each project to set its own source in project level override configuration
[ https://issues.apache.org/jira/browse/KYLIN-3315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu resolved KYLIN-3315. -- Resolution: Fixed Assignee: Dong Li Fix Version/s: v2.4.0 > allow each project to set its own source in project level override > configuration > > > Key: KYLIN-3315 > URL: https://issues.apache.org/jira/browse/KYLIN-3315 > Project: Kylin > Issue Type: Improvement > Components: Metadata >Reporter: Dong Li >Assignee: Dong Li >Priority: Major > Fix For: v2.4.0 > > > Currently, all projects connect to the same source which is set in > kylin.properties with kylin.source.default property. > It's better to allow each project to set its own source in project level > override configuration. > As the result, we can have project A connects to JDBC, and project B connects > to Hive. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3324) NegativeArraySizeException in CreateDictionaryJob$2.getDictionary()
[
https://issues.apache.org/jira/browse/KYLIN-3324?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Billy Liu resolved KYLIN-3324.
--
Resolution: Fixed
Fix Version/s: v2.4.0
> NegativeArraySizeException in CreateDictionaryJob$2.getDictionary()
> ---
>
> Key: KYLIN-3324
> URL: https://issues.apache.org/jira/browse/KYLIN-3324
> Project: Kylin
> Issue Type: Bug
>Reporter: liyang
>Assignee: liyang
>Priority: Major
> Fix For: v2.4.0
>
>
> During cube build, got following exception:
> {noformat}
> 2018-01-31 09:58:14,982 ERROR [Scheduler 311988476 Job
> b931b081-3b44-462b-9bce-e15bf245f1b9-862] common.HadoopShellExecutable :
> error execute
> HadoopShellExecutable{id=b931b081-3b44-462b-9bce-e15bf245f1b9-03, name=Build
> Dimension Dictionary, state=RUNNING}
> java.lang.NegativeArraySizeException
> at
> org.apache.hadoop.io.BytesWritable.setCapacity(BytesWritable.java:144)
> at org.apache.hadoop.io.BytesWritable.setSize(BytesWritable.java:123)
> at org.apache.hadoop.io.BytesWritable.readFields(BytesWritable.java:179)
> at
> org.apache.hadoop.io.SequenceFile$Reader.getCurrentValue(SequenceFile.java:2259)
> at org.apache.hadoop.io.SequenceFile$Reader.next(SequenceFile.java:2387)
> at
> org.apache.kylin.engine.mr.steps.CreateDictionaryJob$2.getDictionary(CreateDictionaryJob.java:87)
> at
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:62)
> at
> org.apache.kylin.cube.cli.DictionaryGeneratorCLI.processSegment(DictionaryGeneratorCLI.java:49)
> at
> org.apache.kylin.engine.mr.steps.CreateDictionaryJob.run(CreateDictionaryJob.java:66)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at
> org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:62)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:64)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:156)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745){noformat}
> Root cause seems to be
> https://stackoverflow.com/questions/24127304/negativearraysizeexception-when-creating-a-sequencefile-with-large-1gb-bytesw
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3300) Upgrade jackson-databind to 2.6.7.1 with security issue fixed
[ https://issues.apache.org/jira/browse/KYLIN-3300?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu updated KYLIN-3300: - Summary: Upgrade jackson-databind to 2.6.7.1 with security issue fixed (was: Upgrade jackson-databind) > Upgrade jackson-databind to 2.6.7.1 with security issue fixed > - > > Key: KYLIN-3300 > URL: https://issues.apache.org/jira/browse/KYLIN-3300 > Project: Kylin > Issue Type: Improvement > Components: Integration >Affects Versions: v2.2.0, v2.3.0 >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Major > Fix For: v2.3.1 > > Attachments: KYLIN-3300.master.001.patch > > > jackson-databind 2.6.3 and 2.6.5 are reported with security issue > (CVE-2017-7525), need ugprade -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3301) Upgrade opensaml to 2.6.6 with security issue fixed
[ https://issues.apache.org/jira/browse/KYLIN-3301?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu updated KYLIN-3301: - Summary: Upgrade opensaml to 2.6.6 with security issue fixed (was: Upgrade opensaml) > Upgrade opensaml to 2.6.6 with security issue fixed > --- > > Key: KYLIN-3301 > URL: https://issues.apache.org/jira/browse/KYLIN-3301 > Project: Kylin > Issue Type: Improvement > Components: Integration >Affects Versions: v2.2.0, v2.3.0 >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Major > Fix For: v2.3.1 > > Attachments: KYLIN-3301.master.002.patch > > > opensaml 2.6.4 is reported with security issue (CVE-2015-1796), need upgrade. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3293) FixedLenHexDimEnc return a wrong code length leads to cut bytes error.
[ https://issues.apache.org/jira/browse/KYLIN-3293?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu resolved KYLIN-3293. -- Resolution: Fixed Fix Version/s: v2.4.0 > FixedLenHexDimEnc return a wrong code length leads to cut bytes error. > -- > > Key: KYLIN-3293 > URL: https://issues.apache.org/jira/browse/KYLIN-3293 > Project: Kylin > Issue Type: Bug >Reporter: jiatao.tao >Assignee: jiatao.tao >Priority: Major > Fix For: v2.4.0 > > > FixedLenHexDimEnc lost its byteLen when deserializing, so in > GTRecord#loadColumnsFromColumnBlocks, the byte after cutting are wrong. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3318) Kylin 2.3 UI top n group by only show dimension columns
[ https://issues.apache.org/jira/browse/KYLIN-3318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16415363#comment-16415363 ] Billy Liu commented on KYLIN-3318: -- [~Shaofengshi] +1, very smart solution. > Kylin 2.3 UI top n group by only show dimension columns > --- > > Key: KYLIN-3318 > URL: https://issues.apache.org/jira/browse/KYLIN-3318 > Project: Kylin > Issue Type: Bug >Reporter: Le Anh Vu >Priority: Major > > In Kylin 2.3.0 Web UI, when I use TopN measure, the group by column drop down > only show me dimension columns. Is it the expected behavior or a bug? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KYLIN-3316) Reported NPE after cube build
[ https://issues.apache.org/jira/browse/KYLIN-3316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu reassigned KYLIN-3316: Assignee: TianZhiwei > Reported NPE after cube build > - > > Key: KYLIN-3316 > URL: https://issues.apache.org/jira/browse/KYLIN-3316 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.3.0 >Reporter: TianZhiwei >Assignee: TianZhiwei >Priority: Major > Labels: build > Fix For: v2.4.0 > > Attachments: 0001-KYLIN-3316-modify-CubingJob.updateMetrics.patch > > > Does not affect the completion of the build task and any build task can be > reproduced -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KYLIN-3277) Kylin should override hiveconf settings when connecting to hive using jdbc
[ https://issues.apache.org/jira/browse/KYLIN-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu reassigned KYLIN-3277: Assignee: Chuqian Yu > Kylin should override hiveconf settings when connecting to hive using jdbc > -- > > Key: KYLIN-3277 > URL: https://issues.apache.org/jira/browse/KYLIN-3277 > Project: Kylin > Issue Type: Bug >Reporter: Chuqian Yu >Assignee: Chuqian Yu >Priority: Major > Labels: patch > Fix For: v2.4.0 > > Attachments: 0001-KYLIN-3277.patch > > > Hi, kylin developers. My cube building procedure failing at Step 2 > "Redistribute Flat Hive Table" because Kylin always trying to submit a mr job > to the default yarn queue. > > I have overrided the mapred.job.queue.name property in both > kylin_hive_conf.xml and kylin.properties but it doesn't work. > > kylin.properties > ``` > kylin.source.hive.beeline-params=-n hive -p hive --hiveconf > mapred.job.queue.name=myQueue -u > "jdbc:hive2://myZk:2181/;serviceDiscoveryMode=zooKeeper;" > ``` > > kylin_hive_conf.xml > ``` > > mapred.job.queue.name > myQueue > > ``` > > After digging into the source code ,I found that kylin try to get the row > count of hive table before redistributing it. But it dose not override the > hive configuration when using jdbc to connect to hive server. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3277) Kylin should override hiveconf settings when connecting to hive using jdbc
[ https://issues.apache.org/jira/browse/KYLIN-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu updated KYLIN-3277: - Fix Version/s: v2.4.0 > Kylin should override hiveconf settings when connecting to hive using jdbc > -- > > Key: KYLIN-3277 > URL: https://issues.apache.org/jira/browse/KYLIN-3277 > Project: Kylin > Issue Type: Bug >Reporter: Chuqian Yu >Assignee: Chuqian Yu >Priority: Major > Labels: patch > Fix For: v2.4.0 > > Attachments: 0001-KYLIN-3277.patch > > > Hi, kylin developers. My cube building procedure failing at Step 2 > "Redistribute Flat Hive Table" because Kylin always trying to submit a mr job > to the default yarn queue. > > I have overrided the mapred.job.queue.name property in both > kylin_hive_conf.xml and kylin.properties but it doesn't work. > > kylin.properties > ``` > kylin.source.hive.beeline-params=-n hive -p hive --hiveconf > mapred.job.queue.name=myQueue -u > "jdbc:hive2://myZk:2181/;serviceDiscoveryMode=zooKeeper;" > ``` > > kylin_hive_conf.xml > ``` > > mapred.job.queue.name > myQueue > > ``` > > After digging into the source code ,I found that kylin try to get the row > count of hive table before redistributing it. But it dose not override the > hive configuration when using jdbc to connect to hive server. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3276) Fix the query cache bug with dynamic parameter
[ https://issues.apache.org/jira/browse/KYLIN-3276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu updated KYLIN-3276: - Summary: Fix the query cache bug with dynamic parameter (was: query with dynamic parameter get the same query result) > Fix the query cache bug with dynamic parameter > -- > > Key: KYLIN-3276 > URL: https://issues.apache.org/jira/browse/KYLIN-3276 > Project: Kylin > Issue Type: Bug > Components: Driver - JDBC, Query Engine >Affects Versions: v2.2.0 > Environment: Kylin 2.2.0, JDK 1.8 >Reporter: ZHAOXIXIN >Assignee: nichunen >Priority: Major > Fix For: v2.3.1 > > > sql:"select distinct uid from table where daytime=? limit 70" > ps.setString(1,"2018-01-03") > store the result to ArrayList, and print the size of ArrayList. > when I set the daytime to different dates, the same result I got. > I changed the sql, just change limit value for example > sql:"select distinct uid from table where daytime=? limit 60" > then I got the right value the same like query from GUI:count (distinct uid) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3270) Fix the blocking issue in Cube optimizing job
[ https://issues.apache.org/jira/browse/KYLIN-3270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu updated KYLIN-3270: - Summary: Fix the blocking issue in Cube optimizing job (was: Add integration test for Optimize Job) > Fix the blocking issue in Cube optimizing job > - > > Key: KYLIN-3270 > URL: https://issues.apache.org/jira/browse/KYLIN-3270 > Project: Kylin > Issue Type: Sub-task > Components: Tools, Build and Test >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Critical > Fix For: v2.3.1 > > Attachments: APACHE-KYLIN-3270-fix-for-2.3.1.patch, > APACHE-KYLIN-3270-master.patch, APACHE-KYLIN-3270.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3277) Kylin should override hiveconf settings when connecting to hive using jdbc
[ https://issues.apache.org/jira/browse/KYLIN-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu updated KYLIN-3277: - Fix Version/s: (was: v2.3.1) > Kylin should override hiveconf settings when connecting to hive using jdbc > -- > > Key: KYLIN-3277 > URL: https://issues.apache.org/jira/browse/KYLIN-3277 > Project: Kylin > Issue Type: Bug >Reporter: Chuqian Yu >Priority: Major > Labels: patch > Attachments: 0001-KYLIN-3277.patch > > > Hi, kylin developers. My cube building procedure failing at Step 2 > "Redistribute Flat Hive Table" because Kylin always trying to submit a mr job > to the default yarn queue. > > I have overrided the mapred.job.queue.name property in both > kylin_hive_conf.xml and kylin.properties but it doesn't work. > > kylin.properties > ``` > kylin.source.hive.beeline-params=-n hive -p hive --hiveconf > mapred.job.queue.name=myQueue -u > "jdbc:hive2://myZk:2181/;serviceDiscoveryMode=zooKeeper;" > ``` > > kylin_hive_conf.xml > ``` > > mapred.job.queue.name > myQueue > > ``` > > After digging into the source code ,I found that kylin try to get the row > count of hive table before redistributing it. But it dose not override the > hive configuration when using jdbc to connect to hive server. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3296) cube合并时，java.lang.ArrayIndexOutOfBoundsException at java.lang.System.arraycopy(Native Method)
[ https://issues.apache.org/jira/browse/KYLIN-3296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16409335#comment-16409335 ] Billy Liu commented on KYLIN-3296: -- Hi [~renzm], this is not standard patch format. The patch should contain your committer info. > cube合并时，java.lang.ArrayIndexOutOfBoundsException at > java.lang.System.arraycopy(Native Method) > - > > Key: KYLIN-3296 > URL: https://issues.apache.org/jira/browse/KYLIN-3296 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.3.0 >Reporter: RenZhiMin >Priority: Major > Labels: patch > Attachments: JIRA.master.3296.patch > > > cube中，设计rowkey时，有个维度设置编码方式是固定长度500。每天采用内存构建算法。在合并cube时，在生成的mr中的map任务执行中出现“java.lang.ArrayIndexOutOfBoundsException > at java.lang.System.arraycopy(Native Method)” > 错误。经查看在生成的mr中的map任务中需要对要合并的cuboiddata数据的rowkey进行切分，切分时，根据每个维度的编码方式获取对应的长度，然后从rowkey中获取，并赋值给SplittedBytes的value中，由于value数组初始化时设置的固定值255，所以在切分大于255的维度值时，出现下标越界错误。 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3285) "Value NNN not exists" error run executing query
[
https://issues.apache.org/jira/browse/KYLIN-3285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Billy Liu updated KYLIN-3285:
-
Fix Version/s: (was: v2.3.1)
> "Value NNN not exists" error run executing query
> -
>
> Key: KYLIN-3285
> URL: https://issues.apache.org/jira/browse/KYLIN-3285
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v2.3.0
>Reporter: Shaofeng SHI
>Priority: Major
> Attachments: cube (1).json, kylin.log
>
>
> Reported by community user zxxb...@163.com:
>
> {color:#b94a48}Kylin version update from 2.2.0 to 2.3.0; cube was build
> before update, and query without error.{color}
> {color:#b94a48}After updatge to version 2.3.0 and query {color}"select
> count(userid) num,day_time from record_ap group by day_time LIMIT
> 1000{color:#b94a48}”{color}
> {color:#b94a48}show errors below:{color}
> Column 0 value '2018-03-06' met dictionary error: Value '2018-03-06'
> (2018-03-06) not exists! while executing SQL: "select count(userid)
> num,day_time from record_ap group by day_time LIMIT 1000”
>
> {color:#b94a48}Where {color}{color:#b94a48}'2018-03-06’ come from? I was
> fully confused.{color}
> {color:#b94a48}Cube have several segments:{color}
> {color:#b94a48}Starttime-endtime{color}
> {color:#b94a48}20180101-20180225{color}
> 20180225-20180304
>
> Attached the log and cube JSON.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3289) Refactor the storage garbage clean up code
[
https://issues.apache.org/jira/browse/KYLIN-3289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Billy Liu updated KYLIN-3289:
-
Fix Version/s: (was: v2.3.1)
> Refactor the storage garbage clean up code
> --
>
> Key: KYLIN-3289
> URL: https://issues.apache.org/jira/browse/KYLIN-3289
> Project: Kylin
> Issue Type: Improvement
>Affects Versions: v2.3.0
>Reporter: Guangyao Li
>Priority: Minor
>
> Kylin will produce some garbage data in storage when it runs.
> Now, the clean up tool "{{kylin.sh org.apache.kylin.tool.StorageCleanupJob}}"
> can show what is garbage data or clean up the garbage by setting options
> "–delete false" or "delete true".
> But Kylin can't show the size of garbage data for users.
> This reconfiguration adds some member variables and methods recording the
> garbage size in the detection process.
> After clean up job running over, Kylin can get the information about garbage
> size.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KYLIN-3270) Add integration test for Optimize Job
[ https://issues.apache.org/jira/browse/KYLIN-3270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu resolved KYLIN-3270. -- Resolution: Fixed > Add integration test for Optimize Job > - > > Key: KYLIN-3270 > URL: https://issues.apache.org/jira/browse/KYLIN-3270 > Project: Kylin > Issue Type: Sub-task > Components: Tools, Build and Test >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Critical > Fix For: v2.3.1 > > Attachments: APACHE-KYLIN-3270-fix-for-2.3.1.patch, > APACHE-KYLIN-3270-master.patch, APACHE-KYLIN-3270.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3301) Upgrade opensaml
[ https://issues.apache.org/jira/browse/KYLIN-3301?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407330#comment-16407330 ] Billy Liu commented on KYLIN-3301: -- +1 > Upgrade opensaml > > > Key: KYLIN-3301 > URL: https://issues.apache.org/jira/browse/KYLIN-3301 > Project: Kylin > Issue Type: Improvement > Components: Integration >Affects Versions: v2.2.0, v2.3.0 >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Major > Attachments: KYLIN-3301.master.001.patch > > > opensaml 2.6.4 is reported with security issue (CVE-2015-1796), need upgrade. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3300) Upgrade jackson-databind
[ https://issues.apache.org/jira/browse/KYLIN-3300?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16407326#comment-16407326 ] Billy Liu commented on KYLIN-3300: -- +1 > Upgrade jackson-databind > > > Key: KYLIN-3300 > URL: https://issues.apache.org/jira/browse/KYLIN-3300 > Project: Kylin > Issue Type: Improvement > Components: Integration >Affects Versions: v2.2.0, v2.3.0 >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Major > Attachments: KYLIN-3300.master.001.patch > > > jackson-databind 2.6.3 and 2.6.5 are reported with security issue > (CVE-2017-7525), need ugprade -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3297) sql在解析>大值 and <小值的时候，kylin出现内存溢出。
[ https://issues.apache.org/jira/browse/KYLIN-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405823#comment-16405823 ] Billy Liu commented on KYLIN-3297: -- Could you show the reproduce steps against the sample cube. > sql在解析>大值 and <小值的时候，kylin出现内存溢出。 > - > > Key: KYLIN-3297 > URL: https://issues.apache.org/jira/browse/KYLIN-3297 > Project: Kylin > Issue Type: Bug > Components: RDBMS Source >Affects Versions: v2.2.0 > Environment: oracle linux 6.8 > cup 8C > memory 48G > apache kylin v2.2.0 >Reporter: 陈家宇 >Priority: Major > > select f.plant_code,f.report_business_area,f.orderby,sum(f.a)a, > case when sum(f.a)=0 then 0 else sum(f.qty)/sum(f.a) end zt from ( > select h.business_area_desc,h.plant_code, > case when p.report_business_area like '%切大片%' then '切大片' > else p.report_business_area end report_business_area, > case when p.report_business_area like '%切大片%' then '00' > else p.orderby end orderby, > sum(p.quantity) qty,sum(a)a from pkn_kpi p > inner join hcm_area h on p.plant_id=h.plant_id > where 1=1 > and h.business_area_desc='上海' > and h.plant_code='S1' > and p.calendar_day>='2018-03-19' > and p.calendar_day<='2018-03-18' > and trim(p.report_business_area) is not null > group by h.business_area_desc,h.plant_code,p.report_business_area,p.orderby > ) f > group by f.plant_code,f.report_business_area,f.orderby > order by orderby > 以上的calendar_day条件，顺序输入反了，变成calendar_day>大值 and > calendar_day<小值，出现了kylin内存溢出，并且崩溃。正常这种情况是没有数据输出才对。 > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3296) cube合并时，java.lang.ArrayIndexOutOfBoundsException at java.lang.System.arraycopy(Native Method)
[ https://issues.apache.org/jira/browse/KYLIN-3296?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405689#comment-16405689 ] Billy Liu commented on KYLIN-3296: -- Nice found. > cube合并时，java.lang.ArrayIndexOutOfBoundsException at > java.lang.System.arraycopy(Native Method) > - > > Key: KYLIN-3296 > URL: https://issues.apache.org/jira/browse/KYLIN-3296 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.3.0 >Reporter: RenZhiMin >Priority: Major > Labels: patch > Attachments: RowKeySplitter.patch > > > cube中，设计rowkey时，有个维度设置编码方式是固定长度500。每天采用内存构建算法。在合并cube时，在生成的mr中的map任务执行中出现“java.lang.ArrayIndexOutOfBoundsException > at java.lang.System.arraycopy(Native Method)” > 错误。经查看在生成的mr中的map任务中需要对要合并的cuboiddata数据的rowkey进行切分，切分时，根据每个维度的编码方式获取对应的长度，然后从rowkey中获取，并赋值给SplittedBytes的value中，由于value数组初始化时设置的固定值255，所以在切分大于255的维度值时，出现下标越界错误。 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3297) sql在解析>大值 and <小值的时候，kylin出现内存溢出。
[ https://issues.apache.org/jira/browse/KYLIN-3297?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16405687#comment-16405687 ] Billy Liu commented on KYLIN-3297: -- Hi [~xer001], thank you for reporting issue. Could you reproduce this issue against the sample cube? It will help other developers identify the root cause more efficiently. > sql在解析>大值 and <小值的时候，kylin出现内存溢出。 > - > > Key: KYLIN-3297 > URL: https://issues.apache.org/jira/browse/KYLIN-3297 > Project: Kylin > Issue Type: Bug > Components: RDBMS Source >Affects Versions: v2.2.0 > Environment: oracle linux 6.8 > cup 8C > memory 48G > apache kylin v2.2.0 >Reporter: 陈家宇 >Priority: Major > > select f.plant_code,f.report_business_area,f.orderby,sum(f.a)a, > case when sum(f.a)=0 then 0 else sum(f.qty)/sum(f.a) end zt from ( > select h.business_area_desc,h.plant_code, > case when p.report_business_area like '%切大片%' then '切大片' > else p.report_business_area end report_business_area, > case when p.report_business_area like '%切大片%' then '00' > else p.orderby end orderby, > sum(p.quantity) qty,sum(a)a from pkn_kpi p > inner join hcm_area h on p.plant_id=h.plant_id > where 1=1 > and h.business_area_desc='上海' > and h.plant_code='S1' > and p.calendar_day>='2018-03-19' > and p.calendar_day<='2018-03-18' > and trim(p.report_business_area) is not null > group by h.business_area_desc,h.plant_code,p.report_business_area,p.orderby > ) f > group by f.plant_code,f.report_business_area,f.orderby > order by orderby > 以上的calendar_day条件，顺序输入反了，变成calendar_day>大值 and > calendar_day<小值，出现了kylin内存溢出，并且崩溃。正常这种情况是没有数据输出才对。 > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3276) query with dynamic parameter get the same query result
[ https://issues.apache.org/jira/browse/KYLIN-3276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu resolved KYLIN-3276. -- Resolution: Fixed > query with dynamic parameter get the same query result > -- > > Key: KYLIN-3276 > URL: https://issues.apache.org/jira/browse/KYLIN-3276 > Project: Kylin > Issue Type: Bug > Components: Driver - JDBC, Query Engine >Affects Versions: v2.2.0 > Environment: Kylin 2.2.0, JDK 1.8 >Reporter: ZHAOXIXIN >Assignee: nichunen >Priority: Major > Fix For: v2.3.1 > > > sql:"select distinct uid from table where daytime=? limit 70" > ps.setString(1,"2018-01-03") > store the result to ArrayList, and print the size of ArrayList. > when I set the daytime to different dates, the same result I got. > I changed the sql, just change limit value for example > sql:"select distinct uid from table where daytime=? limit 60" > then I got the right value the same like query from GUI:count (distinct uid) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3279) Util Class for encryption and decryption
[ https://issues.apache.org/jira/browse/KYLIN-3279?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu resolved KYLIN-3279. -- Resolution: Fixed > Util Class for encryption and decryption > > > Key: KYLIN-3279 > URL: https://issues.apache.org/jira/browse/KYLIN-3279 > Project: Kylin > Issue Type: Improvement > Components: Others >Affects Versions: v2.3.0 >Reporter: nichunen >Assignee: nichunen >Priority: Minor > Fix For: v2.4.0 > > > Extract methods of "encrypt" and "decrypt" in > org.apache.kylin.rest.security.PasswordPlaceholderConfigurer to a Util class > in core-common package. This will make them more reusable. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3281) OLAPProjectRule can't normal working with projectRel[input=sortRel]
[
https://issues.apache.org/jira/browse/KYLIN-3281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Billy Liu resolved KYLIN-3281.
--
Resolution: Fixed
Fix Version/s: v2.4.0
> OLAPProjectRule can't normal working with projectRel[input=sortRel]
>
>
> Key: KYLIN-3281
> URL: https://issues.apache.org/jira/browse/KYLIN-3281
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v2.3.0
>Reporter: yiming.xu
>Assignee: yiming.xu
>Priority: Critical
> Fix For: v2.4.0
>
>
> for queries like:
> select sum("count") from (SELECT LSTG_FORMAT_NAME , COUNT(*) AS "count" FROM
> TEST_KYLIN_FACT GROUP BY "LSTG_FORMAT_NAME" ORDER BY "count" DESC) as a
> the expected logical plan after volcano is:
> {code:java}
> OLAPToEnumerableConverter
> OLAPLimitRel(ctx=[], fetch=[5])
> OLAPAggregateRel(group=[{}], EXPR$0=[SUM($0)], ctx=[])
> OLAPProjectRel(count=[$1], ctx=[])
> OLAPSortRel(sort0=[$1], dir0=[DESC], ctx=[])
> OLAPAggregateRel(group=[{0}], count=[COUNT()], ctx=[])
> OLAPProjectRel(LSTG_FORMAT_NAME=[$3], ctx=[])
> OLAPTableScan(table=[[DEFAULT, TEST_KYLIN_FACT]], ctx=[],
> fields=[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18,
> 19, 20, 21, 22, 23, 24, 25, 26]])
> {code}
> however the actual is:
> {code:java}
> EnumerableLimit(fetch=[5])
> EnumerableAggregate(group=[{}], EXPR$0=[SUM($0)])
> EnumerableCalc(expr#0..1=[{inputs}], count=[$t1])
> EnumerableSort(sort0=[$1], dir0=[DESC])
> EnumerableAggregate(group=[{0}], count=[COUNT()])
> OLAPToEnumerableConverter
> OLAPProjectRel(LSTG_FORMAT_NAME=[$3], ctx=[])
> OLAPTableScan(table=[[DEFAULT, TEST_KYLIN_FACT]], ctx=[],
> fields=[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18,
> 19, 20, 21, 22, 23, 24, 25, 26]])
> {code}
> VolcanoPlanner apply OLAPProjectRule to logicalProject,convert child node
> should use child traitset not current proejct traitset.
> If we use current project trait set, the child node trait set is [olap][0
> DESC], it will create a new subset but the subset doesn't have any rel. the
> cost is inf,
> so volcanoPlanner choose a the EnumerableAggregate.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KYLIN-3284) Refactor all OLAPRel computeSelfCost
[
https://issues.apache.org/jira/browse/KYLIN-3284?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Billy Liu resolved KYLIN-3284.
--
Resolution: Fixed
Fix Version/s: v2.4.0
> Refactor all OLAPRel computeSelfCost
>
>
> Key: KYLIN-3284
> URL: https://issues.apache.org/jira/browse/KYLIN-3284
> Project: Kylin
> Issue Type: Improvement
> Components: Query Engine
>Affects Versions: v2.3.0
>Reporter: yiming.xu
>Assignee: yiming.xu
>Priority: Major
> Fix For: v2.4.0
>
>
> for queries like:
> select lstg_format_name from test_kylin_fact order by case
> when 1=1 then
> cal_dt
> ELSE
> seller_id
> end
> the expected logical plan after volcano is:
> {code:java}
> OLAPToEnumerableConverter
> OLAPLimitRel(ctx=[], fetch=[5])
> OLAPSortRel(sort0=[$1], dir0=[ASC], ctx=[])
> OLAPProjectRel(LSTG_FORMAT_NAME=[$3], EXPR$1=[CASE(=(1, 1), $2,
> CAST($7):DATE)], ctx=[])
> OLAPTableScan(table=[[DEFAULT, TEST_KYLIN_FACT]], ctx=[], fields=[[0,
> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21,
> 22, 23, 24, 25, 26]])
> {code}
> however the actual is:
> {code:java}
> EnumerableSort(sort0=[$1], dir0=[ASC])
> OLAPToEnumerableConverter
> OLAPSortRel(sort0=[$1], dir0=[ASC], ctx=[])
> OLAPProjectRel(LSTG_FORMAT_NAME=[$3], EXPR$1=[CASE(=(1, 1), $2,
> CAST($7):DATE)], ctx=[])
> OLAPTableScan(table=[[DEFAULT, TEST_KYLIN_FACT]], ctx=[], fields=[[0,
> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21,
> 22, 23, 24, 25, 26]])
> {code}
> This is the RelOptPlanner:
> {code:java}
> Sets:
> Set#0, type: RecordType(BIGINT TRANS_ID, BIGINT ORDER_ID, DATE CAL_DT,
> VARCHAR(256) LSTG_FORMAT_NAME, BIGINT LEAF_CATEG_ID, INTEGER LSTG_SITE_ID,
> SMALLINT SLR_SEGMENT_CD, INTEGER SELLER_ID, DECIMAL(19, 4) PRICE, INTEGER
> ITEM_COUNT, VARCHAR(256) TEST_COUNT_DISTINCT_BITMAP, DECIMAL(19, 4)
> DEAL_AMOUNT, INTEGER DEAL_YEAR, VARCHAR(256)
> LEFTJOIN_BUYER_ID_AND_COUNTRY_NAME, VARCHAR(256)
> LEFTJOIN_SELLER_ID_AND_COUNTRY_NAME, VARCHAR(256)
> LEFTJOIN_BUYER_COUNTRY_ABBR, VARCHAR(256) LEFTJOIN_SELLER_COUNTRY_ABBR,
> BIGINT _KY_COUNT__, INTEGER _KY_SUM_TEST_KYLIN_FACT_ITEM_COUNT_, DECIMAL(19,
> 4) _KY_SUM_TEST_KYLIN_FACT_PRICE_, DECIMAL(19, 4)
> _KY_MIN_TEST_KYLIN_FACT_PRICE_, DECIMAL(19, 4)
> _KY_MAX_TEST_KYLIN_FACT_PRICE_, ANY
> _KY_COUNT_DISTINCT_TEST_KYLIN_FACT_SELLER_ID_, ANY
> _KY_COUNT_DISTINCT_TEST_KYLIN_FACT_LSTG_FORMAT_NAME_TEST_KYLIN_FACT_SELLER_ID_,
> ANY _KY_COUNT_DISTINCT_TEST_KYLIN_FACT_TEST_COUNT_DISTINCT_BITMAP_,
> DECIMAL(19, 4) _KY_SUM_TEST_KYLIN_FACT_DEAL_AMOUNT_, ANY
> _KY_PERCENTILE_APPROX_TEST_KYLIN_FACT_PRICE_)
> rel#140:Subset#0.OLAP.[], best=rel#131, importance=0.7291
> rel#131:OLAPTableScan.OLAP.[](table=[DEFAULT,
> TEST_KYLIN_FACT],ctx=,fields=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13,
> 14, 15, 16, 17, 18, 19, 20, 21, 22, 23, 24, 25, 26]), rowcount=100.0,
> cumulative cost={5.0 rows, 5.051 cpu, 0.0 io}
>
> rel#179:OLAPSortRel.OLAP.[2](input=rel#140:Subset#0.OLAP.[],sort0=$2,dir0=ASC,ctx=),
> rowcount=100.0, cumulative cost={2491.7919004335695 rows, 10.05 cpu, 0.0 io}
> rel#169:Subset#0.NONE.[2], best=null, importance=0.7291
>
> rel#167:LogicalSort.NONE.[2](input=rel#140:Subset#0.OLAP.[],sort0=$2,dir0=ASC),
> rowcount=100.0, cumulative cost={inf}
> rel#171:Subset#0.ENUMERABLE.[2], best=rel#182, importance=0.81
>
> rel#181:EnumerableSort.ENUMERABLE.[2](input=rel#177:Subset#0.ENUMERABLE.[],sort0=$2,dir0=ASC),
> rowcount=100.0, cumulative cost={1.0E100 rows, 110.05 cpu, 0.0 io}
>
> rel#182:OLAPToEnumerableConverter.ENUMERABLE.[2](input=rel#180:Subset#0.OLAP.[2]),
> rowcount=100.0, cumulative cost={1.0E100 rows, 10.05 cpu, 0.0 io}
> rel#173:Subset#0.OLAP.[1], best=null, importance=0.6561
> rel#177:Subset#0.ENUMERABLE.[], best=rel#182,
> importance=0.7291
>
> rel#181:EnumerableSort.ENUMERABLE.[2](input=rel#177:Subset#0.ENUMERABLE.[],sort0=$2,dir0=ASC),
> rowcount=100.0, cumulative cost={1.0E100 rows, 110.05 cpu, 0.0 io}
>
> rel#182:OLAPToEnumerableConverter.ENUMERABLE.[2](input=rel#180:Subset#0.OLAP.[2]),
> rowcount=100.0, cumulative cost={1.0E100 rows, 10.05 cpu, 0.0 io}
>
> rel#185:OLAPToEnumerableConverter.ENUMERABLE.[](input=rel#140:Subset#0.OLAP.[]),
> rowcount=100.0, cumulative cost={1.0E100 rows, 5.051 cpu, 0.0 io}
> rel#180:Subset#0.OLAP.[2], best=rel#179, importance=0.6561
>
> rel#179:OLAPSortRel.OLAP.[2](input=rel#140:Subset#0.OLAP.[],sort0=$2,dir0=ASC,ctx=),
> rowcount=100.0, cumulative cost={2491.7919004335695 rows, 10.05 cpu, 0.0 io}
> Set#1,
[jira] [Updated] (KYLIN-3149) Calcite's ReduceExpressionsRule.PROJECT_INSTANCE not working as expected
[
https://issues.apache.org/jira/browse/KYLIN-3149?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Billy Liu updated KYLIN-3149:
-
Fix Version/s: v2.4.0
> Calcite's ReduceExpressionsRule.PROJECT_INSTANCE not working as expected
>
>
> Key: KYLIN-3149
> URL: https://issues.apache.org/jira/browse/KYLIN-3149
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.2.0
>Reporter: hongbin ma
>Assignee: yiming.xu
>Priority: Major
> Fix For: v2.4.0
>
> Attachments: dump.txt
>
>
> for queries like:
> {code:sql}
> select TRANS_ID from kylin_sales group by cast (case
> WHEN '1030101' = '1030101' then substring(COALESCE(OPS_USER_ID,
> ''), 1, 1)
> when '1030101' = '1030102' then substring(COALESCE(OPS_REGION,
> ''), 1, 1)
> when '1030101' = '1030103' then substring(COALESCE(LSTG_FORMAT_NAME,
> ''), 1, 1)
> when '1030101' = '1030104' then substring(COALESCE(LSTG_FORMAT_NAME,
> ''), 1, 1)
> end as varchar(256)), TRANS_ID;
> {code}
> the expected logical plan after volcano is:
> {code}
> EXECUTION PLAN BEFORE REWRITE
> OLAPToEnumerableConverter
> OLAPProjectRel(TRANS_ID=[$1], ctx=[])
> OLAPLimitRel(ctx=[], fetch=[5])
> OLAPAggregateRel(group=[{0, 1}], ctx=[])
> OLAPProjectRel($f0=[SUBSTRING(CASE(IS NOT NULL($9), $9,
> ''), 1, 1)], TRANS_ID=[$0], ctx=[])
> OLAPTableScan(table=[[DEFAULT, KYLIN_SALES]], ctx=[], fields=[[0,
> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13]])
> {code}
> however the actual is:
> {code}
> EXECUTION PLAN BEFORE REWRITE
> OLAPToEnumerableConverter
> OLAPLimitRel(ctx=[], fetch=[5])
> OLAPProjectRel(TRANS_ID=[$1], ctx=[])
> OLAPAggregateRel(group=[{0, 1}], ctx=[])
> OLAPProjectRel($f0=[CAST(CASE(=('1030101', '1030101'),
> SUBSTRING(CASE(IS NOT NULL($9), $9, ''), 1, 1), =('1030101',
> '1030102'), SUBSTRING(CASE(IS NOT NULL($10), $10, ''), 1, 1),
> =('1030101', '1030103'), SUBSTRING(CASE(IS NOT NULL($2), $2, ''),
> 1, 1), =('1030101', '1030104'), SUBSTRING(CASE(IS NOT NULL($2), $2,
> ''), 1, 1), null)):VARCHAR(256) CHARACTER SET "UTF-16LE" COLLATE
> "UTF-16LE$en_US$primary"], TRANS_ID=[$0], ctx=[])
> OLAPTableScan(table=[[DEFAULT, KYLIN_SALES]], ctx=[], fields=[[0,
> 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13]])
> {code}
> looks like Calcite's ReduceExpressionsRule.PROJECT_INSTANCE not working as
> expected. If we dump the internal state of this VolcanoPlanner
> (org.apache.calcite.plan.volcano.VolcanoPlanner#dump), line 19-21 from the
> complete dump is attached:
> {code}
> rel#337:Subset#1.OLAP.[], best=rel#339, importance=0.6561
>
> rel#339:OLAPProjectRel.OLAP.[](input=rel#303:Subset#0.OLAP.[],$f0=CAST(CASE(=('1030101',
> '1030101'), SUBSTRING(CASE(IS NOT NULL($9), $9, ''), 1, 1),
> =('1030101', '1030102'), SUBSTRING(CASE(IS NOT NULL($10), $10,
> ''), 1, 1), =('1030101', '1030103'), SUBSTRING(CASE(IS NOT
> NULL($2), $2, ''), 1, 1), =('1030101', '1030104'),
> SUBSTRING(CASE(IS NOT NULL($2), $2, ''), 1, 1),
> null)):VARCHAR(256) CHARACTER SET "UTF-16LE" COLLATE
> "UTF-16LE$en_US$primary",TRANS_ID=$0,ctx=), rowcount=100.0, cumulative
> cost={15.0 rows, 25.05 cpu, 0.0 io}
>
> rel#348:OLAPProjectRel.OLAP.[](input=rel#303:Subset#0.OLAP.[],$f0=SUBSTRING(CASE(IS
> NOT NULL($9), $9, ''), 1, 1),TRANS_ID=$0,ctx=), rowcount=100.0,
> cumulative cost={15.0 rows, 25.05 cpu, 0.0 io}
> {code}
> we see two rels with same cost: #339 and #348, where #339 is created from
> LogicalProject = (OLAPProjectRule)=> OLAPProject, and #348 is created from
> LogicalProject =( ReduceExpressionsRule) => Reduced LogicalProject
> =(OLAPProjectRule)=> Reduced OLAPProject . Since ReduceExpressionsRule
> require Logical Project rather than OLAP Project, #339 is never reduced.
> The worse thing is that cost of #339 and #348 are same. By current volcano
> planner algorithm the first met rel will be chosen, so unexpected rel is
> chosen
> A simple approach to fix this is to refine the rel choosing algorithm: when
> two rels are equal in cost, choose a "simpler" one. Since we don't have a
> perfect measurement of "simple", we simply choose the rel with smaller
> toString() length
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3283) Support values RelNode
[
https://issues.apache.org/jira/browse/KYLIN-3283?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Billy Liu updated KYLIN-3283:
-
Fix Version/s: v2.4.0
> Support values RelNode
> --
>
> Key: KYLIN-3283
> URL: https://issues.apache.org/jira/browse/KYLIN-3283
> Project: Kylin
> Issue Type: New Feature
> Components: Query Engine
>Affects Versions: v2.3.0
>Reporter: yiming.xu
>Assignee: yiming.xu
>Priority: Major
> Fix For: v2.4.0
>
>
> for queries like:
> select leaf_categ_id, sum(price) as sum_price from test_kylin_fact group by
> leaf_categ_id
> union all
> select 1 as leaf_categ_id2, 2 as sum_price2
> the expected logical plan after volcano is:
> {code:java}
> OLAPToEnumerableConverter
> OLAPLimitRel(ctx=[], fetch=[5])
> OLAPUnionRel(all=[true], ctx=[], all=[true])
> OLAPAggregateRel(group=[{0}], SUM_PRICE=[SUM($1)], ctx=[])
> OLAPProjectRel(LEAF_CATEG_ID=[$4], PRICE=[$8], ctx=[])
> OLAPTableScan(table=[[DEFAULT, TEST_KYLIN_FACT]], ctx=[],
> fields=[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18,
> 19, 20, 21, 22, 23, 24, 25, 26]])
> OLAPProjectRel(LEAF_CATEG_ID2=[1], SUM_PRICE2=[2], ctx=[])
> OLAPValuesRel(type=[RecordType(INTEGER ZERO)], tuples=[[{ 0 }]])
> {code}
> however the actual is:
> {code:java}
> EnumerableLimit(fetch=[5])
> EnumerableUnion(all=[true])
> OLAPToEnumerableConverter
> OLAPAggregateRel(group=[{0}], SUM_PRICE=[SUM($1)], ctx=[])
> OLAPProjectRel(LEAF_CATEG_ID=[$4], PRICE=[$8], ctx=[])
> OLAPTableScan(table=[[DEFAULT, TEST_KYLIN_FACT]], ctx=[],
> fields=[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18,
> 19, 20, 21, 22, 23, 24, 25, 26]])
> EnumerableCalc(expr#0=[{inputs}], expr#1=[1], expr#2=[2],
> LEAF_CATEG_ID2=[$t1], SUM_PRICE2=[$t2])
> EnumerableValues(type=[RecordType(INTEGER ZERO)], tuples=[[{ 0 }]])
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3262) Produce jacoco exec files for integration test
[ https://issues.apache.org/jira/browse/KYLIN-3262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu updated KYLIN-3262: - Fix Version/s: v2.4.0 > Produce jacoco exec files for integration test > -- > > Key: KYLIN-3262 > URL: https://issues.apache.org/jira/browse/KYLIN-3262 > Project: Kylin > Issue Type: Task >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v2.4.0 > > Attachments: APACHE-KYLIN-3262.patch > > > Currently to calculate code coverage, kylin only considers UT. However, IT > also covers lots of code. It's better to include this for code coverage. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3280) The delete button should not be enabled without any segment in cube segment delete confirm dialog
[ https://issues.apache.org/jira/browse/KYLIN-3280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu resolved KYLIN-3280. -- Resolution: Fixed > The delete button should not be enabled without any segment in cube segment > delete confirm dialog > - > > Key: KYLIN-3280 > URL: https://issues.apache.org/jira/browse/KYLIN-3280 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v2.3.0 >Reporter: Peng Xing >Assignee: Peng Xing >Priority: Minor > Fix For: v2.3.1 > > Attachments: delete_segment.png > > > 1. Open cube list, choose one disabled cube, then select 'Delete Segment'; > 2. The Delete button is enabled without any segment, and if you click it, it > has no prompt. > please see the attachment picture. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3280) The delete button should not be enabled without any segment in cube segment delete confirm dialog
[ https://issues.apache.org/jira/browse/KYLIN-3280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu updated KYLIN-3280: - Affects Version/s: (was: v2.4.0) v2.3.0 > The delete button should not be enabled without any segment in cube segment > delete confirm dialog > - > > Key: KYLIN-3280 > URL: https://issues.apache.org/jira/browse/KYLIN-3280 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v2.3.0 >Reporter: Peng Xing >Assignee: Peng Xing >Priority: Minor > Fix For: v2.3.1 > > Attachments: delete_segment.png > > > 1. Open cube list, choose one disabled cube, then select 'Delete Segment'; > 2. The Delete button is enabled without any segment, and if you click it, it > has no prompt. > please see the attachment picture. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3119) A few bugs in the function 'massageSql' of 'QueryUtil.java'
[
https://issues.apache.org/jira/browse/KYLIN-3119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Billy Liu resolved KYLIN-3119.
--
Resolution: Fixed
> A few bugs in the function 'massageSql' of 'QueryUtil.java'
> ---
>
> Key: KYLIN-3119
> URL: https://issues.apache.org/jira/browse/KYLIN-3119
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Reporter: peng.jianhua
>Assignee: peng.jianhua
>Priority: Major
> Fix For: v2.3.1
>
> Attachments:
> 0001-KYLIN-3119-A-few-bugs-in-the-function-massageSql-of-.patch,
> 0001-KYLIN-3119-add-test-cases.patch,
> 0001-kylin-3119-add-more-test-cases-with-enter-and-space.patch,
> 0001-kylin-3119.patch, 01.PNG, 02.PNG
>
>
> in the function 'massageSql' of 'QueryUtil.java',there are a few bugs:
> 01:
> {code:java}
> while (sql.endsWith(";"))
> sql = sql.substring(0, sql.length() - 1);
> {code}
> if the sql end with ';' and after the ';' still has comments,will be
> error.Because the sql will add ("\nLIMIT " + limit) at the end.
> 02:
> {code:java}
> if (limit > 0 && !sql.toLowerCase().contains("limit")) {
> sql += ("\nLIMIT " + limit);
> }
> if (offset > 0 && !sql.toLowerCase().contains("offset")) {
> sql += ("\nOFFSET " + offset);
> }
> {code}
> if the sql already has word 'limit' in it,such as Alias,Subquery,Comments,the
> ‘limit’ in the Input box will be invalid.
> for example,the sql has subquery,and there is 'limit' in subquery.
> {code:java}
> select KYLIN_SALES.PART_DT, count(KYLIN_SALES.PRICE)
> from KYLIN_SALES
> inner join (select ACCOUNT_ID, ACCOUNT_BUYER_LEVEL from KYLIN_ACCOUNT where
> ACCOUNT_COUNTRY = 'US' limit 1) as TT
> on KYLIN_SALES.BUYER_ID = TT.ACCOUNT_ID
> group by KYLIN_SALES.PART_DT
> {code}
> the ‘limit’ in the Input box will be invalid.please refer to 01.png and
> 02.png.
> 03:
> {code:java}
> // https://issues.apache.org/jira/browse/KYLIN-2649
> if (kylinConfig.getForceLimit() > 0 &&
> !sql.toLowerCase().contains("limit")
> && sql.toLowerCase().contains("*")) {
> sql += ("\nLIMIT " + kylinConfig.getForceLimit());
> }
> {code}
> Because KYLIN-2649 is still unresolved,so I didn't change the code,but it
> has same 'limit' word bugs like above.
> And there are some situations that the sql contains \*,such as
> {code:java}count(*),kylin_sales.*{code}
> Maybe the \* in subquery too.
> it seems improperly to deal with the sql just use {code:java}
> sql.toLowerCase().contains("*")
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KYLIN-3287) When a shard by column is in dict encoding, dict building error.
[ https://issues.apache.org/jira/browse/KYLIN-3287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu resolved KYLIN-3287. -- Resolution: Fixed > When a shard by column is in dict encoding, dict building error. > > > Key: KYLIN-3287 > URL: https://issues.apache.org/jira/browse/KYLIN-3287 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.3.0 >Reporter: jiatao.tao >Assignee: jiatao.tao >Priority: Major > Fix For: v2.3.1 > > Attachments: image-2018-03-12-14-14-50-112.png, > image-2018-03-12-14-14-59-193.png, image-2018-03-12-14-15-42-046.png, > image-2018-03-12-14-16-38-657.png, image-2018-03-12-14-17-46-780.png > > > In dict encoding. > !image-2018-03-12-14-14-50-112.png|width=544,height=84! > !image-2018-03-12-14-14-59-193.png|width=535,height=631! > In int encoding > !image-2018-03-12-14-15-42-046.png|width=544,height=87! > !image-2018-03-12-14-16-38-657.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KYLIN-3262) Produce jacoco exec files for integration test
[ https://issues.apache.org/jira/browse/KYLIN-3262?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu reassigned KYLIN-3262: Assignee: Zhong Yanghong > Produce jacoco exec files for integration test > -- > > Key: KYLIN-3262 > URL: https://issues.apache.org/jira/browse/KYLIN-3262 > Project: Kylin > Issue Type: Task >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Attachments: APACHE-KYLIN-3262.patch > > > Currently to calculate code coverage, kylin only considers UT. However, IT > also covers lots of code. It's better to include this for code coverage. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3278) Kylin should not distribute hive table by random at Step1
[ https://issues.apache.org/jira/browse/KYLIN-3278?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu updated KYLIN-3278: - Fix Version/s: v2.3.1 > Kylin should not distribute hive table by random at Step1 > - > > Key: KYLIN-3278 > URL: https://issues.apache.org/jira/browse/KYLIN-3278 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Affects Versions: v2.3.0 >Reporter: Chuqian Yu >Assignee: Wang Cheng >Priority: Major > Fix For: v2.3.1 > > > Hi,Kylin team. > When I upgrade from 2.2.0 to 2.3.0 ,I found that the step.1 "Create > Intermediate Flat Hive Table" of cube build was much slower than before, > because Kylin distribute hive table by random when created it in version > 2.3.0 . I wander is it necessary? Because > kylin.source.hive.redistribute-flat-table was set to true by default, so we > will do the distribution job at Step.2 "Redistribute Flat Hive Table". > Best regards, > Chuqian Yu -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3275) Add unit test for StorageCleanupJob
[ https://issues.apache.org/jira/browse/KYLIN-3275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu resolved KYLIN-3275. -- Resolution: Fixed Fix Version/s: v2.4.0 > Add unit test for StorageCleanupJob > --- > > Key: KYLIN-3275 > URL: https://issues.apache.org/jira/browse/KYLIN-3275 > Project: Kylin > Issue Type: Improvement >Reporter: jiatao.tao >Assignee: jiatao.tao >Priority: Trivial > Fix For: v2.4.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3174) Default scheduler enhancement
[ https://issues.apache.org/jira/browse/KYLIN-3174?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu resolved KYLIN-3174. -- Resolution: Fixed Fix Version/s: v2.4.0 > Default scheduler enhancement > - > > Key: KYLIN-3174 > URL: https://issues.apache.org/jira/browse/KYLIN-3174 > Project: Kylin > Issue Type: Improvement >Reporter: jiatao.tao >Assignee: jiatao.tao >Priority: Minor > Fix For: v2.4.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3271) Optimize sub-path check of ResourceTool
[
https://issues.apache.org/jira/browse/KYLIN-3271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Billy Liu resolved KYLIN-3271.
--
Resolution: Fixed
> Optimize sub-path check of ResourceTool
> ---
>
> Key: KYLIN-3271
> URL: https://issues.apache.org/jira/browse/KYLIN-3271
> Project: Kylin
> Issue Type: Improvement
> Components: Metadata
>Affects Versions: v2.2.0
>Reporter: nichunen
>Assignee: nichunen
>Priority: Minor
> Fix For: v2.4.0
>
>
> kylin uses class org.apache.kylin.common.persistence.ResourceTool to do
> metadata download, upload, remove, etc. The algorithm for resource
> transversal is not very effective. For instance, for an "execute_output" with
> key "/execute_output/\{uuid}", the algorithm will try to check whether it's a
> folder with sub-resources, this makes un-necessary time cost, and in cases of
> metadata with lots of jobs, it may last for a long time before the finish.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KYLIN-3248) Add batch grant API for project ACL.
[ https://issues.apache.org/jira/browse/KYLIN-3248?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu resolved KYLIN-3248. -- Resolution: Fixed Fix Version/s: v2.4.0 > Add batch grant API for project ACL. > > > Key: KYLIN-3248 > URL: https://issues.apache.org/jira/browse/KYLIN-3248 > Project: Kylin > Issue Type: Improvement >Reporter: jiatao.tao >Assignee: jiatao.tao >Priority: Major > Fix For: v2.4.0 > > > Project ACL stored in one file, and leads to adding huge ACLs inefficient. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3266) Improve CI coverage
[ https://issues.apache.org/jira/browse/KYLIN-3266?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu resolved KYLIN-3266. -- Resolution: Fixed Fix Version/s: v2.4.0 > Improve CI coverage > --- > > Key: KYLIN-3266 > URL: https://issues.apache.org/jira/browse/KYLIN-3266 > Project: Kylin > Issue Type: Improvement >Affects Versions: v2.4.0 >Reporter: liyang >Assignee: liyang >Priority: Major > Fix For: v2.4.0 > > > The current CI, aka BuildCubeWithEngine, only test merge on the MR engine. > The merge operation is not tested on Spark engine. Need to improve the test > coverage. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3220) Add manager for project ACL.
[ https://issues.apache.org/jira/browse/KYLIN-3220?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu resolved KYLIN-3220. -- Resolution: Fixed Fix Version/s: v2.4.0 > Add manager for project ACL. > > > Key: KYLIN-3220 > URL: https://issues.apache.org/jira/browse/KYLIN-3220 > Project: Kylin > Issue Type: Improvement >Reporter: jiatao.tao >Assignee: jiatao.tao >Priority: Major > Fix For: v2.4.0 > > > AclRecord need to be cached. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3251) Add a hook that can customer made test_case_data
[ https://issues.apache.org/jira/browse/KYLIN-3251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu resolved KYLIN-3251. -- Resolution: Fixed Fix Version/s: v2.4.0 > Add a hook that can customer made test_case_data > > > Key: KYLIN-3251 > URL: https://issues.apache.org/jira/browse/KYLIN-3251 > Project: Kylin > Issue Type: Improvement >Reporter: jiatao.tao >Assignee: jiatao.tao >Priority: Minor > Fix For: v2.4.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3267) add override MR config at project/cube level only for mem-hungry build steps
[
https://issues.apache.org/jira/browse/KYLIN-3267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Billy Liu resolved KYLIN-3267.
--
Resolution: Fixed
Fix Version/s: v2.4.0
> add override MR config at project/cube level only for mem-hungry build steps
> -
>
> Key: KYLIN-3267
> URL: https://issues.apache.org/jira/browse/KYLIN-3267
> Project: Kylin
> Issue Type: Improvement
>Reporter: Yifei Wu
>Assignee: Yifei Wu
>Priority: Major
> Fix For: v2.4.0
>
>
> Currently, users can override MR config at project/cube level via kylin
> properties {{kylin.engine.mr.config-override.}}
> However, this applies to all MR steps. It is more reasonable to add
> override-configs only for certain memory-hungry steps like in-mem cubing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KYLIN-3233) CacheController can not handle if cacheKey has "/"
[ https://issues.apache.org/jira/browse/KYLIN-3233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu resolved KYLIN-3233. -- Resolution: Fixed Fix Version/s: v2.3.1 > CacheController can not handle if cacheKey has "/" > -- > > Key: KYLIN-3233 > URL: https://issues.apache.org/jira/browse/KYLIN-3233 > Project: Kylin > Issue Type: Improvement >Reporter: jiatao.tao >Assignee: jiatao.tao >Priority: Minor > Fix For: v2.3.1 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3234) ResourceStore should add a API that can recursively list path.
[ https://issues.apache.org/jira/browse/KYLIN-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu resolved KYLIN-3234. -- Resolution: Fixed Fix Version/s: v2.4.0 > ResourceStore should add a API that can recursively list path. > -- > > Key: KYLIN-3234 > URL: https://issues.apache.org/jira/browse/KYLIN-3234 > Project: Kylin > Issue Type: Improvement >Reporter: jiatao.tao >Assignee: jiatao.tao >Priority: Minor > Fix For: v2.4.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3236) The function 'reGenerateAdvancedDict()' has an error logical judgment, which will cause an exception when you edit the cube.
[
https://issues.apache.org/jira/browse/KYLIN-3236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Billy Liu updated KYLIN-3236:
-
Fix Version/s: (was: v2.4.0)
v2.3.1
> The function 'reGenerateAdvancedDict()' has an error logical judgment, which
> will cause an exception when you edit the cube.
>
>
> Key: KYLIN-3236
> URL: https://issues.apache.org/jira/browse/KYLIN-3236
> Project: Kylin
> Issue Type: Bug
> Components: Web
>Affects Versions: v2.3.0
>Reporter: Peng Xing
>Assignee: Peng Xing
>Priority: Major
> Labels: patch
> Fix For: v2.3.1
>
> Attachments:
> 0001-KYLIN-3236-The-function-reGenerateAdvancedDict-has-a.patch, 1.PNG,
> 2.PNG, 3.PNG
>
>
> # When remove deprecated distinct measures, the code has an error logical
> judgment In function 'reGenerateAdvancedDict()', as follows:
> {code:java}
> //remove deprecated distinct measures
> angular.forEach($scope.cubeMetaFrame.dictionaries, function (dict, index)
> {
> if (distinctMeasures.indexOf(dict.column) === -1 &&
> reuseColumns.indexOf(dict.column) === -1) {
> $scope.cubeMetaFrame.dictionaries.splice(index, 1);
> }
> });
> {code}
> This will result in accidental deletion of advanced dictionaries.
> Please review the patch, thanks!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3119) A few bugs in the function 'massageSql' of 'QueryUtil.java'
[
https://issues.apache.org/jira/browse/KYLIN-3119?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Billy Liu updated KYLIN-3119:
-
Fix Version/s: v2.3.1
> A few bugs in the function 'massageSql' of 'QueryUtil.java'
> ---
>
> Key: KYLIN-3119
> URL: https://issues.apache.org/jira/browse/KYLIN-3119
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Reporter: peng.jianhua
>Assignee: peng.jianhua
>Priority: Major
> Fix For: v2.3.1
>
> Attachments:
> 0001-KYLIN-3119-A-few-bugs-in-the-function-massageSql-of-.patch,
> 0001-KYLIN-3119-add-test-cases.patch,
> 0001-kylin-3119-add-more-test-cases-with-enter-and-space.patch,
> 0001-kylin-3119.patch, 01.PNG, 02.PNG
>
>
> in the function 'massageSql' of 'QueryUtil.java',there are a few bugs:
> 01:
> {code:java}
> while (sql.endsWith(";"))
> sql = sql.substring(0, sql.length() - 1);
> {code}
> if the sql end with ';' and after the ';' still has comments,will be
> error.Because the sql will add ("\nLIMIT " + limit) at the end.
> 02:
> {code:java}
> if (limit > 0 && !sql.toLowerCase().contains("limit")) {
> sql += ("\nLIMIT " + limit);
> }
> if (offset > 0 && !sql.toLowerCase().contains("offset")) {
> sql += ("\nOFFSET " + offset);
> }
> {code}
> if the sql already has word 'limit' in it,such as Alias,Subquery,Comments,the
> ‘limit’ in the Input box will be invalid.
> for example,the sql has subquery,and there is 'limit' in subquery.
> {code:java}
> select KYLIN_SALES.PART_DT, count(KYLIN_SALES.PRICE)
> from KYLIN_SALES
> inner join (select ACCOUNT_ID, ACCOUNT_BUYER_LEVEL from KYLIN_ACCOUNT where
> ACCOUNT_COUNTRY = 'US' limit 1) as TT
> on KYLIN_SALES.BUYER_ID = TT.ACCOUNT_ID
> group by KYLIN_SALES.PART_DT
> {code}
> the ‘limit’ in the Input box will be invalid.please refer to 01.png and
> 02.png.
> 03:
> {code:java}
> // https://issues.apache.org/jira/browse/KYLIN-2649
> if (kylinConfig.getForceLimit() > 0 &&
> !sql.toLowerCase().contains("limit")
> && sql.toLowerCase().contains("*")) {
> sql += ("\nLIMIT " + kylinConfig.getForceLimit());
> }
> {code}
> Because KYLIN-2649 is still unresolved,so I didn't change the code,but it
> has same 'limit' word bugs like above.
> And there are some situations that the sql contains \*,such as
> {code:java}count(*),kylin_sales.*{code}
> Maybe the \* in subquery too.
> it seems improperly to deal with the sql just use {code:java}
> sql.toLowerCase().contains("*")
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3258) No check for duplicate cube name when creating a hybrid cube
[ https://issues.apache.org/jira/browse/KYLIN-3258?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu updated KYLIN-3258: - Fix Version/s: v2.3.1 > No check for duplicate cube name when creating a hybrid cube > > > Key: KYLIN-3258 > URL: https://issues.apache.org/jira/browse/KYLIN-3258 > Project: Kylin > Issue Type: Bug > Components: Metadata >Affects Versions: v2.2.0 > Environment: HDP 2.5.6, Kylin 2.2 >Reporter: Vsevolod Ostapenko >Priority: Minor > Fix For: v2.3.1 > > > When loading hybrid cube definitions via REST API, there is no check for > duplicate cube names is the list. If due to a user error or incorrectly > generated list of cubes by an external application/script the same cube name > is listed more than once, new or updated hybrid cube will contain the same > cube listed multiple times. > It does not seem to cause any immediate issues with querying, but it's just > not right. REST API should throw and exception, when the same cube name is > listed multiple times. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3267) add override MR config at project/cube level only for mem-hungry build steps
[
https://issues.apache.org/jira/browse/KYLIN-3267?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Billy Liu updated KYLIN-3267:
-
Issue Type: Improvement (was: Bug)
> add override MR config at project/cube level only for mem-hungry build steps
> -
>
> Key: KYLIN-3267
> URL: https://issues.apache.org/jira/browse/KYLIN-3267
> Project: Kylin
> Issue Type: Improvement
>Reporter: Yifei Wu
>Assignee: Yifei Wu
>Priority: Major
>
> Currently, users can override MR config at project/cube level via kylin
> properties {{kylin.engine.mr.config-override.}}
> However, this applies to all MR steps. It is more reasonable to add
> override-configs only for certain memory-hungry steps like in-mem cubing.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3270) Add integration test for Optimize Job
[ https://issues.apache.org/jira/browse/KYLIN-3270?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu updated KYLIN-3270: - Fix Version/s: v2.3.1 > Add integration test for Optimize Job > - > > Key: KYLIN-3270 > URL: https://issues.apache.org/jira/browse/KYLIN-3270 > Project: Kylin > Issue Type: Sub-task > Components: Tools, Build and Test >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Critical > Fix For: v2.3.1 > > Attachments: APACHE-KYLIN-3270.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3277) Kylin should override hiveconf settings when connecting to hive using jdbc
[ https://issues.apache.org/jira/browse/KYLIN-3277?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu updated KYLIN-3277: - Fix Version/s: v2.3.1 > Kylin should override hiveconf settings when connecting to hive using jdbc > -- > > Key: KYLIN-3277 > URL: https://issues.apache.org/jira/browse/KYLIN-3277 > Project: Kylin > Issue Type: Bug >Reporter: Chuqian Yu >Priority: Major > Labels: patch > Fix For: v2.3.1 > > Attachments: 0001-KYLIN-3277.patch > > > Hi, kylin developers. My cube building procedure failing at Step 2 > "Redistribute Flat Hive Table" because Kylin always trying to submit a mr job > to the default yarn queue. > > I have overrided the mapred.job.queue.name property in both > kylin_hive_conf.xml and kylin.properties but it doesn't work. > > kylin.properties > ``` > kylin.source.hive.beeline-params=-n hive -p hive --hiveconf > mapred.job.queue.name=myQueue -u > "jdbc:hive2://myZk:2181/;serviceDiscoveryMode=zooKeeper;" > ``` > > kylin_hive_conf.xml > ``` > > mapred.job.queue.name > myQueue > > ``` > > After digging into the source code ,I found that kylin try to get the row > count of hive table before redistributing it. But it dose not override the > hive configuration when using jdbc to connect to hive server. > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3276) query with dynamic parameter get the same query result
[ https://issues.apache.org/jira/browse/KYLIN-3276?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu updated KYLIN-3276: - Fix Version/s: v2.3.1 > query with dynamic parameter get the same query result > -- > > Key: KYLIN-3276 > URL: https://issues.apache.org/jira/browse/KYLIN-3276 > Project: Kylin > Issue Type: Bug > Components: Driver - JDBC, Query Engine >Affects Versions: v2.2.0 > Environment: Kylin 2.2.0, JDK 1.8 >Reporter: ZHAOXIXIN >Assignee: nichunen >Priority: Major > Fix For: v2.3.1 > > > sql:"select distinct uid from table where daytime=? limit 70" > ps.setString(1,"2018-01-03") > store the result to ArrayList, and print the size of ArrayList. > when I set the daytime to different dates, the same result I got. > I changed the sql, just change limit value for example > sql:"select distinct uid from table where daytime=? limit 60" > then I got the right value the same like query from GUI:count (distinct uid) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3285) "Value NNN not exists" error run executing query
[
https://issues.apache.org/jira/browse/KYLIN-3285?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Billy Liu updated KYLIN-3285:
-
Fix Version/s: v2.3.1
> "Value NNN not exists" error run executing query
> -
>
> Key: KYLIN-3285
> URL: https://issues.apache.org/jira/browse/KYLIN-3285
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v2.3.0
>Reporter: Shaofeng SHI
>Priority: Major
> Fix For: v2.3.1
>
> Attachments: cube (1).json, kylin.log
>
>
> Reported by community user zxxb...@163.com:
>
> {color:#b94a48}Kylin version update from 2.2.0 to 2.3.0; cube was build
> before update, and query without error.{color}
> {color:#b94a48}After updatge to version 2.3.0 and query {color}"select
> count(userid) num,day_time from record_ap group by day_time LIMIT
> 1000{color:#b94a48}”{color}
> {color:#b94a48}show errors below:{color}
> Column 0 value '2018-03-06' met dictionary error: Value '2018-03-06'
> (2018-03-06) not exists! while executing SQL: "select count(userid)
> num,day_time from record_ap group by day_time LIMIT 1000”
>
> {color:#b94a48}Where {color}{color:#b94a48}'2018-03-06’ come from? I was
> fully confused.{color}
> {color:#b94a48}Cube have several segments:{color}
> {color:#b94a48}Starttime-endtime{color}
> {color:#b94a48}20180101-20180225{color}
> 20180225-20180304
>
> Attached the log and cube JSON.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3280) The delete button should not be enabled without any segment in cube segment delete confirm dialog
[ https://issues.apache.org/jira/browse/KYLIN-3280?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu updated KYLIN-3280: - Fix Version/s: v2.3.1 > The delete button should not be enabled without any segment in cube segment > delete confirm dialog > - > > Key: KYLIN-3280 > URL: https://issues.apache.org/jira/browse/KYLIN-3280 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v2.4.0 >Reporter: Peng Xing >Assignee: Peng Xing >Priority: Minor > Fix For: v2.3.1 > > Attachments: delete_segment.png > > > 1. Open cube list, choose one disabled cube, then select 'Delete Segment'; > 2. The Delete button is enabled without any segment, and if you click it, it > has no prompt. > please see the attachment picture. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3288) "Sqoop To Flat Hive Table" step should specify "mapreduce.queue.name"
[ https://issues.apache.org/jira/browse/KYLIN-3288?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu updated KYLIN-3288: - Fix Version/s: v2.3.1 > "Sqoop To Flat Hive Table" step should specify "mapreduce.queue.name" > - > > Key: KYLIN-3288 > URL: https://issues.apache.org/jira/browse/KYLIN-3288 > Project: Kylin > Issue Type: Bug > Components: RDBMS Source >Reporter: Shaoxiong Zhan >Assignee: Shaoxiong Zhan >Priority: Major > Fix For: v2.3.1 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3287) When a shard by column is in dict encoding, dict building error.
[ https://issues.apache.org/jira/browse/KYLIN-3287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu updated KYLIN-3287: - Fix Version/s: v2.3.1 > When a shard by column is in dict encoding, dict building error. > > > Key: KYLIN-3287 > URL: https://issues.apache.org/jira/browse/KYLIN-3287 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.3.0 >Reporter: jiatao.tao >Assignee: jiatao.tao >Priority: Major > Fix For: v2.3.1 > > Attachments: image-2018-03-12-14-14-50-112.png, > image-2018-03-12-14-14-59-193.png, image-2018-03-12-14-15-42-046.png, > image-2018-03-12-14-16-38-657.png, image-2018-03-12-14-17-46-780.png > > > In dict encoding. > !image-2018-03-12-14-14-50-112.png|width=544,height=84! > !image-2018-03-12-14-14-59-193.png|width=535,height=631! > In int encoding > !image-2018-03-12-14-15-42-046.png|width=544,height=87! > !image-2018-03-12-14-16-38-657.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3287) When a shard by column is in dict encoding, dict building error.
[ https://issues.apache.org/jira/browse/KYLIN-3287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu updated KYLIN-3287: - Affects Version/s: v2.3.0 > When a shard by column is in dict encoding, dict building error. > > > Key: KYLIN-3287 > URL: https://issues.apache.org/jira/browse/KYLIN-3287 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.3.0 >Reporter: jiatao.tao >Assignee: jiatao.tao >Priority: Major > Attachments: image-2018-03-12-14-14-50-112.png, > image-2018-03-12-14-14-59-193.png, image-2018-03-12-14-15-42-046.png, > image-2018-03-12-14-16-38-657.png, image-2018-03-12-14-17-46-780.png > > > In dict encoding. > !image-2018-03-12-14-14-50-112.png|width=544,height=84! > !image-2018-03-12-14-14-59-193.png|width=535,height=631! > In int encoding > !image-2018-03-12-14-15-42-046.png|width=544,height=87! > !image-2018-03-12-14-16-38-657.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3195) Kylin v2.3.0 Release
[ https://issues.apache.org/jira/browse/KYLIN-3195?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu resolved KYLIN-3195. -- Resolution: Fixed > Kylin v2.3.0 Release > > > Key: KYLIN-3195 > URL: https://issues.apache.org/jira/browse/KYLIN-3195 > Project: Kylin > Issue Type: Task > Components: Others >Reporter: Billy Liu >Assignee: Billy Liu >Priority: Major > Fix For: v2.3.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-2884) Add delete segment function for portal
[ https://issues.apache.org/jira/browse/KYLIN-2884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu updated KYLIN-2884: - Fix Version/s: (was: v2.4.0) v2.3.0 > Add delete segment function for portal > -- > > Key: KYLIN-2884 > URL: https://issues.apache.org/jira/browse/KYLIN-2884 > Project: Kylin > Issue Type: Improvement > Components: Web >Reporter: Pan, Julian >Assignee: Pan, Julian >Priority: Major > Fix For: v2.3.0 > > Attachments: KYLIN-2884.patch, KYLIN-2884.patch, KYLIN-2884.patch, > KYLIN-2884.patch, KYLIN-2884.patch, screenshot-1.png, screenshot-2.png > > > Kylin restful already support delete segment function. Could we add a action > button for front-end to user? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3264) kylin.job.retry=1 cause build error
[ https://issues.apache.org/jira/browse/KYLIN-3264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu updated KYLIN-3264: - Fix Version/s: (was: v2.3.0) > kylin.job.retry=1 cause build error > --- > > Key: KYLIN-3264 > URL: https://issues.apache.org/jira/browse/KYLIN-3264 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.3.0 >Reporter: readme_kylin >Priority: Major > > when i set kylin.job.retry to 1,it cause build error at 6 step :Create HTable > . > here is the error info. > java.lang.RuntimeException: HBase table KYLIN230_CFJ854DYZ4 exists! > at > org.apache.kylin.storage.hbase.steps.CubeHTableUtil.createHTable(CubeHTableUtil.java:100) > at > org.apache.kylin.storage.hbase.steps.CreateHTableJob.run(CreateHTableJob.java:111) > at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:97) > at > org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:162) > at > org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:67) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:162) > at > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:300) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3268) Tomcat Security Vulnerability Alert. The version of the tomcat for kylin should upgrade to 7.0.85.
[ https://issues.apache.org/jira/browse/KYLIN-3268?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu resolved KYLIN-3268. -- Resolution: Fixed Fix Version/s: v2.3.0 > Tomcat Security Vulnerability Alert. The version of the tomcat for kylin > should upgrade to 7.0.85. > -- > > Key: KYLIN-3268 > URL: https://issues.apache.org/jira/browse/KYLIN-3268 > Project: Kylin > Issue Type: Bug > Components: Security >Reporter: peng.jianhua >Assignee: peng.jianhua >Priority: Major > Labels: patch > Fix For: v2.3.0 > > Attachments: > 0001-KYLIN-3268-Tomcat-Security-Vulnerability-Alert.-The-.patch > > > [SECURITY] CVE-2018-1305 Security constraint annotations applied too late > CVE-2018-1305 Security constraint annotations applied too late > Severity: High > Versions Affected: Apache Tomcat 9.0.0.M1 to 9.0.4 Apache Tomcat 8.5.0 to > 8.5.27 Apache Tomcat 8.0.0.RC1 to 8.0.49 Apache Tomcat 7.0.0 to 7.0.84 > Description: Security constraints defined by annotations of Servlets were > only applied once a Servlet had been loaded. Because security constraints > defined in this way apply to the URL pattern and any URLs below that point, > it was possible - depending on the order Servlets were loaded - for some > security constraints not to be applied. This could have exposed resources to > users who were not authorised to access them. > Mitigation: Users of the affected versions should apply one of the following > mitigations. Upgrade to: - Apache Tomcat 9.0.5 or later - Apache Tomcat > 8.5.28 or later - Apache Tomcat 8.0.50 or later - Apache Tomcat 7.0.85 or > later > References:https://lists.apache.org/thread.html/d3354bb0a4eda4acc0a66f3eb24a213fdb75d12c7d16060b23e65781@%3Cannounce.tomcat.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3268) Tomcat Security Vulnerability Alert. The version of the tomcat for kylin should upgrade to 7.0.85.
[ https://issues.apache.org/jira/browse/KYLIN-3268?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16379641#comment-16379641 ] Billy Liu commented on KYLIN-3268: -- All security vulnerabilities should be fixed with high priority. Let's do it. > Tomcat Security Vulnerability Alert. The version of the tomcat for kylin > should upgrade to 7.0.85. > -- > > Key: KYLIN-3268 > URL: https://issues.apache.org/jira/browse/KYLIN-3268 > Project: Kylin > Issue Type: Bug > Components: Security >Reporter: peng.jianhua >Assignee: peng.jianhua >Priority: Major > Labels: patch > > [SECURITY] CVE-2018-1305 Security constraint annotations applied too late > CVE-2018-1305 Security constraint annotations applied too late > Severity: High > Versions Affected: Apache Tomcat 9.0.0.M1 to 9.0.4 Apache Tomcat 8.5.0 to > 8.5.27 Apache Tomcat 8.0.0.RC1 to 8.0.49 Apache Tomcat 7.0.0 to 7.0.84 > Description: Security constraints defined by annotations of Servlets were > only applied once a Servlet had been loaded. Because security constraints > defined in this way apply to the URL pattern and any URLs below that point, > it was possible - depending on the order Servlets were loaded - for some > security constraints not to be applied. This could have exposed resources to > users who were not authorised to access them. > Mitigation: Users of the affected versions should apply one of the following > mitigations. Upgrade to: - Apache Tomcat 9.0.5 or later - Apache Tomcat > 8.5.28 or later - Apache Tomcat 8.0.50 or later - Apache Tomcat 7.0.85 or > later > References:https://lists.apache.org/thread.html/d3354bb0a4eda4acc0a66f3eb24a213fdb75d12c7d16060b23e65781@%3Cannounce.tomcat.apache.org%3E -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3264) kylin.job.retry=1 cause build error
[ https://issues.apache.org/jira/browse/KYLIN-3264?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu resolved KYLIN-3264. -- Resolution: Duplicate Fix Version/s: v2.3.0 > kylin.job.retry=1 cause build error > --- > > Key: KYLIN-3264 > URL: https://issues.apache.org/jira/browse/KYLIN-3264 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.3.0 >Reporter: readme_kylin >Priority: Major > Fix For: v2.3.0 > > > when i set kylin.job.retry to 1,it cause build error at 6 step :Create HTable > . > here is the error info. > java.lang.RuntimeException: HBase table KYLIN230_CFJ854DYZ4 exists! > at > org.apache.kylin.storage.hbase.steps.CubeHTableUtil.createHTable(CubeHTableUtil.java:100) > at > org.apache.kylin.storage.hbase.steps.CreateHTableJob.run(CreateHTableJob.java:111) > at org.apache.kylin.engine.mr.MRUtil.runMRJob(MRUtil.java:97) > at > org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:63) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:162) > at > org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:67) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:162) > at > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:300) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3263) AbstractExecutable's retry has problem
[ https://issues.apache.org/jira/browse/KYLIN-3263?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu resolved KYLIN-3263. -- Resolution: Fixed Fix Version/s: v2.3.0 > AbstractExecutable's retry has problem > -- > > Key: KYLIN-3263 > URL: https://issues.apache.org/jira/browse/KYLIN-3263 > Project: Kylin > Issue Type: Bug >Reporter: jiatao.tao >Assignee: jiatao.tao >Priority: Major > Fix For: v2.3.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3258) No check for duplicate cube name when creating a hybrid cube
[ https://issues.apache.org/jira/browse/KYLIN-3258?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16368219#comment-16368219 ] Billy Liu commented on KYLIN-3258: -- Nice found. > No check for duplicate cube name when creating a hybrid cube > > > Key: KYLIN-3258 > URL: https://issues.apache.org/jira/browse/KYLIN-3258 > Project: Kylin > Issue Type: Bug > Components: Metadata >Affects Versions: v2.2.0 > Environment: HDP 2.5.6, Kylin 2.2 >Reporter: Vsevolod Ostapenko >Priority: Minor > > When loading hybrid cube definitions via REST API, there is no check for > duplicate cube names is the list. If due to a user error or incorrectly > generated list of cubes by an external application/script the same cube name > is listed more than once, new or updated hybrid cube will contain the same > cube listed multiple times. > It does not seem to cause any immediate issues with querying, but it's just > not right. REST API should throw and exception, when the same cube name is > listed multiple times. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3260) Error while building cubes
[
https://issues.apache.org/jira/browse/KYLIN-3260?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16368218#comment-16368218
]
Billy Liu commented on KYLIN-3260:
--
Hi [~manuvilakunna...@gmail.com], could you ask you question in the kylin user
mailer? You could get many clues when googling "NoSuchMethodError Kylin".
> Error while building cubes
> --
>
> Key: KYLIN-3260
> URL: https://issues.apache.org/jira/browse/KYLIN-3260
> Project: Kylin
> Issue Type: Bug
> Components: Metadata, Storage - HBase
>Affects Versions: v2.2.0
> Environment: Apache Kylin-2.2.0
> Hadoop-2.6.5
> Hbase-1.1.13
> Hive-2.1.0
> Zookeeper-3.4.10
>Reporter: Manu
>Priority: Blocker
> Labels: Kylin
> Fix For: v2.2.0
>
>
> While building cubes getting this error in {color:#4383b4}#6 Step Name:
> {color}Create HTable
>
> java.lang.NoSuchMethodError:
> org.apache.hadoop.hbase.HTableDescriptor.setValue(Ljava/lang/String;Ljava/lang/String;)V
> at
> org.apache.kylin.storage.hbase.steps.CubeHTableUtil.createHTable(CubeHTableUtil.java:63)
> at
> org.apache.kylin.storage.hbase.steps.CreateHTableJob.run(CreateHTableJob.java:104)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
> at
> org.apache.kylin.engine.mr.common.HadoopShellExecutable.doWork(HadoopShellExecutable.java:62)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125)
> at
> org.apache.kylin.job.execution.DefaultChainedExecutable.doWork(DefaultChainedExecutable.java:64)
> at
> org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:125)
> at
> org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:144)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3259) When a cube is deleted, remove it from the hybrid cube definition
[ https://issues.apache.org/jira/browse/KYLIN-3259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16368215#comment-16368215 ] Billy Liu commented on KYLIN-3259: -- Nice found. > When a cube is deleted, remove it from the hybrid cube definition > - > > Key: KYLIN-3259 > URL: https://issues.apache.org/jira/browse/KYLIN-3259 > Project: Kylin > Issue Type: Improvement > Components: Metadata >Affects Versions: v2.2.0 > Environment: HDP 2.5.6, Kylin 2.2 >Reporter: Vsevolod Ostapenko >Priority: Major > > When a cube is deleted, its references are not automatically removed from > existing hybrid cube definition. That can lead to errors down the road, if > user (or application) retrieves the list of cubes via REST API call and later > tries to update the hybrid cube. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-2884) Add delete segment function for portal
[ https://issues.apache.org/jira/browse/KYLIN-2884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360398#comment-16360398 ] Billy Liu commented on KYLIN-2884: -- Hello [~julianpan], I found one issue here. If the cube detail page is not expanded, click the "Delete Segment" will show "No partition date column defined" !screenshot-1.png! If the cube detail page is expanded, the "Delete Confirm" will pop up. !screenshot-2.png! I could reproduce this issue on the sample cube. > Add delete segment function for portal > -- > > Key: KYLIN-2884 > URL: https://issues.apache.org/jira/browse/KYLIN-2884 > Project: Kylin > Issue Type: Improvement > Components: Web >Reporter: Pan, Julian >Assignee: Pan, Julian >Priority: Major > Fix For: v2.3.0 > > Attachments: KYLIN-2884.patch, KYLIN-2884.patch, KYLIN-2884.patch, > screenshot-1.png, screenshot-2.png > > > Kylin restful already support delete segment function. Could we add a action > button for front-end to user? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-2884) Add delete segment function for portal
[ https://issues.apache.org/jira/browse/KYLIN-2884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu updated KYLIN-2884: - Attachment: screenshot-2.png > Add delete segment function for portal > -- > > Key: KYLIN-2884 > URL: https://issues.apache.org/jira/browse/KYLIN-2884 > Project: Kylin > Issue Type: Improvement > Components: Web >Reporter: Pan, Julian >Assignee: Pan, Julian >Priority: Major > Fix For: v2.3.0 > > Attachments: KYLIN-2884.patch, KYLIN-2884.patch, KYLIN-2884.patch, > screenshot-1.png, screenshot-2.png > > > Kylin restful already support delete segment function. Could we add a action > button for front-end to user? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-2884) Add delete segment function for portal
[ https://issues.apache.org/jira/browse/KYLIN-2884?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu updated KYLIN-2884: - Attachment: screenshot-1.png > Add delete segment function for portal > -- > > Key: KYLIN-2884 > URL: https://issues.apache.org/jira/browse/KYLIN-2884 > Project: Kylin > Issue Type: Improvement > Components: Web >Reporter: Pan, Julian >Assignee: Pan, Julian >Priority: Major > Fix For: v2.3.0 > > Attachments: KYLIN-2884.patch, KYLIN-2884.patch, KYLIN-2884.patch, > screenshot-1.png > > > Kylin restful already support delete segment function. Could we add a action > button for front-end to user? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3252) 当hbase使用的文件系统和MapReduce不一样时，使用清理存储工具时，无法清理hbase所在文件系统的数据文件
[ https://issues.apache.org/jira/browse/KYLIN-3252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16360387#comment-16360387 ] Billy Liu commented on KYLIN-3252: -- I think it's duplicate with https://issues.apache.org/jira/browse/KYLIN-3016 > 当hbase使用的文件系统和MapReduce不一样时，使用清理存储工具时，无法清理hbase所在文件系统的数据文件 > -- > > Key: KYLIN-3252 > URL: https://issues.apache.org/jira/browse/KYLIN-3252 > Project: Kylin > Issue Type: Bug > Components: Tools, Build and Test >Affects Versions: v2.0.0 >Reporter: RenZhiMin >Priority: Minor > Labels: patch > > 大家好，我使用的kylin版本是2.0，Hbase和MapReduce使用的文件系统不是同一个集群，当使用kylin清理存储工具时无法清理Hbase所在的文件系统上的数据 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3197) When ldap is opened, I use an ignored case user to login, the page does not respond.
[
https://issues.apache.org/jira/browse/KYLIN-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Billy Liu updated KYLIN-3197:
-
Fix Version/s: (was: v2.3.0)
Future
> When ldap is opened, I use an ignored case user to login, the page does not
> respond.
>
>
> Key: KYLIN-3197
> URL: https://issues.apache.org/jira/browse/KYLIN-3197
> Project: Kylin
> Issue Type: Bug
> Components: Security
>Affects Versions: v2.3.0
>Reporter: Peng Xing
>Assignee: Peng Xing
>Priority: Major
> Labels: patch
> Fix For: Future
>
> Attachments:
> 0001-KYLIN-3197-When-ldap-is-opened-I-use-an-ignored-case.patch,
> image-2018-01-25-17-22-39-970.png, image-2018-02-06-14-09-32-591.png,
> image-2018-02-08-15-32-25-030.png, image-2018-02-08-15-33-07-277.png,
> image-2018-02-08-15-33-54-480.png, image-2018-02-08-15-35-03-902.png
>
>
> When ldap is opened, I config the kylin.properties, and give wkhGroup the
> admin permission.
> {code:java}
> ## Admin roles in LDAP, for ldap and saml
> kylin.security.acl.admin-role=wkhGroup
> {code}
> then I create a new user named 'wkh' whose group is 'wkhGroup', then I use
> '{color:#ff}wkh{color}' to login in, which is normal.
> But when I use '{color:#ff}WKH{color}' to login in, the page does not
> respond.
> I analyze the backgroud code, and find the function of
> 'org.apache.kylin.rest.security.LDAPAuthoritiesPopulator.getGroupMembershipRoles(String,
> String)' has problem.
> When userDn is
> "uid={color:#ff}wkh{color},ou=People,ou=defaultCluster,dc=zdh,dc=com" and
> username is "{color:#ff}WKH{color}", then authorities will be empty Set
> by the follow code:
> {code:java}
> Set authorities = super.getGroupMembershipRoles(userDn,
> username);
> {code}
> So I have added 'getAdditionalRoles' function to get the authorities again.
> I have test the patch, please review, thanks!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3224) data can't show when use kylin pushdown model
[ https://issues.apache.org/jira/browse/KYLIN-3224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16359843#comment-16359843 ] Billy Liu commented on KYLIN-3224: -- I see. I also did some investigation on this. Different data source has different meta data format, there should be a better way to show the data without modifying the third-party guid.js. > data can't show when use kylin pushdown model > -- > > Key: KYLIN-3224 > URL: https://issues.apache.org/jira/browse/KYLIN-3224 > Project: Kylin > Issue Type: Bug > Components: Query Engine, Web >Affects Versions: v2.2.0, v2.3.0 >Reporter: peng.jianhua >Assignee: peng.jianhua >Priority: Major > Fix For: v2.3.0 > > Attachments: 0001-KYLIN-3224.patch, 01.PNG, 02.PNG, 03.PNG, 04.PNG > > > select * from kylin_sales > use pushdown model,and the result shows like 01.png -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3224) data can't show when use kylin pushdown model
[ https://issues.apache.org/jira/browse/KYLIN-3224?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16359717#comment-16359717 ] Billy Liu commented on KYLIN-3224: -- Agree with [~Zhixiong Chen], we should figure out why the return format is different between cube and pushdown. ui-grid.js is the third party library. We'd better keep it no change. > data can't show when use kylin pushdown model > -- > > Key: KYLIN-3224 > URL: https://issues.apache.org/jira/browse/KYLIN-3224 > Project: Kylin > Issue Type: Bug > Components: Query Engine, Web >Affects Versions: v2.2.0, v2.3.0 >Reporter: peng.jianhua >Assignee: peng.jianhua >Priority: Major > Fix For: v2.3.0 > > Attachments: 0001-KYLIN-3224.patch, 01.PNG, 02.PNG, 03.PNG > > > select * from kylin_sales > use pushdown model,and the result shows like 01.png -- This message was sent by Atlassian JIRA (v7.6.3#76005)
