[jira] [Updated] (KYLIN-5003) Fail to package Kylin due to legacy front end dependencies
[
https://issues.apache.org/jira/browse/KYLIN-5003?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen updated KYLIN-5003:
Summary: Fail to package Kylin due to legacy front end dependencies (was:
Fail to package Kylin due to legancy front end dependencies)
> Fail to package Kylin due to legacy front end dependencies
> --
>
> Key: KYLIN-5003
> URL: https://issues.apache.org/jira/browse/KYLIN-5003
> Project: Kylin
> Issue Type: Bug
> Components: Web
>Affects Versions: v3.1.2
>Reporter: nichunen
>Assignee: nichunen
>Priority: Major
> Fix For: v3.1.3
>
>
> I tried to package Kylin with build/script/package.sh, and it failed with
> error
> {quote}bower angular-unstable#1.1.5 ECMDERR Failed
> to execute "git ls-remote --tags --heads
> https://github.com/johannestroeger/bower-angular-unstable.git;, exit code of
> #128 remote: Repository not found. fatal: repository
> 'https://github.com/johannestroeger/bower-angular-unstable.git/' not
> found{quote}
> And the dependency "phantomjs" can not be installed with "npm install
> phantomjs" now.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KYLIN-5003) Fail to package Kylin due to legancy front end dependencies
nichunen created KYLIN-5003:
---
Summary: Fail to package Kylin due to legancy front end
dependencies
Key: KYLIN-5003
URL: https://issues.apache.org/jira/browse/KYLIN-5003
Project: Kylin
Issue Type: Bug
Components: Web
Affects Versions: v3.1.2
Reporter: nichunen
Assignee: nichunen
Fix For: v3.1.3
I tried to package Kylin with build/script/package.sh, and it failed with error
{quote}bower angular-unstable#1.1.5 ECMDERR Failed to
execute "git ls-remote --tags --heads
https://github.com/johannestroeger/bower-angular-unstable.git;, exit code of
#128 remote: Repository not found. fatal: repository
'https://github.com/johannestroeger/bower-angular-unstable.git/' not
found{quote}
And the dependency "phantomjs" can not be installed with "npm install
phantomjs" now.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-4481) Project-level ACL lookups not working for non-admin SAML-federated users
[
https://issues.apache.org/jira/browse/KYLIN-4481?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17168993#comment-17168993
]
nichunen commented on KYLIN-4481:
-
This change is merged by me, and now I agree with Yanghong, I suggest reverting
the change, and fix the issue with a more reasonable way.
> Project-level ACL lookups not working for non-admin SAML-federated users
>
>
> Key: KYLIN-4481
> URL: https://issues.apache.org/jira/browse/KYLIN-4481
> Project: Kylin
> Issue Type: Bug
> Components: Security
>Affects Versions: v2.6.5, v3.0.1
>Reporter: Rafael Felix Correa
>Assignee: Guangxu Cheng
>Priority: Major
> Fix For: v3.1.0, v3.0.2, v2.6.6
>
>
> Steps to reproduce:
> * setup kylin with SAML as described in
> [http://kylin.apache.org/docs/howto/howto_ldap_and_sso.html].
> kylin.properties:
> {code:java}
> kylin.security.profile=saml
> kylin.security.acl.admin-role=Kylin_Admins
> kylin.security.ldap.connection-server=ldap://openldap:389
> kylin.security.ldap.connection-username=cn=admin,dc=example,dc=org
> # set kylin.security.ldap.connection-password appropriately
> kylin.security.ldap.user-search-base=ou=people,dc=example,dc=org
> kylin.security.ldap.user-search-pattern=(uid={0})
> kylin.security.ldap.user-group-search-base=ou=groups,dc=example,dc=org
> kylin.security.saml.context-context-path=/kylin
> kylin.security.saml.context-scheme=https
> kylin.security.saml.context-server-name=kylin.validdomain.com
> kylin.security.saml.context-server-port=443
> kylin.security.saml.metadata-entity-base-url=https://kylin.validdomain.com/kylin{code}
> * on the LDAP server, make sure you have the following objects in place:
> {code:java}
> # example.user, people, example.org
> dn: uid=example.user,ou=people,dc=example,dc=org
> objectClass: top
> objectClass: account
> objectClass: posixAccount
> objectClass: shadowAccount
> gidNumber: 1
> uidNumber: 5000
> cn: Does not matter
> homeDirectory: /home/doesntmatter
> uid: example.user{code}
> *
> {code:java}
> # Kylin_Users, groups, example.org
> dn: cn=Kylin_Users,ou=groups,dc=example,dc=org
> objectClass: top
> objectClass: groupOfNames
> cn: Kylin_Users
> member: uid=example.user,ou=people,dc=example,dc=org{code}
> * as an ADMIN, create a sample project in kylin and grant QUERY, MANAGEMENT
> or OPERATION access to example.user.
> * now, try logging into kylin.validdomain.com's Web UI as
> [example.u...@validdomain.com.|mailto:example.u...@validdomain.com.]
> Expected result:
> * example.user is logged in, able to select the project from the dropdown
> box at the top left corner and navigate through its properties.
> Actual result:
> * example.user is logged in, but no projects are listed in the dropdown box.
> As if he/she had no permissions in any project.
>
> With LDAP-pure installations (no SAML), this configuration works as expected.
>
> Worth noting:
> [https://github.com/apache/kylin/blob/kylin-3.0.1/server-base/src/main/java/org/apache/kylin/rest/security/SAMLUserDetailsService.java#L40-L54]
> splits the user in the '@' char for performing LDAP lookups. However, by
> editing kylin_metadata manually and appending the @validdomain.com to the
> corresponding object under /acls, the lookup works as it should and the
> non-admin user gets to access the sample project.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KYLIN-4477) Usage of "TLS" is insecure
[
https://issues.apache.org/jira/browse/KYLIN-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen updated KYLIN-4477:
Sprint: Sprint 53 (was: Sprint 52)
> Usage of "TLS" is insecure
> --
>
> Key: KYLIN-4477
> URL: https://issues.apache.org/jira/browse/KYLIN-4477
> Project: Kylin
> Issue Type: Improvement
>Reporter: Md Mahir Asef Kabir
>Assignee: Md Mahir Asef Kabir
>Priority: Major
> Fix For: v3.1.0
>
>
> *Vulnerability Description:* In
> “engine-mr/src/main/java/org/apache/kylin/engine/mr/common/DefaultSslProtocolSocketFactory.java”
> file the following code was written in
> {code:java}
> private static SSLContext createEasySSLContext()
> {code}
> method -
> {code:java}
> SSLContext context = SSLContext.getInstance("TLS");
> {code}
> The vulnerability is, using "TLS” as the argument to SSLContext.getInstance
> method.
> *Reason it’s vulnerable:* TLS 1.0 is vulnerable to man-in-the-middle attacks.
> For further reference, follow
> [this|https://www.comodo.com/e-commerce/ssl-certificates/tls-1-deprecation.php].
> *Suggested Fix:* Using
> {code:java}
> SSLContext.getInstance("TLSv1.3").
> {code}
> *Feedback:* Please select any of the options down below to help us get an
> idea about how you felt about the suggestion -
> # Liked it and will make the suggested changes
> # Liked it but happy with the existing version
> # Didn’t find the suggestion helpful
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KYLIN-4478) Usage of "AES/ECB/PKCS5Padding" is insecure
[
https://issues.apache.org/jira/browse/KYLIN-4478?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen updated KYLIN-4478:
Sprint: Sprint 53 (was: Sprint 52)
> Usage of "AES/ECB/PKCS5Padding" is insecure
> ---
>
> Key: KYLIN-4478
> URL: https://issues.apache.org/jira/browse/KYLIN-4478
> Project: Kylin
> Issue Type: Improvement
>Reporter: Md Mahir Asef Kabir
>Assignee: Md Mahir Asef Kabir
>Priority: Major
> Fix For: v3.1.0
>
>
> *Vulnerability Description:* In
> “core-common/src/main/java/org/apache/kylin/common/util/EncryptUtil.java”
> file the following code was written in public static String encrypt(String
> strToEncrypt) method -
> {code:java}
> Cipher cipher = Cipher.getInstance("AES/ECB/PKCS5Padding");
> {code}
> The vulnerability is, using "AES/ECB/PKCS5Padding” as the argument to
> Cipher.getInstance method.
> *Reason it’s vulnerable:* ”AES/ECB/PKCS5Padding” is not secure. For further
> reference, follow [this|https://zachgrace.com/posts/attacking-ecb/].
> *Suggested Fix:* Using
> {code:java}
> Cipher cipher = Cipher.getInstance("AES/CFB/PKCS5Padding");
> {code}
> *Feedback:* Please select any of the options down below to help us get an
> idea about how you felt about the suggestion -
> # Liked it and will make the suggested changes
> # Liked it but happy with the existing version
> # Didn’t find the suggestion helpful
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-4441) Add restart command to kylin.sh
[ https://issues.apache.org/jira/browse/KYLIN-4441?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126307#comment-17126307 ] nichunen commented on KYLIN-4441: - Verified. > Add restart command to kylin.sh > --- > > Key: KYLIN-4441 > URL: https://issues.apache.org/jira/browse/KYLIN-4441 > Project: Kylin > Issue Type: Improvement > Components: Client - CLI >Affects Versions: all >Reporter: Yaqian Zhang >Assignee: Yaqian Zhang >Priority: Minor > Fix For: v3.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4377) Project's admin cannot operate Hybrid model
[ https://issues.apache.org/jira/browse/KYLIN-4377?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126274#comment-17126274 ] nichunen commented on KYLIN-4377: - Verified. > Project's admin cannot operate Hybrid model > --- > > Key: KYLIN-4377 > URL: https://issues.apache.org/jira/browse/KYLIN-4377 > Project: Kylin > Issue Type: Bug >Reporter: ZhouKang >Assignee: ZhouKang >Priority: Major > Fix For: v3.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4312) Specified cube when querying by API
[ https://issues.apache.org/jira/browse/KYLIN-4312?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126270#comment-17126270 ] nichunen commented on KYLIN-4312: - Verified. > Specified cube when querying by API > --- > > Key: KYLIN-4312 > URL: https://issues.apache.org/jira/browse/KYLIN-4312 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Reporter: langdamao >Assignee: Xiaoxiang Yu >Priority: Major > Fix For: v3.1.0 > > > Support specified cube when using query API. > UseCase1: > For CubeA has dim a,b,c,d ,segments 12.01-now，CubeB has dim > b,c,e，segments 12.08-now， with same hive table and same measure. While > querying "select b,c,count(*) from db.table group by b,c; " , will hit CubeB, > so that result of 12.01-12.08 will disappear. > UseCase2: > For testing,build many cubes with different config, will need to > switch disable|enable all the time. > > See discussion here: > http://mail-archives.apache.org/mod_mbox/kylin-user/202001.mbox/%3c976a402f-3d7e-4b99-b87e-d6b39ff2f...@yidian-inc.com%3e -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4252) Fix the error "Cannot read property 'index' of null" in visualization page
[
https://issues.apache.org/jira/browse/KYLIN-4252?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126269#comment-17126269
]
nichunen commented on KYLIN-4252:
-
Verified.
> Fix the error "Cannot read property 'index' of null" in visualization page
> --
>
> Key: KYLIN-4252
> URL: https://issues.apache.org/jira/browse/KYLIN-4252
> Project: Kylin
> Issue Type: Bug
> Components: Web
>Affects Versions: v3.0.0
>Reporter: wu.kehua
>Assignee: wu.kehua
>Priority: Major
> Fix For: v2.6.5, v3.1.0, v3.0.1
>
> Attachments: image-2019-12-18-10-07-19-107.png
>
>
> # Execute "select PART_DT,sum(PRICE) from KYLIN_SALES group by PART_DT"
> # Expand "Query String"
> # Then click "visualization" to show the chart
> # If we choose the "-- choose metrics --" firstly, then we reopen Metrics
> drop-down box, then we can't see full metric options.
> !image-2019-12-18-10-07-19-107.png!
>
> Then I analyse this operation will cause the error "Cannot read property
> 'index' of null" in visualization page, as follows:
> {code:java}
> message: "Cannot read property 'index' of null"
> stack: "TypeError: Cannot read property 'index' of null?at
> http://kylin_server:7070/kylin/js/scripts.min.20191108164741.js:55197:69?
> at Object.forEach
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-4212) Add a user profile page
[ https://issues.apache.org/jira/browse/KYLIN-4212?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126268#comment-17126268 ] nichunen commented on KYLIN-4212: - Verified. > Add a user profile page > --- > > Key: KYLIN-4212 > URL: https://issues.apache.org/jira/browse/KYLIN-4212 > Project: Kylin > Issue Type: Improvement >Reporter: Congling Xia >Assignee: Congling Xia >Priority: Major > Fix For: v3.1.0 > > Attachments: change_pass.png, entrance.png, profile.png > > > Kylin's user/group API has been extended since KYLIN-4122. User can get more > account details and change password by requesting the APIs. But there is no > webpage for a normal user to view such information or do such changes. > In this improvement, the following changes are desired: > # provide a user profile page to show his/her basic information, > # set a button on it to launch the change password dialog for the user -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4211) PartitionDesc support custom year、month、day partitions name
[
https://issues.apache.org/jira/browse/KYLIN-4211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126266#comment-17126266
]
nichunen commented on KYLIN-4211:
-
Verified.
> PartitionDesc support custom year、month、day partitions name
> ---
>
> Key: KYLIN-4211
> URL: https://issues.apache.org/jira/browse/KYLIN-4211
> Project: Kylin
> Issue Type: Improvement
>Reporter: chuxiao
>Assignee: chuxiao
>Priority: Major
> Fix For: v3.1.0
>
> Attachments: KYLIN-4211.master.001.patch
>
>
> YearMonthDayPartitionConditionBuilder supports partition named year, month,
> day, but can not support partition names like Y, M, D.
> Because there are user using fact table partitioned by Y, M, D, so add
> CustomYearMonthDayFieldPartitionConditionBuilder, support custom year, month,
> day partitions name.
> Partition metadata in model.json like:
> {
> "uuid" : "459d48c1-a8a6-cdf5-6ea7-e2ae48b248e9",
> "last_modified" : 1571652918478,
> "version" : "2.6.0.20500",
> "name" : "kylin_sales_ymd",
> "owner" : "admin",
> "is_draft" : false,
> "description" : "",
> "fact_table" : "BIGDATA_KYLIN.KYLIN_SALES_YMD",
> "lookups" : [ ],
> "dimensions" : [ {
> "table" : "KYLIN_SALES_YMD",
> "columns" : [ "LEAF_CATEG_ID", "TRANS_ID", "SLR_SEGMENT_CD", "SELLER_ID",
> "BUYER_ID", "OPS_USER_ID", "OPS_REGION", "Y", "M", "D" ]
> } ],
> "metrics" : [ "KYLIN_SALES_YMD.PRICE", "KYLIN_SALES_YMD.ITEM_COUNT" ],
> "filter_condition" : "",
> "partition_desc" : {
> "partition_date_column" : "KYLIN_SALES_YMD.Y, KYLIN_SALES_YMD.M,
> KYLIN_SALES_YMD.D",
> "partition_time_column" : null,
> "partition_date_start" : 0,
> "partition_date_format" : "-MM-dd",
> "partition_time_format" : "HH:mm:ss",
> "partition_type" : "APPEND",
> "partition_condition_builder" :
> "org.apache.kylin.metadata.model.PartitionDesc$CustomYearMonthDayFieldPartitionConditionBuilder"
> },
> "capacity" : "MEDIUM"
> }
> Partition_date_column is the year, month, and day partition field split by
> ','.， and Partition_condition_builder is
> org.apache.kylin.metadata.model.PartitionDesc$CustomYearMonthDayFieldPartitionConditionBuilder
> Web can create a normal model and "Edit JSON" on Kylin web
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-3698) check-env.sh should print more details about checking items
[ https://issues.apache.org/jira/browse/KYLIN-3698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126264#comment-17126264 ] nichunen commented on KYLIN-3698: - Verified. > check-env.sh should print more details about checking items > --- > > Key: KYLIN-3698 > URL: https://issues.apache.org/jira/browse/KYLIN-3698 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Affects Versions: v2.5.1 >Reporter: May Zhou >Assignee: Yaqian Zhang >Priority: Minor > Fix For: v3.1.0 > > Original Estimate: 8h > Remaining Estimate: 8h > > In the current version, when users run _check-env.sh_, if there's no error > message, it means everything is OK. > From my perspective, adding more details about the checking items when > executing check-env.sh is better. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4197) DiagnosisInfoCLI block forever at "beeline --version"
[
https://issues.apache.org/jira/browse/KYLIN-4197?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126265#comment-17126265
]
nichunen commented on KYLIN-4197:
-
Verified.
> DiagnosisInfoCLI block forever at "beeline --version"
> -
>
> Key: KYLIN-4197
> URL: https://issues.apache.org/jira/browse/KYLIN-4197
> Project: Kylin
> Issue Type: Improvement
>Reporter: Xiaoxiang Yu
>Assignee: Xiaoxiang Yu
>Priority: Minor
> Fix For: v3.1.0, v3.0.1
>
>
> When using "sh bin/diag.sh xxx xxx" in CDH 5.7, it will blocked at "beeline
> --version" in HDP2.4 .
> Executing "beeline --version" will enter interactive shell in some hadoop
> distribution.
> h3. jstack 26111
> {code:java}
> "main" #1 prio=5 os_prio=0 tid=0x7fae18029800 nid=0x6672 waiting on
> condition [0x7fae1cefe000]
>java.lang.Thread.State: TIMED_WAITING (parking)
> at sun.misc.Unsafe.park(Native Method)
> - parking to wait for <0xf4d89348> (a
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at
> java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:215)
> at
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2078)
> at
> java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1475)
> at
> org.apache.kylin.tool.DiagnosisInfoCLI.executeExtract(DiagnosisInfoCLI.java:266)
> at
> org.apache.kylin.tool.AbstractInfoExtractor.execute(AbstractInfoExtractor.java:117)
> at
> org.apache.kylin.common.util.AbstractApplication.execute(AbstractApplication.java:37)
> at
> org.apache.kylin.tool.DiagnosisInfoCLI.main(DiagnosisInfoCLI.java:102)
> {code}
> h3. jps -m
> {code:java}
> [root@cdh-client ~]# jps -m
> 4982 RunJar
> /root/wangrupeng/kylin-spark/KYLIN-4187-hbase/bin/../tomcat/bin/bootstrap.jar
> org.apache.catalina.startup.Bootstrap start
> 6823 ConsoleProducer --topic useraction_3 --broker-list
> cdh-master:9092,cdh-worker-1:9092,cdh-worker-2:9092
> 32408 Bootstrap start
> 26955 BeeLine --version
> 28796 Jps -m
> 26111 DiagnosisInfoCLI -project -all -destDir haha1
> {code}
> h3. std out
> {code:java}
> afb132f0e6f1/recovered.edits/2.seqid
> SLF4J: Class path contains multiple SLF4J bindings.
> SLF4J: Found binding in
> [jar:file:/root/xiaoxiang.yu/kylin-dist/apache-kylin-3.0.0-SNAPSHOT-bin-4081-master/tool/kylin-tool-3.0.0-SNAPSHOT.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: Found binding in
> [jar:file:/opt/cloudera/parcels/CDH-5.7.6-1.cdh5.7.6.p0.6/jars/slf4j-log4j12-1.7.5.jar!/org/slf4j/impl/StaticLoggerBinder.class]
> SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an
> explanation.
> SLF4J: Actual binding is of type [org.slf4j.impl.Log4jLoggerFactory]
> Java HotSpot(TM) 64-Bit Server VM warning: ignoring option MaxPermSize=512M;
> support was removed in 8.0
> Hive 1.1.0-cdh5.7.6
> Subversion
> file:///data/jenkins/workspace/generic-package-rhel64-6-0/topdir/BUILD/hive-1.1.0-cdh5.7.6
> -r Unknown
> Compiled by jenkins on Tue Feb 21 14:49:03 PST 2017
> From source with checksum 2e88d4c1b4053829c847d58bf016afd1
> Beeline version 1.2.1.spark2 by Apache Hive
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (KYLIN-4480) Implement runtime non-equi join
[
https://issues.apache.org/jira/browse/KYLIN-4480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126029#comment-17126029
]
nichunen edited comment on KYLIN-4480 at 6/4/20, 3:58 PM:
--
Tested with sql
{code:java}
SELECT * FROM
(
SELECT PART_DT, TRANS_ID FROM KYLIN_SALES GROUP BY PART_DT, TRANS_ID
) FACT
LEFT JOIN
(
SELECT CAL_DT as DT_CAL_DT, WEEK_BEG_DT FROM KYLIN_CAL_DT
) DT
ON FACT.PART_DT = DT.DT_CAL_DT AND WEEK_BEG_DT != DATE'2013-03-24'
{code}
!image-2020-06-04-23-57-51-652.png!
was (Author: nichunen):
Tested with sql
{code:java}
SELECT * FROM
(
SELECT PART_DT, TRANS_ID FROM KYLIN_SALES GROUP BY PART_DT, TRANS_ID
) FACT
LEFT JOIN
(
SELECT CAL_DT as DT_CAL_DT, WEEK_BEG_DT FROM KYLIN_CAL_DT
) DT
ON FACT.PART_DT = DT.DT_CAL_DT AND WEEK_BEG_DT = DATE'2013-03-24'
{code}
!image-2020-06-04-23-57-51-652.png!
> Implement runtime non-equi join
> ---
>
> Key: KYLIN-4480
> URL: https://issues.apache.org/jira/browse/KYLIN-4480
> Project: Kylin
> Issue Type: New Feature
> Components: Query Engine
>Reporter: Li Xian
>Assignee: Li Xian
>Priority: Minor
> Fix For: v3.1.0
>
> Attachments: image-2020-06-04-23-57-32-114.png,
> image-2020-06-04-23-57-39-920.png, image-2020-06-04-23-57-51-652.png
>
>
> Currently Kylin does not support non-equi join. This issue is aim to add
> runtime non-equi join feature to Kylin so that queries like
> {code:java}
> select
> |
> non-equi join
> /\
>subquery1 subquery2
> {code}
> can now be executed like below. Kylin will read data from subquery cubes and
> do non-equi join during runtime
> {code:java}
> select
> |
> runtime non-equi join
> /\
>cube1 cube2
> {code}
> I have submitted a PR for this issue
> [https://github.com/apache/kylin/pull/1195]
> [https://github.com/Kyligence/calcite/pull/133]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KYLIN-4480) Implement runtime non-equi join
[
https://issues.apache.org/jira/browse/KYLIN-4480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen resolved KYLIN-4480.
-
Resolution: Fixed
> Implement runtime non-equi join
> ---
>
> Key: KYLIN-4480
> URL: https://issues.apache.org/jira/browse/KYLIN-4480
> Project: Kylin
> Issue Type: New Feature
> Components: Query Engine
>Reporter: Li Xian
>Assignee: Li Xian
>Priority: Minor
> Fix For: v3.1.0
>
> Attachments: image-2020-06-04-23-57-32-114.png,
> image-2020-06-04-23-57-39-920.png, image-2020-06-04-23-57-51-652.png
>
>
> Currently Kylin does not support non-equi join. This issue is aim to add
> runtime non-equi join feature to Kylin so that queries like
> {code:java}
> select
> |
> non-equi join
> /\
>subquery1 subquery2
> {code}
> can now be executed like below. Kylin will read data from subquery cubes and
> do non-equi join during runtime
> {code:java}
> select
> |
> runtime non-equi join
> /\
>cube1 cube2
> {code}
> I have submitted a PR for this issue
> [https://github.com/apache/kylin/pull/1195]
> [https://github.com/Kyligence/calcite/pull/133]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KYLIN-4480) Implement runtime non-equi join
[
https://issues.apache.org/jira/browse/KYLIN-4480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen updated KYLIN-4480:
Attachment: (was: screenshot-1.png)
> Implement runtime non-equi join
> ---
>
> Key: KYLIN-4480
> URL: https://issues.apache.org/jira/browse/KYLIN-4480
> Project: Kylin
> Issue Type: New Feature
> Components: Query Engine
>Reporter: Li Xian
>Assignee: Li Xian
>Priority: Minor
> Fix For: v3.1.0
>
> Attachments: image-2020-06-04-23-57-32-114.png,
> image-2020-06-04-23-57-39-920.png, image-2020-06-04-23-57-51-652.png
>
>
> Currently Kylin does not support non-equi join. This issue is aim to add
> runtime non-equi join feature to Kylin so that queries like
> {code:java}
> select
> |
> non-equi join
> /\
>subquery1 subquery2
> {code}
> can now be executed like below. Kylin will read data from subquery cubes and
> do non-equi join during runtime
> {code:java}
> select
> |
> runtime non-equi join
> /\
>cube1 cube2
> {code}
> I have submitted a PR for this issue
> [https://github.com/apache/kylin/pull/1195]
> [https://github.com/Kyligence/calcite/pull/133]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Comment Edited] (KYLIN-4480) Implement runtime non-equi join
[
https://issues.apache.org/jira/browse/KYLIN-4480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126029#comment-17126029
]
nichunen edited comment on KYLIN-4480 at 6/4/20, 3:57 PM:
--
Tested with sql
{code:java}
SELECT * FROM
(
SELECT PART_DT, TRANS_ID FROM KYLIN_SALES GROUP BY PART_DT, TRANS_ID
) FACT
LEFT JOIN
(
SELECT CAL_DT as DT_CAL_DT, WEEK_BEG_DT FROM KYLIN_CAL_DT
) DT
ON FACT.PART_DT = DT.DT_CAL_DT AND WEEK_BEG_DT = DATE'2013-03-24'
{code}
!image-2020-06-04-23-57-51-652.png!
was (Author: nichunen):
Tested with sql
{code:java}
SELECT * FROM
(
SELECT PART_DT, TRANS_ID FROM KYLIN_SALES GROUP BY PART_DT, TRANS_ID
) FACT
LEFT JOIN
(
SELECT CAL_DT as DT_CAL_DT, WEEK_BEG_DT FROM KYLIN_CAL_DT
) DT
ON FACT.PART_DT = DT.DT_CAL_DT AND WEEK_BEG_DT = DATE'2013-03-24'
{code}
!screenshot-1.png!
> Implement runtime non-equi join
> ---
>
> Key: KYLIN-4480
> URL: https://issues.apache.org/jira/browse/KYLIN-4480
> Project: Kylin
> Issue Type: New Feature
> Components: Query Engine
>Reporter: Li Xian
>Assignee: Li Xian
>Priority: Minor
> Fix For: v3.1.0
>
> Attachments: image-2020-06-04-23-57-32-114.png,
> image-2020-06-04-23-57-39-920.png, image-2020-06-04-23-57-51-652.png,
> screenshot-1.png
>
>
> Currently Kylin does not support non-equi join. This issue is aim to add
> runtime non-equi join feature to Kylin so that queries like
> {code:java}
> select
> |
> non-equi join
> /\
>subquery1 subquery2
> {code}
> can now be executed like below. Kylin will read data from subquery cubes and
> do non-equi join during runtime
> {code:java}
> select
> |
> runtime non-equi join
> /\
>cube1 cube2
> {code}
> I have submitted a PR for this issue
> [https://github.com/apache/kylin/pull/1195]
> [https://github.com/Kyligence/calcite/pull/133]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-4480) Implement runtime non-equi join
[
https://issues.apache.org/jira/browse/KYLIN-4480?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17126029#comment-17126029
]
nichunen commented on KYLIN-4480:
-
Tested with sql
{code:java}
SELECT * FROM
(
SELECT PART_DT, TRANS_ID FROM KYLIN_SALES GROUP BY PART_DT, TRANS_ID
) FACT
LEFT JOIN
(
SELECT CAL_DT as DT_CAL_DT, WEEK_BEG_DT FROM KYLIN_CAL_DT
) DT
ON FACT.PART_DT = DT.DT_CAL_DT AND WEEK_BEG_DT = DATE'2013-03-24'
{code}
!screenshot-1.png!
> Implement runtime non-equi join
> ---
>
> Key: KYLIN-4480
> URL: https://issues.apache.org/jira/browse/KYLIN-4480
> Project: Kylin
> Issue Type: New Feature
> Components: Query Engine
>Reporter: Li Xian
>Assignee: Li Xian
>Priority: Minor
> Fix For: v3.1.0
>
> Attachments: screenshot-1.png
>
>
> Currently Kylin does not support non-equi join. This issue is aim to add
> runtime non-equi join feature to Kylin so that queries like
> {code:java}
> select
> |
> non-equi join
> /\
>subquery1 subquery2
> {code}
> can now be executed like below. Kylin will read data from subquery cubes and
> do non-equi join during runtime
> {code:java}
> select
> |
> runtime non-equi join
> /\
>cube1 cube2
> {code}
> I have submitted a PR for this issue
> [https://github.com/apache/kylin/pull/1195]
> [https://github.com/Kyligence/calcite/pull/133]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KYLIN-4480) Implement runtime non-equi join
[
https://issues.apache.org/jira/browse/KYLIN-4480?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen updated KYLIN-4480:
Attachment: screenshot-1.png
> Implement runtime non-equi join
> ---
>
> Key: KYLIN-4480
> URL: https://issues.apache.org/jira/browse/KYLIN-4480
> Project: Kylin
> Issue Type: New Feature
> Components: Query Engine
>Reporter: Li Xian
>Assignee: Li Xian
>Priority: Minor
> Fix For: v3.1.0
>
> Attachments: screenshot-1.png
>
>
> Currently Kylin does not support non-equi join. This issue is aim to add
> runtime non-equi join feature to Kylin so that queries like
> {code:java}
> select
> |
> non-equi join
> /\
>subquery1 subquery2
> {code}
> can now be executed like below. Kylin will read data from subquery cubes and
> do non-equi join during runtime
> {code:java}
> select
> |
> runtime non-equi join
> /\
>cube1 cube2
> {code}
> I have submitted a PR for this issue
> [https://github.com/apache/kylin/pull/1195]
> [https://github.com/Kyligence/calcite/pull/133]
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KYLIN-4429) For LEAST_RESTRICTIVE consistency, which is the '=' operator, convert implicitly
[
https://issues.apache.org/jira/browse/KYLIN-4429?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen updated KYLIN-4429:
Fix Version/s: v3.1.0
> For LEAST_RESTRICTIVE consistency, which is the '=' operator, convert
> implicitly
>
>
> Key: KYLIN-4429
> URL: https://issues.apache.org/jira/browse/KYLIN-4429
> Project: Kylin
> Issue Type: Sub-task
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>Priority: Major
> Fix For: v3.1.0
>
>
> For the following sql,
> {code}
> select count(*)
> from TEST_KYLIN_FACT
> where cal_dt = date'2012-01-01'
> {code}
> , if the data type of *cal_dt* is varchar, the sql cannot run successfully
> due to no implicitly data type conversion.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KYLIN-4539) Spark build failed due to com.codahale.metrics.json.MetricsModule loaded from kylin job jar
[
https://issues.apache.org/jira/browse/KYLIN-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen resolved KYLIN-4539.
-
Resolution: Fixed
> Spark build failed due to com.codahale.metrics.json.MetricsModule loaded
> from kylin job jar
>
>
> Key: KYLIN-4539
> URL: https://issues.apache.org/jira/browse/KYLIN-4539
> Project: Kylin
> Issue Type: Bug
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
>Priority: Major
> Fix For: v3.1.0
>
>
> {code:sh}
> [Loaded com.fasterxml.jackson.databind.Module from
> file:/data6/yarnenv/local/filecache/533549/spark-jars.zip/jackson-databind-2.9.4.jar]
> [Loaded org.apache.kylin.job.shaded.com.fasterxml.jackson.core.Versioned
> from file:/data8/yarnenv/local/filecache/533839/kylin-job-3.1.0-SNAPSHOT.jar]
> [Loaded org.apache.kylin.job.shaded.com.fasterxml.jackson.databind.Module
> from file:/data8/yarnenv/local/filecache/533839/kylin-job-3.1.0-SNAPSHOT.jar]
> [Loaded com.codahale.metrics.json.MetricsModule from
> file:/data8/yarnenv/local/filecache/533839/kylin-job-3.1.0-SNAPSHOT.jar]
> [Loaded java.lang.VerifyError from
> /data/gaiaadmin/gaiaenv/jdk1.8.0_181/jre/lib/rt.jar] [Loaded
> scala.util.control.NonFatal$ from
> file:/data6/yarnenv/local/filecache/533549/spark-jars.zip/scala-library-2.11.8.jar]
> [Loaded org.apache.spark.SparkUserAppException from
> file:/data6/yarnenv/local/filecache/533549/spark-jars.zip/spark-core_2.11-3.12.0.jar]
> [Loaded
> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4$$anonfun$run$11 from
> file:/data6/yarnenv/local/filecache/533549/spark-jars.zip/spark-yarn_2.11-3.12.0.jar]
> [Loaded org.apache.log4j.spi.ThrowableInformation from
> file:/data6/yarnenv/local/filecache/533549/spark-jars.zip/log4j-1.2.17.jar]
> [Loaded org.apache.log4j.spi.ThrowableRenderer from
> file:/data6/yarnenv/local/filecache/533549/spark-jars.zip/log4j-1.2.17.jar]
> [Loaded org.apache.log4j.DefaultThrowableRenderer from
> file:/data6/yarnenv/local/filecache/533549/spark-jars.zip/log4j-1.2.17.jar]
> [Loaded java.lang.Throwable$PrintStreamOrWriter from
> /data/gaiaadmin/gaiaenv/jdk1.8.0_181/jre/lib/rt.jar] [Loaded
> java.lang.Throwable$WrappedPrintWriter from
> /data/gaiaadmin/gaiaenv/jdk1.8.0_181/jre/lib/rt.jar] [Loaded
> java.io.LineNumberReader from
> /data/gaiaadmin/gaiaenv/jdk1.8.0_181/jre/lib/rt.jar] User class threw
> exception: java.lang.VerifyError: Bad type on operand stack Exception
> Details: Location:
> org/apache/spark/metrics/sink/MetricsServlet.(Ljava/util/Properties;Lcom/codahale/metrics/MetricRegistry;Lorg/apache/spark/SecurityManager;)V
> @116: invokevirtual Reason: Type 'com/codahale/metrics/json/MetricsModule'
> (current frame, stack[2]) is not assignable to
> 'com/fasterxml/jackson/databind/Module' Current Frame: bci: @116 flags: \{ }
> locals: \{ 'org/apache/spark/metrics/sink/MetricsServlet',
> 'java/util/Properties', 'com/codahale/metrics/MetricRegistry',
> 'org/apache/spark/SecurityManager' } stack: \{
> 'org/apache/spark/metrics/sink/MetricsServlet',
> 'com/fasterxml/jackson/databind/ObjectMapper',
> 'com/codahale/metrics/json/MetricsModule' } Bytecode: 0x000: 2a2b b500
> 2a2a 2cb5 002f 2a2d b500 5c2a 0x010: b700 7e2a 1280 b500 322a 1282 b500
> 342a 0x020: 03b5 0037 2a2b 2ab6 0084 b600 8ab5 0039 0x030: 2ab2 008f
> 2b2a b600 91b6 008a b600 95bb 0x040: 0014 592a b700 96b6 009c bb00 1659
> 2ab7 0x050: 009d b600 a1b8 00a7 b500 3b2a bb00 7159 0x060: b700 a8bb
> 00aa 59b2 00b0 b200 b32a b600 0x070: b5b7 00b8 b600 bcb5 003e b1
> java.lang.VerifyError: Bad type on operand stack Exception Details: Location:
> org/apache/spark/metrics/sink/MetricsServlet.(Ljava/util/Properties;Lcom/codahale/metrics/MetricRegistry;Lorg/apache/spark/SecurityManager;)V
> @116: invokevirtual Reason: Type 'com/codahale/metrics/json/MetricsModule'
> (current frame, stack[2]) is not assignable to
> 'com/fasterxml/jackson/databind/Module' Current Frame: bci: @116 flags: \{ }
> locals: \{ 'org/apache/spark/metrics/sink/MetricsServlet',
> 'java/util/Properties', 'com/codahale/metrics/MetricRegistry',
> 'org/apache/spark/SecurityManager' } stack: \{
> 'org/apache/spark/metrics/sink/MetricsServlet',
> 'com/fasterxml/jackson/databind/ObjectMapper',
> 'com/codahale/metrics/json/MetricsModule' } Bytecode: 0x000: 2a2b b500
> 2a2a 2cb5 002f 2a2d b500 5c2a 0x010: b700 7e2a 1280 b500 322a 1282 b500
> 342a 0x020: 03b5 0037 2a2b 2ab6 0084 b600 8ab5 0039 0x030: 2ab2 008f
> 2b2a b600 91b6 008a b600 95bb 0x040: 0014 592a b700 96b6 009c bb00 1659
> 2ab7 0x050: 009d b600 a1b8 00a7 b500 3b2a bb00 7159 0x060: b700 a8bb
> 00aa 59b2 00b0 b200 b32a b600 0x070: b5b7 00b8 b600 bcb5 003e b1 at
>
[jira] [Resolved] (KYLIN-4492) Bad query leads to full gc
[
https://issues.apache.org/jira/browse/KYLIN-4492?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen resolved KYLIN-4492.
-
Resolution: Fixed
> Bad query leads to full gc
> --
>
> Key: KYLIN-4492
> URL: https://issues.apache.org/jira/browse/KYLIN-4492
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>Priority: Major
> Fix For: v3.1.0
>
>
> The following sql goes into dead loop of *RuleQueue.popMatch()* in
> *VolcanoPlanner.findBestExp()* leading to OOM
> {code:java}
> select LSTG_FORMAT_NAME,
> cast(sum(t.sid_cnt) as bigint) as sid_cnts
> from
> (select LSTG_FORMAT_NAME,
> count(distinct SELLER_ID) as sid_cnt
> from TEST_KYLIN_FACT
> group by LSTG_FORMAT_NAME) t
> group by LSTG_FORMAT_NAME
> order by sid_cnts
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KYLIN-4543) Remove the usage for jackson enableDefaultTyping()
[ https://issues.apache.org/jira/browse/KYLIN-4543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen resolved KYLIN-4543. - Resolution: Fixed > Remove the usage for jackson enableDefaultTyping() > -- > > Key: KYLIN-4543 > URL: https://issues.apache.org/jira/browse/KYLIN-4543 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Reporter: Shao Feng Shi >Assignee: Shao Feng Shi >Priority: Minor > Fix For: v3.1.0, v3.0.3 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4322) Cost–benefit of compression HBase result
[ https://issues.apache.org/jira/browse/KYLIN-4322?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-4322: Fix Version/s: (was: v3.1.0) v3.1.1 > Cost–benefit of compression HBase result > > > Key: KYLIN-4322 > URL: https://issues.apache.org/jira/browse/KYLIN-4322 > Project: Kylin > Issue Type: Bug >Reporter: ZhouKang >Assignee: ZhouKang >Priority: Major > Fix For: v3.1.1 > > > kylin.storage.hbase.endpoint-compress-result is TRUE as default. > In our production environment, when the hbase scan result is larger than > 200M, it will take more than 10s to compress data. > We can find this by hbase's log: > ||Size||avg rate||min rate||avg time||max time|| > |<1M|0.12|0.25|0.18ms|0.7s| > |1M ~ 10M|0.39|0.97|0.2s|0.6s| > |10M ~ 100M|0.47|0.81|2s|6.3s| > |>100M|0.95|0.96|15.7s|24.8s| > Notice： > # rate: compressed data size / origin data size > # when the source data size is < 1M, compressed data may larger than the > source data. So the table(Row 1) only calculate then compressed data less > than the source data > # In our environment, 65% compression data (<1M) is larger than source data > When source data is less then 10M, the latency of data transmission is > acceptability. When data is larger then 100M, it will take a long time to > compress data. > > So, I think kylin.storage.hbase.endpoint-compress-result should be FALSE by > default; > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KYLIN-4272) problems of docker/build_image.sh
[
https://issues.apache.org/jira/browse/KYLIN-4272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen reassigned KYLIN-4272:
---
Assignee: ZhouKang
> problems of docker/build_image.sh
> -
>
> Key: KYLIN-4272
> URL: https://issues.apache.org/jira/browse/KYLIN-4272
> Project: Kylin
> Issue Type: Bug
>Reporter: ZhouKang
>Assignee: ZhouKang
>Priority: Major
> Fix For: v3.1.0
>
>
> this script can only be executed in sub dir "docker", if your want to execute
> as following, some error got:
> {code:java}
> // code placeholder
> bash docker/build_image.sh{code}
> And, the source code's dir name must be kylin, you cannot use other name.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (KYLIN-4250) FechRunnner should skip the job to process other jobs instead of throwing exception when the job section metadata is not found
[
https://issues.apache.org/jira/browse/KYLIN-4250?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen reassigned KYLIN-4250:
---
Assignee: chuxiao
> FechRunnner should skip the job to process other jobs instead of throwing
> exception when the job section metadata is not found
> --
>
> Key: KYLIN-4250
> URL: https://issues.apache.org/jira/browse/KYLIN-4250
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v3.0.0-alpha
>Reporter: chuxiao
>Assignee: chuxiao
>Priority: Critical
> Fix For: v2.6.5, v3.1.0, v3.0.1
>
> Attachments: KYLIN-4250.master.001.patch
>
>
> problem:
> Our cluster has two nodes (named build1, build2) building cube jobs, and used
> DistributedScheduler.
> There is a job, id 9f05b84b-cec9-81ee-9336-5a419e451a55, shown built on the
> build1 node.
> The job displays Error, but the first sub task creating hive flat table
> display Ready, and can see the first task's yarn job running through yarn ui.
> After the yarn job is successful, the job re-runs the first sub-task, again
> and again.
> log:
> Looking at the build1 log, the status of this job is changed from ready to
> running, then the first task status is ready to running, then the update job
> information is broadcast, then the update job information broadcast is
> received. But after twenty seconds, a broadcast of the updated job
> information was received.
> After a few minutes, the first task is completed, but the log shows that the
> job status changed from Error to ready! Then the job status changed from
> ready to running, the first task starts running again Repeat the above
> log.
> I suspect that other nodes have changed the job status. Looking at the build2
> node log, there are a lot of exception logs, about there is no output for
> another job id f1b2024a-e6ed-3dd5-5a7d-7c267ead5f1d:
> {code:java}
> 2019-09-20 14:20:58,825 WARN [pool-10-thread-1]
> threadpool.DefaultFetcherRunner:90 : Job Fetcher caught a exception
> java.lang.IllegalArgumentException: there is no related output for job
> id:f1b2024a-e6ed-3dd5-5a7d-7c267ead5f1d
> at
> com.google.common.base.Preconditions.checkArgument(Preconditions.java:92)
> at
> org.apache.kylin.job.execution.ExecutableManager.getOutputDigest(ExecutableManager.java:184)
> at
> org.apache.kylin.job.impl.threadpool.DefaultFetcherRunner.run(DefaultFetcherRunner.java:67)
> at
> java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> at
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> In addition, each build2 receives the broadcast of the build1 update the job
> information， after twenty seconds, the log print changes the first task state
> runinng to ready and broadcasts.
> Restarting the build2 node, not printing the Job Fetcher caught a exception ,
> and the job 9f05b84b-cec9-81ee-9336-5a419e451a55 was successfully executed.
> analysis
> This is due to a job metadata synchronization exception, which triggers a job
> scheduling bug. Build1 node try to run the job, but another build node kills
> the job and changes the job status to Error, causing problems.
> The build2 node may have a metadata synchronization problem, the job with the
> id f1b2024a-e6ed-3dd5-5a7d-7c267ead5f1d exists in ExecutableDao's
> executableDigestMap, and does not exist in ExecutableDao's
> executableOutputDigestMap. Each time FetchRunner foreach the job, it throws
> an exception and fetchFailed is set to true.
> {code:java}
> DefaultFetcherRunner:
> //throw exception
> final Output outputDigest = getExecutableManger().getOutputDigest(id);
> .
> .
> .
> } catch (Throwable th) {
> fetchFailed = true; // this could happen when resource store is
> unavailable
> logger.warn("Job Fetcher caught a exception ", th);
> }
> {code}
> When the build2 first processes the job that build1 is running, since
> fetchFailed is true, the job is not in the list of running jobs in build2,
> the job status is running, FetchRunner.jobStateCount() will kill the job, and
> set the running task status to ready, set the job status to error,
[jira] [Assigned] (KYLIN-4226) Skip current unavailable tables when updating hbase coprocessor
[
https://issues.apache.org/jira/browse/KYLIN-4226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen reassigned KYLIN-4226:
---
Assignee: ZhouKang
> Skip current unavailable tables when updating hbase coprocessor
> ---
>
> Key: KYLIN-4226
> URL: https://issues.apache.org/jira/browse/KYLIN-4226
> Project: Kylin
> Issue Type: Improvement
>Reporter: ZhouKang
>Assignee: ZhouKang
>Priority: Major
> Fix For: v2.6.5, v3.1.0
>
>
> when update hbase coprocessor for htable, there are exceptions cause by
> unavailable tables.
> {code:java}
> // code placeholder
> 2019-08-22 09:12:20,445 INFO [main] util.DeployCoprocessorCLI:181 : Commit
> Information: 2945cb67bae6242f275a038e7bf872a7a9797464;
> Exception in thread "main" org.apache.hadoop.hbase.TableNotFoundException:
> kylin_zjyprc_bigdata_staging:KYLIN_UCHB8D2MVP
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.getTableDescriptor(HBaseAdmin.java:567)
> at
> org.apache.hadoop.hbase.client.HBaseAdmin.getDescriptor(HBaseAdmin.java:365)
> at
> org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI.filterByGitCommit(DeployCoprocessorCLI.java:183)
> at
> org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI.main(DeployCoprocessorCLI.java:131)
> 2019-08-22 09:12:21,207 INFO [close-hbase-conn] hbase.HBaseConnection:137 :
> Closing HBase connections...
> 2019-08-22 09:12:21,208 INFO [close-hbase-conn]
> client.ConnectionImplementation:1889 : Closing master protocol: MasterService
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (KYLIN-4211) PartitionDesc support custom year、month、day partitions name
[
https://issues.apache.org/jira/browse/KYLIN-4211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen reassigned KYLIN-4211:
---
Assignee: chuxiao
> PartitionDesc support custom year、month、day partitions name
> ---
>
> Key: KYLIN-4211
> URL: https://issues.apache.org/jira/browse/KYLIN-4211
> Project: Kylin
> Issue Type: Improvement
>Reporter: chuxiao
>Assignee: chuxiao
>Priority: Major
> Fix For: v3.1.0
>
> Attachments: KYLIN-4211.master.001.patch
>
>
> YearMonthDayPartitionConditionBuilder supports partition named year, month,
> day, but can not support partition names like Y, M, D.
> Because there are user using fact table partitioned by Y, M, D, so add
> CustomYearMonthDayFieldPartitionConditionBuilder, support custom year, month,
> day partitions name.
> Partition metadata in model.json like:
> {
> "uuid" : "459d48c1-a8a6-cdf5-6ea7-e2ae48b248e9",
> "last_modified" : 1571652918478,
> "version" : "2.6.0.20500",
> "name" : "kylin_sales_ymd",
> "owner" : "admin",
> "is_draft" : false,
> "description" : "",
> "fact_table" : "BIGDATA_KYLIN.KYLIN_SALES_YMD",
> "lookups" : [ ],
> "dimensions" : [ {
> "table" : "KYLIN_SALES_YMD",
> "columns" : [ "LEAF_CATEG_ID", "TRANS_ID", "SLR_SEGMENT_CD", "SELLER_ID",
> "BUYER_ID", "OPS_USER_ID", "OPS_REGION", "Y", "M", "D" ]
> } ],
> "metrics" : [ "KYLIN_SALES_YMD.PRICE", "KYLIN_SALES_YMD.ITEM_COUNT" ],
> "filter_condition" : "",
> "partition_desc" : {
> "partition_date_column" : "KYLIN_SALES_YMD.Y, KYLIN_SALES_YMD.M,
> KYLIN_SALES_YMD.D",
> "partition_time_column" : null,
> "partition_date_start" : 0,
> "partition_date_format" : "-MM-dd",
> "partition_time_format" : "HH:mm:ss",
> "partition_type" : "APPEND",
> "partition_condition_builder" :
> "org.apache.kylin.metadata.model.PartitionDesc$CustomYearMonthDayFieldPartitionConditionBuilder"
> },
> "capacity" : "MEDIUM"
> }
> Partition_date_column is the year, month, and day partition field split by
> ','.， and Partition_condition_builder is
> org.apache.kylin.metadata.model.PartitionDesc$CustomYearMonthDayFieldPartitionConditionBuilder
> Web can create a normal model and "Edit JSON" on Kylin web
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Assigned] (KYLIN-4212) Add a user profile page
[ https://issues.apache.org/jira/browse/KYLIN-4212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen reassigned KYLIN-4212: --- Assignee: Congling Xia (was: nichunen) > Add a user profile page > --- > > Key: KYLIN-4212 > URL: https://issues.apache.org/jira/browse/KYLIN-4212 > Project: Kylin > Issue Type: Improvement >Reporter: Congling Xia >Assignee: Congling Xia >Priority: Major > Fix For: v3.1.0 > > Attachments: change_pass.png, entrance.png, profile.png > > > Kylin's user/group API has been extended since KYLIN-4122. User can get more > account details and change password by requesting the APIs. But there is no > webpage for a normal user to view such information or do such changes. > In this improvement, the following changes are desired: > # provide a user profile page to show his/her basic information, > # set a button on it to launch the change password dialog for the user -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KYLIN-4212) Add a user profile page
[ https://issues.apache.org/jira/browse/KYLIN-4212?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen reassigned KYLIN-4212: --- Assignee: nichunen > Add a user profile page > --- > > Key: KYLIN-4212 > URL: https://issues.apache.org/jira/browse/KYLIN-4212 > Project: Kylin > Issue Type: Improvement >Reporter: Congling Xia >Assignee: nichunen >Priority: Major > Fix For: v3.1.0 > > Attachments: change_pass.png, entrance.png, profile.png > > > Kylin's user/group API has been extended since KYLIN-4122. User can get more > account details and change password by requesting the APIs. But there is no > webpage for a normal user to view such information or do such changes. > In this improvement, the following changes are desired: > # provide a user profile page to show his/her basic information, > # set a button on it to launch the change password dialog for the user -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (KYLIN-3947) TS conflict when kylin update metadata in JDBC (MySQL)
[
https://issues.apache.org/jira/browse/KYLIN-3947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen closed KYLIN-3947.
---
Resolution: Fixed
> TS conflict when kylin update metadata in JDBC (MySQL)
> --
>
> Key: KYLIN-3947
> URL: https://issues.apache.org/jira/browse/KYLIN-3947
> Project: Kylin
> Issue Type: Bug
> Components: Security
>Affects Versions: v2.6.1
>Reporter: Iñigo Martinez
>Assignee: Yaqian Zhang
>Priority: Major
> Fix For: Backlog
>
>
> Unable to login in Kylin with admin user. Kylin cluster is composed by two
> nodes, one QUERY node and one ALL node. Our backend for metadata is MySQL
> (not HBASE). Kylin version: 2.6.1
> Solved with same workaround applied in related bug.
> {code:java}
> $KYLIN_HOME/bin/metastore.sh remove /user/admin
> {code}
>
> Tomcat exception follows:
> {code:java}
> Apr 10, 2019 7:30:29 AM org.apache.catalina.core.StandardWrapperValve invoke
> SEVERE: Servlet.service() for servlet [kylin] in context with path [/kylin]
> threw exception
> org.apache.kylin.common.persistence.WriteConflictException: Overwriting
> conflict /user/ADMIN, expect old TS 0, but it is 1554879164739
> at
> org.apache.kylin.common.persistence.JDBCResourceStore$5.execute(JDBCResourceStore.java:482)
> at
> org.apache.kylin.common.persistence.JDBCResourceStore.executeSql(JDBCResourceStore.java:100)
> at
> org.apache.kylin.common.persistence.JDBCResourceStore.checkAndPutResourceInternal(JDBCResourceStore.java:439)
> at
> org.apache.kylin.common.persistence.JDBCResourceStore.checkAndPutResourceImpl(JDBCResourceStore.java:427)
> at
> org.apache.kylin.common.persistence.ResourceStore$6.call(ResourceStore.java:442)
> at
> org.apache.kylin.common.persistence.ResourceStore$6.call(ResourceStore.java:439)
> at
> org.apache.kylin.common.persistence.ExponentialBackoffRetry.doWithRetry(ExponentialBackoffRetry.java:52)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceWithRetry(ResourceStore.java:439)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:427)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResource(ResourceStore.java:421)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResource(ResourceStore.java:401)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResource(ResourceStore.java:380)
> at
> org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192)
> at
> org.apache.kylin.rest.security.KylinUserManager.update(KylinUserManager.java:124)
> at
> org.apache.kylin.rest.service.KylinUserService.updateUser(KylinUserService.java:86)
> at
> org.apache.kylin.rest.service.KylinUserService.createUser(KylinUserService.java:78)
> at
> org.apache.kylin.rest.security.KylinAuthenticationProvider.authenticate(KylinAuthenticationProvider.java:116)
> at
> org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:174)
> at
> org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:199)
> at
> org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:180)
> at
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
> org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
> org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
> org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:64)
> at
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
> org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
> at
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
> org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
> at
>
[jira] [Updated] (KYLIN-3947) TS conflict when kylin update metadata in JDBC (MySQL)
[
https://issues.apache.org/jira/browse/KYLIN-3947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen updated KYLIN-3947:
Fix Version/s: (was: v3.1.0)
Backlog
> TS conflict when kylin update metadata in JDBC (MySQL)
> --
>
> Key: KYLIN-3947
> URL: https://issues.apache.org/jira/browse/KYLIN-3947
> Project: Kylin
> Issue Type: Bug
> Components: Security
>Affects Versions: v2.6.1
>Reporter: Iñigo Martinez
>Assignee: Yaqian Zhang
>Priority: Major
> Fix For: Backlog
>
>
> Unable to login in Kylin with admin user. Kylin cluster is composed by two
> nodes, one QUERY node and one ALL node. Our backend for metadata is MySQL
> (not HBASE). Kylin version: 2.6.1
> Solved with same workaround applied in related bug.
> {code:java}
> $KYLIN_HOME/bin/metastore.sh remove /user/admin
> {code}
>
> Tomcat exception follows:
> {code:java}
> Apr 10, 2019 7:30:29 AM org.apache.catalina.core.StandardWrapperValve invoke
> SEVERE: Servlet.service() for servlet [kylin] in context with path [/kylin]
> threw exception
> org.apache.kylin.common.persistence.WriteConflictException: Overwriting
> conflict /user/ADMIN, expect old TS 0, but it is 1554879164739
> at
> org.apache.kylin.common.persistence.JDBCResourceStore$5.execute(JDBCResourceStore.java:482)
> at
> org.apache.kylin.common.persistence.JDBCResourceStore.executeSql(JDBCResourceStore.java:100)
> at
> org.apache.kylin.common.persistence.JDBCResourceStore.checkAndPutResourceInternal(JDBCResourceStore.java:439)
> at
> org.apache.kylin.common.persistence.JDBCResourceStore.checkAndPutResourceImpl(JDBCResourceStore.java:427)
> at
> org.apache.kylin.common.persistence.ResourceStore$6.call(ResourceStore.java:442)
> at
> org.apache.kylin.common.persistence.ResourceStore$6.call(ResourceStore.java:439)
> at
> org.apache.kylin.common.persistence.ExponentialBackoffRetry.doWithRetry(ExponentialBackoffRetry.java:52)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceWithRetry(ResourceStore.java:439)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:427)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResource(ResourceStore.java:421)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResource(ResourceStore.java:401)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResource(ResourceStore.java:380)
> at
> org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192)
> at
> org.apache.kylin.rest.security.KylinUserManager.update(KylinUserManager.java:124)
> at
> org.apache.kylin.rest.service.KylinUserService.updateUser(KylinUserService.java:86)
> at
> org.apache.kylin.rest.service.KylinUserService.createUser(KylinUserService.java:78)
> at
> org.apache.kylin.rest.security.KylinAuthenticationProvider.authenticate(KylinAuthenticationProvider.java:116)
> at
> org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:174)
> at
> org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:199)
> at
> org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:180)
> at
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
> org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
> org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
> org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:64)
> at
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
> org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
> at
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
>
[jira] [Updated] (KYLIN-4539) Spark build failed due to com.codahale.metrics.json.MetricsModule loaded from kylin job jar
[
https://issues.apache.org/jira/browse/KYLIN-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen updated KYLIN-4539:
Sprint: Sprint 52
> Spark build failed due to com.codahale.metrics.json.MetricsModule loaded
> from kylin job jar
>
>
> Key: KYLIN-4539
> URL: https://issues.apache.org/jira/browse/KYLIN-4539
> Project: Kylin
> Issue Type: Bug
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
>Priority: Major
> Fix For: v3.1.0
>
>
> {code:sh}
> [Loaded com.fasterxml.jackson.databind.Module from
> file:/data6/yarnenv/local/filecache/533549/spark-jars.zip/jackson-databind-2.9.4.jar]
> [Loaded org.apache.kylin.job.shaded.com.fasterxml.jackson.core.Versioned
> from file:/data8/yarnenv/local/filecache/533839/kylin-job-3.1.0-SNAPSHOT.jar]
> [Loaded org.apache.kylin.job.shaded.com.fasterxml.jackson.databind.Module
> from file:/data8/yarnenv/local/filecache/533839/kylin-job-3.1.0-SNAPSHOT.jar]
> [Loaded com.codahale.metrics.json.MetricsModule from
> file:/data8/yarnenv/local/filecache/533839/kylin-job-3.1.0-SNAPSHOT.jar]
> [Loaded java.lang.VerifyError from
> /data/gaiaadmin/gaiaenv/jdk1.8.0_181/jre/lib/rt.jar] [Loaded
> scala.util.control.NonFatal$ from
> file:/data6/yarnenv/local/filecache/533549/spark-jars.zip/scala-library-2.11.8.jar]
> [Loaded org.apache.spark.SparkUserAppException from
> file:/data6/yarnenv/local/filecache/533549/spark-jars.zip/spark-core_2.11-3.12.0.jar]
> [Loaded
> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4$$anonfun$run$11 from
> file:/data6/yarnenv/local/filecache/533549/spark-jars.zip/spark-yarn_2.11-3.12.0.jar]
> [Loaded org.apache.log4j.spi.ThrowableInformation from
> file:/data6/yarnenv/local/filecache/533549/spark-jars.zip/log4j-1.2.17.jar]
> [Loaded org.apache.log4j.spi.ThrowableRenderer from
> file:/data6/yarnenv/local/filecache/533549/spark-jars.zip/log4j-1.2.17.jar]
> [Loaded org.apache.log4j.DefaultThrowableRenderer from
> file:/data6/yarnenv/local/filecache/533549/spark-jars.zip/log4j-1.2.17.jar]
> [Loaded java.lang.Throwable$PrintStreamOrWriter from
> /data/gaiaadmin/gaiaenv/jdk1.8.0_181/jre/lib/rt.jar] [Loaded
> java.lang.Throwable$WrappedPrintWriter from
> /data/gaiaadmin/gaiaenv/jdk1.8.0_181/jre/lib/rt.jar] [Loaded
> java.io.LineNumberReader from
> /data/gaiaadmin/gaiaenv/jdk1.8.0_181/jre/lib/rt.jar] User class threw
> exception: java.lang.VerifyError: Bad type on operand stack Exception
> Details: Location:
> org/apache/spark/metrics/sink/MetricsServlet.(Ljava/util/Properties;Lcom/codahale/metrics/MetricRegistry;Lorg/apache/spark/SecurityManager;)V
> @116: invokevirtual Reason: Type 'com/codahale/metrics/json/MetricsModule'
> (current frame, stack[2]) is not assignable to
> 'com/fasterxml/jackson/databind/Module' Current Frame: bci: @116 flags: \{ }
> locals: \{ 'org/apache/spark/metrics/sink/MetricsServlet',
> 'java/util/Properties', 'com/codahale/metrics/MetricRegistry',
> 'org/apache/spark/SecurityManager' } stack: \{
> 'org/apache/spark/metrics/sink/MetricsServlet',
> 'com/fasterxml/jackson/databind/ObjectMapper',
> 'com/codahale/metrics/json/MetricsModule' } Bytecode: 0x000: 2a2b b500
> 2a2a 2cb5 002f 2a2d b500 5c2a 0x010: b700 7e2a 1280 b500 322a 1282 b500
> 342a 0x020: 03b5 0037 2a2b 2ab6 0084 b600 8ab5 0039 0x030: 2ab2 008f
> 2b2a b600 91b6 008a b600 95bb 0x040: 0014 592a b700 96b6 009c bb00 1659
> 2ab7 0x050: 009d b600 a1b8 00a7 b500 3b2a bb00 7159 0x060: b700 a8bb
> 00aa 59b2 00b0 b200 b32a b600 0x070: b5b7 00b8 b600 bcb5 003e b1
> java.lang.VerifyError: Bad type on operand stack Exception Details: Location:
> org/apache/spark/metrics/sink/MetricsServlet.(Ljava/util/Properties;Lcom/codahale/metrics/MetricRegistry;Lorg/apache/spark/SecurityManager;)V
> @116: invokevirtual Reason: Type 'com/codahale/metrics/json/MetricsModule'
> (current frame, stack[2]) is not assignable to
> 'com/fasterxml/jackson/databind/Module' Current Frame: bci: @116 flags: \{ }
> locals: \{ 'org/apache/spark/metrics/sink/MetricsServlet',
> 'java/util/Properties', 'com/codahale/metrics/MetricRegistry',
> 'org/apache/spark/SecurityManager' } stack: \{
> 'org/apache/spark/metrics/sink/MetricsServlet',
> 'com/fasterxml/jackson/databind/ObjectMapper',
> 'com/codahale/metrics/json/MetricsModule' } Bytecode: 0x000: 2a2b b500
> 2a2a 2cb5 002f 2a2d b500 5c2a 0x010: b700 7e2a 1280 b500 322a 1282 b500
> 342a 0x020: 03b5 0037 2a2b 2ab6 0084 b600 8ab5 0039 0x030: 2ab2 008f
> 2b2a b600 91b6 008a b600 95bb 0x040: 0014 592a b700 96b6 009c bb00 1659
> 2ab7 0x050: 009d b600 a1b8 00a7 b500 3b2a bb00 7159 0x060: b700 a8bb
> 00aa 59b2 00b0 b200 b32a b600 0x070: b5b7 00b8 b600 bcb5 003e b1 at
>
[jira] [Updated] (KYLIN-4537) Give a friendly tips to the user when getting task list fails
[ https://issues.apache.org/jira/browse/KYLIN-4537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-4537: Sprint: Sprint 52 > Give a friendly tips to the user when getting task list fails > - > > Key: KYLIN-4537 > URL: https://issues.apache.org/jira/browse/KYLIN-4537 > Project: Kylin > Issue Type: Improvement > Components: Web >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Major > Fix For: v3.1.0 > > Attachments: image-2020-05-27-21-47-34-128.png, > image-2020-05-27-21-48-57-223.png > > > On the monitoring page, when the task list acquisition fails, it will show > "failed to load jobs", but we don't know the detail reason for the failure, > whether it is due to permission problems or the system crash? > Before: > !image-2020-05-27-21-48-57-223.png|width=491,height=314! > After: > !image-2020-05-27-21-47-34-128.png|width=441,height=310! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4539) Spark build failed due to com.codahale.metrics.json.MetricsModule loaded from kylin job jar
[
https://issues.apache.org/jira/browse/KYLIN-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen updated KYLIN-4539:
Fix Version/s: v3.1.0
> Spark build failed due to com.codahale.metrics.json.MetricsModule loaded
> from kylin job jar
>
>
> Key: KYLIN-4539
> URL: https://issues.apache.org/jira/browse/KYLIN-4539
> Project: Kylin
> Issue Type: Bug
>Reporter: Guangxu Cheng
>Assignee: Guangxu Cheng
>Priority: Major
> Fix For: v3.1.0
>
>
> {code:sh}
> [Loaded com.fasterxml.jackson.databind.Module from
> file:/data6/yarnenv/local/filecache/533549/spark-jars.zip/jackson-databind-2.9.4.jar]
> [Loaded org.apache.kylin.job.shaded.com.fasterxml.jackson.core.Versioned
> from file:/data8/yarnenv/local/filecache/533839/kylin-job-3.1.0-SNAPSHOT.jar]
> [Loaded org.apache.kylin.job.shaded.com.fasterxml.jackson.databind.Module
> from file:/data8/yarnenv/local/filecache/533839/kylin-job-3.1.0-SNAPSHOT.jar]
> [Loaded com.codahale.metrics.json.MetricsModule from
> file:/data8/yarnenv/local/filecache/533839/kylin-job-3.1.0-SNAPSHOT.jar]
> [Loaded java.lang.VerifyError from
> /data/gaiaadmin/gaiaenv/jdk1.8.0_181/jre/lib/rt.jar] [Loaded
> scala.util.control.NonFatal$ from
> file:/data6/yarnenv/local/filecache/533549/spark-jars.zip/scala-library-2.11.8.jar]
> [Loaded org.apache.spark.SparkUserAppException from
> file:/data6/yarnenv/local/filecache/533549/spark-jars.zip/spark-core_2.11-3.12.0.jar]
> [Loaded
> org.apache.spark.deploy.yarn.ApplicationMaster$$anon$4$$anonfun$run$11 from
> file:/data6/yarnenv/local/filecache/533549/spark-jars.zip/spark-yarn_2.11-3.12.0.jar]
> [Loaded org.apache.log4j.spi.ThrowableInformation from
> file:/data6/yarnenv/local/filecache/533549/spark-jars.zip/log4j-1.2.17.jar]
> [Loaded org.apache.log4j.spi.ThrowableRenderer from
> file:/data6/yarnenv/local/filecache/533549/spark-jars.zip/log4j-1.2.17.jar]
> [Loaded org.apache.log4j.DefaultThrowableRenderer from
> file:/data6/yarnenv/local/filecache/533549/spark-jars.zip/log4j-1.2.17.jar]
> [Loaded java.lang.Throwable$PrintStreamOrWriter from
> /data/gaiaadmin/gaiaenv/jdk1.8.0_181/jre/lib/rt.jar] [Loaded
> java.lang.Throwable$WrappedPrintWriter from
> /data/gaiaadmin/gaiaenv/jdk1.8.0_181/jre/lib/rt.jar] [Loaded
> java.io.LineNumberReader from
> /data/gaiaadmin/gaiaenv/jdk1.8.0_181/jre/lib/rt.jar] User class threw
> exception: java.lang.VerifyError: Bad type on operand stack Exception
> Details: Location:
> org/apache/spark/metrics/sink/MetricsServlet.(Ljava/util/Properties;Lcom/codahale/metrics/MetricRegistry;Lorg/apache/spark/SecurityManager;)V
> @116: invokevirtual Reason: Type 'com/codahale/metrics/json/MetricsModule'
> (current frame, stack[2]) is not assignable to
> 'com/fasterxml/jackson/databind/Module' Current Frame: bci: @116 flags: \{ }
> locals: \{ 'org/apache/spark/metrics/sink/MetricsServlet',
> 'java/util/Properties', 'com/codahale/metrics/MetricRegistry',
> 'org/apache/spark/SecurityManager' } stack: \{
> 'org/apache/spark/metrics/sink/MetricsServlet',
> 'com/fasterxml/jackson/databind/ObjectMapper',
> 'com/codahale/metrics/json/MetricsModule' } Bytecode: 0x000: 2a2b b500
> 2a2a 2cb5 002f 2a2d b500 5c2a 0x010: b700 7e2a 1280 b500 322a 1282 b500
> 342a 0x020: 03b5 0037 2a2b 2ab6 0084 b600 8ab5 0039 0x030: 2ab2 008f
> 2b2a b600 91b6 008a b600 95bb 0x040: 0014 592a b700 96b6 009c bb00 1659
> 2ab7 0x050: 009d b600 a1b8 00a7 b500 3b2a bb00 7159 0x060: b700 a8bb
> 00aa 59b2 00b0 b200 b32a b600 0x070: b5b7 00b8 b600 bcb5 003e b1
> java.lang.VerifyError: Bad type on operand stack Exception Details: Location:
> org/apache/spark/metrics/sink/MetricsServlet.(Ljava/util/Properties;Lcom/codahale/metrics/MetricRegistry;Lorg/apache/spark/SecurityManager;)V
> @116: invokevirtual Reason: Type 'com/codahale/metrics/json/MetricsModule'
> (current frame, stack[2]) is not assignable to
> 'com/fasterxml/jackson/databind/Module' Current Frame: bci: @116 flags: \{ }
> locals: \{ 'org/apache/spark/metrics/sink/MetricsServlet',
> 'java/util/Properties', 'com/codahale/metrics/MetricRegistry',
> 'org/apache/spark/SecurityManager' } stack: \{
> 'org/apache/spark/metrics/sink/MetricsServlet',
> 'com/fasterxml/jackson/databind/ObjectMapper',
> 'com/codahale/metrics/json/MetricsModule' } Bytecode: 0x000: 2a2b b500
> 2a2a 2cb5 002f 2a2d b500 5c2a 0x010: b700 7e2a 1280 b500 322a 1282 b500
> 342a 0x020: 03b5 0037 2a2b 2ab6 0084 b600 8ab5 0039 0x030: 2ab2 008f
> 2b2a b600 91b6 008a b600 95bb 0x040: 0014 592a b700 96b6 009c bb00 1659
> 2ab7 0x050: 009d b600 a1b8 00a7 b500 3b2a bb00 7159 0x060: b700 a8bb
> 00aa 59b2 00b0 b200 b32a b600 0x070: b5b7 00b8 b600 bcb5 003e b1 at
>
[jira] [Updated] (KYLIN-3371) Increase of time between steps after Kylin 2.3.1 update
[
https://issues.apache.org/jira/browse/KYLIN-3371?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen updated KYLIN-3371:
Labels: newbie (was: )
> Increase of time between steps after Kylin 2.3.1 update
> ---
>
> Key: KYLIN-3371
> URL: https://issues.apache.org/jira/browse/KYLIN-3371
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v2.3.1
> Environment: -EMR 5.7 (S3 storage for Hive and HBase)
> -Kylin 2.3.1 (updated from 2.2). Deployed on separated EC2 instance.
>Reporter: Roberto Tardío Olmos
>Priority: Major
> Labels: newbie
> Fix For: Backlog
>
> Attachments: Screenshot1.png, Screenshot2.png
>
>
> Hi,
> We have updated from Kylin 2.2 to Kylin 2.3.1. We performed the update
> process with successful. However, we have experimented an increase of time
> wasted between steps during building time. For example, the time between step
> 7 “Build N-Dimension Cuboid:level 1” and “…level2” is now about 30 seconds, a
> lot of time if we take into account that in these steps no processing is
> executed since we use the "layer" algorithm (Screenshot 1). This time between
> steps is much longer than when we used Kylin 2.2, were average time was about
> 5 seconds. (Screenshot 2).
> Could it be related to a change in the code? The configuration we use in
> Kylin 2.3.1 is the same as we used in Kylin 2.2. I have considered this JIRA
> as a possible BUG.
> Thanks in advance,
>
> Screenshot 1:
> !Screenshot1.png!
>
> Screenshot 2
>
> !Screenshot2.png!
>
>
> A sample of the generated log.
> Kylin 2.2
>
> 2018-05-04 14:25:40,797 INFO [Scheduler 542798956 Job
> cb843ecc-5062-49ec-a881-215097e4501a-518] execution.ExecutableManager:421 :
> job id:cb843ecc-5062-49ec-a881-215097e4501a from RUNNING to READY
> 2018-05-04 14:25:45,496 INFO [Scheduler 542798956 Job
> cb843ecc-5062-49ec-a881-215097e4501a-518] execution.AbstractExecutable:111 :
> Executing AbstractExecutable (BUILD CUBE - Captacion_Resumido_Cubo_v3 -
> 2018050400_2018050500 - GMT+02:00 2018-05-04 14:21:34)
>
> Kylin 2.3.1
>
> 2018-05-07 15:25:26,338 INFO [Scheduler 1942482192 Job
> 6d616841-1c4d-46ca-a1b9-a18047fbada5-1006] execution.ExecutableManager:411 :
> job id:6d616841-1c4d-46ca-a1b9-a18047fbada5 from RUNNING to READY
> 2018-05-07 {color:#FF}15:25:48,365{color} INFO [Scheduler 1942482192 Job
> 6d616841-1c4d-46ca-a1b9-a18047fbada5-1006] execution.AbstractExecutable:147 :
> Executing AbstractExecutable (BUILD CUBE - Captacion_Resumido_Cubo_v3 -
> 2018050700_2018050800 - GMT+02:00 2018-05-07 15:19:41)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KYLIN-3310) Use lint for maven-compiler-plugin
[
https://issues.apache.org/jira/browse/KYLIN-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen updated KYLIN-3310:
Labels: newbie (was: )
> Use lint for maven-compiler-plugin
> --
>
> Key: KYLIN-3310
> URL: https://issues.apache.org/jira/browse/KYLIN-3310
> Project: Kylin
> Issue Type: Improvement
> Components: Tools, Build and Test
>Reporter: Ted Yu
>Assignee: Jiatao Tao
>Priority: Major
> Labels: newbie
> Fix For: Backlog
>
>
> lint helps identify structural problems.
> We should enable lint for maven-compiler-plugin
> {code}
> maven-compiler-plugin
> ${maven-compiler-plugin.version}
>
> 1.8
> 1.8
>
> -Xlint:all
> ${compiler.error.flag}
>
> -Xlint:-options
>
> -Xlint:-cast
> -Xlint:-deprecation
> -Xlint:-processing
> -Xlint:-rawtypes
> -Xlint:-serial
> -Xlint:-try
> -Xlint:-unchecked
> -Xlint:-varargs
>
>
>
>
> true
>
> false
>
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KYLIN-3417) Consider replacing ReentrantReadWriteLock with StampedLock
[
https://issues.apache.org/jira/browse/KYLIN-3417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen updated KYLIN-3417:
Labels: newbie (was: )
> Consider replacing ReentrantReadWriteLock with StampedLock
> --
>
> Key: KYLIN-3417
> URL: https://issues.apache.org/jira/browse/KYLIN-3417
> Project: Kylin
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Jiatao Tao
>Priority: Major
> Labels: newbie
> Fix For: Backlog
>
>
> ReentrantReadWriteLock's are only the right solution when there is long hold
> time due to expensive I/O.
> It is expensive for readers.
> We should see if the lighter {{StampedLock}} can be used instead.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KYLIN-3431) Avoid FileInputStream/FileOutputStream
[
https://issues.apache.org/jira/browse/KYLIN-3431?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen updated KYLIN-3431:
Labels: newbie (was: )
> Avoid FileInputStream/FileOutputStream
> --
>
> Key: KYLIN-3431
> URL: https://issues.apache.org/jira/browse/KYLIN-3431
> Project: Kylin
> Issue Type: Improvement
>Reporter: Ted Yu
>Priority: Major
> Labels: newbie
> Fix For: Backlog
>
>
> They rely on finalizers (before Java 11), which create unnecessary GC load.
> The alternatives, {{Files.newInputStream}}, are as easy to use and don't have
> this issue.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Closed] (KYLIN-3133) Fix KYLIN-2717 compatibility issue
[ https://issues.apache.org/jira/browse/KYLIN-3133?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen closed KYLIN-3133. --- Resolution: Won't Fix > Fix KYLIN-2717 compatibility issue > -- > > Key: KYLIN-3133 > URL: https://issues.apache.org/jira/browse/KYLIN-3133 > Project: Kylin > Issue Type: Bug > Components: Metadata, Tools, Build and Test >Affects Versions: v2.2.0 >Reporter: Kaisen Kang >Assignee: Kaisen Kang >Priority: Major > Fix For: Backlog > > Attachments: KYLIN-2717-compatibility.patch > > > Fix KYLIN-2717 compatibility issue: > 1 keep old getTableDesc api so that user could rolling upgrade to v2.2.0 when > user have dozens of QueryServer. > 2 Use tableRef.getTableDesc().getProject() not modelDesc.getProject() to be > compatible with old table resource path format. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KYLIN-3099) Add restart operation for kylin.sh
[ https://issues.apache.org/jira/browse/KYLIN-3099?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen resolved KYLIN-3099. - Resolution: Fixed > Add restart operation for kylin.sh > -- > > Key: KYLIN-3099 > URL: https://issues.apache.org/jira/browse/KYLIN-3099 > Project: Kylin > Issue Type: Improvement > Components: Others >Affects Versions: v2.3.0 >Reporter: peng.jianhua >Assignee: peng.jianhua >Priority: Major > Labels: patch > Fix For: Backlog > > Attachments: 0001-KYLIN-3099-Add-restart-operation-for-kylin.sh.patch > > > When I want to restart kylin server, I should execute stop operation, then > execute start operation, I think this is very troublesome, it is more > convenient to add a restart operation. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-3046) Consider introducing log4j-extras
[ https://issues.apache.org/jira/browse/KYLIN-3046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-3046: Labels: log newbie (was: log) > Consider introducing log4j-extras > -- > > Key: KYLIN-3046 > URL: https://issues.apache.org/jira/browse/KYLIN-3046 > Project: Kylin > Issue Type: Improvement >Reporter: Ted Yu >Priority: Major > Labels: log, newbie > Fix For: Backlog > > > log4j-extras allows log rotation as well as compression. > https://logging.apache.org/log4j/extras/download.html > We should consider using log4j-extras. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-3095) Use ArrayDeque instead of LinkedList for queue implementation
[
https://issues.apache.org/jira/browse/KYLIN-3095?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen updated KYLIN-3095:
Labels: newbie parallel (was: parallel)
> Use ArrayDeque instead of LinkedList for queue implementation
> -
>
> Key: KYLIN-3095
> URL: https://issues.apache.org/jira/browse/KYLIN-3095
> Project: Kylin
> Issue Type: Improvement
>Reporter: Ted Yu
>Assignee: Kaige Liu
>Priority: Minor
> Labels: newbie, parallel
> Fix For: Backlog
>
>
> Use ArrayDeque instead of LinkedList for queue implementation where thread
> safety is not needed.
> From https://docs.oracle.com/javase/7/docs/api/java/util/ArrayDeque.html
> {quote}
> Resizable-array implementation of the Deque interface. Array deques have no
> capacity restrictions; they grow as necessary to support usage. They are not
> thread-safe; in the absence of external synchronization, they do not support
> concurrent access by multiple threads. Null elements are prohibited. This
> class is likely to be faster than Stack when used as a stack, and *faster
> than LinkedList when used as a queue.*
> {quote}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KYLIN-2744) Should return correct type for SUM measure in web
[ https://issues.apache.org/jira/browse/KYLIN-2744?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-2744: Labels: newbie (was: ) > Should return correct type for SUM measure in web > - > > Key: KYLIN-2744 > URL: https://issues.apache.org/jira/browse/KYLIN-2744 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v2.0.0 >Reporter: Kaisen Kang >Assignee: Kaisen Kang >Priority: Major > Labels: newbie > Fix For: Backlog > > Attachments: KYLIN-2744.patch > > > Currently, Kylin return decimal type for the sum measure of double type, > which will result in wrong result. So, We should return correct type for SUM > measure in web. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4496) Metric data missing
[ https://issues.apache.org/jira/browse/KYLIN-4496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-4496: Sprint: (was: Sprint 52) > Metric data missing > --- > > Key: KYLIN-4496 > URL: https://issues.apache.org/jira/browse/KYLIN-4496 > Project: Kylin > Issue Type: Bug > Components: Metrics >Reporter: tianhui >Assignee: tianhui >Priority: Critical > Fix For: v3.1.0 > > > Metric data in memory is never flushed into file system when kylin closing, > So there is some metric data missing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4151) FileSplit ClassCastException in KafkaMRInput
[ https://issues.apache.org/jira/browse/KYLIN-4151?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-4151: Sprint: (was: Sprint 52) > FileSplit ClassCastException in KafkaMRInput > > > Key: KYLIN-4151 > URL: https://issues.apache.org/jira/browse/KYLIN-4151 > Project: Kylin > Issue Type: Bug > Components: Others >Affects Versions: v2.5.2 >Reporter: Tian Hong Wang >Assignee: Shao Feng Shi >Priority: Minor > Fix For: v3.1.0 > > Attachments: 20190828172850.png, KafkaMRInput.patch > > > It wil cause ClassCastException in KafkaMRInput: > org.apache.hadoop.mapreduce.lib.input.FileSplit cannot be cast to > org.apache.hadoop.mapred.FileSplit. > The stack trace see the attachment. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4330) Support the filter function in model desinger for NRT
[ https://issues.apache.org/jira/browse/KYLIN-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-4330: Sprint: (was: Sprint 51) > Support the filter function in model desinger for NRT > -- > > Key: KYLIN-4330 > URL: https://issues.apache.org/jira/browse/KYLIN-4330 > Project: Kylin > Issue Type: Improvement > Components: NRT Streaming >Affects Versions: v2.6.4 > Environment: 阿里云 centos 7 hadoop 2.8.5 >Reporter: kangkang >Priority: Major > Labels: FIlter, model > Fix For: Backlog > > Attachments: 4561578452903_.pic_hd.jpg > > Original Estimate: 336h > Remaining Estimate: 336h > > when i use nrt streaming build for kafka data, can we use filter function > when i set model desinger -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4507) Override getSocketAddress() in TCPMemcachedNodeImpl to auto detect memcached instance ip change
[ https://issues.apache.org/jira/browse/KYLIN-4507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-4507: Fix Version/s: v3.1.0 > Override getSocketAddress() in TCPMemcachedNodeImpl to auto detect memcached > instance ip change > --- > > Key: KYLIN-4507 > URL: https://issues.apache.org/jira/browse/KYLIN-4507 > Project: Kylin > Issue Type: Sub-task >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v3.1.0 > > > When deploying memcached to k8s, it's probably that a memcached instance goes > up & down. During the process, its hostname can be unchanged while its ip > changes. > Therefore, we override the getSocketAddress() in TCPMemcachedNodeImpl to > uncache ip for hostname to make it possible for Kylin to be able to auto > detect memcached instance ip change. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KYLIN-4507) Override getSocketAddress() in TCPMemcachedNodeImpl to auto detect memcached instance ip change
[ https://issues.apache.org/jira/browse/KYLIN-4507?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen resolved KYLIN-4507. - Resolution: Fixed > Override getSocketAddress() in TCPMemcachedNodeImpl to auto detect memcached > instance ip change > --- > > Key: KYLIN-4507 > URL: https://issues.apache.org/jira/browse/KYLIN-4507 > Project: Kylin > Issue Type: Sub-task >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > > When deploying memcached to k8s, it's probably that a memcached instance goes > up & down. During the process, its hostname can be unchanged while its ip > changes. > Therefore, we override the getSocketAddress() in TCPMemcachedNodeImpl to > uncache ip for hostname to make it possible for Kylin to be able to auto > detect memcached instance ip change. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KYLIN-4506) Remove unresolved memcached servers before initialize MemcachedClient
[ https://issues.apache.org/jira/browse/KYLIN-4506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen resolved KYLIN-4506. - Resolution: Fixed > Remove unresolved memcached servers before initialize MemcachedClient > - > > Key: KYLIN-4506 > URL: https://issues.apache.org/jira/browse/KYLIN-4506 > Project: Kylin > Issue Type: Sub-task >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v3.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KYLIN-4504) Cache usage improvement
[ https://issues.apache.org/jira/browse/KYLIN-4504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen resolved KYLIN-4504. - Resolution: Fixed > Cache usage improvement > --- > > Key: KYLIN-4504 > URL: https://issues.apache.org/jira/browse/KYLIN-4504 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: nichunen >Priority: Major > Fix For: v3.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KYLIN-4505) Change guava cache to spring cache for user authentication
[ https://issues.apache.org/jira/browse/KYLIN-4505?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen resolved KYLIN-4505. - Resolution: Fixed > Change guava cache to spring cache for user authentication > -- > > Key: KYLIN-4505 > URL: https://issues.apache.org/jira/browse/KYLIN-4505 > Project: Kylin > Issue Type: Sub-task >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > > Currently the cache for user authentication is local and the invalidation > policy is actively invoking invalidateAll(), which is not good for reuse. > After this change, the cache expiration will be passive and good for reuse. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KYLIN-4537) Give a friendly tips to the user when getting task list fails
[ https://issues.apache.org/jira/browse/KYLIN-4537?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen resolved KYLIN-4537. - Resolution: Fixed > Give a friendly tips to the user when getting task list fails > - > > Key: KYLIN-4537 > URL: https://issues.apache.org/jira/browse/KYLIN-4537 > Project: Kylin > Issue Type: Improvement > Components: Web >Reporter: Guangxu Cheng >Assignee: Guangxu Cheng >Priority: Major > Fix For: v3.1.0 > > Attachments: image-2020-05-27-21-47-34-128.png, > image-2020-05-27-21-48-57-223.png > > > On the monitoring page, when the task list acquisition fails, it will show > "failed to load jobs", but we don't know the detail reason for the failure, > whether it is due to permission problems or the system crash? > Before: > !image-2020-05-27-21-48-57-223.png|width=491,height=314! > After: > !image-2020-05-27-21-47-34-128.png|width=441,height=310! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4506) Remove unresolved memcached servers before initialize MemcachedClient
[ https://issues.apache.org/jira/browse/KYLIN-4506?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-4506: Fix Version/s: v3.1.0 > Remove unresolved memcached servers before initialize MemcachedClient > - > > Key: KYLIN-4506 > URL: https://issues.apache.org/jira/browse/KYLIN-4506 > Project: Kylin > Issue Type: Sub-task >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v3.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KYLIN-4504) Cache usage improvement
[ https://issues.apache.org/jira/browse/KYLIN-4504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen reassigned KYLIN-4504: --- Assignee: nichunen (was: Zhong Yanghong) > Cache usage improvement > --- > > Key: KYLIN-4504 > URL: https://issues.apache.org/jira/browse/KYLIN-4504 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: nichunen >Priority: Major > Fix For: v3.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4493) On HDP3 using spark to build dimension dictionary, it throws NoSuchMethodError.
[
https://issues.apache.org/jira/browse/KYLIN-4493?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen updated KYLIN-4493:
Sprint: Sprint 52 (was: Sprint 51)
> On HDP3 using spark to build dimension dictionary, it throws
> NoSuchMethodError.
> ---
>
> Key: KYLIN-4493
> URL: https://issues.apache.org/jira/browse/KYLIN-4493
> Project: Kylin
> Issue Type: Bug
> Components: Spark Engine
>Affects Versions: v3.0.1, v3.0.2
>Reporter: Zhichao Zhang
>Assignee: Zhichao Zhang
>Priority: Minor
>
> Test env:
> {code:java}
> HDP 3.0.1.0-187
> Hadoop 3.1.1
> Hive 3.1.0
> Kylin 3.0.2
> Spark 2.3.2{code}
>
> Problems:
> when use spark engine and turn on
> *'kylin.engine.spark-dimension-dictionary'*, it will throw error:
> {code:java}
> java.lang.NoSuchMethodError:
> org.apache.hadoop.hive.metastore.HiveMetaStoreClient.(Lorg/apache/hadoop/conf/Configuration;)V
> at
> org.apache.kylin.source.hive.CLIHiveClient.getMetaStoreClient(CLIHiveClient.java:164)
> at
> org.apache.kylin.source.hive.CLIHiveClient.getHiveTableMeta(CLIHiveClient.java:78)
> at org.apache.kylin.source.hive.HiveTable.(HiveTable.java:48)
> at
> org.apache.kylin.source.hive.HiveSource.createReadableTable(HiveSource.java:68)
> at
> org.apache.kylin.source.SourceManager.createReadableTable(SourceManager.java:145)
> at
> org.apache.kylin.engine.spark.SparkBuildDictionary$SnapshotBuildFunction.buildSnapshotTable(SparkBuildDictionary.java:386)
> at
> org.apache.kylin.engine.spark.SparkBuildDictionary$SnapshotBuildFunction.call(SparkBuildDictionary.java:367)
> at
> org.apache.kylin.engine.spark.SparkBuildDictionary$SnapshotBuildFunction.call(SparkBuildDictionary.java:325)
> at
> org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1043)
> at
> org.apache.spark.api.java.JavaPairRDD$$anonfun$pairFunToScalaFun$1.apply(JavaPairRDD.scala:1043)
> at scala.collection.Iterator$$anon$11.next(Iterator.scala:409)
> at scala.collection.Iterator$$anon$13.hasNext(Iterator.scala:462)
> at scala.collection.Iterator$class.foreach(Iterator.scala:893)
> at scala.collection.AbstractIterator.foreach(Iterator.scala:1336)
> at
> scala.collection.generic.Growable$class.$plus$plus$eq(Growable.scala:59)
> at
> scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:104)
> at
> scala.collection.mutable.ArrayBuffer.$plus$plus$eq(ArrayBuffer.scala:48)
> at scala.collection.TraversableOnce$class.to(TraversableOnce.scala:310)
> at scala.collection.AbstractIterator.to(Iterator.scala:1336)
> at
> scala.collection.TraversableOnce$class.toBuffer(TraversableOnce.scala:302)
> at scala.collection.AbstractIterator.toBuffer(Iterator.scala:1336)
> at
> scala.collection.TraversableOnce$class.toArray(TraversableOnce.scala:289)
> at scala.collection.AbstractIterator.toArray(Iterator.scala:1336)
> at
> org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$12.apply(RDD.scala:945)
> at
> org.apache.spark.rdd.RDD$$anonfun$collect$1$$anonfun$12.apply(RDD.scala:945)
> at
> org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)
> at
> org.apache.spark.SparkContext$$anonfun$runJob$5.apply(SparkContext.scala:2074)
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KYLIN-4485) Create a self service interface for cube migration
[ https://issues.apache.org/jira/browse/KYLIN-4485?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-4485: Sprint: Sprint 52 (was: Sprint 51) > Create a self service interface for cube migration > -- > > Key: KYLIN-4485 > URL: https://issues.apache.org/jira/browse/KYLIN-4485 > Project: Kylin > Issue Type: New Feature > Components: Tools, Build and Test >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v3.1.0 > > Attachments: cube-migration-new.png, cube-migration-old.png > > > Current cube migration process is as follows: > !cube-migration-old.png! > There're a few drawbacks: > * kylin admin has to manually check all related aspects to decide whether a > cube is good for migration. If cube design is not good, there'll be many > interaction between users & kylin admins > * kylin admin has to login to the backend server and then run a command to > do the migration, which is tricky and easy to make mistakes > * there's little compatibility check between source metadata & destination > cluster. > A self service interface will refine the migration process as follows: > !cube-migration-new.png! > We can see with the new migration process, there'll be few interactions > between users & kylin admins and much less work for kylin admins. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4419) Make it possible to change metadata in some cases without rebuilding data
[ https://issues.apache.org/jira/browse/KYLIN-4419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-4419: Sprint: Sprint 52 (was: Sprint 51) > Make it possible to change metadata in some cases without rebuilding data > - > > Key: KYLIN-4419 > URL: https://issues.apache.org/jira/browse/KYLIN-4419 > Project: Kylin > Issue Type: Improvement > Components: Metadata >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v3.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-3121) NPE while executing a query with two left outer joins and floating point expressions on nullable fields
[ https://issues.apache.org/jira/browse/KYLIN-3121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-3121: Sprint: Sprint 52 (was: Sprint 51) > NPE while executing a query with two left outer joins and floating point > expressions on nullable fields > --- > > Key: KYLIN-3121 > URL: https://issues.apache.org/jira/browse/KYLIN-3121 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v2.2.0 > Environment: HDP 2.5.6, Kylin 2.2.0 >Reporter: Vsevolod Ostapenko >Assignee: nichunen >Priority: Major > Fix For: v3.1.0 > > > Queries that include two (or more) left outer joins and contain floating > point expressions that operate on the fields that contain integer NULL values > (due to left outer join) fail in-flight with NullPointerExceptions. > As an example, the following query generates NPE on either of the two > expressions: > * 100.0 * t2.media_gap_call_count > * 1.0 * NULLIF(t1.active_call_count, 0) > with > t1 > as > ( > select > d1.cell_name, > count(distinct a1.call_id) as active_call_count > from > zetticsdw.a_vl_hourly_v a1 > inner join > zetticsdw.d_cell_v d1 > on > a1.cell_key = d1.cell_key > where > d1.region_3 = 'Mumbai' > and > a1.thedate = '20171011' > and > a1.thehour = '00' > and > a1.active_call_flg = 1 > group by > d1.cell_name > ), > t2 > as > ( > select > d1.cell_name, > count(distinct a1.call_id) as media_gap_call_count > from > zetticsdw.a_vl_hourly_v a1 > inner join > zetticsdw.d_cell_v d1 > on > a1.cell_key = d1.cell_key > where > d1.region_3 = 'Mumbai' > and > a1.thedate='20171011' > and > a1.thehour = '00' > and > a1.media_gap_call_flg = 1 > group by > d1.cell_name > ) > , > t3 > as > ( > select > d1.cell_name, > sum(a1.ow_call_flg) one_way_call_count, > sum(a1.succ_call_flg) successfull_call_count > from > zetticsdw.a_vl_hourly_v a1 > inner join > zetticsdw.d_cell_v d1 > on > a1.cell_key = d1.cell_key > where > d1.region_3 = 'Mumbai' > and > a1.thedate='20171011' > and > a1.thehour = '00' > group by > d1.cell_name > ) > select >t3.cell_name, >t1.active_call_count, >t2.media_gap_call_count, >t3.one_way_call_count, >t3.successfull_call_count, >-- 100 * t2.media_gap_call_count nom, -- > works >-- 1 * NULLIF(t1.active_call_count, 0) denom-- > works >100.0 * t2.media_gap_call_count nom, -- fails, > NPE of one kind >1.0 * NULLIF(t1.active_call_count, 0) denom -- fails, > NPE of different kind >-- 100.0 * COALESCE(t2.media_gap_call_count, 0) nom,-- > works >-- 1.0 * CAST(NULLIF(t1.active_call_count, 0) as DOUBLE) denom -- > works > from >t3 > left outer join >t1 > on >t3.cell_name = t1.cell_name > left outer join >t2 > on >t3.cell_name = t2.cell_name > In the first case (multiplication of an integer field with a NULL value and a > double) kylin log contains a stack trace similar to the following: > null > at org.apache.calcite.avatica.Helper.createException(Helper.java:56) > at org.apache.calcite.avatica.Helper.createException(Helper.java:41) > at > org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156) > at > org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:218) > at > org.apache.kylin.rest.service.QueryService.execute(QueryService.java:834) > at > org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:561) > at > org.apache.kylin.rest.service.QueryService.query(QueryService.java:181) > at > org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:415) > at > org.apache.kylin.rest.controller.QueryController.query(QueryController.java:78) > at sun.reflect.GeneratedMethodAccessor545.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) > at > org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) > at
[jira] [Updated] (KYLIN-4504) Cache usage improvement
[ https://issues.apache.org/jira/browse/KYLIN-4504?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-4504: Sprint: Sprint 51 > Cache usage improvement > --- > > Key: KYLIN-4504 > URL: https://issues.apache.org/jira/browse/KYLIN-4504 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v3.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Closed] (KYLIN-2738) Change to spring cache for acl
[ https://issues.apache.org/jira/browse/KYLIN-2738?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen closed KYLIN-2738. --- > Change to spring cache for acl > -- > > Key: KYLIN-2738 > URL: https://issues.apache.org/jira/browse/KYLIN-2738 > Project: Kylin > Issue Type: Sub-task >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > > Currently cache is mainly used for query to improve query performance. A > stable cache can also be used to improve the availability. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KYLIN-3361) Support stddev as a new measure
[ https://issues.apache.org/jira/browse/KYLIN-3361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen resolved KYLIN-3361. - Resolution: Fixed > Support stddev as a new measure > --- > > Key: KYLIN-3361 > URL: https://issues.apache.org/jira/browse/KYLIN-3361 > Project: Kylin > Issue Type: New Feature >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v3.1.0 > > > (x ~1~ - +x+) ^2^ + (x ~2~ - +x+) ^2^ + ... + (x ~n~ - +x+) ^2^ = x ~1~ ^2^ + > x ~2~ ^2^ + ... + x ~n~ ^2^ - n +x+ ^2^, where +x+ is the average of x ~1~, x > ~2~, ..., x ~n~. Therefore, to compute stddev, what kylin need to do is to > pre-calculate sum(x ~i~ ^2^), sum(x ~i~) and count -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KYLIN-3487) Create a new measure for precise count distinct
[ https://issues.apache.org/jira/browse/KYLIN-3487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen resolved KYLIN-3487. - Resolution: Fixed > Create a new measure for precise count distinct > --- > > Key: KYLIN-3487 > URL: https://issues.apache.org/jira/browse/KYLIN-3487 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v3.1.0 > > > To compute the precise count distinct, we can use bitmap and global > dictionary. However, there's a limitation for the global dictionary. It maps > from values to ids whose type is integer, which means the number of ids will > be less than 2B. And it's like a Pixiu for which there's increase but no > decrease. > In eBay, there's a requirement of calculating precise count distinct of > session. The session cardinality is large and will grow as time goes on. It > will not be feasible to use the global dictionary when its cardinality > exceeds the upper bound 2B. How can we deal with this? > There's good news that a session never crosses days. With this feature, we > don't need to merge bitmap across days. To calculate precise session > cardinality, we can assign each day a bitmap and directly summarize the > cardinalities estimated by each bitmap. No bitmap merge is needed. > To use bitmap for cardinality calculation, we need to map raw data from value > to an integer id, which is achieved by encoding the value with a dictionary. > Previously, for the ability of merging bitmaps from multiple segments, global > dictionary is used. However, in this case, there's no need of bitmap merge, > the global dictionary is not needed. > And we don't need to filter by or group by session. Then there's no need to > map from value to id and from id to value after the related bitmap is > constructed. Therefore, we don't need to store dictionaries for session. Only > the bitmap is enough. > To deal with segment merge, since bitmaps of each segment are not able to > merge to one bitmap, we use a map for storing multiple bitmaps. In the map, > the key is the segment name and the value is the segment-level bitmap. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KYLIN-4282) support case when in count (distinct)
[ https://issues.apache.org/jira/browse/KYLIN-4282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen resolved KYLIN-4282. - Resolution: Fixed > support case when in count (distinct) > - > > Key: KYLIN-4282 > URL: https://issues.apache.org/jira/browse/KYLIN-4282 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v3.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4282) support case when in count (distinct)
[ https://issues.apache.org/jira/browse/KYLIN-4282?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-4282: Fix Version/s: v3.1.0 > support case when in count (distinct) > - > > Key: KYLIN-4282 > URL: https://issues.apache.org/jira/browse/KYLIN-4282 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v3.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4275) Result of count(case when) is not correct
[
https://issues.apache.org/jira/browse/KYLIN-4275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen updated KYLIN-4275:
Fix Version/s: v3.1.0
> Result of count(case when) is not correct
> -
>
> Key: KYLIN-4275
> URL: https://issues.apache.org/jira/browse/KYLIN-4275
> Project: Kylin
> Issue Type: Bug
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>Priority: Major
> Fix For: v3.1.0
>
>
> For the following sql, the result is not correct:
> {code}
> select count(case when LEAF_CATEG_ID=165888 then 1 else null end)
> from TEST_KYLIN_FACT
> {code}
> The result should be equal to
> {code}
> select count(*)
> from TEST_KYLIN_FACT
> where LEAF_CATEG_ID=165888
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KYLIN-4275) Result of count(case when) is not correct
[
https://issues.apache.org/jira/browse/KYLIN-4275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen resolved KYLIN-4275.
-
Resolution: Fixed
> Result of count(case when) is not correct
> -
>
> Key: KYLIN-4275
> URL: https://issues.apache.org/jira/browse/KYLIN-4275
> Project: Kylin
> Issue Type: Bug
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>Priority: Major
> Fix For: v3.1.0
>
>
> For the following sql, the result is not correct:
> {code}
> select count(case when LEAF_CATEG_ID=165888 then 1 else null end)
> from TEST_KYLIN_FACT
> {code}
> The result should be equal to
> {code}
> select count(*)
> from TEST_KYLIN_FACT
> where LEAF_CATEG_ID=165888
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KYLIN-4281) Precisely set the data type of tuple expression
[ https://issues.apache.org/jira/browse/KYLIN-4281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen resolved KYLIN-4281. - Resolution: Fixed > Precisely set the data type of tuple expression > --- > > Key: KYLIN-4281 > URL: https://issues.apache.org/jira/browse/KYLIN-4281 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v3.1.0 > > > Previously to simplify the calculating of sum(case when), all of the binary > calculation is based on BigDecimal, which is not good for all, especially > when dealing with count(distinct case when), whose inner data type may be hll > or bitmap. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KYLIN-4496) Metric data missing
[ https://issues.apache.org/jira/browse/KYLIN-4496?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen resolved KYLIN-4496. - Resolution: Fixed > Metric data missing > --- > > Key: KYLIN-4496 > URL: https://issues.apache.org/jira/browse/KYLIN-4496 > Project: Kylin > Issue Type: Bug > Components: Metrics >Reporter: tianhui >Assignee: tianhui >Priority: Critical > Fix For: v3.1.0 > > > Metric data in memory is never flushed into file system when kylin closing, > So there is some metric data missing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-3487) Create a new measure for precise count distinct
[ https://issues.apache.org/jira/browse/KYLIN-3487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-3487: Sprint: Sprint 51 > Create a new measure for precise count distinct > --- > > Key: KYLIN-3487 > URL: https://issues.apache.org/jira/browse/KYLIN-3487 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v3.1.0 > > > To compute the precise count distinct, we can use bitmap and global > dictionary. However, there's a limitation for the global dictionary. It maps > from values to ids whose type is integer, which means the number of ids will > be less than 2B. And it's like a Pixiu for which there's increase but no > decrease. > In eBay, there's a requirement of calculating precise count distinct of > session. The session cardinality is large and will grow as time goes on. It > will not be feasible to use the global dictionary when its cardinality > exceeds the upper bound 2B. How can we deal with this? > There's good news that a session never crosses days. With this feature, we > don't need to merge bitmap across days. To calculate precise session > cardinality, we can assign each day a bitmap and directly summarize the > cardinalities estimated by each bitmap. No bitmap merge is needed. > To use bitmap for cardinality calculation, we need to map raw data from value > to an integer id, which is achieved by encoding the value with a dictionary. > Previously, for the ability of merging bitmaps from multiple segments, global > dictionary is used. However, in this case, there's no need of bitmap merge, > the global dictionary is not needed. > And we don't need to filter by or group by session. Then there's no need to > map from value to id and from id to value after the related bitmap is > constructed. Therefore, we don't need to store dictionaries for session. Only > the bitmap is enough. > To deal with segment merge, since bitmaps of each segment are not able to > merge to one bitmap, we use a map for storing multiple bitmaps. In the map, > the key is the segment name and the value is the segment-level bitmap. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-3487) Create a new measure for precise count distinct
[ https://issues.apache.org/jira/browse/KYLIN-3487?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-3487: Fix Version/s: (was: Backlog) v3.1.0 > Create a new measure for precise count distinct > --- > > Key: KYLIN-3487 > URL: https://issues.apache.org/jira/browse/KYLIN-3487 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v3.1.0 > > > To compute the precise count distinct, we can use bitmap and global > dictionary. However, there's a limitation for the global dictionary. It maps > from values to ids whose type is integer, which means the number of ids will > be less than 2B. And it's like a Pixiu for which there's increase but no > decrease. > In eBay, there's a requirement of calculating precise count distinct of > session. The session cardinality is large and will grow as time goes on. It > will not be feasible to use the global dictionary when its cardinality > exceeds the upper bound 2B. How can we deal with this? > There's good news that a session never crosses days. With this feature, we > don't need to merge bitmap across days. To calculate precise session > cardinality, we can assign each day a bitmap and directly summarize the > cardinalities estimated by each bitmap. No bitmap merge is needed. > To use bitmap for cardinality calculation, we need to map raw data from value > to an integer id, which is achieved by encoding the value with a dictionary. > Previously, for the ability of merging bitmaps from multiple segments, global > dictionary is used. However, in this case, there's no need of bitmap merge, > the global dictionary is not needed. > And we don't need to filter by or group by session. Then there's no need to > map from value to id and from id to value after the related bitmap is > constructed. Therefore, we don't need to store dictionaries for session. Only > the bitmap is enough. > To deal with segment merge, since bitmaps of each segment are not able to > merge to one bitmap, we use a map for storing multiple bitmaps. In the map, > the key is the segment name and the value is the segment-level bitmap. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4512) Update NOTICE file
nichunen created KYLIN-4512: --- Summary: Update NOTICE file Key: KYLIN-4512 URL: https://issues.apache.org/jira/browse/KYLIN-4512 Project: Kylin Issue Type: Improvement Components: Others Affects Versions: v3.0.2, v2.6.6 Reporter: nichunen Fix For: v3.1.0 The NOTICE file of Kylin is not in line with ASF policy. http://www.apache.org/dev/licensing-howto.html#alv2-dep If an Apache license piece of code is bundled, we need to copy the required parts into the NOTICE file. And we should update the year in the NOTICE file. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-3361) Support stddev as a new measure
[ https://issues.apache.org/jira/browse/KYLIN-3361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-3361: Sprint: Sprint 51 Issue Type: New Feature (was: Improvement) > Support stddev as a new measure > --- > > Key: KYLIN-3361 > URL: https://issues.apache.org/jira/browse/KYLIN-3361 > Project: Kylin > Issue Type: New Feature >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v3.1.0 > > > (x ~1~ - +x+) ^2^ + (x ~2~ - +x+) ^2^ + ... + (x ~n~ - +x+) ^2^ = x ~1~ ^2^ + > x ~2~ ^2^ + ... + x ~n~ ^2^ - n +x+ ^2^, where +x+ is the average of x ~1~, x > ~2~, ..., x ~n~. Therefore, to compute stddev, what kylin need to do is to > pre-calculate sum(x ~i~ ^2^), sum(x ~i~) and count -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-3361) Support stddev as a new measure
[ https://issues.apache.org/jira/browse/KYLIN-3361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-3361: Parent: (was: KYLIN-3358) Issue Type: Improvement (was: Sub-task) > Support stddev as a new measure > --- > > Key: KYLIN-3361 > URL: https://issues.apache.org/jira/browse/KYLIN-3361 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v3.1.0 > > > (x ~1~ - +x+) ^2^ + (x ~2~ - +x+) ^2^ + ... + (x ~n~ - +x+) ^2^ = x ~1~ ^2^ + > x ~2~ ^2^ + ... + x ~n~ ^2^ - n +x+ ^2^, where +x+ is the average of x ~1~, x > ~2~, ..., x ~n~. Therefore, to compute stddev, what kylin need to do is to > pre-calculate sum(x ~i~ ^2^), sum(x ~i~) and count -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-3361) Support stddev as a new measure
[ https://issues.apache.org/jira/browse/KYLIN-3361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-3361: Fix Version/s: (was: Backlog) v3.1.0 > Support stddev as a new measure > --- > > Key: KYLIN-3361 > URL: https://issues.apache.org/jira/browse/KYLIN-3361 > Project: Kylin > Issue Type: Sub-task >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v3.1.0 > > > (x ~1~ - +x+) ^2^ + (x ~2~ - +x+) ^2^ + ... + (x ~n~ - +x+) ^2^ = x ~1~ ^2^ + > x ~2~ ^2^ + ... + x ~n~ ^2^ - n +x+ ^2^, where +x+ is the average of x ~1~, x > ~2~, ..., x ~n~. Therefore, to compute stddev, what kylin need to do is to > pre-calculate sum(x ~i~ ^2^), sum(x ~i~) and count -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4305) Streaming Receiver cannot limit income query request or cancel long-running query
[
https://issues.apache.org/jira/browse/KYLIN-4305?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen updated KYLIN-4305:
Sprint: Sprint 52
> Streaming Receiver cannot limit income query request or cancel long-running
> query
> -
>
> Key: KYLIN-4305
> URL: https://issues.apache.org/jira/browse/KYLIN-4305
> Project: Kylin
> Issue Type: Improvement
> Components: Real-time Streaming
>Reporter: Xiaoxiang Yu
>Assignee: Xiaoxiang Yu
>Priority: Major
> Fix For: v3.1.0
>
> Attachments: Jietu20191217-221025.png, after_repair_receiver.jstack,
> image-2019-12-17-22-12-01-098.png, streaming_receiver_jstack.log
>
>
> When under heavy load(high rate of query request), receiver can not stand it,
> and most quey may timeout, but the query processing thread cannot be
> cancelled in receiver side, which will cause receiver's crash. You have to
> restart it.
> kylin.log
> {code:java}
> Caused by: java.lang.RuntimeException: timeout when call stream rpc
> at
> org.apache.kylin.storage.stream.rpc.HttpStreamDataSearchClient$QueuedStreamingTupleIterator.hasNext(HttpStreamDataSearchClient.java:298)
> at com.google.common.collect.Iterators$5.hasNext(Iterators.java:596)
> at
> org.apache.kylin.metadata.tuple.CompoundTupleIterator.hasNext(CompoundTupleIterator.java:52)
> at
> org.apache.kylin.query.enumerator.OLAPEnumerator.moveNext(OLAPEnumerator.java:63)
> at Baz$1$1.moveNext(Unknown Source)
> at
> org.apache.calcite.linq4j.EnumerableDefaults.groupBy_(EnumerableDefaults.java:825)
> at
> org.apache.calcite.linq4j.EnumerableDefaults.groupBy(EnumerableDefaults.java:761)
> at
> org.apache.calcite.linq4j.DefaultEnumerable.groupBy(DefaultEnumerable.java:302)
> at Baz.bind(Unknown Source)
> at
> org.apache.calcite.jdbc.CalcitePrepare$CalciteSignature.enumerable(CalcitePrepare.java:365)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.enumerable(CalciteConnectionImpl.java:301)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl._createIterable(CalciteMetaImpl.java:559)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl.createIterable(CalciteMetaImpl.java:550)
> at
> org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:182)
> at
> org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:67)
> at
> org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:44)
> at
> org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:667)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:619)
> at
> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
> ... 83 more
> {code}
> jstack
> {code:java}
> "qtp1901663135-48" #48 prio=5 os_prio=0 tid=0x7f839995f800 nid=0x3cdd
> runnable [0x7f83674fe000]
>java.lang.Thread.State: RUNNABLE
> at java.lang.Thread.yield(Native Method)
> at
> org.apache.kylin.stream.core.query.MultiThreadsResultCollector$1.hasNext(MultiThreadsResultCollector.java:75)
> at
> org.apache.kylin.stream.core.query.RecordsAggregator.aggregate(RecordsAggregator.java:100)
> at
> org.apache.kylin.stream.core.query.StreamingCubeDataSearcher$StreamAggregateSearchResult.iterator(StreamingCubeDataSearcher.java:191)
> at
> org.apache.kylin.stream.server.rest.controller.DataController.query(DataController.java:119)
> at sun.reflect.GeneratedMethodAccessor87.invoke(Unknown Source)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
> at
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
> at
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)
> at
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)
> at
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)
> at
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)
> at
>
[jira] [Updated] (KYLIN-4497) PreparedStatement query do not support "limit ? offset ?"
[
https://issues.apache.org/jira/browse/KYLIN-4497?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen updated KYLIN-4497:
Sprint: Sprint 52
> PreparedStatement query do not support "limit ? offset ?"
> -
>
> Key: KYLIN-4497
> URL: https://issues.apache.org/jira/browse/KYLIN-4497
> Project: Kylin
> Issue Type: Bug
> Components: Driver - JDBC
>Reporter: Chao Long
>Assignee: Chao Long
>Priority: Major
> Fix For: v3.1.0
>
>
> Query
> {code:java}
> String sql = "select part_dt, sum(price) from KYLIN_SALES group by part_dt
> order by part_dt limit ? offset ?";
> conn = getConnection();
> statement = conn.prepareStatement(sql);
> statement.setInt(1, 10);
> statement.setInt(2, 0);
> {code}
> Error log
> {code:java}
> 2020-05-15 09:32:36,062 ERROR [Query b87327f4-7299-32f2-4a2e-c19795a42288-44]
> service.QueryService:542 : Exception while executing query
> java.sql.SQLException: Error while preparing statement [select part_dt,
> sum(price) from KYLIN_SALES group by part_dt order by part_dt limit ? offset
> ?]
> at org.apache.calcite.avatica.Helper.createException(Helper.java:56)
> at org.apache.calcite.avatica.Helper.createException(Helper.java:41)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement_(CalciteConnectionImpl.java:210)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:192)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.prepareStatement(CalciteConnectionImpl.java:89)
> at
> org.apache.calcite.avatica.AvaticaConnection.prepareStatement(AvaticaConnection.java:175)
> at
> org.apache.kylin.rest.service.QueryService.createPreparedContext(QueryService.java:1262)
> at
> org.apache.kylin.rest.service.QueryService.access$600(QueryService.java:141)
> at
> org.apache.kylin.rest.service.QueryService$PreparedContextFactory.create(QueryService.java:1301)
> at
> org.apache.kylin.rest.service.QueryService$PreparedContextFactory.create(QueryService.java:1296)
> at
> org.apache.commons.pool2.BaseKeyedPooledObjectFactory.makeObject(BaseKeyedPooledObjectFactory.java:60)
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.create(GenericKeyedObjectPool.java:1064)
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:358)
> at
> org.apache.commons.pool2.impl.GenericKeyedObjectPool.borrowObject(GenericKeyedObjectPool.java:281)
> at
> org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:680)
> at
> org.apache.kylin.rest.service.QueryService.query(QueryService.java:219)
> at
> org.apache.kylin.rest.service.QueryService.queryAndUpdateCache(QueryService.java:480)
> at
> org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:440)
> at
> org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:378)
> at
> org.apache.kylin.rest.controller.QueryController.query(QueryController.java:93)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> at java.lang.reflect.Method.invoke(Method.java:498)
> at
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
> at
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> at
> com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:209)
> at
> com.thetransactioncompany.cors.CORSFilter.doFilter(CORSFilter.java:244)
> at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
> at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
> at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:219)
> at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
> at
> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:492)
> at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:165)
> at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:104)
> at
> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:1025)
> at
[jira] [Updated] (KYLIN-4477) Usage of "TLS" is insecure
[
https://issues.apache.org/jira/browse/KYLIN-4477?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen updated KYLIN-4477:
Sprint: Sprint 51
> Usage of "TLS" is insecure
> --
>
> Key: KYLIN-4477
> URL: https://issues.apache.org/jira/browse/KYLIN-4477
> Project: Kylin
> Issue Type: Improvement
>Reporter: Md Mahir Asef Kabir
>Assignee: Md Mahir Asef Kabir
>Priority: Major
> Fix For: v3.1.0
>
>
> *Vulnerability Description:* In
> “engine-mr/src/main/java/org/apache/kylin/engine/mr/common/DefaultSslProtocolSocketFactory.java”
> file the following code was written in
> {code:java}
> private static SSLContext createEasySSLContext()
> {code}
> method -
> {code:java}
> SSLContext context = SSLContext.getInstance("TLS");
> {code}
> The vulnerability is, using "TLS” as the argument to SSLContext.getInstance
> method.
> *Reason it’s vulnerable:* TLS 1.0 is vulnerable to man-in-the-middle attacks.
> For further reference, follow
> [this|https://www.comodo.com/e-commerce/ssl-certificates/tls-1-deprecation.php].
> *Suggested Fix:* Using
> {code:java}
> SSLContext.getInstance("TLSv1.3").
> {code}
> *Feedback:* Please select any of the options down below to help us get an
> idea about how you felt about the suggestion -
> # Liked it and will make the suggested changes
> # Liked it but happy with the existing version
> # Didn’t find the suggestion helpful
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KYLIN-4432) duplicated queries with sytax error take unexpect long time when lazy query enabled
[
https://issues.apache.org/jira/browse/KYLIN-4432?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen updated KYLIN-4432:
Sprint: Sprint 51
> duplicated queries with sytax error take unexpect long time when lazy query
> enabled
> ---
>
> Key: KYLIN-4432
> URL: https://issues.apache.org/jira/browse/KYLIN-4432
> Project: Kylin
> Issue Type: Bug
>Reporter: Congling Xia
>Assignee: Congling Xia
>Priority: Major
> Fix For: v3.1.0
>
>
> Hi! Our Kylin server becomes unresponsive recently. All query threads were
> being TIMED_WAITING so that no more thread in the thread pool of Tomcat could
> response new requests. The server logs are full of "Duplicate SQL request"s,
> and many requests retrying for so long time:
> {code:java}
> $ cat kylin.log.25 | grep 14db5fa0-78e0-4ca6-5d04-92995b2b8367-3133991 | head
> -3
> 2020-03-23 12:15:32,279 INFO [Query
> 14db5fa0-78e0-4ca6-5d04-92995b2b8367-3133991] service.QueryService:571 :
> Duplicated SQL request is running, waiting...
> 2020-03-23 12:15:32,379 INFO [Query
> 14db5fa0-78e0-4ca6-5d04-92995b2b8367-3133991] service.QueryService:571 :
> Duplicated SQL request is running, waiting...
> 2020-03-23 12:15:32,479 INFO [Query
> 14db5fa0-78e0-4ca6-5d04-92995b2b8367-3133991] service.QueryService:571 :
> Duplicated SQL request is running, waiting...
> $ cat kylin.log.25 | grep 14db5fa0-78e0-4ca6-5d04-92995b2b8367-3133991 | tail
> -3
> 2020-03-23 12:26:56,920 INFO [Query
> 14db5fa0-78e0-4ca6-5d04-92995b2b8367-3133991] service.QueryService:571 :
> Duplicated SQL request is running, waiting...
> 2020-03-23 12:26:57,020 INFO [Query
> 14db5fa0-78e0-4ca6-5d04-92995b2b8367-3133991] service.QueryService:571 :
> Duplicated SQL request is running, waiting...
> 2020-03-23 12:26:57,121 INFO [Query
> 14db5fa0-78e0-4ca6-5d04-92995b2b8367-3133991] service.QueryService:571 :
> Duplicated SQL request is running, waiting...
> {code}
> This can be re-produced easily:
> * enable lazy query (KYLIN-2897) by setting environment variable
> kylin.query.cache-enabled=true and kylin.query.lazy-query-enabled=true
> * send the same query with syntax error many times in several minutes
> The first query will quickly response with error message, and the following
> queries will executing for long time and will repeatly output "Duplicated SQL
> request is running, waiting...".
> The code in org.apache.kylin.rest.service.QueryService#searchQueryInCache
> indicates duplicated query will wait at most one minute with default setting
> of LazyQueryWaitingTimeoutMilliSeconds but in fact it waits quite longer than
> that.
> {code:java}
> while (response.isRunning()) {
> // Wait at most one minute
> if (System.currentTimeMillis() - response.getLazyQueryStartTime() >=
> getConfig()
> .getLazyQueryWaitingTimeoutMilliSeconds()) {
> cache.evict(sqlRequest.getCacheKey());
> return null;
> }
> logger.info("Duplicated SQL request is running, waiting...");
> try {
> Thread.sleep(100L);
> } catch (InterruptedException e) {
> }
> wrapper = cache.get(sqlRequest.getCacheKey());
> if (wrapper == null) {
> return null;
> }
> response = (SQLResponse) wrapper.get();
> if (response == null) {
> return null;
> }
> }
> {code}
> After some work-around, I find that the dummy response status remains
> unchanged after execution with exception. So the 2nd query will wait
> LazyQueryWaitingTimeoutMilliSeconds, evict the dummy response created by the
> 1st query from cache, and put a new dummy response into the cache. And the
> 3rd query may use the new dummy response's start time to compute timeout, and
> will wait for another LazyQueryWaitingTimeoutMilliSeconds.
> Suppose we send _n_ identical bad queries to the server quickly. The last
> query may wait for nearly _(n-1)_ times of
> LazyQueryWaitingTimeoutMilliSeconds. That's why the Web container's thread
> pool was exhausted – user program uses bad queries to check availability of
> Kylin service and retries many times in a short time.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KYLIN-4355) Add validation for cube re-assignmnet(Realtime OLAP)
[ https://issues.apache.org/jira/browse/KYLIN-4355?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-4355: Sprint: Sprint 52 > Add validation for cube re-assignmnet(Realtime OLAP) > > > Key: KYLIN-4355 > URL: https://issues.apache.org/jira/browse/KYLIN-4355 > Project: Kylin > Issue Type: Bug > Components: Real-time Streaming >Affects Versions: v3.0.0 >Reporter: Xiaoxiang Yu >Assignee: Xiaoxiang Yu >Priority: Minor > Fix For: v3.1.0 > > > Case 1. In assignment, specific partition can be assign to more than one > replica set, thus cause receiver consumed duplicate kafka message. > Case 2. In assignment, you can remove all partition for one repilca set, > which is make no sense at all. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4464) Query ... row_number over(order by c1) ... order by c2 ... get wrong order result
[
https://issues.apache.org/jira/browse/KYLIN-4464?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen updated KYLIN-4464:
Sprint: Sprint 51
> Query ... row_number over(order by c1) ... order by c2 ... get wrong order
> result
> -
>
> Key: KYLIN-4464
> URL: https://issues.apache.org/jira/browse/KYLIN-4464
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Reporter: Chao Long
>Assignee: Chao Long
>Priority: Major
> Fix For: v3.1.0
>
> Attachments: image-2020-04-20-18-19-40-925.png,
> image-2020-04-20-18-22-34-765.png, image-2020-05-07-10-32-39-983.png
>
>
> *SQL*
> {code:java}
> select t.*, row_number() over (order by t.lstg_format_name) as row_num from (
> select lstg_format_name, sum(price) as GMV from
> KYLIN_SALES
> where lstg_format_name is not null
> group by lstg_format_name
> ) as t
> order by t.GMV desc limit 5{quote}
> {code}
>
> *Kylin result (order by lstg_format_name)*
> !image-2020-04-20-18-19-40-925.png!
>
> *Calcite execution plan*
> {code:java}
> EXECUTION PLAN BEFORE REWRITE
> OLAPToEnumerableConverter
> OLAPWindowRel(window#0=[window(partition {} order by [0 DESC] rows between
> UNBOUNDED PRECEDING and CURRENT ROW aggs [ROW_NUMBER()])], ctx=[],
> groups=[[window(partition {} order by [0 DESC] rows between UNBOUNDED
> PRECEDING and CURRENT ROW aggs [ROW_NUMBER()])]])
> OLAPLimitRel(ctx=[], fetch=[5])
> OLAPSortRel(sort0=[$1], dir0=[DESC], ctx=[])
> OLAPAggregateRel(group=[{0}], GMV=[SUM($1)], ctx=[])
> OLAPProjectRel(LSTG_FORMAT_NAME=[$2], PRICE=[$5], ctx=[])
> OLAPFilterRel(condition=[IS NOT NULL($2)], ctx=[])
> OLAPTableScan(table=[[DEFAULT, KYLIN_SALES]], ctx=[],
> fields=[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12]])
> {code}
>
> *Hive result(order by GMV)*
> !image-2020-04-20-18-22-34-765.png!
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KYLIN-4419) Make it possible to change metadata in some cases without rebuilding data
[ https://issues.apache.org/jira/browse/KYLIN-4419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-4419: Sprint: Sprint 51 > Make it possible to change metadata in some cases without rebuilding data > - > > Key: KYLIN-4419 > URL: https://issues.apache.org/jira/browse/KYLIN-4419 > Project: Kylin > Issue Type: Improvement > Components: Metadata >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v3.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4394) Upgrade dependency version for several CVEs
[ https://issues.apache.org/jira/browse/KYLIN-4394?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-4394: Sprint: Sprint 51 > Upgrade dependency version for several CVEs > --- > > Key: KYLIN-4394 > URL: https://issues.apache.org/jira/browse/KYLIN-4394 > Project: Kylin > Issue Type: Improvement > Components: Security >Reporter: XuCongying >Assignee: Yaqian Zhang >Priority: Major > Fix For: v3.1.0 > > Attachments: apache-kylin_CVE-report.md > > > I noticed some of your libraries contained CVEs. I suggest updating their > versions to increase the security of your project. The following is a > detailed content. > * *Vulnerable Library Version:* org.scala-lang : scala-compiler : 2.11.0 > *CVE ID:* > [CVE-2017-15288|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-15288] > *Import Path:* engine-flink/pom.xml, kylin-it/pom.xml, engine-spark/pom.xml > *Suggested Safe Versions:* 2.11.12, 2.12.10, 2.12.4, 2.12.5, 2.12.6, 2.12.7, > 2.12.8, 2.12.9, 2.13.0, 2.13.0-M1, 2.13.0-M2, 2.13.0-M3, 2.13.0-M3-f73b161, > 2.13.0-M4, 2.13.0-M4-pre-20d3c21, 2.13.0-M5, 2.13.0-M5-1775dba, > 2.13.0-M5-5eef812, 2.13.0-M5-6e0cba7, 2.13.0-RC1, 2.13.0-RC2, 2.13.0-RC3, > 2.13.1 > * *Vulnerable Library Version:* org.apache.tomcat : tomcat-catalina : 7.0.91 > *CVE ID:* > [CVE-2016-8735|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-8735], > [CVE-2019-0232|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-0232], > [CVE-2016-6794|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-6794], > [CVE-2016-6816|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-6816], > [CVE-2016-8745|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-8745], > [CVE-2019-17563|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-17563] > *Import Path:* tomcat-ext/pom.xml, server/pom.xml, server-base/pom.xml > *Suggested Safe Versions:* 10.0.0-M1, 7.0.100, 9.0.30, 9.0.31 > * *Vulnerable Library Version:* com.h2database : h2 : 1.4.196 *CVE ID:* > [CVE-2018-10054|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-10054], > > [CVE-2018-14335|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-14335] > *Import Path:* server/pom.xml, kylin-it/pom.xml, source-jdbc/pom.xml, > source-hive/pom.xml, datasource-sdk/pom.xml *Suggested Safe Versions:* > 1.4.198, 1.4.199, 1.4.200 > * *Vulnerable Library Version:* com.google.guava : guava : 14.0 *CVE ID:* > [CVE-2018-10237|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-10237] > *Import Path:* core-storage/pom.xml, stream-receiver/pom.xml, > server/pom.xml, core-cube/pom.xml, core-metadata/pom.xml, jdbc/pom.xml, > tool-assembly/pom.xml, core-metrics/pom.xml *Suggested Safe Versions:* > 24.1.1-android, 24.1.1-jre, 25.0-android, 25.0-jre, 25.1-android, 25.1-jre, > 26.0-android, 26.0-jre, 27.0-android, 27.0-jre, 27.0.1-android, 27.0.1-jre, > 27.1-android, 27.1-jre, 28.0-android, 28.0-jre, 28.1-android, 28.1-jre, > 28.2-android, 28.2-jre > * *Vulnerable Library Version:* org.apache.hive.hcatalog : > hive-hcatalog-core : 1.2.1 *CVE ID:* > [CVE-2015-7521|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-7521] > *Import Path:* metrics-reporter-hive/pom.xml, assembly/pom.xml, > server/pom.xml, kylin-it/pom.xml, source-jdbc/pom.xml, source-hive/pom.xml, > server-base/pom.xml *Suggested Safe Versions:* 1.2.2, 2.0.0, 2.0.1, 2.1.0, > 2.1.1, 2.2.0, 2.3.0, 2.3.1, 2.3.2, 2.3.3, 2.3.4, 2.3.5, 2.3.6, 3.0.0, 3.1.0, > 3.1.1, 3.1.2 > * *Vulnerable Library Version:* org.apache.spark : spark-core_2.11 : 2.3.2 > *CVE ID:* > [CVE-2017-7678|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2017-7678], > [CVE-2018-3826|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-3826], > [CVE-2018-11770|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-11770], > > [CVE-2019-10099|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-10099] > *Import Path:* server/pom.xml, kylin-it/pom.xml, engine-spark/pom.xml, > storage-hbase/pom.xml *Suggested Safe Versions:* 2.4.5 > * *Vulnerable Library Version:* org.apache.kafka : kafka_2.11 : 1.0.0 *CVE > ID:* > [CVE-2018-1288|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2018-1288], > [CVE-2019-17196|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2019-17196] > *Import Path:* assembly/pom.xml, source-kafka/pom.xml, kylin-it/pom.xml, > stream-source-kafka/pom.xml, metrics-reporter-kafka/pom.xml *Suggested Safe > Versions:* 2.1.1, 2.2.0, 2.2.1, 2.2.2, 2.3.0, 2.3.1, 2.4.0 > * *Vulnerable Library Version:* org.apache.hive : hive-jdbc : 1.2.1 *CVE > ID:* > [CVE-2016-3083|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2016-3083], > [CVE-2015-7521|https://cve.mitre.org/cgi-bin/cvename.cgi?name=CVE-2015-7521], >
[jira] [Updated] (KYLIN-4281) Precisely set the data type of tuple expression
[ https://issues.apache.org/jira/browse/KYLIN-4281?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-4281: Sprint: Sprint 51 > Precisely set the data type of tuple expression > --- > > Key: KYLIN-4281 > URL: https://issues.apache.org/jira/browse/KYLIN-4281 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v3.1.0 > > > Previously to simplify the calculating of sum(case when), all of the binary > calculation is based on BigDecimal, which is not good for all, especially > when dealing with count(distinct case when), whose inner data type may be hll > or bitmap. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-3947) TS conflict when kylin update metadata in JDBC (MySQL)
[
https://issues.apache.org/jira/browse/KYLIN-3947?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17110305#comment-17110305
]
nichunen commented on KYLIN-3947:
-
Hi [~inigoml]
Any update?
> TS conflict when kylin update metadata in JDBC (MySQL)
> --
>
> Key: KYLIN-3947
> URL: https://issues.apache.org/jira/browse/KYLIN-3947
> Project: Kylin
> Issue Type: Bug
> Components: Security
>Affects Versions: v2.6.1
>Reporter: Iñigo Martinez
>Assignee: Yaqian Zhang
>Priority: Major
> Fix For: v3.1.0
>
>
> Unable to login in Kylin with admin user. Kylin cluster is composed by two
> nodes, one QUERY node and one ALL node. Our backend for metadata is MySQL
> (not HBASE). Kylin version: 2.6.1
> Solved with same workaround applied in related bug.
> {code:java}
> $KYLIN_HOME/bin/metastore.sh remove /user/admin
> {code}
>
> Tomcat exception follows:
> {code:java}
> Apr 10, 2019 7:30:29 AM org.apache.catalina.core.StandardWrapperValve invoke
> SEVERE: Servlet.service() for servlet [kylin] in context with path [/kylin]
> threw exception
> org.apache.kylin.common.persistence.WriteConflictException: Overwriting
> conflict /user/ADMIN, expect old TS 0, but it is 1554879164739
> at
> org.apache.kylin.common.persistence.JDBCResourceStore$5.execute(JDBCResourceStore.java:482)
> at
> org.apache.kylin.common.persistence.JDBCResourceStore.executeSql(JDBCResourceStore.java:100)
> at
> org.apache.kylin.common.persistence.JDBCResourceStore.checkAndPutResourceInternal(JDBCResourceStore.java:439)
> at
> org.apache.kylin.common.persistence.JDBCResourceStore.checkAndPutResourceImpl(JDBCResourceStore.java:427)
> at
> org.apache.kylin.common.persistence.ResourceStore$6.call(ResourceStore.java:442)
> at
> org.apache.kylin.common.persistence.ResourceStore$6.call(ResourceStore.java:439)
> at
> org.apache.kylin.common.persistence.ExponentialBackoffRetry.doWithRetry(ExponentialBackoffRetry.java:52)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceWithRetry(ResourceStore.java:439)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:427)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResource(ResourceStore.java:421)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResource(ResourceStore.java:401)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResource(ResourceStore.java:380)
> at
> org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192)
> at
> org.apache.kylin.rest.security.KylinUserManager.update(KylinUserManager.java:124)
> at
> org.apache.kylin.rest.service.KylinUserService.updateUser(KylinUserService.java:86)
> at
> org.apache.kylin.rest.service.KylinUserService.createUser(KylinUserService.java:78)
> at
> org.apache.kylin.rest.security.KylinAuthenticationProvider.authenticate(KylinAuthenticationProvider.java:116)
> at
> org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:174)
> at
> org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:199)
> at
> org.springframework.security.web.authentication.www.BasicAuthenticationFilter.doFilterInternal(BasicAuthenticationFilter.java:180)
> at
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
> org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
> org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
> org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:64)
> at
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
> org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
> at
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
> at
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)
> at
>
[jira] [Resolved] (KYLIN-4168) Fix sonar reported static code issues phase 2
[ https://issues.apache.org/jira/browse/KYLIN-4168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen resolved KYLIN-4168. - Resolution: Fixed > Fix sonar reported static code issues phase 2 > - > > Key: KYLIN-4168 > URL: https://issues.apache.org/jira/browse/KYLIN-4168 > Project: Kylin > Issue Type: Improvement >Reporter: Yaqian Zhang >Priority: Major > Fix For: v3.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4294) Add http api for metrics
[ https://issues.apache.org/jira/browse/KYLIN-4294?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-4294: Sprint: Sprint 51 > Add http api for metrics > - > > Key: KYLIN-4294 > URL: https://issues.apache.org/jira/browse/KYLIN-4294 > Project: Kylin > Issue Type: Improvement > Components: REST Service >Affects Versions: v2.6.0 >Reporter: xiang zhang >Assignee: xiang zhang >Priority: Minor > Fix For: v3.1.0 > > Attachments: kylin-4294-instruction.pdf > > > # Expose metrics through http api to facilitate the integration of some > external monitoring components, such as tsdb > # add a python script for tcollector -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KYLIN-4192) Build UHC dictionary with spark
[ https://issues.apache.org/jira/browse/KYLIN-4192?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen resolved KYLIN-4192. - Resolution: Fixed > Build UHC dictionary with spark > --- > > Key: KYLIN-4192 > URL: https://issues.apache.org/jira/browse/KYLIN-4192 > Project: Kylin > Issue Type: Improvement >Reporter: wangrupeng >Assignee: wangrupeng >Priority: Major > Fix For: v3.1.0 > > > To enable this, add following property to kylin.properties > kylin.engine.spark-udc-dictionary=true -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KYLIN-4317) Update doc for KYLIN-4104
[ https://issues.apache.org/jira/browse/KYLIN-4317?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen resolved KYLIN-4317. - Resolution: Fixed > Update doc for KYLIN-4104 > - > > Key: KYLIN-4317 > URL: https://issues.apache.org/jira/browse/KYLIN-4317 > Project: Kylin > Issue Type: Improvement >Reporter: weibin0516 >Assignee: weibin0516 >Priority: Major > Fix For: v3.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4315) Use metadata numRows in beeline client for quick row counting
[ https://issues.apache.org/jira/browse/KYLIN-4315?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-4315: Sprint: Sprint 51 > Use metadata numRows in beeline client for quick row counting > - > > Key: KYLIN-4315 > URL: https://issues.apache.org/jira/browse/KYLIN-4315 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: Congling Xia >Assignee: Congling Xia >Priority: Major > Fix For: v3.1.0 > > > Hi, I find that in `BeelineHiveClient`, method `getHiveTableRows` uses > "select count(*) from " for table row counting. The method is > invoked in flat intermediate table redistribution step in cube building. > This stats can be loaded in metastore. It costs much less time than scanning > all rows in Hive table. Since intermediate tables are created and inserted by > Kylin, statistics will be automatically calculated and stored in metastore > when > `[hive.stats.autogather|https://cwiki.apache.org/confluence/display/Hive/Configuration+Properties#ConfigurationProperties-hive.stats.autogather]` > is enabled (which is the default setting for Hive). > ref Hive wiki for more detail about `numRows` stats: > [https://cwiki.apache.org/confluence/display/Hive/StatsDev#StatsDev-ExistingTables%E2%80%93ANALYZE] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4330) use nrt streaming build for kafka data, can we use filter function when i set model desinger
[ https://issues.apache.org/jira/browse/KYLIN-4330?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-4330: Sprint: Sprint 51 > use nrt streaming build for kafka data, can we use filter function when i set > model desinger > - > > Key: KYLIN-4330 > URL: https://issues.apache.org/jira/browse/KYLIN-4330 > Project: Kylin > Issue Type: New Feature > Components: NRT Streaming >Affects Versions: v2.6.4 > Environment: 阿里云 centos 7 hadoop 2.8.5 >Reporter: kangkang >Priority: Major > Labels: FIlter, model > Fix For: v3.1.0 > > Attachments: 4561578452903_.pic_hd.jpg > > Original Estimate: 336h > Remaining Estimate: 336h > > when i use nrt streaming build for kafka data, can we use filter function > when i set model desinger -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4352) A empty segment will cause incomplete query result in Realtime OLAP
[ https://issues.apache.org/jira/browse/KYLIN-4352?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-4352: Sprint: Sprint 52 > A empty segment will cause incomplete query result in Realtime OLAP > --- > > Key: KYLIN-4352 > URL: https://issues.apache.org/jira/browse/KYLIN-4352 > Project: Kylin > Issue Type: Bug > Components: Real-time Streaming >Affects Versions: v3.0.0 >Reporter: Xiaoxiang Yu >Assignee: Xiaoxiang Yu >Priority: Major > Fix For: v3.1.0 > > > When you query hit a streaming cube, you query will divided into two part, > one historical part, one realtime part. > The boundary of two part is decided by the latest READY segment's *End Time* > . But when you have a segment which segment range is from future(created by > mistake), it will cause all following query get incomplete result. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4351) Provided a metadata repair tool for Realtime OLAP
[ https://issues.apache.org/jira/browse/KYLIN-4351?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-4351: Sprint: Sprint 52 > Provided a metadata repair tool for Realtime OLAP > - > > Key: KYLIN-4351 > URL: https://issues.apache.org/jira/browse/KYLIN-4351 > Project: Kylin > Issue Type: Improvement > Components: Real-time Streaming >Reporter: Xiaoxiang Yu >Assignee: Xiaoxiang Yu >Priority: Major > Fix For: v3.1.0 > > > The metatdata of Realtime Receiver cluster is located in Zookeeper, something > it maybe removed by accident, thus cause the fail of query, and build. > In that case, user have to disable and re-enable the streaming cube. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-3121) NPE while executing a query with two left outer joins and floating point expressions on nullable fields
[ https://issues.apache.org/jira/browse/KYLIN-3121?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-3121: Sprint: Sprint 51 > NPE while executing a query with two left outer joins and floating point > expressions on nullable fields > --- > > Key: KYLIN-3121 > URL: https://issues.apache.org/jira/browse/KYLIN-3121 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v2.2.0 > Environment: HDP 2.5.6, Kylin 2.2.0 >Reporter: Vsevolod Ostapenko >Assignee: nichunen >Priority: Major > Fix For: v3.1.0 > > > Queries that include two (or more) left outer joins and contain floating > point expressions that operate on the fields that contain integer NULL values > (due to left outer join) fail in-flight with NullPointerExceptions. > As an example, the following query generates NPE on either of the two > expressions: > * 100.0 * t2.media_gap_call_count > * 1.0 * NULLIF(t1.active_call_count, 0) > with > t1 > as > ( > select > d1.cell_name, > count(distinct a1.call_id) as active_call_count > from > zetticsdw.a_vl_hourly_v a1 > inner join > zetticsdw.d_cell_v d1 > on > a1.cell_key = d1.cell_key > where > d1.region_3 = 'Mumbai' > and > a1.thedate = '20171011' > and > a1.thehour = '00' > and > a1.active_call_flg = 1 > group by > d1.cell_name > ), > t2 > as > ( > select > d1.cell_name, > count(distinct a1.call_id) as media_gap_call_count > from > zetticsdw.a_vl_hourly_v a1 > inner join > zetticsdw.d_cell_v d1 > on > a1.cell_key = d1.cell_key > where > d1.region_3 = 'Mumbai' > and > a1.thedate='20171011' > and > a1.thehour = '00' > and > a1.media_gap_call_flg = 1 > group by > d1.cell_name > ) > , > t3 > as > ( > select > d1.cell_name, > sum(a1.ow_call_flg) one_way_call_count, > sum(a1.succ_call_flg) successfull_call_count > from > zetticsdw.a_vl_hourly_v a1 > inner join > zetticsdw.d_cell_v d1 > on > a1.cell_key = d1.cell_key > where > d1.region_3 = 'Mumbai' > and > a1.thedate='20171011' > and > a1.thehour = '00' > group by > d1.cell_name > ) > select >t3.cell_name, >t1.active_call_count, >t2.media_gap_call_count, >t3.one_way_call_count, >t3.successfull_call_count, >-- 100 * t2.media_gap_call_count nom, -- > works >-- 1 * NULLIF(t1.active_call_count, 0) denom-- > works >100.0 * t2.media_gap_call_count nom, -- fails, > NPE of one kind >1.0 * NULLIF(t1.active_call_count, 0) denom -- fails, > NPE of different kind >-- 100.0 * COALESCE(t2.media_gap_call_count, 0) nom,-- > works >-- 1.0 * CAST(NULLIF(t1.active_call_count, 0) as DOUBLE) denom -- > works > from >t3 > left outer join >t1 > on >t3.cell_name = t1.cell_name > left outer join >t2 > on >t3.cell_name = t2.cell_name > In the first case (multiplication of an integer field with a NULL value and a > double) kylin log contains a stack trace similar to the following: > null > at org.apache.calcite.avatica.Helper.createException(Helper.java:56) > at org.apache.calcite.avatica.Helper.createException(Helper.java:41) > at > org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156) > at > org.apache.calcite.avatica.AvaticaStatement.executeQuery(AvaticaStatement.java:218) > at > org.apache.kylin.rest.service.QueryService.execute(QueryService.java:834) > at > org.apache.kylin.rest.service.QueryService.queryWithSqlMassage(QueryService.java:561) > at > org.apache.kylin.rest.service.QueryService.query(QueryService.java:181) > at > org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:415) > at > org.apache.kylin.rest.controller.QueryController.query(QueryController.java:78) > at sun.reflect.GeneratedMethodAccessor545.invoke(Unknown Source) > at > sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) > at java.lang.reflect.Method.invoke(Method.java:606) > at > org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205) > at > org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133) > at >
[jira] [Updated] (KYLIN-4120) Failed to query "select * from {lookup}" if a lookup table joined in two different models
[
https://issues.apache.org/jira/browse/KYLIN-4120?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
nichunen updated KYLIN-4120:
Sprint: Sprint 52
> Failed to query "select * from {lookup}" if a lookup table joined in two
> different models
> -
>
> Key: KYLIN-4120
> URL: https://issues.apache.org/jira/browse/KYLIN-4120
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v3.0.0-alpha2
>Reporter: nichunen
>Assignee: Xiaoxiang Yu
>Priority: Major
> Fix For: v3.1.0
>
>
> Steps to reproduce:
> 1. Create two models with same lookup table (create snapshot)
> 2. Define different columns as dimensions in the two models
> (Etc: in model ma, with column lookup.c1, lookup.c2; in model mb, with
> column lookup.c2, lookup.c3)
> 3. Create two cubes with the two models, and build the 2 cubes
> 4. Run query with "select * from {lookup}" with the 2 cubes in ready status,
> it should be failed with exception message like
> "
> No model found for OLAPContext,
> CUBE_NOT_CONTAIN_ALL_COLUMN[1_39bd1072:SSB.DATES.D_DAYNUMINWEEK,
> 1_39bd1072:SSB.DATES.D_DAYOFWEEK, 1_39bd1072:SSB.DATES.D_DATE,
> 1_39bd1072:SSB.DATES.D_DAYNUMINMONTH, 1_39bd1072:SSB.DATES.D_DAYNUMINYEAR,
> 1_39bd1072:SSB.DATES.D_MONTH, 1_39bd1072:SSB.DATES.D_MONTHNUMINYEAR],
> CUBE_NOT_READY, CUBE_NOT_READY,
> CUBE_NOT_CONTAIN_ALL_COLUMN[1_39bd1072:SSB.DATES.D_YEARMONTHNUM,
> 1_39bd1072:SSB.DATES.D_YEAR], rel#218:OLAPTableScan.OLAP.[](table=[SSB,
> DATES],ctx=,fields=[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11]) while executing
> SQL: "select * from DATES LIMIT 5"
> "
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Resolved] (KYLIN-4418) Bug fix for ShardingHash.getShard in HBaseLookupRowEncoder
[ https://issues.apache.org/jira/browse/KYLIN-4418?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen resolved KYLIN-4418. - Resolution: Fixed > Bug fix for ShardingHash.getShard in HBaseLookupRowEncoder > -- > > Key: KYLIN-4418 > URL: https://issues.apache.org/jira/browse/KYLIN-4418 > Project: Kylin > Issue Type: Sub-task > Components: Job Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v3.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KYLIN-4417) Use hash rather than random for ConvergeCuboidDataPartitioner to avoid potential issue
[ https://issues.apache.org/jira/browse/KYLIN-4417?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen resolved KYLIN-4417. - Resolution: Fixed > Use hash rather than random for ConvergeCuboidDataPartitioner to avoid > potential issue > -- > > Key: KYLIN-4417 > URL: https://issues.apache.org/jira/browse/KYLIN-4417 > Project: Kylin > Issue Type: Sub-task > Components: Job Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Minor > Fix For: v3.1.0 > > > If use random, similar issue will occur KYLIN-3388 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4414) Make bulkload locality possible
[ https://issues.apache.org/jira/browse/KYLIN-4414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen updated KYLIN-4414: Fix Version/s: v3.1.0 > Make bulkload locality possible > --- > > Key: KYLIN-4414 > URL: https://issues.apache.org/jira/browse/KYLIN-4414 > Project: Kylin > Issue Type: Sub-task > Components: Job Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Minor > Fix For: v3.1.0 > > > # Firstly, we need to do write data directly to the hbase cluster > # Then we can apply patch http://issues.apache.org/jira/browse/HBASE-12596 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KYLIN-4414) Make bulkload locality possible
[ https://issues.apache.org/jira/browse/KYLIN-4414?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen resolved KYLIN-4414. - Resolution: Fixed > Make bulkload locality possible > --- > > Key: KYLIN-4414 > URL: https://issues.apache.org/jira/browse/KYLIN-4414 > Project: Kylin > Issue Type: Sub-task > Components: Job Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Minor > Fix For: v3.1.0 > > > # Firstly, we need to do write data directly to the hbase cluster > # Then we can apply patch http://issues.apache.org/jira/browse/HBASE-12596 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KYLIN-4415) HTable Creation with Retry
[ https://issues.apache.org/jira/browse/KYLIN-4415?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] nichunen resolved KYLIN-4415. - Resolution: Fixed > HTable Creation with Retry > -- > > Key: KYLIN-4415 > URL: https://issues.apache.org/jira/browse/KYLIN-4415 > Project: Kylin > Issue Type: Sub-task > Components: Job Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v3.1.0 > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
