[jira] [Commented] (KYLIN-3649) segment region count and size are not correct when using mysql as Kylin metadata storage
[
https://issues.apache.org/jira/browse/KYLIN-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668186#comment-16668186
]
Shaofeng SHI commented on KYLIN-3649:
-
Patch merged; Also found the StorageCleanupJob has the same issue, also fixed
and merged in this commit. Thank you Lingang, welcome to contribute more to
Kylin!
> segment region count and size are not correct when using mysql as Kylin
> metadata storage
> -
>
> Key: KYLIN-3649
> URL: https://issues.apache.org/jira/browse/KYLIN-3649
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Affects Versions: v2.5.0
>Reporter: Lingang Deng
>Assignee: Lingang Deng
>Priority: Major
> Fix For: v2.5.1
>
> Attachments: 0001-KYLIN-3649.patch
>
>
> As titles, segment region count and size are not correct.
> {code:java}
> if ("hbase".equals(getConfig().getMetadataUrl().getScheme())) {
> try {
> logger.debug("Loading HTable info " + cubeName + ", " + tableName);
> // use reflection to isolate NoClassDef errors when HBase is not
> available
> hr = (HBaseResponse)
> Class.forName("org.apache.kylin.rest.service.HBaseInfoUtil")//
> .getMethod("getHBaseInfo", new Class[] { String.class,
> KylinConfig.class })//
> .invoke(null, tableName, this.getConfig());
> } catch (Throwable e) {
> throw new IOException(e);
> }
> }
> {code}
> Judgement is not valid when using mysql as Kylin metadata storage.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3649) segment region count and size are not correct when using mysql as Kylin metadata storage
[
https://issues.apache.org/jira/browse/KYLIN-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668184#comment-16668184
]
ASF subversion and git services commented on KYLIN-3649:
Commit bfb90556fbd88ca2cf33f7702786ec3e3fee777a in kylin's branch
refs/heads/2.5.x from chitin
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=bfb9055 ]
KYLIN-3649 segment region count and size are not correct when using mysql as
Kylin metadata storage
> segment region count and size are not correct when using mysql as Kylin
> metadata storage
> -
>
> Key: KYLIN-3649
> URL: https://issues.apache.org/jira/browse/KYLIN-3649
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Affects Versions: v2.5.0
>Reporter: Lingang Deng
>Assignee: Lingang Deng
>Priority: Major
> Fix For: v2.5.1
>
> Attachments: 0001-KYLIN-3649.patch
>
>
> As titles, segment region count and size are not correct.
> {code:java}
> if ("hbase".equals(getConfig().getMetadataUrl().getScheme())) {
> try {
> logger.debug("Loading HTable info " + cubeName + ", " + tableName);
> // use reflection to isolate NoClassDef errors when HBase is not
> available
> hr = (HBaseResponse)
> Class.forName("org.apache.kylin.rest.service.HBaseInfoUtil")//
> .getMethod("getHBaseInfo", new Class[] { String.class,
> KylinConfig.class })//
> .invoke(null, tableName, this.getConfig());
> } catch (Throwable e) {
> throw new IOException(e);
> }
> }
> {code}
> Judgement is not valid when using mysql as Kylin metadata storage.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3649) segment region count and size are not correct when using mysql as Kylin metadata storage
[
https://issues.apache.org/jira/browse/KYLIN-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668185#comment-16668185
]
ASF subversion and git services commented on KYLIN-3649:
Commit 43c7207524f8507e1a47552d71186e6c8d7254c6 in kylin's branch
refs/heads/master from chitin
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=43c7207 ]
KYLIN-3649 segment region count and size are not correct when using mysql as
Kylin metadata storage
> segment region count and size are not correct when using mysql as Kylin
> metadata storage
> -
>
> Key: KYLIN-3649
> URL: https://issues.apache.org/jira/browse/KYLIN-3649
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Affects Versions: v2.5.0
>Reporter: Lingang Deng
>Assignee: Lingang Deng
>Priority: Major
> Fix For: v2.5.1
>
> Attachments: 0001-KYLIN-3649.patch
>
>
> As titles, segment region count and size are not correct.
> {code:java}
> if ("hbase".equals(getConfig().getMetadataUrl().getScheme())) {
> try {
> logger.debug("Loading HTable info " + cubeName + ", " + tableName);
> // use reflection to isolate NoClassDef errors when HBase is not
> available
> hr = (HBaseResponse)
> Class.forName("org.apache.kylin.rest.service.HBaseInfoUtil")//
> .getMethod("getHBaseInfo", new Class[] { String.class,
> KylinConfig.class })//
> .invoke(null, tableName, this.getConfig());
> } catch (Throwable e) {
> throw new IOException(e);
> }
> }
> {code}
> Judgement is not valid when using mysql as Kylin metadata storage.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3649) segment region count and size are not correct when using mysql as Kylin metadata storage
[
https://issues.apache.org/jira/browse/KYLIN-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lingang Deng updated KYLIN-3649:
Attachment: (was: KYLIN-3649.patch)
> segment region count and size are not correct when using mysql as Kylin
> metadata storage
> -
>
> Key: KYLIN-3649
> URL: https://issues.apache.org/jira/browse/KYLIN-3649
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Affects Versions: v2.5.0
>Reporter: Lingang Deng
>Assignee: Lingang Deng
>Priority: Major
> Fix For: v2.5.1
>
> Attachments: 0001-KYLIN-3649.patch
>
>
> As titles, segment region count and size are not correct.
> {code:java}
> if ("hbase".equals(getConfig().getMetadataUrl().getScheme())) {
> try {
> logger.debug("Loading HTable info " + cubeName + ", " + tableName);
> // use reflection to isolate NoClassDef errors when HBase is not
> available
> hr = (HBaseResponse)
> Class.forName("org.apache.kylin.rest.service.HBaseInfoUtil")//
> .getMethod("getHBaseInfo", new Class[] { String.class,
> KylinConfig.class })//
> .invoke(null, tableName, this.getConfig());
> } catch (Throwable e) {
> throw new IOException(e);
> }
> }
> {code}
> Judgement is not valid when using mysql as Kylin metadata storage.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3649) segment region count and size are not correct when using mysql as Kylin metadata storage
[
https://issues.apache.org/jira/browse/KYLIN-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lingang Deng updated KYLIN-3649:
Attachment: 0001-KYLIN-3649.patch
> segment region count and size are not correct when using mysql as Kylin
> metadata storage
> -
>
> Key: KYLIN-3649
> URL: https://issues.apache.org/jira/browse/KYLIN-3649
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Affects Versions: v2.5.0
>Reporter: Lingang Deng
>Assignee: Lingang Deng
>Priority: Major
> Fix For: v2.5.1
>
> Attachments: 0001-KYLIN-3649.patch
>
>
> As titles, segment region count and size are not correct.
> {code:java}
> if ("hbase".equals(getConfig().getMetadataUrl().getScheme())) {
> try {
> logger.debug("Loading HTable info " + cubeName + ", " + tableName);
> // use reflection to isolate NoClassDef errors when HBase is not
> available
> hr = (HBaseResponse)
> Class.forName("org.apache.kylin.rest.service.HBaseInfoUtil")//
> .getMethod("getHBaseInfo", new Class[] { String.class,
> KylinConfig.class })//
> .invoke(null, tableName, this.getConfig());
> } catch (Throwable e) {
> throw new IOException(e);
> }
> }
> {code}
> Judgement is not valid when using mysql as Kylin metadata storage.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3652) Segment auto-merge should not be triggered in some cases
[ https://issues.apache.org/jira/browse/KYLIN-3652?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16668074#comment-16668074 ] Chao Long commented on KYLIN-3652: -- I'd like to fix this issue. > Segment auto-merge should not be triggered in some cases > > > Key: KYLIN-3652 > URL: https://issues.apache.org/jira/browse/KYLIN-3652 > Project: Kylin > Issue Type: Improvement > Components: Metadata >Reporter: Shaofeng SHI >Priority: Major > > The segment auto-merge will be triggered whenever there is segment change: > add, remove, etc. This causes some problem, for example, the merging job has > problem, and the user wants to do something for an existing segment. But once > he discarding the auto-generated merging job, a new job is generated, causing > an endless loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3652) Segment auto-merge should not be triggered in some cases
Shaofeng SHI created KYLIN-3652: --- Summary: Segment auto-merge should not be triggered in some cases Key: KYLIN-3652 URL: https://issues.apache.org/jira/browse/KYLIN-3652 Project: Kylin Issue Type: Improvement Components: Metadata Reporter: Shaofeng SHI The segment auto-merge will be triggered whenever there is segment change: add, remove, etc. This causes some problem, for example, the merging job has problem, and the user wants to do something for an existing segment. But once he discarding the auto-generated merging job, a new job is generated, causing an endless loop. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KYLIN-3649) segment region count and size are not correct when using mysql as Kylin metadata storage
[
https://issues.apache.org/jira/browse/KYLIN-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI reassigned KYLIN-3649:
---
Assignee: Lingang Deng
> segment region count and size are not correct when using mysql as Kylin
> metadata storage
> -
>
> Key: KYLIN-3649
> URL: https://issues.apache.org/jira/browse/KYLIN-3649
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Affects Versions: v2.5.0
>Reporter: Lingang Deng
>Assignee: Lingang Deng
>Priority: Major
> Fix For: v2.5.1
>
> Attachments: KYLIN-3649.patch
>
>
> As titles, segment region count and size are not correct.
> {code:java}
> if ("hbase".equals(getConfig().getMetadataUrl().getScheme())) {
> try {
> logger.debug("Loading HTable info " + cubeName + ", " + tableName);
> // use reflection to isolate NoClassDef errors when HBase is not
> available
> hr = (HBaseResponse)
> Class.forName("org.apache.kylin.rest.service.HBaseInfoUtil")//
> .getMethod("getHBaseInfo", new Class[] { String.class,
> KylinConfig.class })//
> .invoke(null, tableName, this.getConfig());
> } catch (Throwable e) {
> throw new IOException(e);
> }
> }
> {code}
> Judgement is not valid when using mysql as Kylin metadata storage.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3517) Couldn't update coprocessor on HBase 2.0
[
https://issues.apache.org/jira/browse/KYLIN-3517?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667995#comment-16667995
]
ASF subversion and git services commented on KYLIN-3517:
Commit aba864ffdf57fbfbb12142d519d53b09bc4b46b9 in kylin's branch
refs/heads/2.5.x-hadoop3.1 from [~caolijun1166]
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=aba864f ]
KYLIN-3517 Upadate coprocessor on HBase2.0 is avaliable.
Signed-off-by: shaofengshi
> Couldn't update coprocessor on HBase 2.0
>
>
> Key: KYLIN-3517
> URL: https://issues.apache.org/jira/browse/KYLIN-3517
> Project: Kylin
> Issue Type: Bug
> Components: Storage - HBase
>Reporter: Shaofeng SHI
>Assignee: Lijun Cao
>Priority: Major
> Fix For: v2.5.0
>
>
> On HDP 3.0, run update coprocessor, got this error:
>
> {code:java}
> 2018-08-28 00:24:26,683 ERROR [pool-7-thread-1] util.DeployCoprocessorCLI:383
> : Error processing KYLIN_O9JRT8XOQ9
> java.lang.UnsupportedOperationException: HTableDescriptor is read-only
> at
> org.apache.hadoop.hbase.client.ImmutableHTableDescriptor.getDelegateeForModification(ImmutableHTableDescriptor.java:59)
> at
> org.apache.hadoop.hbase.HTableDescriptor.removeCoprocessor(HTableDescriptor.java:768)
> at
> org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI.resetCoprocessor(DeployCoprocessorCLI.java:300)
> at
> org.apache.kylin.storage.hbase.util.DeployCoprocessorCLI$ResetCoprocessorWorker.run(DeployCoprocessorCLI.java:375)
> at
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
> at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
> at java.lang.Thread.run(Thread.java:745)
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-2565) Support Hadoop 3.0
[ https://issues.apache.org/jira/browse/KYLIN-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667993#comment-16667993 ] ASF subversion and git services commented on KYLIN-2565: Commit 2eecd49d37d4da9d6cecccdfafe59406c481c4d5 in kylin's branch refs/heads/2.5.x-hadoop3.1 from shaofengshi [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=2eecd49 ] KYLIN-2565 upgrade to hadoop 3.0 hbase 2.0, pass UT > Support Hadoop 3.0 > -- > > Key: KYLIN-2565 > URL: https://issues.apache.org/jira/browse/KYLIN-2565 > Project: Kylin > Issue Type: New Feature > Components: Job Engine >Reporter: Wang Cheng >Assignee: Shaofeng SHI >Priority: Major > Fix For: v2.5.0 > > > Hadoop3.0-alpha is released, Kylin should also keep compatible with it. Below > is the Hadoop3.0 components requirements: > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0.0+release -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3651) JDBCResourceStore doesn't list all resources
[
https://issues.apache.org/jira/browse/KYLIN-3651?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667990#comment-16667990
]
ASF subversion and git services commented on KYLIN-3651:
Commit d1d87065de99e1bf8a3927165a899ab32122d9da in kylin's branch
refs/heads/2.5.x-hadoop3.1 from shaofengshi
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=d1d8706 ]
KYLIN-3651 JDBCResourceStore doesn't list all resources
> JDBCResourceStore doesn't list all resources
>
>
> Key: KYLIN-3651
> URL: https://issues.apache.org/jira/browse/KYLIN-3651
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Affects Versions: v2.5.0
>Reporter: Shaofeng SHI
>Assignee: Shaofeng SHI
>Priority: Major
> Fix For: v2.5.1
>
>
> The JDBC resource store persists metadata into two tables:
> # kylin_default_instance
> # kylin_default_instance_log
>
> The metadata entry like "/execute_output" will be saved to the _log table;
> But the "list" operation on "/" only lists the subfolders in the first table,
> no in the second table. This will cause partial metadata copy when use
> #ResourceTool.copy.
>
> {code:java}
> ./bin/metastore.sh list /
> [/UUID, /acl, /cube, /cube_desc, /cube_statistics, /dict, /execute, /kafka,
> /model_desc, /project, /streaming, /table, /table_acl, /table_snapshot, /user]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-2565) Support Hadoop 3.0
[ https://issues.apache.org/jira/browse/KYLIN-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667992#comment-16667992 ] ASF subversion and git services commented on KYLIN-2565: Commit 581a5295b86e6ce022b359d1c181528cba2d8006 in kylin's branch refs/heads/2.5.x-hadoop3.1 from [~blue20080] [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=581a529 ] KYLIN-2565, upgrade to Hadoop3.0 > Support Hadoop 3.0 > -- > > Key: KYLIN-2565 > URL: https://issues.apache.org/jira/browse/KYLIN-2565 > Project: Kylin > Issue Type: New Feature > Components: Job Engine >Reporter: Wang Cheng >Assignee: Shaofeng SHI >Priority: Major > Fix For: v2.5.0 > > > Hadoop3.0-alpha is released, Kylin should also keep compatible with it. Below > is the Hadoop3.0 components requirements: > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0.0+release -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3562) TS conflict when kylin update metadata in HBase
[
https://issues.apache.org/jira/browse/KYLIN-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667989#comment-16667989
]
ASF subversion and git services commented on KYLIN-3562:
Commit a084e7e25bac5755ac9607a956ceb10247a3b28e in kylin's branch
refs/heads/2.5.x-hadoop3.1 from tttMelody
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=a084e7e ]
KYLIN-3562, optimization the logic that updating the user when user logged in.
> TS conflict when kylin update metadata in HBase
> ---
>
> Key: KYLIN-3562
> URL: https://issues.apache.org/jira/browse/KYLIN-3562
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.4.0
>Reporter: Lingang Deng
>Assignee: Jiatao Tao
>Priority: Major
> Fix For: v2.4.2, v2.5.1
>
> Attachments: image-2018-09-17-16-40-56-212.png,
> image-2018-09-25-15-03-51-009.png, image-2018-09-25-16-43-50-277.png
>
>
> Error log was as follows,
> {code:java}
> org.apache.kylin.common.persistence.WriteConflictException: Overwriting
> conflict /user/admin, expect old TS 1536928877043, but it is 1536928907207
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:325)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:318)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:303)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:282)
> at
> org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192){code}
>
> what disturbs me the most was that the error was happened several hours,
> then my all build job and query job failed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3518) Coprocessor reports NPE when execute a query on HBase 2.0
[
https://issues.apache.org/jira/browse/KYLIN-3518?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667994#comment-16667994
]
ASF subversion and git services commented on KYLIN-3518:
Commit 6ae921d139fb140dbda192fc897398506b38fc0a in kylin's branch
refs/heads/2.5.x-hadoop3.1 from [~caolijun1166]
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=6ae921d ]
KYLIN-3518 Fix Coprocessor NPE problem on hbase 2
> Coprocessor reports NPE when execute a query on HBase 2.0
> -
>
> Key: KYLIN-3518
> URL: https://issues.apache.org/jira/browse/KYLIN-3518
> Project: Kylin
> Issue Type: Bug
> Components: Storage - HBase
>Reporter: Shaofeng SHI
>Assignee: Lijun Cao
>Priority: Major
> Fix For: v2.5.0
>
>
> On HDP 3.0, build a cube and then run a simple count query, NPE occurred:
>
> {code:java}
> 2018-08-28 01:30:16,969 DEBUG [Query f7bf8004-b516-e372-18df-0d507075d471-71]
> v2.CubeHBaseRPC:315 : hbase.rpc.timeout = 9 ms, use 81000 ms as timeout
> for coprocessor
> 2018-08-28 01:30:16,983 INFO [Query f7bf8004-b516-e372-18df-0d507075d471-71]
> v2.CubeHBaseEndpointRPC:141 : Serialized scanRequestBytes 522 bytes,
> rawScanBytesString 44 bytes
> 2018-08-28 01:30:16,984 INFO [Query f7bf8004-b516-e372-18df-0d507075d471-71]
> v2.CubeHBaseEndpointRPC:143 : The scan 67b41fc6 for segment
> kylin_sales_cube_clone[2012010100_2013010100] is as below with 1
> separate raw scans, shard part of start/end key is set to 0
> 2018-08-28 01:30:16,991 INFO [Query f7bf8004-b516-e372-18df-0d507075d471-71]
> v2.CubeHBaseRPC:288 : Visiting hbase table KYLIN_5Q088VO5I0: cuboid require
> post aggregation, from 0 to 16384 Start:
> \x00\x00\x00\x00\x00\x00\x00\x00\x40\x00\x00\x00\x00
> (\x00\x00\x00\x00\x00\x00\x00\x00@\x00\x00\x00\x00) Stop:
> \x00\x00\x00\x00\x00\x00\x00\x00\x40\x00\xFF\xFF\xFF\x00
> (\x00\x00\x00\x00\x00\x00\x00\x00@\x00\xFF\xFF\xFF\x00), No Fuzzy Key
> 2018-08-28 01:30:16,991 DEBUG [Query f7bf8004-b516-e372-18df-0d507075d471-71]
> v2.CubeHBaseEndpointRPC:148 : Submitting rpc to 1 shards starting from shard
> 0, scan range count 1
> 2018-08-28 01:30:17,010 INFO [Query f7bf8004-b516-e372-18df-0d507075d471-71]
> common.KylinConfig:332 : Loading kylin-defaults.properties from
> file:/root/shaofengshi/apache-kylin-2.5.0-SNAPSHOT-bin/tomcat/webapps/kylin/WEB-INF/lib/kylin-core-common-2.5.0-SNAPSHOT.jar!/kylin-defaults.properties
> 2018-08-28 01:30:17,033 DEBUG [Query f7bf8004-b516-e372-18df-0d507075d471-71]
> common.KylinConfig:291 : KYLIN_CONF property was not set, will seek
> KYLIN_HOME env variable
> 2018-08-28 01:30:17,051 INFO [pool-14-thread-1] hbase.HBaseConnection:110 :
> Creating coprocessor thread pool with max of 2048, core of 2048
> 2018-08-28 01:30:17,094 INFO [Query f7bf8004-b516-e372-18df-0d507075d471-71]
> gtrecord.SequentialCubeTupleIterator:73 : Using SortedIteratorMergerWithLimit
> to merge segment results
> 2018-08-28 01:30:17,097 DEBUG [Query f7bf8004-b516-e372-18df-0d507075d471-71]
> enumerator.OLAPEnumerator:117 : return TupleIterator...
> 2018-08-28 01:30:21,607 INFO [kylin-coproc--pool9-t1]
> client.RpcRetryingCallerImpl:134 : Call exception, tries=6, retries=6,
> started=4410 ms ago, cancelled=false, msg=java.io.IOException
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:468)
> at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:131)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:324)
> at org.apache.hadoop.hbase.ipc.RpcExecutor$Handler.run(RpcExecutor.java:304)
> Caused by: java.lang.NullPointerException
> at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.CubeVisitService.visitCube(CubeVisitService.java:253)
> at
> org.apache.kylin.storage.hbase.cube.v2.coprocessor.endpoint.generated.CubeVisitProtos$CubeVisitService.callMethod(CubeVisitProtos.java:)
> at org.apache.hadoop.hbase.regionserver.HRegion.execService(HRegion.java:8032)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execServiceOnRegion(RSRpcServices.java:2426)
> at
> org.apache.hadoop.hbase.regionserver.RSRpcServices.execService(RSRpcServices.java:2408)
> at
> org.apache.hadoop.hbase.shaded.protobuf.generated.ClientProtos$ClientService$2.callBlockingMethod(ClientProtos.java:42010)
> at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:409)
> ... 3 more
> , details=row '' on table 'KYLIN_5Q088VO5I0' at
> region=KYLIN_5Q088VO5I0,,1535417272444.27b82cb4702db4557a98b9a7e60b7692.,
> hostname=ignite03.com,16020,1534313612401, seqNum=2
> 2018-08-28 01:30:25,633 INFO [kylin-coproc--pool9-t1]
> client.RpcRetryingCallerImpl:134 : Call exception, tries=7, retries=7,
> started=8436 ms ago, cancelled=false, msg=java.io.IOException
> at
[jira] [Commented] (KYLIN-2565) Support Hadoop 3.0
[ https://issues.apache.org/jira/browse/KYLIN-2565?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667996#comment-16667996 ] ASF subversion and git services commented on KYLIN-2565: Commit f411935b4a4dcf83072b54d18bfdf863fb5dd918 in kylin's branch refs/heads/2.5.x-hadoop3.1 from shaofengshi [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=f411935 ] KYLIN-2565 Add cdh60 profile > Support Hadoop 3.0 > -- > > Key: KYLIN-2565 > URL: https://issues.apache.org/jira/browse/KYLIN-2565 > Project: Kylin > Issue Type: New Feature > Components: Job Engine >Reporter: Wang Cheng >Assignee: Shaofeng SHI >Priority: Major > Fix For: v2.5.0 > > > Hadoop3.0-alpha is released, Kylin should also keep compatible with it. Below > is the Hadoop3.0 components requirements: > https://cwiki.apache.org/confluence/display/HADOOP/Hadoop+3.0.0+release -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3646) Add instruction about rowkey sequence in "Advanced setting" page
[ https://issues.apache.org/jira/browse/KYLIN-3646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667991#comment-16667991 ] ASF subversion and git services commented on KYLIN-3646: Commit 9d0cedc15600d0eca0adbd419c037b53945d2d7e in kylin's branch refs/heads/2.5.x-hadoop3.1 from Emiya0306 [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=9d0cedc ] KYLIN-3646 Add instruction about rowkey sequence in "Advanced setting" page > Add instruction about rowkey sequence in "Advanced setting" page > > > Key: KYLIN-3646 > URL: https://issues.apache.org/jira/browse/KYLIN-3646 > Project: Kylin > Issue Type: Improvement > Components: Web >Reporter: Shaofeng SHI >Assignee: Roger >Priority: Major > Fix For: v2.4.2, v2.5.1 > > > The position of dimension in rowkey is important for performance (both build > and query); But this message is not clear for a normal user. We'd better add > instruction on the web GUI, so that user knows to adjust it by some rule. > > In the rowkey part, adding: > "Important: Dimension's positioin on HBase rowkey is critical for > performance. You can drag and drop to adjust the sequence. In short, put > filtering dimension before non-filtering dimension, and put high cardinality > dimension before low cardinality dimension." -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3646) Add instruction about rowkey sequence in "Advanced setting" page
[ https://issues.apache.org/jira/browse/KYLIN-3646?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-3646: Fix Version/s: (was: v2.6.0) v2.5.1 v2.4.2 > Add instruction about rowkey sequence in "Advanced setting" page > > > Key: KYLIN-3646 > URL: https://issues.apache.org/jira/browse/KYLIN-3646 > Project: Kylin > Issue Type: Improvement > Components: Web >Reporter: Shaofeng SHI >Assignee: Roger >Priority: Major > Fix For: v2.4.2, v2.5.1 > > > The position of dimension in rowkey is important for performance (both build > and query); But this message is not clear for a normal user. We'd better add > instruction on the web GUI, so that user knows to adjust it by some rule. > > In the rowkey part, adding: > "Important: Dimension's positioin on HBase rowkey is critical for > performance. You can drag and drop to adjust the sequence. In short, put > filtering dimension before non-filtering dimension, and put high cardinality > dimension before low cardinality dimension." -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3646) Add instruction about rowkey sequence in "Advanced setting" page
[ https://issues.apache.org/jira/browse/KYLIN-3646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667975#comment-16667975 ] ASF subversion and git services commented on KYLIN-3646: Commit 9d0cedc15600d0eca0adbd419c037b53945d2d7e in kylin's branch refs/heads/2.5.x from Emiya0306 [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=9d0cedc ] KYLIN-3646 Add instruction about rowkey sequence in "Advanced setting" page > Add instruction about rowkey sequence in "Advanced setting" page > > > Key: KYLIN-3646 > URL: https://issues.apache.org/jira/browse/KYLIN-3646 > Project: Kylin > Issue Type: Improvement > Components: Web >Reporter: Shaofeng SHI >Assignee: Roger >Priority: Major > Fix For: v2.6.0 > > > The position of dimension in rowkey is important for performance (both build > and query); But this message is not clear for a normal user. We'd better add > instruction on the web GUI, so that user knows to adjust it by some rule. > > In the rowkey part, adding: > "Important: Dimension's positioin on HBase rowkey is critical for > performance. You can drag and drop to adjust the sequence. In short, put > filtering dimension before non-filtering dimension, and put high cardinality > dimension before low cardinality dimension." -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3646) Add instruction about rowkey sequence in "Advanced setting" page
[ https://issues.apache.org/jira/browse/KYLIN-3646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667976#comment-16667976 ] ASF subversion and git services commented on KYLIN-3646: Commit 7afd99be2f408d9b57d3e6fa1e9a5545784d8fa6 in kylin's branch refs/heads/2.4.x from Emiya0306 [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=7afd99b ] KYLIN-3646 Add instruction about rowkey sequence in "Advanced setting" page > Add instruction about rowkey sequence in "Advanced setting" page > > > Key: KYLIN-3646 > URL: https://issues.apache.org/jira/browse/KYLIN-3646 > Project: Kylin > Issue Type: Improvement > Components: Web >Reporter: Shaofeng SHI >Assignee: Roger >Priority: Major > Fix For: v2.6.0 > > > The position of dimension in rowkey is important for performance (both build > and query); But this message is not clear for a normal user. We'd better add > instruction on the web GUI, so that user knows to adjust it by some rule. > > In the rowkey part, adding: > "Important: Dimension's positioin on HBase rowkey is critical for > performance. You can drag and drop to adjust the sequence. In short, put > filtering dimension before non-filtering dimension, and put high cardinality > dimension before low cardinality dimension." -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-2899) Introduce segment level query cache
[ https://issues.apache.org/jira/browse/KYLIN-2899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong updated KYLIN-2899: -- Summary: Introduce segment level query cache (was: Enable segment level query cache) > Introduce segment level query cache > --- > > Key: KYLIN-2899 > URL: https://issues.apache.org/jira/browse/KYLIN-2899 > Project: Kylin > Issue Type: Sub-task > Components: Query Engine >Affects Versions: v2.1.0 >Reporter: Zhong Yanghong >Assignee: Ma Gang >Priority: Major > Fix For: v2.6.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3649) segment region count and size are not correct when using mysql as Kylin metadata storage
[
https://issues.apache.org/jira/browse/KYLIN-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI updated KYLIN-3649:
Fix Version/s: v2.5.1
> segment region count and size are not correct when using mysql as Kylin
> metadata storage
> -
>
> Key: KYLIN-3649
> URL: https://issues.apache.org/jira/browse/KYLIN-3649
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Affects Versions: v2.5.0
>Reporter: Lingang Deng
>Priority: Major
> Fix For: v2.5.1
>
> Attachments: KYLIN-3649.patch
>
>
> As titles, segment region count and size are not correct.
> {code:java}
> if ("hbase".equals(getConfig().getMetadataUrl().getScheme())) {
> try {
> logger.debug("Loading HTable info " + cubeName + ", " + tableName);
> // use reflection to isolate NoClassDef errors when HBase is not
> available
> hr = (HBaseResponse)
> Class.forName("org.apache.kylin.rest.service.HBaseInfoUtil")//
> .getMethod("getHBaseInfo", new Class[] { String.class,
> KylinConfig.class })//
> .invoke(null, tableName, this.getConfig());
> } catch (Throwable e) {
> throw new IOException(e);
> }
> }
> {code}
> Judgement is not valid when using mysql as Kylin metadata storage.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3649) segment region count and size are not correct when using mysql as Kylin metadata storage
[
https://issues.apache.org/jira/browse/KYLIN-3649?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Lingang Deng updated KYLIN-3649:
Attachment: KYLIN-3649.patch
> segment region count and size are not correct when using mysql as Kylin
> metadata storage
> -
>
> Key: KYLIN-3649
> URL: https://issues.apache.org/jira/browse/KYLIN-3649
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Affects Versions: v2.5.0
>Reporter: Lingang Deng
>Priority: Major
> Attachments: KYLIN-3649.patch
>
>
> As titles, segment region count and size are not correct.
> {code:java}
> if ("hbase".equals(getConfig().getMetadataUrl().getScheme())) {
> try {
> logger.debug("Loading HTable info " + cubeName + ", " + tableName);
> // use reflection to isolate NoClassDef errors when HBase is not
> available
> hr = (HBaseResponse)
> Class.forName("org.apache.kylin.rest.service.HBaseInfoUtil")//
> .getMethod("getHBaseInfo", new Class[] { String.class,
> KylinConfig.class })//
> .invoke(null, tableName, this.getConfig());
> } catch (Throwable e) {
> throw new IOException(e);
> }
> }
> {code}
> Judgement is not valid when using mysql as Kylin metadata storage.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-2894) Change the query cache expiration strategy by signature checking
[
https://issues.apache.org/jira/browse/KYLIN-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI updated KYLIN-2894:
Fix Version/s: v2.6.0
> Change the query cache expiration strategy by signature checking
>
>
> Key: KYLIN-2894
> URL: https://issues.apache.org/jira/browse/KYLIN-2894
> Project: Kylin
> Issue Type: Sub-task
> Components: Query Engine
>Reporter: Zhong Yanghong
>Assignee: Wang Ken
>Priority: Major
> Fix For: v2.6.0
>
>
> Currently to invalid query cache, {{CacheService}} will either invoke
> {{cleanDataCache}} or {{cleanAllDataCache}}. Both methods will clear all of
> the query cache, which is very inefficient. In eBay PROD environment, there's
> around 400 cubing jobs per day, which means the query cache will be cleared
> very 4 minutes. Then we introduced a signature based cache invalidation
> strategy. The basic idea is as follows:
> * Add a signature for {{SQLResponse}}, here we choose the cube last build time
> * When fetch {{SQLResponse}} for cache, first check whether the signature is
> consistent. If not, this cached value is overdue and will be invalidate.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-2896) Refine query exception cache
[ https://issues.apache.org/jira/browse/KYLIN-2896?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-2896: Fix Version/s: v2.6.0 > Refine query exception cache > > > Key: KYLIN-2896 > URL: https://issues.apache.org/jira/browse/KYLIN-2896 > Project: Kylin > Issue Type: Sub-task > Components: Query Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v2.6.0 > > > It's not necessary to divide query cache into success and exception. Only one > query cache is enough. And some special exception needs to be cached. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-2897) Improve the query execution for a set of duplicate queries in a short period
[ https://issues.apache.org/jira/browse/KYLIN-2897?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-2897: Fix Version/s: v2.6.0 > Improve the query execution for a set of duplicate queries in a short period > > > Key: KYLIN-2897 > URL: https://issues.apache.org/jira/browse/KYLIN-2897 > Project: Kylin > Issue Type: Sub-task > Components: Query Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > Fix For: v2.6.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-2898) Introduce memcached as a distributed cache for queries
[ https://issues.apache.org/jira/browse/KYLIN-2898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-2898: Affects Version/s: (was: v2.1.0) Fix Version/s: v2.6.0 > Introduce memcached as a distributed cache for queries > -- > > Key: KYLIN-2898 > URL: https://issues.apache.org/jira/browse/KYLIN-2898 > Project: Kylin > Issue Type: Sub-task > Components: Query Engine >Reporter: Zhong Yanghong >Assignee: Wang Ken >Priority: Major > Fix For: v2.6.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-2898) Introduce memcached as a distributed cache for queries
[ https://issues.apache.org/jira/browse/KYLIN-2898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667052#comment-16667052 ] ASF subversion and git services commented on KYLIN-2898: Commit de86ed61271e1cddb98ddfb29639f757823f0b2a in kylin's branch refs/heads/master from Zhong [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=de86ed6 ] KYLIN-2898 If a distributed cache is adopted, small query results are also better to be put into the cache. > Introduce memcached as a distributed cache for queries > -- > > Key: KYLIN-2898 > URL: https://issues.apache.org/jira/browse/KYLIN-2898 > Project: Kylin > Issue Type: Sub-task > Components: Query Engine >Affects Versions: v2.1.0 >Reporter: Zhong Yanghong >Assignee: Wang Ken >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-2897) Improve the query execution for a set of duplicate queries in a short period
[ https://issues.apache.org/jira/browse/KYLIN-2897?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667046#comment-16667046 ] ASF subversion and git services commented on KYLIN-2897: Commit b9aab0105cbf00a3581409a357be3816d0cd89a6 in kylin's branch refs/heads/master from Ma,Gang [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=b9aab01 ] KYLIN-2897 improve the query execution for a set of duplicate queries in a short period > Improve the query execution for a set of duplicate queries in a short period > > > Key: KYLIN-2897 > URL: https://issues.apache.org/jira/browse/KYLIN-2897 > Project: Kylin > Issue Type: Sub-task > Components: Query Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-2899) Enable segment level query cache
[ https://issues.apache.org/jira/browse/KYLIN-2899?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667050#comment-16667050 ] ASF subversion and git services commented on KYLIN-2899: Commit 7b58b161a1d3264e744e3e78b0cffbde5e830e67 in kylin's branch refs/heads/master from Ma,Gang [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=7b58b16 ] KYLIN-2899 Introduce segment level query cache > Enable segment level query cache > > > Key: KYLIN-2899 > URL: https://issues.apache.org/jira/browse/KYLIN-2899 > Project: Kylin > Issue Type: Sub-task > Components: Query Engine >Affects Versions: v2.1.0 >Reporter: Zhong Yanghong >Assignee: Ma Gang >Priority: Major > Fix For: v2.6.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-2896) Refine query exception cache
[ https://issues.apache.org/jira/browse/KYLIN-2896?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667045#comment-16667045 ] ASF subversion and git services commented on KYLIN-2896: Commit 6b5f961724e4b7964dbc1b268a200095d0fbe30d in kylin's branch refs/heads/master from Zhong [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=6b5f961 ] KYLIN-2896 remove query exception cache > Refine query exception cache > > > Key: KYLIN-2896 > URL: https://issues.apache.org/jira/browse/KYLIN-2896 > Project: Kylin > Issue Type: Sub-task > Components: Query Engine >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > > It's not necessary to divide query cache into success and exception. Only one > query cache is enough. And some special exception needs to be cached. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-2898) Introduce memcached as a distributed cache for queries
[ https://issues.apache.org/jira/browse/KYLIN-2898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667048#comment-16667048 ] ASF subversion and git services commented on KYLIN-2898: Commit 73d0fd43af2ad4fa3ae7830341a0e7d96a349ee5 in kylin's branch refs/heads/master from Zhong [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=73d0fd4 ] KYLIN-2898 config memcached > Introduce memcached as a distributed cache for queries > -- > > Key: KYLIN-2898 > URL: https://issues.apache.org/jira/browse/KYLIN-2898 > Project: Kylin > Issue Type: Sub-task > Components: Query Engine >Affects Versions: v2.1.0 >Reporter: Zhong Yanghong >Assignee: Wang Ken >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-2898) Introduce memcached as a distributed cache for queries
[ https://issues.apache.org/jira/browse/KYLIN-2898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667047#comment-16667047 ] ASF subversion and git services commented on KYLIN-2898: Commit cd952ffd23e27b8468db03c38758d552a00c7486 in kylin's branch refs/heads/master from Wang Ken [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=cd952ff ] KYLIN-2898 Introduce memcached as a distributed cache for queries > Introduce memcached as a distributed cache for queries > -- > > Key: KYLIN-2898 > URL: https://issues.apache.org/jira/browse/KYLIN-2898 > Project: Kylin > Issue Type: Sub-task > Components: Query Engine >Affects Versions: v2.1.0 >Reporter: Zhong Yanghong >Assignee: Wang Ken >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-2894) Change the query cache expiration strategy by signature checking
[
https://issues.apache.org/jira/browse/KYLIN-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667044#comment-16667044
]
ASF subversion and git services commented on KYLIN-2894:
Commit 95afdb44e52be0d1e24c2487745906dbd35da8fc in kylin's branch
refs/heads/master from Zhong
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=95afdb4 ]
KYLIN-2894 add unit test
> Change the query cache expiration strategy by signature checking
>
>
> Key: KYLIN-2894
> URL: https://issues.apache.org/jira/browse/KYLIN-2894
> Project: Kylin
> Issue Type: Sub-task
> Components: Query Engine
>Reporter: Zhong Yanghong
>Assignee: Wang Ken
>Priority: Major
>
> Currently to invalid query cache, {{CacheService}} will either invoke
> {{cleanDataCache}} or {{cleanAllDataCache}}. Both methods will clear all of
> the query cache, which is very inefficient. In eBay PROD environment, there's
> around 400 cubing jobs per day, which means the query cache will be cleared
> very 4 minutes. Then we introduced a signature based cache invalidation
> strategy. The basic idea is as follows:
> * Add a signature for {{SQLResponse}}, here we choose the cube last build time
> * When fetch {{SQLResponse}} for cache, first check whether the signature is
> consistent. If not, this cached value is overdue and will be invalidate.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-2898) Introduce memcached as a distributed cache for queries
[ https://issues.apache.org/jira/browse/KYLIN-2898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667049#comment-16667049 ] ASF subversion and git services commented on KYLIN-2898: Commit 2f04803e2c860922a5f33469403f667bce004cc3 in kylin's branch refs/heads/master from Zhong [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=2f04803 ] KYLIN-2898 add unit test > Introduce memcached as a distributed cache for queries > -- > > Key: KYLIN-2898 > URL: https://issues.apache.org/jira/browse/KYLIN-2898 > Project: Kylin > Issue Type: Sub-task > Components: Query Engine >Affects Versions: v2.1.0 >Reporter: Zhong Yanghong >Assignee: Wang Ken >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-2723) Introduce metrics collector for query & job metrics
[ https://issues.apache.org/jira/browse/KYLIN-2723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667051#comment-16667051 ] ASF subversion and git services commented on KYLIN-2723: Commit 78b17f5a70fb2e5d66e9cce4def79e5adb377878 in kylin's branch refs/heads/master from Zhong [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=78b17f5 ] KYLIN-2723 fix potential concurrent issue when add rpc statistics > Introduce metrics collector for query & job metrics > --- > > Key: KYLIN-2723 > URL: https://issues.apache.org/jira/browse/KYLIN-2723 > Project: Kylin > Issue Type: Sub-task >Reporter: Zhong Yanghong >Assignee: Wang Ken >Priority: Major > Fix For: v2.3.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-2894) Change the query cache expiration strategy by signature checking
[
https://issues.apache.org/jira/browse/KYLIN-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16667043#comment-16667043
]
ASF subversion and git services commented on KYLIN-2894:
Commit 7e1cdeda85dc293efe8bd408444d7bbef8401be6 in kylin's branch
refs/heads/master from Ma,Gang
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=7e1cded ]
KYLIN-2894 add a new signature calculator FactTableRealizationSetCalculator
> Change the query cache expiration strategy by signature checking
>
>
> Key: KYLIN-2894
> URL: https://issues.apache.org/jira/browse/KYLIN-2894
> Project: Kylin
> Issue Type: Sub-task
> Components: Query Engine
>Reporter: Zhong Yanghong
>Assignee: Wang Ken
>Priority: Major
>
> Currently to invalid query cache, {{CacheService}} will either invoke
> {{cleanDataCache}} or {{cleanAllDataCache}}. Both methods will clear all of
> the query cache, which is very inefficient. In eBay PROD environment, there's
> around 400 cubing jobs per day, which means the query cache will be cleared
> very 4 minutes. Then we introduced a signature based cache invalidation
> strategy. The basic idea is as follows:
> * Add a signature for {{SQLResponse}}, here we choose the cube last build time
> * When fetch {{SQLResponse}} for cache, first check whether the signature is
> consistent. If not, this cached value is overdue and will be invalidate.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] shaofengshi closed pull request #318: Add document for how to become Apache Committer
shaofengshi closed pull request #318: Add document for how to become Apache Committer URL: https://github.com/apache/kylin/pull/318 This is a PR merged from a forked repository. As GitHub hides the original diff on merge, it is displayed below for the sake of provenance: As this is a foreign pull request (from a fork), the diff is supplied below (as it won't show otherwise due to GitHub magic): diff --git a/website/_data/development-cn.yml b/website/_data/development-cn.yml index d1b2331b53..74a9fd8bac 100644 --- a/website/_data/development-cn.yml +++ b/website/_data/development-cn.yml @@ -23,6 +23,7 @@ - coding_naming_convention - howto_test - howto_contribute +- howto_become_apache_committer - howto_docs - howto_package - howto_hbase_branches diff --git a/website/_data/development.yml b/website/_data/development.yml index 827fb573cb..fa68c1bc06 100644 --- a/website/_data/development.yml +++ b/website/_data/development.yml @@ -23,6 +23,7 @@ - coding_naming_convention - howto_test - howto_contribute + - howto_become_apache_committer - howto_docs - howto_package - howto_hbase_branches diff --git a/website/_dev/howto_become_apache_committer.cn.md b/website/_dev/howto_become_apache_committer.cn.md new file mode 100644 index 00..fa6fd0d02c --- /dev/null +++ b/website/_dev/howto_become_apache_committer.cn.md @@ -0,0 +1,24 @@ +--- +layout: dev-cn +title: 如何成为 Apache Committer +categories: development +permalink: /cn/development/howto_become_apache_committer.html +--- + +## 宽泛的标准 +这篇文章将介绍如何成为一名 Apache Committer；简单来说可分为以下几个步骤： + +1. 理解和认同 Apache 运作方式和理念 (Apache Way)，并以此理念来与其他人协同工作； +2. 参与项目，可以从贡献文档，回答社区问题（礼貌地），贡献 patch 等做起；到这个时候，你就是一名 contributor 了； +3. 成为开源项目某个模块的专家，能够贡献代码、添加测试案例并不 break 其他人的功能，与其他 committer 良好沟通建立互信； + +**另外，这些行为最好是自愿的，候选人喜爱此项目并自我激励，把项目当作是自己的事情。** + +如果做到了上述的全部或大部分，那么接下来就是找到一个项目的 PMC 愿意来提名你为 committer。他需要陈述你的贡献给所有 PMC，并获得至少 3 个 +1 票。 + +总之，Apache 强调 “**Community over code**” （社区第一，胜过代码）， 你需要在 Kylin 社区树立影响力和发言权，大胆发言，让其他人认可你的贡献，并愿意与你建立信任关系。成为 committer 是一个荣誉与责任共存的事情；成为 committer 不是终点，而是一个更高的起点。 + +参考资料： +[https://community.apache.org/contributors/](https://community.apache.org/contributors/) +[https://www.quora.com/How-can-I-become-an-Apache-committer](https://www.quora.com/How-can-I-become-an-Apache-committer) +[https://mahout.apache.org/developers/how-to-become-a-committer](https://mahout.apache.org/developers/how-to-become-a-committer) diff --git a/website/_dev/howto_become_apache_committer.md b/website/_dev/howto_become_apache_committer.md new file mode 100644 index 00..1cd1bbf8e7 --- /dev/null +++ b/website/_dev/howto_become_apache_committer.md @@ -0,0 +1,24 @@ +--- +layout: dev +title: How to become an Apache Committer +categories: development +permalink: /development/howto_become_apache_committer.html +--- + +## The broad standard +This article will show you how to become an Apache Committer; in simple terms, it can be divided into the following steps: + +1. Understand and agree with Apache's way of working and idea (Apache Way) and use it to work with others; +2. Participate in the project, you can start with contributing documents, answering community questions (courtesy), contributing patches, etc. At this time, you are a contributor; +3. Be an expert in a module of an open source project, able to contribute code, add test cases and not break other people's functions, and establish good mutual trust with other committers; + +**In addition, these behaviours are best voluntary. Candidates love the project and are self-motivated, treating the project as their own.** + +If you do all or most of the above, then the next step is to find a project PMC willing to nominate you as a committer. He needs to state your contribution to all PMCs and get at least 3 +1 votes. + +In short, Apache emphasizes "**Community over code**" (community first, better than code), you need to establish influence and voice in the Kylin community, speak boldly, let others recognize your contribution, and are willing to build trust with you. Being a committer is a matter of coexistence of honour and responsibility; becoming a committer is not an end, but a higher starting point. + +Reference: +[https://community.apache.org/contributors/](https://community.apache.org/contributors/) +[https://www.quora.com/How-can-I-become-an-Apache-committer](https://www.quora.com/How-can-I-become-an-Apache-committer) +[https://mahout.apache.org/developers/how-to-become-a-committer](https://mahout.apache.org/developers/how-to-become-a-committer) This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3556) Interned string should not be used as lock object
[
https://issues.apache.org/jira/browse/KYLIN-3556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1920#comment-1920
]
ASF GitHub Bot commented on KYLIN-3556:
---
asfgit commented on issue #319: KYLIN-3556, replace String.intern to avoid
unexpected locking collisions
URL: https://github.com/apache/kylin/pull/319#issuecomment-433846327
Can one of the admins verify this patch?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Interned string should not be used as lock object
> -
>
> Key: KYLIN-3556
> URL: https://issues.apache.org/jira/browse/KYLIN-3556
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Affects Versions: v2.5.0
>Reporter: Ted Yu
>Assignee: Kaige Liu
>Priority: Minor
> Fix For: v2.5.1
>
>
> In JDBCResourceDAO :
> {code}
> public void execute(Connection connection) throws SQLException {
> synchronized (resPath.intern()) {
> {code}
> Locking on an interned string can cause unexpected locking collisions with
> other part of code.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3556) Interned string should not be used as lock object
[
https://issues.apache.org/jira/browse/KYLIN-3556?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1919#comment-1919
]
ASF GitHub Bot commented on KYLIN-3556:
---
etherge opened a new pull request #319: KYLIN-3556, replace String.intern to
avoid unexpected locking collisions
URL: https://github.com/apache/kylin/pull/319
Locking on an interned string can cause unexpected locking collisions with
other part of code. A better option is **_Interner_** in guava.
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Interned string should not be used as lock object
> -
>
> Key: KYLIN-3556
> URL: https://issues.apache.org/jira/browse/KYLIN-3556
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Affects Versions: v2.5.0
>Reporter: Ted Yu
>Assignee: Kaige Liu
>Priority: Minor
> Fix For: v2.5.1
>
>
> In JDBCResourceDAO :
> {code}
> public void execute(Connection connection) throws SQLException {
> synchronized (resPath.intern()) {
> {code}
> Locking on an interned string can cause unexpected locking collisions with
> other part of code.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] asfgit commented on issue #319: KYLIN-3556, replace String.intern to avoid unexpected locking collisions
asfgit commented on issue #319: KYLIN-3556, replace String.intern to avoid unexpected locking collisions URL: https://github.com/apache/kylin/pull/319#issuecomment-433846327 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] etherge opened a new pull request #319: KYLIN-3556, replace String.intern to avoid unexpected locking collisions
etherge opened a new pull request #319: KYLIN-3556, replace String.intern to avoid unexpected locking collisions URL: https://github.com/apache/kylin/pull/319 Locking on an interned string can cause unexpected locking collisions with other part of code. A better option is **_Interner_** in guava. This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] asfgit commented on issue #318: Add document for how to become Apache Committer
asfgit commented on issue #318: Add document for how to become Apache Committer URL: https://github.com/apache/kylin/pull/318#issuecomment-433844365 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] GinaZhai opened a new pull request #318: Add document for how to become Apache Committer
GinaZhai opened a new pull request #318: Add document for how to become Apache Committer URL: https://github.com/apache/kylin/pull/318 This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Resolved] (KYLIN-3599) Bulk Add Measures
[ https://issues.apache.org/jira/browse/KYLIN-3599?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI resolved KYLIN-3599. - Resolution: Fixed > Bulk Add Measures > - > > Key: KYLIN-3599 > URL: https://issues.apache.org/jira/browse/KYLIN-3599 > Project: Kylin > Issue Type: Improvement > Components: Web , Website >Reporter: Pan, Julian >Assignee: Pan, Julian >Priority: Major > Fix For: v2.5.1 > > Attachments: bluk add button.png, select all or clean.png > > > If user want to add hundreds measure, user need add them one by one in > currently website. > Add a bulk add button to resolve this issues in measure step. > P.S just include common measure (sum, max, min, raw, percentile), due to > count_distinct, topN and extended_column need more setting. Image attached > and the default name generate automatically by pattern [measure_columnName]. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3562) TS conflict when kylin update metadata in HBase
[
https://issues.apache.org/jira/browse/KYLIN-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI updated KYLIN-3562:
Fix Version/s: (was: Future)
> TS conflict when kylin update metadata in HBase
> ---
>
> Key: KYLIN-3562
> URL: https://issues.apache.org/jira/browse/KYLIN-3562
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.4.0
>Reporter: Lingang Deng
>Assignee: Jiatao Tao
>Priority: Major
> Fix For: v2.4.2, v2.5.1
>
> Attachments: image-2018-09-17-16-40-56-212.png,
> image-2018-09-25-15-03-51-009.png, image-2018-09-25-16-43-50-277.png
>
>
> Error log was as follows,
> {code:java}
> org.apache.kylin.common.persistence.WriteConflictException: Overwriting
> conflict /user/admin, expect old TS 1536928877043, but it is 1536928907207
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:325)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:318)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:303)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:282)
> at
> org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192){code}
>
> what disturbs me the most was that the error was happened several hours,
> then my all build job and query job failed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KYLIN-3526) No check for hybrid name with special character
[ https://issues.apache.org/jira/browse/KYLIN-3526?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI resolved KYLIN-3526. - Resolution: Fixed > No check for hybrid name with special character > --- > > Key: KYLIN-3526 > URL: https://issues.apache.org/jira/browse/KYLIN-3526 > Project: Kylin > Issue Type: Improvement > Components: Web >Reporter: Na Zhai >Assignee: Roger >Priority: Minor > Fix For: v2.5.1 > > Attachments: 名称没做限制.png > > > When create hybrid，there is no check for hybrid name with special character. > Such as !@&. > Hybrid can be created successfully with name like hy! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3527) Hybrid couldn't save when there is only 1 cube
[ https://issues.apache.org/jira/browse/KYLIN-3527?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI resolved KYLIN-3527. - Resolution: Fixed > Hybrid couldn't save when there is only 1 cube > -- > > Key: KYLIN-3527 > URL: https://issues.apache.org/jira/browse/KYLIN-3527 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v2.5.0 >Reporter: Shaofeng SHI >Assignee: Roger >Priority: Minor > Fix For: v2.5.1 > > Attachments: hybrid-not-saved.png, move-in-out-icons.png > > > I created a hybrid with two cubes; Later I want to remove one cube from it, > but I found when there is only 1 cube, the "Submit" button was disabled. > > The hybrid should allows 0, 1 or more cubes. > > Besides, I found the move in and out icons were not properly displayed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3528) No details page for hybrid
[ https://issues.apache.org/jira/browse/KYLIN-3528?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI resolved KYLIN-3528. - Resolution: Fixed > No details page for hybrid > -- > > Key: KYLIN-3528 > URL: https://issues.apache.org/jira/browse/KYLIN-3528 > Project: Kylin > Issue Type: Improvement > Components: Web >Reporter: Na Zhai >Assignee: Roger >Priority: Minor > Fix For: v2.5.1 > > > When I click hybrid name, there is no detail page for that hybrid. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3629) NullPointException throws when use preparedStatement cache in some case
[
https://issues.apache.org/jira/browse/KYLIN-3629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI resolved KYLIN-3629.
-
Resolution: Fixed
> NullPointException throws when use preparedStatement cache in some case
> ---
>
> Key: KYLIN-3629
> URL: https://issues.apache.org/jira/browse/KYLIN-3629
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Reporter: Ma Gang
>Assignee: Ma Gang
>Priority: Major
> Fix For: v2.5.1
>
> Attachments: KYLIN-3629.patch
>
>
> NullPointException throws when use preparedStatement cache in some case that
> there is no realization exists in the OLAPContext, for example the following
> sql request will cause issue:
> {color:#205081}{"sql":"select t1.lstg_site_id,t1.seller_id, count(1) from
> (select lstg_site_id,seller_id,count(1) from TEST_KYLIN_FACT where
> lstg_site_id=? and seller_id=? group by lstg_site_id,seller_id) t1 left join
> (select lstg_site_id,seller_id,count(1) from TEST_KYLIN_FACT where
> lstg_site_id=? and seller_id=? group by lstg_site_id,seller_id) t2 on
> t1.lstg_site_id = t2.lstg_site_id group by
> t1.lstg_site_id,t1.seller_id","offset":0,"limit":5,"params":[\{"className":"java.lang.Integer","value":"100"},\{"className":"java.lang.Integer","value":"1731"},\{"className":"java.lang.Integer","value":"100"},\{"className":"java.lang.Integer","value":"1732"}],"acceptPartial":true,"project":"default","backdoorToggles":
> {}}{color}
> the exception stacktrace is like:
> {color:#205081}org.apache.kylin.rest.exception.InternalErrorException:
> Unknown error.\n\tat
> org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:424)\n\tat
>
> org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:351)\n\tat
>
> org.apache.kylin.rest.controller.QueryController.query(QueryController.java:86)\n\tat
> sun.reflect.GeneratedMethodAccessor155.invoke(Unknown Source)\n\tat
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat
> java.lang.reflect.Method.invoke(Method.java:498)\n\tat
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)\n\tat
>
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:133)\n\tat
>
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:97)\n\tat
>
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:827)\n\tat
>
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:738)\n\tat
>
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)\n\tat
>
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:967)\n\tat
>
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:901)\n\tat
>
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:970)\n\tat
>
> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:872)\n\tat
> javax.servlet.http.HttpServlet.service(HttpServlet.java:661)\n\tat
> org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:846)\n\tat
> javax.servlet.http.HttpServlet.service(HttpServlet.java:742)\n\tat
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)\n\tat
>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n\tat
> org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)\n\tat
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)\n\tat
>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)\n\tat
>
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:317)\n\tat
>
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:127)\n\tat
>
> org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:91)\n\tat
>
> org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:331)\n\tat
>
> org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)\n\tat
>
>
[jira] [Updated] (KYLIN-3637) Union All not working with count()
[
https://issues.apache.org/jira/browse/KYLIN-3637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI updated KYLIN-3637:
Fix Version/s: (was: v2.5.1)
> Union All not working with count()
> --
>
> Key: KYLIN-3637
> URL: https://issues.apache.org/jira/browse/KYLIN-3637
> Project: Kylin
> Issue Type: Bug
>Reporter: Shaofeng SHI
>Priority: Major
>
> From user@kylin mailing list:
> When I ran sql "SELECT count(TRANS_ID) as TRANS_ID FROM KYLIN_SALES where
> TRANS_ID <> 1 union all select count(TRANS_ID) as TRANS_ID FROM KYLIN_SALES"
> on sample cube.
> Got Error:
> Error while compiling generated Java code: public static class Record2_1
> implements java.io.Serializable { public Long TRANS_ID; public long
> _KY_COUNT__; public Record2_1() {} public boolean equals(Object o) { if
> (this == o) \{ return true; } if (!(o instanceof Record2_1)) { return false;
> } return java.util.Objects.equals(this.TRANS_ID, ((Record2_1) o).TRANS_ID)
> && this._KY_COUNT__ == ((Record2_1) o)._KY_COUNT__; } public int hashCode()
> { int h = 0; h = org.apache.calcite.runtime.Utilities.hash(h,
> this.TRANS_ID); h = org.apache.calcite.runtime.Utilities.hash(h,
> this._KY_COUNT__); return h; } public int compareTo(Record2_1 that) { int c;
> c = org.apache.calcite.runtime.Utilities.compareNullsLast(this.TRANS_ID,
> that.TRANS_ID); if (c != 0) \{ return c; } c =
> org.apache.calcite.runtime.Utilities.compare(this._KY_COUNT__,
> that._KY_COUNT__); if (c != 0) \{ return c; } return 0; } public String
> toString() { return "{TRANS_ID=" + this.TRANS_ID + ", _KY_COUNT__=" +
> this._KY_COUNT__ + "}"; } } public static class Record1_0 implements
> java.io.Serializable \{ public long f0; public Record1_0() {} public boolean
> equals(Object o) \{ if (this == o) { return true; } if (!(o instanceof
> Record1_0)) \{ return false; } return this.f0 == ((Record1_0) o).f0; } public
> int hashCode() { int h = 0; h = org.apache.calcite.runtime.Utilities.hash(h,
> this.f0); return h; } public int compareTo(Record1_0 that) { final int c; c
> = org.apache.calcite.runtime.Utilities.compare(this.f0, that.f0); if (c !=
> 0) \{ return c; } return 0; } public String toString() { return "{f0=" +
> this.f0 + "}"; } } org.apache.calcite.DataContext root; public
> org.apache.calcite.linq4j.Enumerable bind(final
> org.apache.calcite.DataContext root0) { root = root0; final
> org.apache.calcite.linq4j.Enumerable _inputEnumerable =
> ((org.apache.kylin.query.schema.OLAPTable)
> root.getRootSchema().getSubSchema("DEFAULT").getTable("KYLIN_SALES")).executeOLAPQuery(root,
> 1); final org.apache.calcite.linq4j.AbstractEnumerable child = new
> org.apache.calcite.linq4j.AbstractEnumerable(){ public
> org.apache.calcite.linq4j.Enumerator enumerator() { return new
> org.apache.calcite.linq4j.Enumerator(){ public final
> org.apache.calcite.linq4j.Enumerator inputEnumerator =
> _inputEnumerable.enumerator(); public void reset() {
> inputEnumerator.reset(); } public boolean moveNext() { while
> (inputEnumerator.moveNext()) { final Long inp0_ = (Long) ((Object[])
> inputEnumerator.current())[0]; if (inp0_ != null && inp0_.longValue() != 1L)
> { return true; } } return false; } public void close() {
> inputEnumerator.close(); } public Object current() { final Object[] current
> = (Object[]) inputEnumerator.current(); return new Object[] { current[0],
> current[11]}; } }; } }; final org.apache.calcite.linq4j.Enumerable
> _inputEnumerable0 = ((org.apache.kylin.query.schema.OLAPTable)
> root.getRootSchema().getSubSchema("DEFAULT").getTable("KYLIN_SALES")).executeOLAPQuery(root,
> 2); final org.apache.calcite.linq4j.AbstractEnumerable child1 = new
> org.apache.calcite.linq4j.AbstractEnumerable(){ public
> org.apache.calcite.linq4j.Enumerator enumerator() { return new
> org.apache.calcite.linq4j.Enumerator(){ public final
> org.apache.calcite.linq4j.Enumerator inputEnumerator =
> _inputEnumerable0.enumerator(); public void reset() {
> inputEnumerator.reset(); } public boolean moveNext() { return
> inputEnumerator.moveNext(); } public void close() { inputEnumerator.close();
> } public Object current() { final Object[] current = (Object[])
> inputEnumerator.current(); return new Record2_1( (Long) current[0],
> org.apache.calcite.runtime.SqlFunctions.toLong(current[11])); } }; } };
> return
> org.apache.calcite.linq4j.Linq4j.singletonEnumerable(child.aggregate(new
> org.apache.calcite.linq4j.function.Function0() { public Object apply() {
> long a0s0; a0s0 = 0; Record1_0 record0; record0 = new Record1_0();
> record0.f0 = a0s0; return record0; } } .apply(), new
> org.apache.calcite.linq4j.function.Function2() { public Record1_0
> apply(Record1_0 acc, Object[] in) { acc.f0 = acc.f0 +
> org.apache.calcite.runtime.SqlFunctions.toLong(in[1]); return acc; } public
> Record1_0 apply(Object acc, Object in) {
[jira] [Closed] (KYLIN-3637) Union All not working with count()
[
https://issues.apache.org/jira/browse/KYLIN-3637?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI closed KYLIN-3637.
---
Resolution: Duplicate
> Union All not working with count()
> --
>
> Key: KYLIN-3637
> URL: https://issues.apache.org/jira/browse/KYLIN-3637
> Project: Kylin
> Issue Type: Bug
>Reporter: Shaofeng SHI
>Priority: Major
>
> From user@kylin mailing list:
> When I ran sql "SELECT count(TRANS_ID) as TRANS_ID FROM KYLIN_SALES where
> TRANS_ID <> 1 union all select count(TRANS_ID) as TRANS_ID FROM KYLIN_SALES"
> on sample cube.
> Got Error:
> Error while compiling generated Java code: public static class Record2_1
> implements java.io.Serializable { public Long TRANS_ID; public long
> _KY_COUNT__; public Record2_1() {} public boolean equals(Object o) { if
> (this == o) \{ return true; } if (!(o instanceof Record2_1)) { return false;
> } return java.util.Objects.equals(this.TRANS_ID, ((Record2_1) o).TRANS_ID)
> && this._KY_COUNT__ == ((Record2_1) o)._KY_COUNT__; } public int hashCode()
> { int h = 0; h = org.apache.calcite.runtime.Utilities.hash(h,
> this.TRANS_ID); h = org.apache.calcite.runtime.Utilities.hash(h,
> this._KY_COUNT__); return h; } public int compareTo(Record2_1 that) { int c;
> c = org.apache.calcite.runtime.Utilities.compareNullsLast(this.TRANS_ID,
> that.TRANS_ID); if (c != 0) \{ return c; } c =
> org.apache.calcite.runtime.Utilities.compare(this._KY_COUNT__,
> that._KY_COUNT__); if (c != 0) \{ return c; } return 0; } public String
> toString() { return "{TRANS_ID=" + this.TRANS_ID + ", _KY_COUNT__=" +
> this._KY_COUNT__ + "}"; } } public static class Record1_0 implements
> java.io.Serializable \{ public long f0; public Record1_0() {} public boolean
> equals(Object o) \{ if (this == o) { return true; } if (!(o instanceof
> Record1_0)) \{ return false; } return this.f0 == ((Record1_0) o).f0; } public
> int hashCode() { int h = 0; h = org.apache.calcite.runtime.Utilities.hash(h,
> this.f0); return h; } public int compareTo(Record1_0 that) { final int c; c
> = org.apache.calcite.runtime.Utilities.compare(this.f0, that.f0); if (c !=
> 0) \{ return c; } return 0; } public String toString() { return "{f0=" +
> this.f0 + "}"; } } org.apache.calcite.DataContext root; public
> org.apache.calcite.linq4j.Enumerable bind(final
> org.apache.calcite.DataContext root0) { root = root0; final
> org.apache.calcite.linq4j.Enumerable _inputEnumerable =
> ((org.apache.kylin.query.schema.OLAPTable)
> root.getRootSchema().getSubSchema("DEFAULT").getTable("KYLIN_SALES")).executeOLAPQuery(root,
> 1); final org.apache.calcite.linq4j.AbstractEnumerable child = new
> org.apache.calcite.linq4j.AbstractEnumerable(){ public
> org.apache.calcite.linq4j.Enumerator enumerator() { return new
> org.apache.calcite.linq4j.Enumerator(){ public final
> org.apache.calcite.linq4j.Enumerator inputEnumerator =
> _inputEnumerable.enumerator(); public void reset() {
> inputEnumerator.reset(); } public boolean moveNext() { while
> (inputEnumerator.moveNext()) { final Long inp0_ = (Long) ((Object[])
> inputEnumerator.current())[0]; if (inp0_ != null && inp0_.longValue() != 1L)
> { return true; } } return false; } public void close() {
> inputEnumerator.close(); } public Object current() { final Object[] current
> = (Object[]) inputEnumerator.current(); return new Object[] { current[0],
> current[11]}; } }; } }; final org.apache.calcite.linq4j.Enumerable
> _inputEnumerable0 = ((org.apache.kylin.query.schema.OLAPTable)
> root.getRootSchema().getSubSchema("DEFAULT").getTable("KYLIN_SALES")).executeOLAPQuery(root,
> 2); final org.apache.calcite.linq4j.AbstractEnumerable child1 = new
> org.apache.calcite.linq4j.AbstractEnumerable(){ public
> org.apache.calcite.linq4j.Enumerator enumerator() { return new
> org.apache.calcite.linq4j.Enumerator(){ public final
> org.apache.calcite.linq4j.Enumerator inputEnumerator =
> _inputEnumerable0.enumerator(); public void reset() {
> inputEnumerator.reset(); } public boolean moveNext() { return
> inputEnumerator.moveNext(); } public void close() { inputEnumerator.close();
> } public Object current() { final Object[] current = (Object[])
> inputEnumerator.current(); return new Record2_1( (Long) current[0],
> org.apache.calcite.runtime.SqlFunctions.toLong(current[11])); } }; } };
> return
> org.apache.calcite.linq4j.Linq4j.singletonEnumerable(child.aggregate(new
> org.apache.calcite.linq4j.function.Function0() { public Object apply() {
> long a0s0; a0s0 = 0; Record1_0 record0; record0 = new Record1_0();
> record0.f0 = a0s0; return record0; } } .apply(), new
> org.apache.calcite.linq4j.function.Function2() { public Record1_0
> apply(Record1_0 acc, Object[] in) { acc.f0 = acc.f0 +
> org.apache.calcite.runtime.SqlFunctions.toLong(in[1]); return acc; } public
> Record1_0 apply(Object acc, Object in) { return apply(
[jira] [Resolved] (KYLIN-3635) Percentile calculation on Spark engine is wrong
[ https://issues.apache.org/jira/browse/KYLIN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI resolved KYLIN-3635. - Resolution: Fixed > Percentile calculation on Spark engine is wrong > --- > > Key: KYLIN-3635 > URL: https://issues.apache.org/jira/browse/KYLIN-3635 > Project: Kylin > Issue Type: Bug > Components: Spark Engine >Affects Versions: v2.3.0, v2.3.1, v2.4.0, v2.3.2, v2.4.1, v2.5.0 >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Major > Fix For: v2.4.2, v2.5.1 > > > As titled; Received reporting that percentile result is wrong when using > Spark engine. Checked the code and found the object was reused and not reset, > the problem won't happen on normal MR as it would be serialized before next > call, but would be a problem on Spark, as the object was cached in-memory and > be serialized later. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3651) JDBCResourceStore doesn't list all resources
[
https://issues.apache.org/jira/browse/KYLIN-3651?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI resolved KYLIN-3651.
-
Resolution: Fixed
> JDBCResourceStore doesn't list all resources
>
>
> Key: KYLIN-3651
> URL: https://issues.apache.org/jira/browse/KYLIN-3651
> Project: Kylin
> Issue Type: Bug
> Components: Metadata
>Affects Versions: v2.5.0
>Reporter: Shaofeng SHI
>Assignee: Shaofeng SHI
>Priority: Major
> Fix For: v2.5.1
>
>
> The JDBC resource store persists metadata into two tables:
> # kylin_default_instance
> # kylin_default_instance_log
>
> The metadata entry like "/execute_output" will be saved to the _log table;
> But the "list" operation on "/" only lists the subfolders in the first table,
> no in the second table. This will cause partial metadata copy when use
> #ResourceTool.copy.
>
> {code:java}
> ./bin/metastore.sh list /
> [/UUID, /acl, /cube, /cube_desc, /cube_statistics, /dict, /execute, /kafka,
> /model_desc, /project, /streaming, /table, /table_acl, /table_snapshot, /user]
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3562) TS conflict when kylin update metadata in HBase
[
https://issues.apache.org/jira/browse/KYLIN-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1824#comment-1824
]
Jiatao Tao edited comment on KYLIN-3562 at 10/29/18 7:33 AM:
-
We cannot remove the logic that updates user when user login in directly cuz it
sticks with LDAP.
In this commit:
https://github.com/apache/kylin/commit/3bcbaa88b97cf868ddf8f51a54b015ef3b7cc068,
I refine the logic that updates the user, so in most case, it won't update the
user.
The next step may separate LDAP so the design can be cleaner.
was (Author: aron.tao):
We can not remove the logic that updates user when user login in directly cuz
it sticks with LDAP.
In this commit:
https://github.com/apache/kylin/commit/3bcbaa88b97cf868ddf8f51a54b015ef3b7cc068,
I refine the logic that updates the user, so in most case, it won't update the
user.
The next step may separate LDAP so the design can be cleaner.
> TS conflict when kylin update metadata in HBase
> ---
>
> Key: KYLIN-3562
> URL: https://issues.apache.org/jira/browse/KYLIN-3562
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.4.0
>Reporter: Lingang Deng
>Assignee: Jiatao Tao
>Priority: Major
> Fix For: Future, v2.4.2, v2.5.1
>
> Attachments: image-2018-09-17-16-40-56-212.png,
> image-2018-09-25-15-03-51-009.png, image-2018-09-25-16-43-50-277.png
>
>
> Error log was as follows,
> {code:java}
> org.apache.kylin.common.persistence.WriteConflictException: Overwriting
> conflict /user/admin, expect old TS 1536928877043, but it is 1536928907207
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:325)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:318)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:303)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:282)
> at
> org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192){code}
>
> what disturbs me the most was that the error was happened several hours,
> then my all build job and query job failed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3562) TS conflict when kylin update metadata in HBase
[
https://issues.apache.org/jira/browse/KYLIN-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1824#comment-1824
]
Jiatao Tao commented on KYLIN-3562:
---
We can not remove the logic that updates user when user login in directly cuz
it sticks with LDAP.
In this commit:
https://github.com/apache/kylin/commit/3bcbaa88b97cf868ddf8f51a54b015ef3b7cc068,
I refine the logic that updates the user, so in most case, it won't update the
user.
The next step may separate LDAP so the design can be cleaner.
> TS conflict when kylin update metadata in HBase
> ---
>
> Key: KYLIN-3562
> URL: https://issues.apache.org/jira/browse/KYLIN-3562
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.4.0
>Reporter: Lingang Deng
>Assignee: Jiatao Tao
>Priority: Major
> Fix For: Future, v2.4.2, v2.5.1
>
> Attachments: image-2018-09-17-16-40-56-212.png,
> image-2018-09-25-15-03-51-009.png, image-2018-09-25-16-43-50-277.png
>
>
> Error log was as follows,
> {code:java}
> org.apache.kylin.common.persistence.WriteConflictException: Overwriting
> conflict /user/admin, expect old TS 1536928877043, but it is 1536928907207
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:325)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:318)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:303)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:282)
> at
> org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192){code}
>
> what disturbs me the most was that the error was happened several hours,
> then my all build job and query job failed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3562) TS conflict when kylin update metadata in HBase
[
https://issues.apache.org/jira/browse/KYLIN-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1817#comment-1817
]
ASF GitHub Bot commented on KYLIN-3562:
---
shaofengshi closed pull request #317: KYLIN-3562, optimization the logic that
updating the user when user logged in.
URL: https://github.com/apache/kylin/pull/317
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/server-base/src/main/java/org/apache/kylin/rest/security/KylinAuthenticationProvider.java
b/server-base/src/main/java/org/apache/kylin/rest/security/KylinAuthenticationProvider.java
index dd9cbad0a8..d96b3583de 100644
---
a/server-base/src/main/java/org/apache/kylin/rest/security/KylinAuthenticationProvider.java
+++
b/server-base/src/main/java/org/apache/kylin/rest/security/KylinAuthenticationProvider.java
@@ -110,10 +110,12 @@ public Authentication authenticate(Authentication
authentication) throws Authent
}
Assert.notNull(user, "The UserDetail is null.");
-logger.debug("User {} authorities : {}", user.getUsername(),
user.getAuthorities());
-if (!userService.userExists(user.getUsername())) {
+String username = user.getUsername();
+logger.debug("User {} authorities : {}", username,
user.getAuthorities());
+if (!userService.userExists(username)) {
userService.createUser(user);
-} else {
+} else if
(!userService.loadUserByUsername(username).equals(user)) {
+// in case ldap users changing.
userService.updateUser(user);
}
diff --git
a/server-base/src/main/java/org/apache/kylin/rest/security/ManagedUser.java
b/server-base/src/main/java/org/apache/kylin/rest/security/ManagedUser.java
index 00e0045357..48d44405a1 100644
--- a/server-base/src/main/java/org/apache/kylin/rest/security/ManagedUser.java
+++ b/server-base/src/main/java/org/apache/kylin/rest/security/ManagedUser.java
@@ -22,8 +22,8 @@
import java.util.Collection;
import java.util.Iterator;
import java.util.List;
+import java.util.Objects;
-import com.google.common.base.Preconditions;
import org.apache.kylin.common.persistence.RootPersistentEntity;
import org.apache.kylin.rest.service.UserGrantedAuthority;
import org.springframework.security.core.GrantedAuthority;
@@ -41,6 +41,7 @@
import com.fasterxml.jackson.databind.SerializerProvider;
import com.fasterxml.jackson.databind.annotation.JsonDeserialize;
import com.fasterxml.jackson.databind.annotation.JsonSerialize;
+import com.google.common.base.Preconditions;
import com.google.common.collect.Lists;
@SuppressWarnings("serial")
@@ -231,28 +232,22 @@ public boolean isEnabled() {
}
@Override
-public int hashCode() {
-final int prime = 31;
-int result = 1;
-result = prime * result + ((username == null) ? 0 :
username.hashCode());
-return result;
+public boolean equals(Object o) {
+if (this == o)
+return true;
+if (o == null || getClass() != o.getClass())
+return false;
+ManagedUser that = (ManagedUser) o;
+return disabled == that.disabled && defaultPassword ==
that.defaultPassword && locked == that.locked
+&& lockedTime == that.lockedTime && wrongTime == that.wrongTime
+&& Objects.equals(username, that.username) &&
Objects.equals(password, that.password)
+&& Objects.equals(authorities, that.authorities);
}
@Override
-public boolean equals(Object obj) {
-if (this == obj)
-return true;
-if (obj == null)
-return false;
-if (getClass() != obj.getClass())
-return false;
-ManagedUser other = (ManagedUser) obj;
-if (username == null) {
-if (other.username != null)
-return false;
-} else if (!username.equals(other.username))
-return false;
-return true;
+public int hashCode() {
+return Objects.hash(super.hashCode(), username, password, authorities,
disabled, defaultPassword, locked,
+lockedTime, wrongTime);
}
@Override
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> TS conflict when kylin update metadata in HBase
> ---
>
>
[jira] [Commented] (KYLIN-3562) TS conflict when kylin update metadata in HBase
[
https://issues.apache.org/jira/browse/KYLIN-3562?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1818#comment-1818
]
ASF subversion and git services commented on KYLIN-3562:
Commit 3bcbaa88b97cf868ddf8f51a54b015ef3b7cc068 in kylin's branch
refs/heads/master from tttMelody
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=3bcbaa8 ]
KYLIN-3562, optimization the logic that updating the user when user logged in.
> TS conflict when kylin update metadata in HBase
> ---
>
> Key: KYLIN-3562
> URL: https://issues.apache.org/jira/browse/KYLIN-3562
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.4.0
>Reporter: Lingang Deng
>Assignee: Jiatao Tao
>Priority: Major
> Fix For: v2.4.2, v2.5.1
>
> Attachments: image-2018-09-17-16-40-56-212.png,
> image-2018-09-25-15-03-51-009.png, image-2018-09-25-16-43-50-277.png
>
>
> Error log was as follows,
> {code:java}
> org.apache.kylin.common.persistence.WriteConflictException: Overwriting
> conflict /user/admin, expect old TS 1536928877043, but it is 1536928907207
> at
> org.apache.kylin.storage.hbase.HBaseResourceStore.checkAndPutResourceImpl(HBaseResourceStore.java:325)
> at
> org.apache.kylin.common.persistence.ResourceStore.checkAndPutResourceCheckpoint(ResourceStore.java:318)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:303)
> at
> org.apache.kylin.common.persistence.ResourceStore.putResource(ResourceStore.java:282)
> at
> org.apache.kylin.metadata.cachesync.CachedCrudAssist.save(CachedCrudAssist.java:192){code}
>
> what disturbs me the most was that the error was happened several hours,
> then my all build job and query job failed.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] shaofengshi closed pull request #317: KYLIN-3562, optimization the logic that updating the user when user logged in.
shaofengshi closed pull request #317: KYLIN-3562, optimization the logic that
updating the user when user logged in.
URL: https://github.com/apache/kylin/pull/317
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/server-base/src/main/java/org/apache/kylin/rest/security/KylinAuthenticationProvider.java
b/server-base/src/main/java/org/apache/kylin/rest/security/KylinAuthenticationProvider.java
index dd9cbad0a8..d96b3583de 100644
---
a/server-base/src/main/java/org/apache/kylin/rest/security/KylinAuthenticationProvider.java
+++
b/server-base/src/main/java/org/apache/kylin/rest/security/KylinAuthenticationProvider.java
@@ -110,10 +110,12 @@ public Authentication authenticate(Authentication
authentication) throws Authent
}
Assert.notNull(user, "The UserDetail is null.");
-logger.debug("User {} authorities : {}", user.getUsername(),
user.getAuthorities());
-if (!userService.userExists(user.getUsername())) {
+String username = user.getUsername();
+logger.debug("User {} authorities : {}", username,
user.getAuthorities());
+if (!userService.userExists(username)) {
userService.createUser(user);
-} else {
+} else if
(!userService.loadUserByUsername(username).equals(user)) {
+// in case ldap users changing.
userService.updateUser(user);
}
diff --git
a/server-base/src/main/java/org/apache/kylin/rest/security/ManagedUser.java
b/server-base/src/main/java/org/apache/kylin/rest/security/ManagedUser.java
index 00e0045357..48d44405a1 100644
--- a/server-base/src/main/java/org/apache/kylin/rest/security/ManagedUser.java
+++ b/server-base/src/main/java/org/apache/kylin/rest/security/ManagedUser.java
@@ -22,8 +22,8 @@
import java.util.Collection;
import java.util.Iterator;
import java.util.List;
+import java.util.Objects;
-import com.google.common.base.Preconditions;
import org.apache.kylin.common.persistence.RootPersistentEntity;
import org.apache.kylin.rest.service.UserGrantedAuthority;
import org.springframework.security.core.GrantedAuthority;
@@ -41,6 +41,7 @@
import com.fasterxml.jackson.databind.SerializerProvider;
import com.fasterxml.jackson.databind.annotation.JsonDeserialize;
import com.fasterxml.jackson.databind.annotation.JsonSerialize;
+import com.google.common.base.Preconditions;
import com.google.common.collect.Lists;
@SuppressWarnings("serial")
@@ -231,28 +232,22 @@ public boolean isEnabled() {
}
@Override
-public int hashCode() {
-final int prime = 31;
-int result = 1;
-result = prime * result + ((username == null) ? 0 :
username.hashCode());
-return result;
+public boolean equals(Object o) {
+if (this == o)
+return true;
+if (o == null || getClass() != o.getClass())
+return false;
+ManagedUser that = (ManagedUser) o;
+return disabled == that.disabled && defaultPassword ==
that.defaultPassword && locked == that.locked
+&& lockedTime == that.lockedTime && wrongTime == that.wrongTime
+&& Objects.equals(username, that.username) &&
Objects.equals(password, that.password)
+&& Objects.equals(authorities, that.authorities);
}
@Override
-public boolean equals(Object obj) {
-if (this == obj)
-return true;
-if (obj == null)
-return false;
-if (getClass() != obj.getClass())
-return false;
-ManagedUser other = (ManagedUser) obj;
-if (username == null) {
-if (other.username != null)
-return false;
-} else if (!username.equals(other.username))
-return false;
-return true;
+public int hashCode() {
+return Objects.hash(super.hashCode(), username, password, authorities,
disabled, defaultPassword, locked,
+lockedTime, wrongTime);
}
@Override
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
[jira] [Commented] (KYLIN-3635) Percentile calculation on Spark engine is wrong
[ https://issues.apache.org/jira/browse/KYLIN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1785#comment-1785 ] ASF subversion and git services commented on KYLIN-3635: Commit 26e71a762b08445ad173fe42fc84c204ff33fe49 in kylin's branch refs/heads/master from tttMelody [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=26e71a7 ] KYLIN-3635, make a stronger constraint about implementing method reset(). > Percentile calculation on Spark engine is wrong > --- > > Key: KYLIN-3635 > URL: https://issues.apache.org/jira/browse/KYLIN-3635 > Project: Kylin > Issue Type: Bug > Components: Spark Engine >Affects Versions: v2.3.0, v2.3.1, v2.4.0, v2.3.2, v2.4.1, v2.5.0 >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Major > Fix For: v2.4.2, v2.5.1 > > > As titled; Received reporting that percentile result is wrong when using > Spark engine. Checked the code and found the object was reused and not reset, > the problem won't happen on normal MR as it would be serialized before next > call, but would be a problem on Spark, as the object was cached in-memory and > be serialized later. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3635) Percentile calculation on Spark engine is wrong
[
https://issues.apache.org/jira/browse/KYLIN-3635?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1784#comment-1784
]
ASF GitHub Bot commented on KYLIN-3635:
---
shaofengshi closed pull request #316: KYLIN-3635, make a stronger constraint
about implementing method reset().
URL: https://github.com/apache/kylin/pull/316
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/core-metadata/src/main/java/org/apache/kylin/measure/MeasureIngester.java
b/core-metadata/src/main/java/org/apache/kylin/measure/MeasureIngester.java
index ed2cb02e59..b48acf0862 100644
--- a/core-metadata/src/main/java/org/apache/kylin/measure/MeasureIngester.java
+++ b/core-metadata/src/main/java/org/apache/kylin/measure/MeasureIngester.java
@@ -43,9 +43,9 @@
abstract public V valueOf(String[] values, MeasureDesc measureDesc,
Map> dictionaryMap);
-public void reset() {
-
-}
+// Be attention with this, do remember resetting objects if you init in
your implementation.
+// See more details in KYLIN-3635.
+abstract public void reset();
public V reEncodeDictionary(V value, MeasureDesc measureDesc,
Map> oldDicts, Map>
newDicts) {
throw new UnsupportedOperationException();
diff --git
a/core-metadata/src/main/java/org/apache/kylin/measure/basic/BigDecimalIngester.java
b/core-metadata/src/main/java/org/apache/kylin/measure/basic/BigDecimalIngester.java
index c7541abb04..5194606b11 100644
---
a/core-metadata/src/main/java/org/apache/kylin/measure/basic/BigDecimalIngester.java
+++
b/core-metadata/src/main/java/org/apache/kylin/measure/basic/BigDecimalIngester.java
@@ -38,4 +38,9 @@ public BigDecimal valueOf(String[] values, MeasureDesc
measureDesc, Map reEncodeDictionary(List value,
MeasureDesc measureDesc, Map> oldDicts,
Map> newDicts) {
diff --git
a/core-metadata/src/main/java/org/apache/kylin/measure/topn/TopNMeasureType.java
b/core-metadata/src/main/java/org/apache/kylin/measure/topn/TopNMeasureType.java
index 9b6ff0ac20..d7b1bd7382 100644
---
a/core-metadata/src/main/java/org/apache/kylin/measure/topn/TopNMeasureType.java
+++
b/core-metadata/src/main/java/org/apache/kylin/measure/topn/TopNMeasureType.java
@@ -162,6 +162,11 @@ public boolean isMemoryHungry() {
return topNCounter;
}
+@Override
+public void reset() {
+
+}
+
@Override
public TopNCounter
reEncodeDictionary(TopNCounter value, MeasureDesc measureDesc,
Map> oldDicts,
Map> newDicts) {
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Percentile calculation on Spark engine is wrong
> ---
>
> Key: KYLIN-3635
> URL: https://issues.apache.org/jira/browse/KYLIN-3635
> Project: Kylin
> Issue Type: Bug
> Components: Spark Engine
>Affects Versions: v2.3.0, v2.3.1, v2.4.0, v2.3.2, v2.4.1, v2.5.0
>Reporter: Shaofeng SHI
>Assignee: Shaofeng SHI
>Priority: Major
> Fix For: v2.4.2, v2.5.1
>
>
> As titled; Received reporting that percentile result is wrong when using
> Spark engine. Checked the code and found the object was reused and not reset,
> the problem won't happen on normal MR as it would be serialized before next
> call, but would be a problem on Spark, as the object was cached in-memory and
> be serialized later.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] shaofengshi closed pull request #316: KYLIN-3635, make a stronger constraint about implementing method reset().
shaofengshi closed pull request #316: KYLIN-3635, make a stronger constraint
about implementing method reset().
URL: https://github.com/apache/kylin/pull/316
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/core-metadata/src/main/java/org/apache/kylin/measure/MeasureIngester.java
b/core-metadata/src/main/java/org/apache/kylin/measure/MeasureIngester.java
index ed2cb02e59..b48acf0862 100644
--- a/core-metadata/src/main/java/org/apache/kylin/measure/MeasureIngester.java
+++ b/core-metadata/src/main/java/org/apache/kylin/measure/MeasureIngester.java
@@ -43,9 +43,9 @@
abstract public V valueOf(String[] values, MeasureDesc measureDesc,
Map> dictionaryMap);
-public void reset() {
-
-}
+// Be attention with this, do remember resetting objects if you init in
your implementation.
+// See more details in KYLIN-3635.
+abstract public void reset();
public V reEncodeDictionary(V value, MeasureDesc measureDesc,
Map> oldDicts, Map>
newDicts) {
throw new UnsupportedOperationException();
diff --git
a/core-metadata/src/main/java/org/apache/kylin/measure/basic/BigDecimalIngester.java
b/core-metadata/src/main/java/org/apache/kylin/measure/basic/BigDecimalIngester.java
index c7541abb04..5194606b11 100644
---
a/core-metadata/src/main/java/org/apache/kylin/measure/basic/BigDecimalIngester.java
+++
b/core-metadata/src/main/java/org/apache/kylin/measure/basic/BigDecimalIngester.java
@@ -38,4 +38,9 @@ public BigDecimal valueOf(String[] values, MeasureDesc
measureDesc, Map reEncodeDictionary(List value,
MeasureDesc measureDesc, Map> oldDicts,
Map> newDicts) {
diff --git
a/core-metadata/src/main/java/org/apache/kylin/measure/topn/TopNMeasureType.java
b/core-metadata/src/main/java/org/apache/kylin/measure/topn/TopNMeasureType.java
index 9b6ff0ac20..d7b1bd7382 100644
---
a/core-metadata/src/main/java/org/apache/kylin/measure/topn/TopNMeasureType.java
+++
b/core-metadata/src/main/java/org/apache/kylin/measure/topn/TopNMeasureType.java
@@ -162,6 +162,11 @@ public boolean isMemoryHungry() {
return topNCounter;
}
+@Override
+public void reset() {
+
+}
+
@Override
public TopNCounter
reEncodeDictionary(TopNCounter value, MeasureDesc measureDesc,
Map> oldDicts,
Map> newDicts) {
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
With regards,
Apache Git Services
