[jira] [Created] (KYLIN-2748) Calcite code generation can not gc cause OOM
yiming.xu created KYLIN-2748: Summary: Calcite code generation can not gc cause OOM Key: KYLIN-2748 URL: https://issues.apache.org/jira/browse/KYLIN-2748 Project: Kylin Issue Type: Bug Components: Query Engine Affects Versions: v2.0.0 Reporter: yiming.xu Assignee: liyang calcite code generation class can not gc cause OOM. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2720) Should not allow user to access to all tables' metadata of a project
[ https://issues.apache.org/jira/browse/KYLIN-2720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095713#comment-16095713 ] Shaofeng SHI commented on KYLIN-2720: - Sorry for the late; I'm merging it in local, if pass UT, I will push it to remote. Thanks! > Should not allow user to access to all tables' metadata of a project > > > Key: KYLIN-2720 > URL: https://issues.apache.org/jira/browse/KYLIN-2720 > Project: Kylin > Issue Type: Improvement >Reporter: qiumingming >Assignee: qiumingming > Fix For: v2.0.0, v2.1.0 > > Attachments: KYLIN-2720.patch > > > Currently, user can access to all tables and columns metadata of a specific > project as long as he can access to this project, which is not reasonable. > User should just allow to access to tables that he owned cubes dependent to. > However, user can see some other tables in the web UI in current version. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2720) Should not allow user to access to all tables' metadata of a project
[
https://issues.apache.org/jira/browse/KYLIN-2720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095729#comment-16095729
]
Shaofeng SHI commented on KYLIN-2720:
-
[~qmm] There are many errors when run "mvn test", it seems the change breaks
Spring injections; Was it okay in your side?:
{code}
2017-07-21 03:34:23,608 WARN [main] support.GenericApplicationContext:546 :
Exception encountered during context initialization - cancelling refresh
attempt: org.springframework.beans.factory.BeanCreationException: Error
creating bean with name 'queryController': Injection of autowired dependencies
failed; nested exception is
org.springframework.beans.factory.BeanCreationException: Could not autowire
field: private org.apache.kylin.rest.service.QueryService
org.apache.kylin.rest.controller.QueryController.queryService; nested exception
is org.springframework.beans.factory.BeanCreationException: Error creating bean
with name 'queryService': Injection of autowired dependencies failed; nested
exception is org.springframework.beans.factory.BeanCreationException: Could not
autowire field: private org.apache.kylin.rest.service.CubeService
org.apache.kylin.rest.service.QueryService.cubeService; nested exception is
org.springframework.beans.factory.NoSuchBeanDefinitionException: No qualifying
bean of type [org.apache.kylin.rest.service.CubeService] found for dependency:
expected at least 1 bean which qualifies as autowire candidate for this
dependency. Dependency annotations:
{@org.springframework.beans.factory.annotation.Autowired(required=true),
@org.springframework.beans.factory.annotation.Qualifier(value=cubeService)}
2017-07-21 03:34:23,610 ERROR [main] context.TestContextManager:231 : Caught
exception while allowing TestExecutionListener
[org.springframework.test.context.support.DependencyInjectionTestExecutionListener@58d15aa9]
to prepare test instance
[org.apache.kylin.rest.controller.QueryControllerTest@2bbaec57]
java.lang.IllegalStateException: Failed to load ApplicationContext
at
org.springframework.test.context.cache.DefaultCacheAwareContextLoaderDelegate.loadContext(DefaultCacheAwareContextLoaderDelegate.java:124)
at
org.springframework.test.context.support.DefaultTestContext.getApplicationContext(DefaultTestContext.java:83)
at
org.springframework.test.context.support.DependencyInjectionTestExecutionListener.injectDependencies(DependencyInjectionTestExecutionListener.java:117)
at
org.springframework.test.context.support.DependencyInjectionTestExecutionListener.prepareTestInstance(DependencyInjectionTestExecutionListener.java:83)
at
org.springframework.test.context.TestContextManager.prepareTestInstance(TestContextManager.java:228)
at
org.springframework.test.context.junit4.SpringJUnit4ClassRunner.createTest(SpringJUnit4ClassRunner.java:230)
at
org.springframework.test.context.junit4.SpringJUnit4ClassRunner$1.runReflectiveCall(SpringJUnit4ClassRunner.java:289)
at
org.junit.internal.runners.model.ReflectiveCallable.run(ReflectiveCallable.java:12)
at
org.springframework.test.context.junit4.SpringJUnit4ClassRunner.methodBlock(SpringJUnit4ClassRunner.java:291)
at
org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:249)
at
org.springframework.test.context.junit4.SpringJUnit4ClassRunner.runChild(SpringJUnit4ClassRunner.java:89)
at org.junit.runners.ParentRunner$3.run(ParentRunner.java:290)
at org.junit.runners.ParentRunner$1.schedule(ParentRunner.java:71)
at org.junit.runners.ParentRunner.runChildren(ParentRunner.java:288)
at org.junit.runners.ParentRunner.access$000(ParentRunner.java:58)
at org.junit.runners.ParentRunner$2.evaluate(ParentRunner.java:268)
at
org.junit.internal.runners.statements.RunBefores.evaluate(RunBefores.java:26)
at
org.springframework.test.context.junit4.statements.RunBeforeTestClassCallbacks.evaluate(RunBeforeTestClassCallbacks.java:61)
at
org.junit.internal.runners.statements.RunAfters.evaluate(RunAfters.java:27)
at
org.springframework.test.context.junit4.statements.RunAfterTestClassCallbacks.evaluate(RunAfterTestClassCallbacks.java:70)
at org.junit.runners.ParentRunner.run(ParentRunner.java:363)
at
org.springframework.test.context.junit4.SpringJUnit4ClassRunner.run(SpringJUnit4ClassRunner.java:193)
at
org.apache.maven.surefire.junit4.JUnit4Provider.execute(JUnit4Provider.java:367)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeWithRerun(JUnit4Provider.java:274)
at
org.apache.maven.surefire.junit4.JUnit4Provider.executeTestSet(JUnit4Provider.java:238)
at
org.apache.maven.surefire.junit4.JUnit4Provider.invoke(JUnit4Provider.java:161)
at
[jira] [Commented] (KYLIN-2721) Introduce a new metrics framework based on yammer metrics
[ https://issues.apache.org/jira/browse/KYLIN-2721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095648#comment-16095648 ] Billy Liu commented on KYLIN-2721: -- The dropwizard.metrics which is the new version of yammer metrics since 2014 is the most used metrics framework. If we are building new metrics feature, I'd like to suggest this library. It has better community support than earlier version and loose-coupling with hadoop libraries. > Introduce a new metrics framework based on yammer metrics > - > > Key: KYLIN-2721 > URL: https://issues.apache.org/jira/browse/KYLIN-2721 > Project: Kylin > Issue Type: New Feature >Affects Versions: v2.0.0 >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > Attachments: Metrics Framework.png > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2720) Should not allow user to access to all tables' metadata of a project
[
https://issues.apache.org/jira/browse/KYLIN-2720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095750#comment-16095750
]
qiumingming commented on KYLIN-2720:
I see. There is a mistake in the patch: @Qualifier("cubeService") should be
@Qualifier("cubeMgmtService"). I run well in kylin 1.5. But there are some code
differences between master-branch and 1.5. I am sorry for this. A new version
patch has been attached.
> Should not allow user to access to all tables' metadata of a project
>
>
> Key: KYLIN-2720
> URL: https://issues.apache.org/jira/browse/KYLIN-2720
> Project: Kylin
> Issue Type: Improvement
>Reporter: qiumingming
>Assignee: qiumingming
> Fix For: v2.0.0, v2.1.0
>
> Attachments: KYLIN-2720.patch
>
>
> Currently, user can access to all tables and columns metadata of a specific
> project as long as he can access to this project, which is not reasonable.
> User should just allow to access to tables that he owned cubes dependent to.
> However, user can see some other tables in the web UI in current version.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KYLIN-2744) Should return correct type for SUM measure in web
[ https://issues.apache.org/jira/browse/KYLIN-2744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095677#comment-16095677 ] Billy Liu commented on KYLIN-2744: -- Could we allow user to choose which is the best return type for that measure? The default return type could be the same type with column, but allow user to change it to others. We could limit the choice scope, to reduce the mistakes. > Should return correct type for SUM measure in web > - > > Key: KYLIN-2744 > URL: https://issues.apache.org/jira/browse/KYLIN-2744 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v2.0.0 >Reporter: kangkaisen >Assignee: kangkaisen > Attachments: KYLIN-2744.patch > > > Currently, Kylin return decimal type for the sum measure of double type, > which will result in wrong result. So, We should return correct type for SUM > measure in web. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Assigned] (KYLIN-2720) Should not allow user to access to all tables' metadata of a project
[ https://issues.apache.org/jira/browse/KYLIN-2720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI reassigned KYLIN-2720: --- Assignee: qiumingming > Should not allow user to access to all tables' metadata of a project > > > Key: KYLIN-2720 > URL: https://issues.apache.org/jira/browse/KYLIN-2720 > Project: Kylin > Issue Type: Improvement >Reporter: qiumingming >Assignee: qiumingming > Fix For: v2.0.0, v2.1.0 > > Attachments: KYLIN-2720.patch > > > Currently, user can access to all tables and columns metadata of a specific > project as long as he can access to this project, which is not reasonable. > User should just allow to access to tables that he owned cubes dependent to. > However, user can see some other tables in the web UI in current version. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2720) Should not allow user to access to all tables' metadata of a project
[ https://issues.apache.org/jira/browse/KYLIN-2720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qiumingming updated KYLIN-2720: --- Attachment: KYLIN-2720.patch > Should not allow user to access to all tables' metadata of a project > > > Key: KYLIN-2720 > URL: https://issues.apache.org/jira/browse/KYLIN-2720 > Project: Kylin > Issue Type: Improvement >Reporter: qiumingming >Assignee: qiumingming > Fix For: v2.0.0, v2.1.0 > > Attachments: KYLIN-2720.patch > > > Currently, user can access to all tables and columns metadata of a specific > project as long as he can access to this project, which is not reasonable. > User should just allow to access to tables that he owned cubes dependent to. > However, user can see some other tables in the web UI in current version. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2720) Should not allow user to access to all tables' metadata of a project
[ https://issues.apache.org/jira/browse/KYLIN-2720?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] qiumingming updated KYLIN-2720: --- Attachment: (was: KYLIN-2720.patch) > Should not allow user to access to all tables' metadata of a project > > > Key: KYLIN-2720 > URL: https://issues.apache.org/jira/browse/KYLIN-2720 > Project: Kylin > Issue Type: Improvement >Reporter: qiumingming >Assignee: qiumingming > Fix For: v2.0.0, v2.1.0 > > Attachments: KYLIN-2720.patch > > > Currently, user can access to all tables and columns metadata of a specific > project as long as he can access to this project, which is not reasonable. > User should just allow to access to tables that he owned cubes dependent to. > However, user can see some other tables in the web UI in current version. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Updated] (KYLIN-2721) Introduce a new metrics framework based on dropwizard metrics
[ https://issues.apache.org/jira/browse/KYLIN-2721?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong updated KYLIN-2721: -- Summary: Introduce a new metrics framework based on dropwizard metrics (was: Introduce a new metrics framework based on yammer metrics) > Introduce a new metrics framework based on dropwizard metrics > - > > Key: KYLIN-2721 > URL: https://issues.apache.org/jira/browse/KYLIN-2721 > Project: Kylin > Issue Type: New Feature >Affects Versions: v2.0.0 >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > Attachments: Metrics Framework.png > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2721) Introduce a new metrics framework based on yammer metrics
[ https://issues.apache.org/jira/browse/KYLIN-2721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095686#comment-16095686 ] Zhong Yanghong commented on KYLIN-2721: --- [~yimingliu], actually we are now using dropwizard.metrics. I'll change the title. > Introduce a new metrics framework based on yammer metrics > - > > Key: KYLIN-2721 > URL: https://issues.apache.org/jira/browse/KYLIN-2721 > Project: Kylin > Issue Type: New Feature >Affects Versions: v2.0.0 >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > Attachments: Metrics Framework.png > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2720) Should not allow user to access to all tables' metadata of a project
[ https://issues.apache.org/jira/browse/KYLIN-2720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095694#comment-16095694 ] qiumingming commented on KYLIN-2720: Hi Shaofeng SHI , What's going on about this issue. Many days have passed. > Should not allow user to access to all tables' metadata of a project > > > Key: KYLIN-2720 > URL: https://issues.apache.org/jira/browse/KYLIN-2720 > Project: Kylin > Issue Type: Improvement >Reporter: qiumingming > Fix For: v2.0.0, v2.1.0 > > Attachments: KYLIN-2720.patch > > > Currently, user can access to all tables and columns metadata of a specific > project as long as he can access to this project, which is not reasonable. > User should just allow to access to tables that he owned cubes dependent to. > However, user can see some other tables in the web UI in current version. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (KYLIN-2720) Should not allow user to access to all tables' metadata of a project
[ https://issues.apache.org/jira/browse/KYLIN-2720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095694#comment-16095694 ] qiumingming edited comment on KYLIN-2720 at 7/21/17 2:41 AM: - Hi Shaofeng SHI, What's going on about this issue. Many days have passed. was (Author: qmm): Hi Shaofeng SHI , What's going on about this issue. Many days have passed. > Should not allow user to access to all tables' metadata of a project > > > Key: KYLIN-2720 > URL: https://issues.apache.org/jira/browse/KYLIN-2720 > Project: Kylin > Issue Type: Improvement >Reporter: qiumingming > Fix For: v2.0.0, v2.1.0 > > Attachments: KYLIN-2720.patch > > > Currently, user can access to all tables and columns metadata of a specific > project as long as he can access to this project, which is not reasonable. > User should just allow to access to tables that he owned cubes dependent to. > However, user can see some other tables in the web UI in current version. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (KYLIN-2720) Should not allow user to access to all tables' metadata of a project
[ https://issues.apache.org/jira/browse/KYLIN-2720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095694#comment-16095694 ] qiumingming edited comment on KYLIN-2720 at 7/21/17 2:41 AM: - Hi Shaofeng SHI, What's going on about this issue. Many days have passed. was (Author: qmm): Hi Shaofeng SHI, What's going on about this issue. Many days have passed. > Should not allow user to access to all tables' metadata of a project > > > Key: KYLIN-2720 > URL: https://issues.apache.org/jira/browse/KYLIN-2720 > Project: Kylin > Issue Type: Improvement >Reporter: qiumingming > Fix For: v2.0.0, v2.1.0 > > Attachments: KYLIN-2720.patch > > > Currently, user can access to all tables and columns metadata of a specific > project as long as he can access to this project, which is not reasonable. > User should just allow to access to tables that he owned cubes dependent to. > However, user can see some other tables in the web UI in current version. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (KYLIN-2720) Should not allow user to access to all tables' metadata of a project
[ https://issues.apache.org/jira/browse/KYLIN-2720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095694#comment-16095694 ] qiumingming edited comment on KYLIN-2720 at 7/21/17 2:46 AM: - Hi [~Shaofeng SHI], What's going on about this issue. Many days have passed. was (Author: qmm): Hi Shaofeng SHI, What's going on about this issue. Many days have passed. > Should not allow user to access to all tables' metadata of a project > > > Key: KYLIN-2720 > URL: https://issues.apache.org/jira/browse/KYLIN-2720 > Project: Kylin > Issue Type: Improvement >Reporter: qiumingming > Fix For: v2.0.0, v2.1.0 > > Attachments: KYLIN-2720.patch > > > Currently, user can access to all tables and columns metadata of a specific > project as long as he can access to this project, which is not reasonable. > User should just allow to access to tables that he owned cubes dependent to. > However, user can see some other tables in the web UI in current version. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (KYLIN-2720) Should not allow user to access to all tables' metadata of a project
[ https://issues.apache.org/jira/browse/KYLIN-2720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095694#comment-16095694 ] qiumingming edited comment on KYLIN-2720 at 7/21/17 2:46 AM: - Hi [~Shaofengshi], What's going on about this issue. Many days have passed. was (Author: qmm): Hi [~Shaofeng SHI], What's going on about this issue. Many days have passed. > Should not allow user to access to all tables' metadata of a project > > > Key: KYLIN-2720 > URL: https://issues.apache.org/jira/browse/KYLIN-2720 > Project: Kylin > Issue Type: Improvement >Reporter: qiumingming > Fix For: v2.0.0, v2.1.0 > > Attachments: KYLIN-2720.patch > > > Currently, user can access to all tables and columns metadata of a specific > project as long as he can access to this project, which is not reasonable. > User should just allow to access to tables that he owned cubes dependent to. > However, user can see some other tables in the web UI in current version. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Comment Edited] (KYLIN-2720) Should not allow user to access to all tables' metadata of a project
[ https://issues.apache.org/jira/browse/KYLIN-2720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095694#comment-16095694 ] qiumingming edited comment on KYLIN-2720 at 7/21/17 2:40 AM: - Hi Shaofeng SHI , What's going on about this issue. Many days have passed. was (Author: qmm): Hi Shaofeng SHI , What's going on about this issue. Many days have passed. > Should not allow user to access to all tables' metadata of a project > > > Key: KYLIN-2720 > URL: https://issues.apache.org/jira/browse/KYLIN-2720 > Project: Kylin > Issue Type: Improvement >Reporter: qiumingming > Fix For: v2.0.0, v2.1.0 > > Attachments: KYLIN-2720.patch > > > Currently, user can access to all tables and columns metadata of a specific > project as long as he can access to this project, which is not reasonable. > User should just allow to access to tables that he owned cubes dependent to. > However, user can see some other tables in the web UI in current version. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2653) Spark cubing support HBase cluster with kerberos
[
https://issues.apache.org/jira/browse/KYLIN-2653?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095302#comment-16095302
]
liyang commented on KYLIN-2653:
---
{{KylinConfig}} & {{KylinConfigBase}} should be read-only by design. However
the patch breaks the design by loosening {{getAllProperties}} and
{{reloadKylinConfig}} to public. Can this be avoided?
Btw.. we should remove {{KylinConfigBase.setProperty()}} as well someday.
> Spark cubing support HBase cluster with kerberos
>
>
> Key: KYLIN-2653
> URL: https://issues.apache.org/jira/browse/KYLIN-2653
> Project: Kylin
> Issue Type: Bug
> Components: Spark Engine
>Affects Versions: v2.0.0
>Reporter: kangkaisen
>Assignee: kangkaisen
>
> Currently, Spark cubing doesn't support HBase cluster with kerberos.
> Temporarily，we could support HBase cluster with kerberos on Yarn client mode,
> because which is easy.
> In the long term，we should avoid access HBase in Spark cubing.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KYLIN-2715) StorageCleanupJob removes intermediate hive tables of jobs in progress
[
https://issues.apache.org/jira/browse/KYLIN-2715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095311#comment-16095311
]
liyang commented on KYLIN-2715:
---
Ah... I see. The problem maybe not about the last build ID. Should be below
lines:
{code}
if (!state.isFinalState()) {
String path =
JobBuilderSupport.getJobWorkingDir(engineConfig.getHdfsWorkingDirectory(),
jobId);
allHdfsPathsNeedToBeDeleted.remove(path);
{code}
It should remove all paths under the job working dir, not just the job working
dir itself.
> StorageCleanupJob removes intermediate hive tables of jobs in progress
> --
>
> Key: KYLIN-2715
> URL: https://issues.apache.org/jira/browse/KYLIN-2715
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v2.0.0
>Reporter: Alexander Sterligov
>
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KYLIN-2672) Only clean necessary cache for CubeMigrationCLI
[ https://issues.apache.org/jira/browse/KYLIN-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095331#comment-16095331 ] liyang commented on KYLIN-2672: --- True, tables requires reload too. I'm working on KYLIN-2717 that may help this JIRA too. Once table is project local, not shared by multiple projects globally, then reload a TableDesc won't impact multiple projects. Will this behavior solve the problem addressed by this JIRA? > Only clean necessary cache for CubeMigrationCLI > --- > > Key: KYLIN-2672 > URL: https://issues.apache.org/jira/browse/KYLIN-2672 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Affects Versions: v2.0.0 >Reporter: kangkaisen >Assignee: kangkaisen > Attachments: KYLIN-2672.patch > > > Currently, we simply clear ALL cache in CubeMigrationCLI. which will make a > few of queries slower in prod env when we have many tables, models, cubes and > migrate cube often. > So, we could only clean necessary cache for CubeMigrationCLI. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2721) Introduce a new metrics framework based on yammer metrics
[ https://issues.apache.org/jira/browse/KYLIN-2721?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095374#comment-16095374 ] liyang commented on KYLIN-2721: --- I see consensus as well as open questions. * Kylin will have to make a choice between yammer metrics or hadoop metrics2. Using both at the same time does not make sense. * There are different consumers of collected metrics. JMX client is one consumer; "reporting them to multiple storages" is another way of consumption. Design should allow both, and other potential consumers. * The proposal tends to implement JVM and OS metrics (cpuUsage, jvmCpuUsage, jvmMemAvail, javaFDs, sysFDs), which seems a re-invent of wheel to me. Please correct if I misunderstand anything. We can discuss on these open points one by one. > Introduce a new metrics framework based on yammer metrics > - > > Key: KYLIN-2721 > URL: https://issues.apache.org/jira/browse/KYLIN-2721 > Project: Kylin > Issue Type: New Feature >Affects Versions: v2.0.0 >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong > Attachments: Metrics Framework.png > > -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Commented] (KYLIN-2722) Introduce a new measure, called active reservoir, for actively pushing metrics to reporters
[
https://issues.apache.org/jira/browse/KYLIN-2722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095395#comment-16095395
]
liyang commented on KYLIN-2722:
---
Some quick comments on the patch.
* The package name could be better. It should align with existing measure
packages at least, like {{org.apache.kylin.measure.hllc}}. Maybe
{{org.apache.kylin.measure.reservoir}}?
* The config property has a naming convention [1]. Basically the property
namespace should map to java package. Thus it should be
{{kylin.measure.reservoir.active-default-class}} given above package name is
selected.
[1] http://kylin.apache.org/development/coding_naming_convention.html
> Introduce a new measure, called active reservoir, for actively pushing
> metrics to reporters
> ---
>
> Key: KYLIN-2722
> URL: https://issues.apache.org/jira/browse/KYLIN-2722
> Project: Kylin
> Issue Type: Sub-task
>Affects Versions: v2.0.0
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
> Attachments: APACHE-KYLIN-2722.patch
>
>
> For many existing metrics frameworks, they focus on maintaining metrics in
> memory independently for each instance. However, kylin server may consist of
> multiple instances. Thus we extend existing metrics framework by introducing
> *active reservoir* to actively push metrics to reporters which will report
> metrics of its instance to a unified storage.
> Here we introduced two *active reservoirs*. One is called
> {{BlockingReservoir}}, which will buffer the metrics. The other is called
> {{InstantReservoir}}, which owns no buffer and will directly push metrics to
> reporters.
> Generally, one *active reservoir* can push its metrics to multiple reporters
> and one reporter can only listen on one *active reservoir*.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
[jira] [Commented] (KYLIN-2744) Should return correct type for SUM measure in web
[ https://issues.apache.org/jira/browse/KYLIN-2744?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16095347#comment-16095347 ] liyang commented on KYLIN-2744: --- I know the history of sum double as decimal. Sum of double incurs small errors like 1+2=2.9. While this error should be perfectly fine, it worries pilot users in early days of Kylin. To make sure the sum result is super precise, it is then decided to use decimal to sum floats and doubles. I think we could return double for sum double by default, but returning decimal should still be available. At least for backward compatibility. > Should return correct type for SUM measure in web > - > > Key: KYLIN-2744 > URL: https://issues.apache.org/jira/browse/KYLIN-2744 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v2.0.0 >Reporter: kangkaisen >Assignee: kangkaisen > Attachments: KYLIN-2744.patch > > > Currently, Kylin return decimal type for the sum measure of double type, > which will result in wrong result. So, We should return correct type for SUM > measure in web. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
