[jira] [Updated] (KYLIN-3238) Fix NPE when CubeInstance.getLatestReadySegment() returns null
[
https://issues.apache.org/jira/browse/KYLIN-3238?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhong Yanghong updated KYLIN-3238:
--
Description: Current we suppose {{}}
> Fix NPE when CubeInstance.getLatestReadySegment() returns null
> --
>
> Key: KYLIN-3238
> URL: https://issues.apache.org/jira/browse/KYLIN-3238
> Project: Kylin
> Issue Type: Improvement
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>Priority: Major
>
> Current we suppose {{}}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3238) Fix NPE when CubeInstance.getLatestReadySegment() returns null
Zhong Yanghong created KYLIN-3238: - Summary: Fix NPE when CubeInstance.getLatestReadySegment() returns null Key: KYLIN-3238 URL: https://issues.apache.org/jira/browse/KYLIN-3238 Project: Kylin Issue Type: Improvement Reporter: Zhong Yanghong Assignee: Zhong Yanghong -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3235) add null check for SQL
[ https://issues.apache.org/jira/browse/KYLIN-3235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong updated KYLIN-3235: -- Issue Type: Sub-task (was: Improvement) Parent: KYLIN-3237 > add null check for SQL > -- > > Key: KYLIN-3235 > URL: https://issues.apache.org/jira/browse/KYLIN-3235 > Project: Kylin > Issue Type: Sub-task >Reporter: Zhong Yanghong >Assignee: Wang Ken >Priority: Major > Fix For: v2.3.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3237) Improve the way of dealing with NPE
Zhong Yanghong created KYLIN-3237: - Summary: Improve the way of dealing with NPE Key: KYLIN-3237 URL: https://issues.apache.org/jira/browse/KYLIN-3237 Project: Kylin Issue Type: Improvement Reporter: Zhong Yanghong Assignee: Zhong Yanghong -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3199) The login dialog should be closed when ldap user with no permission login correctly
[ https://issues.apache.org/jira/browse/KYLIN-3199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Peng Xing updated KYLIN-3199: - Component/s: (was: Security) Web > The login dialog should be closed when ldap user with no permission login > correctly > --- > > Key: KYLIN-3199 > URL: https://issues.apache.org/jira/browse/KYLIN-3199 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v2.3.0 >Reporter: Peng Xing >Assignee: Peng Xing >Priority: Minor > Labels: patch > Fix For: v2.3.0 > > Attachments: > 0001-KYLIN-3199-The-login-dialog-should-be-closed-when-ld.patch, > ldap_user_login.png > > > 1. Open ldap authentication, but I do not give the admin permission to group > 'xpGroup'; > 2. Create a ldap user 'xp', who belongs to group 'xpGroup', so this user has > none permission. > 3. When user 'xp' login in, the above bar has showed and been enabled, but > the login dialog still show. > 4. Then you can click any button on above bar. > Please refer to 'ldap_user_login.png' > I think the login dialog should be closed when you login in correctly, and > redirect to the 'Model' page, but this user has no permission. > I have modified this issue, please review the patch, thanks! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3236) The function 'reGenerateAdvancedDict()' has an error logical judgment, which will cause an exception when you edit the cube.
[
https://issues.apache.org/jira/browse/KYLIN-3236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peng Xing updated KYLIN-3236:
-
Description:
When remove deprecated distinct measures, the code has an error logical
judgment In function 'reGenerateAdvancedDict()', as follows:
{code:java}
//remove deprecated distinct measures
angular.forEach($scope.cubeMetaFrame.dictionaries, function (dict, index) {
if (distinctMeasures.indexOf(dict.column) === -1 &&
reuseColumns.indexOf(dict.column) === -1) {
$scope.cubeMetaFrame.dictionaries.splice(index, 1);
}
});
{code}
This will result in accidental deletion of advanced dictionaries.
Please review the patch, thanks!
was:
When remove deprecated distinct measures, the code has an error logical
judgment In function 'reGenerateAdvancedDict()', as follows:
{code:java}
//remove deprecated distinct measures
angular.forEach($scope.cubeMetaFrame.dictionaries, function (dict, index) {
if (distinctMeasures.indexOf(dict.column) === -1 &&
reuseColumns.indexOf(dict.column) === -1) {
$scope.cubeMetaFrame.dictionaries.splice(index, 1);
}
});
{code}
This will result in accidental deletion of advanced dictionaries.
> The function 'reGenerateAdvancedDict()' has an error logical judgment, which
> will cause an exception when you edit the cube.
>
>
> Key: KYLIN-3236
> URL: https://issues.apache.org/jira/browse/KYLIN-3236
> Project: Kylin
> Issue Type: Bug
> Components: Web
>Affects Versions: v2.3.0
>Reporter: Peng Xing
>Assignee: Peng Xing
>Priority: Major
> Labels: patch
> Attachments:
> 0001-KYLIN-3236-The-function-reGenerateAdvancedDict-has-a.patch
>
>
> When remove deprecated distinct measures, the code has an error logical
> judgment In function 'reGenerateAdvancedDict()', as follows:
> {code:java}
> //remove deprecated distinct measures
> angular.forEach($scope.cubeMetaFrame.dictionaries, function (dict, index)
> {
> if (distinctMeasures.indexOf(dict.column) === -1 &&
> reuseColumns.indexOf(dict.column) === -1) {
> $scope.cubeMetaFrame.dictionaries.splice(index, 1);
> }
> });
> {code}
> This will result in accidental deletion of advanced dictionaries.
> Please review the patch, thanks!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3236) The function 'reGenerateAdvancedDict()' has an error logical judgment, which will cause an exception when you edit the cube.
[
https://issues.apache.org/jira/browse/KYLIN-3236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peng Xing updated KYLIN-3236:
-
Summary: The function 'reGenerateAdvancedDict()' has an error logical
judgment, which will cause an exception when you edit the cube. (was: The
function 'reGenerateAdvancedDict()' has an error logical judgment, which will
cause the exception when you edit the cube.)
> The function 'reGenerateAdvancedDict()' has an error logical judgment, which
> will cause an exception when you edit the cube.
>
>
> Key: KYLIN-3236
> URL: https://issues.apache.org/jira/browse/KYLIN-3236
> Project: Kylin
> Issue Type: Bug
> Components: Web
>Affects Versions: v2.3.0
>Reporter: Peng Xing
>Assignee: Peng Xing
>Priority: Major
> Labels: patch
> Attachments:
> 0001-KYLIN-3236-The-function-reGenerateAdvancedDict-has-a.patch
>
>
> When remove deprecated distinct measures, the code has an error logical
> judgment In function 'reGenerateAdvancedDict()', as follows:
> {code:java}
> //remove deprecated distinct measures
> angular.forEach($scope.cubeMetaFrame.dictionaries, function (dict, index)
> {
> if (distinctMeasures.indexOf(dict.column) === -1 &&
> reuseColumns.indexOf(dict.column) === -1) {
> $scope.cubeMetaFrame.dictionaries.splice(index, 1);
> }
> });
> {code}
> This will result in accidental deletion of advanced dictionaries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3236) The function 'reGenerateAdvancedDict()' has an error logical judgment, which will cause the exception when you edit the cube.
[
https://issues.apache.org/jira/browse/KYLIN-3236?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Peng Xing updated KYLIN-3236:
-
Attachment: 0001-KYLIN-3236-The-function-reGenerateAdvancedDict-has-a.patch
> The function 'reGenerateAdvancedDict()' has an error logical judgment, which
> will cause the exception when you edit the cube.
> -
>
> Key: KYLIN-3236
> URL: https://issues.apache.org/jira/browse/KYLIN-3236
> Project: Kylin
> Issue Type: Bug
> Components: Web
>Affects Versions: v2.3.0
>Reporter: Peng Xing
>Assignee: Peng Xing
>Priority: Major
> Labels: patch
> Attachments:
> 0001-KYLIN-3236-The-function-reGenerateAdvancedDict-has-a.patch
>
>
> When remove deprecated distinct measures, the code has an error logical
> judgment In function 'reGenerateAdvancedDict()', as follows:
> {code:java}
> //remove deprecated distinct measures
> angular.forEach($scope.cubeMetaFrame.dictionaries, function (dict, index)
> {
> if (distinctMeasures.indexOf(dict.column) === -1 &&
> reuseColumns.indexOf(dict.column) === -1) {
> $scope.cubeMetaFrame.dictionaries.splice(index, 1);
> }
> });
> {code}
> This will result in accidental deletion of advanced dictionaries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3236) The function 'reGenerateAdvancedDict()' has an error logical judgment, which will cause the exception when you edit the cube.
Peng Xing created KYLIN-3236:
Summary: The function 'reGenerateAdvancedDict()' has an error
logical judgment, which will cause the exception when you edit the cube.
Key: KYLIN-3236
URL: https://issues.apache.org/jira/browse/KYLIN-3236
Project: Kylin
Issue Type: Bug
Components: Web
Affects Versions: v2.3.0
Reporter: Peng Xing
Assignee: Peng Xing
When remove deprecated distinct measures, the code has an error logical
judgment In function 'reGenerateAdvancedDict()', as follows:
{code:java}
//remove deprecated distinct measures
angular.forEach($scope.cubeMetaFrame.dictionaries, function (dict, index) {
if (distinctMeasures.indexOf(dict.column) === -1 &&
reuseColumns.indexOf(dict.column) === -1) {
$scope.cubeMetaFrame.dictionaries.splice(index, 1);
}
});
{code}
This will result in accidental deletion of advanced dictionaries.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3235) add null check for SQL
[ https://issues.apache.org/jira/browse/KYLIN-3235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong updated KYLIN-3235: -- Fix Version/s: v2.3.0 > add null check for SQL > -- > > Key: KYLIN-3235 > URL: https://issues.apache.org/jira/browse/KYLIN-3235 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Wang Ken >Priority: Major > Fix For: v2.3.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3235) add null check for SQL
[ https://issues.apache.org/jira/browse/KYLIN-3235?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong resolved KYLIN-3235. --- Resolution: Done > add null check for SQL > -- > > Key: KYLIN-3235 > URL: https://issues.apache.org/jira/browse/KYLIN-3235 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Wang Ken >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3235) add null check for SQL
[ https://issues.apache.org/jira/browse/KYLIN-3235?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16352085#comment-16352085 ] Zhong Yanghong commented on KYLIN-3235: --- https://github.com/apache/kylin/commit/61f36a270aa218fb3b221f695b74f86e6b433d8d > add null check for SQL > -- > > Key: KYLIN-3235 > URL: https://issues.apache.org/jira/browse/KYLIN-3235 > Project: Kylin > Issue Type: Improvement >Reporter: Zhong Yanghong >Assignee: Wang Ken >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3235) add null check for SQL
Zhong Yanghong created KYLIN-3235: - Summary: add null check for SQL Key: KYLIN-3235 URL: https://issues.apache.org/jira/browse/KYLIN-3235 Project: Kylin Issue Type: Improvement Reporter: Zhong Yanghong Assignee: Wang Ken -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KYLIN-2898) Introduce memcached as a distributed cache for queries
[ https://issues.apache.org/jira/browse/KYLIN-2898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong reassigned KYLIN-2898: - Assignee: Wang Ken (was: Zhong Yanghong) > Introduce memcached as a distributed cache for queries > -- > > Key: KYLIN-2898 > URL: https://issues.apache.org/jira/browse/KYLIN-2898 > Project: Kylin > Issue Type: Sub-task > Components: Query Engine >Affects Versions: v2.1.0 >Reporter: Zhong Yanghong >Assignee: Wang Ken >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-2802) Enable cube planner phase one
[ https://issues.apache.org/jira/browse/KYLIN-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong closed KYLIN-2802. - Resolution: Done > Enable cube planner phase one > - > > Key: KYLIN-2802 > URL: https://issues.apache.org/jira/browse/KYLIN-2802 > Project: Kylin > Issue Type: Sub-task >Affects Versions: v2.1.0 >Reporter: Zhong Yanghong >Assignee: Wang Ken >Priority: Major > Fix For: v2.2.0 > > > This phase happens during cube building. To be able to trigger cube planner, > one of the following preconditions should be satisfied. > * The cube is empty > * Job is for refreshing the only one segment. > Since cube planner is based on cuboid row count, if the first segment is > empty and there's no useful row count info, cube planner should not be > triggered in this phase. > We also introduce a threshold. If the total number of defined cuboids is less > than this threshold, cube planner should not be triggered. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (KYLIN-2802) Enable cube planner phase one
[ https://issues.apache.org/jira/browse/KYLIN-2802?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong reopened KYLIN-2802: --- Assignee: Wang Ken (was: Zhong Yanghong) > Enable cube planner phase one > - > > Key: KYLIN-2802 > URL: https://issues.apache.org/jira/browse/KYLIN-2802 > Project: Kylin > Issue Type: Sub-task >Affects Versions: v2.1.0 >Reporter: Zhong Yanghong >Assignee: Wang Ken >Priority: Major > Fix For: v2.2.0 > > > This phase happens during cube building. To be able to trigger cube planner, > one of the following preconditions should be satisfied. > * The cube is empty > * Job is for refreshing the only one segment. > Since cube planner is based on cuboid row count, if the first segment is > empty and there's no useful row count info, cube planner should not be > triggered in this phase. > We also introduce a threshold. If the total number of defined cuboids is less > than this threshold, cube planner should not be triggered. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-2730) Introduce genetic algorithm for cube planner
[ https://issues.apache.org/jira/browse/KYLIN-2730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong closed KYLIN-2730. - Resolution: Done > Introduce genetic algorithm for cube planner > > > Key: KYLIN-2730 > URL: https://issues.apache.org/jira/browse/KYLIN-2730 > Project: Kylin > Issue Type: Sub-task >Affects Versions: v2.1.0 >Reporter: Zhong Yanghong >Assignee: Wang Ken >Priority: Major > Fix For: v2.2.0 > > > The basic idea of this algorithm is to evolve the generation of chromosome, > where a chromosome is a cuboid set to prebuilt. The detailed steps are as > follows: > # Initialize a generation of chromosome > # Evolve the generation by selection, crossover and mutation by N round. > # From the final generation, choose a best chromosome that is a best cuboid > set to recommend. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (KYLIN-2730) Introduce genetic algorithm for cube planner
[ https://issues.apache.org/jira/browse/KYLIN-2730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong reopened KYLIN-2730: --- Assignee: Wang Ken (was: Zhong Yanghong) > Introduce genetic algorithm for cube planner > > > Key: KYLIN-2730 > URL: https://issues.apache.org/jira/browse/KYLIN-2730 > Project: Kylin > Issue Type: Sub-task >Affects Versions: v2.1.0 >Reporter: Zhong Yanghong >Assignee: Wang Ken >Priority: Major > Fix For: v2.2.0 > > > The basic idea of this algorithm is to evolve the generation of chromosome, > where a chromosome is a cuboid set to prebuilt. The detailed steps are as > follows: > # Initialize a generation of chromosome > # Evolve the generation by selection, crossover and mutation by N round. > # From the final generation, choose a best chromosome that is a best cuboid > set to recommend. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-2729) Introduce greedy algorithm for cube planner
[ https://issues.apache.org/jira/browse/KYLIN-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong closed KYLIN-2729. - Resolution: Done > Introduce greedy algorithm for cube planner > --- > > Key: KYLIN-2729 > URL: https://issues.apache.org/jira/browse/KYLIN-2729 > Project: Kylin > Issue Type: Sub-task >Affects Versions: v2.1.0 >Reporter: Zhong Yanghong >Assignee: Wang Ken >Priority: Major > Fix For: v2.2.0 > > > The basic idea of greedy algorithm is to select the highest benefit cuboid > round by round. Notice that the result got from this algorithm is not the > global optimization one -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-2728) Introduce a new cuboid scheduler based on cuboid tree rather than static rules
[
https://issues.apache.org/jira/browse/KYLIN-2728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhong Yanghong closed KYLIN-2728.
-
Resolution: Done
> Introduce a new cuboid scheduler based on cuboid tree rather than static rules
> --
>
> Key: KYLIN-2728
> URL: https://issues.apache.org/jira/browse/KYLIN-2728
> Project: Kylin
> Issue Type: Sub-task
>Affects Versions: v2.1.0
>Reporter: Zhong Yanghong
>Assignee: Ma Gang
>Priority: Major
> Fix For: v2.2.0
>
>
> Rather than depending on static rules to define a parent-child relationship
> between cuboids, a cuboid tree can be constructed for this relationship. In
> this tree, the root is the base cuboid. For a cuboid, it may have many legal
> parent cuboids. We should choose the {color:#f79232}best parent {color}cuboid
> as its parent in this tree. Then how to define the best? It's related two
> rules as follows:
> * If we know the row count, choose the one with the least row count
> * Otherwise, choose the smallest cuboid
> Based on the cuboid tree, we introduced a new cuboid scheduler called
> {{TreeCuboidScheduler}}. And both cube building & query can be based on
> {{TreeCuboidScheduler}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Reopened] (KYLIN-2729) Introduce greedy algorithm for cube planner
[ https://issues.apache.org/jira/browse/KYLIN-2729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhong Yanghong reopened KYLIN-2729: --- Assignee: Wang Ken (was: Zhong Yanghong) > Introduce greedy algorithm for cube planner > --- > > Key: KYLIN-2729 > URL: https://issues.apache.org/jira/browse/KYLIN-2729 > Project: Kylin > Issue Type: Sub-task >Affects Versions: v2.1.0 >Reporter: Zhong Yanghong >Assignee: Wang Ken >Priority: Major > Fix For: v2.2.0 > > > The basic idea of greedy algorithm is to select the highest benefit cuboid > round by round. Notice that the result got from this algorithm is not the > global optimization one -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (KYLIN-2728) Introduce a new cuboid scheduler based on cuboid tree rather than static rules
[
https://issues.apache.org/jira/browse/KYLIN-2728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhong Yanghong reopened KYLIN-2728:
---
Assignee: Ma Gang (was: Zhong Yanghong)
> Introduce a new cuboid scheduler based on cuboid tree rather than static rules
> --
>
> Key: KYLIN-2728
> URL: https://issues.apache.org/jira/browse/KYLIN-2728
> Project: Kylin
> Issue Type: Sub-task
>Affects Versions: v2.1.0
>Reporter: Zhong Yanghong
>Assignee: Ma Gang
>Priority: Major
> Fix For: v2.2.0
>
>
> Rather than depending on static rules to define a parent-child relationship
> between cuboids, a cuboid tree can be constructed for this relationship. In
> this tree, the root is the base cuboid. For a cuboid, it may have many legal
> parent cuboids. We should choose the {color:#f79232}best parent {color}cuboid
> as its parent in this tree. Then how to define the best? It's related two
> rules as follows:
> * If we know the row count, choose the one with the least row count
> * Otherwise, choose the smallest cuboid
> Based on the cuboid tree, we introduced a new cuboid scheduler called
> {{TreeCuboidScheduler}}. And both cube building & query can be based on
> {{TreeCuboidScheduler}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (KYLIN-2728) Introduce a new cuboid scheduler based on cuboid tree rather than static rules
[
https://issues.apache.org/jira/browse/KYLIN-2728?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhong Yanghong closed KYLIN-2728.
-
Resolution: Fixed
> Introduce a new cuboid scheduler based on cuboid tree rather than static rules
> --
>
> Key: KYLIN-2728
> URL: https://issues.apache.org/jira/browse/KYLIN-2728
> Project: Kylin
> Issue Type: Sub-task
>Affects Versions: v2.1.0
>Reporter: Zhong Yanghong
>Assignee: Ma Gang
>Priority: Major
> Fix For: v2.2.0
>
>
> Rather than depending on static rules to define a parent-child relationship
> between cuboids, a cuboid tree can be constructed for this relationship. In
> this tree, the root is the base cuboid. For a cuboid, it may have many legal
> parent cuboids. We should choose the {color:#f79232}best parent {color}cuboid
> as its parent in this tree. Then how to define the best? It's related two
> rules as follows:
> * If we know the row count, choose the one with the least row count
> * Otherwise, choose the smallest cuboid
> Based on the cuboid tree, we introduced a new cuboid scheduler called
> {{TreeCuboidScheduler}}. And both cube building & query can be based on
> {{TreeCuboidScheduler}}.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (KYLIN-2722) Introduce a new measure, called active reservoir, for actively pushing metrics to reporters
[
https://issues.apache.org/jira/browse/KYLIN-2722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhong Yanghong reassigned KYLIN-2722:
-
Assignee: Wang Ken (was: Zhong Yanghong)
> Introduce a new measure, called active reservoir, for actively pushing
> metrics to reporters
> ---
>
> Key: KYLIN-2722
> URL: https://issues.apache.org/jira/browse/KYLIN-2722
> Project: Kylin
> Issue Type: Sub-task
>Reporter: Zhong Yanghong
>Assignee: Wang Ken
>Priority: Major
> Fix For: v2.3.0
>
> Attachments: APACHE-KYLIN-2722.patch
>
>
> For many existing metrics frameworks, they focus on maintaining metrics in
> memory independently for each instance. However, kylin server may consist of
> multiple instances. Thus we extend existing metrics framework by introducing
> *active reservoir* to actively push metrics to reporters which will report
> metrics of its instance to a unified storage.
> Here we introduced two *active reservoirs*. One is called
> {{BlockingReservoir}}, which will buffer the metrics. The other is called
> {{InstantReservoir}}, which owns no buffer and will directly push metrics to
> reporters.
> Generally, one *active reservoir* can push its metrics to multiple reporters
> and one reporter can only listen on one *active reservoir*.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KYLIN-2962) drop error job not delete segment
[ https://issues.apache.org/jira/browse/KYLIN-2962?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhixiong Chen resolved KYLIN-2962. -- Resolution: Fixed Fix Version/s: v2.3.0 > drop error job not delete segment > - > > Key: KYLIN-2962 > URL: https://issues.apache.org/jira/browse/KYLIN-2962 > Project: Kylin > Issue Type: Bug > Components: Job Engine, REST Service >Reporter: Pan, Julian >Assignee: Dong Li >Priority: Major > Labels: scope > Fix For: v2.3.0 > > > When job error and user drop the job, which may cause issue if drop job does > not clean the error segment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-2962) drop error job not delete segment
[ https://issues.apache.org/jira/browse/KYLIN-2962?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16352012#comment-16352012 ] Zhixiong Chen commented on KYLIN-2962: -- This jira is duplicated with KYLIN-3228. https://issues.apache.org/jira/browse/KYLIN-3228 You could check comments in KYLIN-3228. So I close this jira. > drop error job not delete segment > - > > Key: KYLIN-2962 > URL: https://issues.apache.org/jira/browse/KYLIN-2962 > Project: Kylin > Issue Type: Bug > Components: Job Engine, REST Service >Reporter: Pan, Julian >Assignee: Dong Li >Priority: Major > Labels: scope > > When job error and user drop the job, which may cause issue if drop job does > not clean the error segment. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KYLIN-2886) export csv file error
[
https://issues.apache.org/jira/browse/KYLIN-2886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhixiong Chen reassigned KYLIN-2886:
Assignee: Zhixiong Chen (was: Zhong,Jason)
> export csv file error
> -
>
> Key: KYLIN-2886
> URL: https://issues.apache.org/jira/browse/KYLIN-2886
> Project: Kylin
> Issue Type: Bug
> Components: REST Service
>Reporter: lynn
>Assignee: Zhixiong Chen
>Priority: Major
> Labels: scope
> Attachments: 1.png, 21.png
>
>
> SQL:
> select price, sell_id from kylin_sales order by sell_id compute max(price) by
> sell_id;
> when i run a "compute by sub statement"，obviously, it does not support the
> statement.
> !1.png|thumbnail!
> It doesn't matter! But when i want to export the results which previously
> executed successfully to the csv file, it still throws exception!
> !21.png|thumbnail!
> REST Service return message:
> {
> "code": "999",
> "data": null,
> "msg": "Encountered \"compute\" at line 1, column 57. Was expecting one
> of: \"LIMIT\" ... \"OFFSET\" ... \"FETCH\" ...
> \",\" ... \"ASC\" ... \"DESC\" ... \"NULLS\" ... \"NOT\" ...
>\"IN\" ... \"BETWEEN\" ... \"LIKE\" ... \"SIMILAR\" ...
> \"=\" ... \">\" ... \"<\" ... \"<=\" ... \">=\" ...
> \"<>\" ... \"!=\" ... \"+\" ... \"-\" ... \"*\" ... \"/\"
> ... \"||\" ... \"AND\" ... \"OR\" ... \"IS\" ...
> \"MEMBER\" ... \"SUBMULTISET\" ... \"MULTISET\" ... \"[\" ...
> \".\" ... \"(\" ... ",
> "stacktrace": "org.apache.kylin.rest.exception.InternalErrorException:
> Encountered \"compute\" at line 1, column 57. Was expecting one of:
> \"LIMIT\" ... \"OFFSET\" ... \"FETCH\" ... \",\" ...
> \"ASC\" ... \"DESC\" ... \"NULLS\" ... \"NOT\" ... \"IN\" ...
> \"BETWEEN\" ... \"LIKE\" ... \"SIMILAR\" ... \"=\" ...
> \">\" ... \"<\" ... \"<=\" ... \">=\" ... \"<>\" ...
> \"!=\" ... \"+\" ... \"-\" ... \"*\" ... \"/\" ... \"||\"
> ... \"AND\" ... \"OR\" ... \"IS\" ... \"MEMBER\" ...
> \"SUBMULTISET\" ... \"MULTISET\" ... \"[\" ... \".\" ...
> \"(\" ... \n\tat
> org.apache.kylin.rest.service.QueryService.doQueryWithCache(QueryService.java:440)\n\tat
>
> org.apache.kylin.rest.controller.QueryController.downloadQueryResult(QueryController.java:121)\n\tat
> sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)\n\tat
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)\n\tat
>
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)\n\tat
> java.lang.reflect.Method.invoke(Method.java:498)\n\tat
> org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:221)\n\tat
>
> org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:136)\n\tat
>
> org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:110)\n\tat
>
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:832)\n\tat
>
> org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:743)\n\tat
>
> org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:85)\n\tat
>
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:961)\n\tat
>
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:895)\n\tat
>
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:967)\n\tat
>
> org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:858)\n\tat
> javax.servlet.http.HttpServlet.service(HttpServlet.java:624)\n\tat
> org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:843)\n\tat
> javax.servlet.http.HttpServlet.service(HttpServlet.java:731)\n\tat
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)\n\tat
>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)\n\tat
> org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)\n\tat
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)\n\tat
>
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)\n\tat
>
>
[jira] [Assigned] (KYLIN-2900) Web Interface MR Link BUG
[ https://issues.apache.org/jira/browse/KYLIN-2900?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Zhixiong Chen reassigned KYLIN-2900: Assignee: Zhixiong Chen (was: Zhong,Jason) > Web Interface MR Link BUG > - > > Key: KYLIN-2900 > URL: https://issues.apache.org/jira/browse/KYLIN-2900 > Project: Kylin > Issue Type: Bug > Components: REST Service >Reporter: lynn >Assignee: Zhixiong Chen >Priority: Major > Attachments: 1.png > > > When I notice the detail info of the MR Job, I find the Link starts with > “unsafe” occasionally , as the picture "1.png". > !1.png|thumbnail! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3045) Too many open files raised from new Configuration()
[ https://issues.apache.org/jira/browse/KYLIN-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351989#comment-16351989 ] liyang commented on KYLIN-3045: --- The "fixed version=2.3.0" removed. > Too many open files raised from new Configuration() > --- > > Key: KYLIN-3045 > URL: https://issues.apache.org/jira/browse/KYLIN-3045 > Project: Kylin > Issue Type: Bug >Reporter: liyang >Assignee: liyang >Priority: Major > Labels: scope > > Try to refactor MapReduceExecutable to avoid re-creating Configuration too > many times. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3045) Too many open files raised from new Configuration()
[ https://issues.apache.org/jira/browse/KYLIN-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyang updated KYLIN-3045: -- Fix Version/s: (was: v2.3.0) > Too many open files raised from new Configuration() > --- > > Key: KYLIN-3045 > URL: https://issues.apache.org/jira/browse/KYLIN-3045 > Project: Kylin > Issue Type: Bug >Reporter: liyang >Assignee: liyang >Priority: Major > Labels: scope > > Try to refactor MapReduceExecutable to avoid re-creating Configuration too > many times. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3045) Too many open files raised from new Configuration()
[ https://issues.apache.org/jira/browse/KYLIN-3045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351988#comment-16351988 ] liyang commented on KYLIN-3045: --- My attempt didn't work out. Seems it did not hit the root cause. > Too many open files raised from new Configuration() > --- > > Key: KYLIN-3045 > URL: https://issues.apache.org/jira/browse/KYLIN-3045 > Project: Kylin > Issue Type: Bug >Reporter: liyang >Assignee: liyang >Priority: Major > Labels: scope > Fix For: v2.3.0 > > > Try to refactor MapReduceExecutable to avoid re-creating Configuration too > many times. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3227) Automatically remove the blank at the end of lines in properties files
[ https://issues.apache.org/jira/browse/KYLIN-3227?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guangyao Li updated KYLIN-3227: --- Description: In the 'kylin.properties' file: eg: kylin.engine.spark-conf.spark.executor.instances=1 There is a blank space bebind "1" The Kylin will startup failure. was: In the 'kylin.properties' file: eg: kap.storage.columnar.spark-conf.spark.executor.instances=1 There is a blank space bebind "1" The Kylin will startup failure. > Automatically remove the blank at the end of lines in properties files > -- > > Key: KYLIN-3227 > URL: https://issues.apache.org/jira/browse/KYLIN-3227 > Project: Kylin > Issue Type: Bug > Components: Environment >Affects Versions: v2.2.0 >Reporter: Guangyao Li >Assignee: Wang Cheng >Priority: Minor > Fix For: v2.3.0 > > > In the 'kylin.properties' file: > eg: kylin.engine.spark-conf.spark.executor.instances=1 > There is a blank space bebind "1" > The Kylin will startup failure. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3228) Should remove the related segment when deleting a job
[ https://issues.apache.org/jira/browse/KYLIN-3228?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351938#comment-16351938 ] Shaofeng SHI commented on KYLIN-3228: - Copy youcheng's comment in mailing list: "Actually I have checked the code in master, the function dropJob has been improved already, with commit 8191b8ed39c32d8cfa30eaa596c03552db736a70 before delete job, check if exist relatedCube, if exist throw badrequestexception. Maybe just one little improvement, for the web, do not show "drop" when the job status is 'ERROR'" And the web gui issue is duplicated with KYLIN-3215 > Should remove the related segment when deleting a job > - > > Key: KYLIN-3228 > URL: https://issues.apache.org/jira/browse/KYLIN-3228 > Project: Kylin > Issue Type: Bug > Components: REST Service, Web >Affects Versions: v2.2.0 >Reporter: youcheng.zhang >Assignee: Zhixiong Chen >Priority: Major > Fix For: v2.3.0 > > > When delete error job in the web page, kylin doesn't delete the relative > segment, and the segment remain with status not ready, you can't delete it or > refresh it either. > Even after use StorageCleanupJob, the segment still remains. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3228) Should remove the related segment when deleting a job
[ https://issues.apache.org/jira/browse/KYLIN-3228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI resolved KYLIN-3228. - Resolution: Fixed > Should remove the related segment when deleting a job > - > > Key: KYLIN-3228 > URL: https://issues.apache.org/jira/browse/KYLIN-3228 > Project: Kylin > Issue Type: Bug > Components: REST Service, Web >Affects Versions: v2.2.0 >Reporter: youcheng.zhang >Assignee: Zhixiong Chen >Priority: Major > Fix For: v2.3.0 > > > When delete error job in the web page, kylin doesn't delete the relative > segment, and the segment remain with status not ready, you can't delete it or > refresh it either. > Even after use StorageCleanupJob, the segment still remains. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3228) Should remove the related segment when deleting a job
[ https://issues.apache.org/jira/browse/KYLIN-3228?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-3228: Fix Version/s: v2.3.0 > Should remove the related segment when deleting a job > - > > Key: KYLIN-3228 > URL: https://issues.apache.org/jira/browse/KYLIN-3228 > Project: Kylin > Issue Type: Bug > Components: REST Service, Web >Affects Versions: v2.2.0 >Reporter: youcheng.zhang >Assignee: Zhixiong Chen >Priority: Major > Fix For: v2.3.0 > > > When delete error job in the web page, kylin doesn't delete the relative > segment, and the segment remain with status not ready, you can't delete it or > refresh it either. > Even after use StorageCleanupJob, the segment still remains. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3071) Add config to reuse dict to reduce dict size
[ https://issues.apache.org/jira/browse/KYLIN-3071?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu updated KYLIN-3071: - Fix Version/s: v2.3.0 > Add config to reuse dict to reduce dict size > - > > Key: KYLIN-3071 > URL: https://issues.apache.org/jira/browse/KYLIN-3071 > Project: Kylin > Issue Type: Improvement >Reporter: Yang Hao >Assignee: Yang Hao >Priority: Major > Fix For: v2.3.0 > > Attachments: KYLIN-3071.apache-master.001.patch > > > When calling DictionaryManager.trySaveNewDict, and growing dict is not > enabled, it only use the history dict which is equal, it may generate many > dict. We should supply a config to use contains instead of equal to reuse old > dict. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-2932) Simplify the thread model for in-memory cubing
[ https://issues.apache.org/jira/browse/KYLIN-2932?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu updated KYLIN-2932: - Fix Version/s: v2.3.0 > Simplify the thread model for in-memory cubing > -- > > Key: KYLIN-2932 > URL: https://issues.apache.org/jira/browse/KYLIN-2932 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: Wang Ken >Assignee: Wang Ken >Priority: Major > Fix For: v2.3.0 > > Attachments: APACHE-KYLIN-2932.patch > > > The current implementation uses split threads, task threads and main thread > to do the cube building, there is complex join and error handling logic. > The new implement leverages the ForkJoinPool from JDK, the event split logic > is handled in > main thread. Cuboid task and sub-tasks are handled in fork join pool, cube > results are collected > async and can be write to output earlier. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3219) Fix NPE in CubingJob
[ https://issues.apache.org/jira/browse/KYLIN-3219?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu updated KYLIN-3219: - Fix Version/s: v2.3.0 > Fix NPE in CubingJob > > > Key: KYLIN-3219 > URL: https://issues.apache.org/jira/browse/KYLIN-3219 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v2.3.0 >Reporter: Peng Xing >Assignee: Peng Xing >Priority: Minor > Fix For: v2.3.0 > > Attachments: 0001-KYLIN-3219.patch > > > When using spark engine, there is no 'Build Cube In-Mem' step task, > error log: > java.lang.NullPointerException > at org.apache.kylin.engine.mr.CubingJob.updateMetrics(CubingJob.java:284) > at org.apache.kylin.engine.mr.CubingJob.onStatusChange(CubingJob.java:272) > at > org.apache.kylin.job.execution.DefaultChainedExecutable.onExecuteFinished(DefaultChainedExecutable.java:132) > at org.apache.kylin.engine.mr.CubingJob.onExecuteFinished(CubingJob.java:266) > at > org.apache.kylin.job.execution.AbstractExecutable.onExecuteFinishedWithRetry(AbstractExecutable.java:98) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:175) > at > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:300) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622) > at java.lang.Thread.run(Thread.java:745) > 2018-01-31 18:15:20,443 ERROR [pool-11-thread-10] > threadpool.DefaultScheduler:302 : ExecuteException > job:693e2cd6-c7e7-4ed9-b1d6-e7ad86f5c501 > org.apache.kylin.job.exception.ExecuteException: > java.lang.NullPointerException > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:179) > at > org.apache.kylin.job.impl.threadpool.DefaultScheduler$JobRunner.run(DefaultScheduler.java:300) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1152) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:622) > at java.lang.Thread.run(Thread.java:745) > Caused by: java.lang.NullPointerException > at org.apache.kylin.engine.mr.CubingJob.updateMetrics(CubingJob.java:284) > at org.apache.kylin.engine.mr.CubingJob.onStatusChange(CubingJob.java:272) > at > org.apache.kylin.job.execution.DefaultChainedExecutable.onExecuteFinished(DefaultChainedExecutable.java:132) > at org.apache.kylin.engine.mr.CubingJob.onExecuteFinished(CubingJob.java:266) > at > org.apache.kylin.job.execution.AbstractExecutable.onExecuteFinishedWithRetry(AbstractExecutable.java:98) > at > org.apache.kylin.job.execution.AbstractExecutable.execute(AbstractExecutable.java:175) > ... 4 more -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3222) The function of editing 'Advanced Dictionaries' in cube is unavailable.
[ https://issues.apache.org/jira/browse/KYLIN-3222?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu updated KYLIN-3222: - Fix Version/s: v2.3.0 > The function of editing 'Advanced Dictionaries' in cube is unavailable. > --- > > Key: KYLIN-3222 > URL: https://issues.apache.org/jira/browse/KYLIN-3222 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v2.3.0 >Reporter: Peng Xing >Assignee: Peng Xing >Priority: Major > Labels: patch > Fix For: v2.3.0 > > Attachments: > 0001-KYLIN-3222-The-function-of-editing-Advanced-Dictiona.patch, > modify_advanced_dictionary.png, modify_advanced_dictionary_no_effect.png > > > There is a problem about editing 'Advanced Dictionaries' in cube, refer to > 'modify_advanced_dictionary.png' and > 'modify_advanced_dictionary_no_effect.png'. > Please review the patch, thanks! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3214) Initialize ExternalAclProvider when starting kylin
[ https://issues.apache.org/jira/browse/KYLIN-3214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu updated KYLIN-3214: - Fix Version/s: v2.3.0 > Initialize ExternalAclProvider when starting kylin > -- > > Key: KYLIN-3214 > URL: https://issues.apache.org/jira/browse/KYLIN-3214 > Project: Kylin > Issue Type: Improvement > Components: Security >Affects Versions: v2.2.0 >Reporter: peng.jianhua >Assignee: peng.jianhua >Priority: Minor > Fix For: v2.3.0 > > Attachments: 0001-KYLIN-3214.patch > > > Currently, ExternalAclProvider is initialized only when calling acl related > api. > Manage ACL through Ranger，ranger can not get the status of the > ExternalAclProvider in time because of ExternalAclProvider not initialized > when starting kylin. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3221) Some improvements for lookup table
[
https://issues.apache.org/jira/browse/KYLIN-3221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351822#comment-16351822
]
Billy Liu commented on KYLIN-3221:
--
+1
> Some improvements for lookup table
> ---
>
> Key: KYLIN-3221
> URL: https://issues.apache.org/jira/browse/KYLIN-3221
> Project: Kylin
> Issue Type: Improvement
> Components: Job Engine, Metadata, Query Engine
>Reporter: Ma Gang
>Assignee: Ma Gang
>Priority: Major
>
> There are two limitations for current look table design:
> # lookup table size is limited, because table snapshot need to be cached in
> Kylin server, too large snapshot table will break the server.
> # lookup table snapshot references are stored in all segments of the cube,
> cannot support global snapshot table, the global snapshot table means when
> the lookup table is updated, it will take effective for all segments.
> To resolve the above limitations, we decide to do some improvements for the
> existing lookup table design, below is the initial document, any comments and
> suggestions are welcome.
> h2. Metadata
> Will add a new property in CubeDesc to describe how lookup tables will be
> snapshot, it can be defined during the cube design
> |{{@JsonProperty}}{{(}}{{"snapshot_table_desc_list"}}{{)}}
> {{private}} {{List snapshotTableDescList =
> Collections.emptyList();}}|
> SnapshotTableDesc defines how table is stored and whether it is global or
> not, currently we can support two types of store:
> # "metaStore", table snapshot is stored in the metadata store, it is the
> same as current design, and this is the default option.
> # "hbaseStore', table snapshot is stored in an additional hbase table.
> |{{@JsonProperty}}{{(}}{{"table_name"}}{{)}}
> {{private}} {{String tableName;}}
>
> {{@JsonProperty}}{{(}}{{"store_type"}}{{)}}
> {{private}} {{String snapshotStorageType = }}{{"metaStore"}}{{;}}
>
> {{@JsonProperty}}{{(}}{{"global"}}{{)}}
> {{private}} {{boolean}} {{global = }}{{false}}{{;}}|
>
> Add 'snapshots' property in CubeInstance, to store snapshots resource path
> for each table, when the table snapshot is set to global in cube design:
> |{{@JsonProperty}}{{(}}{{"snapshots"}}{{)}}
> {{private}} {{Map snapshots; }}{{// tableName ->
> tableResoucePath mapping}}|
>
> Add new meta model ExtTableSnapshot to describe the extended table snapshot
> information, the information is stored in a new metastore path:
> /ext_table_snapshot/\{tableName}/\{uuid}.snapshot, the metadata including
> following info:
> |{{@JsonProperty}}{{(}}{{"tableName"}}{{)}}
> {{private}} {{String tableName;}}
>
> {{@JsonProperty}}{{(}}{{"signature"}}{{)}}
> {{private}} {{TableSignature signature;}}
>
> {{@JsonProperty}}{{(}}{{"storage_location_identifier"}}{{)}}
> {{private}} {{String storageLocationIdentifier;}}
>
> {{@JsonProperty}}{{(}}{{"size"}}{{)}}
> {{private}} {{long}} {{size;}}
>
> {{@JsonProperty}}{{(}}{{"row_cnt"}}{{)}}
> {{private}} {{long}} {{rowCnt;}}|
>
> Add new section in 'Advance Setting' tab when do cube design, user can set
> table snapshot properties for each table, and by default, it is segment level
> and store to metadata store
> h2. Build
> If user specify 'hbaseStore' storageType for any lookup table, will use
> MapReduce job convert the hive source table to hfiles, and then bulk load
> hfiles to HTable. So it will add two job steps to do the lookup table
> materialization.
> h2. HBase Lookup Table Schema
> all data are stored in raw value
> suppose the lookup table has primary keys: key1,key2
> rowkey will be:
> ||2 bytes||len1 bytes||2 bytes||len2 bytes||
> |key1 value length(len1)|key1 value|key 2 value length(len2)|key2 value|
>
> 1 column family c, multiple columns which column name is the index of the
> column in the table definition
> |c|
> |1|2|...|
>
> h2. Query
> For key lookup query, directly call hbase get api to get entire row according
> to key.
> For queries that need fetch keys according to the derived columns, iterate
> all rows to get related keys.
> For queries that only hit the lookup table, iterate all rows and let calcite
> to do aggregation and filter.
> h2. Management
> For each lookup table, admin can view how many snapshots it has in Kylin, and
> can view each snapshot type/size information and which cube/segments the
> snapshot is referenced, the snapshot tables that have no reference can be
> deleted.
> h2. Cleanup
> When clean up metadata store, need to remove snapshot stored in HBase. And
> need to clean up metadata store periodically by cronjob.
> h2. Future
> # Add coprocessor for lookup table, to improve the performance of lookup
> table query, and queries that filter by derived columns.
> # Add secondly index support for external snapshot table.
--
This message was sent
[jira] [Updated] (KYLIN-3107) An alert dialog will pop up warning "Failed to load bar chat" when the user enter or log out from the "Dashboard" page.
[ https://issues.apache.org/jira/browse/KYLIN-3107?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu updated KYLIN-3107: - Fix Version/s: v2.3.0 > An alert dialog will pop up warning "Failed to load bar chat" when the user > enter or log out from the "Dashboard" page. > --- > > Key: KYLIN-3107 > URL: https://issues.apache.org/jira/browse/KYLIN-3107 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v2.3.0 >Reporter: peng.jianhua >Assignee: peng.jianhua >Priority: Minor > Fix For: v2.3.0 > > Attachments: alert_dialog_pop_up_when_enter_Dashboard_page.PNG, > alert_dialog_pop_up_when_log_out_from_Dashboard_page.PNG > > > An alert dialog will pop up warning "Failed to load bar chat" when the user > enter or log out from the "Dashboard" page. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3224) data can't show when use kylin pushdown model
[ https://issues.apache.org/jira/browse/KYLIN-3224?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu updated KYLIN-3224: - Fix Version/s: v2.3.0 > data can't show when use kylin pushdown model > -- > > Key: KYLIN-3224 > URL: https://issues.apache.org/jira/browse/KYLIN-3224 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v2.2.0, v2.3.0 >Reporter: peng.jianhua >Assignee: peng.jianhua >Priority: Major > Fix For: v2.3.0 > > Attachments: 0001-KYLIN-3224.patch, 01.PNG > > > select * from kylin_sales > use pushdown model,and the result shows like 01.png -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3197) When ldap is opened, I use an ignored case user to login, the page does not respond.
[
https://issues.apache.org/jira/browse/KYLIN-3197?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Billy Liu updated KYLIN-3197:
-
Fix Version/s: v2.3.0
> When ldap is opened, I use an ignored case user to login, the page does not
> respond.
>
>
> Key: KYLIN-3197
> URL: https://issues.apache.org/jira/browse/KYLIN-3197
> Project: Kylin
> Issue Type: Bug
> Components: Security
>Affects Versions: v2.3.0
>Reporter: Peng Xing
>Assignee: Peng Xing
>Priority: Major
> Labels: patch
> Fix For: v2.3.0
>
> Attachments:
> 0001-KYLIN-3197-When-ldap-is-opened-I-use-an-ignored-case.patch,
> image-2018-01-25-17-22-39-970.png
>
>
> When ldap is opened, I config the kylin.properties, and give wkhGroup the
> admin permission.
> {code:java}
> ## Admin roles in LDAP, for ldap and saml
> kylin.security.acl.admin-role=wkhGroup
> {code}
> then I create a new user named 'wkh' whose group is 'wkhGroup', then I use
> '{color:#ff}wkh{color}' to login in, which is normal.
> But when I use '{color:#ff}WKH{color}' to login in, the page does not
> respond.
> I analyze the backgroud code, and find the function of
> 'org.apache.kylin.rest.security.LDAPAuthoritiesPopulator.getGroupMembershipRoles(String,
> String)' has problem.
> When userDn is
> "uid={color:#ff}wkh{color},ou=People,ou=defaultCluster,dc=zdh,dc=com" and
> username is "{color:#ff}WKH{color}", then authorities will be null by the
> follow code:
> {code:java}
> Set authorities = super.getGroupMembershipRoles(userDn,
> username);
> {code}
> So I have added 'getAdditionalRoles' function to get the authorities again.
> I have test the patch, please review, thanks!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3226) When user logs in with only query permission, 'N/A' is displayed in the cube's action list.
[ https://issues.apache.org/jira/browse/KYLIN-3226?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu updated KYLIN-3226: - Fix Version/s: v2.3.0 > When user logs in with only query permission, 'N/A' is displayed in the > cube's action list. > --- > > Key: KYLIN-3226 > URL: https://issues.apache.org/jira/browse/KYLIN-3226 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v2.3.0 >Reporter: Peng Xing >Assignee: Peng Xing >Priority: Major > Labels: patch > Fix For: v2.3.0 > > Attachments: > 0001-KYLIN-3226-When-user-logs-in-with-only-query-permiss.patch, > NA_action.png, only_query_permission.png > > > 1. Only give the 'QUERY' permission of project 'learn_kylin' for user 'xp', > refer to 'only_query_permission.png'. > 2. When user 'xp' logins in, 'N/A' is displayed in the cube's action list, > refer to 'NA_action.png'. > Please review the patch, thanks! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3199) The login dialog should be closed when ldap user with no permission login correctly
[ https://issues.apache.org/jira/browse/KYLIN-3199?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu updated KYLIN-3199: - Fix Version/s: v2.3.0 > The login dialog should be closed when ldap user with no permission login > correctly > --- > > Key: KYLIN-3199 > URL: https://issues.apache.org/jira/browse/KYLIN-3199 > Project: Kylin > Issue Type: Bug > Components: Security >Affects Versions: v2.3.0 >Reporter: Peng Xing >Assignee: Peng Xing >Priority: Minor > Labels: patch > Fix For: v2.3.0 > > Attachments: > 0001-KYLIN-3199-The-login-dialog-should-be-closed-when-ld.patch, > ldap_user_login.png > > > 1. Open ldap authentication, but I do not give the admin permission to group > 'xpGroup'; > 2. Create a ldap user 'xp', who belongs to group 'xpGroup', so this user has > none permission. > 3. When user 'xp' login in, the above bar has showed and been enabled, but > the login dialog still show. > 4. Then you can click any button on above bar. > Please refer to 'ldap_user_login.png' > I think the login dialog should be closed when you login in correctly, and > redirect to the 'Model' page, but this user has no permission. > I have modified this issue, please review the patch, thanks! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3205) Allow one column is used for both dimension and precisely count distinct measure
[ https://issues.apache.org/jira/browse/KYLIN-3205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu resolved KYLIN-3205. -- Resolution: Fixed Fix Version/s: v2.3.0 > Allow one column is used for both dimension and precisely count distinct > measure > > > Key: KYLIN-3205 > URL: https://issues.apache.org/jira/browse/KYLIN-3205 > Project: Kylin > Issue Type: Bug > Components: Metadata >Affects Versions: v2.2.0 >Reporter: kangkaisen >Assignee: kangkaisen >Priority: Major > Fix For: v2.3.0 > > Attachments: KYLIN-3205.patch > > > I Introduced a bug in KYLIN-2316, we should allow one column is used for both > dimension and precisely count distinct measure, as long as the dimension > encoding is not dict. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Issue Comment Deleted] (KYLIN-2929) speed up Dump file performance
[ https://issues.apache.org/jira/browse/KYLIN-2929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu updated KYLIN-2929: - Comment: was deleted (was: Thanks [~feng_xiao_yu], but the patch merged failed. Could you rebase against the master branch first? ) > speed up Dump file performance > -- > > Key: KYLIN-2929 > URL: https://issues.apache.org/jira/browse/KYLIN-2929 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v2.0.0 >Reporter: fengYu >Assignee: fengYu >Priority: Major > Labels: Performance > Fix For: v2.3.0 > > Attachments: > 0001-KYLIN-2929-speed-up-dump-performance-write-dump-file.patch > > > when I work on KYLIN-2926, I find coprocessor will dump to disk once > estimatedMemSize is bigger than spillThreshold, and found that spill data > size is extraordinary smaller that estimatedMemSize, in my case dump file > size is about 8MB and spillThreshold is setting to 3GB. > So, I try to keep the spill data in memory rather than write the file to disk > immediately, and when those in-memory spill data reach the threshold, write > all spill files together. > In my case, the coprocessor process cost time drop from 22s to 16s, it is > about 30% upgrade。 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-2929) speed up Dump file performance
[ https://issues.apache.org/jira/browse/KYLIN-2929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu resolved KYLIN-2929. -- Resolution: Fixed Merged at https://git1-us-west.apache.org/repos/asf?p=kylin.git;a=commit;h=4a29d92e > speed up Dump file performance > -- > > Key: KYLIN-2929 > URL: https://issues.apache.org/jira/browse/KYLIN-2929 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v2.0.0 >Reporter: fengYu >Assignee: fengYu >Priority: Major > Labels: Performance > Fix For: v2.3.0 > > Attachments: > 0001-KYLIN-2929-speed-up-dump-performance-write-dump-file.patch > > > when I work on KYLIN-2926, I find coprocessor will dump to disk once > estimatedMemSize is bigger than spillThreshold, and found that spill data > size is extraordinary smaller that estimatedMemSize, in my case dump file > size is about 8MB and spillThreshold is setting to 3GB. > So, I try to keep the spill data in memory rather than write the file to disk > immediately, and when those in-memory spill data reach the threshold, write > all spill files together. > In my case, the coprocessor process cost time drop from 22s to 16s, it is > about 30% upgrade。 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3215) remove 'drop' option when job status is stopped and error
[ https://issues.apache.org/jira/browse/KYLIN-3215?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu updated KYLIN-3215: - Fix Version/s: v2.3.0 > remove 'drop' option when job status is stopped and error > - > > Key: KYLIN-3215 > URL: https://issues.apache.org/jira/browse/KYLIN-3215 > Project: Kylin > Issue Type: Improvement > Components: Web >Affects Versions: v2.2.0, v2.3.0 >Reporter: peng.jianhua >Assignee: peng.jianhua >Priority: Minor > Fix For: v2.3.0 > > Attachments: 0001-KYLIN-3215.patch > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Reopened] (KYLIN-2933) Fix compilation against the Kafka 1.0.0 release
[
https://issues.apache.org/jira/browse/KYLIN-2933?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI reopened KYLIN-2933:
-
Not finished yet, let me reopen it.
> Fix compilation against the Kafka 1.0.0 release
> ---
>
> Key: KYLIN-2933
> URL: https://issues.apache.org/jira/browse/KYLIN-2933
> Project: Kylin
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: Shaofeng SHI
>Priority: Major
> Fix For: v2.3.0
>
>
> Due to change of class location of SecurityProtocol, KafkaClusterConfig.java
> needs to be modified so that it compiles against the Kafka 1.0.0 release:
> {code}
> -import org.apache.kafka.common.protocol.SecurityProtocol;
> +import org.apache.kafka.common.security.auth.SecurityProtocol;
> {code}
> A compatibility layer may be needed if we support multiple Kafka releases
> simultaneously.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3205) Allow one column is used for both dimension and precisely count distinct measure
[ https://issues.apache.org/jira/browse/KYLIN-3205?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-3205: Fix Version/s: (was: v2.3.0) > Allow one column is used for both dimension and precisely count distinct > measure > > > Key: KYLIN-3205 > URL: https://issues.apache.org/jira/browse/KYLIN-3205 > Project: Kylin > Issue Type: Bug > Components: Metadata >Affects Versions: v2.2.0 >Reporter: kangkaisen >Assignee: kangkaisen >Priority: Major > Attachments: KYLIN-3205.patch > > > I Introduced a bug in KYLIN-2316, we should allow one column is used for both > dimension and precisely count distinct measure, as long as the dimension > encoding is not dict. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3205) Allow one column is used for both dimension and precisely count distinct measure
[ https://issues.apache.org/jira/browse/KYLIN-3205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351774#comment-16351774 ] Shaofeng SHI commented on KYLIN-3205: - Patch merged in master; Thanks kaisen! > Allow one column is used for both dimension and precisely count distinct > measure > > > Key: KYLIN-3205 > URL: https://issues.apache.org/jira/browse/KYLIN-3205 > Project: Kylin > Issue Type: Bug > Components: Metadata >Affects Versions: v2.2.0 >Reporter: kangkaisen >Assignee: kangkaisen >Priority: Major > Attachments: KYLIN-3205.patch > > > I Introduced a bug in KYLIN-2316, we should allow one column is used for both > dimension and precisely count distinct measure, as long as the dimension > encoding is not dict. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3234) ResourceStore should add a API that can recursively list path.
[ https://issues.apache.org/jira/browse/KYLIN-3234?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiatao.tao updated KYLIN-3234: -- Summary: ResourceStore should add a API that can recursively list path. (was: Resource should add a API that can recursively list path.) > ResourceStore should add a API that can recursively list path. > -- > > Key: KYLIN-3234 > URL: https://issues.apache.org/jira/browse/KYLIN-3234 > Project: Kylin > Issue Type: Improvement >Reporter: jiatao.tao >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3234) Resource should add a API that can recursively list path.
jiatao.tao created KYLIN-3234: - Summary: Resource should add a API that can recursively list path. Key: KYLIN-3234 URL: https://issues.apache.org/jira/browse/KYLIN-3234 Project: Kylin Issue Type: Improvement Reporter: jiatao.tao -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3233) CacheController can not handle if cacheKey has "/"
[ https://issues.apache.org/jira/browse/KYLIN-3233?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiatao.tao updated KYLIN-3233: -- Priority: Minor (was: Major) > CacheController can not handle if cacheKey has "/" > -- > > Key: KYLIN-3233 > URL: https://issues.apache.org/jira/browse/KYLIN-3233 > Project: Kylin > Issue Type: Improvement >Reporter: jiatao.tao >Priority: Minor > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3233) CacheController can not handle if cacheKey has "/"
jiatao.tao created KYLIN-3233: - Summary: CacheController can not handle if cacheKey has "/" Key: KYLIN-3233 URL: https://issues.apache.org/jira/browse/KYLIN-3233 Project: Kylin Issue Type: Improvement Reporter: jiatao.tao -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3205) Allow one column is used for both dimension and precisely count distinct measure
[ https://issues.apache.org/jira/browse/KYLIN-3205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351751#comment-16351751 ] kangkaisen commented on KYLIN-3205: --- OK. I agree with you. > Allow one column is used for both dimension and precisely count distinct > measure > > > Key: KYLIN-3205 > URL: https://issues.apache.org/jira/browse/KYLIN-3205 > Project: Kylin > Issue Type: Bug > Components: Metadata >Affects Versions: v2.2.0 >Reporter: kangkaisen >Assignee: kangkaisen >Priority: Major > Fix For: v2.3.0 > > Attachments: KYLIN-3205.patch > > > I Introduced a bug in KYLIN-2316, we should allow one column is used for both > dimension and precisely count distinct measure, as long as the dimension > encoding is not dict. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-684) Remove holistic distinct count and multiple column distinct count from sample cube
[ https://issues.apache.org/jira/browse/KYLIN-684?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-684. -- > Remove holistic distinct count and multiple column distinct count from sample > cube > -- > > Key: KYLIN-684 > URL: https://issues.apache.org/jira/browse/KYLIN-684 > Project: Kylin > Issue Type: Bug > Components: General >Affects Versions: v0.7.1 >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Major > Fix For: v0.7.1 > > > In the sample cube "kylin_sales_cube_desc", there are two advanced measures > which are not formally supported yet. This makes user confusing. So drop them > should be better than keep them there. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-2074) Error happens when click the "SQL" section of cube
[
https://issues.apache.org/jira/browse/KYLIN-2074?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI closed KYLIN-2074.
---
> Error happens when click the "SQL" section of cube
> --
>
> Key: KYLIN-2074
> URL: https://issues.apache.org/jira/browse/KYLIN-2074
> Project: Kylin
> Issue Type: Bug
> Components: REST Service
>Affects Versions: v1.5.4.1
>Reporter: Shaofeng SHI
>Assignee: Shaofeng SHI
>Priority: Minor
> Fix For: v1.6.0
>
>
> Qiutongxin (qiutongxin at jd.com) from mailing list reported this bug: an
> error thrown when click the "SQL" tab of a cube, in kylin.log there is this
> NPE:
> {code}
> ERROR [http-bio-8070-exec-2] controller.BasicController:44 :
> java.lang.NullPointerException
> at
> org.apache.kylin.engine.EngineFactory.batchEngine(EngineFactory.java:44)
> at
> org.apache.kylin.engine.EngineFactory.getJoinedFlatTableDesc(EngineFactory.java:58)
> at
> org.apache.kylin.rest.controller.CubeController.getSql(CubeController.java:154)
> at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> at
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> at
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> {code}
> Checking the REST call between web and server, the segment name be passed in
> the REST URL is 'null':
> http://localhost:7070/kylin/api/cubes/embedded_cube/segs/null/sql
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (KYLIN-2680) kylin2.0 Build Cube with Spark (beta),can't access nameservice1
[
https://issues.apache.org/jira/browse/KYLIN-2680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI closed KYLIN-2680.
---
> kylin2.0 Build Cube with Spark (beta),can't access nameservice1
> ---
>
> Key: KYLIN-2680
> URL: https://issues.apache.org/jira/browse/KYLIN-2680
> Project: Kylin
> Issue Type: Test
>Affects Versions: v2.0.0
> Environment: CDH5.11.0 (HDFS HA)
> apache-kylin-2.0.0-bin-cdh57
>Reporter: Jepson
>Assignee: Shaofeng SHI
>Priority: Critical
> Labels: documentation
> Fix For: document
>
> Attachments: error.png
>
> Original Estimate: 5h
> Remaining Estimate: 5h
>
> I refernece with the doc,configure spark cube.
> [http://kylin.apache.org/docs20/tutorial/cube_spark.html]
> *First use the command with error:*
> [hdfs@hadoop36 hadoop-conf]$ $KYLIN_HOME/bin/kylin.sh start
> Retrieving hadoop conf dir...
> /opt/software/kylin/hadoop-conf is override as the kylin_hadoop_conf_dir
> KYLIN_HOME is set to /opt/software/kylin
> mkdir: Failed on local exception: java.io.IOException: Connection reset by
> peer; Host Details : local host is: "hadoop36.jiuye/192.168.17.36";
> destination host is: "nameservice1":8020;
> Failed to create /kylin. Please make sure the user has right to access /kylin
> [hdfs@hadoop36 hadoop-conf]$
> *Second use the command with error:*
> [hdfs@hadoop36 hadoop-conf]$ $KYLIN_HOME/bin/kylin.sh start
> Retrieving hadoop conf dir...
> /opt/software/kylin/hadoop-conf is override as the kylin_hadoop_conf_dir
> KYLIN_HOME is set to /opt/software/kylin
> 2017-06-22 15:07:04,885 INFO [main] ipc.Client
> (Client.java:handleConnectionTimeout(840)) - Retrying connect to server:
> nameservice1/180.168.41.175:8020. Already tried 0 time(s); maxRetries=45
> This error is that kylin can't access with nameservce1(HDFS HA),
> {color:#59afe1}so I add the hdfs-site.xml, not in the doc,is sloved:
> [hdfs@hadoop36 hadoop-conf]$ ln -s /etc/hadoop/conf/hdfs-site.xml
> $KYLIN_HOME/hadoop-conf/hdfs-site.xml{color}
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (KYLIN-1401) Passwords in kylin.properties should be enctrypted
[
https://issues.apache.org/jira/browse/KYLIN-1401?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI closed KYLIN-1401.
---
> Passwords in kylin.properties should be enctrypted
> --
>
> Key: KYLIN-1401
> URL: https://issues.apache.org/jira/browse/KYLIN-1401
> Project: Kylin
> Issue Type: Improvement
> Components: Client - CLI
>Affects Versions: v1.2
>Reporter: Wu Xiang
>Assignee: Shaofeng SHI
>Priority: Major
> Fix For: v1.5.0
>
>
> When integrating LDAP authentication in Kylin, people might run into the
> following error message. One possible reason is that Kylin actually requires
> passwords in kylin.properties be encrypted.
> {quote}
> Invalid bean definition with name 'ldapSource' defined in class path resource
> \[kylinSecurity.xm\]: Input length must be multiple of 16 when decrypting
> with padded cipher
> {quote}
> Related code for passwords decryption is as follows:
> {code:title=PasswordPlaceholderConfigurer.java|borderStyle=solid}
> protected String resolvePlaceholder(String placeholder, Properties props)
> {
> if (placeholder.toLowerCase().contains("password")) {
> return decrypt(props.getProperty(placeholder));
> } else {
> return props.getProperty(placeholder);
> }
> }
> {code}
> Related discussion in mailing list:
> [Link|http://mail-archives.apache.org/mod_mbox/kylin-user/201602.mbox/%3CCAJxfx2ANpe0rG1Vm1R_Sdh4XZuNP3FCswXmv1_xFUVSpdOkH_A%40mail.gmail.com%3E]
> In this ticket, I would propose a CLI tool for password encryption for Kylin,
> and add related documentation in [How to Enable Security with LDAP and
> SSO|http://kylin.apache.org/docs/howto/howto_ldap_and_sso.html]
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (KYLIN-1745) Java heap space errors for all builds in Kylin 1.5.2
[
https://issues.apache.org/jira/browse/KYLIN-1745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI closed KYLIN-1745.
---
> Java heap space errors for all builds in Kylin 1.5.2
>
>
> Key: KYLIN-1745
> URL: https://issues.apache.org/jira/browse/KYLIN-1745
> Project: Kylin
> Issue Type: Bug
>Affects Versions: v1.5.2
>Reporter: Nick Muerdter
>Assignee: Shaofeng SHI
>Priority: Major
> Fix For: v1.5.2.1
>
>
> When trying to build any cubes in Kylin 1.5.2, I get Java heap space errors
> on step 2 (Extract Fact Table Distinct Columns). I experience the same heap
> space error on my own cube, as well as using the sample cube data (from
> {{sample.sh}}).
> I'm testing on a fairly stock HDP-2.2.9.0-3393 installation. I've tried
> tweaking the memory settings, but even with the map memory tuned to allow
> 8GB, I still get the same heap space error even for the small sample data
> (where 8GB seems like it should be more than plenty).
> If I try the same build against the sample data on a fresh 1.5.1
> installation, the cube builds as expected, but with a fresh 1.5.2
> installation, I get these heap space errors on any build attempts.
> Here's the backtrace from the failed map job:
> {code}
> 2016-05-29 01:51:57,910 WARN [main]
> org.apache.hadoop.metrics2.impl.MetricsConfig: Cannot locate configuration:
> tried hadoop-metrics2-maptask.properties,hadoop-metrics2.properties
> 2016-05-29 01:51:57,977 INFO [main]
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: Scheduled snapshot period
> at 10 second(s).
> 2016-05-29 01:51:57,977 INFO [main]
> org.apache.hadoop.metrics2.impl.MetricsSystemImpl: MapTask metrics system
> started
> 2016-05-29 01:51:57,984 INFO [main] org.apache.hadoop.mapred.YarnChild:
> Executing with tokens:
> 2016-05-29 01:51:58,017 INFO [main] org.apache.hadoop.mapred.YarnChild: Kind:
> mapreduce.job, Service: job_1464481246919_0079, Ident:
> (org.apache.hadoop.mapreduce.security.token.JobTokenIdentifier@548e6d58)
> 2016-05-29 01:51:58,045 INFO [main] org.apache.hadoop.mapred.YarnChild:
> Sleeping for 0ms before retrying again. Got null now.
> 2016-05-29 01:51:58,249 INFO [main] org.apache.hadoop.mapred.YarnChild:
> mapreduce.cluster.local.dir for child:
> /hadoop/yarn/local/usercache/api-umbrella/appcache/application_1464481246919_0079
> 2016-05-29 01:51:58,443 INFO [main]
> org.apache.hadoop.conf.Configuration.deprecation: session.id is deprecated.
> Instead, use dfs.metrics.session-id
> 2016-05-29 01:51:58,792 INFO [main]
> org.apache.hadoop.mapreduce.lib.output.FileOutputCommitter: File Output
> Committer Algorithm version is 1
> 2016-05-29 01:51:58,801 INFO [main] org.apache.hadoop.mapred.Task: Using
> ResourceCalculatorProcessTree : [ ]
> 2016-05-29 01:51:59,078 INFO [main] org.apache.hadoop.mapred.MapTask:
> Processing split: org.apache.hive.hcatalog.mapreduce.HCatSplit@12dae582
> 2016-05-29 01:51:59,218 ERROR [main] org.apache.hadoop.mapred.YarnChild:
> Error running child : java.lang.OutOfMemoryError: Java heap space
> at
> org.apache.hadoop.mapred.MapTask$MapOutputBuffer.init(MapTask.java:983)
> at
> org.apache.hadoop.mapred.MapTask.createSortingCollector(MapTask.java:401)
> at org.apache.hadoop.mapred.MapTask.access$100(MapTask.java:81)
> at
> org.apache.hadoop.mapred.MapTask$NewOutputCollector.(MapTask.java:695)
> at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:767)
> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
> at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167)
> at java.security.AccessController.doPrivileged(Native Method)
> at javax.security.auth.Subject.doAs(Subject.java:422)
> at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
> at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162)
> {code}
> Any ideas? Let me know if I can supply any other details.
> Thanks!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (KYLIN-761) Tables are not shown in the "Query" tab, and couldn't run SQL query after cube be built
[ https://issues.apache.org/jira/browse/KYLIN-761?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-761. -- > Tables are not shown in the "Query" tab, and couldn't run SQL query after > cube be built > --- > > Key: KYLIN-761 > URL: https://issues.apache.org/jira/browse/KYLIN-761 > Project: Kylin > Issue Type: Bug > Components: General >Affects Versions: v0.7.1 > Environment: CDH 5.4 >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Major > Fix For: v0.7.1 > > Attachments: find-hive-dependency.sh > > > User from mailing-list reports their Kylin doesn't show up the table/columns > in "Query" tab after cube be built; In kylin.log, there are exceptions: > “Error instantiating > JsonCustomSchema(name=DEFAULT)” > Caused by: java.lang.ClassNotFoundException: > org.apache.kylin.query.schema.OLAPSchemaFactory > at java.net.URLClassLoader$1.run(URLClassLoader.java:366) > at java.net.URLClassLoader$1.run(URLClassLoader.java:355) > at java.security.AccessController.doPrivileged(Native Method) > at java.net.URLClassLoader.findClass(URLClassLoader.java:354) > at java.lang.ClassLoader.loadClass(ClassLoader.java:425) > at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:308) > at java.lang.ClassLoader.loadClass(ClassLoader.java:358) > at java.lang.Class.forName0(Native Method) > at java.lang.Class.forName(Class.java:191) > at org.apache.calcite.model.ModelHandler.visit(ModelHandler.java:209) > ... 103 more -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-1103) Couldn't create sample cube with ./bin/sample.sh
[ https://issues.apache.org/jira/browse/KYLIN-1103?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-1103. --- > Couldn't create sample cube with ./bin/sample.sh > > > Key: KYLIN-1103 > URL: https://issues.apache.org/jira/browse/KYLIN-1103 > Project: Kylin > Issue Type: Bug > Components: Environment >Affects Versions: 1.x-HBase1.1.3 >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Major > Fix For: 1.x-HBase1.1.3 > > > Get reporting from liuyiming@foxmail.com in kylin dev mailing list: > I have installed HDP 2.3 on my cluster(Hadoop 2.7.1, HBase 1.1.1, Hive 1.2.1) > with the help of Ambari. > I downloaded the latest > apache-kylin-1.2-HBase1.1-incubating-SNAPSHOT-bin.tar.gz. The Kylin could > start normally with slight modification of find-hive-dependency. > Then I run the sample.sh to import the sample project. But the web GUI could > not show the learn_kylin project from the dropdown box. > I have checked Hive, and found three tables(kylin_cal_dt, > kylin_category_groupings,kylin_sales) created under default database. > I have checked HBase, and found three tables(kylin_metadata, > kylin_metadata_acl, kylin_metadata_user) created. > It seems the sample data is imported correctly. But the metadata import > failed. > When executed the following command: > hbase org.apache.hadoop.util.RunJar lib/kylin-job-1.1-incubating-SNAPSHOT.jar > org.apache.kylin.common.persistence.ResourceTool upload sample_cube/metadata/ > It has the following log > "Null inputstream for /". It seems the ResourceTool could not find the > metadata from local sample_cube directory. > [kylin@server kylin]$ ls -al > total 68 > drwxr-xr-x. 8 kylin kylin 4096 Oct 26 22:54 . > drwxr-xr-x. 6 root root 4096 Oct 26 22:47 .. > drwxr-xr-x. 2 kylin kylin 4096 Oct 25 17:33 bin > -rw-r--r--. 1 kylin kylin 823 Oct 25 17:33 commit.sha1 > drwxr-xr-x. 2 kylin kylin 4096 Oct 25 17:33 conf > -rw-r--r--. 1 kylin kylin 539 Oct 25 17:33 DISCLAIMER > drwxr-xr-x. 2 kylin kylin 4096 Oct 25 17:33 lib > -rw-r--r--. 1 kylin kylin 13060 Oct 25 17:33 LICENSE > drwxrwxr-x. 2 kylin kylin 4096 Oct 26 22:48 logs > -rw-r--r--. 1 kylin kylin 180 Oct 25 17:33 NOTICE > -rw-rw-r--. 1 kylin kylin 6 Oct 26 22:54 pid > -rw-r--r--. 1 kylin kylin 849 Oct 25 17:33 README.md > drwxr-xr-x. 4 kylin kylin 4096 Oct 25 17:33 sample_cube > drwxr-xr-x. 9 kylin kylin 4096 Oct 25 17:33 tomcat -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-682) Got "No aggregator for func 'MIN' and return type 'decimal(19,4)'" error when build cube
[ https://issues.apache.org/jira/browse/KYLIN-682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-682. -- > Got "No aggregator for func 'MIN' and return type 'decimal(19,4)'" error when > build cube > > > Key: KYLIN-682 > URL: https://issues.apache.org/jira/browse/KYLIN-682 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v0.7.1 >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Major > Fix For: v0.7.1 > > > Problem from community user Sun: > Try to build our own test cube according to the offical tutorial. When > doing the step of Build Base Cuboid Data, error occurs. > I checked the mapreduce task log and found the following error stacktrace > : > Note that with defining measure step, I choose measure of GMV_MIN and > GMV_MAX both and these two measures are automatically defined as > decimal(19,4) and there is no way to modify that. > Super weird for this kind of exception info. Hope any experts can shed a > light on this. > Best regards, > Sun. >2015-04-14 10:49:06,327 INFO [AsyncDispatcher event handler] > org.apache.hadoop.mapreduce.v2.app.job.impl.TaskAttemptImpl: Diagnostics > report from attempt_1428978814353_0003_m_00_3: Error: > java.lang.IllegalArgumentException: No aggregator for func 'MIN' and > return type 'decimal(19,4)' > at > org.apache.kylin.metadata.measure.MeasureAggregator.create(MeasureAggregat > or.java:58) > at > org.apache.kylin.metadata.measure.MeasureAggregators.(MeasureAggrega > tors.java:49) > at > org.apache.kylin.metadata.measure.MeasureAggregators.(MeasureAggrega > tors.java:39) > at > org.apache.kylin.job.hadoop.cube.CuboidReducer.setup(CuboidReducer.java:73 > ) > at org.apache.hadoop.mapreduce.Reducer.run(Reducer.java:168) > at org.apache.hadoop.mapred.Task$NewCombinerRunner.combine(Task.java:1651) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(MapTask.java > :1630) > at > org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.java:1482) > at > org.apache.hadoop.mapred.MapTask$NewOutputCollector.close(MapTask.java:720 > ) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:790) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:168) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation. > java:1642) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:163) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-644) join table as null error when build the cube
[ https://issues.apache.org/jira/browse/KYLIN-644?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-644. -- > join table as null error when build the cube > > > Key: KYLIN-644 > URL: https://issues.apache.org/jira/browse/KYLIN-644 > Project: Kylin > Issue Type: Bug > Components: Metadata >Affects Versions: v0.7.1 >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Minor > Fix For: v0.7.1 > > > An error occured when I try to build a cube. I examined the log and found > this : > INSERT OVERWRITE TABLE kylin_intermediate_ZSJ_1970010100_ > 2922789940817071255_96c192af_6191_4a13_b827_02c9d5eb1a7d SELECT > FACT_TABLE.QKDYDM > ,FACT_TABLE.NY > FROM DEFAULT.ZSJ_FACT_YT as FACT_TABLE > INNER JOIN DEFAULT.ZSJ_DIM_NY as null > ON FACT_TABLE.NY = null.YEARMONTH > INNER JOIN DEFAULT.ZSJ_DIM_ORG as null > ON FACT_TABLE.QKDYDM = null.YTDM > How can there be 'join xxx as null' in this sentence? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-658) java.lang.IllegalArgumentException: Cannot find rowkey column XXX in cube CubeDesc
[ https://issues.apache.org/jira/browse/KYLIN-658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-658. -- > java.lang.IllegalArgumentException: Cannot find rowkey column XXX in cube > CubeDesc > -- > > Key: KYLIN-658 > URL: https://issues.apache.org/jira/browse/KYLIN-658 > Project: Kylin > Issue Type: Bug > Components: Metadata >Affects Versions: v0.7.1 >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Major > Fix For: v0.7.1 > > > When create a cube, in the final step to save the cube, it throws error in > the back end: > Caused by: java.lang.IllegalArgumentException: Cannot find rowkey column XXX > in cube CubeDesc [name=CUBE_NAME] > at > org.apache.kylin.cube.model.RowKeyDesc.buildRowKey(RowKeyDesc.java:202) > at org.apache.kylin.cube.model.RowKeyDesc.init(RowKeyDesc.java:181) > at org.apache.kylin.cube.model.CubeDesc.init(CubeDesc.java:449) > at > org.apache.kylin.cube.CubeDescManager.createCubeDesc(CubeDescManager.java:150) > at > org.apache.kylin.rest.service.CubeService.createCubeAndDesc(CubeService.java:153) > at > org.apache.kylin.rest.service.CubeService$$FastClassByCGLIB$$17a07c0e.invoke() > at net.sf.cglib.proxy.MethodProxy.invoke(MethodProxy.java:204) > at > org.springframework.aop.framework.Cglib2AopProxy$DynamicAdvisedInterceptor.intercept(Cglib2AopProxy.java:618) > at > org.apache.kylin.rest.service.CubeService$$EnhancerByCGLIB$$8cd7efcb.createCubeAndDesc() > at > org.apache.kylin.rest.controller.CubeController.saveCubeDesc(CubeController.java:331) > ... 83 more -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-681) Failed to build dictionary if the rowkey's dictionary property is "date(yyyy-mm-dd)"
[ https://issues.apache.org/jira/browse/KYLIN-681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-681. -- > Failed to build dictionary if the rowkey's dictionary property is > "date(-mm-dd)" > > > Key: KYLIN-681 > URL: https://issues.apache.org/jira/browse/KYLIN-681 > Project: Kylin > Issue Type: Bug >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Major > Fix For: v0.7.1 > > > I got the following error when building a cube which was migrated from 0.6; > There is a rowkey column whose dictionary property is "date(-mm-dd)": > java.lang.IllegalArgumentException: Unknown dictionary data set > 'date(-mm-dd)', referred from DEFAULT.TABLE_.GA_VST_DT > at > org.apache.kylin.dict.DictionaryManager.decideSourceData(DictionaryManager.java:235) > at > org.apache.kylin.job.hadoop.cube.FactDistinctColumnsMapperBase.setup(FactDistinctColumnsMapperBase.java:59) > at > org.apache.kylin.job.hadoop.cube.FactDistinctHiveColumnsMapper.setup(FactDistinctHiveColumnsMapper.java:56) > at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:142) > at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764) > at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) > at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:167) > at java.security.AccessController.doPrivileged(Native Method) > at javax.security.auth.Subject.doAs(Subject.java:415) > at > org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1650) > at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:162) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-576) Refactor expansion rate expression
[ https://issues.apache.org/jira/browse/KYLIN-576?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-576. -- > Refactor expansion rate expression > -- > > Key: KYLIN-576 > URL: https://issues.apache.org/jira/browse/KYLIN-576 > Project: Kylin > Issue Type: New Feature > Components: General, Metadata >Reporter: Luke Han >Assignee: Shaofeng SHI >Priority: Major > Fix For: v0.7.1 > > > Currently, Expansion Rate calculation just use Source Hive Table Size (Fact > Table) and Cube Size: > Expansion Rate = sum(cube segment size)/hive_table_size > New expression: > Input Size: Flat table (intermedia hive table after join) size > Input Records: Flat table (intermedia hive table after join) records > Expansion Rate = sum(cube segment size)/sum(input_size) -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-553) From cube instance, couldn't easily find the project instance that it belongs to
[ https://issues.apache.org/jira/browse/KYLIN-553?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-553. -- > From cube instance, couldn't easily find the project instance that it belongs > to > > > Key: KYLIN-553 > URL: https://issues.apache.org/jira/browse/KYLIN-553 > Project: Kylin > Issue Type: Improvement > Components: Metadata >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Major > Fix For: v0.7.1 > > Original Estimate: 4h > Remaining Estimate: 4h > > Currently the linkage between a Cube and a project is recorded in the project > meta data, there is no project info recorded in a Cube; There are some > senario that need find out the project for a Cube. It need scan all projects > to get this info. This can be enhanced by wiring the cube - project linkage > just after a cube be loaded from storage. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-436) Can't sync Hive table metadata from other database rather than "default"
[ https://issues.apache.org/jira/browse/KYLIN-436?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-436. -- > Can't sync Hive table metadata from other database rather than "default" > > > Key: KYLIN-436 > URL: https://issues.apache.org/jira/browse/KYLIN-436 > Project: Kylin > Issue Type: Bug > Components: Metadata >Reporter: Luke Han >Assignee: Shaofeng SHI >Priority: Major > Labels: github-import > Fix For: v0.7.1 > > > Hive table metadata can't be sync up into Kylin for Hive tables coming from > other database rather than "default" > Imported from GitHub > Url: https://github.com/KylinOLAP/Kylin/issues/69 > Created by: [lukehan|https://github.com/lukehan] > Labels: bug, > Milestone: v0.6.2 Release > Assignee: [liyang-gmt8|https://github.com/liyang-gmt8] > Created at: Tue Nov 18 23:01:02 CST 2014 > State: closed -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-412) No exception or message when sync up table which can't access
[ https://issues.apache.org/jira/browse/KYLIN-412?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-412. -- > No exception or message when sync up table which can't access > - > > Key: KYLIN-412 > URL: https://issues.apache.org/jira/browse/KYLIN-412 > Project: Kylin > Issue Type: Bug > Components: Metadata >Reporter: Luke Han >Assignee: Shaofeng SHI >Priority: Major > Labels: github-import > Fix For: v0.7.1 > > > When sync up Hive table to Kylin, there's no exception or message for table > which underline account can't access it. > Imported from GitHub > Url: https://github.com/KylinOLAP/Kylin/issues/93 > Created by: [lukehan|https://github.com/lukehan] > Labels: bug, > Milestone: v0.6.5 Release > Created at: Fri Nov 21 21:29:48 CST 2014 > State: open -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-407) Validation: There's should no Hive table column using "binary" data type
[ https://issues.apache.org/jira/browse/KYLIN-407?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-407. -- > Validation: There's should no Hive table column using "binary" data type > > > Key: KYLIN-407 > URL: https://issues.apache.org/jira/browse/KYLIN-407 > Project: Kylin > Issue Type: Improvement > Components: Metadata >Reporter: Luke Han >Assignee: Shaofeng SHI >Priority: Major > Labels: github-import > Fix For: v0.7.1 > > > There's should no binary column be included in the star schema which will be > Kylin cube's input source. > Imported from GitHub > Url: https://github.com/KylinOLAP/Kylin/issues/98 > Created by: [lukehan|https://github.com/lukehan] > Labels: newfeature, > Created at: Mon Nov 24 22:04:16 CST 2014 > State: open -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-198) Support same column name in different lookup table
[ https://issues.apache.org/jira/browse/KYLIN-198?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-198. -- > Support same column name in different lookup table > -- > > Key: KYLIN-198 > URL: https://issues.apache.org/jira/browse/KYLIN-198 > Project: Kylin > Issue Type: Improvement > Components: Metadata >Reporter: Luke Han >Assignee: Shaofeng SHI >Priority: Major > Labels: github-import > Fix For: v1.0 > > > When there are two lookup table using same name in one cube, the cube engine > will raise exception since it do not know which column coming from which > table. > Using full qualified name with _database.tablename.columnname_ to unique > identify column in row key. > Workaround: duplicate lookup table with specified name and join with fact. > For example: duplicate dw_countries to dw_countries_seller and > dw_countries_buyer. > Imported from GitHub > Url: https://github.com/KylinOLAP/Kylin/issues/307 > Created by: [lukehan|https://github.com/lukehan] > Labels: bug, > Milestone: v2.0 Release > Assignee: [qhzhou|https://github.com/qhzhou] > Created at: Fri Dec 26 15:02:51 CST 2014 > State: open -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-2679) Report error when a dimension using "dict" encoding and also configured Global dictionary for "distinct_count" measure
[ https://issues.apache.org/jira/browse/KYLIN-2679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI resolved KYLIN-2679. - Resolution: Duplicate > Report error when a dimension using "dict" encoding and also configured > Global dictionary for "distinct_count" measure > -- > > Key: KYLIN-2679 > URL: https://issues.apache.org/jira/browse/KYLIN-2679 > Project: Kylin > Issue Type: Bug > Components: Metadata, Query Engine >Reporter: Shaofeng SHI >Assignee: kangkaisen >Priority: Major > Labels: scope > Fix For: v2.3.0 > > > This is a problem reported by community user Sonny Heer: > After finally getting the global dictionary to work with building the cube > there are now exceptions during query. > ERROR in query: "AppendTrieDictionary can't retrive value from id" > The cube has a UHC dimension, which also appeared in a count distinct > measure. Then a global dictionary is created for it, but Global dictionary > doesn't support decoding, then there is such an error during query time. > While this wasn't even checked when the cube is created. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-2679) Report error when a dimension using "dict" encoding and also configured Global dictionary for "distinct_count" measure
[ https://issues.apache.org/jira/browse/KYLIN-2679?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-2679. --- > Report error when a dimension using "dict" encoding and also configured > Global dictionary for "distinct_count" measure > -- > > Key: KYLIN-2679 > URL: https://issues.apache.org/jira/browse/KYLIN-2679 > Project: Kylin > Issue Type: Bug > Components: Metadata, Query Engine >Reporter: Shaofeng SHI >Assignee: kangkaisen >Priority: Major > Labels: scope > > This is a problem reported by community user Sonny Heer: > After finally getting the global dictionary to work with building the cube > there are now exceptions during query. > ERROR in query: "AppendTrieDictionary can't retrive value from id" > The cube has a UHC dimension, which also appeared in a count distinct > measure. Then a global dictionary is created for it, but Global dictionary > doesn't support decoding, then there is such an error during query time. > While this wasn't even checked when the cube is created. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3205) Allow one column is used for both dimension and precisely count distinct measure
[ https://issues.apache.org/jira/browse/KYLIN-3205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351748#comment-16351748 ] Shaofeng SHI commented on KYLIN-3205: - Hi kaisen, the patch improved the error message, that is good (at least user won't fact the problem described in KYLIN-2679). But it doesn't implement the requirement. To support one column to use both dict and global dict, the metadata structure need be changed. So I would merge this patch, and keep this Jira as open (with no fixed version), what do you think? > Allow one column is used for both dimension and precisely count distinct > measure > > > Key: KYLIN-3205 > URL: https://issues.apache.org/jira/browse/KYLIN-3205 > Project: Kylin > Issue Type: Bug > Components: Metadata >Affects Versions: v2.2.0 >Reporter: kangkaisen >Assignee: kangkaisen >Priority: Major > Fix For: v2.3.0 > > Attachments: KYLIN-3205.patch > > > I Introduced a bug in KYLIN-2316, we should allow one column is used for both > dimension and precisely count distinct measure, as long as the dimension > encoding is not dict. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3069) Add proper time zone support to the WebUI instead of GMT/PST kludge
[ https://issues.apache.org/jira/browse/KYLIN-3069?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351731#comment-16351731 ] Billy Liu commented on KYLIN-3069: -- [~Zhixiong Chen], if you think the patch is OK, please merge it. > Add proper time zone support to the WebUI instead of GMT/PST kludge > --- > > Key: KYLIN-3069 > URL: https://issues.apache.org/jira/browse/KYLIN-3069 > Project: Kylin > Issue Type: Bug > Components: Web >Affects Versions: v2.2.0 > Environment: HDP 2.5.3, Kylin 2.2.0 >Reporter: Vsevolod Ostapenko >Assignee: peng.jianhua >Priority: Minor > Fix For: v2.3.0 > > Attachments: > 0001-KYLIN-3069-Add-proper-time-zone-support-to-the-WebUI.patch, Screen Shot > 2017-12-05 at 10.01.39 PM.png, kylin_pic1.png, kylin_pic2.png, kylin_pic3.png > > Original Estimate: 168h > Remaining Estimate: 168h > > Time zone handling logic in the WebUI is a kludge, coded to parse only > "GMT-N" time zone specifications and defaulting to PST, if parsing is not > successful (kylin/webapp/app/js/filters/filter.js) > Integrating moment and moment time zone (http://momentjs.com/timezone/docs/) > into the product, would allow correct time zone handling. > For the users who happen to reside in the geographical locations that do > observe day light savings time, usage of GMT-N format is very inconvenient > and info reported by the UI in various places is perplexing. > Needless to say that the GMT moniker itself is long deprecated. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-2898) Introduce memcached as a distributed cache for queries
[ https://issues.apache.org/jira/browse/KYLIN-2898?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu updated KYLIN-2898: - Fix Version/s: (was: v2.3.0) > Introduce memcached as a distributed cache for queries > -- > > Key: KYLIN-2898 > URL: https://issues.apache.org/jira/browse/KYLIN-2898 > Project: Kylin > Issue Type: Sub-task > Components: Query Engine >Affects Versions: v2.1.0 >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-2895) Refine query cache
[ https://issues.apache.org/jira/browse/KYLIN-2895?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu updated KYLIN-2895: - Fix Version/s: (was: v2.3.0) > Refine query cache > -- > > Key: KYLIN-2895 > URL: https://issues.apache.org/jira/browse/KYLIN-2895 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Affects Versions: v2.1.0 >Reporter: Zhong Yanghong >Assignee: Zhong Yanghong >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-2894) Change the query cache expiration strategy by signature checking
[
https://issues.apache.org/jira/browse/KYLIN-2894?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Billy Liu updated KYLIN-2894:
-
Fix Version/s: (was: v2.3.0)
> Change the query cache expiration strategy by signature checking
>
>
> Key: KYLIN-2894
> URL: https://issues.apache.org/jira/browse/KYLIN-2894
> Project: Kylin
> Issue Type: Sub-task
> Components: Query Engine
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>Priority: Major
>
> Currently to invalid query cache, {{CacheService}} will either invoke
> {{cleanDataCache}} or {{cleanAllDataCache}}. Both methods will clear all of
> the query cache, which is very inefficient. In eBay PROD environment, there's
> around 400 cubing jobs per day, which means the query cache will be cleared
> very 4 minutes. Then we introduced a signature based cache invalidation
> strategy. The basic idea is as follows:
> * Add a signature for {{SQLResponse}}, here we choose the cube last build time
> * When fetch {{SQLResponse}} for cache, first check whether the signature is
> consistent. If not, this cached value is overdue and will be invalidate.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-2899) Enable segment level query cache
[ https://issues.apache.org/jira/browse/KYLIN-2899?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu updated KYLIN-2899: - Fix Version/s: (was: v2.3.0) > Enable segment level query cache > > > Key: KYLIN-2899 > URL: https://issues.apache.org/jira/browse/KYLIN-2899 > Project: Kylin > Issue Type: Sub-task > Components: Query Engine >Affects Versions: v2.1.0 >Reporter: Zhong Yanghong >Assignee: Ma Gang >Priority: Major > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-2884) Add delete segment function for portal
[ https://issues.apache.org/jira/browse/KYLIN-2884?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351730#comment-16351730 ] Billy Liu commented on KYLIN-2884: -- Hello [~julianpan], could you update the patch, I'd like to merge it into 2.3 release. > Add delete segment function for portal > -- > > Key: KYLIN-2884 > URL: https://issues.apache.org/jira/browse/KYLIN-2884 > Project: Kylin > Issue Type: Improvement >Reporter: Pan, Julian >Assignee: Pan, Julian >Priority: Major > Fix For: v2.3.0 > > Attachments: KYLIN-2884.patch > > > Kylin restful already support delete segment function. Could we add a action > button for front-end to user? -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-2672) Only clean necessary cache for CubeMigrationCLI
[ https://issues.apache.org/jira/browse/KYLIN-2672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351728#comment-16351728 ] Billy Liu commented on KYLIN-2672: -- [~liyang.g...@gmail.com], could you review this patch? > Only clean necessary cache for CubeMigrationCLI > --- > > Key: KYLIN-2672 > URL: https://issues.apache.org/jira/browse/KYLIN-2672 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Affects Versions: v2.0.0 >Reporter: kangkaisen >Assignee: kangkaisen >Priority: Major > Fix For: v2.3.0 > > Attachments: KYLIN-2672.patch > > > Currently, we simply clear ALL cache in CubeMigrationCLI. which will make a > few of queries slower in prod env when we have many tables, models, cubes and > migrate cube often. > So, we could only clean necessary cache for CubeMigrationCLI. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3194) Tolerate broken job metadata caused by executable ClassNotFoundException
[ https://issues.apache.org/jira/browse/KYLIN-3194?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu resolved KYLIN-3194. -- Resolution: Fixed Fix Version/s: v2.3.0 Commit: https://github.com/apache/kylin/commit/f797c84044f52f9c351ad680bc3a8a3482781121 https://github.com/apache/kylin/commit/da14ee94bcc5b91fabf99954b88819cc513effea > Tolerate broken job metadata caused by executable ClassNotFoundException > > > Key: KYLIN-3194 > URL: https://issues.apache.org/jira/browse/KYLIN-3194 > Project: Kylin > Issue Type: Improvement >Reporter: liyang >Assignee: liyang >Priority: Major > Fix For: v2.3.0 > > > As Kylin evolves, many executable class was renamed or deprecated. These lead > to broken job metadata and ClassNotFoundException can be thrown from many > places in the code. Better to tolerate these error metadata once for all. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-3211) Class IntegerDimEnc shuould give more exception information when the length is exceed the max or less than the min
[
https://issues.apache.org/jira/browse/KYLIN-3211?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Billy Liu resolved KYLIN-3211.
--
Resolution: Fixed
Fix Version/s: v2.3.0
Commit:
https://github.com/apache/kylin/commit/04b3220739713a7d157e6a286fabea5a5036fad4
> Class IntegerDimEnc shuould give more exception information when the length
> is exceed the max or less than the min
> --
>
> Key: KYLIN-3211
> URL: https://issues.apache.org/jira/browse/KYLIN-3211
> Project: Kylin
> Issue Type: Bug
>Reporter: Yifei Wu
>Assignee: Yifei Wu
>Priority: Major
> Fix For: v2.3.0
>
>
> the message IllegalArgumentException in Class IntegerDimEnc is empty
> currently, it may cause confusion when using Kylin or troubleshooting a
> problem.
> {code:java}
> // code placeholder
> public IntDimEnc(int len) {
> if (len <= 0 || len >= CAP.length)
> throw new IllegalArgumentException();
> this.fixedLen = len;
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (KYLIN-452) Automatically create local cluster for running tests
[ https://issues.apache.org/jira/browse/KYLIN-452?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-452. -- > Automatically create local cluster for running tests > > > Key: KYLIN-452 > URL: https://issues.apache.org/jira/browse/KYLIN-452 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Reporter: Luke Han >Assignee: Shaofeng SHI >Priority: Major > Labels: github-import > Fix For: v0.7.1 > > > Phoenix automatically creates a mini-cluster when you run a test. For > example, download the latest Phoenix, do 'mvn -DskipTests install', open > UpsertBigValuesIT.java in Intellij, and press Control-Shift-F10. (It works in > Eclipse too.) The test creates a mini-cluster, runs 4 test methods, and tears > down the cluster. It takes 1 minute to run. No configuration whatsoever is > required. > I know that this kind of automation hard to achieve, but it allows casual > committers (like me) to get involved in the project without spending hours or > days setting up the environment. > Since Phoenix and Kylin are both based on HBase, you could probably > copy-paste their maven XML and the base classes for their tests. > Imported from GitHub > Url: https://github.com/KylinOLAP/Kylin/issues/53 > Created by: [julianhyde|https://github.com/julianhyde] > Labels: enhancement, > Milestone: v0.6.2 Release > Assignee: [shaofengshi|https://github.com/shaofengshi] > Created at: Fri Nov 14 06:01:18 CST 2014 > State: open -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-627) Hive tables' partition column was not sync into Kylin
[ https://issues.apache.org/jira/browse/KYLIN-627?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-627. -- > Hive tables' partition column was not sync into Kylin > - > > Key: KYLIN-627 > URL: https://issues.apache.org/jira/browse/KYLIN-627 > Project: Kylin > Issue Type: Bug > Components: Metadata >Affects Versions: v0.7.1 >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Major > Fix For: v0.7.1 > > > When sync a hive table into Kylin, the partition column was not loaded, which > caused user couldn't use that column in defining Cube. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-628) Couldn't build a new created cube
[ https://issues.apache.org/jira/browse/KYLIN-628?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-628. -- > Couldn't build a new created cube > - > > Key: KYLIN-628 > URL: https://issues.apache.org/jira/browse/KYLIN-628 > Project: Kylin > Issue Type: Bug > Components: Metadata >Affects Versions: v0.7.1 >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Major > Fix For: v0.7.1 > > Original Estimate: 2h > Remaining Estimate: 2h > > When create a new cube, specify a partition start date like 2015-1-1; After > save the Cube, click "Actions -> Build", Kylin shows 2015-1-1 as the begin > date, select a date after that as the end date; When click "Submit", Kylin > pops up an error dialogue, saying "Couldn't allocate segment", and from the > segment name, it shows using 1970-1-1 as the begin date, which is incorrect. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-629) Kylin failed to run mapreduce job if there is no mapreduce.application.classpath in mapred-site.xml
[ https://issues.apache.org/jira/browse/KYLIN-629?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-629. -- > Kylin failed to run mapreduce job if there is no > mapreduce.application.classpath in mapred-site.xml > --- > > Key: KYLIN-629 > URL: https://issues.apache.org/jira/browse/KYLIN-629 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Affects Versions: v0.7.1 >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Major > Fix For: v0.7.1 > > > I deployed 0.7.1 snapshot in our hadoop cluster, sync a table to Kylin, which > trigger Kylin submit a hadoop job to calculate the fields cardinality; But > this job was failed, the error is: > 15/03/10 01:05:25 INFO mapreduce.Job: Job job_1425075571333_139544 failed > with state FAILED due to: Application application_1425075571333_139544 failed > 2 times due to AM Container for appattempt_1425075571333_139544_02 exited > with exitCode: 1 due to: Exception from container-launch: > org.apache.hadoop.util.Shell$ExitCodeException: > at org.apache.hadoop.util.Shell.runCommand(Shell.java:505) > at org.apache.hadoop.util.Shell.run(Shell.java:418) > at > org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:650) > at > org.apache.hadoop.yarn.server.nodemanager.LinuxContainerExecutor.launchContainer(LinuxContainerExecutor.java:279) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:300) > at > org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:81) > at java.util.concurrent.FutureTask.run(FutureTask.java:262) > at > java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) > at > java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) > at java.lang.Thread.run(Thread.java:745) > Adding more log in AbstractHadoopJob.setJobClasspath, get the following debug > message: > [pool-7-thread-1]:[2015-03-10 > 00:33:17,739][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob.setJobClasspath(AbstractHadoopJob.java:146)] > - append job jar: > /export/home/b_kylin/kylin_ii/lib/kylin-job-0.7.1-SNAPSHOT.jar > [pool-7-thread-1]:[2015-03-10 > 00:33:17,740][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob.setJobClasspath(AbstractHadoopJob.java:152)] > - append kylin.hive.dependency: > /apache/hive/conf:/apache/hive/lib/*:/apache/hive-0.13.0.2.1.3.0-563/hcatalog/share/hcatalog/hive-hcatalog-core-0.13.0.2.1.3.0-563.jar > to mapreduce.application.classpath > [pool-7-thread-1]:[2015-03-10 > 00:33:17,740][INFO][org.apache.kylin.job.hadoop.AbstractHadoopJob.setJobClasspath(AbstractHadoopJob.java:164)] > - Hadoop job classpath is: > /apache/hive/conf,/apache/hive/lib/*,/apache/hive-0.13.0.2.1.3.0-563/hcatalog/share/hcatalog/hive-hcatalog-core-0.13.0.2.1.3.0-563.jar > From it I see the job configuration's "mapreduce.application.classpath" was > empty before appending the hive dependencies; After appending, there is only > hive on the classpath, which would cause the hadoop job failed to run. > Doing some search, find the explaination of "mapreduce.application.classpath" > in: > https://hadoop.apache.org/docs/r2.6.0/hadoop-mapreduce-client/hadoop-mapreduce-client-core/mapred-default.xml > Now we know that, the added hive dependency will overwrite the default > classpath, which is not expected. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-746) test case under server folder are problematic when tested against minicluster
[ https://issues.apache.org/jira/browse/KYLIN-746?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-746. -- > test case under server folder are problematic when tested against minicluster > - > > Key: KYLIN-746 > URL: https://issues.apache.org/jira/browse/KYLIN-746 > Project: Kylin > Issue Type: Bug > Components: Tools, Build and Test >Reporter: hongbin ma >Assignee: Shaofeng SHI >Priority: Major > Fix For: v1.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-642) Couldn't refresh cube segment
[ https://issues.apache.org/jira/browse/KYLIN-642?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-642. -- > Couldn't refresh cube segment > - > > Key: KYLIN-642 > URL: https://issues.apache.org/jira/browse/KYLIN-642 > Project: Kylin > Issue Type: Bug > Components: Job Engine >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Major > Fix For: v0.7.1 > > > When user selects a segment to refresh, Kylin will report error like "New > segment range invalid, start date must be earlier than end date, > 142646400 < 142646400" -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-696) Dictionary couldn't recognize a value and throw IllegalArgumentException: "Not a valid value"
[ https://issues.apache.org/jira/browse/KYLIN-696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-696. -- > Dictionary couldn't recognize a value and throw IllegalArgumentException: > "Not a valid value" > - > > Key: KYLIN-696 > URL: https://issues.apache.org/jira/browse/KYLIN-696 > Project: Kylin > Issue Type: Bug > Components: General >Affects Versions: v0.6.5, v0.6.6, v0.7.1 >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Major > Fix For: v0.7.1 > > > I have a cube which have several lookups be left joined with the fact table; > I defined one foreign key as a dimension, then that fk column was added into > the rowkey, with "dictinoary" = "Y"; > The dictionary was successfully built; while the cube build job was failed, > the error trace looks like: > Caused by: java.lang.IllegalArgumentException: Not a valid value: > 01020 > at > org.apache.kylin.dict.TrieDictionary.getIdFromValueBytesImpl(TrieDictionary.java:176) > at > org.apache.kylin.dict.NumberDictionary.getIdFromValueBytesImpl(NumberDictionary.java:172) > at org.apache.kylin.dict.Dictionary.getIdFromValueBytes(Dictionary.java:138) > at > org.apache.kylin.dict.TrieDictionary.getIdFromValueImpl(TrieDictionary.java:161) > at org.apache.kylin.dict.Dictionary.getIdFromValue(Dictionary.java:91) > Then I dumped the dictionary to local and output all its values; In the mean > while I run hive SQL to list the distinct values for that column from the > flat table; Then I found the 1020 is appeared in hive table but not in the > dictionary; -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-634) Script to import sample data and cube metadata
[ https://issues.apache.org/jira/browse/KYLIN-634?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-634. -- > Script to import sample data and cube metadata > -- > > Key: KYLIN-634 > URL: https://issues.apache.org/jira/browse/KYLIN-634 > Project: Kylin > Issue Type: Improvement > Components: Tools, Build and Test >Reporter: Luke Han >Assignee: Shaofeng SHI >Priority: Major > Fix For: v0.7.1 > > > The current binary package will not ship with sample hive data and cube. > Using another package and script to import sample data into hive and kylin > for further experience and testing. > Ship binary package with sample data is perfect solution -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Closed] (KYLIN-639) Get "Table 'xxxx' not found while executing SQL" error after a cube be successfully built
[ https://issues.apache.org/jira/browse/KYLIN-639?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI closed KYLIN-639. -- > Get "Table '' not found while executing SQL" error after a cube be > successfully built > - > > Key: KYLIN-639 > URL: https://issues.apache.org/jira/browse/KYLIN-639 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v0.7.1 >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI >Priority: Major > Fix For: v0.7.1 > > Original Estimate: 24h > Remaining Estimate: 24h > > After building a cube successfully, execute a SQL on the "query" tab, but got > error like: > rom line 1, column 86 to line 1, column 96: Table 'KYLIN_SALES' not found > while executing SQL: "select cal_dt, sum(price) as total_selled, > count(distinct seller_id) as sellers from kylin_sales group by cal_dt order > by cal_dt LIMIT 5" > In the left table tree of "query" tab, the tables that defined in the cube > wasn't displayed as well; But if restart the kylin server, everything goes > well. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3140) Auto merge jobs should not block user build jobs
[ https://issues.apache.org/jira/browse/KYLIN-3140?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16351720#comment-16351720 ] Shaofeng SHI commented on KYLIN-3140: - Okay I see; you want to control the concurrent new jobs, but don't want to it be impacted by the merge jobs. The auto-merge job can not be discarded easily as once the old is discarded, a new will be sumbmitted immediately. > Auto merge jobs should not block user build jobs > > > Key: KYLIN-3140 > URL: https://issues.apache.org/jira/browse/KYLIN-3140 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Reporter: Wang, Gang >Assignee: Shaofeng SHI >Priority: Major > > Although in the latest version, Kylin support concurrent jobs. If the > concurrency is set to 1, there is some possibility that cube build job will > have dead lock. Say, when there is some issue which causes merge job failed, > even when you discard the job, another job will be launched and failed again > due to auto merge policy. And this failed merge job blocks user to build > incremental segment. > Even if the concurrency is set to larger than 1, the auto merge jobs occupy > some concurrency quota. > While, from user perspective, they don't care much about the auto merge jobs, > and the auto merge jobs should not block the building/refresh jobs they > submit manually. > A better way may be separating the auto merge jobs from the job queue, > parameter max-building-segments only limit jobs submitted by users. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
