[jira] [Commented] (KYLIN-4722) Add more statistics to the query results
[ https://issues.apache.org/jira/browse/KYLIN-4722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185568#comment-17185568 ] ASF GitHub Bot commented on KYLIN-4722: --- RupengWang opened a new pull request #1380: URL: https://github.com/apache/kylin/pull/1380 ## Proposed changes Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request. If it fixes a bug or resolves a feature request, be sure to link to that issue. ## Types of changes What types of changes does your code introduce to Kylin? _Put an `x` in the boxes that apply_ - [x] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation Update (if none of the other choices apply) ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [x] I have create an issue on [Kylin's jira](https://issues.apache.org/jira/browse/KYLIN), and have described the bug/feature there in detail - [x] Commit messages in my PR start with the related jira ID, like "KYLIN- Make Kylin project open-source" - [x] Compiling and unit tests pass locally with my changes - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] If this change need a document change, I will prepare another pr against the `document` branch - [ ] Any dependent changes have been merged ## Further comments If this is a relatively large or complex change, kick off the discussion at user@kylin or dev@kylin by explaining why you chose the solution you did and what alternatives you considered, etc... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add more statistics to the query results > > > Key: KYLIN-4722 > URL: https://issues.apache.org/jira/browse/KYLIN-4722 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Affects Versions: v4.0.0-alpha >Reporter: wangrupeng >Assignee: wangrupeng >Priority: Minor > Fix For: v4.0.0-alpha > > > Now, the query result contains scaned rows, scaned bytes. There are some > other statistics can be added like the number of scan files, spark scan time, > etc. > It will be useful to add the number of parquet files scaned when querying, > especially, the shard by column is configured which will decrease the > number of scaned parquet files to improve query efficency. > To read more about shard by column with below link. > [https://cwiki.apache.org/confluence/display/KYLIN/Improving+query+effeciency+by+set+shard+by+column] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [kylin] RupengWang opened a new pull request #1380: KYLIN-4722 && KYLIN-4721
RupengWang opened a new pull request #1380: URL: https://github.com/apache/kylin/pull/1380 ## Proposed changes Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request. If it fixes a bug or resolves a feature request, be sure to link to that issue. ## Types of changes What types of changes does your code introduce to Kylin? _Put an `x` in the boxes that apply_ - [x] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation Update (if none of the other choices apply) ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [x] I have create an issue on [Kylin's jira](https://issues.apache.org/jira/browse/KYLIN), and have described the bug/feature there in detail - [x] Commit messages in my PR start with the related jira ID, like "KYLIN- Make Kylin project open-source" - [x] Compiling and unit tests pass locally with my changes - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] If this change need a document change, I will prepare another pr against the `document` branch - [ ] Any dependent changes have been merged ## Further comments If this is a relatively large or complex change, kick off the discussion at user@kylin or dev@kylin by explaining why you chose the solution you did and what alternatives you considered, etc... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kylin] RupengWang closed pull request #1379: KYLIN-4722 && KYLIN-4721
RupengWang closed pull request #1379: URL: https://github.com/apache/kylin/pull/1379 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KYLIN-4722) Add more statistics to the query results
[ https://issues.apache.org/jira/browse/KYLIN-4722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185565#comment-17185565 ] ASF GitHub Bot commented on KYLIN-4722: --- RupengWang opened a new pull request #1379: URL: https://github.com/apache/kylin/pull/1379 ## Proposed changes Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request. If it fixes a bug or resolves a feature request, be sure to link to that issue. ## Types of changes What types of changes does your code introduce to Kylin? _Put an `x` in the boxes that apply_ - [x] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation Update (if none of the other choices apply) ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [ ] I have create an issue on [Kylin's jira](https://issues.apache.org/jira/browse/KYLIN), and have described the bug/feature there in detail - [ ] Commit messages in my PR start with the related jira ID, like "KYLIN- Make Kylin project open-source" - [ ] Compiling and unit tests pass locally with my changes - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] If this change need a document change, I will prepare another pr against the `document` branch - [ ] Any dependent changes have been merged ## Further comments If this is a relatively large or complex change, kick off the discussion at user@kylin or dev@kylin by explaining why you chose the solution you did and what alternatives you considered, etc... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add more statistics to the query results > > > Key: KYLIN-4722 > URL: https://issues.apache.org/jira/browse/KYLIN-4722 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Affects Versions: v4.0.0-alpha >Reporter: wangrupeng >Assignee: wangrupeng >Priority: Minor > Fix For: v4.0.0-alpha > > > Now, the query result contains scaned rows, scaned bytes. There are some > other statistics can be added like the number of scan files, spark scan time, > etc. > It will be useful to add the number of parquet files scaned when querying, > especially, the shard by column is configured which will decrease the > number of scaned parquet files to improve query efficency. > To read more about shard by column with below link. > [https://cwiki.apache.org/confluence/display/KYLIN/Improving+query+effeciency+by+set+shard+by+column] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4722) Add more statistics to the query results
[ https://issues.apache.org/jira/browse/KYLIN-4722?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185566#comment-17185566 ] ASF GitHub Bot commented on KYLIN-4722: --- RupengWang closed pull request #1379: URL: https://github.com/apache/kylin/pull/1379 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add more statistics to the query results > > > Key: KYLIN-4722 > URL: https://issues.apache.org/jira/browse/KYLIN-4722 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Affects Versions: v4.0.0-alpha >Reporter: wangrupeng >Assignee: wangrupeng >Priority: Minor > Fix For: v4.0.0-alpha > > > Now, the query result contains scaned rows, scaned bytes. There are some > other statistics can be added like the number of scan files, spark scan time, > etc. > It will be useful to add the number of parquet files scaned when querying, > especially, the shard by column is configured which will decrease the > number of scaned parquet files to improve query efficency. > To read more about shard by column with below link. > [https://cwiki.apache.org/confluence/display/KYLIN/Improving+query+effeciency+by+set+shard+by+column] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [kylin] RupengWang opened a new pull request #1379: KYLIN-4722 && KYLIN-4721
RupengWang opened a new pull request #1379: URL: https://github.com/apache/kylin/pull/1379 ## Proposed changes Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request. If it fixes a bug or resolves a feature request, be sure to link to that issue. ## Types of changes What types of changes does your code introduce to Kylin? _Put an `x` in the boxes that apply_ - [x] Bugfix (non-breaking change which fixes an issue) - [x] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation Update (if none of the other choices apply) ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [ ] I have create an issue on [Kylin's jira](https://issues.apache.org/jira/browse/KYLIN), and have described the bug/feature there in detail - [ ] Commit messages in my PR start with the related jira ID, like "KYLIN- Make Kylin project open-source" - [ ] Compiling and unit tests pass locally with my changes - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] If this change need a document change, I will prepare another pr against the `document` branch - [ ] Any dependent changes have been merged ## Further comments If this is a relatively large or complex change, kick off the discussion at user@kylin or dev@kylin by explaining why you chose the solution you did and what alternatives you considered, etc... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kylin] hit-lacus commented on a change in pull request #1376: KYLIN-4698 Delete segment storage path after merging segment, deleting segment and droping cube
hit-lacus commented on a change in pull request #1376: URL: https://github.com/apache/kylin/pull/1376#discussion_r477945116 ## File path: kylin-spark-project/kylin-spark-engine/src/test/java/org/apache/kylin/engine/spark/LocalWithSparkSessionTest.java ## @@ -191,7 +201,15 @@ protected ExecutableState mergeSegments(String cubeName, long start, long end, b CubeSegment mergeSegment = cubeMgr.mergeSegments(cube, new SegmentRange.TSRange(start, end), null, force); NSparkMergingJob mergeJob = NSparkMergingJob.merge(mergeSegment, "ADMIN"); execMgr.addJob(mergeJob); -return wait(mergeJob); Review comment: Why remove `wait` method? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KYLIN-4698) Delete segment storage path after merging segment, deleting segment and droping cube
[ https://issues.apache.org/jira/browse/KYLIN-4698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185562#comment-17185562 ] ASF GitHub Bot commented on KYLIN-4698: --- hit-lacus commented on a change in pull request #1376: URL: https://github.com/apache/kylin/pull/1376#discussion_r477945116 ## File path: kylin-spark-project/kylin-spark-engine/src/test/java/org/apache/kylin/engine/spark/LocalWithSparkSessionTest.java ## @@ -191,7 +201,15 @@ protected ExecutableState mergeSegments(String cubeName, long start, long end, b CubeSegment mergeSegment = cubeMgr.mergeSegments(cube, new SegmentRange.TSRange(start, end), null, force); NSparkMergingJob mergeJob = NSparkMergingJob.merge(mergeSegment, "ADMIN"); execMgr.addJob(mergeJob); -return wait(mergeJob); Review comment: Why remove `wait` method? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Delete segment storage path after merging segment, deleting segment and > droping cube > > > Key: KYLIN-4698 > URL: https://issues.apache.org/jira/browse/KYLIN-4698 > Project: Kylin > Issue Type: Bug > Components: Storage - Parquet >Affects Versions: v4.0.0-alpha >Reporter: Zhichao Zhang >Assignee: Zhichao Zhang >Priority: Minor > > It doesn't delete segment storage path when merge segments, delete segments > and drop cube for Kylin 4.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4698) Delete segment storage path after merging segment, deleting segment and droping cube
[
https://issues.apache.org/jira/browse/KYLIN-4698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185561#comment-17185561
]
ASF GitHub Bot commented on KYLIN-4698:
---
hit-lacus commented on a change in pull request #1376:
URL: https://github.com/apache/kylin/pull/1376#discussion_r477937494
##
File path:
core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
##
@@ -307,6 +307,7 @@ public String getHdfsWorkingDirectory() {
String metaId = getMetadataUrlPrefix().replace(':', '-');
//transform relative path for local metadata
if (metaId.startsWith("../")) {
+metaId = metaId.replace("../", "");
Review comment:
This replace action looks confusing. Could you please add a comment to
explain its purpose?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Delete segment storage path after merging segment, deleting segment and
> droping cube
>
>
> Key: KYLIN-4698
> URL: https://issues.apache.org/jira/browse/KYLIN-4698
> Project: Kylin
> Issue Type: Bug
> Components: Storage - Parquet
>Affects Versions: v4.0.0-alpha
>Reporter: Zhichao Zhang
>Assignee: Zhichao Zhang
>Priority: Minor
>
> It doesn't delete segment storage path when merge segments, delete segments
> and drop cube for Kylin 4.0
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [kylin] hit-lacus commented on a change in pull request #1376: KYLIN-4698 Delete segment storage path after merging segment, deleting segment and droping cube
hit-lacus commented on a change in pull request #1376:
URL: https://github.com/apache/kylin/pull/1376#discussion_r477937494
##
File path:
core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
##
@@ -307,6 +307,7 @@ public String getHdfsWorkingDirectory() {
String metaId = getMetadataUrlPrefix().replace(':', '-');
//transform relative path for local metadata
if (metaId.startsWith("../")) {
+metaId = metaId.replace("../", "");
Review comment:
This replace action looks confusing. Could you please add a comment to
explain its purpose?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[GitHub] [kylin] hit-lacus commented on a change in pull request #1376: KYLIN-4698 Delete segment storage path after merging segment, deleting segment and droping cube
hit-lacus commented on a change in pull request #1376:
URL: https://github.com/apache/kylin/pull/1376#discussion_r477948867
##
File path:
server-base/src/main/java/org/apache/kylin/rest/init/InitialSparkerContext.java
##
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kylin.rest.init;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.kylin.common.KylinConfig;
+import org.apache.spark.sql.SparderContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.beans.factory.InitializingBean;
+
+import java.io.File;
+import java.nio.file.Paths;
+
+/**
+ * Created by zhangzc on 8/26/20.
+ */
+public class InitialSparkerContext implements InitializingBean {
+
+private static final Logger logger =
LoggerFactory.getLogger(InitialSparkerContext.class);
+
+@Override
+public void afterPropertiesSet() throws Exception {
+runInitialSparder();
+}
+
+private void runInitialSparder() {
+logger.info("Spark is starting.");
+SparderContext.init();
+final String kylinHome =
StringUtils.defaultIfBlank(KylinConfig.getKylinHome(), "./");
+final File appidFile = Paths.get(kylinHome, "sparkappid").toFile();
Review comment:
If current spark context is killed and restarted, will new applicationId
update here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (KYLIN-4698) Delete segment storage path after merging segment, deleting segment and droping cube
[
https://issues.apache.org/jira/browse/KYLIN-4698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185557#comment-17185557
]
ASF GitHub Bot commented on KYLIN-4698:
---
hit-lacus commented on a change in pull request #1376:
URL: https://github.com/apache/kylin/pull/1376#discussion_r477948867
##
File path:
server-base/src/main/java/org/apache/kylin/rest/init/InitialSparkerContext.java
##
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kylin.rest.init;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.kylin.common.KylinConfig;
+import org.apache.spark.sql.SparderContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.beans.factory.InitializingBean;
+
+import java.io.File;
+import java.nio.file.Paths;
+
+/**
+ * Created by zhangzc on 8/26/20.
+ */
+public class InitialSparkerContext implements InitializingBean {
+
+private static final Logger logger =
LoggerFactory.getLogger(InitialSparkerContext.class);
+
+@Override
+public void afterPropertiesSet() throws Exception {
+runInitialSparder();
+}
+
+private void runInitialSparder() {
+logger.info("Spark is starting.");
+SparderContext.init();
+final String kylinHome =
StringUtils.defaultIfBlank(KylinConfig.getKylinHome(), "./");
+final File appidFile = Paths.get(kylinHome, "sparkappid").toFile();
Review comment:
If current spark context is killed and restarted, will new applicationId
update here?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Delete segment storage path after merging segment, deleting segment and
> droping cube
>
>
> Key: KYLIN-4698
> URL: https://issues.apache.org/jira/browse/KYLIN-4698
> Project: Kylin
> Issue Type: Bug
> Components: Storage - Parquet
>Affects Versions: v4.0.0-alpha
>Reporter: Zhichao Zhang
>Assignee: Zhichao Zhang
>Priority: Minor
>
> It doesn't delete segment storage path when merge segments, delete segments
> and drop cube for Kylin 4.0
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-4698) Delete segment storage path after merging segment, deleting segment and droping cube
[
https://issues.apache.org/jira/browse/KYLIN-4698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=1718#comment-1718
]
ASF GitHub Bot commented on KYLIN-4698:
---
hit-lacus commented on a change in pull request #1376:
URL: https://github.com/apache/kylin/pull/1376#discussion_r477947020
##
File path:
server-base/src/main/java/org/apache/kylin/rest/init/InitialSparkerContext.java
##
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kylin.rest.init;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.kylin.common.KylinConfig;
+import org.apache.spark.sql.SparderContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.beans.factory.InitializingBean;
+
+import java.io.File;
+import java.nio.file.Paths;
+
+/**
+ * Created by zhangzc on 8/26/20.
+ */
+public class InitialSparkerContext implements InitializingBean {
Review comment:
`Sparker` or `Sparder`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Delete segment storage path after merging segment, deleting segment and
> droping cube
>
>
> Key: KYLIN-4698
> URL: https://issues.apache.org/jira/browse/KYLIN-4698
> Project: Kylin
> Issue Type: Bug
> Components: Storage - Parquet
>Affects Versions: v4.0.0-alpha
>Reporter: Zhichao Zhang
>Assignee: Zhichao Zhang
>Priority: Minor
>
> It doesn't delete segment storage path when merge segments, delete segments
> and drop cube for Kylin 4.0
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [kylin] hit-lacus commented on a change in pull request #1376: KYLIN-4698 Delete segment storage path after merging segment, deleting segment and droping cube
hit-lacus commented on a change in pull request #1376: URL: https://github.com/apache/kylin/pull/1376#discussion_r477946492 ## File path: server-base/src/main/java/org/apache/kylin/rest/init/InitialSparkerContext.java ## @@ -0,0 +1,59 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.kylin.rest.init; + +import org.apache.commons.io.FileUtils; +import org.apache.commons.lang3.StringUtils; +import org.apache.kylin.common.KylinConfig; +import org.apache.spark.sql.SparderContext; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.InitializingBean; + +import java.io.File; +import java.nio.file.Paths; + +/** + * Created by zhangzc on 8/26/20. Review comment: Please not leave your name. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KYLIN-4698) Delete segment storage path after merging segment, deleting segment and droping cube
[ https://issues.apache.org/jira/browse/KYLIN-4698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185554#comment-17185554 ] ASF GitHub Bot commented on KYLIN-4698: --- hit-lacus commented on a change in pull request #1376: URL: https://github.com/apache/kylin/pull/1376#discussion_r477946492 ## File path: server-base/src/main/java/org/apache/kylin/rest/init/InitialSparkerContext.java ## @@ -0,0 +1,59 @@ +/* + * Licensed to the Apache Software Foundation (ASF) under one + * or more contributor license agreements. See the NOTICE file + * distributed with this work for additional information + * regarding copyright ownership. The ASF licenses this file + * to you under the Apache License, Version 2.0 (the + * "License"); you may not use this file except in compliance + * with the License. You may obtain a copy of the License at + * + * http://www.apache.org/licenses/LICENSE-2.0 + * + * Unless required by applicable law or agreed to in writing, software + * distributed under the License is distributed on an "AS IS" BASIS, + * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. + * See the License for the specific language governing permissions and + * limitations under the License. + */ + +package org.apache.kylin.rest.init; + +import org.apache.commons.io.FileUtils; +import org.apache.commons.lang3.StringUtils; +import org.apache.kylin.common.KylinConfig; +import org.apache.spark.sql.SparderContext; +import org.slf4j.Logger; +import org.slf4j.LoggerFactory; +import org.springframework.beans.factory.InitializingBean; + +import java.io.File; +import java.nio.file.Paths; + +/** + * Created by zhangzc on 8/26/20. Review comment: Please not leave your name. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Delete segment storage path after merging segment, deleting segment and > droping cube > > > Key: KYLIN-4698 > URL: https://issues.apache.org/jira/browse/KYLIN-4698 > Project: Kylin > Issue Type: Bug > Components: Storage - Parquet >Affects Versions: v4.0.0-alpha >Reporter: Zhichao Zhang >Assignee: Zhichao Zhang >Priority: Minor > > It doesn't delete segment storage path when merge segments, delete segments > and drop cube for Kylin 4.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [kylin] hit-lacus commented on a change in pull request #1376: KYLIN-4698 Delete segment storage path after merging segment, deleting segment and droping cube
hit-lacus commented on a change in pull request #1376:
URL: https://github.com/apache/kylin/pull/1376#discussion_r477947020
##
File path:
server-base/src/main/java/org/apache/kylin/rest/init/InitialSparkerContext.java
##
@@ -0,0 +1,59 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kylin.rest.init;
+
+import org.apache.commons.io.FileUtils;
+import org.apache.commons.lang3.StringUtils;
+import org.apache.kylin.common.KylinConfig;
+import org.apache.spark.sql.SparderContext;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+import org.springframework.beans.factory.InitializingBean;
+
+import java.io.File;
+import java.nio.file.Paths;
+
+/**
+ * Created by zhangzc on 8/26/20.
+ */
+public class InitialSparkerContext implements InitializingBean {
Review comment:
`Sparker` or `Sparder`?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (KYLIN-4698) Delete segment storage path after merging segment, deleting segment and droping cube
[ https://issues.apache.org/jira/browse/KYLIN-4698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185553#comment-17185553 ] ASF GitHub Bot commented on KYLIN-4698: --- hit-lacus commented on a change in pull request #1376: URL: https://github.com/apache/kylin/pull/1376#discussion_r477945116 ## File path: kylin-spark-project/kylin-spark-engine/src/test/java/org/apache/kylin/engine/spark/LocalWithSparkSessionTest.java ## @@ -191,7 +201,15 @@ protected ExecutableState mergeSegments(String cubeName, long start, long end, b CubeSegment mergeSegment = cubeMgr.mergeSegments(cube, new SegmentRange.TSRange(start, end), null, force); NSparkMergingJob mergeJob = NSparkMergingJob.merge(mergeSegment, "ADMIN"); execMgr.addJob(mergeJob); -return wait(mergeJob); Review comment: Why remove `wait` method? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Delete segment storage path after merging segment, deleting segment and > droping cube > > > Key: KYLIN-4698 > URL: https://issues.apache.org/jira/browse/KYLIN-4698 > Project: Kylin > Issue Type: Bug > Components: Storage - Parquet >Affects Versions: v4.0.0-alpha >Reporter: Zhichao Zhang >Assignee: Zhichao Zhang >Priority: Minor > > It doesn't delete segment storage path when merge segments, delete segments > and drop cube for Kylin 4.0 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [kylin] hit-lacus commented on a change in pull request #1376: KYLIN-4698 Delete segment storage path after merging segment, deleting segment and droping cube
hit-lacus commented on a change in pull request #1376: URL: https://github.com/apache/kylin/pull/1376#discussion_r477945116 ## File path: kylin-spark-project/kylin-spark-engine/src/test/java/org/apache/kylin/engine/spark/LocalWithSparkSessionTest.java ## @@ -191,7 +201,15 @@ protected ExecutableState mergeSegments(String cubeName, long start, long end, b CubeSegment mergeSegment = cubeMgr.mergeSegments(cube, new SegmentRange.TSRange(start, end), null, force); NSparkMergingJob mergeJob = NSparkMergingJob.merge(mergeSegment, "ADMIN"); execMgr.addJob(mergeJob); -return wait(mergeJob); Review comment: Why remove `wait` method? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kylin] hit-lacus commented on a change in pull request #1376: KYLIN-4698 Delete segment storage path after merging segment, deleting segment and droping cube
hit-lacus commented on a change in pull request #1376:
URL: https://github.com/apache/kylin/pull/1376#discussion_r477937494
##
File path:
core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
##
@@ -307,6 +307,7 @@ public String getHdfsWorkingDirectory() {
String metaId = getMetadataUrlPrefix().replace(':', '-');
//transform relative path for local metadata
if (metaId.startsWith("../")) {
+metaId = metaId.replace("../", "");
Review comment:
This replace action looks confusing.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (KYLIN-4698) Delete segment storage path after merging segment, deleting segment and droping cube
[
https://issues.apache.org/jira/browse/KYLIN-4698?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185548#comment-17185548
]
ASF GitHub Bot commented on KYLIN-4698:
---
hit-lacus commented on a change in pull request #1376:
URL: https://github.com/apache/kylin/pull/1376#discussion_r477937494
##
File path:
core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
##
@@ -307,6 +307,7 @@ public String getHdfsWorkingDirectory() {
String metaId = getMetadataUrlPrefix().replace(':', '-');
//transform relative path for local metadata
if (metaId.startsWith("../")) {
+metaId = metaId.replace("../", "");
Review comment:
This replace action looks confusing.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Delete segment storage path after merging segment, deleting segment and
> droping cube
>
>
> Key: KYLIN-4698
> URL: https://issues.apache.org/jira/browse/KYLIN-4698
> Project: Kylin
> Issue Type: Bug
> Components: Storage - Parquet
>Affects Versions: v4.0.0-alpha
>Reporter: Zhichao Zhang
>Assignee: Zhichao Zhang
>Priority: Minor
>
> It doesn't delete segment storage path when merge segments, delete segments
> and drop cube for Kylin 4.0
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [kylin] hit-lacus merged pull request #1375: KYLIN-4715 Wrong function with kylin document about how to optimize c…
hit-lacus merged pull request #1375: URL: https://github.com/apache/kylin/pull/1375 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KYLIN-4715) Wrong function with kylin document about how to optimize cube build
[ https://issues.apache.org/jira/browse/KYLIN-4715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185546#comment-17185546 ] ASF GitHub Bot commented on KYLIN-4715: --- hit-lacus merged pull request #1375: URL: https://github.com/apache/kylin/pull/1375 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Wrong function with kylin document about how to optimize cube build > --- > > Key: KYLIN-4715 > URL: https://issues.apache.org/jira/browse/KYLIN-4715 > Project: Kylin > Issue Type: Bug > Components: Documentation >Reporter: wangrupeng >Assignee: wangrupeng >Priority: Minor > Fix For: v3.1.1 > > Attachments: image-2020-08-25-11-13-55-160.png, > image-2020-08-25-11-14-14-556.png > > > [http://kylin.apache.org/docs/howto/howto_optimize_build.html] > The number of cuboids should be N*(N-1)/2 when with the (N-2) dimensions. > !image-2020-08-25-11-13-55-160.png|width=660,height=337! > !image-2020-08-25-11-14-14-556.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4715) Wrong function with kylin document about how to optimize cube build
[ https://issues.apache.org/jira/browse/KYLIN-4715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185547#comment-17185547 ] ASF subversion and git services commented on KYLIN-4715: Commit 23a7e3c9ada4ad5f298302553e9661c36b235e3d in kylin's branch refs/heads/document from rupengwang [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=23a7e3c ] KYLIN-4715 Wrong function with kylin document about how to optimize cube build > Wrong function with kylin document about how to optimize cube build > --- > > Key: KYLIN-4715 > URL: https://issues.apache.org/jira/browse/KYLIN-4715 > Project: Kylin > Issue Type: Bug > Components: Documentation >Reporter: wangrupeng >Assignee: wangrupeng >Priority: Minor > Fix For: v3.1.1 > > Attachments: image-2020-08-25-11-13-55-160.png, > image-2020-08-25-11-14-14-556.png > > > [http://kylin.apache.org/docs/howto/howto_optimize_build.html] > The number of cuboids should be N*(N-1)/2 when with the (N-2) dimensions. > !image-2020-08-25-11-13-55-160.png|width=660,height=337! > !image-2020-08-25-11-14-14-556.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4715) Wrong function with kylin document about how to optimize cube build
[ https://issues.apache.org/jira/browse/KYLIN-4715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185545#comment-17185545 ] ASF GitHub Bot commented on KYLIN-4715: --- hit-lacus commented on pull request #1375: URL: https://github.com/apache/kylin/pull/1375#issuecomment-681300900 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Wrong function with kylin document about how to optimize cube build > --- > > Key: KYLIN-4715 > URL: https://issues.apache.org/jira/browse/KYLIN-4715 > Project: Kylin > Issue Type: Bug > Components: Documentation >Reporter: wangrupeng >Assignee: wangrupeng >Priority: Minor > Fix For: v3.1.1 > > Attachments: image-2020-08-25-11-13-55-160.png, > image-2020-08-25-11-14-14-556.png > > > [http://kylin.apache.org/docs/howto/howto_optimize_build.html] > The number of cuboids should be N*(N-1)/2 when with the (N-2) dimensions. > !image-2020-08-25-11-13-55-160.png|width=660,height=337! > !image-2020-08-25-11-14-14-556.png! -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [kylin] hit-lacus commented on pull request #1375: KYLIN-4715 Wrong function with kylin document about how to optimize c…
hit-lacus commented on pull request #1375: URL: https://github.com/apache/kylin/pull/1375#issuecomment-681300900 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (KYLIN-4722) Add more statistics to the query results
[ https://issues.apache.org/jira/browse/KYLIN-4722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangrupeng updated KYLIN-4722: -- Description: Now, the query result contains scaned rows, scaned bytes. There are some other statistics can be added like the number of scan files, spark scan time, etc. It will be useful to add the number of parquet files scaned when querying, especially, the shard by column is configured which will decrease the number of scaned parquet files to improve query efficency. To read more about shard by column with below link. [https://cwiki.apache.org/confluence/display/KYLIN/Improving+query+effeciency+by+set+shard+by+column] was: Now, the query result contains scaned rows, scaned bytes. It will be useful to add the number of parquet files scaned when querying, especially, the shard by column is configured which will decrease the number of scaned parquet files to improve query efficency. To read more about shard by column with below link. [https://cwiki.apache.org/confluence/display/KYLIN/Improving+query+effeciency+by+set+shard+by+column] > Add more statistics to the query results > > > Key: KYLIN-4722 > URL: https://issues.apache.org/jira/browse/KYLIN-4722 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Affects Versions: v4.0.0-alpha >Reporter: wangrupeng >Assignee: wangrupeng >Priority: Minor > Fix For: v4.0.0-alpha > > > Now, the query result contains scaned rows, scaned bytes. There are some > other statistics can be added like the number of scan files, spark scan time, > etc. It will be useful to add the number of parquet files scaned when > querying, especially, the shard by column is configured which will decrease > the number of scaned parquet files to improve query efficency. > To read more about shard by column with below link. > [https://cwiki.apache.org/confluence/display/KYLIN/Improving+query+effeciency+by+set+shard+by+column] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4722) Add more statistics to the query results
[ https://issues.apache.org/jira/browse/KYLIN-4722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangrupeng updated KYLIN-4722: -- Description: Now, the query result contains scaned rows, scaned bytes. There are some other statistics can be added like the number of scan files, spark scan time, etc. It will be useful to add the number of parquet files scaned when querying, especially, the shard by column is configured which will decrease the number of scaned parquet files to improve query efficency. To read more about shard by column with below link. [https://cwiki.apache.org/confluence/display/KYLIN/Improving+query+effeciency+by+set+shard+by+column] was: Now, the query result contains scaned rows, scaned bytes. There are some other statistics can be added like the number of scan files, spark scan time, etc. It will be useful to add the number of parquet files scaned when querying, especially, the shard by column is configured which will decrease the number of scaned parquet files to improve query efficency. To read more about shard by column with below link. [https://cwiki.apache.org/confluence/display/KYLIN/Improving+query+effeciency+by+set+shard+by+column] > Add more statistics to the query results > > > Key: KYLIN-4722 > URL: https://issues.apache.org/jira/browse/KYLIN-4722 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Affects Versions: v4.0.0-alpha >Reporter: wangrupeng >Assignee: wangrupeng >Priority: Minor > Fix For: v4.0.0-alpha > > > Now, the query result contains scaned rows, scaned bytes. There are some > other statistics can be added like the number of scan files, spark scan time, > etc. > It will be useful to add the number of parquet files scaned when querying, > especially, the shard by column is configured which will decrease the > number of scaned parquet files to improve query efficency. > To read more about shard by column with below link. > [https://cwiki.apache.org/confluence/display/KYLIN/Improving+query+effeciency+by+set+shard+by+column] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Updated] (KYLIN-4722) Add more statistics to the query results
[ https://issues.apache.org/jira/browse/KYLIN-4722?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] wangrupeng updated KYLIN-4722: -- Summary: Add more statistics to the query results (was: Add the number of files scaned when querying) > Add more statistics to the query results > > > Key: KYLIN-4722 > URL: https://issues.apache.org/jira/browse/KYLIN-4722 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Affects Versions: v4.0.0-alpha >Reporter: wangrupeng >Assignee: wangrupeng >Priority: Minor > Fix For: v4.0.0-alpha > > > Now, the query result contains scaned rows, scaned bytes. It will be useful > to add the number of parquet files scaned when querying, especially, the > shard by column is configured which will decrease the number of scaned > parquet files to improve query efficency. > To read more about shard by column with below link. > [https://cwiki.apache.org/confluence/display/KYLIN/Improving+query+effeciency+by+set+shard+by+column] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4722) Add the number of files scaned when querying
wangrupeng created KYLIN-4722: - Summary: Add the number of files scaned when querying Key: KYLIN-4722 URL: https://issues.apache.org/jira/browse/KYLIN-4722 Project: Kylin Issue Type: Improvement Components: Query Engine Affects Versions: v4.0.0-alpha Reporter: wangrupeng Assignee: wangrupeng Fix For: v4.0.0-alpha Now, the query result contains scaned rows, scaned bytes. It will be useful to add the number of parquet files scaned when querying, especially, the shard by column is configured which will decrease the number of scaned parquet files to improve query efficency. To read more about shard by column with below link. [https://cwiki.apache.org/confluence/display/KYLIN/Improving+query+effeciency+by+set+shard+by+column] -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4721) The default source source type should be CSV not Hive with the local debug mode
wangrupeng created KYLIN-4721: - Summary: The default source source type should be CSV not Hive with the local debug mode Key: KYLIN-4721 URL: https://issues.apache.org/jira/browse/KYLIN-4721 Project: Kylin Issue Type: Bug Components: Metadata Affects Versions: v4.0.0-alpha Reporter: wangrupeng Assignee: wangrupeng Fix For: v4.0.0-alpha When debuging kylin 4.0 with tomcat local mode, Kylin will use the metadata which is located in $KYLIN_SOURCE/examples/test_case_data/sample_local and the source type of tables is hive. The build task will remain pending because it cannot connect the remote hadoop cluster. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [kylin] coveralls commented on pull request #1378: KYLIN-4720 support htable read quota
coveralls commented on pull request #1378: URL: https://github.com/apache/kylin/pull/1378#issuecomment-680769682 ## Pull Request Test Coverage Report for [Build 6291](https://coveralls.io/builds/33009741) * **0** of **6** **(0.0%)** changed or added relevant lines in **2** files are covered. * **5** unchanged lines in **3** files lost coverage. * Overall coverage increased (+**0.004%**) to **28.061%** --- | Changes Missing Coverage | Covered Lines | Changed/Added Lines | % | | :-|--||---: | | [core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java](https://coveralls.io/builds/33009741/source?filename=core-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcommon%2FKylinConfigBase.java#L2691) | 0 | 1 | 0.0% | [storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CubeHTableUtil.java](https://coveralls.io/builds/33009741/source?filename=storage-hbase%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fstorage%2Fhbase%2Fsteps%2FCubeHTableUtil.java#L171) | 0 | 5 | 0.0% | Files with Coverage Reduction | New Missed Lines | % | | :-|--|--: | | [tool/src/main/java/org/apache/kylin/tool/query/ProbabilityGenerator.java](https://coveralls.io/builds/33009741/source?filename=tool%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Ftool%2Fquery%2FProbabilityGenerator.java#L42) | 1 | 81.58% | | [stream-core/src/main/java/org/apache/kylin/stream/core/storage/CheckPointStore.java](https://coveralls.io/builds/33009741/source?filename=stream-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fstream%2Fcore%2Fstorage%2FCheckPointStore.java#L98) | 2 | 73.74% | | [stream-core/src/main/java/org/apache/kylin/stream/core/storage/columnar/ColumnarStoreCache.java](https://coveralls.io/builds/33009741/source?filename=stream-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fstream%2Fcore%2Fstorage%2Fcolumnar%2FColumnarStoreCache.java#L80) | 2 | 57.95% | | Totals | [](https://coveralls.io/builds/33009741) | | :-- | --: | | Change from base [Build 6276](https://coveralls.io/builds/32939506): | 0.004% | | Covered Lines: | 26249 | | Relevant Lines: | 93544 | --- # - [Coveralls](https://coveralls.io) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KYLIN-4720) support htable read quota
[ https://issues.apache.org/jira/browse/KYLIN-4720?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17185059#comment-17185059 ] ASF GitHub Bot commented on KYLIN-4720: --- coveralls commented on pull request #1378: URL: https://github.com/apache/kylin/pull/1378#issuecomment-680769682 ## Pull Request Test Coverage Report for [Build 6291](https://coveralls.io/builds/33009741) * **0** of **6** **(0.0%)** changed or added relevant lines in **2** files are covered. * **5** unchanged lines in **3** files lost coverage. * Overall coverage increased (+**0.004%**) to **28.061%** --- | Changes Missing Coverage | Covered Lines | Changed/Added Lines | % | | :-|--||---: | | [core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java](https://coveralls.io/builds/33009741/source?filename=core-common%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcommon%2FKylinConfigBase.java#L2691) | 0 | 1 | 0.0% | [storage-hbase/src/main/java/org/apache/kylin/storage/hbase/steps/CubeHTableUtil.java](https://coveralls.io/builds/33009741/source?filename=storage-hbase%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fstorage%2Fhbase%2Fsteps%2FCubeHTableUtil.java#L171) | 0 | 5 | 0.0% | Files with Coverage Reduction | New Missed Lines | % | | :-|--|--: | | [tool/src/main/java/org/apache/kylin/tool/query/ProbabilityGenerator.java](https://coveralls.io/builds/33009741/source?filename=tool%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Ftool%2Fquery%2FProbabilityGenerator.java#L42) | 1 | 81.58% | | [stream-core/src/main/java/org/apache/kylin/stream/core/storage/CheckPointStore.java](https://coveralls.io/builds/33009741/source?filename=stream-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fstream%2Fcore%2Fstorage%2FCheckPointStore.java#L98) | 2 | 73.74% | | [stream-core/src/main/java/org/apache/kylin/stream/core/storage/columnar/ColumnarStoreCache.java](https://coveralls.io/builds/33009741/source?filename=stream-core%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fstream%2Fcore%2Fstorage%2Fcolumnar%2FColumnarStoreCache.java#L80) | 2 | 57.95% | | Totals | [](https://coveralls.io/builds/33009741) | | :-- | --: | | Change from base [Build 6276](https://coveralls.io/builds/32939506): | 0.004% | | Covered Lines: | 26249 | | Relevant Lines: | 93544 | --- # - [Coveralls](https://coveralls.io) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > support htable read quota > - > > Key: KYLIN-4720 > URL: https://issues.apache.org/jira/browse/KYLIN-4720 > Project: Kylin > Issue Type: Improvement > Components: Storage - HBase >Affects Versions: v3.1.0, v3.1.1 >Reporter: fengpod >Priority: Minor > > In order to avoid the pressure of massive queries on hbase，htable read quota > is necessary。Through a new configuration “kylin.hbase.htable-quota”，we can > control the read quota when the htable created。The default value of this conf > is -1，means will not limit queries。You can set it to any intger greater than > 0 as needed。 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4720) support htable read quota
fengpod created KYLIN-4720: -- Summary: support htable read quota Key: KYLIN-4720 URL: https://issues.apache.org/jira/browse/KYLIN-4720 Project: Kylin Issue Type: Improvement Components: Storage - HBase Affects Versions: v3.1.0, v3.1.1 Reporter: fengpod In order to avoid the pressure of massive queries on hbase，htable read quota is necessary。Through a new configuration “kylin.hbase.htable-quota”，we can control the read quota when the htable created。The default value of this conf is -1，means will not limit queries。You can set it to any intger greater than 0 as needed。 -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [kylin] fengpod opened a new pull request #1378: support htable read quota
fengpod opened a new pull request #1378: URL: https://github.com/apache/kylin/pull/1378 ## Proposed changes Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request. If it fixes a bug or resolves a feature request, be sure to link to that issue. ## Types of changes What types of changes does your code introduce to Kylin? _Put an `x` in the boxes that apply_ - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation Update (if none of the other choices apply) ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [ ] I have create an issue on [Kylin's jira](https://issues.apache.org/jira/browse/KYLIN), and have described the bug/feature there in detail - [ ] Commit messages in my PR start with the related jira ID, like "KYLIN- Make Kylin project open-source" - [ ] Compiling and unit tests pass locally with my changes - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] If this change need a document change, I will prepare another pr against the `document` branch - [ ] Any dependent changes have been merged ## Further comments If this is a relatively large or complex change, kick off the discussion at user@kylin or dev@kylin by explaining why you chose the solution you did and what alternatives you considered, etc... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kylin] fengpod commented on pull request #1378: support htable read quota
fengpod commented on pull request #1378: URL: https://github.com/apache/kylin/pull/1378#issuecomment-680747685 In order to avoid the pressure of massive queries on hbase，htable read quota is necessary。Through a new configuration “kylin.hbase.htable-quota”，we can control the read quota when the htable created。The default value of this conf is -1，means will not limit queries。You can set it to any intger greater than 0 as needed。 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KYLIN-4679) StorageCleanupJob clean hive table support hive table prefix
[ https://issues.apache.org/jira/browse/KYLIN-4679?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17184998#comment-17184998 ] ASF GitHub Bot commented on KYLIN-4679: --- bigxiaochu commented on pull request #1340: URL: https://github.com/apache/kylin/pull/1340#issuecomment-680718484 @hit-lacus add content This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > StorageCleanupJob clean hive table support hive table prefix > > > Key: KYLIN-4679 > URL: https://issues.apache.org/jira/browse/KYLIN-4679 > Project: Kylin > Issue Type: Improvement >Reporter: chuxiao >Priority: Major > > 我的库有几千张表。StorageCleanupJob的行为是获取库里所有表的元数据再过滤表名。首先，取所有表元数据的行为会给metastore较大压力。其次，我们的metastore当一次取超过1千张表的元数据，会直接报错。所以需要增加提前用表名前缀过滤数据的功能。 > My database has thousands of tables. StorageCleanupJob's behavior is to get > the metadata of all tables in the database and then filter table name. > First, the behavior of getting all tables metadata will give metastore great > pressure. Secondly, when getting metadata of more than 1,000 tables at one > time, our metastore would directly return errors. So it needs to add the > function of filtering data with table name prefix in advance -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [kylin] bigxiaochu commented on pull request #1340: KYLIN-4679 StorageCleanupJob clean hive table support hive table prefix
bigxiaochu commented on pull request #1340: URL: https://github.com/apache/kylin/pull/1340#issuecomment-680718484 @hit-lacus add content This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (KYLIN-4719) Refine kylin-defaults.properties for parquet Storage
[
https://issues.apache.org/jira/browse/KYLIN-4719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoxiang Yu updated KYLIN-4719:
Description:
A lot of properties is use less now, we should
# remove some of them, such as flink engine, jdbc source
# add new properties for new implementation
## sparder context
## global dictionary
## other..
{code:java}
//
FLINK ENGINE CONFIGS ###
#
### Flink conf (default is in flink/conf/flink-conf.yaml)
#kylin.engine.flink-conf.jobmanager.heap.size=2G
#kylin.engine.flink-conf.taskmanager.heap.size=4G
#kylin.engine.flink-conf.taskmanager.numberOfTaskSlots=1
#kylin.engine.flink-conf.taskmanager.memory.preallocate=false
#kylin.engine.flink-conf.job.parallelism=1
#kylin.engine.flink-conf.program.enableObjectReuse=false
#kylin.engine.flink-conf.yarn.queue=
#kylin.engine.flink-conf.yarn.nodelabel=
#
QUERY PUSH DOWN ###
#
##kylin.query.pushdown.runner-class-name=org.apache.kylin.query.pushdown.PushDownRunnerSparkImpl
#
##kylin.query.pushdown.update-enabled=false
#
JDBC Data Source
##kylin.source.jdbc.connection-url=
##kylin.source.jdbc.driver=
##kylin.source.jdbc.dialect=
##kylin.source.jdbc.user=
##kylin.source.jdbc.pass=
##kylin.source.jdbc.sqoop-home=
##kylin.source.jdbc.filed-delimiter=|
#
Livy with Kylin
##kylin.engine.livy-conf.livy-enabled=false
##kylin.engine.livy-conf.livy-url=http://LivyHost:8998
##kylin.engine.livy-conf.livy-key.file=hdfs:///path-to-kylin-job-jar
##kylin.engine.livy-conf.livy-arr.jars=hdfs:///path-to-hadoop-dependency-jar
code placeholder
{code}
was:
A lot of properties is use less now, we should remove some of them, such as:
{code:java}
//
FLINK ENGINE CONFIGS ###
#
### Flink conf (default is in flink/conf/flink-conf.yaml)
#kylin.engine.flink-conf.jobmanager.heap.size=2G
#kylin.engine.flink-conf.taskmanager.heap.size=4G
#kylin.engine.flink-conf.taskmanager.numberOfTaskSlots=1
#kylin.engine.flink-conf.taskmanager.memory.preallocate=false
#kylin.engine.flink-conf.job.parallelism=1
#kylin.engine.flink-conf.program.enableObjectReuse=false
#kylin.engine.flink-conf.yarn.queue=
#kylin.engine.flink-conf.yarn.nodelabel=
#
QUERY PUSH DOWN ###
#
##kylin.query.pushdown.runner-class-name=org.apache.kylin.query.pushdown.PushDownRunnerSparkImpl
#
##kylin.query.pushdown.update-enabled=false
#
JDBC Data Source
##kylin.source.jdbc.connection-url=
##kylin.source.jdbc.driver=
##kylin.source.jdbc.dialect=
##kylin.source.jdbc.user=
##kylin.source.jdbc.pass=
##kylin.source.jdbc.sqoop-home=
##kylin.source.jdbc.filed-delimiter=|
#
Livy with Kylin
##kylin.engine.livy-conf.livy-enabled=false
##kylin.engine.livy-conf.livy-url=http://LivyHost:8998
##kylin.engine.livy-conf.livy-key.file=hdfs:///path-to-kylin-job-jar
##kylin.engine.livy-conf.livy-arr.jars=hdfs:///path-to-hadoop-dependency-jar
code placeholder
{code}
> Refine kylin-defaults.properties for parquet Storage
>
>
> Key: KYLIN-4719
> URL: https://issues.apache.org/jira/browse/KYLIN-4719
> Project: Kylin
> Issue Type: Sub-task
> Components: Environment
>Reporter: Xiaoxiang Yu
>Priority: Major
> Fix For: v4.0.0-beta
>
>
> A lot of properties is use less now, we should
> # remove some of them, such as flink engine, jdbc source
> # add new properties for new implementation
> ## sparder context
> ## global dictionary
> ## other..
>
> {code:java}
> //
> FLINK ENGINE CONFIGS ###
> #
> ### Flink conf (default is in flink/conf/flink-conf.yaml)
> #kylin.engine.flink-conf.jobmanager.heap.size=2G
> #kylin.engine.flink-conf.taskmanager.heap.size=4G
> #kylin.engine.flink-conf.taskmanager.numberOfTaskSlots=1
> #kylin.engine.flink-conf.taskmanager.memory.preallocate=false
> #kylin.engine.flink-conf.job.parallelism=1
> #kylin.engine.flink-conf.program.enableObjectReuse=false
> #kylin.engine.flink-conf.yarn.queue=
> #kylin.engine.flink-conf.yarn.nodelabel=
> #
> QUERY PUSH DOWN ###
> #
> ##kylin.query.pushdown.runner-class-name=org.apache.kylin.query.pushdown.PushDownRunnerSparkImpl
> #
> ##kylin.query.pushdown.update-enabled=false
> #
> JDBC Data Source
> ##kylin.source.jdbc.connection-url=
> ##kylin.source.jdbc.driver=
> ##kylin.source.jdbc.dialect=
> ##kylin.source.jdbc.user=
> ##kylin.source.jdbc.pass=
> ##kylin.source.jdbc.sqoop-home=
> ##kylin.source.jdbc.filed-delimiter=|
> #
> Livy with Kylin
> ##kylin.engine.livy-conf.livy-enabled=false
> ##kylin.engine.livy-conf.livy-url=http://LivyHost:8998
> ##kylin.engine.livy-conf.livy-key.file=hdfs:///path-to-kylin-job-jar
> ##kylin.engine.livy-conf.livy-arr.jars=hdfs:///path-to-hadoop-dependency-jar
> code placeholder
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Updated] (KYLIN-4719) Refine kylin-defaults.properties for parquet Storage
[
https://issues.apache.org/jira/browse/KYLIN-4719?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Xiaoxiang Yu updated KYLIN-4719:
Description:
A lot of properties is use less now, we should remove some of them, such as:
{code:java}
//
FLINK ENGINE CONFIGS ###
#
### Flink conf (default is in flink/conf/flink-conf.yaml)
#kylin.engine.flink-conf.jobmanager.heap.size=2G
#kylin.engine.flink-conf.taskmanager.heap.size=4G
#kylin.engine.flink-conf.taskmanager.numberOfTaskSlots=1
#kylin.engine.flink-conf.taskmanager.memory.preallocate=false
#kylin.engine.flink-conf.job.parallelism=1
#kylin.engine.flink-conf.program.enableObjectReuse=false
#kylin.engine.flink-conf.yarn.queue=
#kylin.engine.flink-conf.yarn.nodelabel=
#
QUERY PUSH DOWN ###
#
##kylin.query.pushdown.runner-class-name=org.apache.kylin.query.pushdown.PushDownRunnerSparkImpl
#
##kylin.query.pushdown.update-enabled=false
#
JDBC Data Source
##kylin.source.jdbc.connection-url=
##kylin.source.jdbc.driver=
##kylin.source.jdbc.dialect=
##kylin.source.jdbc.user=
##kylin.source.jdbc.pass=
##kylin.source.jdbc.sqoop-home=
##kylin.source.jdbc.filed-delimiter=|
#
Livy with Kylin
##kylin.engine.livy-conf.livy-enabled=false
##kylin.engine.livy-conf.livy-url=http://LivyHost:8998
##kylin.engine.livy-conf.livy-key.file=hdfs:///path-to-kylin-job-jar
##kylin.engine.livy-conf.livy-arr.jars=hdfs:///path-to-hadoop-dependency-jar
code placeholder
{code}
was:
A lot of properties is use less now, we should remove some of them, such as:
```
FLINK ENGINE CONFIGS ###
#
### Flink conf (default is in flink/conf/flink-conf.yaml)
#kylin.engine.flink-conf.jobmanager.heap.size=2G
#kylin.engine.flink-conf.taskmanager.heap.size=4G
#kylin.engine.flink-conf.taskmanager.numberOfTaskSlots=1
#kylin.engine.flink-conf.taskmanager.memory.preallocate=false
#kylin.engine.flink-conf.job.parallelism=1
#kylin.engine.flink-conf.program.enableObjectReuse=false
#kylin.engine.flink-conf.yarn.queue=
#kylin.engine.flink-conf.yarn.nodelabel=
```
> Refine kylin-defaults.properties for parquet Storage
>
>
> Key: KYLIN-4719
> URL: https://issues.apache.org/jira/browse/KYLIN-4719
> Project: Kylin
> Issue Type: Sub-task
> Components: Environment
>Reporter: Xiaoxiang Yu
>Priority: Major
> Fix For: v4.0.0-beta
>
>
> A lot of properties is use less now, we should remove some of them, such as:
>
> {code:java}
> //
> FLINK ENGINE CONFIGS ###
> #
> ### Flink conf (default is in flink/conf/flink-conf.yaml)
> #kylin.engine.flink-conf.jobmanager.heap.size=2G
> #kylin.engine.flink-conf.taskmanager.heap.size=4G
> #kylin.engine.flink-conf.taskmanager.numberOfTaskSlots=1
> #kylin.engine.flink-conf.taskmanager.memory.preallocate=false
> #kylin.engine.flink-conf.job.parallelism=1
> #kylin.engine.flink-conf.program.enableObjectReuse=false
> #kylin.engine.flink-conf.yarn.queue=
> #kylin.engine.flink-conf.yarn.nodelabel=
> #
> QUERY PUSH DOWN ###
> #
> ##kylin.query.pushdown.runner-class-name=org.apache.kylin.query.pushdown.PushDownRunnerSparkImpl
> #
> ##kylin.query.pushdown.update-enabled=false
> #
> JDBC Data Source
> ##kylin.source.jdbc.connection-url=
> ##kylin.source.jdbc.driver=
> ##kylin.source.jdbc.dialect=
> ##kylin.source.jdbc.user=
> ##kylin.source.jdbc.pass=
> ##kylin.source.jdbc.sqoop-home=
> ##kylin.source.jdbc.filed-delimiter=|
> #
> Livy with Kylin
> ##kylin.engine.livy-conf.livy-enabled=false
> ##kylin.engine.livy-conf.livy-url=http://LivyHost:8998
> ##kylin.engine.livy-conf.livy-key.file=hdfs:///path-to-kylin-job-jar
> ##kylin.engine.livy-conf.livy-arr.jars=hdfs:///path-to-hadoop-dependency-jar
> code placeholder
> {code}
>
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Created] (KYLIN-4719) Refine kylin-defaults.properties for parquet Storage
Xiaoxiang Yu created KYLIN-4719: --- Summary: Refine kylin-defaults.properties for parquet Storage Key: KYLIN-4719 URL: https://issues.apache.org/jira/browse/KYLIN-4719 Project: Kylin Issue Type: Sub-task Components: Environment Reporter: Xiaoxiang Yu Fix For: v4.0.0-beta A lot of properties is use less now, we should remove some of them, such as: ``` FLINK ENGINE CONFIGS ### # ### Flink conf (default is in flink/conf/flink-conf.yaml) #kylin.engine.flink-conf.jobmanager.heap.size=2G #kylin.engine.flink-conf.taskmanager.heap.size=4G #kylin.engine.flink-conf.taskmanager.numberOfTaskSlots=1 #kylin.engine.flink-conf.taskmanager.memory.preallocate=false #kylin.engine.flink-conf.job.parallelism=1 #kylin.engine.flink-conf.program.enableObjectReuse=false #kylin.engine.flink-conf.yarn.queue= #kylin.engine.flink-conf.yarn.nodelabel= ``` -- This message was sent by Atlassian Jira (v8.3.4#803005)
