[jira] [Commented] (KYLIN-3632) Add configuration that can switch on/off preparedStatement cache in Kylin server
[ https://issues.apache.org/jira/browse/KYLIN-3632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649757#comment-16649757 ] ASF GitHub Bot commented on KYLIN-3632: --- codecov-io commented on issue #290: KYLIN-3632 Add configuration that can switch on/off preparedStatement cache URL: https://github.com/apache/kylin/pull/290#issuecomment-429717664 # [Codecov](https://codecov.io/gh/apache/kylin/pull/290?src=pr=h1) Report > Merging [#290](https://codecov.io/gh/apache/kylin/pull/290?src=pr=desc) into [master](https://codecov.io/gh/apache/kylin/commit/5f6007ff2b5aead4781c876fa3203dd67188d90b?src=pr=desc) will **decrease** coverage by `<.01%`. > The diff coverage is `0%`. [](https://codecov.io/gh/apache/kylin/pull/290?src=pr=tree) ```diff @@ Coverage Diff @@ ## master #290 +/- ## - Coverage 21.29% 21.29% -0.01% + Complexity 4442 -2 Files 1087 1087 Lines 6998369984 +1 Branches 1010910109 - Hits 1490314902 -1 - Misses5367653677 +1 - Partials 1404 1405 +1 ``` | [Impacted Files](https://codecov.io/gh/apache/kylin/pull/290?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...va/org/apache/kylin/rest/service/QueryService.java](https://codecov.io/gh/apache/kylin/pull/290/diff?src=pr=tree#diff-c2VydmVyLWJhc2Uvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3Jlc3Qvc2VydmljZS9RdWVyeVNlcnZpY2UuamF2YQ==) | `0% <0%> (ø)` | `0 <0> (ø)` | :arrow_down: | | [.../java/org/apache/kylin/common/KylinConfigBase.java](https://codecov.io/gh/apache/kylin/pull/290/diff?src=pr=tree#diff-Y29yZS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2NvbW1vbi9LeWxpbkNvbmZpZ0Jhc2UuamF2YQ==) | `14.66% <0%> (-0.03%)` | `36 <0> (ø)` | | | [...he/kylin/dict/lookup/cache/RocksDBLookupTable.java](https://codecov.io/gh/apache/kylin/pull/290/diff?src=pr=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L2xvb2t1cC9jYWNoZS9Sb2Nrc0RCTG9va3VwVGFibGUuamF2YQ==) | `72.97% <0%> (-5.41%)` | `6% <0%> (-1%)` | | | [...a/org/apache/kylin/dict/Number2BytesConverter.java](https://codecov.io/gh/apache/kylin/pull/290/diff?src=pr=tree#diff-Y29yZS1kaWN0aW9uYXJ5L3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9kaWN0L051bWJlcjJCeXRlc0NvbnZlcnRlci5qYXZh) | `81.74% <0%> (-0.8%)` | `17% <0%> (-1%)` | | | [...rg/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://codecov.io/gh/apache/kylin/pull/290/diff?src=pr=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL2lubWVtY3ViaW5nL01lbURpc2tTdG9yZS5qYXZh) | `70.21% <0%> (+0.6%)` | `7% <0%> (ø)` | :arrow_down: | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/290?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/290?src=pr=footer). Last update [5f6007f...3fcb1d5](https://codecov.io/gh/apache/kylin/pull/290?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add configuration that can switch on/off preparedStatement cache in Kylin > server > > > Key: KYLIN-3632 > URL: https://issues.apache.org/jira/browse/KYLIN-3632 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Reporter: Ma Gang >Assignee: Ma Gang >Priority: Minor > Fix For: v2.5.1 > > > From Kylin 2.5 we introduce preparedStatement cache feature, it can be turn > on/off for each request by adding a new field "enableStatementCache" in the > query request, by default it is on. We need to add a switch in server level, > the preparedStatement cache can take effective only when the switch is on, by > default, it will be set to false. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3632) Add configuration that can switch on/off preparedStatement cache in Kylin server
[ https://issues.apache.org/jira/browse/KYLIN-3632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649745#comment-16649745 ] ASF GitHub Bot commented on KYLIN-3632: --- allenma opened a new pull request #290: KYLIN-3632 Add configuration that can switch on/off preparedStatement cache URL: https://github.com/apache/kylin/pull/290 Add configuration that can switch on/off preparedStatement cache in kylin server This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add configuration that can switch on/off preparedStatement cache in Kylin > server > > > Key: KYLIN-3632 > URL: https://issues.apache.org/jira/browse/KYLIN-3632 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Reporter: Ma Gang >Assignee: Ma Gang >Priority: Minor > Fix For: v2.5.1 > > > From Kylin 2.5 we introduce preparedStatement cache feature, it can be turn > on/off for each request by adding a new field "enableStatementCache" in the > query request, by default it is on. We need to add a switch in server level, > the preparedStatement cache can take effective only when the switch is on, by > default, it will be set to false. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3632) Add configuration that can switch on/off preparedStatement cache in Kylin server
[ https://issues.apache.org/jira/browse/KYLIN-3632?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649746#comment-16649746 ] ASF GitHub Bot commented on KYLIN-3632: --- asfgit commented on issue #290: KYLIN-3632 Add configuration that can switch on/off preparedStatement cache URL: https://github.com/apache/kylin/pull/290#issuecomment-429714335 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Add configuration that can switch on/off preparedStatement cache in Kylin > server > > > Key: KYLIN-3632 > URL: https://issues.apache.org/jira/browse/KYLIN-3632 > Project: Kylin > Issue Type: Improvement > Components: Query Engine >Reporter: Ma Gang >Assignee: Ma Gang >Priority: Minor > Fix For: v2.5.1 > > > From Kylin 2.5 we introduce preparedStatement cache feature, it can be turn > on/off for each request by adding a new field "enableStatementCache" in the > query request, by default it is on. We need to add a switch in server level, > the preparedStatement cache can take effective only when the switch is on, by > default, it will be set to false. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3601) The max connection number generated by the PreparedContextPool is inconsistent with the configuration.
[
https://issues.apache.org/jira/browse/KYLIN-3601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649744#comment-16649744
]
huaicui edited comment on KYLIN-3601 at 10/15/18 5:34 AM:
--
We use Gatling to loading test;
This is our test script:
import java.util.concurrent.TimeUnit
import io.gatling.core.Predef._
import io.gatling.http.Predef._
import scala.concurrent.duration._
class KylinSimulation2 extends Simulation {
val httpConf = http
.acceptHeader("application/json")
.contentTypeHeader("application/json")
val headers_json = Map(
"Content-Type" -> "application/json",
"Authorization" -> "Basic QURNSU46S1lMSU4="
)
val scn2 = scenario("KylinServer6").during(60) { // 60 means keep sending
requests for about 60s
exec(
http("KylinQuery4")
.post("http://sap-datanode5:7070/kylin/api/query;)
.headers(headers_json)
.body(StringBody("{ \"sql\":\"select CLIENTAGENT, STARTDAY, COUNT(*)
TOTAL_NUMBER from REPORT_VIEW where ID = ? group by STARTDAY, CLIENTAGENT\",
\"project\":\"Report\", \"params\":[
{\"className\":\"java.lang.String\",\"value\":\"11001\"}
] }")).asJSON
)
}
setUp(
scn2.inject(atOnceUsers(150)) // 150 users(threads)
).protocols(httpConf)
}
We use 150 threads to loading test, kylin.query.statement-cache-max-num-per-key
is 200; We found the number of PrepareStatement instance would be more and more
like the attaches(CreatedCount will be larger and larger); No matter how many
concurrent that will trigger this issue;
was (Author: kofiori):
We use Gatling to loading test;
This is our test script:
import java.util.concurrent.TimeUnit
import io.gatling.core.Predef._
import io.gatling.http.Predef._
import scala.concurrent.duration._
class KylinSimulation2 extends Simulation {
val httpConf = http
.acceptHeader("application/json")
.contentTypeHeader("application/json")
val headers_json = Map(
"Content-Type" -> "application/json",
"Authorization" -> "Basic QURNSU46S1lMSU4="
)
val scn2 = scenario("KylinServer6").during(60) {// 60 means keep sending
requests for about 60s
exec(
http("KylinQuery4")
.post("http://sap-datanode5:7070/kylin/api/query;)
.headers(headers_json)
.body(StringBody("{ \"sql\":\"select CLIENTAGENT, STARTDAY, COUNT(*)
TOTAL_NUMBER from XXRPT_HGSMEETINGUSERREPORT_VIEW where SITEID = ? group by
STARTDAY, CLIENTAGENT\", \"project\":\"CCA_REPORT\",
\"params\":[{\"className\":\"java.lang.String\",\"value\":\"1000182\"}]
}")).asJSON
)
}
setUp(
scn2.inject(atOnceUsers(150))// 150 users(threads)
).protocols(httpConf)
}
We use 150 threads to loading test, kylin.query.statement-cache-max-num-per-key
is 200; We found the number of PrepareStatement instance would be more and more
like the attaches(CreatedCount will be larger and larger); No matter how many
concurrent that will trigger this issue;
> The max connection number generated by the PreparedContextPool is
> inconsistent with the configuration.
> --
>
> Key: KYLIN-3601
> URL: https://issues.apache.org/jira/browse/KYLIN-3601
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v2.5.0
>Reporter: huaicui
>Priority: Major
> Attachments: FirstResponseDistribute.jpg,
> SixthResponseDistribute.jpg, image-2018-09-28-15-14-00-288.png, image.png
>
>
> 因为并发性能不够，使用了magang提供的PrepareStatement方法进行测试。性能有所有提高，但随着测试次数的增加，吞吐率会越来越低而且数据超时也越来越多。经过修改代码在queryAndUpdateCache最后返回前加入日志打印：logger.debug("BorrowedCount:"+preparedContextPool.getBorrowedCount()
> +",DestroyedCount:"+preparedContextPool.getDestroyedCount()
> +",CreatedCount:"+preparedContextPool.getCreatedCount()
> +",ReturnedCount:"+preparedContextPool.getReturnedCount()
> 同时配置文件加入该配置：
> kylin.query.statement-cache-max-num-per-key=200
>
>
> 日志显示，当同一sql并发一段时间后，PreparedContextPool创建了越来越多PrepareStatement，并没有进行阻塞后续来的请求。
> !image-2018-09-28-15-14-00-288.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3601) The max connection number generated by the PreparedContextPool is inconsistent with the configuration.
[
https://issues.apache.org/jira/browse/KYLIN-3601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649744#comment-16649744
]
huaicui commented on KYLIN-3601:
We use Gatling to loading test;
This is our test script:
import java.util.concurrent.TimeUnit
import io.gatling.core.Predef._
import io.gatling.http.Predef._
import scala.concurrent.duration._
class KylinSimulation2 extends Simulation {
val httpConf = http
.acceptHeader("application/json")
.contentTypeHeader("application/json")
val headers_json = Map(
"Content-Type" -> "application/json",
"Authorization" -> "Basic QURNSU46S1lMSU4="
)
val scn2 = scenario("KylinServer6").during(60) {// 60 means keep sending
requests for about 60s
exec(
http("KylinQuery4")
.post("http://sap-datanode5:7070/kylin/api/query;)
.headers(headers_json)
.body(StringBody("{ \"sql\":\"select CLIENTAGENT, STARTDAY, COUNT(*)
TOTAL_NUMBER from XXRPT_HGSMEETINGUSERREPORT_VIEW where SITEID = ? group by
STARTDAY, CLIENTAGENT\", \"project\":\"CCA_REPORT\",
\"params\":[{\"className\":\"java.lang.String\",\"value\":\"1000182\"}]
}")).asJSON
)
}
setUp(
scn2.inject(atOnceUsers(150))// 150 users(threads)
).protocols(httpConf)
}
We use 150 threads to loading test, kylin.query.statement-cache-max-num-per-key
is 200; We found the number of PrepareStatement instance would be more and more
like the attaches(CreatedCount will be larger and larger); No matter how many
concurrent that will trigger this issue;
> The max connection number generated by the PreparedContextPool is
> inconsistent with the configuration.
> --
>
> Key: KYLIN-3601
> URL: https://issues.apache.org/jira/browse/KYLIN-3601
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v2.5.0
>Reporter: huaicui
>Priority: Major
> Attachments: FirstResponseDistribute.jpg,
> SixthResponseDistribute.jpg, image-2018-09-28-15-14-00-288.png, image.png
>
>
> 因为并发性能不够，使用了magang提供的PrepareStatement方法进行测试。性能有所有提高，但随着测试次数的增加，吞吐率会越来越低而且数据超时也越来越多。经过修改代码在queryAndUpdateCache最后返回前加入日志打印：logger.debug("BorrowedCount:"+preparedContextPool.getBorrowedCount()
> +",DestroyedCount:"+preparedContextPool.getDestroyedCount()
> +",CreatedCount:"+preparedContextPool.getCreatedCount()
> +",ReturnedCount:"+preparedContextPool.getReturnedCount()
> 同时配置文件加入该配置：
> kylin.query.statement-cache-max-num-per-key=200
>
>
> 日志显示，当同一sql并发一段时间后，PreparedContextPool创建了越来越多PrepareStatement，并没有进行阻塞后续来的请求。
> !image-2018-09-28-15-14-00-288.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3632) Add configuration that can switch on/off preparedStatement cache in Kylin server
Ma Gang created KYLIN-3632: -- Summary: Add configuration that can switch on/off preparedStatement cache in Kylin server Key: KYLIN-3632 URL: https://issues.apache.org/jira/browse/KYLIN-3632 Project: Kylin Issue Type: Improvement Components: Query Engine Reporter: Ma Gang Assignee: Ma Gang Fix For: v2.5.1 From Kylin 2.5 we introduce preparedStatement cache feature, it can be turn on/off for each request by adding a new field "enableStatementCache" in the query request, by default it is on. We need to add a switch in server level, the preparedStatement cache can take effective only when the switch is on, by default, it will be set to false. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3601) The max connection number generated by the PreparedContextPool is inconsistent with the configuration.
[
https://issues.apache.org/jira/browse/KYLIN-3601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649722#comment-16649722
]
Ma Gang commented on KYLIN-3601:
Looks weird, and should not happen theoretically. [~kofiori] what performance
test tool you use to do the test? Is one concurrent client only send the next
request after it get response for the previous request?
And you should increase the concurrency time by time to do the test, for
example: 1, 5, 10, 20, 50, 100 concurrent clients, and check the issue happens
in which concurrency level.
For borrow fail case, you may add some log after throw NoSuchElementException
when borrow prepareContext from pool.
Also the bottleneck may happen in Hbase side or even in tomcat, could you check
the query log for the timeout query, what makes the query slow? you may dump
some stacktrace randomly when lots of query failed.
> The max connection number generated by the PreparedContextPool is
> inconsistent with the configuration.
> --
>
> Key: KYLIN-3601
> URL: https://issues.apache.org/jira/browse/KYLIN-3601
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v2.5.0
>Reporter: huaicui
>Priority: Major
> Attachments: FirstResponseDistribute.jpg,
> SixthResponseDistribute.jpg, image-2018-09-28-15-14-00-288.png, image.png
>
>
> 因为并发性能不够，使用了magang提供的PrepareStatement方法进行测试。性能有所有提高，但随着测试次数的增加，吞吐率会越来越低而且数据超时也越来越多。经过修改代码在queryAndUpdateCache最后返回前加入日志打印：logger.debug("BorrowedCount:"+preparedContextPool.getBorrowedCount()
> +",DestroyedCount:"+preparedContextPool.getDestroyedCount()
> +",CreatedCount:"+preparedContextPool.getCreatedCount()
> +",ReturnedCount:"+preparedContextPool.getReturnedCount()
> 同时配置文件加入该配置：
> kylin.query.statement-cache-max-num-per-key=200
>
>
> 日志显示，当同一sql并发一段时间后，PreparedContextPool创建了越来越多PrepareStatement，并没有进行阻塞后续来的请求。
> !image-2018-09-28-15-14-00-288.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3601) The max connection number generated by the PreparedContextPool is inconsistent with the configuration.
[
https://issues.apache.org/jira/browse/KYLIN-3601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649694#comment-16649694
]
huaicui commented on KYLIN-3601:
No, We have not solved this issue.
> The max connection number generated by the PreparedContextPool is
> inconsistent with the configuration.
> --
>
> Key: KYLIN-3601
> URL: https://issues.apache.org/jira/browse/KYLIN-3601
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v2.5.0
>Reporter: huaicui
>Priority: Major
> Attachments: FirstResponseDistribute.jpg,
> SixthResponseDistribute.jpg, image-2018-09-28-15-14-00-288.png, image.png
>
>
> 因为并发性能不够，使用了magang提供的PrepareStatement方法进行测试。性能有所有提高，但随着测试次数的增加，吞吐率会越来越低而且数据超时也越来越多。经过修改代码在queryAndUpdateCache最后返回前加入日志打印：logger.debug("BorrowedCount:"+preparedContextPool.getBorrowedCount()
> +",DestroyedCount:"+preparedContextPool.getDestroyedCount()
> +",CreatedCount:"+preparedContextPool.getCreatedCount()
> +",ReturnedCount:"+preparedContextPool.getReturnedCount()
> 同时配置文件加入该配置：
> kylin.query.statement-cache-max-num-per-key=200
>
>
> 日志显示，当同一sql并发一段时间后，PreparedContextPool创建了越来越多PrepareStatement，并没有进行阻塞后续来的请求。
> !image-2018-09-28-15-14-00-288.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (KYLIN-3630) Remove unused fields in the implementations of MeasureType
[
https://issues.apache.org/jira/browse/KYLIN-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jiatao.tao closed KYLIN-3630.
-
thanks shaofeng.
> Remove unused fields in the implementations of MeasureType
> --
>
> Key: KYLIN-3630
> URL: https://issues.apache.org/jira/browse/KYLIN-3630
> Project: Kylin
> Issue Type: Improvement
>Reporter: jiatao.tao
>Assignee: jiatao.tao
>Priority: Minor
> Fix For: v2.6.0
>
>
> In code RawMeasureType#RawMeasureType, we can see that both funcName and
> dataType are use, and this can be remove cuz there's no constraint on its
> constructor.
> {code:java}
> @SuppressWarnings("unused")
> private final DataType dataType;
> public RawMeasureType(String funcName, DataType dataType) {
> this.dataType = dataType;
> }
> {code}
> And after we done that, we can more easily test MeasureType without passing
> in meaningless args(AggregatorMemEstimateTest#112).
> Like codes in org.apache.kylin.measure.dim.DimCountDistinctMeasureType, other
> implementations can be cleaner.
> {code:java}
> @Override
> public MeasureType createMeasureType(String funcName,
> DataType dataType) {
> return new DimCountDistinctMeasureType();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Resolved] (KYLIN-3630) Remove unused fields in the implementations of MeasureType
[
https://issues.apache.org/jira/browse/KYLIN-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jiatao.tao resolved KYLIN-3630.
---
Resolution: Fixed
> Remove unused fields in the implementations of MeasureType
> --
>
> Key: KYLIN-3630
> URL: https://issues.apache.org/jira/browse/KYLIN-3630
> Project: Kylin
> Issue Type: Improvement
>Reporter: jiatao.tao
>Assignee: jiatao.tao
>Priority: Minor
> Fix For: v2.6.0
>
>
> In code RawMeasureType#RawMeasureType, we can see that both funcName and
> dataType are use, and this can be remove cuz there's no constraint on its
> constructor.
> {code:java}
> @SuppressWarnings("unused")
> private final DataType dataType;
> public RawMeasureType(String funcName, DataType dataType) {
> this.dataType = dataType;
> }
> {code}
> And after we done that, we can more easily test MeasureType without passing
> in meaningless args(AggregatorMemEstimateTest#112).
> Like codes in org.apache.kylin.measure.dim.DimCountDistinctMeasureType, other
> implementations can be cleaner.
> {code:java}
> @Override
> public MeasureType createMeasureType(String funcName,
> DataType dataType) {
> return new DimCountDistinctMeasureType();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3601) The max connection number generated by the PreparedContextPool is inconsistent with the configuration.
[
https://issues.apache.org/jira/browse/KYLIN-3601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649644#comment-16649644
]
huaicui edited comment on KYLIN-3601 at 10/15/18 2:11 AM:
--
Concurrent clients less than 150, but
kylin.query.statement-cache-max-num-per-key is 200. So I don't understand this
issue; Why would it be triggered (borrow fail)？
was (Author: kofiori):
Concurrent clients less than 150, but
kylin.query.statement-cache-max-num-per-key is 200. So I don't understand this
issue;
> The max connection number generated by the PreparedContextPool is
> inconsistent with the configuration.
> --
>
> Key: KYLIN-3601
> URL: https://issues.apache.org/jira/browse/KYLIN-3601
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v2.5.0
>Reporter: huaicui
>Priority: Major
> Attachments: FirstResponseDistribute.jpg,
> SixthResponseDistribute.jpg, image-2018-09-28-15-14-00-288.png, image.png
>
>
> 因为并发性能不够，使用了magang提供的PrepareStatement方法进行测试。性能有所有提高，但随着测试次数的增加，吞吐率会越来越低而且数据超时也越来越多。经过修改代码在queryAndUpdateCache最后返回前加入日志打印：logger.debug("BorrowedCount:"+preparedContextPool.getBorrowedCount()
> +",DestroyedCount:"+preparedContextPool.getDestroyedCount()
> +",CreatedCount:"+preparedContextPool.getCreatedCount()
> +",ReturnedCount:"+preparedContextPool.getReturnedCount()
> 同时配置文件加入该配置：
> kylin.query.statement-cache-max-num-per-key=200
>
>
> 日志显示，当同一sql并发一段时间后，PreparedContextPool创建了越来越多PrepareStatement，并没有进行阻塞后续来的请求。
> !image-2018-09-28-15-14-00-288.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3601) The max connection number generated by the PreparedContextPool is inconsistent with the configuration.
[
https://issues.apache.org/jira/browse/KYLIN-3601?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649644#comment-16649644
]
huaicui commented on KYLIN-3601:
Concurrent clients less than 150, but
kylin.query.statement-cache-max-num-per-key is 200. So I don't understand this
issue;
> The max connection number generated by the PreparedContextPool is
> inconsistent with the configuration.
> --
>
> Key: KYLIN-3601
> URL: https://issues.apache.org/jira/browse/KYLIN-3601
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v2.5.0
>Reporter: huaicui
>Priority: Major
> Attachments: FirstResponseDistribute.jpg,
> SixthResponseDistribute.jpg, image-2018-09-28-15-14-00-288.png, image.png
>
>
> 因为并发性能不够，使用了magang提供的PrepareStatement方法进行测试。性能有所有提高，但随着测试次数的增加，吞吐率会越来越低而且数据超时也越来越多。经过修改代码在queryAndUpdateCache最后返回前加入日志打印：logger.debug("BorrowedCount:"+preparedContextPool.getBorrowedCount()
> +",DestroyedCount:"+preparedContextPool.getDestroyedCount()
> +",CreatedCount:"+preparedContextPool.getCreatedCount()
> +",ReturnedCount:"+preparedContextPool.getReturnedCount()
> 同时配置文件加入该配置：
> kylin.query.statement-cache-max-num-per-key=200
>
>
> 日志显示，当同一sql并发一段时间后，PreparedContextPool创建了越来越多PrepareStatement，并没有进行阻塞后续来的请求。
> !image-2018-09-28-15-14-00-288.png!
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3627) calcite-core-1.13.0-kylin-r4
[ https://issues.apache.org/jira/browse/KYLIN-3627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649633#comment-16649633 ] Shaofeng SHI commented on KYLIN-3627: - Not receiving your email in the mail group. It is in [https://github.com/Kyligence/calcite,] you can browser a specific version branch. Welcome to join Kylin mail group. > calcite-core-1.13.0-kylin-r4 > > > Key: KYLIN-3627 > URL: https://issues.apache.org/jira/browse/KYLIN-3627 > Project: Kylin > Issue Type: Bug >Reporter: liuzhixin >Priority: Major > > where to find calcite-core version 1.13.0-kylin-r4 -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3597) Fix sonar reported static code issues
[ https://issues.apache.org/jira/browse/KYLIN-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649621#comment-16649621 ] ASF subversion and git services commented on KYLIN-3597: Commit 5f6007ff2b5aead4781c876fa3203dd67188d90b in kylin's branch refs/heads/master from shaofengshi [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=5f6007f ] KYLIN-3597 improve code smell > Fix sonar reported static code issues > - > > Key: KYLIN-3597 > URL: https://issues.apache.org/jira/browse/KYLIN-3597 > Project: Kylin > Issue Type: Improvement > Components: Others >Reporter: Shaofeng SHI >Priority: Major > Fix For: v2.6.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KYLIN-3619) Some job won't clean up temp directory after finished
[ https://issues.apache.org/jira/browse/KYLIN-3619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI reassigned KYLIN-3619: --- Assignee: jiaoenwei > Some job won't clean up temp directory after finished > - > > Key: KYLIN-3619 > URL: https://issues.apache.org/jira/browse/KYLIN-3619 > Project: Kylin > Issue Type: Bug > Components: Job Engine, REST Service >Reporter: jiaoenwei >Assignee: jiaoenwei >Priority: Major > Fix For: v2.6.0 > > > * org.apache.kylin.source.hive.cardinality.HiveColumnCardinalityJob > * org.apache.kylin.engine.spark.SparkExecutable > These two job did not remove kylin_meta_job temp folder. > After download kybot diag package did not remove temp folder. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3619) Some job won't clean up temp directory after finished
[ https://issues.apache.org/jira/browse/KYLIN-3619?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-3619: Fix Version/s: v2.6.0 > Some job won't clean up temp directory after finished > - > > Key: KYLIN-3619 > URL: https://issues.apache.org/jira/browse/KYLIN-3619 > Project: Kylin > Issue Type: Bug > Components: Job Engine, REST Service >Reporter: jiaoenwei >Assignee: jiaoenwei >Priority: Major > Fix For: v2.6.0 > > > * org.apache.kylin.source.hive.cardinality.HiveColumnCardinalityJob > * org.apache.kylin.engine.spark.SparkExecutable > These two job did not remove kylin_meta_job temp folder. > After download kybot diag package did not remove temp folder. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3630) Remove unused fields in the implementations of MeasureType
[
https://issues.apache.org/jira/browse/KYLIN-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shaofeng SHI updated KYLIN-3630:
Fix Version/s: v2.6.0
Summary: Remove unused fields in the implementations of MeasureType
(was: remove unused fields in the implementations of MeasureType)
> Remove unused fields in the implementations of MeasureType
> --
>
> Key: KYLIN-3630
> URL: https://issues.apache.org/jira/browse/KYLIN-3630
> Project: Kylin
> Issue Type: Improvement
>Reporter: jiatao.tao
>Assignee: jiatao.tao
>Priority: Minor
> Fix For: v2.6.0
>
>
> In code RawMeasureType#RawMeasureType, we can see that both funcName and
> dataType are use, and this can be remove cuz there's no constraint on its
> constructor.
> {code:java}
> @SuppressWarnings("unused")
> private final DataType dataType;
> public RawMeasureType(String funcName, DataType dataType) {
> this.dataType = dataType;
> }
> {code}
> And after we done that, we can more easily test MeasureType without passing
> in meaningless args(AggregatorMemEstimateTest#112).
> Like codes in org.apache.kylin.measure.dim.DimCountDistinctMeasureType, other
> implementations can be cleaner.
> {code:java}
> @Override
> public MeasureType createMeasureType(String funcName,
> DataType dataType) {
> return new DimCountDistinctMeasureType();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Assigned] (KYLIN-3514) Support AUC UDAF
[ https://issues.apache.org/jira/browse/KYLIN-3514?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI reassigned KYLIN-3514: --- Assignee: JiJun Tang > Support AUC UDAF > > > Key: KYLIN-3514 > URL: https://issues.apache.org/jira/browse/KYLIN-3514 > Project: Kylin > Issue Type: New Feature >Reporter: JiJun Tang >Assignee: JiJun Tang >Priority: Minor > Fix For: v2.6.0 > > > auc aggregation measure function: > Compute auc result by aggregate two list (truth and prd) > > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3617) Reduce number of visiting metastore for job scheduler
[
https://issues.apache.org/jira/browse/KYLIN-3617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649603#comment-16649603
]
ASF GitHub Bot commented on KYLIN-3617:
---
shaofengshi closed pull request #286: KYLIN-3617 Use job's cache in job
scheduler
URL: https://github.com/apache/kylin/pull/286
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git a/core-job/src/main/java/org/apache/kylin/job/dao/ExecutableDao.java
b/core-job/src/main/java/org/apache/kylin/job/dao/ExecutableDao.java
index 0cc6c8e5fb..8352005234 100644
--- a/core-job/src/main/java/org/apache/kylin/job/dao/ExecutableDao.java
+++ b/core-job/src/main/java/org/apache/kylin/job/dao/ExecutableDao.java
@@ -23,6 +23,7 @@
import java.util.Collections;
import java.util.List;
import java.util.NavigableSet;
+import java.util.Set;
import org.apache.kylin.common.KylinConfig;
import org.apache.kylin.common.persistence.JsonSerializer;
@@ -241,6 +242,10 @@ private long writeJobOutputResource(String path,
ExecutableOutputPO output) thro
}
}
+public ExecutableOutputPO getJobOutputDigest(String uuid) {
+return executableOutputDigestMap.get(uuid);
+}
+
public List getJobOutputDigests(long timeStart, long
timeEndExclusive) {
List jobOutputDigests = Lists.newArrayList();
for (ExecutableOutputPO po : executableOutputDigestMap.values()) {
@@ -268,6 +273,10 @@ private long writeJobOutputResource(String path,
ExecutableOutputPO output) thro
}
}
+public ExecutablePO getJobDigest(String uuid) {
+return executableDigestMap.get(uuid);
+}
+
public List getJobDigests(long timeStart, long
timeEndExclusive) {
List jobDigests = Lists.newArrayList();
for (ExecutablePO po : executableDigestMap.values()) {
@@ -277,6 +286,11 @@ private long writeJobOutputResource(String path,
ExecutableOutputPO output) thro
return jobDigests;
}
+public List getJobIdsInCache() {
+Set idSet = executableDigestMap.keySet();
+return Lists.newArrayList(idSet);
+}
+
public List getJobIds() throws PersistentException {
try {
NavigableSet resources =
store.listResources(ResourceStore.EXECUTE_RESOURCE_ROOT);
@@ -391,4 +405,13 @@ public void deleteJobOutput(String uuid) throws
PersistentException {
throw new PersistentException(e);
}
}
+
+public void reloadAll() throws IOException {
+try (AutoReadWriteLock.AutoLock lock =
executableDigestMapLock.lockForWrite()) {
+executableDigestCrud.reloadAll();
+}
+try (AutoReadWriteLock.AutoLock lock =
executableOutputDigestMapLock.lockForWrite()) {
+executableOutputDigestCrud.reloadAll();
+}
+}
}
diff --git
a/core-job/src/main/java/org/apache/kylin/job/execution/ExecutableManager.java
b/core-job/src/main/java/org/apache/kylin/job/execution/ExecutableManager.java
index 5cc8a0f7d7..b866618e05 100644
---
a/core-job/src/main/java/org/apache/kylin/job/execution/ExecutableManager.java
+++
b/core-job/src/main/java/org/apache/kylin/job/execution/ExecutableManager.java
@@ -155,6 +155,10 @@ public AbstractExecutable getJob(String uuid) {
}
}
+public AbstractExecutable getJobDigest(String uuid) {
+return parseTo(executableDao.getJobDigest(uuid));
+}
+
public Output getOutput(String uuid) {
try {
final ExecutableOutputPO jobOutput =
executableDao.getJobOutput(uuid);
@@ -166,6 +170,12 @@ public Output getOutput(String uuid) {
}
}
+public Output getOutputDigest(String uuid) {
+final ExecutableOutputPO jobOutput =
executableDao.getJobOutputDigest(uuid);
+Preconditions.checkArgument(jobOutput != null, "there is no related
output for job id:" + uuid);
+return parseOutput(jobOutput);
+}
+
private DefaultOutput parseOutput(ExecutableOutputPO jobOutput) {
final DefaultOutput result = new DefaultOutput();
result.setExtra(jobOutput.getInfo());
@@ -286,6 +296,10 @@ public void updateAllRunningJobsToError() {
}
}
+public List getAllJobIdsInCache() {
+return executableDao.getJobIdsInCache();
+}
+
public void resumeAllRunningJobs() {
try {
final List jobOutputs =
executableDao.getJobOutputs();
@@ -439,6 +453,10 @@ public void updateJobOutput(String jobId, ExecutableState
newStatus, Map
newConcurrentMap(), jobEngineConfig.getConfig());
logger.info("Staring resume all running jobs.");
+ExecutableManager executableManager = getExecutableManager();
executableManager.resumeAllRunningJobs();
[jira] [Commented] (KYLIN-3617) Reduce number of visiting metastore for job scheduler
[ https://issues.apache.org/jira/browse/KYLIN-3617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649604#comment-16649604 ] ASF subversion and git services commented on KYLIN-3617: Commit 52307bab46db6240169492f240a8b69bde5110d8 in kylin's branch refs/heads/master from [~nichunen] [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=52307ba ] KYLIN-3617 Use job's cache in job scheduler > Reduce number of visiting metastore for job scheduler > - > > Key: KYLIN-3617 > URL: https://issues.apache.org/jira/browse/KYLIN-3617 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Affects Versions: v2.4.1 >Reporter: nichunen >Assignee: nichunen >Priority: Major > Fix For: v2.6.0 > > > For KYLIN-3470 introduced cache for jobs' metadata, it's also can be used in > job scheduler to reduce the pressure on metastore -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3310) Use lint for maven-compiler-plugin
[
https://issues.apache.org/jira/browse/KYLIN-3310?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Ted Yu updated KYLIN-3310:
--
Description:
lint helps identify structural problems.
We should enable lint for maven-compiler-plugin
{code}
maven-compiler-plugin
${maven-compiler-plugin.version}
1.8
1.8
-Xlint:all
${compiler.error.flag}
-Xlint:-options
-Xlint:-cast
-Xlint:-deprecation
-Xlint:-processing
-Xlint:-rawtypes
-Xlint:-serial
-Xlint:-try
-Xlint:-unchecked
-Xlint:-varargs
true
false
{code}
was:
lint helps identify structural problems.
We should enable lint for maven-compiler-plugin
{code}
maven-compiler-plugin
${maven-compiler-plugin.version}
1.8
1.8
-Xlint:all
${compiler.error.flag}
-Xlint:-options
-Xlint:-cast
-Xlint:-deprecation
-Xlint:-processing
-Xlint:-rawtypes
-Xlint:-serial
-Xlint:-try
-Xlint:-unchecked
-Xlint:-varargs
true
false
{code}
> Use lint for maven-compiler-plugin
> --
>
> Key: KYLIN-3310
> URL: https://issues.apache.org/jira/browse/KYLIN-3310
> Project: Kylin
> Issue Type: Improvement
> Components: Tools, Build and Test
>Reporter: Ted Yu
>Assignee: jiatao.tao
>Priority: Major
> Fix For: v2.6.0
>
>
> lint helps identify structural problems.
> We should enable lint for maven-compiler-plugin
> {code}
> maven-compiler-plugin
> ${maven-compiler-plugin.version}
>
> 1.8
> 1.8
>
> -Xlint:all
> ${compiler.error.flag}
>
> -Xlint:-options
>
> -Xlint:-cast
> -Xlint:-deprecation
> -Xlint:-processing
> -Xlint:-rawtypes
> -Xlint:-serial
> -Xlint:-try
> -Xlint:-unchecked
> -Xlint:-varargs
>
>
>
>
> true
>
> false
>
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3631) Utilize Arrays#parallelSort for better performance
Ted Yu created KYLIN-3631: - Summary: Utilize Arrays#parallelSort for better performance Key: KYLIN-3631 URL: https://issues.apache.org/jira/browse/KYLIN-3631 Project: Kylin Issue Type: Task Reporter: Ted Yu Arrays#parallelSort was introduced since Java 1.8 We can utilize Arrays#parallelSort to achieve better performance. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3617) Reduce number of visiting metastore for job scheduler
[ https://issues.apache.org/jira/browse/KYLIN-3617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649411#comment-16649411 ] ASF GitHub Bot commented on KYLIN-3617: --- codecov-io commented on issue #286: KYLIN-3617 Use job's cache in job scheduler URL: https://github.com/apache/kylin/pull/286#issuecomment-429630698 # [Codecov](https://codecov.io/gh/apache/kylin/pull/286?src=pr=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@50f1758`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit). > The diff coverage is `43.75%`. [](https://codecov.io/gh/apache/kylin/pull/286?src=pr=tree) ```diff @@ Coverage Diff@@ ## master#286 +/- ## Coverage ? 21.3% Complexity? Files ?1087 Lines ? 69983 Branches ? 10109 Hits ? 14907 Misses? 53673 Partials ?1403 ``` | [Impacted Files](https://codecov.io/gh/apache/kylin/pull/286?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [...lin/job/impl/threadpool/PriorityFetcherRunner.java](https://codecov.io/gh/apache/kylin/pull/286/diff?src=pr=tree#diff-Y29yZS1qb2Ivc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2pvYi9pbXBsL3RocmVhZHBvb2wvUHJpb3JpdHlGZXRjaGVyUnVubmVyLmphdmE=) | `0% <0%> (ø)` | `0 <0> (?)` | | | [...he/kylin/job/impl/threadpool/DefaultScheduler.java](https://codecov.io/gh/apache/kylin/pull/286/diff?src=pr=tree#diff-Y29yZS1qb2Ivc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2pvYi9pbXBsL3RocmVhZHBvb2wvRGVmYXVsdFNjaGVkdWxlci5qYXZh) | `68.53% <100%> (ø)` | `10 <1> (?)` | | | [...n/java/org/apache/kylin/job/dao/ExecutableDao.java](https://codecov.io/gh/apache/kylin/pull/286/diff?src=pr=tree#diff-Y29yZS1qb2Ivc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2pvYi9kYW8vRXhlY3V0YWJsZURhby5qYXZh) | `35.38% <33.33%> (ø)` | `24 <2> (?)` | | | [.../apache/kylin/job/execution/ExecutableManager.java](https://codecov.io/gh/apache/kylin/pull/286/diff?src=pr=tree#diff-Y29yZS1qb2Ivc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2pvYi9leGVjdXRpb24vRXhlY3V0YWJsZU1hbmFnZXIuamF2YQ==) | `34.7% <42.85%> (ø)` | `35 <2> (?)` | | | [...ylin/job/impl/threadpool/DefaultFetcherRunner.java](https://codecov.io/gh/apache/kylin/pull/286/diff?src=pr=tree#diff-Y29yZS1qb2Ivc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2pvYi9pbXBsL3RocmVhZHBvb2wvRGVmYXVsdEZldGNoZXJSdW5uZXIuamF2YQ==) | `72.72% <85.71%> (ø)` | `10 <0> (?)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/286?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/286?src=pr=footer). Last update [50f1758...ee848c9](https://codecov.io/gh/apache/kylin/pull/286?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Reduce number of visiting metastore for job scheduler > - > > Key: KYLIN-3617 > URL: https://issues.apache.org/jira/browse/KYLIN-3617 > Project: Kylin > Issue Type: Improvement > Components: Job Engine >Affects Versions: v2.4.1 >Reporter: nichunen >Assignee: nichunen >Priority: Major > Fix For: v2.6.0 > > > For KYLIN-3470 introduced cache for jobs' metadata, it's also can be used in > job scheduler to reduce the pressure on metastore -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3617) Reduce number of visiting metastore for job scheduler
[
https://issues.apache.org/jira/browse/KYLIN-3617?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649405#comment-16649405
]
ASF GitHub Bot commented on KYLIN-3617:
---
nichunen commented on a change in pull request #286: KYLIN-3617 Use job's cache
in job scheduler
URL: https://github.com/apache/kylin/pull/286#discussion_r224993945
##
File path: core-job/src/main/java/org/apache/kylin/job/dao/ExecutableDao.java
##
@@ -391,4 +405,13 @@ public void deleteJobOutput(String uuid) throws
PersistentException {
throw new PersistentException(e);
}
}
+
+public void reloadAll() throws IOException {
+try (AutoReadWriteLock.AutoLock lock =
executableDigestMapLock.lockForWrite()) {
+executableDigestCrud.reloadAll();
Review comment:
@shaofengshi You are right, it's a mistake, should use lockForWrite during
reloading. PR updated, thanks
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Reduce number of visiting metastore for job scheduler
> -
>
> Key: KYLIN-3617
> URL: https://issues.apache.org/jira/browse/KYLIN-3617
> Project: Kylin
> Issue Type: Improvement
> Components: Job Engine
>Affects Versions: v2.4.1
>Reporter: nichunen
>Assignee: nichunen
>Priority: Major
> Fix For: v2.6.0
>
>
> For KYLIN-3470 introduced cache for jobs' metadata, it's also can be used in
> job scheduler to reduce the pressure on metastore
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3630) remove unused fields in the implementations of MeasureType
[
https://issues.apache.org/jira/browse/KYLIN-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649369#comment-16649369
]
ASF GitHub Bot commented on KYLIN-3630:
---
shaofengshi closed pull request #289: KYLIN-3630, remove unused fields in the
implementations of MeasureType
URL: https://github.com/apache/kylin/pull/289
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/core-metadata/src/main/java/org/apache/kylin/measure/bitmap/BitmapMeasureType.java
b/core-metadata/src/main/java/org/apache/kylin/measure/bitmap/BitmapMeasureType.java
index 403d1b6c6d..f724257de5 100644
---
a/core-metadata/src/main/java/org/apache/kylin/measure/bitmap/BitmapMeasureType.java
+++
b/core-metadata/src/main/java/org/apache/kylin/measure/bitmap/BitmapMeasureType.java
@@ -52,7 +52,7 @@
@Override
public MeasureType createMeasureType(String funcName,
DataType dataType) {
-return new BitmapMeasureType(funcName, dataType);
+return new BitmapMeasureType();
}
@Override
@@ -71,10 +71,7 @@ public String getAggrDataTypeName() {
}
}
-public DataType dataType;
-
-public BitmapMeasureType(String funcName, DataType dataType) {
-this.dataType = dataType;
+public BitmapMeasureType() {
}
@Override
diff --git
a/core-metadata/src/main/java/org/apache/kylin/measure/extendedcolumn/ExtendedColumnMeasureType.java
b/core-metadata/src/main/java/org/apache/kylin/measure/extendedcolumn/ExtendedColumnMeasureType.java
index b38299ac3b..4c20ef3fe2 100644
---
a/core-metadata/src/main/java/org/apache/kylin/measure/extendedcolumn/ExtendedColumnMeasureType.java
+++
b/core-metadata/src/main/java/org/apache/kylin/measure/extendedcolumn/ExtendedColumnMeasureType.java
@@ -56,7 +56,7 @@
@Override
public MeasureType createMeasureType(String funcName,
DataType dataType) {
-return new ExtendedColumnMeasureType(funcName, dataType);
+return new ExtendedColumnMeasureType(dataType);
}
@Override
@@ -75,7 +75,7 @@ public String getAggrDataTypeName() {
}
}
-public ExtendedColumnMeasureType(String funcName, DataType dataType) {
+public ExtendedColumnMeasureType(DataType dataType) {
this.dataType = dataType;
}
diff --git
a/core-metadata/src/main/java/org/apache/kylin/measure/hllc/HLLCMeasureType.java
b/core-metadata/src/main/java/org/apache/kylin/measure/hllc/HLLCMeasureType.java
index 51c5a66217..daa104baf7 100644
---
a/core-metadata/src/main/java/org/apache/kylin/measure/hllc/HLLCMeasureType.java
+++
b/core-metadata/src/main/java/org/apache/kylin/measure/hllc/HLLCMeasureType.java
@@ -43,7 +43,7 @@
@Override
public MeasureType createMeasureType(String funcName,
DataType dataType) {
-return new HLLCMeasureType(funcName, dataType);
+return new HLLCMeasureType(dataType);
}
@Override
@@ -66,7 +66,7 @@ public String getAggrDataTypeName() {
private final DataType dataType;
-public HLLCMeasureType(String funcName, DataType dataType) {
+public HLLCMeasureType(DataType dataType) {
// note at query parsing phase, the data type may be null, because
only function and parameters are known
this.dataType = dataType;
}
diff --git
a/core-metadata/src/main/java/org/apache/kylin/measure/percentile/PercentileMeasureType.java
b/core-metadata/src/main/java/org/apache/kylin/measure/percentile/PercentileMeasureType.java
index 2c79784696..44bd2133b2 100644
---
a/core-metadata/src/main/java/org/apache/kylin/measure/percentile/PercentileMeasureType.java
+++
b/core-metadata/src/main/java/org/apache/kylin/measure/percentile/PercentileMeasureType.java
@@ -39,7 +39,7 @@
public static final String FUNC_PERCENTILE_APPROX = "PERCENTILE_APPROX";
public static final String DATATYPE_PERCENTILE = "percentile";
-public PercentileMeasureType(String funcName, DataType dataType) {
+public PercentileMeasureType(DataType dataType) {
this.dataType = dataType;
}
@@ -47,7 +47,7 @@ public PercentileMeasureType(String funcName, DataType
dataType) {
@Override
public MeasureType createMeasureType(String
funcName, DataType dataType) {
-return new PercentileMeasureType(funcName, dataType);
+return new PercentileMeasureType(dataType);
}
@Override
diff --git
a/core-metadata/src/main/java/org/apache/kylin/measure/raw/RawMeasureType.java
b/core-metadata/src/main/java/org/apache/kylin/measure/raw/RawMeasureType.java
index 2add0602ad..e9f1c82c45 100644
---
[jira] [Commented] (KYLIN-3630) remove unused fields in the implementations of MeasureType
[
https://issues.apache.org/jira/browse/KYLIN-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649370#comment-16649370
]
ASF subversion and git services commented on KYLIN-3630:
Commit 50f175892a558d40d645501745c9ab7d060fa0f6 in kylin's branch
refs/heads/master from tttMelody
[ https://gitbox.apache.org/repos/asf?p=kylin.git;h=50f1758 ]
KYLIN-3630, remove unused fields in the implementations of MeasureType
> remove unused fields in the implementations of MeasureType
> --
>
> Key: KYLIN-3630
> URL: https://issues.apache.org/jira/browse/KYLIN-3630
> Project: Kylin
> Issue Type: Improvement
>Reporter: jiatao.tao
>Assignee: jiatao.tao
>Priority: Minor
>
> In code RawMeasureType#RawMeasureType, we can see that both funcName and
> dataType are use, and this can be remove cuz there's no constraint on its
> constructor.
> {code:java}
> @SuppressWarnings("unused")
> private final DataType dataType;
> public RawMeasureType(String funcName, DataType dataType) {
> this.dataType = dataType;
> }
> {code}
> And after we done that, we can more easily test MeasureType without passing
> in meaningless args(AggregatorMemEstimateTest#112).
> Like codes in org.apache.kylin.measure.dim.DimCountDistinctMeasureType, other
> implementations can be cleaner.
> {code:java}
> @Override
> public MeasureType createMeasureType(String funcName,
> DataType dataType) {
> return new DimCountDistinctMeasureType();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3597) Fix sonar reported static code issues
[
https://issues.apache.org/jira/browse/KYLIN-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649365#comment-16649365
]
ASF GitHub Bot commented on KYLIN-3597:
---
shaofengshi closed pull request #288: KYLIN-3597 Improve code smell
URL: https://github.com/apache/kylin/pull/288
This is a PR merged from a forked repository.
As GitHub hides the original diff on merge, it is displayed below for
the sake of provenance:
As this is a foreign pull request (from a fork), the diff is supplied
below (as it won't show otherwise due to GitHub magic):
diff --git
a/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
b/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
index 6f1dfd9144..386a73ca6f 100644
--- a/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
+++ b/core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
@@ -57,6 +57,12 @@
private static final long serialVersionUID = 1L;
private static final Logger logger =
LoggerFactory.getLogger(KylinConfigBase.class);
+private static final String FALSE = "false";
+private static final String TRUE = "true";
+private static final String DEFAULT = "default";
+private static final String KYLIN_ENGINE_MR_JOB_JAR =
"kylin.engine.mr.job-jar";
+private static final String KYLIN_STORAGE_HBASE_COPROCESSOR_LOCAL_JAR =
"kylin.storage.hbase.coprocessor-local-jar";
+
/*
* DON'T DEFINE CONSTANTS FOR PROPERTY KEYS!
*
@@ -84,13 +90,13 @@ public static String getKylinHomeWithoutWarn() {
public static String getSparkHome() {
String sparkHome = System.getenv("SPARK_HOME");
if (StringUtils.isNotEmpty(sparkHome)) {
-logger.info("SPARK_HOME was set to " + sparkHome);
+logger.info("SPARK_HOME was set to {}", sparkHome);
return sparkHome;
}
sparkHome = System.getProperty("SPARK_HOME");
if (StringUtils.isNotEmpty(sparkHome)) {
-logger.info("SPARK_HOME was set to " + sparkHome);
+logger.info("SPARK_HOME was set to {}", sparkHome);
return sparkHome;
}
@@ -195,7 +201,7 @@ final protected String getRequired(String prop) {
* Use with care, properties should be read-only. This is for testing only.
*/
final public void setProperty(String key, String value) {
-logger.info("Kylin Config was updated with " + key + " : " + value);
+logger.info("Kylin Config was updated with {} : {}", key, value);
properties.setProperty(BCC.check(key), value);
}
@@ -228,7 +234,6 @@ public String getDeployEnv() {
}
private String cachedHdfsWorkingDirectory;
-private String cachedBigCellDirectory;
public String getHdfsWorkingDirectory() {
if (cachedHdfsWorkingDirectory != null)
@@ -301,7 +306,7 @@ public String getZookeeperConnectString() {
}
public boolean isZookeeperAclEnabled() {
-return
Boolean.parseBoolean(getOptional("kylin.env.zookeeper-acl-enabled", "false"));
+return
Boolean.parseBoolean(getOptional("kylin.env.zookeeper-acl-enabled", FALSE));
}
public String getZKAuths() {
@@ -381,7 +386,7 @@ public String getHBaseMappingAdapter() {
}
public boolean isCheckCopyOnWrite() {
-return
Boolean.parseBoolean(getOptional("kylin.metadata.check-copy-on-write",
"false"));
+return
Boolean.parseBoolean(getOptional("kylin.metadata.check-copy-on-write", FALSE));
}
public String getHbaseClientScannerTimeoutPeriod() {
@@ -401,7 +406,7 @@ public String getHbaseClientRetriesNumber() {
//
public boolean isUseForestTrieDictionary() {
-return
Boolean.parseBoolean(getOptional("kylin.dictionary.use-forest-trie", "true"));
+return
Boolean.parseBoolean(getOptional("kylin.dictionary.use-forest-trie", TRUE));
}
public int getTrieDictionaryForestMaxTrieSizeMB() {
@@ -413,11 +418,11 @@ public int getCachedDictMaxEntrySize() {
}
public boolean isGrowingDictEnabled() {
-return
Boolean.parseBoolean(this.getOptional("kylin.dictionary.growing-enabled",
"false"));
+return
Boolean.parseBoolean(this.getOptional("kylin.dictionary.growing-enabled",
FALSE));
}
public boolean isDictResuable() {
-return
Boolean.parseBoolean(this.getOptional("kylin.dictionary.resuable", "false"));
+return
Boolean.parseBoolean(this.getOptional("kylin.dictionary.resuable", FALSE));
}
public int getAppendDictEntrySize() {
@@ -453,7 +458,7 @@ public double getExtTableSnapshotLocalCacheMaxSizeGB() {
}
public boolean isShrunkenDictFromGlobalEnabled() {
-return
Boolean.parseBoolean(this.getOptional("kylin.dictionary.shrunken-from-global-enabled",
"false"));
+
[jira] [Commented] (KYLIN-3597) Fix sonar reported static code issues
[ https://issues.apache.org/jira/browse/KYLIN-3597?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649366#comment-16649366 ] ASF subversion and git services commented on KYLIN-3597: Commit ae9c24a5782841e5079fad44d2c807b07ae72f33 in kylin's branch refs/heads/master from [~caolijun1166] [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=ae9c24a ] KYLIN-3597 Improve code smell > Fix sonar reported static code issues > - > > Key: KYLIN-3597 > URL: https://issues.apache.org/jira/browse/KYLIN-3597 > Project: Kylin > Issue Type: Improvement > Components: Others >Reporter: Shaofeng SHI >Priority: Major > Fix For: v2.6.0 > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-3630) remove unused fields in the implementations of MeasureType
[
https://issues.apache.org/jira/browse/KYLIN-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649352#comment-16649352
]
ASF GitHub Bot commented on KYLIN-3630:
---
codecov-io commented on issue #289: KYLIN-3630, remove unused fields in the
implementations of MeasureType
URL: https://github.com/apache/kylin/pull/289#issuecomment-429619135
# [Codecov](https://codecov.io/gh/apache/kylin/pull/289?src=pr=h1) Report
> :exclamation: No coverage uploaded for pull request base
(`master@ec10114`). [Click here to learn what that
means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit).
> The diff coverage is `100%`.
[](https://codecov.io/gh/apache/kylin/pull/289?src=pr=tree)
```diff
@@ Coverage Diff@@
## master#289 +/- ##
Coverage ? 21.3%
Complexity?4441
Files ?1087
Lines ? 69953
Branches ? 10108
Hits ? 14906
Misses? 53644
Partials ?1403
```
| [Impacted
Files](https://codecov.io/gh/apache/kylin/pull/289?src=pr=tree) | Coverage Δ
| Complexity Δ | |
|---|---|---|---|
|
[...sure/extendedcolumn/ExtendedColumnMeasureType.java](https://codecov.io/gh/apache/kylin/pull/289/diff?src=pr=tree#diff-Y29yZS1tZXRhZGF0YS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUva3lsaW4vbWVhc3VyZS9leHRlbmRlZGNvbHVtbi9FeHRlbmRlZENvbHVtbk1lYXN1cmVUeXBlLmphdmE=)
| `30.27% <100%> (ø)` | `5 <1> (?)` | |
|
[...org/apache/kylin/measure/topn/TopNMeasureType.java](https://codecov.io/gh/apache/kylin/pull/289/diff?src=pr=tree#diff-Y29yZS1tZXRhZGF0YS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUva3lsaW4vbWVhc3VyZS90b3BuL1RvcE5NZWFzdXJlVHlwZS5qYXZh)
| `4.5% <100%> (ø)` | `3 <1> (?)` | |
|
[...apache/kylin/measure/bitmap/BitmapMeasureType.java](https://codecov.io/gh/apache/kylin/pull/289/diff?src=pr=tree#diff-Y29yZS1tZXRhZGF0YS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUva3lsaW4vbWVhc3VyZS9iaXRtYXAvQml0bWFwTWVhc3VyZVR5cGUuamF2YQ==)
| `19.6% <100%> (ø)` | `4 <1> (?)` | |
|
[...a/org/apache/kylin/measure/raw/RawMeasureType.java](https://codecov.io/gh/apache/kylin/pull/289/diff?src=pr=tree#diff-Y29yZS1tZXRhZGF0YS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUva3lsaW4vbWVhc3VyZS9yYXcvUmF3TWVhc3VyZVR5cGUuamF2YQ==)
| `9.37% <100%> (ø)` | `3 <1> (?)` | |
|
[...org/apache/kylin/measure/hllc/HLLCMeasureType.java](https://codecov.io/gh/apache/kylin/pull/289/diff?src=pr=tree#diff-Y29yZS1tZXRhZGF0YS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUva3lsaW4vbWVhc3VyZS9obGxjL0hMTENNZWFzdXJlVHlwZS5qYXZh)
| `65% <100%> (ø)` | `5 <1> (?)` | |
|
[...ylin/measure/percentile/PercentileMeasureType.java](https://codecov.io/gh/apache/kylin/pull/289/diff?src=pr=tree#diff-Y29yZS1tZXRhZGF0YS9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUva3lsaW4vbWVhc3VyZS9wZXJjZW50aWxlL1BlcmNlbnRpbGVNZWFzdXJlVHlwZS5qYXZh)
| `55% <100%> (ø)` | `4 <1> (?)` | |
--
[Continue to review full report at
Codecov](https://codecov.io/gh/apache/kylin/pull/289?src=pr=continue).
> **Legend** - [Click here to learn
more](https://docs.codecov.io/docs/codecov-delta)
> `Δ = absolute (impact)`, `ø = not affected`, `? = missing data`
> Powered by
[Codecov](https://codecov.io/gh/apache/kylin/pull/289?src=pr=footer). Last
update
[ec10114...f32c098](https://codecov.io/gh/apache/kylin/pull/289?src=pr=lastupdated).
Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments).
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> remove unused fields in the implementations of MeasureType
> --
>
> Key: KYLIN-3630
> URL: https://issues.apache.org/jira/browse/KYLIN-3630
> Project: Kylin
> Issue Type: Improvement
>Reporter: jiatao.tao
>Assignee: jiatao.tao
>Priority: Minor
>
> In code RawMeasureType#RawMeasureType, we can see that both funcName and
> dataType are use, and this can be remove cuz there's no constraint on its
> constructor.
> {code:java}
> @SuppressWarnings("unused")
> private final DataType dataType;
> public RawMeasureType(String funcName, DataType dataType) {
> this.dataType = dataType;
> }
> {code}
> And after we done that,
[jira] [Updated] (KYLIN-3630) remove unused fields in the implementations of MeasureType
[
https://issues.apache.org/jira/browse/KYLIN-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jiatao.tao updated KYLIN-3630:
--
Description:
In code RawMeasureType#RawMeasureType, we can see that both funcName and
dataType are use, and this can be remove cuz there's no constraint on its
constructor.
{code:java}
@SuppressWarnings("unused")
private final DataType dataType;
public RawMeasureType(String funcName, DataType dataType) {
this.dataType = dataType;
}
{code}
And after we done that, we can more easily test MeasureType without passing in
meaningless args(AggregatorMemEstimateTest#112).
Like codes in org.apache.kylin.measure.dim.DimCountDistinctMeasureType, other
implementations can be cleaner.
{code:java}
@Override
public MeasureType createMeasureType(String funcName, DataType
dataType) {
return new DimCountDistinctMeasureType();
}
{code}
was:
In code RawMeasureType#RawMeasureType, we can see that both funcName and
dataType are use, and this can be remove cuz there's no constraint on its
constructor.
{code:java}
@SuppressWarnings("unused")
private final DataType dataType;
public RawMeasureType(String funcName, DataType dataType) {
this.dataType = dataType;
}
{code}
And after we done that, we can more easily test MeasureType without passing in
meaningless args.
Like codes in org.apache.kylin.measure.dim.DimCountDistinctMeasureType, other
implementations can be cleaner.
{code:java}
@Override
public MeasureType createMeasureType(String funcName, DataType
dataType) {
return new DimCountDistinctMeasureType();
}
{code}
> remove unused fields in the implementations of MeasureType
> --
>
> Key: KYLIN-3630
> URL: https://issues.apache.org/jira/browse/KYLIN-3630
> Project: Kylin
> Issue Type: Improvement
>Reporter: jiatao.tao
>Assignee: jiatao.tao
>Priority: Minor
>
> In code RawMeasureType#RawMeasureType, we can see that both funcName and
> dataType are use, and this can be remove cuz there's no constraint on its
> constructor.
> {code:java}
> @SuppressWarnings("unused")
> private final DataType dataType;
> public RawMeasureType(String funcName, DataType dataType) {
> this.dataType = dataType;
> }
> {code}
> And after we done that, we can more easily test MeasureType without passing
> in meaningless args(AggregatorMemEstimateTest#112).
> Like codes in org.apache.kylin.measure.dim.DimCountDistinctMeasureType, other
> implementations can be cleaner.
> {code:java}
> @Override
> public MeasureType createMeasureType(String funcName,
> DataType dataType) {
> return new DimCountDistinctMeasureType();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3630) remove unused fields in the implementations of MeasureType
[
https://issues.apache.org/jira/browse/KYLIN-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649344#comment-16649344
]
ASF GitHub Bot commented on KYLIN-3630:
---
tttMelody opened a new pull request #289: KYLIN-3630, remove unused fields in
the implementations of MeasureType
URL: https://github.com/apache/kylin/pull/289
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> remove unused fields in the implementations of MeasureType
> --
>
> Key: KYLIN-3630
> URL: https://issues.apache.org/jira/browse/KYLIN-3630
> Project: Kylin
> Issue Type: Improvement
>Reporter: jiatao.tao
>Assignee: jiatao.tao
>Priority: Minor
>
> In code RawMeasureType#RawMeasureType, we can see that both funcName and
> dataType are use, and this can be remove cuz there's no constraint on its
> constructor.
> {code:java}
> @SuppressWarnings("unused")
> private final DataType dataType;
> public RawMeasureType(String funcName, DataType dataType) {
> this.dataType = dataType;
> }
> {code}
> And after we done that, we can more easily test MeasureType without passing
> in meaningless args.
> Like codes in org.apache.kylin.measure.dim.DimCountDistinctMeasureType, other
> implementations can be cleaner.
> {code:java}
> @Override
> public MeasureType createMeasureType(String funcName,
> DataType dataType) {
> return new DimCountDistinctMeasureType();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-3630) remove unused fields in the implementations of MeasureType
[
https://issues.apache.org/jira/browse/KYLIN-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16649345#comment-16649345
]
ASF GitHub Bot commented on KYLIN-3630:
---
asfgit commented on issue #289: KYLIN-3630, remove unused fields in the
implementations of MeasureType
URL: https://github.com/apache/kylin/pull/289#issuecomment-429617849
Can one of the admins verify this patch?
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> remove unused fields in the implementations of MeasureType
> --
>
> Key: KYLIN-3630
> URL: https://issues.apache.org/jira/browse/KYLIN-3630
> Project: Kylin
> Issue Type: Improvement
>Reporter: jiatao.tao
>Assignee: jiatao.tao
>Priority: Minor
>
> In code RawMeasureType#RawMeasureType, we can see that both funcName and
> dataType are use, and this can be remove cuz there's no constraint on its
> constructor.
> {code:java}
> @SuppressWarnings("unused")
> private final DataType dataType;
> public RawMeasureType(String funcName, DataType dataType) {
> this.dataType = dataType;
> }
> {code}
> And after we done that, we can more easily test MeasureType without passing
> in meaningless args.
> Like codes in org.apache.kylin.measure.dim.DimCountDistinctMeasureType, other
> implementations can be cleaner.
> {code:java}
> @Override
> public MeasureType createMeasureType(String funcName,
> DataType dataType) {
> return new DimCountDistinctMeasureType();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3630) remove unused fields in the implementations of MeasureType
[
https://issues.apache.org/jira/browse/KYLIN-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jiatao.tao updated KYLIN-3630:
--
Description:
In code RawMeasureType#RawMeasureType, we can see that both funcName and
dataType are use, and this can be remove cuz there's no constraint on its
constructor.
{code:java}
@SuppressWarnings("unused")
private final DataType dataType;
public RawMeasureType(String funcName, DataType dataType) {
this.dataType = dataType;
}
{code}
And after we done that, we can mo easily test MeasureType without passing in
meaningless args.
Like codes in org.apache.kylin.measure.dim.DimCountDistinctMeasureType, other
{code:java}
@Override
public MeasureType createMeasureType(String funcName, DataType
dataType) {
return new DimCountDistinctMeasureType();
}
{code}
was:
Like in code org.apache.kylin.measure.raw.RawMeasureType#RawMeasureType, we can
see that both funcName and dataType are use, and this can be remove cuz
there's no constraint on its constructor.
And after we done that, we can mo easily test MeasureType without passing in
meaningless args.
Like codes in org.apache.kylin.measure.dim.DimCountDistinctMeasureType
> remove unused fields in the implementations of MeasureType
> --
>
> Key: KYLIN-3630
> URL: https://issues.apache.org/jira/browse/KYLIN-3630
> Project: Kylin
> Issue Type: Improvement
>Reporter: jiatao.tao
>Assignee: jiatao.tao
>Priority: Minor
>
> In code RawMeasureType#RawMeasureType, we can see that both funcName and
> dataType are use, and this can be remove cuz there's no constraint on its
> constructor.
> {code:java}
> @SuppressWarnings("unused")
> private final DataType dataType;
> public RawMeasureType(String funcName, DataType dataType) {
> this.dataType = dataType;
> }
> {code}
> And after we done that, we can mo easily test MeasureType without passing in
> meaningless args.
> Like codes in org.apache.kylin.measure.dim.DimCountDistinctMeasureType, other
> {code:java}
> @Override
> public MeasureType createMeasureType(String funcName,
> DataType dataType) {
> return new DimCountDistinctMeasureType();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3630) remove unused fields in the implementations of MeasureType
[
https://issues.apache.org/jira/browse/KYLIN-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
jiatao.tao updated KYLIN-3630:
--
Description:
In code RawMeasureType#RawMeasureType, we can see that both funcName and
dataType are use, and this can be remove cuz there's no constraint on its
constructor.
{code:java}
@SuppressWarnings("unused")
private final DataType dataType;
public RawMeasureType(String funcName, DataType dataType) {
this.dataType = dataType;
}
{code}
And after we done that, we can more easily test MeasureType without passing in
meaningless args.
Like codes in org.apache.kylin.measure.dim.DimCountDistinctMeasureType, other
implementations can be cleaner.
{code:java}
@Override
public MeasureType createMeasureType(String funcName, DataType
dataType) {
return new DimCountDistinctMeasureType();
}
{code}
was:
In code RawMeasureType#RawMeasureType, we can see that both funcName and
dataType are use, and this can be remove cuz there's no constraint on its
constructor.
{code:java}
@SuppressWarnings("unused")
private final DataType dataType;
public RawMeasureType(String funcName, DataType dataType) {
this.dataType = dataType;
}
{code}
And after we done that, we can mo easily test MeasureType without passing in
meaningless args.
Like codes in org.apache.kylin.measure.dim.DimCountDistinctMeasureType, other
{code:java}
@Override
public MeasureType createMeasureType(String funcName, DataType
dataType) {
return new DimCountDistinctMeasureType();
}
{code}
> remove unused fields in the implementations of MeasureType
> --
>
> Key: KYLIN-3630
> URL: https://issues.apache.org/jira/browse/KYLIN-3630
> Project: Kylin
> Issue Type: Improvement
>Reporter: jiatao.tao
>Assignee: jiatao.tao
>Priority: Minor
>
> In code RawMeasureType#RawMeasureType, we can see that both funcName and
> dataType are use, and this can be remove cuz there's no constraint on its
> constructor.
> {code:java}
> @SuppressWarnings("unused")
> private final DataType dataType;
> public RawMeasureType(String funcName, DataType dataType) {
> this.dataType = dataType;
> }
> {code}
> And after we done that, we can more easily test MeasureType without passing
> in meaningless args.
> Like codes in org.apache.kylin.measure.dim.DimCountDistinctMeasureType, other
> implementations can be cleaner.
> {code:java}
> @Override
> public MeasureType createMeasureType(String funcName,
> DataType dataType) {
> return new DimCountDistinctMeasureType();
> }
> {code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Updated] (KYLIN-3630) remove unused fields in the implementations of MeasureType
[ https://issues.apache.org/jira/browse/KYLIN-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiatao.tao updated KYLIN-3630: -- Attachment: (was: screenshot-1.png) > remove unused fields in the implementations of MeasureType > -- > > Key: KYLIN-3630 > URL: https://issues.apache.org/jira/browse/KYLIN-3630 > Project: Kylin > Issue Type: Improvement >Reporter: jiatao.tao >Assignee: jiatao.tao >Priority: Minor > > Like in code org.apache.kylin.measure.raw.RawMeasureType#RawMeasureType, we > can see that both funcName and dataType are use, and this can be remove cuz > there's no constraint on its constructor. > And after we done that, we can mo easily test MeasureType without passing in > meaningless args. > Like codes in org.apache.kylin.measure.dim.DimCountDistinctMeasureType -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3630) remove unused fields in the implementations of MeasureType
[ https://issues.apache.org/jira/browse/KYLIN-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiatao.tao updated KYLIN-3630: -- Attachment: (was: image-2018-10-14-18-56-29-010.png) > remove unused fields in the implementations of MeasureType > -- > > Key: KYLIN-3630 > URL: https://issues.apache.org/jira/browse/KYLIN-3630 > Project: Kylin > Issue Type: Improvement >Reporter: jiatao.tao >Assignee: jiatao.tao >Priority: Minor > > Like in code org.apache.kylin.measure.raw.RawMeasureType#RawMeasureType, we > can see that both funcName and dataType are use, and this can be remove cuz > there's no constraint on its constructor. > And after we done that, we can mo easily test MeasureType without passing in > meaningless args. > Like codes in org.apache.kylin.measure.dim.DimCountDistinctMeasureType -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3630) remove unused fields in the implementations of MeasureType
[ https://issues.apache.org/jira/browse/KYLIN-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiatao.tao updated KYLIN-3630: -- Description: Like in code org.apache.kylin.measure.raw.RawMeasureType#RawMeasureType, we can see that both funcName and dataType are use, and this can be remove cuz there's no constraint on its constructor. And after we done that, we can mo easily test MeasureType without passing in meaningless args. Like codes in org.apache.kylin.measure.dim.DimCountDistinctMeasureType was: Like in code org.apache.kylin.measure.raw.RawMeasureType#RawMeasureType, we can see that both funcName and dataType are use, and this can be remove cuz there's no constraint on its constructor. And after we done that, we can mo easily test MeasureType without passing in meaningless args. !screenshot-1.png! Like codes in org.apache.kylin.measure.dim.DimCountDistinctMeasureType !image-2018-10-14-18-56-29-010.png! > remove unused fields in the implementations of MeasureType > -- > > Key: KYLIN-3630 > URL: https://issues.apache.org/jira/browse/KYLIN-3630 > Project: Kylin > Issue Type: Improvement >Reporter: jiatao.tao >Assignee: jiatao.tao >Priority: Minor > > Like in code org.apache.kylin.measure.raw.RawMeasureType#RawMeasureType, we > can see that both funcName and dataType are use, and this can be remove cuz > there's no constraint on its constructor. > And after we done that, we can mo easily test MeasureType without passing in > meaningless args. > Like codes in org.apache.kylin.measure.dim.DimCountDistinctMeasureType -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Updated] (KYLIN-3630) remove unused fields in the implementations of MeasureType
[ https://issues.apache.org/jira/browse/KYLIN-3630?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] jiatao.tao updated KYLIN-3630: -- Attachment: screenshot-1.png > remove unused fields in the implementations of MeasureType > -- > > Key: KYLIN-3630 > URL: https://issues.apache.org/jira/browse/KYLIN-3630 > Project: Kylin > Issue Type: Improvement >Reporter: jiatao.tao >Assignee: jiatao.tao >Priority: Minor > Attachments: image-2018-10-14-18-56-29-010.png, screenshot-1.png > > > !image-2018-10-14-18-56-29-010.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3630) remove unused fields in the implementations of MeasureType
jiatao.tao created KYLIN-3630: - Summary: remove unused fields in the implementations of MeasureType Key: KYLIN-3630 URL: https://issues.apache.org/jira/browse/KYLIN-3630 Project: Kylin Issue Type: Improvement Reporter: jiatao.tao Assignee: jiatao.tao Attachments: image-2018-10-14-18-56-29-010.png !image-2018-10-14-18-56-29-010.png! -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Assigned] (KYLIN-3620) "--" should not be a comment marker use between single quotes in SQL
[ https://issues.apache.org/jira/browse/KYLIN-3620?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lijun Cao reassigned KYLIN-3620: Assignee: Lijun Cao > "--" should not be a comment marker use between single quotes in SQL > > > Key: KYLIN-3620 > URL: https://issues.apache.org/jira/browse/KYLIN-3620 > Project: Kylin > Issue Type: Bug > Components: Query Engine >Affects Versions: v2.4.1 >Reporter: Scott Fan >Assignee: Lijun Cao >Priority: Major > > Hi, > When I exec a SQL like below, there was an error. > SELECT * FROM TABLEA WHERE NAME='Jason--Arthur' > Seems like KYLIN use the – as a comment marker. > It should not be that between single quotes, I think. > > Even I send a REST POST contains SQL like that, or do an ODBC query, there > was an error too. > You may tell me that I can use a \ as escape, but I can't change the SQL > generated by BI tools like tableau. > Thanks > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
