[jira] [Commented] (KYLIN-4852) Update hadoop 2 version to 2.7.3 or above
[ https://issues.apache.org/jira/browse/KYLIN-4852?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17256852#comment-17256852 ] ASF GitHub Bot commented on KYLIN-4852: --- shaofengshi opened a new pull request #1528: URL: https://github.com/apache/kylin/pull/1528 ## Proposed changes Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request. If it fixes a bug or resolves a feature request, be sure to link to that issue. ## Types of changes What types of changes does your code introduce to Kylin? _Put an `x` in the boxes that apply_ - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation Update (if none of the other choices apply) ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [x] I have create an issue on [Kylin's jira](https://issues.apache.org/jira/browse/KYLIN), and have described the bug/feature there in detail - [x] Commit messages in my PR start with the related jira ID, like "KYLIN- Make Kylin project open-source" - [ ] Compiling and unit tests pass locally with my changes - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] If this change need a document change, I will prepare another pr against the `document` branch - [ ] Any dependent changes have been merged ## Further comments If this is a relatively large or complex change, kick off the discussion at user@kylin or dev@kylin by explaining why you chose the solution you did and what alternatives you considered, etc... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Update hadoop 2 version to 2.7.3 or above > - > > Key: KYLIN-4852 > URL: https://issues.apache.org/jira/browse/KYLIN-4852 > Project: Kylin > Issue Type: Improvement > Components: Environment >Reporter: Shao Feng Shi >Priority: Major > > The current hadoop version in Kylin is 2.7.1, which has several known high > risk vulnerabilities, need to upgrade. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [kylin] shaofengshi opened a new pull request #1528: KYLIN-4852 update dependency versions
shaofengshi opened a new pull request #1528: URL: https://github.com/apache/kylin/pull/1528 ## Proposed changes Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request. If it fixes a bug or resolves a feature request, be sure to link to that issue. ## Types of changes What types of changes does your code introduce to Kylin? _Put an `x` in the boxes that apply_ - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation Update (if none of the other choices apply) ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [x] I have create an issue on [Kylin's jira](https://issues.apache.org/jira/browse/KYLIN), and have described the bug/feature there in detail - [x] Commit messages in my PR start with the related jira ID, like "KYLIN- Make Kylin project open-source" - [ ] Compiling and unit tests pass locally with my changes - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] If this change need a document change, I will prepare another pr against the `document` branch - [ ] Any dependent changes have been merged ## Further comments If this is a relatively large or complex change, kick off the discussion at user@kylin or dev@kylin by explaining why you chose the solution you did and what alternatives you considered, etc... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (KYLIN-4852) Update hadoop 2 version to 2.7.3 or above
Shao Feng Shi created KYLIN-4852: Summary: Update hadoop 2 version to 2.7.3 or above Key: KYLIN-4852 URL: https://issues.apache.org/jira/browse/KYLIN-4852 Project: Kylin Issue Type: Improvement Components: Environment Reporter: Shao Feng Shi The current hadoop version in Kylin is 2.7.1, which has several known high risk vulnerabilities, need to upgrade. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [kylin] hit-lacus merged pull request #1527: revert hadoop version change to recover UT
hit-lacus merged pull request #1527: URL: https://github.com/apache/kylin/pull/1527 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kylin] codecov-io commented on pull request #1527: revert hadoop version change to recover UT
codecov-io commented on pull request #1527: URL: https://github.com/apache/kylin/pull/1527#issuecomment-752856996 # [Codecov](https://codecov.io/gh/apache/kylin/pull/1527?src=pr=h1) Report > Merging [#1527](https://codecov.io/gh/apache/kylin/pull/1527?src=pr=desc) (99a4bbe) into [master](https://codecov.io/gh/apache/kylin/commit/3e12b6d621fe8e5c747a5783f64bc535618c8035?el=desc) (3e12b6d) will **decrease** coverage by `0.06%`. > The diff coverage is `2.24%`. [](https://codecov.io/gh/apache/kylin/pull/1527?src=pr=tree) ```diff @@ Coverage Diff @@ ## master#1527 +/- ## - Coverage 25.51% 25.44% -0.07% + Complexity 6765 6762 -3 Files 1505 1507 +2 Lines 9361893784 +166 Branches 1311113132 +21 - Hits 2388623864 -22 - Misses6735667535 +179 - Partials 2376 2385 +9 ``` | [Impacted Files](https://codecov.io/gh/apache/kylin/pull/1527?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [.../java/org/apache/kylin/common/KylinConfigBase.java](https://codecov.io/gh/apache/kylin/pull/1527/diff?src=pr=tree#diff-Y29yZS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2NvbW1vbi9LeWxpbkNvbmZpZ0Jhc2UuamF2YQ==) | `12.51% <0.00%> (-0.05%)` | `51.00 <0.00> (ø)` | | | [.../java/org/apache/kylin/job/util/JobRestClient.java](https://codecov.io/gh/apache/kylin/pull/1527/diff?src=pr=tree#diff-Y29yZS1qb2Ivc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2pvYi91dGlsL0pvYlJlc3RDbGllbnQuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | | | [.../apache/kylin/engine/spark/SparkCubingByLayer.java](https://codecov.io/gh/apache/kylin/pull/1527/diff?src=pr=tree#diff-ZW5naW5lLXNwYXJrL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9lbmdpbmUvc3BhcmsvU3BhcmtDdWJpbmdCeUxheWVyLmphdmE=) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (ø)` | | | [...che/kylin/engine/spark/SparkMergingDictionary.java](https://codecov.io/gh/apache/kylin/pull/1527/diff?src=pr=tree#diff-ZW5naW5lLXNwYXJrL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9lbmdpbmUvc3BhcmsvU3BhcmtNZXJnaW5nRGljdGlvbmFyeS5qYXZh) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (ø)` | | | [...rg/apache/kylin/source/hive/BeelineHiveClient.java](https://codecov.io/gh/apache/kylin/pull/1527/diff?src=pr=tree#diff-c291cmNlLWhpdmUvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3NvdXJjZS9oaXZlL0JlZWxpbmVIaXZlQ2xpZW50LmphdmE=) | `3.42% <0.00%> (ø)` | `5.00 <0.00> (ø)` | | | [...n/java/org/apache/kylin/tool/CubeMigrationCLI.java](https://codecov.io/gh/apache/kylin/pull/1527/diff?src=pr=tree#diff-dG9vbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUva3lsaW4vdG9vbC9DdWJlTWlncmF0aW9uQ0xJLmphdmE=) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (ø)` | | | [...a/org/apache/kylin/tool/job/CubeBuildingCLIV2.java](https://codecov.io/gh/apache/kylin/pull/1527/diff?src=pr=tree#diff-dG9vbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUva3lsaW4vdG9vbC9qb2IvQ3ViZUJ1aWxkaW5nQ0xJVjIuamF2YQ==) | `0.00% <0.00%> (ø)` | `0.00 <0.00> (?)` | | | [...main/java/org/apache/kylin/common/KylinConfig.java](https://codecov.io/gh/apache/kylin/pull/1527/diff?src=pr=tree#diff-Y29yZS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2NvbW1vbi9LeWxpbkNvbmZpZy5qYXZh) | `38.46% <42.85%> (-0.11%)` | `30.00 <1.00> (+1.00)` | :arrow_down: | | [...org/apache/kylin/common/restclient/RestClient.java](https://codecov.io/gh/apache/kylin/pull/1527/diff?src=pr=tree#diff-Y29yZS1jb21tb24vc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL2NvbW1vbi9yZXN0Y2xpZW50L1Jlc3RDbGllbnQuamF2YQ==) | `17.79% <100.00%> (ø)` | `8.00 <0.00> (ø)` | | | [.../apache/kylin/tool/query/ProbabilityGenerator.java](https://codecov.io/gh/apache/kylin/pull/1527/diff?src=pr=tree#diff-dG9vbC9zcmMvbWFpbi9qYXZhL29yZy9hcGFjaGUva3lsaW4vdG9vbC9xdWVyeS9Qcm9iYWJpbGl0eUdlbmVyYXRvci5qYXZh) | `65.78% <0.00%> (-10.53%)` | `10.00% <0.00%> (-1.00%)` | | | ... and [6 more](https://codecov.io/gh/apache/kylin/pull/1527/diff?src=pr=tree-more) | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/1527?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/1527?src=pr=footer). Last update [9a3e5a3...99a4bbe](https://codecov.io/gh/apache/kylin/pull/1527?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an
[GitHub] [kylin] shaofengshi opened a new pull request #1527: revert hadoop version change to recover UT
shaofengshi opened a new pull request #1527: URL: https://github.com/apache/kylin/pull/1527 ## Proposed changes revert a change on master branch yesterday by mistake when I made another change ## Types of changes What types of changes does your code introduce to Kylin? _Put an `x` in the boxes that apply_ - [x] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation Update (if none of the other choices apply) ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [ ] I have create an issue on [Kylin's jira](https://issues.apache.org/jira/browse/KYLIN), and have described the bug/feature there in detail - [ ] Commit messages in my PR start with the related jira ID, like "KYLIN- Make Kylin project open-source" - [x] Compiling and unit tests pass locally with my changes - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] If this change need a document change, I will prepare another pr against the `document` branch - [x] Any dependent changes have been merged ## Further comments This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KYLIN-4658) Union all issue with regarding to windows function & aggregation on
[
https://issues.apache.org/jira/browse/KYLIN-4658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17256829#comment-17256829
]
ASF GitHub Bot commented on KYLIN-4658:
---
kyotoYaho commented on pull request #1526:
URL: https://github.com/apache/kylin/pull/1526#issuecomment-752843139
LGTM
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Union all issue with regarding to windows function & aggregation on
>
>
> Key: KYLIN-4658
> URL: https://issues.apache.org/jira/browse/KYLIN-4658
> Project: Kylin
> Issue Type: Improvement
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>Priority: Major
>
> Test SQL:
> {code}
> select CNT, GMV, sum(GMV) over(partition by SLR_SEGMENT_CD) TOTAL_GMV,
> SLR_SEGMENT_CD, LSTG_FORMAT_NAME
> from
> (select sum(PRICE) GMV, sum(ITEM_COUNT) CNT, SLR_SEGMENT_CD, LSTG_FORMAT_NAME
> from TEST_KYLIN_FACT group by SLR_SEGMENT_CD, LSTG_FORMAT_NAME
> UNION ALL
> select sum(PRICE) GMV, sum(ITEM_COUNT) CNT, SLR_SEGMENT_CD, LSTG_FORMAT_NAME
> from TEST_KYLIN_FACT group by SLR_SEGMENT_CD, LSTG_FORMAT_NAME)
> order by TOTAL_GMV
> {code}
>
> Exception:
> {code}
> Index: 2, Size: 2 while executing SQL: "select * from (select CNT, GMV,
> sum(GMV) over(partition by SLR_SEGMENT_CD) TOTAL_GMV, SLR_SEGMENT_CD,
> LSTG_FORMAT_NAME from (select sum(PRICE) GMV, sum(ITEM_COUNT) CNT,
> SLR_SEGMENT_CD, LSTG_FORMAT_NAME from TEST_KYLIN_FACT group by
> SLR_SEGMENT_CD, LSTG_FORMAT_NAME UNION ALL select sum(PRICE) GMV,
> sum(ITEM_COUNT) CNT, SLR_SEGMENT_CD, LSTG_FORMAT_NAME from TEST_KYLIN_FACT
> group by SLR_SEGMENT_CD, LSTG_FORMAT_NAME) order by TOTAL_GMV) limit 5"
> {code}
> Similar issue for the following sql:
> {code}
> select LSTG_FORMAT_NAME,
>SLR_SEGMENT_CD,
>CAL_DT,
>sum(CNT) as CNT
> from
> (select LSTG_FORMAT_NAME,
> SLR_SEGMENT_CD,
> CAL_DT,
> sum(ITEM_COUNT) CNT
>from TEST_KYLIN_FACT
>where LSTG_FORMAT_NAME = 'ABIN'
>group by LSTG_FORMAT_NAME,
> SLR_SEGMENT_CD,
> CAL_DT
>UNION ALL select 'NON-ABIN' as LSTG_FORMAT_NAME,
> SLR_SEGMENT_CD,
> CAL_DT,
> case
> when SLR_SEGMENT_CD > 1000 then CNT * 2
> else CNT * 3
> end as CNT
>from
> (select SLR_SEGMENT_CD,
> CAL_DT,
> sum(ITEM_COUNT) CNT
> from TEST_KYLIN_FACT
> where LSTG_FORMAT_NAME <> 'ABIN'
> group by SLR_SEGMENT_CD,CAL_DT))
> group by LSTG_FORMAT_NAME,
> SLR_SEGMENT_CD,
> CAL_DT
> order by CNT
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [kylin] kyotoYaho commented on pull request #1526: KYLIN-4658 Fix union all issue with regarding to windows function & aggregation on
kyotoYaho commented on pull request #1526: URL: https://github.com/apache/kylin/pull/1526#issuecomment-752843139 LGTM This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Created] (KYLIN-4851) Better to throw exception when lazy query waiting timeout
Zhong Yanghong created KYLIN-4851: - Summary: Better to throw exception when lazy query waiting timeout Key: KYLIN-4851 URL: https://issues.apache.org/jira/browse/KYLIN-4851 Project: Kylin Issue Type: Improvement Reporter: Zhong Yanghong Assignee: Zhong Yanghong -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4658) Union all issue with regarding to windows function & aggregation on
[
https://issues.apache.org/jira/browse/KYLIN-4658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17256806#comment-17256806
]
ASF GitHub Bot commented on KYLIN-4658:
---
zzcclp commented on pull request #1526:
URL: https://github.com/apache/kylin/pull/1526#issuecomment-752824642
Verified passed. LGTM.
## Before this pr
### Query with window function:
**The result is wrong.**
### Union query:
## After this pr
### Query with window function:
The result is right.
### Union query:
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Union all issue with regarding to windows function & aggregation on
>
>
> Key: KYLIN-4658
> URL: https://issues.apache.org/jira/browse/KYLIN-4658
> Project: Kylin
> Issue Type: Improvement
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>Priority: Major
>
> Test SQL:
> {code}
> select CNT, GMV, sum(GMV) over(partition by SLR_SEGMENT_CD) TOTAL_GMV,
> SLR_SEGMENT_CD, LSTG_FORMAT_NAME
> from
> (select sum(PRICE) GMV, sum(ITEM_COUNT) CNT, SLR_SEGMENT_CD, LSTG_FORMAT_NAME
> from TEST_KYLIN_FACT group by SLR_SEGMENT_CD, LSTG_FORMAT_NAME
> UNION ALL
> select sum(PRICE) GMV, sum(ITEM_COUNT) CNT, SLR_SEGMENT_CD, LSTG_FORMAT_NAME
> from TEST_KYLIN_FACT group by SLR_SEGMENT_CD, LSTG_FORMAT_NAME)
> order by TOTAL_GMV
> {code}
>
> Exception:
> {code}
> Index: 2, Size: 2 while executing SQL: "select * from (select CNT, GMV,
> sum(GMV) over(partition by SLR_SEGMENT_CD) TOTAL_GMV, SLR_SEGMENT_CD,
> LSTG_FORMAT_NAME from (select sum(PRICE) GMV, sum(ITEM_COUNT) CNT,
> SLR_SEGMENT_CD, LSTG_FORMAT_NAME from TEST_KYLIN_FACT group by
> SLR_SEGMENT_CD, LSTG_FORMAT_NAME UNION ALL select sum(PRICE) GMV,
> sum(ITEM_COUNT) CNT, SLR_SEGMENT_CD, LSTG_FORMAT_NAME from TEST_KYLIN_FACT
> group by SLR_SEGMENT_CD, LSTG_FORMAT_NAME) order by TOTAL_GMV) limit 5"
> {code}
> Similar issue for the following sql:
> {code}
> select LSTG_FORMAT_NAME,
>SLR_SEGMENT_CD,
>CAL_DT,
>sum(CNT) as CNT
> from
> (select LSTG_FORMAT_NAME,
> SLR_SEGMENT_CD,
> CAL_DT,
> sum(ITEM_COUNT) CNT
>from TEST_KYLIN_FACT
>where LSTG_FORMAT_NAME = 'ABIN'
>group by LSTG_FORMAT_NAME,
> SLR_SEGMENT_CD,
> CAL_DT
>UNION ALL select 'NON-ABIN' as LSTG_FORMAT_NAME,
> SLR_SEGMENT_CD,
> CAL_DT,
> case
> when SLR_SEGMENT_CD > 1000 then CNT * 2
> else CNT * 3
> end as CNT
>from
> (select SLR_SEGMENT_CD,
> CAL_DT,
> sum(ITEM_COUNT) CNT
> from TEST_KYLIN_FACT
> where LSTG_FORMAT_NAME <> 'ABIN'
> group by SLR_SEGMENT_CD,CAL_DT))
> group by LSTG_FORMAT_NAME,
> SLR_SEGMENT_CD,
> CAL_DT
> order by CNT
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [kylin] zzcclp commented on pull request #1526: KYLIN-4658 Fix union all issue with regarding to windows function & aggregation on
zzcclp commented on pull request #1526: URL: https://github.com/apache/kylin/pull/1526#issuecomment-752824642 Verified passed. LGTM. ## Before this pr ### Query with window function:  **The result is wrong.** ### Union query:  ## After this pr ### Query with window function:  The result is right. ### Union query:  This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[ https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17256805#comment-17256805 ] ASF GitHub Bot commented on KYLIN-4813: --- hit-lacus closed pull request #1525: URL: https://github.com/apache/kylin/pull/1525 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Refine spark logger for Kylin 4 build engine > > > Key: KYLIN-4813 > URL: https://issues.apache.org/jira/browse/KYLIN-4813 > Project: Kylin > Issue Type: Improvement >Affects Versions: v4.0.0-alpha >Reporter: Xiaoxiang Yu >Assignee: Yaqian Zhang >Priority: Major > Fix For: v4.0.0-beta > > > - Separate spark log from kylin log > - Store driver/executor log into HDFS. > - Provided a API to view driver log. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[ https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17256804#comment-17256804 ] ASF GitHub Bot commented on KYLIN-4813: --- hit-lacus commented on pull request #1525: URL: https://github.com/apache/kylin/pull/1525#issuecomment-752824252 It is better to double check the internal or add more test coverage. I will close this at the moment. Let's re-design this later. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Refine spark logger for Kylin 4 build engine > > > Key: KYLIN-4813 > URL: https://issues.apache.org/jira/browse/KYLIN-4813 > Project: Kylin > Issue Type: Improvement >Affects Versions: v4.0.0-alpha >Reporter: Xiaoxiang Yu >Assignee: Yaqian Zhang >Priority: Major > Fix For: v4.0.0-beta > > > - Separate spark log from kylin log > - Store driver/executor log into HDFS. > - Provided a API to view driver log. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [kylin] hit-lacus closed pull request #1525: Fix KYLIN-4813 Avoid putting into the map many times
hit-lacus closed pull request #1525: URL: https://github.com/apache/kylin/pull/1525 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kylin] hit-lacus commented on pull request #1525: Fix KYLIN-4813 Avoid putting into the map many times
hit-lacus commented on pull request #1525: URL: https://github.com/apache/kylin/pull/1525#issuecomment-752824252 It is better to double check the internal or add more test coverage. I will close this at the moment. Let's re-design this later. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KYLIN-4823) Push down having filter error when group by dynamic column
[
https://issues.apache.org/jira/browse/KYLIN-4823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17256802#comment-17256802
]
ASF GitHub Bot commented on KYLIN-4823:
---
hit-lacus closed pull request #1493:
URL: https://github.com/apache/kylin/pull/1493
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Push down having filter error when group by dynamic column
> ---
>
> Key: KYLIN-4823
> URL: https://issues.apache.org/jira/browse/KYLIN-4823
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v3.1.0
>Reporter: hcy
>Priority: Major
>
> 如果cube只有一个segment，且shard by的列存在于group by中，满足having filter push
> down的条件时，如果group by中存在动态列，并且case when then
> 中的表达是为column而不是常量时会报数组越界的错误。配置kylin.query.enable-dynamic-column=true无效，也会报错。
> 测试Cube如下:
> 模型为kylin
> example中的kylin_sales_model，cube为kylin_sales_cube，为了重现错误把BUYER_ID的rowkey设置为shard
> by
> 测试SQL如下:
> SELECT BUYER_ID,
> CASE WHEN LSTG_SITE_ID > 1 then LSTG_SITE_ID else LEAF_CATEG_ID END AS
> dyna_group,
> SUM(PRICE)
> FROM KYLIN_SALES
> GROUP BY
> BUYER_ID,
> CASE WHEN LSTG_SITE_ID > 1 then LSTG_SITE_ID else LEAF_CATEG_ID END
> HAVING SUM(PRICE)>10
> 报错如下：
> {color:#b94a48}Index: 4, Size: 1 while executing SQL: "select * from (SELECT
> BUYER_ID, CASE WHEN LSTG_SITE_ID > 1 then LSTG_SITE_ID else LEAF_CATEG_ID END
> AS dyna_group, SUM(PRICE) FROM KYLIN_SALES GROUP BY BUYER_ID, CASE WHEN
> LSTG_SITE_ID > 1 then LSTG_SITE_ID else LEAF_CATEG_ID END HAVING
> SUM(PRICE)>10) limit 5"{color}
> Caused by: java.lang.IndexOutOfBoundsException: Index: 4, Size: 1
> at java.util.ArrayList.rangeCheck(ArrayList.java:657)
> at java.util.ArrayList.get(ArrayList.java:433)
> at
> org.apache.kylin.storage.gtrecord.GTCubeStorageQueryBase.checkHavingCanPushDown(GTCubeStorageQueryBase.java:552)
> at
> org.apache.kylin.storage.gtrecord.GTCubeStorageQueryBase.getStorageQueryRequest(GTCubeStorageQueryBase.java:189)
> at
> org.apache.kylin.storage.gtrecord.GTCubeStorageQueryBase.search(GTCubeStorageQueryBase.java:89)
> at
> org.apache.kylin.query.enumerator.OLAPEnumerator.queryStorage(OLAPEnumerator.java:117)
> at
> org.apache.kylin.query.enumerator.OLAPEnumerator.moveNext(OLAPEnumerator.java:60)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-4823) Push down having filter error when group by dynamic column
[
https://issues.apache.org/jira/browse/KYLIN-4823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17256801#comment-17256801
]
ASF GitHub Bot commented on KYLIN-4823:
---
hit-lacus commented on pull request #1493:
URL: https://github.com/apache/kylin/pull/1493#issuecomment-752823570
Looks like there is another better solution. I will close this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Push down having filter error when group by dynamic column
> ---
>
> Key: KYLIN-4823
> URL: https://issues.apache.org/jira/browse/KYLIN-4823
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v3.1.0
>Reporter: hcy
>Priority: Major
>
> 如果cube只有一个segment，且shard by的列存在于group by中，满足having filter push
> down的条件时，如果group by中存在动态列，并且case when then
> 中的表达是为column而不是常量时会报数组越界的错误。配置kylin.query.enable-dynamic-column=true无效，也会报错。
> 测试Cube如下:
> 模型为kylin
> example中的kylin_sales_model，cube为kylin_sales_cube，为了重现错误把BUYER_ID的rowkey设置为shard
> by
> 测试SQL如下:
> SELECT BUYER_ID,
> CASE WHEN LSTG_SITE_ID > 1 then LSTG_SITE_ID else LEAF_CATEG_ID END AS
> dyna_group,
> SUM(PRICE)
> FROM KYLIN_SALES
> GROUP BY
> BUYER_ID,
> CASE WHEN LSTG_SITE_ID > 1 then LSTG_SITE_ID else LEAF_CATEG_ID END
> HAVING SUM(PRICE)>10
> 报错如下：
> {color:#b94a48}Index: 4, Size: 1 while executing SQL: "select * from (SELECT
> BUYER_ID, CASE WHEN LSTG_SITE_ID > 1 then LSTG_SITE_ID else LEAF_CATEG_ID END
> AS dyna_group, SUM(PRICE) FROM KYLIN_SALES GROUP BY BUYER_ID, CASE WHEN
> LSTG_SITE_ID > 1 then LSTG_SITE_ID else LEAF_CATEG_ID END HAVING
> SUM(PRICE)>10) limit 5"{color}
> Caused by: java.lang.IndexOutOfBoundsException: Index: 4, Size: 1
> at java.util.ArrayList.rangeCheck(ArrayList.java:657)
> at java.util.ArrayList.get(ArrayList.java:433)
> at
> org.apache.kylin.storage.gtrecord.GTCubeStorageQueryBase.checkHavingCanPushDown(GTCubeStorageQueryBase.java:552)
> at
> org.apache.kylin.storage.gtrecord.GTCubeStorageQueryBase.getStorageQueryRequest(GTCubeStorageQueryBase.java:189)
> at
> org.apache.kylin.storage.gtrecord.GTCubeStorageQueryBase.search(GTCubeStorageQueryBase.java:89)
> at
> org.apache.kylin.query.enumerator.OLAPEnumerator.queryStorage(OLAPEnumerator.java:117)
> at
> org.apache.kylin.query.enumerator.OLAPEnumerator.moveNext(OLAPEnumerator.java:60)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-4823) Push down having filter error when group by dynamic column
[
https://issues.apache.org/jira/browse/KYLIN-4823?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17256803#comment-17256803
]
ASF GitHub Bot commented on KYLIN-4823:
---
hit-lacus edited a comment on pull request #1493:
URL: https://github.com/apache/kylin/pull/1493#issuecomment-752823570
Looks like there is another better solution from yanghong. I will close this.
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Push down having filter error when group by dynamic column
> ---
>
> Key: KYLIN-4823
> URL: https://issues.apache.org/jira/browse/KYLIN-4823
> Project: Kylin
> Issue Type: Bug
> Components: Query Engine
>Affects Versions: v3.1.0
>Reporter: hcy
>Priority: Major
>
> 如果cube只有一个segment，且shard by的列存在于group by中，满足having filter push
> down的条件时，如果group by中存在动态列，并且case when then
> 中的表达是为column而不是常量时会报数组越界的错误。配置kylin.query.enable-dynamic-column=true无效，也会报错。
> 测试Cube如下:
> 模型为kylin
> example中的kylin_sales_model，cube为kylin_sales_cube，为了重现错误把BUYER_ID的rowkey设置为shard
> by
> 测试SQL如下:
> SELECT BUYER_ID,
> CASE WHEN LSTG_SITE_ID > 1 then LSTG_SITE_ID else LEAF_CATEG_ID END AS
> dyna_group,
> SUM(PRICE)
> FROM KYLIN_SALES
> GROUP BY
> BUYER_ID,
> CASE WHEN LSTG_SITE_ID > 1 then LSTG_SITE_ID else LEAF_CATEG_ID END
> HAVING SUM(PRICE)>10
> 报错如下：
> {color:#b94a48}Index: 4, Size: 1 while executing SQL: "select * from (SELECT
> BUYER_ID, CASE WHEN LSTG_SITE_ID > 1 then LSTG_SITE_ID else LEAF_CATEG_ID END
> AS dyna_group, SUM(PRICE) FROM KYLIN_SALES GROUP BY BUYER_ID, CASE WHEN
> LSTG_SITE_ID > 1 then LSTG_SITE_ID else LEAF_CATEG_ID END HAVING
> SUM(PRICE)>10) limit 5"{color}
> Caused by: java.lang.IndexOutOfBoundsException: Index: 4, Size: 1
> at java.util.ArrayList.rangeCheck(ArrayList.java:657)
> at java.util.ArrayList.get(ArrayList.java:433)
> at
> org.apache.kylin.storage.gtrecord.GTCubeStorageQueryBase.checkHavingCanPushDown(GTCubeStorageQueryBase.java:552)
> at
> org.apache.kylin.storage.gtrecord.GTCubeStorageQueryBase.getStorageQueryRequest(GTCubeStorageQueryBase.java:189)
> at
> org.apache.kylin.storage.gtrecord.GTCubeStorageQueryBase.search(GTCubeStorageQueryBase.java:89)
> at
> org.apache.kylin.query.enumerator.OLAPEnumerator.queryStorage(OLAPEnumerator.java:117)
> at
> org.apache.kylin.query.enumerator.OLAPEnumerator.moveNext(OLAPEnumerator.java:60)
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [kylin] hit-lacus edited a comment on pull request #1493: KYLIN-4823 Fix duplicate groupBy columns in OLAPAggregateRel
hit-lacus edited a comment on pull request #1493: URL: https://github.com/apache/kylin/pull/1493#issuecomment-752823570 Looks like there is another better solution from yanghong. I will close this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kylin] hit-lacus closed pull request #1493: KYLIN-4823 Fix duplicate groupBy columns in OLAPAggregateRel
hit-lacus closed pull request #1493: URL: https://github.com/apache/kylin/pull/1493 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kylin] hit-lacus commented on pull request #1493: KYLIN-4823 Fix duplicate groupBy columns in OLAPAggregateRel
hit-lacus commented on pull request #1493: URL: https://github.com/apache/kylin/pull/1493#issuecomment-752823570 Looks like there is another better solution. I will close this. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kylin] shaofengshi commented on pull request #1515: update find-xx-dependency.sh for macOS compatibility
shaofengshi commented on pull request #1515: URL: https://github.com/apache/kylin/pull/1515#issuecomment-752822929 Hi Kang, are you running Kylin service on MacOS? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KYLIN-4827) SparkMergingDictionary parallelize not work
[ https://issues.apache.org/jira/browse/KYLIN-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17256798#comment-17256798 ] ASF subversion and git services commented on KYLIN-4827: Commit 9a3e5a3c5ee7ae18cd2612eb2351b40905a63d99 in kylin's branch refs/heads/master from yangjiang [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=9a3e5a3 ] [KYLIN-4827] Fix SparkMergingDictionary parallelize not work. > SparkMergingDictionary parallelize not work > > > Key: KYLIN-4827 > URL: https://issues.apache.org/jira/browse/KYLIN-4827 > Project: Kylin > Issue Type: Improvement > Components: Spark Engine >Reporter: JiangYang >Assignee: JiangYang >Priority: Major > Fix For: v3.1.2 > > Attachments: Completed Stages (2) copy.png, Pasted Graphic 1.png, > Pasted Graphic 2.png, Stage Id.png, Storage Environment.png, > image-2020-11-25-12-46-05-750.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4827) SparkMergingDictionary parallelize not work
[ https://issues.apache.org/jira/browse/KYLIN-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17256797#comment-17256797 ] ASF GitHub Bot commented on KYLIN-4827: --- shaofengshi merged pull request #1488: URL: https://github.com/apache/kylin/pull/1488 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > SparkMergingDictionary parallelize not work > > > Key: KYLIN-4827 > URL: https://issues.apache.org/jira/browse/KYLIN-4827 > Project: Kylin > Issue Type: Improvement > Components: Spark Engine >Reporter: JiangYang >Assignee: JiangYang >Priority: Major > Fix For: v3.1.2 > > Attachments: Completed Stages (2) copy.png, Pasted Graphic 1.png, > Pasted Graphic 2.png, Stage Id.png, Storage Environment.png, > image-2020-11-25-12-46-05-750.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Resolved] (KYLIN-4827) SparkMergingDictionary parallelize not work
[ https://issues.apache.org/jira/browse/KYLIN-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi resolved KYLIN-4827. -- Resolution: Fixed > SparkMergingDictionary parallelize not work > > > Key: KYLIN-4827 > URL: https://issues.apache.org/jira/browse/KYLIN-4827 > Project: Kylin > Issue Type: Improvement > Components: Spark Engine >Reporter: JiangYang >Assignee: JiangYang >Priority: Major > Fix For: v3.1.2 > > Attachments: Completed Stages (2) copy.png, Pasted Graphic 1.png, > Pasted Graphic 2.png, Stage Id.png, Storage Environment.png, > image-2020-11-25-12-46-05-750.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Commented] (KYLIN-4827) SparkMergingDictionary parallelize not work
[ https://issues.apache.org/jira/browse/KYLIN-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17256795#comment-17256795 ] ASF GitHub Bot commented on KYLIN-4827: --- shaofengshi commented on pull request #1488: URL: https://github.com/apache/kylin/pull/1488#issuecomment-752822387 thanks Ted, it looks good to me! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > SparkMergingDictionary parallelize not work > > > Key: KYLIN-4827 > URL: https://issues.apache.org/jira/browse/KYLIN-4827 > Project: Kylin > Issue Type: Improvement > Components: Spark Engine >Reporter: JiangYang >Assignee: JiangYang >Priority: Major > Fix For: v3.1.2 > > Attachments: Completed Stages (2) copy.png, Pasted Graphic 1.png, > Pasted Graphic 2.png, Stage Id.png, Storage Environment.png, > image-2020-11-25-12-46-05-750.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Assigned] (KYLIN-4827) SparkMergingDictionary parallelize not work
[ https://issues.apache.org/jira/browse/KYLIN-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi reassigned KYLIN-4827: Assignee: JiangYang > SparkMergingDictionary parallelize not work > > > Key: KYLIN-4827 > URL: https://issues.apache.org/jira/browse/KYLIN-4827 > Project: Kylin > Issue Type: Improvement > Components: Spark Engine >Reporter: JiangYang >Assignee: JiangYang >Priority: Major > Fix For: v3.1.2 > > Attachments: Completed Stages (2) copy.png, Pasted Graphic 1.png, > Pasted Graphic 2.png, Stage Id.png, Storage Environment.png, > image-2020-11-25-12-46-05-750.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [kylin] shaofengshi merged pull request #1488: [KYLIN-4827] Fix SparkMergingDictionary parallelize not work.
shaofengshi merged pull request #1488: URL: https://github.com/apache/kylin/pull/1488 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[GitHub] [kylin] shaofengshi commented on pull request #1488: [KYLIN-4827] Fix SparkMergingDictionary parallelize not work.
shaofengshi commented on pull request #1488: URL: https://github.com/apache/kylin/pull/1488#issuecomment-752822387 thanks Ted, it looks good to me! This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Updated] (KYLIN-4827) SparkMergingDictionary parallelize not work
[ https://issues.apache.org/jira/browse/KYLIN-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shao Feng Shi updated KYLIN-4827: - Fix Version/s: v3.1.2 Issue Type: Improvement (was: Bug) > SparkMergingDictionary parallelize not work > > > Key: KYLIN-4827 > URL: https://issues.apache.org/jira/browse/KYLIN-4827 > Project: Kylin > Issue Type: Improvement > Components: Spark Engine >Reporter: JiangYang >Priority: Major > Fix For: v3.1.2 > > Attachments: Completed Stages (2) copy.png, Pasted Graphic 1.png, > Pasted Graphic 2.png, Stage Id.png, Storage Environment.png, > image-2020-11-25-12-46-05-750.png > > -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Issue Comment Deleted] (KYLIN-4682) java.lang.IndexOutOfBoundsException due to not setting havingFilter correctly
[
https://issues.apache.org/jira/browse/KYLIN-4682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhong Yanghong updated KYLIN-4682:
--
Comment: was deleted
(was: It seems rule *FilterAggregateTransposeRule* is not effective. It's
better to set a lower number for computing the cost of *OLAPFilterRel* to make
filter push down as much as possible.)
> java.lang.IndexOutOfBoundsException due to not setting havingFilter correctly
> -
>
> Key: KYLIN-4682
> URL: https://issues.apache.org/jira/browse/KYLIN-4682
> Project: Kylin
> Issue Type: Improvement
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>Priority: Major
>
> SQL:
> {code}
> select LSTG_FORMAT_NAME, LEAF_CATEG_ID, sum(price) as gmv
> from TEST_KYLIN_FACT
> group by LSTG_FORMAT_NAME, LEAF_CATEG_ID
> having LSTG_FORMAT_NAME = 'Auction'
> {code}
> Error stack trace:
> {code}
> Caused by: java.lang.IndexOutOfBoundsException: Index: 3, Size: 1
> at java.util.ArrayList.rangeCheck(ArrayList.java:657)
> at java.util.ArrayList.get(ArrayList.java:433)
> at
> org.apache.kylin.storage.gtrecord.GTCubeStorageQueryBase.checkHavingCanPushDown(GTCubeStorageQueryBase.java:553)
> at
> org.apache.kylin.storage.gtrecord.GTCubeStorageQueryBase.getStorageQueryRequest(GTCubeStorageQueryBase.java:196)
> at
> org.apache.kylin.storage.gtrecord.GTCubeStorageQueryBase.searchInner(GTCubeStorageQueryBase.java:98)
> at
> org.apache.kylin.storage.gtrecord.GTCubeStorageQueryBase.search(GTCubeStorageQueryBase.java:90)
> at
> org.apache.kylin.storage.hybrid.HybridStorageQuery.search(HybridStorageQuery.java:53)
> at
> org.apache.kylin.query.enumerator.OLAPEnumerator.queryStorage(OLAPEnumerator.java:117)
> at
> org.apache.kylin.query.enumerator.OLAPEnumerator.moveNext(OLAPEnumerator.java:60)
> at Baz$1$1.moveNext(Unknown Source)
> at
> org.apache.calcite.linq4j.EnumerableDefaults.groupBy_(EnumerableDefaults.java:825)
> at
> org.apache.calcite.linq4j.EnumerableDefaults.groupBy(EnumerableDefaults.java:761)
> at
> org.apache.calcite.linq4j.DefaultEnumerable.groupBy(DefaultEnumerable.java:302)
> at Baz.bind(Unknown Source)
> at
> org.apache.calcite.jdbc.CalcitePrepare$CalciteSignature.enumerable(CalcitePrepare.java:365)
> at
> org.apache.calcite.jdbc.CalciteConnectionImpl.enumerable(CalciteConnectionImpl.java:301)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl._createIterable(CalciteMetaImpl.java:559)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl.createIterable(CalciteMetaImpl.java:550)
> at
> org.apache.calcite.avatica.AvaticaResultSet.execute(AvaticaResultSet.java:182)
> at
> org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:67)
> at
> org.apache.calcite.jdbc.CalciteResultSet.execute(CalciteResultSet.java:44)
> at
> org.apache.calcite.avatica.AvaticaConnection$1.execute(AvaticaConnection.java:667)
> at
> org.apache.calcite.jdbc.CalciteMetaImpl.prepareAndExecute(CalciteMetaImpl.java:619)
> at
> org.apache.calcite.avatica.AvaticaConnection.prepareAndExecuteInternal(AvaticaConnection.java:675)
> at
> org.apache.calcite.avatica.AvaticaStatement.executeInternal(AvaticaStatement.java:156)
> ... 81 more
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[jira] [Commented] (KYLIN-4813) Refine spark logger for Kylin 4 build engine
[
https://issues.apache.org/jira/browse/KYLIN-4813?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17256772#comment-17256772
]
ASF GitHub Bot commented on KYLIN-4813:
---
zzcclp commented on a change in pull request #1525:
URL: https://github.com/apache/kylin/pull/1525#discussion_r550371200
##
File path:
core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
##
@@ -206,12 +207,15 @@ protected Properties getProperties(Collection
propertyKeys) {
}
protected StrSubstitutor getSubstitutor() {
-// env > properties
-final Map all = Maps.newHashMap();
-all.putAll((Map) properties);
-all.putAll(System.getenv());
+return new StrSubstitutor(getPropertiesMap());
+}
-return new StrSubstitutor(all);
+protected Map getPropertiesMap() {
+if (propertiesMap.isEmpty()) {
+propertiesMap.putAll((Map) properties);
+propertiesMap.putAll(System.getenv());
+}
Review comment:
When calling 'getPropertiesMap()' at the first time, it put all values
of properties and System.getenv() into propertiesMap, but if there are some
values be changed, the values in propertiesMap cann't be changed too?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Refine spark logger for Kylin 4 build engine
>
>
> Key: KYLIN-4813
> URL: https://issues.apache.org/jira/browse/KYLIN-4813
> Project: Kylin
> Issue Type: Improvement
>Affects Versions: v4.0.0-alpha
>Reporter: Xiaoxiang Yu
>Assignee: Yaqian Zhang
>Priority: Major
> Fix For: v4.0.0-beta
>
>
> - Separate spark log from kylin log
> - Store driver/executor log into HDFS.
> - Provided a API to view driver log.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [kylin] zzcclp commented on a change in pull request #1525: Fix KYLIN-4813 Avoid putting into the map many times
zzcclp commented on a change in pull request #1525:
URL: https://github.com/apache/kylin/pull/1525#discussion_r550371200
##
File path:
core-common/src/main/java/org/apache/kylin/common/KylinConfigBase.java
##
@@ -206,12 +207,15 @@ protected Properties getProperties(Collection
propertyKeys) {
}
protected StrSubstitutor getSubstitutor() {
-// env > properties
-final Map all = Maps.newHashMap();
-all.putAll((Map) properties);
-all.putAll(System.getenv());
+return new StrSubstitutor(getPropertiesMap());
+}
-return new StrSubstitutor(all);
+protected Map getPropertiesMap() {
+if (propertiesMap.isEmpty()) {
+propertiesMap.putAll((Map) properties);
+propertiesMap.putAll(System.getenv());
+}
Review comment:
When calling 'getPropertiesMap()' at the first time, it put all values
of properties and System.getenv() into propertiesMap, but if there are some
values be changed, the values in propertiesMap cann't be changed too?
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
[jira] [Commented] (KYLIN-4658) Union all issue with regarding to windows function & aggregation on
[
https://issues.apache.org/jira/browse/KYLIN-4658?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17256549#comment-17256549
]
ASF GitHub Bot commented on KYLIN-4658:
---
kyotoYaho opened a new pull request #1526:
URL: https://github.com/apache/kylin/pull/1526
## Proposed changes
Describe the big picture of your changes here to communicate to the
maintainers why we should accept this pull request. If it fixes a bug or
resolves a feature request, be sure to link to that issue.
## Types of changes
What types of changes does your code introduce to Kylin?
_Put an `x` in the boxes that apply_
- [ ] Bugfix (non-breaking change which fixes an issue)
- [ ] New feature (non-breaking change which adds functionality)
- [ ] Breaking change (fix or feature that would cause existing
functionality to not work as expected)
- [ ] Documentation Update (if none of the other choices apply)
## Checklist
_Put an `x` in the boxes that apply. You can also fill these out after
creating the PR. If you're unsure about any of them, don't hesitate to ask.
We're here to help! This is simply a reminder of what we are going to look for
before merging your code._
- [ ] I have create an issue on [Kylin's
jira](https://issues.apache.org/jira/browse/KYLIN), and have described the
bug/feature there in detail
- [ ] Commit messages in my PR start with the related jira ID, like
"KYLIN- Make Kylin project open-source"
- [ ] Compiling and unit tests pass locally with my changes
- [ ] I have added tests that prove my fix is effective or that my feature
works
- [ ] If this change need a document change, I will prepare another pr
against the `document` branch
- [ ] Any dependent changes have been merged
## Further comments
If this is a relatively large or complex change, kick off the discussion at
user@kylin or dev@kylin by explaining why you chose the solution you did and
what alternatives you considered, etc...
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Union all issue with regarding to windows function & aggregation on
>
>
> Key: KYLIN-4658
> URL: https://issues.apache.org/jira/browse/KYLIN-4658
> Project: Kylin
> Issue Type: Improvement
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>Priority: Major
>
> Test SQL:
> {code}
> select CNT, GMV, sum(GMV) over(partition by SLR_SEGMENT_CD) TOTAL_GMV,
> SLR_SEGMENT_CD, LSTG_FORMAT_NAME
> from
> (select sum(PRICE) GMV, sum(ITEM_COUNT) CNT, SLR_SEGMENT_CD, LSTG_FORMAT_NAME
> from TEST_KYLIN_FACT group by SLR_SEGMENT_CD, LSTG_FORMAT_NAME
> UNION ALL
> select sum(PRICE) GMV, sum(ITEM_COUNT) CNT, SLR_SEGMENT_CD, LSTG_FORMAT_NAME
> from TEST_KYLIN_FACT group by SLR_SEGMENT_CD, LSTG_FORMAT_NAME)
> order by TOTAL_GMV
> {code}
>
> Exception:
> {code}
> Index: 2, Size: 2 while executing SQL: "select * from (select CNT, GMV,
> sum(GMV) over(partition by SLR_SEGMENT_CD) TOTAL_GMV, SLR_SEGMENT_CD,
> LSTG_FORMAT_NAME from (select sum(PRICE) GMV, sum(ITEM_COUNT) CNT,
> SLR_SEGMENT_CD, LSTG_FORMAT_NAME from TEST_KYLIN_FACT group by
> SLR_SEGMENT_CD, LSTG_FORMAT_NAME UNION ALL select sum(PRICE) GMV,
> sum(ITEM_COUNT) CNT, SLR_SEGMENT_CD, LSTG_FORMAT_NAME from TEST_KYLIN_FACT
> group by SLR_SEGMENT_CD, LSTG_FORMAT_NAME) order by TOTAL_GMV) limit 5"
> {code}
> Similar issue for the following sql:
> {code}
> select LSTG_FORMAT_NAME,
>SLR_SEGMENT_CD,
>CAL_DT,
>sum(CNT) as CNT
> from
> (select LSTG_FORMAT_NAME,
> SLR_SEGMENT_CD,
> CAL_DT,
> sum(ITEM_COUNT) CNT
>from TEST_KYLIN_FACT
>where LSTG_FORMAT_NAME = 'ABIN'
>group by LSTG_FORMAT_NAME,
> SLR_SEGMENT_CD,
> CAL_DT
>UNION ALL select 'NON-ABIN' as LSTG_FORMAT_NAME,
> SLR_SEGMENT_CD,
> CAL_DT,
> case
> when SLR_SEGMENT_CD > 1000 then CNT * 2
> else CNT * 3
> end as CNT
>from
> (select SLR_SEGMENT_CD,
> CAL_DT,
> sum(ITEM_COUNT) CNT
> from TEST_KYLIN_FACT
> where LSTG_FORMAT_NAME <> 'ABIN'
> group by SLR_SEGMENT_CD,CAL_DT))
> group by LSTG_FORMAT_NAME,
> SLR_SEGMENT_CD,
> CAL_DT
> order by CNT
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
[GitHub] [kylin] kyotoYaho opened a new pull request #1526: KYLIN-4658 Fix union all issue with regarding to windows function & aggregation on
kyotoYaho opened a new pull request #1526: URL: https://github.com/apache/kylin/pull/1526 ## Proposed changes Describe the big picture of your changes here to communicate to the maintainers why we should accept this pull request. If it fixes a bug or resolves a feature request, be sure to link to that issue. ## Types of changes What types of changes does your code introduce to Kylin? _Put an `x` in the boxes that apply_ - [ ] Bugfix (non-breaking change which fixes an issue) - [ ] New feature (non-breaking change which adds functionality) - [ ] Breaking change (fix or feature that would cause existing functionality to not work as expected) - [ ] Documentation Update (if none of the other choices apply) ## Checklist _Put an `x` in the boxes that apply. You can also fill these out after creating the PR. If you're unsure about any of them, don't hesitate to ask. We're here to help! This is simply a reminder of what we are going to look for before merging your code._ - [ ] I have create an issue on [Kylin's jira](https://issues.apache.org/jira/browse/KYLIN), and have described the bug/feature there in detail - [ ] Commit messages in my PR start with the related jira ID, like "KYLIN- Make Kylin project open-source" - [ ] Compiling and unit tests pass locally with my changes - [ ] I have added tests that prove my fix is effective or that my feature works - [ ] If this change need a document change, I will prepare another pr against the `document` branch - [ ] Any dependent changes have been merged ## Further comments If this is a relatively large or complex change, kick off the discussion at user@kylin or dev@kylin by explaining why you chose the solution you did and what alternatives you considered, etc... This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KYLIN-3392) Support NULL value in Sum, Max, Min Aggregation
[ https://issues.apache.org/jira/browse/KYLIN-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17256509#comment-17256509 ] ASF GitHub Bot commented on KYLIN-3392: --- hit-lacus closed pull request #1524: URL: https://github.com/apache/kylin/pull/1524 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Support NULL value in Sum, Max, Min Aggregation > --- > > Key: KYLIN-3392 > URL: https://issues.apache.org/jira/browse/KYLIN-3392 > Project: Kylin > Issue Type: Bug >Reporter: Yifei Wu >Assignee: Yifei Wu >Priority: Major > Fix For: Future > > Attachments: KYLIN-3392-2.png, KYLIN-3392.png, kylin-3.0.0-alpha2.png > > > It is treated as 0 when confronted with NULL value in KYLIN's basic aggregate > measure (like sum, max, min). However, to distinguish the NULL value with 0 > is very necessary. > It should be like this > *sum(null, null) = null* > *sum(null, 1) = 1* > *max(null, null) = null* > *max(null, -1) = -1* > *min(null, -1）= -1* > in accordance with Hive and SparkSQL -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [kylin] hit-lacus closed pull request #1524: KYLIN-3392 Fix BufferUnderflowException read DecimalAggregator when it is null
hit-lacus closed pull request #1524: URL: https://github.com/apache/kylin/pull/1524 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KYLIN-3392) Support NULL value in Sum, Max, Min Aggregation
[ https://issues.apache.org/jira/browse/KYLIN-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17256438#comment-17256438 ] ASF GitHub Bot commented on KYLIN-3392: --- lgtm-com[bot] commented on pull request #1524: URL: https://github.com/apache/kylin/pull/1524#issuecomment-752409775 This pull request **introduces 1 alert** when merging 09d66680eb9bae85978ebe5afe8d8dc2a074400f into 7cca1c27cd3c2414f7f9b7061009b8a8b55069bf - [view on LGTM.com](https://lgtm.com/projects/g/apache/kylin/rev/pr-b7fc08c1d7413a1327c1a607a5467bf0516efa5b) **new alerts:** * 1 for Boxed variable is never null This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Support NULL value in Sum, Max, Min Aggregation > --- > > Key: KYLIN-3392 > URL: https://issues.apache.org/jira/browse/KYLIN-3392 > Project: Kylin > Issue Type: Bug >Reporter: Yifei Wu >Assignee: Yifei Wu >Priority: Major > Fix For: Future > > Attachments: KYLIN-3392-2.png, KYLIN-3392.png, kylin-3.0.0-alpha2.png > > > It is treated as 0 when confronted with NULL value in KYLIN's basic aggregate > measure (like sum, max, min). However, to distinguish the NULL value with 0 > is very necessary. > It should be like this > *sum(null, null) = null* > *sum(null, 1) = 1* > *max(null, null) = null* > *max(null, -1) = -1* > *min(null, -1）= -1* > in accordance with Hive and SparkSQL -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [kylin] lgtm-com[bot] commented on pull request #1524: KYLIN-3392 Fix BufferUnderflowException read DecimalAggregator when it is null
lgtm-com[bot] commented on pull request #1524: URL: https://github.com/apache/kylin/pull/1524#issuecomment-752409775 This pull request **introduces 1 alert** when merging 09d66680eb9bae85978ebe5afe8d8dc2a074400f into 7cca1c27cd3c2414f7f9b7061009b8a8b55069bf - [view on LGTM.com](https://lgtm.com/projects/g/apache/kylin/rev/pr-b7fc08c1d7413a1327c1a607a5467bf0516efa5b) **new alerts:** * 1 for Boxed variable is never null This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Commented] (KYLIN-3392) Support NULL value in Sum, Max, Min Aggregation
[ https://issues.apache.org/jira/browse/KYLIN-3392?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=17256399#comment-17256399 ] ASF GitHub Bot commented on KYLIN-3392: --- lgtm-com[bot] commented on pull request #1524: URL: https://github.com/apache/kylin/pull/1524#issuecomment-752392577 This pull request **introduces 1 alert** when merging 73c46e541c09d851c48f3bf937af79d325cb737f into 7cca1c27cd3c2414f7f9b7061009b8a8b55069bf - [view on LGTM.com](https://lgtm.com/projects/g/apache/kylin/rev/pr-5b0e6256a8d7d6031dc814185d9ab4c7898c3e97) **new alerts:** * 1 for Boxed variable is never null This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Support NULL value in Sum, Max, Min Aggregation > --- > > Key: KYLIN-3392 > URL: https://issues.apache.org/jira/browse/KYLIN-3392 > Project: Kylin > Issue Type: Bug >Reporter: Yifei Wu >Assignee: Yifei Wu >Priority: Major > Fix For: Future > > Attachments: KYLIN-3392-2.png, KYLIN-3392.png, kylin-3.0.0-alpha2.png > > > It is treated as 0 when confronted with NULL value in KYLIN's basic aggregate > measure (like sum, max, min). However, to distinguish the NULL value with 0 > is very necessary. > It should be like this > *sum(null, null) = null* > *sum(null, 1) = 1* > *max(null, null) = null* > *max(null, -1) = -1* > *min(null, -1）= -1* > in accordance with Hive and SparkSQL -- This message was sent by Atlassian Jira (v8.3.4#803005)
[GitHub] [kylin] lgtm-com[bot] commented on pull request #1524: KYLIN-3392 Fix BufferUnderflowException read DecimalAggregator when it is null
lgtm-com[bot] commented on pull request #1524: URL: https://github.com/apache/kylin/pull/1524#issuecomment-752392577 This pull request **introduces 1 alert** when merging 73c46e541c09d851c48f3bf937af79d325cb737f into 7cca1c27cd3c2414f7f9b7061009b8a8b55069bf - [view on LGTM.com](https://lgtm.com/projects/g/apache/kylin/rev/pr-5b0e6256a8d7d6031dc814185d9ab4c7898c3e97) **new alerts:** * 1 for Boxed variable is never null This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org
[jira] [Assigned] (KYLIN-4658) Union all issue with regarding to windows function & aggregation on
[
https://issues.apache.org/jira/browse/KYLIN-4658?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Zhong Yanghong reassigned KYLIN-4658:
-
Assignee: Zhong Yanghong (was: JiangYang)
> Union all issue with regarding to windows function & aggregation on
>
>
> Key: KYLIN-4658
> URL: https://issues.apache.org/jira/browse/KYLIN-4658
> Project: Kylin
> Issue Type: Improvement
>Reporter: Zhong Yanghong
>Assignee: Zhong Yanghong
>Priority: Major
>
> Test SQL:
> {code}
> select CNT, GMV, sum(GMV) over(partition by SLR_SEGMENT_CD) TOTAL_GMV,
> SLR_SEGMENT_CD, LSTG_FORMAT_NAME
> from
> (select sum(PRICE) GMV, sum(ITEM_COUNT) CNT, SLR_SEGMENT_CD, LSTG_FORMAT_NAME
> from TEST_KYLIN_FACT group by SLR_SEGMENT_CD, LSTG_FORMAT_NAME
> UNION ALL
> select sum(PRICE) GMV, sum(ITEM_COUNT) CNT, SLR_SEGMENT_CD, LSTG_FORMAT_NAME
> from TEST_KYLIN_FACT group by SLR_SEGMENT_CD, LSTG_FORMAT_NAME)
> order by TOTAL_GMV
> {code}
>
> Exception:
> {code}
> Index: 2, Size: 2 while executing SQL: "select * from (select CNT, GMV,
> sum(GMV) over(partition by SLR_SEGMENT_CD) TOTAL_GMV, SLR_SEGMENT_CD,
> LSTG_FORMAT_NAME from (select sum(PRICE) GMV, sum(ITEM_COUNT) CNT,
> SLR_SEGMENT_CD, LSTG_FORMAT_NAME from TEST_KYLIN_FACT group by
> SLR_SEGMENT_CD, LSTG_FORMAT_NAME UNION ALL select sum(PRICE) GMV,
> sum(ITEM_COUNT) CNT, SLR_SEGMENT_CD, LSTG_FORMAT_NAME from TEST_KYLIN_FACT
> group by SLR_SEGMENT_CD, LSTG_FORMAT_NAME) order by TOTAL_GMV) limit 5"
> {code}
> Similar issue for the following sql:
> {code}
> select LSTG_FORMAT_NAME,
>SLR_SEGMENT_CD,
>CAL_DT,
>sum(CNT) as CNT
> from
> (select LSTG_FORMAT_NAME,
> SLR_SEGMENT_CD,
> CAL_DT,
> sum(ITEM_COUNT) CNT
>from TEST_KYLIN_FACT
>where LSTG_FORMAT_NAME = 'ABIN'
>group by LSTG_FORMAT_NAME,
> SLR_SEGMENT_CD,
> CAL_DT
>UNION ALL select 'NON-ABIN' as LSTG_FORMAT_NAME,
> SLR_SEGMENT_CD,
> CAL_DT,
> case
> when SLR_SEGMENT_CD > 1000 then CNT * 2
> else CNT * 3
> end as CNT
>from
> (select SLR_SEGMENT_CD,
> CAL_DT,
> sum(ITEM_COUNT) CNT
> from TEST_KYLIN_FACT
> where LSTG_FORMAT_NAME <> 'ABIN'
> group by SLR_SEGMENT_CD,CAL_DT))
> group by LSTG_FORMAT_NAME,
> SLR_SEGMENT_CD,
> CAL_DT
> order by CNT
> {code}
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
