[GitHub] [kylin] codecov-io commented on issue #727: KYLIN-3628 Fix unexpected exception for select from Lookup Table
codecov-io commented on issue #727: KYLIN-3628 Fix unexpected exception for select from Lookup Table URL: https://github.com/apache/kylin/pull/727#issuecomment-507928724 # [Codecov](https://codecov.io/gh/apache/kylin/pull/727?src=pr=h1) Report > :exclamation: No coverage uploaded for pull request base (`master@7b1927c`). [Click here to learn what that means](https://docs.codecov.io/docs/error-reference#section-missing-base-commit). > The diff coverage is `0%`. [](https://codecov.io/gh/apache/kylin/pull/727?src=pr=tree) ```diff @@Coverage Diff@@ ## master #727 +/- ## = Coverage ? 25.78% Complexity? 6039 = Files ? 1392 Lines ?82821 Branches ?11607 = Hits ?21354 Misses?59413 Partials ? 2054 ``` | [Impacted Files](https://codecov.io/gh/apache/kylin/pull/727?src=pr=tree) | Coverage Δ | Complexity Δ | | |---|---|---|---| | [.../kylin/query/enumerator/LookupTableEnumerator.java](https://codecov.io/gh/apache/kylin/pull/727/diff?src=pr=tree#diff-cXVlcnkvc3JjL21haW4vamF2YS9vcmcvYXBhY2hlL2t5bGluL3F1ZXJ5L2VudW1lcmF0b3IvTG9va3VwVGFibGVFbnVtZXJhdG9yLmphdmE=) | `0% <0%> (ø)` | `0 <0> (?)` | | | [...c/main/java/org/apache/kylin/cube/CubeManager.java](https://codecov.io/gh/apache/kylin/pull/727/diff?src=pr=tree#diff-Y29yZS1jdWJlL3NyYy9tYWluL2phdmEvb3JnL2FwYWNoZS9reWxpbi9jdWJlL0N1YmVNYW5hZ2VyLmphdmE=) | `36.97% <0%> (ø)` | `44 <0> (?)` | | -- [Continue to review full report at Codecov](https://codecov.io/gh/apache/kylin/pull/727?src=pr=continue). > **Legend** - [Click here to learn more](https://docs.codecov.io/docs/codecov-delta) > `Δ = absolute (impact)`, `ø = not affected`, `? = missing data` > Powered by [Codecov](https://codecov.io/gh/apache/kylin/pull/727?src=pr=footer). Last update [7b1927c...542a655](https://codecov.io/gh/apache/kylin/pull/727?src=pr=lastupdated). Read the [comment docs](https://docs.codecov.io/docs/pull-request-comments). This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-4070) No logs print in kylin.log file and met error "Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster"
[
https://issues.apache.org/jira/browse/KYLIN-4070?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877452#comment-16877452
]
XiaoXiang Yu commented on KYLIN-4070:
-
Actually JIRA is not suitable for asking question, mailing list will be better
place. And I think we don't need to modify kylin.sh, and do you have try execute
{quote}sh bin/find-hive-dependency.sh
{quote}
does hive_dependency be configured correctly?
> No logs print in kylin.log file and met error "Error: Could not find or load
> main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster"
> ---
>
> Key: KYLIN-4070
> URL: https://issues.apache.org/jira/browse/KYLIN-4070
> Project: Kylin
> Issue Type: Bug
> Environment: apache-kylin-2.6.2-bin-hbase1x hadoop-2.7.7
> hbase-1.2.11hive-2.3.4
>Reporter: xiangakun
>Priority: Blocker
>
> Hello friends,
>
> I am using kylin with version 2.6.2, and the relied components are lists
> below, I start the kylin server successfully and met the error
> "org.apache.kylin.rest.exception.InternalErrorException:
> org/apache/hadoop/hive/conf/HiveConf" when loading hive meta data. I updated
> the configuration from "export
> HBASE_CLASSPATH_PREFIX=${KYLIN_HOME}/conf:${KYLIN_HOME}/lib/*:${KYLIN_HOME}/ext/*:${HBASE_CLASSPATH_PREFIX}"
> to "export
> HBASE_CLASSPATH_PREFIX=${KYLIN_HOME}/conf:${KYLIN_HOME}/lib/*:${KYLIN_HOME}/ext/*:{color:#FF}${hive_dependency}{color}:${HBASE_CLASSPATH_PREFIX}"
> in kylin.sh file. it did work fine. However, after this action, I found no
> more messages could be produced in logs/kylin.log file. I removed kylin.log
> file, and after restarted the kylin server, kylin.log file was not created
> any more. It's seems too stange.
> What's more, when I create a cube and build it, the error happens in step 3,
> it shows "
> Error: Could not find or load main class
> org.apache.hadoop.mapreduce.v2.app.MRAppMaster".
>
> In fact, I used the same configuration in yarn-site.xml and mapred-site.xml
> files with kylin version 2.1.0 for a long time, it worked fine and never met
> such error.
>
> Does anyone meet the same problem before? Hope receive your kindly help soon.
>
> Thanks in advance.
>
> Best Regards
> xiangakun
>
>
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Closed] (KYLIN-4059) metrics write to hive error
[ https://issues.apache.org/jira/browse/KYLIN-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu closed KYLIN-4059. --- Resolution: Duplicate Close this issue temporarily. > metrics write to hive error > --- > > Key: KYLIN-4059 > URL: https://issues.apache.org/jira/browse/KYLIN-4059 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.6.2 >Reporter: chenchen >Assignee: XiaoXiang Yu >Priority: Major > Attachments: 501561537981_.pic_hd.jpg, 511561539223_.pic.jpg > > > my hadoop version is 2.7.1 > > I configure cube-planer and other metrics。When the query QPS of the machine > is relatively high, the following error will occur(Console Display metrics > Data Write Hive Failure)。 > By looking at the source code, kylin triggers an event to notify hiveProducer > to write data for each query. According to common sense, hive is not suitable > for dealing with this high frequency writing. > I wonder what the solution is? > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-4070) No logs print in kylin.log file and met error "Error: Could not find or load main class org.apache.hadoop.mapreduce.v2.app.MRAppMaster"
xiangakun created KYLIN-4070:
Summary: No logs print in kylin.log file and met error "Error:
Could not find or load main class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster"
Key: KYLIN-4070
URL: https://issues.apache.org/jira/browse/KYLIN-4070
Project: Kylin
Issue Type: Bug
Environment: apache-kylin-2.6.2-bin-hbase1x hadoop-2.7.7
hbase-1.2.11hive-2.3.4
Reporter: xiangakun
Hello friends,
I am using kylin with version 2.6.2, and the relied components are lists below,
I start the kylin server successfully and met the error
"org.apache.kylin.rest.exception.InternalErrorException:
org/apache/hadoop/hive/conf/HiveConf" when loading hive meta data. I updated
the configuration from "export
HBASE_CLASSPATH_PREFIX=${KYLIN_HOME}/conf:${KYLIN_HOME}/lib/*:${KYLIN_HOME}/ext/*:${HBASE_CLASSPATH_PREFIX}"
to "export
HBASE_CLASSPATH_PREFIX=${KYLIN_HOME}/conf:${KYLIN_HOME}/lib/*:${KYLIN_HOME}/ext/*:{color:#FF}${hive_dependency}{color}:${HBASE_CLASSPATH_PREFIX}"
in kylin.sh file. it did work fine. However, after this action, I found no
more messages could be produced in logs/kylin.log file. I removed kylin.log
file, and after restarted the kylin server, kylin.log file was not created any
more. It's seems too stange.
What's more, when I create a cube and build it, the error happens in step 3, it
shows "
Error: Could not find or load main class
org.apache.hadoop.mapreduce.v2.app.MRAppMaster".
In fact, I used the same configuration in yarn-site.xml and mapred-site.xml
files with kylin version 2.1.0 for a long time, it worked fine and never met
such error.
Does anyone meet the same problem before? Hope receive your kindly help soon.
Thanks in advance.
Best Regards
xiangakun
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Commented] (KYLIN-4059) metrics write to hive error
[ https://issues.apache.org/jira/browse/KYLIN-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877441#comment-16877441 ] XiaoXiang Yu commented on KYLIN-4059: - But I find this issue is duplicated with https://issues.apache.org/jira/browse/KYLIN-4026. > metrics write to hive error > --- > > Key: KYLIN-4059 > URL: https://issues.apache.org/jira/browse/KYLIN-4059 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.6.2 >Reporter: chenchen >Assignee: XiaoXiang Yu >Priority: Major > Attachments: 501561537981_.pic_hd.jpg, 511561539223_.pic.jpg > > > my hadoop version is 2.7.1 > > I configure cube-planer and other metrics。When the query QPS of the machine > is relatively high, the following error will occur(Console Display metrics > Data Write Hive Failure)。 > By looking at the source code, kylin triggers an event to notify hiveProducer > to write data for each query. According to common sense, hive is not suitable > for dealing with this high frequency writing. > I wonder what the solution is? > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-4069) HivePushDownConverter.doConvert will change sql semantics in some scenarios
weibin0516 created KYLIN-4069:
-
Summary: HivePushDownConverter.doConvert will change sql semantics
in some scenarios
Key: KYLIN-4069
URL: https://issues.apache.org/jira/browse/KYLIN-4069
Project: Kylin
Issue Type: Bug
Components: Query Engine
Affects Versions: v2.6.2
Reporter: weibin0516
Assignee: weibin0516
HivePushDownConverter.doConvert source code is as follows:
{code:java}
public static String doConvert(String originStr, boolean isPrepare) {
// Step1.Replace " with `
String convertedSql = replaceString(originStr, "\"", "`");
// Step2.Replace extract functions
convertedSql = extractReplace(convertedSql);
// Step3.Replace cast type string
convertedSql = castReplace(convertedSql);
// Step4.Replace sub query
convertedSql = subqueryReplace(convertedSql);
// Step5.Replace char_length with length
convertedSql = replaceString(convertedSql, "CHAR_LENGTH", "LENGTH");
convertedSql = replaceString(convertedSql, "char_length", "length");
// Step6.Replace "||" with concat
convertedSql = concatReplace(convertedSql);
// Step7.Add quote for interval in timestampadd
convertedSql = timestampAddDiffReplace(convertedSql);
// Step8.Replace integer with int
convertedSql = replaceString(convertedSql, "INTEGER", "INT");
convertedSql = replaceString(convertedSql, "integer", "int");
// Step9.Add limit 1 for prepare select sql to speed up
if (isPrepare) {
convertedSql = addLimit(convertedSql);
}
return convertedSql;
}
{code}
It is not advisable to directly replace the sql text. The following example
will convert sql to another error sql:
{code:sql}
SELECT "CHAR_LENGTH" FROM datasource.a
{code}
will convert to
{code:sql}
SELECT `LENGTH` FROM datasource.a
{code}
Every use of replaceString in doConvert will cause such problems.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-4068) Automatically add limit has bug
weibin0516 created KYLIN-4068:
-
Summary: Automatically add limit has bug
Key: KYLIN-4068
URL: https://issues.apache.org/jira/browse/KYLIN-4068
Project: Kylin
Issue Type: Bug
Components: Query Engine
Affects Versions: v2.6.2
Reporter: weibin0516
Assignee: weibin0516
{code:sql}
SELECT E_Name FROM Employees_China
UNION
SELECT E_Name FROM Employees_USA
{code}
will convert to
{code:sql}
SELECT E_Name FROM Employees_China
UNION
SELECT E_Name FROM Employees_USA
LIMIT 5
{code}
This limit is not working on the result of union, but on SELECT E_Name FROM
Employees_USA.
We should use a more secure way to achieve the limit effect.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [kylin] hit-lacus opened a new pull request #727: KYLIN-3628 Fix unexpected exception for select from Lookup Table
hit-lacus opened a new pull request #727: KYLIN-3628 Fix unexpected exception for select from Lookup Table URL: https://github.com/apache/kylin/pull/727 In some case, cube returned by findLatestSnapshot didn't contains the snapshot table for what you need. Because some dimension which exists in model but was removed in cube design phase, query such as "select * from LookupTable" will throw unexpected exception and confused user. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] asfgit commented on issue #727: KYLIN-3628 Fix unexpected exception for select from Lookup Table
asfgit commented on issue #727: KYLIN-3628 Fix unexpected exception for select from Lookup Table URL: https://github.com/apache/kylin/pull/727#issuecomment-507911993 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3628) Query with lookup table always use latest snapshot
[ https://issues.apache.org/jira/browse/KYLIN-3628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877432#comment-16877432 ] ASF GitHub Bot commented on KYLIN-3628: --- hit-lacus commented on pull request #727: KYLIN-3628 Fix unexpected exception for select from Lookup Table URL: https://github.com/apache/kylin/pull/727 In some case, cube returned by findLatestSnapshot didn't contains the snapshot table for what you need. Because some dimension which exists in model but was removed in cube design phase, query such as "select * from LookupTable" will throw unexpected exception and confused user. This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > Query with lookup table always use latest snapshot > -- > > Key: KYLIN-3628 > URL: https://issues.apache.org/jira/browse/KYLIN-3628 > Project: Kylin > Issue Type: Improvement >Reporter: Na Zhai >Assignee: Na Zhai >Priority: Major > Fix For: v2.6.0 > > > If user queries a lookup table, Kylin will randomly selects a Cube (which has > the snapshot of this lookup table) to answer it. This causes uncertainty when > there are multiple cubes (share the same lookup): some cubes are newly built, > some not. If Kylin picks an old cube, the query result is old. > To remove this uncertainty, for such queries, either always use latest > snapshot, or use earlist snapshot. We believe the "latest" version is better. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-4058) HBase tableName is case sensitive, so remove htableName toUpperCase()
[ https://issues.apache.org/jira/browse/KYLIN-4058?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16873052#comment-16873052 ] liuzhixin edited comment on KYLIN-4058 at 7/3/19 2:21 AM: -- refer-pr: [https://github.com/apache/kylin/pull/717] was (Author: zhixin): refer: [https://github.com/apache/kylin/pull/717] > HBase tableName is case sensitive, so remove htableName toUpperCase() > - > > Key: KYLIN-4058 > URL: https://issues.apache.org/jira/browse/KYLIN-4058 > Project: Kylin > Issue Type: Bug > Components: Storage - HBase >Reporter: liuzhixin >Priority: Major > > *Description* > HBase is case sensitive，tableName.toUpperCase() will lead to the created > table could not be found. > If my tableName of HBase is lowercase, kylin can't find the table created in > the HFile generated step. > *Improvement* > just remove toUpperCase() in CubeHFileJob for HBase tableName on > master-hadoop3.1. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [kylin] liuzx32 commented on issue #717: Bug fix: remove htableName toUpperCase, for HBase tableName is case sensitive
liuzx32 commented on issue #717: Bug fix: remove htableName toUpperCase, for HBase tableName is case sensitive URL: https://github.com/apache/kylin/pull/717#issuecomment-507911659 refer-jira: https://issues.apache.org/jira/browse/KYLIN-4058 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-4064) parameter 'engineType' is not working when running integration test
[ https://issues.apache.org/jira/browse/KYLIN-4064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877424#comment-16877424 ] ASF subversion and git services commented on KYLIN-4064: Commit 14382672ae8929d557bd2deb49fbd8d83b2ac544 in kylin's branch refs/heads/engine-flink from chao long [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=1438267 ] KYLIN-4064 Make param engineType working in CI > parameter 'engineType' is not working when running integration test > --- > > Key: KYLIN-4064 > URL: https://issues.apache.org/jira/browse/KYLIN-4064 > Project: Kylin > Issue Type: Bug > Components: Integration >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v3.0.0-beta > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-4057) autoMerge job can not stop
[ https://issues.apache.org/jira/browse/KYLIN-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877419#comment-16877419 ] XiaoXiang Yu commented on KYLIN-4057: - Hi, could you please share with us about how to reproduce this error? > autoMerge job can not stop > -- > > Key: KYLIN-4057 > URL: https://issues.apache.org/jira/browse/KYLIN-4057 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.6.2 >Reporter: chenchen >Assignee: 王汝鹏 >Priority: Major > Attachments: 5031561517158_.pic_hd.jpg > > > In this version，I used DistributedScheduler。 > In my case, A cube config autoMerge, But the job that automatically merges > segments is wrong ，I can't fix this error, I can only discard and drop the > job, but Kylin's monitoring mechanism creates the same new job. > I want to ask if there is any way to fix this problem without banning cube. > The error is reported as follows. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-4057) autoMerge job can not stop
[ https://issues.apache.org/jira/browse/KYLIN-4057?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877419#comment-16877419 ] XiaoXiang Yu edited comment on KYLIN-4057 at 7/3/19 2:04 AM: - Hi, could you please share with us about how to reproduce this error? [~chenfeng_0829] was (Author: hit_lacus): Hi, could you please share with us about how to reproduce this error? > autoMerge job can not stop > -- > > Key: KYLIN-4057 > URL: https://issues.apache.org/jira/browse/KYLIN-4057 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.6.2 >Reporter: chenchen >Assignee: 王汝鹏 >Priority: Major > Attachments: 5031561517158_.pic_hd.jpg > > > In this version，I used DistributedScheduler。 > In my case, A cube config autoMerge, But the job that automatically merges > segments is wrong ，I can't fix this error, I can only discard and drop the > job, but Kylin's monitoring mechanism creates the same new job. > I want to ask if there is any way to fix this problem without banning cube. > The error is reported as follows. > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-4064) parameter 'engineType' is not working when running integration test
[ https://issues.apache.org/jira/browse/KYLIN-4064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877416#comment-16877416 ] ASF GitHub Bot commented on KYLIN-4064: --- nichunen commented on pull request #726: KYLIN-4064 Make param engineType working in CI URL: https://github.com/apache/kylin/pull/726 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > parameter 'engineType' is not working when running integration test > --- > > Key: KYLIN-4064 > URL: https://issues.apache.org/jira/browse/KYLIN-4064 > Project: Kylin > Issue Type: Bug > Components: Integration >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v3.0.0-beta > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-4064) parameter 'engineType' is not working when running integration test
[ https://issues.apache.org/jira/browse/KYLIN-4064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877417#comment-16877417 ] ASF subversion and git services commented on KYLIN-4064: Commit 7b1927c709546c8982d234f5ffa1c199fb9d46c5 in kylin's branch refs/heads/master from chao long [ https://gitbox.apache.org/repos/asf?p=kylin.git;h=7b1927c ] KYLIN-4064 Make param engineType working in CI > parameter 'engineType' is not working when running integration test > --- > > Key: KYLIN-4064 > URL: https://issues.apache.org/jira/browse/KYLIN-4064 > Project: Kylin > Issue Type: Bug > Components: Integration >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v3.0.0-beta > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [kylin] nichunen merged pull request #726: KYLIN-4064 Make param engineType working in CI
nichunen merged pull request #726: KYLIN-4064 Make param engineType working in CI URL: https://github.com/apache/kylin/pull/726 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-3628) Query with lookup table always use latest snapshot
[
https://issues.apache.org/jira/browse/KYLIN-3628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16877286#comment-16877286
]
Vsevolod Ostapenko commented on KYLIN-3628:
---
[~hit_lacus], I build local 2.6.2 with your changes applied on top and
"derived" column check removed.
It seems to work as intended.
One suggestion though, debug message about overriding cube selection in the
findLatestSnapshot() should be logged only when cube selection was really
overridden. For example:
{code:java}
if (!cube.equals(cubeInstance)) {
logger.debug("Picked cube {} over {} as it provides a more recent snapshot of
the lookup table {}", cube, cubeInstance, lookupTableName);
}
{code}
> Query with lookup table always use latest snapshot
> --
>
> Key: KYLIN-3628
> URL: https://issues.apache.org/jira/browse/KYLIN-3628
> Project: Kylin
> Issue Type: Improvement
>Reporter: Na Zhai
>Assignee: Na Zhai
>Priority: Major
> Fix For: v2.6.0
>
>
> If user queries a lookup table, Kylin will randomly selects a Cube (which has
> the snapshot of this lookup table) to answer it. This causes uncertainty when
> there are multiple cubes (share the same lookup): some cubes are newly built,
> some not. If Kylin picks an old cube, the query result is old.
> To remove this uncertainty, for such queries, either always use latest
> snapshot, or use earlist snapshot. We believe the "latest" version is better.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Comment Edited] (KYLIN-3628) Query with lookup table always use latest snapshot
[
https://issues.apache.org/jira/browse/KYLIN-3628?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876282#comment-16876282
]
Vsevolod Ostapenko edited comment on KYLIN-3628 at 7/2/19 7:50 PM:
---
Hi XiaoXiang, thank you for a quick turnaround with a fix.
I looked at the code change, and the following check seems too restrictive:
{code:java}
if (dimensionDesc.isDerived() &&
dimensionDesc.getTable().equalsIgnoreCase(lookupTbl)) {
{code}
Per my internal tests, the entire lookup table is being snapshotted as part of
a cube, if any dimension (non-derived or otherwise) is supplied by that table.
So, dimension doesn't have to be derived.
In fact, in my test cubes there is no derived dimensions at all (all "derived"
properties for all the lookup tables are set to null), it does not prevent
executing "select * from lookupTable" against such cube on a non-patched 2.6.2
system.
To summarize, I believe that the "dimensionDesc.isDerived()" call should be
removed from the expression above.
Edit: I have to stay corrected, for the lookup table only the columns that are
normal or derived dimensions are being snapshotted (not the entire table).
Still, it doesn't change the stance that the "derived" check is not needed in
the expression above.
was (Author: seva_ostapenko):
Hi XiaoXiang, thank you for a quick turnaround with a fix.
I looked at the code change, and the following check seems too restrictive:
{code:java}
if (dimensionDesc.isDerived() &&
dimensionDesc.getTable().equalsIgnoreCase(lookupTbl)) {
{code}
Per my internal tests, the entire lookup table is being snapshotted as part of
a cube, if any dimension (non-derived or otherwise) is supplied by that table.
So, dimension doesn't have to be derived.
In fact, in my test cubes there is no derived dimensions at all (all "derived"
properties for all the lookup tables are set to null), it does not prevent
executing "select * from lookupTable" against such cube on a non-patched 2.6.2
system.
To summarize, I believe that the "dimensionDesc.isDerived()" call should be
removed from the expression above.
> Query with lookup table always use latest snapshot
> --
>
> Key: KYLIN-3628
> URL: https://issues.apache.org/jira/browse/KYLIN-3628
> Project: Kylin
> Issue Type: Improvement
>Reporter: Na Zhai
>Assignee: Na Zhai
>Priority: Major
> Fix For: v2.6.0
>
>
> If user queries a lookup table, Kylin will randomly selects a Cube (which has
> the snapshot of this lookup table) to answer it. This causes uncertainty when
> there are multiple cubes (share the same lookup): some cubes are newly built,
> some not. If Kylin picks an old cube, the query result is old.
> To remove this uncertainty, for such queries, either always use latest
> snapshot, or use earlist snapshot. We believe the "latest" version is better.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-4067) Speed up response of kylin cube page
zhao jintao created KYLIN-4067: -- Summary: Speed up response of kylin cube page Key: KYLIN-4067 URL: https://issues.apache.org/jira/browse/KYLIN-4067 Project: Kylin Issue Type: Improvement Components: Web Affects Versions: v2.6.2 Reporter: zhao jintao Assignee: zhao jintao Fix For: Future Hi Team： My Kylin project has more than 100 cubes. It is very slowly when open kylin web page. I find that kylin load all information of cubes when open cube page. It load all information of all cubes at every cube in kylin project. The url of request is "http://ip:port/kylin/api/cubes?limit=65535=0;. For example, if one project has 10 cubes, this request will be called by 10 times. But this information is only be used to determine whether the name is duplicated when adding a new cube. This page loading mechanism can be optimized. Getting all the information of all cube only needs to be called when adding a new cube. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-4066) No planner for not ROLE_ADMIN user on WebSite
langdamao created KYLIN-4066:
Summary: No planner for not ROLE_ADMIN user on WebSite
Key: KYLIN-4066
URL: https://issues.apache.org/jira/browse/KYLIN-4066
Project: Kylin
Issue Type: Bug
Components: Website
Affects Versions: v2.6.1
Environment: macOs
Reporter: langdamao
Fix For: Future
There is no planner page for none ROLE_ADMIN user ,even though he has ADMIN
access for the project.
I found in webapp/app/partials/cubes/cube_detail.html line44
{code:java}
Planner
{code}
but other place the hasPermission function use as
{code:java}
hasPermission('cube',cube, permissions.ADMINISTRATION.mask)
{code}
the function hasPermission is defined in webapp/app/js/controllers/page.js
line100
{code:java}
$scope.hasPermission = function (accessType, entity){
...
if (accessType === 'cube') {
project = entity.project
} else if (accessType === 'project') {
project = entity && entity.name || entity.selectedProject
} else if (accessType === 'model') {
project = ProjectModel.getProjectByCubeModel(entity.name)
}
...
}{code}
I guess whether it's cause by no param of accessType.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [kylin] nichunen merged pull request #724: Add tech blog: Deep dive into realtime olap
nichunen merged pull request #724: Add tech blog: Deep dive into realtime olap URL: https://github.com/apache/kylin/pull/724 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Assigned] (KYLIN-4059) metrics write to hive error
[ https://issues.apache.org/jira/browse/KYLIN-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu reassigned KYLIN-4059: --- Assignee: XiaoXiang Yu > metrics write to hive error > --- > > Key: KYLIN-4059 > URL: https://issues.apache.org/jira/browse/KYLIN-4059 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.6.2 >Reporter: chenchen >Assignee: XiaoXiang Yu >Priority: Major > Attachments: 501561537981_.pic_hd.jpg, 511561539223_.pic.jpg > > > my hadoop version is 2.7.1 > > I configure cube-planer and other metrics。When the query QPS of the machine > is relatively high, the following error will occur(Console Display metrics > Data Write Hive Failure)。 > By looking at the source code, kylin triggers an event to notify hiveProducer > to write data for each query. According to common sense, hive is not suitable > for dealing with this high frequency writing. > I wonder what the solution is? > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [kylin] coveralls commented on issue #726: KYLIN-4064 Make param engineType working in CI
coveralls commented on issue #726: KYLIN-4064 Make param engineType working in CI URL: https://github.com/apache/kylin/pull/726#issuecomment-507618487 ## Pull Request Test Coverage Report for [Build 4688](https://coveralls.io/builds/24322490) * **0** of **0** changed or added relevant lines in **0** files are covered. * **3** unchanged lines in **1** file lost coverage. * Overall coverage increased (+**0.006%**) to **28.273%** --- | Files with Coverage Reduction | New Missed Lines | % | | :-|--|--: | | [core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://coveralls.io/builds/24322490/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Finmemcubing%2FMemDiskStore.java#L449) | 3 | 78.42% | | Totals | [](https://coveralls.io/builds/24322490) | | :-- | --: | | Change from base [Build 4686](https://coveralls.io/builds/24316858): | 0.006% | | Covered Lines: | 23412 | | Relevant Lines: | 82808 | --- # - [Coveralls](https://coveralls.io) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] coveralls commented on issue #725: KYLIN-4062 Using a static hashmap to replace many "if-else" to improv…
coveralls commented on issue #725: KYLIN-4062 Using a static hashmap to replace many "if-else" to improv… URL: https://github.com/apache/kylin/pull/725#issuecomment-507618083 ## Pull Request Test Coverage Report for [Build 4687](https://coveralls.io/builds/24322443) * **0** of **26** **(0.0%)** changed or added relevant lines in **1** file are covered. * **6** unchanged lines in **3** files lost coverage. * Overall coverage increased (+**0.009%**) to **28.275%** --- | Changes Missing Coverage | Covered Lines | Changed/Added Lines | % | | :-|--||---: | | [query/src/main/java/org/apache/kylin/query/adhoc/PushDownRunnerJdbcImpl.java](https://coveralls.io/builds/24322443/source?filename=query%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fquery%2Fadhoc%2FPushDownRunnerJdbcImpl.java#L43) | 0 | 26 | 0.0% | Files with Coverage Reduction | New Missed Lines | % | | :-|--|--: | | [query/src/main/java/org/apache/kylin/query/adhoc/PushDownRunnerJdbcImpl.java](https://coveralls.io/builds/24322443/source?filename=query%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fquery%2Fadhoc%2FPushDownRunnerJdbcImpl.java#L259) | 1 | 0.0% | | [core-cube/src/main/java/org/apache/kylin/cube/cuboid/TreeCuboidScheduler.java](https://coveralls.io/builds/24322443/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Fcuboid%2FTreeCuboidScheduler.java#L124) | 2 | 68.46% | | [core-cube/src/main/java/org/apache/kylin/cube/inmemcubing/MemDiskStore.java](https://coveralls.io/builds/24322443/source?filename=core-cube%2Fsrc%2Fmain%2Fjava%2Forg%2Fapache%2Fkylin%2Fcube%2Finmemcubing%2FMemDiskStore.java#L449) | 3 | 78.42% | | Totals | [](https://coveralls.io/builds/24322443) | | :-- | --: | | Change from base [Build 4686](https://coveralls.io/builds/24316858): | 0.009% | | Covered Lines: | 23410 | | Relevant Lines: | 82793 | --- # - [Coveralls](https://coveralls.io) This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] asfgit commented on issue #726: KYLIN-4064 Make param engineType working in CI
asfgit commented on issue #726: KYLIN-4064 Make param engineType working in CI URL: https://github.com/apache/kylin/pull/726#issuecomment-507603336 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-4064) parameter 'engineType' is not working when running integration test
[ https://issues.apache.org/jira/browse/KYLIN-4064?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876824#comment-16876824 ] ASF GitHub Bot commented on KYLIN-4064: --- Wayne1c commented on pull request #726: KYLIN-4064 Make param engineType working in CI URL: https://github.com/apache/kylin/pull/726 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org > parameter 'engineType' is not working when running integration test > --- > > Key: KYLIN-4064 > URL: https://issues.apache.org/jira/browse/KYLIN-4064 > Project: Kylin > Issue Type: Bug > Components: Integration >Reporter: Chao Long >Assignee: Chao Long >Priority: Major > Fix For: v3.0.0-beta > > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[GitHub] [kylin] Wayne1c opened a new pull request #726: KYLIN-4064 Make param engineType working in CI
Wayne1c opened a new pull request #726: KYLIN-4064 Make param engineType working in CI URL: https://github.com/apache/kylin/pull/726 This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-4062) Too many "if else" clause in PushDownRunnerJdbcImpl#toSqlType
[
https://issues.apache.org/jira/browse/KYLIN-4062?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876822#comment-16876822
]
ASF GitHub Bot commented on KYLIN-4062:
---
RupengWang commented on pull request #725: KYLIN-4062 Using a static hashmap to
replace many "if-else" to improv…
URL: https://github.com/apache/kylin/pull/725
…e efficiency
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
> Too many "if else" clause in PushDownRunnerJdbcImpl#toSqlType
> -
>
> Key: KYLIN-4062
> URL: https://issues.apache.org/jira/browse/KYLIN-4062
> Project: Kylin
> Issue Type: Improvement
> Components: Query Engine
>Reporter: Shaofeng SHI
>Assignee: 王汝鹏
>Priority: Minor
>
> In this method, it has 30 "if else" which is low efficient; Should use a
> static Hashmap, then only need 1 check.
>
> {code:java}
> if ("string".equalsIgnoreCase(type)) {
> return Types.VARCHAR;
> } else if ("varchar".equalsIgnoreCase(type)) {
> return Types.VARCHAR;
> } else if ("char".equalsIgnoreCase(type)) {
> return Types.CHAR;
> } else if
> ...{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[GitHub] [kylin] asfgit commented on issue #725: KYLIN-4062 Using a static hashmap to replace many "if-else" to improv…
asfgit commented on issue #725: KYLIN-4062 Using a static hashmap to replace many "if-else" to improv… URL: https://github.com/apache/kylin/pull/725#issuecomment-507601578 Can one of the admins verify this patch? This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[GitHub] [kylin] RupengWang opened a new pull request #725: KYLIN-4062 Using a static hashmap to replace many "if-else" to improv…
RupengWang opened a new pull request #725: KYLIN-4062 Using a static hashmap to replace many "if-else" to improv… URL: https://github.com/apache/kylin/pull/725 …e efficiency This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: us...@infra.apache.org With regards, Apache Git Services
[jira] [Commented] (KYLIN-4059) metrics write to hive error
[ https://issues.apache.org/jira/browse/KYLIN-4059?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876756#comment-16876756 ] XiaoXiang Yu commented on KYLIN-4059: - I will check it under CDH5.7/Kylin2.6.2. > metrics write to hive error > --- > > Key: KYLIN-4059 > URL: https://issues.apache.org/jira/browse/KYLIN-4059 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.6.2 >Reporter: chenchen >Priority: Major > Attachments: 501561537981_.pic_hd.jpg, 511561539223_.pic.jpg > > > my hadoop version is 2.7.1 > > I configure cube-planer and other metrics。When the query QPS of the machine > is relatively high, the following error will occur(Console Display metrics > Data Write Hive Failure)。 > By looking at the source code, kylin triggers an event to notify hiveProducer > to write data for each query. According to common sense, hive is not suitable > for dealing with this high frequency writing. > I wonder what the solution is? > -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Commented] (KYLIN-4065) dictionary is lost
[ https://issues.apache.org/jira/browse/KYLIN-4065?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=16876747#comment-16876747 ] XiaoXiang Yu commented on KYLIN-4065: - Hi, [~chenfeng_0829], if you know the root cause, or how to reproduce, please share with us. Or submiting a patch will be better as well. > dictionary is lost > -- > > Key: KYLIN-4065 > URL: https://issues.apache.org/jira/browse/KYLIN-4065 > Project: Kylin > Issue Type: Bug >Affects Versions: v2.6.2 >Reporter: chenchen >Priority: Major > Attachments: 1B75DB41-7884-46A2-A241-F2CC72C344E8.png > > > I've upgraded from 2.3.0 and I haven't had this problem before. This error > occurs when merging and querying > My cube processes Kafka data in batches. This dictionary is a regular rowkey > (dict type) > This anomaly occurs every day or two, but segments are normal and data is not > lost. > I'm too clear about the reason for this bug. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Resolved] (KYLIN-4046) Refine JDBC Source(source.default=8)
[ https://issues.apache.org/jira/browse/KYLIN-4046?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] XiaoXiang Yu resolved KYLIN-4046. - Resolution: Fixed Fix Version/s: v3.0.0 > Refine JDBC Source(source.default=8) > > > Key: KYLIN-4046 > URL: https://issues.apache.org/jira/browse/KYLIN-4046 > Project: Kylin > Issue Type: Bug > Components: RDBMS Source >Reporter: XiaoXiang Yu >Assignee: XiaoXiang Yu >Priority: Critical > Fix For: v3.0.0 > > > Currently, the function of ingest from RDBMS(kylin.source.default=8) to Kylin > has some problem such as: > # case-sensitive > # weak dialect compatibility > # mis-use quote character > We need to fix this problem in kylin.source.default=8. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
