[jira] [Updated] (KYLIN-2421) Add spark engine to Integration Test
[ https://issues.apache.org/jira/browse/KYLIN-2421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-2421: Description: Let Jenkins to test Spark cubing daily. Will change the engine type for "ci_inner_join_cube" to "4" (spark). After this change, you need do a couple changes manually in your HDP 2.4 sandbox to finish the Integration Test: 1. manually install spark-1.6.3-bin-hadoop2.6 a local folder like /usr/local/spark cd /usr/local tar -zxvf spark-1.6.3-bin-hadoop2.6.tgz ln -s spark-1.6.3-bin-hadoop2.6 spark 2. manually upload the spark-assembly jar to HDFS as /kylin/spark/spark-assembly-1.6.3-hadoop2.6.0.jar (avoid repeatedly uploading the jar to HDFS): hadoop fs -mkdir /kylin/spark/ hadoop fs -put /usr/local/spark/lib/spark-assembly-1.6.3-hadoop2.6.0.jar /kylin/spark/ 3. manually create local temp folder for hbase client (if it doesn't exist): mkdir -p /hadoop/hbase/local/jars chmod 777 /hadoop/hbase/local/jars 4. Add SPARK_HOME property when run mvn: mvn verify -fae -Dhdp.version=2.4.0.0-169 -DSPARK_HOME=/usr/local/spark -P sandbox was: Let Jenkins to test Spark cubing daily. Will change the engine type for "ci_inner_join_cube" to "4" (spark). After this change, you need do a couple changes manually in your HDP 2.4 sandbox to finish the Integration Test: 1. manually install spark-1.6.3-bin-hadoop2.6 a local folder like /usr/local/spark 2. manually upload the spark-assembly jar to HDFS as /kylin/spark/spark-assembly-1.6.3-hadoop2.6.0.jar 3. manually create local temp folder for hbase client (if it doesn't exist): mkdir -p /hadoop/hbase/local/jars chmod 777 /hadoop/hbase/local/jars 4. Add SPARK_HOME property when run mvn: mvn verify -fae -Dhdp.version=2.4.0.0-169 -DSPARK_HOME=/usr/local/spark -P sandbox > Add spark engine to Integration Test > > > Key: KYLIN-2421 > URL: https://issues.apache.org/jira/browse/KYLIN-2421 > Project: Kylin > Issue Type: Sub-task > Components: Job Engine >Affects Versions: v2.0.0 >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI > Fix For: v2.0.0 > > > Let Jenkins to test Spark cubing daily. Will change the engine type for > "ci_inner_join_cube" to "4" (spark). > After this change, you need do a couple changes manually in your HDP 2.4 > sandbox to finish the Integration Test: > 1. manually install spark-1.6.3-bin-hadoop2.6 a local folder like > /usr/local/spark > cd /usr/local > tar -zxvf spark-1.6.3-bin-hadoop2.6.tgz > ln -s spark-1.6.3-bin-hadoop2.6 spark > 2. manually upload the spark-assembly jar to HDFS as > /kylin/spark/spark-assembly-1.6.3-hadoop2.6.0.jar (avoid repeatedly uploading > the jar to HDFS): > hadoop fs -mkdir /kylin/spark/ > hadoop fs -put /usr/local/spark/lib/spark-assembly-1.6.3-hadoop2.6.0.jar > /kylin/spark/ > 3. manually create local temp folder for hbase client (if it doesn't exist): > mkdir -p /hadoop/hbase/local/jars > chmod 777 /hadoop/hbase/local/jars > 4. Add SPARK_HOME property when run mvn: > mvn verify -fae -Dhdp.version=2.4.0.0-169 -DSPARK_HOME=/usr/local/spark -P > sandbox -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (KYLIN-2421) Add spark engine to Integration Test
[ https://issues.apache.org/jira/browse/KYLIN-2421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Shaofeng SHI updated KYLIN-2421: Description: Let Jenkins to test Spark cubing daily. Will change the engine type for "ci_inner_join_cube" to "4" (spark). After this change, you need do a couple changes manually in your HDP 2.4 sandbox to finish the Integration Test: 1. manually install spark-1.6.3-bin-hadoop2.6 a local folder like /usr/local/spark 2. manually upload the spark-assembly jar to HDFS as /kylin/spark/spark-assembly-1.6.3-hadoop2.6.0.jar 3. manually create local temp folder for hbase client (if it doesn't exist): mkdir -p /hadoop/hbase/local/jars chmod 777 /hadoop/hbase/local/jars 4. Add SPARK_HOME property when run mvn: mvn verify -fae -Dhdp.version=2.4.0.0-169 -DSPARK_HOME=/usr/local/spark -P sandbox was:Let Jenkins to test Spark cubing daily. > Add spark engine to Integration Test > > > Key: KYLIN-2421 > URL: https://issues.apache.org/jira/browse/KYLIN-2421 > Project: Kylin > Issue Type: Sub-task > Components: Job Engine >Affects Versions: v2.0.0 >Reporter: Shaofeng SHI >Assignee: Shaofeng SHI > Fix For: v2.0.0 > > > Let Jenkins to test Spark cubing daily. Will change the engine type for > "ci_inner_join_cube" to "4" (spark). > After this change, you need do a couple changes manually in your HDP 2.4 > sandbox to finish the Integration Test: > 1. manually install spark-1.6.3-bin-hadoop2.6 a local folder like > /usr/local/spark > 2. manually upload the spark-assembly jar to HDFS as > /kylin/spark/spark-assembly-1.6.3-hadoop2.6.0.jar > 3. manually create local temp folder for hbase client (if it doesn't exist): > mkdir -p /hadoop/hbase/local/jars > chmod 777 /hadoop/hbase/local/jars > 4. Add SPARK_HOME property when run mvn: > mvn verify -fae -Dhdp.version=2.4.0.0-169 -DSPARK_HOME=/usr/local/spark -P > sandbox -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (KYLIN-2423) Model should always include PK/FK as dimensions
liyang created KYLIN-2423: - Summary: Model should always include PK/FK as dimensions Key: KYLIN-2423 URL: https://issues.apache.org/jira/browse/KYLIN-2423 Project: Kylin Issue Type: Improvement Reporter: liyang Assignee: liyang Model should always include PK/FK as dimensions, as they participate in model any way. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (KYLIN-2422) NumberDictionary support for decimal with extra 0 after "."
Shaofeng SHI created KYLIN-2422: --- Summary: NumberDictionary support for decimal with extra 0 after "." Key: KYLIN-2422 URL: https://issues.apache.org/jira/browse/KYLIN-2422 Project: Kylin Issue Type: Improvement Components: Metadata Reporter: Shaofeng SHI Assignee: Shaofeng SHI Fix For: v2.0.0 The NumberDictionary should support decimal with extra 0 after "."; E.g: 54.32 = 54.3200 1234.00 = 1234 -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (KYLIN-2421) Add spark engine to Integration Test
Shaofeng SHI created KYLIN-2421: --- Summary: Add spark engine to Integration Test Key: KYLIN-2421 URL: https://issues.apache.org/jira/browse/KYLIN-2421 Project: Kylin Issue Type: Sub-task Components: Job Engine Affects Versions: v2.0.0 Reporter: Shaofeng SHI Assignee: Shaofeng SHI Fix For: v2.0.0 Let Jenkins to test Spark cubing daily. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (KYLIN-2420) Record zookeeper lock holder information
[ https://issues.apache.org/jira/browse/KYLIN-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roger Shi resolved KYLIN-2420. -- Resolution: Fixed > Record zookeeper lock holder information > > > Key: KYLIN-2420 > URL: https://issues.apache.org/jira/browse/KYLIN-2420 > Project: Kylin > Issue Type: Improvement >Reporter: Roger Shi >Priority: Minor > Attachments: > 0001-KYLIN-2420-Record-zookeeper-lock-holder-information.patch > > > Kylin should record zookeeper lock holder information such as process id and > host name. If one instance can't acquire a lock, at least we know which one > is holding it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (KYLIN-2420) Record zookeeper lock holder information
[ https://issues.apache.org/jira/browse/KYLIN-2420?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Roger Shi updated KYLIN-2420: - Attachment: 0001-KYLIN-2420-Record-zookeeper-lock-holder-information.patch > Record zookeeper lock holder information > > > Key: KYLIN-2420 > URL: https://issues.apache.org/jira/browse/KYLIN-2420 > Project: Kylin > Issue Type: Improvement >Reporter: Roger Shi >Priority: Minor > Attachments: > 0001-KYLIN-2420-Record-zookeeper-lock-holder-information.patch > > > Kylin should record zookeeper lock holder information such as process id and > host name. If one instance can't acquire a lock, at least we know which one > is holding it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (KYLIN-2420) Record zookeeper lock holder information
Roger Shi created KYLIN-2420: Summary: Record zookeeper lock holder information Key: KYLIN-2420 URL: https://issues.apache.org/jira/browse/KYLIN-2420 Project: Kylin Issue Type: Improvement Reporter: Roger Shi Priority: Minor Kylin should record zookeeper lock holder information such as process id and host name. If one instance can't acquire a lock, at least we know which one is holding it. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Commented] (KYLIN-2361) Upgrade to Tomcat 8.X
[ https://issues.apache.org/jira/browse/KYLIN-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852531#comment-15852531 ] Billy Liu commented on KYLIN-2361: -- https://github.com/openwide-java/tomcat-classloader-ordered provides the patch for ordered class loading. > Upgrade to Tomcat 8.X > - > > Key: KYLIN-2361 > URL: https://issues.apache.org/jira/browse/KYLIN-2361 > Project: Kylin > Issue Type: Task > Components: Web >Affects Versions: v1.6.0 >Reporter: Billy Liu >Assignee: Billy Liu >Priority: Minor > Fix For: v2.0.0 > > > Apache Tomcat 8.5.x supports the same Servlet, JSP, EL, and WebSocket > Specification versions as Apache Tomcat 8.0.x. In addition to that, it also > implements the JASPIC 1.1 specification. There are significant changes in > many areas under the hood, resulting in improved performance, stability, and > total cost of ownership. Please refer to the Apache Tomcat 8.5 Changelog for > details. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (KYLIN-2390) Wrong argument order for WinAggResetContextImpl()
[
https://issues.apache.org/jira/browse/KYLIN-2390?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
liyang resolved KYLIN-2390.
---
Resolution: Fixed
Assignee: liyang (was: hongbin ma)
Fix Version/s: v2.0.0
resolved by KYLIN-2419
> Wrong argument order for WinAggResetContextImpl()
> -
>
> Key: KYLIN-2390
> URL: https://issues.apache.org/jira/browse/KYLIN-2390
> Project: Kylin
> Issue Type: Bug
>Reporter: Ted Yu
>Assignee: liyang
> Fix For: v2.0.0
>
>
> In EnumerableWindow#implement() , the order of last two parameters is swapped:
> {code}
> new WinAggResetContextImpl(builder6, agg.state, i_,
> startX, endX,
> hasRows, partitionRowCount, frameRowCount));
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Resolved] (KYLIN-2419) Rollback KYLIN-2292 workaround
[ https://issues.apache.org/jira/browse/KYLIN-2419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyang resolved KYLIN-2419. --- Resolution: Fixed Assignee: liyang Fix Version/s: v2.0.0 > Rollback KYLIN-2292 workaround > -- > > Key: KYLIN-2419 > URL: https://issues.apache.org/jira/browse/KYLIN-2419 > Project: Kylin > Issue Type: Bug >Reporter: liyang >Assignee: liyang > Fix For: v2.0.0 > > > As we have upgraded to calcite 1.11, the workaround of CALCITE-1540 > introduced by KYLIN-2292 must rollback. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Updated] (KYLIN-2419) Rollback KYLIN-2292 workaround
[ https://issues.apache.org/jira/browse/KYLIN-2419?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] liyang updated KYLIN-2419: -- Description: As we have upgraded to calcite 1.11, the workaround of CALCITE-1540 introduced by KYLIN-2292 must rollback. > Rollback KYLIN-2292 workaround > -- > > Key: KYLIN-2419 > URL: https://issues.apache.org/jira/browse/KYLIN-2419 > Project: Kylin > Issue Type: Bug >Reporter: liyang > > As we have upgraded to calcite 1.11, the workaround of CALCITE-1540 > introduced by KYLIN-2292 must rollback. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Created] (KYLIN-2419) Rollback KYLIN-2292 workaround
liyang created KYLIN-2419: - Summary: Rollback KYLIN-2292 workaround Key: KYLIN-2419 URL: https://issues.apache.org/jira/browse/KYLIN-2419 Project: Kylin Issue Type: Bug Reporter: liyang -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Reopened] (KYLIN-2361) Upgrade to Tomcat 8.X
[ https://issues.apache.org/jira/browse/KYLIN-2361?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Billy Liu reopened KYLIN-2361: -- Tomcat 8 classloader could not guarantee the alphabetical order as it was the case with Tomcat 7.x. But Kylin depends on the loader order (atopcalcite and caicite), so the upgrade is pending. See the discussion in https://bz.apache.org/bugzilla/show_bug.cgi?id=57129 for more information. > Upgrade to Tomcat 8.X > - > > Key: KYLIN-2361 > URL: https://issues.apache.org/jira/browse/KYLIN-2361 > Project: Kylin > Issue Type: Task > Components: Web >Affects Versions: v1.6.0 >Reporter: Billy Liu >Assignee: Billy Liu >Priority: Minor > Fix For: v2.0.0 > > > Apache Tomcat 8.5.x supports the same Servlet, JSP, EL, and WebSocket > Specification versions as Apache Tomcat 8.0.x. In addition to that, it also > implements the JASPIC 1.1 specification. There are significant changes in > many areas under the hood, resulting in improved performance, stability, and > total cost of ownership. Please refer to the Apache Tomcat 8.5 Changelog for > details. -- This message was sent by Atlassian JIRA (v6.3.15#6346)
[jira] [Resolved] (KYLIN-2242) Directly write hdfs file in reducer is dangerous
[
https://issues.apache.org/jira/browse/KYLIN-2242?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
kangkaisen resolved KYLIN-2242.
---
Resolution: Fixed
> Directly write hdfs file in reducer is dangerous
>
>
> Key: KYLIN-2242
> URL: https://issues.apache.org/jira/browse/KYLIN-2242
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v1.6.0
>Reporter: kangkaisen
>Assignee: kangkaisen
> Fix For: v2.0.0
>
> Attachments: consoleFull.html, KYLIN-2242.patch
>
>
> currently, Kylin directly write hdfs file in {{FactDistinctColumnsReducer}},
> which is dangerous because the MapReduce Speculative Execution will result in
> more than one reducers write the same hdfs file at the same time.
> After KYLIN-2217, I think this issue will occur with higher probability. we
> should output the value by {{context.wirte}} in reducer.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (KYLIN-2242) Directly write hdfs file in reducer is dangerous
[
https://issues.apache.org/jira/browse/KYLIN-2242?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15852499#comment-15852499
]
kangkaisen commented on KYLIN-2242:
---
OK, Thank you, Shaofeng.
> Directly write hdfs file in reducer is dangerous
>
>
> Key: KYLIN-2242
> URL: https://issues.apache.org/jira/browse/KYLIN-2242
> Project: Kylin
> Issue Type: Bug
> Components: Job Engine
>Affects Versions: v1.6.0
>Reporter: kangkaisen
>Assignee: kangkaisen
> Fix For: v2.0.0
>
> Attachments: consoleFull.html, KYLIN-2242.patch
>
>
> currently, Kylin directly write hdfs file in {{FactDistinctColumnsReducer}},
> which is dangerous because the MapReduce Speculative Execution will result in
> more than one reducers write the same hdfs file at the same time.
> After KYLIN-2217, I think this issue will occur with higher probability. we
> should output the value by {{context.wirte}} in reducer.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
[jira] [Commented] (KYLIN-2406) TPC-H query 20, can triggers NPE
[
https://issues.apache.org/jira/browse/KYLIN-2406?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15851216#comment-15851216
]
Kaige Liu commented on KYLIN-2406:
---
Above query will get this execution plan:
{code}
OLAPToEnumerableConverter
EnumerableCalc(expr#0..17=[{inputs}], expr#18=[>($t2, $t9)], S_NAME=[$t12],
S_ADDRESS=[$t13], $condition=[$t18])
EnumerableJoin(condition=[=($1, $11)], joinType=[inner])
EnumerableJoin(condition=[AND(=($0, $8), =($1, $10))], joinType=[inner])
*OLAPTableScan(table=[[TPCH_FLAT_ORC_2, V_PARTSUPP]], fields=[[0, 1, 2,
3, 4, 5, 6, 7]])*
EnumerableCalc(expr#0..2=[{inputs}], expr#3=[0.5], expr#4=[*($t3,
$t2)], L_PARTKEY=[$t0], SUM_QUANTITY=[$t4], L_SUPPKEY=[$t1])
EnumerableAggregate(group=[{0, 1}], agg#0=[SUM($2)])
EnumerableCalc(expr#0..36=[{inputs}], expr#37=['1992-01-01'],
expr#38=[>=($t8, $t37)], expr#39=['1995-01-01'], expr#40=[<=($t8, $t39)],
expr#41=['CANADA'], expr#42=[=($t28, $t41)], expr#43=['forest%'],
expr#44=[LIKE($t31, $t43)], expr#45=[AND($t38, $t40, $t42, $t44)],
L_PARTKEY=[$t1], L_SUPPKEY=[$t2], L_QUANTITY=[$t3], $condition=[$t45])
OLAPJoinRel(condition=[=($1, $30)], joinType=[inner])
OLAPJoinRel(condition=[=($23, $27)], joinType=[inner])
OLAPJoinRel(condition=[=($2, $20)], joinType=[inner])
OLAPTableScan(table=[[TPCH_FLAT_ORC_2, V_LINEITEM]],
fields=[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19]])
OLAPTableScan(table=[[TPCH_FLAT_ORC_2, SUPPLIER]],
fields=[[0, 1, 2, 3, 4, 5, 6]])
OLAPTableScan(table=[[TPCH_FLAT_ORC_2, NATION]], fields=[[0,
1, 2]])
OLAPTableScan(table=[[TPCH_FLAT_ORC_2, PART]], fields=[[0, 1,
2, 3, 4, 5, 6]])
*OLAPTableScan(table=[[TPCH_FLAT_ORC_2, SUPPLIER]], fields=[[0, 1, 2, 3,
4, 5, 6]])*
{code}
The two OLAPTableScan in same OLAPContext do not match join relation defined in
model.
Need to give a clear error message here.
> TPC-H query 20, can triggers NPE
>
>
> Key: KYLIN-2406
> URL: https://issues.apache.org/jira/browse/KYLIN-2406
> Project: Kylin
> Issue Type: Bug
>Reporter: liyang
>Assignee: Kaige Liu
>
> Below query triggers NPE
> {code}
> with tmp3 as (
> select l_partkey, 0.5 * sum(l_quantity) as sum_quantity, l_suppkey
> from v_lineitem
> inner join supplier on l_suppkey = s_suppkey
> inner join nation on s_nationkey = n_nationkey
> inner join part on l_partkey = p_partkey
> where l_shipdate >= '1992-01-01' and l_shipdate <= '1995-01-01'
> and n_name = 'CANADA'
> and p_name like 'forest%'
> group by l_partkey, l_suppkey
> )
> select
> s_name,
> s_address
> from
> v_partsupp
> inner join tmp3 on ps_partkey = l_partkey and ps_suppkey = l_suppkey
> inner join supplier on ps_suppkey = s_suppkey
> where
> ps_availqty > sum_quantity
> group by
> s_name, s_address
> order by
> s_name
> {code}
> While below query is OK. Only difference being the order of "inner join tmp3"
> and "inner join supplier"
> {code}
> with tmp3 as (
> select l_partkey, 0.5 * sum(l_quantity) as sum_quantity, l_suppkey
> from v_lineitem
> inner join supplier on l_suppkey = s_suppkey
> inner join nation on s_nationkey = n_nationkey
> inner join part on l_partkey = p_partkey
> where l_shipdate >= '1992-01-01' and l_shipdate <= '1995-01-01'
> and n_name = 'CANADA'
> and p_name like 'forest%'
> group by l_partkey, l_suppkey
> )
> select
> s_name,
> s_address
> from
> v_partsupp
> inner join supplier on ps_suppkey = s_suppkey
> inner join tmp3 on ps_partkey = l_partkey and ps_suppkey = l_suppkey
> where
> ps_availqty > sum_quantity
> group by
> s_name, s_address
> order by
> s_name
> {code}
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)
