[jira] [Created] (KYLIN-4768) New Catalog to expose cuboid table
yiming.xu created KYLIN-4768: Summary: New Catalog to expose cuboid table Key: KYLIN-4768 URL: https://issues.apache.org/jira/browse/KYLIN-4768 Project: Kylin Issue Type: New Feature Components: Query Engine Reporter: yiming.xu Assignee: Chunen Ni -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4767) Support catalyst parse SQL with Spark engine
yiming.xu created KYLIN-4767: Summary: Support catalyst parse SQL with Spark engine Key: KYLIN-4767 URL: https://issues.apache.org/jira/browse/KYLIN-4767 Project: Kylin Issue Type: New Feature Components: Query Engine Reporter: yiming.xu current -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4223) Add new storage interface to support write to any formate
yiming.xu created KYLIN-4223: Summary: Add new storage interface to support write to any formate Key: KYLIN-4223 URL: https://issues.apache.org/jira/browse/KYLIN-4223 Project: Kylin Issue Type: Sub-task Reporter: yiming.xu Add a new storage interface to support write to any format, the default format will be parquet. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4222) Add new interface to extract metadata to build
yiming.xu created KYLIN-4222: Summary: Add new interface to extract metadata to build Key: KYLIN-4222 URL: https://issues.apache.org/jira/browse/KYLIN-4222 Project: Kylin Issue Type: Sub-task Reporter: yiming.xu In this spark build engine, a cuboid is a cuboid table, Add new interface to extract metadata, this interface make the build engine to support anywhere if they need to build some data. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4221) Auto spark conf parameter tuning
yiming.xu created KYLIN-4221: Summary: Auto spark conf parameter tuning Key: KYLIN-4221 URL: https://issues.apache.org/jira/browse/KYLIN-4221 Project: Kylin Issue Type: Sub-task Reporter: yiming.xu Spark job need strong conf to run fast and stable, we should have some strategy to auto set spark conf -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4219) New task SparkExecutable
yiming.xu created KYLIN-4219: Summary: New task SparkExecutable Key: KYLIN-4219 URL: https://issues.apache.org/jira/browse/KYLIN-4219 Project: Kylin Issue Type: Sub-task Reporter: yiming.xu Add new SparkExecutable to start a new job. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4220) New build engine
yiming.xu created KYLIN-4220: Summary: New build engine Key: KYLIN-4220 URL: https://issues.apache.org/jira/browse/KYLIN-4220 Project: Kylin Issue Type: Sub-task Reporter: yiming.xu -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4218) Add new engine type
yiming.xu created KYLIN-4218: Summary: Add new engine type Key: KYLIN-4218 URL: https://issues.apache.org/jira/browse/KYLIN-4218 Project: Kylin Issue Type: Sub-task Components: Job Engine Reporter: yiming.xu Add a new engine to support the build engine. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4217) Calcite rel to Spark plan
yiming.xu created KYLIN-4217: Summary: Calcite rel to Spark plan Key: KYLIN-4217 URL: https://issues.apache.org/jira/browse/KYLIN-4217 Project: Kylin Issue Type: Sub-task Components: Query Engine Reporter: yiming.xu Transform calcite rel to spark plan to implement distributed computing. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4216) Add query engine interface
yiming.xu created KYLIN-4216: Summary: Add query engine interface Key: KYLIN-4216 URL: https://issues.apache.org/jira/browse/KYLIN-4216 Project: Kylin Issue Type: Sub-task Components: Query Engine Reporter: yiming.xu Current query engine uses calcite to computing data, add a new interface and implement this. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4215) Add Spark Isolated class Loader
yiming.xu created KYLIN-4215: Summary: Add Spark Isolated class Loader Key: KYLIN-4215 URL: https://issues.apache.org/jira/browse/KYLIN-4215 Project: Kylin Issue Type: Sub-task Reporter: yiming.xu Current Kylin server add Hadoop classpath to tomcat classpath, the new query engine will start a spark driver in the Kylin server, we use a spark Isolated class Loader to void some class problem. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4214) Distributed query engine
yiming.xu created KYLIN-4214: Summary: Distributed query engine Key: KYLIN-4214 URL: https://issues.apache.org/jira/browse/KYLIN-4214 Project: Kylin Issue Type: Sub-task Components: Query Engine Reporter: yiming.xu Sql through calcite analysis optimization will generate a tree of OLAP rel, and spark's catalyst is also generated by parsing SQL after a tree, automatically optimized to become a dataframe to calculate, if the plan of calcite can be converted into a spark plan, then we will achieve distributed computing, calcite is only responsible for parsing SQL and returning result sets, reducing the pressure on the kylin server side. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-4213) The new build engine with Spark-SQL
yiming.xu created KYLIN-4213: Summary: The new build engine with Spark-SQL Key: KYLIN-4213 URL: https://issues.apache.org/jira/browse/KYLIN-4213 Project: Kylin Issue Type: Sub-task Components: Job Engine Affects Versions: Future Reporter: yiming.xu Use Spark-SQL to compute cuboid, build cuboid A, B, C , Sum(D) is sql "select A B C Sum(D) from table group by A, B, C". To void many memory errors or other exceptions, we can auto set spark conf with build job.E.g use adaptive execution. -- This message was sent by Atlassian Jira (v8.3.4#803005)
[jira] [Created] (KYLIN-3391) BadQueryDetector only detect first query
yiming.xu created KYLIN-3391: Summary: BadQueryDetector only detect first query Key: KYLIN-3391 URL: https://issues.apache.org/jira/browse/KYLIN-3391 Project: Kylin Issue Type: Bug Reporter: yiming.xu BadQueryDetector only detect first query. -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3390) QueryInterceptorUtil.queryInterceptors is not thread safe
yiming.xu created KYLIN-3390: Summary: QueryInterceptorUtil.queryInterceptors is not thread safe Key: KYLIN-3390 URL: https://issues.apache.org/jira/browse/KYLIN-3390 Project: Kylin Issue Type: Bug Reporter: yiming.xu With concurrent case , org.apache.kylin.query.relnode.OLAPToEnumerableConverter throw ConcurrentModificationException -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3283) Support values RelNode
yiming.xu created KYLIN-3283:
Summary: Support values RelNode
Key: KYLIN-3283
URL: https://issues.apache.org/jira/browse/KYLIN-3283
Project: Kylin
Issue Type: New Feature
Components: Query Engine
Affects Versions: v2.3.0
Reporter: yiming.xu
Assignee: yiming.xu
for queries like:
select leaf_categ_id, sum(price) as sum_price from test_kylin_fact group by
leaf_categ_id
union all
select 1 as leaf_categ_id2, 2 as sum_price2
the expected logical plan after volcano is:
{code:java}
OLAPToEnumerableConverter
OLAPLimitRel(ctx=[], fetch=[5])
OLAPAggregateRel(group=[{}], EXPR$0=[SUM($0)], ctx=[])
OLAPProjectRel(count=[$1], ctx=[])
OLAPSortRel(sort0=[$1], dir0=[DESC], ctx=[])
OLAPAggregateRel(group=[{0}], count=[COUNT()], ctx=[])
OLAPProjectRel(LSTG_FORMAT_NAME=[$3], ctx=[])
OLAPTableScan(table=[[DEFAULT, TEST_KYLIN_FACT]], ctx=[],
fields=[[0, 1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19,
20, 21, 22, 23, 24, 25, 26]])
{code}
however the actual is:
{code:java}
OLAPToEnumerableConverter
OLAPLimitRel(ctx=[], fetch=[5])
OLAPUnionRel(all=[true], ctx=[], all=[true])
OLAPAggregateRel(group=[{0}], SUM_PRICE=[SUM($1)], ctx=[])
OLAPProjectRel(LEAF_CATEG_ID=[$4], PRICE=[$8], ctx=[])
OLAPTableScan(table=[[DEFAULT, TEST_KYLIN_FACT]], ctx=[], fields=[[0,
1, 2, 3, 4, 5, 6, 7, 8, 9, 10, 11, 12, 13, 14, 15, 16, 17, 18, 19, 20, 21, 22,
23, 24, 25, 26]])
OLAPProjectRel(LEAF_CATEG_ID2=[1], SUM_PRICE2=[2], ctx=[])
OLAPValuesRel(type=[RecordType(INTEGER ZERO)], tuples=[[{ 0 }]])
{code}
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
[jira] [Created] (KYLIN-3281) OlapProjectRule not working with projectrel + sortrel
yiming.xu created KYLIN-3281: Summary: OlapProjectRule not working with projectrel + sortrel Key: KYLIN-3281 URL: https://issues.apache.org/jira/browse/KYLIN-3281 Project: Kylin Issue Type: Bug Components: Query Engine Affects Versions: v2.3.0 Reporter: yiming.xu -- This message was sent by Atlassian JIRA (v7.6.3#76005)
[jira] [Created] (KYLIN-3150) PercentileCounter maxlength is to small
yiming.xu created KYLIN-3150: Summary: PercentileCounter maxlength is to small Key: KYLIN-3150 URL: https://issues.apache.org/jira/browse/KYLIN-3150 Project: Kylin Issue Type: Bug Components: Query Engine Affects Versions: v2.2.0 Reporter: yiming.xu Assignee: liyang With Percentile 1000, the count size more than 10 * 1024. We should use different max length with different compression. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KYLIN-3064) Yarn timeline-service will start jersey client, remove it
yiming.xu created KYLIN-3064: Summary: Yarn timeline-service will start jersey client, remove it Key: KYLIN-3064 URL: https://issues.apache.org/jira/browse/KYLIN-3064 Project: Kylin Issue Type: Bug Reporter: yiming.xu Priority: Minor We not used timeline service, when the yarn.timeline-service.enabled is true, yarn client will start a jersey client, it will use many resources. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KYLIN-2776) new metric framework with kylin
yiming.xu created KYLIN-2776: Summary: new metric framework with kylin Key: KYLIN-2776 URL: https://issues.apache.org/jira/browse/KYLIN-2776 Project: Kylin Issue Type: New Feature Affects Versions: v2.0.0 Reporter: yiming.xu Assignee: yiming.xu Attachments: active_calls.png, calls.png, metric_structure.png, query_count.png, query_duration.png, query_result_rowcount.png, report.json With https://issues.apache.org/jira/browse/KYLIN-2721.We are plan to release a new metric framework. New metric is different hadoop metric and based on dropwizard . which has the following advantage: * Well-defined metric model for frequently-needed metrics (ie JVM metrics) * Well-defined measurements for all metrics (ie max, mean, stddev, mean_rate, etc), * Built-in pluggable reporting frameworks like JMX, Console, Log, JSON We refactor QueryMetric with new metris. New metric add perflog. Perflog trace calls duration time and current active calls record to metric system. Attachment is the difference between the two metric system . -- This message was sent by Atlassian JIRA (v6.4.14#64029)
[jira] [Created] (KYLIN-2748) Calcite code generation can not gc cause OOM
yiming.xu created KYLIN-2748: Summary: Calcite code generation can not gc cause OOM Key: KYLIN-2748 URL: https://issues.apache.org/jira/browse/KYLIN-2748 Project: Kylin Issue Type: Bug Components: Query Engine Affects Versions: v2.0.0 Reporter: yiming.xu Assignee: liyang calcite code generation class can not gc cause OOM. -- This message was sent by Atlassian JIRA (v6.4.14#64029)
