Re:error as Extract Fact Table Distinct Columns: Java heap space

2018-05-30 Thread Ma Gang
Hi, The number of reducers in 'fact distinct columns' step is calculated as: {number of normal columns need to build dict} + {UHC columns number * UHC reducer count} + {number of cuboid row counters} + 1 the UHC reducer number can be configured by "kylin.engine.mr.uhc-reducer-count". You need

Re:[jira] [Created] (KYLIN-3351) Cube Planner not working in apche kylin 2.3.0(open Source)

2018-04-26 Thread Ma Gang
Did you setup System Cube? If there is system cube setup, you can check dashboard tab to see whether the query metrics have been record correctly. At 2018-04-26 14:26:00, "praveenece (JIRA)" wrote: >praveenece created KYLIN-3351: >- > >

Re:回复:如何用API删除job

2018-07-30 Thread Ma Gang
I don't know why you need to drop job in your case, but there is a drop job restful api in kylin: DELETE /api/jobs/{jobId}/drop 在 2018-07-30 17:23:28,"鲨鱼" <786510...@qq.com> 写道: >谢谢! >已试过官网的API,关于对job操作的API的其中三个:resume,pause,discard >

Re:Re: Re: [Discuss] Lookup table improvement: support global/big lookup table

2018-04-04 Thread Ma Gang
he snapshot size, build time, storage path on the GUI? > >With Warm regards > >Billy Liu > > >2018-04-01 18:19 GMT+08:00 ShaoFeng Shi <shaofeng...@apache.org>: >> Thank you, Gang! Looking forward to seeing this feature in Kylin. >> >> 2018-04-01 17:35 GMT+08:0

Re:Query Problem

2018-04-04 Thread Ma Gang
Hi Rahul, Currently Kylin only store pre-aggregate data, and no raw data is stored in the system, so Kylin can only support the aggregate query(has group by clause). For the no-aggregate query, in order to return better results, Kylin just hack to output sum of metric columns, for example, the

Re:Error result with this SQL

2018-04-23 Thread Ma Gang
This should be a bug, currently Kylin cannot handle such sub-query properly(both outside query and sub-query hit cube). The second sql can work is because there is no subquery exists, and intersect_count is Kylin's udf, you should define bitmap measure for passenger_id column. At 2018-04-20

Re:Re: [Discuss] Lookup table improvement: support global/big lookup table

2018-04-01 Thread Ma Gang
Sure ShaoFeng, I add a storageType field in the SnapshotDesc to support different materialized storages. At 2018-04-01 09:57:23, "ShaoFeng Shi" <shaofeng...@apache.org> wrote: >Thank you Ma Gang; This is a good proposal. Externalizing the lookup >snapshots will reduce the

Re:Re: [Announce] Welcome new Apache Kylin committer :Allen Ma

2018-10-16 Thread Ma Gang
Billy Liu Luke Han 于2018年10月16日周二下午11:28写道: > > I am very pleased to announce that the Project Management Committee (PMC) of > Apache Kylin has asked Allen Ma (Gang Ma) to become Apache Kylin committer, > and he has already accepted. > > Allen has already made many contr

Re:Re: Slow Query Performance With 'WHERE' Clause

2018-10-23 Thread Ma Gang
You may post your query related log here, there should be some query log that indicated whether the filter is push down or not, at least in the returned response stats, there's some log show how many rows are filtered in the coprocessor side. At 2018-10-23 16:58:36, "Sachin Aggarwal" wrote:

Re:Problematic thread for Query, BadQueryDetector

2018-10-23 Thread Ma Gang
Hi Shrikant, The log indicated that your query needs data from a snapshot lookup table(your query columns should contain some derived columns defined in the cube), but the snapshot is very large, so the query is very slow. You may check the snapshot size in the lookup table's snapshot tab in

[DISCUSS] New Kylin Streaming Solution From eBay

2018-10-30 Thread Ma Gang
Hi all, eBay Kylin team has developed a new Kylin streaming solution, the basic idea is to build a streaming cluster to ingest data from streaming source(Kafka), and provide query for real-time data, the data preparation latency is milliseconds, which means the data is queryable almost when

Re:Re: Re: [DISCUSS] New Kylin Streaming Solution From eBay

2018-11-01 Thread Ma Gang
etc.? Besides, what's the plan of contributing it to the >community? Thanks! > > >Ma Gang 于2018年11月1日周四 下午2:45写道: > >> Thanks Xiaoxiang, >> Very good questions! Please see my comments started with [Gang]: >> >> >> 1. Is it possible to use Yar

Re:Re: [DISCUSS] New Kylin Streaming Solution From eBay

2018-11-01 Thread Ma Gang
m columnar >storage, why not use a open source mature columnar storage solution ? Have >your ever compare the performance of your custom columnar storage to open >source columnar storage solution ? > > > > >Best wishes, >Xiaoxiang Yu > > >发件人: Ma

Re:[DISCUSS] New Kylin Streaming Solution From eBay

2018-10-30 Thread Ma Gang
Resend the design doc, not sure why the attachment is removed in the previous mail. At 2018-10-30 15:24:01, "Ma Gang" wrote: Hi all, eBay Kylin team has developed a new Kylin streaming solution, the basic idea is to build a streaming cluster to ingest data from streaming so

Re:Re: Re: [DISCUSS] Columnar storage engine for Apache Kylin

2018-10-31 Thread Ma Gang
dicated >HDFS/Spark cluster for Kylin's query, the spark will allocate an executor >which in the same machine with the data block to get data locality. >Besides, some cache technologies like Alluxio, Ignite can provide layed >(memory -> ssd -> hdd), memory speed, LRU cache for HDFS fil

Re:Re: [DISCUSS] New Kylin Streaming Solution From eBay

2018-10-30 Thread Ma Gang
Jira ticket has been created, and the related design doc is attached in the ticket: https://issues.apache.org/jira/browse/KYLIN-3654 在 2018-10-30 21:40:34,"ShaoFeng Shi" 写道: >Hi Gang, > >The design doc is still missing; can you upload it to somewhere and then >provide a l

Re:there is a log at run time a job about the sample cube of apache kylin

2018-10-30 Thread Ma Gang
The log is clear: FAILED: SemanticException [Error 10001]: Line 22:5 Table not found 'KYLIN_SALES' You need to check whether the table is existed in hive or not, you can use the hive client on Kylin's machine to check it. At 2018-10-31 02:24:47, "ebrahim zare" wrote: >hi >When I run the cube,

Re:Re: Re: [DISCUSS] New Kylin Streaming Solution From eBay

2018-10-31 Thread Ma Gang
plug-in >architecture, so that it can support non-HBase storage? As you know we're >implementing the parquet storage. Can this solution support other storages >without much rework? > >Thanks for raising this discussion. > >Ma Gang 于2018年10月31日周三 上午9:57写道: > >> Jira ticket

Re:Re: Re: Slow Query Performance With 'WHERE' Clause

2018-10-23 Thread Ma Gang
dule(s) as required for tracing latencies -- would you please suggest any classes which can provide me these details? Regards, Shrikant Bang. On Tue, Oct 23, 2018 at 3:09 PM Ma Gang wrote: You may post your query related log here, there should be some query log that indicated whether the

Re:Re: [DISCUSS] Columnar storage engine for Apache Kylin

2018-09-28 Thread Ma Gang
I like parquet, it is very efficient format and supported by various projects, but there are some questions if we use parquet as the cube storage format: 1. Is it possible to locate a cuboid quickly in a parquet file? How to save cuboid metadata info in the parquet's FileMetaData, just in the

Re:Kylin concurrent issue for query engine;

2018-09-21 Thread Ma Gang
Could you run jstack on the Kylin process and send out result? So that we can know what the query threads are blocked on. I encountered the same issue when doing load test, and found that most of query threads are blocked on class loading, and the class is code gen by calcite, so we may need

Re:Re: Re:Kylin concurrent issue for query engine;

2018-09-26 Thread Ma Gang
Just send prepare request to Kylin server as before, by default, the server side cache is enable: POST /kylin/api/query {"sql":"select minute_start,count(1) from TABLE where minute_start>=? and minute_start 写道: >+1 to add this into Kylin documentation. > >huaicui <270922...@qq.com>

Kylin real-time streaming is ready on realtime-streaming branch

2018-12-22 Thread Ma Gang
n be found in the attachment of jira: https://issues.apache.org/jira/browse/KYLIN-3654. This is just the first version, any comments and pull request are welcome! Thanks, Ma,Gang

Re:[VOTE] Release apache-kylin-2.6.0 (RC1)

2019-01-09 Thread Ma Gang
+1, mvn test passed At 2019-01-10 09:59:55, "Yichen Zhou" wrote: >+1 >mvn test passed > >Regards, >Yichen > >On Wed, Jan 9, 2019 at 5:58 PM Rongchuan Jin >wrote: > >> +1 >> >> >> 金荣钏/Rongchuan.Jin >> >> >> 在 2019/1/10 上午9:49,“ShaoFeng Shi” 写入: >> >> Checked

Re:Re: Kylin will not delete old hbase table when refresh the segment

2018-12-18 Thread Ma Gang
+1, the behavior can be the same as merge segments, old htables can be deleted directly, not sure it make sense or not to keep old htables to have the capability to roll back the old data. At 2018-12-17 17:49:50, "ShaoFeng Shi" wrote: >It can be improved to do the cleanup automatically. I

Re:[DISCUSS] Kylin 3.0 alpha and beta release before GA

2019-03-25 Thread Ma Gang
Thanks ShaoFeng, The plan looks good. At 2019-03-25 09:24:00, "ShaoFeng Shi" wrote: Hello, About two months ago, we raised the "[Discuss] Moving toward Apache Kylin 3.0" in the developer group, all agree to use 3.0 as the next major release version when the Real-Time feature released. Now

Re:RE: [VOTE] Release apache-kylin-3.0.0-alpha (RC1)

2019-04-11 Thread Ma Gang
+1 bindingmvn test passed At 2019-04-11 13:58:16, "李 栋" wrote: >+1 binding > >Mvn test passed > >Dong Li > >-Original Message- >From: Billy Liu >Sent: Wednesday, April 10, 2019 10:19 PM >To: dev >Subject: Re: [VOTE] Release apache-kylin-3.0.0-alpha (RC1) > >+1 binding > >mvn test

Re:Plan to host the first "Kylin Data Summit" event

2019-05-30 Thread Ma Gang
+1, looking forward to it. At 2019-05-30 14:29:24, "ShaoFeng Shi" wrote: >Hello Kylin developers and users, > > > >We (Kyligence Inc) planned to host the first "Kylin Data Summit" event at >Shanghai, China. This event is going to provide a place to share, discuss >the technology and trends in

Re:Re: [VOTE] Release apache-kylin-2.6.3 (RC1)

2019-07-03 Thread Ma Gang
+1 (binding) mvn test passed test env: Maven home: /dev/tools/apache-maven-3.6.0 Java version: 1.8.0_202, vendor: AdoptOpenJdk, runtime: /Library/Java/JavaVirtualMachines/adoptopenjdk-8.jdk/Contents/Home/jre Default locale: en_CN, platform encoding: UTF-8 OS name: "mac os x", version:

[jira] [Created] (KYLIN-1783) Can't add override property at cube design 'Configuration Overwrites' step.

2016-06-12 Thread Ma Gang (JIRA)
Ma Gang created KYLIN-1783: -- Summary: Can't add override property at cube design 'Configuration Overwrites' step. Key: KYLIN-1783 URL: https://issues.apache.org/jira/browse/KYLIN-1783 Project: Kylin

[jira] [Created] (KYLIN-1819) Exception swallowed when start DefaultScheduler fail

2016-06-24 Thread Ma Gang (JIRA)
Ma Gang created KYLIN-1819: -- Summary: Exception swallowed when start DefaultScheduler fail Key: KYLIN-1819 URL: https://issues.apache.org/jira/browse/KYLIN-1819 Project: Kylin Issue Type: Bug

[jira] [Created] (KYLIN-1827) Send mail notification when runtime exception throws during build/merge cube

2016-06-27 Thread Ma Gang (JIRA)
Ma Gang created KYLIN-1827: -- Summary: Send mail notification when runtime exception throws during build/merge cube Key: KYLIN-1827 URL: https://issues.apache.org/jira/browse/KYLIN-1827 Project: Kylin

[jira] [Created] (KYLIN-1932) Query did not filter unrelated cuboid shards when cuboid is shard on specific column

2016-08-01 Thread Ma Gang (JIRA)
Ma Gang created KYLIN-1932: -- Summary: Query did not filter unrelated cuboid shards when cuboid is shard on specific column Key: KYLIN-1932 URL: https://issues.apache.org/jira/browse/KYLIN-1932 Project

[jira] [Created] (KYLIN-2965) RealizationChooser cost calculation logic different with CubeInstance

2017-10-24 Thread Ma Gang (JIRA)
Ma Gang created KYLIN-2965: -- Summary: RealizationChooser cost calculation logic different with CubeInstance Key: KYLIN-2965 URL: https://issues.apache.org/jira/browse/KYLIN-2965 Project: Kylin

[jira] [Created] (KYLIN-3373) Some improvements for lookup table - UI part change

2018-05-09 Thread Ma Gang (JIRA)
Ma Gang created KYLIN-3373: -- Summary: Some improvements for lookup table - UI part change Key: KYLIN-3373 URL: https://issues.apache.org/jira/browse/KYLIN-3373 Project: Kylin Issue Type: Sub-task

[jira] [Created] (KYLIN-3374) Some improvements for lookup table - metadata change

2018-05-09 Thread Ma Gang (JIRA)
Ma Gang created KYLIN-3374: -- Summary: Some improvements for lookup table - metadata change Key: KYLIN-3374 URL: https://issues.apache.org/jira/browse/KYLIN-3374 Project: Kylin Issue Type: Sub-task

[jira] [Created] (KYLIN-3377) Some improvements for lookup table - snapshot management

2018-05-09 Thread Ma Gang (JIRA)
Ma Gang created KYLIN-3377: -- Summary: Some improvements for lookup table - snapshot management Key: KYLIN-3377 URL: https://issues.apache.org/jira/browse/KYLIN-3377 Project: Kylin Issue Type: Sub

[jira] [Created] (KYLIN-3375) Some improvements for lookup table - build change

2018-05-09 Thread Ma Gang (JIRA)
Ma Gang created KYLIN-3375: -- Summary: Some improvements for lookup table - build change Key: KYLIN-3375 URL: https://issues.apache.org/jira/browse/KYLIN-3375 Project: Kylin Issue Type: Sub-task

[jira] [Created] (KYLIN-3376) Some improvements for lookup table - query change

2018-05-09 Thread Ma Gang (JIRA)
Ma Gang created KYLIN-3376: -- Summary: Some improvements for lookup table - query change Key: KYLIN-3376 URL: https://issues.apache.org/jira/browse/KYLIN-3376 Project: Kylin Issue Type: Sub-task

[jira] [Created] (KYLIN-3396) NPE throws when materialize lookup table to HBase

2018-06-03 Thread Ma Gang (JIRA)
Ma Gang created KYLIN-3396: -- Summary: NPE throws when materialize lookup table to HBase Key: KYLIN-3396 URL: https://issues.apache.org/jira/browse/KYLIN-3396 Project: Kylin Issue Type: Improvement

[jira] [Created] (KYLIN-3209) Optimize job partial statistics path is inconsistent with existing one

2018-01-29 Thread Ma Gang (JIRA)
Ma Gang created KYLIN-3209: -- Summary: Optimize job partial statistics path is inconsistent with existing one Key: KYLIN-3209 URL: https://issues.apache.org/jira/browse/KYLIN-3209 Project: Kylin

[jira] [Created] (KYLIN-3221) Some improvements for lookup table

2018-01-31 Thread Ma Gang (JIRA)
Ma Gang created KYLIN-3221: -- Summary: Some improvements for lookup table Key: KYLIN-3221 URL: https://issues.apache.org/jira/browse/KYLIN-3221 Project: Kylin Issue Type: Improvement

[jira] [Created] (KYLIN-3522) PrepareStatement cache issue

2018-08-31 Thread Ma Gang (JIRA)
Ma Gang created KYLIN-3522: -- Summary: PrepareStatement cache issue Key: KYLIN-3522 URL: https://issues.apache.org/jira/browse/KYLIN-3522 Project: Kylin Issue Type: Improvement Components

[jira] [Created] (KYLIN-3320) CubeStatsReader cannot print stats properly for some cube

2018-03-26 Thread Ma Gang (JIRA)
Ma Gang created KYLIN-3320: -- Summary: CubeStatsReader cannot print stats properly for some cube Key: KYLIN-3320 URL: https://issues.apache.org/jira/browse/KYLIN-3320 Project: Kylin Issue Type

[jira] [Created] (KYLIN-3434) Support prepare statement in Kylin server side

2018-06-29 Thread Ma Gang (JIRA)
Ma Gang created KYLIN-3434: -- Summary: Support prepare statement in Kylin server side Key: KYLIN-3434 URL: https://issues.apache.org/jira/browse/KYLIN-3434 Project: Kylin Issue Type: Improvement

[jira] [Created] (KYLIN-3632) Add configuration that can switch on/off preparedStatement cache in Kylin server

2018-10-14 Thread Ma Gang (JIRA)
Ma Gang created KYLIN-3632: -- Summary: Add configuration that can switch on/off preparedStatement cache in Kylin server Key: KYLIN-3632 URL: https://issues.apache.org/jira/browse/KYLIN-3632 Project: Kylin

[jira] [Created] (KYLIN-3629) NullPointException throws when use preparedStatement cache in some case

2018-10-13 Thread Ma Gang (JIRA)
Ma Gang created KYLIN-3629: -- Summary: NullPointException throws when use preparedStatement cache in some case Key: KYLIN-3629 URL: https://issues.apache.org/jira/browse/KYLIN-3629 Project: Kylin

[jira] [Created] (KYLIN-3654) New Kylin Streaming

2018-10-30 Thread Ma Gang (JIRA)
Ma Gang created KYLIN-3654: -- Summary: New Kylin Streaming Key: KYLIN-3654 URL: https://issues.apache.org/jira/browse/KYLIN-3654 Project: Kylin Issue Type: New Feature Components: Job

[jira] [Created] (KYLIN-3692) New streaming ui implementation

2018-11-16 Thread Ma Gang (JIRA)
Ma Gang created KYLIN-3692: -- Summary: New streaming ui implementation Key: KYLIN-3692 URL: https://issues.apache.org/jira/browse/KYLIN-3692 Project: Kylin Issue Type: Sub-task Reporter

[jira] [Created] (KYLIN-3691) New streaming ui implementation

2018-11-16 Thread Ma Gang (JIRA)
Ma Gang created KYLIN-3691: -- Summary: New streaming ui implementation Key: KYLIN-3691 URL: https://issues.apache.org/jira/browse/KYLIN-3691 Project: Kylin Issue Type: Sub-task Reporter

[jira] [Created] (KYLIN-3690) New streaming backend implementation

2018-11-16 Thread Ma Gang (JIRA)
Ma Gang created KYLIN-3690: -- Summary: New streaming backend implementation Key: KYLIN-3690 URL: https://issues.apache.org/jira/browse/KYLIN-3690 Project: Kylin Issue Type: Sub-task

[jira] [Created] (KYLIN-3536) PrepareStatement cache issue when there are new segments built

2018-09-04 Thread Ma Gang (JIRA)
Ma Gang created KYLIN-3536: -- Summary: PrepareStatement cache issue when there are new segments built Key: KYLIN-3536 URL: https://issues.apache.org/jira/browse/KYLIN-3536 Project: Kylin Issue Type

[jira] [Created] (KYLIN-3745) real-time segment state changed from active to immutable is not sequently

2018-12-27 Thread Ma Gang (JIRA)
Ma Gang created KYLIN-3745: -- Summary: real-time segment state changed from active to immutable is not sequently Key: KYLIN-3745 URL: https://issues.apache.org/jira/browse/KYLIN-3745 Project: Kylin

[jira] [Created] (KYLIN-3747) Use FQDN to register a streaming receiver instead of ip

2018-12-28 Thread Ma Gang (JIRA)
Ma Gang created KYLIN-3747: -- Summary: Use FQDN to register a streaming receiver instead of ip Key: KYLIN-3747 URL: https://issues.apache.org/jira/browse/KYLIN-3747 Project: Kylin Issue Type: Sub

[jira] [Created] (KYLIN-3768) Save streaming metadata a standard kylin path in zookeeper

2019-01-14 Thread Ma Gang (JIRA)
Ma Gang created KYLIN-3768: -- Summary: Save streaming metadata a standard kylin path in zookeeper Key: KYLIN-3768 URL: https://issues.apache.org/jira/browse/KYLIN-3768 Project: Kylin Issue Type: Sub

[jira] [Created] (KYLIN-3787) NPE throws when dimension value has null when query real-time data

2019-01-24 Thread Ma Gang (JIRA)
Ma Gang created KYLIN-3787: -- Summary: NPE throws when dimension value has null when query real-time data Key: KYLIN-3787 URL: https://issues.apache.org/jira/browse/KYLIN-3787 Project: Kylin Issue

[jira] [Created] (KYLIN-3821) Expose real-time streaming data consuming lag info

2019-02-21 Thread Ma Gang (JIRA)
Ma Gang created KYLIN-3821: -- Summary: Expose real-time streaming data consuming lag info Key: KYLIN-3821 URL: https://issues.apache.org/jira/browse/KYLIN-3821 Project: Kylin Issue Type: Improvement

[jira] [Created] (KYLIN-3789) Stream receiver FQDN is too long to show on GUI

2019-01-25 Thread Ma Gang (JIRA)
Ma Gang created KYLIN-3789: -- Summary: Stream receiver FQDN is too long to show on GUI Key: KYLIN-3789 URL: https://issues.apache.org/jira/browse/KYLIN-3789 Project: Kylin Issue Type: Sub-task

[jira] [Created] (KYLIN-3797) Too many or filters may break Kylin server when flatting filter

2019-01-29 Thread Ma Gang (JIRA)
Ma Gang created KYLIN-3797: -- Summary: Too many or filters may break Kylin server when flatting filter Key: KYLIN-3797 URL: https://issues.apache.org/jira/browse/KYLIN-3797 Project: Kylin Issue

[jira] [Created] (KYLIN-3955) Real-time streaming tech blog

2019-04-12 Thread Ma Gang (JIRA)
Ma Gang created KYLIN-3955: -- Summary: Real-time streaming tech blog Key: KYLIN-3955 URL: https://issues.apache.org/jira/browse/KYLIN-3955 Project: Kylin Issue Type: Improvement Components