[Discuss] Lookup table improvement: support global/big lookup table

[magang]

Hi all,

There are two limitations for current lookup table design:

1. lookup table size is limited, because table snapshot need to be cached in
Kylin server, too large snapshot table will break Kylin server, also the
snapshot building may take very long time when it is too large.
2. each segment has its own lookup table snapshot, but some users may need a
global snapshot table, which means when the global table is updated, the
query for all segments need to reflect the change. 

To resolve the above limitations, I have created ticket: 
https://issues.apache.org/jira/browse/KYLIN-3221
  , and put initial new
design doc there, any comments and suggestions are welcome.

--
Sent from: http://apache-kylin.74782.x6.nabble.com/

Re: What algorithm is the caching policy?

[magang]

Kylin use ehcache as the cache solution, by default, it use LRU and
timeToIdleSeconds is set to 1 day. All are configurable in ehcache.xml, by
default it is:










--
Sent from: http://apache-kylin.74782.x6.nabble.com/

Re: Why 'is null' is not evaluable?

[magang]

Yang wrote
> Looks like a bug to me.
> 
> On Thu, Apr 27, 2017 at 3:52 PM, magang 

> mg4work@

>  wrote:
> 
>> Hi,
>>
>> When sql contains filter: 'is null'/'is not null', the filter will not be
>> push down to coprocessor, because the CompareTupleFilter.isEvaluable()
>> method return false, is it by design or just a bug?
>>
>> related code is here:
>> https://github.com/apache/kylin/blob/master/core-
>> metadata/src/main/java/org/apache/kylin/metadata/filter/
>> CompareTupleFilter.java#L216
>>
>> --
>> View this message in context: http://apache-kylin.74782.x6.
>> nabble.com/Why-is-null-is-not-evaluable-tp7796.html
>> Sent from the Apache Kylin mailing list archive at Nabble.com.
>>

Thanks Yang! Will log a ticket to track it

--
View this message in context: 
http://apache-kylin.74782.x6.nabble.com/Why-is-null-is-not-evaluable-tp7796p7861.html
Sent from the Apache Kylin mailing list archive at Nabble.com.

Why 'is null' is not evaluable?

[magang]

Hi,

When sql contains filter: 'is null'/'is not null', the filter will not be
push down to coprocessor, because the CompareTupleFilter.isEvaluable()
method return false, is it by design or just a bug?

related code is here:
https://github.com/apache/kylin/blob/master/core-metadata/src/main/java/org/apache/kylin/metadata/filter/CompareTupleFilter.java#L216

--
View this message in context: 
http://apache-kylin.74782.x6.nabble.com/Why-is-null-is-not-evaluable-tp7796.html
Sent from the Apache Kylin mailing list archive at Nabble.com.