[
https://issues.apache.org/jira/browse/KYLIN-5564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Guangyuan Feng updated KYLIN-5564:
----------------------------------
Description:
Currently, all the data generated by Kylin are saved as *Parquet* files through
Spark, but Kylin has not make full use of the features of Parquet when scanning
data. Among them, BloomFilter must be stressed, because it's the most common
tool to help *READERs* to skip useless data.
Therefore, we introduced an approach to build *BloomFilter* automatically,
conditionally and smartly when constructing segments, on the desired columns
especially according to the query histories.
_After brought in BloomFilter, Spark will have a good performance improvement
in the most cases._
was:
Currently, all the data generated by Kylin are saved as Parquet files through
Spark, but Kylin has not make full use of the features of Parquet when scanning
data. Among them, BloomFilter must be stressed, because it's the most common
tool to help READERs to skip useless data.
Therefore, we introduced a approach to build BloomFilter automatically,
conditionally and smartly when constructing segments, on the desired columns
especially according to the query histories.
After brought in BloomFilter, Spark will have a good performance improvement in
the most cases.
> Introduce Bloom Filter to optimize data scanning based on Spark
> ---------------------------------------------------------------
>
> Key: KYLIN-5564
> URL: https://issues.apache.org/jira/browse/KYLIN-5564
> Project: Kylin
> Issue Type: Improvement
> Components: Query Engine
> Affects Versions: 5.0-alpha
> Reporter: Guangyuan Feng
> Assignee: Guangyuan Feng
> Priority: Major
> Fix For: 5.0-alpha
>
>
> Currently, all the data generated by Kylin are saved as *Parquet* files
> through Spark, but Kylin has not make full use of the features of Parquet
> when scanning data. Among them, BloomFilter must be stressed, because it's
> the most common tool to help *READERs* to skip useless data.
> Therefore, we introduced an approach to build *BloomFilter* automatically,
> conditionally and smartly when constructing segments, on the desired columns
> especially according to the query histories.
>
> _After brought in BloomFilter, Spark will have a good performance improvement
> in the most cases._
>
>
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
