[
https://issues.apache.org/jira/browse/PARQUET-2237?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Mars updated PARQUET-2237:
--------------------------
Description:
If we can accurately judge by the minMax status, we don’t need to load the
dictionary from filesystem and compare one by one anymore.
Similarly , Bloomfilter needs to load from filesystem, it may costs time and
memory. If we can exactly determine the existence/nonexistence of the value
from minMax or dictionary filters , then we can avoid using Bloomfilter to
Improve performance.
For example,
# read data greater than {{x1}} in the block, if minMax in status is all
greater than {{{}x1{}}}, then we don't need to read dictionary and compare one
by one.
# If we already have page dictionaries and have compared one by one, we don't
need to read BloomFilter and compare.
was:
Bloomfilter needs to load from filesystem, it may costs time and memory. If we
can exactly determine the existence/nonexistence of the value from other
filters , then we can avoid using Bloomfilter to Improve performance.
Similarly, if we can accurately judge by the minMax status, we also don’t need
to load the dictionary from filesystem and compare one by one anymore.
For example,
# read data greater than {{x1}} in the block, if minMax in status is all
greater than {{{}x1{}}}, then we don't need to read dictionary and compare one
by one.
# If we already have page dictionaries and compare one by one already, we
don't need to read BloomFilter and compare.
> Improve performance when filters in RowGroupFilter can match exactly
> --------------------------------------------------------------------
>
> Key: PARQUET-2237
> URL: https://issues.apache.org/jira/browse/PARQUET-2237
> Project: Parquet
> Issue Type: Improvement
> Reporter: Mars
> Priority: Major
>
> If we can accurately judge by the minMax status, we don’t need to load the
> dictionary from filesystem and compare one by one anymore.
> Similarly , Bloomfilter needs to load from filesystem, it may costs time and
> memory. If we can exactly determine the existence/nonexistence of the value
> from minMax or dictionary filters , then we can avoid using Bloomfilter to
> Improve performance.
> For example,
> # read data greater than {{x1}} in the block, if minMax in status is all
> greater than {{{}x1{}}}, then we don't need to read dictionary and compare
> one by one.
> # If we already have page dictionaries and have compared one by one, we
> don't need to read BloomFilter and compare.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
