[
https://issues.apache.org/jira/browse/HUDI-3582?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
shibei updated HUDI-3582:
-------------------------
Description:
In query processing, we need to scan many data blocks in HUDI table. However,
most of them may not match the query predicate even after using column
statistic info in the metadata table, row group level or page level statistics
in parquet files, etc.
The total data size of touched blocks determines the query speed, and how to
save IO has become the key point to improving query performance. To address
this problem, we introduce secondary index to improve hudi query performance.
In the initial implementation, secondary index based on lucene will be carried
out.
was:
Nowadays, record level index is mainly implemented for `tagLocation`, and
queries do not benefit from this, see
[https://github.com/apache/hudi/pull/3508] for more detail. In this issue, we
will implement record level index based on Apache Lucene to gain the following
abilities:
1. For point query, we can get file level row number from record level index,
combining parquet column index brought by Spark 3.2 to achieve accurate reading;
2. For `tagLocation`, we can directly get record location info from record
level index.
> Introduce Secondary Index to Improve HUDI Query Performance
> -----------------------------------------------------------
>
> Key: HUDI-3582
> URL: https://issues.apache.org/jira/browse/HUDI-3582
> Project: Apache Hudi
> Issue Type: New Feature
> Components: index
> Reporter: shibei
> Assignee: shibei
> Priority: Blocker
> Fix For: 0.12.0
>
>
> In query processing, we need to scan many data blocks in HUDI table. However,
> most of them may not match the query predicate even after using column
> statistic info in the metadata table, row group level or page level
> statistics in parquet files, etc.
> The total data size of touched blocks determines the query speed, and how to
> save IO has become the key point to improving query performance. To address
> this problem, we introduce secondary index to improve hudi query performance.
> In the initial implementation, secondary index based on lucene will be
> carried out.
--
This message was sent by Atlassian Jira
(v8.20.7#820007)
