Davis Zhang created HUDI-9164:
---------------------------------
Summary: Partition strategy of MDT secondary index does not match
the query pattern it serves
Key: HUDI-9164
URL: https://issues.apache.org/jira/browse/HUDI-9164
Project: Apache Hudi
Issue Type: Bug
Reporter: Davis Zhang
Fix For: 1.1.0
h3. MDT sec idx layout does not favor join efficiently
Regarding the MDT join with an incoming pruning set RDD[Internal Row], the
existing secondary index data layout does not favor batch prefix look up.
MDT secondary index are using record scheme
<data column value><separator><record key value>
So you can see all records comes with the prefix of the column value.
It adopts hash based partitioning, which means it takes Full key, hash it and
decide which file group the partition belongs to.
In case we are given a pruning set containing a bunch of {{<data column
value>}} to look up / join with MDT sec idx, because we only knew the record
prefix, which is the {{<data column value>}} , so we don't know which bucket
the potential MDT records belongs to. As a result, we need to scan all file
groups just to search 1 such value.
In case of join between the pruning set and the MDT record index this leads to
O(m*n) complexity in the worst case, where m and n are size of the 2.
This is not scalable at all.
Needs a improvements on the partition scheme to handle prefix based search at a
large scale.
As of today, we need to scan all file groups to search for record given prefix
as it is hash based partition not range based.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
