[
https://issues.apache.org/jira/browse/HUDI-1503?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Shimin Yang updated HUDI-1503:
------------------------------
Description:
This ticket is to introduce a new hash based index, which can improve the
performance of write operations and speed up the queries at the same
time(removing shuffle for Spark/Hive).
The new hash-based index works with a customized hash-based partitioner, which
partition records based on the hash value of index keys and a fixed bucket
number. So there's no need to visit the existing files to determine which file
group each record belongs.
Meanwhile, the file group id, hash mode and bucket num can be used by the query
engines to eliminate shuffle introduced by aggregation and join.
We implemented an HoodieIndex based on hive hash function which used on
production environment of ByteDance for many very-large volume dataset, and we
hope this feature can be contributed to the community soon.
was:
This ticket is to introduce a new hash based index, which can improve the
performance of write operations and speed up the queries at the same
time(removing shuffle for Spark/Hive).
The new hash-based index works with a customized hash-based partitioner, which
partition records based on the hash value of index keys and a fixed bucket
number. So there's no need to visit the existing files to determine which file
group each record belongs.
Meanwhile, the file group id, hash mode and bucket num can be used by the query
engines to eliminate shuffle introduced by aggregation and join.
We implemented an HoodieIndex based on hive hash function which used in our
production environment in ByteDance for many very-large volume dataset, and we
hope this feature can be contributed to the community soon.
> Implement a Hash(Bucket)-based Index
> ------------------------------------
>
> Key: HUDI-1503
> URL: https://issues.apache.org/jira/browse/HUDI-1503
> Project: Apache Hudi
> Issue Type: Wish
> Components: Index, Performance
> Reporter: Shimin Yang
> Priority: Major
>
> This ticket is to introduce a new hash based index, which can improve the
> performance of write operations and speed up the queries at the same
> time(removing shuffle for Spark/Hive).
> The new hash-based index works with a customized hash-based partitioner,
> which partition records based on the hash value of index keys and a fixed
> bucket number. So there's no need to visit the existing files to determine
> which file group each record belongs.
> Meanwhile, the file group id, hash mode and bucket num can be used by the
> query engines to eliminate shuffle introduced by aggregation and join.
> We implemented an HoodieIndex based on hive hash function which used on
> production environment of ByteDance for many very-large volume dataset, and
> we hope this feature can be contributed to the community soon.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
