[
https://issues.apache.org/jira/browse/HUDI-5556?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Alexey Kudinkin updated HUDI-5556:
----------------------------------
Description:
Currently, most of the HoodieTableMetadata APIs (getColumnStats,
getBloomFilters, etc) provide only for loading the data from MT in-memory which
shifts the burden on the caller to break their requests in chunks to make sure
these fit in memory: for ex, trying to read Bloom Filters from MT, Bloom Index
(caller) have to make sure we're not reading more than 256 filters at a time to
limit its memory footprint.
Instead HoodieTableMetadata API has to be
# Rebased to rely on HoodieData
# Provide levers to the caller whether MT should be read in-memory or
on-cluster
#
was:
Currently, most of the HoodieTableMetadata APIs (getColumnStats,
getBloomFilters, etc) provide only for loading the data from MT in-memory which
shifts the burden on the caller to break their requests in chunks to make sure
these fit in memory: for ex, trying to read Bloom Filters from MT, Bloom Index
(caller) have to make sure we're not reading more than 256 filters at a time to
limit its memory footprint.
Instead HoodieTableMetadata API has to be
# Rebased to rely on HoodieData
# Provide levers to the caller whether MT should be read in-memory or
on-cluster
> Make sure HoodieTableMetadata API allows to read both in-memory and on-cluster
> ------------------------------------------------------------------------------
>
> Key: HUDI-5556
> URL: https://issues.apache.org/jira/browse/HUDI-5556
> Project: Apache Hudi
> Issue Type: Improvement
> Components: metadata
> Reporter: Alexey Kudinkin
> Priority: Blocker
> Fix For: 0.14.0
>
>
> Currently, most of the HoodieTableMetadata APIs (getColumnStats,
> getBloomFilters, etc) provide only for loading the data from MT in-memory
> which shifts the burden on the caller to break their requests in chunks to
> make sure these fit in memory: for ex, trying to read Bloom Filters from MT,
> Bloom Index (caller) have to make sure we're not reading more than 256
> filters at a time to limit its memory footprint.
>
> Instead HoodieTableMetadata API has to be
> # Rebased to rely on HoodieData
> # Provide levers to the caller whether MT should be read in-memory or
> on-cluster
> #
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
