Davis Zhang created HUDI-9613:
---------------------------------
Summary: HoodieBackedTableMetadata.java any API does not handle
Scalable lookup needs to be revised
Key: HUDI-9613
URL: https://issues.apache.org/jira/browse/HUDI-9613
Project: Apache Hudi
Issue Type: Bug
Components: index
Reporter: Davis Zhang
Fix For: 1.2.0
Example
* verride public HoodieData<HoodieRecord<Hood
* ieMetadataPayload>> getRecordsByKeyPrefixes(HoodieData<String> keyPrefixes,
String partitionName, boolean shouldLoadInMemory,
Option<SerializableFunctionUnchecked<String, String>> keyEncodingFn) {
ValidationUtils.checkState(keyPrefixes instanceof HoodieListData,
"getRecordsByKeyPrefixes only support HoodieListData at the moment"); // Apply
key encoding if present
the check is added because
* day 1 the API took java collection as input
* later at interface level, we change it to HoodieData interface, but the
underlying DAG is not necessarily able to handle big amount of RDD data.
the underlying DAG needs to be revise to be scalable before we open these APIs
to take RDD based hoodie data.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
