nsivabalan commented on code in PR #14085:
URL: https://github.com/apache/hudi/pull/14085#discussion_r2429555945
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/block/HoodieDataBlock.java:
##########
@@ -260,19 +261,13 @@ public final <T> ClosableIterator<T>
getEngineRecordIterator(HoodieReaderContext
* @param <T> The type of engine-specific record representation to
return.
* @return An iterator containing the records of interest in specified type.
*/
- public final <T> ClosableIterator<T>
getEngineRecordIterator(HoodieReaderContext<T> readerContext, List<String>
keys, boolean fullKey) {
+ public final <T> ClosableIterator<T>
getEngineRecordIterator(HoodieReaderContext<T> readerContext, List<String>
keys, boolean fullKey) throws IOException {
boolean fullScan = keys.isEmpty();
-
- // Otherwise, we fetch all the records and filter out all the records, but
the
- // ones requested
- ClosableIterator<T> allRecords = getEngineRecordIterator(readerContext);
- if (fullScan) {
- return allRecords;
+ if (!fullScan) {
+ return lookupEngineRecords(keys, fullKey);
+ } else {
+ return ClosableIterator.wrap(Collections.emptyIterator());
}
Review Comment:
Myself and ethan went over this together.
essentially the caller
https://github.com/apache/hudi/blob/4e6ebba26cb097cb26cddcbb3958d99dda476320/hudi-common/src/main/java/org/apache/hudi/common/table/read/buffer/FileGroupRecordBuffer.java#L198
calls this only incase of partitions in MDT apart from FILES partition.
Hence, its safe to assume that if KeySpec is set, its meant for partitions
in MDT apart from FILES partition.
The `enablePointLookups` is indirectly set in thru constructor while
constructing the FG reader and record buffer and the
`BaseHoodieLogRecordReader`. So, the presence of predicate and keySpec means we
are looking to fetch from non FILES partition in MDT.
For col stats, RLI and SI, I added breakpoints and confirmed the behavior.
and hence you may not see we check for `enablePointLookups` here. (if you
are comparing to how we were doing this in 0.14.1 for instance)
##########
hudi-common/src/main/java/org/apache/hudi/common/table/log/block/HoodieHFileDataBlock.java:
##########
@@ -173,8 +173,30 @@ protected <T> ClosableIterator<HoodieRecord<T>>
lookupRecords(List<String> sorte
// Get writer's schema from the header
final ClosableIterator<HoodieRecord<IndexedRecord>> recordIterator =
fullKey ? reader.getRecordsByKeysIterator(sortedKeys, readerSchema)
: reader.getRecordsByKeyPrefixIterator(sortedKeys, readerSchema);
-
return new CloseableMappingIterator<>(recordIterator, data ->
(HoodieRecord<T>) data);
}
}
+
+ @Override
+ protected <T> ClosableIterator<T> lookupEngineRecords(List<String>
sortedKeys, boolean fullKey) throws IOException {
+ HoodieLogBlockContentLocation blockContentLoc =
getBlockContentLocation().get();
+
+ // NOTE: It's important to extend Hadoop configuration here to make sure
configuration
+ // is appropriately carried over
+ StorageConfiguration<?> inlineConf =
getBlockContentLocation().get().getStorage().getConf().getInline();
+ StoragePath inlinePath = InLineFSUtils.getInlineFilePath(
Review Comment:
sure.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
