nsivabalan commented on a change in pull request #4559:
URL: https://github.com/apache/hudi/pull/4559#discussion_r800940993
##########
File path:
hudi-hadoop-mr/src/main/java/org/apache/hudi/hadoop/realtime/HoodieHFileRealtimeInputFormat.java
##########
@@ -38,32 +36,18 @@
import org.apache.log4j.Logger;
import java.io.IOException;
-import java.util.Arrays;
-import java.util.List;
-import java.util.stream.Collectors;
/**
* HoodieRealtimeInputFormat for HUDI datasets which store data in HFile base
file format.
*/
@UseRecordReaderFromInputFormat
@UseFileSplitsFromInputFormat
-public class HoodieHFileRealtimeInputFormat extends HoodieHFileInputFormat {
+public class HoodieHFileRealtimeInputFormat extends
HoodieRealtimeFileInputFormatBase {
private static final Logger LOG =
LogManager.getLogger(HoodieHFileRealtimeInputFormat.class);
- @Override
- public InputSplit[] getSplits(JobConf job, int numSplits) throws IOException
{
- List<FileSplit> fileSplits = Arrays.stream(super.getSplits(job, numSplits))
- .map(is -> (FileSplit) is)
- .collect(Collectors.toList());
- return HoodieRealtimeInputFormatUtils.getRealtimeSplits(job, fileSplits);
- }
-
- @Override
- protected HoodieDefaultTimeline filterInstantsTimeline(HoodieDefaultTimeline
timeline) {
- // no specific filtering for Realtime format
- return timeline;
- }
+ // NOTE: We're only using {@code HoodieHFileInputFormat} to compose {@code
RecordReader}
+ private final HoodieHFileInputFormat hFileInputFormat = new
HoodieHFileInputFormat();
Review comment:
I get it. but in master, we disabled log files for HFile realtime IF and
now we are enabling is it?
excerpt from HoodieHFileInputFormat in master.
```
@Override
protected boolean includeLogFilesForSnapshotView() {
return true;
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
