deepakpanda93 commented on issue #4978:
URL: https://github.com/apache/hudi/issues/4978#issuecomment-4832969128
Hi @CrazyBeeline
The Hive/realtime read path that caused this has been reworked, and log-only
file groups (file groups with only .log files and no base parquet) are now
handled explicitly in 1.x:
- The new FileGroup reader (default in 1.x) reads log-only file slices
natively.
- Even the legacy realtime path now has an explicit branch — for a log-only
split it uses a HoodieEmptyRecordReader as the base reader instead of falling
through to a wrong path:
```
// HoodieParquetRealtimeInputFormat#getRecordReader
if (shouldUseFilegroupReader(jobConf, split)) {
return super.getRecordReader(realtimeSplit, jobConf, reporter);
}
// for log only split, set the parquet reader as empty.
if (isLogFile(realtimeSplit.getPath())) {
return new HoodieRealtimeRecordReader(realtimeSplit, jobConf,
new HoodieEmptyRecordReader(realtimeSplit, jobConf));
}
```
This is exactly the case that was missing in 0.10.1 (querying xxx_rt before
the first compaction, when a file group has only log files).
Would you be able to retry your scenario on a recent Hudi 1.x release and
let us know if it's resolved? If so, we can close this out. If you still hit it
on 1.x, please share the table type, writer (Flink/Spark/HBase index), and the
stack trace and we'll dig in.
Thanks! cc @xiarixiaoyao @nsivabalan
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
