yihua commented on PR #5048:
URL: https://github.com/apache/hudi/pull/5048#issuecomment-1135460941
@boneanxs there are two ways to address the issue:
(1) if the main purpose is to make sure that the incremental reading does
not skip the empty commit metadata which is being written, the read logic of
commit metadata should consider such case. Currently,
`HoodieCommitMetadata.fromBytes()` uses `fromJsonString()` to parse the
metadata and does not throw exception. One way is to add a flag to control the
behavior so that it can throw an exception for incremental reading, thus the
empty commit metadata being written should fail the incremental read at the
moment, guaranteeing no data loss.
```
public static <T> T fromJsonString(String jsonStr, Class<T> clazz) throws
Exception {
if (jsonStr == null || jsonStr.isEmpty()) {
// For empty commit file (no data or somethings bad happen).
return clazz.newInstance();
}
return JsonUtils.getObjectMapper().readValue(jsonStr, clazz);
}
```
(2) If you do want to fix the write side, you can create new API name in
`HoodieWrapperFileSystem` sth like
`HoodieWrapperFileSystem::createImmutableFileInPath()` and differentiate the
logic based on the storage scheme, i.e., `fs.getScheme()`, inside the method.
Only for HDFS, the new logic is introduced and it's guarded by the flag. Then,
the new API is called in `HoodieActiveTimeline` and `HoodieActiveTimeline`
should not be aware of any file system level behavior difference.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
