danny0405 commented on code in PR #11518:
URL: https://github.com/apache/hudi/pull/11518#discussion_r1682132235
##########
hudi-common/src/main/java/org/apache/hudi/common/model/HoodieCommitMetadata.java:
##########
@@ -248,30 +245,16 @@ public static <T> T fromJsonString(String jsonStr,
Class<T> clazz) throws Except
* parse the bytes of deltacommit, and get the base file and the log files
belonging to this
* provided file group.
*/
- // TODO: refactor this method to avoid doing the json tree walking
(HUDI-4822).
public static Option<Pair<String, List<String>>>
getFileSliceForFileGroupFromDeltaCommit(byte[] bytes, HoodieFileGroupId
fileGroupId) {
try {
- String jsonStr = fromUTF8Bytes(
- convertCommitMetadataToJsonBytes(deserializeCommitMetadata(bytes),
org.apache.hudi.avro.model.HoodieCommitMetadata.class));
- if (jsonStr.isEmpty()) {
- return Option.empty();
- }
- JsonNode ptToWriteStatsMap =
JsonUtils.getObjectMapper().readTree(jsonStr).get("partitionToWriteStats");
- Iterator<Map.Entry<String, JsonNode>> pts = ptToWriteStatsMap.fields();
- while (pts.hasNext()) {
- Map.Entry<String, JsonNode> ptToWriteStats = pts.next();
- if (ptToWriteStats.getValue().isArray()) {
- for (JsonNode writeStat : ptToWriteStats.getValue()) {
- HoodieFileGroupId fgId = new
HoodieFileGroupId(ptToWriteStats.getKey(), writeStat.get("fileId").asText());
- if (fgId.equals(fileGroupId)) {
- String baseFile = writeStat.get("baseFile").asText();
- ArrayNode logFilesNode = (ArrayNode) writeStat.get("logFiles");
- List<String> logFiles = new ArrayList<>();
- for (JsonNode logFile : logFilesNode) {
- logFiles.add(logFile.asText());
- }
- return Option.of(Pair.of(baseFile, logFiles));
- }
+ org.apache.hudi.avro.model.HoodieCommitMetadata commitMetadata =
deserializeCommitMetadata(bytes);
Review Comment:
The only invoker `HoodieCDCExtractor.getDependentFileSliceForLogFile` has a
logic like this:
```java
try {
HoodieBaseFile baseFile = fileSlice.getLeft().isEmpty()
? null
: new HoodieBaseFile(storage.getPathInfo(new
StoragePath(partitionPath, fileSlice.getLeft())));
```
That means you should either returns null and check the null val or returns
empty string that matches the check.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
