voonhous commented on code in PR #18417:
URL: https://github.com/apache/hudi/pull/18417#discussion_r3008157006
##########
hudi-common/src/main/java/org/apache/hudi/BaseHoodieTableFileIndex.java:
##########
@@ -484,9 +499,13 @@ private List<StoragePathInfo>
listPartitionPathFiles(List<PartitionPath> partiti
Function.identity()
));
+ HoodieTimer timer = HoodieTimer.start();
try {
- Map<String, List<StoragePathInfo>> fetchedPartitionsMap =
-
tableMetadata.getAllFilesInPartitions(missingPartitionPathsMap.keySet(),
getPartitionPathFilter(activeTimeline));
+ log.info("On {}, out of {} partition paths, {} are missing from cache.
Loading them.",
+ metaClient.getTableConfig().getTableName(), partitionPaths.size(),
missingPartitionPaths.size());
Review Comment:
uard the "missing from cache" log with `if (missingPartitionPaths.size() >
0)` there is no point logging when everything is cached.
##########
hudi-common/src/main/java/org/apache/hudi/BaseHoodieTableFileIndex.java:
##########
@@ -357,18 +358,26 @@ private Map<PartitionPath, List<FileSlice>>
filterFiles(List<PartitionPath> part
.orElseGet(() ->
finalFileSystemView.getLatestFileSlices(partitionPath.path))
.collect(Collectors.toList())
));
+ } finally {
+ log.info("On {} with query instant as {}, it took {}ms to filter {}
files into file slices across {} partitions",
Review Comment:
My main concern here is that the logs generated here will be too noisy.
This fires on every single query, even if it took 1ms. For a busy Spark
application hitting multiple Hudi tables, this alone generates one info line
per query.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
