lcspinter commented on a change in pull request #2916:
URL: https://github.com/apache/hive/pull/2916#discussion_r787541067
##########
File path:
ql/src/java/org/apache/hadoop/hive/ql/txn/compactor/metrics/DeltaFilesMetricReporter.java
##########
@@ -398,13 +156,6 @@ private static long getBaseSize(AcidDirectory dir) throws
IOException {
return baseSize;
}
- private static long getModificationTime(AcidUtils.ParsedDirectory dir,
FileSystem fs) throws IOException {
- return dir.getFiles(fs, Ref.from(false)).stream()
- .map(HadoopShims.HdfsFileStatusWithId::getFileStatus)
- .mapToLong(FileStatus::getModificationTime)
- .max()
- .orElse(new Date().getTime());
- }
private static long getDirSize(AcidUtils.ParsedDirectory dir, FileSystem fs)
throws IOException {
Review comment:
I think we should collect the small delta metrics because they can
indicate issues with streaming (I wish I had such a metric when I investigated
the Northfolk escalation :) )
Also, we are doing the same calculation in the `Initiator` code
(`Initiator#sumDirSize`) to determine the compaction type.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
