dengzhhu653 commented on code in PR #5950:
URL: https://github.com/apache/hive/pull/5950#discussion_r2437918716
##########
ql/src/java/org/apache/hadoop/hive/ql/stats/BasicStatsTask.java:
##########
@@ -182,6 +186,59 @@ public Object process(StatsAggregator statsAggregator)
throws HiveException, Met
parameters.putAll(providedBasicStats);
}
+ try {
+ long totalSize = 0L, numFiles = 0L;
+ // Read stats from partition parameters only
+ // Parameters were already populated via populateQuickStats or
providedBasicStats.putAll
+ String ts = parameters.get(StatsSetupConst.TOTAL_SIZE);
+ String nf = parameters.get(StatsSetupConst.NUM_FILES);
+
+ if (ts != null && nf != null) {
+ try {
+ totalSize = Long.parseLong(ts);
+ numFiles = Long.parseLong(nf);
+ } catch (NumberFormatException ignore) {
+ // If parsing fails, skip the warning without affecting the main
flow
+ totalSize = 0L;
+ numFiles = 0L;
+ }
+ }
+
+ // Only warn when there are multiple files and total size > 0
+ if (numFiles > 100 && totalSize > 0) {
+ long threshold = (conf != null)
+ ?
conf.getLongVar(HiveConf.ConfVars.HIVE_MERGE_MAP_FILES_AVG_SIZE)
+ :
HiveConf.ConfVars.HIVE_MERGE_MAP_FILES_AVG_SIZE.defaultLongVal;
+
+ long avg = totalSize / numFiles;
+
+ if (avg <= threshold) {
Review Comment:
here we can also use
```
String who = (partish.getPartition() == null) ? ("table " +
partish.getTable().getFullyQualifiedName())
: ("partition " +
partish.getPartition().getName());
long threshold =
conf.getLongVar(HiveConf.ConfVars.HIVE_MERGE_MAP_FILES_AVG_SIZE);
SmallFilesWarningUtil.smallFilesWarnings(parameters, 100L,
threshold, who, "[ANALYZE]")
.ifPresent(msg -> {
LOG.warn(msg);
SessionState ss = SessionState.get();
if (ss != null && ss.getConsole() != null)
{
ss.getConsole().printInfo(msg);
}
});
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
