deniskuzZ commented on code in PR #6662:
URL: https://github.com/apache/hive/pull/6662#discussion_r3690603376
##########
ql/src/java/org/apache/hadoop/hive/ql/optimizer/StatsOptimizer.java:
##########
@@ -933,29 +934,60 @@ private Collection<List<ColumnStatisticsObj>>
verifyAndGetPartColumnStats(
}
private Long getRowCnt(TableScanOperator tsOp, Table tbl) throws
HiveException {
- long rowCnt = 0L;
+ if (tbl.isNonNative()) {
+ return getRowCntFromStorageHandler(tsOp, tbl);
+ }
final List<Partish> partishList;
- if (tbl.isPartitioned() &&
StatsUtils.checkCanProvidePartitionStats(tbl)) {
+ if (tbl.isPartitioned()) {
partishList = pctx.getPrunedPartitions(tsOp.getConf().getAlias(),
tsOp).getPartitions().stream()
.map(Partish::buildFor)
- .collect(Collectors.toList());
+ .toList();
} else {
partishList = Lists.newArrayList(Partish.buildFor(tbl));
}
+ long rowCnt = 0L;
for (Partish partish : partishList) {
Map<String, String> basicStats = partish.getPartParameters();
- if (tbl.isNonNative()) {
- if (!tbl.getStorageHandler().canComputeQueryUsingStats(partish)) {
- return null;
- }
- basicStats = tbl.getStorageHandler().getBasicStatistics(partish);
- } else if
(!StatsUtils.areBasicStatsUptoDateForQueryAnswering(partish.getTable(),
partish.getPartParameters())) {
+ if
(!StatsUtils.areBasicStatsUptoDateForQueryAnswering(partish.getTable(),
basicStats)) {
return null;
}
- long partRowCnt =
Long.parseLong(basicStats.get(StatsSetupConst.ROW_COUNT));
- rowCnt += partRowCnt;
+ rowCnt += Long.parseLong(basicStats.get(StatsSetupConst.ROW_COUNT));
}
return rowCnt;
}
+
+ private Long getRowCntFromStorageHandler(TableScanOperator tsOp, Table
tbl) throws HiveException {
+ if (tbl.getMetaTable() != null) {
+ // metadata table scans cannot be answered from the data table's
statistics
+ return null;
+ }
+ HiveStorageHandler storageHandler = tbl.getStorageHandler();
+ if (tbl.isPartitioned()) {
+ PrunedPartitionList prunedList =
pctx.getPrunedPartitions(tsOp.getConf().getAlias(), tsOp);
+ if (!prunedList.getReferredPartCols().isEmpty()) {
Review Comment:
The zero was deliberately moved inside the referredPartCols check. Hoisting
it reintroduces a wrong answer on Iceberg tables converted from unpartitioned
to partitioned: partition enumeration skips the former unpartitioned spec, so
getPartitions() is empty while the table still has rows.
With this change applied — SELECT count(*) returns 0 while SELECT * returns
3 rows.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
