github-actions[bot] commented on code in PR #67542:
URL: https://github.com/apache/doris/pull/67542#discussion_r3954305194
##########
fe/fe-core/src/main/java/org/apache/doris/catalog/OlapTable.java:
##########
@@ -1945,6 +1945,18 @@ public long getRowCountForPartitionIndex(long
partitionId, long indexId, boolean
return index.getRowCount() == -1 ? 0 : index.getRowCount();
}
+ /**
+ * Whether the table has any partition that has ever been loaded.
+ */
+ public boolean hasData() {
+ for (Partition p : idToPartition.values()) {
+ if (p.hasData()) {
Review Comment:
[P1] Make the cloud emptiness check authoritative and job-scoped
`p.hasData()` is not authoritative across FE promotion: a follower can cache
version 1, miss the old master's best-effort async push, then be promoted while
MetaService is already at version 2. Analysis threads have no `ConnectContext`,
so the default `Long.MAX_VALUE` TTL returns stale version 1 until the periodic
background sync, and this path can persist the same empty stats. On cache
misses, this loop also makes one synchronous non-batch RPC per partition, and
per-column tasks can repeat it. Please obtain one fresh batched MetaService
snapshot and share the table result once per analysis job (or synchronously
invalidate on promotion), with failover and request-count tests.
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/analysis/OlapAnalysisTask.java:
##########
@@ -89,7 +89,9 @@ public void doExecute() throws Exception {
return;
}
// For empty table, write empty result directly, no need to run SQL to
collect stats.
- if (info.rowCount == 0 && tableSample != null) {
+ // Only take the fast path when no partition has ever been loaded.
+ if (info.rowCount == 0 && tableSample != null
+ && tbl instanceof OlapTable && !((OlapTable) tbl).hasData()) {
Review Comment:
[P1] Reconcile the stale row count after falling through
Falling through here still leaves the stale zero in two downstream places.
`doSample()` selects `FULL` and changes the result row count to `COUNT(1)`, but
the single UNIQUE/AGG-key branch keeps NDV as literal 0, so a nonempty result
with min/max is rejected and cached as `ColumnStatistic.UNKNOWN`. `runQuery()`
also records the actual count only in `indexesRowCount`; immutable
`jobInfo.rowCount` remains 0 and is copied into
`TableStatsMeta`/`ColStatsMeta`, so `SHOW TABLE STATS` stays at 0 and later
tablet reporting triggers redundant analysis. Please derive both full-scan NDV
and completed-job row count from the authoritative query result, with an
unmocked finalization test.
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/analysis/OlapAnalysisTask.java:
##########
@@ -89,7 +89,9 @@ public void doExecute() throws Exception {
return;
}
// For empty table, write empty result directly, no need to run SQL to
collect stats.
- if (info.rowCount == 0 && tableSample != null) {
+ // Only take the fast path when no partition has ever been loaded.
Review Comment:
[P1] Fence empty results against a concurrent first publish
This decision is not atomic with persisting the empty result. A first
publish or insert-overwrite can advance the partition version after `hasData()`
returns false and set `partitionChanged`, then this task appends empty stats;
all-column job finalization clears that marker (or creates metadata with it
false if none existed), so the stale result can survive the completed load.
Please compare the captured/current table or partition version before accepting
empty output and preserve or reschedule first-load invalidation when the
version advances during the job; add a latch-based race test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
