englefly commented on code in PR #68282:
URL: https://github.com/apache/doris/pull/68282#discussion_r4071362491
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/analysis/AnalysisManager.java:
##########
@@ -1487,6 +1487,29 @@ public void removeTableStats(long tableId) {
}
}
+ /**
+ * TRUNCATE TABLE removes all the data of the table, but the table itself
stays and can be loaded
+ * immediately. The stats record must be kept, otherwise the row count of
the newly loaded data can
+ * never be reported: the backends report the row count of the new empty
tablets with a delay of up to
+ * {@code tablet_stat_update_interval_second}, and without a record there
is nothing to accumulate the
+ * loaded rows into. So reset the record to the state of an empty table
instead of removing it.
+ *
+ * <p>The transition is journaled by the truncate itself: the master DDL
path and
+ * {@link InternalCatalog#replayTruncateTable} both apply it. It must not
be journaled here, a separate
+ * entry is not atomic with the truncate entry, and a crash in between
would replay a zeroed record
+ * onto the data which was never truncated.
+ */
+ public void resetTableStats(OlapTable table, long truncateTxnId) {
+ synchronized (idToTblStats) {
+ TableStatsMeta tableStats = idToTblStats.get(table.getId());
+ if (tableStats == null) {
+ tableStats = new TableStatsMeta(table);
+ idToTblStats.put(table.getId(), tableStats);
Review Comment:
Withdrawn in 343cd946a4a, which makes the fence part of this moot; the rest
is the same family as the previous comment.
The replacement of the reset record by `new TableStatsMeta(...)` in
`updateTableStats()` is the other face
of the stale analysis: the record it publishes carries the pre-truncate
collected values, and (before
this withdrawal) it also dropped the fence and the `-1` watermark made every
later update pass. Both are
handled by the same follow-up: a generation which travels with the analysis
job and is compared when its
result is applied, plus making the first-record publication use the same
synchronized create-if-absent
path as the reset so it cannot overwrite a record created in between.
##########
fe/fe-core/src/main/java/org/apache/doris/statistics/analysis/TableStatsMeta.java:
##########
@@ -130,6 +142,72 @@ public TableStatsMeta(long rowCount, AnalysisInfo
analyzedJob, TableIf table) {
update(analyzedJob, table);
}
+ /**
+ * Create a record for a table which doesn't have one yet, in the state of
an empty table. The rows
+ * loaded into the table are accumulated by {@link
AnalysisManager#replayUpdateRowsRecord}, so a record
+ * has to exist before the first load, otherwise these rows can never be
turned into a row count.
+ */
+ public TableStatsMeta(OlapTable table) {
+ this.ctlId = table.getDatabase().getCatalog().getId();
+ this.ctlName = table.getDatabase().getCatalog().getName();
+ this.dbId = table.getDatabase().getId();
+ this.dbName = table.getDatabase().getFullName();
+ this.tblId = table.getId();
+ this.tblName = table.getName();
+ this.idxId = -1;
+ this.indexesRowCount = buildEmptyIndexRowCount(table);
+ this.updatedRowsBase.set(0);
+ }
+
+ /**
+ * TRUNCATE TABLE removes all the data of the table. Reset this record
back to the state of an empty
+ * table instead of dropping it, so that the rows loaded after the
truncation can still be accumulated
+ * into {@link #updatedRows} and be reported as the row count of the table.
+ */
+ public void reset(OlapTable table, long truncateTxnId) {
+ rowCount = 0;
+ updatedRows.set(0);
+ // Nothing has been collected for the emptied table, so none of the
rows loaded from now on is
+ // included in the collected row count. They are all delta rows.
Review Comment:
Withdrawn in 343cd946a4a. Confirmed for the version you reviewed, and the
ordering you describe is why I did not try to patch it by writing the fence
first: a reader which reads the old value before the write and accumulates
after the clear still survives, since the check and the accumulation are not
atomic with the reset.
With the fence gone, the remaining window is the accepted gap (a delayed
update of a transaction whose
rows the truncation removed can still be added to the record after the
reset), which is bounded by the
backend report and documented in the commit message of 343cd946a4a. Making
it exact needs the generation
in the event, as in the other threads.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
