Re: [DISCUSS] Technical design for rate(), irate(), increase(), and delta() aggregate functions

[Xinqi Zhao]

Hi Hongyin, Yuan, and all,

Sorry for the delayed reply.

Thank you, Hongyin, for following this proposal and offering to participate in 
further discussion and review. I have now completed the final technical design, 
including the ordered-input property, distributed intermediate-state handling, 
and memory-management approach that you mentioned.

Yuan, regarding your question about pushing the four functions down to 
TableScan and combining AggregationNode with TableScan into an 
AggregationTableScanNode: I considered this approach, but decided not to 
support the pushdown in the current design.

The main reasons are as follows.

First, all aggregate functions in one AggregationTableScanNode share a single 
underlying scan direction, either ASC or DESC.

For existing functions such as first() and last(), the direction is mainly a 
performance preference: first() prefers ASC but can still process DESC input, 
while last() prefers DESC but can still process ASC input.

The Ordered implementations of rate(), increase(), irate(), and delta(), 
however, require their time_col input to be strictly ascending. This is a 
correctness requirement rather than a performance preference.

For example:

SELECT
rate(counter, time, window_start, window_end),
last(value, time)
FROM table1
GROUP BY device_id;

rate() requires ASC input, while last() prefers DESC input. The current 
scan-direction selection mechanism cannot represent the mandatory ASC 
requirement of rate(). If DESC is selected, the Ordered rate implementation 
cannot process the input safely.

Second, the ordering semantics of these functions are determined by the 
explicitly supplied time_col argument. This argument may be an INT64 or 
TIMESTAMP column other than the table’s physical TIME column.

The scanAscending property of AggregationTableScanNode only describes the scan 
order of the physical TIME column. It cannot prove that an arbitrary time_col 
argument is ordered.

Therefore, the final design derives a separate inputOrderedByTimeAscending 
property from the OrderingScheme of the finalized physical child. The Ordered 
implementation is selected only when the planner can prove that samples are 
ascending by the actual time_col within each SQL aggregation group.

Third, these functions require the original timestamp/value samples for 
sorting, duplicate-timestamp validation, counter-reset detection, and boundary 
extrapolation. They cannot be calculated only from file, chunk, or page 
statistics, so the main statistical benefit of aggregation pushdown does not 
apply.

Finally, the current optimizer makes the pushdown decision for the complete 
AggregationNode. It does not split one node into pushed-down and 
non-pushed-down aggregate functions.

For example:

SELECT
count(value),
sum(value),
rate(counter, time, window_start, window_end)
FROM table1
GROUP BY device_id;

If rate() is not eligible for pushdown, pushing down only count() and sum() 
would require splitting the AggregationNode and recombining the results. I 
consider that additional planner change outside the scope of this feature.

Based on these considerations, the final design disables aggregation pushdown 
whenever an AggregationNode contains rate(), increase(), irate(), or delta(). 
This is an intentional performance trade-off and does not affect query 
semantics. Existing functions such as count() and sum() remain eligible for 
pushdown when they are used without a rate-family function.

Best regards,
Xinqi Zhao




         原始邮件
         
       
发件人：张洪胤 <sprico...@163.com&gt;
发件时间：2026年7月29日 11:04
收件人：dev <dev@iotdb.apache.org&gt;
主题：Re: [DISCUSS] Technical design for rate(), irate(), increase(), and delta() 
aggregate functions



       Hi&nbsp;Xinqi&nbsp;and&nbsp;Yuan,

Thank&nbsp;you&nbsp;for&nbsp;sharing&nbsp;the&nbsp;design&nbsp;and&nbsp;the&nbsp;detailed&nbsp;review&nbsp;comments.&nbsp;I&nbsp;am&nbsp;following&nbsp;this&nbsp;proposal&nbsp;closely,&nbsp;particularly&nbsp;the&nbsp;ordered-input&nbsp;property,&nbsp;distributed&nbsp;intermediate-state&nbsp;handling,&nbsp;and&nbsp;memory-management&nbsp;approach.

This&nbsp;is&nbsp;an&nbsp;important&nbsp;addition&nbsp;to&nbsp;the&nbsp;IoTDB&nbsp;table&nbsp;model,&nbsp;and&nbsp;I&nbsp;look&nbsp;forward&nbsp;to&nbsp;the&nbsp;next&nbsp;revision&nbsp;and&nbsp;implementation&nbsp;progress.&nbsp;I&nbsp;would&nbsp;also&nbsp;be&nbsp;happy&nbsp;to&nbsp;participate&nbsp;in&nbsp;further&nbsp;discussion&nbsp;and&nbsp;review.

Best&nbsp;regards,
Hongyin&nbsp;Zhang


&gt;&nbsp;2026年7月24日&nbsp;17:02，Yuan&nbsp;Tian&nbsp;<jackietie...@gmail.com&gt;&nbsp;写道：
&gt;&nbsp;
&gt;&nbsp;Hi&nbsp;xinqi,
&gt;&nbsp;
&gt;&nbsp;Do&nbsp;you&nbsp;think&nbsp;about&nbsp;pushing&nbsp;these&nbsp;four&nbsp;functions&nbsp;down&nbsp;to&nbsp;TableScan&nbsp;and
&gt;&nbsp;combining&nbsp;AggNode&nbsp;with&nbsp;TableaScan&nbsp;into&nbsp;an&nbsp;AggTableScanNode.&nbsp;If&nbsp;so,&nbsp;why&nbsp;do
&gt;&nbsp;you&nbsp;choose&nbsp;to&nbsp;not&nbsp;do&nbsp;that?
&gt;&nbsp;
&gt;&nbsp;Best&nbsp;regards,
&gt;&nbsp;---------------------
&gt;&nbsp;Yuan&nbsp;Tian
&gt;&nbsp;
&gt;&nbsp;On&nbsp;Fri,&nbsp;Jul&nbsp;24,&nbsp;2026&nbsp;at&nbsp;10:33 
AM&nbsp;Xinqi&nbsp;Zhao&nbsp;<3322351...@qq.com&gt;&nbsp;wrote:
&gt;&nbsp;
&gt;&gt;&nbsp;Hi&nbsp;Yuan&nbsp;and&nbsp;all,
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;Thank&nbsp;you&nbsp;for&nbsp;the&nbsp;detailed&nbsp;review&nbsp;and&nbsp;suggestions.&nbsp;I&nbsp;have&nbsp;revised&nbsp;the
&gt;&gt;&nbsp;technical&nbsp;design&nbsp;accordingly.
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;Related&nbsp;issue:
&gt;&gt;&nbsp;https://github.com/apache/iotdb/issues/17976
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;The&nbsp;main&nbsp;changes&nbsp;are&nbsp;summarized&nbsp;below.
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;Separate&nbsp;Ordered&nbsp;and&nbsp;Naive&nbsp;accumulator&nbsp;implementations
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;The&nbsp;Ordered&nbsp;and&nbsp;Naive&nbsp;modes&nbsp;have&nbsp;been&nbsp;split&nbsp;into&nbsp;different&nbsp;concrete
&gt;&gt;&nbsp;classes&nbsp;because&nbsp;they&nbsp;maintain&nbsp;different&nbsp;states&nbsp;and&nbsp;have&nbsp;different&nbsp;lifecycle
&gt;&gt;&nbsp;requirements.
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;For&nbsp;each&nbsp;function,&nbsp;an&nbsp;abstract&nbsp;base&nbsp;class&nbsp;now&nbsp;contains&nbsp;the&nbsp;shared
&gt;&gt;&nbsp;validation,&nbsp;window&nbsp;semantics,&nbsp;and&nbsp;final&nbsp;calculation&nbsp;logic,&nbsp;while&nbsp;the
&gt;&gt;&nbsp;concrete&nbsp;subclasses&nbsp;implement&nbsp;the&nbsp;corresponding&nbsp;state&nbsp;management:
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;AbstractRateTableAccumulator
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;OrderedRateAccumulator
&gt;&gt;&nbsp;NaiveRateAccumulator
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;The&nbsp;same&nbsp;structure&nbsp;is&nbsp;used&nbsp;for&nbsp;increase(),&nbsp;irate(),&nbsp;and&nbsp;delta(),&nbsp;as&nbsp;well
&gt;&gt;&nbsp;as&nbsp;for&nbsp;the&nbsp;GroupedAccumulator&nbsp;path.
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;The&nbsp;accumulator&nbsp;path&nbsp;and&nbsp;the&nbsp;algorithm&nbsp;are&nbsp;treated&nbsp;as&nbsp;two&nbsp;independent
&gt;&gt;&nbsp;dimensions:
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;The&nbsp;operator&nbsp;selects&nbsp;TableAccumulator&nbsp;or&nbsp;GroupedAccumulator.
&gt;&gt;&nbsp;The&nbsp;aggregation&nbsp;step&nbsp;and&nbsp;ordering&nbsp;property&nbsp;select&nbsp;Ordered&nbsp;or&nbsp;Naive.
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;Ordered&nbsp;implementations&nbsp;are&nbsp;used&nbsp;only&nbsp;for&nbsp;SINGLE&nbsp;aggregation&nbsp;and&nbsp;keep&nbsp;O(1)
&gt;&gt;&nbsp;state&nbsp;per&nbsp;group.&nbsp;Naive&nbsp;implementations&nbsp;preserve&nbsp;all&nbsp;samples&nbsp;and&nbsp;support
&gt;&gt;&nbsp;SINGLE,&nbsp;PARTIAL,&nbsp;INTERMEDIATE,&nbsp;and&nbsp;FINAL&nbsp;stages.
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;Explicit&nbsp;derivation&nbsp;of&nbsp;the&nbsp;ordered-input&nbsp;property
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;The&nbsp;updated&nbsp;design&nbsp;defines&nbsp;inputOrderedByTimeAscending&nbsp;as&nbsp;a
&gt;&gt;&nbsp;per-aggregation&nbsp;physical&nbsp;property.&nbsp;It&nbsp;is&nbsp;derived&nbsp;in
&gt;&gt;&nbsp;TableDistributedPlanGenerator.visitAggregation()&nbsp;after&nbsp;the&nbsp;final&nbsp;physical
&gt;&gt;&nbsp;child&nbsp;of&nbsp;the&nbsp;AggregationNode&nbsp;has&nbsp;been&nbsp;established.
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;The&nbsp;proof&nbsp;uses&nbsp;the&nbsp;final&nbsp;child’s&nbsp;OrderingScheme.&nbsp;For&nbsp;a&nbsp;particular
&gt;&gt;&nbsp;aggregation:
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;The&nbsp;second&nbsp;argument&nbsp;must&nbsp;be&nbsp;a&nbsp;directly&nbsp;addressable&nbsp;time_col&nbsp;symbol.
&gt;&gt;&nbsp;Before&nbsp;time_col&nbsp;appears&nbsp;in&nbsp;the&nbsp;ordering&nbsp;prefix,&nbsp;only&nbsp;SQL&nbsp;grouping&nbsp;keys&nbsp;are
&gt;&gt;&nbsp;allowed.
&gt;&gt;&nbsp;time_col&nbsp;must&nbsp;use&nbsp;ascending&nbsp;order.
&gt;&gt;&nbsp;Ordering&nbsp;symbols&nbsp;after&nbsp;time_col&nbsp;do&nbsp;not&nbsp;affect&nbsp;the&nbsp;proof.
&gt;&gt;&nbsp;If&nbsp;the&nbsp;ordering&nbsp;is&nbsp;missing,&nbsp;uses&nbsp;descending&nbsp;time,&nbsp;contains&nbsp;a&nbsp;non-grouping
&gt;&gt;&nbsp;symbol&nbsp;before&nbsp;time_col,&nbsp;or&nbsp;otherwise&nbsp;cannot&nbsp;be&nbsp;proven,&nbsp;the&nbsp;property&nbsp;is
&gt;&gt;&nbsp;false.
&gt;&gt;&nbsp;Arbitrary&nbsp;time&nbsp;expressions&nbsp;conservatively&nbsp;use&nbsp;the&nbsp;Naive&nbsp;implementation.
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;This&nbsp;also&nbsp;means&nbsp;that&nbsp;Project,&nbsp;Exchange,&nbsp;Join,&nbsp;Union,&nbsp;HOP/TUMBLE,&nbsp;and&nbsp;other
&gt;&gt;&nbsp;operators&nbsp;are&nbsp;handled&nbsp;through&nbsp;the&nbsp;ordering&nbsp;property&nbsp;of&nbsp;the&nbsp;finalized
&gt;&gt;&nbsp;physical&nbsp;child.&nbsp;If&nbsp;they&nbsp;do&nbsp;not&nbsp;preserve&nbsp;and&nbsp;expose&nbsp;a&nbsp;sufficient
&gt;&gt;&nbsp;OrderingScheme,&nbsp;the&nbsp;Ordered&nbsp;implementation&nbsp;is&nbsp;not&nbsp;selected.
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;Only&nbsp;the&nbsp;following&nbsp;condition&nbsp;enables&nbsp;the&nbsp;Ordered&nbsp;implementation:
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;step&nbsp;==&nbsp;SINGLE&nbsp;&amp;&amp;&nbsp;inputOrderedByTimeAscending
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;PARTIAL,&nbsp;INTERMEDIATE,&nbsp;and&nbsp;FINAL&nbsp;stages&nbsp;always&nbsp;use&nbsp;the&nbsp;Naive
&gt;&gt;&nbsp;implementation.
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;The&nbsp;current&nbsp;version&nbsp;also&nbsp;disables&nbsp;aggregation&nbsp;pushdown&nbsp;to
&gt;&gt;&nbsp;AggregationTableScanNode&nbsp;and&nbsp;ExternalTsFileAggregationScanNode&nbsp;for&nbsp;these
&gt;&gt;&nbsp;functions.&nbsp;The&nbsp;explicit&nbsp;time_col&nbsp;may&nbsp;differ&nbsp;from&nbsp;the&nbsp;physical&nbsp;TIME&nbsp;column,
&gt;&gt;&nbsp;and&nbsp;the&nbsp;functions&nbsp;require&nbsp;raw&nbsp;samples&nbsp;rather&nbsp;than&nbsp;file,&nbsp;chunk,&nbsp;or&nbsp;page
&gt;&gt;&nbsp;statistics.
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;Common&nbsp;runtime&nbsp;validation&nbsp;flow
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;A&nbsp;stateless&nbsp;RateFunctionValidation&nbsp;utility&nbsp;is&nbsp;now&nbsp;shared&nbsp;by&nbsp;Ordered/Naive,
&gt;&gt;&nbsp;Table/Grouped,&nbsp;and&nbsp;the&nbsp;different&nbsp;aggregation&nbsp;stages.
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;The&nbsp;validation&nbsp;order&nbsp;and&nbsp;behavior&nbsp;are&nbsp;explicitly&nbsp;defined:
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;Rows&nbsp;with&nbsp;NULL&nbsp;value_col&nbsp;are&nbsp;ignored.
&gt;&gt;&nbsp;If&nbsp;value_col&nbsp;is&nbsp;non-NULL,&nbsp;the&nbsp;required&nbsp;time&nbsp;and&nbsp;window&nbsp;arguments&nbsp;must&nbsp;be
&gt;&gt;&nbsp;non-NULL.
&gt;&gt;&nbsp;NaN&nbsp;and&nbsp;Infinity&nbsp;are&nbsp;rejected.
&gt;&gt;&nbsp;Negative&nbsp;values&nbsp;are&nbsp;rejected&nbsp;for&nbsp;rate(),&nbsp;increase(),&nbsp;and&nbsp;irate(),&nbsp;while
&gt;&gt;&nbsp;delta()&nbsp;accepts&nbsp;negative&nbsp;values.
&gt;&gt;&nbsp;window_start&nbsp;must&nbsp;be&nbsp;less&nbsp;than&nbsp;window_end.
&gt;&gt;&nbsp;Samples&nbsp;must&nbsp;satisfy&nbsp;window_start&nbsp;<=&nbsp;time_col&nbsp;<&nbsp;window_end.
&gt;&gt;&nbsp;Window&nbsp;boundaries&nbsp;must&nbsp;be&nbsp;consistent&nbsp;within&nbsp;the&nbsp;same&nbsp;SQL&nbsp;group.
&gt;&gt;&nbsp;Duplicate&nbsp;timestamps&nbsp;are&nbsp;rejected.
&gt;&gt;&nbsp;Ordered&nbsp;implementations&nbsp;also&nbsp;reject&nbsp;an&nbsp;unexpected&nbsp;decrease&nbsp;in&nbsp;timestamps.
&gt;&gt;&nbsp;After&nbsp;filtering,&nbsp;fewer&nbsp;than&nbsp;two&nbsp;valid&nbsp;samples&nbsp;produce&nbsp;NULL.
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;Regarding&nbsp;NULL&nbsp;handling,&nbsp;I&nbsp;retained&nbsp;the&nbsp;requirement-level&nbsp;contract&nbsp;that
&gt;&gt;&nbsp;only&nbsp;a&nbsp;NULL&nbsp;value_col&nbsp;causes&nbsp;a&nbsp;row&nbsp;to&nbsp;be&nbsp;ignored.&nbsp;A&nbsp;non-NULL&nbsp;value&nbsp;with&nbsp;a
&gt;&gt;&nbsp;NULL&nbsp;time&nbsp;or&nbsp;required&nbsp;window&nbsp;boundary&nbsp;is&nbsp;treated&nbsp;as&nbsp;invalid&nbsp;input.&nbsp;This
&gt;&gt;&nbsp;rule&nbsp;is&nbsp;now&nbsp;applied&nbsp;consistently&nbsp;in&nbsp;every&nbsp;execution&nbsp;path.
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;Versioned&nbsp;Intermediate&nbsp;State&nbsp;protocol
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;The&nbsp;exact&nbsp;BLOB&nbsp;formats&nbsp;are&nbsp;now&nbsp;specified&nbsp;as&nbsp;follows:
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;rate(),&nbsp;increase(),&nbsp;and&nbsp;delta():
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;version:int32
&gt;&gt;&nbsp;windowStart:int64
&gt;&gt;&nbsp;windowEnd:int64
&gt;&gt;&nbsp;sampleCount:int32
&gt;&gt;&nbsp;repeated&nbsp;{&nbsp;timestamp:int64,&nbsp;value:double&nbsp;}
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;irate():
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;version:int32
&gt;&gt;&nbsp;sampleCount:int32
&gt;&gt;&nbsp;repeated&nbsp;{&nbsp;timestamp:int64,&nbsp;value:double&nbsp;}
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;The&nbsp;initial&nbsp;protocol&nbsp;version&nbsp;is&nbsp;1.
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;The&nbsp;state&nbsp;rules&nbsp;are&nbsp;now&nbsp;explicit:
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;A&nbsp;partial&nbsp;state&nbsp;with&nbsp;no&nbsp;valid&nbsp;samples&nbsp;is&nbsp;represented&nbsp;as&nbsp;NULL.
&gt;&gt;&nbsp;A&nbsp;one-sample&nbsp;state&nbsp;is&nbsp;preserved&nbsp;and&nbsp;encoded&nbsp;normally.
&gt;&gt;&nbsp;NULL&nbsp;intermediate&nbsp;positions&nbsp;are&nbsp;skipped.
&gt;&gt;&nbsp;Window&nbsp;boundaries&nbsp;are&nbsp;checked&nbsp;for&nbsp;consistency&nbsp;when&nbsp;states&nbsp;are&nbsp;merged.
&gt;&gt;&nbsp;Samples&nbsp;from&nbsp;different&nbsp;states&nbsp;are&nbsp;merged&nbsp;without&nbsp;relying&nbsp;on&nbsp;state&nbsp;arrival
&gt;&gt;&nbsp;order.
&gt;&gt;&nbsp;Duplicate&nbsp;timestamps&nbsp;are&nbsp;checked&nbsp;after&nbsp;all&nbsp;states&nbsp;have&nbsp;been&nbsp;merged&nbsp;and
&gt;&gt;&nbsp;sorted.
&gt;&gt;&nbsp;The&nbsp;codec&nbsp;validates&nbsp;the&nbsp;version,&nbsp;header&nbsp;and&nbsp;total&nbsp;length,&nbsp;sample&nbsp;count,
&gt;&gt;&nbsp;window&nbsp;values,&nbsp;timestamps,&nbsp;and&nbsp;sample&nbsp;values&nbsp;before&nbsp;creating&nbsp;the&nbsp;decoded
&gt;&gt;&nbsp;buffer.
&gt;&gt;&nbsp;Serialized-size&nbsp;calculations&nbsp;use&nbsp;exact&nbsp;arithmetic&nbsp;and&nbsp;must&nbsp;fit&nbsp;the
&gt;&gt;&nbsp;ByteBuffer&nbsp;integer&nbsp;capacity.
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;I&nbsp;also&nbsp;corrected&nbsp;the&nbsp;TimeValueBuffer.merge()&nbsp;issue&nbsp;identified&nbsp;in&nbsp;the
&gt;&gt;&nbsp;previous&nbsp;pseudocode:&nbsp;the&nbsp;original&nbsp;size&nbsp;is&nbsp;retained&nbsp;as&nbsp;the&nbsp;copy&nbsp;offset,&nbsp;and
&gt;&gt;&nbsp;the&nbsp;size&nbsp;is&nbsp;updated&nbsp;only&nbsp;once&nbsp;after&nbsp;copying.
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;Incremental&nbsp;grouped&nbsp;memory&nbsp;accounting
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;Grouped&nbsp;Naive&nbsp;accumulators&nbsp;no&nbsp;longer&nbsp;traverse&nbsp;every&nbsp;non-empty&nbsp;group&nbsp;after
&gt;&gt;&nbsp;each&nbsp;input&nbsp;batch.
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;TimeValueBufferBigArray&nbsp;maintains&nbsp;buffersRetainedBytes&nbsp;incrementally.&nbsp;Only
&gt;&gt;&nbsp;buffers&nbsp;modified&nbsp;by&nbsp;the&nbsp;current&nbsp;operation,&nbsp;together&nbsp;with&nbsp;any&nbsp;ObjectBigArray
&gt;&gt;&nbsp;capacity&nbsp;change,&nbsp;contribute&nbsp;to&nbsp;the&nbsp;memory&nbsp;delta.
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;As&nbsp;a&nbsp;result:
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;Appending&nbsp;one&nbsp;sample&nbsp;has&nbsp;amortized&nbsp;O(1)&nbsp;accounting&nbsp;cost.
&gt;&gt;&nbsp;Merging&nbsp;a&nbsp;state&nbsp;is&nbsp;proportional&nbsp;to&nbsp;the&nbsp;number&nbsp;of&nbsp;merged&nbsp;samples.
&gt;&gt;&nbsp;Processing&nbsp;a&nbsp;batch&nbsp;is&nbsp;proportional&nbsp;to&nbsp;the&nbsp;positions&nbsp;or&nbsp;groups&nbsp;actually
&gt;&gt;&nbsp;visited.
&gt;&gt;&nbsp;There&nbsp;is&nbsp;no&nbsp;additional&nbsp;O(blockCount&nbsp;×&nbsp;groupCount)&nbsp;scan.
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;Final&nbsp;sorting&nbsp;is&nbsp;performed&nbsp;in&nbsp;place&nbsp;using&nbsp;introsort&nbsp;on&nbsp;the&nbsp;parallel
&gt;&gt;&nbsp;timestamp&nbsp;and&nbsp;value&nbsp;arrays.&nbsp;Its&nbsp;worst-case&nbsp;time&nbsp;complexity&nbsp;is&nbsp;O(n&nbsp;log&nbsp;n),
&gt;&gt;&nbsp;with&nbsp;O(log&nbsp;n)&nbsp;auxiliary&nbsp;space.
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;Numerical&nbsp;edge&nbsp;cases
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;The&nbsp;numerical&nbsp;rules&nbsp;have&nbsp;also&nbsp;been&nbsp;clarified:
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;Timestamp&nbsp;subtraction&nbsp;uses&nbsp;BigInteger&nbsp;before&nbsp;conversion&nbsp;to&nbsp;seconds,
&gt;&gt;&nbsp;avoiding&nbsp;long&nbsp;overflow&nbsp;while&nbsp;preserving&nbsp;exact&nbsp;tick&nbsp;differences.
&gt;&gt;&nbsp;Window&nbsp;and&nbsp;sample&nbsp;intervals&nbsp;are&nbsp;validated&nbsp;before&nbsp;applying&nbsp;the&nbsp;increase&nbsp;==
&gt;&gt;&nbsp;0&nbsp;shortcut.
&gt;&gt;&nbsp;Intermediate&nbsp;and&nbsp;final&nbsp;arithmetic&nbsp;results&nbsp;are&nbsp;checked&nbsp;for&nbsp;finiteness.
&gt;&gt;&nbsp;A&nbsp;non-finite&nbsp;durationToZero&nbsp;does&nbsp;not&nbsp;cause&nbsp;zero-point&nbsp;truncation&nbsp;and&nbsp;is
&gt;&gt;&nbsp;not&nbsp;itself&nbsp;treated&nbsp;as&nbsp;an&nbsp;error.
&gt;&gt;&nbsp;sampleCount&nbsp;uses&nbsp;int&nbsp;and&nbsp;is&nbsp;updated&nbsp;with&nbsp;exact&nbsp;overflow&nbsp;checks.&nbsp;The
&gt;&gt;&nbsp;serialized&nbsp;state&nbsp;is&nbsp;also&nbsp;limited&nbsp;by&nbsp;the&nbsp;maximum&nbsp;ByteBuffer&nbsp;size.
&gt;&gt;&nbsp;All&nbsp;input&nbsp;values&nbsp;are&nbsp;represented&nbsp;internally&nbsp;as&nbsp;double.&nbsp;Therefore,&nbsp;INT64
&gt;&gt;&nbsp;values&nbsp;whose&nbsp;absolute&nbsp;value&nbsp;exceeds&nbsp;2^53&nbsp;follow&nbsp;IEEE&nbsp;754&nbsp;double&nbsp;semantics,
&gt;&gt;&nbsp;and&nbsp;unit-level&nbsp;differences&nbsp;are&nbsp;not&nbsp;guaranteed&nbsp;to&nbsp;remain&nbsp;exact.&nbsp;This
&gt;&gt;&nbsp;limitation&nbsp;is&nbsp;now&nbsp;documented&nbsp;explicitly.
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;The&nbsp;revised&nbsp;design&nbsp;continues&nbsp;to&nbsp;preserve&nbsp;complete&nbsp;samples&nbsp;during
&gt;&gt;&nbsp;distributed&nbsp;aggregation&nbsp;and&nbsp;performs&nbsp;sorting,&nbsp;duplicate&nbsp;detection,
&gt;&gt;&nbsp;counter-reset&nbsp;correction,&nbsp;and&nbsp;extrapolation&nbsp;only&nbsp;after&nbsp;all&nbsp;states&nbsp;have&nbsp;been
&gt;&gt;&nbsp;merged.
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;Best&nbsp;regards,
&gt;&gt;&nbsp;Xinqi&nbsp;Zhao
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;原始邮件
&gt;&gt;&nbsp;发件人：Yuan&nbsp;Tian&nbsp;<jackietie...@gmail.com&gt;
&gt;&gt;&nbsp;发件时间：2026年7月17日&nbsp;11:57
&gt;&gt;&nbsp;收件人：dev&nbsp;<dev@iotdb.apache.org&gt;
&gt;&gt;&nbsp;主题：Re:&nbsp;[DISCUSS]&nbsp;Technical&nbsp;design&nbsp;for&nbsp;rate(),&nbsp;irate(),&nbsp;increase(),&nbsp;and
&gt;&gt;&nbsp;delta()&nbsp;aggregate&nbsp;functions
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;Hi&nbsp;Xinqi,
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;Thanks&nbsp;for&nbsp;sharing&nbsp;the&nbsp;technical&nbsp;design.&nbsp;I&nbsp;reviewed&nbsp;the&nbsp;detailed&nbsp;design
&gt;&gt;&nbsp;document&nbsp;and&nbsp;left&nbsp;several&nbsp;inline&nbsp;comments.&nbsp;Overall,&nbsp;reusing&nbsp;the&nbsp;existing
&gt;&gt;&nbsp;table-model&nbsp;aggregation&nbsp;framework&nbsp;and&nbsp;preserving&nbsp;complete&nbsp;samples&nbsp;during
&gt;&gt;&nbsp;distributed&nbsp;aggregation&nbsp;looks&nbsp;reasonable.&nbsp;Before&nbsp;implementation,&nbsp;I&nbsp;think
&gt;&gt;&nbsp;the&nbsp;following&nbsp;points&nbsp;should&nbsp;be&nbsp;clarified&nbsp;or&nbsp;corrected.
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;&nbsp;&nbsp;1.&nbsp;Accumulator&nbsp;organization
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;The&nbsp;ordered&nbsp;and&nbsp;buffered&nbsp;modes&nbsp;maintain&nbsp;substantially&nbsp;different&nbsp;states&nbsp;and
&gt;&gt;&nbsp;follow&nbsp;different&nbsp;processing&nbsp;paths.&nbsp;I&nbsp;suggest&nbsp;reconsidering&nbsp;whether&nbsp;both
&gt;&gt;&nbsp;modes&nbsp;should&nbsp;live&nbsp;in&nbsp;the&nbsp;same&nbsp;concrete&nbsp;accumulator&nbsp;class.&nbsp;A&nbsp;common&nbsp;abstract
&gt;&gt;&nbsp;base&nbsp;class&nbsp;could&nbsp;contain&nbsp;shared&nbsp;validation&nbsp;and&nbsp;calculation&nbsp;logic,&nbsp;while&nbsp;the
&gt;&gt;&nbsp;ordered&nbsp;and&nbsp;buffered&nbsp;implementations&nbsp;could&nbsp;be&nbsp;separate&nbsp;subclasses.&nbsp;This&nbsp;may
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;reduce&nbsp;mode-specific&nbsp;branching&nbsp;and&nbsp;make&nbsp;their&nbsp;respective&nbsp;invariants&nbsp;clearer.
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;&nbsp;&nbsp;2.&nbsp;Derivation&nbsp;of&nbsp;the&nbsp;ordered-input&nbsp;property
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;The&nbsp;design&nbsp;says&nbsp;that&nbsp;the&nbsp;ordered&nbsp;implementation&nbsp;is&nbsp;selected&nbsp;when&nbsp;the
&gt;&gt;&nbsp;planner&nbsp;can&nbsp;prove&nbsp;that&nbsp;samples&nbsp;are&nbsp;ordered&nbsp;by&nbsp;time_col,&nbsp;but&nbsp;it&nbsp;does&nbsp;not&nbsp;yet
&gt;&gt;&nbsp;define&nbsp;how&nbsp;this&nbsp;proof&nbsp;is&nbsp;made.
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;The&nbsp;design&nbsp;should&nbsp;specify:
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;&nbsp;&nbsp;-&nbsp;At&nbsp;which&nbsp;physical-planning&nbsp;stage&nbsp;the&nbsp;property&nbsp;is&nbsp;derived.
&gt;&gt;&nbsp;&nbsp;&nbsp;-&nbsp;How&nbsp;per-group&nbsp;ordering,&nbsp;rather&nbsp;than&nbsp;only&nbsp;global&nbsp;ordering,&nbsp;is&nbsp;proven.
&gt;&gt;&nbsp;&nbsp;&nbsp;-&nbsp;How&nbsp;Project,&nbsp;Exchange,&nbsp;Join,&nbsp;Union,&nbsp;HOP/TUMBLE,&nbsp;and&nbsp;other&nbsp;operators
&gt;&gt;&nbsp;&nbsp;&nbsp;affect&nbsp;the&nbsp;property.
&gt;&gt;&nbsp;&nbsp;&nbsp;-&nbsp;Whether&nbsp;only&nbsp;a&nbsp;direct&nbsp;time_col&nbsp;symbol&nbsp;is&nbsp;supported;&nbsp;an&nbsp;arbitrary&nbsp;time
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;&nbsp;&nbsp;expression&nbsp;should&nbsp;conservatively&nbsp;fall&nbsp;back&nbsp;to&nbsp;the&nbsp;buffered&nbsp;implementation.
&gt;&gt;&nbsp;&nbsp;&nbsp;-&nbsp;That&nbsp;the&nbsp;property&nbsp;must&nbsp;be&nbsp;false&nbsp;whenever&nbsp;ordering&nbsp;cannot&nbsp;be&nbsp;proven.
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;Although&nbsp;inputOrderedByTimeAscending&nbsp;is&nbsp;not&nbsp;meaningful&nbsp;for&nbsp;every
&gt;&gt;&nbsp;aggregation&nbsp;function,&nbsp;keeping&nbsp;it&nbsp;in&nbsp;AggregationNode.Aggregation&nbsp;as&nbsp;a
&gt;&gt;&nbsp;serialized&nbsp;per-aggregation&nbsp;physical&nbsp;hint&nbsp;seems&nbsp;acceptable&nbsp;if&nbsp;its&nbsp;scope&nbsp;and
&gt;&gt;&nbsp;exact&nbsp;meaning&nbsp;are&nbsp;clearly&nbsp;documented.&nbsp;The&nbsp;derivation&nbsp;should&nbsp;happen&nbsp;after
&gt;&gt;&nbsp;the&nbsp;physical&nbsp;input&nbsp;is&nbsp;finalized&nbsp;rather&nbsp;than&nbsp;during&nbsp;logical&nbsp;function
&gt;&gt;&nbsp;analysis.
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;&nbsp;&nbsp;3.&nbsp;Runtime&nbsp;validation&nbsp;contract
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;The&nbsp;design&nbsp;should&nbsp;define&nbsp;a&nbsp;common&nbsp;validation&nbsp;flow&nbsp;shared&nbsp;by
&gt;&gt;&nbsp;ordered/buffered,&nbsp;Table/Grouped,&nbsp;and&nbsp;SINGLE/PARTIAL/FINAL&nbsp;paths.&nbsp;In
&gt;&gt;&nbsp;particular:
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;&nbsp;&nbsp;-&nbsp;Ignore&nbsp;samples&nbsp;whose&nbsp;value&nbsp;or&nbsp;time&nbsp;is&nbsp;NULL&nbsp;before&nbsp;validating&nbsp;valid
&gt;&gt;&nbsp;&nbsp;&nbsp;samples.
&gt;&gt;&nbsp;&nbsp;&nbsp;-&nbsp;Reject&nbsp;NaN&nbsp;and&nbsp;Infinity.
&gt;&gt;&nbsp;&nbsp;&nbsp;-&nbsp;Reject&nbsp;negative&nbsp;counter&nbsp;values&nbsp;for&nbsp;rate(),&nbsp;increase(),&nbsp;and&nbsp;irate();
&gt;&gt;&nbsp;&nbsp;&nbsp;delta()&nbsp;may&nbsp;accept&nbsp;negative&nbsp;values.
&gt;&gt;&nbsp;&nbsp;&nbsp;-&nbsp;Require&nbsp;non-NULL&nbsp;window&nbsp;bounds&nbsp;and&nbsp;window_start&nbsp;<&nbsp;window_end.
&gt;&gt;&nbsp;&nbsp;&nbsp;-&nbsp;Require&nbsp;sample&nbsp;timestamps&nbsp;to&nbsp;be&nbsp;within&nbsp;[window_start,&nbsp;window_end).
&gt;&gt;&nbsp;&nbsp;&nbsp;-&nbsp;Require&nbsp;consistent&nbsp;window&nbsp;bounds&nbsp;within&nbsp;the&nbsp;same&nbsp;group.
&gt;&gt;&nbsp;&nbsp;&nbsp;-&nbsp;Reject&nbsp;duplicate&nbsp;timestamps.
&gt;&gt;&nbsp;&nbsp;&nbsp;-&nbsp;Return&nbsp;NULL&nbsp;when&nbsp;fewer&nbsp;than&nbsp;two&nbsp;valid&nbsp;samples&nbsp;remain.
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;Without&nbsp;one&nbsp;explicitly&nbsp;defined&nbsp;common&nbsp;entry&nbsp;point&nbsp;and&nbsp;validation&nbsp;order,&nbsp;the
&gt;&gt;&nbsp;different&nbsp;accumulator&nbsp;paths&nbsp;may&nbsp;produce&nbsp;inconsistent&nbsp;behavior.
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;&nbsp;&nbsp;4.&nbsp;Intermediate-state&nbsp;format&nbsp;and&nbsp;merge&nbsp;behavior
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;The&nbsp;exact&nbsp;binary&nbsp;format&nbsp;should&nbsp;be&nbsp;specified,&nbsp;for&nbsp;example:
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;rate/increase/delta:
&gt;&gt;&nbsp;version&nbsp;|&nbsp;windowStart&nbsp;|&nbsp;windowEnd&nbsp;|&nbsp;sampleCount&nbsp;|&nbsp;samples
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;irate:
&gt;&gt;&nbsp;version&nbsp;|&nbsp;sampleCount&nbsp;|&nbsp;samples
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;The&nbsp;design&nbsp;should&nbsp;also&nbsp;define:
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;&nbsp;&nbsp;-&nbsp;Whether&nbsp;a&nbsp;partial&nbsp;state&nbsp;with&nbsp;no&nbsp;valid&nbsp;samples&nbsp;is&nbsp;NULL&nbsp;or&nbsp;has&nbsp;sampleCount
&gt;&gt;&nbsp;&nbsp;&nbsp;=&nbsp;0.
&gt;&gt;&nbsp;&nbsp;&nbsp;-&nbsp;How&nbsp;a&nbsp;one-sample&nbsp;partial&nbsp;state&nbsp;is&nbsp;preserved.
&gt;&gt;&nbsp;&nbsp;&nbsp;-&nbsp;How&nbsp;addIntermediate()&nbsp;handles&nbsp;NULL&nbsp;positions.
&gt;&gt;&nbsp;&nbsp;&nbsp;-&nbsp;How&nbsp;inconsistent&nbsp;window&nbsp;bounds&nbsp;and&nbsp;duplicate&nbsp;timestamps&nbsp;across&nbsp;states
&gt;&gt;&nbsp;&nbsp;&nbsp;are&nbsp;handled.
&gt;&gt;&nbsp;&nbsp;&nbsp;-&nbsp;How&nbsp;unknown&nbsp;versions,&nbsp;truncated&nbsp;states,&nbsp;trailing&nbsp;bytes,&nbsp;and&nbsp;oversized
&gt;&gt;&nbsp;&nbsp;&nbsp;BLOBs&nbsp;are&nbsp;handled.
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;There&nbsp;is&nbsp;also&nbsp;a&nbsp;correctness&nbsp;issue&nbsp;in&nbsp;the&nbsp;current&nbsp;TimeValueBuffer.merge()
&gt;&gt;&nbsp;pseudocode:&nbsp;it&nbsp;assigns&nbsp;size&nbsp;=&nbsp;mergedSize&nbsp;before&nbsp;copying&nbsp;and&nbsp;then&nbsp;copies
&gt;&gt;&nbsp;from&nbsp;offset&nbsp;size,&nbsp;which&nbsp;will&nbsp;cause&nbsp;an&nbsp;out-of-bounds&nbsp;access.&nbsp;The&nbsp;old&nbsp;size
&gt;&gt;&nbsp;should&nbsp;be&nbsp;retained&nbsp;as&nbsp;the&nbsp;copy&nbsp;offset,&nbsp;and&nbsp;size&nbsp;should&nbsp;be&nbsp;updated&nbsp;only&nbsp;once
&gt;&gt;&nbsp;after&nbsp;the&nbsp;copy.
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;&nbsp;&nbsp;5.&nbsp;Grouped&nbsp;memory&nbsp;accounting
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;The&nbsp;proposed&nbsp;grouped&nbsp;implementation&nbsp;recalculates&nbsp;retained&nbsp;memory&nbsp;by
&gt;&gt;&nbsp;traversing&nbsp;every&nbsp;non-empty&nbsp;group&nbsp;after&nbsp;each&nbsp;input&nbsp;or&nbsp;intermediate&nbsp;batch.
&gt;&gt;&nbsp;This&nbsp;adds&nbsp;approximately&nbsp;O(blockCount&nbsp;×&nbsp;groupCount)&nbsp;work&nbsp;and&nbsp;may&nbsp;approach
&gt;&gt;&nbsp;O(n²)&nbsp;in&nbsp;unfavorable&nbsp;cases.
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;It&nbsp;would&nbsp;be&nbsp;better&nbsp;to&nbsp;maintain&nbsp;the&nbsp;retained-size&nbsp;delta&nbsp;incrementally&nbsp;for
&gt;&gt;&nbsp;only&nbsp;the&nbsp;buffers&nbsp;changed&nbsp;by&nbsp;the&nbsp;current&nbsp;batch,&nbsp;including&nbsp;any&nbsp;ObjectBigArray
&gt;&gt;&nbsp;capacity&nbsp;change.
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;&nbsp;&nbsp;6.&nbsp;Numerical&nbsp;edge&nbsp;cases
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;The&nbsp;extrapolation&nbsp;design&nbsp;should&nbsp;also&nbsp;address&nbsp;the&nbsp;following&nbsp;cases:
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;&nbsp;&nbsp;-&nbsp;Math.subtractExact(later,&nbsp;earlier)&nbsp;may&nbsp;overflow&nbsp;for&nbsp;two&nbsp;otherwise
&gt;&gt;&nbsp;&nbsp;&nbsp;valid&nbsp;INT64&nbsp;timestamps.
&gt;&gt;&nbsp;&nbsp;&nbsp;-&nbsp;The&nbsp;final&nbsp;division&nbsp;performed&nbsp;by&nbsp;rate()&nbsp;or&nbsp;irate()&nbsp;may&nbsp;still&nbsp;produce
&gt;&gt;&nbsp;&nbsp;&nbsp;Infinity&nbsp;even&nbsp;if&nbsp;the&nbsp;extrapolated&nbsp;intermediate&nbsp;result&nbsp;is&nbsp;finite.
&gt;&gt;&nbsp;&nbsp;&nbsp;-&nbsp;The&nbsp;increase&nbsp;==&nbsp;0&nbsp;shortcut&nbsp;should&nbsp;occur&nbsp;only&nbsp;after&nbsp;validating&nbsp;the
&gt;&gt;&nbsp;&nbsp;&nbsp;window&nbsp;and&nbsp;sample&nbsp;interval.
&gt;&gt;&nbsp;&nbsp;&nbsp;-&nbsp;Converting&nbsp;INT64&nbsp;values&nbsp;greater&nbsp;than&nbsp;2^53&nbsp;to&nbsp;double&nbsp;can&nbsp;lose&nbsp;unit
&gt;&gt;&nbsp;&nbsp;&nbsp;increments;&nbsp;the&nbsp;intended&nbsp;semantics&nbsp;should&nbsp;be&nbsp;documented.
&gt;&gt;&nbsp;&nbsp;&nbsp;-&nbsp;The&nbsp;upper&nbsp;bound&nbsp;implied&nbsp;by&nbsp;using&nbsp;an&nbsp;int&nbsp;for&nbsp;sampleCount&nbsp;should&nbsp;be
&gt;&gt;&nbsp;&nbsp;&nbsp;defined.
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;These&nbsp;points&nbsp;do&nbsp;not&nbsp;change&nbsp;the&nbsp;overall&nbsp;direction&nbsp;of&nbsp;the&nbsp;design,&nbsp;but&nbsp;they
&gt;&gt;&nbsp;affect&nbsp;correctness,&nbsp;consistency,&nbsp;and&nbsp;performance.&nbsp;I&nbsp;suggest&nbsp;making&nbsp;them
&gt;&gt;&nbsp;explicit&nbsp;in&nbsp;the&nbsp;technical&nbsp;design&nbsp;before&nbsp;implementation&nbsp;begins.
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;Best&nbsp;regards,
&gt;&gt;&nbsp;---------------------
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;Yuan&nbsp;Tian
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;On&nbsp;Fri,&nbsp;Jul&nbsp;17,&nbsp;2026&nbsp;at&nbsp;11:02 
AM&nbsp;Xinqi&nbsp;Zhao&nbsp;<3322351...@qq.com&gt;&nbsp;wrote:
&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;Hi&nbsp;IoTDB&nbsp;community,
&gt;&gt;&gt;&nbsp;
&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;Following&nbsp;the&nbsp;previous&nbsp;requirements&nbsp;discussion,&nbsp;I&nbsp;have&nbsp;prepared&nbsp;an&nbsp;initial
&gt;&gt;&gt;&nbsp;technical&nbsp;design&nbsp;for&nbsp;adding&nbsp;the&nbsp;Prometheus-like&nbsp;rate(),&nbsp;irate(),
&gt;&gt;&gt;&nbsp;increase(),&nbsp;and&nbsp;delta()&nbsp;aggregate&nbsp;functions&nbsp;to&nbsp;the&nbsp;IoTDB&nbsp;table&nbsp;model.
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;Related&nbsp;issue:
&gt;&gt;&gt;&nbsp;https://github.com/apache/iotdb/issues/17976
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;The&nbsp;main&nbsp;design&nbsp;is&nbsp;summarized&nbsp;below.
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;Integration&nbsp;with&nbsp;the&nbsp;existing&nbsp;aggregation&nbsp;framework
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;The&nbsp;four&nbsp;functions&nbsp;will&nbsp;be&nbsp;implemented&nbsp;entirely&nbsp;within&nbsp;the&nbsp;existing
&gt;&gt;&gt;&nbsp;table-model&nbsp;aggregation&nbsp;framework.&nbsp;No&nbsp;new&nbsp;SQL&nbsp;execution&nbsp;layer&nbsp;or
&gt;&gt;&gt;&nbsp;third-party&nbsp;dependency&nbsp;will&nbsp;be&nbsp;introduced.
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;Both&nbsp;execution&nbsp;paths&nbsp;will&nbsp;be&nbsp;supported:
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;TableAccumulator&nbsp;for&nbsp;global&nbsp;aggregation&nbsp;and&nbsp;streaming&nbsp;aggregation&nbsp;when
&gt;&gt;&gt;&nbsp;groups&nbsp;are&nbsp;fully&nbsp;ordered.
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;GroupedAccumulator&nbsp;for&nbsp;HashAggregationOperator&nbsp;and
&gt;&gt;&gt;&nbsp;StreamingHashAggregationOperator.
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;Each&nbsp;function&nbsp;will&nbsp;provide&nbsp;both&nbsp;TableAccumulator&nbsp;and&nbsp;GroupedAccumulator
&gt;&gt;&gt;&nbsp;implementations.
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;The&nbsp;signatures&nbsp;are:
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;rate(value_col,&nbsp;time_col,&nbsp;window_start,&nbsp;window_end)
&gt;&gt;&gt;&nbsp;increase(value_col,&nbsp;time_col,&nbsp;window_start,&nbsp;window_end)
&gt;&gt;&gt;&nbsp;irate(value_col,&nbsp;time_col)
&gt;&gt;&gt;&nbsp;delta(value_col,&nbsp;time_col,&nbsp;window_start,&nbsp;window_end)
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;The&nbsp;value&nbsp;column&nbsp;supports&nbsp;INT32,&nbsp;INT64,&nbsp;FLOAT,&nbsp;and&nbsp;DOUBLE.&nbsp;Values&nbsp;are
&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;converted&nbsp;to&nbsp;double&nbsp;internally,&nbsp;and&nbsp;all&nbsp;four&nbsp;functions&nbsp;return&nbsp;DOUBLE.&nbsp;Time
&gt;&gt;&gt;&nbsp;arguments&nbsp;support&nbsp;TIMESTAMP&nbsp;and&nbsp;INT64.
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;Ordered&nbsp;and&nbsp;unordered&nbsp;implementations
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;Each&nbsp;accumulator&nbsp;will&nbsp;support&nbsp;two&nbsp;internal&nbsp;modes&nbsp;rather&nbsp;than&nbsp;introducing
&gt;&gt;&gt;&nbsp;separate&nbsp;Ordered&nbsp;and&nbsp;Naive&nbsp;accumulator&nbsp;classes.
&gt;&gt;&gt;&nbsp;
&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;When&nbsp;the&nbsp;aggregation&nbsp;step&nbsp;is&nbsp;SINGLE&nbsp;and&nbsp;the&nbsp;planner&nbsp;can&nbsp;prove&nbsp;that&nbsp;samples
&gt;&gt;&gt;&nbsp;within&nbsp;each&nbsp;group&nbsp;are&nbsp;ordered&nbsp;by&nbsp;time_col,&nbsp;the&nbsp;accumulator&nbsp;will&nbsp;use&nbsp;a
&gt;&gt;&gt;&nbsp;streaming&nbsp;implementation&nbsp;with&nbsp;fixed-size&nbsp;state:
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;Time&nbsp;complexity:&nbsp;O(n)
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;Additional&nbsp;space&nbsp;per&nbsp;group:&nbsp;O(1)
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;Otherwise,&nbsp;the&nbsp;accumulator&nbsp;will&nbsp;buffer&nbsp;all&nbsp;valid&nbsp;samples&nbsp;and&nbsp;sort&nbsp;them
&gt;&gt;&gt;&nbsp;before&nbsp;the&nbsp;final&nbsp;calculation:
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;Time&nbsp;complexity:&nbsp;O(n&nbsp;log&nbsp;n)
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;Additional&nbsp;space&nbsp;per&nbsp;group:&nbsp;O(n)
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;The&nbsp;two&nbsp;modes&nbsp;will&nbsp;have&nbsp;identical&nbsp;function&nbsp;semantics,&nbsp;validation&nbsp;rules,
&gt;&gt;&gt;&nbsp;and&nbsp;results.
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;A&nbsp;new&nbsp;inputOrderedByTimeAscending&nbsp;property&nbsp;will&nbsp;be&nbsp;added&nbsp;to
&gt;&gt;&gt;&nbsp;AggregationNode.Aggregation.&nbsp;The&nbsp;ordered&nbsp;implementation&nbsp;will&nbsp;be&nbsp;selected
&gt;&gt;&gt;&nbsp;only&nbsp;when:
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;step&nbsp;==&nbsp;SINGLE&nbsp;&amp;amp;&amp;amp;&nbsp;inputOrderedByTimeAscending
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;PARTIAL,&nbsp;INTERMEDIATE,&nbsp;and&nbsp;FINAL&nbsp;aggregation&nbsp;stages&nbsp;will&nbsp;always&nbsp;use&nbsp;the
&gt;&gt;&gt;&nbsp;buffered&nbsp;implementation&nbsp;because&nbsp;they&nbsp;need&nbsp;to&nbsp;generate,&nbsp;merge,&nbsp;or&nbsp;consume
&gt;&gt;&gt;&nbsp;intermediate&nbsp;states.
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;The&nbsp;ordered&nbsp;mode&nbsp;will&nbsp;still&nbsp;perform&nbsp;runtime&nbsp;checks&nbsp;for&nbsp;duplicate
&gt;&gt;&gt;&nbsp;timestamps&nbsp;and&nbsp;unexpected&nbsp;out-of-order&nbsp;input.
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;Accumulator&nbsp;organization
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;rate()&nbsp;and&nbsp;increase()&nbsp;will&nbsp;share&nbsp;common&nbsp;base&nbsp;classes&nbsp;because&nbsp;they&nbsp;use&nbsp;the
&gt;&gt;&gt;&nbsp;same&nbsp;counter-reset&nbsp;correction&nbsp;and&nbsp;boundary-extrapolation&nbsp;logic.
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;The&nbsp;main&nbsp;TableAccumulator&nbsp;classes&nbsp;will&nbsp;include:
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;AbstractRateIncreaseAccumulator
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;RateAccumulator
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;IncreaseAccumulator
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;IrateAccumulator
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;DeltaAccumulator
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;Equivalent&nbsp;GroupedAccumulator&nbsp;classes&nbsp;will&nbsp;maintain&nbsp;state&nbsp;independently
&gt;&gt;&gt;&nbsp;for&nbsp;each&nbsp;groupId.
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;In&nbsp;ordered&nbsp;mode,&nbsp;the&nbsp;accumulators&nbsp;retain&nbsp;only&nbsp;fixed&nbsp;state&nbsp;such&nbsp;as&nbsp;the
&gt;&gt;&gt;&nbsp;first,&nbsp;previous,&nbsp;and&nbsp;last&nbsp;samples,&nbsp;sample&nbsp;count,&nbsp;corrected&nbsp;increase,&nbsp;and
&gt;&gt;&gt;&nbsp;window&nbsp;boundaries.
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;In&nbsp;unordered&nbsp;mode,&nbsp;samples&nbsp;will&nbsp;be&nbsp;stored&nbsp;in&nbsp;a&nbsp;new&nbsp;TimeValueBuffer.
&gt;&gt;&gt;&nbsp;Grouped&nbsp;accumulators&nbsp;will&nbsp;use&nbsp;TimeValueBufferBigArray&nbsp;to&nbsp;maintain&nbsp;one
&gt;&gt;&gt;&nbsp;buffer&nbsp;per&nbsp;groupId.
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;Intermediate-state&nbsp;handling
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;For&nbsp;distributed&nbsp;aggregation,&nbsp;PARTIAL,&nbsp;INTERMEDIATE,&nbsp;and&nbsp;FINAL&nbsp;stages&nbsp;must
&gt;&gt;&gt;&nbsp;preserve&nbsp;the&nbsp;complete&nbsp;sample&nbsp;sequence&nbsp;because&nbsp;counter-reset&nbsp;detection,
&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;duplicate-timestamp&nbsp;validation,&nbsp;and&nbsp;the&nbsp;final&nbsp;result&nbsp;depend&nbsp;on&nbsp;global&nbsp;time
&gt;&gt;&gt;&nbsp;ordering.
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;The&nbsp;intermediate&nbsp;state&nbsp;will&nbsp;contain:
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;A&nbsp;state&nbsp;version
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;window_start&nbsp;and&nbsp;window_end,&nbsp;except&nbsp;for&nbsp;irate()
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;The&nbsp;number&nbsp;of&nbsp;valid&nbsp;samples
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;All&nbsp;timestamp/value&nbsp;pairs
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;PARTIAL&nbsp;stages&nbsp;collect&nbsp;and&nbsp;serialize&nbsp;their&nbsp;samples&nbsp;without&nbsp;calculating&nbsp;a
&gt;&gt;&gt;&nbsp;final&nbsp;result.
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;INTERMEDIATE&nbsp;stages&nbsp;deserialize&nbsp;and&nbsp;merge&nbsp;sample&nbsp;buffers&nbsp;without&nbsp;sorting
&gt;&gt;&gt;&nbsp;them.
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;The&nbsp;FINAL&nbsp;stage&nbsp;merges&nbsp;all&nbsp;states,&nbsp;sorts&nbsp;the&nbsp;complete&nbsp;sample&nbsp;set&nbsp;by
&gt;&gt;&gt;&nbsp;timestamp,&nbsp;checks&nbsp;for&nbsp;duplicate&nbsp;timestamps,&nbsp;and&nbsp;then&nbsp;performs&nbsp;the&nbsp;final
&gt;&gt;&gt;&nbsp;calculation.
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;Deserialization&nbsp;will&nbsp;validate&nbsp;the&nbsp;sample&nbsp;count&nbsp;and&nbsp;serialized&nbsp;length&nbsp;to
&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;reject&nbsp;corrupted&nbsp;intermediate&nbsp;states&nbsp;and&nbsp;prevent&nbsp;invalid&nbsp;memory&nbsp;allocation.
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;Memory&nbsp;management
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;The&nbsp;unordered&nbsp;implementation&nbsp;will&nbsp;integrate&nbsp;with&nbsp;the&nbsp;existing&nbsp;query
&gt;&gt;&gt;&nbsp;memory-management&nbsp;mechanism,&nbsp;following&nbsp;a&nbsp;design&nbsp;similar&nbsp;to&nbsp;the&nbsp;exact
&gt;&gt;&gt;&nbsp;percentile()&nbsp;accumulator.
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;TimeValueBuffer&nbsp;and&nbsp;TimeValueBufferBigArray&nbsp;will&nbsp;report&nbsp;their&nbsp;retained
&gt;&gt;&gt;&nbsp;memory.&nbsp;Accumulators&nbsp;will&nbsp;use&nbsp;MemoryReservationManager&nbsp;to&nbsp;reserve&nbsp;or
&gt;&gt;&gt;&nbsp;release&nbsp;the&nbsp;difference&nbsp;whenever&nbsp;buffers&nbsp;grow,&nbsp;merge,&nbsp;or&nbsp;shrink.
&gt;&gt;&gt;&nbsp;
&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;reset()&nbsp;will&nbsp;clear&nbsp;the&nbsp;calculation&nbsp;state,&nbsp;shrink&nbsp;oversized&nbsp;internal&nbsp;arrays
&gt;&gt;&gt;&nbsp;to&nbsp;their&nbsp;initial&nbsp;capacity,&nbsp;and&nbsp;release&nbsp;the&nbsp;corresponding&nbsp;query&nbsp;memory.
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;Shared&nbsp;extrapolation&nbsp;logic
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;A&nbsp;shared&nbsp;ExtrapolationUtil&nbsp;will&nbsp;implement&nbsp;the&nbsp;boundary-extrapolation
&gt;&gt;&gt;&nbsp;algorithm&nbsp;used&nbsp;by&nbsp;rate(),&nbsp;increase(),&nbsp;and&nbsp;delta().&nbsp;It&nbsp;will&nbsp;be&nbsp;responsible
&gt;&gt;&gt;&nbsp;for:
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;Converting&nbsp;timestamp&nbsp;differences&nbsp;to&nbsp;seconds&nbsp;according&nbsp;to&nbsp;the&nbsp;current
&gt;&gt;&gt;&nbsp;timestamp_precision
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;Calculating&nbsp;the&nbsp;average&nbsp;sample&nbsp;interval
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;Applying&nbsp;the&nbsp;1.1-times&nbsp;extrapolation&nbsp;threshold
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;Limiting&nbsp;extrapolation&nbsp;to&nbsp;half&nbsp;an&nbsp;average&nbsp;interval&nbsp;when&nbsp;a&nbsp;boundary&nbsp;is&nbsp;too
&gt;&gt;&gt;&nbsp;far&nbsp;away
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;Applying&nbsp;counter&nbsp;zero-point&nbsp;protection&nbsp;for&nbsp;rate()&nbsp;and&nbsp;increase()
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;Calculating&nbsp;and&nbsp;validating&nbsp;the&nbsp;final&nbsp;extrapolation&nbsp;factor
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;Counter-reset&nbsp;detection&nbsp;and&nbsp;sample&nbsp;traversal&nbsp;will&nbsp;remain&nbsp;in&nbsp;the
&gt;&gt;&gt;&nbsp;accumulators&nbsp;rather&nbsp;than&nbsp;in&nbsp;ExtrapolationUtil.
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;rate()&nbsp;will&nbsp;divide&nbsp;the&nbsp;extrapolated&nbsp;increase&nbsp;by&nbsp;the&nbsp;complete&nbsp;window
&gt;&gt;&gt;&nbsp;duration.&nbsp;increase()&nbsp;will&nbsp;return&nbsp;the&nbsp;extrapolated&nbsp;increase&nbsp;directly.
&gt;&gt;&gt;&nbsp;delta()&nbsp;will&nbsp;extrapolate&nbsp;the&nbsp;raw&nbsp;difference&nbsp;between&nbsp;the&nbsp;last&nbsp;and&nbsp;first
&gt;&gt;&gt;&nbsp;values&nbsp;without&nbsp;counter&nbsp;zero-point&nbsp;protection.&nbsp;irate()&nbsp;will&nbsp;use&nbsp;only&nbsp;the
&gt;&gt;&gt;&nbsp;final&nbsp;two&nbsp;samples&nbsp;and&nbsp;will&nbsp;not&nbsp;perform&nbsp;boundary&nbsp;extrapolation.
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;All&nbsp;time&nbsp;conversions&nbsp;will&nbsp;respect&nbsp;the&nbsp;configured&nbsp;ms,&nbsp;us,&nbsp;or&nbsp;ns&nbsp;timestamp
&gt;&gt;&gt;&nbsp;precision,&nbsp;while&nbsp;rate()&nbsp;and&nbsp;irate()&nbsp;will&nbsp;always&nbsp;return&nbsp;values&nbsp;per&nbsp;second.
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;Please&nbsp;review&nbsp;this&nbsp;initial&nbsp;design,&nbsp;especially&nbsp;the&nbsp;ordered-input&nbsp;property,
&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;the&nbsp;complete-sample&nbsp;intermediate&nbsp;state,&nbsp;and&nbsp;the&nbsp;memory-management&nbsp;approach.
&gt;&gt;&gt;&nbsp;Any&nbsp;comments&nbsp;or&nbsp;alternative&nbsp;suggestions&nbsp;are&nbsp;welcome.
&gt;&gt;&gt;&nbsp;
&gt;&gt;&gt;&nbsp;Best&nbsp;regards,
&gt;&gt;&gt;&nbsp;Xinqi&nbsp;Zhao
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;
&gt;&gt;&nbsp;