Hi everyone, I would like to share an update on the LTTB downsampling table function for the Table Model, tracked in PR #18725 [1]. The discussion has produced a number of decisions that are now implemented and covered by tests, and I would like to summarize them here so that the semantics are recorded on the mailing list and anyone interested can review the current state.
PR: https://github.com/apache/iotdb/pull/18725 1. SIZE defines a bucket, not a separate LTTB invocation In SIZE mode each window is one LTTB bucket and sampling proceeds across buckets. One anchor chain is maintained per partition and per participant column: for the current bucket, A is the previously selected point, B ranges over the eligible points of the bucket, and C is the average point of the next bucket. The selected B becomes the anchor for the following bucket. When SLIDE < SIZE the overlapping windows do not restart the anchor chain, and the state remains independent between partitions and between columns. This supersedes the earlier statement that every window runs independently and that overlapping windows must not share anchors. N and SIZE remain mutually exclusive. 2. Boundary rules - The first eligible point F and the last eligible point L of a column are buckets of their own and are always returned; F is reported by the first window that contains it, L by the last one. - Every other window contributes exselected with the standard largest-triangle rule. - A partition with only two eligiblrefore returns two rows, F and L, not one. - When the next bucket has no eligiis taken from the next bucket that has one; if none exists, the last finite eligible point is used. - If every participant column is NULL in a window, the window emits no row. An endpoint used as A or C is note. 3. SQL conventions The table arguments follow the exis LTTB(DATA => metrics PARTITION BY TIMECOL => 'time', SIZE => 2) The DESCRIPTOR(time) form in the original issue description was a mistake and has been corrected to TIMECOL => 't earlier descriptor wording. 4. Duplicate timestamps In both N and SIZE mode TIMECOL must be unique within each input partition. If two input rows of the same partitio, the query fails with a semantic error reported to the client as 701, naming duplicate timestamps as the cause. Rows are nsolved by input order. The validation runs on the partition input, before per-column NULL filtering; equal timestamps in difflid, and a row shared by several overlapping windows is still a single input row. 5. Implementation notes - The per-processor memory budget was moved out of the udf-api provider interface into an engine-internalned table function does not see it. The operator receives the resolved value, so the reservation and the limit enforced by the pro - LTTB must run once over the partitions merged in order, like FFT: a partitioned GroupNode does not mathin one branch, so fragment-local execution would sample per fragment. 6. Testing Unit tests cover the anchor chain, the bucket boundary rules, NULL handling, non-finite values, integer precisiory bounding and timestamp uniqueness within and across input blocks. Integration tests run in both standalone and cluster topologutput, SLIDE below and above SIZE, lookahead over NULL buckets, the 701 cases (within a partition, across devices, and across distribuid case of equal timestamps in different partitions. Feedback is very welcome, and I would be glad to hear from anyone who wants to review the current state of the PR.nts above should be reconsidered, please reply here or on the PR and I will follow up. Thanks, DaZuiZui
