Caideyipi commented on PR #17871: URL: https://github.com/apache/iotdb/pull/17871#issuecomment-6847461785
### Independent re-review: one complete round Reviewed the complete 86-file diff from baseline `cca1dbbde6d0dffa38495c92c92a48b2626d2f99` to `033b55d1474c978dd1b4ee07138f337809ede990`, plus relevant implementation/API/test context. The review covered MAD merge/null semantics, sampling Row reuse and INT64 precision, DWT levels, Histogram/MinMax extremes, Spline timestamps/coordinates/endpoints and test fixtures, AR/DTW validation, moving-average recovery, ExactOrderStatistics extreme-value median/MAD, and DTW history, cross-partition matching, interleaved/reversed/multilevel merges and serialization. **Outcome:** six actionable findings were fixed in [`111fd493b7a`](https://github.com/apache/iotdb/commit/111fd493b7a4ab63437394e50beaa09ae72b7c94). The final local commit, remote source branch `Caideyipi/iotdb:fix/library-udf-edge-cases`, and PR head were all confirmed as `111fd493b7a4ab63437394e50beaa09ae72b7c94`. The source head was checked immediately before pushing and had no concurrent update. **Draft status is retained** because this round found behavioral and test-logic issues; a fresh independent re-review is required. ### Findings and repairs Locations below refer to the reviewed commit. 1. **[P2] INT64 MAD loses small deviations at large offsets**: [ExactOrderStatistics.java:181](https://github.com/apache/iotdb/blob/033b55d1474c978dd1b4ee07138f337809ede990/library-udf/src/main/java/org/apache/iotdb/library/dprofile/util/ExactOrderStatistics.java#L181). Converting samples/median to double before subtraction makes the MAD of adjacent large INT64 values zero instead of `0.5`. The fix calculates twice the median and doubled absolute deviations with BigInteger before conversion. Added odd/even and extreme-offset regressions. 2. **[P2] A nonzero subnormal histogram width can still misclassify buckets**: [UDTFHistogram.java:90](https://github.com/apache/iotdb/blob/033b55d1474c978dd1b4ee07138f337809ede990/library-udf/src/main/java/org/apache/iotdb/library/dprofile/UDTFHistogram.java#L90). For range `0..5*Double.MIN_VALUE`, two buckets, and inputs `2*MIN_VALUE,3*MIN_VALUE`, rounded division by the bucket width produces `[0,2]` instead of `[1,1]`. The fix uses the normalized ratio for subnormal widths; tests cover positive and negative bounds. 3. **[P2] Spline can emit duplicate timestamps**: [UDTFSpline.java:119](https://github.com/apache/iotdb/blob/033b55d1474c978dd1b4ee07138f337809ede990/library-udf/src/main/java/org/apache/iotdb/library/dprofile/UDTFSpline.java#L119). Input timestamps/values `0..4` with `points=10` generate repeated timestamps, violating the PointCollector requirement for strictly increasing output. The fix caps the output count to the available integer timestamps and uses that count for spacing and endpoint selection. The collector regression enforces increasing timestamps and verifies values. 4. **[P2] Moving averages lose finite small values after large samples leave**: [UDTFMvAvg.java:75](https://github.com/apache/iotdb/blob/033b55d1474c978dd1b4ee07138f337809ede990/library-udf/src/main/java/org/apache/iotdb/library/dprofile/UDTFMvAvg.java#L75). With `window=2` and values `[1e16,1e16,1,1]`, the final average is `0.5` rather than `1`. Recovery triggered only by nonfinite sums misses this cancellation. The fix uses compensated additions/removals and resets compensation when rebuilding after overflow. Regressions also cover positive/negative maximum doubles and subsequent windows. 5. **[P2] DTW merge repeatedly recalculates all historical windows**: [UDAFDTWMatch.java:143](https://github.com/apache/iotdb/blob/033b55d1474c978dd1b4ee07138f337809ede990/library-udf/src/main/java/org/apache/iotdb/library/match/UDAFDTWMatch.java#L143). Each combine copied the complete accumulated input and recomputed every window. Sequentially combining many small partitions therefore makes matching work quadratic in partition count for a fixed pattern. The fix accumulates incoming points and defers a required chronological rebuild to final output, persists the pending-rebuild flag through serialization, and uses two rolling DTW rows. Tests cover repeated small interleaved merges, serialization before output, continuation, repeated output, source immutability, and nonzero-distance threshold boundaries. 6. **[P2] The LOF series regression used an impossible access window**: [UDFWindowAndQueueTest.java:279](https://github.com/apache/iotdb/blob/033b55d1474c978dd1b4ee07138f337809ede990/library-udf/src/test/java/org/apache/iotdb/library/UDFWindowAndQueueTest.java#L279). It configured a two-row window but supplied six rows, so the asserted result could not arise under the configured engine access strategy. The corrected test uses the default access window and embedding size five, with nine input rows including two invalid values, preserving the expected three output timestamps. ### Verification - Four numerical regressions failed against their pre-fix implementations, then passed after repair. - `mvn -o spotless:apply -P with-integration-tests -pl library-udf`: passed. - `mvn -o clean test -pl library-udf -am`: **139 tests, zero failures/errors/skips**. - `mvn -o clean test-compile -pl library-udf -am -P with-zh-locale -DskipTests`: passed. - Lifecycle Checkstyle and `git diff --check`: passed. - Both clean source reactors disabled local and remote build caches using `-Ddevelocity.cache.local.enabled=false -Ddevelocity.cache.remote.enabled=false`. Required IoTDB source dependencies were rebuilt. - All newly added tests were run locally. Validation used the local offline fallback because remote Maven plugin downloads were blocked. - No full-project build or performance benchmark was run. Remote CI was neither waited on nor polled. ### DTW resource limits The retained ordered input remains **O(N + R)** in state memory and serialized size, where N is the effective input count and R the match count; arbitrary temporal interleaving still requires the historical input. The merge repair removes repeated historical DTW recalculation during combines. A pending rebuild at final output still costs O(N * L^2) for pattern length L, with O(L) DTW scratch space. This is an algorithmic assessment, not a measured performance result. This session completed one review round and its repairs; it did not conduct another review after the update. -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
