Caideyipi commented on PR #17871:
URL: https://github.com/apache/iotdb/pull/17871#issuecomment-6847461785

   ### Independent re-review: one complete round
   
   Reviewed the complete 86-file diff from baseline 
`cca1dbbde6d0dffa38495c92c92a48b2626d2f99` to 
`033b55d1474c978dd1b4ee07138f337809ede990`, plus relevant 
implementation/API/test context.
   
   The review covered MAD merge/null semantics, sampling Row reuse and INT64 
precision, DWT levels, Histogram/MinMax extremes, Spline 
timestamps/coordinates/endpoints and test fixtures, AR/DTW validation, 
moving-average recovery, ExactOrderStatistics extreme-value median/MAD, and DTW 
history, cross-partition matching, interleaved/reversed/multilevel merges and 
serialization.
   
   **Outcome:** six actionable findings were fixed in 
[`111fd493b7a`](https://github.com/apache/iotdb/commit/111fd493b7a4ab63437394e50beaa09ae72b7c94).
 The final local commit, remote source branch 
`Caideyipi/iotdb:fix/library-udf-edge-cases`, and PR head were all confirmed as 
`111fd493b7a4ab63437394e50beaa09ae72b7c94`. The source head was checked 
immediately before pushing and had no concurrent update. **Draft status is 
retained** because this round found behavioral and test-logic issues; a fresh 
independent re-review is required.
   
   ### Findings and repairs
   
   Locations below refer to the reviewed commit.
   
   1. **[P2] INT64 MAD loses small deviations at large offsets**: 
[ExactOrderStatistics.java:181](https://github.com/apache/iotdb/blob/033b55d1474c978dd1b4ee07138f337809ede990/library-udf/src/main/java/org/apache/iotdb/library/dprofile/util/ExactOrderStatistics.java#L181).
 Converting samples/median to double before subtraction makes the MAD of 
adjacent large INT64 values zero instead of `0.5`. The fix calculates twice the 
median and doubled absolute deviations with BigInteger before conversion. Added 
odd/even and extreme-offset regressions.
   
   2. **[P2] A nonzero subnormal histogram width can still misclassify 
buckets**: 
[UDTFHistogram.java:90](https://github.com/apache/iotdb/blob/033b55d1474c978dd1b4ee07138f337809ede990/library-udf/src/main/java/org/apache/iotdb/library/dprofile/UDTFHistogram.java#L90).
 For range `0..5*Double.MIN_VALUE`, two buckets, and inputs 
`2*MIN_VALUE,3*MIN_VALUE`, rounded division by the bucket width produces 
`[0,2]` instead of `[1,1]`. The fix uses the normalized ratio for subnormal 
widths; tests cover positive and negative bounds.
   
   3. **[P2] Spline can emit duplicate timestamps**: 
[UDTFSpline.java:119](https://github.com/apache/iotdb/blob/033b55d1474c978dd1b4ee07138f337809ede990/library-udf/src/main/java/org/apache/iotdb/library/dprofile/UDTFSpline.java#L119).
 Input timestamps/values `0..4` with `points=10` generate repeated timestamps, 
violating the PointCollector requirement for strictly increasing output. The 
fix caps the output count to the available integer timestamps and uses that 
count for spacing and endpoint selection. The collector regression enforces 
increasing timestamps and verifies values.
   
   4. **[P2] Moving averages lose finite small values after large samples 
leave**: 
[UDTFMvAvg.java:75](https://github.com/apache/iotdb/blob/033b55d1474c978dd1b4ee07138f337809ede990/library-udf/src/main/java/org/apache/iotdb/library/dprofile/UDTFMvAvg.java#L75).
 With `window=2` and values `[1e16,1e16,1,1]`, the final average is `0.5` 
rather than `1`. Recovery triggered only by nonfinite sums misses this 
cancellation. The fix uses compensated additions/removals and resets 
compensation when rebuilding after overflow. Regressions also cover 
positive/negative maximum doubles and subsequent windows.
   
   5. **[P2] DTW merge repeatedly recalculates all historical windows**: 
[UDAFDTWMatch.java:143](https://github.com/apache/iotdb/blob/033b55d1474c978dd1b4ee07138f337809ede990/library-udf/src/main/java/org/apache/iotdb/library/match/UDAFDTWMatch.java#L143).
 Each combine copied the complete accumulated input and recomputed every 
window. Sequentially combining many small partitions therefore makes matching 
work quadratic in partition count for a fixed pattern. The fix accumulates 
incoming points and defers a required chronological rebuild to final output, 
persists the pending-rebuild flag through serialization, and uses two rolling 
DTW rows. Tests cover repeated small interleaved merges, serialization before 
output, continuation, repeated output, source immutability, and 
nonzero-distance threshold boundaries.
   
   6. **[P2] The LOF series regression used an impossible access window**: 
[UDFWindowAndQueueTest.java:279](https://github.com/apache/iotdb/blob/033b55d1474c978dd1b4ee07138f337809ede990/library-udf/src/test/java/org/apache/iotdb/library/UDFWindowAndQueueTest.java#L279).
 It configured a two-row window but supplied six rows, so the asserted result 
could not arise under the configured engine access strategy. The corrected test 
uses the default access window and embedding size five, with nine input rows 
including two invalid values, preserving the expected three output timestamps.
   
   ### Verification
   
   - Four numerical regressions failed against their pre-fix implementations, 
then passed after repair.
   - `mvn -o spotless:apply -P with-integration-tests -pl library-udf`: passed.
   - `mvn -o clean test -pl library-udf -am`: **139 tests, zero 
failures/errors/skips**.
   - `mvn -o clean test-compile -pl library-udf -am -P with-zh-locale 
-DskipTests`: passed.
   - Lifecycle Checkstyle and `git diff --check`: passed.
   - Both clean source reactors disabled local and remote build caches using 
`-Ddevelocity.cache.local.enabled=false 
-Ddevelocity.cache.remote.enabled=false`. Required IoTDB source dependencies 
were rebuilt.
   - All newly added tests were run locally. Validation used the local offline 
fallback because remote Maven plugin downloads were blocked.
   - No full-project build or performance benchmark was run. Remote CI was 
neither waited on nor polled.
   
   ### DTW resource limits
   
   The retained ordered input remains **O(N + R)** in state memory and 
serialized size, where N is the effective input count and R the match count; 
arbitrary temporal interleaving still requires the historical input. The merge 
repair removes repeated historical DTW recalculation during combines. A pending 
rebuild at final output still costs O(N * L^2) for pattern length L, with O(L) 
DTW scratch space. This is an algorithmic assessment, not a measured 
performance result.
   
   This session completed one review round and its repairs; it did not conduct 
another review after the update.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to