Hi everyone,
I am planning to work on the HighPass, LowPass, and CrossCorrelation (XCorr)
UDFs in the library-udf module, and would like to gather input from anyone
familiar with signal processing or time series analysis.
## Background: what these functions do
High-pass and low-pass filters are fundamental tools in signal processing. A
low-pass filter attenuates high-frequency components while allowing
low-frequency components to pass through, which is useful for smoothing and
denoising sensor data. A high-pass filter does the opposite — it removes
low-frequency trends (e.g., sensor drift, slow temperature changes) and
retains fast-changing components like vibration spikes or abrupt anomalies.
Cross-correlation (XCorr) measures the similarity between two time series as
a function of the relative shift (lag) between them. It is used to detect
whether one signal is a delayed or phase-shifted version of another — for
example, identifying whether a vibration sensor on one machine triggers a
response on a neighboring machine with a time delay.
## Current state in IoTDB
These functions already have initial implementations in library-udf:
- `UDTFHighPass` and `UDTFLowPass` (frequency package): both use FFT
(JTransforms `DoubleFFT_1D`) to convert the entire series into the frequency
domain, zero out the unwanted components, and inverse-transform back. The
cutoff is controlled by a single `wpass` parameter in (0,1). This is a
straightforward approach, but there are other well-established methods (IIR
filters like Butterworth/Chebyshev, FIR windowed-sinc filters) with different
tradeoffs in terms of phase distortion, roll-off steepness, and computational
cost.
- `UDTFXCorr` (dmatch package): uses a naive O(n²) time-domain sliding dot
product to compute the cross-correlation for every lag. While functionally
correct, an FFT-based O(n log n) implementation is the standard choice in
practice for longer series. I also noticed that `CrossCorrelation.java`
appears identically in both `dmatch/util/` and `dprofile/util/` — there may
be an opportunity to consolidate.
## Why I am writing
The space of signal processing algorithms is broad, and design choices
(e.g.,filter type, normalization convention for XCorr, boundary handling,
streaming
vs. batch computation) can significantly affect usability and correctness for
IoT scenarios. Before diving into implementation, I would like to ask:
1. Is anyone in the community experienced with signal processing or time
series analysis who could share perspective on what filter types or correlation
conventions are most relevant for IoTDB users?
2. Are there existing user requests or JIRA issues that describe desired
behavior for these functions?
Any pointers, references, or discussion would be very helpful.
---
我计划对 library-udf 模块中的 HighPass、LowPass 和 CrossCorrelation (XCorr) UDF
进行改进和开发，想向熟悉信号处理或时间序列分析的社区成员请教。
## 背景：这些函数是做什么的
高通滤波和低通滤波是信号处理中的基础工具。低通滤波器衰减高频分量、保留低频分量，常用于传感器数据的平滑和去噪。高通滤波器则相反——它去除低频趋势（如传感器漂移、缓慢的温度变化），保留快速变化的成分，如振动尖峰或突发异常。
互相关 (XCorr)
衡量两条时间序列在不同相对位移（滞后）下的相似度。它用于检测一个信号是否是另一个信号的延迟或相移版本——例如，判断一台机器的振动传感器是否在一定的
时间延迟后引起相邻机器的响应。
## IoTDB 中的现状
这些函数在 library-udf 中已有初步实现：
- `UDTFHighPass` 和 `UDTFLowPass`（frequency 包）：均使用 FFT（JTransforms
`DoubleFFT_1D`）将整条序列变换到频域，将不需要的频率分量置零，再逆变换回时域。
截止频率由唯一的参数 `wpass`（取值 0 到 1）控制。这是一种直接的方法，但业界还有其他成熟方案（如 IIR 滤波器
Butterworth/Chebyshev，FIR 加窗 sinc 滤波器），它们在相位失真、滚降陡度和计算开销方面各有取舍。
- `UDTFXCorr`（dmatch 包）：使用朴素的 O(n²) 时域滑动点积计算每个滞后下的互相关。功能正确，但对于较长序列而言，基于 FFT 的
O(n log n) 实现是实践中更标准的选择。我还注意到 `CrossCorrelation.java` 在 `dmatch/util/` 和
`dprofile/util/`中完全相同——可能有合并的空间。
## 致信目的
信号处理算法的设计空间很广，滤波类型选择、互相关系数的归一化方式、边界处理、批量计算与流式计算等设计决策，都会显著影响函数在 IoT
场景下的可用性和正确性。在着手
开发之前，我想请教：
1. 社区中是否有在信号处理或时间序列分析方面有经验的成员，能就物联网用户最相关的 滤波器类型或相关函数的设计惯例分享一些见解？
2. 是否有现有的用户需求或 JIRA issue 描述了这些函数的期望行为？
任何指点、参考资料或讨论都将非常有帮助。
Best wishes
Yaobin Chen
