Caideyipi commented on code in PR #14168:
URL: https://github.com/apache/iotdb/pull/14168#discussion_r1893577371
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/processor/downsampling/DownSamplingProcessor.java:
##########
@@ -88,11 +154,9 @@ public void customize(
.getRegionId()))
.getDatabaseName()
+ TsFileConstant.PATH_SEPARATOR;
-
- pathLastObjectCache = initPathLastObjectCache(memoryLimitInBytes);
Review Comment:
Why put it down to the child classes? What if we directly call it here..
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/processor/downsampling/DownSamplingFilter.java:
##########
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.pipe.processor.downsampling;
+
+public abstract class DownSamplingFilter {
Review Comment:
Better extends "Accountable" ()
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/processor/downsampling/sdt/SwingingDoorTrendingSamplingProcessor.java:
##########
@@ -164,18 +141,33 @@ protected void processRow(
final SwingingDoorTrendingFilter filter =
pathLastObjectCache.getPartialPathLastObject(timeSeriesSuffix);
- if (filter != null) {
- if (filter.filter(row.getTime(), row.getObject(i))) {
- hasNonNullMeasurements = true;
- } else {
+ if (Objects.nonNull(filter)) {
+ final Boolean result = filterArrivalTimeAndEventTime(filter,
arrivalTime, currentRowTime);
+ if (Objects.isNull(result)) {
+ if (filter.filter(arrivalTime, currentRowTime, row.getObject(i))) {
+ hasNonNullMeasurements = true;
+ } else {
+ remarkableRow.markNull(i);
+ }
+ continue;
+ }
+
+ // It will not be null
+ if (result) {
Review Comment:
!result？
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/processor/downsampling/DownSamplingProcessor.java:
##########
@@ -138,6 +202,37 @@ protected abstract void processRow(
String deviceSuffix,
AtomicReference<Exception> exception);
+ /**
+ * Determine the arrival time interval and event time interval.
+ *
+ * @return true to indicate that the process does not need to be continued
and the data can be
+ * updated directly. False means that the event is discarded directly.
Null means that the
+ * downsampling algorithm can continue to be executed.
+ */
+ protected Boolean filterArrivalTimeAndEventTime(
+ final DownSamplingFilter filter, final long arrivalTime, final long
eventTime) {
+ final long arrivalTimeInterval = Math.abs(arrivalTime -
filter.getLastPointArrivalTime());
Review Comment:
May need further discussion about the precision of the arrival time...
Currently the doc declare it as the same as the IoTDB's precision..
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/processor/downsampling/DownSamplingFilter.java:
##########
@@ -0,0 +1,58 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.pipe.processor.downsampling;
+
+public abstract class DownSamplingFilter {
+
+ protected boolean isFilteredByArrivalTime = true;
Review Comment:
Do we really need this....
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/pipe/processor/downsampling/sdt/SwingingDoorTrendingSamplingProcessor.java:
##########
@@ -49,21 +51,7 @@ public class SwingingDoorTrendingSamplingProcessor extends
DownSamplingProcessor
*/
private double compressionDeviation;
- /**
- * The minimum time distance between two stored data points if current point
time to the last
- * stored point time distance <= compressionMinTimeInterval, current point
will NOT be stored
- * regardless of compression deviation
- */
- private long compressionMinTimeInterval;
-
- /**
- * The maximum time distance between two stored data points if current point
time to the last
- * stored point time distance >= compressionMaxTimeInterval, current point
will be stored
- * regardless of compression deviation
- */
- private long compressionMaxTimeInterval;
-
- private PartialPathLastObjectCache<SwingingDoorTrendingFilter<?>>
pathLastObjectCache;
+ private PartialPathLastObjectCache<SwingingDoorTrendingFilter>
pathLastObjectCache;
Review Comment:
extends DownSamplingProcessor\<SwingingDoorTrendingFilter\>? Then the cache
can be put above(
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
