JackieTien97 commented on code in PR #18826:
URL: https://github.com/apache/iotdb/pull/18826#discussion_r4258223689
##########
iotdb-core/calc-commons/src/main/java/org/apache/iotdb/calc/execution/operator/process/FilterAndProjectOperator.java:
##########
@@ -172,8 +191,8 @@ private TsBlock getFilterTsBlock(TsBlock input) {
if (!hasNonMappableUDF) {
// get result of calculated common sub expressions
for (ColumnTransformer columnTransformer : commonTransformerList) {
- // CASE WHEN clause would clear all its cache
- // evaluate again to acquire cache
+ // CASE evaluation can clear a common expression after the filter used
it. Recompute it
+ // before materializing the column for projection within this same
input block.
columnTransformer.tryEvaluate();
resultColumns.add(columnTransformer.getColumn());
Review Comment:
Recomputing here also re-runs stateful functions on the same block after
CASE has cleared them. Before this PR the multi-block case below failed; now it
silently returns wrong values:
```sql
CREATE TABLE dt2 (time TIMESTAMP TIME, dev STRING TAG, x DOUBLE FIELD);
INSERT INTO dt2(time, dev, x) VALUES (1, 'd0', 1.0), (2, 'd0', 3.0), (3,
'd0', 6.0);
FLUSH;
INSERT INTO dt2(time, dev, x) VALUES (4, 'd0', 10.0), (5, 'd0', 11.0);
FLUSH;
SELECT time, x, diff(x) AS d FROM dt2 WHERE CASE WHEN diff(x) > 1.0 THEN 1.0
ELSE 0.0 END > 0.0;
```
(1C1D with `max_tsblock_line_number=1`, so `[1, 3, 6]` and `[10, 11]` reach
this operator as separate blocks)
- expected: `d = 2.0, 3.0, 4.0` at time 2, 3, 4
- base: `301: Index 2 out of bounds for length 2`
- this PR: time 4 returns `d = -1.0`. The first WHEN already computed `diff`
for `[10, 11]` and advanced its state to 11. CASE then clears that column, and
this line runs `diff` again on the same block (`10 - 11`).
- With `ELSE 1.0` (every row passes), time 1 returns `-5.0` instead of NULL.
A selection mask (see the `DeviceAttributeUpdater` comment) doesn't help
here, because DIFF evaluates every row anyway. The column computed by the first
WHEN was complete and valid; it is lost only because CASE clears its WHEN
subtree (see the root-cause note in the review summary).
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/execution/operator/schema/source/DeviceAttributeUpdater.java:
##########
@@ -145,9 +152,13 @@ private TsBlock getFilterTsBlock() {
final List<Column> filterResultColumns =
new ArrayList<>(Arrays.asList(curBlock.getValueColumns()));
- // Get result of calculated common sub expressions
+ // CASE evaluation can clear a common expression after the filter used it.
Recompute it before
+ // materializing the column for projection within this same batch.
commonTransformerList.forEach(
- columnTransformer ->
filterResultColumns.add(columnTransformer.getColumn()));
+ columnTransformer -> {
+ columnTransformer.tryEvaluate();
+ filterResultColumns.add(columnTransformer.getColumn());
+ });
Review Comment:
**Regression (must fix):** `tryEvaluate()` recomputes a cleared common
expression for every device in the batch, ignoring the filter. If the CASE was
guarding that expression, the UPDATE now evaluates it on devices the guard
excluded:
```sql
CREATE TABLE t (id STRING TAG, src STRING ATTRIBUTE, a STRING ATTRIBUTE, v
DOUBLE FIELD);
INSERT INTO t(time, id, src, a, v) VALUES (1, 'd0', 'n/a', 'seed', 1), (2,
'd1', '5', 'seed', 1);
UPDATE t SET a = CAST(CAST(src AS INT32) + 1 AS STRING)
WHERE CASE WHEN src <> 'n/a' THEN CAST(src AS INT32) > 10 ELSE false END;
```
- base: succeeds; no device matches, so `a` stays `seed`
- this PR: `305: internal error. statemachine throws a runtime exception:
org.apache.iotdb.commons.exception.SemanticException: Cannot cast n/a to INT32
type`
If d1 has `src = '50'` instead (one device matches), base fails with the NPE
this PR targets and this PR fails with the same cast error. The expected result
is `a = '51'` for d1.
The query path has always behaved this way ([`FilterAndProjectOperator`
L191-L199](https://github.com/apache/iotdb/blob/5e0cb1c8a98345af99bb1115eebad1be7ea12ffc/iotdb-core/calc-commons/src/main/java/org/apache/iotdb/calc/execution/operator/process/FilterAndProjectOperator.java#L191-L199)):
`SELECT CAST(s AS INT32) + 1 FROM z4 WHERE CASE WHEN s <> 'n/a' THEN CAST(s AS
INT32) > 10 ELSE false END` fails with `701: Cannot cast n/a to INT32 type` on
both base and this PR.
Suggested fix for both call sites: recompute only the rows that passed the
filter, for example
```java
// ColumnTransformer
public void tryEvaluateWithSelection(boolean[] selection) {
if (!columnCache.hasCached()) {
evaluateWithSelection(selection);
}
}
// here (in FilterAndProjectOperator: filterColumn / positionCount)
boolean[] selection = new boolean[deviceSchemaBatch.size()];
for (int i = 0; i < selection.length; i++) {
selection[i] = satisfy(curFilterColumn, i);
}
for (ColumnTransformer transformer : commonTransformerList) {
transformer.tryEvaluateWithSelection(selection);
filterResultColumns.add(transformer.getColumn());
}
```
With this change, both UPDATEs and the SELECT return the expected results in
my local run, and the calc-commons tests plus the datanode
`*ColumnTransformerTest` / `DevicePredicateHandlerTest` still pass.
This block also duplicates
[`FilterAndProjectOperator.getFilterTsBlock`](https://github.com/apache/iotdb/blob/5e0cb1c8a98345af99bb1115eebad1be7ea12ffc/iotdb-core/calc-commons/src/main/java/org/apache/iotdb/calc/execution/operator/process/FilterAndProjectOperator.java#L191-L199).
The two copies have already drifted once: base had `tryEvaluate()` there but
not here, which is the NPE this PR fixes. A single helper next to
`constructFilteredTsBlock` would keep the fix in one place.
##########
iotdb-core/calc-commons/src/main/java/org/apache/iotdb/calc/transformation/dag/column/ColumnTransformer.java:
##########
@@ -112,7 +119,54 @@ public void close() {
// do nothing
}
+ /**
+ * Clears caches during evaluation. Subclasses should define child edges via
{@link
+ * #forEachChild(Consumer)} instead of overriding this method.
+ *
+ * <p>This legacy entry point remains non-final for compatibility with
existing test doubles.
Review Comment:
Nit: `clearCache()` is still the main evaluation-time cleanup (CASE and the
Unary/Binary/Ternary/Multi selection paths call it), so "legacy entry point" is
misleading. Maybe: "Kept non-final so existing Mockito-based tests can stub it."
##########
integration-test/src/test/java/org/apache/iotdb/relational/it/schema/IoTDBCaseDeviceCacheIT.java:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.relational.it.schema;
+
+import org.apache.iotdb.it.env.EnvFactory;
+import org.apache.iotdb.it.framework.IoTDBTestRunner;
+import org.apache.iotdb.itbase.category.TableClusterIT;
+import org.apache.iotdb.itbase.category.TableLocalStandaloneIT;
+import org.apache.iotdb.itbase.env.BaseEnv;
+
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+
+import java.sql.Connection;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+
+import static org.apache.iotdb.db.it.utils.TestUtils.assertResultSetEqual;
+import static org.apache.iotdb.db.it.utils.TestUtils.prepareTableData;
+
+@RunWith(IoTDBTestRunner.class)
+@Category({TableLocalStandaloneIT.class, TableClusterIT.class})
+public class IoTDBCaseDeviceCacheIT {
+ private static final String DATABASE = "case_device_cache";
+ private static final String CASE_RESULT = "CASE WHEN src = 'match' THEN
'match' ELSE 'other' END";
+
+ @BeforeClass
+ public static void setUp() throws Exception {
+ // Keep every device in the same schema region and force a new batch for
each device.
+
EnvFactory.getEnv().getConfig().getCommonConfig().setDefaultSchemaRegionGroupNumPerDatabase(1);
+
EnvFactory.getEnv().getConfig().getCommonConfig().setMaxTsBlockLineNumber(1);
+ EnvFactory.getEnv().initClusterEnvironment();
+ prepareTableData(new String[] {"CREATE DATABASE " + DATABASE});
+ }
+
+ @AfterClass
+ public static void tearDown() throws Exception {
+ EnvFactory.getEnv().cleanClusterEnvironment();
+ }
+
+ @Test
+ public void testDevicePredicatesAcrossBatches() throws Exception {
+ try (Connection connection =
EnvFactory.getEnv().getConnection(BaseEnv.TABLE_SQL_DIALECT);
+ Statement statement = connection.createStatement()) {
+ createTable(statement, "predicate_cache");
+ String predicate = CASE_RESULT + " = src";
+ assertResultSetEqual(
+ statement.executeQuery("SHOW DEVICES FROM predicate_cache WHERE " +
predicate),
+ "id,src,a,b,",
+ new HashSet<>(
+ Arrays.asList(
+ "d0,match,null,null,",
+ "d1,match,null,null,",
+ "d2,other,null,null,",
+ "d3,other,null,null,")));
+ assertResultSetEqual(
+ statement.executeQuery("COUNT DEVICES FROM predicate_cache WHERE " +
predicate),
+ "count(devices),",
+ Collections.singleton("4,"));
+ }
+ }
+
+ @Test
+ public void testAttributeProjectionAcrossBatches() throws Exception {
+ try (Connection connection =
EnvFactory.getEnv().getConnection(BaseEnv.TABLE_SQL_DIALECT);
+ Statement statement = connection.createStatement()) {
+ createTable(statement, "projection_cache");
+ statement.execute("UPDATE projection_cache SET a = " + CASE_RESULT + ",
b = src");
+ assertResultSetEqual(
+ statement.executeQuery("SHOW DEVICES FROM projection_cache"),
+ "id,src,a,b,",
+ new HashSet<>(
+ Arrays.asList(
+ "d0,match,match,match,",
+ "d1,match,match,match,",
+ "d2,other,other,other,",
+ "d3,other,other,other,")));
+ }
+ }
+
+ @Test
+ public void testAttributeCommonExpressionClearedByCase() throws Exception {
+ try (Connection connection =
EnvFactory.getEnv().getConnection(BaseEnv.TABLE_SQL_DIALECT);
+ Statement statement = connection.createStatement()) {
+ createTable(statement, "common_cache");
+ statement.execute(
+ "UPDATE common_cache SET a = src "
+ + "WHERE CASE WHEN src = 'match' THEN true ELSE false END");
Review Comment:
This test does not exercise the common-expression path it is named after.
`src` is a bare column reference:
`ColumnTransformerBuilder.visitSymbolReference` turns it into a leaf without
consulting `hasSeen`, so `commonTransformerList` stays empty. The same UPDATE
already returns the correct result on base (checked with
`max_tsblock_line_number=1` and one schema region). The unit test at
`DevicePredicateHandlerTest` L167 likewise uses a leaf as the common
transformer, which the planner never produces.
A statement that does go through the common-expression path:
```sql
UPDATE common_cache SET a = upper(src) WHERE CASE WHEN upper(src) = 'MATCH'
THEN true ELSE false END
```
- base: `305: internal error. statemachine throws a runtime exception:
java.lang.NullPointerException: Cannot invoke
"org.apache.tsfile.block.column.Column.isNull(int)" because "curColumn" is null`
- this PR: d0/d1 get `MATCH`, d2/d3 stay NULL
Please also cover the guarded-cast UPDATE from the `DeviceAttributeUpdater`
comment.
##########
iotdb-core/calc-commons/src/test/java/org/apache/iotdb/calc/execution/operator/process/FilterAndProjectOperatorTest.java:
##########
@@ -0,0 +1,555 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.calc.execution.operator.process;
+
+import org.apache.iotdb.calc.execution.operator.CommonOperatorContext;
+import org.apache.iotdb.calc.execution.operator.Operator;
+import org.apache.iotdb.calc.plan.planner.memory.MemoryReservationManager;
+import org.apache.iotdb.calc.transformation.dag.column.ColumnTransformer;
+import
org.apache.iotdb.calc.transformation.dag.column.FailFunctionColumnTransformer;
+import
org.apache.iotdb.calc.transformation.dag.column.TableCaseWhenThenColumnTransformer;
+import
org.apache.iotdb.calc.transformation.dag.column.binary.ArithmeticAdditionColumnTransformer;
+import
org.apache.iotdb.calc.transformation.dag.column.binary.CompareGreaterThanColumnTransformer;
+import
org.apache.iotdb.calc.transformation.dag.column.leaf.ConstantColumnTransformer;
+import
org.apache.iotdb.calc.transformation.dag.column.leaf.IdentityColumnTransformer;
+import
org.apache.iotdb.calc.transformation.dag.column.leaf.LeafColumnTransformer;
+import
org.apache.iotdb.calc.transformation.dag.column.leaf.NullColumnTransformer;
+import
org.apache.iotdb.calc.transformation.dag.column.multi.MappableUDFColumnTransformer;
+import org.apache.iotdb.calc.transformation.dag.udf.UDTFExecutor;
+import org.apache.iotdb.commons.queryengine.plan.planner.plan.node.PlanNodeId;
+
+import org.apache.tsfile.block.column.Column;
+import org.apache.tsfile.block.column.ColumnBuilder;
+import org.apache.tsfile.enums.TSDataType;
+import org.apache.tsfile.read.common.block.TsBlock;
+import org.apache.tsfile.read.common.block.column.DoubleColumn;
+import org.apache.tsfile.read.common.block.column.RunLengthEncodedColumn;
+import org.apache.tsfile.read.common.block.column.TimeColumn;
+import org.junit.Test;
+
+import java.time.ZoneId;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.Iterator;
+import java.util.List;
+import java.util.Optional;
+import java.util.function.Consumer;
+
+import static org.apache.tsfile.read.common.type.BooleanType.BOOLEAN;
+import static org.apache.tsfile.read.common.type.DoubleType.DOUBLE;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+public class FilterAndProjectOperatorTest {
+
+ @Test
+ public void testProjectionDoesNotReuseColumnsAcrossDifferentSizedBlocks()
throws Exception {
+ Graph graph = new Graph();
+ ColumnTransformer input = graph.input(0);
+ ColumnTransformer caseWhen =
+ graph.caseWhen(graph.greaterThan(input, 0), graph.constant(1),
graph.nullValue());
+ try (Operator operator =
+ project(graph, new TsBlock[] {block(1), block(-2, 3), block(4)},
caseWhen, input)) {
+ assertBlock(operator.next(), new Double[] {1.0}, new Double[] {1.0});
+ assertBlock(operator.next(), new Double[] {null, 1.0}, new Double[]
{-2.0, 3.0});
+ assertBlock(operator.next(), new Double[] {1.0}, new Double[] {4.0});
+ }
+ }
+
+ @Test
+ public void testProjectionDoesNotReuseColumnsAcrossEqualSizedBlocks() throws
Exception {
+ Graph graph = new Graph();
+ ColumnTransformer input = graph.input(0);
+ ColumnTransformer caseWhen =
+ graph.caseWhen(graph.greaterThan(input, 0), graph.constant(1),
graph.nullValue());
+ try (Operator operator =
+ project(graph, new TsBlock[] {block(1, 2), block(-3, 4)}, caseWhen,
input)) {
+ assertBlock(operator.next(), new Double[] {1.0, 1.0}, new Double[] {1.0,
2.0});
+ assertBlock(operator.next(), new Double[] {null, 1.0}, new Double[]
{-3.0, 4.0});
+ }
+ }
+
+ @Test
+ public void testProjectionHandlesEmptyBlocks() throws Exception {
+ Graph graph = new Graph();
+ ColumnTransformer input = graph.input(0);
+ ColumnTransformer caseWhen =
+ graph.caseWhen(graph.greaterThan(input, 0), graph.constant(1),
graph.nullValue());
+ try (Operator operator =
+ project(graph, new TsBlock[] {block(1), block(), block(-2)}, caseWhen,
input)) {
+ assertBlock(operator.next(), new Double[] {1.0}, new Double[] {1.0});
+ assertBlock(operator.next(), new Double[0], new Double[0]);
+ assertBlock(operator.next(), new Double[] {null}, new Double[] {-2.0});
+ }
+ }
+
+ @Test
+ public void testProjectionClearsSharedIntermediateExpressions() throws
Exception {
+ Graph graph = new Graph();
+ ColumnTransformer shared = graph.add(graph.input(0), 0);
+ ColumnTransformer caseWhen =
+ graph.caseWhen(graph.greaterThan(shared, 0), graph.constant(1),
graph.nullValue());
+ try (Operator operator =
+ project(graph, new TsBlock[] {block(1), block(-2)}, caseWhen,
graph.add(shared, 1))) {
+ assertBlock(operator.next(), new Double[] {1.0}, new Double[] {2.0});
+ assertBlock(operator.next(), new Double[] {null}, new Double[] {-1.0});
+ }
+ }
+
+ @Test
+ public void testFilterDoesNotReuseColumnsAcrossBlocks() throws Exception {
+ Graph filter = new Graph();
+ ColumnTransformer input = filter.input(0);
+ ColumnTransformer caseWhen =
+ filter.caseWhen(filter.greaterThan(input, 0), filter.constant(1),
filter.constant(0));
+ ColumnTransformer predicate = filter.greaterThan(filter.add(caseWhen,
input), 0);
+ Graph projection = new Graph();
+ try (Operator operator =
+ operator(
+ new TsBlock[] {block(1), block(-2, 3), block(-4)},
+ filter,
+ predicate,
+ Collections.emptyList(),
+ projection,
+ projection.input(0))) {
+ assertBlock(operator.next(), new Double[] {1.0});
+ assertBlock(operator.next(), new Double[] {3.0});
+ assertBlock(operator.next(), new Double[0]);
+ }
+ }
+
+ @Test
+ public void testFilterCommonExpressionsDoNotCrossBlocks() throws Exception {
+ for (boolean runLengthEncodedTime : new boolean[] {false, true}) {
+ Graph filter = new Graph();
+ ColumnTransformer shared = filter.add(filter.input(0), 0);
+ ColumnTransformer caseWhen =
+ filter.caseWhen(filter.greaterThan(shared, 0), filter.constant(1),
filter.constant(0));
+ ColumnTransformer predicate = filter.greaterThan(caseWhen, 0);
+ Graph projection = new Graph();
+ try (Operator operator =
+ operator(
+ new TsBlock[] {
+ block(runLengthEncodedTime, 1),
+ block(runLengthEncodedTime, -2, 3),
+ block(runLengthEncodedTime, 4, -5)
+ },
+ filter,
+ predicate,
+ Collections.singletonList(shared),
+ projection,
+ projection.input(0),
+ projection.input(1))) {
+ assertBlock(operator.next(), new Double[] {1.0}, new Double[] {1.0});
+ assertBlock(operator.next(), new Double[] {3.0}, new Double[] {3.0});
+ assertBlock(operator.next(), new Double[] {4.0}, new Double[] {4.0});
+ }
+ }
+ }
+
+ @Test
+ public void testProjectionClearsCachedDescendantsOfNestedCase() throws
Exception {
+ Graph graph = new Graph();
+ ColumnTransformer shared = graph.add(graph.input(0), 0);
+ ColumnTransformer innerCase =
+ graph.caseWhen(graph.greaterThan(shared, 0), graph.constant(1),
graph.constant(0));
+ ColumnTransformer outerCase =
+ graph.caseWhen(graph.greaterThan(innerCase, 0), graph.constant(2),
graph.constant(0));
+ // Simulate an intermediate cache retained from the previous input block.
Clearing only the
+ // outer CASE itself must not allow this cached value to reach the new
block's WHEN condition.
+ shared.initializeColumnCache(doubleColumn(1));
+ try (Operator operator = project(graph, new TsBlock[] {block(-2)},
outerCase)) {
+ assertBlock(operator.next(), new Double[] {0.0});
+ }
+ }
+
+ @Test
+ public void testThenExpressionIsRecomputedWithoutSelectionForLaterOutput()
throws Exception {
+ Graph graph = new Graph();
+ ColumnTransformer input = graph.input(0);
+ ColumnTransformer shared = graph.add(input, 1);
+ ColumnTransformer caseWhen =
+ graph.caseWhen(graph.greaterThan(input, 0), shared, graph.constant(0));
+ try (Operator operator =
+ project(graph, new TsBlock[] {block(-2, 3), block(-4, 5)}, caseWhen,
shared)) {
+ assertBlock(operator.next(), new Double[] {0.0, 4.0}, new Double[]
{-1.0, 4.0});
+ assertBlock(operator.next(), new Double[] {0.0, 6.0}, new Double[]
{-3.0, 6.0});
+ }
+ }
+
+ @Test
+ public void testCaseStillSkipsUnusedFailBranch() throws Exception {
+ Graph graph = new Graph();
+ ColumnTransformer caseWhen =
+ graph.caseWhen(
+ graph.greaterThan(graph.input(0), 0),
+ graph.constant(1),
+ new FailFunctionColumnTransformer(DOUBLE, "The unused ELSE branch
was evaluated"));
+ try (Operator operator = project(graph, new TsBlock[] {block(1, 2),
block(3)}, caseWhen)) {
+ assertBlock(operator.next(), new Double[] {1.0, 1.0});
+ assertBlock(operator.next(), new Double[] {1.0});
+ }
+ }
+
+ @Test
+ public void testProjectionReusesSharedExpressionsWithinEachBlock() throws
Exception {
+ Graph graph = new Graph();
+ CountingIdentityColumnTransformer input = new
CountingIdentityColumnTransformer();
+ graph.leaves.add(input);
+ ColumnTransformer shared = graph.add(input, 0);
+ try (Operator operator =
+ project(graph, new TsBlock[] {block(1), block(2)}, shared,
graph.add(shared, 1))) {
+ assertBlock(operator.next(), new Double[] {1.0}, new Double[] {2.0});
+ assertEquals(1, input.evaluationCount);
+ assertBlock(operator.next(), new Double[] {2.0}, new Double[] {3.0});
+ assertEquals(2, input.evaluationCount);
+ }
+ }
+
+ @Test
+ public void testCopiedOperatorReusesCollectedTransformers() throws Exception
{
+ Graph graph = new Graph();
+ CountingIdentityColumnTransformer input = new
CountingIdentityColumnTransformer();
+ graph.leaves.add(input);
+ try (FilterAndProjectOperator original = project(graph, new TsBlock[0],
graph.add(input, 1));
+ FilterAndProjectOperator copy =
+ new FilterAndProjectOperator(
+ original, new TsBlockSourceOperator(new TsBlock[] {block(1),
block(2)}))) {
+ assertEquals(1, input.childTraversalCount);
+ assertBlock(copy.next(), new Double[] {2.0});
+ assertBlock(copy.next(), new Double[] {3.0});
+ assertEquals(1, input.childTraversalCount);
+ }
+ }
+
+ @Test
+ public void testProjectionCollectsSharedDagOnlyOnce() throws Exception {
+ Graph graph = new Graph();
+ CountingIdentityColumnTransformer input = new
CountingIdentityColumnTransformer();
+ graph.leaves.add(input);
+ ColumnTransformer shared = input;
+ // Each level references the same child twice. A traversal without
deduplication would visit
+ // the input 4096 times per root, although this DAG contains only 13
distinct transformers.
+ for (int i = 0; i < 12; i++) {
+ shared = graph.add(shared, shared);
+ }
+ try (Operator operator =
+ project(graph, new TsBlock[] {block(1), block(2)}, shared,
graph.add(shared, 1))) {
+ assertEquals(1, input.childTraversalCount);
+ assertBlock(operator.next(), new Double[] {4096.0}, new Double[]
{4097.0});
+ assertEquals(1, input.childTraversalCount);
+ assertBlock(operator.next(), new Double[] {8192.0}, new Double[]
{8193.0});
+ assertEquals(1, input.childTraversalCount);
+ }
+ }
+
+ @Test
+ public void testFilterAndCommonExpressionsCollectSharedDagOnlyOnce() throws
Exception {
+ Graph filter = new Graph();
+ CountingIdentityColumnTransformer input = new
CountingIdentityColumnTransformer();
+ filter.leaves.add(input);
+ ColumnTransformer shared = filter.add(input, 1);
+ ColumnTransformer predicate = filter.greaterThan(shared, 0);
+ Graph projection = new Graph();
+ try (Operator operator =
+ operator(
+ new TsBlock[] {block(1), block(-2, 3)},
+ filter,
+ predicate,
+ Collections.singletonList(shared),
+ projection,
+ projection.input(0),
+ projection.input(1))) {
+ assertEquals(1, input.childTraversalCount);
+ assertBlock(operator.next(), new Double[] {1.0}, new Double[] {2.0});
+ assertEquals(1, input.childTraversalCount);
+ assertBlock(operator.next(), new Double[] {3.0}, new Double[] {4.0});
+ assertEquals(1, input.childTraversalCount);
+ }
+ }
+
+ @Test
+ public void testMappableUdfClearsCaseDescendantsAcrossBlocks() throws
Exception {
+ Graph graph = new Graph();
+ ColumnTransformer input = graph.input(0);
+ ColumnTransformer caseWhen =
+ graph.caseWhen(graph.greaterThan(input, 0), graph.constant(1),
graph.constant(0));
+ ColumnTransformer udf =
+ new MappableUDFColumnTransformer(
+ DOUBLE,
+ new ColumnTransformer[] {reference(caseWhen), reference(input)},
+ new UDTFExecutor("test", ZoneId.of("UTC")) {
+ @Override
+ public void execute(Column[] columns, ColumnBuilder builder) {
+ for (int i = 0; i < columns[0].getPositionCount(); i++) {
+ builder.writeDouble(columns[0].getDouble(i) +
columns[1].getDouble(i));
+ }
+ }
+
+ @Override
+ public void beforeDestroy() {
+ // No UDF resources are allocated by this test executor.
+ }
+ });
+ try (Operator operator =
+ project(graph, new TsBlock[] {block(1), block(-2, 3), block(4)}, udf))
{
+ assertBlock(operator.next(), new Double[] {2.0});
+ assertBlock(operator.next(), new Double[] {-2.0, 4.0});
+ assertBlock(operator.next(), new Double[] {5.0});
+ }
+ }
+
+ @Test
+ public void testReusingCasePreservesPendingSiblingColumn() throws Exception {
+ Graph graph = new Graph();
+ ColumnTransformer input = graph.input(0);
+ ColumnTransformer sharedCase =
+ graph.caseWhen(graph.greaterThan(input, 0), graph.constant(1),
graph.constant(0));
+ ColumnTransformer nestedCase =
+ graph.caseWhen(graph.greaterThan(sharedCase, 0), graph.constant(2),
graph.constant(3));
+ // The first output leaves sharedCase cached. In the second output, input
is evaluated before
+ // nestedCase reuses sharedCase. Its cleanup must preserve input until the
addition reads it.
+ try (Operator operator =
+ project(
+ graph,
+ new TsBlock[] {block(-3, 0, 2), block(4, -1)},
+ sharedCase,
+ graph.add(input, nestedCase))) {
+ assertBlock(operator.next(), new Double[] {0.0, 0.0, 1.0}, new Double[]
{0.0, 3.0, 4.0});
+ assertBlock(operator.next(), new Double[] {1.0, 0.0}, new Double[] {6.0,
2.0});
+ }
+ }
+
+ private static FilterAndProjectOperator project(
+ Graph graph, TsBlock[] blocks, ColumnTransformer... outputs) {
+ return operator(blocks, null, null, Collections.emptyList(), graph,
outputs);
+ }
+
+ private static FilterAndProjectOperator operator(
+ TsBlock[] blocks,
+ Graph filter,
+ ColumnTransformer predicate,
+ List<ColumnTransformer> common,
+ Graph projection,
+ ColumnTransformer... outputs) {
+ List<ColumnTransformer> projectOutputs = new ArrayList<>();
+ for (ColumnTransformer output : outputs) {
+ projectOutputs.add(reference(output));
+ }
+ for (ColumnTransformer transformer : common) {
+ reference(transformer);
+ }
+ return new FilterAndProjectOperator(
+ new TestOperatorContext(),
+ new TsBlockSourceOperator(blocks),
+ Collections.nCopies(1 + common.size(), TSDataType.DOUBLE),
+ filter == null ? Collections.emptyList() : filter.leaves,
+ predicate == null ? null : reference(predicate),
+ common,
+ projection.leaves,
+ projectOutputs,
+ false,
+ predicate != null);
+ }
+
+ private static ColumnTransformer reference(ColumnTransformer transformer) {
+ transformer.addReferenceCount();
+ return transformer;
+ }
+
+ private static DoubleColumn doubleColumn(double... values) {
+ return new DoubleColumn(values.length, Optional.empty(), values);
+ }
+
+ private static TsBlock block(double... values) {
+ return block(false, values);
+ }
+
+ private static TsBlock block(boolean runLengthEncodedTime, double... values)
{
+ Column timeColumn =
+ runLengthEncodedTime
+ ? new RunLengthEncodedColumn(new TimeColumn(1, new long[] {0}),
values.length)
+ : new TimeColumn(values.length, new long[values.length]);
+ return new TsBlock(timeColumn, doubleColumn(values));
+ }
+
+ private static void assertBlock(TsBlock actual, Double[]... expectedColumns)
{
+ assertEquals(expectedColumns.length, actual.getValueColumnCount());
+ assertEquals(expectedColumns[0].length, actual.getPositionCount());
+ for (int columnIndex = 0; columnIndex < expectedColumns.length;
columnIndex++) {
+ Column column = actual.getColumn(columnIndex);
+ Double[] expected = expectedColumns[columnIndex];
+ assertEquals(expected.length, column.getPositionCount());
+ for (int row = 0; row < expected.length; row++) {
+ if (expected[row] == null) {
+ assertTrue(column.isNull(row));
+ } else {
+ assertFalse(column.isNull(row));
+ assertEquals(expected[row], column.getDouble(row), 0.0);
+ }
+ }
+ }
+ }
+
+ private static class Graph {
+ private final List<LeafColumnTransformer> leaves = new ArrayList<>();
+
+ private ColumnTransformer input(int index) {
+ return leaf(new IdentityColumnTransformer(DOUBLE, index));
+ }
+
+ private ColumnTransformer constant(double value) {
+ return leaf(new ConstantColumnTransformer(DOUBLE, doubleColumn(value)));
+ }
+
+ private ColumnTransformer nullValue() {
+ return leaf(new NullColumnTransformer());
+ }
+
+ private ColumnTransformer leaf(LeafColumnTransformer transformer) {
+ leaves.add(transformer);
+ return transformer;
+ }
+
+ private ColumnTransformer add(ColumnTransformer input, double value) {
+ return add(input, constant(value));
+ }
+
+ private ColumnTransformer add(ColumnTransformer left, ColumnTransformer
right) {
+ return new ArithmeticAdditionColumnTransformer(DOUBLE, reference(left),
reference(right));
+ }
+
+ private ColumnTransformer greaterThan(ColumnTransformer input, double
value) {
+ return new CompareGreaterThanColumnTransformer(
+ BOOLEAN, reference(input), reference(constant(value)));
+ }
+
+ private ColumnTransformer caseWhen(
+ ColumnTransformer when, ColumnTransformer then, ColumnTransformer
otherwise) {
+ return new TableCaseWhenThenColumnTransformer(
+ DOUBLE,
+ Collections.singletonList(reference(when)),
+ Collections.singletonList(reference(then)),
+ reference(otherwise));
+ }
+ }
+
+ private static class CountingIdentityColumnTransformer extends
IdentityColumnTransformer {
+ private int evaluationCount;
+ private int childTraversalCount;
+
+ private CountingIdentityColumnTransformer() {
+ super(DOUBLE, 0);
+ }
+
+ @Override
+ protected void evaluate() {
+ evaluationCount++;
+ super.evaluate();
+ }
+
+ @Override
+ protected void forEachChild(Consumer<ColumnTransformer> action) {
+ childTraversalCount++;
+ }
+ }
+
+ private static class TestOperatorContext extends CommonOperatorContext {
+ private TestOperatorContext() {
+ super(0, new PlanNodeId("test"), "test");
+ }
+
+ @Override
+ public MemoryReservationManager getMemoryReservationContext() {
+ return null;
+ }
+
+ @Override
+ public int getFragmentId() {
+ return 0;
+ }
+
+ @Override
+ public int getPipelineId() {
+ return 0;
+ }
+
+ @Override
+ public long ramBytesUsed() {
+ return 0;
+ }
+ }
+
+ private static class TsBlockSourceOperator implements Operator {
Review Comment:
Nit: this duplicates the production `ValuesOperator` in the same package.
`new ValuesOperator(new TestOperatorContext(), Arrays.asList(blocks))` should
do the same job, and it won't need updating when the `Operator` interface
changes.
##########
iotdb-core/calc-commons/src/main/java/org/apache/iotdb/calc/transformation/dag/column/AbstractCaseWhenThenColumnTransformer.java:
##########
@@ -234,6 +235,22 @@ protected void checkType() {
// do nothing
}
+ @Override
+ protected boolean shouldClearChildrenCache() {
+ // Clearing a cached CASE during evaluation must preserve pending sibling
columns.
+ return false;
+ }
Review Comment:
This preserves a pending sibling column only when the shared node is itself
a CASE. CASE's own cleanup
([L222-L227](https://github.com/apache/iotdb/blob/5e0cb1c8a98345af99bb1115eebad1be7ea12ffc/iotdb-core/calc-commons/src/main/java/org/apache/iotdb/calc/transformation/dag/column/AbstractCaseWhenThenColumnTransformer.java#L222-L227))
still clears its WHEN/THEN/ELSE subtrees recursively. Any non-CASE node shared
with an operand that the parent has already computed but not read yet is
therefore wiped within the same block. All of these fail the same way on base
and on this PR, with a single input block:
| SQL | expected | actual |
| --- | --- | --- |
| `SELECT s1 + CASE WHEN s1 > 0 THEN 2 ELSE 3 END FROM p` (s1 = 1, -2) |
3.0, 1.0 | NULL, NULL (tree model too) |
| `SELECT time, s1 FROM p WHERE s1 + CASE WHEN s1 > 0 THEN 2 ELSE 3 END > 0`
| 2 rows | empty set |
| `SELECT s1 > CASE WHEN s1 > 0 THEN 2 ELSE -3 END FROM p` | false, true |
`301: … "leftColumn" is null` |
| `SHOW DEVICES` / `COUNT DEVICES … WHERE src = CASE WHEN src = 'match' THEN
'match' ELSE 'other' END` | all devices | `301: … "leftColumn" is null` |
| `UPDATE t SET a = concat(src, CASE WHEN src = 'match' THEN '-m' ELSE '-o'
END)` | `match-m` / `other-o` | succeeds, but every `a` becomes NULL (old value
lost) |
| `greatest(s1, CASE …)`, `s1 BETWEEN -5 AND CASE …`, `substring(s, 1, CASE
…)`, `CASE WHEN s IS NOT NULL THEN substring(s, 1, length(s) - 1) END` | values
| NULL |
Cause:
[`BinaryColumnTransformer.evaluate()`](https://github.com/apache/iotdb/blob/5e0cb1c8a98345af99bb1115eebad1be7ea12ffc/iotdb-core/calc-commons/src/main/java/org/apache/iotdb/calc/transformation/dag/column/binary/BinaryColumnTransformer.java#L45-L56),
[`TernaryColumnTransformer.evaluate()` /
`evaluateWithSelection()`](https://github.com/apache/iotdb/blob/5e0cb1c8a98345af99bb1115eebad1be7ea12ffc/iotdb-core/calc-commons/src/main/java/org/apache/iotdb/calc/transformation/dag/column/ternary/TernaryColumnTransformer.java#L51-L79),
[`MultiColumnTransformer.evaluate()`](https://github.com/apache/iotdb/blob/5e0cb1c8a98345af99bb1115eebad1be7ea12ffc/iotdb-core/calc-commons/src/main/java/org/apache/iotdb/calc/transformation/dag/column/multi/MultiColumnTransformer.java#L44-L61)
and [`MappableUDFColumnTransformer.evaluate()` /
`evaluateWithSelection()`](https://github.com/apache/iotdb/blob/5e0cb1c8a98345af99bb1115eebad1be7ea12ffc/iotdb-core/calc-commons/src/main/java/org/apache/iotdb/calc/
transformation/dag/column/multi/MappableUDFColumnTransformer.java#L45-L77)
evaluate all children first and call `getColumn()` afterwards. When a later
child is (or contains) a CASE, its cleanup clears the shared node, and the
earlier child's read returns null / 0 positions.
[`BinaryColumnTransformer.evaluateWithSelection()`](https://github.com/apache/iotdb/blob/5e0cb1c8a98345af99bb1115eebad1be7ea12ffc/iotdb-core/calc-commons/src/main/java/org/apache/iotdb/calc/transformation/dag/column/binary/BinaryColumnTransformer.java#L59-L74)
already avoids this by reading the left column before evaluating the right one
(see its comment). I applied that rule everywhere: read each child's column and
position count right after evaluating it, before evaluating the next sibling.
In my local run that fixes every row above, and the calc-commons tests and
datanode transformer tests still pass.
The same argument applies to this opt-out itself: "preserve pending sibling
columns" holds for every node type, not just CASE. The new tests only use the
`CASE … = src` orientation, which happens to avoid the problem (see the IT
comment).
##########
iotdb-core/calc-commons/src/main/java/org/apache/iotdb/calc/execution/operator/process/FilterAndProjectOperator.java:
##########
@@ -152,6 +165,12 @@ public TsBlock next() throws Exception {
private TsBlock getFilterTsBlock(TsBlock input) {
final Column originTimeColumn = input.getTimeColumn();
final int positionCount = originTimeColumn.getPositionCount();
+ // Short-circuit evaluation can reset cache reference counts when a shared
expression is
+ // recomputed. Invalidate the whole DAG before feeding a new block,
including intermediate
+ // expressions whose caches may still have unconsumed references from the
previous block.
+ for (ColumnTransformer columnTransformer : filterCacheTransformers) {
+ columnTransformer.clearCacheForNewInput();
+ }
Review Comment:
Optional (same as base, not blocking): the reset runs when the *next* block
arrives. Columns left over from the previous block therefore stay reachable
while the driver is blocked and after the last block; an example is a shared
expression recomputed after CASE, which keeps an unconsumed reference.
Resetting at the end of `next()` (in a `finally`) would release them as soon as
the output block is built.
##########
iotdb-core/calc-commons/src/main/java/org/apache/iotdb/calc/transformation/dag/column/multi/MultiColumnTransformer.java:
##########
@@ -90,11 +91,8 @@ public List<ColumnTransformer> getChildren() {
}
@Override
- public void clearCache() {
- super.clearCache();
- for (ColumnTransformer columnTransformer : columnTransformerList) {
- columnTransformer.clearCache();
- }
+ protected void forEachChild(Consumer<ColumnTransformer> action) {
+ columnTransformerList.forEach(action);
}
Review Comment:
Not changed by this PR, but same mechanism, same class:
[`evaluateWithSelection()` reads `getColumnCachePositionCount()` at
L72](https://github.com/apache/iotdb/blob/5e0cb1c8a98345af99bb1115eebad1be7ea12ffc/iotdb-core/calc-commons/src/main/java/org/apache/iotdb/calc/transformation/dag/column/multi/MultiColumnTransformer.java#L72)
after `getColumn()` has already released the first child's column (when that
child has a single reference), so the result has 0 rows. Any multi-argument
function under THEN/ELSE whose first argument is referenced only once returns
NULL, on base and on this PR:
```sql
SELECT time, CASE WHEN time > 0 THEN coalesce(s1, 0.0) END FROM p --
expected 1.0, -2.0; actual NULL, NULL
SELECT time, CASE WHEN time > 0 THEN greatest(s1, 0.5) END FROM p --
expected 1.0, 0.5; actual NULL, NULL
```
8bf22945c1 fixed this exact pattern in `LogicAnd` / `LogicOr` /
`LogicalAndMulti` / `LogicalOrMulti` / `MappableUDF` but missed this class.
Reading the position count before `getColumn()`, as `evaluate()` does at
L50-L51, fixes it.
##########
iotdb-core/datanode/src/test/java/org/apache/iotdb/db/queryengine/execution/operator/schema/source/DevicePredicateHandlerTest.java:
##########
@@ -0,0 +1,323 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.queryengine.execution.operator.schema.source;
+
+import org.apache.iotdb.calc.transformation.dag.column.ColumnTransformer;
+import
org.apache.iotdb.calc.transformation.dag.column.TableCaseWhenThenColumnTransformer;
+import
org.apache.iotdb.calc.transformation.dag.column.binary.CompareEqualToColumnTransformer;
+import
org.apache.iotdb.calc.transformation.dag.column.leaf.ConstantColumnTransformer;
+import
org.apache.iotdb.calc.transformation.dag.column.leaf.IdentityColumnTransformer;
+import
org.apache.iotdb.calc.transformation.dag.column.leaf.LeafColumnTransformer;
+import org.apache.iotdb.commons.path.PartialPath;
+import org.apache.iotdb.commons.schema.node.role.IDeviceMNode;
+import org.apache.iotdb.commons.schema.table.TsTable;
+import org.apache.iotdb.commons.schema.table.column.TagColumnSchema;
+import org.apache.iotdb.commons.schema.table.column.TsTableColumnSchema;
+import org.apache.iotdb.db.schemaengine.rescon.MemSchemaRegionStatistics;
+import
org.apache.iotdb.db.schemaengine.schemaregion.mtree.impl.mem.mnode.IMemMNode;
+import
org.apache.iotdb.db.schemaengine.schemaregion.mtree.impl.mem.mnode.info.TableDeviceInfo;
+import
org.apache.iotdb.db.schemaengine.schemaregion.read.resp.info.impl.ShowDevicesResult;
+
+import org.apache.tsfile.common.conf.TSFileConfig;
+import org.apache.tsfile.enums.TSDataType;
+import org.apache.tsfile.read.common.block.column.BinaryColumn;
+import org.apache.tsfile.read.common.block.column.BooleanColumn;
+import org.apache.tsfile.utils.Binary;
+import org.junit.Test;
+
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Optional;
+
+import static org.apache.tsfile.read.common.type.BooleanType.BOOLEAN;
+import static org.apache.tsfile.read.common.type.StringType.STRING;
+import static org.junit.Assert.assertEquals;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.verify;
+import static org.mockito.Mockito.when;
+
+public class DevicePredicateHandlerTest {
+
+ private static final List<TsTableColumnSchema> COLUMNS =
+ Collections.singletonList(new TagColumnSchema("tag", TSDataType.STRING));
+
+ @Test
+ public void testPredicateFilterDoesNotReuseFullBatches() throws Exception {
+ Graph graph = new Graph();
+ ColumnTransformer predicate = graph.matchesCase(graph.input(0));
+ try (DevicePredicateFilter filter = predicateFilter(graph, predicate)) {
+ int batchSize = DevicePredicateHandler.DEFAULT_MAX_TS_BLOCK_LINE_NUMBER;
+ for (String value : new String[] {"match", "other"}) {
+ for (int i = 0; i < batchSize; i++) {
+ filter.addBatch(device(value));
+ }
+ // addBatch evaluates full batches automatically; exhausting the
iterator resets the batch.
+ List<String> result = consume(filter);
+ assertEquals(batchSize, result.size());
+ for (String actual : result) {
+ assertEquals(value, actual);
+ }
+ }
+ }
+ }
+
+ @Test
+ public void testPredicateFilterDoesNotReuseDifferentSizedBatches() throws
Exception {
+ Graph graph = new Graph();
+ ColumnTransformer predicate = graph.matchesCase(graph.input(0));
+ try (DevicePredicateFilter filter = predicateFilter(graph, predicate)) {
+ assertEquals(Collections.singletonList("match"), filterBatch(filter,
"match"));
+ assertEquals(Arrays.asList("other", "match"), filterBatch(filter,
"other", "match"));
+ assertEquals(Collections.emptyList(), filterBatch(filter, "skip"));
+ }
+ }
+
+ @Test
+ public void testBlackListConstructorDoesNotReusePredicateAcrossBatches()
throws Exception {
+ Graph graph = new Graph();
+ ColumnTransformer predicate = graph.matchesCase(graph.input(0));
+ List<IDeviceMNode<IMemMNode>> nodes =
+ Arrays.asList(node("match", 0), node("other", 1), node("match", 2));
+ try (DeviceBlackListConstructor constructor =
+ new DeviceBlackListConstructor(
+ graph.leaves,
+ reference(predicate),
+ "table",
+ COLUMNS,
+ (pointer, name) -> null,
+ mock(MemSchemaRegionStatistics.class),
+ "test",
+ new TsTable("table"))) {
+ constructor.handleDeviceNode(nodes.get(0));
+ constructor.prepareBatchResult();
+ constructor.update();
+ constructor.handleDeviceNode(nodes.get(1));
+ constructor.handleDeviceNode(nodes.get(2));
+ constructor.prepareBatchResult();
+ constructor.update();
+ assertEquals(3, constructor.getPreDeletedNum());
+ for (IDeviceMNode<IMemMNode> node : nodes) {
+ verify(node).preDeactivateSelfOrTemplate();
+ }
+ }
+ }
+
+ @Test
+ public void testAttributeUpdaterDoesNotReuseFilterAcrossBatches() throws
Exception {
+ Graph filter = new Graph();
+ ColumnTransformer predicate = filter.matchesCase(filter.input(0));
+ Graph projection = new Graph();
+ List<List<String>> updates = new ArrayList<>();
+ try (DeviceAttributeUpdater updater =
+ attributeUpdater(
+ filter, predicate, Collections.emptyList(), projection, updates,
projection.input(0))) {
+ updateBatch(updater, "match");
+ updateBatch(updater, "other", "match");
+ assertEquals(
+ Arrays.asList(
+ Arrays.asList("match", "match"),
+ Arrays.asList("other", "other"),
+ Arrays.asList("match", "match")),
+ updates);
+ }
+ }
+
+ @Test
+ public void testAttributeUpdaterDoesNotReuseProjectionAcrossBatches() throws
Exception {
+ Graph projection = new Graph();
+ ColumnTransformer input = projection.input(0);
+ ColumnTransformer caseWhen = projection.caseResult(input);
+ List<List<String>> updates = new ArrayList<>();
+ try (DeviceAttributeUpdater updater =
+ attributeUpdater(
+ null, null, Collections.emptyList(), projection, updates,
caseWhen, input)) {
+ updateBatch(updater, "match");
+ updateBatch(updater, "other", "match");
+ assertEquals(
+ Arrays.asList(
+ Arrays.asList("match", "match", "match"),
+ Arrays.asList("other", "other", "other"),
+ Arrays.asList("match", "match", "match")),
+ updates);
+ }
+ }
+
+ @Test
+ public void testAttributeUpdaterRecomputesCommonExpressionClearedByCase()
throws Exception {
+ Graph filter = new Graph();
+ ColumnTransformer shared = filter.input(0);
Review Comment:
`filter.input(0)` as a common transformer cannot come from the planner (bare
column references never become common expressions). This test therefore
exercises the recompute mechanism, but not a DAG the planner builds. A shared
non-leaf node (for example, a function over `input`) would match what `UPDATE …
SET a = upper(src) WHERE CASE WHEN upper(src) = 'MATCH' …` produces.
##########
iotdb-core/calc-commons/src/main/java/org/apache/iotdb/calc/transformation/dag/column/AbstractCaseWhenThenColumnTransformer.java:
##########
@@ -234,6 +235,22 @@ protected void checkType() {
// do nothing
}
+ @Override
+ protected boolean shouldClearChildrenCache() {
+ // Clearing a cached CASE during evaluation must preserve pending sibling
columns.
+ return false;
+ }
+
+ @Override
+ protected void forEachChild(Consumer<ColumnTransformer> action) {
+ for (Pair<ColumnTransformer, ColumnTransformer> whenThenColumnTransformer :
+ whenThenTransformers) {
+ action.accept(whenThenColumnTransformer.left);
+ action.accept(whenThenColumnTransformer.right);
+ }
+ action.accept(elseTransformer);
+ }
Review Comment:
Nit: CASE now keeps the same child list in three places: here, the clear
loop in `doTransform()`
([L222-L227](https://github.com/apache/iotdb/blob/5e0cb1c8a98345af99bb1115eebad1be7ea12ffc/iotdb-core/calc-commons/src/main/java/org/apache/iotdb/calc/transformation/dag/column/AbstractCaseWhenThenColumnTransformer.java#L222-L227))
and `close()`
([L255-L263](https://github.com/apache/iotdb/blob/5e0cb1c8a98345af99bb1115eebad1be7ea12ffc/iotdb-core/calc-commons/src/main/java/org/apache/iotdb/calc/transformation/dag/column/AbstractCaseWhenThenColumnTransformer.java#L255-L263)).
Since the goal is a single child-edge definition, consider
`forEachChild(ColumnTransformer::clearCache)` in `doTransform()` and
`forEachChild(ColumnTransformer::close)` in `close()`. The same applies to the
`close()` methods of Unary/Binary/Ternary/Multi/MappableUDF. That way a new
child edge cannot be added to one list and missed in another.
##########
integration-test/src/test/java/org/apache/iotdb/relational/it/schema/IoTDBCaseDeviceCacheIT.java:
##########
@@ -0,0 +1,136 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.relational.it.schema;
+
+import org.apache.iotdb.it.env.EnvFactory;
+import org.apache.iotdb.it.framework.IoTDBTestRunner;
+import org.apache.iotdb.itbase.category.TableClusterIT;
+import org.apache.iotdb.itbase.category.TableLocalStandaloneIT;
+import org.apache.iotdb.itbase.env.BaseEnv;
+
+import org.junit.AfterClass;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+
+import java.sql.Connection;
+import java.sql.SQLException;
+import java.sql.Statement;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+
+import static org.apache.iotdb.db.it.utils.TestUtils.assertResultSetEqual;
+import static org.apache.iotdb.db.it.utils.TestUtils.prepareTableData;
+
+@RunWith(IoTDBTestRunner.class)
+@Category({TableLocalStandaloneIT.class, TableClusterIT.class})
+public class IoTDBCaseDeviceCacheIT {
+ private static final String DATABASE = "case_device_cache";
+ private static final String CASE_RESULT = "CASE WHEN src = 'match' THEN
'match' ELSE 'other' END";
+
+ @BeforeClass
+ public static void setUp() throws Exception {
+ // Keep every device in the same schema region and force a new batch for
each device.
+
EnvFactory.getEnv().getConfig().getCommonConfig().setDefaultSchemaRegionGroupNumPerDatabase(1);
+
EnvFactory.getEnv().getConfig().getCommonConfig().setMaxTsBlockLineNumber(1);
+ EnvFactory.getEnv().initClusterEnvironment();
+ prepareTableData(new String[] {"CREATE DATABASE " + DATABASE});
+ }
+
+ @AfterClass
+ public static void tearDown() throws Exception {
+ EnvFactory.getEnv().cleanClusterEnvironment();
+ }
+
+ @Test
+ public void testDevicePredicatesAcrossBatches() throws Exception {
+ try (Connection connection =
EnvFactory.getEnv().getConnection(BaseEnv.TABLE_SQL_DIALECT);
+ Statement statement = connection.createStatement()) {
+ createTable(statement, "predicate_cache");
+ String predicate = CASE_RESULT + " = src";
Review Comment:
All new schema tests (and `DevicePredicateHandlerTest`) put the CASE on the
left: `CASE … = src`. The mirrored predicate `src = CASE … END` already fails
in the first batch on both base and this PR (`301: … "leftColumn" is null`; see
the comment on `shouldClearChildrenCache()`). Please cover both orientations
once that is fixed.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]