Caideyipi commented on code in PR #18606:
URL: https://github.com/apache/iotdb/pull/18606#discussion_r3964636195
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/plan/node/write/InsertTabletNode.java:
##########
@@ -792,108 +749,12 @@ private void writeValues(DataOutputStream stream) throws
IOException {
}
private void serializeColumn(TSDataType dataType, Object column, ByteBuffer
buffer) {
- switch (dataType) {
- case INT32:
- case DATE:
- int[] intValues = (int[]) column;
- for (int j = 0; j < rowCount; j++) {
- ReadWriteIOUtils.write(intValues[j], buffer);
- }
- break;
- case INT64:
- case TIMESTAMP:
- long[] longValues = (long[]) column;
- for (int j = 0; j < rowCount; j++) {
- ReadWriteIOUtils.write(longValues[j], buffer);
- }
- break;
- case FLOAT:
- float[] floatValues = (float[]) column;
- for (int j = 0; j < rowCount; j++) {
- ReadWriteIOUtils.write(floatValues[j], buffer);
- }
- break;
- case DOUBLE:
- double[] doubleValues = (double[]) column;
- for (int j = 0; j < rowCount; j++) {
- ReadWriteIOUtils.write(doubleValues[j], buffer);
- }
- break;
- case BOOLEAN:
- boolean[] boolValues = (boolean[]) column;
- for (int j = 0; j < rowCount; j++) {
- ReadWriteIOUtils.write(BytesUtils.boolToByte(boolValues[j]), buffer);
- }
- break;
- case TEXT:
- case BLOB:
- case STRING:
- case OBJECT:
- Binary[] binaryValues = (Binary[]) column;
- for (int j = 0; j < rowCount; j++) {
- if (binaryValues[j] != null && binaryValues[j].getValues() != null) {
- ReadWriteIOUtils.write(binaryValues[j], buffer);
- } else {
- ReadWriteIOUtils.write(0, buffer);
- }
- }
- break;
- default:
- throw new
UnSupportedDataTypeException(String.format(DATATYPE_UNSUPPORTED, dataType));
- }
+ Type.fromTsDataType(dataType).serializeArray(column, rowCount, buffer);
Review Comment:
`Type.serializeArray` uses TsFile's binary-array format, which writes a
presence byte before each binary value. The Tablet reader in
`RAW_ARRAY_BYTE_BUFFER_DESERIALIZER_SERVICE` still reads the legacy Tablet
format (length + payload only, with nulls represented by the separate bitmap),
so the first presence byte is interpreted as part of the length and the stream
becomes misaligned. CI fails with `BufferUnderflowException` in
`InsertMultiTabletsNodeSerdeTest`. Please preserve the existing Tablet wire
format or update both serialization, size calculation, and all readers together.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/queryengine/plan/planner/plan/node/write/InsertTabletNode.java:
##########
@@ -1032,36 +893,7 @@ private int getRowNumInRange(List<int[]> rangeList) {
}
private int getColumnSize(TSDataType dataType, Object column, int start, int
end) {
- int size = 0;
- switch (dataType) {
- case INT32:
- case DATE:
- size += Integer.BYTES * (end - start);
- break;
- case INT64:
- case TIMESTAMP:
- size += Long.BYTES * (end - start);
- break;
- case FLOAT:
- size += Float.BYTES * (end - start);
- break;
- case DOUBLE:
- size += Double.BYTES * (end - start);
- break;
- case BOOLEAN:
- size += Byte.BYTES * (end - start);
- break;
- case TEXT:
- case BLOB:
- case STRING:
- case OBJECT:
- Binary[] binaryValues = (Binary[]) column;
- for (int j = start; j < end; j++) {
- size += ReadWriteIOUtils.sizeToWrite(binaryValues[j]);
- }
- break;
- }
- return size;
+ return Type.fromTsDataType(dataType).serializedSize(column, start, end);
Review Comment:
The WAL size calculation now uses `Type.serializedSize`, whose binary-array
size includes a presence byte, but `WAL_ARRAY_WRITER_SERVICE` still writes only
the length and payload. The calculated WAL entry size therefore disagrees with
the bytes actually written, which breaks WAL offsets/search indexes. The three
`WALFileTest` failures in CI reproduce this. Please use a size calculation
matching the WAL writer's wire format.
##########
iotdb-core/calc-commons/src/main/java/org/apache/iotdb/calc/execution/operator/source/relational/aggregation/grouped/GroupedMaxAccumulator.java:
##########
@@ -519,4 +287,100 @@ protected void updateBooleanValue(int groupId, boolean
value) {
booleanValues.set(groupId, value);
}
}
+
+ @Override
+ public RuntimeException unsupportedException() {
+ return new UnSupportedDataTypeException(
+ String.format(CalcMessages.UNSUPPORTED_DATA_TYPE_IN_MAX_AGGREGATION,
seriesDataType));
+ }
+
+ @Override
+ public void initializeIntValues() {
+ intValues = new IntBigArray(Integer.MIN_VALUE);
+ }
+
+ @Override
+ public void initializeLongValues() {
+ longValues = new LongBigArray(Long.MIN_VALUE);
+ }
+
+ @Override
+ public void initializeFloatValues() {
+ floatValues = new FloatBigArray(Float.MIN_VALUE);
Review Comment:
`Float.MIN_VALUE` and `Double.MIN_VALUE` are the smallest positive values,
not lower bounds. With this initialization, `MAX` ignores zero and all negative
FLOAT/DOUBLE inputs, leaving `inits` false and returning NULL. The existing CI
tests `testFloatMaxWithNonPositiveInput`, `testDoubleMaxWithNonPositiveInput`,
and `testDoubleMaxWithNonPositiveIntermediateInput` reproduce this regression.
Please keep the previous `NEGATIVE_INFINITY` initialization or use another true
lower bound.
##########
iotdb-core/datanode/src/main/java/org/apache/iotdb/db/utils/TypeServices.java:
##########
@@ -0,0 +1,4012 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.db.utils;
+
+import org.apache.iotdb.calc.exception.QueryProcessException;
+import org.apache.iotdb.calc.execution.aggregation.Accumulator;
+import org.apache.iotdb.calc.execution.aggregation.BinaryModeAccumulator;
+import org.apache.iotdb.calc.execution.aggregation.BooleanModeAccumulator;
+import org.apache.iotdb.calc.execution.aggregation.DoubleModeAccumulator;
+import org.apache.iotdb.calc.execution.aggregation.FloatModeAccumulator;
+import org.apache.iotdb.calc.execution.aggregation.IntModeAccumulator;
+import org.apache.iotdb.calc.execution.aggregation.LongModeAccumulator;
+import org.apache.iotdb.calc.execution.operator.process.fill.IFill;
+import
org.apache.iotdb.calc.execution.operator.process.fill.constant.BinaryConstantFill;
+import
org.apache.iotdb.calc.execution.operator.process.fill.constant.BooleanConstantFill;
+import
org.apache.iotdb.calc.execution.operator.process.fill.constant.DoubleConstantFill;
+import
org.apache.iotdb.calc.execution.operator.process.fill.constant.FloatConstantFill;
+import
org.apache.iotdb.calc.execution.operator.process.fill.constant.IntConstantFill;
+import
org.apache.iotdb.calc.execution.operator.process.fill.constant.LongConstantFill;
+import org.apache.iotdb.calc.i18n.CalcMessages;
+import org.apache.iotdb.calc.plan.planner.CommonOperatorUtils;
+import org.apache.iotdb.calc.transformation.dag.util.CastFunctionUtils;
+import org.apache.iotdb.calc.utils.constant.SqlConstant;
+import org.apache.iotdb.commons.conf.CommonDescriptor;
+import org.apache.iotdb.commons.exception.SemanticException;
+import org.apache.iotdb.commons.exception.pipe.PipeRuntimeNonCriticalException;
+import
org.apache.iotdb.commons.queryengine.plan.relational.sql.ast.BinaryLiteral;
+import
org.apache.iotdb.commons.queryengine.plan.relational.sql.ast.BooleanLiteral;
+import
org.apache.iotdb.commons.queryengine.plan.relational.sql.ast.DoubleLiteral;
+import
org.apache.iotdb.commons.queryengine.plan.relational.sql.ast.FloatLiteral;
+import
org.apache.iotdb.commons.queryengine.plan.relational.sql.ast.GenericLiteral;
+import org.apache.iotdb.commons.queryengine.plan.relational.sql.ast.Literal;
+import
org.apache.iotdb.commons.queryengine.plan.relational.sql.ast.LongLiteral;
+import
org.apache.iotdb.commons.queryengine.plan.relational.sql.ast.StringLiteral;
+import org.apache.iotdb.commons.queryengine.utils.DateTimeUtils;
+import org.apache.iotdb.commons.queryengine.utils.TimestampPrecisionUtils;
+import org.apache.iotdb.db.conf.IoTDBConfig;
+import org.apache.iotdb.db.i18n.DataNodeMiscMessages;
+import org.apache.iotdb.db.i18n.DataNodePipeMessages;
+import org.apache.iotdb.db.i18n.DataNodeQueryMessages;
+import org.apache.iotdb.db.pipe.processor.aggregate.TimeSeriesRuntimeState;
+import
org.apache.iotdb.db.pipe.processor.aggregate.operator.intermediateresult.sametype.numeric.AbstractSameTypeNumericOperator;
+import
org.apache.iotdb.db.pipe.processor.aggregate.window.datastructure.WindowOutput;
+import
org.apache.iotdb.db.queryengine.execution.operator.window.EqualBinaryWindowManager;
+import
org.apache.iotdb.db.queryengine.execution.operator.window.EqualBooleanWindowManager;
+import
org.apache.iotdb.db.queryengine.execution.operator.window.EqualDoubleWindowManager;
+import
org.apache.iotdb.db.queryengine.execution.operator.window.EqualFloatWindowManager;
+import
org.apache.iotdb.db.queryengine.execution.operator.window.EqualIntWindowManager;
+import
org.apache.iotdb.db.queryengine.execution.operator.window.EqualLongWindowManager;
+import
org.apache.iotdb.db.queryengine.execution.operator.window.VariationDoubleWindowManager;
+import
org.apache.iotdb.db.queryengine.execution.operator.window.VariationFloatWindowManager;
+import
org.apache.iotdb.db.queryengine.execution.operator.window.VariationIntWindowManager;
+import
org.apache.iotdb.db.queryengine.execution.operator.window.VariationLongWindowManager;
+import
org.apache.iotdb.db.queryengine.execution.operator.window.VariationWindowManager;
+import
org.apache.iotdb.db.queryengine.execution.operator.window.VariationWindowParameter;
+import org.apache.iotdb.db.queryengine.statistics.StatisticsManager;
+import
org.apache.iotdb.db.queryengine.transformation.dag.transformer.unary.ArithmeticNegationTransformer;
+import
org.apache.iotdb.db.queryengine.transformation.dag.transformer.unary.InTransformer;
+import
org.apache.iotdb.db.queryengine.transformation.dag.transformer.unary.scalar.DiffFunctionTransformer;
+import
org.apache.iotdb.db.queryengine.transformation.dag.transformer.unary.scalar.RoundFunctionTransformer;
+import
org.apache.iotdb.db.queryengine.transformation.datastructure.util.ValueRecorder;
+import
org.apache.iotdb.db.storageengine.dataregion.wal.buffer.IWALByteBufferView;
+import org.apache.iotdb.db.storageengine.dataregion.wal.utils.WALWriteUtils;
+import org.apache.iotdb.db.utils.datastructure.AlignedTVList;
+import org.apache.iotdb.db.utils.datastructure.BinaryTVList;
+import org.apache.iotdb.db.utils.datastructure.BooleanTVList;
+import org.apache.iotdb.db.utils.datastructure.DoubleTVList;
+import org.apache.iotdb.db.utils.datastructure.FloatTVList;
+import org.apache.iotdb.db.utils.datastructure.IntTVList;
+import org.apache.iotdb.db.utils.datastructure.LongTVList;
+import org.apache.iotdb.db.utils.datastructure.TVList;
+import org.apache.iotdb.db.utils.windowing.window.EvictableBatchList;
+import org.apache.iotdb.db.utils.windowing.window.WindowImpl;
+import org.apache.iotdb.pipe.api.access.Row;
+import org.apache.iotdb.rpc.RpcUtils;
+
+import com.google.common.io.BaseEncoding;
+import com.sun.jna.platform.win32.OaIdl;
+import com.sun.jna.platform.win32.OleAuto;
+import com.sun.jna.platform.win32.Variant;
+import com.sun.jna.platform.win32.WTypes;
+import com.sun.jna.platform.win32.WinDef;
+import org.apache.tsfile.block.column.Column;
+import org.apache.tsfile.block.column.ColumnBuilder;
+import org.apache.tsfile.common.conf.TSFileConfig;
+import org.apache.tsfile.encoding.decoder.Decoder;
+import org.apache.tsfile.enums.TSDataType;
+import org.apache.tsfile.external.commons.lang3.StringUtils;
+import org.apache.tsfile.file.metadata.IChunkMetadata;
+import org.apache.tsfile.file.metadata.enums.TSEncoding;
+import org.apache.tsfile.file.metadata.statistics.DateStatistics;
+import org.apache.tsfile.file.metadata.statistics.Statistics;
+import org.apache.tsfile.read.common.BatchData;
+import org.apache.tsfile.read.common.block.TsBlockBuilder;
+import org.apache.tsfile.read.common.block.column.BinaryColumnBuilder;
+import org.apache.tsfile.read.common.type.Type;
+import org.apache.tsfile.read.common.type.service.TypeService;
+import org.apache.tsfile.read.filter.basic.Filter;
+import org.apache.tsfile.read.reader.series.PaginationController;
+import org.apache.tsfile.utils.Binary;
+import org.apache.tsfile.utils.BitMap;
+import org.apache.tsfile.utils.BytesUtils;
+import org.apache.tsfile.utils.DateUtils;
+import org.apache.tsfile.utils.Pair;
+import org.apache.tsfile.utils.RamUsageEstimator;
+import org.apache.tsfile.utils.ReadWriteIOUtils;
+import org.apache.tsfile.utils.TsPrimitiveType;
+import org.apache.tsfile.write.UnSupportedDataTypeException;
+import org.apache.tsfile.write.chunk.AlignedChunkWriterImpl;
+import org.apache.tsfile.write.chunk.ChunkWriterImpl;
+import org.apache.tsfile.write.chunk.ValueChunkWriter;
+import org.eclipse.milo.opcua.stack.core.Identifiers;
+import org.eclipse.milo.opcua.stack.core.types.builtin.DateTime;
+import org.eclipse.milo.opcua.stack.core.types.builtin.NodeId;
+
+import java.io.DataInputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.io.InputStream;
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.sql.Date;
+import java.time.Instant;
+import java.time.LocalDate;
+import java.time.ZoneId;
+import java.time.ZoneOffset;
+import java.util.Arrays;
+import java.util.List;
+import java.util.Objects;
+import java.util.Set;
+import java.util.function.BiConsumer;
+import java.util.function.BiFunction;
+import java.util.function.Function;
+import java.util.function.IntFunction;
+import java.util.function.LongSupplier;
+import java.util.function.Supplier;
+import java.util.function.ToDoubleFunction;
+import java.util.function.ToIntFunction;
+import java.util.function.ToLongFunction;
+
+public class TypeServices {
+
+ public static final int DEFAULT_DATE =
+ DateUtils.parseDateExpressionToInt(LocalDate.of(1970, 1, 1));
+ private static final LocalDate EMPTY_LOCAL_DATE = LocalDate.of(1000, 1, 1);
+
+ public static final class Memory {
+
+ public static final TypeService<LongSupplier>
STATEMENT_VALUE_SIZE_PER_LINE_SERVICE =
+ type ->
+ switch (type.getTypeEnum()) {
+ case BOOLEAN, INT32, DATE, INT64, TIMESTAMP, FLOAT, DOUBLE ->
type::estimateValueSize;
+ case TEXT, BLOB, STRING ->
+ () ->
StatisticsManager.getInstance().getMaxBinarySizeInBytes();
+ case OBJECT, ROW, UNKNOWN, VECTOR ->
+ () -> {
+ throw new UnsupportedOperationException(
+ CalcMessages.UNKNOWN_DATATYPE + type.getTypeEnum());
+ };
+ };
+
+ public static final TypeService<ToLongFunction<Object>>
INSERT_NODE_VALUE_SIZE_SERVICE =
+ type ->
+ switch (type.getTypeEnum()) {
+ case BOOLEAN, INT32, DATE, INT64, TIMESTAMP, FLOAT, DOUBLE ->
+ value ->
+ RamUsageEstimator.alignObjectSize(
+ type.estimateValueSize() +
RamUsageEstimator.NUM_BYTES_OBJECT_HEADER);
+ case TEXT, BLOB, STRING ->
+ value -> Objects.nonNull(value) ? ((Binary)
value).ramBytesUsed() : 0L;
+ case OBJECT, ROW, UNKNOWN, VECTOR -> value -> 0L;
+ };
+
+ static {
+ STATEMENT_VALUE_SIZE_PER_LINE_SERVICE.check();
+ INSERT_NODE_VALUE_SIZE_SERVICE.check();
+ }
+
+ private Memory() {
+ // Utility class
+ }
+ }
+
+ public static final class Transformation {
+
+ public static final TypeService<Function<Object, Column>>
CONSTANT_COLUMN_BUILDER_SERVICE =
+ type ->
+ switch (type.getTypeEnum()) {
+ case BOOLEAN, INT32, INT64, FLOAT, DOUBLE, TEXT ->
+ value -> {
+ final ColumnBuilder builder = type.createColumnBuilder(1);
+ type.writeObject(builder, value);
+ return builder.build();
+ };
+ case DATE, TIMESTAMP, BLOB, STRING, OBJECT, ROW, UNKNOWN, VECTOR
->
+ throw new UnSupportedDataTypeException(
+ DataNodeQueryMessages.UNSUPPORTED_TYPE +
type.getTypeEnum())
+ .setChecked(true);
+ };
+
+ public static final TypeService<
+
Function<org.apache.iotdb.db.queryengine.plan.statement.literal.Literal, IFill>>
+ CONSTANT_FILL_SERVICE =
+ type ->
+ switch (type.getTypeEnum()) {
+ case BOOLEAN -> literal -> new
BooleanConstantFill(literal.getBoolean());
+ case INT32 -> literal -> new
IntConstantFill(literal.getInt());
+ case DATE -> literal -> new
IntConstantFill(literal.getDate());
+ case INT64, TIMESTAMP -> literal -> new
LongConstantFill(literal.getLong());
+ case FLOAT -> literal -> new
FloatConstantFill(literal.getFloat());
+ case DOUBLE -> literal -> new
DoubleConstantFill(literal.getDouble());
+ case TEXT, BLOB, STRING -> literal -> new
BinaryConstantFill(literal.getBinary());
+ case OBJECT, ROW, UNKNOWN, VECTOR ->
+ literal -> {
+ throw new IllegalArgumentException(
+ CommonOperatorUtils.UNKNOWN_DATATYPE +
type.getTypeEnum());
+ };
+ };
+
+ public static final TypeService<IntFunction<ColumnBuilder>>
COLUMN_BUILDER_SERVICE =
+ type ->
+ switch (type.getTypeEnum()) {
+ case BOOLEAN,
+ INT32,
+ DATE,
+ INT64,
+ TIMESTAMP,
+ FLOAT,
+ DOUBLE,
+ TEXT,
+ BLOB,
+ STRING,
+ OBJECT ->
+ type::createColumnBuilder;
+ case ROW, UNKNOWN, VECTOR ->
+ throw new UnSupportedDataTypeException(
+ String.format(
+ DataNodeQueryMessages.UNSUPPORTED_DATA_TYPE_FMT,
type.getTypeEnum()))
+ .setChecked(true);
+ };
+
+ public static final TypeService<ColumnToDoubleConverter>
VALUE_TO_DOUBLE_SERVICE =
+ type ->
+ switch (type.getTypeEnum()) {
+ case INT32, DATE -> type::getInt;
+ case INT64, TIMESTAMP -> type::getLong;
+ case FLOAT -> type::getFloat;
+ case DOUBLE -> type::getDouble;
+ case BOOLEAN -> (column, index) -> type.getBoolean(column,
index) ? 1 : 0;
+ case TEXT, BLOB, STRING, OBJECT, ROW, UNKNOWN, VECTOR ->
+ (column, index) -> {
+ throw new QueryProcessException(
+ DataNodeQueryMessages.UNSUPPORTED_DATA_TYPE_2 +
type.getTypeEnum());
+ };
+ };
+
+ public static final TypeService<ColumnValueWriter>
TRANSFORM_COLUMN_VALUE_WRITER_SERVICE =
+ type ->
+ switch (type.getTypeEnum()) {
+ case BOOLEAN,
+ INT32,
+ DATE,
+ INT64,
+ TIMESTAMP,
+ FLOAT,
+ DOUBLE,
+ TEXT,
+ BLOB,
+ STRING,
+ OBJECT ->
+ type::write;
+ case ROW, UNKNOWN, VECTOR ->
+ (builder, column, index) -> {
+ throw new UnSupportedDataTypeException(
+ String.format(
+ DataNodeQueryMessages.UNSUPPORTED_DATA_TYPE_FMT,
type.getTypeEnum()));
+ };
+ };
+
+ // Schema evolution keeps physically compatible columns, widens supported
values, and exposes
+ // incompatible historical values as nulls instead of interpreting their
raw storage.
+ public static final TypeService<AlteredDataTypeColumnTransformer>
+ ALTERED_DATA_TYPE_COLUMN_TRANSFORMER_SERVICE =
+ type ->
+ switch (type.getTypeEnum()) {
+ case BOOLEAN,
+ INT32,
+ INT64,
+ FLOAT,
+ DOUBLE,
+ TEXT,
+ STRING,
+ TIMESTAMP,
+ DATE,
+ BLOB -> {
+ final TSDataType targetType =
TSDataType.valueOf(type.getTypeEnum().name());
+ yield (source, positionCount) -> {
+ final TSDataType sourceType = source.getDataType();
+ // DATE shares INT32's physical column but was
historically incompatible with
+ // a schema change to INT32, while the other
shared-column pairs are reusable.
+ if (sourceType == targetType
+ || (targetType != TSDataType.INT32
+ && SchemaUtils.isUsingSameColumn(sourceType,
targetType))) {
+ return source;
+ }
+ return targetType.isCompatible(sourceType)
+ ? source.convertTo(targetType)
+ : type.createNullColumn(positionCount);
+ };
+ }
+ case OBJECT -> (source, positionCount) -> source;
+ case ROW, UNKNOWN, VECTOR -> (source, positionCount) -> null;
+ };
+
+ public static final TypeService<RoundTransformer>
ROUND_TRANSFORMER_SERVICE =
+ type ->
+ switch (type.getTypeEnum()) {
+ case INT32 -> RoundFunctionTransformer::transformInt;
+ case INT64 -> RoundFunctionTransformer::transformLong;
+ case FLOAT -> RoundFunctionTransformer::transformFloat;
+ case DOUBLE -> RoundFunctionTransformer::transformDouble;
+ case BOOLEAN, DATE, TIMESTAMP, TEXT, BLOB, STRING, OBJECT, ROW,
UNKNOWN, VECTOR ->
+ (transformer, columns, builder) -> {
+ throw new UnsupportedOperationException(
+ String.format("Unsupported source dataType: %s",
type.getTypeEnum()));
+ };
+ };
+
+ public static final TypeService<DiffTransformer> DIFF_TRANSFORMER_SERVICE =
+ type ->
+ switch (type.getTypeEnum()) {
+ case INT32 -> DiffFunctionTransformer::transformInt;
+ case INT64 -> DiffFunctionTransformer::transformLong;
+ case FLOAT -> DiffFunctionTransformer::transformFloat;
+ case DOUBLE -> DiffFunctionTransformer::transformDouble;
+ case BOOLEAN, DATE, TIMESTAMP, TEXT, BLOB, STRING, OBJECT, ROW,
UNKNOWN, VECTOR ->
+ (transformer, columns, builder) -> {
+ throw new QueryProcessException(
+ DataNodeQueryMessages.UNSUPPORTED_DATA_TYPE_2 +
type.getTypeEnum());
+ };
+ };
+
+ public static final TypeService<NegationTransformer>
NEGATION_TRANSFORMER_SERVICE =
+ type ->
+ switch (type.getTypeEnum()) {
+ case INT32 -> ArithmeticNegationTransformer::transformInt;
+ case INT64 -> ArithmeticNegationTransformer::transformLong;
+ case FLOAT -> ArithmeticNegationTransformer::transformFloat;
+ case DOUBLE -> ArithmeticNegationTransformer::transformDouble;
+ case BOOLEAN, DATE, TIMESTAMP, TEXT, BLOB, STRING, OBJECT, ROW,
UNKNOWN, VECTOR ->
+ (transformer, columns, builder) -> {
+ throw new QueryProcessException(
+ DataNodeQueryMessages.UNSUPPORTED_DATA_TYPE_2 +
type.getTypeEnum());
+ };
+ };
+
+ public static final TypeService<StateWindowSplitter>
STATE_WINDOW_SPLITTER_SERVICE =
+ type ->
+ switch (type.getTypeEnum()) {
+ case INT32 ->
+ (valueRecorder, delta, values, index) -> {
+ if (!valueRecorder.hasRecorded()) {
+ valueRecorder.recordInt(type.getInt(values, index - 1));
+ valueRecorder.setRecorded(true);
+ }
+ final boolean split =
+ Math.abs(type.getInt(values, index) -
valueRecorder.getInt()) > delta;
+ if (split) {
+ valueRecorder.recordInt(type.getInt(values, index));
+ }
+ return split;
+ };
+ case INT64 ->
+ (valueRecorder, delta, values, index) -> {
+ if (!valueRecorder.hasRecorded()) {
+ valueRecorder.recordLong(type.getLong(values, index -
1));
+ valueRecorder.setRecorded(true);
+ }
+ final boolean split =
+ Math.abs(type.getLong(values, index) -
valueRecorder.getLong()) > delta;
+ if (split) {
+ valueRecorder.recordLong(type.getLong(values, index));
+ }
+ return split;
+ };
+ case FLOAT ->
+ (valueRecorder, delta, values, index) -> {
+ if (!valueRecorder.hasRecorded()) {
+ valueRecorder.recordFloat(type.getFloat(values, index -
1));
+ valueRecorder.setRecorded(true);
+ }
+ final boolean split =
+ Math.abs(type.getFloat(values, index) -
valueRecorder.getFloat()) > delta;
+ if (split) {
+ valueRecorder.recordFloat(type.getFloat(values, index));
+ }
+ return split;
+ };
+ case DOUBLE ->
+ (valueRecorder, delta, values, index) -> {
+ if (!valueRecorder.hasRecorded()) {
+ valueRecorder.recordDouble(type.getDouble(values, index
- 1));
+ valueRecorder.setRecorded(true);
+ }
+ final boolean split =
+ Math.abs(type.getDouble(values, index) -
valueRecorder.getDouble()) > delta;
+ if (split) {
+ valueRecorder.recordDouble(type.getDouble(values,
index));
+ }
+ return split;
+ };
+ case BOOLEAN ->
+ (valueRecorder, delta, values, index) -> {
+ if (!valueRecorder.hasRecorded()) {
+ valueRecorder.recordBoolean(type.getBoolean(values,
index - 1));
+ valueRecorder.setRecorded(true);
+ }
+ final boolean split =
+ type.getBoolean(values, index) !=
valueRecorder.getBoolean();
+ if (split) {
+ valueRecorder.recordBoolean(type.getBoolean(values,
index));
+ }
+ return split;
+ };
+ case TEXT ->
+ (valueRecorder, delta, values, index) -> {
+ if (!valueRecorder.hasRecorded()) {
+ valueRecorder.recordString(type.getBinary(values, index
- 1).toString());
+ valueRecorder.setRecorded(true);
+ }
+ final String value = type.getBinary(values,
index).toString();
+ final boolean split =
!value.equals(valueRecorder.getString());
+ if (split) {
+ valueRecorder.recordString(value);
+ }
+ return split;
+ };
+ case DATE, TIMESTAMP, BLOB, STRING, OBJECT, ROW, UNKNOWN, VECTOR
->
+ (valueRecorder, delta, values, index) -> {
+ throw new UnsupportedOperationException(
+
DataNodeQueryMessages.INVALID_DATA_TYPE_FOR_STATE_WINDOW_STRATEGY);
+ };
+ };
+
+ public static final TypeService<InTransformerSetInitializer>
+ IN_TRANSFORMER_SET_INITIALIZER_SERVICE =
+ type ->
+ switch (type.getTypeEnum()) {
+ case INT32, DATE -> InTransformer::initIntSet;
+ case INT64, TIMESTAMP -> InTransformer::initLongSet;
+ case FLOAT -> InTransformer::initFloatSet;
+ case DOUBLE -> InTransformer::initDoubleSet;
+ case BOOLEAN -> InTransformer::initBooleanSet;
+ case TEXT, STRING -> InTransformer::initStringSet;
+ case BLOB, OBJECT, ROW, UNKNOWN, VECTOR ->
+ (transformer, values) -> {
+ throw new UnsupportedOperationException(
+ DataNodeQueryMessages.UNSUPPORTED_DATA_TYPE_3 +
type.getTypeEnum());
+ };
+ };
+
+ public static final TypeService<InTransformerColumnTransformer>
+ IN_TRANSFORMER_COLUMN_TRANSFORMER_SERVICE =
+ type ->
+ switch (type.getTypeEnum()) {
+ case INT32, DATE -> InTransformer::transformInt;
+ case INT64, TIMESTAMP -> InTransformer::transformLong;
+ case FLOAT -> InTransformer::transformFloat;
+ case DOUBLE -> InTransformer::transformDouble;
+ case BOOLEAN -> InTransformer::transformBoolean;
+ case TEXT, STRING -> InTransformer::transformBinary;
+ case BLOB, OBJECT, ROW, UNKNOWN, VECTOR ->
+ (transformer, columns, builder) -> {
+ throw new QueryProcessException(
+ DataNodeQueryMessages.UNSUPPORTED_DATA_TYPE_3 +
type.getTypeEnum());
+ };
+ };
+
+ public static final TypeService<TypeService<CastColumnStrategy>>
CAST_COLUMN_SERVICE =
+ type ->
+ switch (type.getTypeEnum()) {
+ case INT32 ->
+ targetType ->
+ switch (targetType.getTypeEnum()) {
+ case INT32 ->
+ castColumnStrategy(
+ targetType,
+ (column, index, builder) ->
+ targetType.writeInt(builder,
type.getInt(column, index)));
+ case INT64 ->
+ castColumnStrategy(
+ targetType,
+ (column, index, builder) ->
+ targetType.writeLong(builder,
type.getInt(column, index)));
+ case FLOAT ->
+ castColumnStrategy(
+ targetType,
+ (column, index, builder) ->
+ targetType.writeFloat(builder,
type.getInt(column, index)));
+ case DOUBLE ->
+ castColumnStrategy(
+ targetType,
+ (column, index, builder) ->
+ targetType.writeDouble(builder,
type.getInt(column, index)));
+ case BOOLEAN ->
+ castColumnStrategy(
+ targetType,
+ (column, index, builder) ->
+ targetType.writeBoolean(
+ builder, type.getInt(column, index) !=
0));
+ case TEXT ->
+ castColumnStrategy(
+ targetType,
+ (column, index, builder) ->
+ targetType.writeBinary(
+ builder,
+ BytesUtils.valueOf(
+ String.valueOf(type.getInt(column,
index)))));
+ case DATE, TIMESTAMP, STRING, BLOB, OBJECT, ROW,
UNKNOWN, VECTOR ->
+ unsupportedTargetCastStrategy(type);
+ };
+ case INT64 ->
+ targetType ->
+ switch (targetType.getTypeEnum()) {
+ case INT32 ->
+ castColumnStrategy(
+ targetType,
+ (column, index, builder) ->
+ targetType.writeInt(
+ builder,
+ CastFunctionUtils.castLongToInt(
+ type.getLong(column, index))));
+ case INT64 ->
+ castColumnStrategy(
+ targetType,
+ (column, index, builder) ->
+ targetType.writeLong(builder,
type.getLong(column, index)));
+ case FLOAT ->
+ castColumnStrategy(
+ targetType,
+ (column, index, builder) ->
+ targetType.writeFloat(builder,
type.getLong(column, index)));
+ case DOUBLE ->
+ castColumnStrategy(
+ targetType,
+ (column, index, builder) ->
+ targetType.writeDouble(builder,
type.getLong(column, index)));
+ case BOOLEAN ->
+ castColumnStrategy(
+ targetType,
+ (column, index, builder) ->
+ targetType.writeBoolean(
+ builder, type.getLong(column, index)
!= 0L));
+ case TEXT ->
+ castColumnStrategy(
+ targetType,
+ (column, index, builder) ->
+ targetType.writeBinary(
+ builder,
+ BytesUtils.valueOf(
+
String.valueOf(type.getLong(column, index)))));
+ case DATE, TIMESTAMP, STRING, BLOB, OBJECT, ROW,
UNKNOWN, VECTOR ->
+ unsupportedTargetCastStrategy(type);
+ };
+ case FLOAT ->
+ targetType ->
+ switch (targetType.getTypeEnum()) {
+ case INT32 ->
+ castColumnStrategy(
+ targetType,
+ (column, index, builder) ->
+ targetType.writeInt(
+ builder,
+ CastFunctionUtils.castFloatToInt(
+ type.getFloat(column, index))));
+ case INT64 ->
+ castColumnStrategy(
+ targetType,
+ (column, index, builder) ->
+ targetType.writeLong(
+ builder,
+ CastFunctionUtils.castFloatToLong(
+ type.getFloat(column, index))));
+ case FLOAT ->
+ castColumnStrategy(
+ targetType,
+ (column, index, builder) ->
+ targetType.writeFloat(builder,
type.getFloat(column, index)));
+ case DOUBLE ->
+ castColumnStrategy(
+ targetType,
+ (column, index, builder) ->
+ targetType.writeDouble(builder,
type.getFloat(column, index)));
+ case BOOLEAN ->
+ castColumnStrategy(
+ targetType,
+ (column, index, builder) ->
+ targetType.writeBoolean(
+ builder, type.getFloat(column, index)
!= 0.0f));
+ case TEXT ->
+ castColumnStrategy(
+ targetType,
+ (column, index, builder) ->
+ targetType.writeBinary(
+ builder,
+ BytesUtils.valueOf(
+
String.valueOf(type.getFloat(column, index)))));
+ case DATE, TIMESTAMP, STRING, BLOB, OBJECT, ROW,
UNKNOWN, VECTOR ->
+ unsupportedTargetCastStrategy(type);
+ };
+ case DOUBLE ->
+ targetType ->
+ switch (targetType.getTypeEnum()) {
+ case INT32 ->
+ castColumnStrategy(
+ targetType,
+ (column, index, builder) ->
+ targetType.writeInt(
+ builder,
+ CastFunctionUtils.castDoubleToInt(
+ type.getDouble(column, index))));
+ case INT64 ->
+ castColumnStrategy(
+ targetType,
+ (column, index, builder) ->
+ targetType.writeLong(
+ builder,
+ CastFunctionUtils.castDoubleToLong(
+ type.getDouble(column, index))));
+ case FLOAT ->
+ castColumnStrategy(
+ targetType,
+ (column, index, builder) ->
+ targetType.writeFloat(
+ builder,
+ CastFunctionUtils.castDoubleToFloat(
+ type.getDouble(column, index))));
+ case DOUBLE ->
+ castColumnStrategy(
+ targetType,
+ (column, index, builder) ->
+ targetType.writeDouble(builder,
type.getDouble(column, index)));
+ case BOOLEAN ->
+ castColumnStrategy(
+ targetType,
+ (column, index, builder) ->
+ targetType.writeBoolean(
+ builder, type.getDouble(column, index)
!= 0.0));
+ case TEXT ->
+ castColumnStrategy(
+ targetType,
+ (column, index, builder) ->
+ targetType.writeBinary(
+ builder,
+ BytesUtils.valueOf(
+
String.valueOf(type.getDouble(column, index)))));
+ case DATE, TIMESTAMP, STRING, BLOB, OBJECT, ROW,
UNKNOWN, VECTOR ->
+ unsupportedTargetCastStrategy(type);
+ };
+ case BOOLEAN ->
+ targetType ->
+ switch (targetType.getTypeEnum()) {
+ case INT32 ->
+ castColumnStrategy(
+ targetType,
+ (column, index, builder) ->
+ targetType.writeInt(
+ builder, type.getBoolean(column,
index) ? 1 : 0));
+ case INT64 ->
+ castColumnStrategy(
+ targetType,
+ (column, index, builder) ->
+ targetType.writeLong(
+ builder, type.getBoolean(column,
index) ? 1L : 0L));
+ case FLOAT ->
+ castColumnStrategy(
+ targetType,
+ (column, index, builder) ->
+ targetType.writeFloat(
+ builder, type.getBoolean(column,
index) ? 1.0f : 0.0f));
+ case DOUBLE ->
+ castColumnStrategy(
+ targetType,
+ (column, index, builder) ->
+ targetType.writeDouble(
+ builder, type.getBoolean(column,
index) ? 1.0 : 0.0));
+ case BOOLEAN ->
+ castColumnStrategy(
+ targetType,
+ (column, index, builder) ->
+ targetType.writeBoolean(
+ builder, type.getBoolean(column,
index)));
+ case TEXT ->
+ castColumnStrategy(
+ targetType,
+ (column, index, builder) ->
+ targetType.writeBinary(
+ builder,
+ BytesUtils.valueOf(
+
String.valueOf(type.getBoolean(column, index)))));
+ case DATE, TIMESTAMP, STRING, BLOB, OBJECT, ROW,
UNKNOWN, VECTOR ->
+ unsupportedTargetCastStrategy(type);
+ };
+ case TEXT ->
+ targetType ->
+ switch (targetType.getTypeEnum()) {
+ case INT32 ->
+ castBinaryColumnStrategy(
+ type,
+ targetType,
+ (value, builder) ->
+ targetType.writeInt(builder,
Integer.parseInt(value)));
+ case INT64 ->
+ castBinaryColumnStrategy(
+ type,
+ targetType,
+ (value, builder) ->
+ targetType.writeLong(builder,
Long.parseLong(value)));
+ case FLOAT ->
+ castBinaryColumnStrategy(
+ type,
+ targetType,
+ (value, builder) ->
+ targetType.writeFloat(
+ builder,
CastFunctionUtils.castTextToFloat(value)));
+ case DOUBLE ->
+ castBinaryColumnStrategy(
+ type,
+ targetType,
+ (value, builder) ->
+ targetType.writeDouble(
+ builder,
CastFunctionUtils.castTextToDouble(value)));
+ case BOOLEAN ->
+ castBinaryColumnStrategy(
+ type,
+ targetType,
+ (value, builder) ->
+ targetType.writeBoolean(
+ builder,
CastFunctionUtils.castTextToBoolean(value)));
+ case TEXT ->
+ castColumnStrategy(
+ targetType,
+ (column, index, builder) ->
+ targetType.writeBinary(builder,
type.getBinary(column, index)));
+ case DATE, TIMESTAMP, STRING, BLOB, OBJECT, ROW,
UNKNOWN, VECTOR ->
+ unsupportedTargetCastStrategy(type);
+ };
+ case DATE, TIMESTAMP, STRING, BLOB, OBJECT, ROW, UNKNOWN, VECTOR
->
+ targetType -> unsupportedSourceCastStrategy(type);
+ };
+
+ private static CastColumnStrategy castColumnStrategy(
+ final Type targetType, final CastColumnWriter writer) {
+ return new CastColumnStrategy(() -> {}, targetType::createColumnBuilder,
writer);
+ }
+
+ private static CastColumnStrategy castBinaryColumnStrategy(
+ final Type sourceType, final Type targetType, final
BinaryStringCastWriter writer) {
+ return castColumnStrategy(
+ targetType,
+ (column, index, builder) ->
+ writer.write(
+ sourceType.getBinary(column,
index).getStringValue(TSFileConfig.STRING_CHARSET),
+ builder));
+ }
+
+ private static CastColumnStrategy unsupportedSourceCastStrategy(final Type
sourceType) {
+ return unsupportedCastStrategy(
+ String.format("Unsupported source dataType: %s",
sourceType.getTypeEnum()));
+ }
+
+ private static CastColumnStrategy unsupportedTargetCastStrategy(final Type
sourceType) {
+ return unsupportedCastStrategy(
+ String.format("Unsupported target dataType: %s",
sourceType.getTypeEnum()));
+ }
+
+ private static CastColumnStrategy unsupportedCastStrategy(final String
message) {
+ return new CastColumnStrategy(
+ () -> {
+ throw new UnsupportedOperationException(message);
+ },
+ ignored -> {
+ throw new UnsupportedOperationException(message);
+ },
+ (column, index, builder) -> {
+ throw new UnsupportedOperationException(message);
+ });
+ }
+
+ public static final class CastColumnStrategy {
+ private final Runnable validator;
+ private final IntFunction<ColumnBuilder> builderFactory;
+ private final CastColumnWriter writer;
+
+ private CastColumnStrategy(
+ final Runnable validator,
+ final IntFunction<ColumnBuilder> builderFactory,
+ final CastColumnWriter writer) {
+ this.validator = validator;
+ this.builderFactory = builderFactory;
+ this.writer = writer;
+ }
+
+ public void validate() {
+ validator.run();
+ }
+
+ public ColumnBuilder createBuilder(final int expectedEntries) {
+ return builderFactory.apply(expectedEntries);
+ }
+
+ public void cast(final Column column, final int index, final
ColumnBuilder builder) {
+ writer.write(column, index, builder);
+ }
+ }
+
+ @FunctionalInterface
+ private interface CastColumnWriter {
+ void write(Column column, int index, ColumnBuilder builder);
+ }
+
+ @FunctionalInterface
+ private interface BinaryStringCastWriter {
+ void write(String value, ColumnBuilder builder);
+ }
+
+ @FunctionalInterface
+ public interface ColumnToDoubleConverter {
+ double convert(Column column, int index) throws QueryProcessException;
+ }
+
+ @FunctionalInterface
+ public interface ColumnValueWriter {
+ void write(ColumnBuilder builder, Column column, int index);
+ }
+
+ @FunctionalInterface
+ public interface AlteredDataTypeColumnTransformer {
+ Column transform(Column source, int positionCount);
+ }
+
+ @FunctionalInterface
+ public interface RoundTransformer {
+ void transform(RoundFunctionTransformer transformer, Column[] columns,
ColumnBuilder builder)
+ throws QueryProcessException, IOException;
+ }
+
+ @FunctionalInterface
+ public interface DiffTransformer {
+ void transform(DiffFunctionTransformer transformer, Column[] columns,
ColumnBuilder builder)
+ throws QueryProcessException;
+ }
+
+ @FunctionalInterface
+ public interface NegationTransformer {
+ void transform(
+ ArithmeticNegationTransformer transformer, Column[] columns,
ColumnBuilder builder)
+ throws QueryProcessException, IOException;
+ }
+
+ @FunctionalInterface
+ public interface StateWindowSplitter {
+ boolean split(ValueRecorder valueRecorder, double delta, Column values,
int index);
+ }
+
+ @FunctionalInterface
+ public interface InTransformerSetInitializer {
+ void initialize(InTransformer transformer, Set<String> values);
+ }
+
+ @FunctionalInterface
+ public interface InTransformerColumnTransformer {
+ void transform(InTransformer transformer, Column[] columns,
ColumnBuilder builder)
+ throws QueryProcessException;
+ }
+
+ static {
+ CONSTANT_COLUMN_BUILDER_SERVICE.check();
+ CONSTANT_FILL_SERVICE.check();
+ COLUMN_BUILDER_SERVICE.check();
+ VALUE_TO_DOUBLE_SERVICE.check();
+ TRANSFORM_COLUMN_VALUE_WRITER_SERVICE.check();
+ ALTERED_DATA_TYPE_COLUMN_TRANSFORMER_SERVICE.check();
+ ROUND_TRANSFORMER_SERVICE.check();
+ DIFF_TRANSFORMER_SERVICE.check();
+ NEGATION_TRANSFORMER_SERVICE.check();
+ STATE_WINDOW_SPLITTER_SERVICE.check();
+ IN_TRANSFORMER_SET_INITIALIZER_SERVICE.check();
+ IN_TRANSFORMER_COLUMN_TRANSFORMER_SERVICE.check();
+ CAST_COLUMN_SERVICE.check();
+ }
+
+ private Transformation() {
+ // Utility class
+ }
+ }
+
+ public static final class Aggregation {
+
+ public static final TypeService<ExtremeValueAccumulatorStrategy>
+ EXTREME_VALUE_ACCUMULATOR_STRATEGY_SERVICE =
+ type ->
+ switch (type.getTypeEnum()) {
+ case INT32, DATE ->
+ extremeValueAccumulatorStrategy(
+ (accumulator, column, index) ->
+
accumulator.updateIntResult(column.getInt(index)),
+ (accumulator, value) ->
+ accumulator.updateIntResult(((Number)
value).intValue()),
+ (accumulator, column) ->
accumulator.getResult().setInt(column.getInt(0)),
+ (builder, result) ->
builder.writeInt(result.getInt()));
+ case INT64, TIMESTAMP ->
+ extremeValueAccumulatorStrategy(
+ (accumulator, column, index) ->
+
accumulator.updateLongResult(column.getLong(index)),
+ (accumulator, value) ->
+ accumulator.updateLongResult(((Number)
value).longValue()),
+ (accumulator, column) ->
+
accumulator.getResult().setLong(column.getLong(0)),
+ (builder, result) ->
builder.writeLong(result.getLong()));
+ case FLOAT ->
+ extremeValueAccumulatorStrategy(
+ (accumulator, column, index) ->
+
accumulator.updateFloatResult(column.getFloat(index)),
+ (accumulator, value) ->
+ accumulator.updateFloatResult(((Number)
value).floatValue()),
+ (accumulator, column) ->
+
accumulator.getResult().setFloat(column.getFloat(0)),
+ (builder, result) ->
builder.writeFloat(result.getFloat()));
+ case DOUBLE ->
+ extremeValueAccumulatorStrategy(
+ (accumulator, column, index) ->
+
accumulator.updateDoubleResult(column.getDouble(index)),
+ (accumulator, value) ->
+ accumulator.updateDoubleResult(((Number)
value).doubleValue()),
+ (accumulator, column) ->
+
accumulator.getResult().setDouble(column.getDouble(0)),
+ (builder, result) ->
builder.writeDouble(result.getDouble()));
+ case STRING ->
+ extremeValueAccumulatorStrategy(
+ (accumulator, column, index) ->
+
accumulator.updateBinaryResult(column.getBinary(index)),
+ (accumulator, value) ->
accumulator.updateBinaryResult((Binary) value),
+ (accumulator, column) ->
+
accumulator.getResult().setBinary(column.getBinary(0)),
+ (builder, result) ->
builder.writeBinary(result.getBinary()));
+ case BOOLEAN, TEXT, BLOB, OBJECT, ROW, UNKNOWN, VECTOR ->
+ ExtremeValueAccumulatorStrategy.unsupported();
+ };
+
+ public static final TypeService<TimeValueAccumulatorStrategy>
+ TIME_VALUE_ACCUMULATOR_STRATEGY_SERVICE =
+ type ->
+ switch (type.getTypeEnum()) {
+ case INT32, DATE ->
+ timeValueAccumulatorStrategy(
+ (accumulator, column, index, time) ->
+
accumulator.updateIntResult(column.getInt(index), time),
+ (accumulator, statistics, first) ->
+ accumulator.updateIntResult(
+ ((Number) getStatisticsValue(statistics,
first)).intValue(),
+ getStatisticsTime(statistics, first)),
+ (accumulator, column) ->
+
accumulator.getTimeValueResult().setInt(column.getInt(0)),
+ (builder, result) ->
builder.writeInt(result.getInt()));
+ case INT64, TIMESTAMP ->
+ timeValueAccumulatorStrategy(
+ (accumulator, column, index, time) ->
+
accumulator.updateLongResult(column.getLong(index), time),
+ (accumulator, statistics, first) ->
+ accumulator.updateLongResult(
+ ((Number) getStatisticsValue(statistics,
first)).longValue(),
+ getStatisticsTime(statistics, first)),
+ (accumulator, column) ->
+
accumulator.getTimeValueResult().setLong(column.getLong(0)),
+ (builder, result) ->
builder.writeLong(result.getLong()));
+ case FLOAT ->
+ timeValueAccumulatorStrategy(
+ (accumulator, column, index, time) ->
+
accumulator.updateFloatResult(column.getFloat(index), time),
+ (accumulator, statistics, first) ->
+ accumulator.updateFloatResult(
+ ((Number) getStatisticsValue(statistics,
first)).floatValue(),
+ getStatisticsTime(statistics, first)),
+ (accumulator, column) ->
+
accumulator.getTimeValueResult().setFloat(column.getFloat(0)),
+ (builder, result) ->
builder.writeFloat(result.getFloat()));
+ case DOUBLE ->
+ timeValueAccumulatorStrategy(
+ (accumulator, column, index, time) ->
+
accumulator.updateDoubleResult(column.getDouble(index), time),
+ (accumulator, statistics, first) ->
+ accumulator.updateDoubleResult(
+ ((Number) getStatisticsValue(statistics,
first)).doubleValue(),
+ getStatisticsTime(statistics, first)),
+ (accumulator, column) ->
+
accumulator.getTimeValueResult().setDouble(column.getDouble(0)),
+ (builder, result) ->
builder.writeDouble(result.getDouble()));
+ case TEXT, BLOB, STRING, OBJECT ->
+ timeValueAccumulatorStrategy(
+ (accumulator, column, index, time) ->
+
accumulator.updateBinaryResult(column.getBinary(index), time),
+ (accumulator, statistics, first) ->
+ accumulator.updateBinaryResult(
+ getBinaryStatisticsValue(statistics, first),
+ getStatisticsTime(statistics, first)),
+ (accumulator, column) ->
+
accumulator.getTimeValueResult().setBinary(column.getBinary(0)),
+ (builder, result) ->
builder.writeBinary(result.getBinary()));
+ case BOOLEAN ->
+ timeValueAccumulatorStrategy(
+ (accumulator, column, index, time) ->
+
accumulator.updateBooleanResult(column.getBoolean(index), time),
+ (accumulator, statistics, first) ->
+ accumulator.updateBooleanResult(
+ (boolean) getStatisticsValue(statistics,
first),
+ getStatisticsTime(statistics, first)),
+ (accumulator, column) ->
+
accumulator.getTimeValueResult().setBoolean(column.getBoolean(0)),
+ (builder, result) ->
builder.writeBoolean(result.getBoolean()));
+ case ROW, UNKNOWN, VECTOR ->
TimeValueAccumulatorStrategy.unsupported();
+ };
+
+ public static final TypeService<MaxMinByAccumulatorStrategy>
+ MAX_MIN_BY_ACCUMULATOR_STRATEGY_SERVICE =
+ type ->
+ switch (type.getTypeEnum()) {
+ case INT32, DATE ->
+ maxMinByAccumulatorStrategy(
+ type,
+ (accumulator, time, yColumn, yIndex, xColumn,
xIndex) ->
+ accumulator.updateIntResult(
+ time, yColumn.getInt(yIndex), xColumn,
xIndex),
+ (accumulator, time, bytes, offset) ->
+ accumulator.updateIntResult(
+ time,
+ BytesUtils.bytesToInt(bytes, offset),
+ accumulator.readXFromBytesIntermediateInput(
+ bytes, offset + Integer.BYTES),
+ 0));
+ case INT64, TIMESTAMP ->
+ maxMinByAccumulatorStrategy(
+ type,
+ (accumulator, time, yColumn, yIndex, xColumn,
xIndex) ->
+ accumulator.updateLongResult(
+ time, yColumn.getLong(yIndex), xColumn,
xIndex),
+ (accumulator, time, bytes, offset) ->
+ accumulator.updateLongResult(
+ time,
+ BytesUtils.bytesToLongFromOffset(bytes,
Long.BYTES, offset),
+ accumulator.readXFromBytesIntermediateInput(
+ bytes, offset + Long.BYTES),
+ 0));
+ case FLOAT ->
+ maxMinByAccumulatorStrategy(
+ type,
+ (accumulator, time, yColumn, yIndex, xColumn,
xIndex) ->
+ accumulator.updateFloatResult(
+ time, yColumn.getFloat(yIndex), xColumn,
xIndex),
+ (accumulator, time, bytes, offset) ->
+ accumulator.updateFloatResult(
+ time,
+ BytesUtils.bytesToFloat(bytes, offset),
+ accumulator.readXFromBytesIntermediateInput(
+ bytes, offset + Float.BYTES),
+ 0));
+ case DOUBLE ->
+ maxMinByAccumulatorStrategy(
+ type,
+ (accumulator, time, yColumn, yIndex, xColumn,
xIndex) ->
+ accumulator.updateDoubleResult(
+ time, yColumn.getDouble(yIndex), xColumn,
xIndex),
+ (accumulator, time, bytes, offset) ->
+ accumulator.updateDoubleResult(
+ time,
+ BytesUtils.bytesToDouble(bytes, offset),
+ accumulator.readXFromBytesIntermediateInput(
+ bytes, offset + Double.BYTES),
+ 0));
+ case STRING ->
+ maxMinByAccumulatorStrategy(
+ type,
+ (accumulator, time, yColumn, yIndex, xColumn,
xIndex) ->
+ accumulator.updateBinaryResult(
+ time, yColumn.getBinary(yIndex), xColumn,
xIndex),
+ TypeServices.Aggregation::updateBinaryIntermediate);
+ case TEXT, BLOB, OBJECT -> maxMinByAccumulatorStrategy(type,
null, null);
+ case BOOLEAN -> maxMinByAccumulatorStrategy(type, null,
null);
+ case ROW, UNKNOWN, VECTOR ->
MaxMinByAccumulatorStrategy.unsupported(type);
+ };
+
+ public static final TypeService<EventWindowManagerProvider>
+ EVENT_WINDOW_MANAGER_PROVIDER_SERVICE =
+ type ->
+ switch (type.getTypeEnum()) {
+ case INT32 ->
+ eventWindowManagerProvider(
+ EqualIntWindowManager::new,
VariationIntWindowManager::new);
+ case INT64 ->
+ eventWindowManagerProvider(
+ EqualLongWindowManager::new,
VariationLongWindowManager::new);
+ case FLOAT ->
+ eventWindowManagerProvider(
+ EqualFloatWindowManager::new,
VariationFloatWindowManager::new);
+ case DOUBLE ->
+ eventWindowManagerProvider(
+ EqualDoubleWindowManager::new,
VariationDoubleWindowManager::new);
+ case TEXT ->
eventWindowManagerProvider(EqualBinaryWindowManager::new, null);
+ case BOOLEAN ->
eventWindowManagerProvider(EqualBooleanWindowManager::new, null);
+ case DATE, TIMESTAMP, BLOB, STRING, OBJECT, ROW, UNKNOWN,
VECTOR ->
+ EventWindowManagerProvider.unsupported();
+ };
+
+ private static EventWindowManagerProvider eventWindowManagerProvider(
+ final EventWindowManagerFactory equalFactory,
+ final EventWindowManagerFactory variationFactory) {
+ return new EventWindowManagerProvider(equalFactory, variationFactory);
+ }
+
+ private static void updateBinaryIntermediate(
+ final MaxMinByAccumulator accumulator,
+ final long time,
+ final byte[] bytes,
+ final int offset) {
+ final int length = BytesUtils.bytesToInt(bytes, offset);
+ final int valueOffset = offset + Integer.BYTES;
+ accumulator.updateBinaryResult(
+ time,
+ new Binary(BytesUtils.subBytes(bytes, valueOffset, length)),
+ accumulator.readXFromBytesIntermediateInput(bytes, valueOffset +
length),
+ 0);
+ }
+
+ private static MaxMinByAccumulatorStrategy maxMinByAccumulatorStrategy(
+ final Type type,
+ final MaxMinByYColumnUpdater yColumnUpdater,
+ final MaxMinByYBytesUpdater yBytesUpdater) {
+ return new MaxMinByAccumulatorStrategy(type, true, yColumnUpdater,
yBytesUpdater);
+ }
+
+ private static TimeValueAccumulatorStrategy timeValueAccumulatorStrategy(
+ final TimeValueColumnUpdater columnUpdater,
+ final TimeValueStatisticsUpdater statisticsUpdater,
+ final TimeValueFinalSetter finalSetter,
+ final BiConsumer<ColumnBuilder, TsPrimitiveType> resultWriter) {
+ return new TimeValueAccumulatorStrategy(
+ columnUpdater, statisticsUpdater, finalSetter, resultWriter);
+ }
+
+ private static Object getStatisticsValue(final Statistics<?> statistics,
final boolean first) {
+ return first ? statistics.getFirstValue() : statistics.getLastValue();
+ }
+
+ private static long getStatisticsTime(final Statistics<?> statistics,
final boolean first) {
+ return first ? statistics.getStartTime() : statistics.getEndTime();
+ }
+
+ private static Binary getBinaryStatisticsValue(
+ final Statistics<?> statistics, final boolean first) {
+ final Object value = getStatisticsValue(statistics, first);
+ if (statistics instanceof DateStatistics) {
+ return new Binary(TSDataType.getDateStringValue((Integer) value),
StandardCharsets.UTF_8);
+ }
+ return value instanceof Binary
+ ? (Binary) value
+ : new Binary(String.valueOf(value), StandardCharsets.UTF_8);
+ }
+
+ public static final class TimeValueAccumulatorStrategy {
+ private final TimeValueColumnUpdater columnUpdater;
+ private final TimeValueStatisticsUpdater statisticsUpdater;
+ private final TimeValueFinalSetter finalSetter;
+ private final BiConsumer<ColumnBuilder, TsPrimitiveType> resultWriter;
+
+ private TimeValueAccumulatorStrategy(
+ final TimeValueColumnUpdater columnUpdater,
+ final TimeValueStatisticsUpdater statisticsUpdater,
+ final TimeValueFinalSetter finalSetter,
+ final BiConsumer<ColumnBuilder, TsPrimitiveType> resultWriter) {
+ this.columnUpdater = columnUpdater;
+ this.statisticsUpdater = statisticsUpdater;
+ this.finalSetter = finalSetter;
+ this.resultWriter = resultWriter;
+ }
+
+ private static TimeValueAccumulatorStrategy unsupported() {
+ return new TimeValueAccumulatorStrategy(null, null, null, null);
+ }
+
+ public boolean isSupported() {
+ return columnUpdater != null;
+ }
+
+ public void addInput(
+ final TimeValueAccumulator accumulator,
+ final Column[] columns,
+ final BitMap bitMap,
+ final boolean stopAfterFirstValue,
+ final boolean bitmapMarksSelected) {
+ final int count = columns[0].getPositionCount();
+ for (int i = 0; i < count; i++) {
+ if (bitMap != null && bitmapMarksSelected != bitMap.isMarked(i)) {
+ continue;
+ }
+ if (!columns[1].isNull(i)) {
+ columnUpdater.update(accumulator, columns[1], i,
columns[0].getLong(i));
+ if (stopAfterFirstValue) {
+ return;
+ }
+ }
+ }
+ }
+
+ public void addIntermediate(
+ final TimeValueAccumulator accumulator,
+ final Column valueColumn,
+ final Column timeColumn) {
+ columnUpdater.update(accumulator, valueColumn, 0,
timeColumn.getLong(0));
+ }
+
+ public void addStatistics(
+ final TimeValueAccumulator accumulator,
+ final Statistics<?> statistics,
+ final boolean first) {
+ statisticsUpdater.update(accumulator, statistics, first);
+ }
+
+ public void setFinal(final TimeValueAccumulator accumulator, final
Column column) {
+ finalSetter.set(accumulator, column);
+ }
+
+ public void writeResult(final ColumnBuilder builder, final
TsPrimitiveType result) {
+ resultWriter.accept(builder, result);
+ }
+ }
+
+ public static final class MaxMinByAccumulatorStrategy {
+ private final Type type;
+ private final boolean xSupported;
+ private final MaxMinByYColumnUpdater yColumnUpdater;
+ private final MaxMinByYBytesUpdater yBytesUpdater;
+
+ private MaxMinByAccumulatorStrategy(
+ final Type type,
+ final boolean xSupported,
+ final MaxMinByYColumnUpdater yColumnUpdater,
+ final MaxMinByYBytesUpdater yBytesUpdater) {
+ this.type = type;
+ this.xSupported = xSupported;
+ this.yColumnUpdater = yColumnUpdater;
+ this.yBytesUpdater = yBytesUpdater;
+ }
+
+ private static MaxMinByAccumulatorStrategy unsupported(final Type type) {
+ return new MaxMinByAccumulatorStrategy(type, false, null, null);
+ }
+
+ public boolean isXSupported() {
+ return xSupported;
+ }
+
+ public boolean isYSupported() {
+ return yColumnUpdater != null;
+ }
+
+ public void addInput(
+ final MaxMinByAccumulator accumulator, final Column[] columns, final
BitMap bitMap) {
+ final int count = columns[0].getPositionCount();
+ for (int i = 0; i < count; i++) {
+ if (bitMap != null && !bitMap.isMarked(i)) {
+ continue;
+ }
+ if (!columns[2].isNull(i)) {
+ yColumnUpdater.update(accumulator, columns[0].getLong(i),
columns[2], i, columns[1], i);
+ }
+ }
+ }
+
+ public void setXResult(final TsPrimitiveType result, final Column
column, final int index) {
+ type.setTo(result, column, index);
Review Comment:
`Type.setTo(TsPrimitiveType, Column, int)` copies the primitive into the
column; it does not copy the column value into the primitive. This call
therefore leaves `xResult` at its default value and can overwrite the input
column. CI reproduces this in `AccumulatorTest.maxByAccumulatorTest` (`-99`
expected, `0` returned). Please use the correct column-to-primitive copy
operation.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]