chrisdutz commented on code in PR #19176:
URL: https://github.com/apache/iotdb/pull/19176#discussion_r4505952616
##########
iotdb-client/session/src/main/java/org/apache/iotdb/session/rpccompress/TabletEncoder.java:
##########
@@ -118,6 +128,78 @@ private void encodePlainBinary(Tablet tablet, int j,
ByteArrayOutputStream baos)
}
}
+ /**
+ * Writes a PLAIN column of a fixed-width type with the same bytes as {@code
PlainEncoder}
+ * (big-endian; a null repeats the last non-null value, 0 before the first
one), but through one
+ * buffer instead of one synchronized {@code write(int)} call per byte.
+ *
+ * @return false if the type has no fixed width here; the caller then uses
the generic encoder
+ */
+ private static boolean encodePlainFixedWidth(
+ Tablet tablet, int j, TSDataType dataType, PublicBAOS baos) {
+ int rows = tablet.getRowSize();
+ ByteBuffer plain;
+ switch (dataType) {
Review Comment:
Done. The column writers are now `TABLET_PLAIN_VALUE_WRITER_SERVICE` in
`SessionTypeServices`, called through `SessionUtils.encodePlainValue`.
`TSDataTypeSwitchArchitectureTest` passes locally; it failed on the previous
commit.
##########
iotdb-client/session/src/main/java/org/apache/iotdb/session/rpccompress/TabletEncoder.java:
##########
@@ -118,6 +128,78 @@ private void encodePlainBinary(Tablet tablet, int j,
ByteArrayOutputStream baos)
}
}
+ /**
+ * Writes a PLAIN column of a fixed-width type with the same bytes as {@code
PlainEncoder}
+ * (big-endian; a null repeats the last non-null value, 0 before the first
one), but through one
+ * buffer instead of one synchronized {@code write(int)} call per byte.
Review Comment:
Right, `PublicBAOS` doesn't synchronize. The comment now says the cost is a
call and a capacity check per byte. The description now lists the cases where
this path runs: compression off, tablets below `tabletCompressionMinRowSize`,
or PLAIN configured.
##########
iotdb-client/session/src/main/java/org/apache/iotdb/session/rpccompress/TabletEncoder.java:
##########
@@ -57,6 +57,16 @@ public TabletEncoder(CompressionType compressionType,
List<TSEncoding> encodingL
}
public ByteBuffer encodeTime(Tablet tablet) {
+ if (encodingList.get(0) == TSEncoding.PLAIN) {
+ // Same bytes as PlainEncoder (big-endian longs), written in one go
instead of byte by byte.
+ int rows = tablet.getRowSize();
+ ByteBuffer plain = ByteBuffer.allocate(rows * Long.BYTES);
+ for (int i = 0; i < rows; i++) {
+ plain.putLong(tablet.getTimestamp(i));
+ }
+ plain.flip();
+ return compressBuffer(plain);
Review Comment:
Applied.
##########
iotdb-client/session/src/main/java/org/apache/iotdb/session/rpccompress/TabletEncoder.java:
##########
@@ -118,6 +128,78 @@ private void encodePlainBinary(Tablet tablet, int j,
ByteArrayOutputStream baos)
}
}
+ /**
+ * Writes a PLAIN column of a fixed-width type with the same bytes as {@code
PlainEncoder}
+ * (big-endian; a null repeats the last non-null value, 0 before the first
one), but through one
+ * buffer instead of one synchronized {@code write(int)} call per byte.
+ *
+ * @return false if the type has no fixed width here; the caller then uses
the generic encoder
+ */
+ private static boolean encodePlainFixedWidth(
Review Comment:
The boolean return and the `default` fallback are gone. The service handles
every type; ROW, UNKNOWN and VECTOR throw `UnSupportedDataTypeException` like
the generic encoder. INT32, DATE and the binary writers moved into the same
service, so the `if` chain in `encodeValues` is now a single PLAIN/non-PLAIN
branch.
The "repeat the last value" rule still exists in both services: one calls
`encoder.encode(...)` and the other `buffer.putX(...)` per row. Sharing it
would need either a boxed or functional per-value call, which would undo the
speedup, or a filled copy of each column. The two now sit next to each other in
one file, and the test compares them byte for byte, so a drift fails the test.
##########
iotdb-client/session/src/main/java/org/apache/iotdb/session/rpccompress/TabletEncoder.java:
##########
@@ -118,6 +128,78 @@ private void encodePlainBinary(Tablet tablet, int j,
ByteArrayOutputStream baos)
}
}
+ /**
+ * Writes a PLAIN column of a fixed-width type with the same bytes as {@code
PlainEncoder}
+ * (big-endian; a null repeats the last non-null value, 0 before the first
one), but through one
+ * buffer instead of one synchronized {@code write(int)} call per byte.
+ *
+ * @return false if the type has no fixed width here; the caller then uses
the generic encoder
+ */
+ private static boolean encodePlainFixedWidth(
+ Tablet tablet, int j, TSDataType dataType, PublicBAOS baos) {
+ int rows = tablet.getRowSize();
+ ByteBuffer plain;
+ switch (dataType) {
+ case INT64:
+ case TIMESTAMP:
+ {
+ long[] values = (long[]) tablet.getValues()[j];
+ plain = ByteBuffer.allocate(rows * Long.BYTES);
Review Comment:
I did both cheap steps. `PublicBAOS` is pre-sized with
`SessionUtils.getTotalValueOccupation(tablet)`, which is exactly the PLAIN
size, and the `Encoder` is built only for non-PLAIN columns. I left out the
single buffer for the whole tablet.
##########
iotdb-client/session/src/test/java/org/apache/iotdb/session/rpccompress/TabletEncoderPlainTest.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.session.rpccompress;
+
+import org.apache.iotdb.session.util.SessionUtils;
+
+import org.apache.tsfile.encoding.encoder.Encoder;
+import org.apache.tsfile.encoding.encoder.PlainEncoder;
+import org.apache.tsfile.enums.TSDataType;
+import org.apache.tsfile.file.metadata.enums.CompressionType;
+import org.apache.tsfile.file.metadata.enums.TSEncoding;
+import org.apache.tsfile.utils.PublicBAOS;
+import org.apache.tsfile.write.record.Tablet;
+import org.apache.tsfile.write.schema.IMeasurementSchema;
+import org.apache.tsfile.write.schema.MeasurementSchema;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Random;
+
+/**
+ * The bulk PLAIN path of {@link TabletEncoder} must write exactly the bytes
the generic {@link
+ * PlainEncoder} writes value by value, including the "repeat the last
non-null value" rule.
+ */
+public class TabletEncoderPlainTest {
+
+ private static final TSDataType[] TYPES = {
+ TSDataType.INT64, TSDataType.TIMESTAMP, TSDataType.DOUBLE,
TSDataType.FLOAT, TSDataType.BOOLEAN
+ };
+
+ @Test
+ public void plainBulkPathMatchesPlainEncoder() {
+ Random random = new Random(42);
+ for (int iteration = 0; iteration < 100; iteration++) {
+ Tablet tablet = randomTablet(random, iteration);
+ TabletEncoder encoder =
+ new TabletEncoder(
+ CompressionType.UNCOMPRESSED,
Review Comment:
The test now loops over every `CompressionType`, decompresses with
`IUnCompressor` and checks the trailing uncompressed size. Dropping
`compressBuffer` in `encodeTime` now fails it. GZIP gets compressible data.
##########
iotdb-client/session/src/test/java/org/apache/iotdb/session/rpccompress/TabletEncoderPlainTest.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.session.rpccompress;
+
+import org.apache.iotdb.session.util.SessionUtils;
+
+import org.apache.tsfile.encoding.encoder.Encoder;
+import org.apache.tsfile.encoding.encoder.PlainEncoder;
+import org.apache.tsfile.enums.TSDataType;
+import org.apache.tsfile.file.metadata.enums.CompressionType;
+import org.apache.tsfile.file.metadata.enums.TSEncoding;
+import org.apache.tsfile.utils.PublicBAOS;
+import org.apache.tsfile.write.record.Tablet;
+import org.apache.tsfile.write.schema.IMeasurementSchema;
+import org.apache.tsfile.write.schema.MeasurementSchema;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Random;
+
+/**
+ * The bulk PLAIN path of {@link TabletEncoder} must write exactly the bytes
the generic {@link
+ * PlainEncoder} writes value by value, including the "repeat the last
non-null value" rule.
+ */
+public class TabletEncoderPlainTest {
+
+ private static final TSDataType[] TYPES = {
+ TSDataType.INT64, TSDataType.TIMESTAMP, TSDataType.DOUBLE,
TSDataType.FLOAT, TSDataType.BOOLEAN
+ };
+
+ @Test
+ public void plainBulkPathMatchesPlainEncoder() {
+ Random random = new Random(42);
+ for (int iteration = 0; iteration < 100; iteration++) {
+ Tablet tablet = randomTablet(random, iteration);
+ TabletEncoder encoder =
+ new TabletEncoder(
+ CompressionType.UNCOMPRESSED,
+ Collections.nCopies(TYPES.length + 1, TSEncoding.PLAIN));
+
+ PublicBAOS expectedTime = new PublicBAOS();
+ Encoder timeEncoder = new PlainEncoder(TSDataType.INT64, 0);
+ for (int row = 0; row < tablet.getRowSize(); row++) {
+ timeEncoder.encode(tablet.getTimestamp(row), expectedTime);
+ }
+ Assert.assertArrayEquals(
+ "time column, iteration " + iteration,
+ Arrays.copyOf(expectedTime.getBuf(), expectedTime.size()),
+ bytes(encoder.encodeTime(tablet)));
+
+ PublicBAOS expectedValues = new PublicBAOS();
+ for (int column = 0; column < TYPES.length; column++) {
+ SessionUtils.encodeValue(
+ TYPES[column], tablet, column, new PlainEncoder(TYPES[column], 0),
expectedValues);
+ }
+ byte[] actualValues = bytes(encoder.encodeValues(tablet));
+ // encodeValues appends the bit maps after the value columns
+ Assert.assertArrayEquals(
+ "value columns, iteration " + iteration,
+ Arrays.copyOf(expectedValues.getBuf(), expectedValues.size()),
+ Arrays.copyOf(actualValues, expectedValues.size()));
+ }
+ }
+
+ private static Tablet randomTablet(Random random, int iteration) {
+ List<IMeasurementSchema> schemas = new ArrayList<>();
+ for (int column = 0; column < TYPES.length; column++) {
+ schemas.add(new MeasurementSchema("s" + column, TYPES[column]));
+ }
+ int rows = 1 + random.nextInt(300);
+ Tablet tablet = new Tablet("root.sg.d1", schemas, rows);
+ for (int row = 0; row < rows; row++) {
+ tablet.addTimestamp(row, random.nextLong());
+ for (int column = 0; column < TYPES.length; column++) {
+ if (random.nextInt(5) == 0) {
+ continue; // leave a null
+ }
+ switch (TYPES[column]) {
+ case INT64:
+ case TIMESTAMP:
+ tablet.addValue(row, column, random.nextLong());
+ break;
+ case DOUBLE:
+ tablet.addValue(row, column, iteration % 7 == 0 ? Double.NaN :
random.nextGaussian());
Review Comment:
Added `0x7ff8000000000001L` and `0x7fc00001` next to the canonical NaNs.
`putDouble`/`putFloat` now fail the test.
##########
iotdb-client/session/src/test/java/org/apache/iotdb/session/rpccompress/TabletEncoderPlainTest.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.session.rpccompress;
+
+import org.apache.iotdb.session.util.SessionUtils;
+
+import org.apache.tsfile.encoding.encoder.Encoder;
+import org.apache.tsfile.encoding.encoder.PlainEncoder;
+import org.apache.tsfile.enums.TSDataType;
+import org.apache.tsfile.file.metadata.enums.CompressionType;
+import org.apache.tsfile.file.metadata.enums.TSEncoding;
+import org.apache.tsfile.utils.PublicBAOS;
+import org.apache.tsfile.write.record.Tablet;
+import org.apache.tsfile.write.schema.IMeasurementSchema;
+import org.apache.tsfile.write.schema.MeasurementSchema;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Random;
+
+/**
+ * The bulk PLAIN path of {@link TabletEncoder} must write exactly the bytes
the generic {@link
+ * PlainEncoder} writes value by value, including the "repeat the last
non-null value" rule.
+ */
+public class TabletEncoderPlainTest {
+
+ private static final TSDataType[] TYPES = {
+ TSDataType.INT64, TSDataType.TIMESTAMP, TSDataType.DOUBLE,
TSDataType.FLOAT, TSDataType.BOOLEAN
+ };
+
+ @Test
+ public void plainBulkPathMatchesPlainEncoder() {
+ Random random = new Random(42);
+ for (int iteration = 0; iteration < 100; iteration++) {
+ Tablet tablet = randomTablet(random, iteration);
+ TabletEncoder encoder =
+ new TabletEncoder(
+ CompressionType.UNCOMPRESSED,
+ Collections.nCopies(TYPES.length + 1, TSEncoding.PLAIN));
+
+ PublicBAOS expectedTime = new PublicBAOS();
+ Encoder timeEncoder = new PlainEncoder(TSDataType.INT64, 0);
+ for (int row = 0; row < tablet.getRowSize(); row++) {
+ timeEncoder.encode(tablet.getTimestamp(row), expectedTime);
+ }
+ Assert.assertArrayEquals(
+ "time column, iteration " + iteration,
+ Arrays.copyOf(expectedTime.getBuf(), expectedTime.size()),
+ bytes(encoder.encodeTime(tablet)));
+
+ PublicBAOS expectedValues = new PublicBAOS();
+ for (int column = 0; column < TYPES.length; column++) {
+ SessionUtils.encodeValue(
+ TYPES[column], tablet, column, new PlainEncoder(TYPES[column], 0),
expectedValues);
+ }
+ byte[] actualValues = bytes(encoder.encodeValues(tablet));
+ // encodeValues appends the bit maps after the value columns
+ Assert.assertArrayEquals(
+ "value columns, iteration " + iteration,
+ Arrays.copyOf(expectedValues.getBuf(), expectedValues.size()),
+ Arrays.copyOf(actualValues, expectedValues.size()));
Review Comment:
The test now builds the whole expected output, including the bit maps, and
compares the full array. An extra byte now fails it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]