JackieTien97 commented on code in PR #19176:
URL: https://github.com/apache/iotdb/pull/19176#discussion_r4505875960
##########
iotdb-client/session/src/main/java/org/apache/iotdb/session/rpccompress/TabletEncoder.java:
##########
@@ -118,6 +128,78 @@ private void encodePlainBinary(Tablet tablet, int j,
ByteArrayOutputStream baos)
}
}
+ /**
+ * Writes a PLAIN column of a fixed-width type with the same bytes as {@code
PlainEncoder}
+ * (big-endian; a null repeats the last non-null value, 0 before the first
one), but through one
+ * buffer instead of one synchronized {@code write(int)} call per byte.
+ *
+ * @return false if the type has no fixed width here; the caller then uses
the generic encoder
+ */
+ private static boolean encodePlainFixedWidth(
+ Tablet tablet, int j, TSDataType dataType, PublicBAOS baos) {
+ int rows = tablet.getRowSize();
+ ByteBuffer plain;
+ switch (dataType) {
Review Comment:
This `switch` fails
`TSDataTypeSwitchArchitectureTest.repositoryProductionCodeMustUseTypeService`,
which only allows switching on `TSDataType` inside a `TypeService` lambda. It's
the failure in both `unit-test (17, ..., others)` jobs.
Could these column writers move into a new `TypeService` in
`SessionTypeServices`, exposed through `SessionUtils`? `SessionTypeServices` is
package-private in `session.util`, so `TabletEncoder` can't call it directly.
An `if` chain would get past the rule, but it would only sidestep what the rule
is for.
##########
iotdb-client/session/src/main/java/org/apache/iotdb/session/rpccompress/TabletEncoder.java:
##########
@@ -118,6 +128,78 @@ private void encodePlainBinary(Tablet tablet, int j,
ByteArrayOutputStream baos)
}
}
+ /**
+ * Writes a PLAIN column of a fixed-width type with the same bytes as {@code
PlainEncoder}
+ * (big-endian; a null repeats the last non-null value, 0 before the first
one), but through one
+ * buffer instead of one synchronized {@code write(int)} call per byte.
Review Comment:
`baos` here is a `PublicBAOS`, which overrides `write(int)` and
`write(byte[], int, int)` without `synchronized`; only the JDK parent class
synchronizes them. So the cost being avoided is one method call and capacity
check per byte, not a synchronized call. Could the comment and the PR
description say that?
The description also ties this to RPC compression being enabled, which
doesn't match where this path runs. With the default builder, tablets of 10 or
more rows use TS_2DIFF, GORILLA or RLE for these columns and the time column,
so they never get here. This path runs when compression is off, when a tablet
has fewer rows than `tabletCompressionMinRowSize` (the all-PLAIN branch at
`Session.java:3204`), or when PLAIN is configured explicitly.
##########
iotdb-client/session/src/main/java/org/apache/iotdb/session/rpccompress/TabletEncoder.java:
##########
@@ -118,6 +128,78 @@ private void encodePlainBinary(Tablet tablet, int j,
ByteArrayOutputStream baos)
}
}
+ /**
+ * Writes a PLAIN column of a fixed-width type with the same bytes as {@code
PlainEncoder}
+ * (big-endian; a null repeats the last non-null value, 0 before the first
one), but through one
+ * buffer instead of one synchronized {@code write(int)} call per byte.
+ *
+ * @return false if the type has no fixed width here; the caller then uses
the generic encoder
+ */
+ private static boolean encodePlainFixedWidth(
+ Tablet tablet, int j, TSDataType dataType, PublicBAOS baos) {
+ int rows = tablet.getRowSize();
+ ByteBuffer plain;
+ switch (dataType) {
+ case INT64:
+ case TIMESTAMP:
+ {
+ long[] values = (long[]) tablet.getValues()[j];
+ plain = ByteBuffer.allocate(rows * Long.BYTES);
Review Comment:
Optional, not blocking: each column allocates a temporary buffer and copies
it into a `PublicBAOS` that starts at 32 bytes and grows. In a rough timing,
one pre-sized buffer for the whole tablet took about 19-26% less time. That
variant skipped the bitmap work, so treat it as an upper bound.
Two cheap steps would be `new PublicBAOS(estimatedSize)` and building the
`Encoder` at line 93 only where it's used. For the time column,
`asLongBuffer().put()` showed no gain, so that one isn't worth doing.
##########
iotdb-client/session/src/test/java/org/apache/iotdb/session/rpccompress/TabletEncoderPlainTest.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.session.rpccompress;
+
+import org.apache.iotdb.session.util.SessionUtils;
+
+import org.apache.tsfile.encoding.encoder.Encoder;
+import org.apache.tsfile.encoding.encoder.PlainEncoder;
+import org.apache.tsfile.enums.TSDataType;
+import org.apache.tsfile.file.metadata.enums.CompressionType;
+import org.apache.tsfile.file.metadata.enums.TSEncoding;
+import org.apache.tsfile.utils.PublicBAOS;
+import org.apache.tsfile.write.record.Tablet;
+import org.apache.tsfile.write.schema.IMeasurementSchema;
+import org.apache.tsfile.write.schema.MeasurementSchema;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Random;
+
+/**
+ * The bulk PLAIN path of {@link TabletEncoder} must write exactly the bytes
the generic {@link
+ * PlainEncoder} writes value by value, including the "repeat the last
non-null value" rule.
+ */
+public class TabletEncoderPlainTest {
+
+ private static final TSDataType[] TYPES = {
+ TSDataType.INT64, TSDataType.TIMESTAMP, TSDataType.DOUBLE,
TSDataType.FLOAT, TSDataType.BOOLEAN
+ };
+
+ @Test
+ public void plainBulkPathMatchesPlainEncoder() {
+ Random random = new Random(42);
+ for (int iteration = 0; iteration < 100; iteration++) {
+ Tablet tablet = randomTablet(random, iteration);
+ TabletEncoder encoder =
+ new TabletEncoder(
+ CompressionType.UNCOMPRESSED,
+ Collections.nCopies(TYPES.length + 1, TSEncoding.PLAIN));
+
+ PublicBAOS expectedTime = new PublicBAOS();
+ Encoder timeEncoder = new PlainEncoder(TSDataType.INT64, 0);
+ for (int row = 0; row < tablet.getRowSize(); row++) {
+ timeEncoder.encode(tablet.getTimestamp(row), expectedTime);
+ }
+ Assert.assertArrayEquals(
+ "time column, iteration " + iteration,
+ Arrays.copyOf(expectedTime.getBuf(), expectedTime.size()),
+ bytes(encoder.encodeTime(tablet)));
+
+ PublicBAOS expectedValues = new PublicBAOS();
+ for (int column = 0; column < TYPES.length; column++) {
+ SessionUtils.encodeValue(
+ TYPES[column], tablet, column, new PlainEncoder(TYPES[column], 0),
expectedValues);
+ }
+ byte[] actualValues = bytes(encoder.encodeValues(tablet));
+ // encodeValues appends the bit maps after the value columns
+ Assert.assertArrayEquals(
+ "value columns, iteration " + iteration,
+ Arrays.copyOf(expectedValues.getBuf(), expectedValues.size()),
+ Arrays.copyOf(actualValues, expectedValues.size()));
+ }
+ }
+
+ private static Tablet randomTablet(Random random, int iteration) {
+ List<IMeasurementSchema> schemas = new ArrayList<>();
+ for (int column = 0; column < TYPES.length; column++) {
+ schemas.add(new MeasurementSchema("s" + column, TYPES[column]));
+ }
+ int rows = 1 + random.nextInt(300);
+ Tablet tablet = new Tablet("root.sg.d1", schemas, rows);
+ for (int row = 0; row < rows; row++) {
+ tablet.addTimestamp(row, random.nextLong());
+ for (int column = 0; column < TYPES.length; column++) {
+ if (random.nextInt(5) == 0) {
+ continue; // leave a null
+ }
+ switch (TYPES[column]) {
+ case INT64:
+ case TIMESTAMP:
+ tablet.addValue(row, column, random.nextLong());
+ break;
+ case DOUBLE:
+ tablet.addValue(row, column, iteration % 7 == 0 ? Double.NaN :
random.nextGaussian());
Review Comment:
`Double.NaN` is the canonical NaN, so this can't tell `doubleToLongBits`
from the raw bits, and FLOAT has no NaN at all. Replacing those calls with
`plain.putDouble(last)` and `plain.putFloat(last)` still passes this test, even
though non-canonical NaNs then produce different bytes.
Could the test also use values like
`Double.longBitsToDouble(0x7ff8000000000001L)` and
`Float.intBitsToFloat(0x7fc00001)`?
##########
iotdb-client/session/src/main/java/org/apache/iotdb/session/rpccompress/TabletEncoder.java:
##########
@@ -57,6 +57,16 @@ public TabletEncoder(CompressionType compressionType,
List<TSEncoding> encodingL
}
public ByteBuffer encodeTime(Tablet tablet) {
+ if (encodingList.get(0) == TSEncoding.PLAIN) {
+ // Same bytes as PlainEncoder (big-endian longs), written in one go
instead of byte by byte.
+ int rows = tablet.getRowSize();
+ ByteBuffer plain = ByteBuffer.allocate(rows * Long.BYTES);
+ for (int i = 0; i < rows; i++) {
+ plain.putLong(tablet.getTimestamp(i));
+ }
+ plain.flip();
+ return compressBuffer(plain);
Review Comment:
This does the same thing as `SessionUtils.getTimeBuffer(Tablet)`
(`SessionUtils.java:50`), and `SessionUtils` is already imported:
```suggestion
return compressBuffer(SessionUtils.getTimeBuffer(tablet));
```
##########
iotdb-client/session/src/main/java/org/apache/iotdb/session/rpccompress/TabletEncoder.java:
##########
@@ -118,6 +128,78 @@ private void encodePlainBinary(Tablet tablet, int j,
ByteArrayOutputStream baos)
}
}
+ /**
+ * Writes a PLAIN column of a fixed-width type with the same bytes as {@code
PlainEncoder}
+ * (big-endian; a null repeats the last non-null value, 0 before the first
one), but through one
+ * buffer instead of one synchronized {@code write(int)} call per byte.
+ *
+ * @return false if the type has no fixed width here; the caller then uses
the generic encoder
+ */
+ private static boolean encodePlainFixedWidth(
Review Comment:
These are four nearly identical loops, and they repeat the "a null repeats
the last value" rule from `SessionTypeServices.TABLET_VALUE_ENCODER_SERVICE`,
so the two have to be kept in sync by hand. Sonar also reports a cognitive
complexity of 24 here (`java:S3776`, limit 15).
The `boolean` return doesn't add much either. `default` is only reached by
unsupported types such as `ROW` and `VECTOR`, because `INT32`, `DATE` and the
binary types are handled earlier, and the generic encoder then just throws
`UnSupportedDataTypeException`. That also makes the `@return` text a little
misleading. Moving this into a `TypeService` for the `switch` below would be a
good time to clean it up.
##########
iotdb-client/session/src/test/java/org/apache/iotdb/session/rpccompress/TabletEncoderPlainTest.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.session.rpccompress;
+
+import org.apache.iotdb.session.util.SessionUtils;
+
+import org.apache.tsfile.encoding.encoder.Encoder;
+import org.apache.tsfile.encoding.encoder.PlainEncoder;
+import org.apache.tsfile.enums.TSDataType;
+import org.apache.tsfile.file.metadata.enums.CompressionType;
+import org.apache.tsfile.file.metadata.enums.TSEncoding;
+import org.apache.tsfile.utils.PublicBAOS;
+import org.apache.tsfile.write.record.Tablet;
+import org.apache.tsfile.write.schema.IMeasurementSchema;
+import org.apache.tsfile.write.schema.MeasurementSchema;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Random;
+
+/**
+ * The bulk PLAIN path of {@link TabletEncoder} must write exactly the bytes
the generic {@link
+ * PlainEncoder} writes value by value, including the "repeat the last
non-null value" rule.
+ */
+public class TabletEncoderPlainTest {
+
+ private static final TSDataType[] TYPES = {
+ TSDataType.INT64, TSDataType.TIMESTAMP, TSDataType.DOUBLE,
TSDataType.FLOAT, TSDataType.BOOLEAN
+ };
+
+ @Test
+ public void plainBulkPathMatchesPlainEncoder() {
+ Random random = new Random(42);
+ for (int iteration = 0; iteration < 100; iteration++) {
+ Tablet tablet = randomTablet(random, iteration);
+ TabletEncoder encoder =
+ new TabletEncoder(
+ CompressionType.UNCOMPRESSED,
+ Collections.nCopies(TYPES.length + 1, TSEncoding.PLAIN));
+
+ PublicBAOS expectedTime = new PublicBAOS();
+ Encoder timeEncoder = new PlainEncoder(TSDataType.INT64, 0);
+ for (int row = 0; row < tablet.getRowSize(); row++) {
+ timeEncoder.encode(tablet.getTimestamp(row), expectedTime);
+ }
+ Assert.assertArrayEquals(
+ "time column, iteration " + iteration,
+ Arrays.copyOf(expectedTime.getBuf(), expectedTime.size()),
+ bytes(encoder.encodeTime(tablet)));
+
+ PublicBAOS expectedValues = new PublicBAOS();
+ for (int column = 0; column < TYPES.length; column++) {
+ SessionUtils.encodeValue(
+ TYPES[column], tablet, column, new PlainEncoder(TYPES[column], 0),
expectedValues);
+ }
+ byte[] actualValues = bytes(encoder.encodeValues(tablet));
+ // encodeValues appends the bit maps after the value columns
+ Assert.assertArrayEquals(
+ "value columns, iteration " + iteration,
+ Arrays.copyOf(expectedValues.getBuf(), expectedValues.size()),
+ Arrays.copyOf(actualValues, expectedValues.size()));
Review Comment:
Truncating to the expected length means the total length and the bitmap
section are never checked. Writing one extra byte after the last column
(BOOLEAN) still passes, even though it shifts the whole bitmap section.
Could this also assert the total length and the bitmap bytes?
##########
iotdb-client/session/src/test/java/org/apache/iotdb/session/rpccompress/TabletEncoderPlainTest.java:
##########
@@ -0,0 +1,127 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing,
+ * software distributed under the License is distributed on an
+ * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ * KIND, either express or implied. See the License for the
+ * specific language governing permissions and limitations
+ * under the License.
+ */
+
+package org.apache.iotdb.session.rpccompress;
+
+import org.apache.iotdb.session.util.SessionUtils;
+
+import org.apache.tsfile.encoding.encoder.Encoder;
+import org.apache.tsfile.encoding.encoder.PlainEncoder;
+import org.apache.tsfile.enums.TSDataType;
+import org.apache.tsfile.file.metadata.enums.CompressionType;
+import org.apache.tsfile.file.metadata.enums.TSEncoding;
+import org.apache.tsfile.utils.PublicBAOS;
+import org.apache.tsfile.write.record.Tablet;
+import org.apache.tsfile.write.schema.IMeasurementSchema;
+import org.apache.tsfile.write.schema.MeasurementSchema;
+import org.junit.Assert;
+import org.junit.Test;
+
+import java.nio.ByteBuffer;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.List;
+import java.util.Random;
+
+/**
+ * The bulk PLAIN path of {@link TabletEncoder} must write exactly the bytes
the generic {@link
+ * PlainEncoder} writes value by value, including the "repeat the last
non-null value" rule.
+ */
+public class TabletEncoderPlainTest {
+
+ private static final TSDataType[] TYPES = {
+ TSDataType.INT64, TSDataType.TIMESTAMP, TSDataType.DOUBLE,
TSDataType.FLOAT, TSDataType.BOOLEAN
+ };
+
+ @Test
+ public void plainBulkPathMatchesPlainEncoder() {
+ Random random = new Random(42);
+ for (int iteration = 0; iteration < 100; iteration++) {
+ Tablet tablet = randomTablet(random, iteration);
+ TabletEncoder encoder =
+ new TabletEncoder(
+ CompressionType.UNCOMPRESSED,
Review Comment:
Only `UNCOMPRESSED` runs here, so the compressed path isn't covered.
Changing `return compressBuffer(plain);` in `encodeTime` to `return plain;`
still passes this test, although the server would then read the last 4 bytes as
the uncompressed length. `IoTDBSessionCompressedIT` doesn't cover it either: no
session there sets PLAIN for INT64, DOUBLE or FLOAT, and the time column takes
its encoding from the INT64 setting.
Could this loop over the codecs and decompress with `IUnCompressor` before
comparing? GZIP would need less random data for now because of the overflow
described in the review summary.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]