chrisdutz commented on code in PR #19176:
URL: https://github.com/apache/iotdb/pull/19176#discussion_r4506062420
##########
iotdb-client/session/src/main/java/org/apache/iotdb/session/rpccompress/TabletEncoder.java:
##########
@@ -74,25 +74,17 @@ public ByteBuffer encodeTime(Tablet tablet) {
}
public ByteBuffer encodeValues(Tablet tablet) {
- PublicBAOS baos = new PublicBAOS();
+ // the PLAIN size of the tablet; other encodings need less
+ PublicBAOS baos = new
PublicBAOS(SessionUtils.getTotalValueOccupation(tablet));
Review Comment:
Confirmed all three and fixed them in the next commit.
- Allocation: `ThreadMXBean.getThreadAllocatedBytes` around one
`encodeValues` call, 10 INT64 columns under TS_2DIFF, gave 0.52 / 4.02 MiB on
master and 7.65 / 76.31 MiB on the previous commit (100,000 / 1,000,000 rows),
the same as your numbers.
- NPE: a new test, `allNullBinaryColumnWithoutValueArray`, builds that
table-model tablet (all-null STRING TAG column, `null` value array). It fails
with the NPE on the previous commit and passes on master.
- Overflow: 2^27 × 3 × 8 = 3,221,225,472 wraps to -1,073,741,824, plus the
three bit map markers.
The stream is now pre-sized only for PLAIN columns, from a per-type row
width (a new `TABLET_PLAIN_ROW_SIZE_SERVICE`). Binary types count only their
length prefix, so no values are read. The size is summed as a `long` and only
used if it fits in an array. If there are no PLAIN columns, the default stream
is used as before. `getTotalValueOccupation` is private again.
A first version of the fix left out the bit maps, so an all-PLAIN tablet's
stream doubled for the last few bytes (30.5 instead of 15.3 MiB at 100,000
rows). The estimate now adds an upper bound for the bit maps. Measured after
the fix: TS_2DIFF 0.52 / 4.02 MiB (as master), all-PLAIN 15.4 / 153.8 MiB
(master: 16.0 / 256.0).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]