JackieTien97 commented on code in PR #19176:
URL: https://github.com/apache/iotdb/pull/19176#discussion_r4506041449
##########
iotdb-client/session/src/main/java/org/apache/iotdb/session/rpccompress/TabletEncoder.java:
##########
@@ -74,25 +74,17 @@ public ByteBuffer encodeTime(Tablet tablet) {
}
public ByteBuffer encodeValues(Tablet tablet) {
- PublicBAOS baos = new PublicBAOS();
+ // the PLAIN size of the tablet; other encodings need less
+ PublicBAOS baos = new
PublicBAOS(SessionUtils.getTotalValueOccupation(tablet));
Review Comment:
This sizes the stream for PLAIN whatever the column encodings are, which
causes three problems. The earlier suggestion to pre-size should have been
limited to PLAIN columns.
- **More allocation on the default path.** Non-PLAIN columns pay for the
full PLAIN size and don't need it. For regular INT64 series under TS_2DIFF,
each `encodeValues` call now allocates 7.65 MB instead of 0.52 MB at 100,000
rows × 10 columns, and 76.3 MB instead of 4.02 MB at 1,000,000 rows × 10
columns, with about 3-4% more time in a rough timing. Mixed sensor data stayed
about level.
- **NPE on a binary column with no value array.** `getTotalValueOccupation`
reads the binary values (`SessionTypeServices.java:107`). A table-model tablet
built with the raw constructor, whose TEXT, STRING or BLOB TAG column is
entirely null with a null value array, survives `filterNullColumns` and now
throws `NullPointerException`. It encoded fine before. The trigger is narrow:
tablets built with the regular constructor and `addValue` always have the array.
- **Overflow above 2 GB.** The size is an `int`, so 2^27 rows × 3 INT64
columns gives -1073741821 and `new PublicBAOS` throws
`IllegalArgumentException: Negative initial size`, even under TS_2DIFF. This is
rare, and only the sizing step was exercised, not a full run with real arrays
of that size.
Summing only the PLAIN columns into a `long`, without reading binary values,
should address all three, though this hasn't been tried. The stream still grows
when binary values need more, and pre-sizing can be skipped if the sum doesn't
fit in an `int`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]