JackieTien97 opened a new pull request, #19179:
URL: https://github.com/apache/iotdb/pull/19179
## Description
### Problem
`TabletEncoder.compressBuffer` sized its output array as
`ICompressor#getMaxBytesForCompression(n) + 4`. tsfile only estimates that size
for GZIP (`max(40 + n / 2, n)`, which its Javadoc calls insecure) and LZMA2
(`100 + n`), so data that hardly compresses outgrows the array and the client
cannot encode the tablet:
| Codec | Data | Exception |
| --- | --- | --- |
| GZIP | 10 rows of noisy doubles, GORILLA | `GZIPCompressOverflowException`
|
| GZIP | 1000 rows of random INT64, PLAIN | `GZIPCompressOverflowException` |
| GZIP | 5 rows of random INT64, PLAIN | `IndexOutOfBoundsException` (the
output fits the array, but not together with the 4-byte length) |
| LZMA2 | 150000 rows of random INT64, PLAIN (1.2 MB) |
`ArrayIndexOutOfBoundsException` |
`IoTDBSessionCompressedIT` does not catch this because its 10-row values
compress well.
### Fix
GZIP and LZMA2 now use the array-returning `compress(byte[], int, int)`, so
they size their own output. SNAPPY, LZ4 and ZSTD have a real upper bound and
keep writing straight into one array, exactly as before.
The wire format does not change: the compressed bytes followed by the
uncompressed length as a 4-byte big-endian int, as `TabletDecoder#uncompress`
reads it.
### Alternative: one code path for every codec
Using the array-returning call everywhere would remove the branch, but it
costs the codecs that do not have this bug, and LZ4 is the default codec. In a
rough timing on a loaded machine, it added 2-6% to the SNAPPY and LZ4 compress
step on sensor-like data and about 180% on random data (about 70 us per 800
KB), while ZSTD changed by under 1%. It also needs two more arrays per call.
The absolute cost is small, so this is easy to simplify if reviewers prefer one
path.
### Testing
- New `TabletEncoderCompressionTest`: 6 of its 7 tests fail without the fix.
The seventh pins the exact bytes and passes both ways.
- All unit tests in `service-rpc` and `session` pass.
- Compared the bytes sent before and after the fix, in a one-off check that
is not committed, across 3432 encodings (every codec `ICompressor` supports, 15
data and encoding combinations, 0 to 150000 rows). Nothing that could be
encoded before changed, and the 187 encodings that threw before now succeed
(185 GZIP, 2 LZMA2).
- Fed the encoder output to the real `TabletDecoder` in-process, in another
one-off check: all 130 cases decode back to the original values.
Not tested against a real cluster, and `IoTDBSessionCompressedIT` was not
run locally.
This does not depend on #19176, which does not touch `compressBuffer`. With
both applied there is no conflict and the `TabletEncoder` tests pass.
<hr>
This PR has:
- [x] been self-reviewed.
- [x] added comments explaining the "why" and the intent of the code
wherever would not be obvious
for an unfamiliar reader.
- [x] added unit tests or modified existing tests to cover new code paths,
ensuring the threshold
for code coverage.
<hr>
##### Key changed/added classes (or packages if there are too many classes)
in this PR
- `TabletEncoder`
- `TabletEncoderCompressionTest`
🤖 Generated with [Claude Code](https://claude.com/claude-code)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]