Re: [PR] IGNITE-28853 Optimize serialization and deserialization of CompessedMessage [ignite]
wernerdv merged PR #13329: URL: https://github.com/apache/ignite/pull/13329 -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] IGNITE-28853 Optimize serialization and deserialization of CompessedMessage [ignite]
ignitetcbot commented on PR #13329:
URL: https://github.com/apache/ignite/pull/13329#issuecomment-4977033057
### TCBot Test Analysis
* TeamCity: [--> Run :: All
Results](https://ci2.ignite.apache.org/viewLog.html?buildId=9209632&buildTypeId=IgniteTests24Java8_RunAll)
* Branch: `pull/13329/head`
* Base: `master`
* Tested commit:
[a2e63df](https://github.com/apache/ignite/pull/13329/commits/a2e63dff17db71aca26a7bf9cee602012cca7df2)
Possible Blockers (1)
* [Platform C++ CMake (Win x64 |
Release)](https://ci2.ignite.apache.org/viewLog.html?buildId=9210252): 0 tests
Failure on metric
* Failure on metric is a blocker. Base branch critical fail rate is 1,1%
New Tests (6)
* [Basic 1](https://ci2.ignite.apache.org/viewLog.html?buildId=9210598): 6
tests
* `IgniteBasicTestSuite`:
`CompressedMessageTest.testReadFailsOnTruncatedPayload` - PASSED
* `IgniteBasicTestSuite`:
`CompressedMessageTest.testUncompressFailsWithoutChunks` - PASSED
* `IgniteBasicTestSuite`:
`CompressedMessageTest.testUncompressFailsOnUnderstatedDataSize` - PASSED
* `IgniteBasicTestSuite`:
`CompressedMessageTest.testUncompressFailsOnUnderstatedDataSizeMultiChunk` -
PASSED
* `IgniteBasicTestSuite`:
`CompressedMessageTest.testReadFailsOnNegativeDataSize` - PASSED
* `IgniteBasicTestSuite`: `CompressedMessageTest.testReadFailsOnNullChunk`
- PASSED
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
Re: [PR] IGNITE-28853 Optimize serialization and deserialization of CompessedMessage [ignite]
alex-plekhanov commented on code in PR #13329:
URL: https://github.com/apache/ignite/pull/13329#discussion_r3572520464
##
modules/core/src/main/java/org/apache/ignite/internal/direct/DirectMessageReader.java:
##
@@ -503,24 +522,49 @@ private T
readCompressedMessageAndDeserialize(DirectByteBufferStream stream,
if (msg0.dataSize() == 0)
return null;
-byte[] uncompressed = msg0.uncompressed();
+// Reuse the temp reader across fields/messages instead of allocating
a fresh state stack each time.
+if (tmpReader == null)
+tmpReader = new DirectMessageReader(msgFactory, cacheObjProc);
+else
+tmpReader.reset();
-ByteBuffer tmpBuf = ByteBuffer.allocateDirect(uncompressed.length);
+tmpReader.setBuffer(ByteBuffer.wrap(msg0.uncompressed()));
-tmpBuf.put(uncompressed);
-tmpBuf.flip();
+T res;
-DirectMessageReader tmpReader = new DirectMessageReader(msgFactory,
cacheObjProc);
+boolean ok = false;
-tmpReader.setBuffer(tmpBuf);
+try {
+res = fun.apply(tmpReader);
-T res = fun.apply(tmpReader);
+// The payload buffer is always complete, so a partial read means
a corrupted stream, not a lack of data.
+if (!tmpReader.state.item().stream.lastFinished())
+throw new IgniteException("Failed to deserialize compressed
payload: uncompressed data ended " +
+"unexpectedly [dataSize=" + msg0.dataSize() + ']');
Review Comment:
Braces required here
##
modules/core/src/main/java/org/apache/ignite/internal/direct/DirectMessageReader.java:
##
@@ -469,12 +484,16 @@ public ByteBuffer getBuffer() {
@Override public void beforeNestedRead() {
state.forward();
-state.item().stream.setBuffer(buf);
+curStream = state.item().stream;
+
+curStream.setBuffer(buf);
}
/** {@inheritDoc} */
@Override public void afterNestedRead(boolean finished) {
state.backward(finished);
+
+curStream = state.item().stream;
Review Comment:
Do we need additional `curStream.setBuffer(buf);` call here?
Looks like currently buffer instance can't be changed during message
deserialization (`setBuffer` called with the same `buf` instance on each part
of message). At least for communication messages , but I don't analize other
code paths. But if we change buffer when deserializing nested message there can
be an issue when previous item holds reference to the old buffer. Maybe it's
safer to call `curStream.setBuffer(buf);` here too?
##
modules/core/src/main/java/org/apache/ignite/internal/managers/communication/CompressedMessage.java:
##
@@ -90,99 +86,106 @@ public byte[] uncompressed() {
/** @return Next chunk of data or null. */
public byte[] nextChunk() {
-return chunkedReader.nextChunk();
+return chunkIdx < chunks.size() ? chunks.get(chunkIdx++) : null;
}
-/**
- * @param buf Buffer.
- * @return Compressed data.
- */
-private byte[] compress(ByteBuffer buf) {
-byte[] data = new byte[dataSize];
+/** @param buf Buffer. */
+private void compress(ByteBuffer buf) {
+Deflater deflater = new Deflater(compressionLvl, true);
-buf.get(data);
+try {
+deflater.setInput(buf);
+deflater.finish();
-ByteArrayOutputStream baos = new ByteArrayOutputStream(data.length);
-Deflater deflater = new Deflater(compressionLvl, true);
+chunks = new ArrayList<>(dataSize / CHUNK_SIZE + 1);
-try (DeflaterOutputStream dos = new DeflaterOutputStream(baos,
deflater)) {
-dos.write(data);
-dos.finish();
-}
-catch (IOException ex) {
-throw new IgniteException(ex);
+// Incompressible data may expand: raw-deflate worst case below
CHUNK_SIZE is one stored block (~11 bytes overhead).
+byte[] chunk0 = new byte[dataSize >= CHUNK_SIZE ? CHUNK_SIZE :
dataSize + 16];
+
+int len = 0;
+
+while (!deflater.finished()) {
+len += deflater.deflate(chunk0, len, chunk0.length - len);
+
+if (len == chunk0.length && !deflater.finished()) {
+chunks.add(chunk0);
+
+chunk0 = new byte[CHUNK_SIZE];
+len = 0;
+}
+}
+
+if (len > 0)
+chunks.add(len == chunk0.length ? chunk0 :
Arrays.copyOf(chunk0, len));
}
finally {
deflater.end();
}
-
-return baos.toByteArray();
}
/** @return Uncompressed data. */
private byte[] uncompress() {
-if (tmpBuf == null)
-return null;
+if (chunks == null)
+throw new IgniteException("Compressed stream is truncated
Re: [PR] IGNITE-28853 Optimize serialization and deserialization of CompessedMessage [ignite]
anton-vinogradov commented on PR #13329: URL: https://github.com/apache/ignite/pull/13329#issuecomment-4895868777 Prepared ready-to-merge fixes for everything above: wernerdv/ignite#3 (targets this PR's branch, so merging it brings the fixes straight here). Beyond the six review items, self-reviewing the fixes surfaced two more issues that are also addressed there: * an exception thrown from *inside* the payload deserialization (not just a silent partial read) left the reused `tmpReader` with half-read state — the drop is now done in a `finally`; * a lying `dataSize` could still inflate the chunk-list capacity allocation in `readFrom()` (~1.7 MB of `Object[]` for a ~15-byte wire message) — the pre-size is now clamped. Verification: `CompressedMessageTest` 7/7 (5 new cases), `DirectMarshallingMessagesTest` green, `-Pcheckstyle` clean, benchmarks module compiles. Known accepted limitations are listed in the fix PR description. 🤖 Generated with [Claude Code](https://claude.com/claude-code) -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] IGNITE-28853 Optimize serialization and deserialization of CompessedMessage [ignite]
anton-vinogradov commented on PR #13329:
URL: https://github.com/apache/ignite/pull/13329#issuecomment-4895522810
Reviewed the whole change (the `CompressedMessage`/serializer part matches
my #13325, so the findings there carry over; the `DirectMessageReader` part —
`curStream` cache and `tmpReader` reuse — is new and I looked at it from
scratch). The `curStream` cache itself checks out: all three mutation points
(`setBuffer` / `beforeNestedRead` / `afterNestedRead`) keep it in sync, depth
only ever changes inside `try/finally` in `DirectByteBufferStream`, and every
production `reset()` is followed by `setBuffer` before the next read. The
benchmark compiles with `-Pbenchmarks` and runs. Found the following, roughly
by severity.
### 1. `tmpReader` reuse: one partial inner read poisons every subsequent
compressed field on the connection (reproduced)
`reset()` is a shallow reset: `DirectMessageState.reset()` only zeroes
`stack[0].state` and asserts `pos == 0`. It does **not** clear the per-stream
parse state (`msgTypeDone`, half-read `msg`, `readSize`/`readItems`, map
iteration, `tmpArr`) — those are only cleared when a `readMessage`/`readMap`
*finishes successfully* — and `backward(false)` intentionally preserves the
state of abandoned deeper items for resume.
The old code was immune by construction (`new DirectMessageReader(...)` per
compressed field). With reuse, this sequence breaks: a `CompressedMessage`
whose inflated payload is logically incomplete (incompatible peer, corruption,
understated `dataSize` — see #4) makes `fun.apply(tmpReader)` return with
`lastFinished == false` and **no exception**, so the session stays alive. The
next, fully valid compressed field on the same per-connection reader reuses the
poisoned `tmpReader`: `readMessage` sees the stale `msgTypeDone == true`, skips
the type header of the new payload and resumes the *old* half-read message at a
wrong offset → the valid field reads as `null` with `lastRead == false` forever
(outer parser waits for bytes that never come), or a silently corrupted message
is assembled.
I verified this with a quick repro against the branch classes: after one
truncated payload, a following valid field fails to deserialize; with a fresh
reader (old behavior) the same field reads fine.
The clean fix falls out of an invariant this method already has:
`tmpReader`'s buffer is always the *complete* payload, so a partial inner read
is never "need more bytes" — it is always corruption. Instead of propagating it
as `lastRead = false`:
```java
T res = fun.apply(tmpReader);
if (!tmpReader.state.item().stream.lastFinished()) {
tmpReader = null; // Don't reuse a reader with half-read state.
throw new IgniteException("Failed to deserialize compressed payload: " +
"uncompressed data ended unexpectedly [dataSize=" + msg0.dataSize()
+ ']');
}
lastRead = true;
```
This also turns the silent protocol desync from #4 into a localized,
diagnosable error, and dropping `tmpReader` matters because the session is not
guaranteed to die: the legacy selector loop (`IGNITE_NO_SELECTOR_OPTS=true`,
`GridNioServer.processSelectedKeys`) logs the exception and keeps the session —
only the optimized path closes it.
### 2. `tmpReader` retains the last uncompressed payload for the connection
lifetime
`tmpReader` is a field of the per-connection reader (session meta,
`GridDirectParser`), and after the read its `stack[0].stream` keeps referencing
the `ByteBuffer.wrap(uncompressed)`. For exchange traffic
(`GridDhtPartitionsFullMessage` on large topologies) that's megabytes per
connection held until the *next* compressed message on that connection or TCP
session close; on the coordinator — one payload per node connection. The old
per-call reader made it garbage immediately. One line after `fun.apply`:
```java
tmpReader.setBuffer(EMPTY_BUF); // static final ByteBuffer EMPTY_BUF =
ByteBuffer.wrap(new byte[0]);
```
### Carried over from #13325 (identical `CompressedMessage`/serializer code)
These three have no benign trigger — the sender computes `dataSize =
buf.remaining()` correctly by construction — but the reworked receive path made
the wire `dataSize` load-bearing, while master was insensitive to it (`tmpBuf`
grew from actually received chunks, `readAllBytes()` sized the output,
`dataSize` was assert-only). So on a corrupted stream the failure quality
degrades vs master, and the guards are cheap:
**3. `dataSize > 0` + immediate `finalChunk` → bare NPE.** `readFrom()`
legitimately returns `true` with `chunks == null` when the first `finalChunk`
boolean is `true` (state 1 exits before state 2 runs) — the new null-chunk
guard doesn't cover this path. `uncompress()` then returns `null` and the
caller hits `ByteBuffer.wrap(null)` → NPE in the NIO worker. Master threw
`IgniteException
Re: [PR] IGNITE-28853 Optimize serialization and deserialization of CompessedMessage [ignite]
wernerdv commented on PR #13329: URL: https://github.com/apache/ignite/pull/13329#issuecomment-4893207279 Results of the `JmhDirectMessageReaderBenchmark`: - master: ``` Benchmark (entries) Mode Cnt Score Error Units JmhDirectMessageReaderBenchmark.compressedMessage 30 thrpt5 4595,593 ± 7257,161 ops/s JmhDirectMessageReaderBenchmark.compressedMessage500 thrpt5 1582,391 ± 476,917 ops/s ``` - current branch: ``` Benchmark (entries) Mode Cnt Score Error Units JmhDirectMessageReaderBenchmark.compressedMessage 30 thrpt5 47589,313 ± 23902,679 ops/s JmhDirectMessageReaderBenchmark.compressedMessage500 thrpt5 3216,279 ± 586,795 ops/s ``` -- This is an automated message from the Apache Git Service. To respond to the message, please log on to GitHub and use the URL above to go to the specific comment. To unsubscribe, e-mail: [email protected] For queries about this service, please contact Infrastructure at: [email protected]
Re: [PR] IGNITE-28853 Optimize serialization and deserialization of CompessedMessage [ignite]
ignitetcbot commented on PR #13329:
URL: https://github.com/apache/ignite/pull/13329#issuecomment-4891387189
### TCBot Test Analysis
* TeamCity: [--> Run :: All
Results](https://ci2.ignite.apache.org/viewLog.html?buildId=9180826&buildTypeId=IgniteTests24Java8_RunAll)
* Branch: `pull/13329/head`
* Base: `master`
* Tested commit:
[fa51508](https://github.com/apache/ignite/pull/13329/commits/fa5150835ac99dcedae40f51d3348aa1a0804840)
Possible Blockers (0)
No blockers found.
New Tests (2)
* [Basic 1](https://ci2.ignite.apache.org/viewLog.html?buildId=9180681): 1
tests
* `IgniteBasicTestSuite`: `CompressedMessageTest.testReadFailsOnNullChunk`
- PASSED
* [Cache 13](https://ci2.ignite.apache.org/viewLog.html?buildId=9180690): 1
tests
* `IgniteCacheTestSuite13`:
`SystemViewCacheExpiryPolicyTest.testCacheViewExpiryPolicy[factory=javax.cache.configuration.FactoryBuilder$SingletonFactory@689faf79,
actual=SingletonFactory [expiryPlc=EternalExpiryPolicy [create=ETERNAL]]]` -
PASSED
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
