jt2594838 opened a new pull request, #19148:
URL: https://github.com/apache/iotdb/pull/19148
## Description
Large RPC requests can leave expanded transport arrays attached to a Session
long after the request completes. Add an opt-in `thriftMaxReusableBufferSize`
setting to `Session.Builder`, `TableSessionBuilder`, `SessionPool.Builder`, and
`TableSessionPoolBuilder`. Buffer requests above the configured byte threshold
use temporary arrays and reset their reusable storage to
`thriftDefaultBufferSize`.
The threshold applies to raw read/write and compression/decompression
buffers, including newly created, redirected, reconnected, and replacement
pooled connections and background node discovery. Existing constructors and
builders retain their reuse policy through the `Integer.MAX_VALUE` default; `0`
selects temporary storage for every nonempty buffer request. Incremental writes
preserve their serialized prefix when crossing the threshold. Temporary arrays
are released after flush or consumption, and existing frame-size checks remain
in effect.
## Validation
- 160 focused tests passed across RPC buffers, transport factories, Session
recovery, and session pools. Cases cover threshold boundaries, incremental
writes, temporary-buffer lifetimes, binary/compact protocols, old constructors,
independent client settings, and pool/discovery propagation.
- Spotless and `git diff --check` passed.
- Full-reactor test compilation passed for both English and Chinese locales
(50 modules each).
```sh
mvn test -pl iotdb-client/session -am -Drat.skip=true \
-Dtest=AutoResizingBufferTest,TElasticFramedTransportTest,TCompressedElasticFramedTransportTest,DeepCopyRpcTransportFactoryTest,SessionSeedNodeFallbackTest,SessionPoolTest,SessionPoolExceptionTest
\
-Dsurefire.failIfNoSpecifiedTests=false
mvn test-compile -DskipTests -Drat.skip=true
mvn test-compile -P with-zh-locale -DskipTests -Drat.skip=true
```
## Large-request write experiment
A local isolated 1CN1DN instance was tested with Snappy frames, 16
concurrent Sessions, 10 rounds, and 256 timed writes per Session per round.
Each request contained 32,768 rows and 32 DOUBLE columns (about 8.25 MiB before
compression). Both modes completed 40,960 timed requests, and all 20 groups
passed row-count validation.
| Metric | Default reuse policy | 1 MiB threshold |
| --- | ---: | ---: |
| Median throughput | 27.85 M points/s | 27.48 M points/s |
| Retained transport arrays across 16 Sessions | 346 MiB | 64 KiB |
| Worker-thread allocation per request | 25.96 MB | 57.02 MB |
| Client GC time across timed intervals | 3,218 ms | 6,671 ms |
These results demonstrate the retained-memory/allocation tradeoff. Unrelated
background Maven jobs and server memory backpressure overlapped the experiment,
so the throughput difference should not be treated as an isolated
client-performance effect. The benchmark used an existing server distribution
and the modified client classes.
<hr>
This PR has:
- [x] been self-reviewed.
- [x] added API documentation for the configuration and compatibility
behavior.
- [x] added comments explaining temporary-array ownership and
incremental-write behavior.
- [x] added unit tests for transport behavior and configuration propagation.
- [x] been tested against a local IoTDB instance with large concurrent
writes.
Key changed packages: `org.apache.iotdb.rpc`, `org.apache.iotdb.session`,
and `org.apache.iotdb.session.pool`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]