jt2594838 opened a new pull request, #19148:
URL: https://github.com/apache/iotdb/pull/19148

   ## Description
   
   Large RPC requests can leave expanded transport arrays attached to a Session 
long after the request completes. Add an opt-in `thriftMaxReusableBufferSize` 
setting to `Session.Builder`, `TableSessionBuilder`, `SessionPool.Builder`, and 
`TableSessionPoolBuilder`. Buffer requests above the configured byte threshold 
use temporary arrays and reset their reusable storage to 
`thriftDefaultBufferSize`.
   
   The threshold applies to raw read/write and compression/decompression 
buffers, including newly created, redirected, reconnected, and replacement 
pooled connections and background node discovery. Existing constructors and 
builders retain their reuse policy through the `Integer.MAX_VALUE` default; `0` 
selects temporary storage for every nonempty buffer request. Incremental writes 
preserve their serialized prefix when crossing the threshold. Temporary arrays 
are released after flush or consumption, and existing frame-size checks remain 
in effect.
   
   ## Validation
   
   - 160 focused tests passed across RPC buffers, transport factories, Session 
recovery, and session pools. Cases cover threshold boundaries, incremental 
writes, temporary-buffer lifetimes, binary/compact protocols, old constructors, 
independent client settings, and pool/discovery propagation.
   - Spotless and `git diff --check` passed.
   - Full-reactor test compilation passed for both English and Chinese locales 
(50 modules each).
   
   ```sh
   mvn test -pl iotdb-client/session -am -Drat.skip=true \
     
-Dtest=AutoResizingBufferTest,TElasticFramedTransportTest,TCompressedElasticFramedTransportTest,DeepCopyRpcTransportFactoryTest,SessionSeedNodeFallbackTest,SessionPoolTest,SessionPoolExceptionTest
 \
     -Dsurefire.failIfNoSpecifiedTests=false
   mvn test-compile -DskipTests -Drat.skip=true
   mvn test-compile -P with-zh-locale -DskipTests -Drat.skip=true
   ```
   
   ## Large-request write experiment
   
   A local isolated 1CN1DN instance was tested with Snappy frames, 16 
concurrent Sessions, 10 rounds, and 256 timed writes per Session per round. 
Each request contained 32,768 rows and 32 DOUBLE columns (about 8.25 MiB before 
compression). Both modes completed 40,960 timed requests, and all 20 groups 
passed row-count validation.
   
   | Metric | Default reuse policy | 1 MiB threshold |
   | --- | ---: | ---: |
   | Median throughput | 27.85 M points/s | 27.48 M points/s |
   | Retained transport arrays across 16 Sessions | 346 MiB | 64 KiB |
   | Worker-thread allocation per request | 25.96 MB | 57.02 MB |
   | Client GC time across timed intervals | 3,218 ms | 6,671 ms |
   
   These results demonstrate the retained-memory/allocation tradeoff. Unrelated 
background Maven jobs and server memory backpressure overlapped the experiment, 
so the throughput difference should not be treated as an isolated 
client-performance effect. The benchmark used an existing server distribution 
and the modified client classes.
   
   <hr>
   
   This PR has:
   
   - [x] been self-reviewed.
   - [x] added API documentation for the configuration and compatibility 
behavior.
   - [x] added comments explaining temporary-array ownership and 
incremental-write behavior.
   - [x] added unit tests for transport behavior and configuration propagation.
   - [x] been tested against a local IoTDB instance with large concurrent 
writes.
   
   Key changed packages: `org.apache.iotdb.rpc`, `org.apache.iotdb.session`, 
and `org.apache.iotdb.session.pool`.
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: [email protected]

For queries about this service, please contact Infrastructure at:
[email protected]

Reply via email to