JackieTien97 opened a new pull request, #19269:
URL: https://github.com/apache/iotdb/pull/19269
## Description
Queries with very large time-partition requests can stay in front-end
analysis while holding reservations from the operator memory pool. The Thrift
buffer computes 1.5x growth using `int`; above roughly 1.33 GiB this overflows,
so each small field write can copy the entire buffer while increasing its
capacity by only a few bytes. The uncompressed frame limit was only checked at
`flush()`, after serialization had already completed.
This change:
- Computes buffer growth and write positions without integer overflow and
bounds them by the Java array limit.
- Rejects oversized uncompressed frames before allocating or copying the
next write, and closes the transport. Compressed transports retain their
existing limit on the compressed frame size.
- Checks the query deadline before enumerating time partitions and
periodically during enumeration, allowing the existing Coordinator cleanup to
release front-end reservations.
## Reproduction
On `dev/1.3` at `013622816a`, a real 1,500,000,000-byte buffer grew to
1,500,000,008, 1,500,000,016, and 1,500,000,024 bytes for three successive
8-byte requests. Each request copied the entire array. With the fix, the first
request grows the buffer once and the following requests reuse it.
Regression tests also fail on the base revision for delayed frame-size
rejection, write-position overflow, and missing partition-enumeration timeout
checks. The query regression uses the real SQL parser, Coordinator, Analyzer,
and generated ConfigNode Thrift serializer with test schema/partition fetchers
and an in-memory transport. It checks that three successive oversized partition
requests stop at the configured frame limit and fully restore operator-pool
memory. This is a controlled reproduction of the code path, not a replay of the
original production SQL or a live-cluster test.
## Validation
```sh
mvn -B clean test -pl iotdb-core/datanode -am \
-Dtest=AutoResizingBufferTest,TElasticFramedTransportTest,RpcUtilsTest,UrlUtilsTest,TSStatusCodeTest,QueryTimePartitionTest,QueryPartitionMemoryTest,AnalyzeTest,QueryExecutionRetryTest
\
-Dsurefire.failIfNoSpecifiedTests=false -DfailIfNoTests=false
```
53 distinct tests passed, including normal frame boundaries,
compressed-frame compatibility, buffer reuse/shrinking, SQL analysis, query
retry, and memory cleanup. Checkstyle and Spotless checks passed. The
large-array reproduction was run separately with a bounded 5 GiB JVM heap so
normal unit tests do not need multi-gigabyte allocations.
- [x] Self-reviewed.
- [x] Added regression tests.
- [x] Added comments explaining the overflow, early rejection, and timeout
behavior.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]