ColinLeeo opened a new pull request, #18333:
URL: https://github.com/apache/iotdb/pull/18333
## Description
### Problem
A table-model UDF can produce a large result block while processing a
partition containing a large device. Previously, `TableFunctionOperator` could
return the generated `TsBlock` without enforcing the configured row-count and
serialized byte-size limits.
This could propagate an oversized block through the query pipeline and
eventually exceed an RPC frame limit, such as the 64 MB Thrift frame limit.
### Design
`TableFunctionOperator` now splits each final output block inside
`buildTsBlock()`, after pass-through columns have been appended. Checking the
final block is important because pass-through columns contribute to the
serialized result size.
For every output fragment, the operator:
1. Caps the candidate row count using `max_tsblock_line_number`.
2. Serializes the candidate region and compares its exact byte size with
`max_tsblock_size_in_bytes`.
3. Uses binary search to find the largest row prefix that fits when the
candidate is oversized.
4. Queues all fragments in their original order for subsequent `next()`
calls.
Binary search handles variable-width values such as `TEXT`, `STRING`, and
`BLOB` without relying on an average row-size estimate. The size check uses
`TsBlockSerde` because the serialized size is the relevant boundary for blocks
sent through the query pipeline.
The splitting logic is kept in `TableFunctionOperator` instead of inheriting
the retained-block logic from `AbstractOperator`. This avoids maintaining two
result-fragmentation paths and ensures that proper columns and pass-through
columns are always split together.
### Configuration and corner cases
This PR introduces no new configuration. It uses:
- `max_tsblock_size_in_bytes`
- `max_tsblock_line_number`
Rows are indivisible. If a single row is larger than
`max_tsblock_size_in_bytes`, it is returned alone so execution can continue.
Result ordering and empty-output behavior are unchanged.
### Tests
The unit test covers both output forms:
- proper columns only
- proper columns with pass-through columns
It verifies the row-count limit, exact serialized byte-size limit, total row
count, ordering, values, and pass-through alignment.
The integration test configures a 1 KB TsBlock limit and runs a table UDF
that expands four input rows into 256 variable-width output rows. It verifies:
- every expected `(time, repeat_index)` pair is returned
- no result row is duplicated
- payload values are complete
- pass-through columns remain aligned across fragments
The following targeted tests passed:
```shell
mvn test -pl iotdb-core/datanode -am \
-Dtest=TableFunctionOperatorTest \
-DfailIfNoTests=false \
-Dsurefire.failIfNoSpecifiedTests=false
mvn verify -Drat.skip=true -DskipUTs \
-Dit.test=IoTDBUserDefinedTableFunctionIT#testLargeResultIsSplitWithoutDataLoss
\
-DfailIfNoTests=false \
-Dfailsafe.failIfNoSpecifiedTests=false \
-pl integration-test -am \
-PTableSimpleIT -P with-integration-tests
```
<hr>
This PR has:
- [x] been self-reviewed.
- [x] added comments explaining non-obvious design decisions.
- [x] added or updated unit tests.
- [x] added an integration test.
- [x] been tested in a test IoTDB cluster.
<hr>
##### Key changed/added classes
- `TableFunctionOperator`
- Splits final output blocks by row count and exact serialized size.
- Applies splitting after pass-through columns are appended.
- `TableFunctionOperatorTest`
- Covers variable-width results with and without pass-through columns.
- `LargeResultTableFunction`
- Generates wide integration-test output.
- `IoTDBUserDefinedTableFunctionIT`
- Verifies end-to-end result completeness and pass-through alignment.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
