HTHou opened a new pull request, #18461:
URL: https://github.com/apache/iotdb/pull/18461
## Description
Backport the REST write request resource limits from #17481 to `dev/1.3`.
A crafted `insertTablet` or `insertRecords` request can contain an extremely
large JSON body or a very large number of rows, columns, and values. Before
this change, the REST layer deserialized and processed the payload without a
bounded request-body or aggregate memory budget, which could exhaust DataNode
heap under concurrent requests.
This change:
- rejects an oversized request body before JSON deserialization with HTTP
413;
- handles requests without a usable `Content-Length` by counting bytes while
the entity stream is read;
- limits total in-flight REST request-body bytes across concurrent requests
and returns HTTP 503 when the budget is exhausted;
- releases reservations when reading finishes, a request fails, or the
response completes;
- limits rows, columns, and values for REST v1/v2 write requests and returns
HTTP 413;
- supports hot reload for the REST runtime limits.
New configuration parameters:
- `rest_max_request_body_size_in_bytes=16777216`
- `rest_max_total_concurrent_request_body_size_in_bytes=0`
- `rest_max_insert_rows=100000`
- `rest_max_insert_columns=1024`
- `rest_max_insert_values=1000000`
For `rest_max_total_concurrent_request_body_size_in_bytes`, `0` uses half of
the free-memory share configured by `datanode_memory_proportion` (5% of maximum
heap with the default ratio), matching the budget used by the newer branch's
auto-resizing buffer memory control. Negative values disable the corresponding
limit.
## Compatibility
The `dev/1.3` REST implementation is part of the DataNode module and
supports tree-model REST v1/v2 only. This backport therefore adapts the fix to
`javax.ws.rs` and Java 8 and does not include the newer table-model REST
endpoints.
## Tests
```shell
mvn spotless:apply -pl iotdb-core/datanode
mvn test -pl iotdb-core/datanode \
-Dspotless.check.skip=true \
-Dtest='RequestSizeLimitFilterTest,RequestValidationLimitTest,IoTDBRestServiceDescriptorTest'
\
-DfailIfNoTests=false \
-Dsurefire.failIfNoSpecifiedTests=false
```
14 tests passed with no failures or errors.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
