ss77892 opened a new pull request, #11637:
URL: https://github.com/apache/ozone/pull/11637
## What changes were proposed in this pull request?
HDDS-16559. Datanode ReadBlock fails with RESOURCE_EXHAUSTED when the client
response size exceeds the gRPC message limit
For streaming reads (ReadBlock), the client sets the size of each response
in ReadBlockRequestProto.responseDataSize, using
ozone.client.stream.read.response-data-size. The datanode used this value as
is. It only replaced 0 with the 1 MiB default. Two problems follow:
1. Responses too large for gRPC. The client's gRPC inbound message limit is
OzoneConsts.OZONE_SCM_CHUNK_MAX_SIZE (32 MiB). If the configured size is near
or above that limit, each response (data, checksum list and envelope) is too
big, and the read fails with RESOURCE_EXHAUSTED. The client has no way to
recover from this.
2. No bound on the field. responseDataSize is a uint32, and the datanode
uses it to size its read buffer. A request can ask for any buffer size up to
Integer.MAX_VALUE. A value above Integer.MAX_VALUE arrives in Java as a
negative int.
This PR makes the datanode limit the response size in
KeyValueHandler.boundResponseDataSize:
- A value of zero or less falls back to the 1 MiB default.
- A positive value is capped at MAX_READ_BLOCK_RESPONSE_DATA_SIZE, which
is OZONE_SCM_CHUNK_MAX_SIZE - 1 MiB. The checksum list adds at most 32 bytes
per 8 KiB of data, so 1 MiB leaves enough room for it and the envelope.
If the client asks for too much, the datanode now sends smaller responses
and the read succeeds. There are no protocol or client changes. The client
already accepts responses shorter than the requested size (covered by
testReadBlockResponseShorterThanResponseDataSize).
## What is the link to the Apache JIRA
https://issues.apache.org/jira/browse/HDDS-16559
## How was this patch tested?
- New unit test TestKeyValueHandler#testReadBlockResponseDataSizeIsBounded.
It checks how boundResponseDataSize handles 0, negative values, a normal value,
the maximum and Integer.MAX_VALUE. It also runs ReadBlock end to end with
response sizes 0, -1 and Integer.MAX_VALUE and checks the data that comes back.
- New integration test
TestStreamRead#testResponseDataSizeAboveGrpcMessageLimit. It sets the client's
response size to 2 * OZONE_SCM_CHUNK_MAX_SIZE, stream-reads a 40 MB key, and
checks its MD5 against the block file on the datanode.
- Moved the block-file lookup in TestStreamRead into a getBlockFile helper
so both tests can use it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: [email protected]
For queries about this service, please contact Infrastructure at:
[email protected]
---------------------------------------------------------------------
To unsubscribe, e-mail: [email protected]
For additional commands, e-mail: [email protected]
