mumrah opened a new pull request #9008:
URL: https://github.com/apache/kafka/pull/9008
This change makes use of the generated protocols for FetchRequest and
FetchResponse. The main challenge here was how to allow the transferrable bytes
of the record set to be directly sent to the outgoing response without copying
into a buffer.
The proposed solution is similar to the existing multi-send object used in
[FetchResponse](TODO). However, a new writer class [RecordsWriter](TODO) was
introduced to allow interleaving of ByteBufferSend (for headers and other
non-record fields) along with RecordsSend-s which implement the efficient byte
transfer.
Another change introduced here is that FetchRequest and FetchResponse do not
maintain their own copies of the fields from the message. Instead, they hold a
reference to the generated message class (FetchRequestData and
FetchResponseData). Read-only copies of different forms of the message data are
created once open construction to allow for efficient access using the existing
class methods.
For example, in FetchRequest we hold the FetchRequestData, but also compute
and hold:
```java
private final FetchRequestData fetchRequestData;
// These are immutable read-only structures derived from FetchRequestData
private final Map<TopicPartition, PartitionData> fetchData;
private final List<TopicPartition> toForget;
private final FetchMetadata metadata;
```
And in FetchResponse, we similarly hold:
```java
private final FetchResponseData fetchResponseData;
private final LinkedHashMap<TopicPartition, PartitionData<T>>
responseDataMap;
```
If we want, we could deprecate all the accessors on
FetchRequest/FetchResponse and force callers to use the `#data()` method. This
would eliminate the need for these additional data structures.
Finally, most of the other changes are fixing up tests that were actually
using invalid default values for protocol messages (which are now enforced,
thanks to the generated classes) as well as rectifying the JSON schema to match
what the actual defined `Schema`s were (e.g., FETCH_RESPONSE_V11)
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
us...@infra.apache.org
