[
https://issues.apache.org/jira/browse/KAFKA-2066?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15104883#comment-15104883
]
David Jacot commented on KAFKA-2066:
------------------------------------
In the current implementation, FetchResponse contains a map of
FetchResponsePartitionData where each FetchResponsePartitionData contains a
MessageSet. Then, FetchResponseSend is used to directly transfers bytes of each
MessageSet to the channel. This is where the "zero-copy" happens.
Whereas, in the new FetchResponse, messages are represented as a ByteBuffer
which brings all bytes in the user space and therefore breaks the "zero-copy".
In order to keep it, I propose to use Records instead of a ByteBuffer in
FetchResponse and extend the serialization mechanism to support channels. I
need to work out the details but, roughly, It would require to:
1) update FetchResponse (and ProduceRequest to stay consistent);
2) introduce a new Type for Records;
3) add a new method in Type which work a the channel level with a default
implementation which uses the current way; and
4) add some methods in Records interface for the serialization (basically what
is in MessageSet)
4) make MessageSet inherit from Records to make them interoperable
(serialization wise).
[~ijuma], [~granthenke] What do you think?
> Replace FetchRequest / FetchResponse with their
> org.apache.kafka.common.requests equivalents
> --------------------------------------------------------------------------------------------
>
> Key: KAFKA-2066
> URL: https://issues.apache.org/jira/browse/KAFKA-2066
> Project: Kafka
> Issue Type: Sub-task
> Reporter: Gwen Shapira
> Assignee: David Jacot
>
> Replace FetchRequest / FetchResponse with their
> org.apache.kafka.common.requests equivalents.
> Note that they can't be completely removed until we deprecate the
> SimpleConsumer API (and it will require very careful patchwork for the places
> where core modules actually use the SimpleConsumer API).
> This also requires a solution on how to stream from memory-mapped files
> (similar to what existing code does with FileMessageSet.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)
