[
https://issues.apache.org/jira/browse/KAFKA-1532?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14056802#comment-14056802
]
Jay Kreps commented on KAFKA-1532:
----------------------------------
It's not unavoidable, but it is pretty hard. To make this really work we have
to be able to stream partial requests from the network layer, to the
api/processing layer, and to the log layer. This is even more complicated
because additional appends to the log have to be blocked until the message is
completely written.
> Move CRC32 to AFTER the payload
> -------------------------------
>
> Key: KAFKA-1532
> URL: https://issues.apache.org/jira/browse/KAFKA-1532
> Project: Kafka
> Issue Type: Improvement
> Components: core, producer
> Reporter: Julian Morrison
> Assignee: Jun Rao
> Priority: Minor
>
> To support streaming a message of known length but unknown content, take the
> CRC32 out of the message header and make it a message trailer. Then client
> libraries can calculate it after streaming the message to Kafka, without
> materializing the whole message in RAM.
--
This message was sent by Atlassian JIRA
(v6.2#6252)
