[
https://issues.apache.org/jira/browse/KAFKA-736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Neha Narkhede updated KAFKA-736:
--------------------------------
Attachment: kafka-736-v1.patch
Went with the simpler approach of resetting the interest ops on the producer
socket key to ~(READ & WRITE) right after the read is completed. Changes
include -
1. SyncProducer
- If required.request.acks=0, don't read response
- Change the doSend() API to take in requestAcks. This is used by the topic
metadata send as well as the produce request send. Forced the num acks for
topic metadata to be 1 since no other value makes sense
2. SocketServer
- Added some tracing mainly to make it easier to trace what the socket server
is doing
- Reset the interest ops on the selection key to not do writes or reads after a
read is complete. This prevents the server from reading more than one request
from the same producer
- In processNewResponses(), if the send buffer in the producer response is
null, turn the read interest bit on, since this is a fake producer response for
a produce request that had required.request.acks=0. For other produce requests,
it will turn on the write interest bit like it does today. The reason I put the
change here vs in the write() API is because we don't want to waste any time
setting the READ interest bit to reduce the delay in reading the next pipelined
produce request
3. KafkaApis
- Sends a fake producer response while handling a producer request with
required.request.acks=0
4. Unit testing
The reason I changed the required.request.acks=1 in most unit tests is to avoid
having unit tests that are timing dependent. We went through the exercise of
cleaning up our unit test suite to not include time dependent unit tests in the
past, so I haven't included any more in this patch. We should add system tests
though and I will file a JIRA to get that fixed. Let me know if you think we
should still add unit tests for required.request.acks=0
> Add an option to the 0.8 producer to mimic 0.7 producer behavior
> ----------------------------------------------------------------
>
> Key: KAFKA-736
> URL: https://issues.apache.org/jira/browse/KAFKA-736
> Project: Kafka
> Issue Type: Improvement
> Components: producer
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Priority: Blocker
> Labels: p2, replication-performance
> Attachments: kafka-736-draft.patch, kafka-736-v1.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> I profiled a producer throughput benchmark between a producer and a remote
> broker. It turns out that the background send threads spends ~97% of its time
> waiting to read the acknowledgement from the broker.
> I propose we change the current behavior of request.required.acks=0 to mean
> no acknowledgement from the broker. This will mimic the 0.7 producer behavior
> and will enable tuning the producer for very high throughput.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
