[
https://issues.apache.org/jira/browse/KAFKA-736?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]
Neha Narkhede updated KAFKA-736:
--------------------------------
Attachment: kafka-736-v2.patch
Thanks for the review, these are good points. I see your point about separating
kafka logic from the socket server, I guess writing some real code and having
it reviewed is the best way to drill it down in my head :-)
1. RequestChannel.getFakeProducerResponse is a bad name, I think. Since it is a
general API for getting an empty response, renamed it getEmptyResponse. You are
right, it has nothing to do with producer response. I'm not too sure about the
getShutdownReceive right now, will clean that in a follow up patch, right after
this.
2. SocketServer.processNewResponses()
Good point, changed the comment to reflect that
3. SocketServer.processNewResponses()
done
4. SyncProducer.send-
Good point, this will also avoid creating hordes of Option objects.
5. SyncProducer.doSend-
True, this is meant to be generic though currently it is a private api inside
SyncProducer. Changed it to take a boolean readResponse.
6. Sure. Included a util in TestUtils and changed tests to use that instead of
creating a properties object repeatedly
7. Added a unit test for testing ordering of pipelined requests in
SocketServerTest
8. Added a unit test to PrimitiveApiTest for testing pipelined producer
requests and ordering of the consumed data
More changes in the patch -
- Fixed a bug from v1 patch in socket server. Basically, the way I was
resetting the interest ops was illegal. Fixed that to only reset the READ
interest bit
I think I was right about the performance concern, but it is still way off what
I had imagined the degradation to be -
replication factor = 1
no compression
required.request.acks=0
producer-threads batch-size draft-patch v1/v2 patch
1 1 19.31 6.49
2 1 21.70 0.01
1 50 77.84 58.07
2 50 83.72 0.33
1 100 75.18 45.80
2 100 83.53 0.63
> Add an option to the 0.8 producer to mimic 0.7 producer behavior
> ----------------------------------------------------------------
>
> Key: KAFKA-736
> URL: https://issues.apache.org/jira/browse/KAFKA-736
> Project: Kafka
> Issue Type: Improvement
> Components: producer
> Affects Versions: 0.8
> Reporter: Neha Narkhede
> Assignee: Neha Narkhede
> Priority: Blocker
> Labels: p2, replication-performance
> Attachments: kafka-736-draft.patch, kafka-736-v1.patch,
> kafka-736-v2.patch
>
> Original Estimate: 24h
> Remaining Estimate: 24h
>
> I profiled a producer throughput benchmark between a producer and a remote
> broker. It turns out that the background send threads spends ~97% of its time
> waiting to read the acknowledgement from the broker.
> I propose we change the current behavior of request.required.acks=0 to mean
> no acknowledgement from the broker. This will mimic the 0.7 producer behavior
> and will enable tuning the producer for very high throughput.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
