[
https://issues.apache.org/jira/browse/KAFKA-7667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16695730#comment-16695730
]
ASF GitHub Bot commented on KAFKA-7667:
---------------------------------------
kaushiksrinivas opened a new pull request #5939: KAFKA-7667: Synchronous send
support for kafka performance producer java application
URL: https://github.com/apache/kafka/pull/5939
ProducerPerformance java application now supports synchronous blocking sends
to produce messages to kafka brokers.
Till now, performance producer application supported only asynchronous sends
with a callback function registered.
this would be enough when trying to load the kafka brokers without bothering
much on the replication & producer responses.
But when kafka brokers replication capacity has to be loaded, asynchronous
sends would not suffice.
So synchronous blocking send calls are now supported with all the existing
metrics calculation of producer performance.
A boolean command line flag --synchronous-send has been added to
enable/disable this feature.
Blocking .get() calls are now done upon each send, and as of this commit
ResultMetadata is not being used or logged.
A new function has been introduced in the Stats class to update and get the
iteration counter.
Tests done:
Comparison tests were done with and without synchronous sends.
below were the configurations of kafka cluster:
No of brokers: 3
CPU per broker: 10 cores
Memory : 32GB
Topic partitions : 1
replication factor : 3
min.insync.replicas: 1 (default).
producer acks level : 1(default), response after only leader write.
Similar configurations were given for performance producer with & without
synchronous sends enabled.
It was observed that, synchronous producer was producing records
approximately 5 times slower than asynchronous producer
but with guarantee of replication before subsequent sends.
So this clearly indicates the impact of replication in brokers. And these
tests would help optimise resources in
kafka broker from replication performance point of view.
Attached results snapshot of performance producer run with/without
synchronous send flag enabled.
----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
[email protected]
> Need synchronous records send support for kafka performance producer java
> application.
> --------------------------------------------------------------------------------------
>
> Key: KAFKA-7667
> URL: https://issues.apache.org/jira/browse/KAFKA-7667
> Project: Kafka
> Issue Type: Improvement
> Components: tools
> Affects Versions: 2.0.0
> Reporter: kaushik srinivas
> Assignee: kaushik srinivas
> Priority: Major
>
> Why synchronous send support for performance producer ?
> ProducerPerformance java application is used for load testing kafka brokers.
> Load testing involves replicating very high throughput records flowing in to
> kafka brokers and
> many producers in field would use synchronous way of sending data i.e
> blocking until the message has been
> written completely on all the min.insyc.replicas no of brokers.
> Asynchronous sends would satisfy the first requirement of loading kafka
> brokers.
> This requirement would help in performance tuning the kafka brokers when
> producers are deployed with "acks": all, "min.insync.replicas" :
> equal to replication factor and synchronous way of sending.
> Throughput degradation happens with synchronous producers and this would help
> in
> tuning resources for replication in kafka brokers.
> Also benchmarks could be made from kafka producer perspective with
> synchronous way of sending records and tune kafka producer's
> resources appropriately.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
