[jira] [Commented] (KAFKA-4816) Message format changes for idempotent/transactional producer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15941022#comment-15941022
 ] 

ASF GitHub Bot commented on KAFKA-4816:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2614


> Message format changes for idempotent/transactional producer
> 
>
> Key: KAFKA-4816
> URL: https://issues.apache.org/jira/browse/KAFKA-4816
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients, core, producer 
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
> Fix For: 0.11.0.0
>
>
> This task is for the implementation of the message format changes documented 
> here: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging#KIP-98-ExactlyOnceDeliveryandTransactionalMessaging-MessageFormat.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4816) Message format changes for idempotent/transactional producer

[Apurva Mehta (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15935601#comment-15935601
 ] 

Apurva Mehta commented on KAFKA-4816:
-

I compiled the perf numbers for this patch, results available here: 
https://docs.google.com/spreadsheets/d/1dHY6M7qCiX-NFvsgvaE0YoVdNq26uA8608XIh_DUpI4/edit?usp=sharing

There is no observable regression on produce or consume at varying message 
sizes, with different workloads, on different machines.

We do see significant on disk savings at smaller message sizes.

Given the nature of the tests, I think the margin of error in throughput 
results is 10%. Results which show a difference of more than 10% should be 
treated with a grain of salt: they generally don't reproduce across runs and 
across machines.

We see consistent producer side performance improvements at small sizes, 
correlated with lower produce request size.

We see general consume side improvements, but there is no pattern and there is 
significant variation across runs. So we can't make any claims on improving the 
consume performance with this message format. But we can claim safely that 
there is no regression since all runs were generally faster with the new format.

> Message format changes for idempotent/transactional producer
> 
>
> Key: KAFKA-4816
> URL: https://issues.apache.org/jira/browse/KAFKA-4816
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients, core, producer 
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
> Fix For: 0.11.0.0
>
>
> This task is for the implementation of the message format changes documented 
> here: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging#KIP-98-ExactlyOnceDeliveryandTransactionalMessaging-MessageFormat.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

[jira] [Commented] (KAFKA-4816) Message format changes for idempotent/transactional producer

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4816?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15889133#comment-15889133
 ] 

ASF GitHub Bot commented on KAFKA-4816:
---

GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/2614

KAFKA-4816: Message format changes for idempotent/transactional producer



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/confluentinc/kafka exactly-once-message-format

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2614.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2614


commit dbedb762dd04784c410b51a38413210da0cf9529
Author: Jason Gustafson 
Date:   2016-11-18T16:41:27Z

EoS Message Format Changes

commit 780880f24e31fe0b16a71be563b7914fc66dc58d
Author: hachikuji 
Date:   2017-02-15T05:09:44Z

Support variable length integer types in the new message format (#127)

commit 67b72a788003260265e87813ea08908ae163485a
Author: hachikuji 
Date:   2017-02-19T19:27:46Z

Add relative timestamps to new message format (#128)

commit 155e2e6b6c8c3d03187e3fd828fd04c194401ca1
Author: hachikuji 
Date:   2017-02-22T23:00:22Z

Implement full (stubbed out) fetch and produce request schema (#130)

commit 3e92c0c54a0334d73a9876c8c49a7ac4e1a77e87
Author: hachikuji 
Date:   2017-02-23T23:25:28Z

Add basic support for control messages (#132)

commit ac20e0630a3738ee86f5a2b84fc7624fad7c8345
Author: hachikuji 
Date:   2017-02-27T19:32:13Z

Add transactional flag support (#135)

commit 04909edd7079e026663a4da937aae4fb61741b27
Author: hachikuji 
Date:   2017-02-27T22:29:04Z

Add leader epoch for KIP-101 and no longer use attributes for null 
key/value (#136)

commit d31596034f331547f8967b565129957f5a8d5da1
Author: hachikuji 
Date:   2017-02-28T00:50:20Z

Support old produce request versions which use older magic values (#131)

commit a5681ac1db5a72184fcc53b2e777cb03152d2f70
Author: hachikuji 
Date:   2017-02-28T20:48:56Z

Add a few missing test cases (#137)

commit a4bd61da3ca052c99290feb61d8a1bb4e0b5995e
Author: Jason Gustafson 
Date:   2017-02-28T23:43:39Z

Some minor tweaks from looking over the code




> Message format changes for idempotent/transactional producer
> 
>
> Key: KAFKA-4816
> URL: https://issues.apache.org/jira/browse/KAFKA-4816
> Project: Kafka
>  Issue Type: Sub-task
>  Components: clients, core, producer 
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
> Fix For: 0.11.0.0
>
>
> This task is for the implementation of the message format changes documented 
> here: 
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-98+-+Exactly+Once+Delivery+and+Transactional+Messaging#KIP-98-ExactlyOnceDeliveryandTransactionalMessaging-MessageFormat.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)