Hi All, (again)

If it helps the discussion, and almost ready patch implementing this is 
available here:

https://github.com/michaelandrepearce/kafka


The biggest/most core change is obviously the kafka.message.Message object.



Some key bits in this implementation is the server side, and submodules 
(connect, mirrormaker, streams) all updated to be aware of the new “headers”



As a big API change have to use new feature on the client side, you use the 
ConsumerRecord and ProducerRecord (which now extend the new Enhanced versions) 
for K,V records without any code changes, to use the headers you use the 
Enhanced versions HeadersConsumerRecord and HeadersProducerRecord. This was 
needed to avoid causing code compilation failure just by upgrading. If the 
patch were accepted I would imagine it as a way to transition.



I am guessing this needs a KIP rather than just myself raising a JIRA as fairly 
substantial api change but unsure whom can raise these so assistance in the 
process would be gratefully accepted..



Cheers

Mike






From: Michael Pearce <michael.pea...@ig.com>
Date: Saturday, September 17, 2016 at 6:40 AM
To: "dev@kafka.apache.org" <dev@kafka.apache.org>
Subject: ProducerRecord/Consumer MetaData/Headers

Hi All,

First of all apologies if this has been previously discussed I have just joined 
the mail list (I cannot find a JIRA or KIP related nor via good old google 
search)

In our company we are looking to replace some of our more traditional message 
flows with Kafka.

One thing we have found lacking though compared with most messaging systems is 
the ability to set header/metadata separate from our payload. We did think 
about the key, but as this is used for compaction we cannot have changing 
values here which metadata/header values will obviously be.

e.g. these headers/metadata are useful for audit data or platform data that is 
not business payload related e.g. storing
the clientId that generated the message, correlation id of a request/response, 
cluster id where the message was generate (case for MirrorMakers), message uuid 
etc this list is endless.

We would like to propose extending the Record from 
ProducerRecord/ConsumerRecord<K, V> to ProducerRecord/ConsumerRecord<K,V,M> 
where M is metadata/header again being like the key and value a simple byte[] 
so that it is completely upto the end users how to serialize / deserialize it.

What our people’s thoughts?
Any other ideas how to add headers/metadata.

How can I progress this?

Cheers
Mike
The information contained in this email is strictly confidential and for the 
use of the addressee only, unless otherwise indicated. If you are not the 
intended recipient, please do not read, copy, use or disclose to others this 
message or any attachment. Please also notify the sender by replying to this 
email or by telephone (+44(020 7896 0011) and then delete the email and any 
copies of it. Opinions, conclusion (etc) that do not relate to the official 
business of this company shall be understood as neither given nor endorsed by 
it. IG is a trading name of IG Markets Limited (a company registered in England 
and Wales, company number 04008957) and IG Index Limited (a company registered 
in England and Wales, company number 01190902). Registered address at Cannon 
Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited 
(register number 195355) and IG Index Limited (register number 114059) are 
authorised and regulated by the Financial Conduct Authority.

Reply via email to