Thanks Ismael.

So i have made a start at a draft KIP please see this here.

https://cwiki.apache.org/confluence/display/KAFKA/KIP-82+-+Add+Record+Headers

I will start a DISCUSS thread shortly.

________________________________________
From: isma...@gmail.com <isma...@gmail.com> on behalf of Ismael Juma 
<ism...@juma.me.uk>
Sent: Thursday, September 22, 2016 9:42 AM
To: dev@kafka.apache.org
Subject: Re: ProducerRecord/Consumer MetaData/Headers

Sorry for the delay, you should have access now.

Ismael

On Thu, Sep 22, 2016 at 8:15 AM, Michael Pearce <michael.pea...@ig.com>
wrote:

> Hi again,
>
> Sorry to be nudging this, but it seems I'm still unable to create a page
> in the KIP Proposal area. Just don't want to be forgotten about between all
> the emails.
>
> Cheers
> Mike
> ________________________________________
> From: Michael Pearce <michael.pea...@ig.com>
> Sent: Monday, September 19, 2016 6:19 PM
> To: dev@kafka.apache.org
> Subject: Re: ProducerRecord/Consumer MetaData/Headers
>
> Hi Again,
>
> I went to the wiki this afternoon to start writing it up, and seems I
> cannot create a page under the KIP area still. If someone could assist.
>
> Cheers
> Mike
>
>
> On 9/18/16, 7:07 PM, "Michael Pearce" <michael.pea...@ig.com> wrote:
>
>     Hi Ismaelj
>
>     Thanks, my wiki user is michael.andre.pearce.
>
>     Re the link thanks again, actually indeed we started off trying to do
> this after we lost the ability to use the key to hold metadata once the
> compaction feature came, but it actually abusing the payload isn't imo a
> great solution, and has some issues that cannot be overcome and stopping us
> from using in some of our data / message flows. As such I think a solution
> in the broker/message/client needs to be made and formalised. Also then an
> ecosystems of tools could rely on such.
>
>     I will add all details in KIP proposal, once I have access.
>
>     Cheers
>     Mike
>
>
>     ________________________________________
>     From: isma...@gmail.com <isma...@gmail.com> on behalf of Ismael Juma <
> ism...@juma.me.uk>
>     Sent: Sunday, September 18, 2016 9:01:22 AM
>     To: dev@kafka.apache.org
>     Subject: Re: ProducerRecord/Consumer MetaData/Headers
>
>     Hi Mike,
>
>     If you give me your wiki user name, I can give you the required
> permissions
>     to post a KIP. This is definitely a big change and there is no clear
>     consensus if changing the Kafka message format is the right way (it
> would
>     be good not to pay the cost if you don't need it) or if it should be
> done
>     via schemas, for example. Gwen shared some thoughts in the following
>     message:
>
>     http://search-hadoop.com/m/uyzND1OXS8EoGCU2
>
>     Ismael
>
>     On Sun, Sep 18, 2016 at 7:11 AM, Michael Pearce <michael.pea...@ig.com
> >
>     wrote:
>
>     > Hi All, (again)
>     >
>     > If it helps the discussion, and almost ready patch implementing this
> is
>     > available here:
>     >
>     > https://github.com/michaelandrepearce/kafka
>     >
>     >
>     > The biggest/most core change is obviously the kafka.message.Message
> object.
>     >
>     >
>     >
>     > Some key bits in this implementation is the server side, and
> submodules
>     > (connect, mirrormaker, streams) all updated to be aware of the new
> “headers”
>     >
>     >
>     >
>     > As a big API change have to use new feature on the client side, you
> use
>     > the ConsumerRecord and ProducerRecord (which now extend the new
> Enhanced
>     > versions) for K,V records without any code changes, to use the
> headers you
>     > use the Enhanced versions HeadersConsumerRecord and
> HeadersProducerRecord.
>     > This was needed to avoid causing code compilation failure just by
>     > upgrading. If the patch were accepted I would imagine it as a way to
>     > transition.
>     >
>     >
>     >
>     > I am guessing this needs a KIP rather than just myself raising a
> JIRA as
>     > fairly substantial api change but unsure whom can raise these so
> assistance
>     > in the process would be gratefully accepted..
>     >
>     >
>     >
>     > Cheers
>     >
>     > Mike
>     >
>     >
>     >
>     >
>     >
>     >
>     > From: Michael Pearce <michael.pea...@ig.com>
>     > Date: Saturday, September 17, 2016 at 6:40 AM
>     > To: "dev@kafka.apache.org" <dev@kafka.apache.org>
>     > Subject: ProducerRecord/Consumer MetaData/Headers
>     >
>     > Hi All,
>     >
>     > First of all apologies if this has been previously discussed I have
> just
>     > joined the mail list (I cannot find a JIRA or KIP related nor via
> good old
>     > google search)
>     >
>     > In our company we are looking to replace some of our more traditional
>     > message flows with Kafka.
>     >
>     > One thing we have found lacking though compared with most messaging
>     > systems is the ability to set header/metadata separate from our
> payload. We
>     > did think about the key, but as this is used for compaction we
> cannot have
>     > changing values here which metadata/header values will obviously be.
>     >
>     > e.g. these headers/metadata are useful for audit data or platform
> data
>     > that is not business payload related e.g. storing
>     > the clientId that generated the message, correlation id of a
>     > request/response, cluster id where the message was generate (case for
>     > MirrorMakers), message uuid etc this list is endless.
>     >
>     > We would like to propose extending the Record from
>     > ProducerRecord/ConsumerRecord<K, V> to
> ProducerRecord/ConsumerRecord<K,V,M>
>     > where M is metadata/header again being like the key and value a
> simple
>     > byte[] so that it is completely upto the end users how to serialize /
>     > deserialize it.
>     >
>     > What our people’s thoughts?
>     > Any other ideas how to add headers/metadata.
>     >
>     > How can I progress this?
>     >
>     > Cheers
>     > Mike
>     > The information contained in this email is strictly confidential and
> for
>     > the use of the addressee only, unless otherwise indicated. If you
> are not
>     > the intended recipient, please do not read, copy, use or disclose to
> others
>     > this message or any attachment. Please also notify the sender by
> replying
>     > to this email or by telephone (+44(020 7896 0011) and then delete
> the
>     > email and any copies of it. Opinions, conclusion (etc) that do not
> relate
>     > to the official business of this company shall be understood as
> neither
>     > given nor endorsed by it. IG is a trading name of IG Markets Limited
> (a
>     > company registered in England and Wales, company number 04008957)
> and IG
>     > Index Limited (a company registered in England and Wales, company
> number
>     > 01190902). Registered address at Cannon Bridge House, 25 Dowgate
> Hill,
>     > London EC4R 2YA. Both IG Markets Limited (register number 195355)
> and IG
>     > Index Limited (register number 114059) are authorised and regulated
> by the
>     > Financial Conduct Authority.
>     >
>     The information contained in this email is strictly confidential and
> for the use of the addressee only, unless otherwise indicated. If you are
> not the intended recipient, please do not read, copy, use or disclose to
> others this message or any attachment. Please also notify the sender by
> replying to this email or by telephone (+44(020 7896 0011) and then delete
> the email and any copies of it. Opinions, conclusion (etc) that do not
> relate to the official business of this company shall be understood as
> neither given nor endorsed by it. IG is a trading name of IG Markets
> Limited (a company registered in England and Wales, company number
> 04008957) and IG Index Limited (a company registered in England and Wales,
> company number 01190902). Registered address at Cannon Bridge House, 25
> Dowgate Hill, London EC4R 2YA. Both IG Markets Limited (register number
> 195355) and IG Index Limited (register number 114059) are authorised and
> regulated by the Financial Conduct Authority.
>
>
>
The information contained in this email is strictly confidential and for the 
use of the addressee only, unless otherwise indicated. If you are not the 
intended recipient, please do not read, copy, use or disclose to others this 
message or any attachment. Please also notify the sender by replying to this 
email or by telephone (+44(020 7896 0011) and then delete the email and any 
copies of it. Opinions, conclusion (etc) that do not relate to the official 
business of this company shall be understood as neither given nor endorsed by 
it. IG is a trading name of IG Markets Limited (a company registered in England 
and Wales, company number 04008957) and IG Index Limited (a company registered 
in England and Wales, company number 01190902). Registered address at Cannon 
Bridge House, 25 Dowgate Hill, London EC4R 2YA. Both IG Markets Limited 
(register number 195355) and IG Index Limited (register number 114059) are 
authorised and regulated by the Financial Conduct Authority.

Reply via email to