My bad, the link should be this:
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Enriched+Message+Metadata
On Fri, Nov 21, 2014 at 5:29 PM, Timothy Chen tnac...@gmail.com wrote:
Hi Guozhang,
I don't think that is publically accessible, can you update it to the
Kafka wiki?
Tim
On
Hi all,
I have updated the wiki page (
https://iwww.corp.linkedin.com/wiki/cf/display/ENGS/Kafka+Enriched+Message+Metadata)
according to people's comments and discussions offline.
Guozhang
On Thu, Nov 13, 2014 at 9:43 AM, Guozhang Wang wangg...@gmail.com wrote:
Hi Jun,
Sorry for the delay
Hi Guozhang,
I don't think that is publically accessible, can you update it to the
Kafka wiki?
Tim
On Fri, Nov 21, 2014 at 5:24 PM, Guozhang Wang wangg...@gmail.com wrote:
Hi all,
I have updated the wiki page (
Hi Jun,
Sorry for the delay on your comments in the wiki page as well as this
thread; quite swamped now. I will get back to you as soon as I find some
time.
Guozhang
On Tue, Nov 11, 2014 at 6:26 PM, Jun Rao jun...@gmail.com wrote:
Thinking about this a bit more. For adding the auditing
Thinking about this a bit more. For adding the auditing support, I am not
sure if we need to change the message format by adding the application
tags. An alternative way to do that is to add it in the producer client.
For example, for each message payload (doesn't matter what the
serialization
Hi Jun,
Regarding 4) in your comment, after thinking it for a while I cannot come
up a way to it along with log compaction without adding new fields into the
current format on message set. Do you have a better way that do not require
protocol changes?
Guozhang
On Mon, Oct 20, 2014 at 9:53 AM,
Thanks for the detailed comments Jun! Some replies inlined.
On Sun, Oct 19, 2014 at 7:42 PM, Jun Rao jun...@gmail.com wrote:
Hi, Guozhang,
Thanks for the writeup.
A few high level comments.
1. Associating (versioned) schemas to a topic can be a good thing overall.
Yes, this could add a
8. Perhaps I need to see the details of the alternative solution. During
the startup of a follower, in general, it's not enough for the follower to
just see the latest generation of the current leader, since the follower
can be several generations behind. So, if the controlled message only
I have updated the wiki page incorporating received comments. We can
discuss some more details on:
1. How we want to do audit? Whether we want to have in-built auditing on
brokers or even MMs or use an audit consumer to fetch all messages from
just brokers.
2. How we can avoid
Hi, Guozhang,
Thanks for the writeup.
A few high level comments.
1. Associating (versioned) schemas to a topic can be a good thing overall.
Yes, this could add a bit more management overhead in Kafka. However, it
makes sure that the data format contract between a producer and a consumer
is kept
Thanks Joe,
I think we now have a few open questions to discuss around this topic:
1. Shall we make core Kafka properties as first class fields in message
header or put them as tags?
The pros of the first approach is more compacted format and hence less
message header overhead; the cons are
Let me add my view on #2 in less delicate terms than Guozhang did :)
When you¹re trying to run Kafka as a service, having to care about the
format of the message sucks. I have plenty of users who are just fine
using the Avro standard and play nice. Then I have a bunch of users who
don¹t want to
I think we could add schemaId(binary) to the MessageAndMetaData
With the schemaId you can implement different downstream software pattern
on the messages reliably. I wrote up more thoughts on this use
https://cwiki.apache.org/confluence/display/KAFKA/Schema+based+topics it
should strive to
I need to take more time to think about this. Here are a few off-the-cuff
remarks:
- To date we have tried really, really hard to keep the data model for
message simple since after all you can always add whatever you like inside
the message body.
- For system tags, why not just make these fields
Hi Jay,
Thanks for the comments. Replied inline.
Guozhang
On Mon, Oct 13, 2014 at 11:11 AM, Jay Kreps jay.kr...@gmail.com wrote:
I need to take more time to think about this. Here are a few off-the-cuff
remarks:
- To date we have tried really, really hard to keep the data model for
Thanks Guozhang! This is an excellent write-up and the approach nicely
consolidates a number of long-standing issues. It would be great if
everyone can review this carefully and give feedback.
Also, wrt discussion in the past we have used a mix of wiki comments
and the mailing list. Personally, I
Hello all,
I put some thoughts on enhancing our current message metadata format to
solve a bunch of existing issues:
https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Enriched+Message+Metadata
This wiki page is for kicking off some discussions about the feasibility of
adding more info
17 matches
Mail list logo
