Re: [VOTE] KIP-431: Support of printing additional ConsumerRecord fields in DefaultMessageFormatter
Hu No, the property order does not matter. And yes, anything after "<--" is my comment. Do you have any suggestions on making it more readable, while keeping it backward compatible? Can we discuss this in the DISCUSS thread? Regards Badai On Thu, Jul 9, 2020 at 8:05 PM Hu Xi wrote: > > Hi Badai, > > Thanks for the KIP. A quick question from me: > > "CreateTime:1592475472398|key1|3|0|h1=v1,h2=v2|value1<-- offset 3, > partition 0" > > Seems the partition follows the offset. My question is, does the property > order matter? The partition is always printed following the offset no matter > the order for `print.partition=true` and `print.offset=true` is specified. > How do users get aware of the number `3` means the offset instead of the > partition? I am assuming "<-- offset 3, partition 0" is your comment not the > printed words:-) Am I correct? > > > ________________ > 发件人: Badai Aqrandista > 发送时间: 2020年7月9日 17:29 > 收件人: dev > 主题: Re: [VOTE] KIP-431: Support of printing additional ConsumerRecord fields > in DefaultMessageFormatter > > David > > Thanks for the vote. That is a good idea. Will start another KIP once > this one is done. > > Regards > Badai > > On Thu, Jul 9, 2020 at 6:46 PM David Jacot wrote: > > > > Hi Badai, > > > > Thanks for the KIP. I think that it is a nice improvement so I am +1 > > (non-binding). > > > > Long term, I wonder if we could adopt a formatting system similar to > > kafkacat. It > > would reduce the number of properties that one has to set and also allow > > more > > powerful formatting. That could be done as a new formatter for instance. > > > > Example: > > kafkacat -b mybroker -t syslog -f 'Topic %t[%p], offset: %o, key: %k, > > payload: %S bytes: %s\n' > > > > Best, > > David > > > > On Wed, Jul 8, 2020 at 12:30 PM Manikumar wrote: > > > > > +1 (binding) > > > > > > Thanks for the KIP. > > > > > > Thanks, > > > Manikumar > > > > > > > > > > > > On Wed, Jul 8, 2020 at 11:47 AM Matthias J. Sax wrote: > > > > > > > +1 (binding) > > > > > > > > -Matthias > > > > > > > > On 7/7/20 7:16 PM, John Roesler wrote: > > > > > Hi Badai, > > > > > > > > > > Thanks for picking this up. I've reviewed the KIP document and > > > > > the threads you linked. I think we may want to make more > > > > > improvements in the future to the printing of headers in particular, > > > > > but this KIP seems like a clear benefit already. I think we can > > > > > take it incrementally. > > > > > > > > > > I'm +1 (binding) > > > > > > > > > > Thanks, > > > > > -John > > > > > > > > > > On Tue, Jul 7, 2020, at 09:57, Badai Aqrandista wrote: > > > > >> Hi all > > > > >> > > > > >> After resurrecting the discussion thread [1] for KIP-431 and have not > > > > >> received any further feedback for 2 weeks, I would like to resurrect > > > > >> the voting thread [2] for KIP-431. > > > > >> > > > > >> I have updated KIP-431 wiki page > > > > >> ( > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-431%3A+Support+of+printing+additional+ConsumerRecord+fields+in+DefaultMessageFormatter > > > > ) > > > > >> to address Ismael's comment on that thread [3]. > > > > >> > > > > >> Does anyone else have other comments or objections about this KIP? > > > > >> > > > > >> [1] > > > > >> > > > > > > > https://lists.apache.org/thread.html/raabf3268ed05931b8a048fce0d6cdf6a326aee4b0d89713d6e6998d6%40%3Cdev.kafka.apache.org%3E > > > > >> > > > > >> [2] > > > > >> > > > > > > > https://lists.apache.org/thread.html/41fff34873184625370f9e76b8d9257f7a9e7892ab616afe64b4f67c%40%3Cdev.kafka.apache.org%3E > > > > >> > > > > >> [3] > > > > >> > > > > > > > https://lists.apache.org/thread.html/99e9cbaad4a0a49b96db104de450c9f488d4b2b03a09b991bcbadbc7%40%3Cdev.kafka.apache.org%3E > > > > >> > > > > >> -- > > > > >> Thanks, > > > > >> Badai > > > > >> > > > > > > > > > > > > > > > -- > Thanks, > Badai -- Thanks, Badai
回复: [VOTE] KIP-431: Support of printing additional ConsumerRecord fields in DefaultMessageFormatter
Hi Badai, Thanks for the KIP. A quick question from me: "CreateTime:1592475472398|key1|3|0|h1=v1,h2=v2|value1<-- offset 3, partition 0" Seems the partition follows the offset. My question is, does the property order matter? The partition is always printed following the offset no matter the order for `print.partition=true` and `print.offset=true` is specified. How do users get aware of the number `3` means the offset instead of the partition? I am assuming "<-- offset 3, partition 0" is your comment not the printed words:-) Am I correct? 发件人: Badai Aqrandista 发送时间: 2020年7月9日 17:29 收件人: dev 主题: Re: [VOTE] KIP-431: Support of printing additional ConsumerRecord fields in DefaultMessageFormatter David Thanks for the vote. That is a good idea. Will start another KIP once this one is done. Regards Badai On Thu, Jul 9, 2020 at 6:46 PM David Jacot wrote: > > Hi Badai, > > Thanks for the KIP. I think that it is a nice improvement so I am +1 > (non-binding). > > Long term, I wonder if we could adopt a formatting system similar to > kafkacat. It > would reduce the number of properties that one has to set and also allow > more > powerful formatting. That could be done as a new formatter for instance. > > Example: > kafkacat -b mybroker -t syslog -f 'Topic %t[%p], offset: %o, key: %k, > payload: %S bytes: %s\n' > > Best, > David > > On Wed, Jul 8, 2020 at 12:30 PM Manikumar wrote: > > > +1 (binding) > > > > Thanks for the KIP. > > > > Thanks, > > Manikumar > > > > > > > > On Wed, Jul 8, 2020 at 11:47 AM Matthias J. Sax wrote: > > > > > +1 (binding) > > > > > > -Matthias > > > > > > On 7/7/20 7:16 PM, John Roesler wrote: > > > > Hi Badai, > > > > > > > > Thanks for picking this up. I've reviewed the KIP document and > > > > the threads you linked. I think we may want to make more > > > > improvements in the future to the printing of headers in particular, > > > > but this KIP seems like a clear benefit already. I think we can > > > > take it incrementally. > > > > > > > > I'm +1 (binding) > > > > > > > > Thanks, > > > > -John > > > > > > > > On Tue, Jul 7, 2020, at 09:57, Badai Aqrandista wrote: > > > >> Hi all > > > >> > > > >> After resurrecting the discussion thread [1] for KIP-431 and have not > > > >> received any further feedback for 2 weeks, I would like to resurrect > > > >> the voting thread [2] for KIP-431. > > > >> > > > >> I have updated KIP-431 wiki page > > > >> ( > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-431%3A+Support+of+printing+additional+ConsumerRecord+fields+in+DefaultMessageFormatter > > > ) > > > >> to address Ismael's comment on that thread [3]. > > > >> > > > >> Does anyone else have other comments or objections about this KIP? > > > >> > > > >> [1] > > > >> > > > > > https://lists.apache.org/thread.html/raabf3268ed05931b8a048fce0d6cdf6a326aee4b0d89713d6e6998d6%40%3Cdev.kafka.apache.org%3E > > > >> > > > >> [2] > > > >> > > > > > https://lists.apache.org/thread.html/41fff34873184625370f9e76b8d9257f7a9e7892ab616afe64b4f67c%40%3Cdev.kafka.apache.org%3E > > > >> > > > >> [3] > > > >> > > > > > https://lists.apache.org/thread.html/99e9cbaad4a0a49b96db104de450c9f488d4b2b03a09b991bcbadbc7%40%3Cdev.kafka.apache.org%3E > > > >> > > > >> -- > > > >> Thanks, > > > >> Badai > > > >> > > > > > > > > -- Thanks, Badai
Re: [VOTE] KIP-431: Support of printing additional ConsumerRecord fields in DefaultMessageFormatter
David Thanks for the vote. That is a good idea. Will start another KIP once this one is done. Regards Badai On Thu, Jul 9, 2020 at 6:46 PM David Jacot wrote: > > Hi Badai, > > Thanks for the KIP. I think that it is a nice improvement so I am +1 > (non-binding). > > Long term, I wonder if we could adopt a formatting system similar to > kafkacat. It > would reduce the number of properties that one has to set and also allow > more > powerful formatting. That could be done as a new formatter for instance. > > Example: > kafkacat -b mybroker -t syslog -f 'Topic %t[%p], offset: %o, key: %k, > payload: %S bytes: %s\n' > > Best, > David > > On Wed, Jul 8, 2020 at 12:30 PM Manikumar wrote: > > > +1 (binding) > > > > Thanks for the KIP. > > > > Thanks, > > Manikumar > > > > > > > > On Wed, Jul 8, 2020 at 11:47 AM Matthias J. Sax wrote: > > > > > +1 (binding) > > > > > > -Matthias > > > > > > On 7/7/20 7:16 PM, John Roesler wrote: > > > > Hi Badai, > > > > > > > > Thanks for picking this up. I've reviewed the KIP document and > > > > the threads you linked. I think we may want to make more > > > > improvements in the future to the printing of headers in particular, > > > > but this KIP seems like a clear benefit already. I think we can > > > > take it incrementally. > > > > > > > > I'm +1 (binding) > > > > > > > > Thanks, > > > > -John > > > > > > > > On Tue, Jul 7, 2020, at 09:57, Badai Aqrandista wrote: > > > >> Hi all > > > >> > > > >> After resurrecting the discussion thread [1] for KIP-431 and have not > > > >> received any further feedback for 2 weeks, I would like to resurrect > > > >> the voting thread [2] for KIP-431. > > > >> > > > >> I have updated KIP-431 wiki page > > > >> ( > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-431%3A+Support+of+printing+additional+ConsumerRecord+fields+in+DefaultMessageFormatter > > > ) > > > >> to address Ismael's comment on that thread [3]. > > > >> > > > >> Does anyone else have other comments or objections about this KIP? > > > >> > > > >> [1] > > > >> > > > > > https://lists.apache.org/thread.html/raabf3268ed05931b8a048fce0d6cdf6a326aee4b0d89713d6e6998d6%40%3Cdev.kafka.apache.org%3E > > > >> > > > >> [2] > > > >> > > > > > https://lists.apache.org/thread.html/41fff34873184625370f9e76b8d9257f7a9e7892ab616afe64b4f67c%40%3Cdev.kafka.apache.org%3E > > > >> > > > >> [3] > > > >> > > > > > https://lists.apache.org/thread.html/99e9cbaad4a0a49b96db104de450c9f488d4b2b03a09b991bcbadbc7%40%3Cdev.kafka.apache.org%3E > > > >> > > > >> -- > > > >> Thanks, > > > >> Badai > > > >> > > > > > > > > -- Thanks, Badai
Re: [VOTE] KIP-431: Support of printing additional ConsumerRecord fields in DefaultMessageFormatter
Hi Badai, Thanks for the KIP. I think that it is a nice improvement so I am +1 (non-binding). Long term, I wonder if we could adopt a formatting system similar to kafkacat. It would reduce the number of properties that one has to set and also allow more powerful formatting. That could be done as a new formatter for instance. Example: kafkacat -b mybroker -t syslog -f 'Topic %t[%p], offset: %o, key: %k, payload: %S bytes: %s\n' Best, David On Wed, Jul 8, 2020 at 12:30 PM Manikumar wrote: > +1 (binding) > > Thanks for the KIP. > > Thanks, > Manikumar > > > > On Wed, Jul 8, 2020 at 11:47 AM Matthias J. Sax wrote: > > > +1 (binding) > > > > -Matthias > > > > On 7/7/20 7:16 PM, John Roesler wrote: > > > Hi Badai, > > > > > > Thanks for picking this up. I've reviewed the KIP document and > > > the threads you linked. I think we may want to make more > > > improvements in the future to the printing of headers in particular, > > > but this KIP seems like a clear benefit already. I think we can > > > take it incrementally. > > > > > > I'm +1 (binding) > > > > > > Thanks, > > > -John > > > > > > On Tue, Jul 7, 2020, at 09:57, Badai Aqrandista wrote: > > >> Hi all > > >> > > >> After resurrecting the discussion thread [1] for KIP-431 and have not > > >> received any further feedback for 2 weeks, I would like to resurrect > > >> the voting thread [2] for KIP-431. > > >> > > >> I have updated KIP-431 wiki page > > >> ( > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-431%3A+Support+of+printing+additional+ConsumerRecord+fields+in+DefaultMessageFormatter > > ) > > >> to address Ismael's comment on that thread [3]. > > >> > > >> Does anyone else have other comments or objections about this KIP? > > >> > > >> [1] > > >> > > > https://lists.apache.org/thread.html/raabf3268ed05931b8a048fce0d6cdf6a326aee4b0d89713d6e6998d6%40%3Cdev.kafka.apache.org%3E > > >> > > >> [2] > > >> > > > https://lists.apache.org/thread.html/41fff34873184625370f9e76b8d9257f7a9e7892ab616afe64b4f67c%40%3Cdev.kafka.apache.org%3E > > >> > > >> [3] > > >> > > > https://lists.apache.org/thread.html/99e9cbaad4a0a49b96db104de450c9f488d4b2b03a09b991bcbadbc7%40%3Cdev.kafka.apache.org%3E > > >> > > >> -- > > >> Thanks, > > >> Badai > > >> > > > > >
Re: [VOTE] KIP-431: Support of printing additional ConsumerRecord fields in DefaultMessageFormatter
+1 (binding) Thanks for the KIP. Thanks, Manikumar On Wed, Jul 8, 2020 at 11:47 AM Matthias J. Sax wrote: > +1 (binding) > > -Matthias > > On 7/7/20 7:16 PM, John Roesler wrote: > > Hi Badai, > > > > Thanks for picking this up. I've reviewed the KIP document and > > the threads you linked. I think we may want to make more > > improvements in the future to the printing of headers in particular, > > but this KIP seems like a clear benefit already. I think we can > > take it incrementally. > > > > I'm +1 (binding) > > > > Thanks, > > -John > > > > On Tue, Jul 7, 2020, at 09:57, Badai Aqrandista wrote: > >> Hi all > >> > >> After resurrecting the discussion thread [1] for KIP-431 and have not > >> received any further feedback for 2 weeks, I would like to resurrect > >> the voting thread [2] for KIP-431. > >> > >> I have updated KIP-431 wiki page > >> ( > https://cwiki.apache.org/confluence/display/KAFKA/KIP-431%3A+Support+of+printing+additional+ConsumerRecord+fields+in+DefaultMessageFormatter > ) > >> to address Ismael's comment on that thread [3]. > >> > >> Does anyone else have other comments or objections about this KIP? > >> > >> [1] > >> > https://lists.apache.org/thread.html/raabf3268ed05931b8a048fce0d6cdf6a326aee4b0d89713d6e6998d6%40%3Cdev.kafka.apache.org%3E > >> > >> [2] > >> > https://lists.apache.org/thread.html/41fff34873184625370f9e76b8d9257f7a9e7892ab616afe64b4f67c%40%3Cdev.kafka.apache.org%3E > >> > >> [3] > >> > https://lists.apache.org/thread.html/99e9cbaad4a0a49b96db104de450c9f488d4b2b03a09b991bcbadbc7%40%3Cdev.kafka.apache.org%3E > >> > >> -- > >> Thanks, > >> Badai > >> > >
Re: [VOTE] KIP-431: Support of printing additional ConsumerRecord fields in DefaultMessageFormatter
+1 (binding) -Matthias On 7/7/20 7:16 PM, John Roesler wrote: > Hi Badai, > > Thanks for picking this up. I've reviewed the KIP document and > the threads you linked. I think we may want to make more > improvements in the future to the printing of headers in particular, > but this KIP seems like a clear benefit already. I think we can > take it incrementally. > > I'm +1 (binding) > > Thanks, > -John > > On Tue, Jul 7, 2020, at 09:57, Badai Aqrandista wrote: >> Hi all >> >> After resurrecting the discussion thread [1] for KIP-431 and have not >> received any further feedback for 2 weeks, I would like to resurrect >> the voting thread [2] for KIP-431. >> >> I have updated KIP-431 wiki page >> (https://cwiki.apache.org/confluence/display/KAFKA/KIP-431%3A+Support+of+printing+additional+ConsumerRecord+fields+in+DefaultMessageFormatter) >> to address Ismael's comment on that thread [3]. >> >> Does anyone else have other comments or objections about this KIP? >> >> [1] >> https://lists.apache.org/thread.html/raabf3268ed05931b8a048fce0d6cdf6a326aee4b0d89713d6e6998d6%40%3Cdev.kafka.apache.org%3E >> >> [2] >> https://lists.apache.org/thread.html/41fff34873184625370f9e76b8d9257f7a9e7892ab616afe64b4f67c%40%3Cdev.kafka.apache.org%3E >> >> [3] >> https://lists.apache.org/thread.html/99e9cbaad4a0a49b96db104de450c9f488d4b2b03a09b991bcbadbc7%40%3Cdev.kafka.apache.org%3E >> >> -- >> Thanks, >> Badai >> signature.asc Description: OpenPGP digital signature
Re: [VOTE] KIP-431: Support of printing additional ConsumerRecord fields in DefaultMessageFormatter
Hi Badai, Thanks for picking this up. I've reviewed the KIP document and the threads you linked. I think we may want to make more improvements in the future to the printing of headers in particular, but this KIP seems like a clear benefit already. I think we can take it incrementally. I'm +1 (binding) Thanks, -John On Tue, Jul 7, 2020, at 09:57, Badai Aqrandista wrote: > Hi all > > After resurrecting the discussion thread [1] for KIP-431 and have not > received any further feedback for 2 weeks, I would like to resurrect > the voting thread [2] for KIP-431. > > I have updated KIP-431 wiki page > (https://cwiki.apache.org/confluence/display/KAFKA/KIP-431%3A+Support+of+printing+additional+ConsumerRecord+fields+in+DefaultMessageFormatter) > to address Ismael's comment on that thread [3]. > > Does anyone else have other comments or objections about this KIP? > > [1] > https://lists.apache.org/thread.html/raabf3268ed05931b8a048fce0d6cdf6a326aee4b0d89713d6e6998d6%40%3Cdev.kafka.apache.org%3E > > [2] > https://lists.apache.org/thread.html/41fff34873184625370f9e76b8d9257f7a9e7892ab616afe64b4f67c%40%3Cdev.kafka.apache.org%3E > > [3] > https://lists.apache.org/thread.html/99e9cbaad4a0a49b96db104de450c9f488d4b2b03a09b991bcbadbc7%40%3Cdev.kafka.apache.org%3E > > -- > Thanks, > Badai >
Re: [VOTE] KIP-431: Support of printing additional ConsumerRecord fields in DefaultMessageFormatter
Hi all After resurrecting the discussion thread [1] for KIP-431 and have not received any further feedback for 2 weeks, I would like to resurrect the voting thread [2] for KIP-431. I have updated KIP-431 wiki page (https://cwiki.apache.org/confluence/display/KAFKA/KIP-431%3A+Support+of+printing+additional+ConsumerRecord+fields+in+DefaultMessageFormatter) to address Ismael's comment on that thread [3]. Does anyone else have other comments or objections about this KIP? [1] https://lists.apache.org/thread.html/raabf3268ed05931b8a048fce0d6cdf6a326aee4b0d89713d6e6998d6%40%3Cdev.kafka.apache.org%3E [2] https://lists.apache.org/thread.html/41fff34873184625370f9e76b8d9257f7a9e7892ab616afe64b4f67c%40%3Cdev.kafka.apache.org%3E [3] https://lists.apache.org/thread.html/99e9cbaad4a0a49b96db104de450c9f488d4b2b03a09b991bcbadbc7%40%3Cdev.kafka.apache.org%3E -- Thanks, Badai
Re: [VOTE] KIP-431: Support of printing additional ConsumerRecord fields in DefaultMessageFormatter
Hi Ismael,
Thanks for your questions. Answers below.
1. It would be helpful to see an example of the output with everything
enabled.
For consumer record:
new ConsumerRecord[Array[Byte], Array[Byte]](
"someTopic",
partition = 9,
offset = 9876,
timestamp = 123,
timestampType = TimestampType.CREATE_TIME,
checksum = 0L,
serializedKeySize = 0,
serializedValueSize = 0,
key = "someKey",
value = "someValue",
new RecordHeaders(Seq(header("h1", "v1"), header("h2", "v2")).asJava)
)
and everything enabled:
Map("print.key" -> "true",
"print.timestamp" -> "true",
"print.partition" -> "true",
"print.offset" -> "true",
"print.headers" -> "true",
"print.value" -> "true"),
The output would be:
"CreateTime:1234 someKey 9876 9 h1:v1,h2:v2 someValue
"
2. What are the default values for the properties (eg what's the default
header separator).
printTimestamp = false
printKey = false
printOffset = false
printPartition = false
printHeaders = false
printValue = true
keySeparator = "\t"
headersSeparator = ","
lineSeparator = "\n"
3. What is the separator used between key/value and the new fields?
There is no additional separator. keySeparator is used for separating key,
value and any new fields.
It is backward compatible behavior. keySeparator - separate key from
anything else.
Regards,
Mateusz Zakarczemny
pt., 12 kwi 2019 o 22:47 Ismael Juma napisał(a):
> Hi Mateusz,
>
> The KIP looks good. Just a few of questions/suggestions:
>
> 1. It would be helpful to see an example of the output with everything
> enabled.
> 2. What are the default values for the properties (eg what's the default
> header separator).
> 3. What is the separator used between key/value and the new fields?
>
> Ismael
>
> On Fri, Apr 12, 2019 at 9:43 PM Mateusz Zakarczemny <
> m.zakarcze...@gmail.com>
> wrote:
>
> > Hi All,
> > This KIP is in discussion for more than a month. The feedback is positive
> > without any objection comments. Therefore, I would like to start voting.
> >
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-431%3A+Support+of+printing+additional+ConsumerRecord+fields+in+DefaultMessageFormatter
> >
> > Regards,
> > Mateusz Zakarczemny
> >
>
Re: [VOTE] KIP-431: Support of printing additional ConsumerRecord fields in DefaultMessageFormatter
Hi Mateusz, The KIP looks good. Just a few of questions/suggestions: 1. It would be helpful to see an example of the output with everything enabled. 2. What are the default values for the properties (eg what's the default header separator). 3. What is the separator used between key/value and the new fields? Ismael On Fri, Apr 12, 2019 at 9:43 PM Mateusz Zakarczemny wrote: > Hi All, > This KIP is in discussion for more than a month. The feedback is positive > without any objection comments. Therefore, I would like to start voting. > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-431%3A+Support+of+printing+additional+ConsumerRecord+fields+in+DefaultMessageFormatter > > Regards, > Mateusz Zakarczemny >
[VOTE] KIP-431: Support of printing additional ConsumerRecord fields in DefaultMessageFormatter
Hi All, This KIP is in discussion for more than a month. The feedback is positive without any objection comments. Therefore, I would like to start voting. https://cwiki.apache.org/confluence/display/KAFKA/KIP-431%3A+Support+of+printing+additional+ConsumerRecord+fields+in+DefaultMessageFormatter Regards, Mateusz Zakarczemny
