Re: [VOTE] KIP-673: Emit JSONs with new auto-generated schema
Hi David, That is a good point that we should clarify. I think we should not commit to guaranteeing the names or the format of the particular request and response schemas themselves, though we should guarantee that they are parseable as JSON. The pre-existing trace logging did not guarantee this either and it was more difficult to parse as it was not well structured. I believe for most structured logging use cases where we are analyzing cluster behavior this should be sufficient. Thanks, Lucas On Thu, Oct 29, 2020 at 3:47 AM David Jacot wrote: > Hi folks, > > I have looked at Anastasia's PR to implement this KIP and I was wondering > how far we want to go with the backward compatibility of this in the > future. > Now that we rely on the auto-generated protocol, the outputted requests and > responses use the name of the fields defined in the schema. Until now, we > have been renaming fields rather easily in the schemas as they were purely > internal. With this KIP, renaming a field will break the structured request > log. > > Does this imply that we will now consider the schemas as part of our public > API? We don't discuss this point in the KIP so it is subject to > interpretation. > > I think that we should be clear on that point and either commit to only > making > the request log parsable while not guaranteeing the format of the requests > and > the responses; or commit to making the schemas part of our public API. > > Personally, I lean towards the former at the moment. That is > probably sufficient > for the targeted use cases. Breaking changes are annoying but as this is > intended to be used for debugging purposes, that may be OK. > > What do you guys think? > > Best, > David > > > On Thu, Oct 8, 2020 at 7:19 PM Anastasia Vela wrote: > > > Thanks everyone for the votes. In summary, the vote passed and the KIP > was > > accepted with 3 binding and 3 non-binding +1s. > > > > Best, > > Anastasia > > > > On Mon, Oct 5, 2020 at 2:50 PM Jason Gustafson > wrote: > > > > > +1 Thanks for the KIP! > > > > > > On Fri, Oct 2, 2020 at 4:37 PM Colin McCabe > wrote: > > > > > > > Thanks, Anastasia! This will be a lot easier to maintain. > > > > > > > > +1 (binding) > > > > > > > > best, > > > > Colin > > > > > > > > On Wed, Sep 30, 2020, at 23:57, David Jacot wrote: > > > > > Thanks for the KIP, Anastasia. > > > > > > > > > > +1 (non-binding) > > > > > > > > > > On Thu, Oct 1, 2020 at 8:06 AM Tom Bentley > > > wrote: > > > > > > > > > > > Thanks Anastasia, > > > > > > > > > > > > +1 (non-binding) > > > > > > > > > > > > > > > > > > On Thu, Oct 1, 2020 at 6:30 AM Gwen Shapira > > > wrote: > > > > > > > > > > > > > Thank you, this will be quite helpful. > > > > > > > > > > > > > > +1 (binding) > > > > > > > > > > > > > > On Wed, Sep 30, 2020 at 11:19 AM Anastasia Vela < > > > av...@confluent.io> > > > > > > > wrote: > > > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > > > Thanks again for the discussion. I'd like to start the vote > for > > > > this > > > > > > KIP. > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-673%3A+Emit+JSONs+with+new+auto-generated+schema > > > > > > > > > > > > > > > > Thanks, > > > > > > > > Anastasia > > > > > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > > Gwen Shapira > > > > > > > Engineering Manager | Confluent > > > > > > > 650.450.2760 | @gwenshap > > > > > > > Follow us: Twitter | blog > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: [VOTE] KIP-673: Emit JSONs with new auto-generated schema
Hi folks, I have looked at Anastasia's PR to implement this KIP and I was wondering how far we want to go with the backward compatibility of this in the future. Now that we rely on the auto-generated protocol, the outputted requests and responses use the name of the fields defined in the schema. Until now, we have been renaming fields rather easily in the schemas as they were purely internal. With this KIP, renaming a field will break the structured request log. Does this imply that we will now consider the schemas as part of our public API? We don't discuss this point in the KIP so it is subject to interpretation. I think that we should be clear on that point and either commit to only making the request log parsable while not guaranteeing the format of the requests and the responses; or commit to making the schemas part of our public API. Personally, I lean towards the former at the moment. That is probably sufficient for the targeted use cases. Breaking changes are annoying but as this is intended to be used for debugging purposes, that may be OK. What do you guys think? Best, David On Thu, Oct 8, 2020 at 7:19 PM Anastasia Vela wrote: > Thanks everyone for the votes. In summary, the vote passed and the KIP was > accepted with 3 binding and 3 non-binding +1s. > > Best, > Anastasia > > On Mon, Oct 5, 2020 at 2:50 PM Jason Gustafson wrote: > > > +1 Thanks for the KIP! > > > > On Fri, Oct 2, 2020 at 4:37 PM Colin McCabe wrote: > > > > > Thanks, Anastasia! This will be a lot easier to maintain. > > > > > > +1 (binding) > > > > > > best, > > > Colin > > > > > > On Wed, Sep 30, 2020, at 23:57, David Jacot wrote: > > > > Thanks for the KIP, Anastasia. > > > > > > > > +1 (non-binding) > > > > > > > > On Thu, Oct 1, 2020 at 8:06 AM Tom Bentley > > wrote: > > > > > > > > > Thanks Anastasia, > > > > > > > > > > +1 (non-binding) > > > > > > > > > > > > > > > On Thu, Oct 1, 2020 at 6:30 AM Gwen Shapira > > wrote: > > > > > > > > > > > Thank you, this will be quite helpful. > > > > > > > > > > > > +1 (binding) > > > > > > > > > > > > On Wed, Sep 30, 2020 at 11:19 AM Anastasia Vela < > > av...@confluent.io> > > > > > > wrote: > > > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > > > Thanks again for the discussion. I'd like to start the vote for > > > this > > > > > KIP. > > > > > > > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-673%3A+Emit+JSONs+with+new+auto-generated+schema > > > > > > > > > > > > > > Thanks, > > > > > > > Anastasia > > > > > > > > > > > > > > > > > > > > > > > > -- > > > > > > Gwen Shapira > > > > > > Engineering Manager | Confluent > > > > > > 650.450.2760 | @gwenshap > > > > > > Follow us: Twitter | blog > > > > > > > > > > > > > > > > > > > > > > > > > > >
Re: [VOTE] KIP-673: Emit JSONs with new auto-generated schema
Thanks everyone for the votes. In summary, the vote passed and the KIP was accepted with 3 binding and 3 non-binding +1s. Best, Anastasia On Mon, Oct 5, 2020 at 2:50 PM Jason Gustafson wrote: > +1 Thanks for the KIP! > > On Fri, Oct 2, 2020 at 4:37 PM Colin McCabe wrote: > > > Thanks, Anastasia! This will be a lot easier to maintain. > > > > +1 (binding) > > > > best, > > Colin > > > > On Wed, Sep 30, 2020, at 23:57, David Jacot wrote: > > > Thanks for the KIP, Anastasia. > > > > > > +1 (non-binding) > > > > > > On Thu, Oct 1, 2020 at 8:06 AM Tom Bentley > wrote: > > > > > > > Thanks Anastasia, > > > > > > > > +1 (non-binding) > > > > > > > > > > > > On Thu, Oct 1, 2020 at 6:30 AM Gwen Shapira > wrote: > > > > > > > > > Thank you, this will be quite helpful. > > > > > > > > > > +1 (binding) > > > > > > > > > > On Wed, Sep 30, 2020 at 11:19 AM Anastasia Vela < > av...@confluent.io> > > > > > wrote: > > > > > > > > > > > > Hi everyone, > > > > > > > > > > > > Thanks again for the discussion. I'd like to start the vote for > > this > > > > KIP. > > > > > > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-673%3A+Emit+JSONs+with+new+auto-generated+schema > > > > > > > > > > > > Thanks, > > > > > > Anastasia > > > > > > > > > > > > > > > > > > > > -- > > > > > Gwen Shapira > > > > > Engineering Manager | Confluent > > > > > 650.450.2760 | @gwenshap > > > > > Follow us: Twitter | blog > > > > > > > > > > > > > > > > > > > >
Re: [VOTE] KIP-673: Emit JSONs with new auto-generated schema
+1 Thanks for the KIP! On Fri, Oct 2, 2020 at 4:37 PM Colin McCabe wrote: > Thanks, Anastasia! This will be a lot easier to maintain. > > +1 (binding) > > best, > Colin > > On Wed, Sep 30, 2020, at 23:57, David Jacot wrote: > > Thanks for the KIP, Anastasia. > > > > +1 (non-binding) > > > > On Thu, Oct 1, 2020 at 8:06 AM Tom Bentley wrote: > > > > > Thanks Anastasia, > > > > > > +1 (non-binding) > > > > > > > > > On Thu, Oct 1, 2020 at 6:30 AM Gwen Shapira wrote: > > > > > > > Thank you, this will be quite helpful. > > > > > > > > +1 (binding) > > > > > > > > On Wed, Sep 30, 2020 at 11:19 AM Anastasia Vela > > > > wrote: > > > > > > > > > > Hi everyone, > > > > > > > > > > Thanks again for the discussion. I'd like to start the vote for > this > > > KIP. > > > > > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-673%3A+Emit+JSONs+with+new+auto-generated+schema > > > > > > > > > > Thanks, > > > > > Anastasia > > > > > > > > > > > > > > > > -- > > > > Gwen Shapira > > > > Engineering Manager | Confluent > > > > 650.450.2760 | @gwenshap > > > > Follow us: Twitter | blog > > > > > > > > > > > > > >
Re: [VOTE] KIP-673: Emit JSONs with new auto-generated schema
Thanks, Anastasia! This will be a lot easier to maintain. +1 (binding) best, Colin On Wed, Sep 30, 2020, at 23:57, David Jacot wrote: > Thanks for the KIP, Anastasia. > > +1 (non-binding) > > On Thu, Oct 1, 2020 at 8:06 AM Tom Bentley wrote: > > > Thanks Anastasia, > > > > +1 (non-binding) > > > > > > On Thu, Oct 1, 2020 at 6:30 AM Gwen Shapira wrote: > > > > > Thank you, this will be quite helpful. > > > > > > +1 (binding) > > > > > > On Wed, Sep 30, 2020 at 11:19 AM Anastasia Vela > > > wrote: > > > > > > > > Hi everyone, > > > > > > > > Thanks again for the discussion. I'd like to start the vote for this > > KIP. > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-673%3A+Emit+JSONs+with+new+auto-generated+schema > > > > > > > > Thanks, > > > > Anastasia > > > > > > > > > > > > -- > > > Gwen Shapira > > > Engineering Manager | Confluent > > > 650.450.2760 | @gwenshap > > > Follow us: Twitter | blog > > > > > > > > >
Re: [VOTE] KIP-673: Emit JSONs with new auto-generated schema
Thanks for the KIP, Anastasia. +1 (non-binding) On Thu, Oct 1, 2020 at 8:06 AM Tom Bentley wrote: > Thanks Anastasia, > > +1 (non-binding) > > > On Thu, Oct 1, 2020 at 6:30 AM Gwen Shapira wrote: > > > Thank you, this will be quite helpful. > > > > +1 (binding) > > > > On Wed, Sep 30, 2020 at 11:19 AM Anastasia Vela > > wrote: > > > > > > Hi everyone, > > > > > > Thanks again for the discussion. I'd like to start the vote for this > KIP. > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-673%3A+Emit+JSONs+with+new+auto-generated+schema > > > > > > Thanks, > > > Anastasia > > > > > > > > -- > > Gwen Shapira > > Engineering Manager | Confluent > > 650.450.2760 | @gwenshap > > Follow us: Twitter | blog > > > > >
Re: [VOTE] KIP-673: Emit JSONs with new auto-generated schema
Thanks Anastasia, +1 (non-binding) On Thu, Oct 1, 2020 at 6:30 AM Gwen Shapira wrote: > Thank you, this will be quite helpful. > > +1 (binding) > > On Wed, Sep 30, 2020 at 11:19 AM Anastasia Vela > wrote: > > > > Hi everyone, > > > > Thanks again for the discussion. I'd like to start the vote for this KIP. > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-673%3A+Emit+JSONs+with+new+auto-generated+schema > > > > Thanks, > > Anastasia > > > > -- > Gwen Shapira > Engineering Manager | Confluent > 650.450.2760 | @gwenshap > Follow us: Twitter | blog > >
Re: [VOTE] KIP-673: Emit JSONs with new auto-generated schema
Thank you, this will be quite helpful. +1 (binding) On Wed, Sep 30, 2020 at 11:19 AM Anastasia Vela wrote: > > Hi everyone, > > Thanks again for the discussion. I'd like to start the vote for this KIP. > https://cwiki.apache.org/confluence/display/KAFKA/KIP-673%3A+Emit+JSONs+with+new+auto-generated+schema > > Thanks, > Anastasia -- Gwen Shapira Engineering Manager | Confluent 650.450.2760 | @gwenshap Follow us: Twitter | blog
Re: [VOTE] KIP-673: Emit JSONs with new auto-generated schema
Hi Anastasia, This looks like a great change that will make debugging cluster behavior much easier. +1 non binding. Lucas On Wed, Sep 30, 2020 at 11:19 AM Anastasia Vela wrote: > Hi everyone, > > Thanks again for the discussion. I'd like to start the vote for this KIP. > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-673%3A+Emit+JSONs+with+new+auto-generated+schema > > Thanks, > Anastasia >
[VOTE] KIP-673: Emit JSONs with new auto-generated schema
Hi everyone, Thanks again for the discussion. I'd like to start the vote for this KIP. https://cwiki.apache.org/confluence/display/KAFKA/KIP-673%3A+Emit+JSONs+with+new+auto-generated+schema Thanks, Anastasia
