Re: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
Hi Martijn, Thanks for looking at this. I have used the template in a new Google Doc https://docs.google.com/document/d/1J1E-cE-X2H3-kw4rNjLn71OGPQk_Yl1iGX4-eCHWLgE/edit?usp=sharing. I have significantly reduced the content in the Flip, in line with what I see as the template and its usage. If this it too much or too little, I can amend, Kind regards, David. From: Martijn Visser Date: Friday, 12 April 2024 at 18:11 To: dev@flink.apache.org Subject: Re: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format Hi David, I tried, but the format wasn't as the FLIP template expects, so I ended up needing to change the entire formatting and that was just too much work to be honest. If you could make sure that especially the headers match with the FLIP template, and that all of the contents from the FLIP template is there, that would make things much easier. Thanks, Martijn On Fri, Apr 12, 2024 at 6:08 PM David Radley wrote: > Hi, > A gentle nudge. Please could a committer/PMC member raise the Flip for > this, > Kind regards, David. > > > From: David Radley > Date: Monday, 8 April 2024 at 09:40 > To: dev@flink.apache.org > Subject: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format > Hi, > I have posted a Google Doc [0] to the mailing list for a discussion thread > for a Flip proposal to introduce a Apicurio-avro format. The discussions > have been resolved, please could a committer/PMC member copy the contents > from the Google Doc, and create a FLIP number for this,. as per the process > [1], > Kind regards, David. > [0] > > https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_document_d_14LWZPVFQ7F9mryJPdKXb4l32n7B0iWYkcOdEd1xTC7w_edit-3Fusp-3Dsharing=DwIGaQ=BSDicqBQBDjDI9RkVyTcHQ=a_7ppZzQ4vpQjmqdi73nB22RONTV0tEZsZXcfdiBEOA=ir9ageEmhu8pt03AmvMqEG9MHPp8aZLMBcqU2pmOnyg6yHra8b6IRXFylvH_aP8G=pHL2e8waNNtvTDT0a3PQM0bcXrb1Fywv0YW_Ln50jCo= > > [1] > > https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_FLINK_Flink-2BImprovement-2BProposals-23FlinkImprovementProposals-2DCreateyourOwnFLIP=DwIGaQ=BSDicqBQBDjDI9RkVyTcHQ=a_7ppZzQ4vpQjmqdi73nB22RONTV0tEZsZXcfdiBEOA=ir9ageEmhu8pt03AmvMqEG9MHPp8aZLMBcqU2pmOnyg6yHra8b6IRXFylvH_aP8G=_7fvlZYc-gUtkFEhwSz9utYsgbDrUtkHEToTdhtQvQc= > > From: Jeyhun Karimov > Date: Friday, 22 March 2024 at 13:05 > To: dev@flink.apache.org > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format > Hi David, > > Thanks a lot for clarification. > Sounds good to me. > > Regards, > Jeyhun > > On Fri, Mar 22, 2024 at 10:54 AM David Radley > wrote: > > > Hi Jeyhun, > > Thanks for your feedback. > > > > So for outbound messages, the message includes the global ID. We register > > the schema and match on the artifact id. So if the schema then evolved, > > adding a new version, the global ID would still be unique and the same > > version would be targeted. If you wanted to change the Flink table > > definition in line with a higher version, then you could do this – the > > artifact id would need to match for it to use the same schema and a > higher > > artifact version would need to be provided. I notice that Apicurio has > > rules around compatibility that you can configure, I suppose if we > attempt > > to create an artifact that breaks these rules , then the register schema > > will fail and the associated operation should fail (e.g. an insert). I > have > > not tried this. > > > > > > For inbound messages, using the global id in the header – this targets > one > > version of the schema. I can create different messages on the topic built > > with different schema versions, and I can create different tables in > Flink, > > as long as the reader and writer schemas are compatible as per the > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_flink_blob_779459168c46b7b4c600ef52f99a5435f81b9048_flink-2Dformats_flink-2Davro_src_main_java_org_apache_flink_formats_avro_RegistryAvroDeserializationSchema.java-23L109=DwIGaQ=BSDicqBQBDjDI9RkVyTcHQ=a_7ppZzQ4vpQjmqdi73nB22RONTV0tEZsZXcfdiBEOA=ir9ageEmhu8pt03AmvMqEG9MHPp8aZLMBcqU2pmOnyg6yHra8b6IRXFylvH_aP8G=kfPzGTjUx9alvbOMoJoeWEHHQ14qwYxTJXbVWAhYvAc= > > Then this should work. > > > > Does this address your question? > > Kind regards, David. > > > > > > From: Jeyhun Karimov > > Date: Thursday, 21 March 2024 at 21:06 > > To: dev@flink.apache.org > > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format > > Hi David, > > > > Thanks for the FLIP. +1 for it. > > I have a minor comment. > > > > Can you please elaborate more on mechanisms in place to ensure data > > consistenc
Re: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
Hi David, I tried, but the format wasn't as the FLIP template expects, so I ended up needing to change the entire formatting and that was just too much work to be honest. If you could make sure that especially the headers match with the FLIP template, and that all of the contents from the FLIP template is there, that would make things much easier. Thanks, Martijn On Fri, Apr 12, 2024 at 6:08 PM David Radley wrote: > Hi, > A gentle nudge. Please could a committer/PMC member raise the Flip for > this, > Kind regards, David. > > > From: David Radley > Date: Monday, 8 April 2024 at 09:40 > To: dev@flink.apache.org > Subject: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format > Hi, > I have posted a Google Doc [0] to the mailing list for a discussion thread > for a Flip proposal to introduce a Apicurio-avro format. The discussions > have been resolved, please could a committer/PMC member copy the contents > from the Google Doc, and create a FLIP number for this,. as per the process > [1], > Kind regards, David. > [0] > > https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_document_d_14LWZPVFQ7F9mryJPdKXb4l32n7B0iWYkcOdEd1xTC7w_edit-3Fusp-3Dsharing=DwIGaQ=BSDicqBQBDjDI9RkVyTcHQ=a_7ppZzQ4vpQjmqdi73nB22RONTV0tEZsZXcfdiBEOA=ir9ageEmhu8pt03AmvMqEG9MHPp8aZLMBcqU2pmOnyg6yHra8b6IRXFylvH_aP8G=pHL2e8waNNtvTDT0a3PQM0bcXrb1Fywv0YW_Ln50jCo= > > [1] > > https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_FLINK_Flink-2BImprovement-2BProposals-23FlinkImprovementProposals-2DCreateyourOwnFLIP=DwIGaQ=BSDicqBQBDjDI9RkVyTcHQ=a_7ppZzQ4vpQjmqdi73nB22RONTV0tEZsZXcfdiBEOA=ir9ageEmhu8pt03AmvMqEG9MHPp8aZLMBcqU2pmOnyg6yHra8b6IRXFylvH_aP8G=_7fvlZYc-gUtkFEhwSz9utYsgbDrUtkHEToTdhtQvQc= > > From: Jeyhun Karimov > Date: Friday, 22 March 2024 at 13:05 > To: dev@flink.apache.org > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format > Hi David, > > Thanks a lot for clarification. > Sounds good to me. > > Regards, > Jeyhun > > On Fri, Mar 22, 2024 at 10:54 AM David Radley > wrote: > > > Hi Jeyhun, > > Thanks for your feedback. > > > > So for outbound messages, the message includes the global ID. We register > > the schema and match on the artifact id. So if the schema then evolved, > > adding a new version, the global ID would still be unique and the same > > version would be targeted. If you wanted to change the Flink table > > definition in line with a higher version, then you could do this – the > > artifact id would need to match for it to use the same schema and a > higher > > artifact version would need to be provided. I notice that Apicurio has > > rules around compatibility that you can configure, I suppose if we > attempt > > to create an artifact that breaks these rules , then the register schema > > will fail and the associated operation should fail (e.g. an insert). I > have > > not tried this. > > > > > > For inbound messages, using the global id in the header – this targets > one > > version of the schema. I can create different messages on the topic built > > with different schema versions, and I can create different tables in > Flink, > > as long as the reader and writer schemas are compatible as per the > > > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_flink_blob_779459168c46b7b4c600ef52f99a5435f81b9048_flink-2Dformats_flink-2Davro_src_main_java_org_apache_flink_formats_avro_RegistryAvroDeserializationSchema.java-23L109=DwIGaQ=BSDicqBQBDjDI9RkVyTcHQ=a_7ppZzQ4vpQjmqdi73nB22RONTV0tEZsZXcfdiBEOA=ir9ageEmhu8pt03AmvMqEG9MHPp8aZLMBcqU2pmOnyg6yHra8b6IRXFylvH_aP8G=kfPzGTjUx9alvbOMoJoeWEHHQ14qwYxTJXbVWAhYvAc= > > Then this should work. > > > > Does this address your question? > > Kind regards, David. > > > > > > From: Jeyhun Karimov > > Date: Thursday, 21 March 2024 at 21:06 > > To: dev@flink.apache.org > > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format > > Hi David, > > > > Thanks for the FLIP. +1 for it. > > I have a minor comment. > > > > Can you please elaborate more on mechanisms in place to ensure data > > consistency and integrity, particularly in the event of schema conflicts? > > Since each message includes a schema ID for inbound and outbound > messages, > > can you elaborate more on message consistency in the context of schema > > evolution? > > > > Regards, > > Jeyhun > > > > > > > > > > > > On Wed, Mar 20, 2024 at 4:34 PM David Radley > wrote: > > > > > Thank you very much for your feedback Mark. I have made the changes in >
Re: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format
Hi, A gentle nudge. Please could a committer/PMC member raise the Flip for this, Kind regards, David. From: David Radley Date: Monday, 8 April 2024 at 09:40 To: dev@flink.apache.org Subject: [EXTERNAL] RE: [DISCUSS] FLIP-XXX Apicurio-avro format Hi, I have posted a Google Doc [0] to the mailing list for a discussion thread for a Flip proposal to introduce a Apicurio-avro format. The discussions have been resolved, please could a committer/PMC member copy the contents from the Google Doc, and create a FLIP number for this,. as per the process [1], Kind regards, David. [0] https://urldefense.proofpoint.com/v2/url?u=https-3A__docs.google.com_document_d_14LWZPVFQ7F9mryJPdKXb4l32n7B0iWYkcOdEd1xTC7w_edit-3Fusp-3Dsharing=DwIGaQ=BSDicqBQBDjDI9RkVyTcHQ=a_7ppZzQ4vpQjmqdi73nB22RONTV0tEZsZXcfdiBEOA=ir9ageEmhu8pt03AmvMqEG9MHPp8aZLMBcqU2pmOnyg6yHra8b6IRXFylvH_aP8G=pHL2e8waNNtvTDT0a3PQM0bcXrb1Fywv0YW_Ln50jCo= [1] https://urldefense.proofpoint.com/v2/url?u=https-3A__cwiki.apache.org_confluence_display_FLINK_Flink-2BImprovement-2BProposals-23FlinkImprovementProposals-2DCreateyourOwnFLIP=DwIGaQ=BSDicqBQBDjDI9RkVyTcHQ=a_7ppZzQ4vpQjmqdi73nB22RONTV0tEZsZXcfdiBEOA=ir9ageEmhu8pt03AmvMqEG9MHPp8aZLMBcqU2pmOnyg6yHra8b6IRXFylvH_aP8G=_7fvlZYc-gUtkFEhwSz9utYsgbDrUtkHEToTdhtQvQc= From: Jeyhun Karimov Date: Friday, 22 March 2024 at 13:05 To: dev@flink.apache.org Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format Hi David, Thanks a lot for clarification. Sounds good to me. Regards, Jeyhun On Fri, Mar 22, 2024 at 10:54 AM David Radley wrote: > Hi Jeyhun, > Thanks for your feedback. > > So for outbound messages, the message includes the global ID. We register > the schema and match on the artifact id. So if the schema then evolved, > adding a new version, the global ID would still be unique and the same > version would be targeted. If you wanted to change the Flink table > definition in line with a higher version, then you could do this – the > artifact id would need to match for it to use the same schema and a higher > artifact version would need to be provided. I notice that Apicurio has > rules around compatibility that you can configure, I suppose if we attempt > to create an artifact that breaks these rules , then the register schema > will fail and the associated operation should fail (e.g. an insert). I have > not tried this. > > > For inbound messages, using the global id in the header – this targets one > version of the schema. I can create different messages on the topic built > with different schema versions, and I can create different tables in Flink, > as long as the reader and writer schemas are compatible as per the > https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_flink_blob_779459168c46b7b4c600ef52f99a5435f81b9048_flink-2Dformats_flink-2Davro_src_main_java_org_apache_flink_formats_avro_RegistryAvroDeserializationSchema.java-23L109=DwIGaQ=BSDicqBQBDjDI9RkVyTcHQ=a_7ppZzQ4vpQjmqdi73nB22RONTV0tEZsZXcfdiBEOA=ir9ageEmhu8pt03AmvMqEG9MHPp8aZLMBcqU2pmOnyg6yHra8b6IRXFylvH_aP8G=kfPzGTjUx9alvbOMoJoeWEHHQ14qwYxTJXbVWAhYvAc= > Then this should work. > > Does this address your question? > Kind regards, David. > > > From: Jeyhun Karimov > Date: Thursday, 21 March 2024 at 21:06 > To: dev@flink.apache.org > Subject: [EXTERNAL] Re: [DISCUSS] FLIP-XXX Apicurio-avro format > Hi David, > > Thanks for the FLIP. +1 for it. > I have a minor comment. > > Can you please elaborate more on mechanisms in place to ensure data > consistency and integrity, particularly in the event of schema conflicts? > Since each message includes a schema ID for inbound and outbound messages, > can you elaborate more on message consistency in the context of schema > evolution? > > Regards, > Jeyhun > > > > > > On Wed, Mar 20, 2024 at 4:34 PM David Radley wrote: > > > Thank you very much for your feedback Mark. I have made the changes in > the > > latest google document. On reflection I agree with you that the > > globalIdPlacement format configuration should apply to the > deserialization > > as well, so it is declarative. I am also going to have a new > configuration > > option to work with content IDs as well as global IDs. In line with the > > deser Apicurio IdHandler and headerHandlers. > > > > kind regards, David. > > > > > > On 2024/03/20 15:18:37 Mark Nuttall wrote: > > > +1 to this > > > > > > A few small comments: > > > > > > Currently, if users have Avro schemas in an Apicurio Registry (an open > > source Apache 2 licensed schema registry), then the natural way to work > > with those Avro flows is to use the schemas in the Apicurio Repository. > > > 'those Avro flows' ... this is
