Hi Mario, Thanks for the KIP! I think that using the same data model inside and outside of connect is valuable, and is a good motivation for this effort.
However, I have seen many situations where the existing data model is insufficient for Connect plugin developers. * It's a closed type system which cannot have new "physical" types added by third-party developers * There are carve-outs in the implementation for framework logical types (java.util.Date etc) * Compatibility between components isn't managed: custom types can be misinterpreted as dumb physical types which is undesirable to users, who may prefer to fail-fast instead. I think that we should evolve the Connect data model in the future to resolve some or all of these problems. This will be easier and less risky if we change the data model first, before expanding it to non-Connect users. Another way to satisfy the requirements of your KIP would be to adopt an existing external data model in Connect. If there was a suitably general data model out there, we could promote it within the Connect ecosystem and deprecate our current model. Thanks, Greg On Thu, Jan 9, 2025 at 12:19 AM Mario Fiore Vitale <mvit...@redhat.com> wrote: > Hi all, > > I just want to bring the discussion up since it was open during the > Christmas holidays. > > Any other considerations? > > Thanks, > Mario. > > On Sat, Dec 28, 2024 at 3:04 PM Mario Fiore Vitale <mvital...@gmail.com> > wrote: > > > Hey Hector, > > > > the aim of this KIP is to separate the Connect data (Struct, Schema, > etc.) > > so it can be used without having a dependency con connect-api. > > > > So the proposal is more related to economics. > > > > As Kafka itself became an API during the years, maybe also the Connect > data > > can be used as a general data format not only for Connect. > > > > Is it more clear now? > > > > Il ven 20 dic 2024, 22:20 Hector Geraldino (BLOOMBERG/ 919 3RD A) < > > hgerald...@bloomberg.net> ha scritto: > > > > > Hey Mario, > > > > > > From a quick read of this KIP, it's not super clear to me what problem > it > > > is trying to solve. The motivation states that "current implementation > > > restricts the usage of data classes", and I'm not sure what you mean by > > > that. > > > > > > Can you maybe add one or two examples of things that are not possible > to > > > do today and could be achieved by this refactor. If this proposal is > more > > > about improving the ergonomics of users of these classes, that's also > > > valuable IMO. > > > > > > From: dev@kafka.apache.org At: 12/18/24 08:32:28 UTC-5:00To: > > > dev@kafka.apache.org > > > Subject: [DISCUSS] KIP-1122: Create a dedicated data module for Kafka > > > Connect data classes > > > > > > Hi Everyone, > > > > > > I would like to start a discussion on KIP-1122: Create a dedicated data > > > module for Kafka Connect data classes [1]. > > > > > > [1] > > > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-1122%3A+Create+a+dedicated > > > +data+module+for+Kafka+Connect+data+classes > > > > > > Regards, > > > -- > > > Mario Fiore Vitale > > > > > > > > > > > > > > -- > > Mario Fiore Vitale > > Senior Software Engineer > > Red Hat <https://www.redhat.com/> > <https://www.redhat.com/> >
