If an admin team gives another the right to create and configure their
topic, I can't see why specifying topic configuration in connector
properties wouldn't be a great idea. Configuration as code, repeatable,
central and automated, easy iteration. Sign me in
On Wed., 12 Sep. 2018, 4:57 am Gwen Shapira, <g...@confluent.io> wrote:
> Hi Ryanne,
>
> Thanks for the feedback!
>
> Can you explain a bit more what you mean by "if we allow connectors to make
> this
> decision, they should have full control of the process."?
>
> I assume you mean, something like:
> Rather than go though the connect framework, connectors should just create
> their own AdminClient instance and create their own topics?
>
> The problem with this approach is that connectors currently don't have
> their own identity (in the authentication/authorization sense). All
> connectors share the framework identity, if the users need to start
> configuring security for both the framework and connect itself, it gets
> messy rather quickly.
> We actually already do the thing I'm imagining you suggested in some
> connectors right now (create AdminClient and configure topics), and we hope
> to use the new framework capability to clean-up the configuration mess this
> has caused. I spent 4 days trying to figure out what a specific connector
> doesn't work, just to find out that you need to give it its own security
> config because it has an AdminClient so the configuration on the framework
> isn't enough.
>
> From my experience with rather large number of customers, there are some
> companies where the topics are controlled by a central team that owns all
> the machinery to create and configure topics (sometimes via gitops,
> kubernetes custom resources, etc) and they would indeed be very surprised
> if a connector suddenly had opinions about topics. There are also teams
> where the application developers feel like they know their data and
> use-case the best and they are in-charge of making all topic-level
> decisions, usually automated by the app itself. Admin client was created
> for those teams and I think they'll appreciate having this capability in
> connect too. Funny thing is, customers who work with one model usually
> can't believe the other model even exists.
>
> I'd love to propose a compromise and suggest that we'll allow this
> functionality in Connect but also give ops teams the option to disable it
> and avoid surprises. But I'm afraid this wont work - too often the defaults
> are just terrible for specific connectors (CDC connectors sometimes need a
> single partition to maintain consistency) and if there is a chance the
> connector preference won't be used, connectors will have to force it via
> admin client which brings us back to the terrible config situation we
> currently have with Admin client.
>
> Gwen
>
>
> On Tue, Sep 11, 2018 at 7:23 PM, Ryanne Dolan <ryannedo...@gmail.com>
> wrote:
>
> > Randall,
> >
> > I have some concerns with this proposal.
> >
> > Firstly, I don't believe it is the job of a connector to configure
> topics,
> > generally, nor for topic-specific settings to hang out in connector
> > configurations. Automatic creation of topics with default settings is an
> > established pattern elsewhere, and I don't think connectors need to
> diverge
> > from this.
> >
> > I agree there are cases where the default settings don't make sense and
> > it'd be nice to override them. But if we allow connectors to make this
> > decision, they should have full control of the process.
> >
> > Some concerns:
> > - I'd expect the cluster's default settings to apply to newly created
> > topics, regardless of who created them. I wouldn't expect source
> connectors
> > to be a special case. In particular, I'd be surprised if Kafka Connect
> were
> > to ignore my cluster's default settings and apply its own defaults.
> > - It will be possible to add a specific topic to this configuration, in
> > which case any reader would expect the topic to have the specified
> > settings. But this will not generally be true. Thus, the configuration
> will
> > end up lying and misleading, and there won't be any indication that the
> > configuration is lying.
> > - Connectors that want to control settings will end up naming topics
> > accordingly. For example, a connector that wants to control the number of
> > partitions would need a bunch of creation rules for 1 partition, 2
> > partitions and so on. This is a bad pattern to establish. A better
> pattern
> > is to let the connector control the number of partitions directly when
> that
> > feature is required.
> > - The proposal introduces 2 new interfaces to control topic creation
> > (configuration rules and TopicSettings), where there is already a
> perfectly
> > good one (AdminClient).
> >
> > Ryanne
> >
> >
> >
> >
> > On Tue, Sep 4, 2018 at 5:08 PM Randall Hauch <rha...@gmail.com> wrote:
> >
> > > Okay, I think I cleaned up the formatting issues in the KIP wiki page.
> > And
> > > while implementing I realized that it'd be helpful to be able to limit
> > via
> > > the connector configuration and the rules which topics are created. I
> > added
> > > the `topic.creation.${ruleName}.policy` behavior, with possible values
> > of
> > > "create" (the default), "autocreate" (to specify that Connect should
> let
> > > the broker autocreate any matching topics), and "fail" (to specify that
> > > Connect should not allow the creation of topics whose names match the
> > > rule's regular expression).
> > >
> > > Let me know what you think. I'd like to start voting soon, but because
> I
> > > made the above change I'll wait a few days.
> > >
> > > Best regards,
> > >
> > > Randall
> > >
> > > On Wed, Aug 29, 2018 at 9:41 PM Randall Hauch <rha...@gmail.com>
> wrote:
> > >
> > > > Thanks, Magesh.
> > > >
> > > > All, I've made a few very minor changes to some JavaDocs and the
> > > > signatures of the name-value pair methods in TopicSettings
> interface. I
> > > > also described as a fifth rejected alternative why this KIP does not
> > > modify
> > > > any topic settings for existing topics. All of these are pretty
> minor,
> > > but
> > > > I'm happy to hear about issues or suggestions.
> > > >
> > > > Since the above changes were very minor, I'll kick off a vote to
> accept
> > > > this KIP unless I hear something in the next day or two. Note that
> this
> > > KIP
> > > > has been around in virtually the exact form for over a year.
> > > >
> > > > Best regards,
> > > >
> > > > Randall
> > > >
> > > > On Wed, Aug 29, 2018 at 9:17 PM Magesh Nandakumar <
> > mage...@confluent.io>
> > > > wrote:
> > > >
> > > >> Randall,
> > > >>
> > > >> I originally thought that this proposal was a config only topic
> > settings
> > > >> and hence made the comment about configs being pass through. I just
> > > >> realized that the connectors can also override and provide the
> > > >> TopicSettings. With that in mind, I think the proposal looks great.
> > > >> Looking forward to the feature.
> > > >>
> > > >> Thanks,
> > > >> Magesh
> > > >>
> > > >> On Tue, Aug 28, 2018 at 8:53 PM Magesh Nandakumar <
> > mage...@confluent.io
> > > >
> > > >> wrote:
> > > >>
> > > >> > I was wondering if it would be much simpler to just do a
> > pass-through
> > > so
> > > >> > that we can support any topic setting added in Kafka without any
> > code
> > > >> > changes in connect. Since these are for topics that will have the
> > > actual
> > > >> > data stream, users might possibly need more flexibility in terms
> of
> > > how
> > > >> the
> > > >> > topics get created.
> > > >> >
> > > >> > Thanks
> > > >> > Magesh
> > > >> >
> > > >> > On Tue, Aug 28, 2018 at 4:56 PM Randall Hauch <rha...@gmail.com>
> > > wrote:
> > > >> >
> > > >> >> Do you think we should support name-value pairs, too?
> > > >> >>
> > > >> >> On Tue, Aug 28, 2018 at 6:41 PM Magesh Nandakumar <
> > > >> mage...@confluent.io>
> > > >> >> wrote:
> > > >> >>
> > > >> >> > Randall,
> > > >> >> >
> > > >> >> > Thanks a lot for the KIP. I think this would be a great
> addition
> > > for
> > > >> >> many
> > > >> >> > source connectors.
> > > >> >> > One clarification I had was regarding the topic settings that
> can
> > > be
> > > >> >> > configured. Is it limited to the setting exposed in the
> > > TopicSettings
> > > >> >> > interface?
> > > >> >> >
> > > >> >> > Thanks
> > > >> >> > Magesh
> > > >> >> >
> > > >> >> > On Tue, Aug 21, 2018 at 7:59 PM Randall Hauch <
> rha...@gmail.com>
> > > >> wrote:
> > > >> >> >
> > > >> >> > > Okay, after much delay let's try this again for AK 2.1. Has
> > > anyone
> > > >> >> found
> > > >> >> > > any concerns? Stephane suggested that we allow updating topic
> > > >> >> > > configurations (everything but partition count). I'm
> > unconvinced
> > > >> that
> > > >> >> > it's
> > > >> >> > > worth the additional complexity in the implementation and the
> > > >> >> > documentation
> > > >> >> > > to explain the behavior. Changing several of the
> topic-specific
> > > >> >> > > configurations have significant impact on broker behavior /
> > > >> >> > functionality,
> > > >> >> > > so IMO we need to proceed more cautiously.
> > > >> >> > >
> > > >> >> > > Stephane, do you have a particular use case in mind for
> > updating
> > > >> topic
> > > >> >> > > configurations on an existing topic?
> > > >> >> > >
> > > >> >> > > Randall
> > > >> >> > >
> > > >> >> > >
> > > >> >> > > On Fri, Jan 26, 2018 at 4:20 PM Randall Hauch <
> > rha...@gmail.com>
> > > >> >> wrote:
> > > >> >> > >
> > > >> >> > > > The KIP deadline for 1.1 has already passed, but I'd like
> to
> > > >> restart
> > > >> >> > this
> > > >> >> > > > discussion so that we make the next release. I've not yet
> > > >> addressed
> > > >> >> the
> > > >> >> > > > previous comment about *existing* topics, but I'll try to
> do
> > > that
> > > >> >> over
> > > >> >> > > the
> > > >> >> > > > next few weeks. Any other comments/suggestions/questions?
> > > >> >> > > >
> > > >> >> > > > Best regards,
> > > >> >> > > >
> > > >> >> > > > Randall
> > > >> >> > > >
> > > >> >> > > > On Thu, Oct 5, 2017 at 12:13 AM, Randall Hauch <
> > > rha...@gmail.com
> > > >> >
> > > >> >> > wrote:
> > > >> >> > > >
> > > >> >> > > >> Oops. Yes, I meant “replication factor”.
> > > >> >> > > >>
> > > >> >> > > >> > On Oct 4, 2017, at 7:18 PM, Ted Yu <yuzhih...@gmail.com
> >
> > > >> wrote:
> > > >> >> > > >> >
> > > >> >> > > >> > Randall:
> > > >> >> > > >> > bq. AdminClient currently allows changing the
> replication
> > > >> >> factory.
> > > >> >> > > >> >
> > > >> >> > > >> > By 'replication factory' did you mean 'replication
> > factor' ?
> > > >> >> > > >> >
> > > >> >> > > >> > Cheers
> > > >> >> > > >> >
> > > >> >> > > >> >> On Wed, Oct 4, 2017 at 9:58 AM, Randall Hauch <
> > > >> rha...@gmail.com
> > > >> >> >
> > > >> >> > > >> wrote:
> > > >> >> > > >> >>
> > > >> >> > > >> >> Currently the KIP's scope is only topics that don't yet
> > > >> exist,
> > > >> >> and
> > > >> >> > we
> > > >> >> > > >> have
> > > >> >> > > >> >> to cognizant of race conditions between tasks with the
> > same
> > > >> >> > > connector.
> > > >> >> > > >> I
> > > >> >> > > >> >> think it is worthwhile to consider whether the KIP's
> > scope
> > > >> >> should
> > > >> >> > > >> expand to
> > > >> >> > > >> >> also address *existing* partitions, though it may not
> be
> > > >> >> > appropriate
> > > >> >> > > to
> > > >> >> > > >> >> have as much control when changing the topic settings
> for
> > > an
> > > >> >> > existing
> > > >> >> > > >> >> topic. For example, changing the number of partitions
> > > (which
> > > >> the
> > > >> >> > KIP
> > > >> >> > > >> >> considers a "topic-specific setting" even though
> > > technically
> > > >> it
> > > >> >> is
> > > >> >> > > not)
> > > >> >> > > >> >> shouldn't be done blindly due to the partitioning
> > impacts,
> > > >> and
> > > >> >> IIRC
> > > >> >> > > you
> > > >> >> > > >> >> can't reduce them (which we could verify before
> > applying).
> > > >> >> Also, I
> > > >> >> > > >> don't
> > > >> >> > > >> >> think the AdminClient currently allows changing the
> > > >> replication
> > > >> >> > > >> factory. I
> > > >> >> > > >> >> think changing the topic configs is less problematic
> both
> > > >> from
> > > >> >> what
> > > >> >> > > >> makes
> > > >> >> > > >> >> sense for connectors to verify/change and from what the
> > > >> >> AdminClient
> > > >> >> > > >> >> supports.
> > > >> >> > > >> >>
> > > >> >> > > >> >> Even if we decide that it's not appropriate to change
> the
> > > >> >> settings
> > > >> >> > on
> > > >> >> > > >> an
> > > >> >> > > >> >> existing topic, I do think it's advantageous to at
> least
> > > >> notify
> > > >> >> the
> > > >> >> > > >> >> connector (or task) prior to the first record sent to a
> > > given
> > > >> >> topic
> > > >> >> > > so
> > > >> >> > > >> that
> > > >> >> > > >> >> the connector can fail or issue a warning if it doesn't
> > > meet
> > > >> its
> > > >> >> > > >> >> requirements.
> > > >> >> > > >> >>
> > > >> >> > > >> >> Best regards,
> > > >> >> > > >> >>
> > > >> >> > > >> >> Randall
> > > >> >> > > >> >>
> > > >> >> > > >> >> On Wed, Oct 4, 2017 at 12:52 AM, Stephane Maarek <
> > > >> >> > > >> >> steph...@simplemachines.com.au> wrote:
> > > >> >> > > >> >>
> > > >> >> > > >> >>> Hi Randall,
> > > >> >> > > >> >>>
> > > >> >> > > >> >>> Thanks for the KIP. I like it
> > > >> >> > > >> >>> What happens when the target topic is already created
> > but
> > > >> the
> > > >> >> > > configs
> > > >> >> > > >> do
> > > >> >> > > >> >>> not match?
> > > >> >> > > >> >>> i.e. wrong RF, num partitions, or missing / additional
> > > >> configs?
> > > >> >> > Will
> > > >> >> > > >> you
> > > >> >> > > >> >>> attempt to apply the necessary changes or throw an
> > error?
> > > >> >> > > >> >>>
> > > >> >> > > >> >>> Thanks!
> > > >> >> > > >> >>> Stephane
> > > >> >> > > >> >>>
> > > >> >> > > >> >>>
> > > >> >> > > >> >>> On 24/5/17, 5:59 am, "Mathieu Fenniak" <
> > > >> >> > > mathieu.fenn...@replicon.com
> > > >> >> > > >> >
> > > >> >> > > >> >>> wrote:
> > > >> >> > > >> >>>
> > > >> >> > > >> >>> Ah, yes, I see you a highlighted part that
> should've
> > > made
> > > >> >> this
> > > >> >> > > >> clear
> > > >> >> > > >> >>> to me the first read. :-) Much clearer now!
> > > >> >> > > >> >>>
> > > >> >> > > >> >>> By the way, enjoyed your Debezium talk in NYC.
> > > >> >> > > >> >>>
> > > >> >> > > >> >>> Looking forward to this Kafka Connect change; it
> will
> > > >> allow
> > > >> >> me
> > > >> >> > to
> > > >> >> > > >> >>> remove a post-deployment tool that I hacked
> together
> > > for
> > > >> the
> > > >> >> > > >> purpose
> > > >> >> > > >> >>> of ensuring auto-created topics have the right
> > config.
> > > >> >> > > >> >>>
> > > >> >> > > >> >>> Mathieu
> > > >> >> > > >> >>>
> > > >> >> > > >> >>>
> > > >> >> > > >> >>> On Tue, May 23, 2017 at 11:38 AM, Randall Hauch <
> > > >> >> > > rha...@gmail.com>
> > > >> >> > > >> >>> wrote:
> > > >> >> > > >> >>>> Thanks for the quick feedback, Mathieu. Yes, the
> first
> > > >> >> > > >> >> configuration
> > > >> >> > > >> >>> rule
> > > >> >> > > >> >>>> whose regex matches will be applied, and no other
> rules
> > > >> will
> > > >> >> be
> > > >> >> > > >> >>> used. I've
> > > >> >> > > >> >>>> updated the KIP to try to make this more clear, but
> let
> > > me
> > > >> >> know
> > > >> >> > if
> > > >> >> > > >> >>> it's
> > > >> >> > > >> >>>> still not clear.
> > > >> >> > > >> >>>>
> > > >> >> > > >> >>>> Best regards,
> > > >> >> > > >> >>>>
> > > >> >> > > >> >>>> Randall
> > > >> >> > > >> >>>>
> > > >> >> > > >> >>>> On Tue, May 23, 2017 at 10:07 AM, Mathieu Fenniak <
> > > >> >> > > >> >>>> mathieu.fenn...@replicon.com> wrote:
> > > >> >> > > >> >>>>
> > > >> >> > > >> >>>>> Hi Randall,
> > > >> >> > > >> >>>>>
> > > >> >> > > >> >>>>> Awesome, very much looking forward to this.
> > > >> >> > > >> >>>>>
> > > >> >> > > >> >>>>> It isn't 100% clear from the KIP how multiple
> > > config-based
> > > >> >> rules
> > > >> >> > > >> >>> would
> > > >> >> > > >> >>>>> be applied; it looks like the first configuration
> rule
> > > >> whose
> > > >> >> > regex
> > > >> >> > > >> >>>>> matches the topic name will be used, and no other
> > rules
> > > >> will
> > > >> >> be
> > > >> >> > > >> >>>>> applied. Is that correct? (I wasn't sure if it
> might
> > > >> >> cascade
> > > >> >> > > >> >>>>> together multiple matching rules...)
> > > >> >> > > >> >>>>>
> > > >> >> > > >> >>>>> Looks great,
> > > >> >> > > >> >>>>>
> > > >> >> > > >> >>>>> Mathieu
> > > >> >> > > >> >>>>>
> > > >> >> > > >> >>>>>
> > > >> >> > > >> >>>>> On Mon, May 22, 2017 at 1:43 PM, Randall Hauch <
> > > >> >> > rha...@gmail.com>
> > > >> >> > > >> >>> wrote:
> > > >> >> > > >> >>>>>> Hi, all.
> > > >> >> > > >> >>>>>>
> > > >> >> > > >> >>>>>> We recently added the ability for Kafka Connect to
> > > create
> > > >> >> > > >> >>> *internal*
> > > >> >> > > >> >>>>> topics
> > > >> >> > > >> >>>>>> using the new AdminClient, but it still would be
> > great
> > > if
> > > >> >> Kafka
> > > >> >> > > >> >>> Connect
> > > >> >> > > >> >>>>>> could do this for new topics that result from
> source
> > > >> >> connector
> > > >> >> > > >> >>> records.
> > > >> >> > > >> >>>>>> I've outlined an approach to do this in "KIP-158
> > Kafka
> > > >> >> Connect
> > > >> >> > > >> >>> should
> > > >> >> > > >> >>>>> allow
> > > >> >> > > >> >>>>>> source connectors to set topic-specific settings
> for
> > > new
> > > >> >> > > >> >> topics".
> > > >> >> > > >> >>>>>>
> > > >> >> > > >> >>>>>> *
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > >> >> > > >> >>>>> 158%3A+Kafka+Connect+should+
> > allow+source+connectors+to+
> > > >> >> > > >> >>>>> set+topic-specific+settings+for+new+topics
> > > >> >> > > >> >>>>>> <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > >> >> > > >> >>>>> 158%3A+Kafka+Connect+should+
> > allow+source+connectors+to+
> > > >> >> > > >> >>>>> set+topic-specific+settings+for+new+topics>*
> > > >> >> > > >> >>>>>>
> > > >> >> > > >> >>>>>> Please take a look and provide feedback. Thanks!
> > > >> >> > > >> >>>>>>
> > > >> >> > > >> >>>>>> Best regards,
> > > >> >> > > >> >>>>>>
> > > >> >> > > >> >>>>>> Randall
> > > >> >> > > >> >>>>>
> > > >> >> > > >> >>>
> > > >> >> > > >> >>>
> > > >> >> > > >> >>>
> > > >> >> > > >> >>>
> > > >> >> > > >> >>
> > > >> >> > > >>
> > > >> >> > > >
> > > >> >> > > >
> > > >> >> > >
> > > >> >> >
> > > >> >>
> > > >> >
> > > >>
> > > >
> > >
> >
>
>
>
> --
> *Gwen Shapira*
> Product Manager | Confluent
> 650.450.2760 | @gwenshap
> Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
> <http://www.confluent.io/blog>
>
