> Rather than go though the connect framework, connectors should just
create their own AdminClient instance and create their own topics?
Rather, can the framework be improved to expose an AdminClient ready to
use? Then connectors can use this instance without needing separate
identities/principals and associated configuration (which I totally
understand would be a nightmare). I believe that covers all the use-cases?
I just don't see how the "terrible config situation" is remedied by adding
even more configuration.
Also, I'm not sure I can conceive of a use-case in which a single connector
would need multiple default topic settings *based on the topic name*. Can
you give a real-world example? Is this something you've encountered, or are
you just trying for a flexible design?
Ryanne
On Tue, Sep 11, 2018 at 9:57 PM Gwen Shapira <g...@confluent.io> wrote:
> Hi Ryanne,
>
> Thanks for the feedback!
>
> Can you explain a bit more what you mean by "if we allow connectors to make
> this
> decision, they should have full control of the process."?
>
> I assume you mean, something like:
> Rather than go though the connect framework, connectors should just create
> their own AdminClient instance and create their own topics?
>
> The problem with this approach is that connectors currently don't have
> their own identity (in the authentication/authorization sense). All
> connectors share the framework identity, if the users need to start
> configuring security for both the framework and connect itself, it gets
> messy rather quickly.
> We actually already do the thing I'm imagining you suggested in some
> connectors right now (create AdminClient and configure topics), and we hope
> to use the new framework capability to clean-up the configuration mess this
> has caused. I spent 4 days trying to figure out what a specific connector
> doesn't work, just to find out that you need to give it its own security
> config because it has an AdminClient so the configuration on the framework
> isn't enough.
>
> From my experience with rather large number of customers, there are some
> companies where the topics are controlled by a central team that owns all
> the machinery to create and configure topics (sometimes via gitops,
> kubernetes custom resources, etc) and they would indeed be very surprised
> if a connector suddenly had opinions about topics. There are also teams
> where the application developers feel like they know their data and
> use-case the best and they are in-charge of making all topic-level
> decisions, usually automated by the app itself. Admin client was created
> for those teams and I think they'll appreciate having this capability in
> connect too. Funny thing is, customers who work with one model usually
> can't believe the other model even exists.
>
> I'd love to propose a compromise and suggest that we'll allow this
> functionality in Connect but also give ops teams the option to disable it
> and avoid surprises. But I'm afraid this wont work - too often the defaults
> are just terrible for specific connectors (CDC connectors sometimes need a
> single partition to maintain consistency) and if there is a chance the
> connector preference won't be used, connectors will have to force it via
> admin client which brings us back to the terrible config situation we
> currently have with Admin client.
>
> Gwen
>
>
> On Tue, Sep 11, 2018 at 7:23 PM, Ryanne Dolan <ryannedo...@gmail.com>
> wrote:
>
> > Randall,
> >
> > I have some concerns with this proposal.
> >
> > Firstly, I don't believe it is the job of a connector to configure
> topics,
> > generally, nor for topic-specific settings to hang out in connector
> > configurations. Automatic creation of topics with default settings is an
> > established pattern elsewhere, and I don't think connectors need to
> diverge
> > from this.
> >
> > I agree there are cases where the default settings don't make sense and
> > it'd be nice to override them. But if we allow connectors to make this
> > decision, they should have full control of the process.
> >
> > Some concerns:
> > - I'd expect the cluster's default settings to apply to newly created
> > topics, regardless of who created them. I wouldn't expect source
> connectors
> > to be a special case. In particular, I'd be surprised if Kafka Connect
> were
> > to ignore my cluster's default settings and apply its own defaults.
> > - It will be possible to add a specific topic to this configuration, in
> > which case any reader would expect the topic to have the specified
> > settings. But this will not generally be true. Thus, the configuration
> will
> > end up lying and misleading, and there won't be any indication that the
> > configuration is lying.
> > - Connectors that want to control settings will end up naming topics
> > accordingly. For example, a connector that wants to control the number of
> > partitions would need a bunch of creation rules for 1 partition, 2
> > partitions and so on. This is a bad pattern to establish. A better
> pattern
> > is to let the connector control the number of partitions directly when
> that
> > feature is required.
> > - The proposal introduces 2 new interfaces to control topic creation
> > (configuration rules and TopicSettings), where there is already a
> perfectly
> > good one (AdminClient).
> >
> > Ryanne
> >
> >
> >
> >
> > On Tue, Sep 4, 2018 at 5:08 PM Randall Hauch <rha...@gmail.com> wrote:
> >
> > > Okay, I think I cleaned up the formatting issues in the KIP wiki page.
> > And
> > > while implementing I realized that it'd be helpful to be able to limit
> > via
> > > the connector configuration and the rules which topics are created. I
> > added
> > > the `topic.creation.${ruleName}.policy` behavior, with possible values
> > of
> > > "create" (the default), "autocreate" (to specify that Connect should
> let
> > > the broker autocreate any matching topics), and "fail" (to specify that
> > > Connect should not allow the creation of topics whose names match the
> > > rule's regular expression).
> > >
> > > Let me know what you think. I'd like to start voting soon, but because
> I
> > > made the above change I'll wait a few days.
> > >
> > > Best regards,
> > >
> > > Randall
> > >
> > > On Wed, Aug 29, 2018 at 9:41 PM Randall Hauch <rha...@gmail.com>
> wrote:
> > >
> > > > Thanks, Magesh.
> > > >
> > > > All, I've made a few very minor changes to some JavaDocs and the
> > > > signatures of the name-value pair methods in TopicSettings
> interface. I
> > > > also described as a fifth rejected alternative why this KIP does not
> > > modify
> > > > any topic settings for existing topics. All of these are pretty
> minor,
> > > but
> > > > I'm happy to hear about issues or suggestions.
> > > >
> > > > Since the above changes were very minor, I'll kick off a vote to
> accept
> > > > this KIP unless I hear something in the next day or two. Note that
> this
> > > KIP
> > > > has been around in virtually the exact form for over a year.
> > > >
> > > > Best regards,
> > > >
> > > > Randall
> > > >
> > > > On Wed, Aug 29, 2018 at 9:17 PM Magesh Nandakumar <
> > mage...@confluent.io>
> > > > wrote:
> > > >
> > > >> Randall,
> > > >>
> > > >> I originally thought that this proposal was a config only topic
> > settings
> > > >> and hence made the comment about configs being pass through. I just
> > > >> realized that the connectors can also override and provide the
> > > >> TopicSettings. With that in mind, I think the proposal looks great.
> > > >> Looking forward to the feature.
> > > >>
> > > >> Thanks,
> > > >> Magesh
> > > >>
> > > >> On Tue, Aug 28, 2018 at 8:53 PM Magesh Nandakumar <
> > mage...@confluent.io
> > > >
> > > >> wrote:
> > > >>
> > > >> > I was wondering if it would be much simpler to just do a
> > pass-through
> > > so
> > > >> > that we can support any topic setting added in Kafka without any
> > code
> > > >> > changes in connect. Since these are for topics that will have the
> > > actual
> > > >> > data stream, users might possibly need more flexibility in terms
> of
> > > how
> > > >> the
> > > >> > topics get created.
> > > >> >
> > > >> > Thanks
> > > >> > Magesh
> > > >> >
> > > >> > On Tue, Aug 28, 2018 at 4:56 PM Randall Hauch <rha...@gmail.com>
> > > wrote:
> > > >> >
> > > >> >> Do you think we should support name-value pairs, too?
> > > >> >>
> > > >> >> On Tue, Aug 28, 2018 at 6:41 PM Magesh Nandakumar <
> > > >> mage...@confluent.io>
> > > >> >> wrote:
> > > >> >>
> > > >> >> > Randall,
> > > >> >> >
> > > >> >> > Thanks a lot for the KIP. I think this would be a great
> addition
> > > for
> > > >> >> many
> > > >> >> > source connectors.
> > > >> >> > One clarification I had was regarding the topic settings that
> can
> > > be
> > > >> >> > configured. Is it limited to the setting exposed in the
> > > TopicSettings
> > > >> >> > interface?
> > > >> >> >
> > > >> >> > Thanks
> > > >> >> > Magesh
> > > >> >> >
> > > >> >> > On Tue, Aug 21, 2018 at 7:59 PM Randall Hauch <
> rha...@gmail.com>
> > > >> wrote:
> > > >> >> >
> > > >> >> > > Okay, after much delay let's try this again for AK 2.1. Has
> > > anyone
> > > >> >> found
> > > >> >> > > any concerns? Stephane suggested that we allow updating topic
> > > >> >> > > configurations (everything but partition count). I'm
> > unconvinced
> > > >> that
> > > >> >> > it's
> > > >> >> > > worth the additional complexity in the implementation and the
> > > >> >> > documentation
> > > >> >> > > to explain the behavior. Changing several of the
> topic-specific
> > > >> >> > > configurations have significant impact on broker behavior /
> > > >> >> > functionality,
> > > >> >> > > so IMO we need to proceed more cautiously.
> > > >> >> > >
> > > >> >> > > Stephane, do you have a particular use case in mind for
> > updating
> > > >> topic
> > > >> >> > > configurations on an existing topic?
> > > >> >> > >
> > > >> >> > > Randall
> > > >> >> > >
> > > >> >> > >
> > > >> >> > > On Fri, Jan 26, 2018 at 4:20 PM Randall Hauch <
> > rha...@gmail.com>
> > > >> >> wrote:
> > > >> >> > >
> > > >> >> > > > The KIP deadline for 1.1 has already passed, but I'd like
> to
> > > >> restart
> > > >> >> > this
> > > >> >> > > > discussion so that we make the next release. I've not yet
> > > >> addressed
> > > >> >> the
> > > >> >> > > > previous comment about *existing* topics, but I'll try to
> do
> > > that
> > > >> >> over
> > > >> >> > > the
> > > >> >> > > > next few weeks. Any other comments/suggestions/questions?
> > > >> >> > > >
> > > >> >> > > > Best regards,
> > > >> >> > > >
> > > >> >> > > > Randall
> > > >> >> > > >
> > > >> >> > > > On Thu, Oct 5, 2017 at 12:13 AM, Randall Hauch <
> > > rha...@gmail.com
> > > >> >
> > > >> >> > wrote:
> > > >> >> > > >
> > > >> >> > > >> Oops. Yes, I meant “replication factor”.
> > > >> >> > > >>
> > > >> >> > > >> > On Oct 4, 2017, at 7:18 PM, Ted Yu <yuzhih...@gmail.com
> >
> > > >> wrote:
> > > >> >> > > >> >
> > > >> >> > > >> > Randall:
> > > >> >> > > >> > bq. AdminClient currently allows changing the
> replication
> > > >> >> factory.
> > > >> >> > > >> >
> > > >> >> > > >> > By 'replication factory' did you mean 'replication
> > factor' ?
> > > >> >> > > >> >
> > > >> >> > > >> > Cheers
> > > >> >> > > >> >
> > > >> >> > > >> >> On Wed, Oct 4, 2017 at 9:58 AM, Randall Hauch <
> > > >> rha...@gmail.com
> > > >> >> >
> > > >> >> > > >> wrote:
> > > >> >> > > >> >>
> > > >> >> > > >> >> Currently the KIP's scope is only topics that don't yet
> > > >> exist,
> > > >> >> and
> > > >> >> > we
> > > >> >> > > >> have
> > > >> >> > > >> >> to cognizant of race conditions between tasks with the
> > same
> > > >> >> > > connector.
> > > >> >> > > >> I
> > > >> >> > > >> >> think it is worthwhile to consider whether the KIP's
> > scope
> > > >> >> should
> > > >> >> > > >> expand to
> > > >> >> > > >> >> also address *existing* partitions, though it may not
> be
> > > >> >> > appropriate
> > > >> >> > > to
> > > >> >> > > >> >> have as much control when changing the topic settings
> for
> > > an
> > > >> >> > existing
> > > >> >> > > >> >> topic. For example, changing the number of partitions
> > > (which
> > > >> the
> > > >> >> > KIP
> > > >> >> > > >> >> considers a "topic-specific setting" even though
> > > technically
> > > >> it
> > > >> >> is
> > > >> >> > > not)
> > > >> >> > > >> >> shouldn't be done blindly due to the partitioning
> > impacts,
> > > >> and
> > > >> >> IIRC
> > > >> >> > > you
> > > >> >> > > >> >> can't reduce them (which we could verify before
> > applying).
> > > >> >> Also, I
> > > >> >> > > >> don't
> > > >> >> > > >> >> think the AdminClient currently allows changing the
> > > >> replication
> > > >> >> > > >> factory. I
> > > >> >> > > >> >> think changing the topic configs is less problematic
> both
> > > >> from
> > > >> >> what
> > > >> >> > > >> makes
> > > >> >> > > >> >> sense for connectors to verify/change and from what the
> > > >> >> AdminClient
> > > >> >> > > >> >> supports.
> > > >> >> > > >> >>
> > > >> >> > > >> >> Even if we decide that it's not appropriate to change
> the
> > > >> >> settings
> > > >> >> > on
> > > >> >> > > >> an
> > > >> >> > > >> >> existing topic, I do think it's advantageous to at
> least
> > > >> notify
> > > >> >> the
> > > >> >> > > >> >> connector (or task) prior to the first record sent to a
> > > given
> > > >> >> topic
> > > >> >> > > so
> > > >> >> > > >> that
> > > >> >> > > >> >> the connector can fail or issue a warning if it doesn't
> > > meet
> > > >> its
> > > >> >> > > >> >> requirements.
> > > >> >> > > >> >>
> > > >> >> > > >> >> Best regards,
> > > >> >> > > >> >>
> > > >> >> > > >> >> Randall
> > > >> >> > > >> >>
> > > >> >> > > >> >> On Wed, Oct 4, 2017 at 12:52 AM, Stephane Maarek <
> > > >> >> > > >> >> steph...@simplemachines.com.au> wrote:
> > > >> >> > > >> >>
> > > >> >> > > >> >>> Hi Randall,
> > > >> >> > > >> >>>
> > > >> >> > > >> >>> Thanks for the KIP. I like it
> > > >> >> > > >> >>> What happens when the target topic is already created
> > but
> > > >> the
> > > >> >> > > configs
> > > >> >> > > >> do
> > > >> >> > > >> >>> not match?
> > > >> >> > > >> >>> i.e. wrong RF, num partitions, or missing / additional
> > > >> configs?
> > > >> >> > Will
> > > >> >> > > >> you
> > > >> >> > > >> >>> attempt to apply the necessary changes or throw an
> > error?
> > > >> >> > > >> >>>
> > > >> >> > > >> >>> Thanks!
> > > >> >> > > >> >>> Stephane
> > > >> >> > > >> >>>
> > > >> >> > > >> >>>
> > > >> >> > > >> >>> On 24/5/17, 5:59 am, "Mathieu Fenniak" <
> > > >> >> > > mathieu.fenn...@replicon.com
> > > >> >> > > >> >
> > > >> >> > > >> >>> wrote:
> > > >> >> > > >> >>>
> > > >> >> > > >> >>> Ah, yes, I see you a highlighted part that
> should've
> > > made
> > > >> >> this
> > > >> >> > > >> clear
> > > >> >> > > >> >>> to me the first read. :-) Much clearer now!
> > > >> >> > > >> >>>
> > > >> >> > > >> >>> By the way, enjoyed your Debezium talk in NYC.
> > > >> >> > > >> >>>
> > > >> >> > > >> >>> Looking forward to this Kafka Connect change; it
> will
> > > >> allow
> > > >> >> me
> > > >> >> > to
> > > >> >> > > >> >>> remove a post-deployment tool that I hacked
> together
> > > for
> > > >> the
> > > >> >> > > >> purpose
> > > >> >> > > >> >>> of ensuring auto-created topics have the right
> > config.
> > > >> >> > > >> >>>
> > > >> >> > > >> >>> Mathieu
> > > >> >> > > >> >>>
> > > >> >> > > >> >>>
> > > >> >> > > >> >>> On Tue, May 23, 2017 at 11:38 AM, Randall Hauch <
> > > >> >> > > rha...@gmail.com>
> > > >> >> > > >> >>> wrote:
> > > >> >> > > >> >>>> Thanks for the quick feedback, Mathieu. Yes, the
> first
> > > >> >> > > >> >> configuration
> > > >> >> > > >> >>> rule
> > > >> >> > > >> >>>> whose regex matches will be applied, and no other
> rules
> > > >> will
> > > >> >> be
> > > >> >> > > >> >>> used. I've
> > > >> >> > > >> >>>> updated the KIP to try to make this more clear, but
> let
> > > me
> > > >> >> know
> > > >> >> > if
> > > >> >> > > >> >>> it's
> > > >> >> > > >> >>>> still not clear.
> > > >> >> > > >> >>>>
> > > >> >> > > >> >>>> Best regards,
> > > >> >> > > >> >>>>
> > > >> >> > > >> >>>> Randall
> > > >> >> > > >> >>>>
> > > >> >> > > >> >>>> On Tue, May 23, 2017 at 10:07 AM, Mathieu Fenniak <
> > > >> >> > > >> >>>> mathieu.fenn...@replicon.com> wrote:
> > > >> >> > > >> >>>>
> > > >> >> > > >> >>>>> Hi Randall,
> > > >> >> > > >> >>>>>
> > > >> >> > > >> >>>>> Awesome, very much looking forward to this.
> > > >> >> > > >> >>>>>
> > > >> >> > > >> >>>>> It isn't 100% clear from the KIP how multiple
> > > config-based
> > > >> >> rules
> > > >> >> > > >> >>> would
> > > >> >> > > >> >>>>> be applied; it looks like the first configuration
> rule
> > > >> whose
> > > >> >> > regex
> > > >> >> > > >> >>>>> matches the topic name will be used, and no other
> > rules
> > > >> will
> > > >> >> be
> > > >> >> > > >> >>>>> applied. Is that correct? (I wasn't sure if it
> might
> > > >> >> cascade
> > > >> >> > > >> >>>>> together multiple matching rules...)
> > > >> >> > > >> >>>>>
> > > >> >> > > >> >>>>> Looks great,
> > > >> >> > > >> >>>>>
> > > >> >> > > >> >>>>> Mathieu
> > > >> >> > > >> >>>>>
> > > >> >> > > >> >>>>>
> > > >> >> > > >> >>>>> On Mon, May 22, 2017 at 1:43 PM, Randall Hauch <
> > > >> >> > rha...@gmail.com>
> > > >> >> > > >> >>> wrote:
> > > >> >> > > >> >>>>>> Hi, all.
> > > >> >> > > >> >>>>>>
> > > >> >> > > >> >>>>>> We recently added the ability for Kafka Connect to
> > > create
> > > >> >> > > >> >>> *internal*
> > > >> >> > > >> >>>>> topics
> > > >> >> > > >> >>>>>> using the new AdminClient, but it still would be
> > great
> > > if
> > > >> >> Kafka
> > > >> >> > > >> >>> Connect
> > > >> >> > > >> >>>>>> could do this for new topics that result from
> source
> > > >> >> connector
> > > >> >> > > >> >>> records.
> > > >> >> > > >> >>>>>> I've outlined an approach to do this in "KIP-158
> > Kafka
> > > >> >> Connect
> > > >> >> > > >> >>> should
> > > >> >> > > >> >>>>> allow
> > > >> >> > > >> >>>>>> source connectors to set topic-specific settings
> for
> > > new
> > > >> >> > > >> >> topics".
> > > >> >> > > >> >>>>>>
> > > >> >> > > >> >>>>>> *
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > >> >> > > >> >>>>> 158%3A+Kafka+Connect+should+
> > allow+source+connectors+to+
> > > >> >> > > >> >>>>> set+topic-specific+settings+for+new+topics
> > > >> >> > > >> >>>>>> <
> > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-
> > > >> >> > > >> >>>>> 158%3A+Kafka+Connect+should+
> > allow+source+connectors+to+
> > > >> >> > > >> >>>>> set+topic-specific+settings+for+new+topics>*
> > > >> >> > > >> >>>>>>
> > > >> >> > > >> >>>>>> Please take a look and provide feedback. Thanks!
> > > >> >> > > >> >>>>>>
> > > >> >> > > >> >>>>>> Best regards,
> > > >> >> > > >> >>>>>>
> > > >> >> > > >> >>>>>> Randall
> > > >> >> > > >> >>>>>
> > > >> >> > > >> >>>
> > > >> >> > > >> >>>
> > > >> >> > > >> >>>
> > > >> >> > > >> >>>
> > > >> >> > > >> >>
> > > >> >> > > >>
> > > >> >> > > >
> > > >> >> > > >
> > > >> >> > >
> > > >> >> >
> > > >> >>
> > > >> >
> > > >>
> > > >
> > >
> >
>
>
>
> --
> *Gwen Shapira*
> Product Manager | Confluent
> 650.450.2760 <(650)%20450-2760> | @gwenshap
> Follow us: Twitter <https://twitter.com/ConfluentInc> | blog
> <http://www.confluent.io/blog>
>
