Hi Chris,
Thanks for the KIP!
1) With this mechanism it seems Connect administrators have to know in
advanced the names of connectors that will be running. As far as I can
tell, static.connectors and static.tasks will require restarting the
worker to change. Also you don't specify if these configurations can
accept wildcards or not. This could be useful for tasks as it's hard
to know in advance how many tasks will exist, especially as connectors
can in theory ignore tasks.max (I wonder if we should make the runtime
enforce that config).
2) The KIP does not explicitly mention a mechanism for management
systems to figure out what to do. As far as I can tell (and looking at
the proposal in Strimzi), these systems will have to poll the REST API
to do so. If I understand correctly, management systems would first
have to poll the GET /connectors and then GET /connectors/{name}/tasks
for each connector, is that right?
3) In the Strimzi proposal you mention adding custom fields to the
Connector resource to instruct it what to do. I wonder if these should
be added directly in the connector configuration, rather than just
exist in the management system. That way all the data necessary to
make decisions would be available in the same place, the REST API.
4) This is not something you mention in the KIP and I don't think it's
a blocker for this KIP but I wonder if you thought about rack
awareness too. I'm bringing it as I think it's a similar concern when
it comes to deciding where to assign tasks.
Thanks,
Mickael
On Tue, Oct 10, 2023 at 8:39 AM Tom Bentley <tbent...@redhat.com> wrote:
>
> Hi Greg,
>
> Many thanks for the KIP. Here are a few initial questions
>
> 1. Incomplete sentence: "But Connect is not situated to be able to manage
> resources directly, as workers are given a fixed "
> 2. You explain how sessioned is now a subset of static, but what happens in
> a cluster where some workers are using static and some are using either
> eager or compatible?
> 3. "Assign each unassigned static job to a static worker which specifies
> that job, choosing arbitrarily if there are multiple valid workers." I
> think there might be less ambiguous words than "arbitrary" to specify this
> behaviour. Hashing the task name would _appear_ pretty arbitrary to the
> user, but would be deterministic. Picking at random would be
> non-deterministic. Even better if you have a rationale.
> 4. You don't describe how a user, or automated system, starting with a set
> of connectors, should find out the tasks that they want to run. This
> relates to the contract of
> org.apache.kafka.connect.connector.Connector#taskConfigs(int maxTasks).
> AFAIK (please correct me if I'm wrong, because it's a long time since I
> looked at this code) there's nothing that validates that the returned list
> has at most the `maxTasks` and connectors can, of course, return fewer than
> that many tasks. So without deeper knowledge of a particular connector it's
> not clear to the user/operator how to configure their static workers and
> static assignments.
> 5. Is there a lurking assumption that task indices are stable? E.g. that
> the task with index 3 will always be the resource-intensive one. I can see
> that that would often be a reliable assumption, but it's not clear to me
> that it is always the case.
>
> Thanks again,
>
> Tom
>
> On Fri, 6 Oct 2023 at 12:36, Greg Harris <greg.har...@aiven.io.invalid>
> wrote:
>
> > Hey everyone!
> >
> > I'd like to propose an improvement to Kafka Connect's scheduling
> > algorithm, with the goal of improving the operability of connect
> > clusters through resource isolation.
> >
> > Find the KIP here:
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-987%3A+Connect+Static+Assignments
> >
> > This feature is primarily intended to be consumed by cluster
> > management systems, so I've opened a sister proposal in the Strimzi
> > project that uses this feature to provide user-facing resource
> > isolation. The base feature is generic, so it is usable manually and
> > with cluster management systems other than Strimzi.
> >
> > Find the proposal here: https://github.com/strimzi/proposals/pull/96
> >
> > Thanks!
> > Greg Harris
> >
> >
