Re: [DISCUSS] KIP-304: Connect runtime mode improvements for container platforms

Long wall of text: --- Hi Ewen > There are 2 aspects to this. The first is rebalancing. There's actually nothing that fundamentally requires every rebalance operation to be stop-the-world. I've had some thoughts about allowing partial and incremental rebalancing floating around for awhile.

Re: [DISCUSS] KIP-304: Connect runtime mode improvements for container platforms

Hi Ewen thanks for the response, I totally agree that this KIP touches upon multiple rather orthogonal issues, I had a hard time deciding on the name and scope for it :) Let me sum up the biggest pain points that we're facing with distributed mode: * By far the biggest issue we have is the lack

Re: [DISCUSS] KIP-304: Connect runtime mode improvements for container platforms

Hey all, I think think this is a great discussion, and is helping to clarify the real pain points as well as explore a few more options than just what was initially proposed. Stephane, I think why you're ending up in "grand redesign" state is because you're highlighting (and the KIP's motivation

Re: [DISCUSS] KIP-304: Connect runtime mode improvements for container platforms

Hi, thanks for the feedback, you have some valid points that I did not consider thoroughly. Now that I think about it, my current proposal is indeed a bit hacky, namely the dichotomy between running "light"/"heavy" connectors in standalone/distributed mode: indeed there should be no difference,

Re: [DISCUSS] KIP-304: Connect runtime mode improvements for container platforms

Fat fingered too... "Connect source should be able to achieve the same centrality of offsets " On Thu., 17 May 2018, 10:27 pm Stephane Maarek, < steph...@simplemachines.com.au> wrote: > Say you have 50 connectors all with different ACLs and service account. > That's 50 connect clusters to

Re: [DISCUSS] KIP-304: Connect runtime mode improvements for container platforms

Say you have 50 connectors all with different ACLs and service account. That's 50 connect clusters to maintain. So 50*3 internal connect topics to maintain (they can't share the same connect topics because they're different clusters). At default config we're talking 1500 partitions which is a lot

Re: [DISCUSS] KIP-304: Connect runtime mode improvements for container platforms

First sentence fat fingered. “Just curious as to why there’s an issue with the backing topics for Kafka Connect.” -- Rahul Singh rahul.si...@anant.us Anant Corporation On May 17, 2018, 6:17 AM -0400, Stephane Maarek , wrote: > Hi Salius > > I think you're on

Re: [DISCUSS] KIP-304: Connect runtime mode improvements for container platforms

Just curious as to why there’s a pr been for the backing topics for Kafka Connect. I’m a fan of decoupling and love the world that operates in containers. I’m just wondering specifically what the issue is for having topics storing the connect states for source / sinks. Wouldn’t it make sense

Re: [DISCUSS] KIP-304: Connect runtime mode improvements for container platforms

Hi Salius I think you're on the money, but you're not pushing things too far. This is something I've hoped for a long time. Let's talk Kafka Connect v2 Kafka Connect Cluster, as you said, are not convenient to work with (the KIP details drawbacks well). I'm all about containerisation just like

Re: [DISCUSS] KIP-304: Connect runtime mode improvements for container platforms

Hi, the only real usecase for the REST interface I can see is providing health/liveness checks for mesos/kubernetes. It's also true that the API can be left as is and e.g. not exposed publicly on the platform level, but this would still leave opportunities to accidentally mess something up

Re: [DISCUSS] KIP-304: Connect runtime mode improvements for container platforms

Hi, What do you plan to use the read-only REST interface for? Is there something what you cannot get through metrics interface? Otherwise it might be easier to just disable the REST interface (either in the code, or just on the platform level - e.g. in Kubernetes). Also, I do not know what is