Re: [DISCUSS] KIP-507: Securing Internal Connect REST Endpoints

Hi Chris, Thanks a lot for the KIP. This will certainly be a useful feature. I would have preferred to use the topic approach as well but I also understand your point of view about the operational complexity for upgrades. If not with this KIP, I would certainly want to go that route at some point

Re: [VOTE] KIP-458: Connector Client Config Override Policy

10, 2019 at 9:23 AM Magesh Nandakumar wrote: > Thanks a lot, Colin. This KIP has now passed voting with 3 binding votes > ( Randall, Rajini & Colin) and 1 non-binding vote (Chris). > Thanks a lot, everyone for the feedback & discussion on this KIP. > > On Fri, May 10

Re: [VOTE] KIP-458: Connector Client Config Override Policy

> > cheers, > Colin > > On Thu, May 9, 2019, at 18:31, Randall Hauch wrote: > > I'm still +1 and like the simplification. > > > > Randall > > > > On Thu, May 9, 2019 at 5:54 PM Magesh Nandakumar > > wrote: > > > > > I have

Re: [VOTE] KIP-458: Connector Client Config Override Policy

I have updated the KIP to remove the `Ignore` policy and also the useOverrides() method in the interface. Thanks a lot for your thoughts, Colin. I believe this certainly simplifies the KIP. On Thu, May 9, 2019 at 3:44 PM Magesh Nandakumar wrote: > Unless anyone has objections, I'm go

Re: [VOTE] KIP-458: Connector Client Config Override Policy

` properties in their > connector configs does seem pretty tiny. > > I'd be fine with removing the `Ignore` for backward compatibility. Still +1 > either way. > > On Thu, May 9, 2019 at 5:23 PM Magesh Nandakumar > wrote: > > > To add more details regarding the backward compat

Re: [VOTE] KIP-458: Connector Client Config Override Policy

On Thu, May 9, 2019 at 3:15 PM Magesh Nandakumar wrote: > Colin, > > Thanks a lot for the feedback. As you said, the possibilities of someone > having "producer.override.request.timeout.ms" in their connector config > in AK 2.2 or lower is very slim. But the key thi

Re: [VOTE] KIP-458: Connector Client Config Override Policy

at 08:05, Rajini Sivaram wrote: > > Hi Magesh, > > > > Thanks for the KIP, +1 (binding) > > > > Regards, > > > > Rajini > > > > > > On Thu, May 9, 2019 at 3:55 PM Randall Hauch wrote: > > > > > Nice work, Magesh. > > &g

Re: [VOTE] KIP-458: Connector Client Config Override Policy

onnect. > +1 (non-binding) > > Cheers, > > Chris > > On Tue, May 7, 2019 at 9:51 AM Magesh Nandakumar > wrote: > > > Hi All, > > > > I would like to start a vote on > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-458%3A+C

[VOTE] KIP-458: Connector Client Config Override Policy

Hi All, I would like to start a vote on https://cwiki.apache.org/confluence/display/KAFKA/KIP-458%3A+Connector+Client+Config+Override+Policy The discussion thread can be found here . Thanks, Magesh

[VOTE] KIP-458: Connector Client Config Override Policy

:35 AM Magesh Nandakumar wrote: > I have addressed all the outstanding discussion points and I believe we > have consensus on the design. Thanks, everyone for the feedback. I will > start a VOTE thread on this KIP. > > On Mon, May 6, 2019 at 10:13 PM Magesh Nandakumar > wrote: &g

Re: [DISCUSS] KIP-458: Connector Client Config Override Policy

I have addressed all the outstanding discussion points and I believe we have consensus on the design. Thanks, everyone for the feedback. I will start a VOTE thread on this KIP. On Mon, May 6, 2019 at 10:13 PM Magesh Nandakumar wrote: > Randall, > > Thanks a lot for the suggestion

Re: [DISCUSS] KIP-458: Connector Client Config Override Policy

it's the > current behavior, but I think it would help that it is described as the > default for the property. > > Otherwise, this looks good to me. > > Best regards, > > Randall > > On Mon, May 6, 2019 at 8:12 PM Magesh Nandakumar > wrote: > > > Konstantine, &g

Re: [DISCUSS] KIP-458: Connector Client Config Override Policy

nce between the two option > anyway. I'm looking forward to the updated KIP. > > On Mon, May 6, 2019 at 7:38 PM Magesh Nandakumar > wrote: > > > Randall, > > > > Thanks a lot for your feedback. > > > > If I understand it correctly, we could do one of the fol

Re: [DISCUSS] KIP-458: Connector Client Config Override Policy

to note > that Class is used as a raw type in field and return value declarations. We > should use the generic type instead. > > Thanks for this improvement proposal! > Konstantine > > On Mon, May 6, 2019 at 11:11 AM Magesh Nandakumar > wrote: > > > Randall, > > &g

Re: [DISCUSS] KIP-458: Connector Client Config Override Policy

tches the `producer.override`, `consumer.override`, and > `admin.override` prefixes. > > Thanks for working through this, Magesh. > > Randall > > On Mon, May 6, 2019 at 1:11 PM Magesh Nandakumar > wrote: > > > Randall, > > > > I was wondering if you h

Re: [DISCUSS] KIP-458: Connector Client Config Override Policy

Randall, I was wondering if you had any thoughts on the above alternatives to deal with a default policy. If it's possible, I would like to finalize the discussions and start a vote. Let me know your thoughts. Thanks, Magesh On Tue, Apr 30, 2019 at 8:46 PM Magesh Nandakumar wrote: > Rand

Re: [VOTE] KIP-454: Expansion of the ConnectClusterState interface

; > The discussion thread can be found at > https://www.mail-archive.com/dev@kafka.apache.org/msg96911.html > > Thanks to Konstantine Karantasis and Magesh Nandakumar for their thoughtful > feedback! > > Cheers, > > Chris >

Re: [VOT] KIP-449: Add connector contexts to Connect worker logs (vote thread)

This will make connect debugging so much easier. Thanks a lot for driving this Randall. +1 (non-binding) Thanks, Magesh On Tue, Apr 30, 2019 at 7:19 PM Jeremy Custenborder wrote: > +1 non binding > > On Mon, Apr 29, 2019 at 5:34 PM Randall Hauch wrote: > > > > I would like to start the vote

Re: [DISCUSS] KIP-458: Connector Client Config Override Policy

ht this may be confusing to people who may see that behavior and > > wonder how to recreate it themselves, so I suggested leaving that policy > > out and replace it with a check to see if a policy was specified at all. > > > > Would be interested in your thoughts on this, especially

Re: [DISCUSS] KIP-458: Connector Client Config Override Policy

l"? > > Otherwise, I like the idea of this. There have been several requests over > the past year or two for adding subsets of this functionality. Might be > good to find and list all of the related KAFKA issues. > > Randall > > On Fri, Apr 26, 2019 at 4:04 PM Chris Egerton

Re: [VOTE] KIP-411: Make default Kafka Connect worker task client IDs distinct

Looks good to me and a very useful feature. +1 ( non-binding) On Mon, Apr 29, 2019, 4:05 PM Arjun Satish wrote: > Thanks, Paul! This is very useful. > > +1 (non-binding) > > Best, > Arjun > > On Fri, Apr 12, 2019 at 4:13 PM Ryanne Dolan > wrote: > > > +1 (non binding) > > > > Thanks > >

Re: [DISCUSS] KIP-458: Connector Client Config Override Policy

eers, > > Chris > > On Wed, Apr 24, 2019 at 11:14 AM Magesh Nandakumar > wrote: > > > HI Chrise, > > > > You are right about the "admin." prefix creating conflicts. Here are few > > options that I can think of > > > > 1. Use `dlq.admi

Re: [DISCUSS] KIP-454: Expansion of the ConnectClusterState interface

sense to treat it similarly to how we're treating the > ConnectClusterState interface now. > > Thanks, > > Chris > > On Thu, Apr 25, 2019 at 4:18 PM Magesh Nandakumar > wrote: > > > HI Chrise, > > > > Overall it looks good to me. Just one last

Re: [DISCUSS] KIP-454: Expansion of the ConnectClusterState interface

terDetails class but > left out the cluster type information for now; let me know what you think. > > Thanks again for your thoughts! > > Cheers, > > Chris > > On Wed, Apr 24, 2019 at 4:49 PM Magesh Nandakumar > wrote: > > > Hi Chris, > > > > Instead of c

Re: [DISCUSS] KIP-454: Expansion of the ConnectClusterState interface

anks again for your thoughts! > > Cheers, > > Chris > > On Tue, Apr 23, 2019 at 4:18 PM Magesh Nandakumar > wrote: > > > Chris, > > > > I certainly would love to hear others thoughts on #1 but IMO, it might > not > > be as useful as Connector

Re: [DISCUSS] KIP-458: Connector Client Config Override Policy

nobody else identifies a common use > case that would involve a similar client config policy implementation that > would be limited to a small set of whitelisted configs. For now keeping the > PrincipalConnectorClientConfigPolicy sounds fine to me. > > Cheers, > > Chris

Re: [DISCUSS] KIP-458: Connector Client Config Override Policy

, 2019 at 10:27 PM Magesh Nandakumar wrote: > Chrise, > > Thanks a lot for your feedback. I will address them in order of your > questions/comments. > > 1. Thanks for bringing this to my attention about KIP-382. I had a closer > look at the KIP and IIUC, the KIP allo

Re: [DISCUSS] KIP-458: Connector Client Config Override Policy

ll set of overrides and no > others. Why not generalize here and create a policy that accepts a list of > allowed overrides during configuration? > > Thanks again for the KIP. > > Cheers, > > Chris > > On Fri, Apr 19, 2019 at 2:53 PM Magesh Nandakumar > wrote: > >

Re: [DISCUSS] KIP-454: Expansion of the ConnectClusterState interface

sed the configs for their worker through their configure(...) > method, so they'd be able to detect any such overrides and act accordingly. > > Thanks again for your thoughts! > > Cheers, > > Chris > > On Thu, Apr 18, 2019 at 11:08 AM Magesh Nandakumar > wrote: > >

[DISCUSS] KIP-458: Connector Client Config Override Policy

Hi all, I've posted "KIP-458: Connector Client Config Override Policy", which allows users to override the connector client configurations based on a policy defined by the administrator. The KIP can be found at

Re: [DISCUSS] KIP-454: Expansion of the ConnectClusterState interface

Hi Chris, Thanks for the KIP. Overall, it looks good and straightforward to me. I had a few questions on the new methods 1. I'm not sure if an extension would ever require the task configs. An extension generally should only require the health and current state of the connector which includes

Re: [VOTE] KIP-369 Alternative Partitioner to Support "Always Round-Robin" Selection

+1 ( non-binding) On Tue, Sep 4, 2018 at 3:39 AM M. Manna wrote: > Hello, > > I have made necessary changes as per the original discussion thread, and > would like to put it for votes. > > Thank you very much for your suggestion and guidance so far. > > Regards, >

Re: [DISCUSS] KIP-369 Alternative Partitioner to Support "Always Round-Robin" Selection

+1 for this. The only small suggestion would be to possibly call this RondRobinPartitioner which makes the intent obvious. On Thu, Aug 30, 2018 at 5:31 PM Stephen Powis wrote: > Neat, this would be super helpful! I submitted this ages ago: > https://issues.apache.org/jira/browse/KAFKA- > >

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

to the feature. Thanks, Magesh On Tue, Aug 28, 2018 at 8:53 PM Magesh Nandakumar wrote: > I was wondering if it would be much simpler to just do a pass-through so > that we can support any topic setting added in Kafka without any code > changes in connect. Since these are f

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

get created. Thanks Magesh On Tue, Aug 28, 2018 at 4:56 PM Randall Hauch wrote: > Do you think we should support name-value pairs, too? > > On Tue, Aug 28, 2018 at 6:41 PM Magesh Nandakumar > wrote: > > > Randall, > > > > Thanks a lot for the KIP. I thi

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

Randall, Thanks a lot for the KIP. I think this would be a great addition for many source connectors. One clarification I had was regarding the topic settings that can be configured. Is it limited to the setting exposed in the TopicSettings interface? Thanks Magesh On Tue, Aug 21, 2018 at 7:59

Re: [VOTE] KIP-305: Add Connect primitive number converters

+1 (non-binding) Thanks Magesh On Tue, May 22, 2018 at 4:23 PM, Randall Hauch wrote: > (bump so a few new subscribers see this thread.) > > On Tue, May 22, 2018 at 4:39 PM, Randall Hauch wrote: > > > +1 (non-binding) > > > > On Tue, May 22, 2018 at 4:05 PM,

Re: [VOTE] KIP-285: Connect Rest Extension Plugin

om> wrote: > > > > > > > > > +1 > > > > > Original message From: Gwen Shapira < > > > g...@confluent.io> > > > > > Date: 5/17/18 12:02 PM (GMT-08:00) To: dev <dev@kafka.apache.org > >

Re: [DISCUSS] KIP-285: Connect Rest Extension Plugin

given the way we expect it to be used. > > > > -Ewen > > > > On Wed, May 16, 2018 at 7:02 PM Randall Hauch <rha...@gmail.com> wrote: > > > > > Looks good to me. Thanks for quickly making the changes! Great work! > > > > >

Re: [VOTE] KIP-298: Error Handling in Connect kafka

+1 (non-binding) Thanks, Magesh On Fri, May 18, 2018 at 2:38 PM, Konstantine Karantasis < konstant...@confluent.io> wrote: > +1 (non-binding) > > - Konstantine > > On Thu, May 17, 2018 at 12:58 AM, Arjun Satish > wrote: > > > All, > > > > Many thanks for all the

Re: [DISCUSS] KIP-298: Error Handling in Connect

Arjun, Thanks for all the updates. I think it looks great and I don't have any other concerns. Thanks Magesh On Fri, May 18, 2018 at 2:36 PM, Konstantine Karantasis < konstant...@confluent.io> wrote: > The updated version of the KIP that uses the dead-letter-queue only for > sink records and

Re: [VOTE] KIP-297: Externalizing Secrets for Connect Configurations

Thanks Robert, this looks great +1 (non-binding) On Thu, May 17, 2018 at 5:35 PM, Colin McCabe wrote: > Thanks, Robert! > > +1 (non-binding) > > Colin > > > On Thu, May 17, 2018, at 14:15, Robert Yokota wrote: > > Hi Colin, > > > > I've changed the KIP to have a composite

Re: [DISCUSS] KIP-305: Add Connect primitive number converters

Thanks Randall for the KIP. I think it will be super useful and looks pretty straightforward to me. Thanks Magesh On Thu, May 17, 2018 at 4:15 PM, Randall Hauch wrote: > I'd like to start discussion of a very straightforward proposal for Connect > to add converters for the

[VOTE] KIP-285: Connect Rest Extension Plugin

Hello everyone, After a good round of discussions with excellent feedback and no major objections, I would like to start a vote on KIP-285: Connect Rest Extension Plugin. KIP: < https://cwiki.apache.org/confluence/display/KAFKA/KIP-285%3A+Connect+Rest+Extension+Plugin > JIRA:

Re: [DISCUSS] KIP-298: Error Handling in Connect

how we can take logs from a > > class > > > > and redirect it to a different location. Made a note in the PR for > > this. > > > > > > > > 5. When we talk about logging messages, this could mean instances of > > > > SinkRecords or So

Re: [DISCUSS] KIP-285: Connect Rest Extension Plugin

tion file. > > As I said, these are all minor suggestions that only affect the KIP > document. Once these are fixed, I think this is ready to move to voting. > > Best regards, > > Randall > > On Tue, May 15, 2018 at 11:30 AM, Magesh Nandakumar <mage...@confluent.io> > wro

Re: [DISCUSS] KIP-285: Connect Rest Extension Plugin

makes those classes more annoying to use. Is there a reason > for them to be nested or can we just pull them out to the same level as > ConnectorHealth? > > -Ewen > > On Tue, May 15, 2018 at 9:30 AM Magesh Nandakumar <mage...@confluent.io> > wrote: > > >

Re: [DISCUSS] KIP-298: Error Handling in Connect

Hi Arjun, I think this a great KIP and would be a great addition to have in connect. Had a couple of minor questions: 1. What would be the value in logging the connector config using errors.log.include.configs for every message? 2. Not being picky on format here but it might be clearer if the

Re: [DISCUSS] KIP-285: Connect Rest Extension Plugin

Randall- I think I have addressed all the comments. Let me know if we can take this to Vote. Thanks Magesh On Tue, May 8, 2018 at 10:12 PM, Magesh Nandakumar <mage...@confluent.io> wrote: > Hi All, > > I have updated the KIP to reflect changes based on the PR > https://githu

Re: [DISCUSS] KIP-285: Connect Rest Extension Plugin

PM, Randall Hauch <rha...@gmail.com> wrote: > Great work, Magesh. I like the overall approach a lot, so I left some > pretty nuanced comments about specific details. > > Best regards, > > Randall > > On Wed, Apr 25, 2018 at 3:03 PM, Magesh Nandakumar <mage...@co

Re: [DISCUSS] KIP-297: Externalizing Secrets for Connect Configurations

Hi Robert, Thanks for the KIP. I think, this will be a great addition to the framework. I think, will be great if the KIP can elaborate a little bit more on how implementations would look like with an example. Also, would be good to provide a reference implementation as well. The other questions

Re: [DISCUSS] KIP-285: Connect Rest Extension Plugin

gt; Thanks for updating the KIP, Magesh. You've resolved all of my concerns, > > though I have one more: we should specify the package names for all new > > interfaces/classes. > > > > I'm looking forward to more feedback from others. > > > > Best regards,

Re: [DISCUSS] KIP-285: Connect Rest Extension Plugin

to do that all the new interfaces would need to live in the runtime. This might affect some minor details in the KIP. Let me know your thoughts. Thanks Magesh On Thu, Apr 19, 2018 at 10:17 PM, Magesh Nandakumar <mage...@confluent.io> wrote: > Hi All, > > I have updated the KIP with fo

Re: [DISCUSS] KIP-285: Connect Rest Extension Plugin

Kindly let me know your thoughts on the updates. Thanks Magesh On Thu, Apr 19, 2018 at 10:39 AM, Magesh Nandakumar <mage...@confluent.io> wrote: > Hi Randall, > > Thanks for your feedback. I also would like to go with > rest.extension.classes`. > > For exposing Configurab

Re: [DISCUSS] KIP-285: Connect Rest Extension Plugin

in the interface. Hence it doesn't affect the public API by any means. I will update the KIP and let everyone know. Thanks Magesh On Thu, Apr 19, 2018 at 9:51 AM, Randall Hauch <rha...@gmail.com> wrote: > On Thu, Apr 12, 2018 at 10:59 AM, Magesh Nandakumar <mage...@confluent.io> &g

Re: [DISCUSS] KIP-285: Connect Rest Extension Plugin

PI. We also need to be explicit that an implementation must >provide a `META-INF/services/org.apache.kafka.connect. > ConnectRestPlugin` >file (or whatever the package name of the `ConnectRestPlugin` will be) > with >the fully-qualified name of the implementation class(es).

Re: [DISCUSS] KIP-285: Connect Rest Extension Plugin

n and it doesn't work, it will be harder to know if > it is me or Kafka... > > On Wed, Apr 11, 2018 at 9:47 AM, Magesh Nandakumar <mage...@confluent.io> > wrote: > > > Hi Gwen, > > > > Thanks for your feedback. As part of this KIP, I was planning on adding a > >

Re: [DISCUSS] KIP-285: Connect Rest Extension Plugin

Or > the ZK authorizer. > > Do you plan on including a simple implementation as part of this KIP as > well? > > Gwen > > On Tue, Apr 10, 2018 at 10:14 PM, Magesh Nandakumar <mage...@confluent.io> > wrote: > > > Hi, > > > > We have posted KIP-28

[DISCUSS] KIP-285: Connect Rest Extension Plugin

Hi, We have posted KIP-285: Connect Rest Extension Plugin to add the ability to provide Rest Extensions to Connect Rest API. https://cwiki.apache.org/confluence/display/KAFKA/KIP-285%3A+Connect+Rest+Extension+Plugin Please take a look. Your feedback is appreciated. Thanks, Magesh

[DISCUSS] KIP-285: Connect Rest Extension Plugin

Hi, We have posted KIP-285: Connect Rest Extension Plugin to add the ability to provide Rest Extensions to Connect Rest API. https://cwiki.apache.org/confluence/display/KAFKA/KIP- 285%3A+Connect+Rest+Extension+Plugin Please take a look. Your feedback is appreciated. Thanks, Magesh