Re: [VOTE] KIP-411: Make default Kafka Connect worker task client IDs distinct

2019-05-06 Thread Paul Davidson
aul > > > > On Thu, May 2, 2019 at 5:36 PM Arjun Satish > > wrote: > > > > > Paul, > > > > > > You might want to make a note on the KIP regarding the impact on > quotas. > > > > > > Thanks, > > > > > > On

Re: [VOTE] KIP-411: Make default Kafka Connect worker task client IDs distinct

2019-05-03 Thread Paul Davidson
ent > default client.id values are of the form: consumer-{count} and > producer-{count}. Thanks, Paul On Thu, May 2, 2019 at 5:36 PM Arjun Satish wrote: > Paul, > > You might want to make a note on the KIP regarding the impact on quotas. > > Thanks, > > On Thu, May 2,

Re: [VOTE] KIP-411: Make default Kafka Connect worker task client IDs distinct

2019-05-02 Thread Paul Davidson
t; > > Maybe we should mention this in the backward compatibility section for > > the > > > KIP. At the same time, since there is no way atm to turn off this > > feature, > > > should this feature be merged and released in the upcoming v2.3? This > is >

Re: [DISCUSS] KIP-411: Add option to make Kafka Connect task client ID values unique

2019-04-12 Thread Paul Davidson
>> legible. >> > > If >> > > > we >> > > > > let the user specify a pattern or something, we're really just >> making >> > > the >> > > > > user worry about these requirements. >>

Re: [VOTE] KIP-411: Make default Kafka Connect worker task client IDs distinct

2019-04-12 Thread Paul Davidson
Just a reminder that KIP-411 is open for voting. No votes received yet! On Fri, Apr 5, 2019 at 9:07 AM Paul Davidson wrote: > Hi all, > > Since we seem to have agreement in the discussion I would like to start > the vote on KIP-411. > > See: > https://cwiki.apache.org/con

[VOTE] KIP-411: Make default Kafka Connect worker task client IDs distinct

2019-04-05 Thread Paul Davidson
Hi all, Since we seem to have agreement in the discussion I would like to start the vote on KIP-411. See: https://cwiki.apache.org/confluence/display/KAFKA/KIP-411%3A+Make+default+Kafka+Connect+worker+task+client+IDs+distinct Also see the related PR: https://github.com/apache/kafka/pull/6097

Re: [DISCUSS] KIP-411: Add option to make Kafka Connect task client ID values unique

2019-03-01 Thread Paul Davidson
cluster-A-${connectorName}-${task}-consumer > > Thoughts? > > Randall > > On Wed, Feb 20, 2019 at 3:18 PM Ryanne Dolan > wrote: > > > Thanks Paul, this is great. This will make monitoring Connect a ton > easier. > > > > Ryanne > > >

Re: [DISCUSS] KIP-411: Add option to make Kafka Connect task client ID values unique

2019-02-20 Thread Paul Davidson
at 3:39 PM Paul Davidson wrote: > Hi everyone. We seem to have agreement that the ideal approach is to > alter the default client ids. Now I'm wondering about the best process to > proceed. Will the change in default behaviour require a new KIP, given it > will affect existing deploymen

Re: [DISCUSS] KIP-411: Add option to make Kafka Connect task client ID values unique

2019-01-11 Thread Paul Davidson
gh from source tasks, which will be of the form > `connector-producer-`. Maybe it is fine. (The other > comments were minor.) > > Best regards, > > Randall > > On Mon, Jan 7, 2019 at 1:19 PM Paul Davidson > wrote: > > > Thanks all. I've submitted a new PR with a possible implem

Re: [DISCUSS] KIP-411: Add option to make Kafka Connect task client ID values unique

2019-01-07 Thread Paul Davidson
. Will be interested to hear your thoughts on this. Paul On Mon, Jan 7, 2019 at 10:27 AM Ryanne Dolan wrote: > I'd also prefer to avoid the new configuration property if possible. Seems > like a lighter touch without it. > > Ryanne > > On Sun, Jan 6, 2019 at 7:25 PM Paul Davidson

Re: [DISCUSS] KIP-411: Add option to make Kafka Connect task client ID values unique

2019-01-06 Thread Paul Davidson
t; (some for only sinks or sources), so the framework can handle all of this > > without changes to connectors. Further, with connector-specific > overrides, > > you could get task-specific values if interpolation were supported in the > > config value (as we n

Re: [DISCUSS] KIP-411: Add option to make Kafka Connect task client ID values unique

2019-01-06 Thread Paul Davidson
Hi Ewen, Thanks for the feedback! If that's the case, would it maybe be possible to compatibly change the > default to use task IDs in the client ID, but only if we don't see an > existing override from the worker config? This would only change the > behavior when someone is using the default,

[DISCUSS] KIP-411: Add option to make Kafka Connect task client ID values unique

2018-12-20 Thread Paul Davidson
Hi everyone, I would like to start a discussion around the following KIP: *https://cwiki.apache.org/confluence/display/KAFKA/KIP-411%3A+Add+option+to+make+Kafka+Connect+task+client+ID+values+unique

Request permission to create KIP

2018-12-20 Thread Paul Davidson
Hi. I would like to create a new KIP, but I currently don't have permission. Wiki ID: pdavidson Thanks! Paul

Re: [VOTE] KIP-382 MirrorMaker 2.0

2018-12-20 Thread Paul Davidson
t; complex > > > topologies. > > > - Provides new metrics including end-to-end replication latency across > > > multiple data centers/clusters. > > > - Emits offsets required to migrate consumers between clusters. > > > - Tooling for offset translation. > > > - MirrorMaker-compatible legacy mode. > > > > > > Thanks, and happy holidays! > > > Ryanne > > > > > > -- Paul Davidson Principal Engineer, Ajna Team Big Data & Monitoring

Reviewers for PR: Automatic unique client id option for Kafka Connect

2018-12-11 Thread Paul Davidson
is in this issue: https://issues.apache.org/jira/browse/KAFKA-5061. Pull request: https://github.com/apache/kafka/pull/5775 Thanks! -- Paul Davidson Salesforce

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

2018-12-11 Thread Paul Davidson
Hi Ryanne, We had an IM exchange about KIP-382 and Mirus a few weeks back, but I also want to post here to publicly express my support. I'm the primary developer of Mirus, which is a Kafka Connect based replication tool we wrote at Salesforce to replace Mirror Maker internally. We open-sourced

[jira] [Created] (KAFKA-6876) Sender exceptions ignored by WorkerSourceTask producer Callback causing data loss

2018-05-07 Thread Paul Davidson (JIRA)
Paul Davidson created KAFKA-6876: Summary: Sender exceptions ignored by WorkerSourceTask producer Callback causing data loss Key: KAFKA-6876 URL: https://issues.apache.org/jira/browse/KAFKA-6876