Kafka Contributor Status

2018-01-16 Thread Chris Egerton
eligible! Cheers, Chris Egerton

Re: [DISCUSS] KIP-449: Add connector contexts to Connect worker logs

2019-04-15 Thread Chris Egerton
Hi Randall, Thanks for the KIP. Debugging Connect workers definitely becomes harder as the number of connectors and tasks increases, and your proposal would simplify the process of sifting through logs and finding relevant information faster and more accurately. I have a couple small comments:

[DISCUSS] KIP-454: Expansion of the ConnectClusterState interface

2019-04-13 Thread Chris Egerton
Hi all, I've posted "KIP-454: Expansion of the ConnectClusterState interface", which proposes that we add provide more information about the Connect cluster to REST extensions. The KIP can be found at

Re: [DISCUSS] KIP-466: Add support for List serialization and deserialization

2019-05-08 Thread Chris Egerton
Hi Daniyar, I think you may want to tweak your syntax a little: public static Serde> List(Serde innerSerde) { return new ListSerde(innerSerde); } Does that work? Cheers, Chris On Wed, May 8, 2019 at 10:29 AM Development wrote: > Hi John, > > I updated JIRA and KIP. > > I didn’t know

Re: [VOTE] KIP-454: Expansion of the ConnectClusterState interface

2019-05-08 Thread Chris Egerton
> > > > > > +1 (binding) > > > > > > > > On Thu, May 2, 2019 at 8:16 PM Magesh Nandakumar < > mage...@confluent.io > > > > > > > wrote: > > > > > > > > > Thanks a lot for the work on this KIP Chris. > >

Re: [DISCUSS] KIP-454: Expansion of the ConnectClusterState interface

2019-04-30 Thread Chris Egerton
ir implementations. > > Otherwise, the KIP LGTM. > > Konstantine > > On Thu, Apr 25, 2019 at 10:29 PM Magesh Nandakumar > wrote: > > > Thanks a lot, Chris. The KIP looks good to me. > > > > On Thu, Apr 25, 2019 at 7:35 PM Chris Egerton > wro

Re: [DISCUSS] KIP-458: Connector Client Config Override Policy

2019-04-30 Thread Chris Egerton
ith > > the > > > current behavior, whereas the "disallows" policy seems useful but not > > > exactly backward compatible. Should we also offer a "Disallow" policy? > In > > > fact, should the policies be named "Ignore" (defaul

Re: [VOT] KIP-449: Add connector contexts to Connect worker logs (vote thread)

2019-04-30 Thread Chris Egerton
+1 (non-binding) Really looking forward to this. Thanks, Randall! On Tue, Apr 30, 2019, 20:47 Magesh Nandakumar wrote: > This will make connect debugging so much easier. Thanks a lot for driving > this Randall. > > +1 (non-binding) > > Thanks, > Magesh > > On Tue, Apr 30, 2019 at 7:19 PM

Re: [VOTE] KIP-458: Connector Client Config Override Policy

2019-05-07 Thread Chris Egerton
Hi Magesh, This looks great! Very excited to see these changes finally coming to Connect. +1 (non-binding) Cheers, Chris On Tue, May 7, 2019 at 9:51 AM Magesh Nandakumar wrote: > Hi All, > > I would like to start a vote on > >

Re: [DISCUSS] KIP-454: Expansion of the ConnectClusterState interface

2019-05-07 Thread Chris Egerton
be found here: https://www.mail-archive.com/dev@kafka.apache.org/msg97458.html. Current status: +1 binding, +2 non-binding. Cheers, Chris On Tue, Apr 30, 2019 at 12:40 PM Chris Egerton wrote: > Hi Konstantine, > > I've updated the KIP to add default method implementations to

Re: [DISCUSS] KIP-454: Expansion of the ConnectClusterState interface

2019-04-18 Thread Chris Egerton
e. Let me know what you think. > > Thanks, > Magesh > > On Sat, Apr 13, 2019 at 9:00 PM Chris Egerton wrote: > > > Hi all, > > > > I've posted "KIP-454: Expansion of the ConnectClusterState interface", > > which proposes that we add provide more

Re: [DISCUSS] KIP-454: Expansion of the ConnectClusterState interface

2019-04-19 Thread Chris Egerton
what consists a "current" task configuration and how this is > retrieved would be valuable here. > > Best, > Konstantine > > > > On Thu, Apr 18, 2019 at 2:45 PM Chris Egerton wrote: > > > Hi Magesh, > > > > Thanks for your comments. I'll

[VOTE] KIP-454: Expansion of the ConnectClusterState interface

2019-05-02 Thread Chris Egerton
Hi all, I'd like to start a vote for KIP-454: https://cwiki.apache.org/confluence/display/KAFKA/KIP-454%3A+Expansion+of+the+ConnectClusterState+interface The discussion thread can be found at https://www.mail-archive.com/dev@kafka.apache.org/msg96911.html Thanks to Konstantine Karantasis and

Re: [DISCUSS] KIP-458: Connector Client Config Override Policy

2019-04-26 Thread Chris Egerton
your inputs. > > Thanks, > Magesh > > On Thu, Apr 25, 2019 at 2:18 PM Chris Egerton wrote: > > > Hi Magesh, > > > > Agreed that we should avoid `dlq.admin`. I also don't have a strong > opinion > > between `connector.` and `.override`, but I have a slight in

Re: [DISCUSS] KIP-454: Expansion of the ConnectClusterState interface

2019-05-08 Thread Chris Egerton
Hi Chris, > > Thanks for the KIP, looks good. I have just one question. Can ` > ConnectClusterState#connectorConfig()` return any sensitive configs like > passwords? > > Thanks, > > Rajini > > On Wed, May 8, 2019 at 1:30 AM Chris Egerton wrote: > > > H

Re: [DISCUSS] KIP-458: Connector Client Config Override Policy

2019-04-22 Thread Chris Egerton
Hi Magesh, This is an exciting KIP! I have a few questions/comments but overall I like the direction it's headed in and hope to see it included in the Connect framework soon. 1. With the proposed "consumer.", "producer.", and "admin." prefixes, how will this interact with connectors such as the

Re: [DISCUSS] KIP-454: Expansion of the ConnectClusterState interface

2019-04-24 Thread Chris Egerton
Let me know what you think. > > Thanks > Magesh > > On Thu, Apr 18, 2019 at 2:45 PM Chris Egerton wrote: > > > Hi Magesh, > > > > Thanks for your comments. I'll address them in the order you provided > them: > > > > 1 - Reason for exposing task con

Re: [DISCUSS] KIP-458: Connector Client Config Override Policy

2019-04-24 Thread Chris Egerton
e well more than > just > > the list of configs but might also want some restriction on values. If > the > > concern is about users wanting principal and also other configs, it would > > still be possible by means of a custom implementation. As is, I would > > prefer to kee

Re: [DISCUSS] KIP-458: Connector Client Config Override Policy

2019-04-25 Thread Chris Egerton
r.override`, '`consumer.override` and `admin.override` - > provides better clarity that these are overrides. > > I don't have a strong opinion in choosing between #2 and #3. Let me > know what you think. > > Thanks > Magesh > > On Wed, Apr 24, 2019 at 10:25 AM Chris Egert

Re: [DISCUSS] KIP-454: Expansion of the ConnectClusterState interface

2019-04-25 Thread Chris Egerton
can include things like groupid, underlying > kafkaclusterId, standalone or distributed, etc. This will help expose any > cluster related information in the future. > Let me know if that would work? > > Thanks, > Magesh > > On Wed, Apr 24, 2019 at 4:26 PM Chris Egerton wrot

Re: [DISCUSS] KIP-454: Expansion of the ConnectClusterState interface

2019-04-23 Thread Chris Egerton
ations of an interface outside apache/kafka repo. WDYT? > > Konstantine > > On Fri, Apr 19, 2019 at 2:26 PM Chris Egerton wrote: > > > Hi Konstantine, > > > > Thanks for your comments. I'll respond to them in the order you provided > > them: > > > > 1.

Re: [DISCUSS] KIP-454: Expansion of the ConnectClusterState interface

2019-04-25 Thread Chris Egerton
at 4:18 PM Magesh Nandakumar wrote: > HI Chrise, > > Overall it looks good to me. Just one last comment - I was wondering if > ConnectClusterDetail should be an interface just like ConnectClusterState. > > Thanks, > Magesh > > On Thu, Apr 25, 2019 at 3:54 PM Chris Eger

Re: [VOTE] KIP-495: Dynamically Adjust Log Levels in Connect

2019-08-13 Thread Chris Egerton
Nice stuff, Arjun! +1 (non-binding) On Tue, Aug 13, 2019 at 1:55 PM Arjun Satish wrote: > Hey everyone, > > I'd like to start a vote for KIP-495 ( > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-495%3A+Dynamically+Adjust+Log+Levels+in+Connect > ). > This change will make Connect

Re: [DISCUSS] KIP-507: Securing Internal Connect REST Endpoints

2019-08-15 Thread Chris Egerton
sed to give > out session tokens in a secure way. It seems that any attacker could just > join the group and sign requests with the provided token. Am I missing > something? > > Ryanne > > On Wed, Aug 14, 2019, 2:31 PM Chris Egerton wrote: > > > The KIP pag

Re: [DISCUSS] KIP-507: Securing Internal Connect REST Endpoints

2019-08-19 Thread Chris Egerton
t see how these are > related. Why would it matter if the workers sent Kafka messages vs POST > requests among themselves? > > Ryanne > > On Thu, Aug 15, 2019 at 3:57 PM Chris Egerton wrote: > > > Hi Ryanne, > > > > Yes, if the Connect group is left unsecur

[DISCUSS] KIP-507: Securing Internal Connect REST Endpoints

2019-08-14 Thread Chris Egerton
Hi all, I'd like to start discussion on a KIP to secure the internal "POST /connectors//tasks" endpoint for the Connect framework. The proposed changes address a vulnerability in the framework in its current state that allows malicious users to write arbitrary task configurations for connectors;

Re: [DISCUSS] KIP-507: Securing Internal Connect REST Endpoints

2019-08-14 Thread Chris Egerton
The KIP page can be found at https://cwiki.apache.org/confluence/display/KAFKA/KIP-507%3A+Securing+Internal+Connect+REST+Endpoints, by the way. Apologies for neglecting to include it in my initial email! On Wed, Aug 14, 2019 at 12:29 PM Chris Egerton wrote: > Hi all, > > I'd like

Re: [DISCUSS] KIP-507: Securing Internal Connect REST Endpoints

2019-08-21 Thread Chris Egerton
nks Ryanne for the feedback so far, hope to hear from some more of you soon :) Cheers, Chris On Mon, Aug 19, 2019 at 12:02 PM Chris Egerton wrote: > Hi Ryanne, > > The reasoning for this is provided in the KIP: "There would be no clear > way to achieve consensus amongst th

Re: [DISCUSS] KIP-507: Securing Internal Connect REST Endpoints

2019-08-28 Thread Chris Egerton
but I also understand > > your > > > point of view about the operational complexity for upgrades. If not > with > > > this KIP, I would certainly want to go that route at some point in the > > > future. > > > > > > As far as using the rebalan

Re: [DISCUSS] KIP-507: Securing Internal Connect REST Endpoints

2019-09-06 Thread Chris Egerton
the course of the next few days or so I'll be opening this for a vote. Cheers, Chris On Wed, Aug 28, 2019 at 12:21 PM Chris Egerton wrote: > HI all, > > Wow, thanks for all the feedback! Happy to see this proposal getting some > love :) > > > RE Konstantine's comm

Re: [DISCUSS] KIP-507: Securing Internal Connect REST Endpoints

2019-09-11 Thread Chris Egerton
for the given key algorithm. More detail can be found in the KIP if anyone's interested. Cheers, Chris On Tue, Sep 10, 2019 at 3:18 PM Chris Egerton wrote: > Hi Randall, > > Thanks so much for your comprehensive feedback! To avoid creating an > extremely long response I'll address y

Re: [DISCUSS] KIP-507: Securing Internal Connect REST Endpoints

2019-09-10 Thread Chris Egerton
venting a cluster from > enabling this feature? Should a warning or error be logged if this feature > is disabled after being enabled? > > Finally, I have a few nits: > >1. The "Backwards compatibility" section should be more focused on the >upgrade UX. "Al

Re: [DISCUSS] KIP-507: Securing Internal Connect REST Endpoints

2019-09-16 Thread Chris Egerton
f > Connect.")? Seems like users unintentionally misconfiguring some of their > Connect workers is quite common. > > Additionally, it will be vital to design appropriate error messages for > > this scenario so that users can (hopefully) dig themselves out of that > ho

Re: [DISCUSS] KIP-507: Securing Internal Connect REST Endpoints

2019-09-17 Thread Chris Egerton
Hauch wrote: > On Mon, Sep 16, 2019 at 3:06 PM Chris Egerton wrote: > > > Hi Randall, > > > > The new default value for the key size configuration will be "null". I've > > clarified this in the KIP. This will still preserve backwards > compatibility >

Re: [DISCUSS] KIP-495: Dynamically Adjust Log Levels in Connect

2019-08-01 Thread Chris Egerton
Thanks Arjun! Looks good to me. On Thu, Aug 1, 2019 at 12:33 PM Arjun Satish wrote: > Thanks for the feedback, Chris! > > Yes, the example is pretty much how Connect will use the new feature. > Tweaked the section to make this more clear. > > Best, > > On Fri, Jul 26

Re: [DISCUSS] KIP-495: Dynamically Adjust Log Levels in Connect

2019-07-26 Thread Chris Egerton
Hi Arjun, This looks great. The changes to public interface are pretty small and moving the Log4jController class into the clients package seems like the right way to go. One question I have--it looks like the purpose of this KIP is to enable dynamic setting of log levels in the Connect

Re: [DISCUSS] KIP-507: Securing Internal Connect REST Endpoints

2019-09-19 Thread Chris Egerton
e proposed configurations to try to take some of Randall's suggestions into account; the changes can be summarized as replacing "internal.request" with "inter.worker", and changing " inter.worker.key.rotation.interval.ms" to "inter.worker.key.ttl.ms". Cheers, Chr

[VOTE] KIP-507: Securing Internal Connect REST Endpoints

2019-09-19 Thread Chris Egerton
Hi all, I'd like to begin voting on KIP-507: https://cwiki.apache.org/confluence/display/KAFKA/KIP-507%3A+Securing+Internal+Connect+REST+Endpoints Thanks to Ryanne, Magesh, Konstantine, Greg, and Randall for the fruitful discussion! Cheers, Chris

Re: Preliminary blog post about the Apache Kafka 2.4.0 release

2019-11-14 Thread Chris Egerton
Hi Manikumar, It looks like the header for KIP-440 is accurate ("KIP-440: Extend Connect Converter to support headers") but the content appears to correspond to KIP-481 ("SerDe Improvements for Connect Decimal type in JSON") instead. Could we double-check and make sure that the summary for

Re: [DISCUSS] Apache Kafka 2.4.0 release

2019-11-12 Thread Chris Egerton
>>>> On Thu, Oct 3, 2019 at 8:54 AM Manikumar < > > >>>> manikumar.re...@gmail.com> > > >>>> >>> wrote: > > >>>> >>>> > > >>>> >>>>> Hi all, > > >>>> >>

Re: [VOTE] KIP-458: Connector Client Config Override Policy

2019-10-15 Thread Chris Egerton
to have a > non-policy, > >>> > since > >>> > > > >> there's now special behavior for when the policy is not > >>> specified. > >>> > > > >> > >>> > > > >> Perhaps the inability for

Re: [DISCUSS] Apache Kafka 2.4.0 release

2019-09-25 Thread Chris Egerton
Hi Manikumar, Would it be possible to add KIP-507 ( https://cwiki.apache.org/confluence/display/KAFKA/KIP-507%3A+Securing+Internal+Connect+REST+Endpoints) to the 2.4 release, as it has recently been accepted? Cheers, Chris On Tue, Sep 24, 2019 at 9:33 AM Kevin Lu wrote: > Hi Manikumar, > >

Re: [VOTE] KIP-507: Securing Internal Connect REST Endpoints

2019-09-25 Thread Chris Egerton
at 9:10 AM Rajini Sivaram > > wrote: > > > > > Thanks for the KIP, Chris! > > > > > > +1 (binding) > > > > > > Regards, > > > > > > Rajini > > > > > > On Tue, Sep 24, 2019 at 3:12 PM Randall Hauch >

Re: [DISCUSS] KIP-610: Error Reporting in Sink Connectors

2020-05-16 Thread Chris Egerton
Hi Konstantine, I don't believe a callback-based implementation would require additional threads to be spun up. The callback could be invoked by the framework whenever the work for reporting a record is done, on that same thread or, in the case of a DLQ topic reporter, in a callback from the

Re: [DISCUSS] KIP-610: Error Reporting in Sink Connectors

2020-05-16 Thread Chris Egerton
Hi Arjun, This is great news. Given that we're already willing to ask developers to catch a method-not-found exception and it sounds like a new interface can be handled in a similar try/catch block in a same place, I like the idea of a new fleshed-out interface instead of a BiConsumer or a

Re: [DISCUSS] KIP-610: Error Reporting in Sink Connectors

2020-05-17 Thread Chris Egerton
Hi Konstantine, Given that the reporter interface is intentionally agnostic about how records are handled and does not necessarily entail writes to a DLQ, I'm also in favor of not specifying a return type for the reporting mechanism. I'm still unclear on how futures are going to provide any

Re: [DISCUSS] KIP-610: Error Reporting in Sink Connectors

2020-05-15 Thread Chris Egerton
Hi Randall, I think the problem here is that users aren't likely to implement a deprecated method, and may even remove that method from their connector after seeing that it is now deprecated. This would render that connector incompatible with older runtimes. Why deprecate a method that we don't

Re: [DISCUSS] KIP-610: Error Reporting in Sink Connectors

2020-05-15 Thread Chris Egerton
Hi all, I think if we're going to avoid adding methods to the SinkTaskContext due to concerns over compatibility, we might also want to ensure that the approach we follow instead doesn't have similar pitfalls. An overloaded "put", "initialize", or "start" isn't quite as troublesome with regards

Re: [DISCUSS] KIP-610: Error Reporting in Sink Connectors

2020-05-18 Thread Chris Egerton
how much control do we want to provide to the developer/user in this KIP > considering that we can add the functionality relatively easily later? > > Randall, Konstantine, what do you think about adding it later vs now? > > Thanks, > Aakash > > On Mon, May 18, 2020 at 12

Re: [DISCUSS] KIP-610: Error Reporting in Sink Connectors

2020-05-18 Thread Chris Egerton
Hi Aakash, I asked this earlier about whether futures were the right way to go, if we wanted to enable asynchronous behavior at all: > I'm still unclear on how futures are going to provide any benefit to developers, though. Blocking on the return of such a future slightly later on in the process

Re: [DISCUSS] KIP-610: Error Reporting in Sink Connectors

2020-05-19 Thread Chris Egerton
Hi Randall, First off, thank you for the incredibly detailed example. I don't mind walls of text. I found it very helpful. I especially liked the idea about modifying how the framework invokes "SinkTask::preCommit" to take most of the work out of developers' hands in the common case of a

Re: [DISCUSS] KIP-610: Error Reporting in Sink Connectors

2020-05-19 Thread Chris Egerton
Hi Aakash, > If "errors.tolerance=none", should it not be the case that the error reporter does not even report any error; rather, the task just fails after throwing the error? I do understand the point you are saying about duplicates, though. I believe the "errors.tolerance" property dictates

[DISCUSS] KIP-618: Atomic commit of source connector records and offsets

2020-05-22 Thread Chris Egerton
Hi all, I know it's a busy time with the upcoming 2.6 release and I don't expect this to get a lot of traction until that's done, but I've published a KIP for allowing atomic commit of offsets and records for source connectors and would appreciate your feedback:

Re: [Connect] Different validation requirements for connector creation and update

2021-01-21 Thread Chris Egerton
Hi Gunnar, It's not possible to do this in a generalized fashion with the API provided by the framework today. Trying to hack your way around things by setting a flag or storing the connector name in some shared JVM state wouldn't work in a cluster with more than one worker since that state would

KIP-618: Exactly-Once Support for Source Connectors

2021-01-25 Thread Chris Egerton
Hi all, I'd like to resurface KIP-618, which has been expanded from its previous form ("Atomic commit of source connector records and offsets") to something more general and, hopefully, more powerful: exactly-once support for source connectors (including fencing-out of zombie tasks). Would

Re: [DISCUSS] KIP-618: Atomic commit of source connector records and offsets

2021-01-25 Thread Chris Egerton
fset storage > (e.g. kafka topic) managed by the periodic task.commit()? > > On 2020/05/22 06:20:51, Chris Egerton wrote: > > Hi all, > > > > I know it's a busy time with the upcoming 2.6 release and I don't expect > > this to get a lot of traction until that's do

Re: [DISCUSS] KIP-618: Atomic commit of source connector records and offsets

2021-02-02 Thread Chris Egerton
-Jason > > > On Mon, Jan 25, 2021 at 7:29 AM Chris Egerton wrote: > > > Hi Ning, > > > > Apologies for the delay in response. I realized after publishing the KIP > > that there were some finer points I hadn't considered in my design and > that > > it was

Re: [VOTE] KIP-738: Removal of Connect's internal converter properties

2021-06-09 Thread Chris Egerton
gt; > https://lists.apache.org/thread.html/r01e9fa52998a337e435e5d0effca02e74b0552bdec271c1eeca39cd2%40%3Cdev.kafka.apache.org%3E > > > > > > On Tue, May 18, 2021 at 11:26 AM Ryanne Dolan > > > wrote: > > > > > > > +1 (non-binding) >

Re: [VOTE] KIP-618: Exactly-once support for source connectors

2021-06-09 Thread Chris Egerton
at 2:35 PM Ryanne Dolan wrote: > > > +1 (non-binding) > > > > On Thu, Jun 3, 2021, 10:23 AM Chris Egerton > > > wrote: > > > > > Hi all, > > > > > > I'd like to call for a vote on KIP-618, which adds support for > > exactl

Re: [DISCUSS] KIP-618: Atomic commit of source connector records and offsets

2021-06-09 Thread Chris Egerton
de to fail. > And even though in 3.0 we could allow ourselves to cause breaking changes > with a major release, I personally would prefer we not have any such > breaking changes. > > Given that, what would be required for us to eliminate that breaking > change, or change it from a bre

Re: [VOTE] KIP-618: Exactly-once support for source connectors

2021-06-10 Thread Chris Egerton
hat I'm voting for the last > > version discussed in the thread - that includes two phase upgrade and no > > breaking changes in 3.0. > > > > +1 (binding) > > > > > > On Wed, Jun 9, 2021, 5:32 AM Chris Egerton > > wrote: > > > > > Hi all, &g

Re: [VOTE] KIP-618: Exactly-once support for source connectors

2021-06-15 Thread Chris Egerton
Hi Ismael, Friendly reminder that your comment is the only outstanding one. If I don't hear back soon I'll probably close the KIP and we can address any concerns in a follow-up KIP. Cheers, Chris On Thu, Jun 10, 2021 at 12:13 PM Chris Egerton wrote: > Hi all, > > Firstl

Re: [VOTE] KIP-618: Exactly-once support for source connectors

2021-06-15 Thread Chris Egerton
to you all. > > Ismael > > On Tue, Jun 15, 2021 at 5:59 AM Chris Egerton > > wrote: > > > Hi Ismael, > > > > Friendly reminder that your comment is the only outstanding one. If I > don't > > hear back soon I'll probably close the KIP and we can addres

Re: [DISCUSS] KIP-618: Atomic commit of source connector records and offsets

2021-05-13 Thread Chris Egerton
Hi Tom, I'm fine with an implicit mapping of connector-provided null to user-exposed UNKNOWN, if the design continues down that overall path. Allowing users to assert that a connector should support exactly-once sounds reasonable; it's similar to the pre-flight checks we already do for connector

[VOTE] KIP-738: Removal of Connect's internal converter properties

2021-05-18 Thread Chris Egerton
Hi all, I'd like to call for a vote on KIP-738: https://cwiki.apache.org/confluence/display/KAFKA/KIP-738%3A+Removal+of+Connect%27s+internal+converter+properties The discussion thread (which was originally titled with "KIP-736") can be found here:

Re: [DISCUSS] KIP-618: Atomic commit of source connector records and offsets

2021-05-18 Thread Chris Egerton
ay nicely with exactly-once support. Cheers, Chris On Mon, May 17, 2021 at 7:24 PM Randall Hauch wrote: > On Mon, May 10, 2021 at 10:26 AM Chris Egerton > > wrote: > > > RE transaction boundaries: > > > > > First, the offset commit interval is currently 60 seconds

Re: [DISCUSS] KIP-618: Atomic commit of source connector records and offsets

2021-05-10 Thread Chris Egerton
ous clusters" > subsection of the "Permitted failure scenarios" section talks about (I > think) how setting `exactly.once.source.enabled=false` for one worker > effectively breaks the EOS delivery guarantees for the whole Connect > cluster. If that is the case, how does this ro

Re: [DISCUSS] KIP-618: Atomic commit of source connector records and offsets

2021-05-07 Thread Chris Egerton
arbitrarily many duplicates? I don't necessarily see this as a > show-stopper, more I'm trying to understand what's possible with this > design. > > 3. What authorization is needed for Admin.fenceProducers()? > > 4. Maybe I missed it, but when a per-connector offsets storage topic

Re: Testing KRaft mode

2021-05-17 Thread Chris Egerton
Hi Gunnar, As far as the Connect behavior goes, sounds like https://issues.apache.org/jira/browse/KAFKA-12252. To verify, you can either set the 'connect.protocol' property of your Connect cluster to 'compatible' (easy mode) or cherry-pick the recently-merged fix for it (

Re: [DISCUSS] KIP-618: Atomic commit of source connector records and offsets

2021-05-19 Thread Chris Egerton
n as I open and close files. This > would allow me to guard from the dreaded exception a line 732 and wrap > an entire file in a transaction. It either makes it or it doesn't. > > J > > > On Tue, May 18, 2021 at 10:04 AM Chris Egerton > wrote: > > > > Hi Randall,

Re: [DISCUSS] KIP-618: Atomic commit of source connector records and offsets

2021-05-14 Thread Chris Egerton
oday by calling > initTransactions() and then just using the producer as normal (no > beginTransaction()/abortTransaction()/endTransaction()). > > In light of that (and assuming you buy these arguments), I wonder how much > extra effort it would be to do for EOS-enabled clusters as pa

Re: [DISCUSS] KIP-736: Removal of Connect's internal converter properties

2021-05-11 Thread Chris Egerton
Hi all, Due to some confusion this KIP's number conflicted with an existing one. I've renamed it to KIP-738. Given the tight scope of the proposed changes, I'm hoping to open a vote on this soon if there's no discussion to be had. Cheers, Chris On Wed, May 5, 2021 at 2:33 PM Chris Egerton

Re: [DISCUSS] KIP-618: Atomic commit of source connector records and offsets

2021-06-02 Thread Chris Egerton
ike engineering a safety catch and then not enabling it. Wouldn't > it be safer to default to required, so that there's no way someone can > mistakenly not get EoS without explicitly having configured it? > > Thanks, > > Tom > > On Tue, Jun 1, 2021 at 4:48 PM Chris Egerton &

Re: [DISCUSS] KIP-618: Atomic commit of source connector records and offsets

2021-05-21 Thread Chris Egerton
to entertain some of these ideas. > > On Fri, May 14, 2021 at 5:06 PM Chris Egerton > > wrote: > > > [...] > > > That's true, but we do go from three static ACLs (write/describe on a fixed > > transactional ID, and idempotent write on a fixed cluster) to a dynam

[VOTE] KIP-618: Exactly-once support for source connectors

2021-06-03 Thread Chris Egerton
Hi all, I'd like to call for a vote on KIP-618, which adds support for exactly-once delivery guarantees for source connectors in the Kafka Connect framework. I suspect there might be a little more discussion to be had but with the KIP freeze deadline approaching, I wanted to give anyone

Re: [DISCUSS] KIP-618: Atomic commit of source connector records and offsets

2021-05-24 Thread Chris Egerton
pportunity to provide a snapshot of what the current design looks like. Cheers, Chris On Fri, May 21, 2021 at 4:32 PM Chris Egerton wrote: > Hi Tom, > > Wow, I was way off base! I was thinking that the intent of the fencible > producer was to employ it by default with 3.0, as

Re: [DISCUSS] KIP-618: Atomic commit of source connector records and offsets

2021-06-01 Thread Chris Egerton
t; > And one general question: in Debezium, we have some connectors that > > produce records "out-of-bands" to a schema history topic via their own > > custom producer. Is there any way envisionable where such a producer > would > > participate in the transaction managed b

Re: [DISCUSS] KIP-477: Add PATCH method for connector config in Connect REST API

2021-07-12 Thread Chris Egerton
Hi all, Know it's been a while for this KIP but in my personal experience the value of a PATCH method in the REST API has actually increased over time and I'd love to have this option for quick, stop-the-bleeding remediation efforts. One thought that comes to mind is that sometimes it may be

Re: [DISCUSS] KIP-618: Atomic commit of source connector records and offsets

2021-05-04 Thread Chris Egerton
-defined and limited-scope ACLs), I erred on the side of keeping things simple Looking forward to the next round of review and really hoping we can get the ball rolling in time for this to land with 3.0! Cheers, Chris On Mon, Apr 12, 2021 at 7:51 AM Chris Egerton wrote: > Hi Randall, > &

[DISCUSS] KIP-736: Removal of Connect's internal converter properties

2021-05-05 Thread Chris Egerton
Hi all, Got a lightweight KIP out to clean up some old properties from Connect that are already deprecated but are still managing to be footguns for some unfortunate users. Would appreciate your thoughts, keeping in mind that if this change is going to happen any time soon, it needs to land in

Re: [DISCUSS] KIP-618: Atomic commit of source connector records and offsets

2021-03-22 Thread Chris Egerton
matter what, they're still going to see downtime with source connectors. Curious what your thoughts are on this, though. > Also, what are the states of the source tasks during this time? Covered above in response to another question, but to reiterate: they will be UNASSIGNED. 16. Good call, add

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

2021-03-31 Thread Chris Egerton
Hi all, @Sophie - I like the sound of the dual-protocol default. The smooth upgrade path it permits sounds fantastic! @Luke - Do you think we can also include Connect in this KIP? Right now we don't set any custom partition assignment strategies for the consumer groups we bring up for sink

Re: [DISCUSS] KIP-618: Atomic commit of source connector records and offsets

2021-03-22 Thread Chris Egerton
e just point me to that > > section. > > > > Finally, regarding Guozhang's question about "partial rebalances" rather > > than fencing out all producers. In your initial response, you said the > > following: > > > > Connectors can arbitrarily re

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

2021-04-12 Thread Chris Egerton
Whoops, small correction--meant to say ConsumerRebalanceListener::onPartitionsLost, not Consumer::onPartitionsLost On Mon, Apr 12, 2021 at 8:17 AM Chris Egerton wrote: > Hi Sophie, > > This sounds fantastic. I've made a note on KAFKA-12487 about being sure to > implem

Re: [DISCUSS] KIP-726: Make the CooperativeStickyAssignor as the default assignor

2021-04-12 Thread Chris Egerton
instead of > > > #onPartitionsRevoked, and set a flag to block any attempts at > committing > > > those partitions which have been revoked. If a commit is attempted, as > > may > > > be the case if the user does not implement #onPartitionsLost (see > > &g

Re: [DISCUSS] KIP-618: Atomic commit of source connector records and offsets

2021-04-12 Thread Chris Egerton
of rebalancing bug took place and two workers believed they owned the same Connector object. I hope this answers any outstanding questions and look forward to your thoughts. Cheers, Chris On Mon, Mar 22, 2021 at 4:38 PM Chris Egerton wrote: > Hi Randall, > > No complaints about email siz

Re: [DISCUSS] KIP-618: Atomic commit of source connector records and offsets

2021-02-22 Thread Chris Egerton
ctors for which task configurations are present in the config > > topic after the latest task count record" - how will this play with > > the incremental rebalances? isn't this exactly the stop-the-world > > rebalance we want to avoid? > > 6. "the worker will ins

Re: [DISCUSS] KIP-618: Atomic commit of source connector records and offsets

2021-02-22 Thread Chris Egerton
I'm happy to remove it, but seeing as it serves a real use case and comes fairly cheap, I figured it'd be best to include now. Thanks again for your feedback; if you have other thoughts I'd love to hear them! Cheers, Chris On Mon, Feb 22, 2021 at 1:57 PM Chris Egerton wrote: > Hi Gwen, >

Re: [DISCUSS] KIP-618: Atomic commit of source connector records and offsets

2021-02-22 Thread Chris Egerton
d like to make sure > that my understanding here is correct :) > > Guozhang > > > On Mon, Feb 22, 2021 at 11:29 AM Chris Egerton > wrote: > > > Hi Guozhang, > > > > Thanks for taking a look, and for your suggestion! > > > > I think there is room fo

Re: KIP-769: Connect API to retrieve connector configuration definitions

2021-08-19 Thread Chris Egerton
Hi Mickael, I'm wondering about the use case here. The motivation section states that "Connect does not provide a way to see what configurations a connector requires. Instead users have to go look at the connector documentation or in the worst case, look directly at the connector source code.",

Re: [VOTE] 2.8.1 RC1

2021-09-14 Thread Chris Egerton
Hi David, I took a look at the Javadocs and noticed a small issue. Using the search bar at the top of the landing page ( https://home.apache.org/~dajac/kafka-2.8.1-rc1/javadoc/), I entered "KafkaProducer" and clicked on the search item that came up. This brought me to

Re: [DISCUSS] KIP-767 Connect Latency Metrics

2021-09-07 Thread Chris Egerton
Hi Jordan, Thanks for the KIP. I'm curious about a possible alternative where the consumer lag for the source connector can be monitored instead of the newly-proposed metric in the KIP. Although sink tasks can't directly report the successful write of a record to the sink system, they are

Re: [DISCUSS] Apache Kafka 3.1.0 release

2021-09-28 Thread Chris Egerton
Hi David, Wondering if we can get KIP-618 included? The vote passed months ago and a PR has been available since mid-June. Cheers, Chris On Tue, Sep 28, 2021 at 12:53 PM Kirk True wrote: > Hi David, > > Is it possible to try to get KIP-768 in 3.1? I have put it up for a vote > and have much

Re: [DISCUSS] Apache Kafka 3.1.0 release

2021-09-29 Thread Chris Egerton
t; > > > Yes, it is definitely possible if you can get the KIP voted before the > KIP > > freeze > > and the code committed before the feature freeze. Please, let me know > when > > the > > KIP is voted and I will add it to the release plan. > > >

Re: Handling retriable exceptions during Connect source task start

2021-11-29 Thread Chris Egerton
Hi Gunnar, I think there's some risk of introducing this retry behavior if we end up invoking Connector::start or Task::start on the same object multiple times. Unexpected behavior may result, such as double-allocation of resources that are initialized in the start method and which are meant to

Re: KIP-769: Connect API to retrieve connector configuration definitions

2021-12-03 Thread Chris Egerton
t; additional attack vectors, despite not really being able to use the > > resulting information in any other part of the API. > > > > Best regards, > > > > Randall > > > > [1] > https://cwiki.apache.org/confluence/display/KAFKA/KIP-507%3A+Securing+Inte

Re: [DISCUSS] KIP-802: Validation Support for Kafka Connect SMT Options

2021-12-21 Thread Chris Egerton
! Some more answers > inline. > > Am Di., 7. Dez. 2021 um 23:49 Uhr schrieb Chris Egerton > : > > > Hi Gunnar, > > > > Thanks for the KIP! The section on backwards compatibility is especially > > impressive and was enjoyable to read. > > > > Excell

Re: Do we want to add more SMTs to Apache Kafka?

2021-12-29 Thread Chris Egerton
Hi all, I think restricting the set of out-of-the-box SMTs that we provide with Connect is reasonable. I do think Joshua raises a valuable point, though. At the risk of reiterating his ideas, we can gain a few things from improving the existing SMTs provided with Connect: first, we can establish

Re: [DISCUSS] Apache Kafka 3.1.0 release

2021-11-22 Thread Chris Egerton
; if not, as with KAFKA-13469, we may want to revert the changes for the PR that cause this issue (in this case, that'd be the PR for KAFKA-12487). Cheers, Chris On Mon, Nov 22, 2021 at 11:42 AM Chris Egerton wrote: > Hi David, > > I'd like to propose KAFKA-13469 ( > https://issues.apache.org

Re: KIP-769: Connect API to retrieve connector configuration definitions

2021-11-23 Thread Chris Egerton
ature request I'd have is to expose the valid enum > > constants for enum-typed options. That'll help to display the values in a > > drop-down or via radio buttons in a UI, give us tab completion in kcctl, > > etc. > > > > Best, > > > > --Gunnar > > >

  1   2   3   4   5   6   7   >