Re: KIP-28 does not allow Processor to specify partition of output message

Created https://issues.apache.org/jira/browse/KAFKA-2649 and attached a PR with the proposed change. Thanks! On October 14, 2015 at 3:12:34 AM, Guozhang Wang (wangg...@gmail.com) wrote: Thanks! On Tue, Oct 13, 2015 at 9:34 PM, Randall Hauch <rha...@gmail.com> wrote: Ok, cool. I agree w

Re: KIP-28 does not allow Processor to specify partition of output message

Perhaps we should add a new Partitioner interface in Kafka Streams? > > Guozhang > >> On Tue, Oct 13, 2015 at 6:38 PM, Randall Hauch <rha...@gmail.com> wrote: >> This overrides the partitioning logic for all topics, right? That means I >> have to explicitly ca

Re: KIP-28 does not allow Processor to specify partition of output message

up. Thus, a single class representing all partitioning logic is not a bad thing at all. (The code organization wise, all logic does not necessarily in the single class, of course.) On Wed, Oct 14, 2015 at 8:47 AM, Randall Hauch <rha...@gmail.com> wrote: > Crea

KIP-28 does not allow Processor to specify partition of output message

a JIRA and create a PR to add it to the API, but without knowing all of the history I’m wondering if leaving it out of the API was intentional. Thoughts? Best regards, Randall Hauch

Re: KIP-28 does not allow Processor to specify partition of output message

in org.apache.kafka.clients.producer.Partitioner Let me know if it works for you. Guozhang On Tue, Oct 13, 2015 at 10:59 AM, Randall Hauch <rha...@gmail.com> wrote: > The new streams API added with KIP-28 is great. I’ve been using it on a > prototype for a few weeks, and I’m looking forward to it be

Additional Kafka Connect schema logical types?

cleaner, though it does increase the verbosity of the Schema object. Thoughts? Randall Hauch [1]  https://github.com/debezium/debezium/blob/74c5adcc8d30afaa221bbdbecad3bb6f6febbaa5/debezium-core/src/main/java/io/debezium/data/Bits.java [2]  https://github.com/debezium/debezium/blob

Re: Kafka Connect connector monitoring tasks via offset storage reader

://github.com/apache/kafka/blob/trunk/connect/api/src/main/java/org/apache/kafka/connect/connector/Connector.java Thanks,  Liquan  On Tue, May 24, 2016 at 1:31 PM, Randall Hauch <rha...@gmail.com> wrote:  > I have a need for one of my SourceConnector implementations to configure a  > bu

Kafka Connect connector monitoring tasks via offset storage reader

compatible, and non-intrusive. Is there any interest in this? If so, I can create an issue and work on a pull request. Best regards, Randall Hauch

Re: [DISCUSS] KIP-131 : Add access to OffsetStorageReader from SourceConnector

Sorry it's taken me so long to come back to this. Have you considered creating a `SourceConnectorContext` interface that extends `ConnectorContext` and that adds the method to access the offset storage? This would very closely match the existing `SourceTaskContext`. `SourceConnector`

Re: Kafka Connect and Partitions

The source connector creates SourceRecord object and can set a number of fields, including the message's key and value, the Kafka topic name and, if desired, the Kafka topic partition number. If the connector does se the the topic partition to a non-null value, then that's the partition to which

Re: [DISCUSS] KIP-154 Add Kafka Connect configuration properties for creating internal topics

wrote: > So, when Kafka broker is less than 3, and the user has not set the > replication configuration it will throw an error to the user, to correct > the configuration according to his setup? Is this the approach you are > suggesting here? > > > > On Mon, May 8, 2017

[DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

Hi, all. We recently added the ability for Kafka Connect to create *internal* topics using the new AdminClient, but it still would be great if Kafka Connect could do this for new topics that result from source connector records. I've outlined an approach to do this in "KIP-158 Kafka Connect

Re: Kafka Connect: To much restarting with a SourceConnector with dynamic set of tasks

You're not doing anything wrong, but I suspect you're requesting task reconfiguration more frequently than was originally envisioned, which means that the current implementation is not as optimal for your case. I'm not sure how much effort is required to implement this new behavior. The logic for

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

the topic name will be used, and no other rules will be > applied. Is that correct? (I wasn't sure if it might cascade > together multiple matching rules...) > > Looks great, > > Mathieu > > > On Mon, May 22, 2017 at 1:43 PM, Randall Hauch <rha...@gmail.com> wrote:

Re: [DISCUSS] KIP-146: Classloading Isolation in Connect

Very nice work, Konstantine. Conflicting dependencies of connectors is indeed a big issue that makes it hard to manage installed connectors. I do like Gwen's idea about removing the 'module.isolation.enabled' property. However, I would have anticipated always using classpath isolation for *only*

Re: Kafka Connect and Partitions

chema, Object value) > > Many thanks once again for your guidance. I think this puzzle is now > solved :) > Best wishes, > David > > -Original Message- > From: Randall Hauch [mailto:rha...@gmail.com] > Sent: 28 April 2017 16:08 > To: dev@kafka.apache.org

Please give me permission to create/edit Kafka Improvement Proposals

See subject. I'm a contributor to Kafka Connect. Thanks! Randall

[VOTE] KIP-154 Add Kafka Connect configuration properties for creating internal topics

Hi, everyone. Given the simple and non-controversial nature of the KIP, I would like to start the voting process for KIP-154: Add Kafka Connect configuration properties for creating internal topics:

[DISCUSS] KIP-154 Add Kafka Connect configuration properties for creating internal topics

Hi, all. I've been working on KAFKA-4667 to change the distributed worker of Kafka Connect to look for the topics used to store connector and task configurations, offsets, and status, and if those tasks do not exist to create them using the new AdminClient. To make this as useful as possible and

Re: [DISCUSS] KIP-154 Add Kafka Connect configuration properties for creating internal topics

on I recommend leaving them out of the example > config files - we already have issues with users playing with configs > without understanding what they are doing and not liking the results. > >> On Fri, May 5, 2017 at 3:42 PM, Randall Hauch <rha...@gmail.com> wrote: >&g

Re: [DISCUSS] KIP-154 Add Kafka Connect configuration properties for creating internal topics

introduced config > > properties. > > > > This is by no means a blocking comment. > > > > Thanks, > > Konstantine > > > > On Fri, May 5, 2017 at 7:18 PM, Randall Hauch <rha...@gmail.com> wrote: > > > > > Thanks, Gwen. > > &g

Re: [DISCUSS] KIP-196: Add metrics to Kafka Connect framework

Ack. > * Why worker-level offset commit stats? It's not clear to me that these are > useful without considering the specific connector. > So would they make more sense on the tasks? Again, on the worker they're aggregates. > > -Ewen > > > On Mon, Sep 11, 2017 at 9:43 AM, Randall

Re: [DISCUSS] KIP-196: Add metrics to Kafka Connect framework

count/total metrics are only valid since the most recent rebalance and are therefore less meaningful, and were removed. On Mon, Sep 11, 2017 at 6:50 PM, Randall Hauch <rha...@gmail.com> wrote: > Thanks, Ewen. Comments inline below. > > On Mon, Sep 11, 2017 at 5:46 PM, Ewen Ches

Re: [VOTE] KIP-196: Add metrics to Kafka Connect framework

187+-+Add+cumulative+count+metric+for+all+Kafka+rate+metrics On Wed, Sep 13, 2017 at 9:53 PM, Randall Hauch <rha...@gmail.com> wrote: > The KIP has passed with three binding +1 votes (Gwen, Sriram, Jason) and > no -1 or +0 votes. > > Thanks to everyone for the feedback. >

Re: [VOTE] KIP-196: Add metrics to Kafka Connect framework

Subramanian <r...@confluent.io> > wrote: > > > +1 > > > > On Tue, Sep 12, 2017 at 12:41 PM, Gwen Shapira <g...@confluent.io> > wrote: > > > > > My +1 remains :) > > > > > > On Tue, Sep 12, 2017 at 12:31 PM Randall Hauch <rha...@

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

eers > >> On Wed, Oct 4, 2017 at 9:58 AM, Randall Hauch <rha...@gmail.com> wrote: >> >> Currently the KIP's scope is only topics that don't yet exist, and we have >> to cognizant of race conditions between tasks with the same connector. I >> think it is wo

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

pose > of ensuring auto-created topics have the right config. > > Mathieu > > > On Tue, May 23, 2017 at 11:38 AM, Randall Hauch <rha...@gmail.com> > wrote: > > Thanks for the quick feedback, Mathieu. Yes, the first configuration > rule >

Re: [DISCUSS] KIP-208: Add SSL support to Kafka Connect REST interface

ameter with > comma > > > separated list. What do you think? > > > > > > As for the "rest" prefix - if we remove it, some of the same > > configuration > > > options are already used today as the option for connecting from Kafka > > > Conne

Re: [DISCUSS] KIP-208: Add SSL support to Kafka Connect REST interface

to support both "http" and "https" simultaneously, so IMO we should add that from the beginning. On Fri, Oct 13, 2017 at 9:34 AM, Randall Hauch <rha...@gmail.com> wrote: > It'd be useful to specify the default values for the configuration > properties. > > On Tue,

Re: Kafka Connect Source questions

On Thu, Oct 12, 2017 at 1:59 AM, Stephane Maarek < steph...@simplemachines.com.au> wrote: > Hi, > > > > I had a look at the Connect Source Worker code and have two questions: > When a Source Task commits offsets, does it perform compaction / > optimisation before sending off? E.g. I read from 1

Re: [DISCUSS] KIP-208: Add SSL support to Kafka Connect REST interface

It'd be useful to specify the default values for the configuration properties. On Tue, Oct 10, 2017 at 2:53 AM, Jakub Scholz wrote: > FYI: Based on Ewen's suggestion from the related JIRA, I added a > clarification to the KIP that it doesn't do anything around authorization / >

Re: [DISCUSS] KIP-197: Include Connector type in Connector REST API

Hi, Ted. Thanks for creating this KIP and for working on the implementation. The proposal looks great for the "/connectors/{name}" endpoint, but there are several others that we need to consider as well so that the responses are all consistent. In particular, look at "/connectors/{name}/status"

Re: [DISCUSS] KIP-197: Include Connector type in Connector REST API

; wrote: > Thanks for the reminder, Randall. > > I have modified the KIP to include these two endpoints. > > On Fri, Sep 8, 2017 at 11:00 AM, Randall Hauch <rha...@gmail.com> wrote: > > > Hi, Ted. > > > > Thanks for creating this KIP and for

Re: [DISCUSS] KIP-197: Include Connector type in Connector REST API

the KIP accordingly. > > Cheers > > On Fri, Sep 8, 2017 at 3:57 PM, Randall Hauch <rha...@gmail.com> wrote: > > > Hi, Ted. Thanks for the quick turn around. > > > > I didn't remember that the response to the "/connectors/{name}/config&q

Re: [DISCUSS] KIP-197: Include Connector type in Connector REST API

Looks good to me! Thanks again! I say go ahead and ask for a vote in a new thread. Randall On Fri, Sep 8, 2017 at 6:22 PM, Ted Yu <yuzhih...@gmail.com> wrote: > Lowercase constants are generated by the enum. > > Updated KIP again. > > On Fri, Sep 8, 2017 at 4:08 P

Re: [VOTE] KIP-197: Include Connector type in Connector REST API

+1 (non-binding) Randall On Fri, Sep 8, 2017 at 6:32 PM, Ted Yu wrote: > Hi, > Please take a look at the following and cast your vote: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 197+Connect+REST+API+should+include+the+connector+type+ >

Re: [VOTE] KIP-131 - Add access to OffsetStorageReader from SourceConnector

AM Florian Hussonnois <fhussonn...@gmail.com> > wrote: > > > Hi Shapira, > > > > We only expose the OffsetStorageReader to connector which relies on > > KafkaOffsetBackingStore. The store continuesly consumes offsets from > kafka > > so I think we can't have

[VOTE] KIP-199: Add Kafka Connect Offset Tool

I'd like to start the vote on KIP-199 to add a command line tool that will allow Connect operators to read, modify, and update source connector offsets. Details are here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-199%3A+Add+Kafka+Connect+offset+tool Thanks, and best regards. Randall

[VOTE] KIP-196: Add metrics to Kafka Connect framework

I'd like to start the vote on KIP-196 to add metrics to the Kafka Connect framework so the worker processes can be measured. Details are here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-196%3A+Add+metrics+to+Kafka+Connect+framework Thanks, and best regards. Randall

Re: [DISCUSS] KIP-196: Add metrics to Kafka Connect framework

stinction in the Connect framework. > - Batching stats > - Histogram of flush batch size > - Counts of flush trigger method (time vs max batch size) > Should be able to add these. > > Cheers, > > Roger > > On Sun, Sep 10, 2017 at 8:

Re: [DISCUSS] KIP-131 : Add access to OffsetStorageReader from SourceConnector

I will update the KIP and the PR with your last approach which sounds > better. > > Thanks. > > Le 16 août 2017 00:53, "Randall Hauch" <rha...@gmail.com> a écrit : > > Sorry it's taken me so long to come back to this. > > Have you considered creating a `So

[VOTE] KIP-131 - Add access to OffsetStorageReader from SourceConnector

I'd like to open the vote for KIP-131: https://cwiki.apache.org/confluence/display/KAFKA/KIP-131+-+Add+access+to+OffsetStorageReader+from+SourceConnector Thanks to Florian for submitting the KIP and the implementation, and to everyone else that helped review. Best regards, Randall

[DISCUSS] KIP-196: Add metrics to Kafka Connect framework

Hi everyone. I've created a new KIP to add metrics to the Kafka Connect framework: https://cwiki.apache.org/confluence/display/KAFKA/KIP-196%3A+Add+metrics+to+Kafka+Connect+framework The KIP approval deadline is looming, so if you're interested in Kafka Connect metrics please review and provide

Re: [VOTE] KIP-131 - Add access to OffsetStorageReader from SourceConnector

> >> On Thu, Sep 7, 2017 at 5:04 PM Gwen Shapira <g...@confluent.io> wrote: >> >> +1 (binding) >> >> Looking forward to see how connector implementations use this in practice >> :) >> >>> On Thu, Sep 7, 2017 at 3:49 PM Randall Hauch <

Re: [DISCUSS] KIP-196: Add metrics to Kafka Connect framework

t; > > > > Gwen > > > > > > > > On Mon, Sep 11, 2017 at 7:50 PM Ewen Cheslack-Postava < > > e...@confluent.io > > > > > > > > wrote: > > > > > > > > > re: questions about additional metrics, I think we

[DISCUSS] KIP-199: Add Kafka Connect offset reset tool

Hi all, KIP-199 [1] describes a new tool that will allow Connect operators to read, update, and remove offsets stored by Connect runtime. This capability has been often asked for by Connect users. The proposal is simple but flexible. Please review and add feedback. Best regards, Randall [1]

Re: [DISCUSS] KIP-196: Add metrics to Kafka Connect framework

ortant to know both numbers. > > > Gwen > > On Thu, Sep 7, 2017 at 7:50 PM, Randall Hauch <rha...@gmail.com> wrote: > > > Hi everyone. > > > > I've created a new KIP to add metrics to the Kafka Connect framework: > > https://cwiki.apache.org/conf

Re: [DISCUSS] KIP-199: Add Kafka Connect offset reset tool

> Better note this in the KIP. > Addressed in a few places. Hopefully it is more clear now. > > bq. but will remove the offsets for the partition with file "b" > > Please move the sample JSON below the above description. > Done. > > Cheers > > On Sun, Sep 10

Re: [DISCUSS] KIP-196: Add metrics to Kafka Connect framework

t; > implementation just does something inefficient), but I'm not really sure > > about that yet. > > > > -Ewen > > > > On Mon, Sep 11, 2017 at 7:11 PM, Randall Hauch <rha...@gmail.com> wrote: > > > > > Based on Roger and Ewen's feedback, I

Re: [VOTE] KIP-196: Add metrics to Kafka Connect framework

o> wrote: > > > +1 > > > > Thanks for this. Can't wait for more complete monitoring for Connect. > > > > On Mon, Sep 11, 2017 at 7:40 AM Randall Hauch <rha...@gmail.com> wrote: > > > > > I'd like to start the vote on KIP-196 to add metrics to

Re: [DISCUSS] KIP-196: Add metrics to Kafka Connect framework

Okay, I think I've incorporated all feedback except for Gwen and Roger than would like to have timing metrics. Given the deadline and Ewen's concern about degraded performance, I think it's prudent to leave those out of this KIP and proceed as is. On Tue, Sep 12, 2017 at 12:48 PM, Randall Hauch

Re: [DISCUSS] KIP-196: Add metrics to Kafka Connect framework

e attributes > are named identically ("sink-record-lag-avg" instead of > "sink-record-{topic}-{partition}.records-lag-avg"), so monitoring systems > have a consistent string they can use, instead of needing to > prefix-and-suffix matching against the attribute name. And T

Re: [VOTE] KIP-199: Add Kafka Connect Offset Tool

On Mon, Sep 11, 2017 at 2:56 PM, Gwen Shapira <g...@confluent.io> wrote: > > > +1 > > > > On Mon, Sep 11, 2017 at 1:33 PM Ted Yu <yuzhih...@gmail.com> wrote: > > > > > +1 > > > > > > On Mon, Sep 11, 2017 at 7:43 AM, Randall Hauc

Re: [DISCUSS] KIP-199: Add Kafka Connect offset reset tool

his, including offset > delete, is to stop the connector, apply operation, then continue. > ack. > > -Ewen > > > On Sun, Sep 10, 2017 at 3:12 PM, Ted Yu <yuzhih...@gmail.com> wrote: > > > bq. How about calling it kafka-connect-source-offset-tool.sh > > > > This is bett

Re: [DISCUSS] KIP-212: Enforce set of legal characters for connector names

Here's the link to KIP-212: https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=74684586 I do think it's worthwhile to define the rules for connector names. However, I think it would be better to describe the current restrictions for names outside of them appearing within URLs. For

Re: [DISCUSS] KIP 145 - Expose Record Headers in Kafka Connect

after conversion is > integer you can still ask for float and it will type concert if possible. > > Cheers > Mike > > > Sent from my iPhone > > > On 13 Dec 2017, at 01:36, Randall Hauch <rha...@gmail.com> wrote: > > > > Trying to revive this after sev

Re: [DISCUSS] KIP-238: Expose Kafka cluster ID in Connect REST API

Thanks, Ewen. I think the KIP is clear enough about the intent and the changed behavior. On Tue, Dec 12, 2017 at 12:22 AM, Ewen Cheslack-Postava wrote: > And to clarify a bit further: the goal is for both standalone and > distributed mode to display the same basic

Re: [DISCUSS] KIP 145 - Expose Record Headers in Kafka Connect

Trying to revive this after several months of inactivity I've spent quite a bit of time evaluating the current KIP-145 proposal and several of the suggested PRs. The original KIP-145 proposal is relatively minimalist (which is very nice), and it adopts Kafka's approach to headers where header

Re: [DISCUSS] KIP-212: Enforce set of legal characters for connector names

er. > > Let me know what you think. > > Kind regards, > Sönke > > ᐧ > >> On Tue, Nov 14, 2017 at 7:03 PM, Randall Hauch <rha...@gmail.com> wrote: >> >> Thanks for updating the KIP to reflect the current process. However, I >> still questi

Re: [DISCUSS] KIP-212: Enforce set of legal characters for connector names

gt; On Thu, Nov 16, 2017 at 2:59 PM, Sönke Liebau <soenke.lie...@opencore.com> > wrote: > > > Hi Randall, > > > > I had mentioned this edge case in the KIP, but will add some further > > detail to further clarify all changing scenarios post pull request. > >

Re: [DISCUSS] KIP-212: Enforce set of legal characters for connector names

uite small and probably won't > even be noticed by many people.. > > best regards, > Sönke > > On Thu, Nov 9, 2017 at 9:10 PM, Randall Hauch <rha...@gmail.com> wrote: > > > Any progress on updating the PR and withdrawing KIP-212? > > > > On Fri, Oct 27,

Re: [VOTE] KIP-215: Add topic regex support for Connect sinks

+1 (non-binding) Thanks for pushing this through. Great work! Randall Hauch On Wed, Nov 1, 2017 at 9:40 AM, Jeff Klukas <jklu...@simple.com> wrote: > I haven't heard any additional concerns over the proposal, so I'd like to > get the voting process started for: > > https:

Re: [VOTE] KIP-215: Add topic regex support for Connect sinks

; > > > > > Thanks all for looking this over. > > > > > > > > > > On Fri, Nov 3, 2017 at 7:31 PM, Jason Gustafson < > ja...@confluent.io> > > > > > wrote: > > > > > > > > > &g

Re: [DISCUSS] KIP-212: Enforce set of legal characters for connector names

Any progress on updating the PR and withdrawing KIP-212? On Fri, Oct 27, 2017 at 5:19 PM, Randall Hauch <rha...@gmail.com> wrote: > Yes, connector names should not be blank or contain just whitespace. In > fact, I might recommend that we trim whitespace at the front and rear of >

Re: [DISCUSS] KIP-285: Connect Rest Extension Plugin

a replica of the required > entities in the draft implementation. If you can take a look at the PR and > let me know your thoughts, I will update the KIP to reflect the same > > https://github.com/apache/kafka/pull/4931 > > On Tue, Apr 24, 2018 at 11:44 AM, Randall Hauch <rha...@gmail.com

Re: [DISCUSS] KIP-305: Add Connect primitive number converters

g if they exist. > > -Ewen > >> On Fri, May 18, 2018 at 11:55 AM Randall Hauch <rha...@gmail.com> wrote: >> >> Thanks, Ewen. >> >> You make several good points, and I've updated the KIP to hopefully address >> your comments. I think the symmetr

Re: [DISCUSS] KIP-285: Connect Rest Extension Plugin

ncludes details on packages for the interfaces and > > the classes. Let me know your thoughts. > > > > Thanks > > Magesh > > > > On Fri, Apr 27, 2018 at 12:03 PM, Randall Hauch <rha...@gmail.com> > wrote: > > > >> Great work, Magesh. I like the ove

Re: [VOTE] KIP-298: Error Handling in Connect kafka

Thanks, Arjun. +1 (non-binding) Regards, Randall On Mon, May 21, 2018 at 11:14 AM, Guozhang Wang wrote: > Thanks for the KIP. +1 (binding) > > > Guozhang > > On Fri, May 18, 2018 at 3:36 PM, Gwen Shapira wrote: > > > +1 > > > > Thank you! Error handling

Re: [DISCUSS] KIP-242: Mask password fields in Kafka Connect REST response

fashion. > > > > @Ewen - Regarding the concern of accessing information securely - has > > there been any consideration of adding authentication to the connect api? > > > > > On Jan 17, 2018, at 3:55 PM, Randall Hauch <rha...@gmail.com> wrote: > &g

[VOTE] KIP-305: Add Connect primitive number converters

I'd like to start a vote of a very straightforward proposal for Connect to add converters for the basic primitive number types: integer, short, long, double, and float that reuse Kafka's corresponding serdes. Here is the KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-

Re: [VOTE] KIP-305: Add Connect primitive number converters

ack-Postava < > e...@confluent.io> > > wrote: > > > >> +1 (binding) > >> > >> -Ewen > >> > >> On Tue, May 22, 2018 at 9:29 AM Ted Yu <yuzhih...@gmail.com> wrote: > >> > >>> +1 > >>> > &

Re: [VOTE] KIP-305: Add Connect primitive number converters

(bump so a few new subscribers see this thread.) On Tue, May 22, 2018 at 4:39 PM, Randall Hauch <rha...@gmail.com> wrote: > +1 (non-binding) > > On Tue, May 22, 2018 at 4:05 PM, Matthias J. Sax <matth...@confluent.io> > wrote: > >> +1 (binding) >> &g

Re: [VOTE] KIP-305: Add Connect primitive number converters

wrote: > > +1 > > > > Thanks, > > Yeva > > > > > > On Tue, May 22, 2018 at 7:48 PM, Magesh Nandakumar <mage...@confluent.io > > wrote: > > > > > +1 (non-binding) > > > > > > Thanks > > > Magesh > > >

Re: [VOTE] KIP-285: Connect Rest Extension Plugin

+1 (non-binding) Regards, Randall On Fri, May 18, 2018 at 11:32 AM, Konstantine Karantasis < konstant...@confluent.io> wrote: > +1 (non-binding) > > - Konstantine > > On Thu, May 17, 2018 at 10:05 PM, Ewen Cheslack-Postava > > wrote: > > > +1 (binding) > > > > Thanks, > >

Re: [VOTE] KIP-297: Externalizing Secrets for Connect Configurations

Looks great. +1 (non-binding) Regards, Randall On Fri, May 18, 2018 at 10:23 AM, Rajini Sivaram wrote: > Thanks, Robert! Sounds good. And thanks for the KIP. > > +1 (binding) > > Regards, > > Rajini > > On Fri, May 18, 2018 at 4:17 PM, Robert Yokota

Re: [DISCUSS] KIP-301 Schema Inferencing in JsonConverters

There are lots of potential details/gotchas here, with no obvious defaults. Even if we did have optional configs for the type(s) to use for empty arrays and null values, how likely is it that those would apply for all occurrences? Two fields might have empty arrays, but the best schema for each

[DISCUSS] KIP-305: Add Connect primitive number converters

I'd like to start discussion of a very straightforward proposal for Connect to add converters for the basic primitive number types: integer, short, long, double, and float. Here is the KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-305%3A+Add+Connect+primitive+number+converters As

Re: [DISCUSS] KIP-305: Add Connect primitive number converters

> > -Ewen > > On Thu, May 17, 2018 at 6:04 PM Magesh Nandakumar <mage...@confluent.io> > wrote: > > > Thanks Randall for the KIP. I think it will be super useful and looks > > pretty straightforward to me. > > > > Thanks > > Magesh > >

Re: [DISCUSS] KIP-285: Connect Rest Extension Plugin

gt; minor edits per your suggestions. Since there are no major outstanding > issues, i'm moving this to vote. > > Thanks > Magesh > >> On Wed, May 16, 2018 at 4:38 PM, Randall Hauch <rha...@gmail.com> wrote: >> >> A few very minor suggestions: >> &

Re: Doubt about developing a kafka connect with avro

Sounds like you're using Kafka Connect and trying to write a source connector. The Connect framework separates responsibilities: 1. Your source connector is responsible for generating the SourceRecord objects and returning them to the Connect framework via your task's "poll()" method. 2. The

Re: Before creating KIP : Kafka Connect / Add a configuration provider class

Very interesting. Would the proposed configuration provider be set at the connector level or the worker level? The latter would obviously be required to handle all/multiple connector configurations. Either way, the provider class(es) would need to be installed on the worker (really, every worker),

Re: [DISCUSS] KIP-212: Enforce set of legal characters for connector names

> I will further investigate which characters the url decoding that > jetty > > > > brings to the table will let us use and if all of these are correctly > > > > handled during connector creation and report back with a new list of > > > > characters tha

Re: [DISCUSS] KIP 145 - Expose Record Headers in Kafka Connect

All, I've updated KIP-145 to reflect my proposal. The proposal addresses SMTs and a different HeaderConverter default, but I'll be updating my PR ( https://github.com/apache/kafka/pull/4319) soon. Feedback is very welcome! Best regards, Randall On Thu, Dec 14, 2017 at 10:20 AM, Randall Hauch

Re: [DISCUSS] KIP-212: Enforce set of legal characters for connector names

ov 16, 2017 at 5:35 PM, Sönke Liebau < > soenke.lie...@opencore.com> > > wrote: > > > > > Sounds good. I've added a few sentences to this effect to the KIP. > > > > > > On Thu, Nov 16, 2017 at 5:02 PM, Randall Hauch <rha...@gmail.com> >

Re: [DISCUSS] KIP-212: Enforce set of legal characters for connector names

place. > > -Ewen > > On Fri, Jan 5, 2018 at 8:10 AM, Randall Hauch <rha...@gmail.com> wrote: > > > Sönke, I'm happy with the current proposal. > > > > Ewen, the proposal allows any characters in the name as long as they are > > properly escaped/encoded.

Re: [DISCUSS] KIP-208: Add SSL support to Kafka Connect REST interface

ised > > address. > > > So I decided to reuse the existing rest.advertised.host.name and > > > rest.advertised.port options and add additional option > > > rest.advertised.security.protocol to specify whether HTTP or HTTPS > > should > > &

Re: [DISCUSS] KIP 145 - Expose Record Headers in Kafka Connect

SimpleHeaderConverter makes a lot of sense to me. Agree that this > > introduces overhead, but perhaps simply documenting an easy > "optimization" > > will be enough to help those who are concerned about it? Since the > > connector-devs decide whether they'l

Re: [DISCUSS] KIP-208: Add SSL support to Kafka Connect REST interface

Jakub, have you had a chance to update the KIP with the latest changes? Would be great to start the vote today so that it's open long enough to adopt before the deadline on Tuesday. Let me know if I can help. On Wed, Jan 17, 2018 at 1:25 AM, Jakub Scholz wrote: > I have been

Re: [DISCUSS] KIP-212: Enforce set of legal characters for connector names

t; >> > > > >> > > > >> > > > > >> > In general my default approach (without context of a specific > > system) > > > >> would > > > >> > be to accept anything that we can encode in UTF-8 and only apply &g

[VOTE] KIP-145: Expose Record Headers in Kafka Connect

Hi everyone, I'd like to start the voting on this KIP to add support for headers in Connect.: *https://cwiki.apache.org/confluence/display/KAFKA/KIP-145+-+Expose+Record+Headers+in+Kafka+Connect *

Re: [DISCUSS] KIP-242: Mask password fields in Kafka Connect REST response

Vincent, Can the KIP more explicitly say that this is opt-in, and that by default nothing will change? Randall On Tue, Jan 16, 2018 at 11:18 PM, Ewen Cheslack-Postava wrote: > Vincent, > > I think with the addition of a configuration to control this for > compatibility,

Re: [DISCUSS] KIP 145 - Expose Record Headers in Kafka Connect

://github.com/apache/kafka/pull/4319 Still looking for feedback on my previous email and these updates. Randall On Wed, Jan 17, 2018 at 10:57 AM, Randall Hauch <rha...@gmail.com> wrote: > Ewen and Michael, > > Thanks for the feedback. I've updated the KIP based upon your feedback. >

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

, Randall On Thu, Oct 5, 2017 at 12:13 AM, Randall Hauch <rha...@gmail.com> wrote: > Oops. Yes, I meant “replication factor”. > > > On Oct 4, 2017, at 7:18 PM, Ted Yu <yuzhih...@gmail.com> wrote: > > > > Randall: > > bq. AdminClient current

Re: [VOTE] KIP-174 Deprecate and remove internal converter configs in WorkerConfig

+1 (non-binding) On Mon, Jan 8, 2018 at 7:09 PM, Gwen Shapira wrote: > +1 binding > > On Mon, Jan 8, 2018 at 4:59 PM Ewen Cheslack-Postava > wrote: > > > +1 binding. Thanks for the KIP! > > > > -Ewen > > > > On Mon, Jan 8, 2018 at 8:34 AM, Ted Yu

Re: [DISCUSS] KIP 145 - Expose Record Headers in Kafka Connect

Does anyone have any thoughts about this proposal for Connect header support? On Thu, Dec 21, 2017 at 4:14 PM, Randall Hauch <rha...@gmail.com> wrote: > All, > > I've updated KIP-145 to reflect my proposal. The proposal addresses SMTs > and a different HeaderConverter

Re: [DISCUSS] KIP 145 - Expose Record Headers in Kafka Connect

to `ByteArrayConverter` so that by default the framework doesn't have to do much effort if headers aren't used/needed. Thoughts? On Tue, Dec 26, 2017 at 11:47 AM, Randall Hauch <rha...@gmail.com> wrote: > Does anyone have any thoughts about this proposal for Connect header > support? > > On

Re: [DISCUSS] KIP-318: Make Kafka Connect Source idempotent

Thanks for starting this conversation, Stephane. I have a few questions. The worker already accepts nearly all producer properties already, and all `producer.*` properties override any hard-coded properties defined in `Worker.java`. So isn't it currently possible for a user to define these

Re: [VOTE] KIP-316: Command-line overrides for ConnectDistributed worker properties

IMO we should not request a vote without additional time for discussion. Best regards, Randall On Wed, Jun 20, 2018 at 7:29 AM, Jakub Scholz wrote: > +1 (non-binding) > > On Mon, Jun 18, 2018 at 8:42 PM Kevin Lafferty > wrote: > > > Hi all, > > > > I got a couple notes of interest on the

Re: [DISCUSS] KIP-316: Command-line overrides for ConnectDistributed worker properties

My only concern with this proposal is that it adds yet another way to specify configuration properties, making it a bit more difficult to track down how/whether a configuration property has been set. Configuring Kafka Connect is already too challenging, so we need to be very clear that this

Re: [DISCUSS] KIP-158: Kafka Connect should allow source connectors to set topic-specific settings for new topics

, Jan 26, 2018 at 4:20 PM Randall Hauch wrote: > The KIP deadline for 1.1 has already passed, but I'd like to restart this > discussion so that we make the next release. I've not yet addressed the > previous comment about *existing* topics, but I'll try to do that over the > next few week

  1   2   3   4   5   6   7   8   >