[jira] [Created] (KAFKA-6915) MirrorMaker: avoid duplicates when source cluster is unreachable for more than session.timeout.ms

Fabien LD created KAFKA-6915: Summary: MirrorMaker: avoid duplicates when source cluster is unreachable for more than session.timeout.ms Key: KAFKA-6915 URL: https://issues.apache.org/jira/browse/KAFKA-6915

Re: [VOTE] KIP-285: Connect Rest Extension Plugin

+1 (binding) Thanks, Ewen On Thu, May 17, 2018 at 12:16 PM Ted Yu wrote: > +1 > Original message From: Gwen Shapira > Date: 5/17/18 12:02 PM (GMT-08:00) To: dev > Subject: Re: [VOTE] KIP-285: Connect Rest

Re: [DISCUSS] KIP-285: Connect Rest Extension Plugin

Yup, thanks for the changes. The 'health' package in particular feels like a nice fit given the way we expect it to be used. -Ewen On Wed, May 16, 2018 at 7:02 PM Randall Hauch wrote: > Looks good to me. Thanks for quickly making the changes! Great work! > > Best regards, > >

Jenkins build is back to normal : kafka-trunk-jdk8 #2652

See

Re: [VOTE] KIP-297: Externalizing Secrets for Connect Configurations

Thanks for addressing this Robert, it's a pretty common user need. First, +1 (binding) generally. Two very minor comments that I think could be clarified but wouldn't affect votes: * Let's list in the KIP what package the ConfigProvider, ConfigChangeCallback, ConfigData and ConfigTransformer

Re: [DISCUSS] KIP-298: Error Handling in Connect

Konstantine, Thanks for pointing out the typos. Fixed them. I had added the JSON schema which should now include key and header configs in there too. This should have been in the public interfaces section. Thanks very much, On Thu, May 17, 2018 at 9:13 AM, Konstantine Karantasis <

Re: [DISCUSS] KIP-305: Add Connect primitive number converters

Just a couple of minor points that don't really affect the implementation: * For nulls, let's just mention the underlying serializers already support this. I'm actually not sure why they should/need to, but given they do, let's just defer to that implementation. * I'm not sure where Float and

Jenkins build is back to normal : kafka-trunk-jdk7 #3435

See

Build failed in Jenkins: kafka-trunk-jdk10 #112

See Changes: [wangguoz] HOTFIX: use ConsumedInternal in StreamsBuilder -- [...truncated 1.50 MB...] kafka.admin.ResetConsumerGroupOffsetTest > testResetOffsetsToEarliest

Re: [VOTE] KIP-297: Externalizing Secrets for Connect Configurations

+1 Original message From: Magesh Nandakumar Date: 5/17/18 6:05 PM (GMT-08:00) To: dev@kafka.apache.org Subject: Re: [VOTE] KIP-297: Externalizing Secrets for Connect Configurations Thanks Robert, this looks great +1 (non-binding) On Thu, May 17, 2018

Re: [VOTE] KIP-297: Externalizing Secrets for Connect Configurations

Thanks Robert, this looks great +1 (non-binding) On Thu, May 17, 2018 at 5:35 PM, Colin McCabe wrote: > Thanks, Robert! > > +1 (non-binding) > > Colin > > > On Thu, May 17, 2018, at 14:15, Robert Yokota wrote: > > Hi Colin, > > > > I've changed the KIP to have a composite

Re: [DISCUSS] KIP-305: Add Connect primitive number converters

Thanks Randall for the KIP. I think it will be super useful and looks pretty straightforward to me. Thanks Magesh On Thu, May 17, 2018 at 4:15 PM, Randall Hauch wrote: > I'd like to start discussion of a very straightforward proposal for Connect > to add converters for the

Re: [VOTE] KIP-297: Externalizing Secrets for Connect Configurations

Thanks, Robert! +1 (non-binding) Colin On Thu, May 17, 2018, at 14:15, Robert Yokota wrote: > Hi Colin, > > I've changed the KIP to have a composite object returned from get(). It's > probably the most straightforward option. Please let me know if you have > any other concerns. > > Thanks,

Re: [VOTE] #2 KIP-248: Create New ConfigCommand That Uses The New AdminClient

Actually, I just realized that this won't work. The AlterConfigs API is kind of broken right now. DescribeConfigs won't return the "sensitive" configurations like passwords. So doing describe + edit + alter will wipe out all sensitive configs. :( I think we should probably just create a

[jira] [Created] (KAFKA-6914) Kafka Connect - Plugins class should have a constructor that can take in parent ClassLoader

Sriram KS created KAFKA-6914: Summary: Kafka Connect - Plugins class should have a constructor that can take in parent ClassLoader Key: KAFKA-6914 URL: https://issues.apache.org/jira/browse/KAFKA-6914

[DISCUSS] KIP-305: Add Connect primitive number converters

I'd like to start discussion of a very straightforward proposal for Connect to add converters for the basic primitive number types: integer, short, long, double, and float. Here is the KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-305%3A+Add+Connect+primitive+number+converters As

Re: [DISCUSS] KIP-302 - Enable Kafka clients to use all DNS resolved IP addresses

Hi Jonathan, > A solution might be to expose to users the choice of using hostname or > canonical host name on both sides. > Say having one setting that collapses functionalities from both KIPs > (bootstrap expansion + advertised lookup) > and an additional parameter that defines how the

Re: [VOTE] KIP-303: Add Dynamic Routing Support in Kafka Streams' Topology Sink

+1 non-binding On Thu, May 17, 2018 at 4:44 PM, Matthias J. Sax wrote: > +1 (binding) > > > On 5/17/18 12:18 PM, Ted Yu wrote: > > +1 > > Original message From: Gwen Shapira > Date: 5/17/18 11:53 AM (GMT-08:00) To: dev

Re: [VOTE] KIP-303: Add Dynamic Routing Support in Kafka Streams' Topology Sink

+1 (binding) On 5/17/18 12:18 PM, Ted Yu wrote: > +1 > Original message From: Gwen Shapira > Date: 5/17/18 11:53 AM (GMT-08:00) To: dev Subject: > Re: [VOTE] KIP-303: Add Dynamic Routing Support in Kafka Streams' Topology > Sink >

Build failed in Jenkins: kafka-trunk-jdk8 #2651

See Changes: [github] KAFKA-6729: Reuse source topics for source KTable's materialized store's [github] HOTFIX: move Conusmed to o.a.k.streams.kstream (#5033) --

[jira] [Created] (KAFKA-6913) Add primitive numeric converters to Connect

Randall Hauch created KAFKA-6913: Summary: Add primitive numeric converters to Connect Key: KAFKA-6913 URL: https://issues.apache.org/jira/browse/KAFKA-6913 Project: Kafka Issue Type:

Re: [VOTE] KIP-297: Externalizing Secrets for Connect Configurations

Hi Colin, I've changed the KIP to have a composite object returned from get(). It's probably the most straightforward option. Please let me know if you have any other concerns. Thanks, Robert On Thu, May 17, 2018 at 11:44 AM, Robert Yokota wrote: > > > Hi Colin, > > My

Build failed in Jenkins: kafka-trunk-jdk7 #3434

See Changes: [github] KAFKA-6729: Reuse source topics for source KTable's materialized store's [github] HOTFIX: move Conusmed to o.a.k.streams.kstream (#5033) --

Build failed in Jenkins: kafka-trunk-jdk10 #111

See Changes: [github] KAFKA-6729: Reuse source topics for source KTable's materialized store's [github] HOTFIX: move Conusmed to o.a.k.streams.kstream (#5033) --

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

Hey Piyush - my bad. Sorry. On 17 May 2018 at 13:23, Piyush Vijay wrote: > It's still not complete. I'll drop a message here when I'm done with the > updates. > > Thanks > > > Piyush Vijay > > On Thu, May 17, 2018 at 12:04 PM, Andy Coates wrote: > > >

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

It's still not complete. I'll drop a message here when I'm done with the updates. Thanks Piyush Vijay On Thu, May 17, 2018 at 12:04 PM, Andy Coates wrote: > Thanks for the update to the KIP Piyush! > > Reading it through again, I've a couple of questions: > > 1. Why is

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

Hi Balint, Out of curiosity - have you ever tried this project https://github.com/coreos/zetcd which tries to provide Zookeeper API for Etcd? Thanks & Regards Jakub On Wed, Mar 28, 2018 at 6:18 PM Molnár Bálint wrote: > Hi all, > > I have created KIP-273: Kafka to

Re: [DISCUSS] KIP-273 Kafka to support using ETCD beside Zookeeper

Hi, To be honest, I'm not sure I understand who is the "we" in the "We're going to post ...". If there are any discussions or plans to replace Zookeeper, then I think they should be done here on the developer list instead of teasing people for several months with some secretive projects. Thanks

Jenkins build is back to normal : kafka-trunk-jdk8 #2650

See

Re: [VOTE] KIP-303: Add Dynamic Routing Support in Kafka Streams' Topology Sink

+1 Original message From: Gwen Shapira Date: 5/17/18 11:53 AM (GMT-08:00) To: dev Subject: Re: [VOTE] KIP-303: Add Dynamic Routing Support in Kafka Streams' Topology Sink Yay, its about time :) +1 On Thu, May 17, 2018 at 12:38 PM,

Re: [VOTE] KIP-285: Connect Rest Extension Plugin

+1 Original message From: Gwen Shapira Date: 5/17/18 12:02 PM (GMT-08:00) To: dev Subject: Re: [VOTE] KIP-285: Connect Rest Extension Plugin LGTM. +1. On Wed, May 16, 2018 at 8:19 PM, Magesh Nandakumar wrote:

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

Thanks for the update to the KIP Piyush! Reading it through again, I've a couple of questions: 1. Why is there a need for a new 'getMatchingAcls' method, over the existing getAcls method? They both take a Resource instance and return a set of Acls. What is the difference in their behaviour? 2.

Re: [VOTE] KIP-285: Connect Rest Extension Plugin

LGTM. +1. On Wed, May 16, 2018 at 8:19 PM, Magesh Nandakumar wrote: > Hello everyone, > > After a good round of discussions with excellent feedback and no major > objections, I would like to start a vote on KIP-285: Connect Rest Extension > Plugin. > > KIP: < >

Jenkins build is back to normal : kafka-trunk-jdk7 #3433

See

Re: [VOTE] KIP-303: Add Dynamic Routing Support in Kafka Streams' Topology Sink

Google splits the URL into two lines and only the first half applies when you click, so it looks very broken. It is easy enough to find though. On Thu, May 17, 2018 at 12:54 PM, Guozhang Wang wrote: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-303% >

Re: [VOTE] KIP-303: Add Dynamic Routing Support in Kafka Streams' Topology Sink

https://cwiki.apache.org/confluence/display/KAFKA/KIP-303% 3A+Add+Dynamic+Routing+in+Streams+Sink works fine on my browser.. anyone else have the same issue? On Thu, May 17, 2018 at 11:51 AM, Damian Guy wrote: > Thanks Guozhang. +1 > > Note: the link to the KIP is broken >

Re: [VOTE] KIP-303: Add Dynamic Routing Support in Kafka Streams' Topology Sink

Yay, its about time :) +1 On Thu, May 17, 2018 at 12:38 PM, Guozhang Wang wrote: > Hello folks, > > I'd like to start a voting thread on adding dynamic routing functionality > in Streams sink node. Please find a KIP here: > >

Re: [VOTE] KIP-303: Add Dynamic Routing Support in Kafka Streams' Topology Sink

Thanks for the KIP. +1 -Bill On Thu, May 17, 2018 at 2:51 PM, Damian Guy wrote: > Thanks Guozhang. +1 > > Note: the link to the KIP is broken > > On Thu, 17 May 2018 at 11:38 Guozhang Wang wrote: > > > Hello folks, > > > > I'd like to start a voting

Re: [VOTE] KIP-303: Add Dynamic Routing Support in Kafka Streams' Topology Sink

Thanks Guozhang. +1 Note: the link to the KIP is broken On Thu, 17 May 2018 at 11:38 Guozhang Wang wrote: > Hello folks, > > I'd like to start a voting thread on adding dynamic routing functionality > in Streams sink node. Please find a KIP here: > >

Re: [VOTE] KIP-297: Externalizing Secrets for Connect Configurations

Hi Colin, My last response was not that clear, so let me back up and explain a bit more. Some secret managers, such as Vault (and maybe Keywhiz) have the notion of a lease duration or a TTL for a path. Every path can have a different TTL. This is period after which the value of the keys at the

[VOTE] KIP-303: Add Dynamic Routing Support in Kafka Streams' Topology Sink

Hello folks, I'd like to start a voting thread on adding dynamic routing functionality in Streams sink node. Please find a KIP here: https://cwiki.apache.org/confluence/display/KAFKA/KIP- 303%3A+Add+Dynamic+Routing+in+Streams+Sink And the PR itself ready for review as well under KAFKA-4936:

Re: [DISCUSS] KIP-303: Add Dynamic Routing in Streams Sink

Thanks Damian. I think I'm convinced to introduce a new class for the sake of semantics clarity plus being able to expose record context along with it. I'm going to update the KIP and start the voting thread now. On Thu, May 17, 2018 at 10:49 AM, Damian Guy wrote: > Hi

[jira] [Resolved] (KAFKA-6729) KTable should use user source topics if possible and not create changelog topic

[ https://issues.apache.org/jira/browse/KAFKA-6729?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-6729. -- Resolution: Fixed > KTable should use user source topics if possible and not create changelog

[jira] [Resolved] (KAFKA-6426) Kafka SASL/SCRAM authentication does not fail for incorrect username or password.

[ https://issues.apache.org/jira/browse/KAFKA-6426?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-6426. -- Resolution: Cannot Reproduce This looks like configuration issue. Please reopen if you think the issue

[jira] [Resolved] (KAFKA-6682) Kafka reconnection after broker restart

[ https://issues.apache.org/jira/browse/KAFKA-6682?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Manikumar resolved KAFKA-6682. -- Resolution: Duplicate Resolving this as duplicate of KAFKA-6260.  Please reopen if the issue still

Build failed in Jenkins: kafka-trunk-jdk10 #110

See Changes: [wangguoz] MINOR: Use Set instead of List for multiple topics (#5024) -- [...truncated 1.50 MB...] kafka.zk.KafkaZkClientTest > testClusterIdMethods STARTED

Re: [DISCUSS] KIP-303: Add Dynamic Routing in Streams Sink

Hi Guozhang, yes i think it would make sense to add this now as having the additional record context would be valuable. Though i'm happy either way. On Wed, 16 May 2018 at 23:07 Guozhang Wang wrote: > Hi Damian, > > My current plan is to add the "RichKeyValueMapper" when

Re: [DISCUSS] KIP-304: Connect runtime mode improvements for container platforms

Fat fingered too... "Connect source should be able to achieve the same centrality of offsets " On Thu., 17 May 2018, 10:27 pm Stephane Maarek, < steph...@simplemachines.com.au> wrote: > Say you have 50 connectors all with different ACLs and service account. > That's 50 connect clusters to

Re: [DISCUSS] KIP-304: Connect runtime mode improvements for container platforms

Say you have 50 connectors all with different ACLs and service account. That's 50 connect clusters to maintain. So 50*3 internal connect topics to maintain (they can't share the same connect topics because they're different clusters). At default config we're talking 1500 partitions which is a lot

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

On Thu, May 17, 2018, at 09:28, Piyush Vijay wrote: > I was planning to do that. > > Another unrelated detail is the presence of the support for ‘*’ ACL > currently. Looks like we’ll have to keep supporting this as a special case, > even though using a different location for wildcard-suffix ACLs

Re: [VOTE] #2 KIP-248: Create New ConfigCommand That Uses The New AdminClient

Hi Viktor, Since the KIP freeze is coming up really soon, maybe we should just drop the section about changes to AlterConfigs from KIP-248. We don't really need it here, since ConfigCommand can use AlterConfigs as-is. We can pick up the discussion about improving AlterConfigs in a future KIP.

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

I was planning to do that. Another unrelated detail is the presence of the support for ‘*’ ACL currently. Looks like we’ll have to keep supporting this as a special case, even though using a different location for wildcard-suffix ACLs on Zk. On Thu, May 17, 2018 at 9:15 AM Colin McCabe

Re: [VOTE] KIP-297: Externalizing Secrets for Connect Configurations

Hi Robert, Hmm. I thought that if you're using ConfigChangeCallback, you are relying on the ConfigProvider to make a callback to you when the configuration has changed. So isn't that always the "push model" (where the ConfigProvider pushes changes to Connect). If you want the "pull model"

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

Thanks, Piyush. +1 for starting the vote soon. Can you please also add a discussion about escaping? For example, earlier we discussed using backslashes to escape special characters. So that users can create an ACL referring to a literal "foo*" group by creating an ACL for "foo\*"

Re: [DISCUSS] KIP-298: Error Handling in Connect

Thanks Arjun for your quick response. Adding an example for the failure log improves things, but I think it'd be better to also add the schema definition of these Json entries. And I'll agree with Magesh that this format should be public API. Also, does the current example have a copy/paste

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

Hi Rajini/Colin, I will remove the wildcard principals from the scope for now, updating KIP right now and will open it for vote. Thanks Piyush Vijay On Thu, May 17, 2018 at 6:59 AM, Rajini Sivaram wrote: > Hi Piyush, > > I have added a PR

Re: [VOTE] KIP-255: OAuth Authentication via SASL/OAUTHBEARER

Hi Jun. I've updated the KIP to add a new section titled "Summary for Production Use" that includes this information along with a consolidated set of references to the applicable specifications. Thanks for the questions. *We still need another binding vote* (currently there are two binding + 1

Re: [DISCUSS] KIP-290: Support for wildcard suffixed ACLs

Hi Piyush, I have added a PR (https://github.com/apache/kafka/pull/5030) with tests to show how group principals can be used for authorization with custom principal builders. One of the tests uses SASL. It is not quite the same as a full-fledged user groups, but since it works with all security

[jira] [Created] (KAFKA-6912) Add authorization tests for custom principal types

Rajini Sivaram created KAFKA-6912: - Summary: Add authorization tests for custom principal types Key: KAFKA-6912 URL: https://issues.apache.org/jira/browse/KAFKA-6912 Project: Kafka Issue

Re: [DISCUSS] KIP-304: Connect runtime mode improvements for container platforms

First sentence fat fingered. “Just curious as to why there’s an issue with the backing topics for Kafka Connect.” -- Rahul Singh rahul.si...@anant.us Anant Corporation On May 17, 2018, 6:17 AM -0400, Stephane Maarek , wrote: > Hi Salius > > I think you're on

Re: [DISCUSS] KIP-304: Connect runtime mode improvements for container platforms

Just curious as to why there’s a pr been for the backing topics for Kafka Connect. I’m a fan of decoupling and love the world that operates in containers. I’m just wondering specifically what the issue is for having topics storing the connect states for source / sinks. Wouldn’t it make sense

RE: [DISCUSS] KIP-302 - Enable Kafka clients to use all DNS resolved IP addresses

Yes, makes sense. You mentioned multiple times you see no overlap and no issue with your KIP, and that they solve different use cases. Appreciate you have an existing use case that would work, but we need to make sure this isn't confusing to users and that any combination will always work,

Re: [DISCUSS] KIP-304: Connect runtime mode improvements for container platforms

Hi Salius I think you're on the money, but you're not pushing things too far. This is something I've hoped for a long time. Let's talk Kafka Connect v2 Kafka Connect Cluster, as you said, are not convenient to work with (the KIP details drawbacks well). I'm all about containerisation just like

[jira] [Created] (KAFKA-6911) Incorrect check for keystore/truststore dynamic update

Rajini Sivaram created KAFKA-6911: - Summary: Incorrect check for keystore/truststore dynamic update Key: KAFKA-6911 URL: https://issues.apache.org/jira/browse/KAFKA-6911 Project: Kafka Issue

Re: [DISCUSS] KIP-304: Connect runtime mode improvements for container platforms

Hi, the only real usecase for the REST interface I can see is providing health/liveness checks for mesos/kubernetes. It's also true that the API can be left as is and e.g. not exposed publicly on the platform level, but this would still leave opportunities to accidentally mess something up

[VOTE] KIP-298: Error Handling in Connect kafka

All, Many thanks for all the feedback on KIP-298. Highly appreciate the time and effort you all put into it. I've updated the KIP accordingly, and would like to start to start a vote on it. KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP- 298%3A+Error+Handling+in+Connect JIRA:

Re: [DISCUSS] KIP-298: Error Handling in Connect

On Wed, May 16, 2018 at 7:13 PM, Matt Farmer wrote: > Hey Arjun, > > I like deadletterqueue all lower case, so I'm +1 on that. > Super! updated the KIP. > > Yes, in the case we were seeing there were external system failures. > We had issues connecting to S3. While the connector

Re: [DISCUSS] KIP-298: Error Handling in Connect

Magesh, Updated the KIP. Thanks a lot! Best, On Wed, May 16, 2018 at 7:12 PM, Magesh Nandakumar wrote: > Arjun, > > Thanks for all the changes. Technically, the message format used for the > DLQ should be part of the public interface since users could consume it and >

Re: [DISCUSS] KIP-303: Add Dynamic Routing in Streams Sink

Hi Damian, My current plan is to add the "RichKeyValueMapper" when getting in KIP-159 to include the record context in this dynamic routing feature. So just to clarify: are you more concerning that if we are going to do that anyways in the future, we should not add overloaded functions with