Re: [DISCUSS] KIP-317 - Add end-to-end data encryption functionality to Apache Kafka

cryption until the > > actual > > > value is inspected. That way joins etc are possible without the > > encryption > > > key, and RocksDB would not need to decrypt values before materializing > to > > > disk. > > > > > > This is why I'

Re: [DISCUSS] KIP-604: Remove ZooKeeper Flags from the Administrative Tools

duplicates > kafka-leader-election.sh, and it has been deprecated for some time. I > changed the KIP. > > cheers, > Colin > > On Mon, May 4, 2020, at 01:06, Sönke Liebau wrote: > > Hi Colin, > > > > thanks for the kip, lgtm overall with two small comments: &g

Re: [DISCUSS] KIP-604: Remove ZooKeeper Flags from the Administrative Tools

, Colin McCabe wrote: > Hi all, > > I posted a KIP about removing the deprecated --zookeeper flags from our > administrative tools. Check it out here: > https://cwiki.apache.org/confluence/x/kRARCQ > > best, > Colin > -- Sönke Liebau Partner Tel. +49 179 7940878 Open

Re: [DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

But I guess it's not > really a blocker, as long as we stated clearly in the KIP how we are going > to roll things out, and let it partially finish in 2.6. > > Boyang > > On Fri, May 1, 2020 at 2:32 PM Sönke Liebau > wrote: > > > Hi Boyang, > > > >

Re: [DISCUSS] KIP-317 - Add end-to-end data encryption functionality to Apache Kafka

KIP is currently light on implementation details, but > > would like to get some feedback on the general approach before fully > > speccing everything. > > > > The KIP can be found at > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-317%3A+Add+

Re: [DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

gt; > fake principals in testing environment, without actual triggering > > in > > > the > > > production system > > >2. For next *3.0 *release: > > > 1. Introduce serializability to PrincipalBuilder > > > 2. Turn

[DISCUSS] KIP-317 - Add end-to-end data encryption functionality to Apache Kafka

All, I've asked for comments on this KIP in the past, but since I didn't really get any feedback I've decided to reduce the initial scope of the KIP a bit and try again. I have reworked to KIP to provide a limited, but useful set of features for this initial KIP and laid out a very rough roadmap

Re: [DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

t; > Let me know if you think either of these approaches would work. > > On Thu, Apr 23, 2020 at 7:01 AM Sönke Liebau > wrote: > > > I agree that this would be useful to have and shouldn't create issues in > > 99% of all cases. But it would be a breaking change to a public API. > >

Re: [DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

value. Obviously you could optimize this using an > optional > > field so you only serialize the toString() if it differed from the value > > you'd get from KafkaPrincipal.toString(). Maybe non-audit logging using > the > > wrong string value of a principa

Re: [DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

> > > > > > > > > Sorry if this was already discussed, but I didn't see this as > > rejected > > > > > alternative: > > > > > > > > > > Until now, we always did client side routing - the client itself > > found > >

Re: [DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

Hi Boyang, I think what Tom is referring to is that it is very hard to forward enough information to the controller to put it into a position to properly authenticate any request. While the Default KafkaPrincipal can easily be serialized and sent to the controller, as previously seen, those are

Re: [ANNOUNCE] Apache Kafka 2.5.0

mael Juma, Ivan Yurchenko, Jason Gustafson, jiameixie, > > John > > > Roesler, José Armando García Sancio, Jukka Karvanen, Karan Kumar, Kevin > > Lu, > > > Konstantine Karantasis, Lee Dongjin, Lev Zemlyanov, Levani Kokhreidze, > > > Lucas Bradstreet, Manikumar

Re: Preferred Partition Leaders

lica.selector.class). > > > Kafka producers have to send data to partition leaders. There is no way > > to > > > tell that we prefer replica leaders to be running in DC1 and DC2. Kafka > > > will also try to evenly balance leaders across brokers > > > (auto.lead

Re: Apache Feathercast

Thanks Jun! I'll reach out to Rich about a date. On Tue, 14 Apr 2020 at 19:11, Jun Rao wrote: > Hi, Sonke, > > That sounds like a good opportunity for promoting Kafka. Thanks for > volunteering. > > Jun > > On Tue, Apr 14, 2020 at 8:08 AM Sönke Liebau > wrote:

Apache Feathercast

Hi all, Rich is trying to reboot the Feathercast series [1] and looking for projects that would be willing to give a quick intro about what they do. I'd be happy to do this for Kafka if no one has any objections. I am sure there are many people eminently more suited to this task than me though,

Jira Cleanup - Kafka on Windows

All, I stumbled across a recent issue about Kafka crashing on Windows the other day, which keeps coming up on jira. So I went and had a look and found 16 unresolved tickets about this - some of them definitely are duplicates, some of them might be a variation. KAKFA-1194 - The kafka broker

Re: Shutdown Broker Because Log Dirs Failed

. 2020, 16:02: > Thank you so much for the response! > > If this is unresolved, could you suggest any alternative for me to make > this work? If that is possible. > > Any other solution is highly appreciated! Thank you. > > From: Sönke Liebau > Sent: Tuesday, 7 Ap

Re: Shutdown Broker Because Log Dirs Failed

Hi Qarl, there probably (and sadly) is no good answer to your question. Windows is not an officially supported platform to run Kafka on and there is a large number of unresolved tickets in Jira on this topic. Most notable is probably KAFKA-1194 [1]. I haven't closely followed this, I thought we

Re: [DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

ses to migrate existing Zookeeper mutation paths, > including > > > configuration, security and quota changes, to controller-only by always > > > routing these alterations to the controller. > > > > > > Let me know your thoughts! > > > > > > Best, > > > Boyang > > > > > > -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Re: [DISCUSS] KIP-585: Conditional SMT

bjects in their own right (opening up the possibility of > > > people supplying their own Conditions, just like they can supply their > > own > > > SMTs). That might be configured something like this: > > > > > > transforms.conditionalExtract.condition:

Re: [DISCUSS] KIP-585: Conditional SMT

matches: my-prefix-.* > > but using the more flexible scheme that would require something more like > this: > > transforms.conditionalExtract.condition: bar > transforms.conditionalExtract.condition.bar.type: TopicMatch > transforms.conditionalExtract.condition.bar

Re: [DISCUSS] KIP-585: Conditional SMT

Hi Tom, sounds useful to me, thanks for the KIP! The only thought that I had while reading was that this will probably raise questions about more involved conditions fairly quickly. For example the "has-header" will cause an appetite for conditions like "this-header-has-that-value". This would

Jira Cleanup

Hi all, I've spent some time with jira and identified a few tickets that might be worth looking at again / closing if there is no appetite for them. KAFKA-1906 & KAFKA-3925 - Both these ticket

Re: Jira Cleanup

FYI: just closed those five tickets On Wed, 11 Mar 2020 at 22:09, Sönke Liebau wrote: > All, > > I left a few comments on some old but still open jiras in an attempt to > clean up a little bit. > > Since probably no one would notice these comments I thought I'd quickly > l

Jira Cleanup

All, I left a few comments on some old but still open jiras in an attempt to clean up a little bit. Since probably no one would notice these comments I thought I'd quickly list them here to give people a chance to check on them: KAFKA-1265 -

Re: KAFKA-9308: Request for Review of documentation

feedback would be appreciated! Best regards, Sönke On Tue, 11 Feb 2020 at 11:48, Sönke Liebau wrote: > Hi everybody, > > I've reworked the SSL part of the documentation a little in order to fix > (among other things) KAFKA-9308[1] and would love some feedback if someone > can spar

KAFKA-9308: Request for Review of documentation

Hi everybody, I've reworked the SSL part of the documentation a little in order to fix (among other things) KAFKA-9308[1] and would love some feedback if someone can spare a few minutes. Pull request: https://github.com/apache/kafka/pull/8009 [1] https://issues.apache.org/jira/browse/KAFKA-9308

Re: [ANNOUNCE] New Kafka PMC Members: Colin, Vahid and Manikumar

Ismael Juma < > ism...@juma.me.uk> > > > > >> wrote: > > > > >>>>>>> > > > > >>>>>>> Congratulations Colin, Vahid and Manikumar! > > > > >>>>>>> > > > > >>>>>>> Ismael > > > > >>>>>>> > > > > >>>>>>> On Tue, Jan 14, 2020 at 9:30 AM Gwen Shapira < > > g...@confluent.io> > > > > >>>>> wrote: > > > > >>>>>>> > > > > >>>>>>>> Hi everyone, > > > > >>>>>>>> > > > > >>>>>>>> I'm happy to announce that Colin McCabe, Vahid Hashemian and > > > > >>>>> Manikumar > > > > >>>>>>>> Reddy are now members of Apache Kafka PMC. > > > > >>>>>>>> > > > > >>>>>>>> Colin and Manikumar became committers on Sept 2018 and Vahid > > on > > > > Jan > > > > >>>>>>>> 2019. They all contributed many patches, code reviews and > > > > >>>>> participated > > > > >>>>>>>> in many KIP discussions. We appreciate their contributions > > and are > > > > >>>>>>>> looking forward to many more to come. > > > > >>>>>>>> > > > > >>>>>>>> Congrats Colin, Vahid and Manikumar! > > > > >>>>>>>> > > > > >>>>>>>> Gwen, on behalf of Apache Kafka PMC > > > > >>>>>>>> > > > > >>>>>> > > > > >>>>> > > > > >>> > > > > >>> > > > > >> > > > > > > > > > > > > -- > > Thanks! > --Vahid > -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Re: [VOTE] KIP-551: Expose disk read and write metrics

; > > On Fri, Jan 10, 2020 at 2:17 PM Colin McCabe > > > > > wrote: > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > I'd like to start the vote on KIP-551: Expose disk read and > write > > > > &g

Re: [VOTE] KIP-409: Allow creating under-replicated topics and partitions

ve updated the KIP to explicitly > mention this point. > > I hope that answers your question, let me know. > Thanks > > > On Mon, Dec 16, 2019 at 4:38 PM Sönke Liebau > wrote: > > > > Hi Michael, > > > > that sounds like a useful addition! I can't

Re: [VOTE] KIP-409: Allow creating under-replicated topics and partitions

t; > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-409%3A+Allow+creating+under-replicated+topics+and+partitions > > > > > > Thanks > > -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Re: [jira] [Created] (KAFKA-9292) KIP-551: Expose disk read and write metrics

gt; > > -- > This message was sent by Atlassian Jira > (v8.3.4#803005) -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Re: Broker Interceptors

Hi Thomas, I think that idea is worth looking at. As you say, if no interceptor is configured then the performance overhead should be negligible. Basically it is then up to the user to decide if he wants tomtake the performance hit. We should make sure to think about monitoring capabilities like

Handling deserialization exceptions in Kafka consumer

Hi all, I was recently discussing deserialization errors and how they can be handled in Kafka consumers. I believe the current code still throws an exception if deserialization fails which stops consumption unless you seek past that record. This creates an issue for a JDBC sink connector that

Re: Preliminary blog post about the Apache Kafka 2.4.0 release

the upcoming Apache Kafka > 2.4.0 > > release. > > Please take a look and let me know if you want to add/modify details. > > Thanks to all who contributed to this blog post. > > > > > https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache1 > > &

Re: [DISCUSS] KIP-317: Transparent Data Encryption

get the KIP accepted into > Kafka. > > Thanks, > Andrew Schofield (IBM) > > On 06/08/2019, 15:46, "Sönke Liebau" > wrote: > > Hi, > > I have so far received pretty much no comments on the technical details > outlined in the KIP. While I am happy to c

Re: [VOTE] KIP-499 - Unify connection name flag for command line tool

er-groups.sh, kafka-consumer-perf-test.sh, > > kafka-verifiable-consumer.sh, kafka-verifiable-producer.sh > > > > Thanks, > > -Mitch > > > > -- > > Gwen Shapira > > Product Manager | Confluent > > 650.450.2760 | @gwenshap > > Follow us: Twitter | blog > > > -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Re: [DISCUSS] KIP-317: Transparent Data Encryption

scenarios. You may learn from those protocols on how they > handle keys, how long they keep them etc. to implement your E2e solution . > > > Am 08.08.2019 um 08:11 schrieb Maulin Vasavada < > maulin.vasav...@gmail.com>: > > > > Hi Sönke Liebau > > < > http

Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface

thorizer. If you have come across other limitations that you would like > > to see addressed in the new API, please raise these on the discussion > > thread so that we can consider those too. All suggestions and feedback > are > > welcome. > > > > Thank you, > > > > Rajini > -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Re: [DISCUSS] KIP-317: Transparent Data Encryption

few people before I but more work into this. Best regards, Sönke On Tue, 21 May 2019 at 14:15, Sönke Liebau wrote: > Hi everybody, > > I'd like to rekindle the discussion around KIP-317. > I have reworked the KIP a little bit in order to design everything as a > pluggable implemen

Jira Cleanup

All, I left a few comments on some old but still open jiras in an attempt to clean up a little bit. Since probably no one would notice these comments I thought I'd quickly list them here to give people a chance to check on them: KAFKA-822 :

Re: [DISCUSS] KIP-317: Transparent Data Encryption

apache.org/confluence/download/attachments/85479936/kafka_e2e-encryption_control-flow.png?version=1=1558439227551=v2 On Fri, 10 Aug 2018 at 14:05, Sönke Liebau wrote: > Hi Viktor, > > thanks for your input! We could accommodate magic headers by removing any > known fixed bytes pr

Re: Cleaning up command line tools argument parsing a little

but if you think it's > > interesting I can rebase it and we can continue with raising a KIP. > > https://github.com/viktorsomogyi/kafka/tree/kafka_shell > > > > Viktor > > > > On Tue, May 7, 2019 at 11:10 AM Sönke Liebau > > wrote: > > > >> Hi Colin,

Re: Cleaning up command line tools argument parsing a little

on that, but I currently have enough other open things to look at, so I won't draw that one to myself as well for now. But I'll keep it in mind for when some time frees up :) Best regards, Sönke Colin McCabe schrieb am Di., 7. Mai 2019, 00:56: > On Mon, May 6, 2019, at 10:21, Sönke Liebau wr

Re: Cleaning up command line tools argument parsing a little

e as variants to ease the transition. > > best, > Colin > > > On Sun, May 5, 2019, at 00:54, Sönke Liebau wrote: > > Hi Colin, > > > > I totally agree! Especially the differently named bootstrap server > options > > have been annoyi

Re: Cleaning up command line tools argument parsing a little

ectly reasonable > to > > > have multiple spellings of the same arguments. I don't really see a > > > downside to keeping the aliases around indefinitely. > > > > > > Ryanne > > > > > > > > > > > > > > > On Wed, Ap

Re: Cleaning up command line tools argument parsing a little

nd indefinitely. > > > > Ryanne > > > > > > > > > > On Wed, Apr 17, 2019, 7:07 AM Sönke Liebau > > wrote: > > > > > Hi everybody, > > > > > > Jason and I were recently discussing command line argument parsing on > > &

Cleaning up command line tools argument parsing a little

Hi everybody, Jason and I were recently discussing command line argument parsing on KAFKA-8131 (or rather the related pull request) [1]. Command line tools and their arguments are somewhat diverse at the moment. Most of the tools use joptsimple for argument parsing, some newer java tools use

Re: Proposal to Auto Close Inactive Tickets

Hi Addison, in general, I am totally in favor of closing unnecessary tickets! However, over the past few weeks, I have spent quite a bit of time looking at old tickets and evaluating whether those should be closed or are still relevant. For a surprising number of these tickets, I've found that

Re: Kafka Jenkins not showing recent builds in history

Quick update: everything seems to be back to normal. On Mon, Apr 8, 2019 at 9:57 AM Sönke Liebau wrote: > I've opened an INFRA ticket about this [1]. Apparently, it is a known > issue that is solved by a restart of Jenkins. One will be scheduled in the > near future, I'll update the

Re: [DISCUSS] KIP-252: Extend ACLs to allow filtering based on ip ranges and subnets

rity/auth/Acl.scala#L37 [3] https://github.com/apache/kafka/blob/b4fa87cc51f0a7d640dc6ae2cc8f89f006aae652/core/src/main/scala/kafka/security/auth/Acl.scala#L66 On Fri, Apr 5, 2019 at 8:09 AM Colin McCabe wrote: > On Thu, Apr 4, 2019, at 01:52, Sönke Liebau wrote: > > Hi Colin, >

Re: Kafka Jenkins not showing recent builds in history

3, 2019 at 10:41 PM Sönke Liebau wrote: > Hi everybody, > > I looked through recent Jenkins builds for a while today and it somehow > looks off to me. > > Both jobs [1] [2] don't show any builds that are more recent than March > 19th in the "build history". >

Re: [DISCUSS] KIP-252: Extend ACLs to allow filtering based on ip ranges and subnets

lled out. We could use INVALID_REQUEST > as the error code for this situation, or maybe create a new one to be more > specific. > > best, > Colin > > > On Wed, Apr 3, 2019, at 04:58, Sönke Liebau wrote: > > All, > > > > as this thread has now been dormant for about thre

Kafka Jenkins not showing recent builds in history

Hi everybody, I looked through recent Jenkins builds for a while today and it somehow looks off to me. Both jobs [1] [2] don't show any builds that are more recent than March 19th in the "build history". Only the "last failed" and "last unsuccessful" permalinks show recent dates. Build pages can

Re: [VOTE] KIP-443: Return to default segment.ms and segment.index.bytes in Streams repartition topics

t;> wangg...@gmail.com> > >> > > > > wrote: > >> > > > > > > >> > > > > > > Hello folks, > >> > > > > > > > >> > > > > > > I'd like to directly start a voting thread on this simple > KIP > >> to > >> > > change > >> > > > > > the > >> > > > > > > default override values for repartition topics: > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-443%3A+Return+to+default+segment.ms+and+segment.index.bytes+in+Streams+repartition+topics > >> > > > > > > > >> > > > > > > The related PR can be found here as well: > >> > > > > > > https://github.com/apache/kafka/pull/6511 > >> > > > > > > > >> > > > > > > If you have any thoughts or feedbacks, they are more than > >> welcomed > >> > > as > >> > > > > > well. > >> > > > > > > > >> > > > > > > > >> > > > > > > -- Guozhang > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > >> > > >> > > >> > -- > >> > -- Guozhang > >> > > > > > > -- > > -- Guozhang > > > > > -- > -- Guozhang > -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Re: [DISCUSS] KIP-252: Extend ACLs to allow filtering based on ip ranges and subnets

consider the security implications of these ACLs in a mixed version cluster quite minimal and addressable via the release notes. Best, Sönke On Sat, Mar 16, 2019 at 1:32 PM Sönke Liebau wrote: > Just a quick bump, as this has been quiet for a while again. > > On Tue, Jan 8, 2019 at 12:44

Re: [VOTE] KIP-349 Priorities for Source Topics

s also to everyone who participated in the discussion. > > > > > > Best regards, > > > -- > > > Nick > > > > > > > On Jan 25, 2019, at 5:51 AM, Sönke Liebau < > soenke.lie...@opencore.com.INVALID> wrote: > > > > >

Re: [VOTE] KIP-349 Priorities for Source Topics

2019/01/28 02:26:31, n...@afshartous.com wrote: > > Hi Sönke, > > > > Thanks for taking the time to review. I’ve put KIP-349 into > hibernation. > > > > Thanks also to everyone who participated in the discussion. > > > > Best regards, > &g

Re: [DISCUSS] KIP-252: Extend ACLs to allow filtering based on ip ranges and subnets

Just a quick bump, as this has been quiet for a while again. On Tue, Jan 8, 2019 at 12:44 PM Sönke Liebau wrote: > Hi Colin, > > thanks for your response! > > in theory we could get away without any additional path changes I > think.. I am still somewhat unsure about the best

Re: Apache Kafka Memory Leakage???

he heap memory goes to about 60 or 70MB, and then falls > back, and the pattern continues. Max heap consumed at an instant is about > 70MB. > > Maybe there is a problem in my platform on updating the stats. I shall > convey it to my team. > > thanks so much for the help. &

Re: Apache Kafka Memory Leakage???

ng as I can share all the details and my screen as well. > After that, we can see what to do next. If the call is not possible then > its ok but it would be great to have it. > Thanks, > > > > On Tue, Mar 5, 2019 at 4:29 PM Sönke Liebau > wrote: > >> Hi Syed, >&

Re: Apache Kafka Memory Leakage???

tterize the > kafka-clients library. > What the steps next to confirm its a bug in kafka-clients? And if its a > bug whats the process to get it resolved? > > Thanks, > > > > On Tue, Mar 5, 2019 at 2:43 PM Sönke Liebau > wrote: > >> Hi Syed, >> >> A

Re: Apache Kafka Memory Leakage???

kafka-clients library. > > Thanks, > > > > On Mon, Mar 4, 2019 at 8:33 PM Sönke Liebau > wrote: > >> Hi Syed, >> >> and you are sure that this memory is actually allocated? I still have my >> reservations about that metric to be honest. Is there

Re: Apache Kafka Memory Leakage???

mp;& isRunning.get()); > > > *Note: *Debugger is a wrapper class that just writes the given string to a > local file using PrintStream's println() method. > > And I don't see any diff in the metrics. I still see the huge amount of > memory allocated. > > See the image attac

Re: Apache Kafka Memory Leakage???

(Duration) > method. Still, I see memory issue. > Is there a way we can get on a one-one call and discuss this pls? Let me > know your availability. I can share zoom meeting link. > > Thanks, > > > > On Sat, Mar 2, 2019 at 2:15 AM Sönke Liebau > wrote: >> &g

Re: Apache Kafka Memory Leakage???

current offset > if (isSyncCommit && isSuggest) { > commitOffset(offsets); > offsets.clear(); > } > } > } while ((msgCount > 0 || infiniteLoop) && isRunning.get()); > &g

Re: Speeding up integration tests

by approximately an > > hour. This would save resources and give quicker feedback for PR builds. > > > > What are your thoughts? > > Has anyone thought about this or were there any attempts made? > > > > Best, > > Viktor > > > > > -- > Best, > Stanislav -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Re: [DISCUSSION] KIP-422: Add support for user/client configuration in the Kafka Admin Client

igurations for users, instead of requiring users to directly > >> talk to ZK. > >> > >> The link for this KIP is > >> following: > >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=97555704 > >> > >> I'd be happy to r

Some jira cleanup

All, I left a few comments on some old but still open jiras in an attempt to clean up a little bit. Since probably no one would notice these comments I thought I'd quickly list them here to give people a chance to check on them: KAFKA-217 :

Re: [VOTE] KIP-349 Priorities for Source Topics

ng started with something easier. > > > Yes it does. And after 6 months of (sometimes circular) discussion I’d like > to either move towards a vote or set the status of this KIP to dormant until > if and when someone else picks up it up. > > Does anybody else have input on eithe

Re: Delayed messages with skewed data when subscribing to many partitions

> would need a wire protocol change. > Perhaps a little consumer side logic that starts fragmenting fetch > requests if it notices that responses always have data from the > maximum number of partitions. > > Best regards, > Sönke > > [1] > https://cwiki.apache.org/confluence/display/KAFKA/KIP-349%3A+Priorities+for+Source+Topics -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Delayed messages with skewed data when subscribing to many partitions

Hi all, we've just had a case where we suspect that messages get delayed from being consumed under certain circumstances. I don't necessarily think this is a bug, hence have not opened a jira yet but wanted to discuss here - there's probably a best practice that I just don't know about. The

Re: [DISCUSS] KIP-252: Extend ACLs to allow filtering based on ip ranges and subnets

ows storing everything in one compacted topic. Any thoughts on this? Best regards, Sönke [1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-290%3A+Support+for+Prefixed+ACLs On Sat, Dec 22, 2018 at 5:51 AM Colin McCabe wrote: > > Hi Sönke, > > One path forward would be to forbid

Re: [VOTE] KIP-382 MirrorMaker 2.0

; - Automatically syncs topic configuration between clusters. > > > > > > - Manages downstream topic ACL. > > > > > > - Supports "active/active" cluster pairs, as well as any number of > > > > active > > > > > > clusters. >

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

at. Legacy MirrorMaker doesn't have a REST API after all. For > organizations that want on-the-fly configuration of their replication flows, > there's Connect. > > This has been brought up by nearly everyone, so I'm happy to oblige. > > Ryanne > -- Sönke Liebau Partner Tel. +4

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

> > > > > > > > > > Looking ahead a bit, I can imagine an external tool > computing > > > the > > > > > > > spanning > > > > > > > tree of topics among a set of clusters based on > inter-cluster > > >

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

in the target cluster, so no > > state information is lost if the source cluster disappears. Both of these > > features are facilitated by the Connect framework and were inspired by > > Uber's uReplicator. > > > > > If the single connect cluster model is indeed use

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

cluster. If Connect supported multiple workers in the same cluster, > it would start to look a lot like a MM2 cluster. > > Ryanne > > On Tue, Dec 4, 2018 at 3:26 PM Sönke Liebau > wrote: > > > Hi Ryanne, > > > > thanks for your response! > > > >

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

Service and > Replication-as-a-Service are sufficiently different use-cases that we > should expect the APIs and configuration files to be at least slightly > different, even if both use the same framework underneath. That said, I do > plan to contribute a few improvements to the Co

Re: [DISCUSS] KIP-398: Support reading trust store from classpath

ailable here: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-398%3A+Support+reading+trust+store+from+classpath > < > https://cwiki.apache.org/confluence/display/KAFKA/KIP-398:+Support+reading+trust+store+from+classpath > > > > Any feedback or comments would be most welcome.

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

ka clusters. Much like Brooklin, MM2 does the work > of setting up connectors between clusters as needed. This > Replication-as-a-Service is a huge win for larger deployments, as well as > for organizations that haven't adopted Connect. > > [1] > https://www.slideshare.net/ConfluentIn

Re: [DISCUSS] KIP-252: Extend ACLs to allow filtering based on ip ranges and subnets

doesn't seem right. Looking forward to your input :) Best regards, Sönke On Tue, Jun 19, 2018 at 5:32 PM Sönke Liebau wrote: > Picking this back up, now that KIP-290 has been merged.. > > As Colin mentioned in an earlier mail this change could create a > potential security issue if not

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

gt; > >>> > > anyways and you usually can deal with at least once > >> donwstream > >> > > > pretty > >> > > > >>> > > easy). But I am very certain its not gonna get somewhere > if > >> > > offset > >> > > > >>> > > commit and record produce cluster are not the same. > >> > > > >>> > > > >> > > > >>> > > Pretty sure without this _design choice_ you can skip on > >> that > >> > > > exactly > >> > > > >>> > > once already > >> > > > >>> > > > >> > > > >>> > > Best Jan > >> > > > >>> > > > >> > > > >>> > > On 16.10.2018 18:16, Ryanne Dolan wrote: > >> > > > >>> > > > > But one big obstacle in this was > >> > > > >>> > > > always that group coordination happened on the source > >> cluster. > >> > > > >>> > > > > >> > > > >>> > > > Jan, thank you for bringing up this issue with legacy > >> > > > MirrorMaker. > >> > > > >>> I > >> > > > >>> > > > totally agree with you. This is one of several problems > >> with > >> > > > >>> MirrorMaker > >> > > > >>> > > > I intend to solve in MM2, and I already have a design > and > >> > > > >>> prototype that > >> > > > >>> > > > solves this and related issues. But as you pointed out, > >> this > >> > > KIP > >> > > > is > >> > > > >>> > > > already rather complex, and I want to focus on the core > >> feature > >> > > > set > >> > > > >>> > > > rather than performance optimizations for now. If we can > >> agree > >> > > on > >> > > > >>> what > >> > > > >>> > > > MM2 looks like, it will be very easy to agree to improve > >> its > >> > > > >>> performance > >> > > > >>> > > > and reliability. > >> > > > >>> > > > > >> > > > >>> > > > That said, I look forward to your support on a > subsequent > >> KIP > >> > > > that > >> > > > >>> > > > addresses consumer coordination and rebalance issues. > Stay > >> > > tuned! > >> > > > >>> > > > > >> > > > >>> > > > Ryanne > >> > > > >>> > > > > >> > > > >>> > > > On Tue, Oct 16, 2018 at 6:58 AM Jan Filipiak < > >> > > > >>> jan.filip...@trivago.com > >> > > > >>> > > > <mailto:jan.filip...@trivago.com>> wrote: > >> > > > >>> > > > > >> > > > >>> > > > Hi, > >> > > > >>> > > > > >> > > > >>> > > > Currently MirrorMaker is usually run collocated with > >> the > >> > > > target > >> > > > >>> > > > cluster. > >> > > > >>> > > > This is all nice and good. But one big obstacle in > >> this was > >> > > > >>> > > > always that group coordination happened on the > source > >> > > > cluster. > >> > > > >>> So > >> > > > >>> > > when > >> > > > >>> > > > then network was congested, you sometimes loose > group > >> > > > >>> membership and > >> > > > >>> > > > have to rebalance and all this. > >> > > > >>> > > > > >> > > > >>> > > > So one big request from we would be the support of > >> having > >> > > > >>> > > coordination > >> > > > >>> > > > cluster != source cluster. > >> > > > >>> > > > > >> > > > >>> > > > I would generally say a LAN is better than a WAN for > >> doing > >> > > > >>> group > >> > > > >>> > > > coordinaton and there is no reason we couldn't have > a > >> group > >> > > > >>> consuming > >> > > > >>> > > > topics from a different cluster and committing > >> offsets to > >> > > > >>> another > >> > > > >>> > > > one right? > >> > > > >>> > > > > >> > > > >>> > > > Other than that. It feels like the KIP has too much > >> > > features > >> > > > >>> where > >> > > > >>> > > many > >> > > > >>> > > > of them are not really wanted and counter productive > >> but I > >> > > > >>> will just > >> > > > >>> > > > wait and see how the discussion goes. > >> > > > >>> > > > > >> > > > >>> > > > Best Jan > >> > > > >>> > > > > >> > > > >>> > > > > >> > > > >>> > > > On 15.10.2018 18:16, Ryanne Dolan wrote: > >> > > > >>> > > > > Hey y'all! > >> > > > >>> > > > > > >> > > > >>> > > > > Please take a look at KIP-382: > >> > > > >>> > > > > > >> > > > >>> > > > > > >> > > > >>> > > > > >> > > > >>> > > > >> > > > >>> > >> > > > > >> > > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0 > >> > > > >>> > > > > > >> > > > >>> > > > > Thanks for your feedback and support. > >> > > > >>> > > > > > >> > > > >>> > > > > Ryanne > >> > > > >>> > > > > > >> > > > >>> > > > > >> > > > >>> > > > >> > > > >>> > >> > > > >> > >> > > > > >> > > > >> > > > >> > > -- > >> > > Best, > >> > > Alex Mironov > >> > > > >> > > >> > > > -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Re: [ANNOUNCE] New Committer: Manikumar Reddy

/confluence/display/KAFKA/KIP-249%3A+Add+Delegation+Token+Operations+to+KafkaAdminClient > > > > > > > > He has broad experience working with many of the core components in > > Kafka > > > > and he has reviewed over 80 PRs. He has also made huge progress > > > addressing > > > > some of our technical debt. > > > > > > > > We appreciate the contributions and we are looking forward to more. > > > > Congrats Manikumar! > > > > > > > > Jason, on behalf of the Apache Kafka PMC > > > > > > > > > > -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Re: [Review] - Pull request #5681 review

in advance. > > - > Srinivas > > - Typed on tiny keys. pls ignore typos.{mobile app} > -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Re: [DISCUSS] KIP-317: Transparent Data Encryption

ty they provide to make sure the users are fully aware of the > security implications. > > Cheers, > Viktor > > On Tue, Jun 19, 2018 at 11:55 AM Sönke Liebau > wrote: > > > Hi Stephane, > > > > thanks for pointing out the broken pictures, I fixed those. > >

Re: [DISCUSS] KIP-252: Extend ACLs to allow filtering based on ip ranges and subnets

gree :) But I couldn't come up with a better way yet. Any thoughts? Best regards, Sönke [1] https://github.com/apache/kafka/pull/5079#pullrequestreview-124512689 On Thu, May 3, 2018 at 10:57 PM, Sönke Liebau wrote: > Technically I absolutely agree with you, this would indeed create > iss

Re: [DISCUSS] KIP-317: Transparent Data Encryption

pression => encryption => send. > and the inverse for a consumer. > > Regards > Stephane > > On 19 June 2018 at 06:46, Sönke Liebau > wrote: > >> Hi everybody, >> >> I've created a draft version of KIP-317 which describes the addition >> of

[DISCUSS] KIP-317: Transparent Data Encryption

Hi everybody, I've created a draft version of KIP-317 which describes the addition of transparent data encryption functionality to Kafka. Please consider this as a basis for discussion - I am aware that this is not at a level of detail sufficient for implementation, but I wanted to get some

Re: [DISCUSS] KIP-252: Extend ACLs to allow filtering based on ip ranges and subnets

haven't been upgraded, they will not honor the DENY ACL, > possibly causing a security issue. > > In general, it seems like we need some kind of versioning system in ACLs to > handle these cases. > > best, > Colin > > On Thu, May 3, 2018, at 08:11, Sönke Liebau wrote: >> H

Re: [DISCUSS] KIP-252: Extend ACLs to allow filtering based on ip ranges and subnets

in KIP-290 that cover most of the suggestions already. Does anybody have any other objections to starting a vote on this KIP? Regards, Sönke On Fri, Feb 2, 2018 at 5:11 PM, Sönke Liebau <soenke.lie...@opencore.com> wrote: > Hi Manikumar, > > you are right, 5713 is a bit ambiguous abo

Are there plans to migrate some/all of the command line tools to use the new AdminClient?

I've dug around jira and the list of KIPs for a bit now, but could not really find anything specific on plans to move the command line tools over to the new AdminClient. Did I miss something or is that not currently planned? Most of the current command line tools require access to Zookeeper,

Re: Documentation build system

> everyone contributing to the project, I figured I'd send this quick note to >> increase the likelihood people see it and have a chance to weigh in. >> >> Thanks, >> -Ewen >> > > > > -- > -- Guozhang -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Re: [DISCUSS] KIP-252: Extend ACLs to allow filtering based on ip ranges and subnets

. Yes, wildcards (*) in hostname doesn't makes sense. Range and subnet support will give us the flexibility. On Thu, Feb 1, 2018 at 5:56 PM, Sönke Liebau < soenke.lie...@opencore.com.invalid> wrote: > Hi Manikumar, > > the current proposal indeed leaves out IPv6 addresses, as I was unsure

Re: [DISCUSS] KIP-252: Extend ACLs to allow filtering based on ip ranges and subnets

low >> specifying ip ranges and subnets for the -allow-host and --deny-host >> parameters of the acl tool. >> >> The KIP can be found at >> https://cwiki.apache.org/confluence/display/KAFKA/KIP- >> 252+-+Extend+ACLs+to+allow+filtering+based+on+ip+ranges+and+subnets >&g

[DISCUSS] KIP-252: Extend ACLs to allow filtering based on ip ranges and subnets

Hey everybody, following a brief inital discussion a couple of days ago on this list I'd like to get a discussion going on KIP-252 which would allow specifying ip ranges and subnets for the -allow-host and --deny-host parameters of the acl tool. The KIP can be found at

Re: [VOTE] KIP-212: Enforce set of legal characters for connector names

n 23, 2018, 1:17 PM Randall Hauch <rha...@gmail.com> wrote: >> > >> > > +1 (non-binding) >> > > >> > > On Mon, Jan 22, 2018 at 6:35 PM, Sönke Liebau < >> > > soenke.lie...@opencore.com.invalid> wrote: >> > > >>

Re: [DISCUSS] KIP-212: Enforce set of legal characters for connector names

ng issues somewhere else in the code because we created an illegal escape sequence again, or are we happy with that behavior for now? Kind regards, Sönke [1] https://docs.oracle.com/javase/7/docs/api/java/lang/Character.html#isISOControl(int) Am 21.01.2018 23:35 schrieb "Sönke Lieba

Re: [DISCUSS] Improving ACLs by allowing ip ranges and subnet expressions?

P cache on a local network, or impersonate a DNS server. For users who want some access controls, but don't care about security, maybe we should make it easier to use and create users without enabling kerberos or similar? best, Colin On Wed, Jan 24, 2018, at 12:59, Sönke Liebau wrote: > Hi

[DISCUSS] Improving ACLs by allowing ip ranges and subnet expressions?

Hi everyone, the current ACL functionality in Kafka is a bit limited concerning host based rules when specifying multiple hosts. A common scenario for this would be that if have a YARN cluster running Spark jobs that access Kafka and want to create ACLs based on the ip addresses of the cluster

  1   2   >