Re: [DISCUSS] KIP-317 - Add end-to-end data encryption functionality to Apache Kafka

tion key. Maybe only record _values_ should be > > > encrypted, and maybe Kafka Streams could defer decryption until the > > actual > > > value is inspected. That way joins etc are possible without the > > encryption > > > key, and RocksDB would n

Re: [DISCUSS] KIP-604: Remove ZooKeeper Flags from the Administrative Tools

just duplicates > kafka-leader-election.sh, and it has been deprecated for some time. I > changed the KIP. > > cheers, > Colin > > On Mon, May 4, 2020, at 01:06, Sönke Liebau wrote: > > Hi Colin, > > > > thanks for the kip, lgtm overall with two small comments:

Re: [DISCUSS] KIP-604: Remove ZooKeeper Flags from the Administrative Tools

, Colin McCabe wrote: > Hi all, > > I posted a KIP about removing the deprecated --zookeeper flags from our > administrative tools. Check it out here: > https://cwiki.apache.org/confluence/x/kRARCQ > > best, > Colin > -- Sönke Liebau Partner Tel. +49 179 7940878 Open

Re: [DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

ipal. But I guess it's not > really a blocker, as long as we stated clearly in the KIP how we are going > to roll things out, and let it partially finish in 2.6. > > Boyang > > On Fri, May 1, 2020 at 2:32 PM Sönke Liebau > wrote: > > > Hi Boyang, > > >

Re: [DISCUSS] KIP-317 - Add end-to-end data encryption functionality to Apache Kafka

in a final version. > > > > I am aware that the KIP is currently light on implementation details, but > > would like to get some feedback on the general approach before fully > > speccing everything. > > > > The KIP can be found at > > > > > https

Re: [DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

ith > > > fake principals in testing environment, without actual triggering > > in > > > the > > > production system > > >2. For next *3.0 *release: > > > 1. Introduce serializability to PrincipalBuilder > > > 2

[DISCUSS] KIP-317 - Add end-to-end data encryption functionality to Apache Kafka

All, I've asked for comments on this KIP in the past, but since I didn't really get any feedback I've decided to reduce the initial scope of the KIP a bit and try again. I have reworked to KIP to provide a limited, but useful set of features for this initial KIP and laid out a very rough roadmap

Re: [DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

gt; Let me know if you think either of these approaches would work. > > On Thu, Apr 23, 2020 at 7:01 AM Sönke Liebau > wrote: > > > I agree that this would be useful to have and shouldn't create issues in > > 99% of all cases. But it would be a breaking change to a public API.

Re: [DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

idden toString to > > return the right value. Obviously you could optimize this using an > optional > > field so you only serialize the toString() if it differed from the value > > you'd get from KafkaPrincipal.toString(). Maybe non-audit logging using > the > >

Re: [DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

pira > > wrote: > > > > > > > > > Hey Boyang, > > > > > > > > > > Sorry if this was already discussed, but I didn't see this as > > rejected > > > > > alternative: > > > > > > > > > > U

Re: [DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

Hi Boyang, I think what Tom is referring to is that it is very hard to forward enough information to the controller to put it into a position to properly authenticate any request. While the Default KafkaPrincipal can easily be serialized and sent to the controller, as previously seen, those are j

Re: [ANNOUNCE] Apache Kafka 2.5.0

mael Juma, Ivan Yurchenko, Jason Gustafson, jiameixie, > > John > > > Roesler, José Armando García Sancio, Jukka Karvanen, Karan Kumar, Kevin > > Lu, > > > Konstantine Karantasis, Lee Dongjin, Lev Zemlyanov, Levani Kokhreidze, > > > Lucas Bradstreet, Manikumar

Re: Preferred Partition Leaders

eplica.selector.class). > > > Kafka producers have to send data to partition leaders. There is no way > > to > > > tell that we prefer replica leaders to be running in DC1 and DC2. Kafka > > > will also try to evenly balance leaders across brokers > > > (auto

Re: Apache Feathercast

Thanks Jun! I'll reach out to Rich about a date. On Tue, 14 Apr 2020 at 19:11, Jun Rao wrote: > Hi, Sonke, > > That sounds like a good opportunity for promoting Kafka. Thanks for > volunteering. > > Jun > > On Tue, Apr 14, 2020 at 8:08 AM Sönke Liebau > wrot

Apache Feathercast

Hi all, Rich is trying to reboot the Feathercast series [1] and looking for projects that would be willing to give a quick intro about what they do. I'd be happy to do this for Kafka if no one has any objections. I am sure there are many people eminently more suited to this task than me though,

Jira Cleanup - Kafka on Windows

All, I stumbled across a recent issue about Kafka crashing on Windows the other day, which keeps coming up on jira. So I went and had a look and found 16 unresolved tickets about this - some of them definitely are duplicates, some of them might be a variation. KAKFA-1194 - The kafka broker cannot

Re: Shutdown Broker Because Log Dirs Failed

. Apr. 2020, 16:02: > Thank you so much for the response! > > If this is unresolved, could you suggest any alternative for me to make > this work? If that is possible. > > Any other solution is highly appreciated! Thank you. > > From: Sönke Liebau > Sent: Tuesday,

Re: Shutdown Broker Because Log Dirs Failed

Hi Qarl, there probably (and sadly) is no good answer to your question. Windows is not an officially supported platform to run Kafka on and there is a large number of unresolved tickets in Jira on this topic. Most notable is probably KAFKA-1194 [1]. I haven't closely followed this, I thought we w

Re: [DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

> > This KIP proposes to migrate existing Zookeeper mutation paths, > including > > > configuration, security and quota changes, to controller-only by always > > > routing these alterations to the controller. > > > > > > Let me know your thoughts! > > > > > > Best, > > > Boyang > > > > > > -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Re: [DISCUSS] KIP-585: Conditional SMT

t; One thing I did briefly consider what the ability to treat conditions > as > > > Configurable objects in their own right (opening up the possibility of > > > people supplying their own Conditions, just like they can supply their > > own > > > SMTs). That might

Re: [DISCUSS] KIP-585: Conditional SMT

gt; transforms.conditionalExtract.condition: topic-matches: my-prefix-.* > > but using the more flexible scheme that would require something more like > this: > > transforms.conditionalExtract.condition: bar > transforms.conditionalExtract.condition.bar.type: TopicMat

Re: [DISCUSS] KIP-585: Conditional SMT

Hi Tom, sounds useful to me, thanks for the KIP! The only thought that I had while reading was that this will probably raise questions about more involved conditions fairly quickly. For example the "has-header" will cause an appetite for conditions like "this-header-has-that-value". This would nec

Jira Cleanup

Hi all, I've spent some time with jira and identified a few tickets that might be worth looking at again / closing if there is no appetite for them. KAFKA-1906 & KAFKA-3925 - Both these ticket re

Re: Jira Cleanup

FYI: just closed those five tickets On Wed, 11 Mar 2020 at 22:09, Sönke Liebau wrote: > All, > > I left a few comments on some old but still open jiras in an attempt to > clean up a little bit. > > Since probably no one would notice these comments I thought I'd quickly &

Jira Cleanup

All, I left a few comments on some old but still open jiras in an attempt to clean up a little bit. Since probably no one would notice these comments I thought I'd quickly list them here to give people a chance to check on them: KAFKA-1265 - SBT

Re: KAFKA-9308: Request for Review of documentation

feedback would be appreciated! Best regards, Sönke On Tue, 11 Feb 2020 at 11:48, Sönke Liebau wrote: > Hi everybody, > > I've reworked the SSL part of the documentation a little in order to fix > (among other things) KAFKA-9308[1] and would love some feedback if someone > can

KAFKA-9308: Request for Review of documentation

Hi everybody, I've reworked the SSL part of the documentation a little in order to fix (among other things) KAFKA-9308[1] and would love some feedback if someone can spare a few minutes. Pull request: https://github.com/apache/kafka/pull/8009 [1] https://issues.apache.org/jira/browse/KAFKA-9308

Re: [ANNOUNCE] New Kafka PMC Members: Colin, Vahid and Manikumar

Ismael Juma < > ism...@juma.me.uk> > > > > >> wrote: > > > > >>>>>>> > > > > >>>>>>> Congratulations Colin, Vahid and Manikumar! > > > > >>>>>>> > > > > >>>>>>> Ismael > > > > >>>>>>> > > > > >>>>>>> On Tue, Jan 14, 2020 at 9:30 AM Gwen Shapira < > > g...@confluent.io> > > > > >>>>> wrote: > > > > >>>>>>> > > > > >>>>>>>> Hi everyone, > > > > >>>>>>>> > > > > >>>>>>>> I'm happy to announce that Colin McCabe, Vahid Hashemian and > > > > >>>>> Manikumar > > > > >>>>>>>> Reddy are now members of Apache Kafka PMC. > > > > >>>>>>>> > > > > >>>>>>>> Colin and Manikumar became committers on Sept 2018 and Vahid > > on > > > > Jan > > > > >>>>>>>> 2019. They all contributed many patches, code reviews and > > > > >>>>> participated > > > > >>>>>>>> in many KIP discussions. We appreciate their contributions > > and are > > > > >>>>>>>> looking forward to many more to come. > > > > >>>>>>>> > > > > >>>>>>>> Congrats Colin, Vahid and Manikumar! > > > > >>>>>>>> > > > > >>>>>>>> Gwen, on behalf of Apache Kafka PMC > > > > >>>>>>>> > > > > >>>>>> > > > > >>>>> > > > > >>> > > > > >>> > > > > >> > > > > > > > > > > > > -- > > Thanks! > --Vahid > -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Re: [VOTE] KIP-551: Expose disk read and write metrics

; > > On Fri, Jan 10, 2020 at 2:17 PM Colin McCabe > > > > > wrote: > > > > > > > > > > > > > > Hi all, > > > > > > > > > > > > > > I'd like to start the vote on KIP-551: Expose disk read and > write > > > &g

Re: [VOTE] KIP-409: Allow creating under-replicated topics and partitions

ave updated the KIP to explicitly > mention this point. > > I hope that answers your question, let me know. > Thanks > > > On Mon, Dec 16, 2019 at 4:38 PM Sönke Liebau > wrote: > > > > Hi Michael, > > > > that sounds like a useful addition!

Re: [VOTE] KIP-409: Allow creating under-replicated topics and partitions

rtitions > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-409%3A+Allow+creating+under-replicated+topics+and+partitions > > > > > > Thanks > > -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Re: [jira] [Created] (KAFKA-9292) KIP-551: Expose disk read and write metrics

gt; > > > > -- > This message was sent by Atlassian Jira > (v8.3.4#803005) -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Re: Broker Interceptors

Hi Thomas, I think that idea is worth looking at. As you say, if no interceptor is configured then the performance overhead should be negligible. Basically it is then up to the user to decide if he wants tomtake the performance hit. We should make sure to think about monitoring capabilities like t

Handling deserialization exceptions in Kafka consumer

Hi all, I was recently discussing deserialization errors and how they can be handled in Kafka consumers. I believe the current code still throws an exception if deserialization fails which stops consumption unless you seek past that record. This creates an issue for a JDBC sink connector that us

Re: Preliminary blog post about the Apache Kafka 2.4.0 release

bout the upcoming Apache Kafka > 2.4.0 > > release. > > Please take a look and let me know if you want to add/modify details. > > Thanks to all who contributed to this blog post. > > > > > https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache1 > &

Re: [DISCUSS] KIP-317: Transparent Data Encryption

nd you'll need that before you can get the KIP accepted into > Kafka. > > Thanks, > Andrew Schofield (IBM) > > On 06/08/2019, 15:46, "Sönke Liebau" > wrote: > > Hi, > > I have so far received pretty much no comments on the technical details > ou

Re: [VOTE] KIP-499 - Unify connection name flag for command line tool

er-groups.sh, kafka-consumer-perf-test.sh, > > kafka-verifiable-consumer.sh, kafka-verifiable-producer.sh > > > > Thanks, > > -Mitch > > > > -- > > Gwen Shapira > > Product Manager | Confluent > > 650.450.2760 | @gwenshap > > Follow us: Twitter | blog > > > -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Re: [DISCUSS] KIP-317: Transparent Data Encryption

ttps scenarios. You may learn from those protocols on how they > handle keys, how long they keep them etc. to implement your E2e solution . > > > Am 08.08.2019 um 08:11 schrieb Maulin Vasavada < > maulin.vasav...@gmail.com>: > > > > Hi Sönke Liebau > > < >

Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface

thorizer. If you have come across other limitations that you would like > > to see addressed in the new API, please raise these on the discussion > > thread so that we can consider those too. All suggestions and feedback > are > > welcome. > > > > Thank you, > > > > Rajini > -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Re: [DISCUSS] KIP-317: Transparent Data Encryption

few people before I but more work into this. Best regards, Sönke On Tue, 21 May 2019 at 14:15, Sönke Liebau wrote: > Hi everybody, > > I'd like to rekindle the discussion around KIP-317. > I have reworked the KIP a little bit in order to design everything as a > pluggable implem

Jira Cleanup

All, I left a few comments on some old but still open jiras in an attempt to clean up a little bit. Since probably no one would notice these comments I thought I'd quickly list them here to give people a chance to check on them: KAFKA-822 : Reassi

Re: [DISCUSS] KIP-317: Transparent Data Encryption

https://cwiki.apache.org/confluence/download/attachments/85479936/kafka_e2e-encryption_control-flow.png?version=1&modificationDate=1558439227551&api=v2 On Fri, 10 Aug 2018 at 14:05, Sönke Liebau wrote: > Hi Viktor, > > thanks for your input! We could accommodate magic heade

Re: Cleaning up command line tools argument parsing a little

is currently based on a quite old trunk but if you think it's > > interesting I can rebase it and we can continue with raising a KIP. > > https://github.com/viktorsomogyi/kafka/tree/kafka_shell > > > > Viktor > > > > On Tue, May 7, 2019 at 11:10 AM Sönke Liebau > &

Re: Cleaning up command line tools argument parsing a little

27;d love to work on that, but I currently have enough other open things to look at, so I won't draw that one to myself as well for now. But I'll keep it in mind for when some time frees up :) Best regards, Sönke Colin McCabe schrieb am Di., 7. Mai 2019, 00:56: > On Mon, May 6, 2

Re: Cleaning up command line tools argument parsing a little

while as variants to ease the transition. > > best, > Colin > > > On Sun, May 5, 2019, at 00:54, Sönke Liebau wrote: > > Hi Colin, > > > > I totally agree! Especially the differently named bootstrap server > options > > have been annoy

Re: Cleaning up command line tools argument parsing a little

ig vs consumer-config. I think it's perfectly reasonable > to > > > have multiple spellings of the same arguments. I don't really see a > > > downside to keeping the aliases around indefinitely. > > > > > > Ryanne > > > > > > >

Re: Cleaning up command line tools argument parsing a little

gt; downside to keeping the aliases around indefinitely. > > > > Ryanne > > > > > > > > > > On Wed, Apr 17, 2019, 7:07 AM Sönke Liebau > > wrote: > > > > > Hi everybody, > > > > > > Jason and I were recently discussing comm

Cleaning up command line tools argument parsing a little

Hi everybody, Jason and I were recently discussing command line argument parsing on KAFKA-8131 (or rather the related pull request) [1]. Command line tools and their arguments are somewhat diverse at the moment. Most of the tools use joptsimple for argument parsing, some newer java tools use argp

Re: Proposal to Auto Close Inactive Tickets

Hi Addison, in general, I am totally in favor of closing unnecessary tickets! However, over the past few weeks, I have spent quite a bit of time looking at old tickets and evaluating whether those should be closed or are still relevant. For a surprising number of these tickets, I've found that the

Re: Kafka Jenkins not showing recent builds in history

Quick update: everything seems to be back to normal. On Mon, Apr 8, 2019 at 9:57 AM Sönke Liebau wrote: > I've opened an INFRA ticket about this [1]. Apparently, it is a known > issue that is solved by a restart of Jenkins. One will be scheduled in the > near future, I'll u

Re: [DISCUSS] KIP-252: Extend ACLs to allow filtering based on ip ranges and subnets

cala/kafka/security/auth/Acl.scala#L37 [3] https://github.com/apache/kafka/blob/b4fa87cc51f0a7d640dc6ae2cc8f89f006aae652/core/src/main/scala/kafka/security/auth/Acl.scala#L66 On Fri, Apr 5, 2019 at 8:09 AM Colin McCabe wrote: > On Thu, Apr 4, 2019, at 01:52, Sönke Liebau wrote: > > H

Re: Kafka Jenkins not showing recent builds in history

n Wed, Apr 3, 2019 at 10:41 PM Sönke Liebau wrote: > Hi everybody, > > I looked through recent Jenkins builds for a while today and it somehow > looks off to me. > > Both jobs [1] [2] don't show any builds that are more recent than March > 19th in the "build his

Re: [DISCUSS] KIP-252: Extend ACLs to allow filtering based on ip ranges and subnets

cult, but it should be spelled out. We could use INVALID_REQUEST > as the error code for this situation, or maybe create a new one to be more > specific. > > best, > Colin > > > On Wed, Apr 3, 2019, at 04:58, Sönke Liebau wrote: > > All, > > > > as this thread

Kafka Jenkins not showing recent builds in history

Hi everybody, I looked through recent Jenkins builds for a while today and it somehow looks off to me. Both jobs [1] [2] don't show any builds that are more recent than March 19th in the "build history". Only the "last failed" and "last unsuccessful" permalinks show recent dates. Build pages can

Re: [VOTE] KIP-443: Return to default segment.ms and segment.index.bytes in Streams repartition topics

19 at 7:08 PM Guozhang Wang < > >> wangg...@gmail.com> > >> > > > > wrote: > >> > > > > > > >> > > > > > > Hello folks, > >> > > > > > > > >> > > > > > > I'd like to directly start a voting thread on this simple > KIP > >> to > >> > > change > >> > > > > > the > >> > > > > > > default override values for repartition topics: > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-443%3A+Return+to+default+segment.ms+and+segment.index.bytes+in+Streams+repartition+topics > >> > > > > > > > >> > > > > > > The related PR can be found here as well: > >> > > > > > > https://github.com/apache/kafka/pull/6511 > >> > > > > > > > >> > > > > > > If you have any thoughts or feedbacks, they are more than > >> welcomed > >> > > as > >> > > > > > well. > >> > > > > > > > >> > > > > > > > >> > > > > > > -- Guozhang > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > >> > > >> > > >> > -- > >> > -- Guozhang > >> > > > > > > -- > > -- Guozhang > > > > > -- > -- Guozhang > -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Re: [DISCUSS] KIP-252: Extend ACLs to allow filtering based on ip ranges and subnets

nally consider the security implications of these ACLs in a mixed version cluster quite minimal and addressable via the release notes. Best, Sönke On Sat, Mar 16, 2019 at 1:32 PM Sönke Liebau wrote: > Just a quick bump, as this has been quiet for a while again. > > On Tue, Jan 8, 2019 at

Re: [VOTE] KIP-349 Priorities for Source Topics

s also to everyone who participated in the discussion. > > > > > > Best regards, > > > -- > > > Nick > > > > > > > On Jan 25, 2019, at 5:51 AM, Sönke Liebau < > soenke.lie...@opencore.com.INVALID> wrote: > > > > >

Re: [VOTE] KIP-349 Priorities for Source Topics

2019/01/28 02:26:31, n...@afshartous.com wrote: > > Hi Sönke, > > > > Thanks for taking the time to review. I’ve put KIP-349 into > hibernation. > > > > Thanks also to everyone who participated in the discussion. > > > > Best regards, > > -- &

Re: [DISCUSS] KIP-252: Extend ACLs to allow filtering based on ip ranges and subnets

Just a quick bump, as this has been quiet for a while again. On Tue, Jan 8, 2019 at 12:44 PM Sönke Liebau wrote: > Hi Colin, > > thanks for your response! > > in theory we could get away without any additional path changes I > think.. I am still somewhat unsure about the best

Re: Apache Kafka Memory Leakage???

he heap memory goes to about 60 or 70MB, and then falls > back, and the pattern continues. Max heap consumed at an instant is about > 70MB. > > Maybe there is a problem in my platform on updating the stats. I shall > convey it to my team. > > thanks so much for the help. &

Re: Apache Kafka Memory Leakage???

uesting as I can share all the details and my screen as well. > After that, we can see what to do next. If the call is not possible then > its ok but it would be great to have it. > Thanks, > > > > On Tue, Mar 5, 2019 at 4:29 PM Sönke Liebau > wrote: > >> Hi Syed,

Re: Apache Kafka Memory Leakage???

can to betterize the > kafka-clients library. > What the steps next to confirm its a bug in kafka-clients? And if its a > bug whats the process to get it resolved? > > Thanks, > > > > On Tue, Mar 5, 2019 at 2:43 PM Sönke Liebau > wrote: > >> Hi Syed, >>

Re: Apache Kafka Memory Leakage???

the kafka-clients library. > > Thanks, > > > > On Mon, Mar 4, 2019 at 8:33 PM Sönke Liebau > wrote: > >> Hi Syed, >> >> and you are sure that this memory is actually allocated? I still have my >> reservations about that metric to be honest. Is the

Re: Apache Kafka Memory Leakage???

; isRunning.get()); > > > *Note: *Debugger is a wrapper class that just writes the given string to a > local file using PrintStream's println() method. > > And I don't see any diff in the metrics. I still see the huge amount of > memory allocated. > > See the imag

Re: Apache Kafka Memory Leakage???

poll(Duration) > method. Still, I see memory issue. > Is there a way we can get on a one-one call and discuss this pls? Let me > know your availability. I can share zoom meeting link. > > Thanks, > > > > On Sat, Mar 2, 2019 at 2:15 AM Sönke Liebau > wrote: >> &g

Re: Apache Kafka Memory Leakage???

hange the current offset > if (isSyncCommit && isSuggest) { > commitOffset(offsets); > offsets.clear(); > } > } > } while ((msgCount > 0 || infiniteLoop) && isRunning.get()

Re: Speeding up integration tests

case we could cut the testing time back by approximately an > > hour. This would save resources and give quicker feedback for PR builds. > > > > What are your thoughts? > > Has anyone thought about this or were there any attempts made? > > > > Best, > > Viktor > > > > > -- > Best, > Stanislav -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Re: [DISCUSSION] KIP-422: Add support for user/client configuration in the Kafka Admin Client

;> or client configurations for users, instead of requiring users to directly > >> talk to ZK. > >> > >> The link for this KIP is > >> following: > >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=97555704 > >> > >&g

Some jira cleanup

All, I left a few comments on some old but still open jiras in an attempt to clean up a little bit. Since probably no one would notice these comments I thought I'd quickly list them here to give people a chance to check on them: KAFKA-217 : Client

Re: [VOTE] KIP-349 Priorities for Source Topics

'd recommend getting started with something easier. > > > Yes it does. And after 6 months of (sometimes circular) discussion I’d like > to either move towards a vote or set the status of this KIP to dormant until > if and when someone else picks up it up. > > Does anybody

Re: Delayed messages with skewed data when subscribing to many partitions

not checked due to the limits, but that > would need a wire protocol change. > Perhaps a little consumer side logic that starts fragmenting fetch > requests if it notices that responses always have data from the > maximum number of partitions. > > Best regards, > Sönke > > [1] > https://cwiki.apache.org/confluence/display/KAFKA/KIP-349%3A+Priorities+for+Source+Topics -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Delayed messages with skewed data when subscribing to many partitions

Hi all, we've just had a case where we suspect that messages get delayed from being consumed under certain circumstances. I don't necessarily think this is a bug, hence have not opened a jira yet but wanted to discuss here - there's probably a best practice that I just don't know about. The scena

Re: [DISCUSS] KIP-252: Extend ACLs to allow filtering based on ip ranges and subnets

ning and allows storing everything in one compacted topic. Any thoughts on this? Best regards, Sönke [1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-290%3A+Support+for+Prefixed+ACLs On Sat, Dec 22, 2018 at 5:51 AM Colin McCabe wrote: > > Hi Sönke, > > One path forward

Re: [VOTE] KIP-382 MirrorMaker 2.0

; > > - Automatically syncs topic configuration between clusters. > > > > > > - Manages downstream topic ACL. > > > > > > - Supports "active/active" cluster pairs, as well as any number of > > > > active > > > > > > clu

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

a REST API to > do that. Legacy MirrorMaker doesn't have a REST API after all. For > organizations that want on-the-fly configuration of their replication flows, > there's Connect. > > This has been brought up by nearly everyone, so I'm happy to oblige. > > Ryanne

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

> D.C.B.A... > > > > > > > topics > > > > > > > when you create a fully-connected mesh topology. > > > > > > > > > > > > > > Looking ahead a bit, I can imagine an external tool > computing > > > the > > &

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

or stores its own offsets in the target cluster, so no > > state information is lost if the source cluster disappears. Both of these > > features are facilitated by the Connect framework and were inspired by > > Uber's uReplicator. > > > > > If the single conne

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

afka cluster. If Connect supported multiple workers in the same cluster, > it would start to look a lot like a MM2 cluster. > > Ryanne > > On Tue, Dec 4, 2018 at 3:26 PM Sönke Liebau > wrote: > > > Hi Ryanne, > > > > thanks for your response! > > > >

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

a single Kafka cluster. I think Connect-as-a-Service and > Replication-as-a-Service are sufficiently different use-cases that we > should expect the APIs and configuration files to be at least slightly > different, even if both use the same framework underneath. That said, I do > plan

Re: [DISCUSS] KIP-398: Support reading trust store from classpath

ailable here: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-398%3A+Support+reading+trust+store+from+classpath > < > https://cwiki.apache.org/confluence/display/KAFKA/KIP-398:+Support+reading+trust+store+from+classpath > > > > Any feedback or comments would be most welcome.

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

; across any number of Kafka clusters. Much like Brooklin, MM2 does the work > of setting up connectors between clusters as needed. This > Replication-as-a-Service is a huge win for larger deployments, as well as > for organizations that haven't adopted Connect. > > [1] > htt

Re: [DISCUSS] KIP-252: Extend ACLs to allow filtering based on ip ranges and subnets

doesn't seem right. Looking forward to your input :) Best regards, Sönke On Tue, Jun 19, 2018 at 5:32 PM Sönke Liebau wrote: > Picking this back up, now that KIP-290 has been merged.. > > As Colin mentioned in an earlier mail this change could create a > potential security issue i

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

k a look how streams does exactly once. > >> (its a > >> > > > trap > >> > > > >>> > > anyways and you usually can deal with at least once > >> donwstream > >> > > > pretty > >> > > > >>> > > easy). But I am very certain its not gonna get somewhere > if > >> > > offset > >> > > > >>> > > commit and record produce cluster are not the same. > >> > > > >>> > > > >> > > > >>> > > Pretty sure without this _design choice_ you can skip on > >> that > >> > > > exactly > >> > > > >>> > > once already > >> > > > >>> > > > >> > > > >>> > > Best Jan > >> > > > >>> > > > >> > > > >>> > > On 16.10.2018 18:16, Ryanne Dolan wrote: > >> > > > >>> > > > > But one big obstacle in this was > >> > > > >>> > > > always that group coordination happened on the source > >> cluster. > >> > > > >>> > > > > >> > > > >>> > > > Jan, thank you for bringing up this issue with legacy > >> > > > MirrorMaker. > >> > > > >>> I > >> > > > >>> > > > totally agree with you. This is one of several problems > >> with > >> > > > >>> MirrorMaker > >> > > > >>> > > > I intend to solve in MM2, and I already have a design > and > >> > > > >>> prototype that > >> > > > >>> > > > solves this and related issues. But as you pointed out, > >> this > >> > > KIP > >> > > > is > >> > > > >>> > > > already rather complex, and I want to focus on the core > >> feature > >> > > > set > >> > > > >>> > > > rather than performance optimizations for now. If we can > >> agree > >> > > on > >> > > > >>> what > >> > > > >>> > > > MM2 looks like, it will be very easy to agree to improve > >> its > >> > > > >>> performance > >> > > > >>> > > > and reliability. > >> > > > >>> > > > > >> > > > >>> > > > That said, I look forward to your support on a > subsequent > >> KIP > >> > > > that > >> > > > >>> > > > addresses consumer coordination and rebalance issues. > Stay > >> > > tuned! > >> > > > >>> > > > > >> > > > >>> > > > Ryanne > >> > > > >>> > > > > >> > > > >>> > > > On Tue, Oct 16, 2018 at 6:58 AM Jan Filipiak < > >> > > > >>> jan.filip...@trivago.com > >> > > > >>> > > > <mailto:jan.filip...@trivago.com>> wrote: > >> > > > >>> > > > > >> > > > >>> > > > Hi, > >> > > > >>> > > > > >> > > > >>> > > > Currently MirrorMaker is usually run collocated with > >> the > >> > > > target > >> > > > >>> > > > cluster. > >> > > > >>> > > > This is all nice and good. But one big obstacle in > >> this was > >> > > > >>> > > > always that group coordination happened on the > source > >> > > > cluster. > >> > > > >>> So > >> > > > >>> > > when > >> > > > >>> > > > then network was congested, you sometimes loose > group > >> > > > >>> membership and > >> > > > >>> > > > have to rebalance and all this. > >> > > > >>> > > > > >> > > > >>> > > > So one big request from we would be the support of > >> having > >> > > > >>> > > coordination > >> > > > >>> > > > cluster != source cluster. > >> > > > >>> > > > > >> > > > >>> > > > I would generally say a LAN is better than a WAN for > >> doing > >> > > > >>> group > >> > > > >>> > > > coordinaton and there is no reason we couldn't have > a > >> group > >> > > > >>> consuming > >> > > > >>> > > > topics from a different cluster and committing > >> offsets to > >> > > > >>> another > >> > > > >>> > > > one right? > >> > > > >>> > > > > >> > > > >>> > > > Other than that. It feels like the KIP has too much > >> > > features > >> > > > >>> where > >> > > > >>> > > many > >> > > > >>> > > > of them are not really wanted and counter productive > >> but I > >> > > > >>> will just > >> > > > >>> > > > wait and see how the discussion goes. > >> > > > >>> > > > > >> > > > >>> > > > Best Jan > >> > > > >>> > > > > >> > > > >>> > > > > >> > > > >>> > > > On 15.10.2018 18:16, Ryanne Dolan wrote: > >> > > > >>> > > > > Hey y'all! > >> > > > >>> > > > > > >> > > > >>> > > > > Please take a look at KIP-382: > >> > > > >>> > > > > > >> > > > >>> > > > > > >> > > > >>> > > > > >> > > > >>> > > > >> > > > >>> > >> > > > > >> > > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0 > >> > > > >>> > > > > > >> > > > >>> > > > > Thanks for your feedback and support. > >> > > > >>> > > > > > >> > > > >>> > > > > Ryanne > >> > > > >>> > > > > > >> > > > >>> > > > > >> > > > >>> > > > >> > > > >>> > >> > > > >> > >> > > > > >> > > > >> > > > >> > > -- > >> > > Best, > >> > > Alex Mironov > >> > > > >> > > >> > > > -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Re: [ANNOUNCE] New Committer: Manikumar Reddy

/confluence/display/KAFKA/KIP-249%3A+Add+Delegation+Token+Operations+to+KafkaAdminClient > > > > > > > > He has broad experience working with many of the core components in > > Kafka > > > > and he has reviewed over 80 PRs. He has also made huge progress > > > addressing > > > > some of our technical debt. > > > > > > > > We appreciate the contributions and we are looking forward to more. > > > > Congrats Manikumar! > > > > > > > > Jason, on behalf of the Apache Kafka PMC > > > > > > > > > > -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Re: [Review] - Pull request #5681 review

you in advance. > > - > Srinivas > > - Typed on tiny keys. pls ignore typos.{mobile app} > -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Re: [DISCUSS] KIP-317: Transparent Data Encryption

he level of > security they provide to make sure the users are fully aware of the > security implications. > > Cheers, > Viktor > > On Tue, Jun 19, 2018 at 11:55 AM Sönke Liebau > wrote: > > > Hi Stephane, > > > > thanks for pointing out the broken pictures

Re: [DISCUSS] KIP-252: Extend ACLs to allow filtering based on ip ranges and subnets

way to go about things ... I agree :) But I couldn't come up with a better way yet. Any thoughts? Best regards, Sönke [1] https://github.com/apache/kafka/pull/5079#pullrequestreview-124512689 On Thu, May 3, 2018 at 10:57 PM, Sönke Liebau wrote: > Technically I absolutely agree with you

Re: [DISCUSS] KIP-317: Transparent Data Encryption

=> > compression => encryption => send. > and the inverse for a consumer. > > Regards > Stephane > > On 19 June 2018 at 06:46, Sönke Liebau > wrote: > >> Hi everybody, >> >> I've created a draft version of KIP-317 which describes the addition >>

[DISCUSS] KIP-317: Transparent Data Encryption

Hi everybody, I've created a draft version of KIP-317 which describes the addition of transparent data encryption functionality to Kafka. Please consider this as a basis for discussion - I am aware that this is not at a level of detail sufficient for implementation, but I wanted to get some feedb

Re: [DISCUSS] KIP-252: Extend ACLs to allow filtering based on ip ranges and subnets

t been upgraded, they will not honor the DENY ACL, > possibly causing a security issue. > > In general, it seems like we need some kind of versioning system in ACLs to > handle these cases. > > best, > Colin > > On Thu, May 3, 2018, at 08:11, Sönke Liebau wrote: >&g

Re: [DISCUSS] KIP-252: Extend ACLs to allow filtering based on ip ranges and subnets

nderway in KIP-290 that cover most of the suggestions already. Does anybody have any other objections to starting a vote on this KIP? Regards, Sönke On Fri, Feb 2, 2018 at 5:11 PM, Sönke Liebau wrote: > Hi Manikumar, > > you are right, 5713 is a bit ambiguous about which fields are co

Are there plans to migrate some/all of the command line tools to use the new AdminClient?

I've dug around jira and the list of KIPs for a bit now, but could not really find anything specific on plans to move the command line tools over to the new AdminClient. Did I miss something or is that not currently planned? Most of the current command line tools require access to Zookeeper, which

Re: Documentation build system

the project, I figured I'd send this quick note to >> increase the likelihood people see it and have a chance to weigh in. >> >> Thanks, >> -Ewen >> > > > > -- > -- Guozhang -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Re: [DISCUSS] KIP-252: Extend ACLs to allow filtering based on ip ranges and subnets

n't makes sense. Range and subnet support will give us the flexibility. On Thu, Feb 1, 2018 at 5:56 PM, Sönke Liebau < soenke.lie...@opencore.com.invalid> wrote: > Hi Manikumar, > > the current proposal indeed leaves out IPv6 addresses, as I was unsure > whether Kafka fully

Re: [DISCUSS] KIP-252: Extend ACLs to allow filtering based on ip ranges and subnets

gards, Sönke On Thu, Feb 1, 2018 at 10:14 AM, Manikumar wrote: > Hi, > > 1. Do we support IPv6 CIDR/ranges? > > 2. KAFKA-5713 is mentioned in Related JIRAs section. But there is no > mention of wildcard support in the KIP. > > > Thanks, > > On Thu, Feb 1, 2018 a

[DISCUSS] KIP-252: Extend ACLs to allow filtering based on ip ranges and subnets

Hey everybody, following a brief inital discussion a couple of days ago on this list I'd like to get a discussion going on KIP-252 which would allow specifying ip ranges and subnets for the -allow-host and --deny-host parameters of the acl tool. The KIP can be found at https://cwiki.apache.org/co

Re: [VOTE] KIP-212: Enforce set of legal characters for connector names

Tue, Jan 23, 2018 at 10:14 PM, Jason Gustafson > wrote: > >> +1 >> >> On Tue, Jan 23, 2018 at 2:01 PM, Gwen Shapira wrote: >> >> > +1 (binding) >> > >> > On Tue, Jan 23, 2018, 1:17 PM Randall Hauch wrote: >> > >> > > +

Re: [DISCUSS] KIP-212: Enforce set of legal characters for connector names

his maybe causing issues somewhere else in the code because we created an illegal escape sequence again, or are we happy with that behavior for now? Kind regards, Sönke [1] https://docs.oracle.com/javase/7/docs/api/java/lang/Character.html#isISOControl(int) Am 21.01.2018 23:35 schrieb "Sönke

Re: [DISCUSS] Improving ACLs by allowing ip ranges and subnet expressions?

on a local network, or impersonate a DNS server. For users who want some access controls, but don't care about security, maybe we should make it easier to use and create users without enabling kerberos or similar? best, Colin On Wed, Jan 24, 2018, at 12:59, Sönke Liebau wrote: > Hi everyone,

[DISCUSS] Improving ACLs by allowing ip ranges and subnet expressions?

Hi everyone, the current ACL functionality in Kafka is a bit limited concerning host based rules when specifying multiple hosts. A common scenario for this would be that if have a YARN cluster running Spark jobs that access Kafka and want to create ACLs based on the ip addresses of the cluster nod

  1   2   >