Re: [DISCUSS] 2017 October release planning and release version

I am happy to volunteer for working on documenting the release logistics and related topics if someone is needed for that part. Best regards, Sönke Am 24.07.2017 9:28 nachm. schrieb "Guozhang Wang" : > Thanks for everyone who has shared their thoughts and feedbacks to this >

[DISCUSS] KAFKA-5669: Define allowed characters for connector names

As stated in the jira, we should come up with either a set of allowed or a set of disallowed characters for connector names to avoid a couple of issues that currently exist. I tend towards a whitelisting approach to avoid missing a lot of things that then need to be fixed later on. According to

pull request #2755: request for review

The pull request has been open for a while now, can someone please have a look at whether this is still viable and something we want to do? I've updated and resolved conflicts yesterday.

[DISCUSS] KAFKA-4930 & KAFKA 4938 - Treatment of name parameter in create connector requests

While working on KAFKA-4930 and KAFKA-4938 I came across some sort of fundamental questions about the rest api for creating connectors in Kafka Connect that I'd like to put up for discussion. Currently requests that do not contain a "name" element on the top level are not accepted by the API, but

Re: [DISCUSS] KAFKA-4930 & KAFKA 4938 - Treatment of name parameter in create connector requests

d some ideas > how this should be resolved. I hope he responds, but I think that for your > PR, just accept current mess as given... > > Gwen > > On Tue, Jul 4, 2017 at 3:28 AM Sönke Liebau > <soenke.lie...@opencore.com.invalid> wrote: > > > While working on KAFKA

Re: kafka connect questions

lay Teahouse <clayteaho...@gmail.com> > > wrote: > > > > > Hello All, > > > > > > I'd appreciate your help with the following questions. > > > > > > 1) Is there kafka connect for listening to tcp sockets? > > > > > > 2) I

Re: [DISCUSS] KAFKA-4930 & KAFKA 4938 - Treatment of name parameter in create connector requests

is. Best regards, Sönke On Wed, Jul 5, 2017 at 9:31 AM, Sönke Liebau <soenke.lie...@opencore.com> wrote: > Hi, > > regarding "breaking existing functionality" .. yes...that was me getting > confused about intended and existing functionality :) > You are right, th

Re: Jenkins problem with JDK 8 and Scala 2.12 ?

e Advisor > > Twitter : @ppatierno<http://twitter.com/ppatierno> > Linkedin : paolopatierno<http://it.linkedin.com/in/paolopatierno> > Blog : DevExperience<http://paolopatierno.wordpress.com/> > -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Re: [DISCUSS] KIP-186: Increase offsets retention default to 7 days

sers: > >> KIP-186: Increase offsets retention default to 7 days > >> > >> https://cwiki.apache.org/confluence/display/KAFKA/KIP- > >> 186%3A+Increase+offsets+retention+default+to+7+days > >> > >> Note that in addition to the KIP text itself, the linked

Re: [DISCUSS] KAFKA-4930 & KAFKA 4938 - Treatment of name parameter in create connector requests

:) Thanks! Jira: https://issues.apache.org/jira/browse/KAFKA-4930 PR: https://github.com/apache/kafka/pull/2755 On Thu, Jul 6, 2017 at 8:19 PM, Gwen Shapira <g...@confluent.io> wrote: > This sounds great. I'll try to review later today :) > > On Thu, Jul 6, 2017 at 12:35 AM Sönke Lieba

[DISCUSS] KIP-212: Enforce set of legal characters for connector names

All, I've created a KIP to discuss enforcing of rules on what characters are allowed in connector names. Since this may break api calls that are currently working I figured a KIP is the better way to go than to just create a jira. I'd love to hear your input on this!

Request for Review: KAFKA-4930

Hi everybody, could one of the committers please have a quick look at my PR [1] for KAFKA-4930? I think I've addressed all comments I received so far and its been sitting for a while ready for review. Kind regards, Sönke [1] https://github.com/apache/kafka/pull/2755

Request for access to the KIP wiki page

Hi all, could someone please grant my user (userid: soenkeliebau) permissions to the KIP page of the wiki, I'd like to create a proposal. Thanks! Sönke

Re: [DISCUSS] KIP-212: Enforce set of legal characters for connector names

the > > documentation and specify what names are valid. > > > > WDYT? > > > > Randall > > > > On Fri, Oct 27, 2017 at 1:08 PM, Colin McCabe <cmcc...@apache.org> > wrote: > > > >> On Wed, Oct 25, 2017, at 01:07, Sönke Liebau wrote: >

Re: [DISCUSS] KIP-212: Enforce set of legal characters for connector names

/KAFKA/KIP-212%3A+Enforce+set+of+legal+characters+for+connector+names On Thu, Nov 16, 2017 at 2:59 PM, Sönke Liebau <soenke.lie...@opencore.com> wrote: > Hi Randall, > > I had mentioned this edge case in the KIP, but will add some further > detail to further clarify all changing s

Re: [DISCUSS] KIP-212: Enforce set of legal characters for connector names

and/or trailing whitespace characters will still be able to be > updated and deleted. I think that decreases the likelihood of this change > negatively impacting existing users. Basically, going forward, the names of > new connectors will be trimmed. > > WDYT? > > On

Re: [DISCUSS] KIP-212: Enforce set of legal characters for connector names

e we want to change/correct the existing > behavior. But we do need to clarify in the KIP these edge cases that will > change. > > Thanks for the continued work on this, Sönke. > > Regards, > > Randall > > > On Nov 16, 2017, at 1:56 AM, Sönke Liebau > > <soen

Re: [DISCUSS] KIP-212: Enforce set of legal characters for connector names

@Randall: are you happy with the KIP as it stands so I can call for a vote, or are there any outstanding items still to discuss? Same question to anybody else who'd like to participate of course :) On Thu, Nov 16, 2017 at 5:35 PM, Sönke Liebau <soenke.lie...@opencore.com> wrote: > So

Re: KIP-199 - Could offset management be part of the Connect REST API?

hen to have some > functionality as a separate tool vs. in the API, but I thought I'd bring up > the idea and see what others think. > > Thanks, > > --Gunnar > > [1] > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 199%3A+Add+Kafka+Connect+offset+tool > --

Re: [DISCUSS] KIP-212: Enforce set of legal characters for connector names

or updating the KIP to reflect the current process. However, I > still question whether it is necessary to have a KIP - it depends on > whether it was possible with prior versions to have connectors with > zero-length or blank names. Have you tried both of these cases? > > On Fri, Nov

Re: [DISCUSS] KIP-212: Enforce set of legal characters for connector names

an fixing some > > bugs related to proper encoding/decoding. > > > > Thoughts? > > > > > > On Mon, Oct 23, 2017 at 3:44 PM, Sönke Liebau < > > soenke.lie...@opencore.com.invalid> wrote: > > > > > All, > > > > > > I've cr

Re: [DISCUSS] KIP-212: Enforce set of legal characters for connector names

at 8:40 PM, Colin McCabe <cmcc...@apache.org> wrote: > On Tue, Oct 24, 2017, at 11:28, Sönke Liebau wrote: > > Hi, > > > > after reading your messages I'll grant that I might have picked a > > somewhat > > draconic option to solve these issues. > > > &g

Re: Use self contained tokens instead of ACL

r > > - > ATTENTION: > The information in this e-mail is confidential and only meant for the > intended recipient. If you are not the intended recipient, don't use or > disclose it in any way. Please let the sender know and delet

[DISCUSS] KIP-317: Transparent Data Encryption

Hi everybody, I've created a draft version of KIP-317 which describes the addition of transparent data encryption functionality to Kafka. Please consider this as a basis for discussion - I am aware that this is not at a level of detail sufficient for implementation, but I wanted to get some

Re: [DISCUSS] KIP-252: Extend ACLs to allow filtering based on ip ranges and subnets

gree :) But I couldn't come up with a better way yet. Any thoughts? Best regards, Sönke [1] https://github.com/apache/kafka/pull/5079#pullrequestreview-124512689 On Thu, May 3, 2018 at 10:57 PM, Sönke Liebau wrote: > Technically I absolutely agree with you, this would indeed create > iss

Re: [DISCUSS] KIP-252: Extend ACLs to allow filtering based on ip ranges and subnets

in KIP-290 that cover most of the suggestions already. Does anybody have any other objections to starting a vote on this KIP? Regards, Sönke On Fri, Feb 2, 2018 at 5:11 PM, Sönke Liebau <soenke.lie...@opencore.com> wrote: > Hi Manikumar, > > you are right, 5713 is a bit ambiguous abo

Re: [DISCUSS] KIP-252: Extend ACLs to allow filtering based on ip ranges and subnets

haven't been upgraded, they will not honor the DENY ACL, > possibly causing a security issue. > > In general, it seems like we need some kind of versioning system in ACLs to > handle these cases. > > best, > Colin > > On Thu, May 3, 2018, at 08:11, Sönke Liebau wrote: >> H

Re: Use self contained tokens instead of ACL

connect to the service. If we really encounter malicious > behaviour, the prefer way is to shut down the misbehaving client. > > Kind Regards, > Peter > > > -Original Message- > From: Sönke Liebau [mailto:soenke.lie...@opencore.com.INVALID] > Sent: Donne

Re: Use self contained tokens instead of ACL

the fast replay. We don’t want to use Kerberos since we want to > do the authorization on Application level and without involvement of a 3rd > party during runtime. > > -----Original Message- > From: Sönke Liebau [mailto:soenke.lie...@opencore.com.INVALID] > Sent: Mittwoch, 25. Ok

Re: Before creating KIP : Kafka Connect / Add a configuration provider class

l deployed connectors, to provide default configs or > to > > provide sensitive configs like user/password. > > > > I've met these requirements on different projects. > > > > Do you think, this feature merits a new KIP ? > > > > Thanks, > > > > -- > > Florian HUSSONNOIS > > > -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Re: Use self contained tokens instead of ACL

; 48+Delegation+token+support+for+Kafka > > currently the PR is under review process. Maybe this can used be as > starting point for your requirement. > > https://github.com/apache/kafka/pull/3616 > > > > On Fri, Oct 27, 2017 at 2:34 PM, Sönke Liebau < > soe

Re: [DISCUSS] KIP-212: Enforce set of legal characters for connector names

about control >> characters in the KIP and kick off a VOTE thread? >> >> -Ewen >> >> On Fri, Jan 12, 2018 at 1:43 PM, Colin McCabe <cmcc...@apache.org> wrote: >> >> > On Fri, Jan 12, 2018, at 08:03, Sönke Liebau wrote: >> > > Hi everybody, &g

Re: [DISCUSS] KIP-212: Enforce set of legal characters for connector names

not specifically listed the escape sequences as I will have to dig into that topic a bit more to come up with a useful solution I think, but the general principle is stated. Let me know what you think. Best regards, Sönke On Sun, Jan 21, 2018 at 8:37 PM, Sönke Liebau <soenke.lie...@opencore.com> wrote

[DISCUSS] Improving ACLs by allowing ip ranges and subnet expressions?

Hi everyone, the current ACL functionality in Kafka is a bit limited concerning host based rules when specifying multiple hosts. A common scenario for this would be that if have a YARN cluster running Spark jobs that access Kafka and want to create ACLs based on the ip addresses of the cluster

Re: [DISCUSS] KIP-212: Enforce set of legal characters for connector names

ng issues somewhere else in the code because we created an illegal escape sequence again, or are we happy with that behavior for now? Kind regards, Sönke [1] https://docs.oracle.com/javase/7/docs/api/java/lang/Character.html#isISOControl(int) Am 21.01.2018 23:35 schrieb "Sönke Lieba

Re: [VOTE] KIP-212: Enforce set of legal characters for connector names

n 23, 2018, 1:17 PM Randall Hauch <rha...@gmail.com> wrote: >> > >> > > +1 (non-binding) >> > > >> > > On Mon, Jan 22, 2018 at 6:35 PM, Sönke Liebau < >> > > soenke.lie...@opencore.com.invalid> wrote: >> > > >>

Re: [DISCUSS] Improving ACLs by allowing ip ranges and subnet expressions?

P cache on a local network, or impersonate a DNS server. For users who want some access controls, but don't care about security, maybe we should make it easier to use and create users without enabling kerberos or similar? best, Colin On Wed, Jan 24, 2018, at 12:59, Sönke Liebau wrote: > Hi

Re: [DISCUSS] KIP-212: Enforce set of legal characters for connector names

gt; > > any >> > > > user input and document why we can't handle certain inputs and how >> the >> > > user >> > > > should adapt if we can't. In general I try to work under the >> robustness >> > > > principle: *Be conservative in what

Re: [DISCUSS] KIP-252: Extend ACLs to allow filtering based on ip ranges and subnets

low >> specifying ip ranges and subnets for the -allow-host and --deny-host >> parameters of the acl tool. >> >> The KIP can be found at >> https://cwiki.apache.org/confluence/display/KAFKA/KIP- >> 252+-+Extend+ACLs+to+allow+filtering+based+on+ip+ranges+and+subnets >&g

Re: [DISCUSS] KIP-252: Extend ACLs to allow filtering based on ip ranges and subnets

. Yes, wildcards (*) in hostname doesn't makes sense. Range and subnet support will give us the flexibility. On Thu, Feb 1, 2018 at 5:56 PM, Sönke Liebau < soenke.lie...@opencore.com.invalid> wrote: > Hi Manikumar, > > the current proposal indeed leaves out IPv6 addresses, as I was unsure

[DISCUSS] KIP-252: Extend ACLs to allow filtering based on ip ranges and subnets

Hey everybody, following a brief inital discussion a couple of days ago on this list I'd like to get a discussion going on KIP-252 which would allow specifying ip ranges and subnets for the -allow-host and --deny-host parameters of the acl tool. The KIP can be found at

Re: Documentation build system

> everyone contributing to the project, I figured I'd send this quick note to >> increase the likelihood people see it and have a chance to weigh in. >> >> Thanks, >> -Ewen >> > > > > -- > -- Guozhang -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Are there plans to migrate some/all of the command line tools to use the new AdminClient?

I've dug around jira and the list of KIPs for a bit now, but could not really find anything specific on plans to move the command line tools over to the new AdminClient. Did I miss something or is that not currently planned? Most of the current command line tools require access to Zookeeper,

Re: [DISCUSS] KIP-317: Transparent Data Encryption

pression => encryption => send. > and the inverse for a consumer. > > Regards > Stephane > > On 19 June 2018 at 06:46, Sönke Liebau > wrote: > >> Hi everybody, >> >> I've created a draft version of KIP-317 which describes the addition >> of

Re: [DISCUSS] KIP-317: Transparent Data Encryption

ty they provide to make sure the users are fully aware of the > security implications. > > Cheers, > Viktor > > On Tue, Jun 19, 2018 at 11:55 AM Sönke Liebau > wrote: > > > Hi Stephane, > > > > thanks for pointing out the broken pictures, I fixed those. > >

[VOTE] KIP-212: Enforce set of legal characters for connector names

All, this KIP has been discussed for quite some time now and I believe we addressed all major concerns in the current revision, so I'd like to start a vote. KIP can be found here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-212%3A+Enforce+set+of+legal+characters+for+connector+names

Re: [Review] - Pull request #5681 review

in advance. > > - > Srinivas > > - Typed on tiny keys. pls ignore typos.{mobile app} > -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Re: [ANNOUNCE] New Committer: Manikumar Reddy

/confluence/display/KAFKA/KIP-249%3A+Add+Delegation+Token+Operations+to+KafkaAdminClient > > > > > > > > He has broad experience working with many of the core components in > > Kafka > > > > and he has reviewed over 80 PRs. He has also made huge progress > > > addressing > > > > some of our technical debt. > > > > > > > > We appreciate the contributions and we are looking forward to more. > > > > Congrats Manikumar! > > > > > > > > Jason, on behalf of the Apache Kafka PMC > > > > > > > > > > -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

at. Legacy MirrorMaker doesn't have a REST API after all. For > organizations that want on-the-fly configuration of their replication flows, > there's Connect. > > This has been brought up by nearly everyone, so I'm happy to oblige. > > Ryanne > -- Sönke Liebau Partner Tel. +4

Re: [VOTE] KIP-382 MirrorMaker 2.0

; - Automatically syncs topic configuration between clusters. > > > > > > - Manages downstream topic ACL. > > > > > > - Supports "active/active" cluster pairs, as well as any number of > > > > active > > > > > > clusters. >

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

in the target cluster, so no > > state information is lost if the source cluster disappears. Both of these > > features are facilitated by the Connect framework and were inspired by > > Uber's uReplicator. > > > > > If the single connect cluster model is indeed use

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

> > > > > > > > > > Looking ahead a bit, I can imagine an external tool > computing > > > the > > > > > > > spanning > > > > > > > tree of topics among a set of clusters based on > inter-cluster > > >

Re: [DISCUSS] KIP-252: Extend ACLs to allow filtering based on ip ranges and subnets

doesn't seem right. Looking forward to your input :) Best regards, Sönke On Tue, Jun 19, 2018 at 5:32 PM Sönke Liebau wrote: > Picking this back up, now that KIP-290 has been merged.. > > As Colin mentioned in an earlier mail this change could create a > potential security issue if not

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

cluster. If Connect supported multiple workers in the same cluster, > it would start to look a lot like a MM2 cluster. > > Ryanne > > On Tue, Dec 4, 2018 at 3:26 PM Sönke Liebau > wrote: > > > Hi Ryanne, > > > > thanks for your response! > > > >

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

Service and > Replication-as-a-Service are sufficiently different use-cases that we > should expect the APIs and configuration files to be at least slightly > different, even if both use the same framework underneath. That said, I do > plan to contribute a few improvements to the Co

Re: [DISCUSS] KIP-398: Support reading trust store from classpath

ailable here: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-398%3A+Support+reading+trust+store+from+classpath > < > https://cwiki.apache.org/confluence/display/KAFKA/KIP-398:+Support+reading+trust+store+from+classpath > > > > Any feedback or comments would be most welcome.

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

ka clusters. Much like Brooklin, MM2 does the work > of setting up connectors between clusters as needed. This > Replication-as-a-Service is a huge win for larger deployments, as well as > for organizations that haven't adopted Connect. > > [1] > https://www.slideshare.net/ConfluentIn

Re: [DISCUSS] KIP-382: MirrorMaker 2.0

gt; > >>> > > anyways and you usually can deal with at least once > >> donwstream > >> > > > pretty > >> > > > >>> > > easy). But I am very certain its not gonna get somewhere > if > >> > > offset > >> > > > >>> > > commit and record produce cluster are not the same. > >> > > > >>> > > > >> > > > >>> > > Pretty sure without this _design choice_ you can skip on > >> that > >> > > > exactly > >> > > > >>> > > once already > >> > > > >>> > > > >> > > > >>> > > Best Jan > >> > > > >>> > > > >> > > > >>> > > On 16.10.2018 18:16, Ryanne Dolan wrote: > >> > > > >>> > > > > But one big obstacle in this was > >> > > > >>> > > > always that group coordination happened on the source > >> cluster. > >> > > > >>> > > > > >> > > > >>> > > > Jan, thank you for bringing up this issue with legacy > >> > > > MirrorMaker. > >> > > > >>> I > >> > > > >>> > > > totally agree with you. This is one of several problems > >> with > >> > > > >>> MirrorMaker > >> > > > >>> > > > I intend to solve in MM2, and I already have a design > and > >> > > > >>> prototype that > >> > > > >>> > > > solves this and related issues. But as you pointed out, > >> this > >> > > KIP > >> > > > is > >> > > > >>> > > > already rather complex, and I want to focus on the core > >> feature > >> > > > set > >> > > > >>> > > > rather than performance optimizations for now. If we can > >> agree > >> > > on > >> > > > >>> what > >> > > > >>> > > > MM2 looks like, it will be very easy to agree to improve > >> its > >> > > > >>> performance > >> > > > >>> > > > and reliability. > >> > > > >>> > > > > >> > > > >>> > > > That said, I look forward to your support on a > subsequent > >> KIP > >> > > > that > >> > > > >>> > > > addresses consumer coordination and rebalance issues. > Stay > >> > > tuned! > >> > > > >>> > > > > >> > > > >>> > > > Ryanne > >> > > > >>> > > > > >> > > > >>> > > > On Tue, Oct 16, 2018 at 6:58 AM Jan Filipiak < > >> > > > >>> jan.filip...@trivago.com > >> > > > >>> > > > <mailto:jan.filip...@trivago.com>> wrote: > >> > > > >>> > > > > >> > > > >>> > > > Hi, > >> > > > >>> > > > > >> > > > >>> > > > Currently MirrorMaker is usually run collocated with > >> the > >> > > > target > >> > > > >>> > > > cluster. > >> > > > >>> > > > This is all nice and good. But one big obstacle in > >> this was > >> > > > >>> > > > always that group coordination happened on the > source > >> > > > cluster. > >> > > > >>> So > >> > > > >>> > > when > >> > > > >>> > > > then network was congested, you sometimes loose > group > >> > > > >>> membership and > >> > > > >>> > > > have to rebalance and all this. > >> > > > >>> > > > > >> > > > >>> > > > So one big request from we would be the support of > >> having > >> > > > >>> > > coordination > >> > > > >>> > > > cluster != source cluster. > >> > > > >>> > > > > >> > > > >>> > > > I would generally say a LAN is better than a WAN for > >> doing > >> > > > >>> group > >> > > > >>> > > > coordinaton and there is no reason we couldn't have > a > >> group > >> > > > >>> consuming > >> > > > >>> > > > topics from a different cluster and committing > >> offsets to > >> > > > >>> another > >> > > > >>> > > > one right? > >> > > > >>> > > > > >> > > > >>> > > > Other than that. It feels like the KIP has too much > >> > > features > >> > > > >>> where > >> > > > >>> > > many > >> > > > >>> > > > of them are not really wanted and counter productive > >> but I > >> > > > >>> will just > >> > > > >>> > > > wait and see how the discussion goes. > >> > > > >>> > > > > >> > > > >>> > > > Best Jan > >> > > > >>> > > > > >> > > > >>> > > > > >> > > > >>> > > > On 15.10.2018 18:16, Ryanne Dolan wrote: > >> > > > >>> > > > > Hey y'all! > >> > > > >>> > > > > > >> > > > >>> > > > > Please take a look at KIP-382: > >> > > > >>> > > > > > >> > > > >>> > > > > > >> > > > >>> > > > > >> > > > >>> > > > >> > > > >>> > >> > > > > >> > > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-382%3A+MirrorMaker+2.0 > >> > > > >>> > > > > > >> > > > >>> > > > > Thanks for your feedback and support. > >> > > > >>> > > > > > >> > > > >>> > > > > Ryanne > >> > > > >>> > > > > > >> > > > >>> > > > > >> > > > >>> > > > >> > > > >>> > >> > > > >> > >> > > > > >> > > > >> > > > >> > > -- > >> > > Best, > >> > > Alex Mironov > >> > > > >> > > >> > > > -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Re: Delayed messages with skewed data when subscribing to many partitions

> would need a wire protocol change. > Perhaps a little consumer side logic that starts fragmenting fetch > requests if it notices that responses always have data from the > maximum number of partitions. > > Best regards, > Sönke > > [1] > https://cwiki.apache.org/confluence/display/KAFKA/KIP-349%3A+Priorities+for+Source+Topics -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Delayed messages with skewed data when subscribing to many partitions

Hi all, we've just had a case where we suspect that messages get delayed from being consumed under certain circumstances. I don't necessarily think this is a bug, hence have not opened a jira yet but wanted to discuss here - there's probably a best practice that I just don't know about. The

Re: [DISCUSS] KIP-252: Extend ACLs to allow filtering based on ip ranges and subnets

ows storing everything in one compacted topic. Any thoughts on this? Best regards, Sönke [1] https://cwiki.apache.org/confluence/display/KAFKA/KIP-290%3A+Support+for+Prefixed+ACLs On Sat, Dec 22, 2018 at 5:51 AM Colin McCabe wrote: > > Hi Sönke, > > One path forward would be to forbid

Re: [DISCUSS] KIP-252: Extend ACLs to allow filtering based on ip ranges and subnets

consider the security implications of these ACLs in a mixed version cluster quite minimal and addressable via the release notes. Best, Sönke On Sat, Mar 16, 2019 at 1:32 PM Sönke Liebau wrote: > Just a quick bump, as this has been quiet for a while again. > > On Tue, Jan 8, 2019 at 12:44

Re: [VOTE] KIP-349 Priorities for Source Topics

2019/01/28 02:26:31, n...@afshartous.com wrote: > > Hi Sönke, > > > > Thanks for taking the time to review. I’ve put KIP-349 into > hibernation. > > > > Thanks also to everyone who participated in the discussion. > > > > Best regards, > &g

Re: [VOTE] KIP-349 Priorities for Source Topics

s also to everyone who participated in the discussion. > > > > > > Best regards, > > > -- > > > Nick > > > > > > > On Jan 25, 2019, at 5:51 AM, Sönke Liebau < > soenke.lie...@opencore.com.INVALID> wrote: > > > > >

Kafka Jenkins not showing recent builds in history

Hi everybody, I looked through recent Jenkins builds for a while today and it somehow looks off to me. Both jobs [1] [2] don't show any builds that are more recent than March 19th in the "build history". Only the "last failed" and "last unsuccessful" permalinks show recent dates. Build pages can

Re: [DISCUSS] KIP-252: Extend ACLs to allow filtering based on ip ranges and subnets

lled out. We could use INVALID_REQUEST > as the error code for this situation, or maybe create a new one to be more > specific. > > best, > Colin > > > On Wed, Apr 3, 2019, at 04:58, Sönke Liebau wrote: > > All, > > > > as this thread has now been dormant for about thre

Re: [VOTE] KIP-443: Return to default segment.ms and segment.index.bytes in Streams repartition topics

t;> wangg...@gmail.com> > >> > > > > wrote: > >> > > > > > > >> > > > > > > Hello folks, > >> > > > > > > > >> > > > > > > I'd like to directly start a voting thread on this simple > KIP > >> to > >> > > change > >> > > > > > the > >> > > > > > > default override values for repartition topics: > >> > > > > > > > >> > > > > > > > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > >> > https://cwiki.apache.org/confluence/display/KAFKA/KIP-443%3A+Return+to+default+segment.ms+and+segment.index.bytes+in+Streams+repartition+topics > >> > > > > > > > >> > > > > > > The related PR can be found here as well: > >> > > > > > > https://github.com/apache/kafka/pull/6511 > >> > > > > > > > >> > > > > > > If you have any thoughts or feedbacks, they are more than > >> welcomed > >> > > as > >> > > > > > well. > >> > > > > > > > >> > > > > > > > >> > > > > > > -- Guozhang > >> > > > > > > > >> > > > > > > >> > > > > > >> > > > >> > > >> > > >> > -- > >> > -- Guozhang > >> > > > > > > -- > > -- Guozhang > > > > > -- > -- Guozhang > -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Re: Kafka Jenkins not showing recent builds in history

3, 2019 at 10:41 PM Sönke Liebau wrote: > Hi everybody, > > I looked through recent Jenkins builds for a while today and it somehow > looks off to me. > > Both jobs [1] [2] don't show any builds that are more recent than March > 19th in the "build history". >

Re: [DISCUSS] KIP-252: Extend ACLs to allow filtering based on ip ranges and subnets

rity/auth/Acl.scala#L37 [3] https://github.com/apache/kafka/blob/b4fa87cc51f0a7d640dc6ae2cc8f89f006aae652/core/src/main/scala/kafka/security/auth/Acl.scala#L66 On Fri, Apr 5, 2019 at 8:09 AM Colin McCabe wrote: > On Thu, Apr 4, 2019, at 01:52, Sönke Liebau wrote: > > Hi Colin, >

Re: Speeding up integration tests

by approximately an > > hour. This would save resources and give quicker feedback for PR builds. > > > > What are your thoughts? > > Has anyone thought about this or were there any attempts made? > > > > Best, > > Viktor > > > > > -- > Best, > Stanislav -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Re: Apache Kafka Memory Leakage???

current offset > if (isSyncCommit && isSuggest) { > commitOffset(offsets); > offsets.clear(); > } > } > } while ((msgCount > 0 || infiniteLoop) && isRunning.get()); > &g

Re: Apache Kafka Memory Leakage???

(Duration) > method. Still, I see memory issue. > Is there a way we can get on a one-one call and discuss this pls? Let me > know your availability. I can share zoom meeting link. > > Thanks, > > > > On Sat, Mar 2, 2019 at 2:15 AM Sönke Liebau > wrote: >> &g

Re: Apache Kafka Memory Leakage???

mp;& isRunning.get()); > > > *Note: *Debugger is a wrapper class that just writes the given string to a > local file using PrintStream's println() method. > > And I don't see any diff in the metrics. I still see the huge amount of > memory allocated. > > See the image attac

Re: Apache Kafka Memory Leakage???

he heap memory goes to about 60 or 70MB, and then falls > back, and the pattern continues. Max heap consumed at an instant is about > 70MB. > > Maybe there is a problem in my platform on updating the stats. I shall > convey it to my team. > > thanks so much for the help. &

Re: Apache Kafka Memory Leakage???

kafka-clients library. > > Thanks, > > > > On Mon, Mar 4, 2019 at 8:33 PM Sönke Liebau > wrote: > >> Hi Syed, >> >> and you are sure that this memory is actually allocated? I still have my >> reservations about that metric to be honest. Is there

Re: Apache Kafka Memory Leakage???

tterize the > kafka-clients library. > What the steps next to confirm its a bug in kafka-clients? And if its a > bug whats the process to get it resolved? > > Thanks, > > > > On Tue, Mar 5, 2019 at 2:43 PM Sönke Liebau > wrote: > >> Hi Syed, >> >> A

Re: Apache Kafka Memory Leakage???

ng as I can share all the details and my screen as well. > After that, we can see what to do next. If the call is not possible then > its ok but it would be great to have it. > Thanks, > > > > On Tue, Mar 5, 2019 at 4:29 PM Sönke Liebau > wrote: > >> Hi Syed, >&

Re: [DISCUSS] KIP-252: Extend ACLs to allow filtering based on ip ranges and subnets

Just a quick bump, as this has been quiet for a while again. On Tue, Jan 8, 2019 at 12:44 PM Sönke Liebau wrote: > Hi Colin, > > thanks for your response! > > in theory we could get away without any additional path changes I > think.. I am still somewhat unsure about the best

Re: [DISCUSSION] KIP-422: Add support for user/client configuration in the Kafka Admin Client

igurations for users, instead of requiring users to directly > >> talk to ZK. > >> > >> The link for this KIP is > >> following: > >> https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=97555704 > >> > >> I'd be happy to r

Re: [VOTE] KIP-349 Priorities for Source Topics

ng started with something easier. > > > Yes it does. And after 6 months of (sometimes circular) discussion I’d like > to either move towards a vote or set the status of this KIP to dormant until > if and when someone else picks up it up. > > Does anybody else have input on eithe

Some jira cleanup

All, I left a few comments on some old but still open jiras in an attempt to clean up a little bit. Since probably no one would notice these comments I thought I'd quickly list them here to give people a chance to check on them: KAFKA-217 :

Re: Kafka Jenkins not showing recent builds in history

Quick update: everything seems to be back to normal. On Mon, Apr 8, 2019 at 9:57 AM Sönke Liebau wrote: > I've opened an INFRA ticket about this [1]. Apparently, it is a known > issue that is solved by a restart of Jenkins. One will be scheduled in the > near future, I'll update the

Re: Proposal to Auto Close Inactive Tickets

Hi Addison, in general, I am totally in favor of closing unnecessary tickets! However, over the past few weeks, I have spent quite a bit of time looking at old tickets and evaluating whether those should be closed or are still relevant. For a surprising number of these tickets, I've found that

Re: Cleaning up command line tools argument parsing a little

but if you think it's > > interesting I can rebase it and we can continue with raising a KIP. > > https://github.com/viktorsomogyi/kafka/tree/kafka_shell > > > > Viktor > > > > On Tue, May 7, 2019 at 11:10 AM Sönke Liebau > > wrote: > > > >> Hi Colin,

Re: [DISCUSS] KIP-317: Transparent Data Encryption

apache.org/confluence/download/attachments/85479936/kafka_e2e-encryption_control-flow.png?version=1=1558439227551=v2 On Fri, 10 Aug 2018 at 14:05, Sönke Liebau wrote: > Hi Viktor, > > thanks for your input! We could accommodate magic headers by removing any > known fixed bytes pr

Re: Cleaning up command line tools argument parsing a little

on that, but I currently have enough other open things to look at, so I won't draw that one to myself as well for now. But I'll keep it in mind for when some time frees up :) Best regards, Sönke Colin McCabe schrieb am Di., 7. Mai 2019, 00:56: > On Mon, May 6, 2019, at 10:21, Sönke Liebau wr

Re: Cleaning up command line tools argument parsing a little

e as variants to ease the transition. > > best, > Colin > > > On Sun, May 5, 2019, at 00:54, Sönke Liebau wrote: > > Hi Colin, > > > > I totally agree! Especially the differently named bootstrap server > options > > have been annoyi

Re: Cleaning up command line tools argument parsing a little

ectly reasonable > to > > > have multiple spellings of the same arguments. I don't really see a > > > downside to keeping the aliases around indefinitely. > > > > > > Ryanne > > > > > > > > > > > > > > > On Wed, Ap

Cleaning up command line tools argument parsing a little

Hi everybody, Jason and I were recently discussing command line argument parsing on KAFKA-8131 (or rather the related pull request) [1]. Command line tools and their arguments are somewhat diverse at the moment. Most of the tools use joptsimple for argument parsing, some newer java tools use

Re: Cleaning up command line tools argument parsing a little

nd indefinitely. > > > > Ryanne > > > > > > > > > > On Wed, Apr 17, 2019, 7:07 AM Sönke Liebau > > wrote: > > > > > Hi everybody, > > > > > > Jason and I were recently discussing command line argument parsing on > > &

Re: [DISCUSS] KIP-504 - Add new Java Authorizer Interface

thorizer. If you have come across other limitations that you would like > > to see addressed in the new API, please raise these on the discussion > > thread so that we can consider those too. All suggestions and feedback > are > > welcome. > > > > Thank you, > > > > Rajini > -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Re: [DISCUSS] KIP-317: Transparent Data Encryption

scenarios. You may learn from those protocols on how they > handle keys, how long they keep them etc. to implement your E2e solution . > > > Am 08.08.2019 um 08:11 schrieb Maulin Vasavada < > maulin.vasav...@gmail.com>: > > > > Hi Sönke Liebau > > < > http

Re: [DISCUSS] KIP-317: Transparent Data Encryption

few people before I but more work into this. Best regards, Sönke On Tue, 21 May 2019 at 14:15, Sönke Liebau wrote: > Hi everybody, > > I'd like to rekindle the discussion around KIP-317. > I have reworked the KIP a little bit in order to design everything as a > pluggable implemen

Re: [VOTE] KIP-499 - Unify connection name flag for command line tool

er-groups.sh, kafka-consumer-perf-test.sh, > > kafka-verifiable-consumer.sh, kafka-verifiable-producer.sh > > > > Thanks, > > -Mitch > > > > -- > > Gwen Shapira > > Product Manager | Confluent > > 650.450.2760 | @gwenshap > > Follow us: Twitter | blog > > > -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Re: [DISCUSS] KIP-317: Transparent Data Encryption

get the KIP accepted into > Kafka. > > Thanks, > Andrew Schofield (IBM) > > On 06/08/2019, 15:46, "Sönke Liebau" > wrote: > > Hi, > > I have so far received pretty much no comments on the technical details > outlined in the KIP. While I am happy to c

Jira Cleanup

All, I left a few comments on some old but still open jiras in an attempt to clean up a little bit. Since probably no one would notice these comments I thought I'd quickly list them here to give people a chance to check on them: KAFKA-822 :

Re: Broker Interceptors

Hi Thomas, I think that idea is worth looking at. As you say, if no interceptor is configured then the performance overhead should be negligible. Basically it is then up to the user to decide if he wants tomtake the performance hit. We should make sure to think about monitoring capabilities like

Re: [jira] [Created] (KAFKA-9292) KIP-551: Expose disk read and write metrics

gt; > > -- > This message was sent by Atlassian Jira > (v8.3.4#803005) -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Re: [VOTE] KIP-409: Allow creating under-replicated topics and partitions

t; > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-409%3A+Allow+creating+under-replicated+topics+and+partitions > > > > > > Thanks > > -- Sönke Liebau Partner Tel. +49 179 7940878 OpenCore GmbH & Co. KG - Thomas-Mann-Straße 8 - 22880 Wedel - Germany

Re: Preliminary blog post about the Apache Kafka 2.4.0 release

the upcoming Apache Kafka > 2.4.0 > > release. > > Please take a look and let me know if you want to add/modify details. > > Thanks to all who contributed to this blog post. > > > > > https://blogs.apache.org/preview/kafka/?previewEntry=what-s-new-in-apache1 > > &

  1   2   >