Re: KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

2020-05-12 Thread Tom Bentley
Hi David, Thanks for the reply. >> If I understand the proposed throttling algorithm, an initial request > would > >> be allowed (possibly making K negative) and only subsequent requests > >> (before K became positive) would receive the QUOTA_VIOLATED. That would > >> mean it was still possible

Re: [DISCUSS] KIP-585: Conditional SMT

2020-05-11 Thread Tom Bentley
t.pattern: my-prefix-.* > >transforms.t1.then: t1 > >transforms.t1.then.t1.type: ExtractField$Key > >transforms.t1.then.t1.field: c1 > > >I would love to see conditional application of transforms in SMT in > Kafka 2.6. > >Let's

Re: [DISCUSS] KIP-317 - Add end-to-end data encryption functionality to Apache Kafka

2020-05-07 Thread Tom Bentley
with this in addition to the encryption/decryption keys, so I think it should still work. Unless I've overlooked something else. Cheers, Tom On Thu, May 7, 2020 at 6:04 PM Tom Bentley wrote: > Hi Rayanne, > > You raise some good points there. > > Similarly, if the whole record is encryp

Re: [DISCUSS] KIP-317 - Add end-to-end data encryption functionality to Apache Kafka

2020-05-07 Thread Tom Bentley
Hi Rayanne, You raise some good points there. Similarly, if the whole record is encrypted, it becomes impossible to do > joins, group bys etc, which just need the record key and maybe don't have > access to the encryption key. Maybe only record _values_ should be > encrypted, and maybe Kafka

Re: [DISCUSS] KIP-317 - Add end-to-end data encryption functionality to Apache Kafka

2020-05-07 Thread Tom Bentley
Hi Sönke, Replies inline 1. The functionality in this first phase could indeed be achieved with > custom serializers, that would then need to wrap the actual serializer that > is to be used. However, looking forward I intend to add functionality that > allows configuration to be configured

Re: [DISCUSS] KIP-554: Add Broker-side SCRAM Config API

2020-05-05 Thread Tom Bentley
Hi Colin, SCRAM is better than SASL PLAIN because it doesn't send the password over the wire in the clear. Presumably this property is important for some users who have chosen to use SCRAM. This proposal does send the password in the clear when setting the password. That doesn't mean it can't be

Re: [DISCUSS] KIP-589 Add API to Update Replica State in Controller

2020-05-05 Thread Tom Bentley
Sorry if I'm repeating stuff you already figured out, but I just wanted to > be more clear about this (I found it confusing too until David explained it > to me originally...) > > best, > Colin > > > On Sat, May 2, 2020, at 10:30, Tom Bentley wrote: > > Hi David, > &

Re: [DISCUSS] KIP-585: Conditional SMT

2020-05-05 Thread Tom Bentley
predicate alias to use > transforms: t2 > transforms.t2.?: has-my-prefix > transforms.t2.type:ExtractField$Key > transforms.t2.field: c1 > > > One more question. > > Do people think the negation ought to be in the predicate config or in the > referring transformati

Re: [DISCUSS] KIP-585: Conditional SMT

2020-05-04 Thread Tom Bentley
prefix > predicates.has-my-prefix.type: TopicNameMatch > predicates.has-my-prefix.negate: true > predicates.has-my-prefix.pattern: my-prefix-.* > > Excited to see how this is evolving and I think we're heading towards > something really useful for the Connect framework. > >

Re: [DISCUSS] KIP-589 Add API to Update Replica State in Controller

2020-05-02 Thread Tom Bentley
quest, then all topics > on the broker are implied. And if a topic is given with no partitions, all > partitions for that topic (on the broker) are implied. Does this make > sense? > > Thanks again! I'll update the KIP and leave a message here once it's > revised. > > David &

Re: [DISCUSS] KIP-317 - Add end-to-end data encryption functionality to Apache Kafka

2020-05-01 Thread Tom Bentley
Hi Sönke, I never looked at the original version, but what you describe in the new version makes sense to me. Here are a few things which sprang to mind while I was reading: 1. It wasn't immediately obvious why this can't be achieved using custom serializers and deserializers. 2. It would be

Re: [DISCUSS] KIP-585: Conditional SMT

2020-05-01 Thread Tom Bentley
fix-.* > transforms.t2.type: org.apache.kafka.connect.transforms.ExtractField$Key > transforms.t2.field: c1 > > The "?!" means the transform guarded by the predicate runs if the Predicate > returns false. > > I think adding a Filter SMT that uses a Predicate is a nice idea. >

Re: KIP-599: Throttle Create Topic, Create Partition and Delete Topic Operations

2020-04-30 Thread Tom Bentley
Hi David, Thanks for the KIP. If I understand the proposed throttling algorithm, an initial request would be allowed (possibly making K negative) and only subsequent requests (before K became positive) would receive the QUOTA_VIOLATED. That would mean it was still possible to block the

Re: [DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

2020-04-29 Thread Tom Bentley
gt; > > > > > > We'll probably want to at least add a note on that to the docs - > unless > > > it > > > > is there already, I've only looked for about 30 seconds.. > > > > > > > > Best regards, > > > > Sönke > > > >

Re: [DISCUSS] KIP-589 Add API to Update Replica State in Controller

2020-04-29 Thread Tom Bentley
Hi David, Thanks for the KIP! In the rejecting the alternative of having an RPC for log dir failures you say: It was also rejected to prevent "leaking" the notion of a log dir to the > public API. > I'm not quite sure I follow that argument, since we already have RPCs for changing replica log

Re: [DISCUSS] KIP-585: Conditional SMT

2020-04-28 Thread Tom Bentley
in the future, there become compatibility concerns since > SMTs are > > > pretty tightly-coupled with the worker on which they run (although > > > technically, with some classpath/plugin path shuffling, you can run > > > different versions of the out-of-t

Re: [DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

2020-04-23 Thread Tom Bentley
Hi folks, Colin wrote: > The final broker knows it can trust the principal name in the envelope > (since EnvelopeRequest requires CLUSTERACTION on CLUSTER). So it can use > that principal name for authorization (given who you are, what can you > do?) The forwarded principal name will also be

Re: [DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

2020-04-22 Thread Tom Bentley
ouldn't pass the authorization in > the > > first place. > > > > In the meantime, could you give more context on the custom Kafka > principal > > you are referring to? How does that get encoded today, and how server and > > client could both agree on

Re: [DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

2020-04-21 Thread Tom Bentley
Hi Boyang, The answer to my original question about the request principal was that the forwarding broker would authorize the request and the controller would trust the request since it was from another broker. AFAIU you added the principal purely for logging purposes. In the "EnvelopeRequest

Re: [DISCUSS] KIP-578: Add configuration to limit number of partitions

2020-04-21 Thread Tom Bentley
; complexity that I am not sure we want to undertake. > > Do you see sense in what I am saying? > > Thanks. > > On Mon, Apr 20, 2020 at 3:59 PM Tom Bentley wrote: > > > Hi Gokul, > > > > Leaving aside the question of how Kafka scales, I think the propose

Re: [DISCUSS] KIP-578: Add configuration to limit number of partitions

2020-04-20 Thread Tom Bentley
Hi Gokul, Leaving aside the question of how Kafka scales, I think the proposed solution, limiting the number of partitions in a cluster or per-broker, is a policy which ought to be addressable via the pluggable policies (e.g. create.topic.policy.class.name). Unfortunately although there's a

Re: [DISCUSS] KIP-585: Conditional SMT

2020-04-08 Thread Tom Bentley
Since no one objected I've updated the KIP with the revised way to configure this transformation. On Mon, Apr 6, 2020 at 2:52 PM Tom Bentley wrote: > To come back about a point Chris made: > > 1. Instead of the new "ConfigDef config(Map props)" method, >> what would

Re: [DISCUSS] KIP-590: Redirect Zookeeper Mutation Protocols to The Controller

2020-04-06 Thread Tom Bentley
Hi Boyang, Thanks for the KIP! When a broker proxies a request to the controller how does the authenticated principal get propagated? I think a couple of things might complicate this: 1. A PrincipalBuilder might be in use, 2. A Principal does not have to be serializable. Kind regards, Tom

Re: [DISCUSS] KIP-585: Conditional SMT

2020-04-06 Thread Tom Bentley
ke a recipe for confusion. Also problematic is that there's no use for the Config returned from Transformations.validate(). So I'm not convinced that the superficial similarity really gains anything. Kind regards, Tom On Mon, Apr 6, 2020 at 2:29 PM Tom Bentley wrote: > Hi, > > Hi al

Re: [DISCUSS] KIP-585: Conditional SMT

2020-04-06 Thread Tom Bentley
itions and determining their > precedence. > > Would love to see this feature in one or another way in Connect. > > Best, > > --Gunnar > > > > Am Do., 2. Apr. 2020 um 18:48 Uhr schrieb Tom Bentley >: > > > > Hi Chris and Sönke, > > > > Us

Re: [DISCUSS] KIP-585: Conditional SMT

2020-04-02 Thread Tom Bentley
oach of naming > conditions via separate properties, but things may get a little nasty with > literal conditions included there, especially with the possibility of > nesting. Finally, the shift in syntax here should make it easier to handle > cases like the header value matching bro

Re: [DISCUSS] KIP-585: Conditional SMT

2020-04-01 Thread Tom Bentley
ract.condition.hasMyHeader: field-value:my-value > > > Also, while writing that, I noticed that you have a version with and > without "name" for your transformation in the KIP: > > transforms.conditionalExtract.condition.hasMyHeader: has-header:my-header > and > transforms.c

Re: [DISCUSS] KIP-585: Conditional SMT

2020-04-01 Thread Tom Bentley
Does anyone have any comments, feedback or thoughts about this? Thanks, Tom On Tue, Mar 24, 2020 at 11:56 AM Tom Bentley wrote: > Hi, > > I've opened KIP-585 which is intended to provide a mechanism to > conditionally apply SMTs in Kafka Connect. I'd be grateful for any > fe

[jira] [Resolved] (KAFKA-1368) Upgrade log4j

2020-04-01 Thread Tom Bentley (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-1368?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom Bentley resolved KAFKA-1368. Resolution: Duplicate > Upgrade log4j > - > > Key

[jira] [Created] (KAFKA-9775) IllegalFormatConversionException from kafka-consumer-perf-test.sh

2020-03-27 Thread Tom Bentley (Jira)
Tom Bentley created KAFKA-9775: -- Summary: IllegalFormatConversionException from kafka-consumer-perf-test.sh Key: KAFKA-9775 URL: https://issues.apache.org/jira/browse/KAFKA-9775 Project: Kafka

[DISCUSS] KIP-585: Conditional SMT

2020-03-24 Thread Tom Bentley
Hi, I've opened KIP-585 which is intended to provide a mechanism to conditionally apply SMTs in Kafka Connect. I'd be grateful for any feedback: https://cwiki.apache.org/confluence/display/KAFKA/KIP-585%3A+Conditional+SMT Many thanks, Tom

[jira] [Resolved] (KAFKA-9692) Flaky test - kafka.admin.ReassignPartitionsClusterTest#znodeReassignmentShouldOverrideApiTriggeredReassignment

2020-03-20 Thread Tom Bentley (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-9692?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom Bentley resolved KAFKA-9692. Resolution: Done This test got deleted as part of KAFKA-8820, when much of the testing changed

Conditionally applying SMTs

2020-03-18 Thread Tom Bentley
Hi, I opened https://issues.apache.org/jira/browse/KAFKA-9673 a week or so ago sketching an idea for conditionally applying SMTs, which improve the experience for users of connectors, such as Debezium, which can send records to multiple different topics where SMTs need to be applied to for some

Re: [VOTE] 2.5.0 RC1

2020-03-10 Thread Tom Bentley
Hi David, I verified signatures, built the tagged branch and ran unit and integration tests. I found some flaky tests, as follows: https://issues.apache.org/jira/browse/KAFKA-9691 (new) https://issues.apache.org/jira/browse/KAFKA-9692 (new) https://issues.apache.org/jira/browse/KAFKA-9283

[jira] [Created] (KAFKA-9692) Flaky test - kafka.admin.ReassignPartitionsClusterTest#znodeReassignmentShouldOverrideApiTriggeredReassignment

2020-03-10 Thread Tom Bentley (Jira)
Tom Bentley created KAFKA-9692: -- Summary: Flaky test - kafka.admin.ReassignPartitionsClusterTest#znodeReassignmentShouldOverrideApiTriggeredReassignment Key: KAFKA-9692 URL: https://issues.apache.org/jira/browse

[jira] [Created] (KAFKA-9691) Flaky test kafka.admin.TopicCommandWithAdminClientTest#testDescribeUnderReplicatedPartitionsWhenReassignmentIsInProgress

2020-03-10 Thread Tom Bentley (Jira)
Tom Bentley created KAFKA-9691: -- Summary: Flaky test kafka.admin.TopicCommandWithAdminClientTest#testDescribeUnderReplicatedPartitionsWhenReassignmentIsInProgress Key: KAFKA-9691 URL: https://issues.apache.org/jira

Re: Subject: [VOTE] 2.4.1 RC0

2020-03-09 Thread Tom Bentley
+1 (non-binding) Built from source, all unit tests passed. Thanks Bill. On Mon, Mar 9, 2020 at 3:44 AM Gwen Shapira wrote: > +1 (binding) > > Verified signatures, built jars from source, quickstart passed and local > unit tests all passed. > > Thank you for the release Bill! > > Gwen Shapira

ConfigProvider notification

2020-03-09 Thread Tom Bentley
Colin asked that discussion about the future of the ConfigProvider notification mechanism happen on the list (rather than in https://issues.apache.org/jira/browse/KAFKA-9635 where I originally asked). The background that I've pieced together is that the subscribe family of methods were originally

[jira] [Created] (KAFKA-9673) Conditionally apply SMTs

2020-03-06 Thread Tom Bentley (Jira)
Tom Bentley created KAFKA-9673: -- Summary: Conditionally apply SMTs Key: KAFKA-9673 URL: https://issues.apache.org/jira/browse/KAFKA-9673 Project: Kafka Issue Type: New Feature

[jira] [Created] (KAFKA-9663) KafkaStreams.metadataForKey, queryMetadataForKey docs don't mention null

2020-03-05 Thread Tom Bentley (Jira)
Tom Bentley created KAFKA-9663: -- Summary: KafkaStreams.metadataForKey, queryMetadataForKey docs don't mention null Key: KAFKA-9663 URL: https://issues.apache.org/jira/browse/KAFKA-9663 Project: Kafka

[jira] [Created] (KAFKA-9651) Divide by zero in DefaultPartitioner

2020-03-04 Thread Tom Bentley (Jira)
Tom Bentley created KAFKA-9651: -- Summary: Divide by zero in DefaultPartitioner Key: KAFKA-9651 URL: https://issues.apache.org/jira/browse/KAFKA-9651 Project: Kafka Issue Type: Bug

[jira] [Created] (KAFKA-9650) Include human readable quantities for default config docs

2020-03-04 Thread Tom Bentley (Jira)
Tom Bentley created KAFKA-9650: -- Summary: Include human readable quantities for default config docs Key: KAFKA-9650 URL: https://issues.apache.org/jira/browse/KAFKA-9650 Project: Kafka Issue

[jira] [Created] (KAFKA-9635) Should ConfigProvider.subscribe be decrecated?

2020-03-02 Thread Tom Bentley (Jira)
Tom Bentley created KAFKA-9635: -- Summary: Should ConfigProvider.subscribe be decrecated? Key: KAFKA-9635 URL: https://issues.apache.org/jira/browse/KAFKA-9635 Project: Kafka Issue Type: Bug

[jira] [Created] (KAFKA-9634) ConfigProvider does not document thread safety

2020-03-02 Thread Tom Bentley (Jira)
Tom Bentley created KAFKA-9634: -- Summary: ConfigProvider does not document thread safety Key: KAFKA-9634 URL: https://issues.apache.org/jira/browse/KAFKA-9634 Project: Kafka Issue Type: Bug

[jira] [Created] (KAFKA-9633) ConfigProvider.close() not called

2020-03-02 Thread Tom Bentley (Jira)
Tom Bentley created KAFKA-9633: -- Summary: ConfigProvider.close() not called Key: KAFKA-9633 URL: https://issues.apache.org/jira/browse/KAFKA-9633 Project: Kafka Issue Type: Bug

Re: [ANNOUNCE] New committer: Konstantine Karantasis

2020-02-26 Thread Tom Bentley
Congratulations! On Thu, Feb 27, 2020 at 6:43 AM David Jacot wrote: > Congrats! > > Le jeu. 27 févr. 2020 à 06:58, Vahid Hashemian > a écrit : > > > Congratulations Konstantine! > > > > Regards, > > --Vahid > > > > On Wed, Feb 26, 2020 at 6:49 PM Boyang Chen > > wrote: > > > > > Congrats

[jira] [Resolved] (KAFKA-5554) Hilight config settings for particular common use cases

2020-02-25 Thread Tom Bentley (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-5554?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom Bentley resolved KAFKA-5554. Resolution: Abandoned > Hilight config settings for particular common use ca

[jira] [Resolved] (KAFKA-6359) Work for KIP-236

2020-02-25 Thread Tom Bentley (Jira)
[ https://issues.apache.org/jira/browse/KAFKA-6359?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Tom Bentley resolved KAFKA-6359. Resolution: Implemented This was addressed by KIP-455 instead. > Work for KIP-

Re: [VOTE] KIP-558: Add Connect REST API endpoints to view the topics used by connectors in Kafka Connect

2020-01-21 Thread Tom Bentley
+1 (non-binding). Thanks for the KIP Konstantine. On Sat, Jan 18, 2020 at 2:18 AM Konstantine Karantasis < konstant...@confluent.io> wrote: > Hi all, > > I'd like to open the vote on KIP-558 that had a constructive flurry of > discussions in the past few days, in order to give this KIP the

Re: [DISCUSS] KIP-558: Track the set of actively used topics by connectors in Kafka Connect

2020-01-16 Thread Tom Bentley
topic responses, especially if the result of the > /topics > > is an array of the individual responses from /connectors/{name}/topic. > The > > benefit of a single request is that the answer to "which connectors use > > topic X" can be computed using simple to

[jira] [Created] (KAFKA-9435) Replace DescribeLogDirs request/response with automated protocol

2020-01-15 Thread Tom Bentley (Jira)
Tom Bentley created KAFKA-9435: -- Summary: Replace DescribeLogDirs request/response with automated protocol Key: KAFKA-9435 URL: https://issues.apache.org/jira/browse/KAFKA-9435 Project: Kafka

[jira] [Created] (KAFKA-9434) Replace AlterReplicaLogDirs request/response with automated protocol

2020-01-15 Thread Tom Bentley (Jira)
Tom Bentley created KAFKA-9434: -- Summary: Replace AlterReplicaLogDirs request/response with automated protocol Key: KAFKA-9434 URL: https://issues.apache.org/jira/browse/KAFKA-9434 Project: Kafka

[jira] [Created] (KAFKA-9432) Replace DescribeConfigs request/response with automated protocol

2020-01-15 Thread Tom Bentley (Jira)
Tom Bentley created KAFKA-9432: -- Summary: Replace DescribeConfigs request/response with automated protocol Key: KAFKA-9432 URL: https://issues.apache.org/jira/browse/KAFKA-9432 Project: Kafka

[jira] [Created] (KAFKA-9433) Replace AlterConfigs request/response with automated protocol

2020-01-15 Thread Tom Bentley (Jira)
Tom Bentley created KAFKA-9433: -- Summary: Replace AlterConfigs request/response with automated protocol Key: KAFKA-9433 URL: https://issues.apache.org/jira/browse/KAFKA-9433 Project: Kafka

Re: [DISCUSS] KIP-558: Track the set of actively used topics by connectors in Kafka Connect

2020-01-15 Thread Tom Bentley
Hi Konstantine, Thanks for the KIP, I can see how it could be useful. a) Did you consider using a metric for this? I don't think it would satisfy all the use cases you have in mind, but you could mention it in the rejected alternatives. b) If the topic name contains the string "-connector" then

Re: [VOTE] On the new KIP-158: Kafka Connect allows source connectors to set topic settings when creating new topics

2020-01-14 Thread Tom Bentley
+1 (non-binding). Thanks for the KIP! On Tue, Jan 14, 2020 at 6:45 PM Randall Hauch wrote: > Thanks for the updated KIP, Konstantine. I have a few minor nits, but all > are around the implementation details. > > +1 (binding) > > Best regards, > > Randall > > On Mon, Jan 13, 2020 at 10:16 AM

Re: [ANNOUNCE] New Kafka PMC Members: Colin, Vahid and Manikumar

2020-01-14 Thread Tom Bentley
Congratulations! On Tue, Jan 14, 2020 at 6:57 PM Rajini Sivaram wrote: > Congratulations Colin, Vahid and Manikumar! > > Regards, > Rajini > > On Tue, Jan 14, 2020 at 6:32 PM Mickael Maison > wrote: > > > Congrats Colin, Vahid and Manikumar! > > > > On Tue, Jan 14, 2020 at 5:43 PM Ismael Juma

Re: [VOTE] KIP-518: Allow listing consumer groups per state

2020-01-13 Thread Tom Bentley
+1 (non-binding). Thanks for the KIP. On Mon, Jan 13, 2020 at 2:15 PM Dongjin Lee wrote: > +1 (non-binding) > > On Mon, Jan 13, 2020 at 7:16 PM Mickael Maison > wrote: > > > Hi all. > > > > With 2.5.0 approaching, bumping this thread once more as feedback or > > votes would be nice. > > > >

Re: [DISCUSS] KIP-158 UPDATED: Enable source connectors to create new topics with specific configs in Kafka Connect during runtime

2019-12-19 Thread Tom Bentley
Thanks Konstantine, lgtm. On Thu, Dec 19, 2019 at 5:34 PM Ryanne Dolan wrote: > Thanks for the reply Konstantine. Makes sense. > > Ryanne > > On Tue, Dec 17, 2019, 6:41 PM Konstantine Karantasis < > konstant...@confluent.io> wrote: > > > Thanks Randall and Ryanne for your comments. > > > > I'm

Re: [DISCUSS] KIP-158 UPDATED: Enable source connectors to create new topics with specific configs in Kafka Connect during runtime

2019-12-18 Thread Tom Bentley
Hi Konstantine, Thanks for the KIP. I can see this would be a useful addition to Kafka Connect. 10. The documentation of topic.creation.$alias.include says "A list of strings that represent either exact topic names or regular expressions that may match topic names." How do you know whether the

Re: [VOTE] KIP-409: Allow creating under-replicated topics and partitions

2019-12-18 Thread Tom Bentley
+1 non-binding. Thanks! On Wed, Dec 18, 2019 at 1:05 PM Sönke Liebau wrote: > Hi Mickael, > > thanks for your response! That all makes perfect sense and I cannot > give any actual use cases for where what I asked about would be useful > :) > It was more the idle thought if this might be low

Re: [VOTE] KIP-545 support automated consumer offset sync across clusters in MM 2.0

2019-12-12 Thread Tom Bentley
+1 (non-binding) On Thu, Dec 12, 2019 at 6:33 PM Andrew Schofield wrote: > +1 (non-binding) > > On 12/12/2019, 14:20, "Mickael Maison" wrote: > > +1 (binding) > Thanks for the KIP! > > On Thu, Dec 5, 2019 at 12:56 AM Ryanne Dolan > wrote: > > > > Bump. We've got 2

Re: Broker Interceptors

2019-12-06 Thread Tom Bentley
l > > On Fri, Dec 6, 2019 at 1:56 AM Tom Bentley wrote: > > > Hi, > > > > Couldn't this be done without exposing broker internals at the slightly > > higher level of AbstractRequest and AbstractResponse? Those classes are > > public. If the observer interface us

Re: Broker Interceptors

2019-12-06 Thread Tom Bentley
Hi, Couldn't this be done without exposing broker internals at the slightly higher level of AbstractRequest and AbstractResponse? Those classes are public. If the observer interface used Java default methods then adding a new request type would not break existing implementations. I'm thinking

What happened to KIP-158?

2019-11-25 Thread Tom Bentley
Hi, KIP-158 proposed a way to allow source connectors a way to configure their topics prior to creation. This was discussed, but the vote didn't get enough commiter votes at the time (though I think a couple of the +1s came from people who are now committers). Meanwhile in KIP-382 (MirrorMaker

Re: [ANNOUNCE] New committer: John Roesler

2019-11-13 Thread Tom Bentley
Congratulations John! On Wed, Nov 13, 2019 at 8:16 AM Bruno Cadonna wrote: > Congrats, John! > > Best, > Bruno > > On Tue, Nov 12, 2019 at 10:56 PM Guozhang Wang wrote: > > > > Hi Everyone, > > > > The PMC of Apache Kafka is pleased to announce a new Kafka committer, > John > > Roesler. > > >

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

2019-11-08 Thread Tom Bentley
Thanks for those insights Ying. On Thu, Nov 7, 2019 at 9:26 PM Ying Zheng wrote: > > > > > > > > Thanks, I missed that point. However, there's still a point at which the > > consumer fetches start getting served from remote storage (even if that > > point isn't as soon as the local log

Re: [ANNOUNCE] New committer: Mickael Maison

2019-11-07 Thread Tom Bentley
Congratulations Mickael! On Fri, Nov 8, 2019 at 6:41 AM Vahid Hashemian wrote: > Congrats Mickael, > > Well deserved! > > --Vahid > > On Thu, Nov 7, 2019 at 9:10 PM Maulin Vasavada > wrote: > > > Congratulations Mickael! > > > > On Thu, Nov 7, 2019 at 8:27 PM Manikumar > > wrote: > > > > >

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

2019-11-06 Thread Tom Bentley
Hi Satish, >So that means a consumer which gets behind by half an hour will find its > reads being served from remote storage. And, if I understand the proposed > algorithm, each such consumer fetch request could result in a separate > fetch request from the remote storage. I.e. there's no

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

2019-11-06 Thread Tom Bentley
Hi Ying, Because only inactive segments can be shipped to remote storage, to be able > to ship log data as soon > as possible, we will roll log segment very fast (e.g. every half hour). > So that means a consumer which gets behind by half an hour will find its reads being served from remote

Re: [VOTE] KIP-541: Create a fetch.max.bytes configuration for the broker

2019-10-25 Thread Tom Bentley
+1 nb. Thanks! On Fri, Oct 25, 2019 at 7:43 AM Ismael Juma wrote: > +1 (binding) > > On Thu, Oct 24, 2019, 4:56 PM Colin McCabe wrote: > > > Hi all, > > > > I'd like to start the vote on KIP-541: Create a fetch.max.bytes > > configuration for the broker. > > > > KIP:

Re: [DISCUSS] KIP-506: Allow setting SCRAM password via Admin interface

2019-10-22 Thread Tom Bentley
Not repeating yourself is the most efficient encoding. > > I guess the other question is, do we want an API to show what the password > is for an existing user? It seems undeniably useful. And if we provide > the ability to set the password, there is little point in not provid

Re: [DISCUSS] KIP-280: Enhanced log compaction

2019-10-22 Thread Tom Bentley
Hi Senthilnathan, In the motivation isn't it a little misleading to say "On the producer side, we clearly preserve an order for the two messages, "? IMHO, the semantics of the producer are clear that having an observed order of sending records from different producers is not sufficient to

Re: [DISCUSS] KIP-518: Allow listing consumer groups per state

2019-10-22 Thread Tom Bentley
Hi Mickael, Thanks for the KIP. The use of String to represent the desired state in the API seems less typesafe than would be ideal. Is there a reason not to use the existing ConsumerGroupState enum (even if the state is serialized as a String)? While you say that the list-of-names result from

Re: [VOTE] KIP-538: Add a metric tracking the number of open connections with a given SSL cipher type

2019-10-22 Thread Tom Bentley
+1 (non-binding). Thanks! On Tue, Oct 22, 2019 at 1:03 AM Gwen Shapira wrote: > +1 > > Thanks for leading this :) > > On Mon, Oct 21, 2019 at 3:43 PM Colin McCabe wrote: > > > > Hi all, > > > > I'd like to start the vote on KIP-538: Add a metric tracking the number > of open connections with a

Re: Daily build artifacts

2019-09-28 Thread Tom Bentley
Hi Matthias, Out of interest, is the reason simply that no one has had the inclination/time to set this up, or some overriding reason like Apache policy prohibits it (which I know it does for docker images)? Thanks, Tom On Thu, Sep 12, 2019 at 5:46 PM Matthias J. Sax wrote: > There are no

Re: [VOTE] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

2019-09-27 Thread Tom Bentley
On Thu, Sep 26, 2019 at 2:16 PM Jun Rao wrote: > [...] > 100. It would be useful to think through how to support SASL scram in the > new world. If the only broker listener is SASL scram, we need to bootstrap > the scram credential in each broker before we can start the broker. > Currently, this

Re: Status of user Admin API for quotas

2019-09-27 Thread Tom Bentley
reassignment quotas should work. We > > probably want to change that quota to actually throttle only reassignment > > traffic, not just any non-ISR traffic as it does now. Or add a different > > quota type? > > > > best, > > Colin > > > > &

Re: [DISCUSS] KIP-502: Connect SinkTask.put(...) to specify ArrayList in Signature

2019-09-25 Thread Tom Bentley
Hi Cyrus, Thanks for the KIP. In the motivation I can see how having a guaranteed iteration order might be useful for some connectors, but I couldn't see how a Connector developer would benefit from using List.get(int). Could you elaborate? Can you offer any numbers to show that the cost of

Status of user Admin API for quotas

2019-09-20 Thread Tom Bentley
Hi, I was wondering, what is the current status of efforts to add an AdminClient API for managing user quotas? My trawl of the list archives didn't turn up anything current (KIP-248 was rejected KIP-422 was discussed), but perhaps I missed something. Many thanks, Tom

Re: [VOTE] KIP-517: Add consumer metrics to observe user poll behavior

2019-09-20 Thread Tom Bentley
+1 (non-binding). On Fri, Sep 20, 2019 at 7:00 AM Maulin Vasavada wrote: > +1 (non-binding). Thanks for the KIP. > > On Thu, Sep 19, 2019 at 10:38 PM Manikumar > wrote: > > > +1 (binding), Thanks for the KIP. > > > > On Fri, Sep 20, 2019 at 12:41 AM Harsha Chintalapani > > wrote: > > > > > +1

Re: [VOTE] KIP-520: Augment Consumer.committed(partition) to allow multiple partitions

2019-09-12 Thread Tom Bentley
+1 (non-binding). Thanks! On Thu, Sep 12, 2019 at 9:58 AM M. Manna wrote: > Gushing, > > Good kip. +1 (binding) from me. > > Thanks, > MAnna > > On Thu, 12 Sep 2019 at 09:55, Bruno Cadonna wrote: > > > Guozhang, Thanks for the KIP. > > > > +1 (non-binding) > > > > Best, > > Bruno > > > > On

Re: [VOTE] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

2019-09-12 Thread Tom Bentley
+1 (non-binding) On Mon, Sep 9, 2019 at 4:28 PM Colin McCabe wrote: > Hi all, > > I'd like to start the vote for KIP-500: Replace ZooKeeper with a > Self-Managed Metadata Quorum. > > The DISCUSS thread from the mailing list is here: > >

Re: Help needed with PR #6592

2019-09-06 Thread Tom Bentley
Hi Daniyar, You'll need to edit checkstyle/import-control.xml to allow it. Cheers, Tom On Fri, Sep 6, 2019 at 3:40 PM Development wrote: > Hi, > > I’ve been maintaining this PR "KAFKA-8326: Introduce List Serde" > https://github.com/apache/kafka/pull/6592 < >

[jira] [Created] (KAFKA-8862) Misleading exception message for non-existant partition

2019-09-03 Thread Tom Bentley (Jira)
Tom Bentley created KAFKA-8862: -- Summary: Misleading exception message for non-existant partition Key: KAFKA-8862 URL: https://issues.apache.org/jira/browse/KAFKA-8862 Project: Kafka Issue Type

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

2019-08-13 Thread Tom Bentley
ve Principals for PLAINTEXT sessions or use the default ANONYMOUS > Principal. > > Thanks > > > On Mon, Aug 12, 2019 at 10:52 AM Tom Bentley wrote: > > > > Hi folks, > > > > As far as I can see the motivation for KIP-201 is still valid, and as far > > as I'm a

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

2019-08-12 Thread Tom Bentley
thought I'd check whether anyone has any more comments. So please let me know any feedback for this KIP. Many thanks, Tom On Fri, Apr 12, 2019 at 10:46 AM Tom Bentley wrote: > Hi Rajini, > > I've made a number of changes to the KIP. > > 1. I've added RequestedTopicState.r

[DISCUSS] KIP-506: Allow setting SCRAM password via Admin interface

2019-08-12 Thread Tom Bentley
Hi All, I've written KIP-506 proposing an RPC and Admin interface for setting SCRAM user passwords. I think there could be an interesting discussion over the relative merits of hashing on the broker or client. In any case I'd be grateful for any comments you may have:

Re: [VOTE] KIP-499 - Unify connection name flag for command line tool

2019-08-09 Thread Tom Bentley
+1 (non-binding). Thanks! On Fri, Aug 9, 2019 at 12:37 PM Satish Duggana wrote: > +1 (non-binding) Thanks for the KIP, so useful. > > On Fri, Aug 9, 2019 at 4:42 PM Mickael Maison > wrote: > > > +1 (non binding) > > Thanks for the KIP! > > > > On Fri, Aug 9, 2019 at 9:36 AM Andrew Schofield >

[jira] [Created] (KAFKA-8780) Set SCRAM passwords via the Admin interface

2019-08-09 Thread Tom Bentley (JIRA)
Tom Bentley created KAFKA-8780: -- Summary: Set SCRAM passwords via the Admin interface Key: KAFKA-8780 URL: https://issues.apache.org/jira/browse/KAFKA-8780 Project: Kafka Issue Type: New

Re: [DISCUSS] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

2019-08-05 Thread Tom Bentley
Hi Colin, Thanks for the KIP. Currently ZooKeeper provides a convenient notification mechanism for knowing that broker and topic configuration has changed. While KIP-500 does suggest that incremental metadata update is expected to come to clients eventually, that would seem to imply that for

Re: [DISCUSS] KIP-482: The Kafka Protocol should Support Optional Fields

2019-07-02 Thread Tom Bentley
Hi Colin, In the example given, of a FooResponse, both the optional fields have the tag 0x0001. The text says "This number must be unique within the context it appears in.". My first thought was the example tags, must be wrong. But I think I misunderstood what you meant by "the context" – I

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

2019-04-12 Thread Tom Bentley
s, it may be > good to start a new vote thread since the other one is quite old and > perhaps the KIP has changed since then. > > > On Wed, Apr 10, 2019 at 3:58 AM Tom Bentley wrote: > > > Hi Rajini, > > > > I'd be happy to do that. I'll try to get it done in the

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

2019-04-09 Thread Tom Bentley
d a while and reading through the threads, it looks like there > has been a lot of interest in it. > > Thank you, > > Rajini > > > On Wed, Jan 9, 2019 at 11:25 AM Tom Bentley wrote: > > > Hi Anna and Mickael, > > > > Anna, did you have any comments

Re: [VOTE] KIP-183 - Change PreferredReplicaLeaderElectionCommand to use AdminClient

2019-01-28 Thread Tom Bentley
look at https://cwiki.apache.org/confluence/pages/diffpagesbyversion.action?pageId=73632065=20=18 and https://github.com/apache/kafka/commit/269b65279c746bc54c611141a5a6509f9b310f11 Kind regards, Tom On Fri, 8 Sep 2017 at 16:30, Tom Bentley wrote: > Since no one has objected, I concl

Re: [DISCUSS] Kafka 2.2.0 in February 2018

2019-01-18 Thread Tom Bentley
Hi Matthias, Hopefully we can get the work for KIP 183 merged too. Thanks, Tom On Fri, 18 Jan 2019, 19:33 Matthias J. Sax Just a quick update on the release. > > > We have 22 KIP atm: > > 81, 207, 258, 289, 313, 328, 331, 339, 341, 351, 359, 361, 367, 368, > 371, 376, 377, 380, 386, 393, 394,

Re: [ANNOUNCE] New Committer: Vahid Hashemian

2019-01-16 Thread Tom Bentley
Congratulations Vahid! On Wed, Jan 16, 2019 at 10:13 AM Edoardo Comar wrote: > Bravo Vahid!!! > > Edo > > > > From: Mickael Maison > To: Users > Cc: dev > Date: 16/01/2019 08:48 > Subject:Re: [ANNOUNCE] New Committer: Vahid Hashemian > > > > Congratulations Vahid! Well

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

2019-01-09 Thread Tom Bentley
Dec 2018 at 18:11, Tom Bentley wrote: > Hi Anna, > > Firstly, let me apologise again about having missed your previous emails > about this. > > Thank you for the feedback. You raise some valid points about ambiguity. > The problem with pulling the metadata in

[jira] [Created] (KAFKA-7789) SSL-related unit tests hang when run on Fedora 29

2019-01-07 Thread Tom Bentley (JIRA)
Tom Bentley created KAFKA-7789: -- Summary: SSL-related unit tests hang when run on Fedora 29 Key: KAFKA-7789 URL: https://issues.apache.org/jira/browse/KAFKA-7789 Project: Kafka Issue Type: Bug

Re: [DISCUSS] KIP-201: Rationalising Policy interfaces

2018-12-13 Thread Tom Bentley
the policy maker is more interested in what the request is changing? Kind regards, Tom On Fri, 7 Dec 2018 at 08:41, Tom Bentley wrote: > Hi Anna and Mickael, > > Sorry for remaining silent on this for so long. I should have time to look > at this again next week. > > Kind re

<    1   2   3   4   5   6   >