Re: [DISCUSS] KIP-377: TopicCommand to use AdminClient

as far as I experienced. I'd be happy to receive some ideas/feedback on this. Cheers, Viktor On Fri, Sep 28, 2018 at 4:51 PM Viktor Somogyi-Vass wrote: > Hi All, > > I made an update to the KIP. Just in short: > Currently KafkaAdminClient.describeTopics() and > KafkaAdminCl

Re: [ANNOUNCE] New Committer: Manikumar Reddy

Congratulations Manikumar, well deserved! On Fri, 12 Oct 2018, 06:30 Andras Beni, wrote: > Congratulations, Manikumar! > > Srinivas Reddy ezt írta (időpont: 2018. okt. > 12., P 3:00): > > > Congratulations Mani. We'll deserved 👍 > > > > - > > Srinivas > > > > - Typed on tiny keys. pls ignore ty

Re: [DISCUSS] KIP-377: TopicCommand to use AdminClient

ating this. > > We should move the protocol discussion to a separate thread. Probably > also look at KIP-142 as well. > > best, > Colin > > > On Tue, Oct 9, 2018, at 07:45, Viktor Somogyi-Vass wrote: > > Hi All, > > > > Would like to bump this as the con

Re: [DISCUSS] KIP-377: TopicCommand to use AdminClient

(yet?) > > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-142%3A+Add+ListTopicsRequest+to+efficiently+list+all+the+topics+in+a+cluster > > > > There have been other discussions though. In general there are a lot of > > features that would be nic

[VOTE] KIP-377: TopicCommand to use AdminClient

Hi All, I'd like to start a vote on KIP-377: https://cwiki.apache.org/confluence/display/KAFKA/KIP-377%3A+TopicCommand+to+use+AdminClient. Summary: The KIP basically proposes to add --bootstrap-server and --command-config option to TopicsCommand and implement topic administration with AdminClient

Re: [VOTE] KIP-377: TopicCommand to use AdminClient

be phased out in the future? > > best, > Colin > > On Wed, Oct 24, 2018, at 05:47, Mickael Maison wrote: > > +1 (non-binding) > > Thanks for the KIP! > > On Wed, Oct 24, 2018 at 1:28 PM Viktor Somogyi-Vass > > wrote: > > > > > > Hi All, &g

Re: [VOTE] KIP-377: TopicCommand to use AdminClient

king on this. > > On Wed, Oct 24, 2018, 7:28 AM Viktor Somogyi-Vass > wrote: > > > Hi All, > > > > I'd like to start a vote on KIP-377: > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-377%3A+TopicCommand+to+use+AdminClient > &g

Re: [VOTE] KIP-374: Add '--help' option to all available Kafka CLI commands

+1 (non-binding) Thanks for this KIP. Viktor On Wed, Oct 31, 2018 at 2:43 PM Srinivas Reddy wrote: > Hi All, > > I would like to call for a vote on KIP-374: > https://cwiki.apache.org/confluence/x/FgSQBQ > > Summary: > Currently, the '--help' option is recognized by some Kafka commands > but n

Re: [ANNOUNCE] New Kafka PMC member: Dong Lin

Congrats Dong! :) On Tue, Aug 21, 2018 at 10:09 AM James Cheng wrote: > Congrats Dong! > > -James > > > On Aug 20, 2018, at 3:54 AM, Ismael Juma wrote: > > > > Hi everyone, > > > > Dong Lin became a committer in March 2018. Since then, he has remained > > active in the community and contributed

[VOTE] KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

Hi All, I'd like to start a vote on this KIP ( https://cwiki.apache.org/confluence/pages/viewpage.action?pageId=87298242) which aims to refactor ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer. To summarize what's the motivation: When headers were introduced by KIP-82 the Ext

Re: [DISCUSS] KIP-357: Add support to list ACLs per principal

Hi Manikumar, Implementation-wise is it just a filter over the returned ACL listing or do you plan to add new methods to the Authorizer as well? Thanks, Viktor On Fri, Aug 17, 2018 at 9:18 PM Priyank Shah wrote: > +1(non-binding) > > Thanks. > Priyank > > On 8/16/18, 6:01 AM, "Manikumar" wro

Re: [VOTE] KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

; > > ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer > > > +1 > > > > > > Thanks for the KIP! > > > > > > On Wed, Aug 22, 2018 at 2:48 PM Viktor Somogyi-Vass < > > > viktorsomo...@gmail.com> > > > wrote

Re: [VOTE] KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

e of the methods, but pretty > > bad if one doesn't. :) > > > > default T deserialize(String topic, byte[] data) { > > return deserialize(topic, null, data); > > } > > > > default T deserialize(String topic, Headers headers, byte[] data) { // >

Re: [VOTE] KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

lize(topic, data) and deserialize(topic, data). > > Implementing these methods is a trivial burden for users and it feels > like > > there's no good solution which allows both methods to have default > > implementations. > > > > Also, ack on KIP-331

Re: [DISCUSS] KIP-363: Allow performance tools to print final results to output file

Hi Attila, Thanks for the KIP, I think overall it looks good. I have three comments: 1. Would you mind adding an example? (Later on we'd need anyway for the public doc.) 2. Do you want to add any 3rd party CSV reader/writer library or will you implement that too? 3. What is the separator or is tha

Re: [VOTE] KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

> > > Hey Viktor, > > > > Good summary. I agree that option 1) seems like the simplest choice and, > as > > you note, we can always add the default implementation later. I'll leave > > Ismael to make a case for the circular forwarding approach ;) > >

Re: [VOTE] KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

ou agree, I suggest adding it to the testing > plan section. > > Ismael > > On Mon, Aug 27, 2018 at 7:57 AM Viktor Somogyi-Vass < > viktorsomo...@gmail.com> > wrote: > > > Thanks guys, I've updated my KIP with this info (so to keep solution #1). > > If

Re: [VOTE] KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

, 2018 at 10:44 AM Manikumar > > wrote: > > > > > +1 (non-binding) > > > > > > Thanks for the KIP. > > > > > > On Wed, Aug 29, 2018 at 1:41 AM Jason Gustafson > > > wrote: > > > > > > > +1 Thanks for the upda

Re: [VOTE] KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

Apologies, miscounted the binding votes but the good news is that we need only one. Cheers, Viktor On Mon, Sep 3, 2018 at 11:09 AM Viktor Somogyi-Vass wrote: > Hi All, > > I have completed my binary compatibility testing on this as well. Created > a small script & exam

Re: [VOTE] KIP-363: Allow performance tools to print final results to output file

+1 (non-binding), Thanks for the KIP! Viktor On Wed, Sep 5, 2018 at 10:14 AM Attila Sasvári wrote: > Hi All, > > I'd like to start the vote on KIP-363: > * > https://cwiki.apache.org/confluence/display/KAFKA/KIP-363%3A+Allow+performance+tools+to+print+final+results+to+output+file > < > https:/

Re: [VOTE] KIP-336: Consolidate ExtendedSerializer/Serializer and ExtendedDeserializer/Deserializer

, Viktor On Thu, Sep 6, 2018 at 11:03 PM Ismael Juma wrote: > Thanks, +1 (binding). > > On Mon, Sep 3, 2018 at 6:28 AM Viktor Somogyi-Vass < > viktorsomo...@gmail.com> > wrote: > > > Apologies, miscounted the binding votes but the good news is that we need > >

Re: [DISCUSS]: KIP-339: Create a new ModifyConfigs API

I think so, I'm +1 on this. On Sat, Sep 15, 2018 at 8:14 AM Colin McCabe wrote: > On Wed, Aug 15, 2018, at 05:04, Viktor Somogyi wrote: > > Hi, > > > > To weigh-in, I agree with Colin on the API naming, overloads shouldn't > > change behavior. I think all of the Java APIs I've used so far follow

Re: [ANNOUNCE] New Kafka PMC member: Dong Lin

t; > > > > > > wrote: > > > > > > > > > > > > > > > > > > > Congrats Dong! > > > > > > > > > > > > > > > > > > > > -Ray > > > > > > > > > > &

[DISCUSS] KIP-375: TopicCommand to use AdminClient

Hi All, I wrote up a relatively simple KIP about improving the Kafka protocol and the TopicCommand tool to support the new Java based AdminClient and hopefully to deprecate the Zookeeper side of it. I would be happy to receive some opinions about this. In general I think this would be an importan

Re: [DISCUSS] KIP-375: TopicCommand to use AdminClient

> (like SSL and such)? If the former, it seems unrelated. > > On Mon, Sep 24, 2018 at 7:25 AM Viktor Somogyi-Vass < > viktorsomo...@gmail.com> > wrote: > > > Hi All, > > > > I wrote up a relatively simple KIP about improving the Kafka protocol and > > the

Re: [DISCUSS] KIP-375: TopicCommand to use AdminClient

e sure > we're not duplicating efforts. > > Thanks > Eno > > On Mon, Sep 24, 2018 at 8:26 PM, Viktor Somogyi-Vass < > viktorsomo...@gmail.com> wrote: > > > Hi Gwen, > > > > Thanks for your feedback. It is the latter, so passing extra connection >

Re: [ANNOUNCE] New committer: Colin McCabe

Congrats Colin, well deserved! :) On Tue, Sep 25, 2018 at 10:58 AM Stanislav Kozlovski wrote: > Congrats Colin! > > On Tue, Sep 25, 2018 at 9:51 AM Edoardo Comar wrote: > > > Congratulations Colin ! > > -- > > > > Edoardo Comar > > > > IBM Event S

Re: [DISCUSS] KIP-375: TopicCommand to use AdminClient

AM Eno Thereska > > wrote: > > > > > This would be very useful. > > > Could you clarify a bit the difference to > > > https://issues.apache.org/jira/browse/KAFKA-5561 since I didn't get it > > > from > > > the JIRA notes. It&

Re: [DISCUSS] KIP-375 Kafka Clients - make Metadata#TOPIC_EXPIRY_MS configurable

Hi Pavel, May I kindly ask you to increment your KIP number? :) As it turns out I have already created one with this number a little bit earlier. Apologies for the inconvenience. Viktor On Wed, Sep 26, 2018 at 4:48 PM Pavel Moukhataev wrote: > > https://cwiki.apache.org/confluence/display/KAFK

Re: [DISCUSS] KIP-375: TopicCommand to use AdminClient

77%3A+TopicCommand+to+use+AdminClient I will also start a new discussion thread (and link this one), so please post comments there. Apologies for any confusion. Cheers, Viktor On Wed, Sep 26, 2018 at 12:26 PM Viktor Somogyi-Vass < viktorsomo...@gmail.com> wrote: > Hi Manikumar, > >

[DISCUSS] KIP-377: TopicCommand to use AdminClient

Hi All, This is the continuation of the old KIP-375 with the same title: https://lists.apache.org/thread.html/dc71d08de8cd2f082765be22c9f88bc9f8b39bb8e0929a3a4394e9da@%3Cdev.kafka.apache.org%3E The problem there was that two KIPs were created around the same time and I chose to reorganize mine a

Re: [DISCUSS] KIP-375 Kafka Clients - make Metadata#TOPIC_EXPIRY_MS configurable

Hi Pavel, I have changed my KIP's number to 377 so all sorted out. Cheers, Viktor On Wed, Sep 26, 2018 at 4:53 PM Viktor Somogyi-Vass wrote: > Hi Pavel, > > May I kindly ask you to increment your KIP number? :) > As it turns out I have already created one with this nu

Re: [DISCUSS] KIP-377: TopicCommand to use AdminClient

in > many Cloud environments where direct zookeeper access is not > available. > > > On Thu, Sep 27, 2018 at 10:34 AM Viktor Somogyi-Vass > wrote: > > > > Hi All, > > > > This is the continuation of the old KIP-375 with the sam

Re: [DISCUSS] KIP-377: TopicCommand to use AdminClient

e yet (for instance because of some replicas offline), therefore it is not possible to implement the current command's "marked for deletion" feature. To get around this I introduced some changes in the Metadata protocol. Thanks, Viktor On Fri, Sep 28, 2018 at 4:48 PM Viktor Somogyi-V

Re: [VOTE] 2.1.1 RC1

Hi, Ran the ducktapes but the streams upgrade tests failed because the dev version was not updated. This will be fixed in https://github.com/apache/kafka/pull/6217. Once it's merged I'll rerun them. Viktor On Wed, 30 Jan 2019, 22:19 Jonathan Santilli Hello, > > I have downloaded the source code

Re: [ANNOUNCE] New Committer: Bill Bejeck

Congrats Bill! :) On Thu, Feb 14, 2019 at 4:50 PM Dongjin Lee wrote: > Congrats Bill, and thank you again for your great book and contributions on > Kafka Streams! > > Best, > Dongjin > > On Thu, 14 Feb 2019 at 7:31 PM Kamal Chandraprakash < > kamal.chandraprak...@gmail.com> wrote: > > > Congrat

Re: [DISCUSSION] KIP-422: Add support for user/client configuration in the Kafka Admin Client

Hi Guys, I wanted to reject that KIP, split it up and revamp it as in the meantime there were some overlapping works I just didn't get to it due to other higher priority work. One of the splitted KIPs would have been the quota part of that and I'd be happy if that lived in this KIP if Yaodong thin

Re: [VOTE] #2 KIP-248: Create New ConfigCommand That Uses The New AdminClient

Hi Everyone, Sorry for dropping the ball on this. I'd like to discard this KIP as there were some overlapping works in the meanwhile and I think now some design decisions could be made differently. I'll try to revamp this and take the parts that are not implemented yet and compile them into smalle

Re: [ANNOUNCE] New Committer: Randall Hauch

Congrats Randall! :) On Fri, Feb 15, 2019 at 10:15 AM Satish Duggana wrote: > Congratulations Randall! > > On Fri, Feb 15, 2019 at 1:51 PM Mickael Maison > wrote: > > > > Congrats Randall! > > > > On Fri, Feb 15, 2019 at 6:37 AM James Cheng > wrote: > > > > > > Congrats, Randall! Well deserved

[DISCUSS] KIP-434: Add Replica Fetcher and Log Cleaner Count Metrics

Hi All, I'd like to start a discussion about exposing count gauge metrics for the replica fetcher and log cleaner thread counts. It isn't a long KIP and the motivation is very simple: monitoring the thread counts in these cases would help with the investigation of various issues and might help in

Re: [DISCUSS] KIP-236 Interruptible Partition Reassignment

Hi Folks, I also have a pending active work on the incremental partition reassignment stuff here: https://issues.apache.org/jira/browse/KAFKA-6794 I think it would be good to cooperate on this to make both work compatible with each other. I'll write up a KIP about this today so it'll be easier to

Re: [DISCUSS] KIP-236 Interruptible Partition Reassignment

I've published the above mentioned KIP here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-435%3A+Incremental+Partition+Reassignment Will start a discussion about it soon. On Fri, Feb 22, 2019 at 12:45 PM Viktor Somogyi-Vass < viktorsomo...@gmail.com> wrote: > Hi Folks, >

[DISCUSS] KIP-435: Incremental Partition Reassignment

Hi Folks, I've created a KIP about an improvement of the reassignment algorithm we have. It aims to enable partition-wise incremental reassignment. The motivation for this is to avoid excess load that the current replication algorithm implicitly carries as in that case there are points in the algo

Re: [DISCUSS] KIP-236 Interruptible Partition Reassignment

ition reassignment completely. This way the reassignment algorithm (whatever it is) could be reused as a whole. Did you consider this or are there any obstacles that prevents doing this? Regards, Viktor On Fri, Feb 22, 2019 at 2:24 PM Viktor Somogyi-Vass wrote: > I've published the above

Re: [DISCUSS] KIP-433: Provide client API version to authorizer

Hi Ying, I was also thinking about your KIP yesterday I second Gwen on this. There are several other authorization components that are compatible with Kafka and putting this in there means that each and every component has to implement this functionality. At the first read I thought that this func

Re: [DISCUSS] KIP-435: Incremental Partition Reassignment

t; > Thanks, > Harsha > > On Fri, Feb 22, 2019, at 5:44 AM, Viktor Somogyi-Vass wrote: > > Hi Folks, > > > > I've created a KIP about an improvement of the reassignment algorithm we > > have. It aims to enable partition-wise incremental reassignment. The >

Re: [DISCUSS] KIP-434: Add Replica Fetcher and Log Cleaner Count Metrics

the case that any threads have died > > We had discussed a similar topic about thread revival and metrics in > KIP-346. Have you had a chance to look over that discussion? Here is the > mailing discussion for that - > > http://mail-archives.apache.org/mod_mbox/kafka-dev/

Re: [DISCUSS] KIP-236 Interruptible Partition Reassignment

ere is enough interest from the community, this "Planned Future > Changes" for incremental reassignments can also be delivered in KIP-236, > otherwise, another KIP. The current PR: > https://github.com/apache/kafka/pull/6296 only focuses/addresses the > pending Reassignment C

Re: [VOTE] 2.2.0 RC0

Hi Matthias, I've noticed a minor line break issue in the upgrade docs. I've created a small PR for that: https://github.com/apache/kafka/pull/6320 Best, Viktor On Sun, Feb 24, 2019 at 10:16 PM Stephane Maarek wrote: > Hi Matthias > > Thanks for this > Running through the list of KIPs. I think

Speeding up integration tests

Hi Folks, I've been observing lately that unit tests usually take 2.5 hours to run and a very big portion of these are the core tests where a new cluster is spun up for every test. This takes most of the time. I ran a test (TopicCommandWithAdminClient with 38 test inside) through the profiler and

Re: Speeding up integration tests

Juma wrote: > It's an idea that has come up before and worth exploring eventually. > However, I'd first try to optimize the server startup/shutdown process. If > we measure where the time is going, maybe some opportunities will present > themselves. > > Ismael

Re: [DISCUSS] KIP-236 Interruptible Partition Reassignment

nses not to rollback to keep that > topic/partition online. However, if none of brokers in RAR is online, it > may make sense to rollback to OAR and remove it from > /admin/reassign_partitions, since the cluster state is already so bad, that > topic/partition is offline anyway no matter rollbac

Re: [DISCUSS] KIP-434: Add Replica Fetcher and Log Cleaner Count Metrics

consider as well. > > All in all, I think this is a great KIP and very much needed in my opinion. > I can't wait to see this roll out > > Best, > Stanislav > > On Mon, Feb 25, 2019 at 10:29 AM Viktor Somogyi-Vass < > viktorsomo...@gmail.com> wrote: > > &

Re: [DISCUSS] KIP-236 Interruptible Partition Reassignment

t; Buffer(1, 3, 4)), test_topic-11 -> > Map(replicas -> Buffer(2, 3, 5), original_replicas -> Buffer(2, 3, 4))) > Successfully submitted cancellation of reassignments. > The cancelled pending reassignments throttle was removed. > Please run --verify to have the previous rea

Re: [DISCUSS] KIP-438: Expose task, connector IDs in Connect API

I'm generally +1 on this, although have a couple of basic questions. Am I getting that right that you basically want to expose the task id from ConnectorTaskId? And if so, then I guess you'll provide the implementation too? Thanks, Viktor On Tue, Mar 5, 2019 at 6:49 PM Ryanne Dolan wrote: > Hey

Re: [VOTE] KIP-392: Allow consumers to fetch from the closest replica

+1 (non-binding) very good proposal. On Mon, Mar 25, 2019 at 7:19 PM David Arthur wrote: > +1 > > Thanks, Jason! > > On Mon, Mar 25, 2019 at 1:23 PM Eno Thereska > wrote: > > > +1 (non-binding) > > Thanks for updating the KIP and addressing my previous comments. > > > > Eno > > > > On Mon, Mar

Re: Failing test on PR

Hi Mateusz, Just write a comment that only says "retest this please" (without the quotation marks). Best, Viktor On Wed, Mar 27, 2019 at 1:02 PM Mateusz Zakarczemny wrote: > Hi, > I'm working on https://github.com/apache/kafka/pull/4807 PR. Last PR build > failed in some random place. My chang

Re: Speeding up integration tests

ry intriguing. Does it rerun the test classes that > > failed > > > or some known, marked classes? If it is the former, I can see a lot of > > > value in having that automated in our PR builds. I wonder what others > > think > > > of this > > > > > > On Th

Re: [DISCUSS] KIP-434: Add Replica Fetcher and Log Cleaner Count Metrics

Hey Folks, Since there were no discussions on this in the past two weeks I'll create a VOTE thread soon. Thanks, Viktor On Thu, Mar 14, 2019 at 7:05 PM Viktor Somogyi-Vass wrote: > Hey Stanislav, > > Sorry for the delay on this. In the meantime I realized that the dead >

[VOTE] KIP-434: Add Replica Fetcher and Log Cleaner Count Metrics

Hi All, I'd like to start a vote on KIP-434. This basically would add a metrics to count dead threads in ReplicaFetcherManager and LogCleaner to allow monitoring systems to alert based on this. The KIP link: https://cwiki.apache.org/confluence/display/KAFKA/KIP-434%3A+Add+Replica+Fetcher+and+Log+

Re: [VOTE] KIP-434: Add Replica Fetcher and Log Cleaner Count Metrics

> On Thu, Mar 28, 2019 at 8:05 AM Viktor Somogyi-Vass < > viktorsomo...@gmail.com> > wrote: > > > Hi All, > > > > I'd like to start a vote on KIP-434. > > This basically would add a metrics to count dead threads in > > ReplicaFetcherManager

[VOTE] KIP-434: Dead replica fetcher and log cleaner metrics

Hi All, I'd like to start a vote on KIP-434. This basically would add a metrics to count dead threads in ReplicaFetcherManager and LogCleaner to allow monitoring systems to alert based on this. The KIP link: https://cwiki.apache.org/confluence/display/KAFKA/KIP-434%3A+Add+Replica+Fetcher+and+Log+

Re: [VOTE] KIP-434: Dead replica fetcher and log cleaner metrics

t; > Also, would it make sense to name these metrics consistency, something like > `log-cleaner-dead-thread-count` and `replica-fetcher-dead-thread-count`? > > Thanks, > Dhruvil > > On Thu, Mar 28, 2019 at 11:27 AM Viktor Somogyi-Vass < > viktorsomo...@gmail.com> wrote:

Re: [VOTE] KIP-434: Dead replica fetcher and log cleaner metrics

(especially for the components that leverage metrics for monitoring). Thanks, Viktor On Thu, Mar 28, 2019 at 8:44 PM Viktor Somogyi-Vass wrote: > Hi Dhruvil, > > Thanks for the feedback and the vote. I fixed the typo in the KIP. > The naming is interesting though. Unfortunately kafka ov

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

Hey Harsha, I have a rather technical question to this. How do you plan to package this extension? Does this mean that Kafka will depend on HDFS? I think it'd be nice to somehow separate this off to a different package in the project so that it could be built and released separately from the main

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

ill depend on HDFS? > I think it'd be nice to somehow separate this off to a different package in > the project so that it could be built and released separately from the main > Kafka packages." > > We would like all of this code to be part of Apache Kafka . In early days >

Re: [DISCUSS] KIP-435: Incremental Partition Reassignment

t; > > > > > > > > > > > With that said, I think we may not actually need a KIP for the > > current > > > > > > proposal since it doesn't change any APIs. To make it more > > generally > > > > > > useful,

Re: [DISCUSS] KIP-435: Incremental Partition Reassignment

ght not be suitable for that. Best, Viktor On Mon, 15 Apr 2019, 18:22 Viktor Somogyi-Vass, wrote: > Hey Guys, > > I'll reply to you all in this email: > > @Jun: > 1. yes, it'd be a good idea to add this feature, I'll write this into the > KIP. I was actually thin

Re: [VOTE] KIP-434: Dead replica fetcher and log cleaner metrics

Hi Folks, This thread sunk a bit but I'd like to bump it hoping to get some feedback and/or votes. Thanks, Viktor On Thu, Mar 28, 2019 at 8:47 PM Viktor Somogyi-Vass wrote: > Sorry, the end of the message cut off. > > So I tried to be consistent with the convention in LogMana

Re: [ANNOUNCE] New Kafka PMC member: Matthias J. Sax

Congrats Matthias! On Tue, Apr 23, 2019 at 4:24 AM Becket Qin wrote: > Congrats, Matthias! > > On Sat, Apr 20, 2019 at 10:28 AM Matthias J. Sax > wrote: > > > Thank you all! > > > > -Matthias > > > > > > On 4/19/19 3:58 PM, Lei Chen wrote: > > > Congratulations Matthias! Well deserved! > > > >

Re: [ANNOUNCE] New Kafka PMC member: Sriharsh Chintalapan

Congrats Harsha! On Tue, Apr 23, 2019 at 6:13 AM Becket Qin wrote: > Congrats, Harsh! > > On Tue, Apr 23, 2019 at 5:41 AM Colin McCabe wrote: > > > Congratulations, Harsh! > > > > cheers, > > Colin > > > > > > On Thu, Apr 18, 2019, at 11:46, Jun Rao wrote: > > > Hi, Everyone, > > > > > > Srihar

Re: [VOTE] KIP-434: Dead replica fetcher and log cleaner metrics

7 PM Jason Gustafson wrote: > Hi Viktor, > > This looks good. Just one question I had is whether we may as well cover > the log dir fetchers as well. > > Thanks, > Jason > > > On Thu, Apr 25, 2019 at 7:46 AM Viktor Somogyi-Vass < > viktorsomo...@gmail.com> > w

Re: [VOTE] KIP-434: Dead replica fetcher and log cleaner metrics

Let me have a second thought, I'll just add the clientId instead to follow the convention, so it'll change DeadFetcherThreadCount but with the clientId tag. On Fri, Apr 26, 2019 at 11:29 AM Viktor Somogyi-Vass < viktorsomo...@gmail.com> wrote: > Hi Jason, > > Yea I thin

Re: [DISCUSS] KIP-455: Create an Administrative API for Replica Reassignment

Hi Colin, How would the changed kafka-reassign-partitions.sh command work? Would it automatically figure out the controller node if you pass some bootstrap nodes with --bootstrap-server or are we deferring this implementation to the users? Thanks, Viktor On Sat, Apr 20, 2019 at 1:51 AM Colin McC

Re: [DISCUSS] KIP-435: Incremental Partition Reassignment

se of reassignment for a single partition from (0, 1, 2, 3, 4) to (5, 6, 7, 8, 9) we would form the batches (0, 1) → (5, 6); (2, 3) → (7, 8) and 4 → 9 and would execute the reassignment in this order. Let me know what you think. Best, Viktor On Mon, Apr 15, 2019 at 7:01 PM Viktor Somogyi-Vass wrote: >

Re: [VOTE] KIP-434: Dead replica fetcher and log cleaner metrics

> > the older versions that are more prone to fetcher failures. Thoughts? > > > > -Jason > > > > On Fri, Apr 26, 2019 at 5:18 AM Viktor Somogyi-Vass < > > viktorsomo...@gmail.com> > > wrote: > > > > > Let me have a second thought, I

Re: [DISCUSS] KIP-455: Create an Administrative API for Replica Reassignment

Hey Colin & George, Thinking on George's points I was wondering if it's feasible to submit a big reassignment to the controller and thus Zookeeper as frequent writes are slow as the quorum has to synchronize. Perhaps it should be the responsibility of KIP-435

Re: [DISCUSS] KIP-455: Create an Administrative API for Replica Reassignment

Hi Colin, Thanks for explaining all this, it makes sense. Viktor On Sun, May 5, 2019 at 8:18 AM Colin McCabe wrote: > On Thu, May 2, 2019, at 09:35, Viktor Somogyi-Vass wrote: > > Hey Colin & George, > > > > Thinking on George's points I was wondering if it&

Re: Cleaning up command line tools argument parsing a little

On Wed, May 8, 2019 at 11:35 AM Viktor Somogyi-Vass wrote: > Hi Sönke, > > In KAFKA-1774 I have created a Kafka Shell java tool that unfortunately > didn't get much attention so far from the creators of the jira. It works > similarly to what Colin mentioned, like "kafka.

Re: Cleaning up command line tools argument parsing a little

Hi Sönke, In KAFKA-1774 I have created a Kafka Shell java tool that unfortunately didn't get much attention so far from the creators of the jira. It works similarly to what Colin mentioned, like "kafka.sh topics create -n my-topic -p 3 -r 3" or has long names like "kafka.sh topics create --name my

Re: [VOTE] KIP-455: Create an Administrative API for Replica Reassignment

+1 (non-binding) Thanks Colin, this is great stuff. Does a jira (or maybe even a PR :) ) for this exist yet? Viktor On Thu, May 9, 2019 at 7:23 AM Colin McCabe wrote: > Hi all, > > I'd like to start the vote for KIP-455: Create an Administrative API for > Replica Reassignment. I think this KI

Re: [ANNOUNCE] Apache Kafka 2.2.1

Thanks Vahid! On Tue, Jun 4, 2019 at 5:20 PM Colin McCabe wrote: > Thanks, Vahid. > > best, > Colin > > On Mon, Jun 3, 2019, at 07:23, Vahid Hashemian wrote: > > The Apache Kafka community is pleased to announce the release for Apache > > Kafka 2.2.1 > > > > This is a bugfix release for Kafka 2.

Re: [VOTE] KIP-434: Dead replica fetcher and log cleaner metrics

Hi Folks, This vote sunk a bit, I'd like to draw some attention to this again in the hope I get some feedback or votes. Thanks, Viktor On Tue, May 7, 2019 at 4:28 PM Harsha wrote: > Thanks for the kip. LGTM +1. > > -Harsha > > On Mon, Apr 29, 2019, at 8:14 AM, Vikto

Re: [VOTE] KIP-434: Dead replica fetcher and log cleaner metrics

; wrote: > > > > > Thanks Viktor, proposed metrics are really useful to monitor > > replication > > > status on brokers. > > > > > > +1 (non-binding) > > > > > > On Thu, Jun 6, 2019 at 2:05 AM Colin McCabe > > wro

Re: [DISCUSS] KIP-435: Incremental Partition Reassignment

Hi All, I have added another improvement to this, which is to limit the parallel leader movements. I think I'll soon (maybe late this week or early next) start a vote on this too if there are no additional feedback. Thanks, Viktor On Mon, Apr 29, 2019 at 1:26 PM Viktor Somogyi-Vass

Re: [DISCUSS] KIP-435: Incremental Partition Reassignment

'll be able to > get it done soonish. > > best, > Colin > > > On Tue, Jun 25, 2019, at 08:09, Viktor Somogyi-Vass wrote: > > Hi All, > > > > I have added another improvement to this, which is to limit the parallel > > leader movements. I think I

Re: [DISCUSS] KIP-373: Allow users to create delegation tokens for other users

Hi Folks, I took over this issue from Manikumar. Recently another motivation have been raised in Spark for this (SPARK-28173) and I think it'd be great to continue this task. I updated the KIP and will wait for a few days to get some feedback then proceed for the vote. Thanks, Viktor On Tue, Dec

Re: [DISCUSS] KIP-435: Incremental Partition Reassignment

want a way of knowing what target replicas are currently > being worked on. So maybe we'll have to add a field to the structures > returned by listPartitionReassignments. > > best, > Colin > > > On Wed, Jun 26, 2019, at 06:20, Viktor Somogyi-Vass wrote: > > Hey Colin, &g

[DISCUSS] KIP-435: Internal Partition Reassignment Batching

Hi All, I've renamed my KIP as its name was a bit confusing so we'll continue it in this thread. The previous thread for record: https://lists.apache.org/thread.html/0e97e30271f80540d4da1947bba94832639767e511a87bb2ba530fe7@%3Cdev.kafka.apache.org%3E A short summary of the KIP: In case of a vast p

Re: [DISCUSS] KIP-455: Create an Administrative API for Replica Reassignment

Hey Colin, So in my understanding this is how the controller handles a reassignment in a simple scenario: 1. When an AlterPartitionReassignmentRequest arrives it updates the partitions' ZK data in /brokers/topics/[topic]/partitions/[partitionId]/state with targetReplicas 2. Sends out LeaderAndIsr

Re: [DISCUSS] KIP-435: Internal Partition Reassignment Batching

e small nits: > - could we have a section in the KIP where we explicitly define what each > config does? This can be inferred from the KIP as is but requires careful > reading, whereas some developers might want to skim through the change to > get a quick sense. It also improves read

Re: [DISCUSS] KIP-435: Internal Partition Reassignment Batching

rebalance inside the broker would do as well? > > Then again, the "reassignment.parallel.leader.movements" config has a bit > of a different meaning when used in the context of a real reassignment > (where data moves from the leader to N more replicas) versus in the > preferr

Re: [DISCUSS] KIP-373: Allow users to create delegation tokens for other users

Hi Folks, I'm starting a vote on this. Viktor On Thu, Jun 27, 2019 at 12:02 PM Viktor Somogyi-Vass < viktorsomo...@gmail.com> wrote: > Hi Folks, > > I took over this issue from Manikumar. Recently another motivation have > been raised in Spark for this (SPARK-28173) an

[VOTE] KIP-373: Allow users to create delegation tokens for other users

Hi All, I'd like to start a vote on this KIP. https://cwiki.apache.org/confluence/display/KAFKA/KIP-373%3A+Allow+users+to+create+delegation+tokens+for+other+users To summarize it: the proposed feature would allow users (usually superusers) to create delegation tokens for other users. This is espe

Re: [DISCUSS] KIP-500: Replace ZooKeeper with a Self-Managed Metadata Quorum

Hey Colin, I think this is a long-awaited KIP, thanks for driving it. I'm excited to see this in Kafka once. I collected my questions (and I accept the "TBD" answer as they might be a bit deep for this high level :) ). 1.) Are there any specific reasons for the Controller just periodically persist

Re: [VOTE] KIP-373: Allow users to create delegation tokens for other users

Hi All, Bumping this, I'd be happy to get some additional feedback and/or votes. Thanks, Viktor On Wed, Jul 31, 2019 at 11:04 AM Viktor Somogyi-Vass < viktorsomo...@gmail.com> wrote: > Hi All, > > I'd like to start a vote on this KIP. > > https://cwiki.apache.o

Re: Alternative of poll(0) without pulling records

Hey Jungtaek, Thanks for your interest, sometimes I also think such an API would be a good thing. I don't see any strong reasons neither in KIP-288 nor in KIP-266 why such an API shouldn't be created, so go ahead with it, although you'll need to create a short KIP for this as the KafkaConsumer cla

Re: [DISCUSS] KIP-373: Allow users to create delegation tokens for other users

e "User". What do you think? > > > Thanks, > Manikumar > > > On Wed, Jul 31, 2019 at 2:26 PM Viktor Somogyi-Vass < > viktorsomo...@gmail.com> > wrote: > > > Hi Folks, > > > > I'm starting a vote on this. > > > > V

Re: [DISCUSS] KIP-435: Internal Partition Reassignment Batching

so have simpler life. Let me know what you think. Viktor On Thu, Jul 11, 2019 at 7:16 PM Viktor Somogyi-Vass wrote: > Hi Stan, > > I meant the following (maybe rare) scenario - we have replicas [1, 2, 3] on > a lot of partitions and the user runs a massive rebalance to change the

Re: [VOTE] KIP-497: Add inter-broker API to alter ISR

Hey Jason, +1 from me too. One note though: since it's a new protocol we could perhaps rename CurrentZkVersion to something like "IsrEpoch" or "IsrVersion". I think that'd reflect its purpose better. Best, Viktor On Wed, Aug 7, 2019 at 8:37 PM Jason Gustafson wrote: > Hi All, > > I'd like to s

  1   2   3   >