Re: Kafka Mailing List ?

2022-02-17 Thread Divij Vaidya
tivity and I think the mailing list will probably be more useful for most purposes. Hope this helps! Regards, Divij Vaidya On Thu, Feb 17, 2022 at 5:59 PM Amer, Jeremy wrote: > Hi! > > > > Just reading „Kafka:the definitive Guide”, and saw that there is a mailing >

Re: [DISCUSS] KIP-826: Define platforms supported

2022-03-30 Thread Divij Vaidya
for providing an acceptable (baselined against other platforms) level of performance that works for the majority of use cases. Regards, Divij Vaidya On Wed, Mar 30, 2022 at 1:00 PM Mickael Maison wrote: > Hi, > > I recently started a thread > (https://lists.apach

Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API

2022-04-08 Thread Divij Vaidya
this KIP is approved. Regards, Divij Vaidya On Fri, Apr 8, 2022 at 11:35 AM Mickael Maison wrote: > Hi Divij, > > Thanks for taking a look! > > 1. In order to retrieve the sizes, the plan is to use getTotalSpace() > and getUsableSpace() from java.nio.file.FileStore. The imp

Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API

2022-04-07 Thread Divij Vaidya
criteria for this change. We want to avoid regression. Regards, Divij Vaidya On Thu, Apr 7, 2022 at 11:17 AM Mickael Maison wrote: > Hi, > > I wrote a small KIP to expose the total and usable space of logdirs > via the DescribeLogDirs API: > > https://cwiki.apache.org/conflu

Requesting permission for JIRA/Wiki

2022-04-08 Thread Divij Vaidya
I would like to open a PR for https://issues.apache.org/jira/browse/KAFKA-13530 Please give me permission to: 1. create and assign JIRA items 2. create new cwiki JIRA username: divijvaidya cwiki username: divijvaidya Thank you. Regards, Divij Vaidya

Re: Requesting permission for JIRA/Wiki

2022-04-08 Thread Divij Vaidya
Correction: I would like to open a PR for https://issues.apache.org/jira/browse/KAFKA-12319 and not https://issues.apache.org/jira/browse/KAFKA-13530 as stated earlier. Divij Vaidya On Sat, Apr 9, 2022 at 12:19 AM Divij Vaidya wrote: > I would like to open a PR for >

Re: Complete Kafka replication protocol description

2023-09-11 Thread Divij Vaidya
for adding the TLA+ spec as part of the community owned project itself. Right now, things are a bit busy on my end, but I am looking forward to exploring what you shared above in the coming weeks (perhaps a first review by end of sept). Thank you again for starting this conversation. -- Divij Vaidya

Re: [DISCUSS] KIP-977: Partition-Level Throughput Metrics

2023-09-11 Thread Divij Vaidya
please quantify (with an experiment) the expected perf impact of adding the partition dimension? This could be done as part of "test plan" section and would serve as a data point for users to understand the potential impact if they decide to turn it on. -- Divij Vaidya On Sat, Sep 9, 2023

Re: [VOTE] KIP-942: Add Power(ppc64le) support

2023-09-14 Thread Divij Vaidya
uite 5 times won't be a burden to the CI > > infrastructure. The advantage we get is that, unlike nightly builds > > which have a chance of being ignored, branch builds are actively > > monitored by folks in the community. Hence, we will be able to add > > this new suite w

Re: FYI - CI failures due to Apache Infra (Issue with creating launcher for agent)

2023-08-31 Thread Divij Vaidya
This should be fixed now. Please comment on https://issues.apache.org/jira/browse/INFRA-24927 if you find a case where you still hit this problem. -- Divij Vaidya On Mon, Aug 28, 2023 at 12:05 PM Luke Chen wrote: > > Thanks for the info, Divij! > > Luke > > On Mon, Aug 28, 202

Re: [DISCUSS] Apache Kafka 3.5.2 release

2023-10-17 Thread Divij Vaidya
Thank you for volunteering Luke. -- Divij Vaidya On Tue, Oct 17, 2023 at 3:26 PM Bill Bejeck wrote: > Thanks for driving the release, Luke. > > +1 > -Bill > > On Tue, Oct 17, 2023 at 5:05 AM Satish Duggana > wrote: > > > Thanks Luke for volunteering for 3.5.2 r

Re: [ANNOUNCE] New Kafka PMC Member: Satish Duggana

2023-10-27 Thread Divij Vaidya
Congratulations Satish! And thank you for your contributions so far. -- Divij Vaidya On Fri, Oct 27, 2023 at 5:18 PM Lucas Brutschy wrote: > Congrats! > > On Fri, Oct 27, 2023 at 5:06 PM Manikumar > wrote: > > > > Congrats! > > > > On Fri, Oct

Re: [DISCUSS] KIP-977: Partition-Level Throughput Metrics

2023-11-01 Thread Divij Vaidya
as well) 4. what happens when there are contradictory regEx rules, e.g. a topic defined in high as well as low. It is generally solved by defining precedence. In our case, we can choose that high has more precedence than low. What do you think? -- Divij Vaidya On Wed, Nov 1, 2023 at 2:07 PM Qichao Chu wro

Re: Request for Removal From the Mail Thread

2023-11-03 Thread Divij Vaidya
Hey Dasun Subscribe/unsubscribe to the mailing list is self service. You can find the details at https://kafka.apache.org/contact -- Divij Vaidya On Fri, Nov 3, 2023 at 7:37 PM Dasun Nirmitha wrote: > Hello Apache Kafka Mail Thread, > Thanks for having me in your mail thread for a

Re: [DISCUSS] KIP-977: Partition-Level Throughput Metrics

2023-11-07 Thread Divij Vaidya
time to think about the implications of current approach towards extendibility to other dimensions in future (such as e.g. client language instead of partition). I will get back to you before the end of this week. -- Divij Vaidya On Tue, Nov 7, 2023 at 4:05 PM Qichao Chu wrote: > Hi Di

Re: [DISCUSS] KIP-977: Partition-Level Throughput Metrics

2023-10-30 Thread Divij Vaidya
am happy with having a Kafka level dynamic config that we can use to filter our metric/dimensionality since we have a precedence at KIP-544. Hence, my suggestion to push this filtering to metric library can be ignored. -- Divij Vaidya On Sat, Oct 28, 2023 at 11:37 AM Qichao Chu wrote:

Re: [VOTE] 3.6.0 RC0

2023-09-19 Thread Divij Vaidya
e as part of tiered storage 3. verified that public keys for RM are available at https://keys.openpgp.org/ 4. verified that public keys for RM are available at https://people.apache.org/keys/committer/ -- Divij Vaidya On Tue, Sep 19, 2023 at 12:41 PM Sagar wrote: > > Hey Satish, > >

Re: Apache Kafka 3.6.0 release

2023-09-21 Thread Divij Vaidya
Hey Satish I filed a PR to fix the website formatting bug in 3.6 documentation - https://github.com/apache/kafka/pull/14419 Please take a look when you get a chance. -- Divij Vaidya On Tue, Sep 19, 2023 at 5:36 PM Chris Egerton wrote: > > Hi Satish, > > I think this qualifies

Re: [ANNOUNCE] New Kafka PMC Member: Justine Olshan

2023-09-23 Thread Divij Vaidya
Congratulations Justine! On Sat 23. Sep 2023 at 07:06, Chris Egerton wrote: > Congrats Justine! > On Fri, Sep 22, 2023, 20:47 Guozhang Wang > wrote: > > > Congratulations! > > > > On Fri, Sep 22, 2023 at 8:44 PM Tzu-Li (Gordon) Tai > > > wrote: > > > > > > Congratulations Justine! > > > > > >

Re: [ANNOUNCE] New committer: Yash Mayya

2023-09-21 Thread Divij Vaidya
Congratulations Yash! Divij Vaidya On Thu, Sep 21, 2023 at 6:18 PM Sagar wrote: > > Congrats Yash ! > On Thu, 21 Sep 2023 at 9:38 PM, Ashwin wrote: > > > Awesome ! Congratulations Yash !! > > > > On Thu, Sep 21, 2023 at 9:25 PM Edoardo Comar > > w

Re: Apache Kafka 3.6.0 release

2023-09-22 Thread Divij Vaidya
. -- Divij Vaidya On Fri, Sep 22, 2023 at 1:26 AM Satish Duggana wrote: > > Thanks Divij for raising a PR for doc formatting issue. > > On Thu, 21 Sep, 2023, 2:22 PM Divij Vaidya, wrote: > > > Hey Satish > > > > I filed a PR to fix the website formatting bug

Re: [kafka-clients] [VOTE] 3.6.0 RC1

2023-09-20 Thread Divij Vaidya
Hey Satish My comments about documentation misses from RC0 vote thread [1] are still not addressed (such as missing metric documentation, formatting problems etc). Could you please mention why we shouldn't consider them as blockers to make RC1 as the final release? [1]

Re: [VOTE] 3.6.0 RC2

2023-10-02 Thread Divij Vaidya
but we should add it as a fast follow-up. 2. Metric added in https://github.com/apache/kafka/commit/a900794ace4dcf1f9dadee27fbd8b63979532a18 isn't available in documentation (cc: David). I don't consider this as a release blocker but we should add it as a fast follow-up. -- Divij Vaidya On Mon

Re: Apache Kafka 3.6.0 release

2023-09-27 Thread Divij Vaidya
Hey team I need help in determining whether https://github.com/apache/kafka/pull/14457 is a release blocker bug or not. If someone is familiar with replication protocol (on the log diverange and reconciliation process), please add your comments on the PR. -- Divij Vaidya On Wed, Sep 27, 2023

Re: Apache Kafka 3.6.0 release

2023-09-27 Thread Divij Vaidya
Ismael, Thank you for checking. Multiple other folks have validated after I left the comment here that it doesn't impact log truncation and hence won't lead to data loss. I agree that it's not a blocker. (ref: https://github.com/apache/kafka/pull/14457) -- Divij Vaidya On Wed, Sep 27, 2023 at 8

Re: Apache Kafka 3.6.0 release

2023-09-27 Thread Divij Vaidya
it here as an FYI. -- Divij Vaidya On Fri, Sep 22, 2023 at 11:16 AM Divij Vaidya wrote: > > Found a bug while testing TS feature in 3.6 - > https://issues.apache.org/jira/browse/KAFKA-15481 > > I don't consider it as a blocker for release since it's a concurrency > bug that s

Re: [kafka-clients] [VOTE] 3.6.0 RC1

2023-09-25 Thread Divij Vaidya
could post the results of system tests before I vote on this. I am particularly interested in looking at producer/consumer performance results since we have quite a few changes in this release. What is the plan on the system tests? -- Divij Vaidya On Mon, Sep 25, 2023 at 9:10 AM Luke Chen wrote: >

Re: [kafka-clients] [VOTE] 3.6.0 RC1

2023-09-25 Thread Divij Vaidya
Correction: posted the wrong JIRA in my previous email. Instead of https://issues.apache.org/jira/browse/KAFKA-15001, please consider this https://issues.apache.org/jira/browse/KAFKA-15487 -- Divij Vaidya On Mon, Sep 25, 2023 at 10:04 AM Divij Vaidya wrote: > > Hi Satish > >

Re: [DISCUSS] 3.5.2 Release

2023-10-12 Thread Divij Vaidya
ational stability-wise, otherwise). For 3.5.2, I think there are sufficient pending (including some CVE fixes) to start a bug fix release. We just need a volunteer for the release manager. -- Divij Vaidya On Thu, Oct 12, 2023 at 9:57 AM Levani Kokhreidze wrote: > Hello, > > KAFKA-1557

[DISCUSS] Change website "blog" to "news"

2023-10-24 Thread Divij Vaidya
es on PMC). Later, we can work towards adding RSS feed to enable "subscription" to this news and add additional filters such as CVE clarifications etc. What do you think? -- Divij Vaidya

[DISCUSS] How to detect (and prevent) complex bugs in Kafka?

2023-10-24 Thread Divij Vaidya
ache.org/jira/browse/KAFKA-15653 [2] https://lists.apache.org/thread/pfrkk0yb394l5qp8h5mv9vwthx15084j -- Divij Vaidya

Re: [VOTE] KIP-942: Add Power(ppc64le) support

2023-08-28 Thread Divij Vaidya
get is that, unlike nightly builds which have a chance of being ignored, branch builds are actively monitored by folks in the community. Hence, we will be able to add this new suite without adding a new routine in the maintenance. -- Divij Vaidya On Fri, Aug 25, 2023 at 6:49 PM Colin McCabe

FYI - CI failures due to Apache Infra (Issue with creating launcher for agent)

2023-08-28 Thread Divij Vaidya
hile, the workaround is to restart the tests. Cheers! -- Divij Vaidya

Re: [VOTE] KIP-942: Add Power(ppc64le) support

2023-08-14 Thread Divij Vaidya
+1 (binding) -- Divij Vaidya On Wed, Jul 26, 2023 at 9:04 AM Vaibhav Nazare wrote: > > I'd like to call a vote on KIP-942

Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API

2022-04-19 Thread Divij Vaidya
wondering whether it is a problem with underlying storage not providing space information or something else. Divij Vaidya On Fri, Apr 15, 2022 at 3:40 PM Mickael Maison wrote: > Hi Luke, > > 7. I've updated the KIP to clarify these sizes are in bytes. > > Thanks, > Mickael

Re: tiered storage - remote data to topic binding

2023-11-08 Thread Divij Vaidya
? If not, RemoteLogSegmentMetadata also contains a map, CustomMetadata which is opaque to the broker and is populated by RemoteLogMetadataManager. You can choose to feed any attributes you require into this custom metadata and read it in RSM. Does this answer your question? -- Divij Vaidya On Wed, Nov 8, 2023

Re: [DISCUSS] Should we continue to merge without a green build? No!

2023-11-13 Thread Divij Vaidya
reviewers in allocating their bandwidth effectively. -- Divij Vaidya On Sun, Nov 12, 2023 at 2:58 AM Justine Olshan wrote: > I will say that I have also seen tests that seem to be more flaky > intermittently. It may be ok for some time and suddenly the CI is > overloaded and we see is

Re: [DISCUSS] KIP-977: Partition-Level Throughput Metrics

2023-11-13 Thread Divij Vaidya
Thank you for updating the KIP Qichao. I don't have any more questions or suggestions. Looks good to move forward from my perspective. -- Divij Vaidya On Fri, Nov 10, 2023 at 2:25 PM Qichao Chu wrote: > Thank you again for the nice suggestions, Jorge! > I will wait for Divij's re

Re: [DISCUSS] KIP-838 Simulate batching and compression

2022-05-16 Thread Divij Vaidya
ecompression cost by the consumers etc. 3\ Can you please add the algorithm that would be used to determine whether batching is recommended? Divij Vaidya On Mon, May 16, 2022 at 8:42 AM Sergio Daniel Troiano wrote: > Hey guys! > > I would like to start an early discussion on this: &g

Re: [VOTE] KIP-831: Add metric for log recovery progress

2022-05-16 Thread Divij Vaidya
+1 (non-binding) Divij Vaidya On Mon, May 16, 2022 at 10:12 AM Luke Chen wrote: > Hi all, > > I'd like to start a vote on KIP to expose metrics for log recovery > progress. These metrics would let the admins have a way to monitor the log > recovery progress. > > Deta

Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API

2022-05-16 Thread Divij Vaidya
Thanks for addressing my comments Mickael. No more comments/suggestions from my side. LGTM. Divij Vaidya On Tue, May 10, 2022 at 6:10 PM Mickael Maison wrote: > Hi Colin, > > Thanks for the suggestion. > > I guess there are pros and cons with both methods. In my mind

Re: [kafka-clients] Re: [VOTE] 3.2.0 RC1

2022-05-16 Thread Divij Vaidya
Perhaps, we want to merge in https://github.com/apache/kafka/pull/12164 before this release? Divij Vaidya On Mon, May 16, 2022 at 11:30 AM Bruno Cadonna wrote: > Mickael, > > Thanks a lot! > > Totally, missed that! > > Best, > Bruno > > On 16.05.22 10:13, Mick

Re: [VOTE] KIP-827: Expose logdirs total and usable space via Kafka API

2022-05-18 Thread Divij Vaidya
+1 non binding. Divij Vaidya On Tue, May 17, 2022 at 6:16 PM Igor Soarez wrote: > Thanks for this KIP Mickael. > > +1 non binding > > -- > Igor > > On Tue, May 17, 2022, at 2:48 PM, Luke Chen wrote: > > Hi Mickael, > > > > +1 (binding) fro

Re: CI on Kafka seems to be failing even on trunk branch

2022-05-16 Thread Divij Vaidya
don't think they are relevant, how should I > proceed? > > Can I still request for reviews? > > Kind regards > -- Divij Vaidya

Re: virtual KIP meeting for KIP-405

2022-07-07 Thread Divij Vaidya
Hello Are we still having a regular virtual meeting to discuss the progress and outstanding issues? If yes, can you please add me to it (email address: divijvaidy...@gmail.com). Thank you. On 2020/08/20 22:19:57 Jun Rao wrote: > Hi, everyone, > > We plan to have weekly virtual meetings for

[DISCUSS] KIP-852 Optimize calculation of size for log in remote tier

2022-07-01 Thread Divij Vaidya
! -- Divij Vaidya Sr. Software Engineer Amazon

Re: [DISCUSS] Apache Kafka 3.3.0 Release

2022-06-16 Thread Divij Vaidya
elease process? Regards, Divij Vaidya On Wed, Jun 15, 2022 at 11:10 PM José Armando García Sancio wrote: > Hi all, > > This is a friendly reminder that the KIP freeze date is today, June 15th, > 2022. > > The feature freeze date is July 6th, 2022. > > Thanks, > -José >

Re: [DISCUSS] Should we automatically close stale PRs?

2022-06-16 Thread Divij Vaidya
FWIW, I think the "triaged" label is a great idea for reducing the workload on committers. It would perhaps help in reducing our open PR count and increase the velocity of contributions to the project. -- Divij Vaidya On Wed, Jun 8, 2022 at 3:49 PM Viktor Somogyi-Vass wrote: &g

Matomo site Id required for Apache Kafka

2022-07-19 Thread Divij Vaidya
for making the website GDPR compliant is https://issues.apache.org/jira/browse/KAFKA-13868 Thank you. Regards, Divij Vaidya

What are best practices for hosting content on project website?

2022-07-21 Thread Divij Vaidya
ailing list <https://lists.apache.org/thread/mtvwog7md956t5q79o41p99v6qqon5qm> to get some answers for the same. Regards, Divij Vaidya

Re: [DISCUSS] KIP-852 Optimize calculation of size for log in remote tier

2022-07-13 Thread Divij Vaidya
dle expired remote log segments (which should be periodic). Does that address your concern? Divij Vaidya On Tue, Jul 12, 2022 at 11:01 AM Luke Chen wrote: > Hi Divij, > > Thanks for the KIP! > > I think it makes sense to delegate the responsibility of calculation

Re: [DISCUSS] Apache Kafka 3.3.0 Release

2022-07-13 Thread Divij Vaidya
ulation for edge cases - https://github.com/apache/kafka/pull/12184 2. Fix resource leaks - https://github.com/apache/kafka/pull/12228 And the complete list would be at: https://github.com/search?q=is%3Aopen+is%3Apr+author%3Adivijvaidya+is%3Apr+repo%3Aapache%2Fkafka+created%3A2022-04-01..2022-07-30=I

[DISCUSS] Website changes required for Apache projects

2022-07-13 Thread Divij Vaidya
ne with front end experience help us with this one?* Note that we need to make these changes by July 22nd and hence your immediate attention would be greatly appreciated. Cheers! -- Divij Vaidya

Re: [DISCUSS] Website changes required for Apache projects

2022-07-27 Thread Divij Vaidya
org/faq/committers.html>, please feel free to create a new ticket. Thanks everyone for chiming in on this discussion thread. -- Divij Vaidya On Fri, Jul 22, 2022 at 5:08 PM Mickael Maison wrote: > Hi, > > Don't get me wrong, the videos are great and it's definitively the > type

Re: [DISCUSS]: Including TLA+ in the repo

2022-07-26 Thread Divij Vaidya
in TLA+ in our integration tests so that we can also have proof of correctness for the implementation. Do you have any thoughts on this? -- Divij Vaidya On Tue, Jul 26, 2022 at 5:21 PM Haruki Okada wrote: > Hi. > > That also sounds good to me. > > TLA+ spec helps us to understa

Re: [DISCUSS] Website changes required for Apache projects

2022-07-21 Thread Divij Vaidya
still an open question about how we decide what content gets added and what doesn't. I would propose that the model should be the same as accepting code changes i.e. it goes through a community review requiring votes committers/PMC members. Regards, Divij Vaidya On Thu, Jul 21, 2022 at 3:57 A

Hosting videos on project website

2022-07-21 Thread Divij Vaidya
discussion about hosting third party content: https://lists.apache.org/thread/8drsbn0hgdhq4g1qgvm9g8pb5t4x42px -- Divij Vaidya

Re: [DISCUSS] Website changes required for Apache projects

2022-07-19 Thread Divij Vaidya
ke a placeholder image and use JS to trigger playback after the user clicks. I would suggest going with option#1 right now due to time constraints and create a ticket to do (more user friendly) option#2 in the future.* What do you think?* -- Divij Vaidya On Wed, Jul 13, 2022 at 5:10 PM Div

Re: [DISCUSS] KIP-827: Expose logdirs total and usable space via Kafka API

2022-05-03 Thread Divij Vaidya
nguish between situation 1 and situation 2 from code perspective and hence, we will send UNKNOWN in both cases? Divij Vaidya On Mon, May 2, 2022 at 6:24 PM Mickael Maison wrote: > Hi Divij, > > The new fields default to -1 in the protocol too. So in case a broker > hits an Exception

Re: [DISCUSS] KIP-660: Pluggable ReplicaPlacer

2022-05-03 Thread Divij Vaidya
/getter in future but it wouldn't require any code modification to the custom implementations of ReplicaPlacer. Divij Vaidya On Mon, May 2, 2022 at 6:28 PM Mickael Maison wrote: > Hi, > > If there are no further comments, I'll start a vote in the next few days. > > Thanks, > Micka

Last sprint to finish line: Replace EasyMock/Powermock with Mockito

2022-08-01 Thread Divij Vaidya
width from a committer for this week to help us reach the finish line?* Current pending PR requests: 1. https://github.com/apache/kafka/pull/12459 2. https://github.com/apache/kafka/pull/12465 3. https://github.com/apache/kafka/pull/12418 Regards, Divij Vaidya

Re: Kafka/Azul swag

2022-08-21 Thread Divij Vaidya
Hello Geertjan You are on the right track by approaching Apache's trademarks groups. You would find the process to request for usage of Apache Kafka logo on non-software merchandise at https://www.apache.org/foundation/marks/merchandise -- Divij Vaidya On Wed, Aug 17, 2022 at 9:51 AM Geertjan

Re: [DISCUSS] KIP-872: Add Serializer#serializeToByteBuffer() to reduce memory copying

2022-09-28 Thread Divij Vaidya
doesn't have to guess about memory lifetime of the objects they provided to the opaque library. What do you think? -- Divij Vaidya On Sun, Sep 25, 2022 at 5:59 PM ShunKang Lin wrote: > Hi all, I'd like to start a new discussion thread on KIP-872 (Kafka Client) > which proposes th

Re: [VOTE] 3.3.0 RC2

2022-09-28 Thread Divij Vaidya
Please ignore my previous email. Seems like that is a known issue and we plan to fix it after the release. Divij Vaidya On Wed, Sep 28, 2022 at 3:08 PM Divij Vaidya wrote: > Hey folks > > I noticed a non-blocking bug with the documentation page where the arrow > to left

Re: [VOTE] 3.3.0 RC2

2022-09-28 Thread Divij Vaidya
this bug. Reproducer: 1. Visit https://kafka.apache.org/33/documentation.html on a chrome browser. 2. Observe that arrow to expand left nav overlaps with text. 3. Remove "33/" from the url to observe the current documentation. 4. Observe that the current website does not have this bug. Di

Re: [DISCUSS] KIP-852 Optimize calculation of size for log in remote tier

2022-09-28 Thread Divij Vaidya
Hey folks Does anyone else have any thoughts on this before I propose this for a vote? -- Divij Vaidya On Mon, Sep 5, 2022 at 12:57 PM Satish Duggana wrote: > Thanks for the KIP Divij! > > This is a nice improvement to avoid recalculation of size. Customized RLMMs > can implem

Re: [DISCUSS] KIP-870: Retention policy based on record event time

2022-09-28 Thread Divij Vaidya
us in nature and may not provide "at max" guarantee. It can provide "at least" guarantee. -- Divij Vaidya On Wed, Sep 28, 2022 at 3:47 PM Николай Ижиков wrote: > Hello! > > Please, share your feedback? > Is proposed retention policy can be useful for Kafka us

Re: [DISCUSS] KIP-872: Add Serializer#serializeToByteBuffer() to reduce memory copying

2022-09-29 Thread Divij Vaidya
of the provided input buffer nor the pointers). -- Divij Vaidya On Wed, Sep 28, 2022 at 5:35 PM ShunKang Lin wrote: > Hi Divij Vaidya, > > Thanks for your comments. > > 1. I checked the code of KafkaProducer#doSend() > and RecordAccumulator#append(), if KafkaProducer#doSend(

Re: [DISCUSS]: Including TLA+ in the repo

2022-08-10 Thread Divij Vaidya
; > [1]: > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-848%3A+The+Next+Generation+of+the+Consumer+Rebalance+Protocol#KIP848:TheNextGenerationoftheConsumerRebalanceProtocol-TestPlan > > > [2]: > > > > > > https://cwiki.apache.org/confluence/display/KAFKA/KIP-853%3A+KRaft+Voter+Changes#KIP853:KRaftVoterChanges-TestPlan > > > [3]: https://github.com/hachikuji/kafka-specification > > > [4]: > > > > > > https://github.com/Vanlightly/raft-tlaplus/tree/main/specifications/pull-raft > > > [5]: https://www.apache.org/legal/resolved.html > > > > > -- > -- Guozhang > -- Divij Vaidya

Re: ARM/PowerPC builds

2022-08-04 Thread Divij Vaidya
Thank you. This would greatly improve the PR experience since now, there is higher probability for it to be green. Side question though, do we know why ARM tests are timing out? Should we start a JIRA with Apache Infra to root cause? — Divij Vaidya On Thu, Aug 4, 2022 at 12:42 AM Colin McCabe

Re: Last sprint to finish line: Replace EasyMock/Powermock with Mockito

2022-08-04 Thread Divij Vaidya
e=true -Dorg.gradle.parallel=false` to verify that no test assertion has been accidentally removed during the change. *Chris*, would you like to add anything else to the general guidance above which would help reduce the code review churn? -- Divij Vaidya On Mon, Aug 1, 2022 at 6:49 PM Divij Vaidya wrote:

Re: [VOTE] 3.3.2 RC1

2022-12-22 Thread Divij Vaidya
: Authorization failed. (kafka.admin.TopicCommand$) -- Divij Vaidya On Thu, Dec 22, 2022 at 7:12 AM Yash Mayya wrote: > Hi Chris, > > I did the following release validations - > > - Verified the MD5 / SHA-1 / SHA-512 checksums and the PGP signatures > - Built from source using Jav

Re: [DISCUSS] KIP-852 Optimize calculation of size for log in remote tier

2022-12-20 Thread Divij Vaidya
adding a new API, then we need to cache the size in RLM itself (instead of RLMM implementation) and incrementally manage it. The downside of longer archive time at startup (due to initial scale) still remains valid in this situation. -- Divij Vaidya On Fri, Dec 16, 2022 at 12:43 AM Jun Rao wrote

Re: [ANNOUNCE] New committer: Satish Duggana

2022-12-23 Thread Divij Vaidya
The PMC of Apache Kafka is pleased to announce a new Kafka committer > > > Satish > > > > Duggana. > > > > > > > > Satish has been a long time Kafka contributor since 2017. He is the > > main > > > > driver behind KIP-405 that integrates Kafka with remote storage, a > > > > significant and much anticipated feature in Kafka. > > > > > > > > Congratulations, Satish! > > > > > > > > Thanks, > > > > > > > > Jun (on behalf of the Apache Kafka PMC) > > > > > > > > > > -- Divij Vaidya

Re: [DISCUSS] Apache Kafka 3.4.0 release

2022-12-09 Thread Divij Vaidya
on this subject. -- Divij Vaidya On Thu, Dec 8, 2022 at 3:48 AM Sophie Blee-Goldman wrote: > Thanks everyone for the updates, that all sounds good. > > Divij: I pinged some of the relevant reviewers to give your PRs a final > pass, but will > leave it to their judgement f

Re: [DISCUSS] KIP-852 Optimize calculation of size for log in remote tier

2022-12-07 Thread Divij Vaidya
tion hasn't found a need for a > similar new method. For public interface, ideally we want to make it more > general. > > Thanks, > > Jun > > On Mon, Nov 21, 2022 at 7:11 AM Divij Vaidya > wrote: > > > Thank you Jun and Alex for your comments. > > > > P

Re: [DISCUSS] Apache Kafka 3.4.0 release

2022-12-07 Thread Divij Vaidya
://issues.apache.org/jira/browse/KAFKA-7109 2. https://github.com/apache/kafka/pull/12228 -- Divij Vaidya On Wed, Dec 7, 2022 at 3:18 AM Sophie Blee-Goldman wrote: > Hey all, > > First off, just a heads up that code freeze will be *tomorrow, Dec 6th* so > please make sure > to merge an

Re: [DISCUSS] KIP-852 Optimize calculation of size for log in remote tier

2022-12-14 Thread Divij Vaidya
ar buffer / streaming interface from RSM (so that we don't have to wait to fetch the entire segment before starting to send records to the consumer), caching the segments fetched from RSM locally (I would assume all RSM plugin implementations to do this, might as well add it to RLM) etc. -- Divij Vaidya

Re: [DISCUSS] KIP-852 Optimize calculation of size for log in remote tier

2022-11-21 Thread Divij Vaidya
s the main concern for using the cache to fetch total size. Regards, Divij Vaidya On Fri, Nov 18, 2022 at 12:59 PM Alexandre Dupriez < alexandre.dupr...@gmail.com> wrote: > Hi Divij, > > Thanks for the KIP. Please find some comments based on what I read on > this thread so far

Re: [DISCUSS] KIP-852 Optimize calculation of size for log in remote tier

2022-11-17 Thread Divij Vaidya
e has been very constructive for me. Regards, Divij Vaidya On Thu, Nov 10, 2022 at 10:49 PM Jun Rao wrote: > Hi, Divij, > > Thanks for the reply. > > It's true that the data in Kafka could be kept longer with KIP-405. How > much data do you envision to have per broke

Re: [DISCUSS] KIP-852 Optimize calculation of size for log in remote tier

2022-11-10 Thread Divij Vaidya
mplementation of log retention check in RemoteLogManager. -- Divij Vaidya On Thu, Nov 10, 2022 at 6:16 AM Luke Chen wrote: > Hi Divij, > > One more question about the metric: > I think the metric will be updated when > (1) each time we run the log retention check (that is, >

Re: [DISCUSS] KIP-872: Add Serializer#serializeToByteBuffer() to reduce memory copying

2022-11-07 Thread Divij Vaidya
haps, this is worth clarifying in the KIP? -- Divij Vaidya On Sun, Nov 6, 2022 at 4:23 PM John Roesler wrote: > Thanks for the reply, ShunKang! > > You’re absolutely right, we should not change the behavior of the existing > method. > > Regarding the new method, I was thinki

Re: [DISCUSS] KIP-852 Optimize calculation of size for log in remote tier

2022-11-08 Thread Divij Vaidya
ur motivation here is to avoid polluting the interface with optimization specific APIs and I will agree with that goal. But I believe that this new API proposed in the KIP brings in significant improvement and there is no other work around available to achieve the same performance. Regards, D

[VOTE] KIP-852: Optimize calculation of size for log in remote tier

2022-11-01 Thread Divij Vaidya
+in+remote+tier Regards Divij Vaidya

Re: [ANNOUNCE] New Kafka PMC Member: Chris Egerton

2023-03-09 Thread Divij Vaidya
Congratulations Chris! I am in awe with the amount of effort you put in code reviews and helping out the community members. Very well deserved. -- Divij Vaidya On Thu, Mar 9, 2023 at 9:49 PM Jorge Esteban Quilcate Otoya < quilcate.jo...@gmail.com> wrote: > So well deserved! Congra

Re: Hosting Kafka Videos on ASF YouTube channel

2023-03-09 Thread Divij Vaidya
Sure, that makes sense Bill. -- Divij Vaidya On Thu, Mar 9, 2023 at 5:26 PM Bill Bejeck wrote: > Hi Divij, > > Thanks for your comments. > > I understand what you're saying, and I agree with you to a point. > However, IHMO, it's necessary right now as we've onl

Re: [VOTE] KIP-902: Upgrade Zookeeper to 3.8.1

2023-03-20 Thread Divij Vaidya
emoval being far ahead down the line. -- Divij Vaidya On Wed, Mar 15, 2023 at 12:59 PM Christo Lolov wrote: > Hello Colin, > > Thank you for taking the time to review the proposal! > > I have attached a compatibility matrix to aid the explanation below - if > the mailing syste

Re: Hosting Kafka Videos on ASF YouTube channel

2023-03-09 Thread Divij Vaidya
similar for Apache TinkerPop after conferring with j...@apache.org. Reference email thread https://lists.apache.org/thread/kmdfdf65jvcdpzftwmpvbwbyjrmpfhol and YouTube channel https://www.youtube.com/@apachetinkerpop -- Divij Vaidya On Mon, Mar 6, 2023 at 9:26 PM Brian Proffitt wrote: > Yes

Re: [VOTE] KIP-902: Upgrade Zookeeper to 3.8.1

2023-02-23 Thread Divij Vaidya
2.4.x is an EOL version. Vote +1 (non binding) -- Divij Vaidya On Thu, Feb 23, 2023 at 3:32 PM Christo Lolov wrote: > Hello! > > I would like to start the vote for KIP-902, which upgrades Zookeeper to > version 3.8.1. > > The KIP can be found at > https://cwiki.apache

Re: [DISCUSS] Re-visit end of life policy

2023-04-20 Thread Divij Vaidya
to support the proposal. -- Divij Vaidya On Thu, Apr 20, 2023 at 3:52 AM Matthias J. Sax wrote: > While I understand the desire, I tend to agree with Ismael. > > In general, it's a significant amount of work not just to do the actual > releases, but also the cherry-pick bug-fi

Re: [DISCUSS] Re-visit end of life policy

2023-04-18 Thread Divij Vaidya
in the community etc. -- Divij Vaidya On Thu, Apr 13, 2023 at 11:53 PM Ismael Juma wrote: > Clarification below. > > I did not understand your point about maintenance expense to ensure > > compatibility. I am confused because, IMO, irrespective of our bug fix > > suppo

Re: [DISCUSS] KIP-872: Add Serializer#serializeToByteBuffer() to reduce memory copying

2023-04-18 Thread Divij Vaidya
3. Ok. Seems like there is no way around to enforce better semantics and maintain backward compatibility as well! Let's go ahead with what you proposed and create a JIRA to fix the semantics in version 4.x. My comment is resolved here. -- Divij Vaidya On Mon, Apr 10, 2023 at 6:47 AM ShunKang

Re: Adding reviewers with Github actions

2023-04-18 Thread Divij Vaidya
ticular person using the following link: https://github.com/search?q=is%3Apr+reviewed-by%3A +repo%3Aapache%2Fkafka+repo%3Aapache%2Fkafka-site=issues (replace with the GitHub username) -- Divij Vaidya On Tue, Apr 18, 2023 at 4:09 PM Viktor Somogyi-Vass wrote: > I'm not that familiar with Actions

Re: [DISCUSS] KIP-872: Add Serializer#serializeToByteBuffer() to reduce memory copying

2023-04-25 Thread Divij Vaidya
emaining()). -- Divij Vaidya On Wed, Apr 19, 2023 at 6:24 AM ShunKang Lin wrote: > Thank you for your comment, Divij. > > 4. Do you still have any questions about #4? > > 5. I add test case for ByteBufferSerializer backward compatibility: > > https://github.com/ap

Re: KIP-919: Allow AdminClient to Talk Directly with the KRaft Controller Quorum

2023-04-25 Thread Divij Vaidya
;kafka-metadata-shell.sh is at an "evolving" level of interface stability" -> I thought that with KRaft being production ready, the evolving mode for kraft-related tools has also moved to production. Do we have a timeline when this would move to production? -- Divij Vaidya On T

Re: Max Multi Tenancy topic naming - Cloudflare

2023-04-30 Thread Divij Vaidya
how many topics can be supported in a cluster? Or did you want to ask if there is an upper ceiling to the number of ACL configurations applied to isolate permissions to namespaces? -- Divij Vaidya On Fri, Apr 28, 2023 at 7:58 PM Kenneth Eversole wrote: > Hello Everyone, > > I ju

Re: [DISCUSS] KIP-895: Dynamically refresh partition count of __consumer_offsets

2023-04-05 Thread Divij Vaidya
st improve (not fix but just improve!) the current situation by reducing the time to recovery using this KIP. Let's circle back on this discussion as soon as KIP is updated with more details. -- Divij Vaidya On Tue, Apr 4, 2023 at 8:00 PM Alexandre Dupriez < alexandre.dupr...@gmail.com> wrot

Re: [DISCUSS] KIP-895: Dynamically refresh partition count of __consumer_offsets

2023-04-04 Thread Divij Vaidya
faced this problem? [1] https://lists.apache.org/thread/zoowjshvdpkh5p0p7vqjd9fq8xvkr1nd -- Divij Vaidya On Wed, Jan 18, 2023 at 9:52 AM Christo Lolov wrote: > Greetings, > > I am bumping the below DISCUSSion thread for KIP-895. The KIP presents a > situation where con

[DISCUSS] Re-visit end of life policy

2023-04-13 Thread Divij Vaidya
aws-Approvals> don't mention the voting action required for changing EOL policy, I would propose a vote for this change by Lazy Majority <https://cwiki.apache.org/confluence/display/KAFKA/Bylaws#Bylaws-Approvals> . Thoughts? -- Divij Vaidya

Re: [DISCUSS] Re-visit end of life policy

2023-04-13 Thread Divij Vaidya
. Apart from the above points, Ismael, what do you think about having different durations of security support and bug fix support? -- Divij Vaidya On Thu, Apr 13, 2023 at 4:13 PM Ismael Juma wrote: > Hi, > > I think we're discussing two separate things and each has a very differe

  1   2   3   4   >