[jira] [Created] (KAFKA-13965) Document broker-side socket-server-metrics

James Cheng created KAFKA-13965: --- Summary: Document broker-side socket-server-metrics Key: KAFKA-13965 URL: https://issues.apache.org/jira/browse/KAFKA-13965 Project: Kafka Issue Type

Re: Are timestamps available for records stored in Kafka Streams state stores?

you > can access via the TimestampedKeyValueStore/TimstampedWindowStore. > > What's not implemented yet are timestamped session stores. > > Guozhang > > On Thu, May 19, 2022 at 12:49 PM James Cheng wrote: > >> Hi, >> >> I'm trying to see if timestamps are

Are timestamps available for records stored in Kafka Streams state stores?

Hi, I'm trying to see if timestamps are available for records that are stored in Kafka Streams state stores. I saw "KIP-258: Allow to Store Record Timestamps in RocksDB" https://cwiki.apache.org/confluence/display/KAFKA/KIP-258%3A+Allow+to+Store+Record+Timestamps+in+RocksDB

Re: [ANNOUNCE] Apache Kafka 3.2.0

Bruno, Congrats on the release! There is a small typo on the page. > KIP-791 > > adds method recordMetada() to the StateStoreContext, Should be > KIP-791 >

Re: [VOTE] KIP-831: Add metric for log recovery progress

+1 (non-binding) -James Sent from my iPhone > On May 16, 2022, at 12:12 AM, Luke Chen wrote: > > Hi all, > > I'd like to start a vote on KIP to expose metrics for log recovery > progress. These metrics would let the admins have a way to monitor the log > recovery progress. > > Details can

Re: [DISCUSS] KIP-831: Add metric for log recovery progress

0 >> - 1 => 5 >> - /tmp/log2 >> - 0 => 0 >> - 1 => 0 >> >> >> That said, the `RemainingBytesToRecovery` metric is difficult to achieve >> as you expected. I think the current proposal wi

Re: [DISCUSS] KIP-831: Add metric for log recovery progress

ere >> <https://cwiki.apache.org/confluence/display/KAFKA/KIP-831%3A+Add+metric+for+log+recovery+progress#KIP831:Addmetricforlogrecoveryprogress-ProposedChanges> >> shows what it will look like. >> >> Thanks for the suggestion. >> >> Thank you. >>

Re: [DISCUSS] KIP-831: Add metric for log recovery progress

The KIP describes RemainingLogsToRecovery, which seems to be the number of partitions in each log.dir. We have some partitions which are much much larger than others. Those large partitions have many many more segments than others. Is there a way the metric can reflect partition size? Could

Re: [DISCUSS] KIP-729 Custom validation of records on the broker prior to log append

One use case we would like is to require that producers are sending compressed messages. Would this KIP (or KIP-686) allow the broker to detect that? From looking at both KIPs, it doesn't look it would help with my particular use case. Both of the KIPs are at the Record-level. Thanks, -James

Re: [DISCUSS] KIP-760: Increase minimum value of segment.ms and segment.bytes

Badai, Thanks for the KIP. We sometimes want to force compaction on a topic. This might be because there is a bad record in the topic, and we want to force it to get deleted. The way we do this is, we set segment.ms to a small value and write a record, in order to force a segment roll. And we

Re: [ANNOUNCE] Apache Kafka 2.6.1

Thank you Mickael for running the release. Good job everyone! -James Sent from my iPhone > On Jan 11, 2021, at 2:17 PM, Mickael Maison wrote: > > The Apache Kafka community is pleased to announce the release for > Apache Kafka 2.6.1. > > This is a bug fix release and it includes fixes and

Re: Can I get a review for a documentation update (KAFKA-10473)?

Hi, Can I get a review for this small documentation update? Thanks, -James > On Oct 9, 2020, at 5:23 PM, James Cheng wrote: > > Hi, > > Would someone be able to review this pull request for me? > > This is a small documentation change. > > Thanks, > -James &

[jira] [Created] (KAFKA-10801) Docs on configuration have multiple places using the same HTML anchor tag

James Cheng created KAFKA-10801: --- Summary: Docs on configuration have multiple places using the same HTML anchor tag Key: KAFKA-10801 URL: https://issues.apache.org/jira/browse/KAFKA-10801 Project

Re: [DISCUSS] KIP-640 Add log compression analysis tool

Chris, This (understandably) requires access to the log segment files on disk. Managed Kafka services are becoming more popular (Confluent Cloud, Amazon MSK) and they do not expose the log segment files on disk. It’d be great to have an equivalent functionality that would work on managed

Re: [ANNOUNCE] New committer: David Jacot

Congratulations, David! -James > On Oct 16, 2020, at 9:01 AM, Gwen Shapira wrote: > > The PMC for Apache Kafka has invited David Jacot as a committer, and > we are excited to say that he accepted! > > David Jacot has been contributing to Apache Kafka since July 2015 (!) > and has been very

Re: [ANNOUNCE] New committer: Chia-Ping Tsai

Congratulations Chia-Ping! -James > On Oct 19, 2020, at 10:24 AM, Guozhang Wang wrote: > > Hello all, > > I'm happy to announce that Chia-Ping Tsai has accepted his invitation to > become an Apache Kafka committer. > > Chia-Ping has been contributing to Kafka since March 2018 and has made 74

Re: [ANNOUNCE] New committer: A. Sophie Blee-Goldman

Congratulations, Sophie! -James > On Oct 19, 2020, at 9:40 AM, Matthias J. Sax wrote: > > Hi all, > > I am excited to announce that A. Sophie Blee-Goldman has accepted her > invitation to become an Apache Kafka committer. > > Sophie is actively contributing to Kafka since Feb 2019 and has >

Re: Can I get a review for a documentation update (KAFKA-10473)?

Hi, Would someone be able to review this pull request for me? This is a small documentation change. Thanks, -James > On Sep 24, 2020, at 11:53 PM, James Cheng wrote: > > Hi, > > Can I get a review from one of the commiters for this documentation update? >

Can I get a review for a documentation update (KAFKA-10473)?

Hi, Can I get a review from one of the commiters for this documentation update? I am adding docs for the following JMX metrics: kafka.log,type=Log,name=Size kafka.log,type=Log,name=NumLogSegments kafka.log,type=Log,name=LogStartOffset

Re: How do I place a JIRA in status "Patch Available"? (For KAFKA-10473)

fluence/pages/diffpagesbyversion.action?pageId=59689925=36=37> Thanks for the help, -James > On Sep 11, 2020, at 12:53 PM, John Roesler wrote: > > Hi James, > > Sorry, I overlooked your reply until now. I've granted you > access. > > Thanks, > -John > > On Wed, 2020-09-09

Re: How do I place a JIRA in status "Patch Available"? (For KAFKA-10473)

n. It’s a > separate account from Jira. > > Thanks, > John > >> On Wed, Sep 9, 2020, at 20:45, James Cheng wrote: >> Thanks John. That worked. >> >> I clicked the button that says "Submit Patch", and a dialog box popped >> up. I didn

Re: How do I place a JIRA in status "Patch Available"? (For KAFKA-10473)

w. Do this by clicking the "Submit Patch" button in JIRA, and then in the resulting dialog, click "Submit Patch". -James > On Sep 9, 2020, at 6:24 PM, John Roesler wrote: > > Hi James, > > I think the button on Jira says “Add Patch” or something confusing like tha

How do I place a JIRA in status "Patch Available"? (For KAFKA-10473)

I have a JIRA that I am working on, and a pull request available for it. [KAFKA-10473] Website is missing docs on JMX metrics for partition size-on-disk (kafka.log:type=Log,name=*) https://issues.apache.org/jira/browse/KAFKA-10473 https://github.com/apache/kafka/pull/9276 The "Contributing Code

[jira] [Created] (KAFKA-10473) Website is missing docs on JMX metrics for partition size-on-disk

James Cheng created KAFKA-10473: --- Summary: Website is missing docs on JMX metrics for partition size-on-disk Key: KAFKA-10473 URL: https://issues.apache.org/jira/browse/KAFKA-10473 Project: Kafka

Re: [DISCUSS] KIP-579: new exception on min.insync.replicas > replication.factor

about that idea, too. Some broker default settings can also be set dynamically during runtime, so you will also have to catch/reject those. -James > Thanks, > Paolo > > On Wed, 1 Apr 2020 at 05:29, James Cheng wrote: > >> I agree that we should prevent the creation of

Re: [DISCUSS] KIP-579: new exception on min.insync.replicas > replication.factor

I agree that we should prevent the creation of such a topic configuration. That would mean catching it at topic-creation time, as well as catching it on any topic-configuration changes that might make min.isr > replication factor. Not sure how we would detect things if someone changed the

[jira] [Created] (KAFKA-9697) ControlPlaneNetworkProcessorAvgIdlePercent is always NaN

James Cheng created KAFKA-9697: -- Summary: ControlPlaneNetworkProcessorAvgIdlePercent is always NaN Key: KAFKA-9697 URL: https://issues.apache.org/jira/browse/KAFKA-9697 Project: Kafka Issue

[jira] [Created] (KAFKA-9696) Document the control plane metrics that were added in KIP-402

James Cheng created KAFKA-9696: -- Summary: Document the control plane metrics that were added in KIP-402 Key: KAFKA-9696 URL: https://issues.apache.org/jira/browse/KAFKA-9696 Project: Kafka

Re: [ANNOUNCE] New Kafka PMC Members: Colin, Vahid and Manikumar

Congrats Colin, Vahid, and Manikumar! -James > On Jan 14, 2020, at 10:59 AM, Tom Bentley wrote: > > Congratulations! > > On Tue, Jan 14, 2020 at 6:57 PM Rajini Sivaram > wrote: > >> Congratulations Colin, Vahid and Manikumar! >> >> Regards, >> Rajini >> >> On Tue, Jan 14, 2020 at 6:32 PM

[jira] [Created] (KAFKA-9389) Document how to use kafka-reassign-partitions.sh to change log dirs for a partition

James Cheng created KAFKA-9389: -- Summary: Document how to use kafka-reassign-partitions.sh to change log dirs for a partition Key: KAFKA-9389 URL: https://issues.apache.org/jira/browse/KAFKA-9389

[jira] [Resolved] (KAFKA-8745) DumpLogSegments doesn't show keys, when the message is null

[ https://issues.apache.org/jira/browse/KAFKA-8745?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Cheng resolved KAFKA-8745. Fix Version/s: 2.4.0 Reviewer: Guozhang Wang Resolution: Fixed > DumpLogSegme

Re: UnderReplicatedPartitions = 0 and UnderMinPartitionIsrCount > 0

Alexandre, You are right that this is a problem. There is a JIRA on this from a while back. https://issues.apache.org/jira/plugins/servlet/mobile#issue/KAFKA-4680 I don’t think anyone is currently working on it right now. -James Sent from my iPhone > On Aug 13, 2019, at 1:17 AM, Alexandre

[jira] [Created] (KAFKA-8745) DumpLogSegments doesn't show keys, when the message is null

James Cheng created KAFKA-8745: -- Summary: DumpLogSegments doesn't show keys, when the message is null Key: KAFKA-8745 URL: https://issues.apache.org/jira/browse/KAFKA-8745 Project: Kafka Issue

Re: [ANNOUNCE] New Kafka PMC member: Matthias J. Sax

Congrats!! -James Sent from my iPhone > On Apr 18, 2019, at 2:35 PM, Guozhang Wang wrote: > > Hello Everyone, > > I'm glad to announce that Matthias J. Sax is now a member of Kafka PMC. > > Matthias has been a committer since Jan. 2018, and since then he continued > to be active in the

Re: [ANNOUNCE] New Committer: Randall Hauch

Congrats, Randall! Well deserved! -James Sent from my iPhone > On Feb 14, 2019, at 6:16 PM, Guozhang Wang wrote: > > Hello all, > > The PMC of Apache Kafka is happy to announce another new committer joining > the project today: we have invited Randall Hauch as a project committer and > he

Re: [DISCUSS] Kafka 2.2.0 in February 2018

Matthias, You said “Friday 2/14”, but 2/14 is this Thursday. -James Sent from my iPhone > On Feb 11, 2019, at 2:31 PM, Matthias J. Sax wrote: > > Hello, > > this is a short reminder, that feature freeze for AK 2.2 release is end > of this week, Friday 2/14. > > Currently, there are two

[jira] [Created] (KAFKA-7884) Docs for message.format.version and log.message.format.version show invalid (corrupt?) "valid values"

James Cheng created KAFKA-7884: -- Summary: Docs for message.format.version and log.message.format.version show invalid (corrupt?) "valid values" Key: KAFKA-7884 URL: https://issues.apache.org/jira/browse/

Re: [DISCUSS] Kafka 2.2.0 in February 2018

I think KIP-291 is supposed to go into 2.2.0. https://cwiki.apache.org/confluence/display/KAFKA/KIP-291%3A+Separating+controller+connections+and+requests+from+the+data+plane At least, according to the associated JIRA, the pull request seems to have been merged just a couple days ago.

Re: [ANNOUNCE] New Committer: Vahid Hashemian

Congrats, Vahid!! -James > On Jan 15, 2019, at 2:44 PM, Jason Gustafson wrote: > > Hi All, > > The PMC for Apache Kafka has invited Vahid Hashemian as a project committer > and > we are > pleased to announce that he has accepted! > > Vahid has made numerous contributions to the Kafka

[jira] [Created] (KAFKA-7797) Replication throttling configs aren't in the docs

James Cheng created KAFKA-7797: -- Summary: Replication throttling configs aren't in the docs Key: KAFKA-7797 URL: https://issues.apache.org/jira/browse/KAFKA-7797 Project: Kafka Issue Type

Re: [ANNOUNCE] Apache Kafka 2.1.0

Thanks Dong for running the release, and congrats to everyone in the community! -James Sent from my iPhone > On Nov 21, 2018, at 10:09 AM, Dong Lin wrote: > > The Apache Kafka community is pleased to announce the release for Apache > Kafka 2.1.0 > > > This is a major release and includes

Re: [ANNOUNCE] New committer: Colin McCabe

Congrats, Colin! -James > On Sep 25, 2018, at 1:39 AM, Ismael Juma wrote: > > Hi all, > > The PMC for Apache Kafka has invited Colin McCabe as a committer and we are > pleased to announce that he has accepted! > > Colin has contributed 101 commits and 8 KIPs including significant >

Re: [ANNOUNCE] New Kafka PMC member: Dong Lin

Congrats Dong! -James > On Aug 20, 2018, at 3:54 AM, Ismael Juma wrote: > > Hi everyone, > > Dong Lin became a committer in March 2018. Since then, he has remained > active in the community and contributed a number of patches, reviewed > several pull requests and participated in numerous KIP

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

Guozhang, in a previous message, you proposed said this: > On Jul 30, 2018, at 3:56 PM, Guozhang Wang wrote: > > 1. We bump up the JoinGroupRequest with additional fields: > > 1.a) a flag indicating "static" or "dynamic" membership protocols. > 1.b) with "static" membership, we also add the

Re: [DISCUSS] KIP-346 - Limit blast radius of log compaction failure

t;>>>> On Wed, Jul 25, 2018 at 11:55 AM Dhruvil Shah >>>>> wrote: >>>>>> >>>>>>> For the cleaner thread specifically, I do not think respawning will >>>>> help at >>>>>>> all because we are more than likely

Re: [DISCUSS] KIP-346 - Limit blast radius of log compaction failure

think are not true for the log cleaner. >>>>> >>>>> On Wed, Jul 25, 2018 at 11:08 AM Ron Dagostino >>> wrote: >>>>> >>>>>> <<>> in >>>>> an >>>>>> infinite loop which consumes resources and f

Re: [ANNOUNCE] Apache Kafka 2.0.0 Released

Congrats and great job, everyone! Thanks Rajini for driving the release! -James Sent from my iPhone > On Jul 30, 2018, at 3:25 AM, Rajini Sivaram wrote: > > The Apache Kafka community is pleased to announce the release for > > Apache Kafka 2.0.0. > > > > > > This is a major release and

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

c, the > tolerance of client side error is not unlimited. > > > Thank you! > > > > From: James Cheng > Sent: Saturday, July 28, 2018 1:26 AM > To: dev@kafka.apache.org > Subject: Re: [DISCUSS] KIP-345: Reduce multiple consumer re

Re: [DISCUSS] KIP-345: Reduce multiple consumer rebalances by specifying member id

> On Jul 26, 2018, at 11:09 PM, Guozhang Wang wrote: > > Hi Boyang, > > Thanks for the proposed KIP. I made a pass over the wiki and here are some > comments / questions: > > 1. In order to preserve broker compatibility, we need to make sure the > broker version discovery logic can be

Re: [DISCUSS] KIP-346 - Limit blast radius of log compaction failure

Hi Stanislav! Thanks for this KIP! I agree that it would be good if the LogCleaner were more tolerant of errors. Currently, as you said, once it dies, it stays dead. Things are better now than they used to be. We have the metric

[jira] [Created] (KAFKA-7144) Kafka Streams doesn't properly balance partition assignment

James Cheng created KAFKA-7144: -- Summary: Kafka Streams doesn't properly balance partition assignment Key: KAFKA-7144 URL: https://issues.apache.org/jira/browse/KAFKA-7144 Project: Kafka Issue

Re: [DISCUSS] KIP-325: Extend Consumer Group Command to Show Beginning Offsets

The “Motivation” section of the KIP says that the starting offset will be useful but doesn’t say why. Can you add a use-case or two to describe how it will be useful? In our case (see https://github.com/wushujames/kafka-utilities/blob/master/ConsumerGroupLag/README.md), we found the starting

Re: kafka ack=all and min-isr

I wrote a blog post on min.isr that explains it in more detail: https://logallthethings.com/2016/07/11/min-insync-replicas-what-does-it-actually-do/ The post is 2 years old, but I think it's still correct. -James > On Jun 7, 2018, at 10:31 PM, Carl Samuelson wrote: > > Hi > > Hopefully this

[jira] [Created] (KAFKA-6970) Kafka streams lets the user call init() and close() on a state store, when inside Processors

James Cheng created KAFKA-6970: -- Summary: Kafka streams lets the user call init() and close() on a state store, when inside Processors Key: KAFKA-6970 URL: https://issues.apache.org/jira/browse/KAFKA-6970

[jira] [Created] (KAFKA-6967) TopologyTestDriver does not allow pre-populating state stores that have change logging

James Cheng created KAFKA-6967: -- Summary: TopologyTestDriver does not allow pre-populating state stores that have change logging Key: KAFKA-6967 URL: https://issues.apache.org/jira/browse/KAFKA-6967

Re: Kafka KIP meeting on Apr. 9 at 9:00am PDT

Jun, Can you add me as well? wushuja...@gmail.com Thanks, -James > On Apr 5, 2018, at 1:34 PM, Jun Rao wrote: > > Hi, Ted, Vahid, > > Added you to the invite. > > Thanks, > > Jun > > > On Thu, Apr 5, 2018 at 10:42 AM, Vahid S Hashemian <

Re: [ANNOUNCE] Apache Kafka 1.1.0 Released

Thanks Damian and Rajini for running the release! Congrats and good job everyone! -James Sent from my iPhone > On Mar 29, 2018, at 2:27 AM, Rajini Sivaram wrote: > > The Apache Kafka community is pleased to announce the release for > > Apache Kafka 1.1.0. > > > Kafka

Re: [VOTE] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

+1 (non-binding) Thanks for all the hard work on this, Vahid! -James > On Mar 28, 2018, at 10:34 AM, Vahid S Hashemian > wrote: > > Hi all, > > As I believe the feedback and suggestions on this KIP have been addressed > so far, I'd like to start a vote. > The KIP

Re: [ANNOUNCE] New Committer: Dong Lin

Congrats, Dong! -James > On Mar 28, 2018, at 10:58 AM, Becket Qin wrote: > > Hello everyone, > > The PMC of Apache Kafka is pleased to announce that Dong Lin has accepted > our invitation to be a new Kafka committer. > > Dong started working on Kafka about four years

Re: [VOTE] KIP-268: Simplify Kafka Streams Rebalance Metadata Upgrade

+1 (non-binding) -James > On Mar 21, 2018, at 2:28 AM, Damian Guy wrote: > > +1 > > On Wed, 21 Mar 2018 at 01:44 abel-sun wrote: > >> >> Thanks you of your offer ,agree with you! >> >> On 2018/03/21 00:56:11, Richard Yu

Re: [DISCUSS] KIP-268: Simplify Kafka Streams Rebalance Metadata Upgrade

s for all the answers to my questions. -James > > -Matthias > > On 3/21/18 11:16 PM, James Cheng wrote: >> >> >> >>> On Mar 21, 2018, at 11:18 AM, Matthias J. Sax <matth...@confluent.io> wrote: >>> >>> Thanks for following

Re: [DISCUSS] KIP-272: Add API version tag to broker's RequestsPerSec metric

rics,name=RequestsPerSec,request=Produce > > And the KIP is proposing to change it to: > > kafka.network:type=RequestMetrics,name=RequestsPerSec,request=Produce,version=1.0.0 I think that would make the KIP easier to understand. Thanks, -James Sent from my iPhone > On Mar 21, 2018,

Re: [DISCUSS] KIP-272: Add API version tag to broker's RequestsPerSec metric

> with another intuitive name. >> >> Thanks, >> Allen >> >> >> >>> On Wed, Mar 21, 2018 at 2:01 AM, Ted Yu <yuzhih...@gmail.com> wrote: >>> >>> Creating new metric and deprecating existing one seems better from >>> compatibili

Re: [DISCUSS] KIP-268: Simplify Kafka Streams Rebalance Metadata Upgrade

uired updating the on-disk schema. In the case of an updated RocksDB state store but with a downgraded rebalance metadata... that should work, right? Because we still have updated code (which understands the on-disk format) but that it simply gets its partition assignments via the downgraded

Re: [DISCUSS] KIP-272: Add API version tag to broker's RequestsPerSec metric

Manikumar brings up a good point. This is a breaking change to the existing metric. Do we want to break compatibility, or do we want to add a new metric and (optionally) deprecate the existing metric? For reference, in KIP-153 [1], we changed an existing metric without doing proper

Re: [DISCUSS] KIP-268: Simplify Kafka Streams Rebalance Metadata Upgrade

Sorry, I see that the VOTE started already, but I have a late question on this KIP. In the "version probing" protocol: > Detailed upgrade protocol from metadata version X to Y (with X >= 1.2): > On startup/rolling-bounce, an instance does not know what version the leader > understands and

Re: Log Retention Period of Kafka Messages

> On Mar 12, 2018, at 10:13 AM, Kyle Tinker > wrote: > > You have a couple options: > 1) You can adjust log.segment.bytes to make the segments smaller so that > individual segments can be trimmed > 2) You can set log.roll.hours, which will roll to a new log

Re: [DISCUSS] KIP-258: Allow to Store Record Timestamps in RocksDB

Matthias, For all the upgrade paths, is it possible to get rid of the 2nd rolling bounce? For the in-place upgrade, it seems like primary difference between the 1st rolling bounce and the 2nd rolling bounce is to decide whether to send Subscription Version 2 or Subscription Version 3.

Re: [VOTE] KIP-186: Increase offsets retention default to 7 days

+1 (non-binding) -James > On Mar 7, 2018, at 1:20 PM, Jay Kreps wrote: > > +1 > > I think we can improve this in the future, but this simple change will > avoid a lot of pain. Thanks for reviving it Ewen. > > -Jay > > On Mon, Mar 5, 2018 at 11:35 AM, Ewen Cheslack-Postava

Re: [ANNOUNCE] Apache Kafka 1.0.1 Released

Congrats, everyone! Thanks for driving the release, Ewen! -James > On Mar 6, 2018, at 1:22 PM, Guozhang Wang wrote: > > Ewen, thanks for driving the release!! > > > Guozhang > > On Tue, Mar 6, 2018 at 1:14 PM, Ewen Cheslack-Postava wrote: > >> The

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

Vahid, Under rejected alternatives, we had decided that we did NOT want to do per-partition expiration, and instead we wait until the entire group is empty and then (after the right time has passed) expire the entire group at once. I thought of one scenario that might benefit from

Re: [ANNOUNCE] New Kafka PMC Member: Rajini Sivaram

Congrats Rajini! -James Sent from my iPhone > On Jan 17, 2018, at 10:48 AM, Gwen Shapira wrote: > > Dear Kafka Developers, Users and Fans, > > Rajini Sivaram became a committer in April 2017. Since then, she remained > active in the community and contributed major

Re: [VOTE] KIP-247: Add public test utils for Kafka Streams

+1 (non-binding) -James Sent from my iPhone > On Jan 17, 2018, at 6:09 PM, Matthias J. Sax wrote: > > Hi, > > I would like to start the vote for KIP-247: > https://cwiki.apache.org/confluence/display/KAFKA/KIP-247%3A+Add+public+test+utils+for+Kafka+Streams > > >

Re: [ANNOUNCE] New committer: Matthias J. Sax

Congrats, Matthias!! Well deserved! -James > On Jan 12, 2018, at 2:59 PM, Guozhang Wang wrote: > > Hello everyone, > > The PMC of Apache Kafka is pleased to announce Matthias J. Sax as our > newest Kafka committer. > > Matthias has made tremendous contributions to Kafka

Pull Request emails are not being sent to the mailing list

Since Dec 21st, it seems that email notifications of Github pull requests are not being sent to the dev mailing list. Is this a bug, or have PR emails moved to somewhere else? If it’s a bug, can we look into this? The last email PR that was sent was this one:

[jira] [Created] (KAFKA-6312) Add documentation about kafka-consumer-groups.sh's ability to set/change offsets

James Cheng created KAFKA-6312: -- Summary: Add documentation about kafka-consumer-groups.sh's ability to set/change offsets Key: KAFKA-6312 URL: https://issues.apache.org/jira/browse/KAFKA-6312 Project

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

I think the discussion may have already cover this but just in case... How does the leader decide when a newly written message is "committed" enough to hand out to consumers? When a message is produced and is stored to the disk of the leader, the message is not considered "committed" until it

Re: [DISCUSS] KIP-225 - Use tags for consumer “records.lag” metrics

'm happy to adapt my KIP if the community thinks we should duplicate the > metric for a while. > > Thanks! > > On Thu, Nov 16, 2017 at 8:13 AM, James Cheng <wushuja...@gmail.com> wrote: > >> This KIP will break backwards compatibility for anyone who is using the >

Re: [DISCUSS] KIP-225 - Use tags for consumer “records.lag” metrics

This KIP will break backwards compatibility for anyone who is using the existing attribute names. Kafka devs, I believe that metrics are a supported interface, and so this would be a breaking change. In order to do this, we would need a deprecation timeframe for the old metric, and a

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

How fast does the in-memory cache grow? As a random datapoint... 10 months ago we set our offsets.retention.minutes to 1 year. So, for the past 10 months, we essentially have not expired any offsets. Via JMX, one of our brokers says

[jira] [Created] (KAFKA-6197) Difficult to get to the Kafka Streams javadocs

James Cheng created KAFKA-6197: -- Summary: Difficult to get to the Kafka Streams javadocs Key: KAFKA-6197 URL: https://issues.apache.org/jira/browse/KAFKA-6197 Project: Kafka Issue Type

Re: [ANNOUNCE] New committer: Onur Karaman

Congrats Onur! Well deserved! -James > On Nov 6, 2017, at 9:24 AM, Jun Rao wrote: > > Hi, everyone, > > The PMC of Apache Kafka is pleased to announce a new Kafka committer Onur > Karaman. > > Onur's most significant work is the improvement of Kafka controller, which > is

Re: [ANNOUNCE] Apache Kafka 1.0.0 Released

ras Beni, Andrey Dyachkov, Andy Chambers, > Apurva Mehta, Armin Braun, Attila Kreiner, Balint Molnar, Bart De Vylder, > Ben Stopford, Bharat Viswanadham, Bill Bejeck, Boyang Chen, Bryan Baugher, > Colin P. Mccabe, Koen De Groote, Dale Peakall, Damian Guy, Dana Powers, > Dejan Stojadinov

[jira] [Resolved] (KAFKA-6088) Kafka Consumer slows down when reading from highly compacted topics

[ https://issues.apache.org/jira/browse/KAFKA-6088?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] James Cheng resolved KAFKA-6088. Resolution: Won't Fix It is fixed in kafka client 0.11.0.0, and 0.11.0.0 clients can be used

[jira] [Created] (KAFKA-6088) Kafka Consumer slows down when reading from highly compacted topics

James Cheng created KAFKA-6088: -- Summary: Kafka Consumer slows down when reading from highly compacted topics Key: KAFKA-6088 URL: https://issues.apache.org/jira/browse/KAFKA-6088 Project: Kafka

[jira] [Created] (KAFKA-6054) ERROR "SubscriptionInfo - unable to decode subscription data: version=2" when upgrading from 0.10.0.0 to 0.10.2.1

James Cheng created KAFKA-6054: -- Summary: ERROR "SubscriptionInfo - unable to decode subscription data: version=2" when upgrading from 0.10.0.0 to 0.10.2.1 Key: KAFKA-6054 URL: https://issues.apache.org/j

[jira] [Created] (KAFKA-5952) Refactor Consumer Fetcher metrics

James Cheng created KAFKA-5952: -- Summary: Refactor Consumer Fetcher metrics Key: KAFKA-5952 URL: https://issues.apache.org/jira/browse/KAFKA-5952 Project: Kafka Issue Type: Sub-task

[jira] [Created] (KAFKA-5951) Autogenerate Producer RecordAccumulator metrics

James Cheng created KAFKA-5951: -- Summary: Autogenerate Producer RecordAccumulator metrics Key: KAFKA-5951 URL: https://issues.apache.org/jira/browse/KAFKA-5951 Project: Kafka Issue Type: Sub

Re: 1.0.0 KIPs Update

only after today. If it's new > functionality, then it depends on size, risk and importance. If you think > it's important and should target 1.0.0, feel free to make your case in the > JIRA or PR. > > Thanks, > Ismael > > On Wed, Sep 20, 2017 at 10:59 PM, James Cheng <wu

Re: 1.0.0 KIPs Update

Hi. I'm a little unclear on what types of things are subject to the "Feature Freeze" today (Sept 20th). For changes such as https://github.com/apache/kafka/pull/3799 and https://issues.apache.org/jira/browse/KAFKA-3480

Re: [DISCUSS] KIP-196: Add metrics to Kafka Connect framework

Thanks for the KIP, Randall. The KIP has one MBean per metric name. Can I suggest an alternate grouping? kafka.connect:type=connector-metrics,connector=([-.\w]+) connector-type connector-class connector-version status

Re: [ANNOUNCE] New Kafka PMC member: Jiangjie (Becket) Qin

Congrats Becket! -James > On Aug 23, 2017, at 10:20 PM, Joel Koshy wrote: > > Hi everyone, > > Jiangjie (Becket) Qin has been a Kafka committer in October 2016 and has > contributed significantly to several major patches, reviews and discussions > since. I am glad to

Re: [DISCUSS] KIP-186: Increase offsets retention default to 7 days

+1 from me! -James > On Aug 8, 2017, at 5:24 PM, Ewen Cheslack-Postava wrote: > > Hi all, > > I posted a simple new KIP for a problem we see with a lot of users: > KIP-186: Increase offsets retention default to 7 days > >

Re: Consumer offsets partitions size much bigger than others

It's possible that the log-cleaning thread has crashed. That is the thread that implements log compaction. Look in the log-cleaner.log file in your kafka debuglog directory to see if there is any indication that it has crashed (error messages, stack traces, etc). What version of kafka are you

[jira] [Created] (KAFKA-5597) Autogenerate Producer sender metrics

James Cheng created KAFKA-5597: -- Summary: Autogenerate Producer sender metrics Key: KAFKA-5597 URL: https://issues.apache.org/jira/browse/KAFKA-5597 Project: Kafka Issue Type: Sub-task

Re: [ANNOUNCE] New Kafka PMC member Ismael Juma

Congrats Ismael! -James > On Jul 5, 2017, at 1:55 PM, Jun Rao wrote: > > Hi, Everyone, > > Ismael Juma has been active in the Kafka community since he became > a Kafka committer about a year ago. I am glad to announce that Ismael is > now a member of Kafka PMC. > >

Re: [ANNOUNCE] New Kafka PMC member Jason Gustafson

Congrats Jason! -James > On Jul 11, 2017, at 10:32 PM, Guozhang Wang wrote: > > Hi Everyone, > > Jason Gustafson has been very active in contributing to the Kafka community > since he became a Kafka committer last September and has done lots of > significant work including

Re: Aggressive log compaction ratio appears to have no negative effect on log-compacted topics

Jeff, The discussion thread from a while back on KIP-58 has some discussion around "log.cleaner.min.cleanable.ratio". KIP-58 page: https://cwiki.apache.org/confluence/display/KAFKA/KIP-58+-+Make+Log+Compaction+Point+Configurable Discussion thread (linked off that page):

Re: Mirroring multiple clusters into one

we never mirror topics that start with "mirror." This prevents us from creating mirroring loops. > Thanks. > --Vahid > > > > From: James Cheng <wushuja...@gmail.com> > To: us...@kafka.apache.org > Cc: dev <dev@kafka.apache.org> > Date:

Re: Mirroring multiple clusters into one

I'm not sure what the "official" recommendation is. At TiVo, we *do* run all our mirrormakers near the target cluster. It works fine for us, but we're still fairly inexperienced, so I'm not sure how strong of a data point we should be. I think the thought process is, if you are mirroring from a

  1   2   3   4   >