[jira] [Created] (KAFKA-9142) topic __transaction_state disk space grows too big

panpan.liu created KAFKA-9142: - Summary: topic __transaction_state disk space grows too big Key: KAFKA-9142 URL: https://issues.apache.org/jira/browse/KAFKA-9142 Project: Kafka Issue Type:

[jira] [Created] (KAFKA-9141) Global state update error: missing recovery or wrong log message

Chris Toomey created KAFKA-9141: --- Summary: Global state update error: missing recovery or wrong log message Key: KAFKA-9141 URL: https://issues.apache.org/jira/browse/KAFKA-9141 Project: Kafka

Build failed in Jenkins: kafka-2.2-jdk8-old #184

See Changes: [wangguoz] HOTFIX: fix bug in VP test where it greps for the wrong log message -- Started by an SCM change Running as SYSTEM [EnvInject] - Loading node

Build failed in Jenkins: kafka-2.4-jdk8 #51

See Changes: [wangguoz] HOTFIX: fix bug in VP test where it greps for the wrong log message -- [...truncated 2.70 MB...]

Build failed in Jenkins: kafka-2.1-jdk8 #239

See Changes: [wangguoz] HOTFIX: fix bug in VP test where it greps for the wrong log message -- Started by an SCM change Running as SYSTEM [EnvInject] - Loading node

[jira] [Created] (KAFKA-9140) Consumer gets stuck rejoining the group indefinitely

Sophie Blee-Goldman created KAFKA-9140: -- Summary: Consumer gets stuck rejoining the group indefinitely Key: KAFKA-9140 URL: https://issues.apache.org/jira/browse/KAFKA-9140 Project: Kafka

Re: [DISCUSS] KIP-535: Allow state stores to serve stale reads during rebalance

>> I'm having some trouble wrapping my head around what race conditions might occur, other than the fundamentally broken state in which different instances are running totally different topologies. 3. @both Without the topic partitions that the tasks can map back to, we have to rely on

[jira] [Created] (KAFKA-9139) Dynamic broker config types aren't being discovered

Brian Byrne created KAFKA-9139: -- Summary: Dynamic broker config types aren't being discovered Key: KAFKA-9139 URL: https://issues.apache.org/jira/browse/KAFKA-9139 Project: Kafka Issue Type:

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

Hi, Satish, Thanks for the response. 21. Could you elaborate a bit why the positions in remote segment is different from the local one? I thought that they are identical copies. Jun On Fri, Nov 1, 2019 at 4:26 AM Satish Duggana wrote: > Hi Jun, > Thanks for looking into the updated KIP and

Re: [DISCUSS] KIP-405: Kafka Tiered Storage

Hi Jun, Can you please take a look at Satish's reply. Let us know if that answers your question. I would like to get yours and the rest of the community thoughts on the general direction we are going as we continue to make progress. Thanks, Harsha On Fri, Nov 1, 2019 at 3:06 AM Satish

[jira] [Created] (KAFKA-9138) Add system test covering Foreign Key joins (KIP-213)

John Roesler created KAFKA-9138: --- Summary: Add system test covering Foreign Key joins (KIP-213) Key: KAFKA-9138 URL: https://issues.apache.org/jira/browse/KAFKA-9138 Project: Kafka Issue Type:

Re: [DISCUSS] KIP-280: Enhanced log compaction

Eric, I think that's a good point, in `Headers.java` we also designed the API to only have `Header lastHeader(String key);`. I think picking the last header for that key would makes more sense since internally it is organized as a list such that newer headers could consider "overwriting" the

Re: [DISCUSS] KIP-280: Enhanced log compaction

Hi Senthilnathan, Regarding Matthias's point 6, what is the reasoning for choosing the first occurrence of the configured header? I believe this corresponds to the oldest value for given key. If there are multiple values for a key, it seems more intuitive that the newest value is the one that

Re: [DISCUSS] KIP-535: Allow state stores to serve stale reads during rebalance

Regarding 3) I'm wondering, does your concern still apply even now that the pluggable PartitionGrouper interface has been deprecated? Now that we can be sure that the DefaultPartitionGrouper is used to generate the taskId -> partitions mapping, we should be able to convert any taskId to any

Re: [DISCUSS] KIP-535: Allow state stores to serve stale reads during rebalance

Hey Vinoth, thanks for the reply! 3. I get that it's not the main focus of this KIP, but if it's ok, it would be nice to hash out this point right now. It only came up because this KIP-535 is substantially extending the pattern in question. If we push it off until later, then the reviewers are

Re: [VOTE] KIP-280: Enhanced log compaction

I only have one minor comment on the DISCUSS thread, otherwise I'm +1 (binding). On Mon, Nov 4, 2019 at 9:53 AM Senthilnathan Muthusamy wrote: > Hi all, > > I would like to start the vote on the updated KIP-280: Enhanced log > compaction. Thanks to Guozhang, Matthias & Tom for the valuable

Re: [DISCUSS] KIP-280: Enhanced log compaction

Hello Senthilnathan, Thanks for revamping on the KIP. I have only one comment about the wiki otherwise LGTM. 1. We should emphasize that the newly introduced config yields to the existing "log.cleanup.policy", i.e. if the latter's value is `delete` not `compact`, then the previous config would

[VOTE] KIP-280: Enhanced log compaction

Hi all, I would like to start the vote on the updated KIP-280: Enhanced log compaction. Thanks to Guozhang, Matthias & Tom for the valuable feedback on the discussion thread... KIP: https://cwiki.apache.org/confluence/display/KAFKA/KIP-280%3A+Enhanced+log+compaction JIRA:

RE: [DISCUSS] KIP-280: Enhanced log compaction

Hi all, I will start the vote thread shortly for this updated KIP. If there are any more thoughts I would love to hear them. Thanks, Senthil -Original Message- From: Senthilnathan Muthusamy Sent: Thursday, October 31, 2019 3:51 AM To: dev@kafka.apache.org Subject: RE: [DISCUSS]

Re: [DISCUSS] KIP-535: Allow state stores to serve stale reads during rebalance

3. Right now, we still get the topic partitions assigned as a part of the top level Assignment object (the one that wraps AssignmentInfo) and use that to convert taskIds back. This list of only contains assignments for that particular instance. Attempting to also reverse map for "all" the tasksIds

Re: [VOTE] KIP-523 Add KStream#toTable to the Streams DSL

+1 (binding), thanks Aishwarya! On Sun, Nov 3, 2019 at 11:46 AM aishwarya kumar wrote: > This thread has been open for more than 72 hours. So far there are 2 > binding and 1 non-binding votes, looking to conclude this quickly!! > > Best, > Aishwarya > > On Mon, Oct 28, 2019 at 5:00 PM John

[jira] [Created] (KAFKA-9137) Maintenance of FetchSession cache causing FETCH_SESSION_ID_NOT_FOUND in live sessions

Lucas Bradstreet created KAFKA-9137: --- Summary: Maintenance of FetchSession cache causing FETCH_SESSION_ID_NOT_FOUND in live sessions Key: KAFKA-9137 URL: https://issues.apache.org/jira/browse/KAFKA-9137

Re: [DISCUSS] Apache Kafka 2.4.0 release

Hi all, The underlying issue of KAFKA-8677 can cause data loss in consumers. I have included KAFKA-8677 as blocker to 2.4 release. Thanks to Guozhang for identifying the issue. https://issues.apache.org/jira/browse/KAFKA-8677 PR: https://github.com/apache/kafka/pull/7613 Thanks On Fri, Nov 1,

Re: [VOTE] KIP-543: Expand ConfigCommand's non-ZK functionality

No problem :) I also agree that it should be out of scope right now. On Thu, Oct 31, 2019 at 10:10 PM Colin McCabe wrote: > Sorry, that should read "Hi Viktor",-C. > > On Thu, Oct 31, 2019, at 14:08, Colin McCabe wrote: > > Hi Vikto, > > > > That's an interesting idea. However, I think

Re: [DISCUSS] KIP-542: Partition Reassignment Throttling

Exactly. I also can't envision scenarios where we would like to throttle the reassignment traffic to only a subset of the reassigning replicas. The other day I was wondering about that with specialized quotas we can solve the incremental partition reassignment too. Basically the controller would

Re: [DISCUSS] KIP-536: Propagate broker timestamp to Admin API

Cluster metadata is sent to clients on a very regular basis. Adding start-time there seems quite repetitive. Especially considering that this information is only useful in very specific cases. Can we add this capability to the Admin API in a way that won't impact normal client workflow? On Mon,

Why standby tasks read from the StandbyTasks::checkpointedOffsets in assignStandbyPartitions()

Hi, Please let me know if this is not the correct forum to ask this. But I have a doubt, I was hoping someone can clear it for me. In TaskManager:: updateNewAndRestoringTasks(), the function  assignStandbyPartitions() gets called for all the running standby tasks where it populates the Map:

[jira] [Created] (KAFKA-9136) get consumer latest commited timestamp

zhangzhisheng created KAFKA-9136: Summary: get consumer latest commited timestamp Key: KAFKA-9136 URL: https://issues.apache.org/jira/browse/KAFKA-9136 Project: Kafka Issue Type: Wish

Re: [DISCUSS] KIP-536: Propagate broker timestamp to Admin API

Thank you for the feedback, Stanislav! I agree that it would be good to harmonise the naming, and start-time-ms definitely more descriptive. I have updated the proposal to reflect this, and also added the updated json RPC changes. Please have a look. /noa > On 1 Nov 2019, at 09:13, Stanislav

Re: [DISCUSS] KIP-542: Partition Reassignment Throttling

Hi Viktor, > As for the first question I think is no need for *.throttled.replicas in case of reassignment because the LeaderAndIsrRequest exactly specifies the replicas needed to be throttled. Exactly. I also can't envision scenarios where we would like to throttle the reassignment traffic to

Re: [DISCUSS] KIP-542: Partition Reassignment Throttling

Hey Stan, > We will introduce two new configs in order to eventually replace *.replication.throttled.rate. Just to clarify, you mean to replace said config in the context of reassignment throttling, right? We are not planning to remove that config Yes, I don't want to remove that config either.

[jira] [Created] (KAFKA-9135) Kafka producer/consumer are creating too many open file

Dhirendra Singh created KAFKA-9135: -- Summary: Kafka producer/consumer are creating too many open file Key: KAFKA-9135 URL: https://issues.apache.org/jira/browse/KAFKA-9135 Project: Kafka