[jira] [Created] (KAFKA-6471) seekToEnd and seek give unclear results for Consumer with read_committed isolation level

Coen Damen created KAFKA-6471: - Summary: seekToEnd and seek give unclear results for Consumer with read_committed isolation level Key: KAFKA-6471 URL: https://issues.apache.org/jira/browse/KAFKA-6471

[jira] [Created] (KAFKA-6472) WordCount example code error

JONYhao created KAFKA-6472: -- Summary: WordCount example code error Key: KAFKA-6472 URL: https://issues.apache.org/jira/browse/KAFKA-6472 Project: Kafka Issue Type: Bug Components:

Build failed in Jenkins: kafka-trunk-jdk9 #326

See Changes: [wangguoz] KAFKA-6461 TableTableJoinIntegrationTest is unstable if caching is -- [...truncated 1.87 MB...] org.apache.kafka.streams.kstream.WindowTest >

[jira] [Resolved] (KAFKA-6421) consumer group can't show when use kafka-consumer-groups tool if the leader of __consumer_offsets partition for this group changed

[ https://issues.apache.org/jira/browse/KAFKA-6421?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-6421. Resolution: Fixed > consumer group can't show when use kafka-consumer-groups tool if the

Re: [DISCUSS] KIP 226 - Dynamic Broker Configuration

Hi Rajini, I think the proposal makes sense. One suggestion: can we just allow the config to be passed? That is, leave out the properties config for now. On Tue, Jan 23, 2018 at 3:01 PM, Rajini Sivaram wrote: > Since we are running out of time to get the whole

KIP-250 Add Support for Quorum-based Producer Acknowledgment

Hey folks. I would like to add a feature to support the quorum-based acknowledgment for the producer request. We have been running a modified version of Kafka on our testing cluster for weeks, the improvement of P999 is significant with very stable latency. Additionally, I have a proposal to

Re: [VOTE] KIP-227: Introduce Fetch Requests that are Incremental to Increase Partition Scalability

With 3 +1 votes from Ismael, Jun, and Jason, the vote passes. Thanks, Colin On Tue, Jan 23, 2018, at 22:17, Colin McCabe wrote: > On Tue, Jan 23, 2018, at 21:47, Ismael Juma wrote: > > Colin, > > > > You get a cumulative count for rates since we added > >

Re: KIP-250 Add Support for Quorum-based Producer Acknowledgment

Hey Litao, Thanks for the KIP. I have one quick comment before you provide more detail on how to select the leader with the largest LEO. Do you think it would make sense to change the default behavior of acks=-1, such that broker will acknowledge the message once the message has been replicated

Re: Regarding : Store stream for infinite time

Thanks Matthias. This is really helpful. Regards, Aman On Wed, Jan 24, 2018 at 2:47 AM, Matthias J. Sax wrote: > It's simpler to set retention time to Long.MAX_VALUE instead of log > compaction and generating unique keys. This effectively disable > retention because the

Build failed in Jenkins: kafka-trunk-jdk9 #327

See Changes: [github] KAFKA-6287; Consumer group command should list simple consumer groups -- [...truncated 1.87 MB...] org.apache.kafka.streams.tools.StreamsResetterTest

Build failed in Jenkins: kafka-trunk-jdk8 #2347

See Changes: [github] KAFKA-6287; Consumer group command should list simple consumer groups -- [...truncated 3.40 MB...] kafka.utils.CoreUtilsTest > testReadInt PASSED

Re: [VOTE] KIP-227: Introduce Fetch Requests that are Incremental to Increase Partition Scalability

Colin, You get a cumulative count for rates since we added https://cwiki.apache.org/confluence/display/KAFKA/KIP-187+-+Add+cumulative+count+metric+for+all+Kafka+rate+metrics Ismael On Tue, Jan 23, 2018 at 4:21 PM, Colin McCabe wrote: > On Tue, Jan 23, 2018, at 11:57, Jun

Re: [VOTE] KIP-227: Introduce Fetch Requests that are Incremental to Increase Partition Scalability

Thanks for the KIP, +1 (binding). On Tue, Dec 19, 2017 at 7:28 PM, Colin McCabe wrote: > Hi all, > > I'd like to start the vote on KIP-227: Incremental Fetch Requests. > > The KIP is here: https://cwiki.apache.org/confluence/display/KAFKA/KIP- >

[DISCUSS] KIP-250 Add Support for Quorum-based Producer Acknowledgment

Hey folks. I would like to add a feature to support the quorum-based acknowledgment for the producer request. We have been running a modified version of Kafka on our testing cluster for weeks, the improvement of P999 is significant with very stable latency. Additionally, I have a proposal to

Re: [VOTE] KIP-227: Introduce Fetch Requests that are Incremental to Increase Partition Scalability

On Tue, Jan 23, 2018, at 21:47, Ismael Juma wrote: > Colin, > > You get a cumulative count for rates since we added > https://cwiki.apache.org/confluence/display/KAFKA/KIP-187+-+Add+cumulative+count+metric+for+all+Kafka+rate+metrics> Oh, good point. C. > Ismael > > On Tue, Jan 23, 2018 at 4:21

Re: [VOTE] KIP-232: Detect outdated metadata using leaderEpoch and partitionEpoch

Hey Guozhang, Thanks much for reviewing the KIP! In the scenario you described, let's assume that broker A has messages with offset up to 10, and broker B has messages with offset up to 20. If consumer wants to consume message with offset 9, it will not receive OffsetOutOfRangeException from

Re: [VOTE] KIP-232: Detect outdated metadata using leaderEpoch and partitionEpoch

Thanks Dong, I made a pass over the wiki and it lgtm. Just a quick question: can we completely eliminate the OffsetOutOfRangeException with this approach? Say if there is consecutive leader changes such that the cached metadata's partition epoch is 1, and the metadata fetch response returns with

[DISCUSS] KIP-250 Add Support for Quorum-based Producer Acknowledgment

Hey folks. I would like to add a feature to support the quorum-based acknowledgment for the producer request. We have been running a modified version of Kafka on our testing cluster for weeks, the improvement of P999 is significant with very stable latency. Additionally, I have a proposal to

Jenkins build is back to normal : kafka-1.0-jdk7 #130

See

Jenkins build is back to normal : kafka-trunk-jdk7 #3112

See

[jira] [Created] (KAFKA-6477) Add Support for Quorum-based Producer Acknowledge

Litao Deng created KAFKA-6477: - Summary: Add Support for Quorum-based Producer Acknowledge Key: KAFKA-6477 URL: https://issues.apache.org/jira/browse/KAFKA-6477 Project: Kafka Issue Type: New

Build failed in Jenkins: kafka-trunk-jdk8 #2346

See Changes: [wangguoz] KAFKA-6461 TableTableJoinIntegrationTest is unstable if caching is -- [...truncated 1.85 MB...]

[jira] [Resolved] (KAFKA-6287) Inconsistent protocol type for empty consumer groups

[ https://issues.apache.org/jira/browse/KAFKA-6287?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-6287. Resolution: Fixed > Inconsistent protocol type for empty consumer groups >

Streams - retry configuration publishing changelog updates?

I'm seeing this timeout in my Kafka Streams application logs: Caused by: org.apache.kafka.common.errors.TimeoutException: Expiring 7 record(s) for my-table-changelog-18: 30011 ms has passed since last append What kind of retries to I get by default publishing to changelogs? I have these

Re: [VOTE] KIP-212: Enforce set of legal characters for connector names

+1 On Tue, Jan 23, 2018 at 2:01 PM, Gwen Shapira wrote: > +1 (binding) > > On Tue, Jan 23, 2018, 1:17 PM Randall Hauch wrote: > > > +1 (non-binding) > > > > On Mon, Jan 22, 2018 at 6:35 PM, Sönke Liebau < > > soenke.lie...@opencore.com.invalid> wrote: > > >

[jira] [Resolved] (KAFKA-6336) when using assign() with kafka consumer the KafkaConsumerGroup command doesnt show those consumers

[ https://issues.apache.org/jira/browse/KAFKA-6336?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson resolved KAFKA-6336. Resolution: Fixed > when using assign() with kafka consumer the KafkaConsumerGroup command

Regarding : Store stream for infinite time

Hi All, We have a use case to store stream for infinite time (given we have enough storage). We are planning to solve this by Log Compaction. If each message key is unique and Log compaction is enabled, it will store whole stream for infinite time. Just wanted to check if my assumption is

Re: [VOTE] KIP-229: DeleteGroups API

Thanks to those who provided feedback and/or voted on this KIP. The KIP passes with the following vote count: * 3 binding +1's by Rajini, Guozhang, Jason * 5 non-binding +1's by Jeff, Ted, Edo, Mickael, Colin Regards. --Vahid From: "Vahid S Hashemian" To:

Re: Regarding : Store stream for infinite time

Thanks Svante. Regards, Aman On Tue, Jan 23, 2018 at 11:38 PM, Svante Karlsson wrote: > Yes, it will store the last value for each key > > 2018-01-23 18:30 GMT+01:00 Aman Rastogi : > > > Hi All, > > > > We have a use case to store stream for

Re: [VOTE] KIP-212: Enforce set of legal characters for connector names

Thanks for the KIP! +1 (non-binding) Konstantine On Tue, Jan 23, 2018 at 10:48 AM, Ewen Cheslack-Postava wrote: > +1 (binding) > > On Tue, Jan 23, 2018 at 7:28 AM, Matt Farmer wrote: > > > +1 from me (non-binding) =) > > > > > On Jan 22, 2018, at 7:35 PM,

Re: Official Kafka Gitter Room?

Hi, Gitter is used on apache spark project : Spark with Scala/Lobby - A place to discuss and ask questions about using Scala for Spark programming. Regards, JL

Re: Regarding : Store stream for infinite time

Yes, it will store the last value for each key 2018-01-23 18:30 GMT+01:00 Aman Rastogi : > Hi All, > > We have a use case to store stream for infinite time (given we have enough > storage). > > We are planning to solve this by Log Compaction. If each message key is >

Wiki Page Request Permission

Hey. I would like to add a KIP; however, I even can NOT see the 'Create' button on the header of the KIP home page. Could you please grant me the permission? My username is "bryan.deng" Thanks a lot!

Re: [VOTE] KIP-145: Expose Record Headers in Kafka Connect

Thanks for the votes, everyone. This KIP passes with three binding +1 votes and 4 non-binding +1s (including mine). Randall On Tue, Jan 23, 2018 at 3:19 PM, Michael Pearce wrote: > +1 (non-binding) > > I personally think the current proposed Converters described in the

Re: KIP-250 Add Support for Quorum-based Producer Acknowledgment

I agree with Dong, we should see if it's possible to change the default behavior so that as soon as min.insync.replicas brokers respond than the broker acknowledges the message back to the client without waiting for additional brokers who are in the in-sync replica list to respond. (I actually

Re: [VOTE] KIP-212: Enforce set of legal characters for connector names

Thanks, Sönke. This KIP passes with three binding +1s from Ewen, Gwen, and Jason. I've updated the KIP pages accordingly. On Tue, Jan 23, 2018 at 10:14 PM, Jason Gustafson wrote: > +1 > > On Tue, Jan 23, 2018 at 2:01 PM, Gwen Shapira wrote: > > > +1

Re: [VOTE] KIP-232: Detect outdated metadata using leaderEpoch and partitionEpoch

"If consumer wants to consume message with offset 16, then consumer must have already fetched message with offset 15" --> this may not be always true right? What if consumer just call seek(16) after construction and then poll without committed offset ever stored before? Admittedly it is rare but

Re: [DISCUSS] KIP-211: Revise Expiration Semantics of Consumer Group Offsets

Bumping this as I'd like to see it land... It's one of the "features" that tends to catch Kafka n00bs unawares and typically results in message skippage/loss, vs the proposed solution is much more intuitive behavior. Plus it's more wire efficient because consumers no longer need to commit

答复: KIP-250 Add Support for Quorum-based Producer Acknowledgment

Currently, with holding the assigned replicas(AR) for all partitions, controller is now able to elect new leaders by selecting the first replica of AR which occurs in both live replica set and ISR. If switching to the LEO-based strategy, controller context might need to be enriched or

Re: [VOTE] KIP-212: Enforce set of legal characters for connector names

+1 (binding) On Tue, Jan 23, 2018 at 7:28 AM, Matt Farmer wrote: > +1 from me (non-binding) =) > > > On Jan 22, 2018, at 7:35 PM, Sönke Liebau > > > wrote: > > > > All, > > > > this KIP has been discussed for quite some time now and I believe

Re: [VOTE] KIP-227: Introduce Fetch Requests that are Incremental to Increase Partition Scalability

Hi, Collin, Thanks for the updated KIP. +1. Just a minor comment. It seems that it's better for TotalIncrementalFetchSessionsEvicted to be a rate, instead of just an ever-growing count. Jun On Mon, Jan 22, 2018 at 4:35 PM, Jason Gustafson wrote: > > > > What if we want to

Re: [VOTE] KIP-145: Expose Record Headers in Kafka Connect

Hey Randall, It seemed a bit cleaner to me, but I'm not sure if whether there is any advantage to keeping the interfaces decoupled. I'd probably suggest going for the simpler API unless you can think of a good reason not to. -Jason On Tue, Jan 23, 2018 at 8:47 AM, Randall Hauch

[jira] [Created] (KAFKA-6473) Add MockProcessorContext to public test-utils

Matthias J. Sax created KAFKA-6473: -- Summary: Add MockProcessorContext to public test-utils Key: KAFKA-6473 URL: https://issues.apache.org/jira/browse/KAFKA-6473 Project: Kafka Issue Type:

Re: Wiki Page Request Permission

Hi, Litao, Thanks for your interest. Just gave you the wiki permission. Jun On Tue, Jan 23, 2018 at 9:33 AM, Litao Deng wrote: > Hey. > > I would like to add a KIP; however, I even can NOT see the 'Create' button > on the header of the KIP home page. > > Could

Re: [VOTE] KIP-231: Improve the Required ACL of ListGroups API

Hi Jason, Thanks for the vote and feedback. You are correct, I missed the possibility of using the wildcard permission. I'll update the rejected alternatives section accordingly. Thanks again. --Vahid From: Jason Gustafson To: dev@kafka.apache.org Date:

Re: [VOTE] KIP-212: Enforce set of legal characters for connector names

+1 (non-binding) On Mon, Jan 22, 2018 at 6:35 PM, Sönke Liebau < soenke.lie...@opencore.com.invalid> wrote: > All, > > this KIP has been discussed for quite some time now and I believe we > addressed all major concerns in the current revision, so I'd like to > start a vote. > > KIP can be found

Build failed in Jenkins: kafka-trunk-jdk9 #325

See Changes: [jason] KAFKA-6241; Enable dynamic updates of broker SSL keystore (#4263) -- [...truncated 1.45 MB...] kafka.server.DelayedOperationTest >

Build failed in Jenkins: kafka-trunk-jdk7 #3111

See Changes: [jason] KAFKA-6241; Enable dynamic updates of broker SSL keystore (#4263) -- [...truncated 1.90 MB...] org.apache.kafka.streams.tools.StreamsResetterTest >

[jira] [Created] (KAFKA-6474) Rewrite test to use new public TopologyTestDriver

Matthias J. Sax created KAFKA-6474: -- Summary: Rewrite test to use new public TopologyTestDriver Key: KAFKA-6474 URL: https://issues.apache.org/jira/browse/KAFKA-6474 Project: Kafka Issue

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

Hey Jun, Do you think the current KIP looks OK? I am wondering if we can open the voting thread. Thanks! Dong On Fri, Jan 19, 2018 at 3:08 PM, Dong Lin wrote: > Hey Jun, > > I think we can probably have a static method in Util class to decode the > byte[]. Both

Re: Regarding : Store stream for infinite time

It's simpler to set retention time to Long.MAX_VALUE instead of log compaction and generating unique keys. This effectively disable retention because the value is huge. The advantage is, that you don't need to worry about generating unique keys. -Matthias On 1/23/18 10:39 AM, Aman Rastogi

RE: [VOTE] KIP-145: Expose Record Headers in Kafka Connect

+1 (non-binding) I personally think the current proposed Converters described in the KIP are good, as Randall states it keeps it more in line with the pattern of the Converter methods, I think this is a good reason. -Original Message- From: Jason Gustafson [mailto:ja...@confluent.io]

Re: [VOTE] KIP-231: Improve the Required ACL of ListGroups API

+1. It's a little odd that we are still left with the anomalous cluster permission for ListGroups. You mention in the rejected alternatives that reducing permissions would make it impossible for admins to list groups, but I'm not sure why that would be the case. Wouldn't a wildcard group describe

Re: [VOTE] KIP-145: Expose Record Headers in Kafka Connect

I mostly just followed the pattern of the Converter methods, which also take the individual components. Really, the only advantage of the current approach is that the HeaderConverter implementations are a bit more decoupled as they are not aware of the Header/Headers API nor the implementation

Re: [VOTE] KIP-212: Enforce set of legal characters for connector names

+1 from me (non-binding) =) > On Jan 22, 2018, at 7:35 PM, Sönke Liebau > wrote: > > All, > > this KIP has been discussed for quite some time now and I believe we > addressed all major concerns in the current revision, so I'd like to > start a vote. > >

Re: [DISCUSS] KIP 226 - Dynamic Broker Configuration

Since we are running out of time to get the whole ConfigCommand converted to using the new AdminClient for 1.1.0 (KIP-248), we need a way to enable ConfigCommand to handle broker config updates (implemented by KIP-226). As a simple first step, it would make sense to use the existing ConfigCommand

Re: 1.1 KIPs

Hi, Today is the KIP deadline. Tomorrow I'll update the release page with any KIPS that have recently been voted on and accepted according to the process that can be found here: https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Improvement+Proposals#KafkaImprovementProposals-Process

[jira] [Created] (KAFKA-6476) Document dynamic config update

Rajini Sivaram created KAFKA-6476: - Summary: Document dynamic config update Key: KAFKA-6476 URL: https://issues.apache.org/jira/browse/KAFKA-6476 Project: Kafka Issue Type: Sub-task

[jira] [Created] (KAFKA-6475) ConfigException on the broker results in UnknownServerException in the admin client

Xavier Léauté created KAFKA-6475: Summary: ConfigException on the broker results in UnknownServerException in the admin client Key: KAFKA-6475 URL: https://issues.apache.org/jira/browse/KAFKA-6475

Re: [VOTE] KIP-212: Enforce set of legal characters for connector names

+1 (binding) On Tue, Jan 23, 2018, 1:17 PM Randall Hauch wrote: > +1 (non-binding) > > On Mon, Jan 22, 2018 at 6:35 PM, Sönke Liebau < > soenke.lie...@opencore.com.invalid> wrote: > > > All, > > > > this KIP has been discussed for quite some time now and I believe we > >

[jira] [Resolved] (KAFKA-6461) TableTableJoinIntegrationTest is unstable if caching is enabled

[ https://issues.apache.org/jira/browse/KAFKA-6461?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax resolved KAFKA-6461. Resolution: Fixed > TableTableJoinIntegrationTest is unstable if caching is enabled >

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

Hey Jun, Jason, Thanks much for all the review! I will open the voting thread. Regards, Dong On Tue, Jan 23, 2018 at 3:37 PM, Jun Rao wrote: > Hi, Dong, > > The current KIP looks good to me. > > Thanks, > > Jun > > On Tue, Jan 23, 2018 at 12:29 PM, Dong Lin

Re: [VOTE] KIP-86: Configurable SASL callback handlers

Hi, Rajini, Thanks for the KIP. +1 from me. Jun On Thu, Jan 18, 2018 at 8:58 AM, tao xiao wrote: > +1 (non-binding) > > On Fri, 19 Jan 2018 at 00:47 Rajini Sivaram > wrote: > > > Hi all, > > > > I would like to restart the vote for KIP-86: > >

Re: [VOTE] KIP-249: Add Delegation Token Operations to Kafka Admin Client

Hi, Mani, Thanks for the KIP. +1 Jun On Sun, Jan 21, 2018 at 7:44 AM, Manikumar wrote: > Hi All, > > I would like to start a vote on KIP-249 which would add delegation token > operations > to Java Admin Client. > > We have merged DelegationToken API PR recently. We

[VOTE] KIP-232: Detect outdated metadata using leaderEpoch and partitionEpoch

Hi all, I would like to start the voting process for KIP-232: https://cwiki.apache.org/confluence/display/KAFKA/KIP-232%3A+Detect+outdated+metadata+using+leaderEpoch+and+partitionEpoch The KIP will help fix a concurrency issue in Kafka which currently can cause message loss or message

Re: [VOTE] KIP-227: Introduce Fetch Requests that are Incremental to Increase Partition Scalability

On Tue, Jan 23, 2018, at 11:57, Jun Rao wrote: > Hi, Collin, > > Thanks for the updated KIP. +1. Just a minor comment. It seems that it's > better for TotalIncrementalFetchSessionsEvicted to be a rate, instead of > just an ever-growing count. Thanks. Perhaps we can add the rate in addition to

Build failed in Jenkins: kafka-trunk-jdk8 #2345

See Changes: [jason] KAFKA-6241; Enable dynamic updates of broker SSL keystore (#4263) -- [...truncated 123.92 KB...] kafka.tools.ConsoleProducerTest >

Re: [VOTE] KIP-247: Add public test utils for Kafka Streams

One minor change to the KIP. The class TopologyTestDriver will be in package `org.apache.kafka.streams` (instead of `o.a.k.streams.test`). +1 (binding). I am closing this vote as accepted with 3 binding votes (Damian, Guozhang, Matthias) and 2 non-binding votes (Bill, James). Thanks for the

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

Hi, Dong, The current KIP looks good to me. Thanks, Jun On Tue, Jan 23, 2018 at 12:29 PM, Dong Lin wrote: > Hey Jun, > > Do you think the current KIP looks OK? I am wondering if we can open the > voting thread. > > Thanks! > Dong > > On Fri, Jan 19, 2018 at 3:08 PM, Dong