[jira] [Created] (KAFKA-6385) Rack awareness ignored by kafka-reassign-partitions

Gal Barak created KAFKA-6385: Summary: Rack awareness ignored by kafka-reassign-partitions Key: KAFKA-6385 URL: https://issues.apache.org/jira/browse/KAFKA-6385 Project: Kafka Issue Type: Bug

Re: [DISCUSS] KIP 226 - Dynamic Broker Configuration

Hi Jason, Thank you! 2. Updated the KIP: Reconfigurable extends Configurable 4. Changed is_default to config_source in config_entry in the protocol. It will be less confusing that way. The method isDefault() will just return configSource == DEFAULT_CONFIG. Have also included the changes to the

[jira] [Reopened] (KAFKA-6365) How to add a client to list of available clients?

[ https://issues.apache.org/jira/browse/KAFKA-6365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lev Gorodinski reopened KAFKA-6365: --- Hi, looks I don't have edit permissions on

[GitHub] kafka pull request #4297: KAFKA-6317: Maven artifact for kafka should not de...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/4297 ---

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

Hey Dong, > I think it is a good idea to let coordinator do the additional sanity check > to ensure the leader epoch from OffsetCommitRequest never decreases. This > can help us detect bug. The next question will be what should we do if > OffsetCommitRequest provides a smaller leader epoch. One

Re: [VOTE] KIP-243: Make ProducerConfig and ConsumerConfig constructors public

+1 On Tue, Dec 19, 2017 at 1:49 AM, Tom Bentley wrote: > +1 > > On 18 December 2017 at 23:28, Vahid S Hashemian > > wrote: > > > +1 > > > > Thanks for the KIP. > > > > --Vahid > > > > > > > > From: Ted Yu > > To:

Re: [VOTE] KIP-220: Add AdminClient into Kafka Streams' ClientSupplier

Hello, I'm calling for a re-vote on the additional API changes in this KIP. While working on the implementation I found it's better to add the following functions for programmability: * Overloaded KafkaStreams constructor to allow overriding the `Time` object. * Overloaded

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

On Tue, Dec 19, 2017, at 02:16, Jan Filipiak wrote: > Sorry for coming back at this so late. > > > > On 11.12.2017 07:12, Colin McCabe wrote: > > On Sun, Dec 10, 2017, at 22:10, Colin McCabe wrote: > >> On Fri, Dec 8, 2017, at 01:16, Jan Filipiak wrote: > >>> Hi, > >>> > >>> sorry for the late

Re: [VOTE] KIP-243: Make ProducerConfig and ConsumerConfig constructors public

+1 On Tue, Dec 19, 2017 at 12:09 PM, Guozhang Wang wrote: > +1 > > On Tue, Dec 19, 2017 at 1:49 AM, Tom Bentley > wrote: > > > +1 > > > > On 18 December 2017 at 23:28, Vahid S Hashemian < > vahidhashem...@us.ibm.com > > > > > wrote: > > > > > +1 > > >

[jira] [Resolved] (KAFKA-6365) How to add a client to list of available clients?

[ https://issues.apache.org/jira/browse/KAFKA-6365?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Lev Gorodinski resolved KAFKA-6365. --- Resolution: Fixed > How to add a client to list of available clients? >

Jenkins build is back to normal : kafka-trunk-jdk8 #2285

See

[GitHub] kafka pull request #4343: KAFKA-6383: complete shutdown for CREATED StreamTh...

GitHub user rodesai opened a pull request: https://github.com/apache/kafka/pull/4343 KAFKA-6383: complete shutdown for CREATED StreamThreads When transitioning StreamThread from CREATED to PENDING_SHUTDOWN free up resources from the caller, rather than the stream thread,

Re: [DISCUSS] KIP 226 - Dynamic Broker Configuration

4. I wasn't sure what to do, so I left it in there, so that synonyms is a self-contained list. 6. We will never store passwords unencrypted, we will forbid them from being altered if the secret is not configured. Thank you, Rajini On Tue, Dec 19, 2017 at 7:14 PM, Jason Gustafson

[jira] [Created] (KAFKA-6388) Error while trying to roll a segment that already exists

David Hay created KAFKA-6388: Summary: Error while trying to roll a segment that already exists Key: KAFKA-6388 URL: https://issues.apache.org/jira/browse/KAFKA-6388 Project: Kafka Issue Type:

Re: [VOTE] Allowing write access to GitHub repositories (aka GitBox)

GitBox migration will happen today. Committers, please make sure to associate your github ID with your apache.org account via id.apache.org, and make sure to enable 2 factor authentication in GitHub. Ismael On Fri, Dec 15, 2017 at 3:40 PM, Ismael Juma wrote: > Thanks to

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

Hi, Dong, Thanks for the reply. 10. I was actually just thinking the case when the consumer consumes old data. If the current leader epoch is 3 and the consumer is consuming records generated in leader epoch 1, the epoch associated with the offset should be 1. However, as you pointed out, the

Build failed in Jenkins: kafka-trunk-jdk7 #3049

See Changes: [wangguoz] MINOR: Broker down for significant amt of time system test -- Started by an SCM change [EnvInject] - Loading node environment variables. Building

Re: [VOTE] KIP-220: Add AdminClient into Kafka Streams' ClientSupplier

+1 On 12/19/17 9:23 AM, Guozhang Wang wrote: > Hello, > > I'm calling for a re-vote on the additional API changes in this KIP. While > working on the implementation I found it's better to add the following > functions for programmability: > > * Overloaded KafkaStreams constructor to allow

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

Hey Dong, One more thought came to mind. Have you considered edge cases around topic deletion? I think currently if a topic is deleted and then re-created, the leader epoch will start back at the beginning. It seems like that could cause trouble for this solution. One thing that helps is that we

[GitHub] kafka pull request #4313: MINOR: broker down for significant amt of time sys...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/4313 ---

Re: [DISCUSS] KIP 226 - Dynamic Broker Configuration

Hi Rajini, 4. Changed is_default to config_source in config_entry in the protocol. It > will be less confusing that way. The method isDefault() will just > return configSource > Would we still include the active config in the list of synonyms? 6. It is a nice idea to have an automatically

Re: [DISCUSS] KIP-240: AdminClient.listReassignments AdminClient.describeReassignments

Hello Tom, when you were working out KIP-236, did you consider migrating the reassignment state from zookeeper to an internal kafka topic, keyed by partition and log compacted? It would allow an admin client and controller to easily subscribe for those changes, without the need to extend the

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

Hey Jason, Thanks for the comments. These make sense. I have updated the KIP to include a new error INVALID_LEADER_EPOCH. This will be a non-retriable error which may be thrown from consumer's API. When application receives this exception, the only choice will be to revert Kafka client library to

Re: [VOTE] KIP-220: Add AdminClient into Kafka Streams' ClientSupplier

+1 On Tue, Dec 19, 2017 at 1:21 PM, Matthias J. Sax wrote: > +1 > > On 12/19/17 9:23 AM, Guozhang Wang wrote: > > Hello, > > > > I'm calling for a re-vote on the additional API changes in this KIP. > While > > working on the implementation I found it's better to add the

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

Hey Jason, Yeah this may sound a bit confusing. Let me explain my thoughts. If there is no bug in the client library, after consumer rebalance or consumer restart, consume will fetch the previously committed offset and fetch the committed metadata until the leader epoch in the metadata >= the

[jira] [Created] (KAFKA-6386) Deprecate KafkaStreams constructor talking StreamsConfig parameter

Matthias J. Sax created KAFKA-6386: -- Summary: Deprecate KafkaStreams constructor talking StreamsConfig parameter Key: KAFKA-6386 URL: https://issues.apache.org/jira/browse/KAFKA-6386 Project: Kafka

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

Hey Dong, Thanks for the updates. Just one question: When application receives > this exception, the only choice will be to revert Kafka client library to > an earlier version. Not sure I follow this. Wouldn't we just restart the consumer? That would cause it to fetch the previous committed

[VOTE] KIP-231: Improve the Required ACL of ListGroups API

I believe the concerns on this KIP have been addressed so far. Therefore, I'd like to start a vote. https://cwiki.apache.org/confluence/display/KAFKA/KIP-231%3A+Improve+the+Required+ACL+of+ListGroups+API Thanks. --Vahid

Re: [DISCUSS] KIP-218: Make KafkaFuture.Function java 8 lambda compatible

Hello Ismael. Thanks for you feedback. > 1. The KIP seems to rely on the pull request for some of the details of the > proposal. Generally, the KIP should stand on its own. Looking back at what I wrote in the KIP, I agree that its style is too descriptive and relies too much on the content of

Re: [DISCUSS] KIP-218: Make KafkaFuture.Function java 8 lambda compatible

Hi Steven, As a general rule, we don't freeze KIPs after the vote passes. It's reasonably common for things to come up during code review, for example. If we think of improvements, we shouldn't refrain from doing them because of of the vote. If we do minor changes after the KIP passes, we usually

Re: [VOTE] KIP-220: Add AdminClient into Kafka Streams' ClientSupplier

+1 On Tue, 19 Dec 2017 at 19:13 Bill Bejeck wrote: > +1 > > On Tue, Dec 19, 2017 at 1:21 PM, Matthias J. Sax > wrote: > > > +1 > > > > On 12/19/17 9:23 AM, Guozhang Wang wrote: > > > Hello, > > > > > > I'm calling for a re-vote on the additional API

[GitHub] kafka pull request #4344: KAFKA-6321: Consolidate calls to KafkaConsumer's `...

GitHub user vahidhashemian opened a pull request: https://github.com/apache/kafka/pull/4344 KAFKA-6321: Consolidate calls to KafkaConsumer's `beginningOffsets()` and `endOffsets()` in ConsumerGroupCommand ### Committer Checklist (excluded from commit message) - [ ] Verify

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

I think you're saying that depending on the bug, in the worst case, you may have to downgrade the client. I think that's fair. Note that one advantage of making this a fatal error is that we'll be more likely to hit unexpected edge cases in system tests. -Jason On Tue, Dec 19, 2017 at 11:26 AM,

Build failed in Jenkins: kafka-trunk-jdk8 #2287

See Changes: [wangguoz] KAFKA-6150: KIP-204 part III; Change repartition topic segment size and -- [...truncated 414.27 KB...] kafka.utils.CoreUtilsTest > testReadInt

Jenkins build is back to normal : kafka-trunk-jdk7 #3050

See

Build failed in Jenkins: kafka-trunk-jdk8 #2286

See Changes: [wangguoz] MINOR: Broker down for significant amt of time system test -- [...truncated 415.01 KB...] kafka.utils.CoreUtilsTest > testReadInt STARTED

[VOTE] KIP-227: Introduce Fetch Requests that are Incremental to Increase Partition Scalability

Hi all, I'd like to start the vote on KIP-227: Incremental Fetch Requests. The KIP is here: https://cwiki.apache.org/confluence/display/KAFKA/KIP-227%3A+Introduce+Incremental+FetchRequests+to+Increase+Partition+Scalability and discussion thread earlier:

Re: [VOTE] KIP-220: Add AdminClient into Kafka Streams' ClientSupplier

+1 On Tue, Dec 19, 2017 at 1:06 PM Damian Guy wrote: > +1 > > On Tue, 19 Dec 2017 at 19:13 Bill Bejeck wrote: > > > +1 > > > > On Tue, Dec 19, 2017 at 1:21 PM, Matthias J. Sax > > wrote: > > > > > +1 > > > > > > On 12/19/17 9:23

[jira] [Created] (KAFKA-6387) Worker's producer and consumer configs should inherit from worker configs

Randall Hauch created KAFKA-6387: Summary: Worker's producer and consumer configs should inherit from worker configs Key: KAFKA-6387 URL: https://issues.apache.org/jira/browse/KAFKA-6387 Project:

Re: [VOTE] KIP-220: Add AdminClient into Kafka Streams' ClientSupplier

Thanks Gwen, Damian, I will go ahead and merge the corresponding PR then. Guozhang On Tue, Dec 19, 2017 at 2:04 PM, Gwen Shapira wrote: > +1 > > On Tue, Dec 19, 2017 at 1:06 PM Damian Guy wrote: > > > +1 > > > > On Tue, 19 Dec 2017 at 19:13 Bill

[GitHub] kafka pull request #4315: KAFKA-6150: KIP-204 part III; Change repartition t...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/4315 ---

Re: [VOTE] Allowing write access to GitHub repositories (aka GitBox)

Forgot the link to the relevant Infra JIRA: https://issues.apache.org/jira/browse/INFRA-15676 On Tue, Dec 19, 2017 at 11:59 PM, Ismael Juma wrote: > GitBox migration will happen today. Committers, please make sure to > associate your github ID with your apache.org account via

Unable to start Sink connectors[Confluent Kafka]

Starting Sink connector on Confluent Kafka setup throws up the below exception, understand that it is because of some configured interceptor, any advice on how to disable it? [2017-12-20 06:51:11,235] ERROR Task {} failed initialization and will not be started.

Re: [DISCUSS] KIP-218: Make KafkaFuture.Function java 8 lambda compatible

Thanks for the KIP, Steven. A few minor comments: 1. The KIP seems to rely on the pull request for some of the details of the proposal. Generally, the KIP should stand on its own. For example, the public interfaces section should include the signature of interfaces and methods being added. 2. Do

Re: [VOTE] KIP-243: Make ProducerConfig and ConsumerConfig constructors public

+1 On 18 December 2017 at 23:28, Vahid S Hashemian wrote: > +1 > > Thanks for the KIP. > > --Vahid > > > > From: Ted Yu > To: dev@kafka.apache.org > Date: 12/18/2017 02:45 PM > Subject:Re: [VOTE] KIP-243: Make ProducerConfig and

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

Sorry for coming back at this so late. On 11.12.2017 07:12, Colin McCabe wrote: On Sun, Dec 10, 2017, at 22:10, Colin McCabe wrote: On Fri, Dec 8, 2017, at 01:16, Jan Filipiak wrote: Hi, sorry for the late reply, busy times :-/ I would ask you one thing maybe. Since the timeout argument

Re: [DISCUSS] KIP-236 Interruptible Partition Reassignment

Hi Jun, 10. Another concern of mine is on consistency with the current pattern. The > current pattern for change notification based on ZK is (1) we first write > the actual value in the entity path and then write the change notification > path, and (2) the change notification path only includes