Re: [DISCUSS] KIP 226 - Dynamic Broker Configuration

Hi Jun, Thank you for the review. 1. No, I am hoping to migrate partitions to new threads. We just need to ensure they don't run concurrently. 2. AdminClient has a validateOnly option for AlterConfigs. Any exceptions or return value of false from Reconfigurable#validate would fail the

[jira] [Resolved] (KAFKA-6289) NetworkClient should not return internal failed api version responses from poll

[ https://issues.apache.org/jira/browse/KAFKA-6289?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajini Sivaram resolved KAFKA-6289. --- Resolution: Fixed Fix Version/s: 1.0.1 1.1.0 Issue resolved by pull

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

On 08.12.2017 10:43, Ismael Juma wrote: One correction below. On Fri, Dec 8, 2017 at 11:16 AM, Jan Filipiak wrote: We only check max.message.bytes to late to guard against consumer stalling. we dont have a notion of max.networkpacket.size before we allocate the

[VOTE] KIP-218: Make KafkaFuture.Function java 8 lambda compatible

Hello everybody, I think KIP-218 is crystallized enough to start voting. KIP documentation: https://cwiki.apache.org/confluence/display/KAFKA/KIP-218%3A+Make+KafkaFuture.Function+java+8+lambda+compatible Thanks, Steven

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

Hi, sorry for the late reply, busy times :-/ I would ask you one thing maybe. Since the timeout argument seems to be settled I have no further argument form your side except the "i don't want to". Can you see that connection.max.idle.max is the exact time that expresses "We expect the client

[GitHub] kafka pull request #4280: KAFKA-6289: NetworkClient should not expose failed...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/4280 ---

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

One correction below. On Fri, Dec 8, 2017 at 11:16 AM, Jan Filipiak wrote: > We only check max.message.bytes to late to guard against consumer stalling. > we dont have a notion of max.networkpacket.size before we allocate the > bytebuffer to read it into. We do:

[jira] [Created] (KAFKA-6332) Kafka system tests should use nc instead of log grep to detect start-up

Ismael Juma created KAFKA-6332: -- Summary: Kafka system tests should use nc instead of log grep to detect start-up Key: KAFKA-6332 URL: https://issues.apache.org/jira/browse/KAFKA-6332 Project: Kafka

[jira] [Created] (KAFKA-6330) KafkaZkClient request queue time metric

Ismael Juma created KAFKA-6330: -- Summary: KafkaZkClient request queue time metric Key: KAFKA-6330 URL: https://issues.apache.org/jira/browse/KAFKA-6330 Project: Kafka Issue Type: Bug

[jira] [Created] (KAFKA-6331) Transient failure in kafka.api.AdminClientIntegrationTest.testLogStartOffsetCheckpointkafka.api.AdminClientIntegrationTest.testAlterReplicaLogDirs

Guozhang Wang created KAFKA-6331: Summary: Transient failure in kafka.api.AdminClientIntegrationTest.testLogStartOffsetCheckpointkafka.api.AdminClientIntegrationTest.testAlterReplicaLogDirs Key: KAFKA-6331 URL:

[jira] [Resolved] (KAFKA-5002) Stream does't seem to consider partitions for processing which are being consumed

[ https://issues.apache.org/jira/browse/KAFKA-5002?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-5002. -- Resolution: Not A Problem > Stream does't seem to consider partitions for processing which are

Build failed in Jenkins: kafka-trunk-jdk8 #2265

See Changes: [rajinisivaram] KAFKA-6289; NetworkClient should not expose failed internal ApiVersions -- [...truncated 3.37 MB...] kafka.utils.CoreUtilsTest > testCsvList

[jira] [Resolved] (KAFKA-5702) Refactor StreamThread to separate concerns and enable better testability

[ https://issues.apache.org/jira/browse/KAFKA-5702?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-5702. -- Resolution: Fixed > Refactor StreamThread to separate concerns and enable better testability >

[GitHub] kafka pull request #4305: KAFKA-6318: StreamsResetter should return non-zero...

GitHub user shivsantham opened a pull request: https://github.com/apache/kafka/pull/4305 KAFKA-6318: StreamsResetter should return non-zero return code on error If users specify a non-existing topic as input parameter, StreamsResetter only prints out an error message that the topic

Re: [VOTE] KIP-218: Make KafkaFuture.Function java 8 lambda compatible

+1 On Fri, Dec 8, 2017 at 3:49 AM, Steven Aerts wrote: > Hello everybody, > > > I think KIP-218 is crystallized enough to start voting. > > KIP documentation: > > https://cwiki.apache.org/confluence/display/KAFKA/KIP- >

[GitHub] kafka pull request #4306: KAFKA-6331; Fix transient failure in AdminClientIn...

GitHub user lindong28 opened a pull request: https://github.com/apache/kafka/pull/4306 KAFKA-6331; Fix transient failure in AdminClientIntegrationTest.testAlterReplicaLogDirs *More detailed description of your change, if necessary. The PR title and PR message become the

Re: [VOTE] KIP-218: Make KafkaFuture.Function java 8 lambda compatible

+1 On 8 December 2017 at 18:34, Ted Yu wrote: > +1 > > On Fri, Dec 8, 2017 at 3:49 AM, Steven Aerts > wrote: > > > Hello everybody, > > > > > > I think KIP-218 is crystallized enough to start voting. > > > > KIP documentation: > > > >

[GitHub] kafka pull request #4307: KAFKA-6307 mBeanName should be removed before retu...

GitHub user tedyu opened a pull request: https://github.com/apache/kafka/pull/4307 KAFKA-6307 mBeanName should be removed before returning from JmxReporter#removeAttribute() ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation

Re: [VOTE] KIP-218: Make KafkaFuture.Function java 8 lambda compatible

+1 (non binding) Thanks for the KIP On Fri, Dec 8, 2017 at 6:53 PM, Tom Bentley wrote: > +1 > > On 8 December 2017 at 18:34, Ted Yu wrote: > >> +1 >> >> On Fri, Dec 8, 2017 at 3:49 AM, Steven Aerts >> wrote: >> >> > Hello

Re: KStream: Error Reading and Writing Avro records

Cross posted at SO: https://stackoverflow.com/questions/47712933/kstream-error-reading-and-writing-avro-records I put an answer there. -Matthias On 12/7/17 10:29 PM, Somasundaram Sekar wrote: > I’m trying to write avro record that I read from a topic into another > topic, intentions it to

Re: [VOTE] KIP-218: Make KafkaFuture.Function java 8 lambda compatible

Hi Steve, I just posted in the discussion thread, there's just one tiny fix I think would be useful to add while we're making changes to this API. Do you mind having a look? On Fri, Dec 8, 2017 at 11:37 AM Mickael Maison wrote: > +1 (non binding) > Thanks for the KIP >

Re: [DISCUSS] KIP-237: More Controller Health Metrics

Hi, Dong, Thanks for the KIP. It looks reasonable. Just one minor comment. In the following metric, it seems that RequestRate is better than EventRate. kafka.controller:type=ControllerChannelManager,name=EventRateAndQueueTimeMs, brokerId=someId Jun On Wed, Dec 6, 2017 at 6:21 PM, Dong Lin

[jira] [Created] (KAFKA-6337) Error for partition [__consumer_offsets,15] to broker

Abhi created KAFKA-6337: --- Summary: Error for partition [__consumer_offsets,15] to broker Key: KAFKA-6337 URL: https://issues.apache.org/jira/browse/KAFKA-6337 Project: Kafka Issue Type: Bug

Re: [DISCUSS] KIP-236 Interruptible Partition Reassignment

Hi, Tom, Thanks for the KIP. It definitely addresses one of the pain points in partition reassignment. Another issue that it also addresses is the ZK node size limit when writing the reassignment JSON. My only concern is that the KIP needs to create one watcher per reassigned partition. This

[jira] [Created] (KAFKA-6335) SimpleAclAuthorizerTest#testHighConcurrencyModificationOfResourceAcls fails intermittently

Ted Yu created KAFKA-6335: - Summary: SimpleAclAuthorizerTest#testHighConcurrencyModificationOfResourceAcls fails intermittently Key: KAFKA-6335 URL: https://issues.apache.org/jira/browse/KAFKA-6335 Project:

[jira] [Created] (KAFKA-6336) when using assign() with kafka consumer the KafkaConsumerGroup command doesnt show those consumers

Neerja Khattar created KAFKA-6336: - Summary: when using assign() with kafka consumer the KafkaConsumerGroup command doesnt show those consumers Key: KAFKA-6336 URL:

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

Hi, Jiangjie, What I described is almost the same as yours. The only extra thing is to scan the log segment from the identified index entry a bit more to find a file position that ends at a message set boundary and is less than the partition level fetch size. This way, we still preserve the

Re: [DISCUSS] KIP-218: Make KafkaFuture.Function java 8 lambda compatible

Hi Steven, I noticed you are making KafkaFuture.addWaiter(...) public as part of your PR. This is a very useful method to add – and you should mention it in the KIP – however addWaiter currently doesn't guard against exceptions thrown inside of the BiConsumer function, which is something we

Re: [DISCUSS] KIP-232: Detect outdated metadata by adding ControllerMetadataEpoch field

Hi, Dong, Thanks for the reply. A few more points below. For dealing with how to prevent a consumer switching from a new leader to an old leader, you suggestion that refreshes metadata on consumer restart until it sees a metadata version >= the one associated with the offset works too, as long

[GitHub] kafka pull request #4309: KAFKA-4218: Enable access to key in ValueTransform...

GitHub user jeyhunkarimov opened a pull request: https://github.com/apache/kafka/pull/4309 KAFKA-4218: Enable access to key in ValueTransformer and ValueMapper This PR is the partial implementation for KIP-149. As the discussion for this KIP is still ongoing, I made a PR on the

Re: [DISCUSS] KIP 226 - Dynamic Broker Configuration

Hi, Rajini, Thanks for the reply. They all make sense. 5. Got it. Note that currently, only live brokers are registered in ZK. Another thing is that I am not sure that we want every broker to read all broker registrations directly from ZK. It's probably better to have the controller propagate

[GitHub] kafka pull request #3570: KAFKA-4218: KIP-149, Enabling withKey interfaces i...

Github user jeyhunkarimov closed the pull request at: https://github.com/apache/kafka/pull/3570 ---

[jira] [Created] (KAFKA-6338) java.lang.NoClassDefFoundError: org/apache/kafka/common/network/LoginType

Ronald van de Kuil created KAFKA-6338: - Summary: java.lang.NoClassDefFoundError: org/apache/kafka/common/network/LoginType Key: KAFKA-6338 URL: https://issues.apache.org/jira/browse/KAFKA-6338

[GitHub] kafka pull request #4308: MINOR: make addWaiter public and fix exception han...

GitHub user xvrl opened a pull request: https://github.com/apache/kafka/pull/4308 MINOR: make addWaiter public and fix exception handling KafkaFuture.thenApply(...) only allows invoking a callback on normal completion. Making KafkaFuture.addWaiter(...) public makes possible to

[jira] [Created] (KAFKA-6334) Minor documentation typo

Andrew Olson created KAFKA-6334: --- Summary: Minor documentation typo Key: KAFKA-6334 URL: https://issues.apache.org/jira/browse/KAFKA-6334 Project: Kafka Issue Type: Bug Components:

[jira] [Created] (KAFKA-6333) java.awt.headless should not be on commandline

Fabrice Bacchella created KAFKA-6333: Summary: java.awt.headless should not be on commandline Key: KAFKA-6333 URL: https://issues.apache.org/jira/browse/KAFKA-6333 Project: Kafka Issue