[GitHub] kafka pull request #4248: KAFKA-6258; SSLTransportLayer should keep reading ...

GitHub user lindong28 opened a pull request: https://github.com/apache/kafka/pull/4248 KAFKA-6258; SSLTransportLayer should keep reading from socket until either the buffer is full or the socket has no more data You can merge this pull request into a Git repository by running:

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

Hi, Jay, I guess in your proposal the leader has to cache the last offset given back for each partition so that it knows from which offset to serve the next fetch request. This is doable but it means that the leader needs to do an additional index lookup per partition to serve a fetch request.

[jira] [Created] (KAFKA-6258) SSLTransportLayer should keep reading from socket until either the buffer is full or the socket has no more data

Dong Lin created KAFKA-6258: --- Summary: SSLTransportLayer should keep reading from socket until either the buffer is full or the socket has no more data Key: KAFKA-6258 URL:

[GitHub] kafka pull request #4247: KAFKA-6250: Use existing internal topics without r...

GitHub user gavrie opened a pull request: https://github.com/apache/kafka/pull/4247 KAFKA-6250: Use existing internal topics without requiring ACL When using Kafka Connect with a cluster that doesn't allow the user to create topics (due to ACL configuration), Connect fails when

Re: [DISCUSS] KIP 226 - Dynamic Broker Configuration

Hi Ted, You can quote the config name, but it is not necessary for deleting a config since the name doesn't contain any special characters that requires quoting. On Mon, Nov 20, 2017 at 9:20 PM, Ted Yu wrote: > Thanks for the quick response. > > It seems the config

[jira] [Created] (KAFKA-6248) Enable configuration of internal topics of Kafka Streams applications

Tim Van Laer created KAFKA-6248: --- Summary: Enable configuration of internal topics of Kafka Streams applications Key: KAFKA-6248 URL: https://issues.apache.org/jira/browse/KAFKA-6248 Project: Kafka

Re: Adding log4j-extras to Apache Kafka?

Hi Colin, Currently we are moving away from directly referencing log4j to using slf4j instead (KAFKA-1044). As this jira only aims to remove some runtime dependencies we still need more work but eventually users will be able to change to their own implementation. Despite all this the current

[jira] [Created] (KAFKA-6249) Interactive query downtime when node goes down even with standby replicas

Charles Crain created KAFKA-6249: Summary: Interactive query downtime when node goes down even with standby replicas Key: KAFKA-6249 URL: https://issues.apache.org/jira/browse/KAFKA-6249 Project:

[jira] [Resolved] (KAFKA-2864) Bad zookeeper host causes broker to shutdown uncleanly and stall producers

[ https://issues.apache.org/jira/browse/KAFKA-2864?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-2864. Resolution: Duplicate Duplicate of KAFKA-5473. > Bad zookeeper host causes broker to shutdown

[jira] [Created] (KAFKA-6250) Kafka Connect requires permission to create internal topics even if they exist

Gavrie Philipson created KAFKA-6250: --- Summary: Kafka Connect requires permission to create internal topics even if they exist Key: KAFKA-6250 URL: https://issues.apache.org/jira/browse/KAFKA-6250

Build failed in Jenkins: kafka-trunk-jdk8 #2225

See Changes: [ismael] KAFKA-6247; Install Kibosh on Vagrant and fix release downloads in -- [...truncated 385.93 KB...] kafka.utils.CoreUtilsTest > testReadBytes STARTED

[jira] [Created] (KAFKA-6252) A metric named 'XX' already exists, can't register another one.

Alexis Sellier created KAFKA-6252: - Summary: A metric named 'XX' already exists, can't register another one. Key: KAFKA-6252 URL: https://issues.apache.org/jira/browse/KAFKA-6252 Project: Kafka

[jira] [Created] (KAFKA-6251) Update kafka-configs.sh to use the new AdminClient

Rajini Sivaram created KAFKA-6251: - Summary: Update kafka-configs.sh to use the new AdminClient Key: KAFKA-6251 URL: https://issues.apache.org/jira/browse/KAFKA-6251 Project: Kafka Issue

[jira] [Resolved] (KAFKA-4041) kafka unable to reconnect to zookeeper behind an ELB

[ https://issues.apache.org/jira/browse/KAFKA-4041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-4041. Resolution: Duplicate > kafka unable to reconnect to zookeeper behind an ELB >

[jira] [Reopened] (KAFKA-4041) kafka unable to reconnect to zookeeper behind an ELB

[ https://issues.apache.org/jira/browse/KAFKA-4041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma reopened KAFKA-4041: > kafka unable to reconnect to zookeeper behind an ELB >

[jira] [Resolved] (KAFKA-4041) kafka unable to reconnect to zookeeper behind an ELB

[ https://issues.apache.org/jira/browse/KAFKA-4041?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-4041. Resolution: Fixed Marking as a duplicate of KAFKA-5473. In that JIRA, we will recreate the

[GitHub] kafka pull request #4240: KAFKA-6247. Fix system test dependency issues

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/4240 ---

[jira] [Resolved] (KAFKA-6247) Fix system test dependency issues

[ https://issues.apache.org/jira/browse/KAFKA-6247?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-6247. Resolution: Fixed Assignee: Colin P. McCabe Fix Version/s: 1.1.0 > Fix system test

Kafka consumer query.

Hi Team, My Kafka consumer has 2 threads and the number of partitions is let's say 10 so overall 5 partitions per consumer thread. I am saving the time at which a particular record needs to be processed. Now if record1 on partition1 needs to be picked 10 hours from now Thread should move to next

[GitHub] kafka pull request #4243: MINOR: Update Powermock to fix PushHttpMetricsRepo...

GitHub user ewencp opened a pull request: https://github.com/apache/kafka/pull/4243 MINOR: Update Powermock to fix PushHttpMetricsReporterTest failures Fixes test failures where old versions of Powermock don't handle nested classes accessing parent field members when using

[DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

Hi all, I created a KIP to improve the scalability and latency of FetchRequest: https://cwiki.apache.org/confluence/display/KAFKA/KIP-227%3A+Introduce+Incremental+FetchRequests+to+Increase+Partition+Scalability Please take a look. cheers, Colin

[GitHub] kafka pull request #4243: MINOR: Update Powermock to fix PushHttpMetricsRepo...

Github user ewencp closed the pull request at: https://github.com/apache/kafka/pull/4243 ---

[GitHub] kafka pull request #4243: MINOR: Update Powermock to fix PushHttpMetricsRepo...

GitHub user ewencp reopened a pull request: https://github.com/apache/kafka/pull/4243 MINOR: Update Powermock to fix PushHttpMetricsReporterTest failures Fixes test failures where old versions of Powermock don't handle nested classes accessing parent field members when using

Build failed in Jenkins: kafka-trunk-jdk7 #2987

See Changes: [me] KAFKA-6168: Connect Schema comparison is slow for large schemas -- [...truncated 389.55 KB...] kafka.security.auth.SimpleAclAuthorizerTest >

[jira] [Created] (KAFKA-6257) KafkaConsumer was hung when bootstrap servers was not existed

Brian Clark created KAFKA-6257: -- Summary: KafkaConsumer was hung when bootstrap servers was not existed Key: KAFKA-6257 URL: https://issues.apache.org/jira/browse/KAFKA-6257 Project: Kafka

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

Hi, Colin, Thanks for the KIP. A few comments below. 1. Currently, if num.replica.fetchers is configured with a value larger than 1, a broker will be using multiple fetcher threads to fetch from another broker. So, there will be multiple concurrent fetch requests from a given follower broker.

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

I think the general thrust of this makes a ton of sense. I don't love that we're introducing a second type of fetch request. I think the motivation is for compatibility, right? But isn't that what versioning is for? Basically to me although the modification we're making makes sense, the resulting

Build failed in Jenkins: kafka-trunk-jdk8 #2229

See Changes: [wangguoz] KAFKA-6214: enable use of in-memory store for standby tasks -- [...truncated 384.54 KB...] kafka.security.auth.ResourceTypeTest >

Build failed in Jenkins: kafka-trunk-jdk7 #2990

See Changes: [wangguoz] KAFKA-6214: enable use of in-memory store for standby tasks -- [...truncated 1.84 MB...]

[jira] [Resolved] (KAFKA-6168) Connect Schema comparison is slow for large schemas

[ https://issues.apache.org/jira/browse/KAFKA-6168?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava resolved KAFKA-6168. -- Resolution: Fixed Fix Version/s: 1.1.0 Issue resolved by pull request

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

Fill out the JIRA number. bq. If the leader receives an *IncrementalFetchRequest* with a UUID that does not match that of the latest *FetchResponse* *By *latest *FetchResponse, you mean latest response for the broker Id, right ?* *Cheers* On Tue, Nov 21, 2017 at 1:02 PM, Colin McCabe

[jira] [Created] (KAFKA-6253) Improve sink connector topic regex validation

Ewen Cheslack-Postava created KAFKA-6253: Summary: Improve sink connector topic regex validation Key: KAFKA-6253 URL: https://issues.apache.org/jira/browse/KAFKA-6253 Project: Kafka

Build failed in Jenkins: kafka-trunk-jdk8 #2226

See Changes: [me] KAFKA-6168: Connect Schema comparison is slow for large schemas -- [...truncated 1.80 MB...]

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

On Tue, Nov 21, 2017, at 14:35, Ted Yu wrote: > Fill out the JIRA number. OK, will do. > > bq. If the leader receives an *IncrementalFetchRequest* with a UUID that > does not match that of the latest *FetchResponse* > > *By *latest *FetchResponse, you mean latest response for the broker Id, >

Re: Adding log4j-extras to Apache Kafka?

Hi Viktor, I was under the impression that slf4j is just a facade over the underlying log library. So the good thing about moving to slf4j is that it will allow us to more easily migrate to a new log library (logback, log4j2, etc.) in the future if we want. slf4j is also better for library

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

On Tue, Nov 21, 2017, at 15:48, Ted Yu wrote: > For compatibility, I assume that the follower with new code would detect > when the leader doesn't support this feature and fall back to the > existing > full request. > > Cheers In general, the protocol version which the brokers use to communicate

[GitHub] kafka pull request #4244: MINOR: improve flaky Streams system test

GitHub user mjsax opened a pull request: https://github.com/apache/kafka/pull/4244 MINOR: improve flaky Streams system test Handle TimeoutException in Producer callback and retry sending input data ### Committer Checklist (excluded from commit message) - [ ] Verify

[GitHub] kafka pull request #4151: KAFKA-3073: Add topic regex support for Connect si...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/4151 ---

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

Jan, Thanks for explaining the Serde issue! This makes a lot of sense. I discussed with Guozhang about this issue and came up with the following idea that bridges both APIs: We still introduce CombinedKey as a public interface and exploit it to manage the key in the store and the changelog

[jira] [Created] (KAFKA-6254) Introduce Incremental FetchRequests to Increase Partition Scalability

Colin P. McCabe created KAFKA-6254: -- Summary: Introduce Incremental FetchRequests to Increase Partition Scalability Key: KAFKA-6254 URL: https://issues.apache.org/jira/browse/KAFKA-6254 Project:

[GitHub] kafka pull request #4176: KAFKA-6168 Connect Schema comparison is slow for l...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/4176 ---

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

For compatibility, I assume that the follower with new code would detect when the leader doesn't support this feature and fall back to the existing full request. Cheers On Tue, Nov 21, 2017 at 3:44 PM, Colin McCabe wrote: > On Tue, Nov 21, 2017, at 14:35, Ted Yu wrote: > >

[GitHub] kafka pull request #4213: KAFKA-4115: Increasing the heap settings for conne...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/4213 ---

[jira] [Resolved] (KAFKA-2193) Intermittent network + DNS issues can cause brokers to permanently drop out of a cluster

[ https://issues.apache.org/jira/browse/KAFKA-2193?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-2193. Resolution: Duplicate Duplicate of KAFKA-5473. > Intermittent network + DNS issues can cause

[jira] [Resolved] (KAFKA-4871) Kafka doesn't respect TTL on Zookeeper hostname - crash if zookeeper IP changes

[ https://issues.apache.org/jira/browse/KAFKA-4871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-4871. Resolution: Duplicate Duplicate of KAFKA-5473. > Kafka doesn't respect TTL on Zookeeper hostname -

[jira] [Created] (KAFKA-6256) Flaky Unit test: KStreamKTableJoinIntegrationTest.shouldCountClicksPerRegionWithNonZeroByteCache

Matthias J. Sax created KAFKA-6256: -- Summary: Flaky Unit test: KStreamKTableJoinIntegrationTest.shouldCountClicksPerRegionWithNonZeroByteCache Key: KAFKA-6256 URL:

[jira] [Resolved] (KAFKA-6214) Using standby replicas with an in memory state store causes Streams to crash

[ https://issues.apache.org/jira/browse/KAFKA-6214?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-6214. -- Resolution: Fixed Fix Version/s: 1.0.1 1.1.0 Issue resolved by pull

[GitHub] kafka pull request #4239: KAFKA-6214: enable use of in-memory store for stan...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/4239 ---

Build failed in Jenkins: kafka-trunk-jdk7 #2989

See Changes: [me] KAFKA-4115: Increasing the heap settings for Connect scripts -- [...truncated 387.83 KB...] kafka.server.DelayedOperationTest >

Build failed in Jenkins: kafka-trunk-jdk8 #2228

See Changes: [me] KAFKA-4115: Increasing the heap settings for Connect scripts -- [...truncated 387.40 KB...] kafka.security.auth.ResourceTypeTest > testJavaConversions

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

Jan, Not sure I understand your argument that "we still going to present change.oldValue to the filter even though the record context() is for change.newValue". Are you referring to `KTableFilter#process()`? If yes could you point to me which LOC are you concerning about? Guozhang On Mon, Nov

Re: [DISCUSS] KIP-213 Support non-key joining in KTable

Just to clarify, though "CombinedKey" will still be exposed as public APIs it would not be used in any of the returned key types, so users do not need to worry providing a serde for it at all; it will only be used in the Mapper parameter, and internally Streams library would know how to serde it

[jira] [Created] (KAFKA-6255) Add ProduceBench to Trogdor

Colin P. McCabe created KAFKA-6255: -- Summary: Add ProduceBench to Trogdor Key: KAFKA-6255 URL: https://issues.apache.org/jira/browse/KAFKA-6255 Project: Kafka Issue Type: Sub-task

Access to confluence

Hi There, Could you please provide access to cwiki.apache.org so that i can start working on my KIP. I would also need access to https://cwiki.apache.org/confluence/display/KAFKA/Kafka+Streams page. My id: sssanthalin...@kafka.apache.org Thanks, Siva

[GitHub] kafka pull request #4245: KAFKA-6255: Add ProduceBench to Trogdor

GitHub user cmccabe opened a pull request: https://github.com/apache/kafka/pull/4245 KAFKA-6255: Add ProduceBench to Trogdor You can merge this pull request into a Git repository by running: $ git pull https://github.com/cmccabe/kafka KAFKA-6255 Alternatively you can review

Re: Adding log4j-extras to Apache Kafka?

Hi Colin, I think it's reasonable to include log4j-extras with the broker in the next release (1.1.0). In the 2.0.0 timeframe, we may decide to move to log4j2 (or logback), but people can benefit from log4j-extras in the meantime. Ismael On Tue, Nov 21, 2017 at 11:50 PM, Colin McCabe

Build failed in Jenkins: kafka-0.11.0-jdk7 #339

See Changes: [me] MINOR: Update Powermock to fix PushHttpMetricsReporterTest failures -- [...truncated 2.45 MB...] org.apache.kafka.streams.processor.TopologyBuilderTest

Re: Adding log4j-extras to Apache Kafka?

Is there anything that would prevents us from moving directly to log4j2 as the default log backend – independently of our efforts to move remaining pieces to sl4fj? Unless we have some very custom code, it should be possible to rely on log4j-1.2-api.jar to avoid having to migrate everything at

Build failed in Jenkins: kafka-trunk-jdk7 #2988

See Changes: [me] KAFKA-3073: Add topic regex support for Connect sinks -- [...truncated 384.66 KB...] kafka.server.epoch.LeaderEpochFileCacheTest >

Re: Adding log4j-extras to Apache Kafka?

I was under the impression that the log4j2 configuration file syntax was different than that of the original log4j. The log4j 2 documentation seems to confirm this (see https://logging.apache.org/log4j/2.x/manual/migration.html ): > Although the Log4j 2 configuration syntax is different than

Re: Adding log4j-extras to Apache Kafka?

The slf4j conversion is nearly done, I expect it to be merged by tomorrow. The main challenge in moving to log4j2 by default is that the configuration is not compatible. If there's a way to support the old configuration, then we can go straight to log4j2. Ismael On Wed, Nov 22, 2017 at 12:52

Build failed in Jenkins: kafka-trunk-jdk8 #2227

See Changes: [me] KAFKA-3073: Add topic regex support for Connect sinks -- [...truncated 1.80 MB...] org.apache.kafka.streams.integration.RegexSourceIntegrationTest >

[GitHub] kafka pull request #4246: MINOR: improve flaky Streams tests

GitHub user mjsax opened a pull request: https://github.com/apache/kafka/pull/4246 MINOR: improve flaky Streams tests Use TestUtil test directory for state directory instead of default /tmp/kafka-streams ### Committer Checklist (excluded from commit message) - [ ]