[GitHub] kafka pull request #4259: KAFKA-5631 : Use Jackson for serialising to JSON

GitHub user umesh9794 opened a pull request: https://github.com/apache/kafka/pull/4259 KAFKA-5631 : Use Jackson for serialising to JSON This PR replaces the existing `Json.encode` to use Jackson serialization. Since the change is spread more than one module, it relies on the

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

I think the discussion may have already cover this but just in case... How does the leader decide when a newly written message is "committed" enough to hand out to consumers? When a message is produced and is stored to the disk of the leader, the message is not considered "committed" until it

[GitHub] kafka pull request #4258: [KAFKA-4499] Add all() and fetchAll() API for quer...

GitHub user ConcurrencyPractitioner opened a pull request: https://github.com/apache/kafka/pull/4258 [KAFKA-4499] Add all() and fetchAll() API for querying *More detailed description of your change, if necessary. The PR title and PR message become the squashed commit

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

Oh, I see the issue now. The broker uses sendfile() and sends some message data without knowing what the ending offset is. To learn that, we would need another index access. However, when we do that index->offset lookup, we know that the next offset- >index lookup (done in the following fetch

答复: REPLY: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer

Jun, Sound reasonable. Already submitted a new patch and the KIP was also updated. Please review again. Thanks.

[jira] [Created] (KAFKA-6265) GlobalKTable missing #queryableStoreName()

Antony Stubbs created KAFKA-6265: Summary: GlobalKTable missing #queryableStoreName() Key: KAFKA-6265 URL: https://issues.apache.org/jira/browse/KAFKA-6265 Project: Kafka Issue Type: Bug

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

Thanks for the KIP, Colin. It is an interesting idea. Thinking about the fetch protocol, at a high level, currently the following conveys two type of information: 1) what partitions I am interested in 2) where I am on those partitions, i.e. offsets An extreme optimization would be letting the

[GitHub] kafka pull request #4257: KAFKA-6174; Add methods in Options classes to keep...

GitHub user lindong28 opened a pull request: https://github.com/apache/kafka/pull/4257 KAFKA-6174; Add methods in Options classes to keep binary compatibility with 0.11 From 0.11 to 1.0, we moved `DescribeClusterOptions timeoutMs(Integer timeoutMs)` from DescribeClusterOptions to

[GitHub] kafka pull request #4256: KAFKA-6328: Fix 1.0.0 upgrade instructions relatin...

GitHub user hachikuji opened a pull request: https://github.com/apache/kafka/pull/4256 KAFKA-6328: Fix 1.0.0 upgrade instructions relating to the message format version The upgrade instructions concerning the message format versions did not account for upgrades from versions prior

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

Hi, Colin, After step 3a, do we need to update the cached offset in the leader to be the last offset in the data returned in the fetch response? If so, we need another offset index lookup since the leader only knows that it gives out X bytes in the fetch response, but not the last offset in those

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

Hi Jay, On Thu, Nov 23, 2017 at 12:01 AM, Jay Kreps wrote: > I was also thinking there could be mechanical improvements that would help > efficiency such as sharing topic name or TopicPartition objects to reduce > the footprint in a flyweight style. Coincidentally, I was

[jira] [Created] (KAFKA-6264) Log cleaner thread may die on legacy segment containing messages whose offsets are too large

Jiangjie Qin created KAFKA-6264: --- Summary: Log cleaner thread may die on legacy segment containing messages whose offsets are too large Key: KAFKA-6264 URL: https://issues.apache.org/jira/browse/KAFKA-6264

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

On Wed, Nov 22, 2017, at 16:01, Jay Kreps wrote: > Hey Colin, > > WRT memory management I think what you are saying is that you would add a > field to the fetch request which would request that the server cache the > set of partitions and the response would have a field indicating whether > that

[jira] [Created] (KAFKA-6263) Expose metric for group metadata loading duration

Jason Gustafson created KAFKA-6263: -- Summary: Expose metric for group metadata loading duration Key: KAFKA-6263 URL: https://issues.apache.org/jira/browse/KAFKA-6263 Project: Kafka Issue

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

Hey Colin, WRT memory management I think what you are saying is that you would add a field to the fetch request which would request that the server cache the set of partitions and the response would have a field indicating whether that happened or not. This would allow a bound on memory. I was

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

On Wed, Nov 22, 2017, at 14:09, Jun Rao wrote: > Hi, Colin, > > When fetching data for a partition, the leader needs to translate the > fetch offset to a position in a log segment with an index lookup. If the fetch > request now also needs to cache the offset for the next fetch request, > there

Jenkins build is back to normal : kafka-trunk-jdk7 #2994

See

Build failed in Jenkins: kafka-trunk-jdk9 #215

See Changes: [junrao] KAFKA-5646; Use KafkaZkClient in DynamicConfigManager and AdminManager -- [...truncated 1.44 MB...] kafka.zk.ZKPathTest >

[GitHub] kafka pull request #4255: KAFKA-6259: Make KafkaStreams.cleanup() clean glob...

GitHub user mjsax opened a pull request: https://github.com/apache/kafka/pull/4255 KAFKA-6259: Make KafkaStreams.cleanup() clean global state directory ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test

Re: Delete Topic to start Fresh

Topic deletion happens async and there is no guarantee when it will happen. You cannot recreate the topic, until is was deleted. Thus a try-catch-retry strategy would work. Or you check if topic still exists before trying to recreate it. -Matthias On 11/22/17 3:00 AM, Rumani Kumari wrote: >

[GitHub] kafka pull request #4254: KAFKA-6074 Use ZookeeperClient in ReplicaManager a...

GitHub user tedyu opened a pull request: https://github.com/apache/kafka/pull/4254 KAFKA-6074 Use ZookeeperClient in ReplicaManager and Partition Replace ZkUtils with KafkaZkClient in ReplicaManager and Partition Utilize existing unit tests ### Committer Checklist

[GitHub] kafka pull request #4166: KAFKA-6074 Use ZookeeperClient in ReplicaManager a...

Github user tedyu closed the pull request at: https://github.com/apache/kafka/pull/4166 ---

Re: REPLY: [DISCUSS]KIP-223 - Add per-topic min lead and per-partition lead metrics to KafkaConsumer

Hi, Hu, There are two types of names. One is the sensor name, which has to be unique. It's fine if the sensor name includes the topic/partition as the prefix since the sensor name is only a string and is not exposed to jmx. The second name is the metric name, which will be used in jmx. Currently,

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

Hi, Colin, When fetching data for a partition, the leader needs to translate the fetch offset to a position in a log segment with an index lookup. If the fetch request now also needs to cache the offset for the next fetch request, there will be an extra offset index lookup. The offset index

[GitHub] kafka pull request #4246: MINOR: improve flaky Streams tests

Github user mjsax closed the pull request at: https://github.com/apache/kafka/pull/4246 ---

[GitHub] kafka pull request #4253: MINOR: improve error message for Streams test

GitHub user mjsax opened a pull request: https://github.com/apache/kafka/pull/4253 MINOR: improve error message for Streams test ### Committer Checklist (excluded from commit message) - [ ] Verify design and implementation - [ ] Verify test coverage and CI build

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

On Wed, Nov 22, 2017, at 13:43, Colin McCabe wrote: > On Wed, Nov 22, 2017, at 13:08, Jay Kreps wrote: > > Okay yeah, what I said didn't really work or make sense. Ismael's > > interpretation is better. > > > > Couple of things to point out: > > > >1. I'm less sure that replication has a

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

On Wed, Nov 22, 2017, at 13:08, Jay Kreps wrote: > Okay yeah, what I said didn't really work or make sense. Ismael's > interpretation is better. > > Couple of things to point out: > >1. I'm less sure that replication has a high partition count and >consumers don't. There are definitely

[jira] [Created] (KAFKA-6262) Consumer should not uses metadata that is older than the existing metadata

Dong Lin created KAFKA-6262: --- Summary: Consumer should not uses metadata that is older than the existing metadata Key: KAFKA-6262 URL: https://issues.apache.org/jira/browse/KAFKA-6262 Project: Kafka

[GitHub] kafka pull request #4194: KAFKA-5646: Use KafkaZkClient in DynamicConfigMana...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/4194 ---

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

On Tue, Nov 21, 2017, at 22:11, Jun Rao wrote: > Hi, Jay, > > I guess in your proposal the leader has to cache the last offset given > back for each partition so that it knows from which offset to serve the next > fetch request. Hi Jun, Just to clarify, the leader has to cache the last offset

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

Okay yeah, what I said didn't really work or make sense. Ismael's interpretation is better. Couple of things to point out: 1. I'm less sure that replication has a high partition count and consumers don't. There are definitely use cases for consumers that subscribe to everything (e.g.

[GitHub] kafka pull request #4252: [WIP] Migrate Streams Dev Guide content to AK

://github.com/joel-hamill/kafka 20171122-migrate-cp-dev-guide Alternatively you can review and apply these changes as the patch at: https://github.com/apache/kafka/pull/4252.patch To close this pull request, make a commit to your master/trunk branch with (at least) the following in the commit

Re: [DISCUSS] KIP 226 - Dynamic Broker Configuration

Hi Tom, No, I am not proposing this as a way to configure replication quotas. When you describe broker configs using AdminClient, you will see all the configs persisted in /configs/brokers/ in ZooKeeper and this includes leader.replication.throttled.rate, follower.replication.throttled.rate etc.

Jenkins build is back to normal : kafka-trunk-jdk8 #2233

See

Build failed in Jenkins: kafka-trunk-jdk9 #214

See Changes: [me] KAFKA-4827: Correctly encode special chars while creating URI objects -- [...truncated 1.42 MB...] kafka.zk.ZKPathTest >

Re: [DISCUSS] KIP 226 - Dynamic Broker Configuration

Hi Rajini, Just to clarify, are you proposing this as a way to configure interbroker throttling/quotas? I don't think you are, just wanted to check (since KIP-179 proposes a different mechanism for setting them which supports their automatic removal). Cheers, Tom On 22 November 2017 at 18:28,

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

Hi Jay, On Tue, Nov 21, 2017, at 19:03, Jay Kreps wrote: > I think the general thrust of this makes a ton of sense. > > I don't love that we're introducing a second type of fetch request. I > think the motivation is for compatibility, right? But isn't that what > versioning s for? Basically to

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

On Tue, Nov 21, 2017, at 18:56, Jun Rao wrote: > Hi, Colin, > > Thanks for the KIP. A few comments below. > > 1. Currently, if num.replica.fetchers is configured with a value larger > than 1, a broker will be using multiple fetcher threads to fetch from > another broker. So, there will be

Re: [DISCUSS] KIP 226 - Dynamic Broker Configuration

I have made an update to the KIP to optionally return all the config synonyms in *DescribeConfigsResponse*. The synonyms are returned in the order of precedence. AlterConfigsResponse will not be modified by the KIP. Since many configs already have various overrides (e.g. topic configs with broker

[GitHub] kafka pull request #4251: KAFKA-6253: Improve sink connector topic regex val...

GitHub user jklukas opened a pull request: https://github.com/apache/kafka/pull/4251 KAFKA-6253: Improve sink connector topic regex validation KAFKA-3073 added topic regex support for sink connectors. The addition requires that you only specify one of topics or topics.regex

[jira] [Resolved] (KAFKA-4827) Kafka connect: error with special characters in connector name

[ https://issues.apache.org/jira/browse/KAFKA-4827?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava resolved KAFKA-4827. -- Resolution: Fixed Fix Version/s: (was: 0.11.0.3)

[GitHub] kafka pull request #4205: KAFKA-4827: Correctly encode special chars while c...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/4205 ---

Re: [VOTE] KIP-201: Rationalising policy interfaces

Hi everyone, I just wanted to highlight to committers that although this KIP has three non-binding votes, it currently lacks any binding votes: Any feedback would be appreciated. Cheers, Tom On 7 November 2017 at 20:42, Stephane Maarek wrote: > Okay makes

Delete Topic to start Fresh

Hi, I have recently started working on kafka. I tried deleting a kafka Topic and as a result the topic is "marked for Deletion", When does this topic actually gets deleted? Can i recreate this topic while it is in status "marked for deletion"? Thanks, Rumani Kumari

Build failed in Jenkins: kafka-trunk-jdk8 #2232

See Changes: [ismael] KAFKA-1044; Eliminate direct and non-optional log4j references from -- [...truncated 385.61 KB...] kafka.api.PlaintextConsumerTest >

[GitHub] kafka pull request #4243: MINOR: Update Powermock to fix PushHttpMetricsRepo...

Github user ewencp closed the pull request at: https://github.com/apache/kafka/pull/4243 ---

[GitHub] kafka pull request #4250: KAFKA-6261: Fix exception thrown by request loggin...

GitHub user ijuma opened a pull request: https://github.com/apache/kafka/pull/4250 KAFKA-6261: Fix exception thrown by request logging if acks=0 Only expect responseAsString to be set if request logging is enabled _and_ responseSend is defined. Also fixed a couple of

Build failed in Jenkins: kafka-trunk-jdk7 #2992

See Changes: [ismael] KAFKA-1044; Eliminate direct and non-optional log4j references from -- [...truncated 385.34 KB...] kafka.network.SocketServerTest >

[jira] [Created] (KAFKA-6261) Request logging throws exception if acks=0

Ismael Juma created KAFKA-6261: -- Summary: Request logging throws exception if acks=0 Key: KAFKA-6261 URL: https://issues.apache.org/jira/browse/KAFKA-6261 Project: Kafka Issue Type: Bug

[jira] [Created] (KAFKA-6260) KafkaStream 1.0.0 does not strarts correctly with Broker 0.11.0.1

Seweryn Habdank-Wojewodzki created KAFKA-6260: - Summary: KafkaStream 1.0.0 does not strarts correctly with Broker 0.11.0.1 Key: KAFKA-6260 URL: https://issues.apache.org/jira/browse/KAFKA-6260

[jira] [Resolved] (KAFKA-1044) change log4j to slf4j

[ https://issues.apache.org/jira/browse/KAFKA-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-1044. Resolution: Fixed Fix Version/s: 1.1.0 > change log4j to slf4j > -- > >

[GitHub] kafka pull request #3477: KAFKA-1044: eliminating log4j from core

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3477 ---

[GitHub] kafka pull request #4249: fix typo in ProducerConfig doc

GitHub user nick-zh opened a pull request: https://github.com/apache/kafka/pull/4249 fix typo in ProducerConfig doc You can merge this pull request into a Git repository by running: $ git pull https://github.com/nick-zh/kafka trunk Alternatively you can review and apply

[jira] [Created] (KAFKA-6259) Make KafkaStreams.cleanup() clean global state directory

Damian Guy created KAFKA-6259: - Summary: Make KafkaStreams.cleanup() clean global state directory Key: KAFKA-6259 URL: https://issues.apache.org/jira/browse/KAFKA-6259 Project: Kafka Issue Type:

Re: [DISCUSS]: KIP-159: Introducing Rich functions to Streams

Jan, i think you comment with respect to filtering is valid, though not for this KIP. We have separate JIRAs for topology optimization of which this falls into. Thanks, Damian On Wed, 22 Nov 2017 at 02:25 Guozhang Wang wrote: > Jan, > > Not sure I understand your argument

Build failed in Jenkins: kafka-trunk-jdk8 #2231

See -- [...truncated 383.90 KB...] kafka.security.auth.ResourceTypeTest > testJavaConversions PASSED kafka.security.auth.ResourceTypeTest > testFromString STARTED

[jira] [Resolved] (KAFKA-6251) Update kafka-configs.sh to use the new AdminClient

[ https://issues.apache.org/jira/browse/KAFKA-6251?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Rajini Sivaram resolved KAFKA-6251. --- Resolution: Duplicate Fix Version/s: (was: 1.1.0) > Update kafka-configs.sh to use

Jenkins build is back to normal : kafka-0.11.0-jdk7 #340

See

Build failed in Jenkins: kafka-trunk-jdk8 #2230

See Changes: [damian.guy] MINOR: improve flaky Streams system test -- [...truncated 385.26 KB...] kafka.security.auth.ResourceTypeTest > testJavaConversions PASSED

Build failed in Jenkins: kafka-trunk-jdk7 #2991

See Changes: [damian.guy] MINOR: improve flaky Streams system test -- [...truncated 386.01 KB...] kafka.log.BrokerCompressionTest > testBrokerSideCompression[16] STARTED

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

Thanks for the KIP, Colin. As others have said, I think something along these lines is definitely useful, we just need to work out the various details. Jun and Jay, I think you are discussing 2 orthogonal things: 1. Should we have a separate IncrementalFetchRequest or do we reuse FetchRequest?

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

We have seen clusters with a few thousand topic getting idle CPU of 20 percent or more. These may be due to these fetch request. It seems your kip would address scalability by a lot in terms of (dormant) partitions, so I'm excited for this change On 22 Nov. 2017 10:01 pm, "Mickael Maison"

Re: [DISCUSS] KIP-227: Introduce Incremental FetchRequests to Increase Partition Scalability

That's an interesting idea. In our clusters, we definitively feel the cost of unused partitions and I think it's one of these areas where Kafka could improve. On Wed, Nov 22, 2017 at 6:11 AM, Jun Rao wrote: > Hi, Jay, > > I guess in your proposal the leader has to cache the

[GitHub] kafka pull request #4244: MINOR: improve flaky Streams system test

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/4244 ---

[VOTE] KIP-225 - Use tags for consumer “records.lag” metrics

Hi, I would like to start the voting thread for KIP-225. This KIP proposes to correct some lag metrics emitted by the consumer. The KIP wiki is here: https://cwiki.apache.org/confluence/x/uaBzB The discussion thread is here: http://search-hadoop.com/m/Kafka/uyzND1F33uL19AYx/threaded Also could