[jira] [Created] (KAFKA-6626) Performance bottleneck in Kafka Connect sendRecords

2018-03-07 Thread JIRA
Maciej BryƄski created KAFKA-6626: - Summary: Performance bottleneck in Kafka Connect sendRecords Key: KAFKA-6626 URL: https://issues.apache.org/jira/browse/KAFKA-6626 Project: Kafka Issue Typ

[jira] [Created] (KAFKA-6625) kafka offset reset when I upgrade kafka from 0.11.0 to 1.0.0 version

2018-03-07 Thread barry010 (JIRA)
barry010 created KAFKA-6625: --- Summary: kafka offset reset when I upgrade kafka from 0.11.0 to 1.0.0 version Key: KAFKA-6625 URL: https://issues.apache.org/jira/browse/KAFKA-6625 Project: Kafka Iss

Re: [VOTE] KIP-186: Increase offsets retention default to 7 days

2018-03-07 Thread Ted Yu
+1 Original message From: Guozhang Wang Date: 3/7/18 9:17 PM (GMT-08:00) To: dev@kafka.apache.org Subject: Re: [VOTE] KIP-186: Increase offsets retention default to 7 days +1 (binding). On Wed, Mar 7, 2018 at 5:04 PM, James Cheng wrote: > +1 (non-binding) > > -James > > >

Re: [VOTE] KIP-186: Increase offsets retention default to 7 days

2018-03-07 Thread Guozhang Wang
+1 (binding). On Wed, Mar 7, 2018 at 5:04 PM, James Cheng wrote: > +1 (non-binding) > > -James > > > On Mar 7, 2018, at 1:20 PM, Jay Kreps wrote: > > > > +1 > > > > I think we can improve this in the future, but this simple change will > > avoid a lot of pain. Thanks for reviving it Ewen. > > >

Re: [DISCUSS] KIP-267: Add Processor Unit Test Support to Kafka Streams Test Utils

2018-03-07 Thread John Roesler
On Wed, Mar 7, 2018 at 8:03 PM, John Roesler wrote: > Thanks Ted, > > Sure thing; I updated the example code in the KIP with a little snippet. > > -John > > On Wed, Mar 7, 2018 at 7:18 PM, Ted Yu wrote: > >> Looks good. >> >> See if you can add punctuator into the sample code. >> >> On Wed, Mar

[DISCUSS] KIP-267: Add Processor Unit Test Support to Kafka Streams Test Utils

2018-03-07 Thread John Roesler
Dear Kafka community, I am proposing KIP-267 to augment the public Streams test utils API. The goal is to simplify testing of Kafka Streams applications. Please find details in the wiki:https://cwiki.apache.org/confluence/display/KAFKA/KIP-267%3A+Add+Processor+Unit+Test+Support+to+Kafka+Streams+T

[jira] [Created] (KAFKA-6624) log segment deletion could cause a disk to be marked offline incorrectly

2018-03-07 Thread Jun Rao (JIRA)
Jun Rao created KAFKA-6624: -- Summary: log segment deletion could cause a disk to be marked offline incorrectly Key: KAFKA-6624 URL: https://issues.apache.org/jira/browse/KAFKA-6624 Project: Kafka I

[jira] [Created] (KAFKA-6623) Consider renaming inefficient RecordBatch operations

2018-03-07 Thread Jason Gustafson (JIRA)
Jason Gustafson created KAFKA-6623: -- Summary: Consider renaming inefficient RecordBatch operations Key: KAFKA-6623 URL: https://issues.apache.org/jira/browse/KAFKA-6623 Project: Kafka Issue

Re: [VOTE] KIP-186: Increase offsets retention default to 7 days

2018-03-07 Thread James Cheng
+1 (non-binding) -James > On Mar 7, 2018, at 1:20 PM, Jay Kreps wrote: > > +1 > > I think we can improve this in the future, but this simple change will > avoid a lot of pain. Thanks for reviving it Ewen. > > -Jay > > On Mon, Mar 5, 2018 at 11:35 AM, Ewen Cheslack-Postava > wrote: > >> I'd

[jira] [Created] (KAFKA-6622) GroupMetadataManager.loadGroupsAndOffsets decompresses record batch needlessly

2018-03-07 Thread radai rosenblatt (JIRA)
radai rosenblatt created KAFKA-6622: --- Summary: GroupMetadataManager.loadGroupsAndOffsets decompresses record batch needlessly Key: KAFKA-6622 URL: https://issues.apache.org/jira/browse/KAFKA-6622 Pr

Re: [DISCUSS] KIP-253: Support in-order message delivery with partition expansion

2018-03-07 Thread Matthias J. Sax
Jun, thanks for your comment. This should actually work for Streams, because we don't rely on producer "hashing" but specify the partition number explicitly on send(). About not allowing to change the number of partition for changelog topics: for Streams, this seems to imply that we need to creat

[DISCUSS] KIP-258: Allow to Store Record Timestamps in RocksDB

2018-03-07 Thread Matthias J. Sax
Hi, I want to propose KIP-258 for the Streams API to allow storing timestamps in RocksDB. This feature is the basis to resolve multiple tickets (issues and feature requests). Looking forward to your comments about this! https://cwiki.apache.org/confluence/display/KAFKA/KIP-258%3A+Allow+to+Store+

Re: [DISCUSS] KIP-253: Support in-order message delivery with partition expansion

2018-03-07 Thread Jay Kreps
Hey Jason, I agree. Even apart from this proposal the partitioning strategy is really an essential part of the metadata for a topic and had we been less lazy we probably would have included it with the topic metadata. I think in terms of grandfathering this in you could have existing topics just

Re: [DISCUSS] KIP-253: Support in-order message delivery with partition expansion

2018-03-07 Thread Guozhang Wang
If we want to maintain ordering than for a pluggable producer-side partitioner, it needs to coherent with the linear hashing scheme, i.e. we effectively restrict what kind of customization users can do. However such restrictions are hard to enforce programmatically unless we change the API, so mayb

Re: [VOTE] KIP-186: Increase offsets retention default to 7 days

2018-03-07 Thread Jay Kreps
+1 I think we can improve this in the future, but this simple change will avoid a lot of pain. Thanks for reviving it Ewen. -Jay On Mon, Mar 5, 2018 at 11:35 AM, Ewen Cheslack-Postava wrote: > I'd like to kick off voting for KIP-186: > https://cwiki.apache.org/confluence/display/KAFKA/KIP- > 1

Re: [ANNOUNCE] Apache Kafka 1.0.1 Released

2018-03-07 Thread Ismael Juma
Thanks for running the release Ewen and great work everyone! Ismael On Tue, Mar 6, 2018 at 1:14 PM, Ewen Cheslack-Postava wrote: > The Apache Kafka community is pleased to announce the release for Apache > Kafka > 1.0.1. > > This is a bugfix release for the 1.0 branch that was first released wi

[jira] [Created] (KAFKA-6621) Update Streams docs with details for UncaughtExceptionHandler

2018-03-07 Thread Matthias J. Sax (JIRA)
Matthias J. Sax created KAFKA-6621: -- Summary: Update Streams docs with details for UncaughtExceptionHandler Key: KAFKA-6621 URL: https://issues.apache.org/jira/browse/KAFKA-6621 Project: Kafka

Question regarding network outage

2018-03-07 Thread Guru Balse
We have a 0.9 kafka cluster consists of 7 kafka-brokers colocated with 7 zookeepers Producers/consumers are going full tilt at this cluster and we measure delays in response time while we run disruptive tests Delays in response time are 'acceptable' (a few seconds max) for the following disruptio

RE: [VOTE] KIP-186: Increase offsets retention default to 7 days

2018-03-07 Thread Skrzypek, Jonathan
+1 (non-binding) Jonathan Skrzypek -Original Message- From: Bill Bejeck [mailto:bbej...@gmail.com] Sent: 06 March 2018 22:03 To: dev@kafka.apache.org Subject: Re: [VOTE] KIP-186: Increase offsets retention default to 7 days +1 Thanks, Bill On Tue, Mar 6, 2018 at 2:27 PM, Matthias J.

Re: [DISCUSS] KIP-253: Support in-order message delivery with partition expansion

2018-03-07 Thread Jason Gustafson
Hi Dong, What is not clear to me is how the use of linear hashing affects the partitioning logic in the producer, which the user is currently allowed to customize through the Partitioner interface. It sounds like we are effectively deprecating that interface since we will only provide ordering gua

Re: [DISCUSS] KIP-257 - Configurable Quota Management

2018-03-07 Thread Rajini Sivaram
Hi Viktor, Thanks for reviewing the KIP. 1. Yes, that is correct. Typically quotas would depend only on the current partition state. But if you did want to track deleted partitions, you can calculate the diff. 2. I can't think of an use case where ISRs or other replica information would be useful

[jira] [Created] (KAFKA-6620) Documentation about "exactly_once" doesn't mention "transaction.state.log.min.isr"

2018-03-07 Thread Daniel Qian (JIRA)
Daniel Qian created KAFKA-6620: -- Summary: Documentation about "exactly_once" doesn't mention "transaction.state.log.min.isr" Key: KAFKA-6620 URL: https://issues.apache.org/jira/browse/KAFKA-6620 Project