[DISCUSS] KIP-185: Make exactly once in order delivery per partition the default producer setting

2017-08-08 Thread Apurva Mehta
Hi, I've put together a new KIP which proposes to ship Kafka with its strongest delivery guarantees by default. We currently ship with at most once semantics and don't provide any ordering guarantees per partition. The proposal is is to provide exactly once in order delivery per partition by

[GitHub] kafka pull request #3647: KAFKA-4501: Java 9 compilation fixes

2017-08-08 Thread ijuma
GitHub user ijuma opened a pull request: https://github.com/apache/kafka/pull/3647 KAFKA-4501: Java 9 compilation fixes Compilation fixes: - Avoid ambiguity error when appending to Properties in Scala code - Use position() and limit() to fix ambiguity issue - Disable

[GitHub] kafka pull request #3613: KAFKA-2360: Extract producer-specific configs out ...

2017-08-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3613 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] kafka pull request #3645: MINOR: support retrieving cluster_id in system tes...

2017-08-08 Thread xvrl
GitHub user xvrl opened a pull request: https://github.com/apache/kafka/pull/3645 MINOR: support retrieving cluster_id in system tests @ewencp You can merge this pull request into a Git repository by running: $ git pull https://github.com/xvrl/kafka system-test-cluster-id

[GitHub] kafka pull request #3646: MINOR: Remove unneeded error handlers in deprecate...

2017-08-08 Thread hachikuji
GitHub user hachikuji opened a pull request: https://github.com/apache/kafka/pull/3646 MINOR: Remove unneeded error handlers in deprecated request objects These handlers were previously used on the broker to handle uncaught exceptions, but now the broker users the new Java request

[GitHub] kafka pull request #3619: MINOR: Update dependencies for 1.0.0 release

2017-08-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3619 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[DISCUSS] KIP-186: Increase offsets retention default to 7 days

2017-08-08 Thread Ewen Cheslack-Postava
Hi all, I posted a simple new KIP for a problem we see with a lot of users: KIP-186: Increase offsets retention default to 7 days https://cwiki.apache.org/confluence/display/KAFKA/KIP-186%3A+Increase+offsets+retention+default+to+7+days Note that in addition to the KIP text itself, the linked

[GitHub] kafka pull request #3572: MINOR: Remove unused GroupState.state field

2017-08-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3572 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Resolved] (KAFKA-5710) KafkaAdminClient should remove inflight call correctly after response is received

2017-08-08 Thread Ismael Juma (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5710?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-5710. Resolution: Duplicate Duplicate of KAFKA-5658. > KafkaAdminClient should remove inflight call

[GitHub] kafka pull request #3584: KAFKA-5658. Fix AdminClient request timeout handli...

2017-08-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3584 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Jenkins problem with JDK 8 and Scala 2.12 ?

2017-08-08 Thread Paolo Patierno
Hi devs, is there any problems with Jenkins for building with JDK 8 and Scala 2.12. I see the following error in this console log (https://builds.apache.org/job/kafka-pr-jdk8-scala2.12/6618/consoleFull) ERROR: Error fetching remote repo 'origin' hudson.plugins.git.GitException: Failed to

Re: Jenkins problem with JDK 8 and Scala 2.12 ?

2017-08-08 Thread Sönke Liebau
I think the JDK8 Jenkins server had disk space issues yesterday and failed all builds, maybe this is still the same / caused by that as well. On Tue, Aug 8, 2017 at 9:52 AM, Paolo Patierno wrote: > Hi devs, > > is there any problems with Jenkins for building with JDK 8 and

[GitHub] kafka pull request #3642: KafkaAdminClient should remove inflight call corre...

2017-08-08 Thread lindong28
GitHub user lindong28 opened a pull request: https://github.com/apache/kafka/pull/3642 KafkaAdminClient should remove inflight call correctly after response is received You can merge this pull request into a Git repository by running: $ git pull

[jira] [Created] (KAFKA-5710) KafkaAdminClient should remove inflight call correctly after response is received

2017-08-08 Thread Dong Lin (JIRA)
Dong Lin created KAFKA-5710: --- Summary: KafkaAdminClient should remove inflight call correctly after response is received Key: KAFKA-5710 URL: https://issues.apache.org/jira/browse/KAFKA-5710 Project: Kafka

[GitHub] kafka pull request #3642: KafkaAdminClient should remove inflight call corre...

2017-08-08 Thread lindong28
Github user lindong28 closed the pull request at: https://github.com/apache/kafka/pull/3642 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

2017-08-08 Thread Jun Rao
Hi, Dong, I think Tom was suggesting to have the AlterTopicsRequest sent to any broker, which just writes the reassignment json to ZK. The controller will pick up the reassignment and act on it as usual. This should work, right? Having a separate AlterTopicsRequest and AlterReplicaDirRequest

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

2017-08-08 Thread Dong Lin
Hey Jun, Thanks for the comment! Yes, it should work. The tool can send request to any broker and broker can just write the reassignment znode. My previous intuition is that it may be better to only send this request to controller. But I don't have good reasons for this restriction. My

Build failed in Jenkins: kafka-trunk-jdk8 #1891

2017-08-08 Thread Apache Jenkins Server
See Changes: [me] MINOR: support retrieving cluster_id in system tests -- [...truncated 878.30 KB...] kafka.integration.AutoOffsetResetTest >

答复: [DISCUSS] KIP-178: Size-based log directory selection strategy

2017-08-08 Thread Hu Xi
Hi Lin, Yes, it is a problem since it's not easy for us to predict the possible disk spaces each partition occupies in the future. How about the algorithm below: 1. Introduce a data structure to maintain the current free spaces for each log directory in `log.dirs`. Say, a

[GitHub] kafka pull request #3645: MINOR: support retrieving cluster_id in system tes...

2017-08-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3645 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Resolved] (KAFKA-5704) Auto topic creation causes failure with older clusters

2017-08-08 Thread Ewen Cheslack-Postava (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5704?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava resolved KAFKA-5704. -- Resolution: Fixed Fix Version/s: 1.0.0 Issue resolved by pull request

Re: [DISCUSS] KIP-183 - Change PreferredReplicaLeaderElectionCommand to use AdminClient

2017-08-08 Thread Ewen Cheslack-Postava
Thanks for the KIP. Generally the move away from ZK and to native Kafka requests is good, so I'm generally +1 on this. A couple of comments/questions. * You gave the signature electPreferredReplicaLeader(Collection partitions) for the admin client. The old command allows not specifying the topic

Build failed in Jenkins: kafka-0.11.0-jdk7 #263

2017-08-08 Thread Apache Jenkins Server
See Changes: [me] MINOR: support retrieving cluster_id in system tests -- [...truncated 2.43 MB...]

Re: [DISCUSS] KIP-184 Rename LogCleaner and related classes to LogCompactor

2017-08-08 Thread Pranav Maniar
Thanks Guozhang for the suggestion. For now, I have updated KIP incorporating your suggestion. Personally I think implicitly enabling compaction whenever policy is set to compact is more appropriate. Because new users like me will always assume that setting policy to compact will enable

[jira] [Created] (KAFKA-5715) ConsumerGroupCommand failed to show in ascending order for partitions without consumers

2017-08-08 Thread huxihx (JIRA)
huxihx created KAFKA-5715: - Summary: ConsumerGroupCommand failed to show in ascending order for partitions without consumers Key: KAFKA-5715 URL: https://issues.apache.org/jira/browse/KAFKA-5715 Project:

[GitHub] kafka pull request #3641: KAFKA-5704 Corrected Connect distributed startup b...

2017-08-08 Thread asfgit
Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/3641 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

Re: [DISCUSS] KIP-174 - Deprecate and remove internal converter configs in WorkerConfig

2017-08-08 Thread Ewen Cheslack-Postava
Great, looking good. I'd probably be a bit more concrete about the Proposed Changes (e.g., "will log an warning if the config is specified" and "since the JsonConverter is the default, the configs will be removed immediately from the example worker configuration files"). Other than that this LGTM

Re: [DISCUSS] KIP-179: Change ReassignPartitionsCommand to use AdminClient

2017-08-08 Thread Jun Rao
Hi, Tom, Thanks for the KIP. A few minor comments below. 1. Implementation wise, the broker handles adding partitions differently from changing replica assignment. For the former, we directly update the topic path in ZK with the new partitions. For the latter, we write the new partition

[GitHub] kafka pull request #3648: KAFKA-5715: ConsumerGroupCommand should always sho...

2017-08-08 Thread huxihx
GitHub user huxihx opened a pull request: https://github.com/apache/kafka/pull/3648 KAFKA-5715: ConsumerGroupCommand should always show partitions in ascending order Currently, ConsumerGroupCommand shows in ascending order for partitions with active consumer assigned, but failed

Re: 答复: [VOTE] KIP-177: Consumer perf tool should count rebalance time

2017-08-08 Thread Ewen Cheslack-Postava
Hu, The vote period is supposed to give people who might still want to vote -1 a chance to do so, so we'd leave it open until at least 72 hours after the vote was initiated. However, now that enough time has passed feel free to close the vote and move the KIP to Adopted. I'll give it a +1 as

答复: 答复: [VOTE] KIP-177: Consumer perf tool should count rebalance time

2017-08-08 Thread Hu Xi
Ewen, Thanks for the information. Will do that asap. 发件人: Ewen Cheslack-Postava 发送时间: 2017年8月9日 11:33 收件人: dev@kafka.apache.org 主题: Re: 答复: [VOTE] KIP-177: Consumer perf tool should count rebalance time Hu, The vote period is supposed to

Re: [VOTE] KIP-163: Lower the Minimum Required ACL Permission of OffsetFetch

2017-08-08 Thread Ewen Cheslack-Postava
+1, thanks for the KIP! On Sun, Aug 6, 2017 at 5:22 AM, Mickael Maison wrote: > +1 > While it's not fixing the Read/Write inconsistency it's a step in the > right direction > > On Thu, Aug 3, 2017 at 4:46 PM, Jason Gustafson > wrote: > > +1 > > >

Re: [DISCUSS] KIP-184 Rename LogCleaner and related classes to LogCompactor

2017-08-08 Thread Ewen Cheslack-Postava
A simple log message is standard, but the KIP should probably specify what happens when the deprecated config is encountered. Other than that, the change LGTM. Other things that might be worth addressing * log.compactor.min.compaction.lag.ms seems a bit redundant with compactor and compaction.

Re: [VOTE] KIP-176 : Remove deprecated new-consumer option for tools

2017-08-08 Thread Paolo Patierno
Hi Tom, good question. Due to the new version policies, the deprecation will be in the coming 1.0.0 release but then, due to a breaking change on the API, the removal should be on 2.0.0 I guess. Thanks, Paolo Patierno Senior Software Engineer (IoT) @ Red Hat Microsoft MVP on Windows

Build failed in Jenkins: kafka-0.11.0-jdk7 #262

2017-08-08 Thread Apache Jenkins Server
See Changes: [ismael] KAFKA-5658; Fix AdminClient request timeout handling bug resulting in -- [...truncated 859.11 KB...]

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

2017-08-08 Thread Dong Lin
Hey Tom, Thanks for the quick reply. Please see my comment inline. On Tue, Aug 8, 2017 at 11:06 AM, Tom Bentley wrote: > Hi Dong, > > Replies inline, as usual > > > As I originally envisaged it, KIP-179's support for reassigning > partitions > > > > would have

[GitHub] kafka pull request #3644: KAFKA-5711: batch restore should handle deletes

2017-08-08 Thread bbejeck
GitHub user bbejeck opened a pull request: https://github.com/apache/kafka/pull/3644 KAFKA-5711: batch restore should handle deletes You can merge this pull request into a Git repository by running: $ git pull https://github.com/bbejeck/kafka

[jira] [Created] (KAFKA-5714) Allow whitespaces in the principal name

2017-08-08 Thread Alla Tumarkin (JIRA)
Alla Tumarkin created KAFKA-5714: Summary: Allow whitespaces in the principal name Key: KAFKA-5714 URL: https://issues.apache.org/jira/browse/KAFKA-5714 Project: Kafka Issue Type:

[VOTE] KIP-176 : Remove deprecated new-consumer option for tools

2017-08-08 Thread Paolo Patierno
Hi devs, I didn't see any more comments about this KIP. The JIRAs related to the first step (so making --new-consumer as deprecated with warning messages) are merged. I'd like to start a vote for this KIP. Thanks, Paolo Patierno Senior Software Engineer (IoT) @ Red Hat Microsoft MVP on

Re: [DISCUSS] KIP-174 - Deprecate and remove internal converter configs in WorkerConfig

2017-08-08 Thread UMESH CHAUDHARY
Hi Ewen, Sorry, I am bit late in responding this. Thanks for your inputs and I've updated the KIP by adding more details to it. Regards, Umesh On Mon, 31 Jul 2017 at 21:51 Ewen Cheslack-Postava wrote: > On Sun, Jul 30, 2017 at 10:21 PM, UMESH CHAUDHARY

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

2017-08-08 Thread Tom Bentley
> > > > Also, how do you think things would work in the context of KIP-179? Would > > the tool still invoke these requests or would it be done by the broker > > receiving the alterTopics/reassignPartitions protocol call? > > > > My gut feel is that the tool will still invoke these requests. But I

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

2017-08-08 Thread Tom Bentley
Hi Dong, Thanks for your reply. Yeah I agree with you that the total disk capacity can be useful > particularly if it is different across brokers but it is probably of > limited use in most cases. I also expect that most users would have their > own customized tool across to determine the new

Re: [VOTE] KIP-176 : Remove deprecated new-consumer option for tools

2017-08-08 Thread Tom Bentley
The KIP is here for any one, like me, who hasn't seen it yet: https://cwiki.apache.org/confluence/display/KAFKA/KIP-176:+Remove+deprecated+new-consumer+option+for+tools Paolo, the KIP says "On the next release cycle we could totally remove the option." Exactly which release are you proposing that

[jira] [Created] (KAFKA-5711) Bulk Restore Should Handle Deletes

2017-08-08 Thread Bill Bejeck (JIRA)
Bill Bejeck created KAFKA-5711: -- Summary: Bulk Restore Should Handle Deletes Key: KAFKA-5711 URL: https://issues.apache.org/jira/browse/KAFKA-5711 Project: Kafka Issue Type: Bug

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

2017-08-08 Thread Dong Lin
Thanks for your reply. Yes, my original idea is that user can continue to collect the static information for reassignment as they are doing now. It is the status quo. I agree it can be beneficial to have a tool in Kafka to collect other information that may be needed for reassignment so that user

[jira] [Reopened] (KAFKA-5705) Kafka Server start failed and reports "unsafe memory access operation"

2017-08-08 Thread Jason Gustafson (JIRA)
[ https://issues.apache.org/jira/browse/KAFKA-5705?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jason Gustafson reopened KAFKA-5705: > Kafka Server start failed and reports "unsafe memory access operation" >

[jira] [Created] (KAFKA-5712) Expose ProducerConfig in the KafkaProducer clients

2017-08-08 Thread Leo Xuzhang Lin (JIRA)
Leo Xuzhang Lin created KAFKA-5712: -- Summary: Expose ProducerConfig in the KafkaProducer clients Key: KAFKA-5712 URL: https://issues.apache.org/jira/browse/KAFKA-5712 Project: Kafka Issue

Re: [DISCUSS] KIP-113: Support replicas movement between log directories

2017-08-08 Thread Tom Bentley
Hi Dong, Replies inline, as usual > As I originally envisaged it, KIP-179's support for reassigning partitions > > would have more-or-less taken the logic currently in the > > ReassignPartitionsCommand (that is, writing JSON to the > > ZkUtils.ReassignPartitionsPath) > > and put it behind a

[jira] [Created] (KAFKA-5713) Improve '--group' option to understand strings with wildcards

2017-08-08 Thread Alla Tumarkin (JIRA)
Alla Tumarkin created KAFKA-5713: Summary: Improve '--group' option to understand strings with wildcards Key: KAFKA-5713 URL: https://issues.apache.org/jira/browse/KAFKA-5713 Project: Kafka

Way to check if custom SMT has been added to the classpath or even if it i working.

2017-08-08 Thread satyajit vegesna
Hi All, i have created a custom SMT and have deployed. I would like to know if there is a way to check if the transform is working or not.(def not working as the messages are not getting transformed) I am also trying to remote debug using intellij and nothing seam working, as i do not see any

Re: [DISCUSS] KIP-182: Reduce Streams DSL overloads and allow easier use of custom storage engines

2017-08-08 Thread Guozhang Wang
Damian, Thanks for the proposal, I had a few comments on the APIs: 1. Printed#withFile seems not needed, as users should always spec if it is to sysOut or to File at the beginning. In addition as a second thought, I think serdes are not useful for prints anyways since we assume `toString` is

[GitHub] kafka pull request #3643: MINOR: Error when incompatible console producer co...

2017-08-08 Thread aishraj
GitHub user aishraj opened a pull request: https://github.com/apache/kafka/pull/3643 MINOR: Error when incompatible console producer configs. Spinned off from [KAFKA-2526](https://issues.apache.org/jira/browse/KAFKA-2526), to throw an exception if the console producer is passed a

Re: Kafka Client for Swift 4

2017-08-08 Thread Jason Gustafson
You should be able to edit the wiki now. Thanks for the contribution! -Jason On Mon, Aug 7, 2017 at 1:54 PM, Kellan Burket Cummings < kellan.bur...@gmail.com> wrote: > Thanks, Jason, > It's kellan.burket > - Kellan > > On Thu, Aug 3, 2017 at 6:39 PM, Jason Gustafson >