Re: [kafka-clients] [VOTE] 0.10.1.0 RC3

[Dana Powers]

+1 -- passes kafka-python integration tests

On Mon, Oct 17, 2016 at 1:28 PM, Jun Rao  wrote:
> Thanks for preparing the release. Verified quick start on scala 2.11 binary.
> +1
>
> Jun
>
> On Fri, Oct 14, 2016 at 4:29 PM, Jason Gustafson  wrote:
>>
>> Hello Kafka users, developers and client-developers,
>>
>> One more RC for 0.10.1.0. We're hoping this is the final one so that we
>> can meet the release target date of Oct. 17 (Monday). Please let me know as
>> soon as possible if you find any major problems.
>>
>> Release plan:
>> https://cwiki.apache.org/confluence/display/KAFKA/Release+Plan+0.10.1.
>>
>> Release notes for the 0.10.1.0 release:
>> http://home.apache.org/~jgus/kafka-0.10.1.0-rc3/RELEASE_NOTES.html
>>
>> *** Please download, test and vote by Monday, Oct 17, 5pm PT
>>
>> Kafka's KEYS file containing PGP keys we use to sign the release:
>> http://kafka.apache.org/KEYS
>>
>> * Release artifacts to be voted upon (source and binary):
>> http://home.apache.org/~jgus/kafka-0.10.1.0-rc3/
>>
>> * Maven artifacts to be voted upon:
>> https://repository.apache.org/content/groups/staging/
>>
>> * Javadoc:
>> http://home.apache.org/~jgus/kafka-0.10.1.0-rc3/javadoc/
>>
>> * Tag to be voted upon (off 0.10.1 branch) is the 0.10.1.0-rc3 tag:
>>
>> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=50f30a44f31fca1bd9189d2814388d51bd56b06b
>>
>> * Documentation:
>> http://kafka.apache.org/0101/documentation.html
>>
>> * Protocol:
>> http://kafka.apache.org/0101/protocol.html
>>
>> * Tests:
>> Unit tests: https://builds.apache.org/job/kafka-0.10.1-jdk7/71/
>> System tests:
>> http://testing.confluent.io/confluent-kafka-0-10-1-system-test-results/?prefix=2016-10-13--001.1476369986--apache--0.10.1--ee212d1/
>>
>> (Note that these tests do not include a couple patches merged today. I
>> will send links to updated test builds as soon as they are available)
>>
>> Thanks,
>>
>> Jason
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "kafka-clients" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to kafka-clients+unsubscr...@googlegroups.com.
>> To post to this group, send email to kafka-clie...@googlegroups.com.
>> Visit this group at https://groups.google.com/group/kafka-clients.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/kafka-clients/CAJDuW%3DBm0HCOjiHiwnW3WJ_i5u_0e4J2G_mZ_KBkB_WEmo7pNg%40mail.gmail.com.
>> For more options, visit https://groups.google.com/d/optout.
>
>
> --
> You received this message because you are subscribed to the Google Groups
> "kafka-clients" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kafka-clients+unsubscr...@googlegroups.com.
> To post to this group, send email to kafka-clie...@googlegroups.com.
> Visit this group at https://groups.google.com/group/kafka-clients.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/kafka-clients/CAFc58G_Atqyc7-O13EGnRNibng5UPo-a_2h00N2%3D%3DMtWktm%3D1g%40mail.gmail.com.
>
> For more options, visit https://groups.google.com/d/optout.

Re: [kafka-clients] [VOTE] 0.10.1.0 RC3

[Manikumar]

+1 (non-binding)

verified quick start and artifacts.

On Sat, Oct 15, 2016 at 4:59 AM, Jason Gustafson  wrote:

> Hello Kafka users, developers and client-developers,
>
> One more RC for 0.10.1.0. We're hoping this is the final one so that we
> can meet the release target date of Oct. 17 (Monday). Please let me know as
> soon as possible if you find any major problems.
>
> Release plan: https://cwiki.apache.org/confluence/display/KAFKA/Rele
> ase+Plan+0.10.1.
>
> Release notes for the 0.10.1.0 release:
> http://home.apache.org/~jgus/kafka-0.10.1.0-rc3/RELEASE_NOTES.html
>
> *** Please download, test and vote by Monday, Oct 17, 5pm PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> http://home.apache.org/~jgus/kafka-0.10.1.0-rc3/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/
>
> * Javadoc:
> http://home.apache.org/~jgus/kafka-0.10.1.0-rc3/javadoc/
>
> * Tag to be voted upon (off 0.10.1 branch) is the 0.10.1.0-rc3 tag:
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
> 50f30a44f31fca1bd9189d2814388d51bd56b06b
>
> * Documentation:
> http://kafka.apache.org/0101/documentation.html
>
> * Protocol:
> http://kafka.apache.org/0101/protocol.html
>
> * Tests:
> Unit tests: https://builds.apache.org/job/kafka-0.10.1-jdk7/71/
> System tests: http://testing.confluent.io/confluent-kafka-
> 0-10-1-system-test-results/?prefix=2016-10-13--001.
> 1476369986--apache--0.10.1--ee212d1/
>
> (Note that these tests do not include a couple patches merged today. I
> will send links to updated test builds as soon as they are available)
>
> Thanks,
>
> Jason
>
> --
> You received this message because you are subscribed to the Google Groups
> "kafka-clients" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kafka-clients+unsubscr...@googlegroups.com.
> To post to this group, send email to kafka-clie...@googlegroups.com.
> Visit this group at https://groups.google.com/group/kafka-clients.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/kafka-clients/CAJDuW%3DBm0HCOjiHiwnW3WJ_i5u_0e4J2G_
> mZ_KBkB_WEmo7pNg%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

[jira] [Updated] (KAFKA-4275) Check of State-Store-assignment to Processor-Nodes is not enabled

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4275?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang updated KAFKA-4275:
-
   Resolution: Fixed
Fix Version/s: 0.10.2.0
   0.10.1.0
   Status: Resolved  (was: Patch Available)

Issue resolved by pull request 1992
[https://github.com/apache/kafka/pull/1992]

> Check of State-Store-assignment to Processor-Nodes is not enabled
> -
>
> Key: KAFKA-4275
> URL: https://issues.apache.org/jira/browse/KAFKA-4275
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
> Fix For: 0.10.1.0, 0.10.2.0
>
>
> In {{ProcessorContextImpl#getStateStores()}} we should check if a store was 
> connected to the processor and thus, if the processor is allowed to access 
> the store. This check is currently disabled.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4275) Check of State-Store-assignment to Processor-Nodes is not enabled

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4275?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15584451#comment-15584451
 ] 

ASF GitHub Bot commented on KAFKA-4275:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1992


> Check of State-Store-assignment to Processor-Nodes is not enabled
> -
>
> Key: KAFKA-4275
> URL: https://issues.apache.org/jira/browse/KAFKA-4275
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Matthias J. Sax
>Assignee: Matthias J. Sax
> Fix For: 0.10.1.0, 0.10.2.0
>
>
> In {{ProcessorContextImpl#getStateStores()}} we should check if a store was 
> connected to the processor and thus, if the processor is allowed to access 
> the store. This check is currently disabled.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1992: KAFKA-4275: Check of State-Store-assignment to Pro...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1992


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Jenkins build is back to normal : kafka-trunk-jdk8 #982

[Apache Jenkins Server]

See 

[jira] [Commented] (KAFKA-1351) String.format is very expensive in Scala

[Nacho Solis (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15584156#comment-15584156
 ] 

Nacho Solis commented on KAFKA-1351:


Thanks [~jjkoshy].

Yes, this is a pet peeve of mine.  Logging can be slow for many reasons.  Not 
only is string formatting a performance hog; but even with lazy evaluation 
things will still take while when you use derived values.

The newer versions of Scala provide a good alternative for this, macros.   
Coming from the C world I always find it surprising when the language (or 
pre-compiler) doesn't provide that type of functionality.  Granted, it can make 
the code hard to follow if used incorrectly, but it can greatly simplify the 
code while still giving you the desired effect.

We could write our own macro-logging layer or we can use something already out 
there like [scala-logging|https://github.com/typesafehub/scala-logging] which 
wraps SLF4J.  It requires Scala 2.11 or higher.

> String.format is very expensive in Scala
> 
>
> Key: KAFKA-1351
> URL: https://issues.apache.org/jira/browse/KAFKA-1351
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.7.2, 0.8.0, 0.8.1
>Reporter: Neha Narkhede
>  Labels: newbie
> Attachments: KAFKA-1351.patch, KAFKA-1351_2014-04-07_18:02:18.patch, 
> KAFKA-1351_2014-04-09_15:40:11.patch
>
>
> As found in KAFKA-1350, logging is causing significant overhead in the 
> performance of a Kafka server. There are several info statements that use 
> String.format which is particularly expensive. We should investigate adding 
> our own version of String.format that merely uses string concatenation under 
> the covers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk7 #1634

[Apache Jenkins Server]

See 

Changes:

[jason] MINOR: Code refactor - cleaning up some boolean assignments

--
[...truncated 1266 lines...]

kafka.admin.TopicCommandTest > testConfigPreservationAcrossPartitionAlteration 
PASSED

kafka.admin.TopicCommandTest > testAlterIfExists STARTED

kafka.admin.TopicCommandTest > testAlterIfExists PASSED

kafka.admin.TopicCommandTest > testDeleteIfExists STARTED

kafka.admin.TopicCommandTest > testDeleteIfExists PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > shouldFailIfBlankArg STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > shouldFailIfBlankArg PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowVerifyWithoutReassignmentOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowVerifyWithoutReassignmentOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithoutBrokersOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithoutBrokersOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowTopicsOptionWithVerify STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowTopicsOptionWithVerify PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithThrottleOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithThrottleOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > shouldFailIfNoArgs STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > shouldFailIfNoArgs PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowExecuteWithoutReassignmentOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowExecuteWithoutReassignmentOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowBrokersListWithVerifyOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowBrokersListWithVerifyOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldCorrectlyParseValidMinimumExecuteOptions STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldCorrectlyParseValidMinimumExecuteOptions PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithReassignmentOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithReassignmentOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldCorrectlyParseValidMinimumGenerateOptions STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldCorrectlyParseValidMinimumGenerateOptions PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithoutBrokersAndTopicsOptions STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithoutBrokersAndTopicsOptions PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowThrottleWithVerifyOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowThrottleWithVerifyOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldAllowThrottleOptionOnExecute STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldAllowThrottleOptionOnExecute PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowExecuteWithBrokers STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowExecuteWithBrokers PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowExecuteWithTopicsOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowExecuteWithTopicsOption PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldCorrectlyParseValidMinimumVerifyOptions STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldCorrectlyParseValidMinimumVerifyOptions PASSED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithoutTopicsOption STARTED

kafka.admin.ReassignPartitionsCommandArgsTest > 
shouldNotAllowGenerateWithoutTopicsOption PASSED

kafka.admin.AdminTest > testBasicPreferredReplicaElection STARTED
ERROR: Could not install GRADLE_2_4_RC_2_HOME
java.lang.NullPointerException
at 
hudson.plugins.toolenv.ToolEnvBuildWrapper$1.buildEnvVars(ToolEnvBuildWrapper.java:46)
at hudson.model.AbstractBuild.getEnvironment(AbstractBuild.java:931)
at hudson.plugins.git.GitSCM.getParamExpandedRepos(GitSCM.java:404)
at 
hudson.plugins.git.GitSCM.compareRemoteRevisionWithImpl(GitSCM.java:609)
at hudson.plugins.git.GitSCM.compareRemoteRevisionWith(GitSCM.java:574)
at hudson.scm.SCM.compareRemoteRevisionWith(SCM.java:381)
at hudson.scm.SCM.poll(SCM.java:398)
at hudson.model.AbstractProject._poll(AbstractProject.java:1446)
at hudson.model.AbstractProject.poll(AbstractProject.java:1349)
at hudson.triggers.SCMTrigger$Runner.runPolling(SCMTrigger.java:528)
at 

[GitHub] kafka pull request #2013: MINOR: Code refactor - cleaning up some boolean as...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/2013


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2013: MINOR: Code refactor - cleaning up some boolean as...

[imandhan]

Github user imandhan closed the pull request at:

https://github.com/apache/kafka/pull/2013


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #2013: MINOR: Code refactor - cleaning up some boolean as...

[imandhan]

GitHub user imandhan reopened a pull request:

https://github.com/apache/kafka/pull/2013

MINOR: Code refactor - cleaning up some boolean assignments



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/imandhan/kafka KAFKA-refactor

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2013.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2013


commit a9d3b4369a11943c389e0ccc20ccfbafee0ce6f4
Author: Ishita Mandhan 
Date:   2016-10-12T00:51:29Z

MINOR: Code Refactor

Refactored some if else statements which had a boolean type and a boolean 
in the conditional, to remove explicitly stated booleans; cleaning up the code 
a little. For example - if (x==1) true else false is changed to (x==1)

commit db02aafdfcbc9b37587abd7a7f84f9d4edd66454
Author: Ishita Mandhan 
Date:   2016-10-12T00:51:29Z

MINOR: Code Refactor

Refactored some if else statements which had a boolean type and a boolean 
in the conditional, to remove explicitly stated booleans; cleaning up the code 
a little. For example - if (x==1) true else false is changed to (x==1)

commit 1813f6e759edeb99b8e91fae7e077afa7522fce4
Author: Ishita Mandhan 
Date:   2016-10-12T01:04:37Z

Merge branch 'KAFKA-refactor' of https://github.com/imandhan/kafka into 
KAFKA-refactor

commit 6050ca9a3c8ec7236df85c2778d593074355e7ce
Author: Ishita Mandhan 
Date:   2016-10-12T01:04:37Z

Merge branch 'KAFKA-refactor' of https://github.com/imandhan/kafka into 
KAFKA-refactor

commit dccea554822f7e41e84f7db035ff881d0022468d
Author: Ishita Mandhan 
Date:   2016-10-12T02:59:08Z

Merge branch 'KAFKA-refactor' of https://github.com/imandhan/kafka into 
KAFKA-refactor




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: KStreams / add support for sink processor with dynamic topics

[Matthias J. Sax]

-BEGIN PGP SIGNED MESSAGE-
Hash: SHA512

Hi,

using DSL you cannot do this. However, if you use Processor API you can.

There are similar question on the mailing list already. For example:
http://search-hadoop.com/m/uyzND1lghNN1tzbf41=kafka+stream+to+new+t
opic+based+on+message+key

As we got this request multiple times already, it might be worth
adding it IMHO. Not sure what the opinion of other is? We should make
sure that the feature gets accepted before you put a lot of effort in
it. :)


- -Matthias

On 10/17/16 2:10 PM, Florian Hussonnois wrote:
> Hi All,
> 
> Currently, it seems not possible with KStream to produce messages
> to topics which are not known until runtime.
> 
> For a new project I am evaluating the Kafka Connect / Kafka
> Streams architecture but without that feature I cannot retain the
> KStreams API.
> 
> Our use case is pretty basic. We have xml messages in input of our 
> topology. Each message is splitted into subtypes and formatted in
> Avro before being sent to a dedicated topic.
> 
> So the output topics depend of the subtype of each message.
> 
> I think it would be nice to add methods into the KStream interface
> to provide such feature.
> 
> If you think that feature would be usefull I can create a jira and 
> contribute to it. Also, do I need to create a new KIP as this
> requires changes on a public API ?
> 
> Thanks,
> 
-BEGIN PGP SIGNATURE-
Comment: GPGTools - https://gpgtools.org

iQIcBAEBCgAGBQJYBWIhAAoJECnhiMLycopPfTQQAI69Uii5xd8KvaEo/Aeqs0Xw
AzdPHekdVoHANzo1h45W1x3/lnyeMU/n2v09Agsz46cxb+Xbz9NOKGqT3v9Ye0Ic
Eyl5yib1B6sWr41rGuYmwDH8zBoC8dPfGZiWhfXL4Sypey3RWzQlVAUWg8Ob4tqF
rFeubMjWp7yopKRe/7n//JHF029hVK/ePk1vdEsI+2lBI4N7q9ONT/1wKkeCAtdd
CCkI2WP/WbHzCcUVmOL41KoqgQFnmrH7BtLH67jumzEIR16H+ZenGZmS1uzde56E
9mEsl4wmAvfB5GJu6y7JnS7FnQotw7pV7ZneQrA2q8eCZHZqs2fkXf+6ZJNYRir+
rysqt8wJG69ZN9bSNO1Q6/fNbRiSjYi0I7JnzkErP6scfDKlf3bWzlw6Ejc0+iUr
Cd0x2m/RlCepVleMT0UZNDlJd0Ml9Q77npP1lyntHVYHjVvtZLdlB5BQYdTMAx3N
KCLZ+WkY2CBKcwh/KuMr9kW2eWSxH89JSwEule+1bN4vSKyBA6vtrwDoshf6N23g
dEhTiY5NsgkvAe1JEK6d7PLN2Tq1Tq4OJNoP8PZlqW+YSFl41klQUblo8yT1jSlF
iCyQS4rgNRabjBs1iZnZNoZ5eodoJMcUyWPhHUYHne+MXuSr1cNNGeNcbS5W0UyE
dPCe2IiY4zErzxW/Mjmw
=4DpY
-END PGP SIGNATURE-

[jira] [Commented] (KAFKA-1351) String.format is very expensive in Scala

[Joel Koshy (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583851#comment-15583851
 ] 

Joel Koshy commented on KAFKA-1351:
---

Yes - this is still an issue. cc [~nsolis]

> String.format is very expensive in Scala
> 
>
> Key: KAFKA-1351
> URL: https://issues.apache.org/jira/browse/KAFKA-1351
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.7.2, 0.8.0, 0.8.1
>Reporter: Neha Narkhede
>  Labels: newbie
> Attachments: KAFKA-1351.patch, KAFKA-1351_2014-04-07_18:02:18.patch, 
> KAFKA-1351_2014-04-09_15:40:11.patch
>
>
> As found in KAFKA-1350, logging is causing significant overhead in the 
> performance of a Kafka server. There are several info statements that use 
> String.format which is particularly expensive. We should investigate adding 
> our own version of String.format that merely uses string concatenation under 
> the covers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-4225) Replication Quotas: Control Leader & Follower Limit Separately

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson resolved KAFKA-4225.

Resolution: Fixed

> Replication Quotas: Control Leader & Follower Limit Separately
> --
>
> Key: KAFKA-4225
> URL: https://issues.apache.org/jira/browse/KAFKA-4225
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.1.0
>Reporter: Ben Stopford
>Assignee: Ben Stopford
> Fix For: 0.10.1.0
>
>
> As we split the throttled replicas into Leader and Follower configs, it makes 
> sense to also split the throttle limit:
> replication.quota.throttled.rate
>=>
> replication.leader.quota.throttled.rate
> & 
> replication.follower.quota.throttled.rate
> So that admins have fine grain control over both sides of the replication 
> process and the properties match for leader/follower applicability.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Reopened] (KAFKA-4225) Replication Quotas: Control Leader & Follower Limit Separately

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson reopened KAFKA-4225:


> Replication Quotas: Control Leader & Follower Limit Separately
> --
>
> Key: KAFKA-4225
> URL: https://issues.apache.org/jira/browse/KAFKA-4225
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.1.0
>Reporter: Ben Stopford
>Assignee: Ben Stopford
> Fix For: 0.10.1.0
>
>
> As we split the throttled replicas into Leader and Follower configs, it makes 
> sense to also split the throttle limit:
> replication.quota.throttled.rate
>=>
> replication.leader.quota.throttled.rate
> & 
> replication.follower.quota.throttled.rate
> So that admins have fine grain control over both sides of the replication 
> process and the properties match for leader/follower applicability.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Closed] (KAFKA-4225) Replication Quotas: Control Leader & Follower Limit Separately

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson closed KAFKA-4225.
--

Not sure why this is no longer marked "resolved," but closing now.

> Replication Quotas: Control Leader & Follower Limit Separately
> --
>
> Key: KAFKA-4225
> URL: https://issues.apache.org/jira/browse/KAFKA-4225
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.1.0
>Reporter: Ben Stopford
>Assignee: Ben Stopford
> Fix For: 0.10.1.0
>
>
> As we split the throttled replicas into Leader and Follower configs, it makes 
> sense to also split the throttle limit:
> replication.quota.throttled.rate
>=>
> replication.leader.quota.throttled.rate
> & 
> replication.follower.quota.throttled.rate
> So that admins have fine grain control over both sides of the replication 
> process and the properties match for leader/follower applicability.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-4225) Replication Quotas: Control Leader & Follower Limit Separately

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4225?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-4225:
---
Status: Resolved  (was: Closed)

> Replication Quotas: Control Leader & Follower Limit Separately
> --
>
> Key: KAFKA-4225
> URL: https://issues.apache.org/jira/browse/KAFKA-4225
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.1.0
>Reporter: Ben Stopford
>Assignee: Ben Stopford
> Fix For: 0.10.1.0
>
>
> As we split the throttled replicas into Leader and Follower configs, it makes 
> sense to also split the throttle limit:
> replication.quota.throttled.rate
>=>
> replication.leader.quota.throttled.rate
> & 
> replication.follower.quota.throttled.rate
> So that admins have fine grain control over both sides of the replication 
> process and the properties match for leader/follower applicability.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4309) Allow "pluggable" properties in KafkaService in System Tests

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4309?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583527#comment-15583527
 ] 

ASF GitHub Bot commented on KAFKA-4309:
---

GitHub user benstopford opened a pull request:

https://github.com/apache/kafka/pull/2034

KAFKA-4309: Allow "pluggable" properties in KafkaService in System Tests



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/benstopford/kafka 
throttling-system-test-kafka-changes

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2034.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2034


commit 81704e9aac6583b29a615f4095b7ee0e66dec06d
Author: Ben Stopford 
Date:   2016-10-17T21:25:39Z

KAFKA-4309: Allow "pluggable" properties in KafkaService in System Tests.




> Allow "pluggable" properties in KafkaService in System Tests
> 
>
> Key: KAFKA-4309
> URL: https://issues.apache.org/jira/browse/KAFKA-4309
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ben Stopford
>




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2034: KAFKA-4309: Allow "pluggable" properties in KafkaS...

[benstopford]

GitHub user benstopford opened a pull request:

https://github.com/apache/kafka/pull/2034

KAFKA-4309: Allow "pluggable" properties in KafkaService in System Tests



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/benstopford/kafka 
throttling-system-test-kafka-changes

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2034.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2034


commit 81704e9aac6583b29a615f4095b7ee0e66dec06d
Author: Ben Stopford 
Date:   2016-10-17T21:25:39Z

KAFKA-4309: Allow "pluggable" properties in KafkaService in System Tests.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Stream processing meetup at LinkedIn (Sunnyvale) on Wednesday, November 2 at 6pm

[Joel Koshy]

Hi everyone,

We would like to invite you to a Stream Processing Meetup at LinkedIn’s
Sunnyvale campus on Wednesday, November 2 at 6pm.

Please RSVP here (if you intend to attend in person):
http://www.meetup.com/Stream-Processing-Meetup-LinkedIn/events/234454163

We have the following three talks scheduled:

   - *Stream Processing using Apache Samza at LinkedIn: Past, Present, and
   Future*
  - *by Kartik Paramasivam* (LinkedIn)
   - *Kafka Cruise Control: Auto Management of the Kafka Clusters*
  - *by Becket (Jiangjie) Qin* (LinkedIn)
   - *Plumber - Intuit's Samza Ecosystem*
  - *by Shekar Tippur* (Intuit)

Hope to see you there!

Joel

[jira] [Created] (KAFKA-4309) Allow "pluggable" properties in KafkaService in System Tests

[Ben Stopford (JIRA)]

Ben Stopford created KAFKA-4309:
---

 Summary: Allow "pluggable" properties in KafkaService in System 
Tests
 Key: KAFKA-4309
 URL: https://issues.apache.org/jira/browse/KAFKA-4309
 Project: Kafka
  Issue Type: Bug
Reporter: Ben Stopford






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2602) Incorrect descriptions about serializers for ProducerConfig in 0.8.2 and 0.8.3 documentation

[Andrew Musselman (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583514#comment-15583514
 ] 

Andrew Musselman commented on KAFKA-2602:
-

[~hachikuji] [~ijuma] [~junrao] [~granthenke] any guidance whether this is 
valid/still an open issue?

> Incorrect descriptions about serializers for ProducerConfig in 0.8.2 and 
> 0.8.3 documentation 
> -
>
> Key: KAFKA-2602
> URL: https://issues.apache.org/jira/browse/KAFKA-2602
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2.1, 0.9.0.0
>Reporter: Zhuo Liu
>Priority: Minor
>  Labels: newbie
> Fix For: 0.8.2.0
>
> Attachments: KAFKA-2602_2015-09-30_14_22_22.patch
>
>
> The documentation still says "serializer.class" and "key.serializer.class",
> which is no longer correct. 
> We need to add descriptions about "key.serializer" and "value.serializer" in 
> the new Producer Config section (section 3.4).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

KStreams / add support for sink processor with dynamic topics

[Florian Hussonnois]

Hi All,

Currently, it seems not possible with KStream to produce messages to topics
which are not known until runtime.

For a new project I am evaluating the Kafka Connect / Kafka Streams
architecture but without that feature I cannot retain the KStreams API.

Our use case is pretty basic. We have xml messages in input of our
topology.
Each message is splitted into subtypes and formatted in Avro before being
sent to a dedicated topic.

So the output topics depend of the subtype of each message.

I think it would be nice to add methods into the KStream interface to
provide such feature.

If you think that feature would be usefull I can create a jira and
contribute to it.
Also, do I need to create a new KIP as this requires changes on a public
API ?

Thanks,

-- 
Florian HUSSONNOIS

[GitHub] kafka pull request #2033: Documentation for Throttled Replication

[benstopford]

GitHub user benstopford opened a pull request:

https://github.com/apache/kafka/pull/2033

Documentation for Throttled Replication



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/benstopford/kafka throttling-docs

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2033.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2033


commit becce0b29a3db3d70ecf1ddc45138d3403088de4
Author: Ben Stopford 
Date:   2016-10-17T20:58:39Z

KAFKA-4266 - formatting




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [kafka-clients] [VOTE] 0.10.1.0 RC3

[Jun Rao]

Thanks for preparing the release. Verified quick start on scala 2.11
binary. +1

Jun

On Fri, Oct 14, 2016 at 4:29 PM, Jason Gustafson  wrote:

> Hello Kafka users, developers and client-developers,
>
> One more RC for 0.10.1.0. We're hoping this is the final one so that we
> can meet the release target date of Oct. 17 (Monday). Please let me know as
> soon as possible if you find any major problems.
>
> Release plan: https://cwiki.apache.org/confluence/display/KAFKA/Rele
> ase+Plan+0.10.1.
>
> Release notes for the 0.10.1.0 release:
> http://home.apache.org/~jgus/kafka-0.10.1.0-rc3/RELEASE_NOTES.html
>
> *** Please download, test and vote by Monday, Oct 17, 5pm PT
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> http://home.apache.org/~jgus/kafka-0.10.1.0-rc3/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging/
>
> * Javadoc:
> http://home.apache.org/~jgus/kafka-0.10.1.0-rc3/javadoc/
>
> * Tag to be voted upon (off 0.10.1 branch) is the 0.10.1.0-rc3 tag:
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
> 50f30a44f31fca1bd9189d2814388d51bd56b06b
>
> * Documentation:
> http://kafka.apache.org/0101/documentation.html
>
> * Protocol:
> http://kafka.apache.org/0101/protocol.html
>
> * Tests:
> Unit tests: https://builds.apache.org/job/kafka-0.10.1-jdk7/71/
> System tests: http://testing.confluent.io/confluent-kafka-
> 0-10-1-system-test-results/?prefix=2016-10-13--001.
> 1476369986--apache--0.10.1--ee212d1/
>
> (Note that these tests do not include a couple patches merged today. I
> will send links to updated test builds as soon as they are available)
>
> Thanks,
>
> Jason
>
> --
> You received this message because you are subscribed to the Google Groups
> "kafka-clients" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kafka-clients+unsubscr...@googlegroups.com.
> To post to this group, send email to kafka-clie...@googlegroups.com.
> Visit this group at https://groups.google.com/group/kafka-clients.
> To view this discussion on the web visit https://groups.google.com/d/
> msgid/kafka-clients/CAJDuW%3DBm0HCOjiHiwnW3WJ_i5u_0e4J2G_
> mZ_KBkB_WEmo7pNg%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

[jira] [Commented] (KAFKA-2602) Incorrect descriptions about serializers for ProducerConfig in 0.8.2 and 0.8.3 documentation

[Zhuo Liu (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583338#comment-15583338
 ] 

Zhuo Liu commented on KAFKA-2602:
-

Hi [~andrew.musselman], I see one fix here 
https://issues.apache.org/jira/browse/KAFKA-2131,
however, in 0.8.2.x, 0.8.1.x and 0.8.0.x official documentations (e.g., 
http://kafka.apache.org/082/documentation.html),
the doc for value.serializer.class is still missing.
see also https://groups.google.com/forum/#!topic/kafka-clients/Psh1tmVbktY


> Incorrect descriptions about serializers for ProducerConfig in 0.8.2 and 
> 0.8.3 documentation 
> -
>
> Key: KAFKA-2602
> URL: https://issues.apache.org/jira/browse/KAFKA-2602
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2.1, 0.9.0.0
>Reporter: Zhuo Liu
>Priority: Minor
>  Labels: newbie
> Fix For: 0.8.2.0
>
> Attachments: KAFKA-2602_2015-09-30_14_22_22.patch
>
>
> The documentation still says "serializer.class" and "key.serializer.class",
> which is no longer correct. 
> We need to add descriptions about "key.serializer" and "value.serializer" in 
> the new Producer Config section (section 3.4).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3745) Consider adding join key to ValueJoiner interface

[Matthias J. Sax (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583262#comment-15583262
 ] 

Matthias J. Sax commented on KAFKA-3745:


Yes. AFAIK, we got more request for this.

> Consider adding join key to ValueJoiner interface
> -
>
> Key: KAFKA-3745
> URL: https://issues.apache.org/jira/browse/KAFKA-3745
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.0.0
>Reporter: Greg Fodor
>Priority: Minor
>  Labels: api, newbie
>
> In working with Kafka Stream joining, it's sometimes the case that a join key 
> is not actually present in the values of the joins themselves (if, for 
> example, a previous transform generated an ephemeral join key.) In such 
> cases, the actual key of the join is not available in the ValueJoiner 
> implementation to be used to construct the final joined value. This can be 
> worked around by explicitly threading the join key into the value if needed, 
> but it seems like extending the interface to pass the join key along as well 
> would be helpful.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: delete topic causing spikes in fetch/metadata requests

[Zakee]

These errors are possibly caused by delete topic (esp. using v0.8.2.x), refer 
below issues.

https://issues.apache.org/jira/browse/KAFKA-2231
https://issues.apache.org/jira/browse/KAFKA-1194

The replica fetchers still querying their leaders for partitions of deleted 
topic. May need a restart of all brokers and possibly ZK (cleanup of data 
directory) to get rid of these errors. I had the same issue (using 0.10.1.0) 
which claims the issue above is fixed. However, rolling restart of all Zk 
instances only helped get rid of the error in my case.

Thanks.

> On Oct 16, 2016, at 8:37 AM, sunil kalva  wrote:
> 
> Hi
> Can you guys help me with this issue
> 
> On Oct 12, 2016 10:35 PM, "sunil kalva"  wrote:
> 
>> 
>> We are using kafka 0.8.2.2 (client and server), when ever we delete a
>> topic we see lot of errors in broker logs like below, and there is also a
>> spike in fetch/metadata requests. Can i correlate these errors with topic
>> delete or its a known issue. Since there is spike in metadata requests and
>> fetch requests broker throughput has comedown.
>> 
>> 
>> 
>> 
>> --
>> [2016-10-12 16:04:55,054] ERROR [Replica Manager on Broker 4]: Error when
>> processing fetch request for partition [xyz,0] offset 161946645 from
>> consumer with correlation id 0. Possible cause: Request for offset
>> 161946645 but we only have log segments in the range 185487049 to
>> 202816546. (kafka.server.ReplicaManager)
>> [2016-10-12 16:04:55,056] ERROR [Replica Manager on Broker 4]: Error when
>> processing fetch request for partition [xyz,0] offset 161946645 from
>> consumer with correlation id 0. Possible cause: Request for offset
>> 161946645 but we only have log segments in the range 185487049 to
>> 202816546. (kafka.server.ReplicaManager)
>> [2016-10-12 16:04:55,057] ERROR [Replica Manager on Broker 4]: Error when
>> processing fetch request for partition [xyz,0] offset 161946645 from
>> consumer with correlation id 0. Possible cause: Request for offset
>> 161946645 but we only have log segments in the range 185487049 to
>> 202816546. (kafka.server.ReplicaManager)
>> [2016-10-12 16:04:55,059] ERROR [Replica Manager on Broker 4]: Error when
>> processing fetch request for partition [xyz,0] offset 161946645 from
>> consumer with correlation id 0. Possible cause: Request for offset
>> 161946645 but we only have log segments in the range 185487049 to
>> 202816546. (kafka.server.ReplicaManager)
>> [2016-10-12 16:04:55,060] ERROR [Replica Manager on Broker 4]: Error when
>> processing fetch request for partition [xyz,0] offset 161946645 from
>> consumer with correlation id 0. Possible cause: Request for offset
>> 161946645 but we only have log segments in the range 185487049 to
>> 202816546. (kafka.server.ReplicaManager)
>> [2016-10-12 16:04:55,062] ERROR [Replica Manager on Broker 4]: Error when
>> processing fetch request for partition [xyz,0] offset 161946645 from
>> consumer with correlation id 0. Possible cause: Request for offset
>> 161946645 but we only have log segments in the range 185487049 to
>> 202816546. (kafka.server.ReplicaManager)
>> [2016-10-12 16:04:55,064] ERROR [Replica Manager on Broker 4]: Error when
>> processing fetch request for partition [xyz,0] offset 161946645 from
>> consumer with correlation id 0. Possible cause: Request for offset
>> 161946645 but we only have log segments in the range 185487049 to
>> 202816546. (kafka.server.ReplicaManager)
>> [2016-10-12 16:04:55,065] ERROR [Replica Manager on Broker 4]: Error when
>> processing fetch request for partition [xyz,0] offset 161946645 from
>> consumer with correlation id 0. Possible cause: Request for offset
>> 161946645 but we only have log segments in the range 185487049 to
>> 202816546. (kafka.server.ReplicaManager)
>> [2016-10-12 16:04:55,067] ERROR [Replica Manager on Broker 4]: Error when
>> processing fetch request for partition [xyz,0] offset 161946645 from
>> consumer with correlation id 0. Possible cause: Request for offset
>> 161946645 but we only have log segments in the range 185487049 to
>> 202816546. (kafka.server.ReplicaManager)
>> [2016-10-12 16:04:55,068] ERROR [Replica Manager on Broker 4]: Error when
>> processing fetch request for partition [xyz,0] offset 161946645 from
>> consumer with correlation id 0. Possible cause: Request for offset
>> 161946645 but we only have log segments in the range 185487049 to
>> 202816546. (kafka.server.ReplicaManager)
>> [2016-10-12 16:04:55,070] ERROR [Replica Manager on Broker 4]: Error when
>> processing fetch request for partition [xyz,0] offset 161946645 from
>> consumer with correlation id 0. Possible cause: Request for offset
>> 161946645 but we only have log segments in the range 185487049 

[jira] [Commented] (KAFKA-3545) Generalized Serdes for List/Map

[Andrew Musselman (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583196#comment-15583196
 ] 

Andrew Musselman commented on KAFKA-3545:
-

Not sure who to notifiy for closing tickets; [~gwenshap] could you suggest 
someone?

> Generalized Serdes for List/Map
> ---
>
> Key: KAFKA-3545
> URL: https://issues.apache.org/jira/browse/KAFKA-3545
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Greg Fodor
>Priority: Minor
>  Labels: api, newbie
> Fix For: 0.10.1.0
>
>
> In working with Kafka Streams I've found it's often the case I want to 
> perform a "group by" operation, where I repartition a stream based on a 
> foreign key and then do an aggregation of all the values into a single 
> collection, so the stream becomes one where each entry has a value that is a 
> serialized list of values that belonged to the key. (This seems unrelated to 
> the 'group by' operation talked about in KAFKA-3544.) Basically the same 
> typical group by operation found in systems like Cascading.
> In order to create these intermediate list values I needed to define custom 
> avro schemas that simply wrap the elements of interest into a list. It seems 
> desirable that there be some basic facility for constructing simple Serdes of 
> Lists/Maps/Sets of other types, potentially using avro's serialization under 
> the hood. If this existed in the core library it would also enable the 
> addition of higher level operations on streams that can use these Serdes to 
> perform simple operations like the "group by" example I mention.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3545) Generalized Serdes for List/Map

[Greg Fodor (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583149#comment-15583149
 ] 

Greg Fodor commented on KAFKA-3545:
---

Yes

> Generalized Serdes for List/Map
> ---
>
> Key: KAFKA-3545
> URL: https://issues.apache.org/jira/browse/KAFKA-3545
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Greg Fodor
>Priority: Minor
>  Labels: api, newbie
> Fix For: 0.10.1.0
>
>
> In working with Kafka Streams I've found it's often the case I want to 
> perform a "group by" operation, where I repartition a stream based on a 
> foreign key and then do an aggregation of all the values into a single 
> collection, so the stream becomes one where each entry has a value that is a 
> serialized list of values that belonged to the key. (This seems unrelated to 
> the 'group by' operation talked about in KAFKA-3544.) Basically the same 
> typical group by operation found in systems like Cascading.
> In order to create these intermediate list values I needed to define custom 
> avro schemas that simply wrap the elements of interest into a list. It seems 
> desirable that there be some basic facility for constructing simple Serdes of 
> Lists/Maps/Sets of other types, potentially using avro's serialization under 
> the hood. If this existed in the core library it would also enable the 
> addition of higher level operations on streams that can use these Serdes to 
> perform simple operations like the "group by" example I mention.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-3545) Generalized Serdes for List/Map

[Greg Fodor (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Greg Fodor resolved KAFKA-3545.
---
   Resolution: Fixed
Fix Version/s: 0.10.1.0

> Generalized Serdes for List/Map
> ---
>
> Key: KAFKA-3545
> URL: https://issues.apache.org/jira/browse/KAFKA-3545
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Greg Fodor
>Priority: Minor
>  Labels: api, newbie
> Fix For: 0.10.1.0
>
>
> In working with Kafka Streams I've found it's often the case I want to 
> perform a "group by" operation, where I repartition a stream based on a 
> foreign key and then do an aggregation of all the values into a single 
> collection, so the stream becomes one where each entry has a value that is a 
> serialized list of values that belonged to the key. (This seems unrelated to 
> the 'group by' operation talked about in KAFKA-3544.) Basically the same 
> typical group by operation found in systems like Cascading.
> In order to create these intermediate list values I needed to define custom 
> avro schemas that simply wrap the elements of interest into a list. It seems 
> desirable that there be some basic facility for constructing simple Serdes of 
> Lists/Maps/Sets of other types, potentially using avro's serialization under 
> the hood. If this existed in the core library it would also enable the 
> addition of higher level operations on streams that can use these Serdes to 
> perform simple operations like the "group by" example I mention.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2111) Command Line Standardization - Add Help Arguments & List Required Fields

[Andrew Musselman (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583142#comment-15583142
 ] 

Andrew Musselman commented on KAFKA-2111:
-

[~mwarhaftig] any progress on this; are we keeping this open as a placeholder 
still?

> Command Line Standardization - Add Help Arguments & List Required Fields
> 
>
> Key: KAFKA-2111
> URL: https://issues.apache.org/jira/browse/KAFKA-2111
> Project: Kafka
>  Issue Type: Improvement
>  Components: admin
>Reporter: Matt Warhaftig
>Priority: Minor
>  Labels: newbie
>
> KIP-14 is the standardization of tool command line arguments.  As an offshoot 
> of that proposal there are standardization changes that don't need to be part 
> of the KIP since they are less invasive.  They are:
> - Properly format argument descriptions (into sentences) and add any missing 
> "REQUIRED" notes.
> - Add 'help' argument to any top-level tool scripts that were missing it.
> This JIRA is for tracking them.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3545) Generalized Serdes for List/Map

[Andrew Musselman (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3545?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583131#comment-15583131
 ] 

Andrew Musselman commented on KAFKA-3545:
-

[~guozhang] and [~gfodor] looks like the two tickets you mention are resolved; 
can we close this one?

> Generalized Serdes for List/Map
> ---
>
> Key: KAFKA-3545
> URL: https://issues.apache.org/jira/browse/KAFKA-3545
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Greg Fodor
>Priority: Minor
>  Labels: api, newbie
>
> In working with Kafka Streams I've found it's often the case I want to 
> perform a "group by" operation, where I repartition a stream based on a 
> foreign key and then do an aggregation of all the values into a single 
> collection, so the stream becomes one where each entry has a value that is a 
> serialized list of values that belonged to the key. (This seems unrelated to 
> the 'group by' operation talked about in KAFKA-3544.) Basically the same 
> typical group by operation found in systems like Cascading.
> In order to create these intermediate list values I needed to define custom 
> avro schemas that simply wrap the elements of interest into a list. It seems 
> desirable that there be some basic facility for constructing simple Serdes of 
> Lists/Maps/Sets of other types, potentially using avro's serialization under 
> the hood. If this existed in the core library it would also enable the 
> addition of higher level operations on streams that can use these Serdes to 
> perform simple operations like the "group by" example I mention.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3745) Consider adding join key to ValueJoiner interface

[Andrew Musselman (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3745?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583079#comment-15583079
 ] 

Andrew Musselman commented on KAFKA-3745:
-

Still keeping this open?

> Consider adding join key to ValueJoiner interface
> -
>
> Key: KAFKA-3745
> URL: https://issues.apache.org/jira/browse/KAFKA-3745
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.0.0
>Reporter: Greg Fodor
>Priority: Minor
>  Labels: api, newbie
>
> In working with Kafka Stream joining, it's sometimes the case that a join key 
> is not actually present in the values of the joins themselves (if, for 
> example, a previous transform generated an ephemeral join key.) In such 
> cases, the actual key of the join is not available in the ValueJoiner 
> implementation to be used to construct the final joined value. This can be 
> worked around by explicitly threading the join key into the value if needed, 
> but it seems like extending the interface to pass the join key along as well 
> would be helpful.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-189) Create a script for dump segment tool

[Andrew Musselman (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-189?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583075#comment-15583075
 ] 

Andrew Musselman commented on KAFKA-189:


Any interest in this; been open a while.

> Create a script for dump segment tool
> -
>
> Key: KAFKA-189
> URL: https://issues.apache.org/jira/browse/KAFKA-189
> Project: Kafka
>  Issue Type: New Feature
>  Components: core
>Reporter: Taylor Gautier
>Priority: Minor
>  Labels: newbie, patch
> Attachments: KAFKA-189.diff
>
>
> In the kafka jar there is a useful segment dump tool.  It would be useful to 
> expose this as a script.  The current usage is:
> bin/kafka-run-class.sh kafka.tools.DumpLogSegments logfilename -noprint
> Change this to:
> bin/kafka-dump-log-segments.sh logfilename 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2424) Consider introducing lint-like tool for Scala

[Andrew Musselman (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2424?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583065#comment-15583065
 ] 

Andrew Musselman commented on KAFKA-2424:
-

Looks like Abide doesn't have a way to include it in Gradle builds yet and 
Scapegoat doesn't even mention Gradle; any sense of when Abide will support 
Gradle?

> Consider introducing lint-like tool for Scala
> -
>
> Key: KAFKA-2424
> URL: https://issues.apache.org/jira/browse/KAFKA-2424
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Ismael Juma
>  Labels: newbie
>
> Typesafe is working on abide and the first release is expected next month:
> https://github.com/scala/scala-abide
> An alternative is scapegoat:
> https://github.com/sksamuel/scalac-scapegoat-plugin



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2607) Review `Time` interface and its usage

[Andrew Musselman (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2607?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583043#comment-15583043
 ] 

Andrew Musselman commented on KAFKA-2607:
-

Looks like the PR has conflicts now; is this still in progress or is it still 
blocked by KAFKA-2247?

> Review `Time` interface and its usage
> -
>
> Key: KAFKA-2607
> URL: https://issues.apache.org/jira/browse/KAFKA-2607
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2.2
>Reporter: Ismael Juma
>  Labels: newbie
>
> Two of `Time` interface's methods are `milliseconds` and `nanoseconds` which 
> are implemented in `SystemTime` as follows:
> {code}
> @Override
> public long milliseconds() {
> return System.currentTimeMillis();
> }
> @Override
> public long nanoseconds() {
> return System.nanoTime();
> }
> {code}
> The issue with this interface is that it makes it seem that the difference is 
> about the unit (`ms` versus `ns`) whereas it's much more than that:
> https://blogs.oracle.com/dholmes/entry/inside_the_hotspot_vm_clocks
> We should probably change the names of the methods and review our usage to 
> see if we're using the right one in the various places.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2247) Merge kafka.utils.Time and kafka.common.utils.Time

[Andrew Musselman (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2247?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583038#comment-15583038
 ] 

Andrew Musselman commented on KAFKA-2247:
-

[~aauradkar] [~ijuma] Do you have a path to resolve this? Looks like KAFKA-2607 
is blocked by this but there's a patch for it, though it has conflicts now: 
https://github.com/apache/kafka/pull/837

> Merge kafka.utils.Time and kafka.common.utils.Time
> --
>
> Key: KAFKA-2247
> URL: https://issues.apache.org/jira/browse/KAFKA-2247
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Aditya Auradkar
>Assignee: Aditya Auradkar
>Priority: Minor
>
> We currently have 2 different versions of Time in clients and core. These 
> need to be merged.
> It's worth noting that `kafka.utils.MockTime` includes a `scheduler` that is 
> used by some tests while `o.a.kafka.common.utils.Time` does not. We either 
> need to add this functionality or change the tests not to need it anymore.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3408) consumer rebalance fail

[Andrew Musselman (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3408?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583027#comment-15583027
 ] 

Andrew Musselman commented on KAFKA-3408:
-

[~sdhzlzhk] Can you provide more repro info?

> consumer rebalance fail
> ---
>
> Key: KAFKA-3408
> URL: https://issues.apache.org/jira/browse/KAFKA-3408
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.9.0.1
> Environment: centos linux
>Reporter: zhongkai liu
>  Labels: newbie
>
> I use "/bin/kafka-console-consumer" command to start two consumers of group 
> "page_group",then the first conumer console report rebalance failure like 
> this:
> ERROR [page_view_group1_slave2-1458095694092-80c33086], error during 
> syncedRebalance (kafka.consumer.ZookeeperConsumerConnector)
> kafka.common.ConsumerRebalanceFailedException: 
> page_view_group1_slave2-1458095694092-80c33086 can't rebalance after 10 
> retries
> at 
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener.syncedRebalance(ZookeeperConsumerConnector.scala:660)
> at 
> kafka.consumer.ZookeeperConsumerConnector$ZKRebalancerListener$$anon$1.run(ZookeeperConsumerConnector.scala:579)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1703) The bat script failed to start on windows

[Andrew Musselman (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1703?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583022#comment-15583022
 ] 

Andrew Musselman commented on KAFKA-1703:
-

Has this been vetted/merged?

> The bat script failed to start on windows
> -
>
> Key: KAFKA-1703
> URL: https://issues.apache.org/jira/browse/KAFKA-1703
> Project: Kafka
>  Issue Type: Bug
>  Components: build
>Affects Versions: 0.8.1.1
>Reporter: ChengRen
>  Labels: newbie
> Attachments: kafka-run-class.bat
>
>
> The bat script in bin\windows can not start zookeeper and kafka correctly 
> (where my os is just installed and only jdk ready). I modified the 
> kafka-run-class.bat and add jars in libs folder to classpath.
> for %%i in (%BASE_DIR%\core\lib\*.jar) do (
>   call :concat %%i
> )
>  added  begin
> for %%i in (%BASE_DIR%\..\libs\*.jar) do (
>   call :concat %%i
> )
>  added  end
> for %%i in (%BASE_DIR%\perf\target\scala-2.8.0/kafka*.jar) do (
>   call :concat %%i
> )
> Now it runs correctly.
> Under bin\windows:
> zookeeper-server-start.bat ..\..\config\zookeeper.properties
>kafka-server-start.bat ..\..\config\kafka.properties



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1715) better advertising of the bound and working interface

[Andrew Musselman (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1715?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582992#comment-15582992
 ] 

Andrew Musselman commented on KAFKA-1715:
-

[~joestein] any more description/guidance, or is this not an issue any more?

> better advertising of the bound and working interface
> -
>
> Key: KAFKA-1715
> URL: https://issues.apache.org/jira/browse/KAFKA-1715
> Project: Kafka
>  Issue Type: Bug
>Reporter: Joe Stein
>  Labels: newbie
>
> As part of the auto discovery of brokers and meta data messaging we should 
> try to advertise the interface that is bound and working better. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2200) kafkaProducer.send() should not call callback.onCompletion()

[Andrew Musselman (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2200?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583016#comment-15583016
 ] 

Andrew Musselman commented on KAFKA-2200:
-

Still an issue?

> kafkaProducer.send() should not call callback.onCompletion()
> 
>
> Key: KAFKA-2200
> URL: https://issues.apache.org/jira/browse/KAFKA-2200
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jiangjie Qin
>  Labels: newbie
>
> KafkaProducer.send() should not call callback.onCompletion() because this 
> might break the callback firing order.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2167) ZkUtils updateEphemeralPath JavaDoc (spelling and correctness)

[Andrew Musselman (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2167?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15583009#comment-15583009
 ] 

Andrew Musselman commented on KAFKA-2167:
-

Looks like this ticket should be closed.

> ZkUtils updateEphemeralPath JavaDoc (spelling and correctness)
> --
>
> Key: KAFKA-2167
> URL: https://issues.apache.org/jira/browse/KAFKA-2167
> Project: Kafka
>  Issue Type: Bug
>Reporter: Jon Bringhurst
>  Labels: newbie
>
> I'm not 100% sure on this, but it seems like "persistent" should instead say 
> "ephemeral" in the JavaDoc. Also, note that "parrent" is misspelled.
> {noformat}
>   /**
>* Update the value of a persistent node with the given path and data.
>* create parrent directory if necessary. Never throw NodeExistException.
>*/
>   def updateEphemeralPath(client: ZkClient, path: String, data: String): Unit 
> = {
> try {
>   client.writeData(path, data)
> }
> catch {
>   case e: ZkNoNodeException => {
> createParentPath(client, path)
> client.createEphemeral(path, data)
>   }
>   case e2 => throw e2
> }
>   }
> {noformat}
> should be:
> {noformat}
>   /**
>* Update the value of an ephemeral node with the given path and data.
>* create parent directory if necessary. Never throw NodeExistException.
>*/
>   def updateEphemeralPath(client: ZkClient, path: String, data: String): Unit 
> = {
> try {
>   client.writeData(path, data)
> }
> catch {
>   case e: ZkNoNodeException => {
> createParentPath(client, path)
> client.createEphemeral(path, data)
>   }
>   case e2 => throw e2
> }
>   }
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2602) Incorrect descriptions about serializers for ProducerConfig in 0.8.2 and 0.8.3 documentation

[Andrew Musselman (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2602?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582987#comment-15582987
 ] 

Andrew Musselman commented on KAFKA-2602:
-

Is this still open/should this patch be applied?

> Incorrect descriptions about serializers for ProducerConfig in 0.8.2 and 
> 0.8.3 documentation 
> -
>
> Key: KAFKA-2602
> URL: https://issues.apache.org/jira/browse/KAFKA-2602
> Project: Kafka
>  Issue Type: Improvement
>Affects Versions: 0.8.2.1, 0.9.0.0
>Reporter: Zhuo Liu
>Priority: Minor
>  Labels: newbie
> Fix For: 0.8.2.0
>
> Attachments: KAFKA-2602_2015-09-30_14_22_22.patch
>
>
> The documentation still says "serializer.class" and "key.serializer.class",
> which is no longer correct. 
> We need to add descriptions about "key.serializer" and "value.serializer" in 
> the new Producer Config section (section 3.4).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2544) Replication tools wiki page needs to be updated

[Andrew Musselman (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2544?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582980#comment-15582980
 ] 

Andrew Musselman commented on KAFKA-2544:
-

[~gwenshap] do you know who could add Lauren to the wiki contributor list?

> Replication tools wiki page needs to be updated
> ---
>
> Key: KAFKA-2544
> URL: https://issues.apache.org/jira/browse/KAFKA-2544
> Project: Kafka
>  Issue Type: Improvement
>  Components: website
>Affects Versions: 0.8.2.1
>Reporter: Stevo Slavic
>Priority: Minor
>  Labels: documentation, newbie
>
> https://cwiki.apache.org/confluence/display/KAFKA/Replication+tools is 
> outdated, mentions tools which have been heavily refactored or replaced by 
> other tools, e.g. add partition tool, list/create topics tools, etc.
> Please have the replication tools wiki page updated.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1351) String.format is very expensive in Scala

[Andrew Musselman (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1351?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582973#comment-15582973
 ] 

Andrew Musselman commented on KAFKA-1351:
-

Still open?

> String.format is very expensive in Scala
> 
>
> Key: KAFKA-1351
> URL: https://issues.apache.org/jira/browse/KAFKA-1351
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.7.2, 0.8.0, 0.8.1
>Reporter: Neha Narkhede
>  Labels: newbie
> Attachments: KAFKA-1351.patch, KAFKA-1351_2014-04-07_18:02:18.patch, 
> KAFKA-1351_2014-04-09_15:40:11.patch
>
>
> As found in KAFKA-1350, logging is causing significant overhead in the 
> performance of a Kafka server. There are several info statements that use 
> String.format which is particularly expensive. We should investigate adding 
> our own version of String.format that merely uses string concatenation under 
> the covers.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Kafka KIP meeting Oct 19 at 11:00am PST

[Jun Rao]

Hi, Everyone.,

We plan to have a Kafka KIP meeting this coming Wednesday at 11:00am PST. If
you plan to attend but haven't received an invite, please let me know. The
following is the tentative agenda.

Agenda:
KIP-82: add record header

Thanks,

Jun

[jira] [Commented] (KAFKA-4308) Inconsistent parameters between console producer and consumer

[Andrew Musselman (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582896#comment-15582896
 ] 

Andrew Musselman commented on KAFKA-4308:
-

It is; mark this as a dupe?

> Inconsistent parameters between console producer and consumer
> -
>
> Key: KAFKA-4308
> URL: https://issues.apache.org/jira/browse/KAFKA-4308
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.1.0
>Reporter: Gwen Shapira
>  Labels: newbie
>
> kafka-console-producer uses --broker-list while kafka-console-consumer uses 
> --bootstrap-server.
> Let's add --bootstrap-server to the producer for some consistency?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4308) Inconsistent parameters between console producer and consumer

[Manasvi Gupta (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582890#comment-15582890
 ] 

Manasvi Gupta commented on KAFKA-4308:
--

I think this one is duplicate of KAFKA-4307

> Inconsistent parameters between console producer and consumer
> -
>
> Key: KAFKA-4308
> URL: https://issues.apache.org/jira/browse/KAFKA-4308
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.1.0
>Reporter: Gwen Shapira
>  Labels: newbie
>
> kafka-console-producer uses --broker-list while kafka-console-consumer uses 
> --bootstrap-server.
> Let's add --bootstrap-server to the producer for some consistency?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Work started] (KAFKA-4307) Inconsistent parameters between console producer and consumer

[Manasvi Gupta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-4307 started by Manasvi Gupta.

> Inconsistent parameters between console producer and consumer
> -
>
> Key: KAFKA-4307
> URL: https://issues.apache.org/jira/browse/KAFKA-4307
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.1.0
>Reporter: Gwen Shapira
>Assignee: Manasvi Gupta
>  Labels: newbie
>
> kafka-console-producer uses --broker-list while kafka-console-consumer uses 
> --bootstrap-server.
> Let's add --bootstrap-server to the producer for some consistency?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-4307) Inconsistent parameters between console producer and consumer

[Manasvi Gupta (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4307?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Manasvi Gupta reassigned KAFKA-4307:


Assignee: Manasvi Gupta

> Inconsistent parameters between console producer and consumer
> -
>
> Key: KAFKA-4307
> URL: https://issues.apache.org/jira/browse/KAFKA-4307
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.1.0
>Reporter: Gwen Shapira
>Assignee: Manasvi Gupta
>  Labels: newbie
>
> kafka-console-producer uses --broker-list while kafka-console-consumer uses 
> --bootstrap-server.
> Let's add --bootstrap-server to the producer for some consistency?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-205) Audit swallowing of exceptions/throwables

[Andrew Musselman (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-205?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582871#comment-15582871
 ] 

Andrew Musselman commented on KAFKA-205:


Still a need for this audit?

> Audit swallowing of exceptions/throwables
> -
>
> Key: KAFKA-205
> URL: https://issues.apache.org/jira/browse/KAFKA-205
> Project: Kafka
>  Issue Type: Task
>Reporter: Chris Burroughs
>  Labels: newbie
>
> There are several cases like KAFKA-204 where we swallow Error, or even 
> Throwable.  This is probably not the right thing to do (is 
> java.lang.StackOverflowError
> ever recoverable?).  



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4308) Inconsistent parameters between console producer and consumer

[Andrew Musselman (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582846#comment-15582846
 ] 

Andrew Musselman commented on KAFKA-4308:
-

Oh I just meant go ahead, hadn't seen your comment; I don't have permissions to 
assign tickets.

> Inconsistent parameters between console producer and consumer
> -
>
> Key: KAFKA-4308
> URL: https://issues.apache.org/jira/browse/KAFKA-4308
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.1.0
>Reporter: Gwen Shapira
>  Labels: newbie
>
> kafka-console-producer uses --broker-list while kafka-console-consumer uses 
> --bootstrap-server.
> Let's add --bootstrap-server to the producer for some consistency?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4308) Inconsistent parameters between console producer and consumer

[Umesh Chaudhary (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582838#comment-15582838
 ] 

Umesh Chaudhary commented on KAFKA-4308:


Sure Andrew, can you please assign this to me?

> Inconsistent parameters between console producer and consumer
> -
>
> Key: KAFKA-4308
> URL: https://issues.apache.org/jira/browse/KAFKA-4308
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.1.0
>Reporter: Gwen Shapira
>  Labels: newbie
>
> kafka-console-producer uses --broker-list while kafka-console-consumer uses 
> --bootstrap-server.
> Let's add --bootstrap-server to the producer for some consistency?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4308) Inconsistent parameters between console producer and consumer

[Andrew Musselman (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582824#comment-15582824
 ] 

Andrew Musselman commented on KAFKA-4308:
-

Ha, Umesh go for it.

> Inconsistent parameters between console producer and consumer
> -
>
> Key: KAFKA-4308
> URL: https://issues.apache.org/jira/browse/KAFKA-4308
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.1.0
>Reporter: Gwen Shapira
>  Labels: newbie
>
> kafka-console-producer uses --broker-list while kafka-console-consumer uses 
> --bootstrap-server.
> Let's add --bootstrap-server to the producer for some consistency?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4308) Inconsistent parameters between console producer and consumer

[Andrew Musselman (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582822#comment-15582822
 ] 

Andrew Musselman commented on KAFKA-4308:
-

I'll do it and make a PR if that's the right workflow.

> Inconsistent parameters between console producer and consumer
> -
>
> Key: KAFKA-4308
> URL: https://issues.apache.org/jira/browse/KAFKA-4308
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.1.0
>Reporter: Gwen Shapira
>  Labels: newbie
>
> kafka-console-producer uses --broker-list while kafka-console-consumer uses 
> --bootstrap-server.
> Let's add --bootstrap-server to the producer for some consistency?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4308) Inconsistent parameters between console producer and consumer

[Umesh Chaudhary (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4308?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582819#comment-15582819
 ] 

Umesh Chaudhary commented on KAFKA-4308:


Hi [~gwenshap], I would like to work on this.
If possible, please assign it to me.

> Inconsistent parameters between console producer and consumer
> -
>
> Key: KAFKA-4308
> URL: https://issues.apache.org/jira/browse/KAFKA-4308
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.1.0
>Reporter: Gwen Shapira
>  Labels: newbie
>
> kafka-console-producer uses --broker-list while kafka-console-consumer uses 
> --bootstrap-server.
> Let's add --bootstrap-server to the producer for some consistency?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4308) Inconsistent parameters between console producer and consumer

[Gwen Shapira (JIRA)]

Gwen Shapira created KAFKA-4308:
---

 Summary: Inconsistent parameters between console producer and 
consumer
 Key: KAFKA-4308
 URL: https://issues.apache.org/jira/browse/KAFKA-4308
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.10.1.0
Reporter: Gwen Shapira


kafka-console-producer uses --broker-list while kafka-console-consumer uses 
--bootstrap-server.

Let's add --bootstrap-server to the producer for some consistency?





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4307) Inconsistent parameters between console producer and consumer

[Gwen Shapira (JIRA)]

Gwen Shapira created KAFKA-4307:
---

 Summary: Inconsistent parameters between console producer and 
consumer
 Key: KAFKA-4307
 URL: https://issues.apache.org/jira/browse/KAFKA-4307
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.10.1.0
Reporter: Gwen Shapira


kafka-console-producer uses --broker-list while kafka-console-consumer uses 
--bootstrap-server.

Let's add --bootstrap-server to the producer for some consistency?





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] 0.10.1.0 RC3

[Gwen Shapira]

+1 (binding)

Verified license, notice, signatures, unit tests and ran a bunch of
quickstart scenarios.



On Sat, Oct 15, 2016 at 12:16 PM, Eno Thereska  wrote:
> +1 (non-binding)
>
>> On 15 Oct 2016, at 17:47, Jason Gustafson  wrote:
>>
>> As promised, here are updated test results:
>>
>> Unit tests: https://builds.apache.org/job/kafka-0.10.1-jdk7/75/
>> System tests:
>> http://testing.confluent.io/confluent-kafka-0-10-1-system-test-results/?prefix=2016-10-14--001.1476485874--apache--0.10.1--fc1551d/
>>
>> -Jason
>>
>> On Fri, Oct 14, 2016 at 4:29 PM, Jason Gustafson  wrote:
>>
>>> Hello Kafka users, developers and client-developers,
>>>
>>> One more RC for 0.10.1.0. We're hoping this is the final one so that we
>>> can meet the release target date of Oct. 17 (Monday). Please let me know as
>>> soon as possible if you find any major problems.
>>>
>>> Release plan: https://cwiki.apache.org/confluence/display/KAFKA/Rele
>>> ase+Plan+0.10.1.
>>>
>>> Release notes for the 0.10.1.0 release:
>>> http://home.apache.org/~jgus/kafka-0.10.1.0-rc3/RELEASE_NOTES.html
>>>
>>> *** Please download, test and vote by Monday, Oct 17, 5pm PT
>>>
>>> Kafka's KEYS file containing PGP keys we use to sign the release:
>>> http://kafka.apache.org/KEYS
>>>
>>> * Release artifacts to be voted upon (source and binary):
>>> http://home.apache.org/~jgus/kafka-0.10.1.0-rc3/
>>>
>>> * Maven artifacts to be voted upon:
>>> https://repository.apache.org/content/groups/staging/
>>>
>>> * Javadoc:
>>> http://home.apache.org/~jgus/kafka-0.10.1.0-rc3/javadoc/
>>>
>>> * Tag to be voted upon (off 0.10.1 branch) is the 0.10.1.0-rc3 tag:
>>> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=
>>> 50f30a44f31fca1bd9189d2814388d51bd56b06b
>>>
>>> * Documentation:
>>> http://kafka.apache.org/0101/documentation.html
>>>
>>> * Protocol:
>>> http://kafka.apache.org/0101/protocol.html
>>>
>>> * Tests:
>>> Unit tests: https://builds.apache.org/job/kafka-0.10.1-jdk7/71/
>>> System tests: http://testing.confluent.io/confluent-kafka-
>>> 0-10-1-system-test-results/?prefix=2016-10-13--001.
>>> 1476369986--apache--0.10.1--ee212d1/
>>>
>>> (Note that these tests do not include a couple patches merged today. I
>>> will send links to updated test builds as soon as they are available)
>>>
>>> Thanks,
>>>
>>> Jason
>>>
>



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog

[jira] [Created] (KAFKA-4306) Connect workers won't shut down if brokers are not available

[Gwen Shapira (JIRA)]

Gwen Shapira created KAFKA-4306:
---

 Summary: Connect workers won't shut down if brokers are not 
available
 Key: KAFKA-4306
 URL: https://issues.apache.org/jira/browse/KAFKA-4306
 Project: Kafka
  Issue Type: Bug
  Components: KafkaConnect
Affects Versions: 0.10.1.0
Reporter: Gwen Shapira
Assignee: Ewen Cheslack-Postava


If brokers are not available and we try to shut down connect workers, sink 
connectors will be stuck in a loop retrying to commit offsets:

2016-10-17 09:39:14,907] INFO Marking the coordinator 192.168.1.9:9092 (id: 
2147483647 rack: null) dead for group connect-dump-kafka-config1 
(org.apache.kafka.clients.consumer.internals.AbstractCoordinator:600)
[2016-10-17 09:39:14,907] ERROR Commit of 
WorkerSinkTask{id=dump-kafka-config1-0} offsets threw an unexpected exception:  
(org.apache.kafka.connect.runtime.WorkerSinkTask:194)
org.apache.kafka.clients.consumer.RetriableCommitFailedException: Offset commit 
failed with a retriable exception. You should retry committing offsets.
Caused by: org.apache.kafka.common.errors.GroupCoordinatorNotAvailableException

We should probably limit the number of retries before doing "unclean" shutdown.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4238) consumer-subscription not working, when accessing a newly created topic immediately after its creation with the AdminUtils

[Jason Gustafson (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582715#comment-15582715
 ] 

Jason Gustafson commented on KAFKA-4238:


We've debated a little bit on the current behavior of relying on 
{{metadata.max.age.ms}} when a subscribed topic doesn't exist. An alternative 
might be to retry fetching metadata more rapidly (say using 
{{retry.backoff.ms}}) in this case. It's a little annoying to have to tune down 
the metadata refresh interval when the consumer is really only waiting for 
initial topic creation. That would definitely help out in test cases where 
there is often a race condition with topic creation and consumer subscription.

> consumer-subscription not working, when accessing a newly created topic 
> immediately after its creation with the AdminUtils
> --
>
> Key: KAFKA-4238
> URL: https://issues.apache.org/jira/browse/KAFKA-4238
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.0.0, 0.10.0.1
>Reporter: Florian Witteler
>
> I created a test-project to reproduce the bug.
> https://github.com/FloWi/kafka-topic-creation-bug
> We use a docker container that creates a fresh topic before a testsuite gets 
> executed (see {{trait FreshKafkaTopics}}). That trait uses the AdminUtils to 
> create the topic. 
> If we access the newly created topic directly after its creation, the 
> subscriber is broken. It sometimes works though (<5%), so it seems to be a 
> race-condition.
> If I put a {{Thread.sleep(1000)}} after the topic-creation, everything's fine 
> though.
> So, the problem is twofold:
> - {{AdminUtils.createTopic}} should block until the topic-creation is 
> completed
> - {{new KafkaConsumer[String, 
> String](props).subscribe(util.Arrays.asList(topic))}} should throw an 
> exception, when the topic is "not ready"



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4238) consumer-subscription not working, when accessing a newly created topic immediately after its creation with the AdminUtils

[Florian Witteler (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4238?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582593#comment-15582593
 ] 

Florian Witteler commented on KAFKA-4238:
-

Thanks, [~hachikuji]! This workaround works fine. I'd have favoured, if 
AdminUtils.createTopic would block until the topic is ready. But anyway - since 
it is only test-code, I can live with that.

Regards,
Florian

> consumer-subscription not working, when accessing a newly created topic 
> immediately after its creation with the AdminUtils
> --
>
> Key: KAFKA-4238
> URL: https://issues.apache.org/jira/browse/KAFKA-4238
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.0.0, 0.10.0.1
>Reporter: Florian Witteler
>
> I created a test-project to reproduce the bug.
> https://github.com/FloWi/kafka-topic-creation-bug
> We use a docker container that creates a fresh topic before a testsuite gets 
> executed (see {{trait FreshKafkaTopics}}). That trait uses the AdminUtils to 
> create the topic. 
> If we access the newly created topic directly after its creation, the 
> subscriber is broken. It sometimes works though (<5%), so it seems to be a 
> race-condition.
> If I put a {{Thread.sleep(1000)}} after the topic-creation, everything's fine 
> though.
> So, the problem is twofold:
> - {{AdminUtils.createTopic}} should block until the topic-creation is 
> completed
> - {{new KafkaConsumer[String, 
> String](props).subscribe(util.Arrays.asList(topic))}} should throw an 
> exception, when the topic is "not ready"



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1733) Producer.send will block indeterminately when broker is unavailable.

[Dru Panchal (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1733?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15582565#comment-15582565
 ] 

Dru Panchal commented on KAFKA-1733:


As seen in the following file: The fix was made for the 0.9.0 release and it 
was also back-ported into 0.8.2
https://github.com/apache/kafka/blob/0.9.0/core/src/main/scala/kafka/network/BlockingChannel.scala
https://github.com/apache/kafka/blob/0.8.2/core/src/main/scala/kafka/network/BlockingChannel.scala

Also, as seen in the following file: The last release where this issue still 
exists is: 0.8.1
https://github.com/apache/kafka/blob/0.8.1/core/src/main/scala/kafka/network/BlockingChannel.scala


> Producer.send will block indeterminately when broker is unavailable.
> 
>
> Key: KAFKA-1733
> URL: https://issues.apache.org/jira/browse/KAFKA-1733
> Project: Kafka
>  Issue Type: Bug
>  Components: core, producer 
>Affects Versions: 0.8.1.1
>Reporter: Marc Chung
>Assignee: Marc Chung
> Fix For: 0.8.2.0, 0.9.0.0
>
> Attachments: kafka-1733-add-connectTimeoutMs.patch
>
>
> This is a follow up to the conversation here:
> https://mail-archives.apache.org/mod_mbox/kafka-dev/201409.mbox/%3ccaog_4qymoejhkbo0n31+a-ujx0z5unsisd5wbrmn-xtx7gi...@mail.gmail.com%3E
> During ClientUtils.fetchTopicMetadata, if the broker is unavailable, 
> socket.connect will block indeterminately. Any retry policy 
> (message.send.max.retries) further increases the time spent waiting for the 
> socket to connect.
> The root fix is to add a connection timeout value to the BlockingChannel's 
> socket configuration, like so:
> {noformat}
> -channel.socket.connect(new InetSocketAddress(host, port))
> +channel.socket.connect(new InetSocketAddress(host, port), connectTimeoutMs)
> {noformat}
> The simplest thing to do here would be to have a constant, default value that 
> would be applied to every socket configuration. 
> Is that acceptable? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1733) Producer.send will block indeterminately when broker is unavailable.

[Dru Panchal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dru Panchal updated KAFKA-1733:
---
Fix Version/s: 0.8.2.0

> Producer.send will block indeterminately when broker is unavailable.
> 
>
> Key: KAFKA-1733
> URL: https://issues.apache.org/jira/browse/KAFKA-1733
> Project: Kafka
>  Issue Type: Bug
>  Components: core, producer 
>Affects Versions: 0.8.1.1
>Reporter: Marc Chung
>Assignee: Marc Chung
> Fix For: 0.8.2.0, 0.9.0.0
>
> Attachments: kafka-1733-add-connectTimeoutMs.patch
>
>
> This is a follow up to the conversation here:
> https://mail-archives.apache.org/mod_mbox/kafka-dev/201409.mbox/%3ccaog_4qymoejhkbo0n31+a-ujx0z5unsisd5wbrmn-xtx7gi...@mail.gmail.com%3E
> During ClientUtils.fetchTopicMetadata, if the broker is unavailable, 
> socket.connect will block indeterminately. Any retry policy 
> (message.send.max.retries) further increases the time spent waiting for the 
> socket to connect.
> The root fix is to add a connection timeout value to the BlockingChannel's 
> socket configuration, like so:
> {noformat}
> -channel.socket.connect(new InetSocketAddress(host, port))
> +channel.socket.connect(new InetSocketAddress(host, port), connectTimeoutMs)
> {noformat}
> The simplest thing to do here would be to have a constant, default value that 
> would be applied to every socket configuration. 
> Is that acceptable? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-1733) Producer.send will block indeterminately when broker is unavailable.

[Dru Panchal (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1733?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Dru Panchal updated KAFKA-1733:
---
Affects Version/s: 0.8.1.1

> Producer.send will block indeterminately when broker is unavailable.
> 
>
> Key: KAFKA-1733
> URL: https://issues.apache.org/jira/browse/KAFKA-1733
> Project: Kafka
>  Issue Type: Bug
>  Components: core, producer 
>Affects Versions: 0.8.1.1
>Reporter: Marc Chung
>Assignee: Marc Chung
> Fix For: 0.9.0.0
>
> Attachments: kafka-1733-add-connectTimeoutMs.patch
>
>
> This is a follow up to the conversation here:
> https://mail-archives.apache.org/mod_mbox/kafka-dev/201409.mbox/%3ccaog_4qymoejhkbo0n31+a-ujx0z5unsisd5wbrmn-xtx7gi...@mail.gmail.com%3E
> During ClientUtils.fetchTopicMetadata, if the broker is unavailable, 
> socket.connect will block indeterminately. Any retry policy 
> (message.send.max.retries) further increases the time spent waiting for the 
> socket to connect.
> The root fix is to add a connection timeout value to the BlockingChannel's 
> socket configuration, like so:
> {noformat}
> -channel.socket.connect(new InetSocketAddress(host, port))
> +channel.socket.connect(new InetSocketAddress(host, port), connectTimeoutMs)
> {noformat}
> The simplest thing to do here would be to have a constant, default value that 
> would be applied to every socket configuration. 
> Is that acceptable? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-4305) Possible race condition in log segment truncation and Kafka Request Handler

[Hari (JIRA)]

Hari created KAFKA-4305:
---

 Summary: Possible race condition in log segment truncation and 
Kafka Request Handler 
 Key: KAFKA-4305
 URL: https://issues.apache.org/jira/browse/KAFKA-4305
 Project: Kafka
  Issue Type: Bug
  Components: core
Affects Versions: 0.10.0.1
Reporter: Hari
Priority: Minor


We started to observe this trace in our Kafka Server logs after migration to 
0.10.0.1

{code}
[2016-10-16 20:55:04,825] ERROR [KafkaApi-303] Error while responding to offset 
request (kafka.server.KafkaApis)
java.lang.NullPointerException
at kafka.server.KafkaApis.fetchOffsetsBefore(KafkaApis.scala:585)
at kafka.server.KafkaApis.fetchOffsets(KafkaApis.scala:573)
at kafka.server.KafkaApis$$anonfun$27.apply(KafkaApis.scala:530)
at kafka.server.KafkaApis$$anonfun$27.apply(KafkaApis.scala:521)
at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at 
scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
at scala.collection.Iterator$class.foreach(Iterator.scala:727)
at scala.collection.AbstractIterator.foreach(Iterator.scala:1157)
at scala.collection.IterableLike$class.foreach(IterableLike.scala:72)
at scala.collection.AbstractIterable.foreach(Iterable.scala:54)
at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
at scala.collection.AbstractTraversable.map(Traversable.scala:105)
at kafka.server.KafkaApis.handleOffsetRequest(KafkaApis.scala:521)
at kafka.server.KafkaApis.handle(KafkaApis.scala:78)
at kafka.server.KafkaRequestHandler.run(KafkaRequestHandler.scala:60)
at java.lang.Thread.run(Thread.java:745)
{code}

>From a quick investigation, I suspect this could happen because of race 
>condition with log segment truncation and Kafka request handler ?

Any thoughts ?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3559) Task creation time taking too long in rebalance callback

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15581874#comment-15581874
 ] 

ASF GitHub Bot commented on KAFKA-3559:
---

GitHub user enothereska opened a pull request:

https://github.com/apache/kafka/pull/2032

KAFKA-3559: Recycle old tasks when possible



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/enothereska/kafka 
KAFKA-3559-onPartitionAssigned

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2032.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2032


commit 28a50430e4136ba75f0d9b957a67f22e7b1e86a0
Author: Eno Thereska 
Date:   2016-10-17T10:46:45Z

Recycle old tasks when possible




> Task creation time taking too long in rebalance callback
> 
>
> Key: KAFKA-3559
> URL: https://issues.apache.org/jira/browse/KAFKA-3559
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Eno Thereska
>  Labels: architecture
> Fix For: 0.10.2.0
>
>
> Currently in Kafka Streams, we create stream tasks upon getting newly 
> assigned partitions in rebalance callback function {code} onPartitionAssigned 
> {code}, which involves initialization of the processor state stores as well 
> (including opening the rocksDB, restore the store from changelog, etc, which 
> takes time).
> With a large number of state stores, the initialization time itself could 
> take tens of seconds, which usually is larger than the consumer session 
> timeout. As a result, when the callback is completed, the consumer is already 
> treated as failed by the coordinator and rebalance again.
> We need to consider if we can optimize the initialization process, or move it 
> out of the callback function, and while initializing the stores one-by-one, 
> use poll call to send heartbeats to avoid being kicked out by coordinator.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #2032: KAFKA-3559: Recycle old tasks when possible

[enothereska]

GitHub user enothereska opened a pull request:

https://github.com/apache/kafka/pull/2032

KAFKA-3559: Recycle old tasks when possible



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/enothereska/kafka 
KAFKA-3559-onPartitionAssigned

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/2032.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #2032


commit 28a50430e4136ba75f0d9b957a67f22e7b1e86a0
Author: Eno Thereska 
Date:   2016-10-17T10:46:45Z

Recycle old tasks when possible




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Closed] (KAFKA-4299) Consumer offsets reset for all topics after increasing partitions for one topic

[Juho Autio (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Juho Autio closed KAFKA-4299.
-

> Consumer offsets reset for all topics after increasing partitions for one 
> topic
> ---
>
> Key: KAFKA-4299
> URL: https://issues.apache.org/jira/browse/KAFKA-4299
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.0.1
>Reporter: Juho Autio
>
> I increased partitions for one existing topic (2->10), but was surprised to 
> see that it entirely reset the committed offsets of my consumer group.
> All topics & partitions were reset to the earliest offset available, and the 
> consumer read everything again.
> Documentation doesn't mention anything like this. Is this how it's supposed 
> to work, or a bug?
> I would've expected the consumer offsets to not decrease at all, especially 
> for the topics that I didn't even touch.
> For the altered topic I would've expected that consuming the previously 
> existing partitions 0 and 1 would've continued from the position where they 
> were, and naturally starting to read the new added partitions from 0.
> I added partitions according to the "Modifying topics" section of Kafka 
> 0.10.0 Documentation:
> {quote}
> To add partitions you can do
> {code}
>  > bin/kafka-topics.sh --zookeeper $ZOOKEEPER_HOST --alter --topic 
> altered_topic --partitions 10
> {code}
> {quote}
> Previously this topic had 2 partitions.
> For the consumer I'm using 
> {{kafka.javaapi.consumer.ConsumerConnector.createMessageStreamsByFilter()}}.
> And version is:
> {code}
> org.apache.kafka
> kafka_2.11
> 0.10.0.1
> {code}
> Kafka cluster itself is {{kafka_2.11-0.10.0.1}}.
> This is quite problematic because we can't afford waiting for consumers to 
> read the full buffer from the beginning (for all topics!) when increasing 
> partitions for a topic.
> Some possibly relevant settings we have for the consumer:
> {code}
> kafka.partition.assignment.strategy = "range"
> kafka.auto.offset.reset = "smallest"
> kafka.auto.commit.enable = "false"
> kafka.offsets.storage = "kafka"
> kafka.dual.commit.enabled = false
> kafka.consumer.timeout.ms = "2000"
> kafka.auto.create.topics.enable = true
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-4299) Consumer offsets reset for all topics after increasing partitions for one topic

[Juho Autio (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4299?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Juho Autio resolved KAFKA-4299.
---
Resolution: Cannot Reproduce

I did another partition modification in production and this time it succeeded 
without any consumer offsets being reset. I hope reset never happens again.

> Consumer offsets reset for all topics after increasing partitions for one 
> topic
> ---
>
> Key: KAFKA-4299
> URL: https://issues.apache.org/jira/browse/KAFKA-4299
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.10.0.1
>Reporter: Juho Autio
>
> I increased partitions for one existing topic (2->10), but was surprised to 
> see that it entirely reset the committed offsets of my consumer group.
> All topics & partitions were reset to the earliest offset available, and the 
> consumer read everything again.
> Documentation doesn't mention anything like this. Is this how it's supposed 
> to work, or a bug?
> I would've expected the consumer offsets to not decrease at all, especially 
> for the topics that I didn't even touch.
> For the altered topic I would've expected that consuming the previously 
> existing partitions 0 and 1 would've continued from the position where they 
> were, and naturally starting to read the new added partitions from 0.
> I added partitions according to the "Modifying topics" section of Kafka 
> 0.10.0 Documentation:
> {quote}
> To add partitions you can do
> {code}
>  > bin/kafka-topics.sh --zookeeper $ZOOKEEPER_HOST --alter --topic 
> altered_topic --partitions 10
> {code}
> {quote}
> Previously this topic had 2 partitions.
> For the consumer I'm using 
> {{kafka.javaapi.consumer.ConsumerConnector.createMessageStreamsByFilter()}}.
> And version is:
> {code}
> org.apache.kafka
> kafka_2.11
> 0.10.0.1
> {code}
> Kafka cluster itself is {{kafka_2.11-0.10.0.1}}.
> This is quite problematic because we can't afford waiting for consumers to 
> read the full buffer from the beginning (for all topics!) when increasing 
> partitions for a topic.
> Some possibly relevant settings we have for the consumer:
> {code}
> kafka.partition.assignment.strategy = "range"
> kafka.auto.offset.reset = "smallest"
> kafka.auto.commit.enable = "false"
> kafka.offsets.storage = "kafka"
> kafka.dual.commit.enabled = false
> kafka.consumer.timeout.ms = "2000"
> kafka.auto.create.topics.enable = true
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention

[Becket Qin]

Hey David,

Thanks for replies to the questions.

I think one major thing still not clear at this point is that whether the
brokers will only apply the consumed log retention to a specific set of
interested consumer groups, or it does not have such a set of consumer
groups.

For example, for topic T, assume we know that there will be two downstream
consumer groups CG1 and CG2 consuming data from topic T. Will we add a
topic configurations such as
"log.retention.commitoffset.interested.groups=CG1,CG2" to topic T so that
the brokers only care about CG1 and CG2. The committed offsets of other
groups are not interested and won't have any impact on the committed offset
based log retention.

It seems the current proposal does not have an "interested consumer group
set" configuration, so that means any random consumer group may affect the
committed offset based log retention.

I think the committed offset based log retention seems more useful in cases
where we already know which consumer groups will be consuming from this
topic, so we will only wait for those consumer groups but ignore the
others. If a group will be consumed by many unknown or unpredictable
consumer groups, it seems the existing time based log retention is much
simple and clear enough. So I would argue we don't need to address the case
that some groups may come later in the committed offset based retention.

That said, there may still be value to keep the data for some time even
after all the interested consumer groups have consumed the messages. For
example, in a pipelined stream processing DAG, we may want to keep the data
of an intermediate topic for some time in case the job fails. So we can
resume from a previously succeeded stage instead of restart the entire
pipeline. Or we can use the intermediate topic for some debugging work.

Thanks,

Jiangjie (Becket) Qin



On Sun, Oct 16, 2016 at 2:15 AM, 东方甲乙 <254479...@qq.com> wrote:

> Hi Dong,
> The KIP is used to solve both these 2 cases, we specify a small
> consumed log retention time to deleted the consumed data and avoid losing
> un-consumed data.
> And the specify a large force log retention time used as higher bound for
> the data.  I will update the KIP for this info.
> Another solution I think may be ok is to support an API to delete the
> inactive group?  If the group is in inactive, but it's commit offset is
> also in the __commit_offsets topic and
> stay in the offset cache,  we can delete it via this API.
>
>
> Thanks,
> David
>
>
> -- 原始邮件 --
> 发件人: "Dong Lin";;
> 发送时间: 2016年10月14日(星期五) 凌晨5:01
> 收件人: "dev";
>
> 主题: Re: [DISCUSS] KIP-68 Add a consumed log retention before log retention
>
>
>
> Hi David,
>
> As explained in the motivation section of the KIP, the problem is that if
> log retention is too small, we may lose data; and if log retention is too
> large, then we waste disk space. Therefore, we need to solve one if the two
> problems -- allow data to be persisted longer for consumption if log
> retention is set too small, or allow data to be expired earlier if log
> retention is too large. I think the KIP probably needs to make this clear
> and explain which one is rejected and why. Note that the choice of the two
> affects the solution -- if we want to address the first problem then
> log.retention.ms should be used as lower bound on the actual retention
> time, and if we want to address the second problem then the
> log.retention.ms
> should be used as higher bound on the actual retention time.
>
> In both cases, we probably need to figure out a way to determine "active
> consumer group". Maybe we can compare the time-since-last-commit against a
> threshold to determine this. In addition, the threshold can be overridden
> either per-topic or per-groupId. If we go along this route, the rejected
> solution (per-topic vs. per-groupId) should probably be explained in the
> KIP.
>
>
> Thanks,
> Dong
>
>
>
> On Thu, Oct 13, 2016 at 10:23 AM, Dong Lin  wrote:
>
> > Hi David,
> >
> > Thanks for your explanation. There still seems to be issue with this
> > solution. Please see my comment inline.
> >
> >
> > On Thu, Oct 13, 2016 at 8:46 AM, 东方甲乙 <254479...@qq.com> wrote:
> >
> >> Hi Dong,
> >> Sorry for the delay, here are the comments:
> >> 1.I think we should distinguish these two cases:
> >> (1) group has no member, but has commit offset :  In this case we should
> >> consider its commit offset
> >> (2) group has no member, no commit offset:  Skip this group
> >> Is it ok?
> >>
> >>
> >> ListGroup API can list the groups,  but this API only show the Online
> >> Group, so we should enhance the listGroup API to list those groups in
> the
> >> case (1)
> >>
> >> Say some user starts a consumer to consume topic A with
> > enable.auto.commit = true. Later they change the group name in the
> config.
> > Then the proposed solution will never execute consumed log retention for
> > 

Re: [DISCUSS] KIP-82 - Add Record Headers

[Michael Pearce]

Hi Jun,

Sounds good.

Look forward to the invite.

Cheers,
Mike

From: Jun Rao 
Sent: Monday, October 17, 2016 5:55:57 AM
To: dev@kafka.apache.org
Subject: Re: [DISCUSS] KIP-82 - Add Record Headers

Hi, Michael,

We do have online KIP discussion meetings from time to time. How about we
discuss this KIP Wed (Oct 19) at 11:00am PST? I will send out an invite (we
typically do the meeting through Zoom and will post the video recording to
Kafka wiki).

Thanks,

Jun

On Wed, Oct 12, 2016 at 1:22 AM, Michael Pearce 
wrote:

> @Jay and Dana
>
> We have internally had a few discussions of how we may address this if we
> had a common apache kafka message wrapper for headers that can be used
> client side only to, and address the compaction issue.
> I have detailed this solution separately and linked from the main KIP-82
> wiki.
>
> Here’s a direct link –
> https://cwiki.apache.org/confluence/display/KAFKA/
> Headers+Value+Message+Wrapper
>
> We feel this solution though doesn’t manage to address all the use cases
> being mentioned still and also has some compatibility drawbacks e.g.
> backwards forwards compatibility especially on different language clients
> Also we still require with this solution, as still need to address
> compaction issue / tombstones, we need to make server side changes and as
> many message/record version changes.
>
> We believe the proposed solution in KIP-82 does address all these needs
> and is cleaner still, and more benefits.
> Please have a read, and comment. Also if you have any improvements on the
> proposed KIP-82 or an alternative solution/option your input is appreciated.
>
> @All
> As Joel has mentioned to get this moving along, and able to discuss more
> fluidly, it would be great if we can organize to meet up virtually online
> e.g. webex or something.
> I am aware, that the majority are based in America, myself is in the UK.
> @Kostya I assume you’re in Eastern Europe or Russia based on your email
> address (please correct this assumption), I hope the time difference isn’t
> too much that the below would suit you if you wish to join
>
> Can I propose next Wednesday 19th October at 18:30 BST , 10:30 PST, 20:30
> MSK we try meetup online?
>
> Would this date/time suit the majority?
> Also what is the preferred method? I can host via Adobe Connect style
> webex (which my company uses) but it isn’t the best IMHO, so more than
> happy to have someone suggest a better alternative.
>
> Best,
> Mike
>
>
>
>
> On 10/8/16, 7:26 AM, "Michael Pearce"  wrote:
>
> >> I agree with the critique of compaction not having a value. I think
> we should consider fixing that directly.
>
> > Agree that the compaction issue is troubling: compacted "null"
> deletes
> are incompatible w/ headers that must be packed into the message
> value. Are there any alternatives on compaction delete semantics that
> could address this? The KIP wiki discussion I think mostly assumes
> that compaction-delete is what it is and can't be changed/fixed.
>
> This KIP is about dealing with quite a few use cases and issues,
> please see both the KIP use cases detailed by myself and also the
> additional use cases wiki added by LinkedIn linked from the main KIP.
>
> The compaction is something that happily is addressed with headers,
> but most defiantly isn't the sole reason or use case for them, headers
> solves many issues and use cases. Thus their elegance and simplicity, and
> why they're so common in transport mechanisms and so succesfull, as stated
> like http, tcp, jms.
>
> 
> From: Dana Powers 
> Sent: Friday, October 7, 2016 11:09 PM
> To: dev@kafka.apache.org
> Subject: Re: [DISCUSS] KIP-82 - Add Record Headers
>
> > I agree with the critique of compaction not having a value. I think
> we should consider fixing that directly.
>
> Agree that the compaction issue is troubling: compacted "null" deletes
> are incompatible w/ headers that must be packed into the message
> value. Are there any alternatives on compaction delete semantics that
> could address this? The KIP wiki discussion I think mostly assumes
> that compaction-delete is what it is and can't be changed/fixed.
>
> -Dana
>
> On Fri, Oct 7, 2016 at 1:38 PM, Michael Pearce 
> wrote:
> >
> > Hi Jay,
> >
> > Thanks for the comments and feedback.
> >
> > I think its quite clear that if a problem keeps arising then it is
> clear that it needs resolving, and addressing properly.
> >
> > Fair enough at linkedIn, and historically for the very first use
> cases addressing this maybe not have been a big priority. But as Kafka is
> now Apache open source and being picked up by many including my company, it
> is clear and evident that this is a requirement and issue