Build failed in Jenkins: kafka-trunk-jdk8 #743

See Changes: [ismael] KAFKA-3849; Add explanation on why polling every second in MirrorMaker -- [...truncated 6745 lines...] kafka.coordinator.GroupMetadataManagerTest > testStoreNonEmptyGroup

Build failed in Jenkins: kafka-trunk-jdk7 #1412

See Changes: [ismael] KAFKA-3849; Add explanation on why polling every second in MirrorMaker -- [...truncated 3325 lines...] kafka.security.auth.SimpleAclAuthorizerTest > testLoadCache PASSED

Jenkins build is back to normal : kafka-trunk-jdk8 #742

See

Re: [DISCUSS] KIP-53 Add custom policies for reconnect attempts to NetworkdClient

Thanks for the confirmation, Ismael. I will write something up for further discussion. -Dana On Jul 5, 2016 4:43 PM, "Ismael Juma" wrote: > Hi Dana, > > Thanks for the PR. Technically, a (simple) KIP is required when proposing > new configs. > > Ismael > > On Sun, Jun 19,

[jira] [Updated] (KAFKA-3849) Add explanation on why polling every second in MirrorMaker is required

[ https://issues.apache.org/jira/browse/KAFKA-3849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-3849: --- Description: (was: MirrorMaker has consumer poll time, consumer timeout time for new consumer,

[jira] [Updated] (KAFKA-3849) Add explanation on why polling every second in MirrorMaker is required

[ https://issues.apache.org/jira/browse/KAFKA-3849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-3849: --- Summary: Add explanation on why polling every second in MirrorMaker is required (was: Avoid polling

[jira] [Updated] (KAFKA-3849) Avoid polling every second in MirrorMaker if no data is available.

[ https://issues.apache.org/jira/browse/KAFKA-3849?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma updated KAFKA-3849: --- Resolution: Fixed Fix Version/s: (was: 0.10.0.1) 0.10.1.0

[jira] [Commented] (KAFKA-3849) Avoid polling every second in MirrorMaker if no data is available.

[ https://issues.apache.org/jira/browse/KAFKA-3849?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368875#comment-15368875 ] ASF GitHub Bot commented on KAFKA-3849: --- Github user asfgit closed the pull request at:

[GitHub] kafka pull request #1515: KAFKA-3849: Add explanation on why polling every s...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1515 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] kafka pull request #1595: MINOR: Typo fix in comments

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1595 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[GitHub] kafka pull request #1592: Update design.html

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1592 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Commented] (KAFKA-3844) Sort configuration items in log

[ https://issues.apache.org/jira/browse/KAFKA-3844?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368844#comment-15368844 ] ASF GitHub Bot commented on KAFKA-3844: --- Github user asfgit closed the pull request at:

[jira] [Resolved] (KAFKA-3844) Sort configuration items in log

[ https://issues.apache.org/jira/browse/KAFKA-3844?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ismael Juma resolved KAFKA-3844. Resolution: Fixed Fix Version/s: 0.10.1.0 Issue resolved by pull request 1559

[GitHub] kafka pull request #1559: KAFKA-3844; Sort configuration items in log

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1559 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Created] (KAFKA-3941) Avoid applying eviction listener in InMemoryKeyValueLoggedStore

Guozhang Wang created KAFKA-3941: Summary: Avoid applying eviction listener in InMemoryKeyValueLoggedStore Key: KAFKA-3941 URL: https://issues.apache.org/jira/browse/KAFKA-3941 Project: Kafka

Build failed in Jenkins: kafka-trunk-jdk8 #741

See Changes: [junrao] KAFKA-3111; Fix ConsumerPerformance reporting to use time-based instead -- [...truncated 5610 lines...] kafka.log.LogCleanerIntegrationTest > cleanerTest[0] PASSED

Build failed in Jenkins: kafka-trunk-jdk7 #1410

See Changes: [junrao] KAFKA-3111; Fix ConsumerPerformance reporting to use time-based instead -- [...truncated 9017 lines...] kafka.log.LogCleanerIntegrationTest > cleanerTest[0] PASSED

[jira] [Commented] (KAFKA-3887) StreamBounceTest.test_bounce and StreamSmokeTest.test_streams failing

[ https://issues.apache.org/jira/browse/KAFKA-3887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368817#comment-15368817 ] Guozhang Wang commented on KAFKA-3887: -- The system test still failing but seems in a different way:

[jira] [Reopened] (KAFKA-3887) StreamBounceTest.test_bounce and StreamSmokeTest.test_streams failing

[ https://issues.apache.org/jira/browse/KAFKA-3887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang reopened KAFKA-3887: -- > StreamBounceTest.test_bounce and StreamSmokeTest.test_streams failing >

[jira] [Commented] (KAFKA-3940) Log should check the return value of dir.mkdirs()

[ https://issues.apache.org/jira/browse/KAFKA-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368797#comment-15368797 ] Ismael Juma commented on KAFKA-3940: I think we should just change all usages of File.mkdirs() to

[GitHub] kafka pull request #788: KAFKA-3111: Fix ConsumerPerformance reporting to us...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/788 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Commented] (KAFKA-3111) java.lang.ArithmeticException: / by zero in ConsumerPerformance

[ https://issues.apache.org/jira/browse/KAFKA-3111?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368670#comment-15368670 ] ASF GitHub Bot commented on KAFKA-3111: --- Github user asfgit closed the pull request at:

[jira] [Updated] (KAFKA-3111) java.lang.ArithmeticException: / by zero in ConsumerPerformance

[ https://issues.apache.org/jira/browse/KAFKA-3111?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-3111: --- Resolution: Fixed Fix Version/s: 0.10.1.0 Status: Resolved (was: Patch Available) Issue

Re: SSL: Broker works but consumer/producer fail

Which version of kafka are you using. -Harsha On Fri, Jul 8, 2016 at 1:09 AM Narendra Bidari wrote: > Hi Vineet, > > The setup of ssl Kafka requires to make one too many steps precise > correct. I have listed some below . Hope it helps >

[jira] [Updated] (KAFKA-2941) Docs for key/value converter in Kafka connect are unclear

[ https://issues.apache.org/jira/browse/KAFKA-2941?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava updated KAFKA-2941: - Status: Patch Available (was: Open) > Docs for key/value converter in Kafka

[jira] [Commented] (KAFKA-2941) Docs for key/value converter in Kafka connect are unclear

[ https://issues.apache.org/jira/browse/KAFKA-2941?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368573#comment-15368573 ] ASF GitHub Bot commented on KAFKA-2941: --- GitHub user ewencp opened a pull request:

[GitHub] kafka pull request #1599: KAFKA-2941: Clarify docs for key and value Convert...

GitHub user ewencp opened a pull request: https://github.com/apache/kafka/pull/1599 KAFKA-2941: Clarify docs for key and value Converters You can merge this pull request into a Git repository by running: $ git pull https://github.com/ewencp/kafka

[jira] [Updated] (KAFKA-3163) KIP-33 - Add a time based log index to Kafka

[ https://issues.apache.org/jira/browse/KAFKA-3163?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-3163: --- Attachment: 00113931.timeindex 00113931.log Also, I ran the following

[jira] [Commented] (KAFKA-3939) add new consumer metrics in docs

[ https://issues.apache.org/jira/browse/KAFKA-3939?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368391#comment-15368391 ] James Cheng commented on KAFKA-3939: I am addressing this work in

Jenkins build is back to normal : kafka-trunk-jdk8 #740

See

[jira] [Assigned] (KAFKA-3940) Log should check the return value of dir.mkdirs()

[ https://issues.apache.org/jira/browse/KAFKA-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ishita Mandhan reassigned KAFKA-3940: - Assignee: Ishita Mandhan > Log should check the return value of dir.mkdirs() >

[jira] [Commented] (KAFKA-3940) Log should check the return value of dir.mkdirs()

[ https://issues.apache.org/jira/browse/KAFKA-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368315#comment-15368315 ] Jun Rao commented on KAFKA-3940: Also, instead of using File.mkdirs(), it may be better to use

[jira] [Updated] (KAFKA-3940) Log should check the return value of dir.mkdirs()

[ https://issues.apache.org/jira/browse/KAFKA-3940?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Jun Rao updated KAFKA-3940: --- Labels: newbie (was: ) > Log should check the return value of dir.mkdirs() >

[jira] [Created] (KAFKA-3940) Log should check the return value of dir.mkdirs()

Jun Rao created KAFKA-3940: -- Summary: Log should check the return value of dir.mkdirs() Key: KAFKA-3940 URL: https://issues.apache.org/jira/browse/KAFKA-3940 Project: Kafka Issue Type: Bug

Build failed in Jenkins: kafka-trunk-jdk7 #1409

See Changes: [me] KAFKA-3920: Add Schema source connector to Kafka Connect -- [...truncated 5676 lines...] kafka.log.LogCleanerIntegrationTest > cleanerTest[0] PASSED

[jira] [Resolved] (KAFKA-3920) Add Schema source connector to Kafka Connect

[ https://issues.apache.org/jira/browse/KAFKA-3920?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Ewen Cheslack-Postava resolved KAFKA-3920. -- Resolution: Fixed Issue resolved by pull request 1574

[GitHub] kafka pull request #1574: KAFKA-3920: Add Schema source connector to Kafka C...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1574 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Commented] (KAFKA-3920) Add Schema source connector to Kafka Connect

[ https://issues.apache.org/jira/browse/KAFKA-3920?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15368075#comment-15368075 ] ASF GitHub Bot commented on KAFKA-3920: --- Github user asfgit closed the pull request at:

Build failed in Jenkins: kafka-trunk-jdk8 #739

See Changes: [wangguoz] KAFKA-3887: KAFKA-3817 follow-up to avoid forwarding value if it is null -- [...truncated 3356 lines...] kafka.coordinator.GroupMetadataManagerTest >

Build failed in Jenkins: kafka-0.10.0-jdk7 #145

See Changes: [wangguoz] KAFKA-3887: KAFKA-3817 follow-up to avoid forwarding value if it is -- [...truncated 1670 lines...] kafka.integration.SaslPlaintextTopicMetadataTest >

Re: [VOTE] KIP-67: Queryable state for Kafka Streams

Damian, about the rolling upgrade comment: An instance A will contact another instance B by the latter's endpoint, right? So if A has no further information available than B's host and port, then how should instance A know whether it should call B at /v1/ or at /v2/? I agree that my suggestion

Re: [VOTE] KIP-67: Queryable state for Kafka Streams

Addendum in case my previous email wasn't clear: > So for any given instance of a streams application there will never be both a v1 and v2 alive at the same time That's right. But the current live instance will be able to tell other instances, via its endpoint setting, whether it wants to be

[jira] [Created] (KAFKA-3939) add new consumer metrics in docs

Jun Rao created KAFKA-3939: -- Summary: add new consumer metrics in docs Key: KAFKA-3939 URL: https://issues.apache.org/jira/browse/KAFKA-3939 Project: Kafka Issue Type: Task Components:

[jira] [Commented] (KAFKA-3101) Optimize Aggregation Outputs

[ https://issues.apache.org/jira/browse/KAFKA-3101?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15367923#comment-15367923 ] Bill Bejeck commented on KAFKA-3101: [~guozhang] [~enothereska] Would adding flatbuffers

Build failed in Jenkins: kafka-trunk-jdk7 #1408

See Changes: [wangguoz] KAFKA-3887: KAFKA-3817 follow-up to avoid forwarding value if it is -- [...truncated 3325 lines...] kafka.message.ByteBufferMessageSetTest > testValidBytes PASSED

[jira] [Updated] (KAFKA-3559) Task creation time taking too long in rebalance callback

[ https://issues.apache.org/jira/browse/KAFKA-3559?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-3559: - Issue Type: Sub-task (was: Bug) Parent: KAFKA-3938 > Task creation time taking too long

[jira] [Updated] (KAFKA-3938) Fix consumer session timeout issue in Kafka Streams

[ https://issues.apache.org/jira/browse/KAFKA-3938?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-3938: - Labels: architecture (was: ) > Fix consumer session timeout issue in Kafka Streams >

[jira] [Created] (KAFKA-3938) Fix consumer session timeout issue in Kafka Streams

Guozhang Wang created KAFKA-3938: Summary: Fix consumer session timeout issue in Kafka Streams Key: KAFKA-3938 URL: https://issues.apache.org/jira/browse/KAFKA-3938 Project: Kafka Issue

[jira] [Updated] (KAFKA-3758) KStream job fails to recover after Kafka broker stopped

[ https://issues.apache.org/jira/browse/KAFKA-3758?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-3758: - Issue Type: Sub-task (was: Bug) Parent: KAFKA-3938 > KStream job fails to recover after

[jira] [Updated] (KAFKA-3752) Provide a way for KStreams to recover from unclean shutdown

[ https://issues.apache.org/jira/browse/KAFKA-3752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-3752: - Issue Type: Sub-task (was: Improvement) Parent: KAFKA-3938 > Provide a way for KStreams

[jira] [Updated] (KAFKA-3708) Rethink exception handling in KafkaStreams

[ https://issues.apache.org/jira/browse/KAFKA-3708?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang updated KAFKA-3708: - Issue Type: Sub-task (was: Bug) Parent: KAFKA-3938 > Rethink exception handling in

[jira] [Resolved] (KAFKA-3887) StreamBounceTest.test_bounce and StreamSmokeTest.test_streams failing

[ https://issues.apache.org/jira/browse/KAFKA-3887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Guozhang Wang resolved KAFKA-3887. -- Resolution: Fixed Fix Version/s: 0.10.0.1 Issue resolved by pull request 1597

[GitHub] kafka pull request #1597: KAFKA-3817 Follow-up: Avoid forwarding old value i...

Github user asfgit closed the pull request at: https://github.com/apache/kafka/pull/1597 --- If your project is set up for it, you can reply to this email and have your reply appear on GitHub as well. If your project does not have this feature enabled and wishes so, or if the feature is

[jira] [Commented] (KAFKA-3817) KTableRepartitionMap should handle null inputs

[ https://issues.apache.org/jira/browse/KAFKA-3817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15367817#comment-15367817 ] ASF GitHub Bot commented on KAFKA-3817: --- Github user asfgit closed the pull request at:

Re: [VOTE] KIP-67: Queryable state for Kafka Streams

Michael - i'm ok with changing it to a string. Any one else have a strong opinion on this? FWIW - i don't think it will work fine as is during the rolling upgrade scenario as the service that is listening on the port needs to be embedded within each instance. So for any given instance of a

[jira] [Commented] (KAFKA-3919) Broker faills to start after ungraceful shutdown due to non-monotonically incrementing offsets in logs

[ https://issues.apache.org/jira/browse/KAFKA-3919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15367781#comment-15367781 ] Jun Rao commented on KAFKA-3919: [~BigAndy], yes, the fix could be involved and we haven't nailed down a

[jira] [Created] (KAFKA-3937) Kafka Clients Leak Native Memory For Longer Than Needed With Compressed Messages

Tom Crayford created KAFKA-3937: --- Summary: Kafka Clients Leak Native Memory For Longer Than Needed With Compressed Messages Key: KAFKA-3937 URL: https://issues.apache.org/jira/browse/KAFKA-3937

Re: [VOTE] KIP-67: Queryable state for Kafka Streams

I have one further comment about `StreamsConfig.USER_ENDPOINT_CONFIG`. I think we should consider to not restricting the value of this setting to only `host:port` pairs. By design, this setting is capturing user-driven metadata to define an endpoint, so why restrict the creativity or flexibility

[jira] [Updated] (KAFKA-3936) Validate user parameters as early as possible

[ https://issues.apache.org/jira/browse/KAFKA-3936?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ] Matthias J. Sax updated KAFKA-3936: --- Description: Currently, parameters handed in by the user via public API, are not validated.

[jira] [Created] (KAFKA-3936) Validate user parameters as early as possible

Matthias J. Sax created KAFKA-3936: -- Summary: Validate user parameters as early as possible Key: KAFKA-3936 URL: https://issues.apache.org/jira/browse/KAFKA-3936 Project: Kafka Issue Type:

Re: [DISCUSS] Client Side Auto Topic Creation

Hi Jun, I agree that it's closer to the existing behaviour, which some people may be used to by now. However, I am not sure that it won't surprise people. As Grant said, auto-topic creation is a common source of confusion and it interacts badly with topic deletion. If we need to provide

Re: [VOTE] KIP-67: Queryable state for Kafka Streams

+1 On 07/08/2016 11:03 AM, Eno Thereska wrote: > +1 (non-binding) > >> On 7 Jul 2016, at 18:31, Sriram Subramanian wrote: >> >> +1 >> >> On Thu, Jul 7, 2016 at 9:53 AM, Henry Cai >> wrote: >> >>> +1 >>> >>> On Thu, Jul 7, 2016 at 6:48 AM, Michael

[jira] [Commented] (KAFKA-3933) Kafka OOM During Log Recovery Due to Leaked Native Memory

[ https://issues.apache.org/jira/browse/KAFKA-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15367625#comment-15367625 ] Tom Crayford commented on KAFKA-3933: - Ismael: I've pushed the PR here:

[jira] [Commented] (KAFKA-3933) Kafka OOM During Log Recovery Due to Leaked Native Memory

[ https://issues.apache.org/jira/browse/KAFKA-3933?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15367613#comment-15367613 ] ASF GitHub Bot commented on KAFKA-3933: --- GitHub user tcrayford opened a pull request:

[GitHub] kafka pull request #1598: KAFKA-3933: close deepIterator during log recovery

GitHub user tcrayford opened a pull request: https://github.com/apache/kafka/pull/1598 KAFKA-3933: close deepIterator during log recovery Avoids leaking native memory and hence crashing brokers on bootup due to running out of memory. Introduces

Re: [DISCUSS] Client Side Auto Topic Creation

I think the use case for not blowing up the consumer is simply to not create an implicit ordering in which your services have to come up. On 07/07/2016 04:47 PM, Jun Rao wrote: It seems that it makes sense for the writer to trigger auto topic creation, but not the reader. So, my preference is

[jira] [Commented] (KAFKA-3919) Broker faills to start after ungraceful shutdown due to non-monotonically incrementing offsets in logs

[ https://issues.apache.org/jira/browse/KAFKA-3919?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15367489#comment-15367489 ] Andy Coates commented on KAFKA-3919: [~junrao] Yes, we lost a good number of brokers in a power

Re: [VOTE] KIP-67: Queryable state for Kafka Streams

+1 (non-binding) > On 7 Jul 2016, at 18:31, Sriram Subramanian wrote: > > +1 > > On Thu, Jul 7, 2016 at 9:53 AM, Henry Cai > wrote: > >> +1 >> >> On Thu, Jul 7, 2016 at 6:48 AM, Michael Noll wrote: >> >>> +1

Re: SSL: Broker works but consumer/producer fail

Hi Vineet, The setup of ssl Kafka requires to make one too many steps precise correct. I have listed some below . Hope it helps https://github.com/Symantec/kafka-security-0.9 Sent from my iPhone Regards > On Jul 7, 2016, at 4:49 PM, Vineet Kumar wrote: > > Hi > I

Re: Using KIP Process in Apache Flink

Thanks, appreciate it! :-) On Thu, 7 Jul 2016 at 17:21 Ismael Juma wrote: > Hi Aljoscha, > > Thanks for sharing your intention to use a process similar to our KIP > process. You are more than welcome to copy the structures and docs that we > have for the KIP process. :) > >

Kafka 0.10 Java Client TimeoutException: Batch containing 1 record(s) expired

Hi all, I have a single node, multi (3) broker Zookeeper / Kafka setup. I am using the Kafka 0.10 Java client. I wrote following simple remote (on a different Server than Kafka) Producer (in the code I replaced my public IP address with MYIP): Properties config = new Properties(); try {