[jira] [Commented] (KAFKA-4017) Return more helpful responses when misconfigured connectors are submitted

[Ewen Cheslack-Postava (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4017?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15407224#comment-15407224
 ] 

Ewen Cheslack-Postava commented on KAFKA-4017:
--

[~cotedm] Have the config that caused the problem handy? Probably not 
incredibly hard to reproduce, but would help to make sure we address the 
original problem. In particular, the error doesn't look like something I 
normally see, though I expect given the error message that you included a 
foo=bar in the JSON instead of foo: bar?

> Return more helpful responses when misconfigured connectors are submitted
> -
>
> Key: KAFKA-4017
> URL: https://issues.apache.org/jira/browse/KAFKA-4017
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Affects Versions: 0.10.0.0
>Reporter: Dustin Cote
>Assignee: Ewen Cheslack-Postava
>
> Currently if a user submits a connector with a malformed configuration with 
> connect in distributed mode, the response is:
> {code}
> 
> 
> 
> Error 500 
> 
> 
> HTTP ERROR: 500
> Problem accessing /connectors. Reason:
> Request failed.
> Powered by Jetty://
> 
> 
> {code}
> If the user decides to then go look at the connect server side logging, they 
> can maybe parse the stack traces to find out what happened, but are at first 
> greeted by:
> {code}
> [2016-08-03 16:14:07,797] WARN /connectors 
> (org.eclipse.jetty.server.HttpChannel:384)
> java.lang.NoSuchMethodError: 
> javax.servlet.http.HttpServletRequest.isAsyncStarted()Z
> at 
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:684)
> at 
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
> at 
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1127)
> at 
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:515)
> at 
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:185)
> at 
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1061)
> at 
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:141)
> at 
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:110)
> at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at 
> org.eclipse.jetty.server.handler.StatisticsHandler.handle(StatisticsHandler.java:159)
> at 
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:97)
> at org.eclipse.jetty.server.Server.handle(Server.java:499)
> at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:311)
> at 
> org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:257)
> at 
> org.eclipse.jetty.io.AbstractConnection$2.run(AbstractConnection.java:544)
> at 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:635)
> at 
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:555)
> at java.lang.Thread.run(Thread.java:745)
> {code}
> It would be better if Connect can handle this scenario more gracefully and 
> make it more clear what the problem is even directly to the client.  In the 
> example above, you can eventually locate the problem in the server logs as:
> {code}
> [2016-08-03 16:14:07,795] WARN  (org.eclipse.jetty.servlet.ServletHandler:620)
> javax.servlet.ServletException: 
> org.glassfish.jersey.server.ContainerException: 
> com.fasterxml.jackson.databind.JsonMappingException: Unexpected character 
> ('=' (code 61)): was expecting a colon to separate field name and value
>  at [Source: 
> org.glassfish.jersey.message.internal.ReaderInterceptorExecutor$UnCloseableInputStream@20fb9cff;
>  line: 1, column: 147] (through reference chain: 
> org.apache.kafka.connect.runtime.rest.entities.CreateConnectorRequest["config"])
> at 
> org.glassfish.jersey.servlet.WebComponent.serviceImpl(WebComponent.java:489)
> at 
> org.glassfish.jersey.servlet.WebComponent.service(WebComponent.java:427)
> at 
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:388)
> at 
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:341)
> at 
> org.glassfish.jersey.servlet.ServletContainer.service(ServletContainer.java:228)
> at 
> org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:812)
> at 
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:587)
> at 
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:221)
> at 
> 

Jenkins build is back to normal : kafka-trunk-jdk8 #799

[Apache Jenkins Server]

See 

Build failed in Jenkins: kafka-trunk-jdk8 #798

[Apache Jenkins Server]

See 

Changes:

[wangguoz] MINOR: rename StateStoreProvider.getStores() ->

--
[...truncated 8287 lines...]
org.apache.kafka.common.metrics.MetricsTest > testPercentiles STARTED

org.apache.kafka.common.metrics.MetricsTest > testPercentiles PASSED

org.apache.kafka.common.metrics.MetricsTest > testSampledStatInitialValue 
STARTED

org.apache.kafka.common.metrics.MetricsTest > testSampledStatInitialValue PASSED

org.apache.kafka.common.metrics.MetricsTest > testDuplicateMetricName STARTED

org.apache.kafka.common.metrics.MetricsTest > testDuplicateMetricName PASSED

org.apache.kafka.common.metrics.MetricsTest > testQuotas STARTED

org.apache.kafka.common.metrics.MetricsTest > testQuotas PASSED

org.apache.kafka.common.metrics.MetricsTest > testHierarchicalSensors STARTED

org.apache.kafka.common.metrics.MetricsTest > testHierarchicalSensors PASSED

org.apache.kafka.common.metrics.JmxReporterTest > testJmxRegistration STARTED

org.apache.kafka.common.metrics.JmxReporterTest > testJmxRegistration PASSED

org.apache.kafka.common.metrics.stats.HistogramTest > testHistogram STARTED

org.apache.kafka.common.metrics.stats.HistogramTest > testHistogram PASSED

org.apache.kafka.common.metrics.stats.HistogramTest > testConstantBinScheme 
STARTED

org.apache.kafka.common.metrics.stats.HistogramTest > testConstantBinScheme 
PASSED

org.apache.kafka.common.metrics.stats.HistogramTest > testLinearBinScheme 
STARTED

org.apache.kafka.common.metrics.stats.HistogramTest > testLinearBinScheme PASSED

org.apache.kafka.common.utils.CrcTest > testUpdateInt STARTED

org.apache.kafka.common.utils.CrcTest > testUpdateInt PASSED

org.apache.kafka.common.utils.CrcTest > testUpdate STARTED

org.apache.kafka.common.utils.CrcTest > testUpdate PASSED

org.apache.kafka.common.utils.UtilsTest > testAbs STARTED

org.apache.kafka.common.utils.UtilsTest > testAbs PASSED

org.apache.kafka.common.utils.UtilsTest > testMin STARTED

org.apache.kafka.common.utils.UtilsTest > testMin PASSED

org.apache.kafka.common.utils.UtilsTest > testJoin STARTED

org.apache.kafka.common.utils.UtilsTest > testJoin PASSED

org.apache.kafka.common.utils.UtilsTest > testReadBytes STARTED

org.apache.kafka.common.utils.UtilsTest > testReadBytes PASSED

org.apache.kafka.common.utils.UtilsTest > testGetHost STARTED

org.apache.kafka.common.utils.UtilsTest > testGetHost PASSED

org.apache.kafka.common.utils.UtilsTest > testGetPort STARTED

org.apache.kafka.common.utils.UtilsTest > testGetPort PASSED

org.apache.kafka.common.utils.UtilsTest > testCloseAll STARTED

org.apache.kafka.common.utils.UtilsTest > testCloseAll PASSED

org.apache.kafka.common.utils.UtilsTest > testFormatAddress STARTED

org.apache.kafka.common.utils.UtilsTest > testFormatAddress PASSED

org.apache.kafka.common.utils.AbstractIteratorTest > testEmptyIterator STARTED

org.apache.kafka.common.utils.AbstractIteratorTest > testEmptyIterator PASSED

org.apache.kafka.common.utils.AbstractIteratorTest > testIterator STARTED

org.apache.kafka.common.utils.AbstractIteratorTest > testIterator PASSED

org.apache.kafka.common.ClusterTest > testBootstrap STARTED

org.apache.kafka.common.ClusterTest > testBootstrap PASSED

org.apache.kafka.common.cache.LRUCacheTest > testEviction STARTED

org.apache.kafka.common.cache.LRUCacheTest > testEviction PASSED

org.apache.kafka.common.cache.LRUCacheTest > testPutGet STARTED

org.apache.kafka.common.cache.LRUCacheTest > testPutGet PASSED

org.apache.kafka.common.cache.LRUCacheTest > testRemove STARTED

org.apache.kafka.common.cache.LRUCacheTest > testRemove PASSED

org.apache.kafka.common.network.SslTransportLayerTest > 
testClientAuthenticationRequestedValidProvided STARTED

org.apache.kafka.common.network.SslTransportLayerTest > 
testClientAuthenticationRequestedValidProvided PASSED

org.apache.kafka.common.network.SslTransportLayerTest > testUnsupportedCiphers 
STARTED

org.apache.kafka.common.network.SslTransportLayerTest > testUnsupportedCiphers 
PASSED

org.apache.kafka.common.network.SslTransportLayerTest > 
testUnsupportedTLSVersion STARTED

org.apache.kafka.common.network.SslTransportLayerTest > 
testUnsupportedTLSVersion PASSED

org.apache.kafka.common.network.SslTransportLayerTest > 
testClientAuthenticationRequiredNotProvided STARTED

org.apache.kafka.common.network.SslTransportLayerTest > 
testClientAuthenticationRequiredNotProvided PASSED

org.apache.kafka.common.network.SslTransportLayerTest > 
testClientAuthenticationRequestedNotProvided STARTED

org.apache.kafka.common.network.SslTransportLayerTest > 
testClientAuthenticationRequestedNotProvided PASSED

org.apache.kafka.common.network.SslTransportLayerTest > 
testInvalidKeystorePassword STARTED

org.apache.kafka.common.network.SslTransportLayerTest > 
testInvalidKeystorePassword PASSED

org.apache.kafka.common.network.SslTransportLayerTest > 
testClientAuthenticationDisabledNotProvided STARTED

[jira] [Comment Edited] (KAFKA-3989) Add JMH module for Benchmarks

[Bill Bejeck (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15407054#comment-15407054
 ] 

Bill Bejeck edited comment on KAFKA-3989 at 8/4/16 3:03 AM:


I opted to try the gradle shadow plugin first 
(https://github.com/johnrengelman/shadow/) 

I've been able to get JMH working as a sub-module but needed to make the 
following changes to the build.gradle script:

dependencies {  
  .
  // Added this entry for gradle shadow plugin
classpath 'com.github.jengelman.gradle.plugins:shadow:1.2.3' //
  }

Then added this to apply plugin the jmh-benchmarks module:

project(':jmh-benchmarks') {
  apply plugin: 'com.github.johnrengelman.shadow'
}

Is this acceptable ? 



was (Author: bbejeck):
I opted to try the gradle shadow plugin first 
(https://github.com/johnrengelman/shadow/) 

I've been able to get JMH working as a sub-module but needed to make the 
following changes to the build.gradle script:

`dependencies {  
  .
  // Added this entry for gradle shadow plugin
classpath 'com.github.jengelman.gradle.plugins:shadow:1.2.3' //
  }
`
Then added this to apply plugin the jmh-benchmarks module:

project(':jmh-benchmarks') {
  apply plugin: 'com.github.johnrengelman.shadow'
}

Is this acceptable ? 


> Add JMH module for Benchmarks
> -
>
> Key: KAFKA-3989
> URL: https://issues.apache.org/jira/browse/KAFKA-3989
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Bill Bejeck
>Assignee: Bill Bejeck
>
> JMH is a Java harness for building, running, and analyzing benchmarks written 
> in Java or JVM languages.  To run properly JMH needs to be in it's own 
> module.   This task will also investigate using the jmh -gradle pluging 
> [https://github.com/melix/jmh-gradle-plugin] which enables the use of JMH 
> from gradle.  This is related to 
> [https://issues.apache.org/jira/browse/KAFKA-3973]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3989) Add JMH module for Benchmarks

[Bill Bejeck (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15407054#comment-15407054
 ] 

Bill Bejeck commented on KAFKA-3989:


I opted to try the gradle shadow plugin first 
(https://github.com/johnrengelman/shadow/) 

I've been able to get JMH working as a sub-module but needed to make the 
following changes to the build.gradle script:

dependencies {  
  .
  // Added this entry for gradle shadow plugin
classpath 'com.github.jengelman.gradle.plugins:shadow:1.2.3' //
  }

Then added this to apply plugin the jmh-benchmarks module:

project(':jmh-benchmarks') {
  apply plugin: 'com.github.johnrengelman.shadow'
}

Is this acceptable ? 


> Add JMH module for Benchmarks
> -
>
> Key: KAFKA-3989
> URL: https://issues.apache.org/jira/browse/KAFKA-3989
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Bill Bejeck
>Assignee: Bill Bejeck
>
> JMH is a Java harness for building, running, and analyzing benchmarks written 
> in Java or JVM languages.  To run properly JMH needs to be in it's own 
> module.   This task will also investigate using the jmh -gradle pluging 
> [https://github.com/melix/jmh-gradle-plugin] which enables the use of JMH 
> from gradle.  This is related to 
> [https://issues.apache.org/jira/browse/KAFKA-3973]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-3989) Add JMH module for Benchmarks

[Bill Bejeck (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3989?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15407054#comment-15407054
 ] 

Bill Bejeck edited comment on KAFKA-3989 at 8/4/16 3:03 AM:


I opted to try the gradle shadow plugin first 
(https://github.com/johnrengelman/shadow/) 

I've been able to get JMH working as a sub-module but needed to make the 
following changes to the build.gradle script:

`dependencies {  
  .
  // Added this entry for gradle shadow plugin
classpath 'com.github.jengelman.gradle.plugins:shadow:1.2.3' //
  }
`
Then added this to apply plugin the jmh-benchmarks module:

project(':jmh-benchmarks') {
  apply plugin: 'com.github.johnrengelman.shadow'
}

Is this acceptable ? 



was (Author: bbejeck):
I opted to try the gradle shadow plugin first 
(https://github.com/johnrengelman/shadow/) 

I've been able to get JMH working as a sub-module but needed to make the 
following changes to the build.gradle script:

dependencies {  
  .
  // Added this entry for gradle shadow plugin
classpath 'com.github.jengelman.gradle.plugins:shadow:1.2.3' //
  }

Then added this to apply plugin the jmh-benchmarks module:

project(':jmh-benchmarks') {
  apply plugin: 'com.github.johnrengelman.shadow'
}

Is this acceptable ? 


> Add JMH module for Benchmarks
> -
>
> Key: KAFKA-3989
> URL: https://issues.apache.org/jira/browse/KAFKA-3989
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Bill Bejeck
>Assignee: Bill Bejeck
>
> JMH is a Java harness for building, running, and analyzing benchmarks written 
> in Java or JVM languages.  To run properly JMH needs to be in it's own 
> module.   This task will also investigate using the jmh -gradle pluging 
> [https://github.com/melix/jmh-gradle-plugin] which enables the use of JMH 
> from gradle.  This is related to 
> [https://issues.apache.org/jira/browse/KAFKA-3973]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3894) Log Cleaner thread crashes and never restarts

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406970#comment-15406970
 ] 

Jun Rao commented on KAFKA-3894:


[~tcrayford-heroku], I chatted with [~jkreps] on this a bit. There are a couple 
things that we can do to address this issue.

a. We can potentially make the allocation of the dedup buffer more dynamic. We 
can start with something small like 100MB. If needed, we can grow the dedup 
buffer up to the configured size. This will allow us to set a larger default 
dedup buffer size (say 1GB). If there are not lots of keys, the broker won't be 
using that much memory. This will allow the default configuration to 
accommodate more keys.

b. To handle the edge case where a segment still has more keys than the 
increased dedup buffer can handle. We can do the #3 approach as you suggested. 
Basically, if the dedup buffer is full when only a partial segment is loaded, 
we remember the next offset (say L). We scan all old log segments including 
this one as before. The only difference is that when scanning the last segment, 
we force creating a new segment starting at offset L and simply copy the 
existing messages after L to the new segment. Then, after we swapped in the new 
segments, we will move the cleaner marker to offset L. This adds a bit of 
inefficiency since we have to scan the last swapped-in segment again. However, 
this will allow the cleaner to always make progress regardless of the # of 
keys. I am not sure that I understand the case you mentioned that won't work in 
both approach #3 and #4.

> Log Cleaner thread crashes and never restarts
> -
>
> Key: KAFKA-3894
> URL: https://issues.apache.org/jira/browse/KAFKA-3894
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Affects Versions: 0.8.2.2, 0.9.0.1
> Environment: Oracle JDK 8
> Ubuntu Precise
>Reporter: Tim Carey-Smith
>  Labels: compaction
>
> The log-cleaner thread can crash if the number of keys in a topic grows to be 
> too large to fit into the dedupe buffer. 
> The result of this is a log line: 
> {quote}
> broker=0 pri=ERROR t=kafka-log-cleaner-thread-0 at=LogCleaner 
> \[kafka-log-cleaner-thread-0\], Error due to  
> java.lang.IllegalArgumentException: requirement failed: 9750860 messages in 
> segment MY_FAVORITE_TOPIC-2/47580165.log but offset map can fit 
> only 5033164. You can increase log.cleaner.dedupe.buffer.size or decrease 
> log.cleaner.threads
> {quote}
> As a result, the broker is left in a potentially dangerous situation where 
> cleaning of compacted topics is not running. 
> It is unclear if the broader strategy for the {{LogCleaner}} is the reason 
> for this upper bound, or if this is a value which must be tuned for each 
> specific use-case. 
> Of more immediate concern is the fact that the thread crash is not visible 
> via JMX or exposed as some form of service degradation. 
> Some short-term remediations we have made are:
> * increasing the size of the dedupe buffer
> * monitoring the log-cleaner threads inside the JVM



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-4018) Streams causing older slf4j-log4j library to be packaged along with newer version

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma reassigned KAFKA-4018:
--

Assignee: Ismael Juma

> Streams causing older slf4j-log4j library to be packaged along with newer 
> version
> -
>
> Key: KAFKA-4018
> URL: https://issues.apache.org/jira/browse/KAFKA-4018
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ismael Juma
>Assignee: Ismael Juma
> Fix For: 0.10.0.1
>
>
> This is a regression caused by 
> https://github.com/apache/kafka/commit/0bb1d3ae53fc3f80fb096369951187630519eb04.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-4018) Streams causing older slf4j-log4j library to be packaged along with newer version

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4018?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-4018.

Resolution: Fixed

Issue resolved by pull request 1704
[https://github.com/apache/kafka/pull/1704]

> Streams causing older slf4j-log4j library to be packaged along with newer 
> version
> -
>
> Key: KAFKA-4018
> URL: https://issues.apache.org/jira/browse/KAFKA-4018
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ismael Juma
> Fix For: 0.10.0.1
>
>
> This is a regression caused by 
> https://github.com/apache/kafka/commit/0bb1d3ae53fc3f80fb096369951187630519eb04.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1704: KAFKA-4018: Streams causing older slf4j-log4j libr...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1704


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-4018) Streams causing older slf4j-log4j library to be packaged along with newer version

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406936#comment-15406936
 ] 

ASF GitHub Bot commented on KAFKA-4018:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1704


> Streams causing older slf4j-log4j library to be packaged along with newer 
> version
> -
>
> Key: KAFKA-4018
> URL: https://issues.apache.org/jira/browse/KAFKA-4018
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ismael Juma
> Fix For: 0.10.0.1
>
>
> This is a regression caused by 
> https://github.com/apache/kafka/commit/0bb1d3ae53fc3f80fb096369951187630519eb04.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1211) Hold the produce request with ack > 1 in purgatory until replicas' HW has larger than the produce offset

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406915#comment-15406915
 ] 

Jun Rao commented on KAFKA-1211:


[~fpj], very good questions.

1. Yes, the idea is for the follower to copy LGS from the leader. About the 
possibility of leading to an inconsistent state. We just need to make sure the 
log is consistent with respect to the local leader-generation-checkpoint file 
up to the log end offset. One potential issue with the current proposal is when 
the follower truncates the file and then flushes the checkpoint file. If the 
follower crashes at this point and the truncation hasn't been flushed, we may 
treat some of the messages after the truncation point  to be in a wrong leader 
generation. To fix that, we can change the protocol a bit. The basic idea is 
that the follower will never flush the checkpoint ahead of the log. Specially, 
when the follower gets the LGS from the leader, it stores it in memory. After 
truncation, the follower only flushes the prefix of LGS whose start offset is 
up to the log end offset. As the follower starts fetching data, everytime the 
fetched messages cross the leader generation boundary (according to the cached 
LGS), the follower will add a new lead generation entry to the checkpoint file 
and flushes it.

2. LLG doesn't have to be persisted and only needs to be cached in memory. The 
idea of LLG is really to detect any leader generation changes since the 
follower issued the RetreiveLeaderGeneration request. Once this is detected, 
the follower can handle it properly. If the follower crashes and restarts, it 
can always re-get the LLG from the current leader.

> Hold the produce request with ack > 1 in purgatory until replicas' HW has 
> larger than the produce offset
> 
>
> Key: KAFKA-1211
> URL: https://issues.apache.org/jira/browse/KAFKA-1211
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
> Fix For: 0.11.0.0
>
>
> Today during leader failover we will have a weakness period when the 
> followers truncate their data before fetching from the new leader, i.e., 
> number of in-sync replicas is just 1. If during this time the leader has also 
> failed then produce requests with ack >1 that have get responded will still 
> be lost. To avoid this scenario we would prefer to hold the produce request 
> in purgatory until replica's HW has larger than the offset instead of just 
> their end-of-log offsets.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3752) Provide a way for KStreams to recover from unclean shutdown

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406893#comment-15406893
 ] 

Guozhang Wang commented on KAFKA-3752:
--

[~theduderog] Do you still have the logs on the server side?

> Provide a way for KStreams to recover from unclean shutdown
> ---
>
> Key: KAFKA-3752
> URL: https://issues.apache.org/jira/browse/KAFKA-3752
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Affects Versions: 0.10.0.0
>Reporter: Roger Hoover
>  Labels: architecture
>
> If a KStream application gets killed with SIGKILL (e.g. by the Linux OOM 
> Killer), it may leave behind lock files and fail to recover.
> It would be useful to have an options (say --force) to tell KStreams to 
> proceed even if it finds old LOCK files.
> {noformat}
> [2016-05-24 17:37:52,886] ERROR Failed to create an active task #0_0 in 
> thread [StreamThread-1]:  
> (org.apache.kafka.streams.processor.internals.StreamThread:583)
> org.apache.kafka.streams.errors.ProcessorStateException: Error while creating 
> the state manager
>   at 
> org.apache.kafka.streams.processor.internals.AbstractTask.(AbstractTask.java:71)
>   at 
> org.apache.kafka.streams.processor.internals.StreamTask.(StreamTask.java:86)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.createStreamTask(StreamThread.java:550)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.addStreamTasks(StreamThread.java:577)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.access$000(StreamThread.java:68)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread$1.onPartitionsAssigned(StreamThread.java:123)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.onJoinComplete(ConsumerCoordinator.java:222)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$1.onSuccess(AbstractCoordinator.java:232)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$1.onSuccess(AbstractCoordinator.java:227)
>   at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133)
>   at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107)
>   at 
> org.apache.kafka.clients.consumer.internals.RequestFuture$2.onSuccess(RequestFuture.java:182)
>   at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133)
>   at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$SyncGroupResponseHandler.handle(AbstractCoordinator.java:436)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$SyncGroupResponseHandler.handle(AbstractCoordinator.java:422)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:679)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator$CoordinatorResponseHandler.onSuccess(AbstractCoordinator.java:658)
>   at 
> org.apache.kafka.clients.consumer.internals.RequestFuture$1.onSuccess(RequestFuture.java:167)
>   at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.fireSuccess(RequestFuture.java:133)
>   at 
> org.apache.kafka.clients.consumer.internals.RequestFuture.complete(RequestFuture.java:107)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient$RequestFutureCompletionHandler.onComplete(ConsumerNetworkClient.java:426)
>   at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:278)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.clientPoll(ConsumerNetworkClient.java:360)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:224)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:192)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerNetworkClient.poll(ConsumerNetworkClient.java:163)
>   at 
> org.apache.kafka.clients.consumer.internals.AbstractCoordinator.ensureActiveGroup(AbstractCoordinator.java:243)
>   at 
> org.apache.kafka.clients.consumer.internals.ConsumerCoordinator.ensurePartitionAssignment(ConsumerCoordinator.java:345)
>   at 
> org.apache.kafka.clients.consumer.KafkaConsumer.pollOnce(KafkaConsumer.java:977)
>   at 
> org.apache.kafka.clients.consumer.KafkaConsumer.poll(KafkaConsumer.java:937)
>   at 
> org.apache.kafka.streams.processor.internals.StreamThread.runLoop(StreamThread.java:295)
>   at 
> 

Re: [kafka-clients] [VOTE] 0.10.0.1 RC1

[Ismael Juma]

Thanks Manikumar. I filed KAFKA-4018 with the details of when we regressed
as well as a fix.

Ismael

On Wed, Aug 3, 2016 at 4:18 PM, Manikumar Reddy 
wrote:

> Hi,
>
> There are two versions of slf4j-log4j jar in the build. (1.6.1, 1.7.21).
> slf4j-log4j12-1.6.1.jar is coming from streams:examples module.
>
> Thanks,
> Manikumar
>
> On Tue, Aug 2, 2016 at 8:31 PM, Ismael Juma  wrote:
>
>> Hello Kafka users, developers and client-developers,
>>
>> This is the second candidate for the release of Apache Kafka 0.10.0.1.
>> This is a bug fix release and it includes fixes and improvements from 52
>> JIRAs (including a few critical bugs). See the release notes for more
>> details:
>>
>> http://home.apache.org/~ijuma/kafka-0.10.0.1-rc1/RELEASE_NOTES.html
>>
>> When compared to RC0, RC1 contains fixes for two bugs (KAFKA-4008
>> and KAFKA-3950) and a couple of test stabilisation fixes.
>>
>> *** Please download, test and vote by Friday, 5 August, 8am PT ***
>>
>> Kafka's KEYS file containing PGP keys we use to sign the release:
>> http://kafka.apache.org/KEYS
>>
>> * Release artifacts to be voted upon (source and binary):
>> http://home.apache.org/~ijuma/kafka-0.10.0.1-rc1/
>>
>> * Maven artifacts to be voted upon:
>> https://repository.apache.org/content/groups/staging
>>
>> * Javadoc:
>> http://home.apache.org/~ijuma/kafka-0.10.0.1-rc1/javadoc/
>>
>> * Tag to be voted upon (off 0.10.0 branch) is the 0.10.0.1-rc1 tag:
>>
>> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=108580e4594d694827c953264969fe1ce2a7
>>
>> * Documentation:
>> http://kafka.apache.org/0100/documentation.html
>>
>> * Protocol:
>> http://kafka.apache.org/0100/protocol.html
>>
>> * Successful Jenkins builds for the 0.10.0 branch:
>> Unit/integration tests: *https://builds.apache.org/job/kafka-0.10.0-jdk7/179/
>> *
>> System tests: *https://jenkins.confluent.io/job/system-test-kafka-0.10.0/136/
>> *
>>
>> Thanks,
>> Ismael
>>
>> --
>> You received this message because you are subscribed to the Google Groups
>> "kafka-clients" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to kafka-clients+unsubscr...@googlegroups.com.
>> To post to this group, send email to kafka-clie...@googlegroups.com.
>> Visit this group at https://groups.google.com/group/kafka-clients.
>> To view this discussion on the web visit
>> https://groups.google.com/d/msgid/kafka-clients/CAD5tkZaRxAjQbwS_1q4MqskSYKxQWBFmdPVf_PP020bjY9%3DCgQ%40mail.gmail.com
>> 
>> .
>> For more options, visit https://groups.google.com/d/optout.
>>
>
>

[jira] [Commented] (KAFKA-4018) Streams causing older slf4j-log4j library to be packaged along with newer version

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406855#comment-15406855
 ] 

ASF GitHub Bot commented on KAFKA-4018:
---

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/1704

KAFKA-4018: Streams causing older slf4j-log4j library to be packaged along 
with newer version

This is a regression caused by 0bb1d3ae.

After that commit, Streams no longer has a direct dependency on 
slf4j-log4j12, but zkclient
has a dependency on an older version of slf4j-log4j12, so we get a 
transitive dependency on
the older version.

The fix is to simply exclude the undesired dependencies from the zkclient 
dependency.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka 
kafka-4018-streams-duplicate-slf4j-log4j

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1704.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1704


commit 79ef53042dccf1604b68aa470ec6f62810d0c853
Author: Ismael Juma 
Date:   2016-08-03T23:54:56Z

MINOR: Streams causing older slf4j-log4j library to be packaged along with 
newer version




> Streams causing older slf4j-log4j library to be packaged along with newer 
> version
> -
>
> Key: KAFKA-4018
> URL: https://issues.apache.org/jira/browse/KAFKA-4018
> Project: Kafka
>  Issue Type: Bug
>Reporter: Ismael Juma
> Fix For: 0.10.0.1
>
>
> This is a regression caused by 
> https://github.com/apache/kafka/commit/0bb1d3ae53fc3f80fb096369951187630519eb04.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1704: KAFKA-4018: Streams causing older slf4j-log4j libr...

[ijuma]

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/1704

KAFKA-4018: Streams causing older slf4j-log4j library to be packaged along 
with newer version

This is a regression caused by 0bb1d3ae.

After that commit, Streams no longer has a direct dependency on 
slf4j-log4j12, but zkclient
has a dependency on an older version of slf4j-log4j12, so we get a 
transitive dependency on
the older version.

The fix is to simply exclude the undesired dependencies from the zkclient 
dependency.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka 
kafka-4018-streams-duplicate-slf4j-log4j

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1704.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1704


commit 79ef53042dccf1604b68aa470ec6f62810d0c853
Author: Ismael Juma 
Date:   2016-08-03T23:54:56Z

MINOR: Streams causing older slf4j-log4j library to be packaged along with 
newer version




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (KAFKA-4018) Streams causing older slf4j-log4j library to be packaged along with newer version

[Ismael Juma (JIRA)]

Ismael Juma created KAFKA-4018:
--

 Summary: Streams causing older slf4j-log4j library to be packaged 
along with newer version
 Key: KAFKA-4018
 URL: https://issues.apache.org/jira/browse/KAFKA-4018
 Project: Kafka
  Issue Type: Bug
Reporter: Ismael Juma
 Fix For: 0.10.0.1


This is a regression caused by 
https://github.com/apache/kafka/commit/0bb1d3ae53fc3f80fb096369951187630519eb04.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Jenkins build is back to normal : kafka-0.10.0-jdk7 #181

[Apache Jenkins Server]

See 

Jenkins build is back to normal : kafka-trunk-jdk7 #1459

[Apache Jenkins Server]

See 

[jira] [Commented] (KAFKA-3875) Transient test failure: kafka.api.SslProducerSendTest.testSendNonCompressedMessageWithCreateTime

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406740#comment-15406740
 ] 

ASF GitHub Bot commented on KAFKA-3875:
---

GitHub user junrao opened a pull request:

https://github.com/apache/kafka/pull/1703

KAFKA-3875: Transient test failure: 
kafka.api.SslProducerSendTest.testSendNonCompressedMessageWithCreateTime

1. The IllegalStateException is actually thrown from 
testCloseWithZeroTimeoutFromSenderThread() due to a bug. We call 
producer.close() in the callback. Once the first callback is called, producing 
records in the callback will hit the IllegalStateException. This only pollutes 
the output, but doesn't fail the test. I fixed this by only calling 
producer.send() in the first callback.
2. It's not clear which test throws TimeoutException and it's not 
reproducible locally. One thing is that the error message in TimeoutException 
is mis-leading since the timeout is not necessarily due to metadata. Improved 
this by making the error message in TimeoutException clearer.
3. It's not clear what actually failed 
testSendNonCompressedMessageWithCreateTime(). One thing I found is that since 
we set the linger time to MAX_LONG and are sending small messages, those 
produced messages won't be drained until we call producer.close(1L, 
TimeUnit.MILLISECONDS). Normally, 10 secs should be enough for the records to 
be sent. My only hypothesis is that since SSL is more expensive, occasionally, 
10 secs is still not enough. So, I bumped up the timeout from 10 secs to 20 
secs.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/junrao/kafka kafka-3875

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1703.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1703


commit c67640e7a45f95ff68cd140795fb9058c11f3dcb
Author: Jun Rao 
Date:   2016-08-03T22:29:25Z

KAFKA-3875: Transient test failure: 
kafka.api.SslProducerSendTest.testSendNonCompressedMessageWithCreateTime




> Transient test failure: 
> kafka.api.SslProducerSendTest.testSendNonCompressedMessageWithCreateTime
> 
>
> Key: KAFKA-3875
> URL: https://issues.apache.org/jira/browse/KAFKA-3875
> Project: Kafka
>  Issue Type: Sub-task
>  Components: unit tests
>Reporter: Ismael Juma
>Assignee: Jun Rao
>  Labels: transient-unit-test-failure
>
> It failed in a couple of builds.
> {code}
> java.lang.AssertionError: Should have offset 100 but only successfully sent 0 
> expected:<100> but was:<0>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:834)
>   at org.junit.Assert.assertEquals(Assert.java:645)
>   at 
> kafka.api.BaseProducerSendTest.sendAndVerifyTimestamp(BaseProducerSendTest.scala:228)
>   at 
> kafka.api.BaseProducerSendTest.testSendNonCompressedMessageWithCreateTime(BaseProducerSendTest.scala:170)
> {code}
> And standard out:
> {code}
> org.scalatest.junit.JUnitTestFailedError: Send callback returns the following 
> exception
>   at 
> org.scalatest.junit.AssertionsForJUnit$class.newAssertionFailedException(AssertionsForJUnit.scala:103)
>   at 
> org.scalatest.junit.JUnitSuite.newAssertionFailedException(JUnitSuite.scala:79)
>   at org.scalatest.Assertions$class.fail(Assertions.scala:1348)
>   at org.scalatest.junit.JUnitSuite.fail(JUnitSuite.scala:79)
>   at 
> kafka.api.BaseProducerSendTest$callback$4$.onCompletion(BaseProducerSendTest.scala:209)
>   at 
> org.apache.kafka.clients.producer.internals.RecordBatch.done(RecordBatch.java:109)
>   at 
> org.apache.kafka.clients.producer.internals.RecordBatch.maybeExpire(RecordBatch.java:155)
>   at 
> org.apache.kafka.clients.producer.internals.RecordAccumulator.abortExpiredBatches(RecordAccumulator.java:245)
>   at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:211)
>   at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:134)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.kafka.common.errors.TimeoutException: Batch containing 
> 1 record(s) expired due to timeout while requesting metadata from brokers for 
> topic-0
> java.lang.IllegalStateException: Cannot send after the producer is closed.
>   at 
> org.apache.kafka.clients.producer.internals.RecordAccumulator.append(RecordAccumulator.java:172)
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:466)
>   at 
> 

[GitHub] kafka pull request #1703: KAFKA-3875: Transient test failure: kafka.api.SslP...

[junrao]

GitHub user junrao opened a pull request:

https://github.com/apache/kafka/pull/1703

KAFKA-3875: Transient test failure: 
kafka.api.SslProducerSendTest.testSendNonCompressedMessageWithCreateTime

1. The IllegalStateException is actually thrown from 
testCloseWithZeroTimeoutFromSenderThread() due to a bug. We call 
producer.close() in the callback. Once the first callback is called, producing 
records in the callback will hit the IllegalStateException. This only pollutes 
the output, but doesn't fail the test. I fixed this by only calling 
producer.send() in the first callback.
2. It's not clear which test throws TimeoutException and it's not 
reproducible locally. One thing is that the error message in TimeoutException 
is mis-leading since the timeout is not necessarily due to metadata. Improved 
this by making the error message in TimeoutException clearer.
3. It's not clear what actually failed 
testSendNonCompressedMessageWithCreateTime(). One thing I found is that since 
we set the linger time to MAX_LONG and are sending small messages, those 
produced messages won't be drained until we call producer.close(1L, 
TimeUnit.MILLISECONDS). Normally, 10 secs should be enough for the records to 
be sent. My only hypothesis is that since SSL is more expensive, occasionally, 
10 secs is still not enough. So, I bumped up the timeout from 10 secs to 20 
secs.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/junrao/kafka kafka-3875

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1703.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1703


commit c67640e7a45f95ff68cd140795fb9058c11f3dcb
Author: Jun Rao 
Date:   2016-08-03T22:29:25Z

KAFKA-3875: Transient test failure: 
kafka.api.SslProducerSendTest.testSendNonCompressedMessageWithCreateTime




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3875) Transient test failure: kafka.api.SslProducerSendTest.testSendNonCompressedMessageWithCreateTime

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3875?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406737#comment-15406737
 ] 

Jun Rao commented on KAFKA-3875:


I took a look at this and found a few things.

1. The IllegalStateException is actually thrown from 
testCloseWithZeroTimeoutFromSenderThread() due to a bug. We call 
producer.close() in the callback. Once the first callback is called, producing 
records in the callback will hit the IllegalStateException. This only pollutes 
the output,  but doesn't fail the test. I fixed this by only calling 
producer.send() in the first callback.

2. It's not clear which test throws TimeoutException and it's not reproducible 
locally. One thing is that the error message in TimeoutException is mis-leading 
since the timeout is not necessarily due to metadata. Improved this by making 
the error message in TimeoutException clearer.

3. It's not clear what actually failed 
testSendNonCompressedMessageWithCreateTime(). One thing I found is that since 
we set the linger time to MAX_LONG and are sending small messages, those 
produced messages won't be drained until we call producer.close(1L, 
TimeUnit.MILLISECONDS). Normally, 10 secs should be enough for the records to 
be sent. My only hypothesis is that since SSL is more expensive, occasionally, 
10 secs is still not enough. So, I bumped up the timeout from 10 secs to 20 
secs.

> Transient test failure: 
> kafka.api.SslProducerSendTest.testSendNonCompressedMessageWithCreateTime
> 
>
> Key: KAFKA-3875
> URL: https://issues.apache.org/jira/browse/KAFKA-3875
> Project: Kafka
>  Issue Type: Sub-task
>  Components: unit tests
>Reporter: Ismael Juma
>Assignee: Jun Rao
>  Labels: transient-unit-test-failure
>
> It failed in a couple of builds.
> {code}
> java.lang.AssertionError: Should have offset 100 but only successfully sent 0 
> expected:<100> but was:<0>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:834)
>   at org.junit.Assert.assertEquals(Assert.java:645)
>   at 
> kafka.api.BaseProducerSendTest.sendAndVerifyTimestamp(BaseProducerSendTest.scala:228)
>   at 
> kafka.api.BaseProducerSendTest.testSendNonCompressedMessageWithCreateTime(BaseProducerSendTest.scala:170)
> {code}
> And standard out:
> {code}
> org.scalatest.junit.JUnitTestFailedError: Send callback returns the following 
> exception
>   at 
> org.scalatest.junit.AssertionsForJUnit$class.newAssertionFailedException(AssertionsForJUnit.scala:103)
>   at 
> org.scalatest.junit.JUnitSuite.newAssertionFailedException(JUnitSuite.scala:79)
>   at org.scalatest.Assertions$class.fail(Assertions.scala:1348)
>   at org.scalatest.junit.JUnitSuite.fail(JUnitSuite.scala:79)
>   at 
> kafka.api.BaseProducerSendTest$callback$4$.onCompletion(BaseProducerSendTest.scala:209)
>   at 
> org.apache.kafka.clients.producer.internals.RecordBatch.done(RecordBatch.java:109)
>   at 
> org.apache.kafka.clients.producer.internals.RecordBatch.maybeExpire(RecordBatch.java:155)
>   at 
> org.apache.kafka.clients.producer.internals.RecordAccumulator.abortExpiredBatches(RecordAccumulator.java:245)
>   at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:211)
>   at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:134)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.kafka.common.errors.TimeoutException: Batch containing 
> 1 record(s) expired due to timeout while requesting metadata from brokers for 
> topic-0
> java.lang.IllegalStateException: Cannot send after the producer is closed.
>   at 
> org.apache.kafka.clients.producer.internals.RecordAccumulator.append(RecordAccumulator.java:172)
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:466)
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:430)
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:353)
>   at 
> kafka.api.BaseProducerSendTest$CloseCallback$1$$anonfun$onCompletion$1.apply(BaseProducerSendTest.scala:415)
>   at 
> kafka.api.BaseProducerSendTest$CloseCallback$1$$anonfun$onCompletion$1.apply(BaseProducerSendTest.scala:415)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
>   at scala.collection.immutable.Range.foreach(Range.scala:160)
>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
>   at scala.collection.AbstractTraversable.map(Traversable.scala:104)
>   at 
> 

Build failed in Jenkins: kafka-trunk-jdk8 #797

[Apache Jenkins Server]

See 

Changes:

[wangguoz] MINOR: Fixed documentation for KStream left join KStream-KTable

--
[...truncated 1648 lines...]

kafka.api.PlaintextConsumerTest > testPauseStateNotPreservedByRebalance STARTED

kafka.api.PlaintextConsumerTest > testPauseStateNotPreservedByRebalance PASSED

kafka.api.PlaintextConsumerTest > testUnsubscribeTopic STARTED

kafka.api.PlaintextConsumerTest > testUnsubscribeTopic PASSED

kafka.api.PlaintextConsumerTest > testListTopics STARTED

kafka.api.PlaintextConsumerTest > testListTopics PASSED

kafka.api.PlaintextConsumerTest > testAutoCommitOnRebalance STARTED

kafka.api.PlaintextConsumerTest > testAutoCommitOnRebalance PASSED

kafka.api.PlaintextConsumerTest > testSimpleConsumption STARTED

kafka.api.PlaintextConsumerTest > testSimpleConsumption PASSED

kafka.api.PlaintextConsumerTest > testPartitionReassignmentCallback STARTED

kafka.api.PlaintextConsumerTest > testPartitionReassignmentCallback PASSED

kafka.api.PlaintextConsumerTest > testCommitSpecifiedOffsets STARTED

kafka.api.PlaintextConsumerTest > testCommitSpecifiedOffsets PASSED

kafka.api.SslProducerSendTest > testSendNonCompressedMessageWithCreateTime 
STARTED

kafka.api.SslProducerSendTest > testSendNonCompressedMessageWithCreateTime 
PASSED

kafka.api.SslProducerSendTest > testSendCompressedMessageWithLogAppendTime 
STARTED

kafka.api.SslProducerSendTest > testSendCompressedMessageWithLogAppendTime 
PASSED

kafka.api.SslProducerSendTest > testClose STARTED

kafka.api.SslProducerSendTest > testClose PASSED

kafka.api.SslProducerSendTest > testFlush STARTED

kafka.api.SslProducerSendTest > testFlush PASSED

kafka.api.SslProducerSendTest > testSendToPartition STARTED

kafka.api.SslProducerSendTest > testSendToPartition PASSED

kafka.api.SslProducerSendTest > testSendOffset STARTED

kafka.api.SslProducerSendTest > testSendOffset PASSED

kafka.api.SslProducerSendTest > testAutoCreateTopic STARTED

kafka.api.SslProducerSendTest > testAutoCreateTopic PASSED

kafka.api.SslProducerSendTest > testSendWithInvalidCreateTime STARTED

kafka.api.SslProducerSendTest > testSendWithInvalidCreateTime PASSED

kafka.api.SslProducerSendTest > testSendCompressedMessageWithCreateTime STARTED

kafka.api.SslProducerSendTest > testSendCompressedMessageWithCreateTime PASSED

kafka.api.SslProducerSendTest > testCloseWithZeroTimeoutFromCallerThread STARTED

kafka.api.SslProducerSendTest > testCloseWithZeroTimeoutFromCallerThread PASSED

kafka.api.SslProducerSendTest > testCloseWithZeroTimeoutFromSenderThread STARTED

kafka.api.SslProducerSendTest > testCloseWithZeroTimeoutFromSenderThread PASSED

kafka.api.SslProducerSendTest > testSendNonCompressedMessageWithLogApendTime 
STARTED

kafka.api.SslProducerSendTest > testSendNonCompressedMessageWithLogApendTime 
PASSED

kafka.api.AuthorizerIntegrationTest > testCommitWithTopicWrite STARTED

kafka.api.AuthorizerIntegrationTest > testCommitWithTopicWrite PASSED

kafka.api.AuthorizerIntegrationTest > testConsumeWithNoTopicAccess STARTED

kafka.api.AuthorizerIntegrationTest > testConsumeWithNoTopicAccess PASSED

kafka.api.AuthorizerIntegrationTest > 
testCreatePermissionNeededToReadFromNonExistentTopic STARTED

kafka.api.AuthorizerIntegrationTest > 
testCreatePermissionNeededToReadFromNonExistentTopic PASSED

kafka.api.AuthorizerIntegrationTest > testOffsetFetchWithNoAccess STARTED

kafka.api.AuthorizerIntegrationTest > testOffsetFetchWithNoAccess PASSED

kafka.api.AuthorizerIntegrationTest > testListOfsetsWithTopicDescribe STARTED

kafka.api.AuthorizerIntegrationTest > testListOfsetsWithTopicDescribe PASSED

kafka.api.AuthorizerIntegrationTest > testProduceWithTopicRead STARTED

kafka.api.AuthorizerIntegrationTest > testProduceWithTopicRead PASSED

kafka.api.AuthorizerIntegrationTest > testListOffsetsWithNoTopicAccess STARTED

kafka.api.AuthorizerIntegrationTest > testListOffsetsWithNoTopicAccess PASSED

kafka.api.AuthorizerIntegrationTest > 
testCreatePermissionNeededForWritingToNonExistentTopic STARTED

kafka.api.AuthorizerIntegrationTest > 
testCreatePermissionNeededForWritingToNonExistentTopic PASSED

kafka.api.AuthorizerIntegrationTest > testConsumeWithNoGroupAccess STARTED

kafka.api.AuthorizerIntegrationTest > testConsumeWithNoGroupAccess PASSED

kafka.api.AuthorizerIntegrationTest > testConsumeWithTopicDescribe STARTED

kafka.api.AuthorizerIntegrationTest > testConsumeWithTopicDescribe PASSED

kafka.api.AuthorizerIntegrationTest > testOffsetFetchWithNoTopicAccess STARTED

kafka.api.AuthorizerIntegrationTest > testOffsetFetchWithNoTopicAccess PASSED

kafka.api.AuthorizerIntegrationTest > testCommitWithNoTopicAccess STARTED

kafka.api.AuthorizerIntegrationTest > testCommitWithNoTopicAccess PASSED

kafka.api.AuthorizerIntegrationTest > testProduceWithNoTopicAccess STARTED

kafka.api.AuthorizerIntegrationTest > testProduceWithNoTopicAccess PASSED

kafka.api.AuthorizerIntegrationTest > 

[GitHub] kafka pull request #1699: MINOR: rename StateStoreProvider.getStores() -> St...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1699


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-2063) Bound fetch response size

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406730#comment-15406730
 ] 

Guozhang Wang commented on KAFKA-2063:
--

Another note from Streams is that in situations where streams' consumers are 
lagging, then we could likely to be getting one partition at a time in 
{{consumer.poll}}, and hence the buffered data per-partition may not have data 
for all partitions for some periods of time. Hence:

1. If you are doing a join, for example, then the latency may be impacted a bit 
since no outputs will be generated until the consumer has fetched records from 
both partitions. But this impact should be small.

2. We are estimating the stream time based on buffered records' timestamps, and 
hence when there is no data fetched from some of the partitions for some time, 
the stream time will not be advanced, and as a result event-time based 
mechanism such as punctuate function, and window store expiration, etc, will 
not be triggered.

> Bound fetch response size
> -
>
> Key: KAFKA-2063
> URL: https://issues.apache.org/jira/browse/KAFKA-2063
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jay Kreps
>
> Currently the only bound on the fetch response size is 
> max.partition.fetch.bytes * num_partitions. There are two problems:
> 1. First this bound is often large. You may chose 
> max.partition.fetch.bytes=1MB to enable messages of up to 1MB. However if you 
> also need to consume 1k partitions this means you may receive a 1GB response 
> in the worst case!
> 2. The actual memory usage is unpredictable. Partition assignment changes, 
> and you only actually get the full fetch amount when you are behind and there 
> is a full chunk of data ready. This means an application that seems to work 
> fine will suddenly OOM when partitions shift or when the application falls 
> behind.
> We need to decouple the fetch response size from the number of partitions.
> The proposal for doing this would be to add a new field to the fetch request, 
> max_bytes which would control the maximum data bytes we would include in the 
> response.
> The implementation on the server side would grab data from each partition in 
> the fetch request until it hit this limit, then send back just the data for 
> the partitions that fit in the response. The implementation would need to 
> start from a random position in the list of topics included in the fetch 
> request to ensure that in a case of backlog we fairly balance between 
> partitions (to avoid first giving just the first partition until that is 
> exhausted, then the next partition, etc).
> This setting will make the max.partition.fetch.bytes field in the fetch 
> request much less useful and we  should discuss just getting rid of it.
> I believe this also solves the same thing we were trying to address in 
> KAFKA-598. The max_bytes setting now becomes the new limit that would need to 
> be compared to max_message size. This can be much larger--e.g. setting a 50MB 
> max_bytes setting would be okay, whereas now if you set 50MB you may need to 
> allocate 50MB*num_partitions.
> This will require evolving the fetch request protocol version to add the new 
> field and we should do a KIP for it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Preserve offset in Mirror Maker

[Gwen Shapira]

MirrorMaker actually doesn't have a default - it uses what you
configured in the consumer.properties file you use.

Either:
auto.offset.reset = latest (biggest in old versions)
or
auto.offset.reset = earliest (smallest in old versions)

So you can choose whether when MirrorMaker first comes up, if it
starts from beginning or end.

Note that this is just for first start. Any other restart after, it
should use the checkpoints like normal consumers and not lose data
(especially in 0.9.0 and above).

Gwen

On Wed, Aug 3, 2016 at 2:15 PM, Sunil Parmar  wrote:
> We're using mirror maker to mirror data from one data center to another data 
> center ( 1 to 1 ). We noticed that by default the mirror maker by default 
> start from latest offset.  How to change mirror maker producer config to 
> start from last check pointed offset in case of crash without losing data ?
>
> -Sunil
>

[jira] [Assigned] (KAFKA-3875) Transient test failure: kafka.api.SslProducerSendTest.testSendNonCompressedMessageWithCreateTime

[Jun Rao (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3875?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jun Rao reassigned KAFKA-3875:
--

Assignee: Jun Rao

> Transient test failure: 
> kafka.api.SslProducerSendTest.testSendNonCompressedMessageWithCreateTime
> 
>
> Key: KAFKA-3875
> URL: https://issues.apache.org/jira/browse/KAFKA-3875
> Project: Kafka
>  Issue Type: Sub-task
>  Components: unit tests
>Reporter: Ismael Juma
>Assignee: Jun Rao
>  Labels: transient-unit-test-failure
>
> It failed in a couple of builds.
> {code}
> java.lang.AssertionError: Should have offset 100 but only successfully sent 0 
> expected:<100> but was:<0>
>   at org.junit.Assert.fail(Assert.java:88)
>   at org.junit.Assert.failNotEquals(Assert.java:834)
>   at org.junit.Assert.assertEquals(Assert.java:645)
>   at 
> kafka.api.BaseProducerSendTest.sendAndVerifyTimestamp(BaseProducerSendTest.scala:228)
>   at 
> kafka.api.BaseProducerSendTest.testSendNonCompressedMessageWithCreateTime(BaseProducerSendTest.scala:170)
> {code}
> And standard out:
> {code}
> org.scalatest.junit.JUnitTestFailedError: Send callback returns the following 
> exception
>   at 
> org.scalatest.junit.AssertionsForJUnit$class.newAssertionFailedException(AssertionsForJUnit.scala:103)
>   at 
> org.scalatest.junit.JUnitSuite.newAssertionFailedException(JUnitSuite.scala:79)
>   at org.scalatest.Assertions$class.fail(Assertions.scala:1348)
>   at org.scalatest.junit.JUnitSuite.fail(JUnitSuite.scala:79)
>   at 
> kafka.api.BaseProducerSendTest$callback$4$.onCompletion(BaseProducerSendTest.scala:209)
>   at 
> org.apache.kafka.clients.producer.internals.RecordBatch.done(RecordBatch.java:109)
>   at 
> org.apache.kafka.clients.producer.internals.RecordBatch.maybeExpire(RecordBatch.java:155)
>   at 
> org.apache.kafka.clients.producer.internals.RecordAccumulator.abortExpiredBatches(RecordAccumulator.java:245)
>   at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:211)
>   at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:134)
>   at java.lang.Thread.run(Thread.java:745)
> Caused by: org.apache.kafka.common.errors.TimeoutException: Batch containing 
> 1 record(s) expired due to timeout while requesting metadata from brokers for 
> topic-0
> java.lang.IllegalStateException: Cannot send after the producer is closed.
>   at 
> org.apache.kafka.clients.producer.internals.RecordAccumulator.append(RecordAccumulator.java:172)
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.doSend(KafkaProducer.java:466)
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:430)
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.send(KafkaProducer.java:353)
>   at 
> kafka.api.BaseProducerSendTest$CloseCallback$1$$anonfun$onCompletion$1.apply(BaseProducerSendTest.scala:415)
>   at 
> kafka.api.BaseProducerSendTest$CloseCallback$1$$anonfun$onCompletion$1.apply(BaseProducerSendTest.scala:415)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:234)
>   at scala.collection.immutable.Range.foreach(Range.scala:160)
>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:234)
>   at scala.collection.AbstractTraversable.map(Traversable.scala:104)
>   at 
> kafka.api.BaseProducerSendTest$CloseCallback$1.onCompletion(BaseProducerSendTest.scala:415)
>   at 
> org.apache.kafka.clients.producer.internals.RecordBatch.done(RecordBatch.java:107)
>   at 
> org.apache.kafka.clients.producer.internals.Sender.completeBatch(Sender.java:318)
>   at 
> org.apache.kafka.clients.producer.internals.Sender.handleProduceResponse(Sender.java:278)
>   at 
> org.apache.kafka.clients.producer.internals.Sender.access$100(Sender.java:57)
>   at 
> org.apache.kafka.clients.producer.internals.Sender$1.onComplete(Sender.java:364)
>   at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:278)
>   at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:235)
>   at 
> org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:134)
>   at java.lang.Thread.run(Thread.java:745)
> {code}
> https://jenkins.confluent.io/job/kafka-trunk/905/
> https://jenkins.confluent.io/job/kafka-trunk/919 (the output is similar to 
> the first build)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Preserve offset in Mirror Maker

[Sunil Parmar]

We're using mirror maker to mirror data from one data center to another data 
center ( 1 to 1 ). We noticed that by default the mirror maker by default start 
from latest offset.  How to change mirror maker producer config to start from 
last check pointed offset in case of crash without losing data ?

-Sunil

[jira] [Commented] (KAFKA-2063) Bound fetch response size

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2063?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406623#comment-15406623
 ] 

Guozhang Wang commented on KAFKA-2063:
--

Re 1) Currently we use pause / resume per partition to avoid one partition 
getting much more data and hence starving other partitions. And as long as 
there is some round-robin / randomization on the server side, I think it should 
be fine for Streams.

> Bound fetch response size
> -
>
> Key: KAFKA-2063
> URL: https://issues.apache.org/jira/browse/KAFKA-2063
> Project: Kafka
>  Issue Type: Improvement
>Reporter: Jay Kreps
>
> Currently the only bound on the fetch response size is 
> max.partition.fetch.bytes * num_partitions. There are two problems:
> 1. First this bound is often large. You may chose 
> max.partition.fetch.bytes=1MB to enable messages of up to 1MB. However if you 
> also need to consume 1k partitions this means you may receive a 1GB response 
> in the worst case!
> 2. The actual memory usage is unpredictable. Partition assignment changes, 
> and you only actually get the full fetch amount when you are behind and there 
> is a full chunk of data ready. This means an application that seems to work 
> fine will suddenly OOM when partitions shift or when the application falls 
> behind.
> We need to decouple the fetch response size from the number of partitions.
> The proposal for doing this would be to add a new field to the fetch request, 
> max_bytes which would control the maximum data bytes we would include in the 
> response.
> The implementation on the server side would grab data from each partition in 
> the fetch request until it hit this limit, then send back just the data for 
> the partitions that fit in the response. The implementation would need to 
> start from a random position in the list of topics included in the fetch 
> request to ensure that in a case of backlog we fairly balance between 
> partitions (to avoid first giving just the first partition until that is 
> exhausted, then the next partition, etc).
> This setting will make the max.partition.fetch.bytes field in the fetch 
> request much less useful and we  should discuss just getting rid of it.
> I believe this also solves the same thing we were trying to address in 
> KAFKA-598. The max_bytes setting now becomes the new limit that would need to 
> be compared to max_message size. This can be much larger--e.g. setting a 50MB 
> max_bytes setting would be okay, whereas now if you set 50MB you may need to 
> allocate 50MB*num_partitions.
> This will require evolving the fetch request protocol version to add the new 
> field and we should do a KIP for it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1692: MINOR: Fixed documentation for KStream left join K...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1692


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (KAFKA-1143) Consumer should cache topic partition info

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-1143?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-1143.
--
   Resolution: Duplicate
Fix Version/s: (was: 0.10.1.0)

This ticket was proposed for ZK-based high-level consumer and the new consumer 
has already cached metadata.

> Consumer should cache topic partition info
> --
>
> Key: KAFKA-1143
> URL: https://issues.apache.org/jira/browse/KAFKA-1143
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
>
> So that
> 1. It can check if rebalances are necessary when topic/partition watcher 
> fires (they can be triggered for state change event even the data does not 
> change at all).
> 2. Rebalance does not need to read again from ZK.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: Kafka Streams for Remote Server

[Guozhang Wang]

Hi Misha,

If it works locally, then I would still suspect that it is due to a
transient timing issue: note that the create-topic script is non-blocking,
i.e. even when it returns it does not necessarily guarantee that the leader
metadata information has been completed propagating to brokers.



Guozhang


On Thu, Jul 28, 2016 at 1:48 AM, mishadoff  wrote:

> Thanks Guozhang,
>
> Yes, I rely on auto-create, and it works locally. Maybe I need to tweak
> some timeout conf for that?
> Also I identified, that even I manually create a topic, it lists but I can
> not produce messages to this topic with the same exception.
> Producing to other topics works well, so it seems like server problem?
>
> — Misha
>
> > On Jul 28, 2016, at 01:44, Guozhang Wang  wrote:
> >
> > Misha,
> >
> > Did you pre-create the sink topic before starting your application or you
> > are relying on the broker-side auto-create for that topic?
> >
> > If you are relying on auto-create, then there is a transient period where
> > the topic is created but the metadata has not been propagated to the
> > brokers so they do not know they are the leader of the created topic
> > partitions yet. And I'd recommend not relying on it since it is really
> > meant for debugging environment only.
> >
> > Guozhang
> >
> >
> > On Wed, Jul 27, 2016 at 5:45 AM, mishadoff  wrote:
> >
> >> Hello,
> >>
> >> I’ve a simplest ever kafka streams application which just reads from one
> >> kafka topic A and write to another topic B.
> >>
> >> When I run it on my local environment (local zk, local kafka broker,
> local
> >> kafka streams app) everything works fine, topic B created and filled
> with
> >> messages from A
> >> If I run it on existing kafka cluster (remote zk, remote kafka, LOCAL
> >> kafka streams) my app is not working anymore.
> >>
> >> It succesfully read the remote topic A, succesfully process the message
> >> and generate a producer record, creates a B topic in remote kafka, bud
> >> during send I get an error
> >>
> >> ```
> >> 15:36:47.242 [kafka-producer-network-thread |
> >> example-message-counter3-1-StreamThread-1-producer] ERROR
> >> o.a.k.s.p.internals.RecordCollector - Error sending record: null
> >> org.apache.kafka.common.errors.NotLeaderForPartitionException: This
> server
> >> is not the leader for that topic-partition
> >> ```
> >>
> >> Could you point me to direction where to start debug or what problems
> >> might cause this behaviour?
> >>
> >> Thanks,
> >> — Misha
> >
> >
> >
> >
> > --
> > -- Guozhang
>
>


-- 
-- Guozhang

Reg: SSL setup

[BigData dev]

Hi,
Can you please provide information on Self signed certificate setup in
Kafka. As in Kafka documentation only CA signed setup is provided.

http://kafka.apache.org/documentation.html#security_ssl


As because, we need to provide parameters trustore, keystore during
configuration.

Or to work with self signed certificate, do we need to import all nodes
certificates to trustore on all machines?

Can you please provide information on this, if you have worked on this.


Thanks,
Bharat

[jira] [Commented] (KAFKA-3999) Consumer bytes-fetched metric uses decompressed message size

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406234#comment-15406234
 ] 

ASF GitHub Bot commented on KAFKA-3999:
---

Github user vahidhashemian closed the pull request at:

https://github.com/apache/kafka/pull/1698


> Consumer bytes-fetched metric uses decompressed message size
> 
>
> Key: KAFKA-3999
> URL: https://issues.apache.org/jira/browse/KAFKA-3999
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.9.0.1, 0.10.0.0
>Reporter: Jason Gustafson
>Assignee: Vahid Hashemian
>Priority: Minor
>
> It looks like the computation for the bytes-fetched metrics uses the size of 
> the decompressed message set. I would have expected it to be based off of the 
> raw size of the fetch responses. Perhaps it would be helpful to expose both 
> the raw and decompressed fetch sizes? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3999) Consumer bytes-fetched metric uses decompressed message size

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3999?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406235#comment-15406235
 ] 

ASF GitHub Bot commented on KAFKA-3999:
---

GitHub user vahidhashemian reopened a pull request:

https://github.com/apache/kafka/pull/1698

KAFKA-3999: Record raw size of fetch responses as part of consumer metrics

Currently, only the decompressed size of fetch responses is recorded. This 
PR adds a sensor to keep track of the raw size as well.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vahidhashemian/kafka KAFKA-3999

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1698.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1698


commit ae9b0abf60e20923761eda7d5f351625a2b2c0dc
Author: Vahid Hashemian 
Date:   2016-08-02T23:07:27Z

KAFKA-3999: Record raw size of fetch responses

Currently, only the decompressed size of fetch responses is recorded. This 
PR adds a sensor to keep track of their raw size as well.




> Consumer bytes-fetched metric uses decompressed message size
> 
>
> Key: KAFKA-3999
> URL: https://issues.apache.org/jira/browse/KAFKA-3999
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.9.0.1, 0.10.0.0
>Reporter: Jason Gustafson
>Assignee: Vahid Hashemian
>Priority: Minor
>
> It looks like the computation for the bytes-fetched metrics uses the size of 
> the decompressed message set. I would have expected it to be based off of the 
> raw size of the fetch responses. Perhaps it would be helpful to expose both 
> the raw and decompressed fetch sizes? 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1698: KAFKA-3999: Record raw size of fetch responses as ...

[vahidhashemian]

Github user vahidhashemian closed the pull request at:

https://github.com/apache/kafka/pull/1698


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #1698: KAFKA-3999: Record raw size of fetch responses as ...

[vahidhashemian]

GitHub user vahidhashemian reopened a pull request:

https://github.com/apache/kafka/pull/1698

KAFKA-3999: Record raw size of fetch responses as part of consumer metrics

Currently, only the decompressed size of fetch responses is recorded. This 
PR adds a sensor to keep track of the raw size as well.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vahidhashemian/kafka KAFKA-3999

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1698.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1698


commit ae9b0abf60e20923761eda7d5f351625a2b2c0dc
Author: Vahid Hashemian 
Date:   2016-08-02T23:07:27Z

KAFKA-3999: Record raw size of fetch responses

Currently, only the decompressed size of fetch responses is recorded. This 
PR adds a sensor to keep track of their raw size as well.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3042) updateIsr should stop after failed several times due to zkVersion issue

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3042?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406121#comment-15406121
 ] 

Guozhang Wang commented on KAFKA-3042:
--

I agree that we should consider improving the controller logic as a general 
story, at the same time I feel it would not harm (we need to think through the 
corner cases for sure) to combine these two requests as from a debugging 
perspective.

> updateIsr should stop after failed several times due to zkVersion issue
> ---
>
> Key: KAFKA-3042
> URL: https://issues.apache.org/jira/browse/KAFKA-3042
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.8.2.1
> Environment: jdk 1.7
> centos 6.4
>Reporter: Jiahongchao
> Fix For: 0.10.1.0
>
> Attachments: controller.log, server.log.2016-03-23-01, 
> state-change.log
>
>
> sometimes one broker may repeatly log
> "Cached zkVersion 54 not equal to that in zookeeper, skip updating ISR"
> I think this is because the broker consider itself as the leader in fact it's 
> a follower.
> So after several failed tries, it need to find out who is the leader



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-1211) Hold the produce request with ack > 1 in purgatory until replicas' HW has larger than the produce offset

[Flavio Junqueira (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1211?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406063#comment-15406063
 ] 

Flavio Junqueira commented on KAFKA-1211:
-

Thanks for the clarification, [~junrao]. There are a couple of specific points 
that still aren't entirely clear to me:

# We are trying to preserve the generation when we copy messages to a follower, 
correct? In step 3.4, when we say that the follower flushes the LGS, we are 
more specifically trying to replicate the leader LGS, is that right? What 
happens if either the follower crashes or the leader changes between persisting 
the new LGS and fetching the new messages from the leader? I'm concerned that 
we will leave the LGS and the log of the broker in an inconsistent state.
# When we say in step 3.4 that the follower needs to remember the LLG, I 
suppose this is just during this recovery period. Otherwise, once we have 
completed the sync up, the follower knows that the latest generation is the 
LLG. During sync up, there is the question I'm raising above, but it is also 
not super clear whether we need to persist the LLG independently to make sure 
that we don't have a situation in which the follower crashes, comes back, and 
accepts messages from a different generation.

> Hold the produce request with ack > 1 in purgatory until replicas' HW has 
> larger than the produce offset
> 
>
> Key: KAFKA-1211
> URL: https://issues.apache.org/jira/browse/KAFKA-1211
> Project: Kafka
>  Issue Type: Bug
>Reporter: Guozhang Wang
>Assignee: Guozhang Wang
> Fix For: 0.11.0.0
>
>
> Today during leader failover we will have a weakness period when the 
> followers truncate their data before fetching from the new leader, i.e., 
> number of in-sync replicas is just 1. If during this time the leader has also 
> failed then produce requests with ack >1 that have get responded will still 
> be lost. To avoid this scenario we would prefer to hold the produce request 
> in purgatory until replica's HW has larger than the offset instead of just 
> their end-of-log offsets.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [kafka-clients] [VOTE] 0.10.0.1 RC1

[Manikumar Reddy]

Hi,

There are two versions of slf4j-log4j jar in the build. (1.6.1, 1.7.21).
slf4j-log4j12-1.6.1.jar is coming from streams:examples module.

Thanks,
Manikumar

On Tue, Aug 2, 2016 at 8:31 PM, Ismael Juma  wrote:

> Hello Kafka users, developers and client-developers,
>
> This is the second candidate for the release of Apache Kafka 0.10.0.1.
> This is a bug fix release and it includes fixes and improvements from 52
> JIRAs (including a few critical bugs). See the release notes for more
> details:
>
> http://home.apache.org/~ijuma/kafka-0.10.0.1-rc1/RELEASE_NOTES.html
>
> When compared to RC0, RC1 contains fixes for two bugs (KAFKA-4008
> and KAFKA-3950) and a couple of test stabilisation fixes.
>
> *** Please download, test and vote by Friday, 5 August, 8am PT ***
>
> Kafka's KEYS file containing PGP keys we use to sign the release:
> http://kafka.apache.org/KEYS
>
> * Release artifacts to be voted upon (source and binary):
> http://home.apache.org/~ijuma/kafka-0.10.0.1-rc1/
>
> * Maven artifacts to be voted upon:
> https://repository.apache.org/content/groups/staging
>
> * Javadoc:
> http://home.apache.org/~ijuma/kafka-0.10.0.1-rc1/javadoc/
>
> * Tag to be voted upon (off 0.10.0 branch) is the 0.10.0.1-rc1 tag:
>
> https://git-wip-us.apache.org/repos/asf?p=kafka.git;a=tag;h=108580e4594d694827c953264969fe1ce2a7
>
> * Documentation:
> http://kafka.apache.org/0100/documentation.html
>
> * Protocol:
> http://kafka.apache.org/0100/protocol.html
>
> * Successful Jenkins builds for the 0.10.0 branch:
> Unit/integration tests: *https://builds.apache.org/job/kafka-0.10.0-jdk7/179/
> *
> System tests: *https://jenkins.confluent.io/job/system-test-kafka-0.10.0/136/
> *
>
> Thanks,
> Ismael
>
> --
> You received this message because you are subscribed to the Google Groups
> "kafka-clients" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to kafka-clients+unsubscr...@googlegroups.com.
> To post to this group, send email to kafka-clie...@googlegroups.com.
> Visit this group at https://groups.google.com/group/kafka-clients.
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/kafka-clients/CAD5tkZaRxAjQbwS_1q4MqskSYKxQWBFmdPVf_PP020bjY9%3DCgQ%40mail.gmail.com
> 
> .
> For more options, visit https://groups.google.com/d/optout.
>

[jira] [Commented] (KAFKA-3916) Connection from controller to broker disconnects

[Michal Turek (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3916?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15406027#comment-15406027
 ] 

Michal Turek commented on KAFKA-3916:
-

We saw this issue yesterday, I don't know if this helps, but it may be useful 
while debugging.

- Kafka 0.9.0.1 and 0.9.0.1 clients.
- There is ISR shrink and immediate ISR expand visible in graphs based on JMX 
of Kafka brokers.
- Consumers were unable to commit offsets at that time.

{noformat}
2016-08-02 14:25:29.589 INFO  o.a.k.c.c.internals.ConsumerCoordinator   
[Consumer-7]: Offset commit for group ... failed due to REQUEST_TIMED_OUT, will 
find new coordinator and retry
2016-08-02 14:25:52.560 INFO  o.a.k.c.c.internals.ConsumerCoordinator   
[Consumer-2]: Offset commit for group ... failed due to REQUEST_TIMED_OUT, will 
find new coordinator and retry
2016-08-02 14:25:52.562 INFO  o.a.k.c.c.internals.ConsumerCoordinator   
[Consumer-0]: Offset commit for group ... failed due to REQUEST_TIMED_OUT, will 
find new coordinator and retry
2016-08-02 14:25:52.563 INFO  o.a.k.c.c.internals.ConsumerCoordinator   
[Consumer-5]: Offset commit for group ... failed due to REQUEST_TIMED_OUT, will 
find new coordinator and retry
2016-08-02 14:25:52.570 INFO  o.a.k.c.c.internals.ConsumerCoordinator   
[Consumer-6]: Offset commit for group ... failed due to REQUEST_TIMED_OUT, will 
find new coordinator and retry
2016-08-02 14:25:52.570 INFO  o.a.k.c.c.internals.ConsumerCoordinator   
[Consumer-3]: Offset commit for group ... failed due to REQUEST_TIMED_OUT, will 
find new coordinator and retry
2016-08-02 14:25:52.570 INFO  o.a.k.c.c.internals.ConsumerCoordinator   
[Consumer-4]: Offset commit for group ... failed due to REQUEST_TIMED_OUT, will 
find new coordinator and retry
2016-08-02 14:25:52.572 INFO  o.a.k.c.c.internals.ConsumerCoordinator   
[Consumer-1]: Offset commit for group ... failed due to REQUEST_TIMED_OUT, will 
find new coordinator and retry
2016-08-02 14:25:52.572 INFO  o.a.k.c.c.internals.AbstractCoordinator   
[Consumer-7]: Marking the coordinator 2147483646 dead.
2016-08-02 14:25:52.573 INFO  o.a.k.c.c.internals.AbstractCoordinator   
[Consumer-2]: Marking the coordinator 2147483646 dead.
2016-08-02 14:25:52.573 INFO  o.a.k.c.c.internals.AbstractCoordinator   
[Consumer-0]: Marking the coordinator 2147483646 dead.
2016-08-02 14:25:52.574 INFO  o.a.k.c.c.internals.AbstractCoordinator   
[Consumer-5]: Marking the coordinator 2147483646 dead.
2016-08-02 14:25:52.575 INFO  o.a.k.c.c.internals.AbstractCoordinator   
[Consumer-6]: Marking the coordinator 2147483646 dead.
2016-08-02 14:25:52.575 INFO  o.a.k.c.c.internals.AbstractCoordinator   
[Consumer-3]: Marking the coordinator 2147483646 dead.
2016-08-02 14:25:52.575 INFO  o.a.k.c.c.internals.AbstractCoordinator   
[Consumer-4]: Marking the coordinator 2147483646 dead.
2016-08-02 14:25:52.576 INFO  o.a.k.c.c.internals.AbstractCoordinator   
[Consumer-1]: Marking the coordinator 2147483646 dead.
2016-08-02 14:25:52.576 WARN  o.a.k.c.c.internals.ConsumerCoordinator   
[Consumer-7]: Auto offset commit failed: The request timed out.
2016-08-02 14:25:52.577 WARN  o.a.k.c.c.internals.ConsumerCoordinator   
[Consumer-2]: Auto offset commit failed: The request timed out.
2016-08-02 14:25:52.577 WARN  o.a.k.c.c.internals.ConsumerCoordinator   
[Consumer-0]: Auto offset commit failed: The request timed out.
2016-08-02 14:25:52.577 WARN  o.a.k.c.c.internals.ConsumerCoordinator   
[Consumer-5]: Auto offset commit failed: The request timed out.
2016-08-02 14:25:52.578 WARN  o.a.k.c.c.internals.ConsumerCoordinator   
[Consumer-6]: Auto offset commit failed: The request timed out.
2016-08-02 14:25:52.578 WARN  o.a.k.c.c.internals.ConsumerCoordinator   
[Consumer-3]: Auto offset commit failed: The request timed out.
2016-08-02 14:25:52.578 WARN  o.a.k.c.c.internals.ConsumerCoordinator   
[Consumer-4]: Auto offset commit failed: The request timed out.
2016-08-02 14:25:52.579 WARN  o.a.k.c.c.internals.ConsumerCoordinator   
[Consumer-1]: Auto offset commit failed: The request timed out.
{noformat}

> Connection from controller to broker disconnects
> 
>
> Key: KAFKA-3916
> URL: https://issues.apache.org/jira/browse/KAFKA-3916
> Project: Kafka
>  Issue Type: Bug
>  Components: controller
>Affects Versions: 0.9.0.1
>Reporter: Dave Powell
>
> We recently upgraded from 0.8.2.1 to 0.9.0.1. Since then, several times per 
> day, the controllers in our clusters have their connection to all brokers 
> disconnected, and then successfully reconnected a few hundred ms later. Each 
> time this occurs we see a brief spike in our 99th percentile produce and 
> consume times, reaching several hundred ms.
> Here is an example of what 

[jira] [Commented] (KAFKA-3817) KTableRepartitionMap should handle null inputs

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405982#comment-15405982
 ] 

Guozhang Wang commented on KAFKA-3817:
--

Good point, mind filing a minor PR?

> KTableRepartitionMap should handle null inputs
> --
>
> Key: KAFKA-3817
> URL: https://issues.apache.org/jira/browse/KAFKA-3817
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.0.0
>Reporter: Jeff Klukas
>Assignee: Guozhang Wang
> Fix For: 0.10.0.1
>
>
> When calling {{KTable.groupBy}} on the result of a KTable-KTable join, NPEs 
> are raised:
> {{org.apache.kafka.streams.kstream.internals.KTableRepartitionMap$
> > KTableMapProcessor.process(KTableRepartitionMap.java:88)}}
> The root cause is that the join is expected to emit null values when no match 
> is found, but KTableRepartitionMap is not set up to handle this case.
> On the users email list, [~guozhang] described a plan of action:
> I think this is actually a bug in KTableRepartitionMap
> that it actually should expect null grouped keys; this would be a
> straight-forward fix for this operator, but I can make a pass over all the
> repartition operators just to make sure they are all gracefully handling
> null keys.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1701: MINOR: increase usability of shell-scripts

[igalic]

GitHub user igalic opened a pull request:

https://github.com/apache/kafka/pull/1701

MINOR: increase usability of shell-scripts

we do this by ensuring that the --zookeeper parameter consistently
defaults to localhost:2181 — as it already does in some places.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/igalic/kafka sh-defaults

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1701.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1701


commit 24387cba230503af30b6efbebea9b442b26a6d5c
Author: Igor Galić 
Date:   2016-08-03T13:21:58Z

MINOR: increase usability of shell-scripts

we do this by ensuring that the --zookeeper parameter consistently
defaults to localhost:2181 — as it already does in some places.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Work started] (KAFKA-4016) Kafka Streams join benchmark

[Eno Thereska (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-4016?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-4016 started by Eno Thereska.
---
> Kafka Streams join benchmark
> 
>
> Key: KAFKA-4016
> URL: https://issues.apache.org/jira/browse/KAFKA-4016
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Eno Thereska
>Assignee: Eno Thereska
> Fix For: 0.10.1.0
>
>
> As part of the streams benchmark, we need benchmarks for KStream-KStream, 
> KStream-KTable and KTable-KTable joins. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-4016) Kafka Streams join benchmark

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-4016?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405894#comment-15405894
 ] 

ASF GitHub Bot commented on KAFKA-4016:
---

GitHub user enothereska opened a pull request:

https://github.com/apache/kafka/pull/1700

KAFKA-4016: Added join benchmarks



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/enothereska/kafka join-benchmarks

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1700.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1700


commit 84006e952ab2653ee105a6298550e49e00e055b9
Author: Eno Thereska 
Date:   2016-08-03T13:09:38Z

Added join benchmarks




> Kafka Streams join benchmark
> 
>
> Key: KAFKA-4016
> URL: https://issues.apache.org/jira/browse/KAFKA-4016
> Project: Kafka
>  Issue Type: Improvement
>  Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Eno Thereska
>Assignee: Eno Thereska
> Fix For: 0.10.1.0
>
>
> As part of the streams benchmark, we need benchmarks for KStream-KStream, 
> KStream-KTable and KTable-KTable joins. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1700: KAFKA-4016: Added join benchmarks

[enothereska]

GitHub user enothereska opened a pull request:

https://github.com/apache/kafka/pull/1700

KAFKA-4016: Added join benchmarks



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/enothereska/kafka join-benchmarks

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1700.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1700


commit 84006e952ab2653ee105a6298550e49e00e055b9
Author: Eno Thereska 
Date:   2016-08-03T13:09:38Z

Added join benchmarks




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Created] (KAFKA-4016) Kafka Streams join benchmark

[Eno Thereska (JIRA)]

Eno Thereska created KAFKA-4016:
---

 Summary: Kafka Streams join benchmark
 Key: KAFKA-4016
 URL: https://issues.apache.org/jira/browse/KAFKA-4016
 Project: Kafka
  Issue Type: Improvement
  Components: streams
Affects Versions: 0.10.1.0
Reporter: Eno Thereska
Assignee: Eno Thereska
 Fix For: 0.10.1.0


As part of the streams benchmark, we need benchmarks for KStream-KStream, 
KStream-KTable and KTable-KTable joins. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-0.10.0-jdk7 #180

[Apache Jenkins Server]

See 

Changes:

[ismael] KAFKA-3667; Improve Section 7.2 Encryption and Authentication using SSL

--
[...truncated 5715 lines...]
org.apache.kafka.streams.kstream.TimeWindowsTest > windowSizeMustNotBeNegative 
PASSED

org.apache.kafka.streams.kstream.TimeWindowsTest > 
advanceIntervalMustNotBeNegative PASSED

org.apache.kafka.streams.kstream.TimeWindowsTest > 
advanceIntervalMustNotBeLargerThanWindowSize PASSED

org.apache.kafka.streams.kstream.TimeWindowsTest > windowsForTumblingWindows 
PASSED

org.apache.kafka.streams.kstream.TimeWindowsTest > windowsForHoppingWindows 
PASSED

org.apache.kafka.streams.kstream.TimeWindowsTest > 
windowsForBarelyOverlappingHoppingWindows PASSED

org.apache.kafka.streams.kstream.internals.KStreamFilterTest > testFilterNot 
PASSED

org.apache.kafka.streams.kstream.internals.KStreamFilterTest > testFilter PASSED

org.apache.kafka.streams.kstream.internals.KStreamTransformValuesTest > 
testTransform PASSED

org.apache.kafka.streams.kstream.internals.KStreamWindowAggregateTest > 
testAggBasic PASSED

org.apache.kafka.streams.kstream.internals.KStreamWindowAggregateTest > 
testJoin PASSED

org.apache.kafka.streams.kstream.internals.KStreamFlatMapValuesTest > 
testFlatMapValues PASSED

org.apache.kafka.streams.kstream.internals.KTableFilterTest > 
testSendingOldValue PASSED

org.apache.kafka.streams.kstream.internals.KTableFilterTest > 
testNotSendingOldValue PASSED

org.apache.kafka.streams.kstream.internals.KTableFilterTest > 
testSkipNullOnMaterialization PASSED

org.apache.kafka.streams.kstream.internals.KTableFilterTest > testKTable PASSED

org.apache.kafka.streams.kstream.internals.KTableFilterTest > testValueGetter 
PASSED

org.apache.kafka.streams.kstream.internals.KStreamKStreamLeftJoinTest > 
testLeftJoin PASSED

org.apache.kafka.streams.kstream.internals.KStreamKStreamLeftJoinTest > 
testWindowing PASSED

org.apache.kafka.streams.kstream.internals.KTableForeachTest > testForeach 
PASSED

org.apache.kafka.streams.kstream.internals.KTableKTableOuterJoinTest > 
testSendingOldValue PASSED

org.apache.kafka.streams.kstream.internals.KTableKTableOuterJoinTest > testJoin 
PASSED

org.apache.kafka.streams.kstream.internals.KTableKTableOuterJoinTest > 
testNotSendingOldValue PASSED

org.apache.kafka.streams.kstream.internals.KStreamMapTest > testMap PASSED

org.apache.kafka.streams.kstream.internals.KStreamBranchTest > 
testKStreamBranch PASSED

org.apache.kafka.streams.kstream.internals.KGroupedTableImplTest > 
testGroupedCountOccurences PASSED

org.apache.kafka.streams.kstream.internals.KTableSourceTest > 
testNotSedingOldValue PASSED

org.apache.kafka.streams.kstream.internals.KTableSourceTest > 
testSedingOldValue PASSED

org.apache.kafka.streams.kstream.internals.KTableSourceTest > testKTable PASSED

org.apache.kafka.streams.kstream.internals.KTableSourceTest > testValueGetter 
PASSED

org.apache.kafka.streams.kstream.internals.KStreamForeachTest > testForeach 
PASSED

org.apache.kafka.streams.kstream.internals.KTableMapValuesTest > 
testSendingOldValue PASSED

org.apache.kafka.streams.kstream.internals.KTableMapValuesTest > 
testNotSendingOldValue PASSED

org.apache.kafka.streams.kstream.internals.KTableMapValuesTest > testKTable 
PASSED

org.apache.kafka.streams.kstream.internals.KTableMapValuesTest > 
testValueGetter PASSED

org.apache.kafka.streams.kstream.internals.KStreamTransformTest > testTransform 
PASSED

org.apache.kafka.streams.kstream.internals.KStreamMapValuesTest > 
testFlatMapValues PASSED

org.apache.kafka.streams.kstream.internals.KTableImplTest > testStateStore 
PASSED

org.apache.kafka.streams.kstream.internals.KTableImplTest > testRepartition 
PASSED

org.apache.kafka.streams.kstream.internals.KTableImplTest > 
testStateStoreLazyEval PASSED

org.apache.kafka.streams.kstream.internals.KTableImplTest > testKTable PASSED

org.apache.kafka.streams.kstream.internals.KTableImplTest > testValueGetter 
PASSED

org.apache.kafka.streams.kstream.internals.KStreamKStreamJoinTest > 
testOuterJoin PASSED

org.apache.kafka.streams.kstream.internals.KStreamKStreamJoinTest > testJoin 
PASSED

org.apache.kafka.streams.kstream.internals.KStreamKStreamJoinTest > 
testWindowing PASSED

org.apache.kafka.streams.kstream.internals.KeyValuePrinterProcessorTest > 
testPrintKeyValueWithProvidedSerde PASSED

org.apache.kafka.streams.kstream.internals.KeyValuePrinterProcessorTest > 
testPrintKeyValueDefaultSerde PASSED

org.apache.kafka.streams.kstream.internals.KTableAggregateTest > testAggBasic 
PASSED

org.apache.kafka.streams.kstream.internals.KTableAggregateTest > 
testAggRepartition PASSED

org.apache.kafka.streams.kstream.internals.KStreamFlatMapTest > testFlatMap 
PASSED

org.apache.kafka.streams.kstream.internals.KTableMapKeysTest > 
testMapKeysConvertingToStream PASSED

org.apache.kafka.streams.kstream.internals.KTableKTableLeftJoinTest > 
testSendingOldValue PASSED

[jira] [Commented] (KAFKA-1194) The kafka broker cannot delete the old log files after the configured time

[Harald Kirsch (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1194?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405810#comment-15405810
 ] 

Harald Kirsch commented on KAFKA-1194:
--

Just stumbled over yet another problem instance. During startup, Kafka notices 
a corrupt log/index file and tries to repair it. Here is the stack trace:

{noformat}
[2016-08-03 13:56:17,467] INFO Found log file 
d:\Search\kafka\fileshare-1\.log.swap from interrupted swap 
operation, repairing. (kafka.log.Log)
[2016-08-03 13:56:18,436] ERROR There was an error in one of the threads during 
logs loading: kafka.common.KafkaStorageException: Failed to change the index 
file suffix from .swap to  for log segment 0 (kafka.log.LogManager)
[2016-08-03 13:56:18,436] FATAL Fatal error during KafkaServer startup. Prepare 
to shutdown (kafka.server.KafkaServer)
kafka.common.KafkaStorageException: Failed to change the index file suffix from 
.swap to  for log segment 0
at kafka.log.LogSegment.kafkaStorageException$1(LogSegment.scala:268)
at kafka.log.LogSegment.changeFileSuffixes(LogSegment.scala:274)
at kafka.log.Log.replaceSegments(Log.scala:886)
at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:230)
at kafka.log.Log$$anonfun$loadSegments$5.apply(Log.scala:214)
at scala.collection.immutable.Set$Set1.foreach(Set.scala:74)
at kafka.log.Log.loadSegments(Log.scala:214)
at kafka.log.Log.(Log.scala:101)
at 
kafka.log.LogManager$$anonfun$loadLogs$2$$anonfun$3$$anonfun$apply$10$$anonfun$apply$1.apply$mcV$sp(LogManager.scala:151)
at kafka.utils.CoreUtils$$anon$1.run(CoreUtils.scala:56)
at 
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
at java.util.concurrent.FutureTask.run(FutureTask.java:266)
at 
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
at 
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
at java.lang.Thread.run(Thread.java:745)
Caused by: java.nio.file.FileSystemException: 
d:\Search\kafka\fileshare-1\.index.swap -> 
d:\Search\kafka\fileshare-1\.index: The process cannot 
access the file because it is being used by another process.

at 
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
at 
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:387)
at 
sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
at java.nio.file.Files.move(Files.java:1395)
at 
org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:670)
at kafka.log.OffsetIndex.renameTo(OffsetIndex.scala:365)
... 14 more
Suppressed: java.nio.file.FileSystemException: 
d:\Search\kafka\fileshare-1\.index.swap -> 
d:\Search\kafka\fileshare-1\.index: The process cannot 
access the file because it is being used by another process.

at 
sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:86)
at 
sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:97)
at sun.nio.fs.WindowsFileCopy.move(WindowsFileCopy.java:301)
at 
sun.nio.fs.WindowsFileSystemProvider.move(WindowsFileSystemProvider.java:287)
at java.nio.file.Files.move(Files.java:1395)
at 
org.apache.kafka.common.utils.Utils.atomicMoveWithFallback(Utils.java:667)
... 15 more
[2016-08-03 13:56:18,451] INFO shutting down (kafka.server.KafkaServer)
[2016-08-03 13:56:18,467] INFO shut down completed (kafka.server.KafkaServer)
{noformat}

> The kafka broker cannot delete the old log files after the configured time
> --
>
> Key: KAFKA-1194
> URL: https://issues.apache.org/jira/browse/KAFKA-1194
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.8.1
> Environment: window
>Reporter: Tao Qin
>Assignee: Jay Kreps
>  Labels: features, patch
> Fix For: 0.10.1.0
>
> Attachments: KAFKA-1194.patch, kafka-1194-v1.patch, 
> kafka-1194-v2.patch
>
>   Original Estimate: 72h
>  Remaining Estimate: 72h
>
> We tested it in windows environment, and set the log.retention.hours to 24 
> hours.
> # The minimum age of a log file to be eligible for deletion
> log.retention.hours=24
> After several days, the kafka broker still cannot delete the old log file. 
> And we get the following exceptions:
> [2013-12-19 01:57:38,528] ERROR Uncaught exception in scheduled task 
> 'kafka-log-retention' (kafka.utils.KafkaScheduler)

[jira] [Commented] (KAFKA-3665) Default ssl.endpoint.identification.algorithm should be https

[Ryan P (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3665?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405659#comment-15405659
 ] 

Ryan P commented on KAFKA-3665:
---

RFC-6066 describes an extension to the TLS protocol to handle the VIP case. 

https://tools.ietf.org/html/rfc6066#page-6

Setting the SNI field within a client request, JSSE sets this field by default, 
allows the VIP to determine the correct named virtual host for the request and 
set the connection up accordingly from the start.



> Default ssl.endpoint.identification.algorithm should be https
> -
>
> Key: KAFKA-3665
> URL: https://issues.apache.org/jira/browse/KAFKA-3665
> Project: Kafka
>  Issue Type: Bug
>  Components: security
>Affects Versions: 0.9.0.1, 0.10.0.0
>Reporter: Ismael Juma
>Assignee: Ismael Juma
> Fix For: 0.10.1.0
>
>
> The default `ssl.endpoint.identification.algorithm` is `null` which is not a 
> secure default (man in the middle attacks are possible).
> We should probably use `https` instead. A more conservative alternative would 
> be to update the documentation instead of changing the default.
> A paper on the topic (thanks to Ryan Pridgeon for the reference): 
> http://www.cs.utexas.edu/~shmat/shmat_ccs12.pdf



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3945) kafka-console-producer.sh does not accept request-required-acks=all

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3945:
---
Fix Version/s: 0.10.1.0

> kafka-console-producer.sh does not accept request-required-acks=all
> ---
>
> Key: KAFKA-3945
> URL: https://issues.apache.org/jira/browse/KAFKA-3945
> Project: Kafka
>  Issue Type: Bug
>Reporter: James Cheng
>Assignee: Vahid Hashemian
> Fix For: 0.10.1.0
>
>
> The Java producer accepts 0, 1, -1, and all as valid values for acks. The 
> kafka-console-producer.sh however does not accept "all" as a valid value.
> {code}
> $ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test 
> --request-required-acks all
> Cannot parse argument 'all' of option request-required-acks
> {code}
> The code seems to expect it to be an integer:
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/ConsoleProducer.scala#L178-L182
> If I pass in an Integer, though, the error message for it says that "all" is 
> a valid value.
> {code}
> $ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test 
> --request-required-acks 2
> org.apache.kafka.common.config.ConfigException: Invalid value 2 for 
> configuration acks: String must be one of: all, -1, 0, 1
>   at 
> org.apache.kafka.common.config.ConfigDef$ValidString.ensureValid(ConfigDef.java:827)
>   at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:427)
>   at 
> org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:55)
>   at 
> org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:62)
>   at 
> org.apache.kafka.clients.producer.ProducerConfig.(ProducerConfig.java:338)
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:188)
>   at kafka.producer.NewShinyProducer.(BaseProducer.scala:40)
>   at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:45)
>   at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala)
> {code}
> Either the kafka-console-producer.sh needs to be updated to accept "all", or 
> the documentation for kafka-console-producer should be updated to say that 
> "all" is not a valid value. Either way, the two should be in sync with each 
> other.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3945) kafka-console-producer.sh does not accept request-required-acks=all

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3945?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3945:
---
Resolution: Fixed
  Reviewer: Ismael Juma
Status: Resolved  (was: Patch Available)

> kafka-console-producer.sh does not accept request-required-acks=all
> ---
>
> Key: KAFKA-3945
> URL: https://issues.apache.org/jira/browse/KAFKA-3945
> Project: Kafka
>  Issue Type: Bug
>Reporter: James Cheng
>Assignee: Vahid Hashemian
>
> The Java producer accepts 0, 1, -1, and all as valid values for acks. The 
> kafka-console-producer.sh however does not accept "all" as a valid value.
> {code}
> $ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test 
> --request-required-acks all
> Cannot parse argument 'all' of option request-required-acks
> {code}
> The code seems to expect it to be an integer:
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/ConsoleProducer.scala#L178-L182
> If I pass in an Integer, though, the error message for it says that "all" is 
> a valid value.
> {code}
> $ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test 
> --request-required-acks 2
> org.apache.kafka.common.config.ConfigException: Invalid value 2 for 
> configuration acks: String must be one of: all, -1, 0, 1
>   at 
> org.apache.kafka.common.config.ConfigDef$ValidString.ensureValid(ConfigDef.java:827)
>   at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:427)
>   at 
> org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:55)
>   at 
> org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:62)
>   at 
> org.apache.kafka.clients.producer.ProducerConfig.(ProducerConfig.java:338)
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:188)
>   at kafka.producer.NewShinyProducer.(BaseProducer.scala:40)
>   at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:45)
>   at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala)
> {code}
> Either the kafka-console-producer.sh needs to be updated to accept "all", or 
> the documentation for kafka-console-producer should be updated to say that 
> "all" is not a valid value. Either way, the two should be in sync with each 
> other.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3945) kafka-console-producer.sh does not accept request-required-acks=all

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3945?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405654#comment-15405654
 ] 

ASF GitHub Bot commented on KAFKA-3945:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1618


> kafka-console-producer.sh does not accept request-required-acks=all
> ---
>
> Key: KAFKA-3945
> URL: https://issues.apache.org/jira/browse/KAFKA-3945
> Project: Kafka
>  Issue Type: Bug
>Reporter: James Cheng
>Assignee: Vahid Hashemian
>
> The Java producer accepts 0, 1, -1, and all as valid values for acks. The 
> kafka-console-producer.sh however does not accept "all" as a valid value.
> {code}
> $ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test 
> --request-required-acks all
> Cannot parse argument 'all' of option request-required-acks
> {code}
> The code seems to expect it to be an integer:
> https://github.com/apache/kafka/blob/trunk/core/src/main/scala/kafka/tools/ConsoleProducer.scala#L178-L182
> If I pass in an Integer, though, the error message for it says that "all" is 
> a valid value.
> {code}
> $ bin/kafka-console-producer.sh --broker-list localhost:9092 --topic test 
> --request-required-acks 2
> org.apache.kafka.common.config.ConfigException: Invalid value 2 for 
> configuration acks: String must be one of: all, -1, 0, 1
>   at 
> org.apache.kafka.common.config.ConfigDef$ValidString.ensureValid(ConfigDef.java:827)
>   at org.apache.kafka.common.config.ConfigDef.parse(ConfigDef.java:427)
>   at 
> org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:55)
>   at 
> org.apache.kafka.common.config.AbstractConfig.(AbstractConfig.java:62)
>   at 
> org.apache.kafka.clients.producer.ProducerConfig.(ProducerConfig.java:338)
>   at 
> org.apache.kafka.clients.producer.KafkaProducer.(KafkaProducer.java:188)
>   at kafka.producer.NewShinyProducer.(BaseProducer.scala:40)
>   at kafka.tools.ConsoleProducer$.main(ConsoleProducer.scala:45)
>   at kafka.tools.ConsoleProducer.main(ConsoleProducer.scala)
> {code}
> Either the kafka-console-producer.sh needs to be updated to accept "all", or 
> the documentation for kafka-console-producer should be updated to say that 
> "all" is not a valid value. Either way, the two should be in sync with each 
> other.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1618: KAFKA-3945: Change the type of 'acks' config in co...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1618


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-3667) Improve Section 7.2 Encryption and Authentication using SSL to include proper hostname verification configuration

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3667?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3667:
---
Assignee: Ryan P

> Improve Section 7.2 Encryption and Authentication using SSL to include proper 
> hostname verification configuration
> -
>
> Key: KAFKA-3667
> URL: https://issues.apache.org/jira/browse/KAFKA-3667
> Project: Kafka
>  Issue Type: Improvement
>  Components: security
>Reporter: Ryan P
>Assignee: Ryan P
> Fix For: 0.10.0.1
>
>
> Kafka's documentation should include additional guidance on how to properly 
> enable SSL with hostname verification. 
> 1. Hostname verification will not be performed if 
> ssl.endpoint.identification.algorithm has not been set. 
> Failing to enable this will leave Kafka susceptible to 'man-in-the-middle 
> attacks' as describe in the [oracle java api docs. 
> |https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/X509ExtendedTrustManager.html]
> 2. The docs should also include instructions on how to strictly comply with 
> [RFC-2818|https://tools.ietf.org/html/rfc2818#section-3.1]. This will require 
> adding the DNS SAN extension. 
> [keytool|http://docs.oracle.com/javase/7/docs/technotes/tools/windows/keytool.html]
> It's worth noting in the docs that placing the FQDN in the CN is still valid 
> despite being less than ideal as well. 
> 3. KAFKA-3665 aims to set the default value for 
> ssl.endpoint.identification.algorithm to HTTPS. This improvement JIRA aims to 
> document the behavior changes introduced. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3667) Improve Section 7.2 Encryption and Authentication using SSL to include proper hostname verification configuration

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3667?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405638#comment-15405638
 ] 

ASF GitHub Bot commented on KAFKA-3667:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1384


> Improve Section 7.2 Encryption and Authentication using SSL to include proper 
> hostname verification configuration
> -
>
> Key: KAFKA-3667
> URL: https://issues.apache.org/jira/browse/KAFKA-3667
> Project: Kafka
>  Issue Type: Improvement
>  Components: security
>Reporter: Ryan P
>
> Kafka's documentation should include additional guidance on how to properly 
> enable SSL with hostname verification. 
> 1. Hostname verification will not be performed if 
> ssl.endpoint.identification.algorithm has not been set. 
> Failing to enable this will leave Kafka susceptible to 'man-in-the-middle 
> attacks' as describe in the [oracle java api docs. 
> |https://docs.oracle.com/javase/7/docs/api/javax/net/ssl/X509ExtendedTrustManager.html]
> 2. The docs should also include instructions on how to strictly comply with 
> [RFC-2818|https://tools.ietf.org/html/rfc2818#section-3.1]. This will require 
> adding the DNS SAN extension. 
> [keytool|http://docs.oracle.com/javase/7/docs/technotes/tools/windows/keytool.html]
> It's worth noting in the docs that placing the FQDN in the CN is still valid 
> despite being less than ideal as well. 
> 3. KAFKA-3665 aims to set the default value for 
> ssl.endpoint.identification.algorithm to HTTPS. This improvement JIRA aims to 
> document the behavior changes introduced. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1384: KAFKA-3667:Improve Section 7.2 Encryption and Auth...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1384


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3817) KTableRepartitionMap should handle null inputs

[Jan Filipiak (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3817?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15405568#comment-15405568
 ] 

Jan Filipiak commented on KAFKA-3817:
-

[~guozhang] 

So the ability to handle null values in the repartition map processor is really 
important. Nevertheless, I fear that this fix introduced a major problem 
regarding the ordering of the events. We talked about this 
https://github.com/apache/kafka/blob/ba9456de2ebf1a34bdf5f6f62a701875822e1923/streams/src/main/java/org/apache/kafka/streams/kstream/internals/KTableAggregate.java#L82
 before, but with this change a downstream KTableAggregate does not stand a 
chance to keep this behaviour. The mapper might change the key of the record, 
but that is not guaranteed to happen. So the  KTableAggregate will not remove 
first and then add the new, but it will add the new and remove then. This is 
okay for counting (one might see negativ values(might be a problem when Serde 
only handles unsigned)) but its gameover for building complex type objects. 

I would propose to change the order of the 2 if clauses that are now present 
here: 
https://github.com/apache/kafka/blob/730bf9a37a08b2ca41dcda52d2c70e92e85980f7/streams/src/main/java/org/apache/kafka/streams/kstream/internals/KTableRepartitionMap.java#L83
 that should fix it.

> KTableRepartitionMap should handle null inputs
> --
>
> Key: KAFKA-3817
> URL: https://issues.apache.org/jira/browse/KAFKA-3817
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Affects Versions: 0.10.0.0
>Reporter: Jeff Klukas
>Assignee: Guozhang Wang
> Fix For: 0.10.0.1
>
>
> When calling {{KTable.groupBy}} on the result of a KTable-KTable join, NPEs 
> are raised:
> {{org.apache.kafka.streams.kstream.internals.KTableRepartitionMap$
> > KTableMapProcessor.process(KTableRepartitionMap.java:88)}}
> The root cause is that the join is expected to emit null values when no match 
> is found, but KTableRepartitionMap is not set up to handle this case.
> On the users email list, [~guozhang] described a plan of action:
> I think this is actually a bug in KTableRepartitionMap
> that it actually should expect null grouped keys; this would be a
> straight-forward fix for this operator, but I can make a pass over all the
> repartition operators just to make sure they are all gracefully handling
> null keys.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1699: MINOR: rename StateStoreProvider.getStores() -> St...

[dguy]

GitHub user dguy opened a pull request:

https://github.com/apache/kafka/pull/1699

MINOR: rename StateStoreProvider.getStores() -> StateStoreProvider.stores()

Rename StateStoreProvider.getStores(...) to StateStoreProvider.stores(...) 
as this is consistent with the naming of other 'getters' in the public API.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dguy/kafka minor-method-rename

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1699.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1699


commit 965cca2d0748b214bd5819d801b0fdc37038bfd5
Author: Damian Guy 
Date:   2016-08-03T08:14:30Z

rename StateStoreProvider.getStores() -> StateStoreProvider.stores()




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---