Re: [DISCUSS] KIP-70: Revise Partition Assignment Semantics on New Consumer's Subscription Change

[Ewen Cheslack-Postava]

Agreed w/ Jason re: compatibility. It seems like such an edge case to
actually rely on this and I'd consider the current behavior essentially a
bug given how surprising it is. While normally a stickler for
compatibility, I think this is a case where its fine to make the change.

-Ewen

On Wed, Jul 20, 2016 at 9:48 AM, Jason Gustafson  wrote:

> Hey Vahid,
>
> Thanks for writing this up. This seems like a nice improvement over the
> existing somewhat surprising behavior. Currently if you have a consumer
> which changes subscriptions, then you will need to handle separately any
> cleanup for assigned partitions for topics which are no longer subscribed.
> With this change, the user can handle this exclusively in the
> onPartitionsRevoked() callback which seems less error prone. This also
> makes it unnecessary for us to do any special handling when autocommit is
> enabled since all partitions will still be assigned when we do the final
> offset commit prior to rebalancing. The main question mark in my mind is
> compatibility, but it seems unlikely that anyone depends on the current
> behavior. My hunch is that users probably expect it already works this way,
> so from that perspective, it's almost more of a bug fix.
>
> Thanks,
> Jason
>
> On Tue, Jul 19, 2016 at 5:55 PM, Vahid S Hashemian <
> vahidhashem...@us.ibm.com> wrote:
>
> > Hi all,
> >
> > We have started a new KIP under
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-70%3A+Revise+Partition+Assignment+Semantics+on+New+Consumer%27s+Subscription+Change
> >
> > Your feedback is much appreciated.
> >
> > Regards,
> > Vahid Hashemian
> >
> >
>



-- 
Thanks,
Ewen

Re: [DISCUSS] KIP-66 Kafka Connect Transformers for messages

[Ewen Cheslack-Postava]

Jun, The problem with it not being 1-1 is that Connect relies heavily on
offsets, so we'd need to be able to track offsets at this finer
granularity. Filtering is ok, but flatMap isn't. If you convert one message
to many, what are the offsets for the new messages? One possibility would
be to assume that transformations are deterministic and then "enhance" the
offsets with an extra integer field that indicates its position in the
subset. For sources this seems attractive since you can then reset to
whatever the connector-provided offset is and then filter out any of the
"sub"-messages that are earlier than the recorded "sub"-offset. But this
might not actually work for sources since a) the offsets will include extra
fields that the connector doesn't expect (might be ok since we handle that
data as schemaless anyway) and b) if we allow multiple transformations
(which seems likely given that people might want to do things like
rearrange fields + filter messages) then offsets start getting quite
complex as we add sub-sub-offsets and sub-sub-sub-offsets. It's doable, but
seems messy.

Things aren't as easy on the sink side. Since we track offsets using Kafka
offsets we either need to use the extra metadata space to store the
sub-offsets or we need to ensure that we only ever need to commit offsets
on Kafka message boundaries. We might be able to get away with just
delivering the entire set of generated messages in a single put() call,
which the connector is expected to either fully accept or fully reject (via
exception). However, this may end up interacting poorly with assumptions
connectors might make if we expose things like max.poll.records, where they
might expect one record at a time.

I'm not really convinced of the benefit of support this -- at some point it
seems better to use Streams to do transformations if you need flatMap. I
can't think of many generic transformations that would use 1-to-many, and
single message transforms really should be quite general -- that's the
reason for providing a separate interface isolated from Connectors or
Converters.

Gwen, re: using null and sending to dead letter queue, it would be useful
to think about how this might interact with other uses of a dead letter
queue. Similar ideas have been raised for messages that either can't be
parsed or which the connector chokes on repeatedly. If we use a dead letter
queue for those, do we want these messages (which are explicitly filtered
by a transform setup by the user) to end up in the same location?

-Ewen

On Sun, Jul 17, 2016 at 9:53 PM, Jun Rao  wrote:

> Does the transformation need to be 1-to-1? For example, some users model
> each Kafka message as schema + a batch of binary records. When using a sink
> connector to push the Kafka data to a sink, if would be useful if the
> transformer can convert each Kafka message to multiple records.
>
> Thanks,
>
> Jun
>
> On Sat, Jul 16, 2016 at 1:25 PM, Nisarg Shah  wrote:
>
> > Gwen,
> >
> > Yup, that sounds great! Instead of keeping it up to the transformers to
> > handle null, we can instead have the topic as null. Sounds good. To get
> rid
> > of a message, set the topic to a special one (could be as simple as
> null).
> >
> > Like I said before, the more interesting part would be ‘adding’ a new
> > message to the existing list, based on say the current message in the
> > transformer. Does that feature warrant to be included?
> >
> > > On Jul 14, 2016, at 22:25, Gwen Shapira  wrote:
> > >
> > > I used to work on Apache Flume, where we used to allow users to filter
> > > messages completely in the transformation and then we got rid of it,
> > > because we spent too much time trying to help users who had "message
> > > loss", where the loss was actually a bug in the filter...
> > >
> > > What we couldn't do in Flume, but perhaps can do in the simple
> > > transform for Connect is the ability to route messages to different
> > > topics, with "null" as one of the possible targets. This will allow
> > > you to implement a dead-letter-queue functionality and redirect
> > > messages that don't pass filter to an "errors" topic without getting
> > > rid of them completely, while also allowing braver users to get rid of
> > > messages by directing them to "null".
> > >
> > > Does that make sense?
> > >
> > > Gwen
> > >
> > > On Thu, Jul 14, 2016 at 8:33 PM, Nisarg Shah 
> wrote:
> > >> Thank you for your inputs Gwen and Michael.
> > >>
> > >> The original reason why I suggested a set based processing is because
> > of the flexibility is provides. The JIRA had a comment by a user
> requesting
> > a feature that could be achieved with this.
> > >>
> > >> After reading Gwen and Michael's points, (I went through the
> > documentation and the code in detail) and agree with what you have to
> say.
> > Also, fewer guarantees make what I had in mind less certain and thus
> > simplifying it to a single message based 

[jira] [Updated] (KAFKA-3984) Broker doesn't retry reconnecting to an expired Zookeeper connection

[Braedon Vickers (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3984?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Braedon Vickers updated KAFKA-3984:
---
Description: 
We've been having issues with the network connectivity of our Kafka cluster, 
and this seems to be triggering an issue where the brokers stop trying to 
reconnect to Zookeeper, leaving us with a broken cluster even when the network 
has recovered.

When network issues begin we see {{java.net.NoRouteToHostException}} exceptions 
from {{org.apache.zookeeper.ClientCnxn}} as it attempts to re-establish the 
connection. If the network issue resolves itself while we are only getting 
these errors the broker seems to reconnect fine.

However, a lot of the time we end up with a message like this:
{code}[2016-07-22 00:21:44,181] FATAL Could not establish session with 
zookeeper (kafka.server.KafkaHealthcheck)
org.I0Itec.zkclient.exception.ZkException: Unable to connect to 
at org.I0Itec.zkclient.ZkConnection.connect(ZkConnection.java:71)
at org.I0Itec.zkclient.ZkClient.reconnect(ZkClient.java:1279)
...
Caused by: java.net.UnknownHostException: 
at java.net.InetAddress.getAllByName(InetAddress.java:1126)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
at 
org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:61)
at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
...
{code}
(apologies for the partial stack traces - I'm having to try and reconstruct 
them from a less than ideal centralised logging setup.)

If this happens, the broker stops trying to reconnect to Zookeeper, and we have 
to restart it.

It looks like while the {{org.apache.zookeeper.Zookeeper}} client's state isn't 
{{Expired}} it will keep retrying the connection, and will recover OK when the 
network is back. However, once it changes to {{Expired}} (not entirely sure how 
that happens - based on the session timeout perhaps?) zkclient closes the 
existing client and attempts to create a new one. If the network is still down, 
the client constructor throws a {{java.net.UnknownHostException}}, zkclient 
calls {{handleSessionEstablishmentError()}} on {{KafkaHealthcheck}}, 
{{KafkaHealthcheck.handleSessionEstablishmentError()}} logs a "Fatal" error and 
does nothing else.

It seems like some form of retry needs to happen here, or the broker is stuck 
with no Zookeeper connection 
indefinitely.{{KafkaHealthcheck.handleSessionEstablishmentError()}} used to 
kill the JVM, but that was removed in 
https://issues.apache.org/jira/browse/KAFKA-2405. Killing the JVM would be 
better than doing nothing, as then your init system could restart it, allowing 
it to recover once the network was back.

Our cluster is running 0.9.0.1, so not sure if it affects 0.10.0.0 as well. 
However, it seems likely, as there doesn't seem to be any code changes in kafka 
or zkclient that would affect this behaviour.

  was:
We've been having issues with the network connectivity of our Kafka cluster, 
and this seems to be triggering an issue where the brokers stop trying to 
reconnect to Zookeeper, leaving us with a broken cluster even when the network 
has recovered.

When network issues begin we see {{java.net.NoRouteToHostException}} exceptions 
from {{org.apache.zookeeper.ClientCnxn}} as it attempts to re-establish the 
connection. If the network issue resolves itself while we are only getting 
these errors the broker seems to reconnect fine.

However, a lot of the time we end up with a message like this:
{code}[2016-07-22 00:21:44,181] FATAL Could not establish session with 
zookeeper (kafka.server.KafkaHealthcheck)
org.I0Itec.zkclient.exception.ZkException: Unable to connect to 
at org.I0Itec.zkclient.ZkConnection.connect(ZkConnection.java:71)
at org.I0Itec.zkclient.ZkClient.reconnect(ZkClient.java:1279)
...
Caused by: java.net.UnknownHostException: 
at java.net.InetAddress.getAllByName(InetAddress.java:1126)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
at 
org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:61)
at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
...
{code}
(apologies for the partial stack traces - I'm having to try and reconstruct 
them from a less than ideal centralised logging setup.)

If this happens, the broker stops trying to reconnect to Zookeeper, and we have 
to restart it.

It looks like while the {{org.apache.zookeeper.Zookeeper}} client's state isn't 
{{Expired}} it will keep retrying the connection, and will recover OK when the 
network is back. However, once it changes to {{Expired}} (not entirely sure how 
that happens - based on the session timeout perhaps?) zkclient closes the 
existing client and attempts to create a new one. If the network is still down, 
the client constructor throws a {{java.net.UnknownHostException}}, zkclient 
calls 

[jira] [Created] (KAFKA-3984) Broker doesn't retry reconnecting to an expired Zookeeper connection

[Braedon Vickers (JIRA)]

Braedon Vickers created KAFKA-3984:
--

 Summary: Broker doesn't retry reconnecting to an expired Zookeeper 
connection
 Key: KAFKA-3984
 URL: https://issues.apache.org/jira/browse/KAFKA-3984
 Project: Kafka
  Issue Type: Bug
Affects Versions: 0.9.0.1
Reporter: Braedon Vickers


We've been having issues with the network connectivity of our Kafka cluster, 
and this seems to be triggering an issue where the brokers stop trying to 
reconnect to Zookeeper, leaving us with a broken cluster even when the network 
has recovered.

When network issues begin we see {{java.net.NoRouteToHostException}} exceptions 
from {{org.apache.zookeeper.ClientCnxn}} as it attempts to re-establish the 
connection. If the network issue resolves itself while we are only getting 
these errors the broker seems to reconnect fine.

However, a lot of the time we end up with a message like this:
{code}[2016-07-22 00:21:44,181] FATAL Could not establish session with 
zookeeper (kafka.server.KafkaHealthcheck)
org.I0Itec.zkclient.exception.ZkException: Unable to connect to 
at org.I0Itec.zkclient.ZkConnection.connect(ZkConnection.java:71)
at org.I0Itec.zkclient.ZkClient.reconnect(ZkClient.java:1279)
...
Caused by: java.net.UnknownHostException: 
at java.net.InetAddress.getAllByName(InetAddress.java:1126)
at java.net.InetAddress.getAllByName(InetAddress.java:1192)
at 
org.apache.zookeeper.client.StaticHostProvider.(StaticHostProvider.java:61)
at org.apache.zookeeper.ZooKeeper.(ZooKeeper.java:445)
...
{code}
(apologies for the partial stack traces - I'm having to try and reconstruct 
them from a less than ideal centralised logging setup.)

If this happens, the broker stops trying to reconnect to Zookeeper, and we have 
to restart it.

It looks like while the {{org.apache.zookeeper.Zookeeper}} client's state isn't 
{{Expired}} it will keep retrying the connection, and will recover OK when the 
network is back. However, once it changes to {{Expired}} (not entirely sure how 
that happens - based on the session timeout perhaps?) zkclient closes the 
existing client and attempts to create a new one. If the network is still down, 
the client constructor throws a {{java.net.UnknownHostException}}, zkclient 
calls {{handleSessionEstablishmentError()}} on {{KafkaHealthcheck}}, 
{{KafkaHealthcheck.handleSessionEstablishmentError()}} logs a "Fatal" error and 
does nothing else.

It seems like some form of retry needs to happen here, or the broker is stuck 
with no Zookeeper connection 
indefinitely.{{KafkaHealthcheck.handleSessionEstablishmentError()}} used to 
kill the JVM, but that was removed in 
https://issues.apache.org/jira/browse/KAFKA-2405. Killing the JVM would be 
better than doing nothing, as then your init system could restart it, allowing 
it to recover once the network was back.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk7 #1435

[Apache Jenkins Server]

See 

Changes:

[me] KAFKA-3782: Fix transient failure in connect distributed bounce test

--
[...truncated 10686 lines...]

org.apache.kafka.common.serialization.SerializationTest > 
testByteBufferSerializer PASSED

org.apache.kafka.common.config.ConfigDefTest > testBasicTypes STARTED

org.apache.kafka.common.config.ConfigDefTest > testBasicTypes PASSED

org.apache.kafka.common.config.ConfigDefTest > testNullDefault STARTED

org.apache.kafka.common.config.ConfigDefTest > testNullDefault PASSED

org.apache.kafka.common.config.ConfigDefTest > testParseForValidate STARTED

org.apache.kafka.common.config.ConfigDefTest > testParseForValidate PASSED

org.apache.kafka.common.config.ConfigDefTest > testInvalidDefaultRange STARTED

org.apache.kafka.common.config.ConfigDefTest > testInvalidDefaultRange PASSED

org.apache.kafka.common.config.ConfigDefTest > testGroupInference STARTED

org.apache.kafka.common.config.ConfigDefTest > testGroupInference PASSED

org.apache.kafka.common.config.ConfigDefTest > testValidators STARTED

org.apache.kafka.common.config.ConfigDefTest > testValidators PASSED

org.apache.kafka.common.config.ConfigDefTest > testValidateCannotParse STARTED

org.apache.kafka.common.config.ConfigDefTest > testValidateCannotParse PASSED

org.apache.kafka.common.config.ConfigDefTest > testValidate STARTED

org.apache.kafka.common.config.ConfigDefTest > testValidate PASSED

org.apache.kafka.common.config.ConfigDefTest > testInvalidDefaultString STARTED

org.apache.kafka.common.config.ConfigDefTest > testInvalidDefaultString PASSED

org.apache.kafka.common.config.ConfigDefTest > testSslPasswords STARTED

org.apache.kafka.common.config.ConfigDefTest > testSslPasswords PASSED

org.apache.kafka.common.config.ConfigDefTest > testInvalidDefault STARTED

org.apache.kafka.common.config.ConfigDefTest > testInvalidDefault PASSED

org.apache.kafka.common.config.ConfigDefTest > testMissingRequired STARTED

org.apache.kafka.common.config.ConfigDefTest > testMissingRequired PASSED

org.apache.kafka.common.config.ConfigDefTest > testNullDefaultWithValidator 
STARTED

org.apache.kafka.common.config.ConfigDefTest > testNullDefaultWithValidator 
PASSED

org.apache.kafka.common.config.ConfigDefTest > testDefinedTwice STARTED

org.apache.kafka.common.config.ConfigDefTest > testDefinedTwice PASSED

org.apache.kafka.common.config.ConfigDefTest > testBadInputs STARTED

org.apache.kafka.common.config.ConfigDefTest > testBadInputs PASSED

org.apache.kafka.common.config.ConfigDefTest > testValidateMissingConfigKey 
STARTED

org.apache.kafka.common.config.ConfigDefTest > testValidateMissingConfigKey 
PASSED

org.apache.kafka.common.config.AbstractConfigTest > testOriginalsWithPrefix 
STARTED

org.apache.kafka.common.config.AbstractConfigTest > testOriginalsWithPrefix 
PASSED

org.apache.kafka.common.config.AbstractConfigTest > testConfiguredInstances 
STARTED

org.apache.kafka.common.config.AbstractConfigTest > testConfiguredInstances 
PASSED

org.apache.kafka.common.config.AbstractConfigTest > testUnused STARTED

org.apache.kafka.common.config.AbstractConfigTest > testUnused PASSED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testMissingUsernameSaslPlain STARTED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testMissingUsernameSaslPlain PASSED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testUnauthenticatedApiVersionsRequestOverPlaintext STARTED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testUnauthenticatedApiVersionsRequestOverPlaintext PASSED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testMechanismPluggability STARTED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testMechanismPluggability PASSED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testApiVersionsRequestWithUnsupportedVersion STARTED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testApiVersionsRequestWithUnsupportedVersion PASSED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testMissingPasswordSaslPlain STARTED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testMissingPasswordSaslPlain PASSED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testInvalidLoginModule STARTED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testInvalidLoginModule PASSED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testInvalidMechanism STARTED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testInvalidMechanism PASSED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testDisabledMechanism STARTED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testDisabledMechanism PASSED

[GitHub] kafka pull request #1644: MINOR: Update to Gradle 2.14.1

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1644


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (KAFKA-3782) Transient failure with kafkatest.tests.connect.connect_distributed_test.ConnectDistributedTest.test_bounce.clean=True

[Ewen Cheslack-Postava (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ewen Cheslack-Postava resolved KAFKA-3782.
--
   Resolution: Fixed
Fix Version/s: 0.10.0.1

Issue resolved by pull request 1650
[https://github.com/apache/kafka/pull/1650]

> Transient failure with 
> kafkatest.tests.connect.connect_distributed_test.ConnectDistributedTest.test_bounce.clean=True
> -
>
> Key: KAFKA-3782
> URL: https://issues.apache.org/jira/browse/KAFKA-3782
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.10.0.0
>Reporter: Liquan Pei
>Assignee: Jason Gustafson
>Priority: Minor
> Fix For: 0.10.1.0, 0.10.0.1
>
>
> For commit 946ae60
> max() arg is an empty sequence
> Traceback (most recent call last):
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.0/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
>  line 106, in run_all_tests
> data = self.run_single_test()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.0/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
>  line 162, in run_single_test
> return self.current_test_context.function(self.current_test)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.0/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/mark/_mark.py",
>  line 331, in wrapper
> return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.0/kafka/tests/kafkatest/tests/connect/connect_distributed_test.py",
>  line 321, in test_bounce
> sink_seqno_max = max(sink_seqnos)
> ValueError: max() arg is an empty sequence



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1650: KAFKA-3782: Fix transient failure in connect distr...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1650


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3782) Transient failure with kafkatest.tests.connect.connect_distributed_test.ConnectDistributedTest.test_bounce.clean=True

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15388801#comment-15388801
 ] 

ASF GitHub Bot commented on KAFKA-3782:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1650


> Transient failure with 
> kafkatest.tests.connect.connect_distributed_test.ConnectDistributedTest.test_bounce.clean=True
> -
>
> Key: KAFKA-3782
> URL: https://issues.apache.org/jira/browse/KAFKA-3782
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.10.0.0
>Reporter: Liquan Pei
>Assignee: Jason Gustafson
>Priority: Minor
> Fix For: 0.10.1.0, 0.10.0.1
>
>
> For commit 946ae60
> max() arg is an empty sequence
> Traceback (most recent call last):
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.0/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
>  line 106, in run_all_tests
> data = self.run_single_test()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.0/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
>  line 162, in run_single_test
> return self.current_test_context.function(self.current_test)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.0/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/mark/_mark.py",
>  line 331, in wrapper
> return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.0/kafka/tests/kafkatest/tests/connect/connect_distributed_test.py",
>  line 321, in test_bounce
> sink_seqno_max = max(sink_seqnos)
> ValueError: max() arg is an empty sequence



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-4 ACL Admin Schema

[Ismael Juma]

Hi Grant,

Thanks for the KIP.  A few questions and comments:

1. My main concern is that we are skipping the discussion on the desired
model for controlling ACL access and updates. I understand the desire to
reduce the scope, but this seems to be a fundamental aspect of the design
that we need to get right. Without a plan for that, it is difficult to
evaluate if that part of the current proposal is fine.
2. Are the Java objects in "org.apache.kafka.common.security.auth" going to
be public API? If so, we should explain why they should be public and
describe them in the KIP. If not, we should mention that.
3. It would be nice to have a name for a (Resource, ACL) pair. The current
protocol uses `requests`/`responses` for the list of such pairs, but it
would be nice to have something more descriptive, if possible. Any ideas?
4. There is no CreateAcls or DeleteAcls (unlike CreateTopics and
DeleteTopics, for example). It would be good to explain the reasoning for
this choice (Jason also asked this question).
5. What is the plan for when we add standard exceptions to the Authorizer
interface? Will we bump the protocol version?

Thanks,
Ismael

On Thu, Jul 14, 2016 at 5:09 PM, Grant Henke  wrote:

> The KIP-4 Delete Topic Schema vote has passed and the patch
>  is available for review. Now I
> would like to start the discussion for the Acls request/response and server
> side implementations. This includes the ListAclsRequest/Response and the
> AlterAclsRequest/Response.
>
> Details for this implementation can be read here:
> *
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-ACLAdminSchema(KAFKA-3266)
> <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-ACLAdminSchema(KAFKA-3266)
> >*
>
> I have included the exact content below for clarity:
>
> > ACL Admin Schema (KAFKA-3266
> > )
> >
> > *Note*: Some of this work/code overlaps with "KIP-50 - Move Authorizer to
> > o.a.k.common package
> > <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-50+-+Move+Authorizer+to+o.a.k.common+package
> >".
> > KIP-4 does not change the Authorizer interface at all, but does provide
> > java objects in "org.apache.kafka.common.security.auth" to be used in the
> > protocol request/response classes. It also provides translations between
> > the Java and Scala versions for server side compatibility with
> > the Authorizer interface.
> >
> > List ACLs Request
> >
> >
> >
> > ListAcls Request (Version: 0) => principal resource
> >   principal => NULLABLE_STRING
> >   resource => resource_type resource_name
> > resource_type => INT8
> > resource_name => STRING
> >
> > Request semantics:
> >
> >1. Can be sent to any broker
> >2. If a non-null principal is provided the returned ACLs will be
> >filtered by that principle, otherwise ACLs for all principals will be
> >listed.
> >3. If a resource with a resource_type != -1 is provided ACLs will be
> >filtered by that resource, otherwise ACLs for all resources will be
> listed.
> >4. Any principle can list their own ACLs where the permission type is
> >"Allow", Otherwise the principle must be authorized to the "All"
> Operation
> >on the "Cluster" resource to list ACLs.
> >- Unauthorized requests will receive a ClusterAuthorizationException
> >   - This avoids adding a new operation that an existing authorizer
> >   implementation may not be aware of.
> >   - This can be reviewed and further refined/restricted as a follow
> >   up ACLs review after this KIP. See Follow Up Changes
> >   <
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-4+-+Command+line+and+centralized+administrative+operations#KIP-4-Commandlineandcentralizedadministrativeoperations-follow-up-changes
> >
> >   .
> >5. Requesting a resource or principle that does not have any ACLs will
> >not result in an error, instead empty response list is returned
> >
> > List ACLs Response
> >
> >
> >
> > ListAcls Response (Version: 0) => [responses] error_code
> >   responses => resource [acls]
> > resource => resource_type resource_name
> >   resource_type => INT8
> >   resource_name => STRING
> > acls => acl_principle acl_permission_type acl_host acl_operation
> >   acl_principle => STRING
> >   acl_permission_type => INT8
> >   acl_host => STRING
> >   acl_operation => INT8
> >   error_code => INT16
> >
> > Alter ACLs Request
> >
> >
> >
> > AlterAcls Request (Version: 0) => [requests]
> >   requests => resource [actions]
> > resource => resource_type resource_name
> >   resource_type => INT8
> >   resource_name => STRING
> > actions => action 

Jenkins build is back to normal : kafka-trunk-jdk8 #769

[Apache Jenkins Server]

See 

Build failed in Jenkins: kafka-trunk-jdk7 #1434

[Apache Jenkins Server]

See 

Changes:

[wangguoz] MINOR: Upgrade RocksDB to 4.8.0

[ismael] HOTFIX: Adding init file so streams benchmark is autodiscovered

--
[...truncated 13085 lines...]

org.apache.kafka.common.record.KafkaLZ4Test > testKafkaLZ4[3] PASSED

org.apache.kafka.common.record.MemoryRecordsTest > testHasRoomForMethod[0] 
STARTED

org.apache.kafka.common.record.MemoryRecordsTest > testHasRoomForMethod[0] 
PASSED

org.apache.kafka.common.record.MemoryRecordsTest > testIterator[0] STARTED

org.apache.kafka.common.record.MemoryRecordsTest > testIterator[0] PASSED

org.apache.kafka.common.record.MemoryRecordsTest > testHasRoomForMethod[1] 
STARTED

org.apache.kafka.common.record.MemoryRecordsTest > testHasRoomForMethod[1] 
PASSED

org.apache.kafka.common.record.MemoryRecordsTest > testIterator[1] STARTED

org.apache.kafka.common.record.MemoryRecordsTest > testIterator[1] PASSED

org.apache.kafka.common.record.MemoryRecordsTest > testHasRoomForMethod[2] 
STARTED

org.apache.kafka.common.record.MemoryRecordsTest > testHasRoomForMethod[2] 
PASSED

org.apache.kafka.common.record.MemoryRecordsTest > testIterator[2] STARTED

org.apache.kafka.common.record.MemoryRecordsTest > testIterator[2] PASSED

org.apache.kafka.common.record.MemoryRecordsTest > testHasRoomForMethod[3] 
STARTED

org.apache.kafka.common.record.MemoryRecordsTest > testHasRoomForMethod[3] 
PASSED

org.apache.kafka.common.record.MemoryRecordsTest > testIterator[3] STARTED

org.apache.kafka.common.record.MemoryRecordsTest > testIterator[3] PASSED

org.apache.kafka.common.security.auth.KafkaPrincipalTest > 
testPrincipalNameCanContainSeparator STARTED

org.apache.kafka.common.security.auth.KafkaPrincipalTest > 
testPrincipalNameCanContainSeparator PASSED

org.apache.kafka.common.security.auth.KafkaPrincipalTest > 
testEqualsAndHashCode STARTED

org.apache.kafka.common.security.auth.KafkaPrincipalTest > 
testEqualsAndHashCode PASSED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testMissingUsernameSaslPlain STARTED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testMissingUsernameSaslPlain PASSED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testUnauthenticatedApiVersionsRequestOverPlaintext STARTED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testUnauthenticatedApiVersionsRequestOverPlaintext PASSED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testMechanismPluggability STARTED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testMechanismPluggability PASSED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testApiVersionsRequestWithUnsupportedVersion STARTED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testApiVersionsRequestWithUnsupportedVersion PASSED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testMissingPasswordSaslPlain STARTED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testMissingPasswordSaslPlain PASSED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testInvalidLoginModule STARTED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testInvalidLoginModule PASSED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testInvalidMechanism STARTED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testInvalidMechanism PASSED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testDisabledMechanism STARTED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testDisabledMechanism PASSED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testPacketSizeTooBig STARTED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testPacketSizeTooBig PASSED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testInvalidUsernameSaslPlain STARTED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testInvalidUsernameSaslPlain PASSED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testMultipleServerMechanisms STARTED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testMultipleServerMechanisms PASSED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testValidSaslPlainOverPlaintext STARTED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testValidSaslPlainOverPlaintext PASSED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testValidSaslPlainOverSsl STARTED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testValidSaslPlainOverSsl PASSED

org.apache.kafka.common.security.authenticator.SaslAuthenticatorTest > 
testInvalidApiVersionsRequestSequence STARTED

[GitHub] kafka pull request #1650: KAFKA-3782: Fix transient failure in connect distr...

[hachikuji]

GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/1650

KAFKA-3782: Fix transient failure in connect distributed bounce test



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hachikuji/kafka KAFKA-3782

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1650.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1650


commit 76931271265e3257322ea6b32f28618bdce5841d
Author: Jason Gustafson 
Date:   2016-07-18T18:47:14Z

KAFKA-3782: Fix transient failure in connect distributed bounce test

commit af098a94d0a9a001ce48e7a36e155ebb92faedb7
Author: Jason Gustafson 
Date:   2016-07-22T00:54:07Z

reduce source throughput to decrease chance of OOM




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3782) Transient failure with kafkatest.tests.connect.connect_distributed_test.ConnectDistributedTest.test_bounce.clean=True

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15388692#comment-15388692
 ] 

ASF GitHub Bot commented on KAFKA-3782:
---

GitHub user hachikuji opened a pull request:

https://github.com/apache/kafka/pull/1650

KAFKA-3782: Fix transient failure in connect distributed bounce test



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/hachikuji/kafka KAFKA-3782

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1650.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1650


commit 76931271265e3257322ea6b32f28618bdce5841d
Author: Jason Gustafson 
Date:   2016-07-18T18:47:14Z

KAFKA-3782: Fix transient failure in connect distributed bounce test

commit af098a94d0a9a001ce48e7a36e155ebb92faedb7
Author: Jason Gustafson 
Date:   2016-07-22T00:54:07Z

reduce source throughput to decrease chance of OOM




> Transient failure with 
> kafkatest.tests.connect.connect_distributed_test.ConnectDistributedTest.test_bounce.clean=True
> -
>
> Key: KAFKA-3782
> URL: https://issues.apache.org/jira/browse/KAFKA-3782
> Project: Kafka
>  Issue Type: Bug
>  Components: KafkaConnect
>Affects Versions: 0.10.0.0
>Reporter: Liquan Pei
>Assignee: Jason Gustafson
>Priority: Minor
> Fix For: 0.10.1.0
>
>
> For commit 946ae60
> max() arg is an empty sequence
> Traceback (most recent call last):
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.0/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
>  line 106, in run_all_tests
> data = self.run_single_test()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.0/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
>  line 162, in run_single_test
> return self.current_test_context.function(self.current_test)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.0/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/mark/_mark.py",
>  line 331, in wrapper
> return functools.partial(f, *args, **kwargs)(*w_args, **w_kwargs)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka-0.10.0/kafka/tests/kafkatest/tests/connect/connect_distributed_test.py",
>  line 321, in test_bounce
> sink_seqno_max = max(sink_seqnos)
> ValueError: max() arg is an empty sequence



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3887) StreamBounceTest.test_bounce and StreamSmokeTest.test_streams failing

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15388630#comment-15388630
 ] 

Guozhang Wang commented on KAFKA-3887:
--

This seems to be a regression that was introduced in 
https://issues.apache.org/jira/browse/KAFKA-3870 and gets fixed in 
https://issues.apache.org/jira/browse/KAFKA-3911 already. Will keep an eye on 
tonight's run to see if it goes away itself.

> StreamBounceTest.test_bounce and StreamSmokeTest.test_streams failing
> -
>
> Key: KAFKA-3887
> URL: https://issues.apache.org/jira/browse/KAFKA-3887
> Project: Kafka
>  Issue Type: Bug
>  Components: streams, system tests
>Reporter: Ismael Juma
>Assignee: Guozhang Wang
>  Labels: transient-system-test-failure
> Fix For: 0.10.0.1
>
>
> StreamBounceTest.test_bounce and StreamSmokeTest.test_streams has been 
> failing semi-regularly. Output from the latest failure:
> {code}
> 
> test_id:
> 2016-06-20--001.kafkatest.tests.streams.streams_bounce_test.StreamsBounceTest.test_bounce
> status: FAIL
> run time:   4 minutes 13.916 seconds
> Streams Smoke Test process on ubuntu@worker5 took too long to exit
> Traceback (most recent call last):
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
>  line 106, in run_all_tests
> data = self.run_single_test()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
>  line 162, in run_single_test
> return self.current_test_context.function(self.current_test)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/streams/streams_bounce_test.py",
>  line 67, in test_bounce
> self.driver.wait()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/services/streams.py",
>  line 94, in wait
> wait_until(lambda: not node.account.alive(pid), timeout_sec=120, 
> err_msg="Streams Smoke Test process on " + str(node.account) + " took too 
> long to exit")
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/utils/util.py",
>  line 36, in wait_until
> raise TimeoutError(err_msg)
> TimeoutError: Streams Smoke Test process on ubuntu@worker5 took too long to 
> exit
> 
> test_id:
> 2016-06-20--001.kafkatest.tests.streams.streams_smoke_test.StreamsSmokeTest.test_streams
> status: FAIL
> run time:   4 minutes 36.426 seconds
> Streams Smoke Test process on ubuntu@worker9 took too long to exit
> Traceback (most recent call last):
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
>  line 106, in run_all_tests
> data = self.run_single_test()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
>  line 162, in run_single_test
> return self.current_test_context.function(self.current_test)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/streams/streams_smoke_test.py",
>  line 67, in test_streams
> self.driver.wait()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/services/streams.py",
>  line 94, in wait
> wait_until(lambda: not node.account.alive(pid), timeout_sec=120, 
> err_msg="Streams Smoke Test process on " + str(node.account) + " took too 
> long to exit")
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/utils/util.py",
>  line 36, in wait_until
> raise TimeoutError(err_msg)
> TimeoutError: Streams Smoke Test process on ubuntu@worker9 took too long to 
> exit
> {code}
> https://jenkins.confluent.io/job/system-test-kafka/255/console



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk8 #768

[Apache Jenkins Server]

See 

Changes:

[wangguoz] MINOR: Upgrade RocksDB to 4.8.0

--
[...truncated 5234 lines...]

kafka.log.LogCleanerIntegrationTest > cleanerTest[2] PASSED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[2] STARTED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[2] PASSED

kafka.log.LogCleanerIntegrationTest > cleanerTest[3] STARTED

kafka.log.LogCleanerIntegrationTest > cleanerTest[3] PASSED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[3] STARTED

kafka.log.LogCleanerIntegrationTest > testCleanerWithMessageFormatV0[3] PASSED

kafka.log.LogTest > testParseTopicPartitionNameForMissingTopic STARTED

kafka.log.LogTest > testParseTopicPartitionNameForMissingTopic PASSED

kafka.log.LogTest > testIndexRebuild STARTED

kafka.log.LogTest > testIndexRebuild PASSED

kafka.log.LogTest > testLogRolls STARTED

kafka.log.LogTest > testLogRolls PASSED

kafka.log.LogTest > testMessageSizeCheck STARTED

kafka.log.LogTest > testMessageSizeCheck PASSED

kafka.log.LogTest > testAsyncDelete STARTED

kafka.log.LogTest > testAsyncDelete PASSED

kafka.log.LogTest > testReadOutOfRange STARTED

kafka.log.LogTest > testReadOutOfRange PASSED

kafka.log.LogTest > testAppendWithOutOfOrderOffsetsThrowsException STARTED

kafka.log.LogTest > testAppendWithOutOfOrderOffsetsThrowsException PASSED

kafka.log.LogTest > testReadAtLogGap STARTED

kafka.log.LogTest > testReadAtLogGap PASSED

kafka.log.LogTest > testTimeBasedLogRoll STARTED

kafka.log.LogTest > testTimeBasedLogRoll PASSED

kafka.log.LogTest > testLoadEmptyLog STARTED

kafka.log.LogTest > testLoadEmptyLog PASSED

kafka.log.LogTest > testMessageSetSizeCheck STARTED

kafka.log.LogTest > testMessageSetSizeCheck PASSED

kafka.log.LogTest > testIndexResizingAtTruncation STARTED

kafka.log.LogTest > testIndexResizingAtTruncation PASSED

kafka.log.LogTest > testCompactedTopicConstraints STARTED

kafka.log.LogTest > testCompactedTopicConstraints PASSED

kafka.log.LogTest > testThatGarbageCollectingSegmentsDoesntChangeOffset STARTED

kafka.log.LogTest > testThatGarbageCollectingSegmentsDoesntChangeOffset PASSED

kafka.log.LogTest > testAppendAndReadWithSequentialOffsets STARTED

kafka.log.LogTest > testAppendAndReadWithSequentialOffsets PASSED

kafka.log.LogTest > testDeleteOldSegmentsMethod STARTED

kafka.log.LogTest > testDeleteOldSegmentsMethod PASSED

kafka.log.LogTest > testParseTopicPartitionNameForNull STARTED

kafka.log.LogTest > testParseTopicPartitionNameForNull PASSED

kafka.log.LogTest > testAppendAndReadWithNonSequentialOffsets STARTED

kafka.log.LogTest > testAppendAndReadWithNonSequentialOffsets PASSED

kafka.log.LogTest > testParseTopicPartitionNameForMissingSeparator STARTED

kafka.log.LogTest > testParseTopicPartitionNameForMissingSeparator PASSED

kafka.log.LogTest > testCorruptIndexRebuild STARTED

kafka.log.LogTest > testCorruptIndexRebuild PASSED

kafka.log.LogTest > testBogusIndexSegmentsAreRemoved STARTED

kafka.log.LogTest > testBogusIndexSegmentsAreRemoved PASSED

kafka.log.LogTest > testCompressedMessages STARTED

kafka.log.LogTest > testCompressedMessages PASSED

kafka.log.LogTest > testAppendMessageWithNullPayload STARTED

kafka.log.LogTest > testAppendMessageWithNullPayload PASSED

kafka.log.LogTest > testCorruptLog STARTED

kafka.log.LogTest > testCorruptLog PASSED

kafka.log.LogTest > testLogRecoversToCorrectOffset STARTED

kafka.log.LogTest > testLogRecoversToCorrectOffset PASSED

kafka.log.LogTest > testReopenThenTruncate STARTED

kafka.log.LogTest > testReopenThenTruncate PASSED

kafka.log.LogTest > testParseTopicPartitionNameForMissingPartition STARTED

kafka.log.LogTest > testParseTopicPartitionNameForMissingPartition PASSED

kafka.log.LogTest > testParseTopicPartitionNameForEmptyName STARTED

kafka.log.LogTest > testParseTopicPartitionNameForEmptyName PASSED

kafka.log.LogTest > testOpenDeletesObsoleteFiles STARTED

kafka.log.LogTest > testOpenDeletesObsoleteFiles PASSED

kafka.log.LogTest > testSizeBasedLogRoll STARTED

kafka.log.LogTest > testSizeBasedLogRoll PASSED

kafka.log.LogTest > testTimeBasedLogRollJitter STARTED

kafka.log.LogTest > testTimeBasedLogRollJitter PASSED

kafka.log.LogTest > testParseTopicPartitionName STARTED

kafka.log.LogTest > testParseTopicPartitionName PASSED

kafka.log.LogTest > testTruncateTo STARTED

kafka.log.LogTest > testTruncateTo PASSED

kafka.log.LogTest > testCleanShutdownFile STARTED

kafka.log.LogTest > testCleanShutdownFile PASSED

kafka.log.LogConfigTest > testFromPropsEmpty STARTED

kafka.log.LogConfigTest > testFromPropsEmpty PASSED

kafka.log.LogConfigTest > testKafkaConfigToProps STARTED

kafka.log.LogConfigTest > testKafkaConfigToProps PASSED

kafka.log.LogConfigTest > testFromPropsInvalid STARTED

kafka.log.LogConfigTest > testFromPropsInvalid PASSED

kafka.log.CleanerTest > testBuildOffsetMap STARTED

kafka.log.CleanerTest > 

[GitHub] kafka pull request #1645: HOTFIX: Adding init file so streams benchmark is a...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1645


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3929) Add prefix for underlying clients configs in StreamConfig

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15388593#comment-15388593
 ] 

ASF GitHub Bot commented on KAFKA-3929:
---

GitHub user dguy opened a pull request:

https://github.com/apache/kafka/pull/1649

KAFKA-3929: Add prefix for underlying clients configs in StreamConfig

Add prefixes for consumer and producer configs to StreamsConfig, but be 
backward compatible.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dguy/kafka kafka-3929

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1649.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1649


commit 67f5e22eed250f65535bb1c93cf17b55324ddd0f
Author: Damian Guy 
Date:   2016-07-21T23:14:12Z

enable prefixes for consumer and producer configs in streams config




> Add prefix for underlying clients configs in StreamConfig
> -
>
> Key: KAFKA-3929
> URL: https://issues.apache.org/jira/browse/KAFKA-3929
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Damian Guy
>  Labels: api
>
> There are a couple of configs that have the same name for producer / consumer 
> configs, e.g. take a look at {{CommonClientConfigs}}, and also for producer / 
> consumer interceptors there are commonly named configs as well.
> This is semi-related to KAFKA-3740 since we need to add "sub-class" configs 
> for RocksDB as well, and we'd better have some prefix mechanism for such 
> hierarchical configs in general.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1649: KAFKA-3929: Add prefix for underlying clients conf...

[dguy]

GitHub user dguy opened a pull request:

https://github.com/apache/kafka/pull/1649

KAFKA-3929: Add prefix for underlying clients configs in StreamConfig

Add prefixes for consumer and producer configs to StreamsConfig, but be 
backward compatible.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dguy/kafka kafka-3929

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1649.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1649


commit 67f5e22eed250f65535bb1c93cf17b55324ddd0f
Author: Damian Guy 
Date:   2016-07-21T23:14:12Z

enable prefixes for consumer and producer configs in streams config




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #1647: MINOR: Upgrade RocksDB to 4.8.0

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1647


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: kafka-trunk-jdk8 #767

[Apache Jenkins Server]

See 

Changes:

[wangguoz] KAFKA-3911: KTable source materialization

--
[...truncated 4655 lines...]

kafka.server.ServerShutdownTest > testCleanShutdownAfterFailedStartup STARTED

kafka.server.ServerShutdownTest > testCleanShutdownAfterFailedStartup PASSED

kafka.server.ServerShutdownTest > testConsecutiveShutdown STARTED

kafka.server.ServerShutdownTest > testConsecutiveShutdown PASSED

kafka.server.ServerShutdownTest > testCleanShutdown STARTED

kafka.server.ServerShutdownTest > testCleanShutdown PASSED

kafka.server.ServerShutdownTest > testCleanShutdownWithDeleteTopicEnabled 
STARTED

kafka.server.ServerShutdownTest > testCleanShutdownWithDeleteTopicEnabled PASSED

kafka.server.ServerStartupTest > testBrokerStateRunningAfterZK STARTED

kafka.server.ServerStartupTest > testBrokerStateRunningAfterZK PASSED

kafka.server.ServerStartupTest > testBrokerCreatesZKChroot STARTED

kafka.server.ServerStartupTest > testBrokerCreatesZKChroot PASSED

kafka.server.ServerStartupTest > testConflictBrokerRegistration STARTED

kafka.server.ServerStartupTest > testConflictBrokerRegistration PASSED

kafka.server.ServerStartupTest > testBrokerSelfAware STARTED

kafka.server.ServerStartupTest > testBrokerSelfAware PASSED

kafka.server.EdgeCaseRequestTest > testInvalidApiVersionRequest STARTED

kafka.server.EdgeCaseRequestTest > testInvalidApiVersionRequest PASSED

kafka.server.EdgeCaseRequestTest > testMalformedHeaderRequest STARTED

kafka.server.EdgeCaseRequestTest > testMalformedHeaderRequest PASSED

kafka.server.EdgeCaseRequestTest > testProduceRequestWithNullClientId STARTED

kafka.server.EdgeCaseRequestTest > testProduceRequestWithNullClientId PASSED

kafka.server.EdgeCaseRequestTest > testInvalidApiKeyRequest STARTED

kafka.server.EdgeCaseRequestTest > testInvalidApiKeyRequest PASSED

kafka.server.EdgeCaseRequestTest > testHeaderOnlyRequest STARTED

kafka.server.EdgeCaseRequestTest > testHeaderOnlyRequest PASSED

kafka.server.CreateTopicsRequestTest > testValidCreateTopicsRequests STARTED

kafka.server.CreateTopicsRequestTest > testValidCreateTopicsRequests PASSED

kafka.server.CreateTopicsRequestTest > testErrorCreateTopicsRequests STARTED

kafka.server.CreateTopicsRequestTest > testErrorCreateTopicsRequests PASSED

kafka.server.CreateTopicsRequestTest > testInvalidCreateTopicsRequests STARTED

kafka.server.CreateTopicsRequestTest > testInvalidCreateTopicsRequests PASSED

kafka.server.CreateTopicsRequestTest > testNotController STARTED

kafka.server.CreateTopicsRequestTest > testNotController PASSED

kafka.server.OffsetCommitTest > testUpdateOffsets STARTED

kafka.server.OffsetCommitTest > testUpdateOffsets PASSED

kafka.server.OffsetCommitTest > testLargeMetadataPayload STARTED

kafka.server.OffsetCommitTest > testLargeMetadataPayload PASSED

kafka.server.OffsetCommitTest > testCommitAndFetchOffsets STARTED

kafka.server.OffsetCommitTest > testCommitAndFetchOffsets PASSED

kafka.server.OffsetCommitTest > testOffsetExpiration STARTED

kafka.server.OffsetCommitTest > testOffsetExpiration PASSED

kafka.server.OffsetCommitTest > testNonExistingTopicOffsetCommit STARTED

kafka.server.OffsetCommitTest > testNonExistingTopicOffsetCommit PASSED

kafka.server.ReplicaManagerTest > testHighWaterMarkDirectoryMapping STARTED

kafka.server.ReplicaManagerTest > testHighWaterMarkDirectoryMapping PASSED

kafka.server.ReplicaManagerTest > 
testFetchBeyondHighWatermarkReturnEmptyResponse STARTED

kafka.server.ReplicaManagerTest > 
testFetchBeyondHighWatermarkReturnEmptyResponse PASSED

kafka.server.ReplicaManagerTest > testIllegalRequiredAcks STARTED

kafka.server.ReplicaManagerTest > testIllegalRequiredAcks PASSED

kafka.server.ReplicaManagerTest > testClearPurgatoryOnBecomingFollower STARTED

kafka.server.ReplicaManagerTest > testClearPurgatoryOnBecomingFollower PASSED

kafka.server.ReplicaManagerTest > testHighwaterMarkRelativeDirectoryMapping 
STARTED

kafka.server.ReplicaManagerTest > testHighwaterMarkRelativeDirectoryMapping 
PASSED

kafka.server.IsrExpirationTest > testIsrExpirationForSlowFollowers STARTED

kafka.server.IsrExpirationTest > testIsrExpirationForSlowFollowers PASSED

kafka.server.IsrExpirationTest > testIsrExpirationForStuckFollowers STARTED

kafka.server.IsrExpirationTest > testIsrExpirationForStuckFollowers PASSED

kafka.server.IsrExpirationTest > testIsrExpirationIfNoFetchRequestMade STARTED

kafka.server.IsrExpirationTest > testIsrExpirationIfNoFetchRequestMade PASSED

kafka.server.ControlledShutdownLeaderSelectorTest > testSelectLeader STARTED

kafka.server.ControlledShutdownLeaderSelectorTest > testSelectLeader PASSED

kafka.server.ClientQuotaManagerTest > testQuotaViolation STARTED

kafka.server.ClientQuotaManagerTest > testQuotaViolation PASSED

kafka.server.ClientQuotaManagerTest > testExpireQuotaSensors STARTED

kafka.server.ClientQuotaManagerTest > testExpireQuotaSensors PASSED

Jenkins build is back to normal : kafka-trunk-jdk7 #1433

[Apache Jenkins Server]

See 

[jira] [Commented] (KAFKA-3983) It would be helpful if SocketServer's Acceptors logged both the SocketChannel port and the processor ID upon registra

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15388519#comment-15388519
 ] 

ASF GitHub Bot commented on KAFKA-3983:
---

GitHub user rnpridgeon opened a pull request:

https://github.com/apache/kafka/pull/1648

KAFKA-3983 - Add additional information to debug

Add additional information to Acceptor debug message upon connection 
acceptance 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rnpridgeon/kafka trunk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1648.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1648


commit 59077eff1e26eb6bae590188d9d1d6fa1166cfe9
Author: rnpridgeon 
Date:   2016-07-21T22:09:47Z

KAFKA-3983 - Add additional information to debug logging to aid in 
debugging efforts




> It would be helpful if SocketServer's Acceptors logged both the SocketChannel 
> port and the processor ID upon registra
> -
>
> Key: KAFKA-3983
> URL: https://issues.apache.org/jira/browse/KAFKA-3983
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: Ryan P
>Assignee: Ryan P
>Priority: Minor
> Fix For: 0.10.0.1
>
>
> Currently Acceptors log the following message prior to registering passing 
> the accepted channel to a processor. 
> "Accepted connection from %s on %s [%d] sendBufferSize [actual|requested]: 
> [%d|%d] recvBufferSize [actual|requested]: [%d|%d]"
> It would be helpful to include the port number and the processor ID in this 
> message to aid in debugging efforts. Making it easier to track the amount of 
> time between acceptance and processing (connection configuration) 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1648: KAFKA-3983 - Add additional information to debug

[rnpridgeon]

GitHub user rnpridgeon opened a pull request:

https://github.com/apache/kafka/pull/1648

KAFKA-3983 - Add additional information to debug

Add additional information to Acceptor debug message upon connection 
acceptance 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/rnpridgeon/kafka trunk

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1648.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1648


commit 59077eff1e26eb6bae590188d9d1d6fa1166cfe9
Author: rnpridgeon 
Date:   2016-07-21T22:09:47Z

KAFKA-3983 - Add additional information to debug logging to aid in 
debugging efforts




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Build failed in Jenkins: kafka-trunk-jdk8 #766

[Apache Jenkins Server]

See 

Changes:

[wangguoz] KAFKA-3858: Add functions to print stream topologies

--
[...truncated 4641 lines...]

kafka.server.LogRecoveryTest > testHWCheckpointNoFailuresSingleLogSegment PASSED

kafka.server.LogRecoveryTest > testHWCheckpointWithFailuresSingleLogSegment 
STARTED

kafka.server.LogRecoveryTest > testHWCheckpointWithFailuresSingleLogSegment 
PASSED

kafka.server.AbstractFetcherThreadTest > testConsumerLagRemovedWithPartition 
STARTED

kafka.server.AbstractFetcherThreadTest > testConsumerLagRemovedWithPartition 
PASSED

kafka.server.AbstractFetcherThreadTest > testMetricsRemovedOnShutdown STARTED

kafka.server.AbstractFetcherThreadTest > testMetricsRemovedOnShutdown PASSED

kafka.server.ServerShutdownTest > testCleanShutdownAfterFailedStartup STARTED

kafka.server.ServerShutdownTest > testCleanShutdownAfterFailedStartup PASSED

kafka.server.ServerShutdownTest > testConsecutiveShutdown STARTED

kafka.server.ServerShutdownTest > testConsecutiveShutdown PASSED

kafka.server.ServerShutdownTest > testCleanShutdown STARTED

kafka.server.ServerShutdownTest > testCleanShutdown PASSED

kafka.server.ServerShutdownTest > testCleanShutdownWithDeleteTopicEnabled 
STARTED

kafka.server.ServerShutdownTest > testCleanShutdownWithDeleteTopicEnabled PASSED

kafka.server.ServerStartupTest > testBrokerStateRunningAfterZK STARTED

kafka.server.ServerStartupTest > testBrokerStateRunningAfterZK PASSED

kafka.server.ServerStartupTest > testBrokerCreatesZKChroot STARTED

kafka.server.ServerStartupTest > testBrokerCreatesZKChroot PASSED

kafka.server.ServerStartupTest > testConflictBrokerRegistration STARTED

kafka.server.ServerStartupTest > testConflictBrokerRegistration PASSED

kafka.server.ServerStartupTest > testBrokerSelfAware STARTED

kafka.server.ServerStartupTest > testBrokerSelfAware PASSED

kafka.server.EdgeCaseRequestTest > testInvalidApiVersionRequest STARTED

kafka.server.EdgeCaseRequestTest > testInvalidApiVersionRequest PASSED

kafka.server.EdgeCaseRequestTest > testMalformedHeaderRequest STARTED

kafka.server.EdgeCaseRequestTest > testMalformedHeaderRequest PASSED

kafka.server.EdgeCaseRequestTest > testProduceRequestWithNullClientId STARTED

kafka.server.EdgeCaseRequestTest > testProduceRequestWithNullClientId PASSED

kafka.server.EdgeCaseRequestTest > testInvalidApiKeyRequest STARTED

kafka.server.EdgeCaseRequestTest > testInvalidApiKeyRequest PASSED

kafka.server.EdgeCaseRequestTest > testHeaderOnlyRequest STARTED

kafka.server.EdgeCaseRequestTest > testHeaderOnlyRequest PASSED

kafka.server.CreateTopicsRequestTest > testValidCreateTopicsRequests STARTED

kafka.server.CreateTopicsRequestTest > testValidCreateTopicsRequests PASSED

kafka.server.CreateTopicsRequestTest > testErrorCreateTopicsRequests STARTED

kafka.server.CreateTopicsRequestTest > testErrorCreateTopicsRequests PASSED

kafka.server.CreateTopicsRequestTest > testInvalidCreateTopicsRequests STARTED

kafka.server.CreateTopicsRequestTest > testInvalidCreateTopicsRequests PASSED

kafka.server.CreateTopicsRequestTest > testNotController STARTED

kafka.server.CreateTopicsRequestTest > testNotController PASSED

kafka.server.OffsetCommitTest > testUpdateOffsets STARTED

kafka.server.OffsetCommitTest > testUpdateOffsets PASSED

kafka.server.OffsetCommitTest > testLargeMetadataPayload STARTED

kafka.server.OffsetCommitTest > testLargeMetadataPayload PASSED

kafka.server.OffsetCommitTest > testCommitAndFetchOffsets STARTED

kafka.server.OffsetCommitTest > testCommitAndFetchOffsets PASSED

kafka.server.OffsetCommitTest > testOffsetExpiration STARTED

kafka.server.OffsetCommitTest > testOffsetExpiration PASSED

kafka.server.OffsetCommitTest > testNonExistingTopicOffsetCommit STARTED

kafka.server.OffsetCommitTest > testNonExistingTopicOffsetCommit PASSED

kafka.server.ReplicaManagerTest > testHighWaterMarkDirectoryMapping STARTED

kafka.server.ReplicaManagerTest > testHighWaterMarkDirectoryMapping PASSED

kafka.server.ReplicaManagerTest > 
testFetchBeyondHighWatermarkReturnEmptyResponse STARTED

kafka.server.ReplicaManagerTest > 
testFetchBeyondHighWatermarkReturnEmptyResponse PASSED

kafka.server.ReplicaManagerTest > testIllegalRequiredAcks STARTED

kafka.server.ReplicaManagerTest > testIllegalRequiredAcks PASSED

kafka.server.ReplicaManagerTest > testClearPurgatoryOnBecomingFollower STARTED

kafka.server.ReplicaManagerTest > testClearPurgatoryOnBecomingFollower PASSED

kafka.server.ReplicaManagerTest > testHighwaterMarkRelativeDirectoryMapping 
STARTED

kafka.server.ReplicaManagerTest > testHighwaterMarkRelativeDirectoryMapping 
PASSED

kafka.server.IsrExpirationTest > testIsrExpirationForSlowFollowers STARTED

kafka.server.IsrExpirationTest > testIsrExpirationForSlowFollowers PASSED

kafka.server.IsrExpirationTest > testIsrExpirationForStuckFollowers STARTED

kafka.server.IsrExpirationTest > testIsrExpirationForStuckFollowers PASSED

[jira] [Updated] (KAFKA-3983) It would be helpful if SocketServer's Acceptors logged both the SocketChannel port and the processor ID upon registra

[Ryan P (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ryan P updated KAFKA-3983:
--
Description: 
Currently Acceptors log the following message prior to registering passing the 
accepted channel to a processor. 

"Accepted connection from %s on %s [%d] sendBufferSize [actual|requested]: 
[%d|%d] recvBufferSize [actual|requested]: [%d|%d]"

It would be helpful to include the port number and the processor ID in this 
message to aid in debugging efforts. Making it easier to track the amount of 
time between acceptance and processing (connection configuration) 



  was:
Currently Acceptors log the following message prior to registering passing the 
accepted channel to a processor. 

"Accepted connection from %s on %s, registered wtih [%d] sendBufferSize 
[actual|requested]: [%d|%d] recvBufferSize [actual|requested]: [%d|%d]"

It would be helpful to include the port number and the processor ID in this 
message to aid in debugging efforts. Making it easier to track the amount of 
time between acceptance and processing (connection configuration) 




> It would be helpful if SocketServer's Acceptors logged both the SocketChannel 
> port and the processor ID upon registra
> -
>
> Key: KAFKA-3983
> URL: https://issues.apache.org/jira/browse/KAFKA-3983
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: Ryan P
>Assignee: Ryan P
>Priority: Minor
> Fix For: 0.10.0.1
>
>
> Currently Acceptors log the following message prior to registering passing 
> the accepted channel to a processor. 
> "Accepted connection from %s on %s [%d] sendBufferSize [actual|requested]: 
> [%d|%d] recvBufferSize [actual|requested]: [%d|%d]"
> It would be helpful to include the port number and the processor ID in this 
> message to aid in debugging efforts. Making it easier to track the amount of 
> time between acceptance and processing (connection configuration) 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3983) It would be helpful if SocketServer's Acceptors logged both the SocketChannel port and the processor ID upon registra

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3983?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3983:
---
Fix Version/s: 0.10.0.1

> It would be helpful if SocketServer's Acceptors logged both the SocketChannel 
> port and the processor ID upon registra
> -
>
> Key: KAFKA-3983
> URL: https://issues.apache.org/jira/browse/KAFKA-3983
> Project: Kafka
>  Issue Type: Bug
>  Components: core
>Reporter: Ryan P
>Assignee: Ryan P
>Priority: Minor
> Fix For: 0.10.0.1
>
>
> Currently Acceptors log the following message prior to registering passing 
> the accepted channel to a processor. 
> "Accepted connection from %s on %s, registered wtih [%d] sendBufferSize 
> [actual|requested]: [%d|%d] recvBufferSize [actual|requested]: [%d|%d]"
> It would be helpful to include the port number and the processor ID in this 
> message to aid in debugging efforts. Making it easier to track the amount of 
> time between acceptance and processing (connection configuration) 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3983) It would be helpful if SocketServer's Acceptors logged both the SocketChannel port and the processor ID upon registra

[Ryan P (JIRA)]

Ryan P created KAFKA-3983:
-

 Summary: It would be helpful if SocketServer's Acceptors logged 
both the SocketChannel port and the processor ID upon registra
 Key: KAFKA-3983
 URL: https://issues.apache.org/jira/browse/KAFKA-3983
 Project: Kafka
  Issue Type: Bug
  Components: core
Reporter: Ryan P
Assignee: Ryan P
Priority: Minor


Currently Acceptors log the following message prior to registering passing the 
accepted channel to a processor. 

"Accepted connection from %s on %s, registered wtih [%d] sendBufferSize 
[actual|requested]: [%d|%d] recvBufferSize [actual|requested]: [%d|%d]"

It would be helpful to include the port number and the processor ID in this 
message to aid in debugging efforts. Making it easier to track the amount of 
time between acceptance and processing (connection configuration) 





--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3911) Enforce KTable materialization

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15388493#comment-15388493
 ] 

ASF GitHub Bot commented on KAFKA-3911:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1638


> Enforce KTable materialization
> --
>
> Key: KAFKA-3911
> URL: https://issues.apache.org/jira/browse/KAFKA-3911
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Eno Thereska
>Assignee: Eno Thereska
> Fix For: 0.10.1.0
>
>
> Always enforce KTable materialization upon creation (i.e. even for 
> "builder.table(..).to()" we also materialize it into a state store; this will 
> indeed incur unnecessary overhead but should be very rare, and it helps for 
> consistency of materialization).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1638: KAFKA-3911: KTable source materialization

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1638


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (KAFKA-3911) Enforce KTable materialization

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3911?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-3911.
--
Resolution: Fixed

Issue resolved by pull request 1638
[https://github.com/apache/kafka/pull/1638]

> Enforce KTable materialization
> --
>
> Key: KAFKA-3911
> URL: https://issues.apache.org/jira/browse/KAFKA-3911
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Eno Thereska
>Assignee: Eno Thereska
> Fix For: 0.10.1.0
>
>
> Always enforce KTable materialization upon creation (i.e. even for 
> "builder.table(..).to()" we also materialize it into a state store; this will 
> indeed incur unnecessary overhead but should be very rare, and it helps for 
> consistency of materialization).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-3980) JmxReporter uses excessive memory causing OutOfMemoryException

[Andrew Jorgensen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15388382#comment-15388382
 ] 

Andrew Jorgensen edited comment on KAFKA-3980 at 7/21/16 9:37 PM:
--

I was able to extract a list of all the ids from the map in the JmxReporter and 
have uploaded them here 
(https://drive.google.com/file/d/0B_65les2Npo5OHFyMVpXSjd1cXc/view?usp=sharing).
 The compressed size is 29M and uncompressed it is 224M. In the list you can 
see that the type=Fetch keys have clearly defined names but all the 
type=Produce seem to have completely randomized ids. I think that in general 
this makes sense, we dont really need a id on the producer side but the 
JmxReporter should not grow unbounded.

EDIT: Another data point is that I believe we are using the   0.8.1.1 client to 
talk to a 0.9.0.1 cluster. Not sure if the version mismatch there is 
contributing. It looks like the default client-id in the 0.8.1.1 client was an 
empty string where-as the new one uses an AtomicInteger.




was (Author: ajorgensen):
I was able to extract a list of all the ids from the map in the JmxReporter and 
have uploaded them here 
(https://drive.google.com/file/d/0B_65les2Npo5OHFyMVpXSjd1cXc/view?usp=sharing).
 The compressed size is 29M and uncompressed it is 224M. In the list you can 
see that the type=Fetch keys have clearly defined names but all the 
type=Produce seem to have completely randomized ids. I think that in general 
this makes sense, we dont really need a id on the producer side but the 
JmxReporter should not grow unbounded.

> JmxReporter uses excessive memory causing OutOfMemoryException
> --
>
> Key: KAFKA-3980
> URL: https://issues.apache.org/jira/browse/KAFKA-3980
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.1
>Reporter: Andrew Jorgensen
>
> I have some nodes in a kafka cluster that occasionally will run out of memory 
> whenever I restart the producers. I was able to take a heap dump from both a 
> recently restarted Kafka node which weighed in at about 20 MB and a node that 
> has been running for 2 months is using over 700MB of memory. Looking at the 
> heap dump it looks like the JmxReporter is holding on to metrics and causing 
> them to build up over time. 
> !http://imgur.com/N6Cd0Ku.png!
> !http://imgur.com/kQBqA2j.png!
> The ultimate problem this causes is that there is a chance when I restart the 
> producers it will cause the node to experience an Java heap space exception 
> and OOM. The nodes  then fail to startup correctly and write a -1 as the 
> leader number to the partitions they were responsible for effectively 
> resetting the offset and rendering that partition unavailable. The kafka 
> process then needs to go be restarted in order to re-assign the node to the 
> partition that it owns.
> I have a few questions:
> 1. I am not quite sure why there are so many client id entries in that 
> JmxReporter map.
> 2. Is there a way to have the JmxReporter release metrics after a set amount 
> of time or a way to turn certain high cardinality metrics like these off?
> I can provide any logs or heap dumps if more information is needed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3980) JmxReporter uses excessive memory causing OutOfMemoryException

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15388471#comment-15388471
 ] 

Ismael Juma commented on KAFKA-3980:


[~omkreddy], we do expire client quota metrics actually. We don't expire 
anything else though.

> JmxReporter uses excessive memory causing OutOfMemoryException
> --
>
> Key: KAFKA-3980
> URL: https://issues.apache.org/jira/browse/KAFKA-3980
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.1
>Reporter: Andrew Jorgensen
>
> I have some nodes in a kafka cluster that occasionally will run out of memory 
> whenever I restart the producers. I was able to take a heap dump from both a 
> recently restarted Kafka node which weighed in at about 20 MB and a node that 
> has been running for 2 months is using over 700MB of memory. Looking at the 
> heap dump it looks like the JmxReporter is holding on to metrics and causing 
> them to build up over time. 
> !http://imgur.com/N6Cd0Ku.png!
> !http://imgur.com/kQBqA2j.png!
> The ultimate problem this causes is that there is a chance when I restart the 
> producers it will cause the node to experience an Java heap space exception 
> and OOM. The nodes  then fail to startup correctly and write a -1 as the 
> leader number to the partitions they were responsible for effectively 
> resetting the offset and rendering that partition unavailable. The kafka 
> process then needs to go be restarted in order to re-assign the node to the 
> partition that it owns.
> I have a few questions:
> 1. I am not quite sure why there are so many client id entries in that 
> JmxReporter map.
> 2. Is there a way to have the JmxReporter release metrics after a set amount 
> of time or a way to turn certain high cardinality metrics like these off?
> I can provide any logs or heap dumps if more information is needed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-3929) Add prefix for underlying clients configs in StreamConfig

[Damian Guy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Damian Guy reassigned KAFKA-3929:
-

Assignee: Damian Guy  (was: Ishita Mandhan)

> Add prefix for underlying clients configs in StreamConfig
> -
>
> Key: KAFKA-3929
> URL: https://issues.apache.org/jira/browse/KAFKA-3929
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Damian Guy
>  Labels: api
>
> There are a couple of configs that have the same name for producer / consumer 
> configs, e.g. take a look at {{CommonClientConfigs}}, and also for producer / 
> consumer interceptors there are commonly named configs as well.
> This is semi-related to KAFKA-3740 since we need to add "sub-class" configs 
> for RocksDB as well, and we'd better have some prefix mechanism for such 
> hierarchical configs in general.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1647: MINOR: Upgrade RocksDB to 4.8.0

[ijuma]

GitHub user ijuma opened a pull request:

https://github.com/apache/kafka/pull/1647

MINOR: Upgrade RocksDB to 4.8.0



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/ijuma/kafka kafka-rocksdb-4.8

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1647.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1647


commit 1e3fe84ba484b583dc474dfc6a9078115312dd83
Author: Ismael Juma 
Date:   2016-07-21T21:16:58Z

Upgrade RocksDB to 4.8.0




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3929) Add prefix for underlying clients configs in StreamConfig

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15388440#comment-15388440
 ] 

Guozhang Wang commented on KAFKA-3929:
--

Thanks! I will let Damian to pick it up then.

> Add prefix for underlying clients configs in StreamConfig
> -
>
> Key: KAFKA-3929
> URL: https://issues.apache.org/jira/browse/KAFKA-3929
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Ishita Mandhan
>  Labels: api
>
> There are a couple of configs that have the same name for producer / consumer 
> configs, e.g. take a look at {{CommonClientConfigs}}, and also for producer / 
> consumer interceptors there are commonly named configs as well.
> This is semi-related to KAFKA-3740 since we need to add "sub-class" configs 
> for RocksDB as well, and we'd better have some prefix mechanism for such 
> hierarchical configs in general.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-2561) Optionally support OpenSSL for SSL/TLS

[Hendrik Saly (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-2561?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15388432#comment-15388432
 ] 

Hendrik Saly commented on KAFKA-2561:
-

Here is a working draft: 
https://github.com/salyh/kafka/commit/9337c56df9b8387bf42f756faf5be08118259139

First sketch to make SslFactory ready for native OpenSSl support leveraging 
netty and netty tcnative.
Requires netty 4.0.30 (common, handler, buffer, codec) and tcnative 
fork-1.1.33.19 for the respective OS and of course OpenSSL installed (recent 
1.0.1 or better 1.0.2). Could not get the gradle dependency stuff to work so 
maybe one can add the required dependencies. 



> Optionally support OpenSSL for SSL/TLS 
> ---
>
> Key: KAFKA-2561
> URL: https://issues.apache.org/jira/browse/KAFKA-2561
> Project: Kafka
>  Issue Type: New Feature
>  Components: security
>Affects Versions: 0.9.0.0
>Reporter: Ismael Juma
>
> JDK's `SSLEngine` is unfortunately a bit slow (KAFKA-2431 covers this in more 
> detail). We should consider supporting OpenSSL for SSL/TLS. Initial 
> experiments on my laptop show that it performs a lot better:
> {code}
> start.time, end.time, data.consumed.in.MB, MB.sec, data.consumed.in.nMsg, 
> nMsg.sec, config
> 2015-09-21 14:41:58:245, 2015-09-21 14:47:02:583, 28610.2295, 94.0081, 
> 3000, 98574.6111, Java 8u60/server auth JDK 
> SSLEngine/TLS_ECDHE_RSA_WITH_AES_128_CBC_SHA
> 2015-09-21 14:38:24:526, 2015-09-21 14:40:19:941, 28610.2295, 247.8900, 
> 3000, 259931.5514, Java 8u60/server auth 
> OpenSslEngine/TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256
> 2015-09-21 14:49:03:062, 2015-09-21 14:50:27:764, 28610.2295, 337.7751, 
> 3000, 354182.9000, Java 8u60/plaintext
> {code}
> Extracting the throughput figures:
> * JDK SSLEngine: 94 MB/s
> * OpenSSL SSLEngine: 247 MB/s
> * Plaintext: 337 MB/s (code from trunk, so no zero-copy due to KAFKA-2517)
> In order to get these figures, I used Netty's `OpenSslEngine` by hacking 
> `SSLFactory` to use Netty's `SslContextBuilder` and made a few changes to 
> `SSLTransportLayer` in order to workaround differences in behaviour between 
> `OpenSslEngine` and JDK's SSLEngine (filed 
> https://github.com/netty/netty/issues/4235 and 
> https://github.com/netty/netty/issues/4238 upstream).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk7 #1431

[Apache Jenkins Server]

See 

Changes:

[wangguoz] KAFKA-3740: Enable configuration of RocksDBStores

--
[...truncated 4426 lines...]

kafka.utils.ReplicationUtilsTest > testGetLeaderIsrAndEpochForPartition PASSED

kafka.utils.timer.TimerTest > testAlreadyExpiredTask STARTED

kafka.utils.timer.TimerTest > testAlreadyExpiredTask PASSED

kafka.utils.timer.TimerTest > testTaskExpiration STARTED

kafka.utils.timer.TimerTest > testTaskExpiration PASSED

kafka.utils.timer.TimerTaskListTest > testAll STARTED

kafka.utils.timer.TimerTaskListTest > testAll PASSED

kafka.utils.SchedulerTest > testMockSchedulerNonPeriodicTask STARTED

kafka.utils.SchedulerTest > testMockSchedulerNonPeriodicTask PASSED

kafka.utils.SchedulerTest > testMockSchedulerPeriodicTask STARTED

kafka.utils.SchedulerTest > testMockSchedulerPeriodicTask PASSED

kafka.utils.SchedulerTest > testNonPeriodicTask STARTED

kafka.utils.SchedulerTest > testNonPeriodicTask PASSED

kafka.utils.SchedulerTest > testRestart STARTED

kafka.utils.SchedulerTest > testRestart PASSED

kafka.utils.SchedulerTest > testReentrantTaskInMockScheduler STARTED

kafka.utils.SchedulerTest > testReentrantTaskInMockScheduler PASSED

kafka.utils.SchedulerTest > testPeriodicTask STARTED

kafka.utils.SchedulerTest > testPeriodicTask PASSED

kafka.utils.CommandLineUtilsTest > testParseEmptyArg STARTED

kafka.utils.CommandLineUtilsTest > testParseEmptyArg PASSED

kafka.utils.CommandLineUtilsTest > testParseSingleArg STARTED

kafka.utils.CommandLineUtilsTest > testParseSingleArg PASSED

kafka.utils.CommandLineUtilsTest > testParseArgs STARTED

kafka.utils.CommandLineUtilsTest > testParseArgs PASSED

kafka.utils.CommandLineUtilsTest > testParseEmptyArgAsValid STARTED

kafka.utils.CommandLineUtilsTest > testParseEmptyArgAsValid PASSED

kafka.utils.UtilsTest > testAbs STARTED

kafka.utils.UtilsTest > testAbs PASSED

kafka.utils.UtilsTest > testReplaceSuffix STARTED

kafka.utils.UtilsTest > testReplaceSuffix PASSED

kafka.utils.UtilsTest > testCircularIterator STARTED

kafka.utils.UtilsTest > testCircularIterator PASSED

kafka.utils.UtilsTest > testReadBytes STARTED

kafka.utils.UtilsTest > testReadBytes PASSED

kafka.utils.UtilsTest > testCsvList STARTED

kafka.utils.UtilsTest > testCsvList PASSED

kafka.utils.UtilsTest > testReadInt STARTED

kafka.utils.UtilsTest > testReadInt PASSED

kafka.utils.UtilsTest > testCsvMap STARTED

kafka.utils.UtilsTest > testCsvMap PASSED

kafka.utils.UtilsTest > testInLock STARTED

kafka.utils.UtilsTest > testInLock PASSED

kafka.utils.UtilsTest > testSwallow STARTED

kafka.utils.UtilsTest > testSwallow PASSED

kafka.utils.JsonTest > testJsonEncoding STARTED

kafka.utils.JsonTest > testJsonEncoding PASSED

kafka.utils.IteratorTemplateTest > testIterator STARTED

kafka.utils.IteratorTemplateTest > testIterator PASSED

kafka.metrics.KafkaTimerTest > testKafkaTimer STARTED

kafka.metrics.KafkaTimerTest > testKafkaTimer PASSED

kafka.metrics.MetricsTest > testMetricsReporterAfterDeletingTopic STARTED

kafka.metrics.MetricsTest > testMetricsReporterAfterDeletingTopic PASSED

kafka.metrics.MetricsTest > 
testBrokerTopicMetricsUnregisteredAfterDeletingTopic STARTED

kafka.metrics.MetricsTest > 
testBrokerTopicMetricsUnregisteredAfterDeletingTopic PASSED

kafka.metrics.MetricsTest > testMetricsLeak STARTED

kafka.metrics.MetricsTest > testMetricsLeak PASSED

kafka.consumer.TopicFilterTest > testWhitelists STARTED

kafka.consumer.TopicFilterTest > testWhitelists PASSED

kafka.consumer.TopicFilterTest > 
testWildcardTopicCountGetTopicCountMapEscapeJson STARTED

kafka.consumer.TopicFilterTest > 
testWildcardTopicCountGetTopicCountMapEscapeJson PASSED

kafka.consumer.TopicFilterTest > testBlacklists STARTED

kafka.consumer.TopicFilterTest > testBlacklists PASSED

kafka.consumer.PartitionAssignorTest > testRoundRobinPartitionAssignor STARTED

kafka.consumer.PartitionAssignorTest > testRoundRobinPartitionAssignor PASSED

kafka.consumer.PartitionAssignorTest > testRangePartitionAssignor STARTED

kafka.consumer.PartitionAssignorTest > testRangePartitionAssignor PASSED

kafka.consumer.ZookeeperConsumerConnectorTest > testBasic STARTED

kafka.consumer.ZookeeperConsumerConnectorTest > testBasic PASSED

kafka.consumer.ZookeeperConsumerConnectorTest > testCompressionSetConsumption 
STARTED

kafka.consumer.ZookeeperConsumerConnectorTest > testCompressionSetConsumption 
PASSED

kafka.consumer.ZookeeperConsumerConnectorTest > testLeaderSelectionForPartition 
STARTED

kafka.consumer.ZookeeperConsumerConnectorTest > testLeaderSelectionForPartition 
PASSED

kafka.consumer.ZookeeperConsumerConnectorTest > testConsumerDecoder STARTED

kafka.consumer.ZookeeperConsumerConnectorTest > testConsumerDecoder PASSED

kafka.consumer.ZookeeperConsumerConnectorTest > testConsumerRebalanceListener 
STARTED

kafka.consumer.ZookeeperConsumerConnectorTest > testConsumerRebalanceListener 
PASSED

[jira] [Commented] (KAFKA-1044) change log4j to slf4j

[Shannon Carey (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1044?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15388393#comment-15388393
 ] 

Shannon Carey commented on KAFKA-1044:
--

In addition, can we change the dependency on slf4j-log4j12 from "compile" scope 
to "runtime" scope and mark it "optional"?

> change log4j to slf4j 
> --
>
> Key: KAFKA-1044
> URL: https://issues.apache.org/jira/browse/KAFKA-1044
> Project: Kafka
>  Issue Type: Bug
>  Components: log
>Affects Versions: 0.8.0
>Reporter: shijinkui
> Fix For: 0.10.1.0
>
>
> can u chanage the log4j to slf4j, in my project, i use logback, it's conflict 
> with log4j.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3980) JmxReporter uses excessive memory causing OutOfMemoryException

[Andrew Jorgensen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15388382#comment-15388382
 ] 

Andrew Jorgensen commented on KAFKA-3980:
-

I was able to extract a list of all the ids from the map in the JmxReporter and 
have uploaded them here 
(https://drive.google.com/file/d/0B_65les2Npo5OHFyMVpXSjd1cXc/view?usp=sharing).
 The compressed size is 29M and uncompressed it is 224M. In the list you can 
see that the type=Fetch keys have clearly defined names but all the 
type=Produce seem to have completely randomized ids. I think that in general 
this makes sense, we dont really need a id on the producer side but the 
JmxReporter should not grow unbounded.

> JmxReporter uses excessive memory causing OutOfMemoryException
> --
>
> Key: KAFKA-3980
> URL: https://issues.apache.org/jira/browse/KAFKA-3980
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.1
>Reporter: Andrew Jorgensen
>
> I have some nodes in a kafka cluster that occasionally will run out of memory 
> whenever I restart the producers. I was able to take a heap dump from both a 
> recently restarted Kafka node which weighed in at about 20 MB and a node that 
> has been running for 2 months is using over 700MB of memory. Looking at the 
> heap dump it looks like the JmxReporter is holding on to metrics and causing 
> them to build up over time. 
> !http://imgur.com/N6Cd0Ku.png!
> !http://imgur.com/kQBqA2j.png!
> The ultimate problem this causes is that there is a chance when I restart the 
> producers it will cause the node to experience an Java heap space exception 
> and OOM. The nodes  then fail to startup correctly and write a -1 as the 
> leader number to the partitions they were responsible for effectively 
> resetting the offset and rendering that partition unavailable. The kafka 
> process then needs to go be restarted in order to re-assign the node to the 
> partition that it owns.
> I have a few questions:
> 1. I am not quite sure why there are so many client id entries in that 
> JmxReporter map.
> 2. Is there a way to have the JmxReporter release metrics after a set amount 
> of time or a way to turn certain high cardinality metrics like these off?
> I can provide any logs or heap dumps if more information is needed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3982) Issue with processing order of consumer properties in console consumer

[Vahid Hashemian (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vahid Hashemian updated KAFKA-3982:
---
Description: 
With the recent introduction of {{consumer.property}} argument in console 
consumer, both new and old consumer could overwrite certain properties provided 
using this new argument.

Specifically, the old consumer would overwrite the values provided for 
[{{auto.offset.reset}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L173]
 and 
[{{zookeeper.connect}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L174],
 and the new consumer would overwrite the values provided for 
[{{auto.offset.reset}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L196],
 
[{{bootstrap.servers}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L197],
 
[{{key.deserializer}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L198],
 and 
[{{key.deserializer}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L199].

For example, running the old consumer as {{bin/kafka-console-consumer.sh 
--zookeeper localhost:2181 --topic foo --consumer-property 
auto.offset.reset=none}} the value that's eventually selected for 
{{auto.offset.reset}} will be {{largest}}, overwriting what the user provides 
in the command line.

This seems to be because the properties provided via {{consumer.property}} 
argument are not considered when finalizing the configuration of the consumer.

Some properties can now be provided in three different places (directly in the 
command line, via the {{consumer.property}} argument, and via the 
{{consumer.config}} argument, in the same order of precedence).

  was:
With the recent introduction of {{consumer.property}} argument in console 
consumer, both new and old consumer could overwrite certain properties provided 
using this new argument.

Specifically, the old consumer would overwrite the values provided for 
[{{auto.offset.reset}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L173]
 and 
[{{zookeeper.connect}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L174],
 and the new consumer would overwrite the values provided for 
[{{auto.offset.reset}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L196],
 
[{{bootstrap.servers}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L197],
 
[{{key.deserializer}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L198],
 and 
[{{key.deserializer}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L199].

For example, running the old consumer as {{bin/kafka-console-consumer.sh 
--zookeeper localhost:2181 --topic foo --consumer-property 
auto.offset.reset=none}} the value that's eventually selected for 
{{auto.offset.reset}} will be {{largest}}, overwriting what the user provides 
in the command line.

This seems to be because the properties provided via {{consumer.property}} 
argument are not considered when finalizing the configuration of the consumer.

Since these properties can now be provided in three different places (directly 
in the command line, via the {{consumer.property}} argument, and via the 
{{consumer.config}} argument) it would help to clarify the precedence order 
among the three.


> Issue with processing order of consumer properties in console consumer
> --
>
> Key: KAFKA-3982
> URL: https://issues.apache.org/jira/browse/KAFKA-3982
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Reporter: Vahid Hashemian
>Assignee: Vahid Hashemian
>Priority: Minor
>
> With the recent introduction of {{consumer.property}} argument in console 
> consumer, both new and old consumer could overwrite certain properties 
> provided using this new argument.
> Specifically, the old consumer would overwrite the values provided for 
> 

[jira] [Updated] (KAFKA-3982) Issue with processing order of consumer properties in console consumer

[Vahid Hashemian (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vahid Hashemian updated KAFKA-3982:
---
Description: 
With the recent introduction of {{consumer.property}} argument in console 
consumer, both new and old consumer could overwrite certain properties provided 
using this new argument.

Specifically, the old consumer would overwrite the values provided for 
[{{auto.offset.reset}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L173]
 and 
[{{zookeeper.connect}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L174],
 and the new consumer would overwrite the values provided for 
[{{auto.offset.reset}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L196],
 
[{{bootstrap.servers}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L197],
 
[{{key.deserializer}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L198],
 and 
[{{key.deserializer}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L199].

For example, running the old consumer as {{bin/kafka-console-consumer.sh 
--zookeeper localhost:2181 --topic foo --consumer-property 
auto.offset.reset=none}} the value that's eventually selected for 
{{auto.offset.reset}} will be {{largest}}, overwriting what the user provides 
in the command line.

This seems to be because the properties provided via {{consumer.property}} 
argument are not considered when finalizing the configuration of the consumer.

Since these properties can now be provided in three different places (directly 
in the command line, via the {{consumer.property}} argument, and via the 
{{consumer.config}} argument) it would help to clarify the precedence order 
among the three.

  was:
With the recent introduction of {{consumer.property}} argument in console 
consumer, both new and old consumer could overwrite certain properties provided 
using this new argument.

Specifically, the old consumer would overwrite the values provided for 
[{{auto.offset.reset}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L173]
 and 
[{{zookeeper.connect}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L174],
 and the new consumer would overwrite the values provided for 
[{{auto.offset.reset}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L196],
 
[{{key.deserializer}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L198],
 and 
[{{key.deserializer}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L199].

For example, running the old consumer as {{bin/kafka-console-consumer.sh 
--zookeeper localhost:2181 --topic foo --consumer-property 
auto.offset.reset=none}} the value that's eventually selected for 
{{auto.offset.reset}} will be {{largest}}, overwriting what the user provides 
in the command line.

This seems to be because the properties provided via {{consumer.property}} 
argument are not considered when finalizing the configuration of the consumer.


> Issue with processing order of consumer properties in console consumer
> --
>
> Key: KAFKA-3982
> URL: https://issues.apache.org/jira/browse/KAFKA-3982
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Reporter: Vahid Hashemian
>Assignee: Vahid Hashemian
>Priority: Minor
>
> With the recent introduction of {{consumer.property}} argument in console 
> consumer, both new and old consumer could overwrite certain properties 
> provided using this new argument.
> Specifically, the old consumer would overwrite the values provided for 
> [{{auto.offset.reset}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L173]
>  and 
> [{{zookeeper.connect}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L174],
>  and the new consumer would overwrite the values provided for 
> 

[jira] [Commented] (KAFKA-3858) Add functions to print stream topologies

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3858?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15388346#comment-15388346
 ] 

ASF GitHub Bot commented on KAFKA-3858:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1619


> Add functions to print stream topologies
> 
>
> Key: KAFKA-3858
> URL: https://issues.apache.org/jira/browse/KAFKA-3858
> Project: Kafka
>  Issue Type: New Feature
>  Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Roger Hoover
>Assignee: Eno Thereska
> Fix For: 0.10.1.0
>
>
> For debugging and development, it would be very useful to be able to print 
> Kafka streams topologies.  At a minimum, it would be great to be able to see 
> the logical topology including with Kafka topics linked by sub-topologies.  I 
> think that this information does not depend on partitioning.  For more 
> detail, it would be great to be able to print the same logical topology but 
> also showing number of tasks (an perhaps task ids?).  Finally, it would be 
> great to show the physical topology after the tasks have been mapped to JVMs 
> + threads.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1619: KAFKA-3858: Add functions to print stream topologi...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1619


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Resolved] (KAFKA-3858) Add functions to print stream topologies

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3858?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-3858.
--
   Resolution: Fixed
Fix Version/s: 0.10.1.0

Issue resolved by pull request 1619
[https://github.com/apache/kafka/pull/1619]

> Add functions to print stream topologies
> 
>
> Key: KAFKA-3858
> URL: https://issues.apache.org/jira/browse/KAFKA-3858
> Project: Kafka
>  Issue Type: New Feature
>  Components: streams
>Affects Versions: 0.10.1.0
>Reporter: Roger Hoover
>Assignee: Eno Thereska
> Fix For: 0.10.1.0
>
>
> For debugging and development, it would be very useful to be able to print 
> Kafka streams topologies.  At a minimum, it would be great to be able to see 
> the logical topology including with Kafka topics linked by sub-topologies.  I 
> think that this information does not depend on partitioning.  For more 
> detail, it would be great to be able to print the same logical topology but 
> also showing number of tasks (an perhaps task ids?).  Finally, it would be 
> great to show the physical topology after the tasks have been mapped to JVMs 
> + threads.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk8 #764

[Apache Jenkins Server]

See 

Changes:

[wangguoz] KAFKA-3740: Enable configuration of RocksDBStores

--
[...truncated 1349 lines...]
kafka.server.AbstractFetcherThreadTest > testMetricsRemovedOnShutdown STARTED

kafka.server.AbstractFetcherThreadTest > testMetricsRemovedOnShutdown PASSED

kafka.server.ServerShutdownTest > testCleanShutdownAfterFailedStartup STARTED

kafka.server.ServerShutdownTest > testCleanShutdownAfterFailedStartup PASSED

kafka.server.ServerShutdownTest > testConsecutiveShutdown STARTED

kafka.server.ServerShutdownTest > testConsecutiveShutdown PASSED

kafka.server.ServerShutdownTest > testCleanShutdown STARTED

kafka.server.ServerShutdownTest > testCleanShutdown PASSED

kafka.server.ServerShutdownTest > testCleanShutdownWithDeleteTopicEnabled 
STARTED

kafka.server.ServerShutdownTest > testCleanShutdownWithDeleteTopicEnabled PASSED

kafka.server.ServerStartupTest > testBrokerStateRunningAfterZK STARTED

kafka.server.ServerStartupTest > testBrokerStateRunningAfterZK PASSED

kafka.server.ServerStartupTest > testBrokerCreatesZKChroot STARTED

kafka.server.ServerStartupTest > testBrokerCreatesZKChroot PASSED

kafka.server.ServerStartupTest > testConflictBrokerRegistration STARTED

kafka.server.ServerStartupTest > testConflictBrokerRegistration PASSED

kafka.server.ServerStartupTest > testBrokerSelfAware STARTED

kafka.server.ServerStartupTest > testBrokerSelfAware PASSED

kafka.server.EdgeCaseRequestTest > testInvalidApiVersionRequest STARTED

kafka.server.EdgeCaseRequestTest > testInvalidApiVersionRequest PASSED

kafka.server.EdgeCaseRequestTest > testMalformedHeaderRequest STARTED

kafka.server.EdgeCaseRequestTest > testMalformedHeaderRequest PASSED

kafka.server.EdgeCaseRequestTest > testProduceRequestWithNullClientId STARTED

kafka.server.EdgeCaseRequestTest > testProduceRequestWithNullClientId PASSED

kafka.server.EdgeCaseRequestTest > testInvalidApiKeyRequest STARTED

kafka.server.EdgeCaseRequestTest > testInvalidApiKeyRequest PASSED

kafka.server.EdgeCaseRequestTest > testHeaderOnlyRequest STARTED

kafka.server.EdgeCaseRequestTest > testHeaderOnlyRequest PASSED

kafka.server.CreateTopicsRequestTest > testValidCreateTopicsRequests STARTED

kafka.server.CreateTopicsRequestTest > testValidCreateTopicsRequests PASSED

kafka.server.CreateTopicsRequestTest > testErrorCreateTopicsRequests STARTED

kafka.server.CreateTopicsRequestTest > testErrorCreateTopicsRequests PASSED

kafka.server.CreateTopicsRequestTest > testInvalidCreateTopicsRequests STARTED

kafka.server.CreateTopicsRequestTest > testInvalidCreateTopicsRequests PASSED

kafka.server.CreateTopicsRequestTest > testNotController STARTED

kafka.server.CreateTopicsRequestTest > testNotController PASSED

kafka.server.OffsetCommitTest > testUpdateOffsets STARTED

kafka.server.OffsetCommitTest > testUpdateOffsets PASSED

kafka.server.OffsetCommitTest > testLargeMetadataPayload STARTED

kafka.server.OffsetCommitTest > testLargeMetadataPayload PASSED

kafka.server.OffsetCommitTest > testCommitAndFetchOffsets STARTED

kafka.server.OffsetCommitTest > testCommitAndFetchOffsets PASSED

kafka.server.OffsetCommitTest > testOffsetExpiration STARTED

kafka.server.OffsetCommitTest > testOffsetExpiration PASSED

kafka.server.OffsetCommitTest > testNonExistingTopicOffsetCommit STARTED

kafka.server.OffsetCommitTest > testNonExistingTopicOffsetCommit PASSED

kafka.server.ReplicaManagerTest > testHighWaterMarkDirectoryMapping STARTED

kafka.server.ReplicaManagerTest > testHighWaterMarkDirectoryMapping PASSED

kafka.server.ReplicaManagerTest > 
testFetchBeyondHighWatermarkReturnEmptyResponse STARTED

kafka.server.ReplicaManagerTest > 
testFetchBeyondHighWatermarkReturnEmptyResponse PASSED

kafka.server.ReplicaManagerTest > testIllegalRequiredAcks STARTED

kafka.server.ReplicaManagerTest > testIllegalRequiredAcks PASSED

kafka.server.ReplicaManagerTest > testClearPurgatoryOnBecomingFollower STARTED

kafka.server.ReplicaManagerTest > testClearPurgatoryOnBecomingFollower PASSED

kafka.server.ReplicaManagerTest > testHighwaterMarkRelativeDirectoryMapping 
STARTED

kafka.server.ReplicaManagerTest > testHighwaterMarkRelativeDirectoryMapping 
PASSED

kafka.server.IsrExpirationTest > testIsrExpirationForSlowFollowers STARTED

kafka.server.IsrExpirationTest > testIsrExpirationForSlowFollowers PASSED

kafka.server.IsrExpirationTest > testIsrExpirationForStuckFollowers STARTED

kafka.server.IsrExpirationTest > testIsrExpirationForStuckFollowers PASSED

kafka.server.IsrExpirationTest > testIsrExpirationIfNoFetchRequestMade STARTED

kafka.server.IsrExpirationTest > testIsrExpirationIfNoFetchRequestMade PASSED

kafka.server.ControlledShutdownLeaderSelectorTest > testSelectLeader STARTED

kafka.server.ControlledShutdownLeaderSelectorTest > testSelectLeader PASSED

kafka.server.ClientQuotaManagerTest > testQuotaViolation STARTED

kafka.server.ClientQuotaManagerTest > testQuotaViolation PASSED

[jira] [Updated] (KAFKA-3982) Issue with processing order of consumer properties in console consumer

[Vahid Hashemian (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3982?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Vahid Hashemian updated KAFKA-3982:
---
Description: 
With the recent introduction of {{consumer.property}} argument in console 
consumer, both new and old consumer could overwrite certain properties provided 
using this new argument.

Specifically, the old consumer would overwrite the values provided for 
[{{auto.offset.reset}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L173]
 and 
[{{zookeeper.connect}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L174],
 and the new consumer would overwrite the values provided for 
[{{auto.offset.reset}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L196],
 
[{{key.deserializer}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L198],
 and 
[{{key.deserializer}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L199].

For example, running the old consumer as {{bin/kafka-console-consumer.sh 
--zookeeper localhost:2181 --topic foo --consumer-property 
auto.offset.reset=none}} the value that's eventually selected for 
{{auto.offset.reset}} will be {{largest}}, overwriting what the user provides 
in the command line.

This seems to be because the properties provided via {{consumer.property}} 
argument are not considered when finalizing the configuration of the consumer.

  was:
With the recent introduction of {{consumer.property}} argument in console 
consumer, both new and old consumer could overwrite certain properties provided 
using this new argument.

Specifically, the old consumer would overwrite the values provided for 
[{{auto.offset.reset}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L173]
 and 
[{{zookeeper.connect}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L174],
 and the new consumer would overwrite the values provided for 
[{{auto.offset.reset}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L196],
 
[{{key.deserializer}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L198],
 and 
[{{key.deserializer}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L199].

This seems to be because the properties provided via {{consumer.property}} 
argument are not considered when finalizing the configuration of the consumer.


> Issue with processing order of consumer properties in console consumer
> --
>
> Key: KAFKA-3982
> URL: https://issues.apache.org/jira/browse/KAFKA-3982
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Reporter: Vahid Hashemian
>Assignee: Vahid Hashemian
>Priority: Minor
>
> With the recent introduction of {{consumer.property}} argument in console 
> consumer, both new and old consumer could overwrite certain properties 
> provided using this new argument.
> Specifically, the old consumer would overwrite the values provided for 
> [{{auto.offset.reset}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L173]
>  and 
> [{{zookeeper.connect}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L174],
>  and the new consumer would overwrite the values provided for 
> [{{auto.offset.reset}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L196],
>  
> [{{key.deserializer}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L198],
>  and 
> [{{key.deserializer}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L199].
> For example, running the old consumer as {{bin/kafka-console-consumer.sh 
> --zookeeper localhost:2181 --topic foo --consumer-property 
> auto.offset.reset=none}} the value that's eventually selected for 
> {{auto.offset.reset}} will be {{largest}}, overwriting 

[jira] [Commented] (KAFKA-3929) Add prefix for underlying clients configs in StreamConfig

[Ishita Mandhan (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15388216#comment-15388216
 ] 

Ishita Mandhan commented on KAFKA-3929:
---

I've started but haven't made too much progress into it so if the fix needs to 
be done asap, you can pick it up. 

> Add prefix for underlying clients configs in StreamConfig
> -
>
> Key: KAFKA-3929
> URL: https://issues.apache.org/jira/browse/KAFKA-3929
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Ishita Mandhan
>  Labels: api
>
> There are a couple of configs that have the same name for producer / consumer 
> configs, e.g. take a look at {{CommonClientConfigs}}, and also for producer / 
> consumer interceptors there are commonly named configs as well.
> This is semi-related to KAFKA-3740 since we need to add "sub-class" configs 
> for RocksDB as well, and we'd better have some prefix mechanism for such 
> hierarchical configs in general.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Proposal for an End-to-End encryption in Kafka

[Hendrik Dev]

Proposal for an End-to-End encryption in Kafka:

https://github.com/salyh/kafka/commit/151a0340645f4165cb120f79df544326e5a0db6f

Introduce EncryptingSerializer and DecryptingDeserializer.
They act as wrappers for inner (de)serializer and provide transparent
AES and RSA based end-to-end encryption.
So all messages stored in Kafka which passed the EncryptingSerializer
are stored in an encrypted manner.
The DecryptingDeserializer can detect if a message is encrypted and if
not its just passed through as it is.

Here is how it works:
https://github.com/salyh/kafka/blob/151a0340645f4165cb120f79df544326e5a0db6f/clients/src/main/java/org/apache/kafka/common/serialization/EncrpytingSerializer.java

Currently the code is not threadsafe and not optimized or benchmarked.
Its just a proof of concept to start a discussion if such a
contribution would be useful.
Depending on the RSA keysize a overhead of additional 41+(RSA keysize
in bytes) bytes per message are added and for each message produced
(or consumed) a AES en-/decryption needs to be performed. RSA
encryption of the AES key happens not very often cause this could be
cached - so RSA en-/decryption is likely not a big issue regarding
performance.

Note: AES encryption on modern hardware with a recent Java runtime is
typically hardware accelerated (by leveraging AES-NI and -XX:+UseAES
-XX:+UseAESIntrinsics)

Feedback and comments appreciated.

Thanks
Hendrik

-- 
Hendrik Saly (salyh, hendrikdev22)
@hendrikdev22
PGP: 0x22D7F6EC
salyh  apache  org

Re: [DISCUSS] KIP-4 ACL Admin Schema

[Jason Gustafson]

Hi Grant,

This looks good to me. One minor comment. You mention that "delete" actions
will get processed before "add" actions, which makes sense to me. An
alternative to avoid the confusion in the first place would be to replace
the AlterAcls APIs with separate AddAcls and DeleteAcls APIs. Was this
option already rejected?

Thanks,
Jason

On Thu, Jul 21, 2016 at 7:57 AM, Grant Henke  wrote:

> Anyone else have any feedback on this protocol and implementation? I plan
> to start a vote soon.
>
> Thank you,
> Grant
>
> On Fri, Jul 15, 2016 at 1:04 PM, Gwen Shapira  wrote:
>
> > > My goal in the protocol design was to keep the request simple and be
> able
> > > to answer what I think are the 3 most common questions/requests
> > >
> > >- What ACLs are on the cluster?
> > >- What access do I/they have?
> > >- Who has access to this resource?
> >
> > Thanks for clarifying. I think this is good. Perhaps just document
> > this goal next to the protocol for the record :)
> >
> > > Isn't KIP-50 itself one gigantic compatibility concern? I don't see
> > >> how your suggestions make it any worse...
> > >
> > >
> > >
> > >>  Yes, I also think we should take this chance to improve the
> Authorizer
> > interface
> > >> to make it more suitable for the ACL Admin requests.
> > >
> > >
> > > I agree we can address this in KIP-50. What I was getting at was that I
> > > wanted to handle that discussion there. We voted on KIP-50 before 0.10
> > was
> > > released with the intention that we could get it in. Now that 0.10 is
> > > released and a longer time has gone by I am not sure if the opinion of
> > > "breaking is okay" has changed. I will always prefer a backward
> > compatible
> > > approach if possible.
> >
> > Well, the entire KIP-50 discussion - both regarding compatibility and
> > possible increased scope is probably out of context here. Especially
> > since this proposal was written carefully to avoid any assumptions
> > regarding other work. I suggest taking this in a separate thread.
> >
> > Gwen
> >
> > > Thank you,
> > > Grant
> > >
> > >
> > > On Fri, Jul 15, 2016 at 7:22 AM, Ismael Juma 
> wrote:
> > >
> > >> On Fri, Jul 15, 2016 at 6:45 AM, Gwen Shapira 
> > wrote:
> > >> >
> > >> > >>  - I suggest this be addressed in KIP-50 as well, though
> it
> > >> has
> > >> > >>  some compatibility concerns.
> > >> >
> > >> > Isn't KIP-50 itself one gigantic compatibility concern? I don't see
> > >> > how your suggestions make it any worse...
> > >> >
> > >>
> > >> Yes, I also think we should take this chance to improve the Authorizer
> > >> interface to make it more suitable for the ACL Admin requests.
> > >>
> > >> Ismael
> > >>
> > >
> > >
> > >
> > > --
> > > Grant Henke
> > > Software Engineer | Cloudera
> > > gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
> >
>
>
>
> --
> Grant Henke
> Software Engineer | Cloudera
> gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
>

[jira] [Created] (KAFKA-3982) Issue with processing order of consumer properties in console consumer

[Vahid Hashemian (JIRA)]

Vahid Hashemian created KAFKA-3982:
--

 Summary: Issue with processing order of consumer properties in 
console consumer
 Key: KAFKA-3982
 URL: https://issues.apache.org/jira/browse/KAFKA-3982
 Project: Kafka
  Issue Type: Bug
  Components: consumer
Reporter: Vahid Hashemian
Assignee: Vahid Hashemian
Priority: Minor


With the recent introduction of {{consumer.property}} argument in console 
consumer, both new and old consumer could overwrite certain properties provided 
using this new argument.

Specifically, the old consumer would overwrite the values provided for 
[{{auto.offset.reset}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L173]
 and 
[{{zookeeper.connect}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L174],
 and the new consumer would overwrite the values provided for 
[{{auto.offset.reset}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L196],
 
[{{key.deserializer}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L198],
 and 
[{{key.deserializer}}|https://github.com/apache/kafka/blob/10bbffd75439e10fe9db6cf0aa48a7da7e386ef3/core/src/main/scala/kafka/tools/ConsoleConsumer.scala#L199].

This seems to be because the properties provided via {{consumer.property}} 
argument are not considered when finalizing the configuration of the consumer.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3929) Add prefix for underlying clients configs in StreamConfig

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3929?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15388137#comment-15388137
 ] 

Guozhang Wang commented on KAFKA-3929:
--

[~imandhan] Are you already working on this ticket? 

> Add prefix for underlying clients configs in StreamConfig
> -
>
> Key: KAFKA-3929
> URL: https://issues.apache.org/jira/browse/KAFKA-3929
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Ishita Mandhan
>  Labels: api
>
> There are a couple of configs that have the same name for producer / consumer 
> configs, e.g. take a look at {{CommonClientConfigs}}, and also for producer / 
> consumer interceptors there are commonly named configs as well.
> This is semi-related to KAFKA-3740 since we need to add "sub-class" configs 
> for RocksDB as well, and we'd better have some prefix mechanism for such 
> hierarchical configs in general.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3740) Enable configuration of RocksDBStores

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15388136#comment-15388136
 ] 

ASF GitHub Bot commented on KAFKA-3740:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1640


> Enable configuration of RocksDBStores
> -
>
> Key: KAFKA-3740
> URL: https://issues.apache.org/jira/browse/KAFKA-3740
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Damian Guy
>  Labels: api, newbie
> Fix For: 0.10.1.0
>
>
> Today most of the rocksDB configs are hard written inside {{RocksDBStore}}, 
> or the default values are directly used. We need to make them configurable 
> for advanced users. For example, some default values may not work perfectly 
> for some scenarios: 
> https://github.com/HenryCaiHaiying/kafka/commit/ccc4e25b110cd33eea47b40a2f6bf17ba0924576
>  
> One way of doing that is to introduce a "RocksDBStoreConfigs" objects similar 
> to "StreamsConfig", which defines all related rocksDB options configs, that 
> can be passed as key-value pairs to "StreamsConfig".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-3740) Enable configuration of RocksDBStores

[Guozhang Wang (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3740?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Guozhang Wang resolved KAFKA-3740.
--
   Resolution: Fixed
Fix Version/s: 0.10.1.0

Issue resolved by pull request 1640
[https://github.com/apache/kafka/pull/1640]

> Enable configuration of RocksDBStores
> -
>
> Key: KAFKA-3740
> URL: https://issues.apache.org/jira/browse/KAFKA-3740
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Damian Guy
>  Labels: api, newbie
> Fix For: 0.10.1.0
>
>
> Today most of the rocksDB configs are hard written inside {{RocksDBStore}}, 
> or the default values are directly used. We need to make them configurable 
> for advanced users. For example, some default values may not work perfectly 
> for some scenarios: 
> https://github.com/HenryCaiHaiying/kafka/commit/ccc4e25b110cd33eea47b40a2f6bf17ba0924576
>  
> One way of doing that is to introduce a "RocksDBStoreConfigs" objects similar 
> to "StreamsConfig", which defines all related rocksDB options configs, that 
> can be passed as key-value pairs to "StreamsConfig".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1640: KAFKA-3740: Enable configuration of RocksDBStores

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1640


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Reopened] (KAFKA-3887) StreamBounceTest.test_bounce and StreamSmokeTest.test_streams failing

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3887?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson reopened KAFKA-3887:


Still seeing this failure in nightly runs: 
http://testing.confluent.io/confluent-kafka-system-test-results/?prefix=2016-07-20--001.1469035275--apache--trunk--9cad9db/.

> StreamBounceTest.test_bounce and StreamSmokeTest.test_streams failing
> -
>
> Key: KAFKA-3887
> URL: https://issues.apache.org/jira/browse/KAFKA-3887
> Project: Kafka
>  Issue Type: Bug
>  Components: streams, system tests
>Reporter: Ismael Juma
>Assignee: Guozhang Wang
>  Labels: transient-system-test-failure
> Fix For: 0.10.0.1
>
>
> StreamBounceTest.test_bounce and StreamSmokeTest.test_streams has been 
> failing semi-regularly. Output from the latest failure:
> {code}
> 
> test_id:
> 2016-06-20--001.kafkatest.tests.streams.streams_bounce_test.StreamsBounceTest.test_bounce
> status: FAIL
> run time:   4 minutes 13.916 seconds
> Streams Smoke Test process on ubuntu@worker5 took too long to exit
> Traceback (most recent call last):
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
>  line 106, in run_all_tests
> data = self.run_single_test()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
>  line 162, in run_single_test
> return self.current_test_context.function(self.current_test)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/streams/streams_bounce_test.py",
>  line 67, in test_bounce
> self.driver.wait()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/services/streams.py",
>  line 94, in wait
> wait_until(lambda: not node.account.alive(pid), timeout_sec=120, 
> err_msg="Streams Smoke Test process on " + str(node.account) + " took too 
> long to exit")
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/utils/util.py",
>  line 36, in wait_until
> raise TimeoutError(err_msg)
> TimeoutError: Streams Smoke Test process on ubuntu@worker5 took too long to 
> exit
> 
> test_id:
> 2016-06-20--001.kafkatest.tests.streams.streams_smoke_test.StreamsSmokeTest.test_streams
> status: FAIL
> run time:   4 minutes 36.426 seconds
> Streams Smoke Test process on ubuntu@worker9 took too long to exit
> Traceback (most recent call last):
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
>  line 106, in run_all_tests
> data = self.run_single_test()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
>  line 162, in run_single_test
> return self.current_test_context.function(self.current_test)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/streams/streams_smoke_test.py",
>  line 67, in test_streams
> self.driver.wait()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/services/streams.py",
>  line 94, in wait
> wait_until(lambda: not node.account.alive(pid), timeout_sec=120, 
> err_msg="Streams Smoke Test process on " + str(node.account) + " took too 
> long to exit")
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/utils/util.py",
>  line 36, in wait_until
> raise TimeoutError(err_msg)
> TimeoutError: Streams Smoke Test process on ubuntu@worker9 took too long to 
> exit
> {code}
> https://jenkins.confluent.io/job/system-test-kafka/255/console



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] Optimise memory used by replication process by using adaptive fetch message size

[Jay Kreps]

I think the memory usage for consumers can be improved a lot, but I think
there may be a better way then what you are proposing.

The problem is exactly what you describe: the bound the user sets is
per-partition, but the number of partitions may be quite high. The consumer
could provide a bound on the response size by only requesting a subset of
the partitions, but this would mean that if there was no data available on
those partitions the consumer wouldn't be checking other partitions, which
would add latency.

I think the solution is to add a new "max response size" parameter to the
fetch request so the server checks all partitions but doesn't send back
more than this amount in total. This has to be done carefully to ensure
fairness (i.e. if one partition has unbounded amounts of data it shouldn't
indefinitely starve other partitions).

This will fix memory management both in the replicas and for consumers.

There is a JIRA for this: https://issues.apache.org/jira/browse/KAFKA-2063

I think it isn't too hard to do and would be a huge aid to the memory
profile of both the clients and server.

I also don't think there is much use in setting a max size that expands
dynamically since in any case you have to be able to support the maximum,
so you might as well always use that rather than expanding and contracting
dynamically. That is, if your max fetch response size is 64MB you need to
budget 64MB of free memory, so making it smaller some of the time doesn't
really help you.

-Jay

On Thu, Jul 21, 2016 at 2:49 AM, Andrey L. Neporada <
anepor...@yandex-team.ru> wrote:

> Hi all!
>
> We noticed that our Kafka cluster uses a lot of memory for replication.
> Our Kafka usage pattern is following:
>
> 1. Most messages are small (tens or hundreds kilobytes at most), but some
> (rare) messages can be several megabytes.So, we have to set
> replica.fetch.max.bytes = max.message.bytes = 8MB
> 2. Each Kafka broker handles several thousands of partitions from multiple
> topics.
>
> In this scenario total memory required for replication (i.e.
> replica.fetch.max.bytes * numOfPartitions) is unreasonably big.
>
> So we would like to propose following approach to fix this problem:
>
> 1. Introduce new config parameter replica.fetch.base.bytes - which is the
> initial size of replication data chunk. By default this parameter should be
> equal to replica.fetch.max.bytes so the replication process will work as
> before.
>
> 2. If the ReplicaFetcherThread fails when trying to replicate message
> bigger than current replication chunk, we increase it twofold (or up to
> replica.fetch.max.bytes, whichever is smaller) and retry.
>
> 3. If the chunk is replicated successfully we try to decrease the size of
> replication chunk back to replica.fetch.base.bytes.
>
>
> By choosing replica.fetch.base.bytes in optimal way (in our case ~200K),
> we we able to significatly decrease memory usage without any noticeable
> impact on replication efficiency.
>
> Here is JIRA ticket (with PR):
> https://issues.apache.org/jira/browse/KAFKA-3979
>
> Your comments and feedback are highly appreciated!
>
>
> Thanks,
> Andrey.

Re: [DISCUSS] KIP-4 ACL Admin Schema

[Grant Henke]

Anyone else have any feedback on this protocol and implementation? I plan
to start a vote soon.

Thank you,
Grant

On Fri, Jul 15, 2016 at 1:04 PM, Gwen Shapira  wrote:

> > My goal in the protocol design was to keep the request simple and be able
> > to answer what I think are the 3 most common questions/requests
> >
> >- What ACLs are on the cluster?
> >- What access do I/they have?
> >- Who has access to this resource?
>
> Thanks for clarifying. I think this is good. Perhaps just document
> this goal next to the protocol for the record :)
>
> > Isn't KIP-50 itself one gigantic compatibility concern? I don't see
> >> how your suggestions make it any worse...
> >
> >
> >
> >>  Yes, I also think we should take this chance to improve the Authorizer
> interface
> >> to make it more suitable for the ACL Admin requests.
> >
> >
> > I agree we can address this in KIP-50. What I was getting at was that I
> > wanted to handle that discussion there. We voted on KIP-50 before 0.10
> was
> > released with the intention that we could get it in. Now that 0.10 is
> > released and a longer time has gone by I am not sure if the opinion of
> > "breaking is okay" has changed. I will always prefer a backward
> compatible
> > approach if possible.
>
> Well, the entire KIP-50 discussion - both regarding compatibility and
> possible increased scope is probably out of context here. Especially
> since this proposal was written carefully to avoid any assumptions
> regarding other work. I suggest taking this in a separate thread.
>
> Gwen
>
> > Thank you,
> > Grant
> >
> >
> > On Fri, Jul 15, 2016 at 7:22 AM, Ismael Juma  wrote:
> >
> >> On Fri, Jul 15, 2016 at 6:45 AM, Gwen Shapira 
> wrote:
> >> >
> >> > >>  - I suggest this be addressed in KIP-50 as well, though it
> >> has
> >> > >>  some compatibility concerns.
> >> >
> >> > Isn't KIP-50 itself one gigantic compatibility concern? I don't see
> >> > how your suggestions make it any worse...
> >> >
> >>
> >> Yes, I also think we should take this chance to improve the Authorizer
> >> interface to make it more suitable for the ACL Admin requests.
> >>
> >> Ismael
> >>
> >
> >
> >
> > --
> > Grant Henke
> > Software Engineer | Cloudera
> > gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
>



-- 
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke

Re: [DISCUSS] Optimise memory used by replication process by using adaptive fetch message size

[Andrey L. Neporada]

Hi!

Sorry for not being clear enough.
The problem is on follower side, not on leader side.
It is the follower who is allocating replica.fetch.max.bytes-sized buffers for 
fetch responses somewhere in Java client code.

Andrey.



> On 21 Jul 2016, at 16:28, Tom Crayford  wrote:
> 
> Hi there,
> 
> From my understanding of the protocol (and from digging in the source code
> a bunch) I can't see anywhere where Kafka overallocates memory based on the
> fetch request's max bytes, but maybe I have missed something. If there is
> such a place, then I'd recommend fixing that issue instead - it seems more
> pressing and will alleviate your issue (unless I'm misunderstanding
> something and we *have* to overallocate somewhere).
> 
> I looked in the fetch request path up and down, and in the leader, tracing
> from KafkaApis -> ReplicaManager -> Log -> LogSegment, then to
> FetchResponse and FetchResponseSend (in case you want some pointers to some
> code).
> 
> I may be missing something here, but there seems to be a deeper issue here,
> 
> Tom Crayford
> Heroku Kafka
> 
> On Thu, Jul 21, 2016 at 10:49 AM, Andrey L. Neporada <
> anepor...@yandex-team.ru> wrote:
> 
>> Hi all!
>> 
>> We noticed that our Kafka cluster uses a lot of memory for replication.
>> Our Kafka usage pattern is following:
>> 
>> 1. Most messages are small (tens or hundreds kilobytes at most), but some
>> (rare) messages can be several megabytes.So, we have to set
>> replica.fetch.max.bytes = max.message.bytes = 8MB
>> 2. Each Kafka broker handles several thousands of partitions from multiple
>> topics.
>> 
>> In this scenario total memory required for replication (i.e.
>> replica.fetch.max.bytes * numOfPartitions) is unreasonably big.
>> 
>> So we would like to propose following approach to fix this problem:
>> 
>> 1. Introduce new config parameter replica.fetch.base.bytes - which is the
>> initial size of replication data chunk. By default this parameter should be
>> equal to replica.fetch.max.bytes so the replication process will work as
>> before.
>> 
>> 2. If the ReplicaFetcherThread fails when trying to replicate message
>> bigger than current replication chunk, we increase it twofold (or up to
>> replica.fetch.max.bytes, whichever is smaller) and retry.
>> 
>> 3. If the chunk is replicated successfully we try to decrease the size of
>> replication chunk back to replica.fetch.base.bytes.
>> 
>> 
>> By choosing replica.fetch.base.bytes in optimal way (in our case ~200K),
>> we we able to significatly decrease memory usage without any noticeable
>> impact on replication efficiency.
>> 
>> Here is JIRA ticket (with PR):
>> https://issues.apache.org/jira/browse/KAFKA-3979
>> 
>> Your comments and feedback are highly appreciated!
>> 
>> 
>> Thanks,
>> Andrey.

Re: [DISCUSS] Optimise memory used by replication process by using adaptive fetch message size

[Tom Crayford]

Hi there,

>From my understanding of the protocol (and from digging in the source code
a bunch) I can't see anywhere where Kafka overallocates memory based on the
fetch request's max bytes, but maybe I have missed something. If there is
such a place, then I'd recommend fixing that issue instead - it seems more
pressing and will alleviate your issue (unless I'm misunderstanding
something and we *have* to overallocate somewhere).

I looked in the fetch request path up and down, and in the leader, tracing
from KafkaApis -> ReplicaManager -> Log -> LogSegment, then to
FetchResponse and FetchResponseSend (in case you want some pointers to some
code).

I may be missing something here, but there seems to be a deeper issue here,

Tom Crayford
Heroku Kafka

On Thu, Jul 21, 2016 at 10:49 AM, Andrey L. Neporada <
anepor...@yandex-team.ru> wrote:

> Hi all!
>
> We noticed that our Kafka cluster uses a lot of memory for replication.
> Our Kafka usage pattern is following:
>
> 1. Most messages are small (tens or hundreds kilobytes at most), but some
> (rare) messages can be several megabytes.So, we have to set
> replica.fetch.max.bytes = max.message.bytes = 8MB
> 2. Each Kafka broker handles several thousands of partitions from multiple
> topics.
>
> In this scenario total memory required for replication (i.e.
> replica.fetch.max.bytes * numOfPartitions) is unreasonably big.
>
> So we would like to propose following approach to fix this problem:
>
> 1. Introduce new config parameter replica.fetch.base.bytes - which is the
> initial size of replication data chunk. By default this parameter should be
> equal to replica.fetch.max.bytes so the replication process will work as
> before.
>
> 2. If the ReplicaFetcherThread fails when trying to replicate message
> bigger than current replication chunk, we increase it twofold (or up to
> replica.fetch.max.bytes, whichever is smaller) and retry.
>
> 3. If the chunk is replicated successfully we try to decrease the size of
> replication chunk back to replica.fetch.base.bytes.
>
>
> By choosing replica.fetch.base.bytes in optimal way (in our case ~200K),
> we we able to significatly decrease memory usage without any noticeable
> impact on replication efficiency.
>
> Here is JIRA ticket (with PR):
> https://issues.apache.org/jira/browse/KAFKA-3979
>
> Your comments and feedback are highly appreciated!
>
>
> Thanks,
> Andrey.

[DISCUSS] Optimise memory used by replication process by using adaptive fetch message size

[Andrey L. Neporada]

Hi all!

We noticed that our Kafka cluster uses a lot of memory for replication. Our 
Kafka usage pattern is following:

1. Most messages are small (tens or hundreds kilobytes at most), but some 
(rare) messages can be several megabytes.So, we have to set 
replica.fetch.max.bytes = max.message.bytes = 8MB
2. Each Kafka broker handles several thousands of partitions from multiple 
topics.

In this scenario total memory required for replication (i.e. 
replica.fetch.max.bytes * numOfPartitions) is unreasonably big.

So we would like to propose following approach to fix this problem:

1. Introduce new config parameter replica.fetch.base.bytes - which is the 
initial size of replication data chunk. By default this parameter should be 
equal to replica.fetch.max.bytes so the replication process will work as before.

2. If the ReplicaFetcherThread fails when trying to replicate message bigger 
than current replication chunk, we increase it twofold (or up to 
replica.fetch.max.bytes, whichever is smaller) and retry.

3. If the chunk is replicated successfully we try to decrease the size of 
replication chunk back to replica.fetch.base.bytes.
   

By choosing replica.fetch.base.bytes in optimal way (in our case ~200K), we we 
able to significatly decrease memory usage without any noticeable impact on 
replication efficiency.

Here is JIRA ticket (with PR): https://issues.apache.org/jira/browse/KAFKA-3979

Your comments and feedback are highly appreciated!


Thanks,
Andrey.

[GitHub] kafka pull request #1646: add an annotation

[wangzzu]

GitHub user wangzzu opened a pull request:

https://github.com/apache/kafka/pull/1646

add an annotation



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/wangzzu/kafka wangzzu

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1646.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1646


commit 7895a126a2037340a85ba78d1c0055d1039ef0fb
Author: wangzzu 
Date:   2016-07-21T08:09:37Z

add an annotation




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Comment Edited] (KAFKA-3980) JmxReporter uses excessive memory causing OutOfMemoryException

[Andrew Jorgensen (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3980?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15387185#comment-15387185
 ] 

Andrew Jorgensen edited comment on KAFKA-3980 at 7/21/16 5:59 AM:
--

As far as I know we are not restarting with different client ids. We have 
increase the number of producers recently but if we are restarting with 
separate client ids it is not on purpose. When are those client ids generated? 
Looking at the heap dump they look pretty generated and not manually set, could 
restarting the producers frequently cause new ids to be generated each time 
they restart which add up over time? FWIW we have another cluster that is on 
0.8.1.1, so the older producer, which does not seem to exhibit the same 
symptoms despite being restarted multiple times.

EDIT: I'm not sure if its related but we are using this ruby-kafka client for 
some of the producers. The others are in java and using the provided kafka 
client. 
https://github.com/zendesk/ruby-kafka/blob/master/lib/kafka/client.rb#L44. It 
appears that the client sets itself to `ruby-client` by default.


was (Author: ajorgensen):
As far as I know we are not restarting with different client ids. We have 
increase the number of producers recently but if we are restarting with 
separate client ids it is not on purpose. When are those client ids generated? 
Looking at the heap dump they look pretty generated and not manually set, could 
restarting the producers frequently cause new ids to be generated each time 
they restart which add up over time? FWIW we have another cluster that is on 
0.8.1.1, so the older producer, which does not seem to exhibit the same 
symptoms despite being restarted multiple times.

EDIT: I'm not sure if its related but we are using this ruby-kafka client: 
https://github.com/zendesk/ruby-kafka/blob/master/lib/kafka/client.rb#L44. It 
appears that the client sets itself to `ruby-client` by default.

> JmxReporter uses excessive memory causing OutOfMemoryException
> --
>
> Key: KAFKA-3980
> URL: https://issues.apache.org/jira/browse/KAFKA-3980
> Project: Kafka
>  Issue Type: Bug
>Affects Versions: 0.9.0.1
>Reporter: Andrew Jorgensen
>
> I have some nodes in a kafka cluster that occasionally will run out of memory 
> whenever I restart the producers. I was able to take a heap dump from both a 
> recently restarted Kafka node which weighed in at about 20 MB and a node that 
> has been running for 2 months is using over 700MB of memory. Looking at the 
> heap dump it looks like the JmxReporter is holding on to metrics and causing 
> them to build up over time. 
> !http://imgur.com/N6Cd0Ku.png!
> !http://imgur.com/kQBqA2j.png!
> The ultimate problem this causes is that there is a chance when I restart the 
> producers it will cause the node to experience an Java heap space exception 
> and OOM. The nodes  then fail to startup correctly and write a -1 as the 
> leader number to the partitions they were responsible for effectively 
> resetting the offset and rendering that partition unavailable. The kafka 
> process then needs to go be restarted in order to re-assign the node to the 
> partition that it owns.
> I have a few questions:
> 1. I am not quite sure why there are so many client id entries in that 
> JmxReporter map.
> 2. Is there a way to have the JmxReporter release metrics after a set amount 
> of time or a way to turn certain high cardinality metrics like these off?
> I can provide any logs or heap dumps if more information is needed.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)