[jira] [Commented] (KAFKA-1429) Yet another deadlock in controller shutdown

[Jun Rao (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-1429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15343716#comment-15343716
 ] 

Jun Rao commented on KAFKA-1429:


[~pengwei], yes, it's a real bug. Thanks for finding and reporting this. It 
seems that we will also need to fix SessionExpirationListener in 
KafkaController. For easy review, is it convenient for you to submit the patch 
through a pull request (see 
https://cwiki.apache.org/confluence/display/KAFKA/Contributing+Code+Changes)?

> Yet another deadlock in controller shutdown
> ---
>
> Key: KAFKA-1429
> URL: https://issues.apache.org/jira/browse/KAFKA-1429
> Project: Kafka
>  Issue Type: Bug
>  Components: controller
>Affects Versions: 0.8.1
>Reporter: Dmitry Bugaychenko
>Assignee: Neha Narkhede
> Attachments: kafka_0.9.0.0_controller_dead_lock.patch
>
>
> Found one more case of deadlock in controller during shutdown:
> {code}
> ZkClient-EventThread-57-192.168.41.148:2181,192.168.36.250:2181,192.168.41.207:2181
>  id=57 state=TIMED_WAITING
> - waiting on <0x288a66ec> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> - locked <0x288a66ec> (a 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject)
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.parkNanos(LockSupport.java:226)
> at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2082)
> at 
> java.util.concurrent.ThreadPoolExecutor.awaitTermination(ThreadPoolExecutor.java:1468)
> at kafka.utils.KafkaScheduler.shutdown(KafkaScheduler.scala:88)
> at 
> kafka.controller.KafkaController$$anonfun$onControllerResignation$1.apply$mcV$sp(KafkaController.scala:339)
> at 
> kafka.controller.KafkaController$$anonfun$onControllerResignation$1.apply(KafkaController.scala:337)
> at 
> kafka.controller.KafkaController$$anonfun$onControllerResignation$1.apply(KafkaController.scala:337)
> at kafka.utils.Utils$.inLock(Utils.scala:538)
> at 
> kafka.controller.KafkaController.onControllerResignation(KafkaController.scala:337)
> at 
> kafka.controller.KafkaController$SessionExpirationListener$$anonfun$handleNewSession$1.apply$mcZ$sp(KafkaController.scala:1068)
> at 
> kafka.controller.KafkaController$SessionExpirationListener$$anonfun$handleNewSession$1.apply(KafkaController.scala:1067)
> at 
> kafka.controller.KafkaController$SessionExpirationListener$$anonfun$handleNewSession$1.apply(KafkaController.scala:1067)
> at kafka.utils.Utils$.inLock(Utils.scala:538)
> at 
> kafka.controller.KafkaController$SessionExpirationListener.handleNewSession(KafkaController.scala:1067)
> at org.I0Itec.zkclient.ZkClient$4.run(ZkClient.java:472)
> at org.I0Itec.zkclient.ZkEventThread.run(ZkEventThread.java:71)
> Locked synchronizers: count = 1
>   - java.util.concurrent.locks.ReentrantLock$NonfairSync@22b9b31a
> kafka-scheduler-0 id=172 state=WAITING
> - waiting on <0x22b9b31a> (a 
> java.util.concurrent.locks.ReentrantLock$NonfairSync)
> - locked <0x22b9b31a> (a 
> java.util.concurrent.locks.ReentrantLock$NonfairSync)
>  owned by 
> ZkClient-EventThread-57-192.168.41.148:2181,192.168.36.250:2181,192.168.41.207:2181
>  id=57
> at sun.misc.Unsafe.park(Native Method)
> at java.util.concurrent.locks.LockSupport.park(LockSupport.java:186)
> at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:834)
> at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:867)
> at 
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1197)
> at 
> java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:214)
> at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:290)
> at kafka.utils.Utils$.inLock(Utils.scala:536)
> at 
> kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4$$anonfun$apply$17.apply(KafkaController.scala:1110)
> at 
> kafka.controller.KafkaController$$anonfun$kafka$controller$KafkaController$$checkAndTriggerPartitionRebalance$4$$anonfun$apply$17.apply(KafkaController.scala:1108)
> at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
> at 
> scala.collection.mutable.HashMap$$anonfun$foreach$1.apply(HashMap.scala:98)
> at 
> scala.collection.mutable.HashTable$class.foreachEntry(HashTable.scala:226)
> at scala.collection.mutable.HashMap.foreachEntry(HashMap.scala:39)
> at scala.collection.mutable.HashMap.foreach(HashMap.scala:98)
> at 
> 

[jira] [Work started] (KAFKA-3890) Kafka Streams: task assignment is not maintained on cluster restart or rolling restart

[Henry Cai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-3890 started by Henry Cai.

> Kafka Streams: task assignment is not maintained on cluster restart or 
> rolling restart
> --
>
> Key: KAFKA-3890
> URL: https://issues.apache.org/jira/browse/KAFKA-3890
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Henry Cai
>Assignee: Henry Cai
>  Labels: api, newbie
>
> Currently the task assignment in TaskAssignor is not deterministic.  During 
> cluster restart or rolling restart, even though the participating worker 
> nodes are the same, but the TaskAssignor is not able to maintain a 
> deterministic mapping, so about 20% partitions will be reassigned which would 
> cause state repopulation on cluster restart time.
> When the participating worker nodes are not changed, we really just want to 
> keep the old task assignment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3890) Kafka Streams: task assignment is not maintained on cluster restart or rolling restart

[Henry Cai (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3890?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15343209#comment-15343209
 ] 

Henry Cai commented on KAFKA-3890:
--

PR: https://github.com/apache/kafka/pull/1538

> Kafka Streams: task assignment is not maintained on cluster restart or 
> rolling restart
> --
>
> Key: KAFKA-3890
> URL: https://issues.apache.org/jira/browse/KAFKA-3890
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Henry Cai
>Assignee: Henry Cai
>  Labels: api, newbie
>
> Currently the task assignment in TaskAssignor is not deterministic.  During 
> cluster restart or rolling restart, even though the participating worker 
> nodes are the same, but the TaskAssignor is not able to maintain a 
> deterministic mapping, so about 20% partitions will be reassigned which would 
> cause state repopulation on cluster restart time.
> When the participating worker nodes are not changed, we really just want to 
> keep the old task assignment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1538: AFKA-3890 Kafka Streams: task assignment is not ma...

[HenryCaiHaiying]

GitHub user HenryCaiHaiying opened a pull request:

https://github.com/apache/kafka/pull/1538

AFKA-3890 Kafka Streams: task assignment is not maintained on cluster 
restart or rolling restart

Current task assignment in TaskAssignor is not deterministic.

During cluster restart or rolling restart, we have the same set of 
participating worker nodes.  But the current TaskAssignor is not able to 
maintain a deterministic mapping, so about 20% partitions will be reassigned 
which would cause state repopulation.
When the topology of work nodes (# of worker nodes, the TaskIds they are 
carrying with) is not changed, we really just want to keep the old task 
assignment.

Add the code to check whether the node topology is changing or not:
- when the prevAssignedTasks from the old clientStates is the same as the 
new task list
- when there is no new node joining (its prevAssignTasks would be either 
empty or conflict with some other nodes)
- when there is no node dropping out (the total of prevAssignedTasks from 
other nodes would not be equal to the new task list)

When the topology is not changing, we would just use the old mapping.

I also added the code to check whether the previous assignment is balanced 
(whether each node's task list is within [1/2 average -- 2 * average]), if it's 
not balanced, we will still start the a new task assignment.


You can merge this pull request into a Git repository by running:

$ git pull https://github.com/HenryCaiHaiying/kafka upstream

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1538.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1538


commit 2d6c4a18ebea38fae4e7e47a45c8d699445d2738
Author: Henry Cai 
Date:   2016-06-20T06:06:20Z

Add the capability to maintain the old task assignment when the node 
topology is not changing

During cluster restart or rolling restart, the node topology is not 
changing.  But the current TaskAssignor is not able to maintain a deterministic 
mapping, so about 20% partitions will be reassigned which would cause state 
repopulation.
When the node topology is not changed, we really just want to keep the old 
task assignment.

Add the code to check whether the node topology is changing or not:
- when the prevAssignedTasks from the old clientStates is the same as the 
new task list
- when there is no new node joining (its prevAssignTasks would be either 
empty or conflict with some other nodes)
- when there is no node dropping (the total of prevAssignedTasks from other 
nodes would not be equal to the new task list)

When the topology is not changing, we would just use the old mapping.




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Updated] (KAFKA-3890) Kafka Streams: task assignment is not maintained on cluster restart or rolling restart

[Henry Cai (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3890?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Henry Cai updated KAFKA-3890:
-
Description: 
Currently the task assignment in TaskAssignor is not deterministic.  During 
cluster restart or rolling restart, even though the participating worker nodes 
are the same, but the TaskAssignor is not able to maintain a deterministic 
mapping, so about 20% partitions will be reassigned which would cause state 
repopulation on cluster restart time.

When the participating worker nodes are not changed, we really just want to 
keep the old task assignment.


  was:
Today most of the rocksDB configs are hard written inside {{RocksDBStore}}, or 
the default values are directly used. We need to make them configurable for 
advanced users. For example, some default values may not work perfectly for 
some scenarios: 
https://github.com/HenryCaiHaiying/kafka/commit/ccc4e25b110cd33eea47b40a2f6bf17ba0924576
 

One way of doing that is to introduce a "RocksDBStoreConfigs" objects similar 
to "StreamsConfig", which defines all related rocksDB options configs, that can 
be passed as key-value pairs to "StreamsConfig".


> Kafka Streams: task assignment is not maintained on cluster restart or 
> rolling restart
> --
>
> Key: KAFKA-3890
> URL: https://issues.apache.org/jira/browse/KAFKA-3890
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Henry Cai
>Assignee: Henry Cai
>  Labels: api, newbie
>
> Currently the task assignment in TaskAssignor is not deterministic.  During 
> cluster restart or rolling restart, even though the participating worker 
> nodes are the same, but the TaskAssignor is not able to maintain a 
> deterministic mapping, so about 20% partitions will be reassigned which would 
> cause state repopulation on cluster restart time.
> When the participating worker nodes are not changed, we really just want to 
> keep the old task assignment.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3890) Kafka Streams: task assignment is not maintained on cluster restart or rolling restart

[Henry Cai (JIRA)]

Henry Cai created KAFKA-3890:


 Summary: Kafka Streams: task assignment is not maintained on 
cluster restart or rolling restart
 Key: KAFKA-3890
 URL: https://issues.apache.org/jira/browse/KAFKA-3890
 Project: Kafka
  Issue Type: Bug
  Components: streams
Reporter: Henry Cai
Assignee: Henry Cai


Today most of the rocksDB configs are hard written inside {{RocksDBStore}}, or 
the default values are directly used. We need to make them configurable for 
advanced users. For example, some default values may not work perfectly for 
some scenarios: 
https://github.com/HenryCaiHaiying/kafka/commit/ccc4e25b110cd33eea47b40a2f6bf17ba0924576
 

One way of doing that is to introduce a "RocksDBStoreConfigs" objects similar 
to "StreamsConfig", which defines all related rocksDB options configs, that can 
be passed as key-value pairs to "StreamsConfig".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3846) Connect record types should include timestamps

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3846?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15343117#comment-15343117
 ] 

ASF GitHub Bot commented on KAFKA-3846:
---

GitHub user shikhar opened a pull request:

https://github.com/apache/kafka/pull/1537

KAFKA-3846: include timestamp in Connect record types

KIP to come

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/shikhar/kafka kafka-3846

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1537.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1537


commit 2dc7e91c1ec06fcf987dda07a63acf45e1b2d13e
Author: Shikhar Bhushan 
Date:   2016-06-22T00:42:48Z

KAFKA-3846: include timestamp in Connect record types; add Builder for 
`SourceRecord`

`SinkRecord` gets `timestampType` and `timestamp`
`SourceRecord` gets `timestamp`
`SourceRecord.Builder` is the new preferred way to construct `SourceRecord`s




> Connect record types should include timestamps
> --
>
> Key: KAFKA-3846
> URL: https://issues.apache.org/jira/browse/KAFKA-3846
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Affects Versions: 0.10.0.0
>Reporter: Ewen Cheslack-Postava
>Assignee: Shikhar Bhushan
>  Labels: needs-kip
> Fix For: 0.10.1.0
>
>
> Timestamps were added to records in the previous release, however this does 
> not get propagated automatically to Connect because it uses custom wrappers  
> to add fields and rename some for clarity.
> The addition of timestamps should be trivial, but can be really useful (e.g. 
> in sink connectors that would like to include timestamp info if available but 
> when it is not stored in the value).
> This is public API so it will need a KIP despite being very uncontentious.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1537: KAFKA-3846: include timestamp in Connect record ty...

[shikhar]

GitHub user shikhar opened a pull request:

https://github.com/apache/kafka/pull/1537

KAFKA-3846: include timestamp in Connect record types

KIP to come

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/shikhar/kafka kafka-3846

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1537.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1537


commit 2dc7e91c1ec06fcf987dda07a63acf45e1b2d13e
Author: Shikhar Bhushan 
Date:   2016-06-22T00:42:48Z

KAFKA-3846: include timestamp in Connect record types; add Builder for 
`SourceRecord`

`SinkRecord` gets `timestampType` and `timestamp`
`SourceRecord` gets `timestamp`
`SourceRecord.Builder` is the new preferred way to construct `SourceRecord`s




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3740) Add configs for RocksDBStores

[Henry Cai (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15343115#comment-15343115
 ] 

Henry Cai commented on KAFKA-3740:
--

Yes, you or Roger can pick it up.

> Add configs for RocksDBStores
> -
>
> Key: KAFKA-3740
> URL: https://issues.apache.org/jira/browse/KAFKA-3740
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Henry Cai
>  Labels: api, newbie
>
> Today most of the rocksDB configs are hard written inside {{RocksDBStore}}, 
> or the default values are directly used. We need to make them configurable 
> for advanced users. For example, some default values may not work perfectly 
> for some scenarios: 
> https://github.com/HenryCaiHaiying/kafka/commit/ccc4e25b110cd33eea47b40a2f6bf17ba0924576
>  
> One way of doing that is to introduce a "RocksDBStoreConfigs" objects similar 
> to "StreamsConfig", which defines all related rocksDB options configs, that 
> can be passed as key-value pairs to "StreamsConfig".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] KIP-55: Secure quotas for authenticated users

[Jun Rao]

Yes, for consistency, perhaps we can allow client-id quota to be configured
dynamically too and mark the static config in the broker as deprecated. If
both are set, the dynamic one wins.

Thanks,

Jun

On Tue, Jun 21, 2016 at 3:56 AM, Ismael Juma  wrote:

> On Tue, Jun 21, 2016 at 12:50 PM, Rajini Sivaram <
> rajinisiva...@googlemail.com> wrote:
>
> > It is actually quite tempting to do the same for client-id quotas as
> well,
> > but I suppose we can't break existing users who have configured defaults
> in
> > server.properties and providing two ways of setting client-id defaults
> > would be just too confusing.
> >
>
> Using two different approaches for client-id versus user quota defaults is
> also not great. We could deprecate the server.properties default configs
> for client-id quotas and remove them in the future. In the meantime, we
> would have to live with 2 level defaults.
>
> Jun, what are your thoughts on this?
>
> Ismael
>

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[Jun Rao]

Parth,

Thanks for the reply. A couple of comments inline below.

On Tue, Jun 21, 2016 at 10:36 AM, parth brahmbhatt <
brahmbhatt.pa...@gmail.com> wrote:

> 1. Who / how are tokens renewed? By original requester only? or using
> Kerberos
> auth only?
> My recommendation is to do this only using Kerberos auth and only threw the
> renewer specified during the acquisition request.
>
>
Hmm, not sure that I follow this. Are you saying that any client
authenticated with the delegation token can renew, i.e. there is no renewer
needed?

Also, just to be clear, any authenticated client (either through SASL or
SSL) can request a delegation token for the authenticated user, right?


> 2. Are tokens stored on each broker or in ZK?
> My recommendation is still to store in ZK or not store them at all. The
> whole controller based distribution is too much overhead with not much to
> achieve.
>
> 3. How are tokens invalidated / expired?
> Either by expiration time out or through an explicit request to invalidate.
>
> 4. Which encryption algorithm is used?
> SCRAM
>
> 5. What is the impersonation proposal (it wasn't in the KIP but was
> discussed
> in this thread)?
> There is no imperonation proposal. I tried and explained how its a
> different problem and why its not really necessary to discuss that as part
> of this KIP.  This KIP will not support any impersonation, it will just be
> another way to authenticate.
>
> 6. Do we need new ACLs, if so - for what actions?
> We do not need new ACLs.
>
>
Could we document the format of the new request/response and their
associated Resource and Operation for ACL?


> 7. How would the delegation token be configured in the client?
> Should be through config. I wasn't planning on supporting JAAS for tokens.
> I don't believe hadoop does this either.
>
> Thanks
> Parth
>
>
>
> On Thu, Jun 16, 2016 at 4:03 PM, Jun Rao  wrote:
>
> > Harsha,
> >
> > Another question.
> >
> > 9. How would the delegation token be configured in the client? The
> standard
> > way is to do this through JAAS. However, we will need to think through if
> > this is convenient in a shared environment. For example, when a new task
> is
> > added to a Storm worker node, do we need to dynamically add a new section
> > in the JAAS file? It may be more convenient if we can pass in the token
> > through the config directly w/o going through JAAS.
> >
> > Are you or Parth still actively working on this KIP?
> >
> > Thanks,
> >
> > Jun
> >
> >
> >
> > On Sun, Jun 12, 2016 at 2:18 PM, Jun Rao  wrote:
> >
> > > Just to add on that list.
> > >
> > > 2. It would be good to document the format of the data stored in ZK.
> > > 7. Earlier, there was a discussion on whether the tokens should be
> > > propagated through ZK like config/acl/quota, or through the controller.
> > > Currently, the controller is only designed for propagating topic
> > metadata,
> > > but not other data.
> > > 8. Should we use SCRAM to send the token instead of DIGEST-MD5 since
> it's
> > > deprecated?
> > >
> > > Also, the images in the wiki seem broken.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > > On Fri, Jun 10, 2016 at 10:02 AM, Gwen Shapira 
> > wrote:
> > >
> > >> From what I can see, remaining questions are:
> > >>
> > >> 1. Who / how are tokens renewed? By original requester only? or using
> > >> Kerberos auth only?
> > >> 2. Are tokens stored on each broker or in ZK?
> > >> 3. How are tokens invalidated / expired?
> > >> 4. Which encryption algorithm is used?
> > >> 5. What is the impersonation proposal (it wasn't in the KIP but was
> > >> discussed in this thread)?
> > >> 6. Do we need new ACLs, if so - for what actions?
> > >>
> > >> Gwen
> > >>
> > >> On Thu, Jun 9, 2016 at 7:48 PM, Harsha  wrote:
> > >> > Jun & Ismael,
> > >> >  Unfortunately I couldn't attend the KIP
> > meeting
> > >> >  when delegation tokens discussed.
> Appreciate
> > if
> > >> >  you can update the thread if you have any
> > >> >  further questions.
> > >> > Thanks,
> > >> > Harsha
> > >> >
> > >> > On Tue, May 24, 2016, at 11:32 AM, Liquan Pei wrote:
> > >> >> It seems that the links to images in the KIP are broken.
> > >> >>
> > >> >> Liquan
> > >> >>
> > >> >> On Tue, May 24, 2016 at 9:33 AM, parth brahmbhatt <
> > >> >> brahmbhatt.pa...@gmail.com> wrote:
> > >> >>
> > >> >> > 110. What does getDelegationTokenAs mean?
> > >> >> > In the current proposal we only allow a user to get delegation
> > token
> > >> for
> > >> >> > the identity that it authenticated as using another mechanism,
> i.e.
> > >> A user
> > >> >> > that authenticate using a keytab for principal us...@example.com
> > >> will get
> > >> >> > delegation tokens for that user only. In future I think we will
> > have
> > >> to
> > >> >> > extend support such that we allow some set of users (
> > >> >> > kafka-rest-u...@example.com, 

Re: [DISCUSS] KIP-4 Delete Topic Schema

[Guozhang Wang]

I see we have the similar setting for CreateTopic request timeout <= 0 as
well, so maybe it has been discussed and I simply overlooked.. otherwise my
question is for both of these cases.

Guozhang

On Tue, Jun 21, 2016 at 4:07 PM, Guozhang Wang  wrote:

> Thanks Grant, looks good to me overall. One minor comment below:
>
> >   - The error code in the response will either contain an argument
> >   validation exception or a timeout exception. If you receive a
> timeout
> >   exception, because you asked for 0 timeout, you can assume the
> message was
> >   valid and the topic deletion was triggered.
>
> In the timeout <= 0 case, if the client should always ignore and treat the 
> timeout
> error code as "OK", should we just return no error code in this case?
>
>
> Guozhang
>
>
> On Tue, Jun 21, 2016 at 8:17 AM, Grant Henke  wrote:
>
>> Hi Ismael,
>>
>> Thanks for the review. See my responses below.
>>
>> One potential issue is that the number of topics in the response won't
>> > match the number of topics in the request. Worth checking what others
>> think
>> > of this one.
>>
>>
>> Based on the choice of how we handled duplicate topics in the create
>> topics
>> protocol, I took the same approach here. At one point create topics would
>> disconnect because I could't return an error per topic request. In the end
>> the preference was to return and error code even though the cardinality
>> would be different.
>>
>> 4. When requesting to delete a topic that is already marked for
>> > > >deletion, the request will wait for the wait for the timeout and
>> > > return as
>> > > >usual
>> >
>> > Do you mean that it will wait up to the timeout until the delete is
>> > "complete" as per the definition in `6`? Or will it wait unconditionally
>> > until the timeout expires? It would be good to make that clear.
>> >
>>
>> That's exactly what I meant. I updated the wiki.
>>
>> This could leak topic name information (as per KAFKA-3396, which was filed
>> > by you). We would probably want to return `InvalidTopic` for the case
>> where
>> > the user doesn't have a valid `DESCRIBE TOPIC` ACL, right?
>>
>>
>> Good point. I will update the wiki and patch.
>>
>> Unauthorized requests will receive a TopicAuthorizationException if they
>> are authorized to the "Describe" Operation on the "Topic" resource.
>> Otherwise they will receive an InvalidTopicException as if the topic does
>> not exist.
>>
>> I was wondering (and this applies to the create topic as well), is there
>> > any value in a flag that says whether the timeout expired or not?
>>
>>
>> In both the create and delete response we return a TimeoutException error
>> code for the topics that did not "complete" in time. This allows the
>> client
>> to know which topics actions completed and which timed out. I updated the
>> wiki to explicitly call this out in the response section.
>>
>> Thanks,
>> Grant
>>
>> On Tue, Jun 21, 2016 at 5:44 AM, Ismael Juma  wrote:
>>
>> > Thanks Grant. A few comments inline.
>> >
>> > On Mon, Jun 20, 2016 at 9:09 PM, Grant Henke 
>> wrote:
>> >
>> > > >2. If there are multiple instructions for the same topic in one
>> > > >request the extra request will be ignored
>> > > >   - This is because the list of topics is modeled server side
>> as a
>> > > set
>> > > >   - Multiple deletes results in the same end goal, so handling
>> this
>> > > >   error for the user should be okay
>> > >
>> >
>> > One potential issue is that the number of topics in the response won't
>> > match the number of topics in the request. Worth checking what others
>> think
>> > of this one.
>> >
>> > 4. When requesting to delete a topic that is already marked for
>> > > >deletion, the request will wait for the wait for the timeout and
>> > > return as
>> > > >usual
>> >
>> >
>> > Do you mean that it will wait up to the timeout until the delete is
>> > "complete" as per the definition in `6`? Or will it wait unconditionally
>> > until the timeout expires? It would be good to make that clear.
>> >
>> >
>> > > >5. The principal must be authorized to the "Delete" Operation on
>> the
>> > >
>> > >"Topic" resource to delete the topic.
>> > > >   - Unauthorized requests will receive a
>> > TopicAuthorizationException
>> > >
>> >
>> > This could leak topic name information (as per KAFKA-3396, which was
>> filed
>> > by you). We would probably want to return `InvalidTopic` for the case
>> where
>> > the user doesn't have a valid `DESCRIBE TOPIC` ACL, right?
>> >
>> >
>> > > >- Why have a timeout at all? Deletes could take a while?
>> > >
>> >
>> > I was wondering (and this applies to the create topic as well), is there
>> > any value in a flag that says whether the timeout expired or not?
>> >
>> > Thanks,
>> > Ismael
>> >
>>
>>
>>
>> --
>> Grant Henke
>> Software Engineer | Cloudera
>> gr...@cloudera.com | twitter.com/gchenke | 

Re: [DISCUSS] KIP-4 Delete Topic Schema

[Guozhang Wang]

Thanks Grant, looks good to me overall. One minor comment below:

>   - The error code in the response will either contain an argument
>   validation exception or a timeout exception. If you receive a
timeout
>   exception, because you asked for 0 timeout, you can assume the
message was
>   valid and the topic deletion was triggered.

In the timeout <= 0 case, if the client should always ignore and treat
the timeout
error code as "OK", should we just return no error code in this case?


Guozhang


On Tue, Jun 21, 2016 at 8:17 AM, Grant Henke  wrote:

> Hi Ismael,
>
> Thanks for the review. See my responses below.
>
> One potential issue is that the number of topics in the response won't
> > match the number of topics in the request. Worth checking what others
> think
> > of this one.
>
>
> Based on the choice of how we handled duplicate topics in the create topics
> protocol, I took the same approach here. At one point create topics would
> disconnect because I could't return an error per topic request. In the end
> the preference was to return and error code even though the cardinality
> would be different.
>
> 4. When requesting to delete a topic that is already marked for
> > > >deletion, the request will wait for the wait for the timeout and
> > > return as
> > > >usual
> >
> > Do you mean that it will wait up to the timeout until the delete is
> > "complete" as per the definition in `6`? Or will it wait unconditionally
> > until the timeout expires? It would be good to make that clear.
> >
>
> That's exactly what I meant. I updated the wiki.
>
> This could leak topic name information (as per KAFKA-3396, which was filed
> > by you). We would probably want to return `InvalidTopic` for the case
> where
> > the user doesn't have a valid `DESCRIBE TOPIC` ACL, right?
>
>
> Good point. I will update the wiki and patch.
>
> Unauthorized requests will receive a TopicAuthorizationException if they
> are authorized to the "Describe" Operation on the "Topic" resource.
> Otherwise they will receive an InvalidTopicException as if the topic does
> not exist.
>
> I was wondering (and this applies to the create topic as well), is there
> > any value in a flag that says whether the timeout expired or not?
>
>
> In both the create and delete response we return a TimeoutException error
> code for the topics that did not "complete" in time. This allows the client
> to know which topics actions completed and which timed out. I updated the
> wiki to explicitly call this out in the response section.
>
> Thanks,
> Grant
>
> On Tue, Jun 21, 2016 at 5:44 AM, Ismael Juma  wrote:
>
> > Thanks Grant. A few comments inline.
> >
> > On Mon, Jun 20, 2016 at 9:09 PM, Grant Henke 
> wrote:
> >
> > > >2. If there are multiple instructions for the same topic in one
> > > >request the extra request will be ignored
> > > >   - This is because the list of topics is modeled server side as
> a
> > > set
> > > >   - Multiple deletes results in the same end goal, so handling
> this
> > > >   error for the user should be okay
> > >
> >
> > One potential issue is that the number of topics in the response won't
> > match the number of topics in the request. Worth checking what others
> think
> > of this one.
> >
> > 4. When requesting to delete a topic that is already marked for
> > > >deletion, the request will wait for the wait for the timeout and
> > > return as
> > > >usual
> >
> >
> > Do you mean that it will wait up to the timeout until the delete is
> > "complete" as per the definition in `6`? Or will it wait unconditionally
> > until the timeout expires? It would be good to make that clear.
> >
> >
> > > >5. The principal must be authorized to the "Delete" Operation on
> the
> > >
> > >"Topic" resource to delete the topic.
> > > >   - Unauthorized requests will receive a
> > TopicAuthorizationException
> > >
> >
> > This could leak topic name information (as per KAFKA-3396, which was
> filed
> > by you). We would probably want to return `InvalidTopic` for the case
> where
> > the user doesn't have a valid `DESCRIBE TOPIC` ACL, right?
> >
> >
> > > >- Why have a timeout at all? Deletes could take a while?
> > >
> >
> > I was wondering (and this applies to the create topic as well), is there
> > any value in a flag that says whether the timeout expired or not?
> >
> > Thanks,
> > Ismael
> >
>
>
>
> --
> Grant Henke
> Software Engineer | Cloudera
> gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke
>



-- 
-- Guozhang

[jira] [Commented] (KAFKA-3889) Change default Scala version to 2.11

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15342984#comment-15342984
 ] 

Ismael Juma commented on KAFKA-3889:


Scala 2.10 is used if you do `./gradlew test` or `./gradlew jar` (instead of 
`./gradlew testAll` and `./gradlew jarAll`). During development, we typically 
use the `test` variant while Jenkins uses the `testAll` variant. The latter 
takes longer obviously.

> Change default Scala version to 2.11
> 
>
> Key: KAFKA-3889
> URL: https://issues.apache.org/jira/browse/KAFKA-3889
> Project: Kafka
>  Issue Type: Improvement
>  Components: build, tools
>Reporter: Kevin Lafferty
>Priority: Minor
>
> The [downloads page|http://kafka.apache.org/downloads.html] says that Scala 
> 2.11 is recommended. However, 2.10 is the default build version as well as 
> the default in the bin scripts.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3740) Add configs for RocksDBStores

[Guozhang Wang (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3740?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15342979#comment-15342979
 ] 

Guozhang Wang commented on KAFKA-3740:
--

Since we need to pass the StreamsConfig object to the storage layers anyways, 
one approach to do that is to expose the configs via `ProcessorContext`, by 
leveraging `AbstractConfig.originalsWithPrefix`. For example, users can specify 
configs like "rocksdb.xx.xx" and then with a new API:

Map properties ProcessorContext.configs("prefix");

[~h...@pinterest.com] If you are busy recently could myself or [~theduderog] 
pick this up?


> Add configs for RocksDBStores
> -
>
> Key: KAFKA-3740
> URL: https://issues.apache.org/jira/browse/KAFKA-3740
> Project: Kafka
>  Issue Type: Bug
>  Components: streams
>Reporter: Guozhang Wang
>Assignee: Henry Cai
>  Labels: api, newbie
>
> Today most of the rocksDB configs are hard written inside {{RocksDBStore}}, 
> or the default values are directly used. We need to make them configurable 
> for advanced users. For example, some default values may not work perfectly 
> for some scenarios: 
> https://github.com/HenryCaiHaiying/kafka/commit/ccc4e25b110cd33eea47b40a2f6bf17ba0924576
>  
> One way of doing that is to introduce a "RocksDBStoreConfigs" objects similar 
> to "StreamsConfig", which defines all related rocksDB options configs, that 
> can be passed as key-value pairs to "StreamsConfig".



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3889) Change default Scala version to 2.11

[Kevin Lafferty (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15342973#comment-15342973
 ] 

Kevin Lafferty commented on KAFKA-3889:
---

Well, both versions are built. It's just that the default and recommended 
versions are different, which is odd.

> Change default Scala version to 2.11
> 
>
> Key: KAFKA-3889
> URL: https://issues.apache.org/jira/browse/KAFKA-3889
> Project: Kafka
>  Issue Type: Improvement
>  Components: build, tools
>Reporter: Kevin Lafferty
>Priority: Minor
>
> The [downloads page|http://kafka.apache.org/downloads.html] says that Scala 
> 2.11 is recommended. However, 2.10 is the default build version as well as 
> the default in the bin scripts.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3889) Change default Scala version to 2.11

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15342939#comment-15342939
 ] 

Ismael Juma commented on KAFKA-3889:


This is intentional. We build with the lowest version supported in order to 
ensure that we don't use any features from a newer version.

> Change default Scala version to 2.11
> 
>
> Key: KAFKA-3889
> URL: https://issues.apache.org/jira/browse/KAFKA-3889
> Project: Kafka
>  Issue Type: Improvement
>  Components: build, tools
>Reporter: Kevin Lafferty
>Priority: Minor
>
> The [downloads page|http://kafka.apache.org/downloads.html] says that Scala 
> 2.11 is recommended. However, 2.10 is the default build version as well as 
> the default in the bin scripts.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Updated] (KAFKA-3889) Change default Scala version to 2.11

[Kevin Lafferty (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3889?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Kevin Lafferty updated KAFKA-3889:
--
Description: The [downloads page|http://kafka.apache.org/downloads.html] 
says that Scala 2.11 is recommended. However, 2.10 is the default build version 
as well as the default in the bin scripts.

> Change default Scala version to 2.11
> 
>
> Key: KAFKA-3889
> URL: https://issues.apache.org/jira/browse/KAFKA-3889
> Project: Kafka
>  Issue Type: Improvement
>  Components: build, tools
>Reporter: Kevin Lafferty
>Priority: Minor
>
> The [downloads page|http://kafka.apache.org/downloads.html] says that Scala 
> 2.11 is recommended. However, 2.10 is the default build version as well as 
> the default in the bin scripts.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3889) Change default Scala version to 2.11

[Kevin Lafferty (JIRA)]

Kevin Lafferty created KAFKA-3889:
-

 Summary: Change default Scala version to 2.11
 Key: KAFKA-3889
 URL: https://issues.apache.org/jira/browse/KAFKA-3889
 Project: Kafka
  Issue Type: Improvement
  Components: build, tools
Reporter: Kevin Lafferty
Priority: Minor






--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Comment Edited] (KAFKA-3887) StreamBounceTest.test_bounce and StreamSmokeTest.test_streams failing

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15342841#comment-15342841
 ] 

Ismael Juma edited comment on KAFKA-3887 at 6/21/16 10:17 PM:
--

First build to fail: 
http://confluent-kafka-system-test-results.s3-us-west-2.amazonaws.com/2016-06-11--001.1465663177--apache--trunk--da85171/report.html

After that, it's been failing in every build.


was (Author: ijuma):
First build to fail: 
http://confluent-kafka-system-test-results.s3-us-west-2.amazonaws.com/2016-06-11--001.1465663177--apache--trunk--da85171/report.html

After that, it's been failing almost every build.

> StreamBounceTest.test_bounce and StreamSmokeTest.test_streams failing
> -
>
> Key: KAFKA-3887
> URL: https://issues.apache.org/jira/browse/KAFKA-3887
> Project: Kafka
>  Issue Type: Bug
>  Components: streams, system tests
>Reporter: Ismael Juma
>Assignee: Eno Thereska
>  Labels: transient-system-test-failure
>
> StreamBounceTest.test_bounce and StreamSmokeTest.test_streams has been 
> failing semi-regularly. Output from the latest failure:
> {code}
> 
> test_id:
> 2016-06-20--001.kafkatest.tests.streams.streams_bounce_test.StreamsBounceTest.test_bounce
> status: FAIL
> run time:   4 minutes 13.916 seconds
> Streams Smoke Test process on ubuntu@worker5 took too long to exit
> Traceback (most recent call last):
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
>  line 106, in run_all_tests
> data = self.run_single_test()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
>  line 162, in run_single_test
> return self.current_test_context.function(self.current_test)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/streams/streams_bounce_test.py",
>  line 67, in test_bounce
> self.driver.wait()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/services/streams.py",
>  line 94, in wait
> wait_until(lambda: not node.account.alive(pid), timeout_sec=120, 
> err_msg="Streams Smoke Test process on " + str(node.account) + " took too 
> long to exit")
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/utils/util.py",
>  line 36, in wait_until
> raise TimeoutError(err_msg)
> TimeoutError: Streams Smoke Test process on ubuntu@worker5 took too long to 
> exit
> 
> test_id:
> 2016-06-20--001.kafkatest.tests.streams.streams_smoke_test.StreamsSmokeTest.test_streams
> status: FAIL
> run time:   4 minutes 36.426 seconds
> Streams Smoke Test process on ubuntu@worker9 took too long to exit
> Traceback (most recent call last):
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
>  line 106, in run_all_tests
> data = self.run_single_test()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
>  line 162, in run_single_test
> return self.current_test_context.function(self.current_test)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/streams/streams_smoke_test.py",
>  line 67, in test_streams
> self.driver.wait()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/services/streams.py",
>  line 94, in wait
> wait_until(lambda: not node.account.alive(pid), timeout_sec=120, 
> err_msg="Streams Smoke Test process on " + str(node.account) + " took too 
> long to exit")
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/utils/util.py",
>  line 36, in wait_until
> raise TimeoutError(err_msg)
> TimeoutError: Streams Smoke Test process on ubuntu@worker9 took too long to 
> exit
> {code}
> https://jenkins.confluent.io/job/system-test-kafka/255/console



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] KIP-62: Allow consumer to send heartbeats from a background thread

[Jason Gustafson]

Thanks Guozhang. I missed Jun's question.

@Ismael Done.

On Tue, Jun 21, 2016 at 2:37 PM, Guozhang Wang  wrote:

> Hello Jun,
>
> Just clarifying, it will be using the max.poll.interval.ms config, in the
> wiki we use the term "process timeout" for it which exposed in the consumer
> configs as "max.poll.interval.ms". I have updated the wiki to make it more
> clear.
>
> Guozhang
>
>
> On Tue, Jun 21, 2016 at 2:14 PM, Jason Gustafson 
> wrote:
>
> > Hi All,
> >
> > Looks like the vote has passed with +6 binding and +5 non-binding. Thanks
> > everyone for help reviewing the proposal.
> >
> > -Jason
> >
> > On Mon, Jun 20, 2016 at 2:04 PM, Jay Kreps  wrote:
> >
> > > +1
> > >
> > > -Jay
> > >
> > > On Mon, Jun 20, 2016 at 1:39 PM, Jason Gustafson 
> > > wrote:
> > >
> > > > Hi All, I've changed the default max.poll.interval.ms in the KIP to
> 5
> > > > minutes. Unless there are any objections, perhaps we can skip the
> > revote
> > > > since this is a small change. In any case, I'll leave the vote open
> for
> > > > another day.
> > > >
> > > > Thanks,
> > > > Jason
> > > >
> > > > On Mon, Jun 20, 2016 at 12:39 PM, Ismael Juma 
> > wrote:
> > > >
> > > > > On Mon, Jun 20, 2016 at 9:32 PM, Jay Kreps 
> wrote:
> > > > >
> > > > > > Also, checked exceptions? Really Ewen??? :-)
> > > > > >
> > > > >
> > > > > Haha, yeah. I thought checked exceptions were universally disliked.
> > > > People
> > > > > who favour static typing tend to prefer Disjunction/Either and the
> > rest
> > > > > tend to prefer unchecked exceptions.
> > > > >
> > > > > Ismael
> > > > >
> > > >
> > >
> >
>
>
>
> --
> -- Guozhang
>

[jira] [Assigned] (KAFKA-3846) Connect record types should include timestamps

[Shikhar Bhushan (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3846?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Shikhar Bhushan reassigned KAFKA-3846:
--

Assignee: Shikhar Bhushan  (was: Ewen Cheslack-Postava)

> Connect record types should include timestamps
> --
>
> Key: KAFKA-3846
> URL: https://issues.apache.org/jira/browse/KAFKA-3846
> Project: Kafka
>  Issue Type: Improvement
>  Components: KafkaConnect
>Affects Versions: 0.10.0.0
>Reporter: Ewen Cheslack-Postava
>Assignee: Shikhar Bhushan
>  Labels: needs-kip
> Fix For: 0.10.1.0
>
>
> Timestamps were added to records in the previous release, however this does 
> not get propagated automatically to Connect because it uses custom wrappers  
> to add fields and rename some for clarity.
> The addition of timestamps should be trivial, but can be really useful (e.g. 
> in sink connectors that would like to include timestamp info if available but 
> when it is not stored in the value).
> This is public API so it will need a KIP despite being very uncontentious.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3887) StreamBounceTest.test_bounce and StreamSmokeTest.test_streams failing

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3887?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15342841#comment-15342841
 ] 

Ismael Juma commented on KAFKA-3887:


First build to fail: 
http://confluent-kafka-system-test-results.s3-us-west-2.amazonaws.com/2016-06-11--001.1465663177--apache--trunk--da85171/report.html

After that, it's been failing almost every build.

> StreamBounceTest.test_bounce and StreamSmokeTest.test_streams failing
> -
>
> Key: KAFKA-3887
> URL: https://issues.apache.org/jira/browse/KAFKA-3887
> Project: Kafka
>  Issue Type: Bug
>  Components: streams, system tests
>Reporter: Ismael Juma
>Assignee: Eno Thereska
>  Labels: transient-system-test-failure
>
> StreamBounceTest.test_bounce and StreamSmokeTest.test_streams has been 
> failing semi-regularly. Output from the latest failure:
> {code}
> 
> test_id:
> 2016-06-20--001.kafkatest.tests.streams.streams_bounce_test.StreamsBounceTest.test_bounce
> status: FAIL
> run time:   4 minutes 13.916 seconds
> Streams Smoke Test process on ubuntu@worker5 took too long to exit
> Traceback (most recent call last):
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
>  line 106, in run_all_tests
> data = self.run_single_test()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
>  line 162, in run_single_test
> return self.current_test_context.function(self.current_test)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/streams/streams_bounce_test.py",
>  line 67, in test_bounce
> self.driver.wait()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/services/streams.py",
>  line 94, in wait
> wait_until(lambda: not node.account.alive(pid), timeout_sec=120, 
> err_msg="Streams Smoke Test process on " + str(node.account) + " took too 
> long to exit")
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/utils/util.py",
>  line 36, in wait_until
> raise TimeoutError(err_msg)
> TimeoutError: Streams Smoke Test process on ubuntu@worker5 took too long to 
> exit
> 
> test_id:
> 2016-06-20--001.kafkatest.tests.streams.streams_smoke_test.StreamsSmokeTest.test_streams
> status: FAIL
> run time:   4 minutes 36.426 seconds
> Streams Smoke Test process on ubuntu@worker9 took too long to exit
> Traceback (most recent call last):
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
>  line 106, in run_all_tests
> data = self.run_single_test()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
>  line 162, in run_single_test
> return self.current_test_context.function(self.current_test)
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/streams/streams_smoke_test.py",
>  line 67, in test_streams
> self.driver.wait()
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/services/streams.py",
>  line 94, in wait
> wait_until(lambda: not node.account.alive(pid), timeout_sec=120, 
> err_msg="Streams Smoke Test process on " + str(node.account) + " took too 
> long to exit")
>   File 
> "/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/utils/util.py",
>  line 36, in wait_until
> raise TimeoutError(err_msg)
> TimeoutError: Streams Smoke Test process on ubuntu@worker9 took too long to 
> exit
> {code}
> https://jenkins.confluent.io/job/system-test-kafka/255/console



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3888) Allow consumer to send heartbeats in background thread (KIP-62)

[Jason Gustafson (JIRA)]

Jason Gustafson created KAFKA-3888:
--

 Summary: Allow consumer to send heartbeats in background thread 
(KIP-62)
 Key: KAFKA-3888
 URL: https://issues.apache.org/jira/browse/KAFKA-3888
 Project: Kafka
  Issue Type: Improvement
  Components: consumer
Affects Versions: 0.10.1.0
Reporter: Jason Gustafson
Assignee: Jason Gustafson


This ticket covers the implementation of KIP-62 as documented here: 
https://cwiki.apache.org/confluence/display/KAFKA/KIP-62%3A+Allow+consumer+to+send+heartbeats+from+a+background+thread.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [VOTE] KIP-62: Allow consumer to send heartbeats from a background thread

[Guozhang Wang]

Hello Jun,

Just clarifying, it will be using the max.poll.interval.ms config, in the
wiki we use the term "process timeout" for it which exposed in the consumer
configs as "max.poll.interval.ms". I have updated the wiki to make it more
clear.

Guozhang


On Tue, Jun 21, 2016 at 2:14 PM, Jason Gustafson  wrote:

> Hi All,
>
> Looks like the vote has passed with +6 binding and +5 non-binding. Thanks
> everyone for help reviewing the proposal.
>
> -Jason
>
> On Mon, Jun 20, 2016 at 2:04 PM, Jay Kreps  wrote:
>
> > +1
> >
> > -Jay
> >
> > On Mon, Jun 20, 2016 at 1:39 PM, Jason Gustafson 
> > wrote:
> >
> > > Hi All, I've changed the default max.poll.interval.ms in the KIP to 5
> > > minutes. Unless there are any objections, perhaps we can skip the
> revote
> > > since this is a small change. In any case, I'll leave the vote open for
> > > another day.
> > >
> > > Thanks,
> > > Jason
> > >
> > > On Mon, Jun 20, 2016 at 12:39 PM, Ismael Juma 
> wrote:
> > >
> > > > On Mon, Jun 20, 2016 at 9:32 PM, Jay Kreps  wrote:
> > > >
> > > > > Also, checked exceptions? Really Ewen??? :-)
> > > > >
> > > >
> > > > Haha, yeah. I thought checked exceptions were universally disliked.
> > > People
> > > > who favour static typing tend to prefer Disjunction/Either and the
> rest
> > > > tend to prefer unchecked exceptions.
> > > >
> > > > Ismael
> > > >
> > >
> >
>



-- 
-- Guozhang

Re: [VOTE] KIP-62: Allow consumer to send heartbeats from a background thread

[Ismael Juma]

Awesome, please update the KIP page Jason. :)

Ismael

On Tue, Jun 21, 2016 at 11:14 PM, Jason Gustafson 
wrote:

> Hi All,
>
> Looks like the vote has passed with +6 binding and +5 non-binding. Thanks
> everyone for help reviewing the proposal.
>
> -Jason
>
> On Mon, Jun 20, 2016 at 2:04 PM, Jay Kreps  wrote:
>
> > +1
> >
> > -Jay
> >
> > On Mon, Jun 20, 2016 at 1:39 PM, Jason Gustafson 
> > wrote:
> >
> > > Hi All, I've changed the default max.poll.interval.ms in the KIP to 5
> > > minutes. Unless there are any objections, perhaps we can skip the
> revote
> > > since this is a small change. In any case, I'll leave the vote open for
> > > another day.
> > >
> > > Thanks,
> > > Jason
> > >
> > > On Mon, Jun 20, 2016 at 12:39 PM, Ismael Juma 
> wrote:
> > >
> > > > On Mon, Jun 20, 2016 at 9:32 PM, Jay Kreps  wrote:
> > > >
> > > > > Also, checked exceptions? Really Ewen??? :-)
> > > > >
> > > >
> > > > Haha, yeah. I thought checked exceptions were universally disliked.
> > > People
> > > > who favour static typing tend to prefer Disjunction/Either and the
> rest
> > > > tend to prefer unchecked exceptions.
> > > >
> > > > Ismael
> > > >
> > >
> >
>

Jenkins build is back to normal : kafka-trunk-jdk7 #1383

[Apache Jenkins Server]

See 

Re: KafkaConsumer - Poll consumes more records than needed

[Jason Gustafson]

Hi there,

>From 0.10 forward, the consumer supports a configuration option
"max.poll.records" to limit the number of records returned from each call
to poll(). Note also that if you use commitSync or commitAsync (with no
arguments), the consumer assumes that you intend to commit offsets for all
records returned from the previous poll. If that is not the case, you can
also pass the exact offsets you need to commit using the overloaded
versions of those APIs.

Thanks,
Jason

On Tue, Jun 21, 2016 at 5:31 AM, Asier Aranbarri Beldarrain <
aranbar...@gmail.com> wrote:

> Hi everyone! I'm new to this mailing list; Hope I can find solutions and
> who knows, maybe some day help other Kafka developers.
>
> I'm facing an issue regarding the new (>=0.9) KafkaConsumer and its poll()
> method:
>
> What would be the correct way to specify a maximum amount of records to be
> processed without consuming more messages than needed?
>
> For example, I need to consume 10 and just 10 messages from a Kafka topic.
> I start the consumer, and the poll method gets 15 messages. I process 10 of
> them and then manually commit the offset.
>
> BUT:
>
> If I start the consumer again, it will start consuming from the 16th
> message, not the 11th. This is a huge problem for me, as I need a reader
> that consumes an specific number of messages.
>
> In Kafka 0.8, this was easy since you could use the Stream Iterator to
> process the messages and stop in the exact moment you need to. But polling
> isn't exact, so the consumer could consume a higher amount of records that
> I need to process.
>
> Any way to solve this? Any help would be kindly appreciated.
>
> Thanks in advance!
>

Re: [VOTE] KIP-62: Allow consumer to send heartbeats from a background thread

[Jason Gustafson]

Hi All,

Looks like the vote has passed with +6 binding and +5 non-binding. Thanks
everyone for help reviewing the proposal.

-Jason

On Mon, Jun 20, 2016 at 2:04 PM, Jay Kreps  wrote:

> +1
>
> -Jay
>
> On Mon, Jun 20, 2016 at 1:39 PM, Jason Gustafson 
> wrote:
>
> > Hi All, I've changed the default max.poll.interval.ms in the KIP to 5
> > minutes. Unless there are any objections, perhaps we can skip the revote
> > since this is a small change. In any case, I'll leave the vote open for
> > another day.
> >
> > Thanks,
> > Jason
> >
> > On Mon, Jun 20, 2016 at 12:39 PM, Ismael Juma  wrote:
> >
> > > On Mon, Jun 20, 2016 at 9:32 PM, Jay Kreps  wrote:
> > >
> > > > Also, checked exceptions? Really Ewen??? :-)
> > > >
> > >
> > > Haha, yeah. I thought checked exceptions were universally disliked.
> > People
> > > who favour static typing tend to prefer Disjunction/Either and the rest
> > > tend to prefer unchecked exceptions.
> > >
> > > Ismael
> > >
> >
>

[jira] [Resolved] (KAFKA-3872) OOM while running Kafka Streams integration tests

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-3872.

   Resolution: Fixed
Fix Version/s: 0.10.1.0

Issue resolved by pull request 1533
[https://github.com/apache/kafka/pull/1533]

> OOM while running Kafka Streams integration tests
> -
>
> Key: KAFKA-3872
> URL: https://issues.apache.org/jira/browse/KAFKA-3872
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams, unit tests
>Reporter: Ismael Juma
>Assignee: Eno Thereska
>  Labels: transient-unit-test-failure
> Fix For: 0.10.1.0
>
>
> Failures:
> org.apache.kafka.streams.integration.FanoutIntegrationTest.classMethod
> org.apache.kafka.streams.integration.WordCountIntegrationTest.classMethod
> Stracktrace:
> {code}
> java.lang.OutOfMemoryError: Java heap space
>   at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57)
>   at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
>   at kafka.log.SkimpyOffsetMap.(OffsetMap.scala:43)
>   at kafka.log.LogCleaner$CleanerThread.(LogCleaner.scala:193)
>   at kafka.log.LogCleaner$$anonfun$1.apply(LogCleaner.scala:83)
>   at kafka.log.LogCleaner$$anonfun$1.apply(LogCleaner.scala:83)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at scala.collection.immutable.Range.foreach(Range.scala:141)
>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
>   at scala.collection.AbstractTraversable.map(Traversable.scala:105)
>   at kafka.log.LogCleaner.(LogCleaner.scala:83)
>   at kafka.log.LogManager.(LogManager.scala:66)
>   at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:609)
>   at kafka.server.KafkaServer.startup(KafkaServer.scala:183)
>   at kafka.utils.TestUtils$.createServer(TestUtils.scala:132)
>   at kafka.utils.TestUtils.createServer(TestUtils.scala)
>   at 
> org.apache.kafka.streams.integration.utils.KafkaEmbedded.(KafkaEmbedded.java:79)
>   at 
> org.apache.kafka.streams.integration.utils.EmbeddedSingleNodeKafkaCluster.start(EmbeddedSingleNodeKafkaCluster.java:54)
>   at 
> {code}
> Two builds:
> https://builds.apache.org/job/kafka-trunk-jdk8/704/
> https://builds.apache.org/job/kafka-trunk-jdk8/705



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1533: KAFKA-3872: Reduce log cleaner buffer size to 2 MB

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1533


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3872) OOM while running Kafka Streams integration tests

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15342659#comment-15342659
 ] 

ASF GitHub Bot commented on KAFKA-3872:
---

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1533


> OOM while running Kafka Streams integration tests
> -
>
> Key: KAFKA-3872
> URL: https://issues.apache.org/jira/browse/KAFKA-3872
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams, unit tests
>Reporter: Ismael Juma
>Assignee: Eno Thereska
>  Labels: transient-unit-test-failure
> Fix For: 0.10.1.0
>
>
> Failures:
> org.apache.kafka.streams.integration.FanoutIntegrationTest.classMethod
> org.apache.kafka.streams.integration.WordCountIntegrationTest.classMethod
> Stracktrace:
> {code}
> java.lang.OutOfMemoryError: Java heap space
>   at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57)
>   at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
>   at kafka.log.SkimpyOffsetMap.(OffsetMap.scala:43)
>   at kafka.log.LogCleaner$CleanerThread.(LogCleaner.scala:193)
>   at kafka.log.LogCleaner$$anonfun$1.apply(LogCleaner.scala:83)
>   at kafka.log.LogCleaner$$anonfun$1.apply(LogCleaner.scala:83)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at scala.collection.immutable.Range.foreach(Range.scala:141)
>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
>   at scala.collection.AbstractTraversable.map(Traversable.scala:105)
>   at kafka.log.LogCleaner.(LogCleaner.scala:83)
>   at kafka.log.LogManager.(LogManager.scala:66)
>   at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:609)
>   at kafka.server.KafkaServer.startup(KafkaServer.scala:183)
>   at kafka.utils.TestUtils$.createServer(TestUtils.scala:132)
>   at kafka.utils.TestUtils.createServer(TestUtils.scala)
>   at 
> org.apache.kafka.streams.integration.utils.KafkaEmbedded.(KafkaEmbedded.java:79)
>   at 
> org.apache.kafka.streams.integration.utils.EmbeddedSingleNodeKafkaCluster.start(EmbeddedSingleNodeKafkaCluster.java:54)
>   at 
> {code}
> Two builds:
> https://builds.apache.org/job/kafka-trunk-jdk8/704/
> https://builds.apache.org/job/kafka-trunk-jdk8/705



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3887) StreamBounceTest.test_bounce and StreamSmokeTest.test_streams failing

[Ismael Juma (JIRA)]

Ismael Juma created KAFKA-3887:
--

 Summary: StreamBounceTest.test_bounce and 
StreamSmokeTest.test_streams failing
 Key: KAFKA-3887
 URL: https://issues.apache.org/jira/browse/KAFKA-3887
 Project: Kafka
  Issue Type: Bug
  Components: streams, system tests
Reporter: Ismael Juma
Assignee: Eno Thereska


StreamBounceTest.test_bounce and StreamSmokeTest.test_streams has been failing 
semi-regularly. Output from the latest failure:

{code}

test_id:
2016-06-20--001.kafkatest.tests.streams.streams_bounce_test.StreamsBounceTest.test_bounce
status: FAIL
run time:   4 minutes 13.916 seconds


Streams Smoke Test process on ubuntu@worker5 took too long to exit
Traceback (most recent call last):
  File 
"/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
 line 106, in run_all_tests
data = self.run_single_test()
  File 
"/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
 line 162, in run_single_test
return self.current_test_context.function(self.current_test)
  File 
"/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/streams/streams_bounce_test.py",
 line 67, in test_bounce
self.driver.wait()
  File 
"/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/services/streams.py",
 line 94, in wait
wait_until(lambda: not node.account.alive(pid), timeout_sec=120, 
err_msg="Streams Smoke Test process on " + str(node.account) + " took too long 
to exit")
  File 
"/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/utils/util.py",
 line 36, in wait_until
raise TimeoutError(err_msg)
TimeoutError: Streams Smoke Test process on ubuntu@worker5 took too long to exit


test_id:
2016-06-20--001.kafkatest.tests.streams.streams_smoke_test.StreamsSmokeTest.test_streams
status: FAIL
run time:   4 minutes 36.426 seconds


Streams Smoke Test process on ubuntu@worker9 took too long to exit
Traceback (most recent call last):
  File 
"/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
 line 106, in run_all_tests
data = self.run_single_test()
  File 
"/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/tests/runner.py",
 line 162, in run_single_test
return self.current_test_context.function(self.current_test)
  File 
"/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/tests/streams/streams_smoke_test.py",
 line 67, in test_streams
self.driver.wait()
  File 
"/var/lib/jenkins/workspace/system-test-kafka/kafka/tests/kafkatest/services/streams.py",
 line 94, in wait
wait_until(lambda: not node.account.alive(pid), timeout_sec=120, 
err_msg="Streams Smoke Test process on " + str(node.account) + " took too long 
to exit")
  File 
"/var/lib/jenkins/workspace/system-test-kafka/kafka/venv/local/lib/python2.7/site-packages/ducktape-0.5.1-py2.7.egg/ducktape/utils/util.py",
 line 36, in wait_until
raise TimeoutError(err_msg)
TimeoutError: Streams Smoke Test process on ubuntu@worker9 took too long to exit
{code}

https://jenkins.confluent.io/job/system-test-kafka/255/console



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3263) Add Markdown support for ConfigDef

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3263?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15342595#comment-15342595
 ] 

ASF GitHub Bot commented on KAFKA-3263:
---

Github user jcustenborder closed the pull request at:

https://github.com/apache/kafka/pull/952


> Add Markdown support for ConfigDef
> --
>
> Key: KAFKA-3263
> URL: https://issues.apache.org/jira/browse/KAFKA-3263
> Project: Kafka
>  Issue Type: Improvement
>  Components: clients
>Affects Versions: 0.9.0.1
>Reporter: Jeremy Custenborder
>Priority: Minor
>
> The ability to output markdown for ConfigDef would be nice given a lot of 
> people use README.md files in their repositories.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #952: KAFKA-3263 - Support for markdown generation.

[jcustenborder]

Github user jcustenborder closed the pull request at:

https://github.com/apache/kafka/pull/952


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-63: Unify store and downstream caching in streams

[Roger Hoover]

Thanks, Eno.

On Tue, Jun 21, 2016 at 2:22 AM, Eno Thereska 
wrote:

> Hi Roger,
>
> I realised I never got back to you on this one, sorry. Some answers inline:
>
> > On 3 Jun 2016, at 22:48, Roger Hoover  wrote:
> >
> > Hi Eno,
> >
> > Does this mean that Kafka Streams will disable the RocksDB write buffer?
>
> For the purposes of this KIP we might not want to change the current
> RocksDB state of things. However, Guozhang has written about future plans
> in the memory management page <
> https://cwiki.apache.org/confluence/display/KAFKA/Discussion:+Memory+Management+in+Kafka+Streams>
> in the wiki.
>
>
> > Is it currently safe to recover a Kafka Streams application after SIGKILL
> > on the same machine?  If not, will this make it safe to do so?
> >
> > If RocksDB is not flushed before offsets are commited in Kafka and is
> > killed with SIGKILL, will the data in the write buffer be lost (since
> Kafka
> > Streams disables the transaction log)?  That data will be present in the
> > Kafka changelog but will it get applied to the recovered RocksDB database
> > on restart?
>
> These are good questions on failure modes. This KIP will not change the
> failure behaviour from what it currently is. I believe there will be
> subsequent KIPs where the issues around writing atomically to multiple
> places will be considered, and failure semantics will be strengthened. Stay
> tuned.
>
> Thanks
> Eno
>
>
>
> >
> > Thanks,
> >
> > Roger
> >
> > On Fri, Jun 3, 2016 at 2:39 PM, Eno Thereska 
> wrote:
> >
> >> Hi Gwen,
> >>
> >> Yes. As an example, if cache.max.bytes.buffering set to X, and if users
> >> have A aggregation operators and T KTable.to() operators, then X*(A + T)
> >> total bytes will be allocated for caching.
> >>
> >> Eno
> >>
> >>> On 3 Jun 2016, at 21:37, Gwen Shapira  wrote:
> >>>
> >>> Just to clarify: "cache.max.bytes.buffering" is per processor?
> >>>
> >>>
> >>> On Thu, Jun 2, 2016 at 11:30 AM, Eno Thereska 
> >> wrote:
>  Hi there,
> 
>  I have created KIP-63: Unify store and downstream caching in streams
> 
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-63%3A+Unify+store+and+downstream+caching+in+streams
> >> <
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/KIP-63:+Unify+store+and+downstream+caching+in+streams
> >>>
> 
> 
>  Feedback is appreciated.
> 
>  Thank you
>  Eno
> >>
> >>
>
>

[jira] [Updated] (KAFKA-3886) Consumer should handle wakeups while rebalancing more gracefully

[Jason Gustafson (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3886?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Jason Gustafson updated KAFKA-3886:
---
Description: 
If the user calls wakeup() while a rebalance in progress, we currently lose 
track of the state of that rebalance. In the worst case, this can result in an 
additional unneeded rebalance when the user calls poll() again. 

The other thing that can happen is that the rebalance could complete inside 
another blocking call (e.g. {{commitSync()}}). There may be scenarios where 
this can cause us to commit offsets outside the generation an assignment is 
valid for. For example: 

1. Consumer is initially assigned partition A
2. The consumer starts rebalancing, but is interrupted with a call to wakeup().
3. User calls commitSync with offsets (A, 5)
4. Before offset commit is sent, an interrupted rebalance completes and changes 
the assignment to include only partition B.
5. Now we proceed with the unsafe offset commit on partition A.

In this case, we should probably ensure that it is not possible to commit 
offsets after an assignment has been revoked. Other cases, such as position(), 
may be handled similarly.

  was:
If the user calls wakeup() while a rebalance in progress, we currently lose 
track of the state of that rebalance. In the worst case, this can result in an 
additional unneeded rebalance when the user calls poll() again. 

The other thing that can happen is that the rebalance could complete inside 
another blocking call (e.g. {{commitSync()}}). There may be scenarios where 
this can cause us to commit offsets outside. For example: 

1. Consumer is initially assigned partition A
2. The consumer starts rebalancing, but is interrupted with a call to wakeup().
3. User calls commitSync with offsets (A, 5)
4. Before offset commit is sent, an interrupted rebalance completes and changes 
the assignment to include only partition B.
5. Now we proceed with the unsafe offset commit on partition A.

In this case, we should probably ensure that it is not possible to commit 
offsets after an assignment has been revoked. Other cases, such as position(), 
may be handled similarly.


> Consumer should handle wakeups while rebalancing more gracefully
> 
>
> Key: KAFKA-3886
> URL: https://issues.apache.org/jira/browse/KAFKA-3886
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
>Affects Versions: 0.10.0.0
>Reporter: Jason Gustafson
>Assignee: Jason Gustafson
>
> If the user calls wakeup() while a rebalance in progress, we currently lose 
> track of the state of that rebalance. In the worst case, this can result in 
> an additional unneeded rebalance when the user calls poll() again. 
> The other thing that can happen is that the rebalance could complete inside 
> another blocking call (e.g. {{commitSync()}}). There may be scenarios where 
> this can cause us to commit offsets outside the generation an assignment is 
> valid for. For example: 
> 1. Consumer is initially assigned partition A
> 2. The consumer starts rebalancing, but is interrupted with a call to 
> wakeup().
> 3. User calls commitSync with offsets (A, 5)
> 4. Before offset commit is sent, an interrupted rebalance completes and 
> changes the assignment to include only partition B.
> 5. Now we proceed with the unsafe offset commit on partition A.
> In this case, we should probably ensure that it is not possible to commit 
> offsets after an assignment has been revoked. Other cases, such as 
> position(), may be handled similarly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3886) Consumer should handle wakeups while rebalancing more gracefully

[Jason Gustafson (JIRA)]

Jason Gustafson created KAFKA-3886:
--

 Summary: Consumer should handle wakeups while rebalancing more 
gracefully
 Key: KAFKA-3886
 URL: https://issues.apache.org/jira/browse/KAFKA-3886
 Project: Kafka
  Issue Type: Bug
  Components: consumer
Affects Versions: 0.10.0.0
Reporter: Jason Gustafson
Assignee: Jason Gustafson


If the user calls wakeup() while a rebalance in progress, we currently lose 
track of the state of that rebalance. In the worst case, this can result in an 
additional unneeded rebalance when the user calls poll() again. 

The other thing that can happen is that the rebalance could complete inside 
another blocking call (e.g. {{commitSync()}}). There may be scenarios where 
this can cause us to commit offsets outside. For example: 

1. Consumer is initially assigned partition A
2. The consumer starts rebalancing, but is interrupted with a call to wakeup().
3. User calls commitSync with offsets (A, 5)
4. Before offset commit is sent, an interrupted rebalance completes and changes 
the assignment to include only partition B.
5. Now we proceed with the unsafe offset commit on partition A.

In this case, we should probably ensure that it is not possible to commit 
offsets after an assignment has been revoked. Other cases, such as position(), 
may be handled similarly.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1534: MINOR: update streams.html with KStream API change...

[asfgit]

Github user asfgit closed the pull request at:

https://github.com/apache/kafka/pull/1534


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3176) Allow console consumer to consume from particular partitions when new consumer is used.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15342372#comment-15342372
 ] 

ASF GitHub Bot commented on KAFKA-3176:
---

GitHub user vahidhashemian reopened a pull request:

https://github.com/apache/kafka/pull/1536

MINOR: KAFKA-3176 Follow-up to fix minor issues



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vahidhashemian/kafka minor/KAFKA-3176-Followup

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1536.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1536


commit 843026de9c224235fe46f3fc0e882cfd4f4fbf15
Author: Vahid Hashemian 
Date:   2016-06-21T17:36:53Z

MINOR: KAFKA-3176 Follow-up to fix minor issues




> Allow console consumer to consume from particular partitions when new 
> consumer is used.
> ---
>
> Key: KAFKA-3176
> URL: https://issues.apache.org/jira/browse/KAFKA-3176
> Project: Kafka
>  Issue Type: New Feature
>  Components: tools
>Affects Versions: 0.9.0.0
>Reporter: Jiangjie Qin
>Assignee: Vahid Hashemian
> Fix For: 0.10.1.0
>
>
> Previously we have simple consumer shell which can consume from a particular 
> partition. Moving forward we will deprecate simple consumer, it would be 
> useful to allow console consumer to consumer from a particular partition when 
> new consumer is used.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3176) Allow console consumer to consume from particular partitions when new consumer is used.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15342371#comment-15342371
 ] 

ASF GitHub Bot commented on KAFKA-3176:
---

Github user vahidhashemian closed the pull request at:

https://github.com/apache/kafka/pull/1536


> Allow console consumer to consume from particular partitions when new 
> consumer is used.
> ---
>
> Key: KAFKA-3176
> URL: https://issues.apache.org/jira/browse/KAFKA-3176
> Project: Kafka
>  Issue Type: New Feature
>  Components: tools
>Affects Versions: 0.9.0.0
>Reporter: Jiangjie Qin
>Assignee: Vahid Hashemian
> Fix For: 0.10.1.0
>
>
> Previously we have simple consumer shell which can consume from a particular 
> partition. Moving forward we will deprecate simple consumer, it would be 
> useful to allow console consumer to consumer from a particular partition when 
> new consumer is used.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1536: MINOR: KAFKA-3176 Follow-up to fix minor issues

[vahidhashemian]

GitHub user vahidhashemian reopened a pull request:

https://github.com/apache/kafka/pull/1536

MINOR: KAFKA-3176 Follow-up to fix minor issues



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vahidhashemian/kafka minor/KAFKA-3176-Followup

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1536.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1536


commit 843026de9c224235fe46f3fc0e882cfd4f4fbf15
Author: Vahid Hashemian 
Date:   2016-06-21T17:36:53Z

MINOR: KAFKA-3176 Follow-up to fix minor issues




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[GitHub] kafka pull request #1536: MINOR: KAFKA-3176 Follow-up to fix minor issues

[vahidhashemian]

Github user vahidhashemian closed the pull request at:

https://github.com/apache/kafka/pull/1536


---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3176) Allow console consumer to consume from particular partitions when new consumer is used.

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15342313#comment-15342313
 ] 

ASF GitHub Bot commented on KAFKA-3176:
---

GitHub user vahidhashemian opened a pull request:

https://github.com/apache/kafka/pull/1536

MINOR: KAFKA-3176 Follow-up to fix minor issues



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vahidhashemian/kafka minor/KAFKA-3176-Followup

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1536.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1536


commit 843026de9c224235fe46f3fc0e882cfd4f4fbf15
Author: Vahid Hashemian 
Date:   2016-06-21T17:36:53Z

MINOR: KAFKA-3176 Follow-up to fix minor issues




> Allow console consumer to consume from particular partitions when new 
> consumer is used.
> ---
>
> Key: KAFKA-3176
> URL: https://issues.apache.org/jira/browse/KAFKA-3176
> Project: Kafka
>  Issue Type: New Feature
>  Components: tools
>Affects Versions: 0.9.0.0
>Reporter: Jiangjie Qin
>Assignee: Vahid Hashemian
> Fix For: 0.10.1.0
>
>
> Previously we have simple consumer shell which can consume from a particular 
> partition. Moving forward we will deprecate simple consumer, it would be 
> useful to allow console consumer to consumer from a particular partition when 
> new consumer is used.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1536: MINOR: KAFKA-3176 Follow-up to fix minor issues

[vahidhashemian]

GitHub user vahidhashemian opened a pull request:

https://github.com/apache/kafka/pull/1536

MINOR: KAFKA-3176 Follow-up to fix minor issues



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/vahidhashemian/kafka minor/KAFKA-3176-Followup

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1536.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1536


commit 843026de9c224235fe46f3fc0e882cfd4f4fbf15
Author: Vahid Hashemian 
Date:   2016-06-21T17:36:53Z

MINOR: KAFKA-3176 Follow-up to fix minor issues




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Commented] (KAFKA-3727) Consumer.poll() stuck in loop on non-existent topic manually assigned

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3727?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15342295#comment-15342295
 ] 

ASF GitHub Bot commented on KAFKA-3727:
---

GitHub user edoardocomar opened a pull request:

https://github.com/apache/kafka/pull/1535

KAFKA-3727 - ConsumerListener for UnknownTopicOrPartitionException

Added a ConsumerListener to KafkaConsumer

Modified Fetcher and Cluster to account for Unknown Topics

Added unit test

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/edoardocomar/kafka KAFKA-3727b

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1535.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1535


commit 68515ec1d945c438ebd0b41a64406eac5c971dbb
Author: Edoardo Comar 
Date:   2016-06-20T22:16:23Z

KAFKA-3727 - ConsumerListener for UnknownTopicOrPartitionException

Added a ConsumerListener to KafkaConsumer

Modified Fetcher and Cluster to account for Unknown Topics

Added unit test




> Consumer.poll() stuck in loop on non-existent topic manually assigned
> -
>
> Key: KAFKA-3727
> URL: https://issues.apache.org/jira/browse/KAFKA-3727
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Reporter: Edoardo Comar
>Assignee: Edoardo Comar
>Priority: Critical
>
> The behavior of a consumer on poll() for a non-existing topic is surprisingly 
> different/inconsistent 
> between a consumer that subscribed to the topic and one that had the 
> topic-partition manually assigned.
> The "subscribed" consumer will return an empty collection
> The "assigned" consumer will *loop forever* - this feels a bug to me.
> sample snippet to reproduce:
> {quote}
> KafkaConsumer assignKc = new KafkaConsumer<>(props1);
> KafkaConsumer subsKc = new KafkaConsumer<>(props2);
> List tps = new ArrayList<>();
> tps.add(new TopicPartition("topic-not-exists", 0));
> assignKc.assign(tps);
> subsKc.subscribe(Arrays.asList("topic-not-exists"));
> System.out.println("* subscribe k consumer ");
> ConsumerRecords crs2 = subsKc.poll(1000L); 
> print("subscribeKc", crs2); // returns empty
> System.out.println("* assign k consumer ");
> ConsumerRecords crs1 = assignKc.poll(1000L); 
>// will loop forever ! 
> print("assignKc", crs1);
> {quote}
> the logs for the "assigned" consumer show:
> [2016-05-18 17:33:09,907] DEBUG Updated cluster metadata version 8 to 
> Cluster(nodes = [192.168.10.18:9093 (id: 0 rack: null)], partitions = []) 
> (org.apache.kafka.clients.Metadata)
> [2016-05-18 17:33:09,908] DEBUG Partition topic-not-exists-0 is unknown for 
> fetching offset, wait for metadata refresh 
> (org.apache.kafka.clients.consumer.internals.Fetcher)
> [2016-05-18 17:33:10,010] DEBUG Sending metadata request 
> {topics=[topic-not-exists]} to node 0 (org.apache.kafka.clients.NetworkClient)
> [2016-05-18 17:33:10,011] WARN Error while fetching metadata with correlation 
> id 9 : {topic-not-exists=UNKNOWN_TOPIC_OR_PARTITION} 
> (org.apache.kafka.clients.NetworkClient)



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1535: KAFKA-3727 - ConsumerListener for UnknownTopicOrPa...

[edoardocomar]

GitHub user edoardocomar opened a pull request:

https://github.com/apache/kafka/pull/1535

KAFKA-3727 - ConsumerListener for UnknownTopicOrPartitionException

Added a ConsumerListener to KafkaConsumer

Modified Fetcher and Cluster to account for Unknown Topics

Added unit test

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/edoardocomar/kafka KAFKA-3727b

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1535.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1535


commit 68515ec1d945c438ebd0b41a64406eac5c971dbb
Author: Edoardo Comar 
Date:   2016-06-20T22:16:23Z

KAFKA-3727 - ConsumerListener for UnknownTopicOrPartitionException

Added a ConsumerListener to KafkaConsumer

Modified Fetcher and Cluster to account for Unknown Topics

Added unit test




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [DISCUSS] KIP-48 Support for delegation tokens as an authentication mechanism

[parth brahmbhatt]

1. Who / how are tokens renewed? By original requester only? or using Kerberos
auth only?
My recommendation is to do this only using Kerberos auth and only threw the
renewer specified during the acquisition request.

2. Are tokens stored on each broker or in ZK?
My recommendation is still to store in ZK or not store them at all. The
whole controller based distribution is too much overhead with not much to
achieve.

3. How are tokens invalidated / expired?
Either by expiration time out or through an explicit request to invalidate.

4. Which encryption algorithm is used?
SCRAM

5. What is the impersonation proposal (it wasn't in the KIP but was discussed
in this thread)?
There is no imperonation proposal. I tried and explained how its a
different problem and why its not really necessary to discuss that as part
of this KIP.  This KIP will not support any impersonation, it will just be
another way to authenticate.

6. Do we need new ACLs, if so - for what actions?
We do not need new ACLs.

7. How would the delegation token be configured in the client?
Should be through config. I wasn't planning on supporting JAAS for tokens.
I don't believe hadoop does this either.

Thanks
Parth



On Thu, Jun 16, 2016 at 4:03 PM, Jun Rao  wrote:

> Harsha,
>
> Another question.
>
> 9. How would the delegation token be configured in the client? The standard
> way is to do this through JAAS. However, we will need to think through if
> this is convenient in a shared environment. For example, when a new task is
> added to a Storm worker node, do we need to dynamically add a new section
> in the JAAS file? It may be more convenient if we can pass in the token
> through the config directly w/o going through JAAS.
>
> Are you or Parth still actively working on this KIP?
>
> Thanks,
>
> Jun
>
>
>
> On Sun, Jun 12, 2016 at 2:18 PM, Jun Rao  wrote:
>
> > Just to add on that list.
> >
> > 2. It would be good to document the format of the data stored in ZK.
> > 7. Earlier, there was a discussion on whether the tokens should be
> > propagated through ZK like config/acl/quota, or through the controller.
> > Currently, the controller is only designed for propagating topic
> metadata,
> > but not other data.
> > 8. Should we use SCRAM to send the token instead of DIGEST-MD5 since it's
> > deprecated?
> >
> > Also, the images in the wiki seem broken.
> >
> > Thanks,
> >
> > Jun
> >
> > On Fri, Jun 10, 2016 at 10:02 AM, Gwen Shapira 
> wrote:
> >
> >> From what I can see, remaining questions are:
> >>
> >> 1. Who / how are tokens renewed? By original requester only? or using
> >> Kerberos auth only?
> >> 2. Are tokens stored on each broker or in ZK?
> >> 3. How are tokens invalidated / expired?
> >> 4. Which encryption algorithm is used?
> >> 5. What is the impersonation proposal (it wasn't in the KIP but was
> >> discussed in this thread)?
> >> 6. Do we need new ACLs, if so - for what actions?
> >>
> >> Gwen
> >>
> >> On Thu, Jun 9, 2016 at 7:48 PM, Harsha  wrote:
> >> > Jun & Ismael,
> >> >  Unfortunately I couldn't attend the KIP
> meeting
> >> >  when delegation tokens discussed. Appreciate
> if
> >> >  you can update the thread if you have any
> >> >  further questions.
> >> > Thanks,
> >> > Harsha
> >> >
> >> > On Tue, May 24, 2016, at 11:32 AM, Liquan Pei wrote:
> >> >> It seems that the links to images in the KIP are broken.
> >> >>
> >> >> Liquan
> >> >>
> >> >> On Tue, May 24, 2016 at 9:33 AM, parth brahmbhatt <
> >> >> brahmbhatt.pa...@gmail.com> wrote:
> >> >>
> >> >> > 110. What does getDelegationTokenAs mean?
> >> >> > In the current proposal we only allow a user to get delegation
> token
> >> for
> >> >> > the identity that it authenticated as using another mechanism, i.e.
> >> A user
> >> >> > that authenticate using a keytab for principal us...@example.com
> >> will get
> >> >> > delegation tokens for that user only. In future I think we will
> have
> >> to
> >> >> > extend support such that we allow some set of users (
> >> >> > kafka-rest-u...@example.com, storm-nim...@example.com) to acquire
> >> >> > delegation tokens on behalf of other users whose identity they have
> >> >> > verified independently.  Kafka brokers will have ACLs to control
> >> which
> >> >> > users are allowed to impersonate other users and get tokens on
> >> behalf of
> >> >> > them. Overall Impersonation is a whole different problem in my
> >> opinion and
> >> >> > I think we can tackle it in separate KIP.
> >> >> >
> >> >> > 111. What's the typical rate of getting and renewing delegation
> >> tokens?
> >> >> > Typically this should be very very low, 1 request per minute is a
> >> >> > relatively high estimate. However it depends on the token
> >> expiration. I am
> >> >> > less worried about the extra load it puts on controller vs the
> added
> >> >> > complexity and the value it 

RE: Error while sending message to Kafka broker on SSL with Kafka 0.10.0.0

[Subhash Agrawal]

Now I changed the test producer call like this.
C:\development\kafka\kafka_2.11-0.10.0.0\kafka_2.11-0.10.0.0\bin\windows>.\kafka-console-producer.bat
 --broker-list localhost:8393 --topic test --prod
ucer.config ..\..\config\producer.properties

and updated producer.properties like this
security.protocol=SSL
ssl.truststore.location=C:/test.jks
ssl.truststore.password=test

Now I see this error message.
C:\development\kafka\kafka_2.11-0.10.0.0\kafka_2.11-0.10.0.0\bin\windows>.\kafka-console-producer.bat
 --broker-list localhost:8393 --topic test --prod
ucer.config ..\..\config\producer.properties
[2016-06-21 10:23:34,738] WARN The configuration metadata.broker.list = 
localhost:8393 was supplied but isn't a known config. (org.apache.kafka.client
s.producer.ProducerConfig)

[2016-06-21 10:23:55,180] WARN Failed to send SSL Close message  
(org.apache.kafka.common.network.SslTransportLayer)
java.io.IOException: An established connection was aborted by the software in 
your host machine
at sun.nio.ch.SocketDispatcher.write0(Native Method)
at sun.nio.ch.SocketDispatcher.write(SocketDispatcher.java:51)
at sun.nio.ch.IOUtil.writeFromNativeBuffer(IOUtil.java:89)
at sun.nio.ch.IOUtil.write(IOUtil.java:60)
at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:450)
at 
org.apache.kafka.common.network.SslTransportLayer.flush(SslTransportLayer.java:195)
at 
org.apache.kafka.common.network.SslTransportLayer.close(SslTransportLayer.java:163)
at org.apache.kafka.common.utils.Utils.closeAll(Utils.java:690)
at 
org.apache.kafka.common.network.KafkaChannel.close(KafkaChannel.java:47)
at org.apache.kafka.common.network.Selector.close(Selector.java:471)
at 
org.apache.kafka.common.network.Selector.pollSelectionKeys(Selector.java:348)
at org.apache.kafka.common.network.Selector.poll(Selector.java:283)
at org.apache.kafka.clients.NetworkClient.poll(NetworkClient.java:260)
at 
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:229)
at 
org.apache.kafka.clients.producer.internals.Sender.run(Sender.java:134)

Please let me know what I am missing.

Thanks
Subhash Agrawal.

From: Subhash Agrawal
Sent: Monday, June 20, 2016 4:20 PM
To: dev@kafka.apache.org; 'us...@kafka.apache.org'
Subject: Error while sending message to Kafka broker on SSL with Kafka 0.10.0.0

Hi All,
I am seeing error while sending message via test producer on SSL port. I am 
able to successfully send message to non-SSL port.

Here is my broker configuration.

listeners=SSL://:9093
security.inter.broker.protocol=SSL
ssl.client.auth=required
ssl.keystore.location=C:/test.jks
ssl.keystore.password=test
ssl.key.password=test
ssl.truststore.location= C:/test.jks
ssl.truststore.password=test

and I see no error while starting up kafka server as per its console output.

 [2016-06-20 15:59:32,627] INFO Registered broker 0 at path /brokers/ids/0 with 
addresses: SSL -> EndPoint(SAGRAWAL-PC.opentext.net,9093,SSL) (kafka.ut
ils.ZkUtils)
[2016-06-20 15:59:32,635] INFO Kafka version : 0.10.0.0 
(org.apache.kafka.common.utils.AppInfoParser)
[2016-06-20 15:59:32,636] INFO Kafka commitId : b8642491e78c5a13 
(org.apache.kafka.common.utils.AppInfoParser)
[2016-06-20 15:59:32,637] INFO [Kafka Server 0], started 
(kafka.server.KafkaServer)

C:\development\kafka\kafka_2.11-0.10.0.0\kafka_2.11-0.10.0.0\bin\windows>.\kafka-topics.bat
 --create --zookeeper localhost:2181 --replication-factor 1
 --partitions 1 --topic test
Created topic "test".

When I want to send message using test producer on SSL port, I keep on seeing 
this warning message.

C:\development\kafka\kafka_2.11-0.10.0.0\kafka_2.11-0.10.0.0\bin\windows>.\kafka-console-producer.bat
 --broker-list localhost:9093 --topic test
aaa
[2016-06-20 16:13:38,565] WARN Bootstrap broker localhost:9093 disconnected 
(org.apache.kafka.clients.NetworkClient)
[2016-06-20 16:13:38,781] WARN Bootstrap broker localhost:9093 disconnected 
(org.apache.kafka.clients.NetworkClient)
[2016-06-20 16:13:39,010] WARN Bootstrap broker localhost:9093 disconnected 
(org.apache.kafka.clients.NetworkClient)
Terminate batch job (Y/N)? y

Any idea what should I  look into?

Thanks
Subhash Agrawal

[jira] [Resolved] (KAFKA-3828) Consumer thread stalls after consumer re balance for some partition

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma resolved KAFKA-3828.

Resolution: Duplicate

Thanks for the update!

> Consumer thread stalls after consumer re balance for some partition 
> 
>
> Key: KAFKA-3828
> URL: https://issues.apache.org/jira/browse/KAFKA-3828
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
> Environment: Operating System : CentOS release 6.4
> Kafka Cluster: Stand alone cluster with one broker and one zookeeper.
>Reporter: Joseph Aliase
>Assignee: Neha Narkhede
>
> In process of testing the new Kafka Consumer API we came across this issue. 
> We started single broker Kafka Cluster with broker listening on port 9092 and 
> zookeeper on 2181.
> We created a topic test with partition 6. We started a consumer with below 
> configuration:
> bootstrap.servers= host-name:9092
> group.id=consumer-group
> key.deserializer=StringDeserializer.class.getName()
> value.deserializer=StringDeserializer.class.getName()
> session.timeout.ms=3
> heartbeat.interval.ms=1
> We started producing data into topic test:
> sh kafka-producer-perf-test.sh --topic test --num-records 100 
> --record-size 10 --throughput 500 --producer-props 
> bootstrap.servers=localhost:9092
> Consumer instance is started with 6 threads to consume data from 6 partition. 
> We then restart another consumer instance with 6 threads. Consumer re-balance 
> occurs and 6 partitions is divided equally among this two instance.
> Then we start another consumer instance with 6 threads again we could see 
> re-balance occurring with partition getting divided among three consumer 
> instance. Everything works well.
> Then if we stop one consumer instance and partitions get re-balanced between 
> two instance. 
> If we stop and restart the another running instances and repeat the steps for 
> few time we could see the issue occurring where we could see Consumer is 
> holding the partition's but not consuming any data from that partition. 
> Partition data remain unconsumed until we stop the consumer instance which is 
> holding the partition. 
> We were not able to reproduce this issue we publish data to topic at very low 
> rate however issue could be easily reproduced when data is being published at 
> high rate.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3828) Consumer thread stalls after consumer re balance for some partition

[Joseph Aliase (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3828?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15342026#comment-15342026
 ] 

Joseph Aliase commented on KAFKA-3828:
--

Upgrading the consumer fixed the issue. Thanks

> Consumer thread stalls after consumer re balance for some partition 
> 
>
> Key: KAFKA-3828
> URL: https://issues.apache.org/jira/browse/KAFKA-3828
> Project: Kafka
>  Issue Type: Bug
>  Components: consumer
> Environment: Operating System : CentOS release 6.4
> Kafka Cluster: Stand alone cluster with one broker and one zookeeper.
>Reporter: Joseph Aliase
>Assignee: Neha Narkhede
>
> In process of testing the new Kafka Consumer API we came across this issue. 
> We started single broker Kafka Cluster with broker listening on port 9092 and 
> zookeeper on 2181.
> We created a topic test with partition 6. We started a consumer with below 
> configuration:
> bootstrap.servers= host-name:9092
> group.id=consumer-group
> key.deserializer=StringDeserializer.class.getName()
> value.deserializer=StringDeserializer.class.getName()
> session.timeout.ms=3
> heartbeat.interval.ms=1
> We started producing data into topic test:
> sh kafka-producer-perf-test.sh --topic test --num-records 100 
> --record-size 10 --throughput 500 --producer-props 
> bootstrap.servers=localhost:9092
> Consumer instance is started with 6 threads to consume data from 6 partition. 
> We then restart another consumer instance with 6 threads. Consumer re-balance 
> occurs and 6 partitions is divided equally among this two instance.
> Then we start another consumer instance with 6 threads again we could see 
> re-balance occurring with partition getting divided among three consumer 
> instance. Everything works well.
> Then if we stop one consumer instance and partitions get re-balanced between 
> two instance. 
> If we stop and restart the another running instances and repeat the steps for 
> few time we could see the issue occurring where we could see Consumer is 
> holding the partition's but not consuming any data from that partition. 
> Partition data remain unconsumed until we stop the consumer instance which is 
> holding the partition. 
> We were not able to reproduce this issue we publish data to topic at very low 
> rate however issue could be easily reproduced when data is being published at 
> high rate.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-4 Delete Topic Schema

[Grant Henke]

Hi Ismael,

Thanks for the review. See my responses below.

One potential issue is that the number of topics in the response won't
> match the number of topics in the request. Worth checking what others think
> of this one.


Based on the choice of how we handled duplicate topics in the create topics
protocol, I took the same approach here. At one point create topics would
disconnect because I could't return an error per topic request. In the end
the preference was to return and error code even though the cardinality
would be different.

4. When requesting to delete a topic that is already marked for
> > >deletion, the request will wait for the wait for the timeout and
> > return as
> > >usual
>
> Do you mean that it will wait up to the timeout until the delete is
> "complete" as per the definition in `6`? Or will it wait unconditionally
> until the timeout expires? It would be good to make that clear.
>

That's exactly what I meant. I updated the wiki.

This could leak topic name information (as per KAFKA-3396, which was filed
> by you). We would probably want to return `InvalidTopic` for the case where
> the user doesn't have a valid `DESCRIBE TOPIC` ACL, right?


Good point. I will update the wiki and patch.

Unauthorized requests will receive a TopicAuthorizationException if they
are authorized to the "Describe" Operation on the "Topic" resource.
Otherwise they will receive an InvalidTopicException as if the topic does
not exist.

I was wondering (and this applies to the create topic as well), is there
> any value in a flag that says whether the timeout expired or not?


In both the create and delete response we return a TimeoutException error
code for the topics that did not "complete" in time. This allows the client
to know which topics actions completed and which timed out. I updated the
wiki to explicitly call this out in the response section.

Thanks,
Grant

On Tue, Jun 21, 2016 at 5:44 AM, Ismael Juma  wrote:

> Thanks Grant. A few comments inline.
>
> On Mon, Jun 20, 2016 at 9:09 PM, Grant Henke  wrote:
>
> > >2. If there are multiple instructions for the same topic in one
> > >request the extra request will be ignored
> > >   - This is because the list of topics is modeled server side as a
> > set
> > >   - Multiple deletes results in the same end goal, so handling this
> > >   error for the user should be okay
> >
>
> One potential issue is that the number of topics in the response won't
> match the number of topics in the request. Worth checking what others think
> of this one.
>
> 4. When requesting to delete a topic that is already marked for
> > >deletion, the request will wait for the wait for the timeout and
> > return as
> > >usual
>
>
> Do you mean that it will wait up to the timeout until the delete is
> "complete" as per the definition in `6`? Or will it wait unconditionally
> until the timeout expires? It would be good to make that clear.
>
>
> > >5. The principal must be authorized to the "Delete" Operation on the
> >
> >"Topic" resource to delete the topic.
> > >   - Unauthorized requests will receive a
> TopicAuthorizationException
> >
>
> This could leak topic name information (as per KAFKA-3396, which was filed
> by you). We would probably want to return `InvalidTopic` for the case where
> the user doesn't have a valid `DESCRIBE TOPIC` ACL, right?
>
>
> > >- Why have a timeout at all? Deletes could take a while?
> >
>
> I was wondering (and this applies to the create topic as well), is there
> any value in a flag that says whether the timeout expired or not?
>
> Thanks,
> Ismael
>



-- 
Grant Henke
Software Engineer | Cloudera
gr...@cloudera.com | twitter.com/gchenke | linkedin.com/in/granthenke

Re: [DISCUSS] Java 8 as a minimum requirement

[Damian Guy]

+1

On Tue, 21 Jun 2016 at 09:59 Marcus Gründler 
wrote:

> Hi Ismael,
>
> thanks for the pointer to the latest WebSphere documentation - I wasn’t
> aware
> of that release.
>
> We currently have customers that run our software frontend on an older
> WebSphere version that runs on Java 7 and push data to kafka brokers in the
> backend. Replacing Kafka brokers wouldn’t be an issue here since we are in
> control of the backend part.
>
> The WebSphere server where our frontend part (and therefore our kafka
> client)
> is running is a kind of general infrastructure of that customer and upgrade
> cycles are slow and independent of our release cycles.
>
> Of course that could be solved by a kind of proxy in front of the kafka
> brokers,
> so maybe we shouldn’t pay too much tribute to legacy systems :-)
>
> Regards, Marcus
>
>
> > Am 17.06.2016 um 15:44 schrieb Ismael Juma :
> >
> > Hi Marcus,
> >
> > Thanks for your feedback.
> >
> > With regards to IBM WebSphere, the latest stable release (8.5.5) supports
> > Java 8 according to the documentation:
> >
> > http://www-01.ibm.com/support/docview.wss?uid=swg27005002
> >
> > Having said that, it is fair to discuss servers and clients separately.
> In
> > Kafka, you can't use newer clients with older brokers, but you can use
> > older clients with newer brokers. As such, the scenario we're talking
> about
> > is that of users who can upgrade their brokers and clients to the latest
> > Kafka version, but are stuck with an older version of WebSphere, right?
> Are
> > you aware of such users?
> >
> > Ismael
> > On Fri, Jun 17, 2016 at 10:34 AM, Marcus Gründler <
> > marcus.gruend...@aixigo.de> wrote:
> >
> >> -1
> >> Hi Ismael,
> >>
> >> Although I really like the Java 8 features and understand the advantages
> >> you
> >> mentioned about Java 8 migration, I would suggest to stay with Java 7 as
> >> a minimum requirement for a while.
> >>
> >> I think there are two aspects to consider - Kafka Server and Kafka
> >> clients. On
> >> the server part it would make sense to switch to Java 8 because you can
> run
> >> the broker independently from any enclosing runtime (no JEE server etc.)
> >>
> >> But if you change the requirement for Kafka clients, you would cut Kafka
> >> support for quite a lot of real world deployments that run for example
> on
> >> an IBM WebSphere JEE Server (*sigh*). Since up to today there is no
> >> WebSphere version that supports Java 8.
> >>
> >> And I think a split of Kafka server with Java8 and Kafka client JARs in
> >> Java7
> >> would be too complicated to maintain.
> >>
> >> So my conclusion is - stay with Java 7 for a while.
> >>
> >> Regards, Marcus
> >>
> >>
> >>> Am 16.06.2016 um 22:45 schrieb Ismael Juma :
> >>>
> >>> Hi all,
> >>>
> >>> I would like to start a discussion on making Java 8 a minimum
> requirement
> >>> for Kafka's next feature release (let's say Kafka 0.10.1.0 for now).
> This
> >>> is the first discussion on the topic so the idea is to understand how
> >>> people feel about it. If people feel it's too soon, then we can pick up
> >> the
> >>> conversation again after Kafka 0.10.1.0. If the feedback is mostly
> >>> positive, I will start a vote thread.
> >>>
> >>> Let's start with some dates. Java 7 hasn't received public updates
> since
> >>> April 2015[1], Java 8 was released in March 2014[2] and Java 9 is
> >> scheduled
> >>> to be released in March 2017[3].
> >>>
> >>> The first argument for dropping support for Java 7 is that the last
> >> public
> >>> release by Oracle contains a large number of known security
> >>> vulnerabilities. The effectiveness of Kafka's security features is
> >> reduced
> >>> if the underlying runtime is not itself secure.
> >>>
> >>> The second argument for moving to Java 8 is that it adds a number of
> >>> compelling features:
> >>>
> >>> * Lambda expressions and method references (particularly useful for the
> >>> Kafka Streams DSL)
> >>> * Default methods (very useful for maintaining compatibility when
> adding
> >>> methods to interfaces)
> >>> * java.util.stream (helpful for making collection transformations more
> >>> concise)
> >>> * Lots of improvements to java.util.concurrent (CompletableFuture,
> >>> DoubleAdder, DoubleAccumulator, StampedLock, LongAdder,
> LongAccumulator)
> >>> * Other nice things: SplittableRandom, Optional (and many others I have
> >> not
> >>> mentioned)
> >>>
> >>> The third argument is that it will simplify our testing matrix, we
> won't
> >>> have to test with Java 7 any longer (this is particularly useful for
> >> system
> >>> tests that take hours to run). It will also make it easier to support
> >> Scala
> >>> 2.12, which requires Java 8.
> >>>
> >>> The fourth argument is that many other open-source projects have taken
> >> the
> >>> leap already. Examples are Cassandra[4], Lucene[5], Akka[6], Hadoop
> 3[7],
> >>> Jetty[8], Eclipse[9], IntelliJ[10] and many others[11]. Even Android
> 

KafkaConsumer - Poll consumes more records than needed

[Asier Aranbarri Beldarrain]

Hi everyone! I'm new to this mailing list; Hope I can find solutions and
who knows, maybe some day help other Kafka developers.

I'm facing an issue regarding the new (>=0.9) KafkaConsumer and its poll()
method:

What would be the correct way to specify a maximum amount of records to be
processed without consuming more messages than needed?

For example, I need to consume 10 and just 10 messages from a Kafka topic.
I start the consumer, and the poll method gets 15 messages. I process 10 of
them and then manually commit the offset.

BUT:

If I start the consumer again, it will start consuming from the 16th
message, not the 11th. This is a huge problem for me, as I need a reader
that consumes an specific number of messages.

In Kafka 0.8, this was easy since you could use the Stream Iterator to
process the messages and stop in the exact moment you need to. But polling
isn't exact, so the consumer could consume a higher amount of records that
I need to process.

Any way to solve this? Any help would be kindly appreciated.

Thanks in advance!

[jira] [Updated] (KAFKA-3885) Kafka new producer cannot failover

[wateray (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3885?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

wateray updated KAFKA-3885:
---
Description: 
This bug can reproduce by the following steps.
The cluster has 2 brokers.
 a) start a new producer, then send messages, it works well.
 b) Then kill one broker,  it works well.
 c) Then restart the broker,  it works well.
 d) Then kill the other broker,  the producer can't failover.

The the producer print log infinity.
org.apache.kafka.common.errors.TimeoutException: Batch containing 1 record(s) 
expired due to timeout while requesting metadata from brokers for 
lwb_test_p50_r2-29



When producer sends msg, it detected that metadata should update.
But at this code, class: NetworkClient ,method: leastLoadedNode
List nodes = this.metadataUpdater.fetchNodes();

nodes only return one result, and the returned node is the killed node, so the 
producer cannot failover!








  was:
This bug can reproduce by the following steps.
The cluster has 2 brokers.
 a) start a new producer, then send message, it works well.
 b) Then kill one broker,  it works well.
 c) Then restart the broker,  it works well.
 d) Then kill the other broker,  the producer can't failover.

The the producer print log infinity.
org.apache.kafka.common.errors.TimeoutException: Batch containing 1 record(s) 
expired due to timeout while requesting metadata from brokers for 
lwb_test_p50_r2-29



When producer sends msg, it detected that metadata should update.
But at this code, class: NetworkClient ,method: leastLoadedNode
List nodes = this.metadataUpdater.fetchNodes();

nodes only return one result, and the returned node is the killed node, so the 
producer cannot failover!









> Kafka new producer cannot failover
> --
>
> Key: KAFKA-3885
> URL: https://issues.apache.org/jira/browse/KAFKA-3885
> Project: Kafka
>  Issue Type: Bug
>  Components: clients
>Affects Versions: 0.9.0.0, 0.8.2.2, 0.9.0.1, 0.10.0.0
>Reporter: wateray
>
> This bug can reproduce by the following steps.
> The cluster has 2 brokers.
>  a) start a new producer, then send messages, it works well.
>  b) Then kill one broker,  it works well.
>  c) Then restart the broker,  it works well.
>  d) Then kill the other broker,  the producer can't failover.
> The the producer print log infinity.
> org.apache.kafka.common.errors.TimeoutException: Batch containing 1 record(s) 
> expired due to timeout while requesting metadata from brokers for 
> lwb_test_p50_r2-29
> 
> When producer sends msg, it detected that metadata should update.
> But at this code, class: NetworkClient ,method: leastLoadedNode
> List nodes = this.metadataUpdater.fetchNodes();
> nodes only return one result, and the returned node is the killed node, so 
> the producer cannot failover!



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Created] (KAFKA-3885) Kafka new producer cannot failover

[wateray (JIRA)]

wateray created KAFKA-3885:
--

 Summary: Kafka new producer cannot failover
 Key: KAFKA-3885
 URL: https://issues.apache.org/jira/browse/KAFKA-3885
 Project: Kafka
  Issue Type: Bug
  Components: clients
Affects Versions: 0.10.0.0, 0.9.0.1, 0.8.2.2, 0.9.0.0
Reporter: wateray


This bug can reproduce by the following steps.
The cluster has 2 brokers.
 a) start a new producer, then send message, it works well.
 b) Then kill one broker,  it works well.
 c) Then restart the broker,  it works well.
 d) Then kill the other broker,  the producer can't failover.

The the producer print log infinity.
org.apache.kafka.common.errors.TimeoutException: Batch containing 1 record(s) 
expired due to timeout while requesting metadata from brokers for 
lwb_test_p50_r2-29



When producer sends msg, it detected that metadata should update.
But at this code, class: NetworkClient ,method: leastLoadedNode
List nodes = this.metadataUpdater.fetchNodes();

nodes only return one result, and the returned node is the killed node, so the 
producer cannot failover!










--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Resolved] (KAFKA-3874) Transient test failure: org.apache.kafka.streams.integration.JoinIntegrationTest.shouldCountClicksPerRegion

[Damian Guy (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3874?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Damian Guy resolved KAFKA-3874.
---
   Resolution: Fixed
Fix Version/s: 0.10.1.0

Should be fixed by:
https://github.com/apache/kafka/commit/44c8308ab1a6b950c5b12386d7864881b5d052a0

> Transient test failure: 
> org.apache.kafka.streams.integration.JoinIntegrationTest.shouldCountClicksPerRegion
> ---
>
> Key: KAFKA-3874
> URL: https://issues.apache.org/jira/browse/KAFKA-3874
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams, unit tests
>Reporter: Ismael Juma
>Assignee: Damian Guy
>  Labels: transient-unit-test-failure
> Fix For: 0.10.1.0
>
>
> The failure I am recording happened in Jenkins, but I had a similar failure 
> locally (and two other integration tests failed in that run).
> {code}
> Expected: <[KeyValue(europe, 13), KeyValue(americas, 4), KeyValue(asia, 25), 
> KeyValue(americas, 23), KeyValue(europe, 69), KeyValue(americas, 101), 
> KeyValue(europe, 109), KeyValue(asia, 124)]>
>  but: was <[KeyValue(europe, 122), KeyValue(americas, 105), 
> KeyValue(asia, 149), KeyValue(americas, 124), KeyValue(europe, 178), 
> KeyValue(americas, 202), KeyValue(europe, 218), KeyValue(asia, 248)]>
>   at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
>   at org.junit.Assert.assertThat(Assert.java:956)
>   at org.junit.Assert.assertThat(Assert.java:923)
>   at 
> org.apache.kafka.streams.integration.JoinIntegrationTest.shouldCountClicksPerRegion(JoinIntegrationTest.java:258)
> {code}
> https://builds.apache.org/job/kafka-trunk-git-pr-jdk7/4254/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3874) Transient test failure: org.apache.kafka.streams.integration.JoinIntegrationTest.shouldCountClicksPerRegion

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341632#comment-15341632
 ] 

Ismael Juma commented on KAFKA-3874:


[~damianguy], if it should be fixed, then feel free to close it. We can always 
reopen if it happens again.

> Transient test failure: 
> org.apache.kafka.streams.integration.JoinIntegrationTest.shouldCountClicksPerRegion
> ---
>
> Key: KAFKA-3874
> URL: https://issues.apache.org/jira/browse/KAFKA-3874
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams, unit tests
>Reporter: Ismael Juma
>Assignee: Damian Guy
>  Labels: transient-unit-test-failure
>
> The failure I am recording happened in Jenkins, but I had a similar failure 
> locally (and two other integration tests failed in that run).
> {code}
> Expected: <[KeyValue(europe, 13), KeyValue(americas, 4), KeyValue(asia, 25), 
> KeyValue(americas, 23), KeyValue(europe, 69), KeyValue(americas, 101), 
> KeyValue(europe, 109), KeyValue(asia, 124)]>
>  but: was <[KeyValue(europe, 122), KeyValue(americas, 105), 
> KeyValue(asia, 149), KeyValue(americas, 124), KeyValue(europe, 178), 
> KeyValue(americas, 202), KeyValue(europe, 218), KeyValue(asia, 248)]>
>   at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
>   at org.junit.Assert.assertThat(Assert.java:956)
>   at org.junit.Assert.assertThat(Assert.java:923)
>   at 
> org.apache.kafka.streams.integration.JoinIntegrationTest.shouldCountClicksPerRegion(JoinIntegrationTest.java:258)
> {code}
> https://builds.apache.org/job/kafka-trunk-git-pr-jdk7/4254/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3874) Transient test failure: org.apache.kafka.streams.integration.JoinIntegrationTest.shouldCountClicksPerRegion

[Damian Guy (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341627#comment-15341627
 ] 

Damian Guy commented on KAFKA-3874:
---

This should be fixed with the commit yesterday. I'll keep an eye on it to make 
sure.

> Transient test failure: 
> org.apache.kafka.streams.integration.JoinIntegrationTest.shouldCountClicksPerRegion
> ---
>
> Key: KAFKA-3874
> URL: https://issues.apache.org/jira/browse/KAFKA-3874
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams, unit tests
>Reporter: Ismael Juma
>Assignee: Damian Guy
>  Labels: transient-unit-test-failure
>
> The failure I am recording happened in Jenkins, but I had a similar failure 
> locally (and two other integration tests failed in that run).
> {code}
> Expected: <[KeyValue(europe, 13), KeyValue(americas, 4), KeyValue(asia, 25), 
> KeyValue(americas, 23), KeyValue(europe, 69), KeyValue(americas, 101), 
> KeyValue(europe, 109), KeyValue(asia, 124)]>
>  but: was <[KeyValue(europe, 122), KeyValue(americas, 105), 
> KeyValue(asia, 149), KeyValue(americas, 124), KeyValue(europe, 178), 
> KeyValue(americas, 202), KeyValue(europe, 218), KeyValue(asia, 248)]>
>   at org.hamcrest.MatcherAssert.assertThat(MatcherAssert.java:20)
>   at org.junit.Assert.assertThat(Assert.java:956)
>   at org.junit.Assert.assertThat(Assert.java:923)
>   at 
> org.apache.kafka.streams.integration.JoinIntegrationTest.shouldCountClicksPerRegion(JoinIntegrationTest.java:258)
> {code}
> https://builds.apache.org/job/kafka-trunk-git-pr-jdk7/4254/



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1534: MINOR: update streams.html with KStream API change...

[dguy]

GitHub user dguy opened a pull request:

https://github.com/apache/kafka/pull/1534

MINOR: update streams.html with KStream API changes

@mjsax @guozhangwang 

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/dguy/kafka update-streams-doc

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1534.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1534


commit d7a0968b444bdf43cecd79e048097525d42d3ed9
Author: Damian Guy 
Date:   2016-06-21T11:44:49Z

update streams.html with KStream API changes




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

Re: [VOTE] KIP-55: Secure quotas for authenticated users

[Ismael Juma]

On Tue, Jun 21, 2016 at 12:50 PM, Rajini Sivaram <
rajinisiva...@googlemail.com> wrote:

> It is actually quite tempting to do the same for client-id quotas as well,
> but I suppose we can't break existing users who have configured defaults in
> server.properties and providing two ways of setting client-id defaults
> would be just too confusing.
>

Using two different approaches for client-id versus user quota defaults is
also not great. We could deprecate the server.properties default configs
for client-id quotas and remove them in the future. In the meantime, we
would have to live with 2 level defaults.

Jun, what are your thoughts on this?

Ismael

Re: [VOTE] KIP-55: Secure quotas for authenticated users

[Rajini Sivaram]

Jun,

I think that is a good idea. It keeps all user quotas in one place with an
intuitive hierarchy, avoids properties that are applied based on various
conditions and also enables dynamic updates. Dynamic updates obviously need
to get applied to all clients using the default, but I think that should be
ok.

It is actually quite tempting to do the same for client-id quotas as well,
but I suppose we can't break existing users who have configured defaults in
server.properties and providing two ways of setting client-id defaults
would be just too confusing.

I have updated the KIP to remove the default user quota properties and set
default user quotas in /config/users.

Thank you,

Rajini

On Tue, Jun 21, 2016 at 6:04 AM, Jun Rao  wrote:

> Rajini,
>
> Another thing that's probably worth thinking through is whether it's better
> to make the default user quota dynamic as well. So, instead of adding
> quota.user.producer.default
> and quota.user.consumer.default in the broker config, another way is to set
> them using sth like the following.
>
> bin/kafka-configs  --zookeeper localhost:2181 --alter --add-config
> 'producer_byte_rate=10,consumer_byte_rate=20' --entity-name=* --entity-type
> users
>
> We may add other types of quotas in the future and we probably don't want
> to keep adding static configs.
>
> Thanks,
>
> Jun
>
>
>
> On Mon, Jun 20, 2016 at 2:32 AM, Rajini Sivaram <
> rajinisiva...@googlemail.com> wrote:
>
> > Gwen/Jay,
> >
> > Have you had a chance to look at the updated KIP? It will be good to get
> > your feedback as well before restarting vote on the updated KIP.
> >
> > If there are no objections, I will start the vote tomorrow.
> >
> >
> > On Fri, Jun 17, 2016 at 6:59 PM, Rajini Sivaram <
> > rajinisiva...@googlemail.com> wrote:
> >
> > > Thank you, Jun. I have removed user_principal from the KIP.
> > >
> > > On Fri, Jun 17, 2016 at 6:00 PM, Jun Rao  wrote:
> > >
> > >> Rajini,
> > >>
> > >> 10. Yes, then we can probably leave out the user_principal field and
> > keep
> > >> the version to be 1.
> > >>
> > >> Other than that, the KIP looks good to me.
> > >>
> > >> Thanks,
> > >>
> > >> Jun
> > >>
> > >> On Fri, Jun 17, 2016 at 3:29 AM, Rajini Sivaram <
> > >> rajinisiva...@googlemail.com> wrote:
> > >>
> > >> > Jun,
> > >> >
> > >> > 10. Since entity_type "users" is new, shouldn't the JSON for these
> > >> entities
> > >> > have version 1? I have moved "user_principal" out of the config in
> the
> > >> > samples and added to the  entries as well. But
> actually,
> > >> do
> > >> > we need to store the non-encoded principal at all? The node name is
> > >> > URL-encoded user principal, so it is fairly readable if you are
> > looking
> > >> in
> > >> > ZK and *kafka_configs.sh* will show the non-encoded principal by
> > >> decoding
> > >> > the name from the path (since it needs to do encoding anyway because
> > the
> > >> > names specified on the command line will be non-encoded principals,
> it
> > >> can
> > >> > do decoding too). Perhaps that is sufficient?
> > >> >
> > >> > 11. I liked the second approach since it looks neat and
> future-proof.
> > >> Have
> > >> > updated the KIP.
> > >> >
> > >> > 12. Yes, that is correct.
> > >> >
> > >> > Many thanks,
> > >> >
> > >> > Rajini
> > >> >
> > >> >
> > >> > On Thu, Jun 16, 2016 at 11:36 PM, Jun Rao  wrote:
> > >> >
> > >> > > Rajini,
> > >> > >
> > >> > > Thanks for the update. A few more questions/comments.
> > >> > >
> > >> > > 10. For the quota value stored in ZK, since we are adding an
> > optional
> > >> > > user_principal field in the json, we should bump the version from
> 1
> > >> to 2.
> > >> > > Also, user_principal is not really part of the config values. So,
> > >> perhaps
> > >> > > we should represent it as the following?
> > >> > > {
> > >> > > "version":2,
> > >> > > "config": {
> > >> > > "producer_byte_rate":"1024",
> > >> > > "consumer_byte_rate":"2048"
> > >> > > },
> > >> > > "user_principal" : "user1"
> > >> > > }
> > >> > >
> > >> > >  Also, we should store user_principal in the following json too,
> > >> right?
> > >> > > // Zookeeper persistence path
> /users//clients/clientA
> > >> > > {
> > >> > > "version":1,
> > >> > > "config": {
> > >> > > "producer_byte_rate":"10",
> > >> > > "consumer_byte_rate":"30"
> > >> > > }
> > >> > > }
> > >> > >
> > >> > > 11. For the change notification path, would it be better to change
> > it
> > >> to
> > >> > > something like the following and bump up version to 2?
> > >> > > // Change notification for quota of 
> > >> > > {
> > >> > > "version":2,
> > >> > > [
> > >> > >   { "entity_type": "users",
> > >> > > "entity_name": "user2"
> > >> > >   },
> > >> > >   { "entity_type": "clients",
> > >> > > "entity_name": "clientA"
> > >> > >   }
> > >> > > ]
> > >> > >  }
> > >> > >
> > >> > > 

Re: [DISCUSS] KIP-4 Delete Topic Schema

[Ismael Juma]

Thanks Grant. A few comments inline.

On Mon, Jun 20, 2016 at 9:09 PM, Grant Henke  wrote:

> >2. If there are multiple instructions for the same topic in one
> >request the extra request will be ignored
> >   - This is because the list of topics is modeled server side as a
> set
> >   - Multiple deletes results in the same end goal, so handling this
> >   error for the user should be okay
>

One potential issue is that the number of topics in the response won't
match the number of topics in the request. Worth checking what others think
of this one.

4. When requesting to delete a topic that is already marked for
> >deletion, the request will wait for the wait for the timeout and
> return as
> >usual


Do you mean that it will wait up to the timeout until the delete is
"complete" as per the definition in `6`? Or will it wait unconditionally
until the timeout expires? It would be good to make that clear.


> >5. The principal must be authorized to the "Delete" Operation on the
>
>"Topic" resource to delete the topic.
> >   - Unauthorized requests will receive a TopicAuthorizationException
>

This could leak topic name information (as per KAFKA-3396, which was filed
by you). We would probably want to return `InvalidTopic` for the case where
the user doesn't have a valid `DESCRIBE TOPIC` ACL, right?


> >- Why have a timeout at all? Deletes could take a while?
>

I was wondering (and this applies to the create topic as well), is there
any value in a flag that says whether the timeout expired or not?

Thanks,
Ismael

[jira] [Created] (KAFKA-3884) KGroupedStreamIntegrationTest.shouldAggregate seems to be hanging

[Ismael Juma (JIRA)]

Ismael Juma created KAFKA-3884:
--

 Summary: KGroupedStreamIntegrationTest.shouldAggregate seems to be 
hanging
 Key: KAFKA-3884
 URL: https://issues.apache.org/jira/browse/KAFKA-3884
 Project: Kafka
  Issue Type: Sub-task
  Components: streams, unit tests
Reporter: Ismael Juma
Assignee: Guozhang Wang


Build failed after 180 minutes and the last 2 lines were:

{code}
org.apache.kafka.streams.integration.KGroupedStreamIntegrationTest > 
shouldReduce PASSED

org.apache.kafka.streams.integration.KGroupedStreamIntegrationTest > 
shouldAggregate STARTED
{code}

https://builds.apache.org/job/kafka-trunk-jdk8/712/console



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Commented] (KAFKA-3502) Build is killed during kafka streams tests due to `pure virtual method called` error

[Ismael Juma (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3502?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341438#comment-15341438
 ] 

Ismael Juma commented on KAFKA-3502:


This happened again:

"org.apache.kafka.streams.processor.TopologyBuilderTest > 
testAddStateStoreWithNonExistingProcessor PASSED
pure virtual method called
terminate called without an active exception

216 tests completed, 2 failed"

https://builds.apache.org/job/kafka-trunk-jdk7/1379/console

> Build is killed during kafka streams tests due to `pure virtual method 
> called` error
> 
>
> Key: KAFKA-3502
> URL: https://issues.apache.org/jira/browse/KAFKA-3502
> Project: Kafka
>  Issue Type: Sub-task
>Reporter: Ashish K Singh
>  Labels: transient-unit-test-failure
>
> Build failed due to failure in streams' test. Not clear which test led to 
> this.
> Jenkins console: 
> https://builds.apache.org/job/kafka-trunk-git-pr-jdk7/3210/console
> {code}
> org.apache.kafka.streams.kstream.internals.KTableFilterTest > testValueGetter 
> PASSED
> org.apache.kafka.streams.kstream.internals.KStreamFlatMapTest > testFlatMap 
> PASSED
> org.apache.kafka.streams.kstream.internals.KTableAggregateTest > testAggBasic 
> PASSED
> org.apache.kafka.streams.kstream.internals.KStreamFlatMapValuesTest > 
> testFlatMapValues PASSED
> org.apache.kafka.streams.kstream.KStreamBuilderTest > testMerge PASSED
> org.apache.kafka.streams.kstream.KStreamBuilderTest > testFrom PASSED
> org.apache.kafka.streams.kstream.KStreamBuilderTest > testNewName PASSED
> pure virtual method called
> terminate called without an active exception
> :streams:test FAILED
> FAILURE: Build failed with an exception.
> * What went wrong:
> Execution failed for task ':streams:test'.
> > Process 'Gradle Test Executor 4' finished with non-zero exit value 134
> {code}
> Tried reproducing the issue locally, but could not.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Re: [DISCUSS] KIP-63: Unify store and downstream caching in streams

[Eno Thereska]

Hi Roger,

I realised I never got back to you on this one, sorry. Some answers inline:

> On 3 Jun 2016, at 22:48, Roger Hoover  wrote:
> 
> Hi Eno,
> 
> Does this mean that Kafka Streams will disable the RocksDB write buffer?

For the purposes of this KIP we might not want to change the current RocksDB 
state of things. However, Guozhang has written about future plans in the memory 
management page 

 in the wiki.


> Is it currently safe to recover a Kafka Streams application after SIGKILL
> on the same machine?  If not, will this make it safe to do so?
> 
> If RocksDB is not flushed before offsets are commited in Kafka and is
> killed with SIGKILL, will the data in the write buffer be lost (since Kafka
> Streams disables the transaction log)?  That data will be present in the
> Kafka changelog but will it get applied to the recovered RocksDB database
> on restart?

These are good questions on failure modes. This KIP will not change the failure 
behaviour from what it currently is. I believe there will be subsequent KIPs 
where the issues around writing atomically to multiple places will be 
considered, and failure semantics will be strengthened. Stay tuned.

Thanks
Eno



> 
> Thanks,
> 
> Roger
> 
> On Fri, Jun 3, 2016 at 2:39 PM, Eno Thereska  wrote:
> 
>> Hi Gwen,
>> 
>> Yes. As an example, if cache.max.bytes.buffering set to X, and if users
>> have A aggregation operators and T KTable.to() operators, then X*(A + T)
>> total bytes will be allocated for caching.
>> 
>> Eno
>> 
>>> On 3 Jun 2016, at 21:37, Gwen Shapira  wrote:
>>> 
>>> Just to clarify: "cache.max.bytes.buffering" is per processor?
>>> 
>>> 
>>> On Thu, Jun 2, 2016 at 11:30 AM, Eno Thereska 
>> wrote:
 Hi there,
 
 I have created KIP-63: Unify store and downstream caching in streams
 
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-63%3A+Unify+store+and+downstream+caching+in+streams
>> <
>> https://cwiki.apache.org/confluence/display/KAFKA/KIP-63:+Unify+store+and+downstream+caching+in+streams
>>> 
 
 
 Feedback is appreciated.
 
 Thank you
 Eno
>> 
>> 

Jenkins build is back to normal : kafka-trunk-jdk8 #713

[Apache Jenkins Server]

See 

Re: [DISCUSS] Java 8 as a minimum requirement

[Marcus Gründler]

Hi Ismael,

thanks for the pointer to the latest WebSphere documentation - I wasn’t aware 
of that release.

We currently have customers that run our software frontend on an older 
WebSphere version that runs on Java 7 and push data to kafka brokers in the 
backend. Replacing Kafka brokers wouldn’t be an issue here since we are in 
control of the backend part.

The WebSphere server where our frontend part (and therefore our kafka client) 
is running is a kind of general infrastructure of that customer and upgrade 
cycles are slow and independent of our release cycles.

Of course that could be solved by a kind of proxy in front of the kafka 
brokers, 
so maybe we shouldn’t pay too much tribute to legacy systems :-)

Regards, Marcus


> Am 17.06.2016 um 15:44 schrieb Ismael Juma :
> 
> Hi Marcus,
> 
> Thanks for your feedback.
> 
> With regards to IBM WebSphere, the latest stable release (8.5.5) supports
> Java 8 according to the documentation:
> 
> http://www-01.ibm.com/support/docview.wss?uid=swg27005002
> 
> Having said that, it is fair to discuss servers and clients separately. In
> Kafka, you can't use newer clients with older brokers, but you can use
> older clients with newer brokers. As such, the scenario we're talking about
> is that of users who can upgrade their brokers and clients to the latest
> Kafka version, but are stuck with an older version of WebSphere, right? Are
> you aware of such users?
> 
> Ismael
> On Fri, Jun 17, 2016 at 10:34 AM, Marcus Gründler <
> marcus.gruend...@aixigo.de> wrote:
> 
>> -1
>> Hi Ismael,
>> 
>> Although I really like the Java 8 features and understand the advantages
>> you
>> mentioned about Java 8 migration, I would suggest to stay with Java 7 as
>> a minimum requirement for a while.
>> 
>> I think there are two aspects to consider - Kafka Server and Kafka
>> clients. On
>> the server part it would make sense to switch to Java 8 because you can run
>> the broker independently from any enclosing runtime (no JEE server etc.)
>> 
>> But if you change the requirement for Kafka clients, you would cut Kafka
>> support for quite a lot of real world deployments that run for example on
>> an IBM WebSphere JEE Server (*sigh*). Since up to today there is no
>> WebSphere version that supports Java 8.
>> 
>> And I think a split of Kafka server with Java8 and Kafka client JARs in
>> Java7
>> would be too complicated to maintain.
>> 
>> So my conclusion is - stay with Java 7 for a while.
>> 
>> Regards, Marcus
>> 
>> 
>>> Am 16.06.2016 um 22:45 schrieb Ismael Juma :
>>> 
>>> Hi all,
>>> 
>>> I would like to start a discussion on making Java 8 a minimum requirement
>>> for Kafka's next feature release (let's say Kafka 0.10.1.0 for now). This
>>> is the first discussion on the topic so the idea is to understand how
>>> people feel about it. If people feel it's too soon, then we can pick up
>> the
>>> conversation again after Kafka 0.10.1.0. If the feedback is mostly
>>> positive, I will start a vote thread.
>>> 
>>> Let's start with some dates. Java 7 hasn't received public updates since
>>> April 2015[1], Java 8 was released in March 2014[2] and Java 9 is
>> scheduled
>>> to be released in March 2017[3].
>>> 
>>> The first argument for dropping support for Java 7 is that the last
>> public
>>> release by Oracle contains a large number of known security
>>> vulnerabilities. The effectiveness of Kafka's security features is
>> reduced
>>> if the underlying runtime is not itself secure.
>>> 
>>> The second argument for moving to Java 8 is that it adds a number of
>>> compelling features:
>>> 
>>> * Lambda expressions and method references (particularly useful for the
>>> Kafka Streams DSL)
>>> * Default methods (very useful for maintaining compatibility when adding
>>> methods to interfaces)
>>> * java.util.stream (helpful for making collection transformations more
>>> concise)
>>> * Lots of improvements to java.util.concurrent (CompletableFuture,
>>> DoubleAdder, DoubleAccumulator, StampedLock, LongAdder, LongAccumulator)
>>> * Other nice things: SplittableRandom, Optional (and many others I have
>> not
>>> mentioned)
>>> 
>>> The third argument is that it will simplify our testing matrix, we won't
>>> have to test with Java 7 any longer (this is particularly useful for
>> system
>>> tests that take hours to run). It will also make it easier to support
>> Scala
>>> 2.12, which requires Java 8.
>>> 
>>> The fourth argument is that many other open-source projects have taken
>> the
>>> leap already. Examples are Cassandra[4], Lucene[5], Akka[6], Hadoop 3[7],
>>> Jetty[8], Eclipse[9], IntelliJ[10] and many others[11]. Even Android will
>>> support Java 8 in the next version (although it will take a while before
>>> most phones will use that version sadly). This reduces (but does not
>>> eliminate) the chance that we would be the first project that would
>> cause a
>>> user to consider a Java upgrade.
>>> 
>>> The main argument for not making the 

[jira] [Issue Comment Deleted] (KAFKA-3482) [DOC] - Remove the part about LinkedIn running ZK 3.3.X

[Ismael Juma (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3482?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Ismael Juma updated KAFKA-3482:
---
Comment: was deleted

(was: GitHub user bbejeck opened a pull request:

https://github.com/apache/kafka/pull/1532

KAFKA-3482: consolidate utility methods to TestUtils, added StreamsTe…

…stUtils, added method for pausing tests to TestUtils

Changes made:
 1. Added utility method for creating consumer configs.
 2. Added methods for creating producer, consumer configs with default 
values for de/serializers.
 3. Pulled out method for waiting for test state to TestUtils (not using 
Thread.sleep).
 4. Added utility class for creating streams configs and methods providing 
default de/serializers.

You can merge this pull request into a Git repository by running:

$ git pull https://github.com/bbejeck/kafka 
KAFKA_3842_add_helper_functions_test_utils

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1532.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1532


commit 7f53185c0e8b446b32274ae4b54250daa999c699
Author: bbejeck 
Date:   2016-06-21T02:45:23Z

KAFKA-3482: consolidate utility methods to TestUtils, added 
StreamsTestUtils, added method for pausing tests to TestUtils


)

> [DOC] - Remove the part about LinkedIn running ZK 3.3.X
> ---
>
> Key: KAFKA-3482
> URL: https://issues.apache.org/jira/browse/KAFKA-3482
> Project: Kafka
>  Issue Type: Bug
>Reporter: Gwen Shapira
>
> LinkedIn, and everyone else is currently running ZK 3.4.5 / 3.4.6 or maybe 
> higher.
> 3.3.X is definitely not recommended any more. Lets get it out of our docs.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Build failed in Jenkins: kafka-trunk-jdk7 #1382

[Apache Jenkins Server]

See 

Changes:

[me] KAFKA-3176: Add partition/offset options to the new consumer

--
[...truncated 5129 lines...]

kafka.api.SaslSslEndToEndAuthorizationTest > testNoConsumeAcl PASSED

kafka.api.SaslSslEndToEndAuthorizationTest > testProduceConsumeViaAssign STARTED

kafka.api.SaslSslEndToEndAuthorizationTest > testProduceConsumeViaAssign PASSED

kafka.api.SaslSslEndToEndAuthorizationTest > testNoProduceAcl STARTED

kafka.api.SaslSslEndToEndAuthorizationTest > testNoProduceAcl PASSED

kafka.api.SaslSslEndToEndAuthorizationTest > testNoGroupAcl STARTED

kafka.api.SaslSslEndToEndAuthorizationTest > testNoGroupAcl PASSED

kafka.api.SaslSslEndToEndAuthorizationTest > testProduceConsumeViaSubscribe 
STARTED

kafka.api.SaslSslEndToEndAuthorizationTest > testProduceConsumeViaSubscribe 
PASSED

kafka.api.PlaintextProducerSendTest > testSerializerConstructors STARTED

kafka.api.PlaintextProducerSendTest > testSerializerConstructors PASSED

kafka.api.PlaintextProducerSendTest > testWrongSerializer STARTED

kafka.api.PlaintextProducerSendTest > testWrongSerializer PASSED

kafka.api.PlaintextProducerSendTest > 
testSendNonCompressedMessageWithCreateTime STARTED

kafka.api.PlaintextProducerSendTest > 
testSendNonCompressedMessageWithCreateTime PASSED

kafka.api.PlaintextProducerSendTest > 
testSendCompressedMessageWithLogAppendTime STARTED

kafka.api.PlaintextProducerSendTest > 
testSendCompressedMessageWithLogAppendTime PASSED

kafka.api.PlaintextProducerSendTest > testClose STARTED

kafka.api.PlaintextProducerSendTest > testClose PASSED

kafka.api.PlaintextProducerSendTest > testFlush STARTED

kafka.api.PlaintextProducerSendTest > testFlush PASSED

kafka.api.PlaintextProducerSendTest > testSendToPartition STARTED

kafka.api.PlaintextProducerSendTest > testSendToPartition PASSED

kafka.api.PlaintextProducerSendTest > testSendOffset STARTED

kafka.api.PlaintextProducerSendTest > testSendOffset PASSED

kafka.api.PlaintextProducerSendTest > testAutoCreateTopic STARTED

kafka.api.PlaintextProducerSendTest > testAutoCreateTopic PASSED

kafka.api.PlaintextProducerSendTest > testSendWithInvalidCreateTime STARTED

kafka.api.PlaintextProducerSendTest > testSendWithInvalidCreateTime PASSED

kafka.api.PlaintextProducerSendTest > testSendCompressedMessageWithCreateTime 
STARTED

kafka.api.PlaintextProducerSendTest > testSendCompressedMessageWithCreateTime 
PASSED

kafka.api.PlaintextProducerSendTest > testCloseWithZeroTimeoutFromCallerThread 
STARTED

kafka.api.PlaintextProducerSendTest > testCloseWithZeroTimeoutFromCallerThread 
PASSED

kafka.api.PlaintextProducerSendTest > testCloseWithZeroTimeoutFromSenderThread 
STARTED

kafka.api.PlaintextProducerSendTest > testCloseWithZeroTimeoutFromSenderThread 
PASSED

kafka.api.PlaintextProducerSendTest > 
testSendNonCompressedMessageWithLogApendTime STARTED

kafka.api.PlaintextProducerSendTest > 
testSendNonCompressedMessageWithLogApendTime PASSED

kafka.api.PlaintextConsumerTest > testPartitionsForAutoCreate STARTED

kafka.api.PlaintextConsumerTest > testPartitionsForAutoCreate PASSED

kafka.api.PlaintextConsumerTest > testShrinkingTopicSubscriptions STARTED

kafka.api.PlaintextConsumerTest > testShrinkingTopicSubscriptions PASSED

kafka.api.PlaintextConsumerTest > testAsyncCommit STARTED

kafka.api.PlaintextConsumerTest > testAsyncCommit PASSED

kafka.api.PlaintextConsumerTest > testMultiConsumerSessionTimeoutOnStopPolling 
STARTED

kafka.api.PlaintextConsumerTest > testMultiConsumerSessionTimeoutOnStopPolling 
PASSED

kafka.api.PlaintextConsumerTest > testPartitionsForInvalidTopic STARTED

kafka.api.PlaintextConsumerTest > testPartitionsForInvalidTopic PASSED

kafka.api.PlaintextConsumerTest > testSeek STARTED

kafka.api.PlaintextConsumerTest > testSeek PASSED

kafka.api.PlaintextConsumerTest > testPositionAndCommit STARTED

kafka.api.PlaintextConsumerTest > testPositionAndCommit PASSED

kafka.api.PlaintextConsumerTest > testMultiConsumerSessionTimeoutOnClose STARTED

kafka.api.PlaintextConsumerTest > testMultiConsumerSessionTimeoutOnClose PASSED

kafka.api.PlaintextConsumerTest > testFetchRecordTooLarge STARTED

kafka.api.PlaintextConsumerTest > testFetchRecordTooLarge PASSED

kafka.api.PlaintextConsumerTest > testMultiConsumerDefaultAssignment STARTED

kafka.api.PlaintextConsumerTest > testMultiConsumerDefaultAssignment PASSED

kafka.api.PlaintextConsumerTest > testAutoCommitOnClose STARTED

kafka.api.PlaintextConsumerTest > testAutoCommitOnClose PASSED

kafka.api.PlaintextConsumerTest > testExpandingTopicSubscriptions STARTED

kafka.api.PlaintextConsumerTest > testExpandingTopicSubscriptions PASSED

kafka.api.PlaintextConsumerTest > testInterceptors STARTED

kafka.api.PlaintextConsumerTest > testInterceptors PASSED

kafka.api.PlaintextConsumerTest > testPatternUnsubscription STARTED

kafka.api.PlaintextConsumerTest > testPatternUnsubscription PASSED

Build failed in Jenkins: kafka-trunk-jdk8 #712

[Apache Jenkins Server]

See 

Changes:

[me] KAFKA-3865: Fix transient failure in

--
[...truncated 10740 lines...]

org.apache.kafka.common.record.RecordTest > testFields[189] PASSED

org.apache.kafka.common.record.RecordTest > testChecksum[190] STARTED

org.apache.kafka.common.record.RecordTest > testChecksum[190] PASSED

org.apache.kafka.common.record.RecordTest > testEquality[190] STARTED

org.apache.kafka.common.record.RecordTest > testEquality[190] PASSED

org.apache.kafka.common.record.RecordTest > testFields[190] STARTED

org.apache.kafka.common.record.RecordTest > testFields[190] PASSED

org.apache.kafka.common.record.RecordTest > testChecksum[191] STARTED

org.apache.kafka.common.record.RecordTest > testChecksum[191] PASSED

org.apache.kafka.common.record.RecordTest > testEquality[191] STARTED

org.apache.kafka.common.record.RecordTest > testEquality[191] PASSED

org.apache.kafka.common.record.RecordTest > testFields[191] STARTED

org.apache.kafka.common.record.RecordTest > testFields[191] PASSED
:examples:checkstyleMain
:examples:compileTestJava UP-TO-DATE
:examples:processTestResources UP-TO-DATE
:examples:testClasses UP-TO-DATE
:examples:checkstyleTest UP-TO-DATE
:examples:test UP-TO-DATE
:log4j-appender:checkstyleMain
:log4j-appender:compileTestJavawarning: [options] bootstrap class path not set 
in conjunction with -source 1.7
1 warning

:log4j-appender:processTestResources UP-TO-DATE
:log4j-appender:testClasses
:log4j-appender:checkstyleTest
:log4j-appender:test

org.apache.kafka.log4jappender.KafkaLog4jAppenderTest > testLog4jAppends STARTED

org.apache.kafka.log4jappender.KafkaLog4jAppenderTest > testLog4jAppends PASSED

org.apache.kafka.log4jappender.KafkaLog4jAppenderTest > testKafkaLog4jConfigs 
STARTED

org.apache.kafka.log4jappender.KafkaLog4jAppenderTest > testKafkaLog4jConfigs 
PASSED
:streams:checkstyleMain
:streams:compileTestJavawarning: [options] bootstrap class path not set in 
conjunction with -source 1.7
Note: Some input files use unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
1 warning

:streams:processTestResources
:streams:testClasses
:streams:checkstyleTest
:streams:test

org.apache.kafka.streams.KeyValueTest > shouldHaveSaneEqualsAndHashCode STARTED

org.apache.kafka.streams.KeyValueTest > shouldHaveSaneEqualsAndHashCode PASSED

org.apache.kafka.streams.state.internals.RocksDBWindowStoreTest > 
testPutAndFetch STARTED

org.apache.kafka.streams.state.internals.RocksDBWindowStoreTest > 
testPutAndFetch PASSED

org.apache.kafka.streams.state.internals.RocksDBWindowStoreTest > 
testPutAndFetchBefore STARTED

org.apache.kafka.streams.state.internals.RocksDBWindowStoreTest > 
testPutAndFetchBefore PASSED

org.apache.kafka.streams.state.internals.RocksDBWindowStoreTest > 
testInitialLoading STARTED

org.apache.kafka.streams.state.internals.RocksDBWindowStoreTest > 
testInitialLoading PASSED

org.apache.kafka.streams.state.internals.RocksDBWindowStoreTest > testRestore 
STARTED

org.apache.kafka.streams.state.internals.RocksDBWindowStoreTest > testRestore 
PASSED

org.apache.kafka.streams.state.internals.RocksDBWindowStoreTest > testRolling 
STARTED

org.apache.kafka.streams.state.internals.RocksDBWindowStoreTest > testRolling 
PASSED

org.apache.kafka.streams.state.internals.RocksDBWindowStoreTest > 
testSegmentMaintenance STARTED

org.apache.kafka.streams.state.internals.RocksDBWindowStoreTest > 
testSegmentMaintenance PASSED

org.apache.kafka.streams.state.internals.RocksDBWindowStoreTest > 
testPutSameKeyTimestamp STARTED

org.apache.kafka.streams.state.internals.RocksDBWindowStoreTest > 
testPutSameKeyTimestamp PASSED

org.apache.kafka.streams.state.internals.RocksDBWindowStoreTest > 
testPutAndFetchAfter STARTED

org.apache.kafka.streams.state.internals.RocksDBWindowStoreTest > 
testPutAndFetchAfter PASSED

org.apache.kafka.streams.state.internals.StoreChangeLoggerTest > testAddRemove 
STARTED

org.apache.kafka.streams.state.internals.StoreChangeLoggerTest > testAddRemove 
PASSED

org.apache.kafka.streams.state.internals.OffsetCheckpointTest > testReadWrite 
STARTED

org.apache.kafka.streams.state.internals.OffsetCheckpointTest > testReadWrite 
PASSED

org.apache.kafka.streams.state.internals.InMemoryLRUCacheStoreTest > testEvict 
STARTED

org.apache.kafka.streams.state.internals.InMemoryLRUCacheStoreTest > testEvict 
PASSED

org.apache.kafka.streams.state.internals.InMemoryLRUCacheStoreTest > testSize 
STARTED

org.apache.kafka.streams.state.internals.InMemoryLRUCacheStoreTest > testSize 
PASSED

org.apache.kafka.streams.state.internals.InMemoryLRUCacheStoreTest > 
testPutIfAbsent STARTED

org.apache.kafka.streams.state.internals.InMemoryLRUCacheStoreTest > 
testPutIfAbsent PASSED

org.apache.kafka.streams.state.internals.InMemoryLRUCacheStoreTest > 
testRestoreWithDefaultSerdes STARTED

org.apache.kafka.streams.state.internals.InMemoryLRUCacheStoreTest > 

Build failed in Jenkins: kafka-0.10.0-jdk7 #132

[Apache Jenkins Server]

See 

Changes:

[me] MINOR: Catch Throwable in commitSourceTask()

[me] KAFKA-3865: Fix transient failure in

--
[...truncated 6345 lines...]

org.apache.kafka.connect.runtime.WorkerSourceTaskTest > testStartPaused PASSED

org.apache.kafka.connect.runtime.WorkerSourceTaskTest > testPause PASSED

org.apache.kafka.connect.runtime.WorkerSourceTaskTest > testPollsInBackground 
PASSED

org.apache.kafka.connect.runtime.WorkerSourceTaskTest > testFailureInPoll PASSED

org.apache.kafka.connect.runtime.WorkerSourceTaskTest > testCommit PASSED

org.apache.kafka.connect.runtime.WorkerSourceTaskTest > testCommitFailure PASSED

org.apache.kafka.connect.runtime.WorkerSourceTaskTest > 
testSendRecordsConvertsData PASSED

org.apache.kafka.connect.runtime.WorkerSourceTaskTest > testSendRecordsRetries 
PASSED

org.apache.kafka.connect.runtime.WorkerSourceTaskTest > 
testSendRecordsTaskCommitRecordFail PASSED

org.apache.kafka.connect.runtime.WorkerSourceTaskTest > testSlowTaskStart PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testPutConnectorConfig PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testJoinAssignment PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRebalance PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRebalanceFailedConnector PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testHaltCleansUpWorker PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testCreateConnector PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testCreateConnectorAlreadyExists PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testDestroyConnector PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartConnector PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartUnknownConnector PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartConnectorRedirectToLeader PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartConnectorRedirectToOwner PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartTask PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartUnknownTask PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartTaskRedirectToLeader PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testRestartTaskRedirectToOwner PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorConfigAdded PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorConfigUpdate PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorPaused PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorResumed PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testUnknownConnectorPaused PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorPausedRunningTaskOnly PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testConnectorResumedRunningTaskOnly PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testTaskConfigAdded PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testJoinLeaderCatchUpFails PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testAccessors PASSED

org.apache.kafka.connect.runtime.distributed.DistributedHerderTest > 
testInconsistentConfigs PASSED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testNormalJoinGroupFollower PASSED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testMetadata PASSED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testLeaderPerformAssignment1 PASSED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testLeaderPerformAssignment2 PASSED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testJoinLeaderCannotAssign PASSED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testRejoinGroup PASSED

org.apache.kafka.connect.runtime.distributed.WorkerCoordinatorTest > 
testNormalJoinGroupLeader PASSED

org.apache.kafka.connect.runtime.standalone.StandaloneHerderTest > 
testPutConnectorConfig PASSED

org.apache.kafka.connect.runtime.standalone.StandaloneHerderTest > 
testPutTaskConfigs PASSED

org.apache.kafka.connect.runtime.standalone.StandaloneHerderTest > 
testCreateSourceConnector PASSED

org.apache.kafka.connect.runtime.standalone.StandaloneHerderTest > 
testCreateSinkConnector PASSED

Re: [VOTE] KIP-4 Create Topics Schema

[Manikumar Reddy]

+1 (non-binding)

On Tue, Jun 21, 2016 at 9:16 AM, Ewen Cheslack-Postava 
wrote:

> +1 (binding) and thanks for the work on this Grant!
>
> -Ewen
>
> On Mon, Jun 20, 2016 at 12:18 PM, Gwen Shapira  wrote:
>
> > +1 (binding)
> >
> > On Mon, Jun 20, 2016 at 12:13 PM, Tom Crayford 
> > wrote:
> > > +1 (non-binding)
> > >
> > > On Mon, Jun 20, 2016 at 8:07 PM, Harsha  wrote:
> > >
> > >> +1 (binding)
> > >> -Harsha
> > >>
> > >> On Mon, Jun 20, 2016, at 11:33 AM, Ismael Juma wrote:
> > >> > +1 (binding)
> > >> >
> > >> > On Mon, Jun 20, 2016 at 8:27 PM, Dana Powers  >
> > >> > wrote:
> > >> >
> > >> > > +1 -- thanks for the update
> > >> > >
> > >> > > On Mon, Jun 20, 2016 at 10:49 AM, Grant Henke <
> ghe...@cloudera.com>
> > >> wrote:
> > >> > > > I have update the patch and wiki based on the feedback in the
> > >> discussion
> > >> > > > thread. The only change is that instead of logging and
> > disconnecting
> > >> in
> > >> > > the
> > >> > > > case of invalid messages (duplicate topics or both arguments) we
> > now
> > >> > > return
> > >> > > > and InvalidRequest error back to the client for that topic.
> > >> > > >
> > >> > > > I would like to restart the vote now including that change. If
> you
> > >> have
> > >> > > > already voted, please revote in this thread.
> > >> > > >
> > >> > > > Thank you,
> > >> > > > Grant
> > >> > > >
> > >> > > > On Sun, Jun 19, 2016 at 8:57 PM, Ewen Cheslack-Postava <
> > >> > > e...@confluent.io>
> > >> > > > wrote:
> > >> > > >
> > >> > > >> Don't necessarily want to add noise here, but I'm -1 based on
> the
> > >> > > >> disconnect part. See discussion in other thread. (I'm +1
> > otherwise,
> > >> and
> > >> > > >> happy to have my vote applied assuming we clean up that one
> > issue.)
> > >> > > >>
> > >> > > >> -Ewen
> > >> > > >>
> > >> > > >> On Thu, Jun 16, 2016 at 6:05 PM, Harsha 
> wrote:
> > >> > > >>
> > >> > > >> > +1 (binding)
> > >> > > >> > Thanks,
> > >> > > >> > Harsha
> > >> > > >> >
> > >> > > >> > On Thu, Jun 16, 2016, at 04:15 PM, Guozhang Wang wrote:
> > >> > > >> > > +1.
> > >> > > >> > >
> > >> > > >> > > On Thu, Jun 16, 2016 at 3:47 PM, Ismael Juma <
> > ism...@juma.me.uk
> > >> >
> > >> > > >> wrote:
> > >> > > >> > >
> > >> > > >> > > > +1 (binding)
> > >> > > >> > > >
> > >> > > >> > > > On Thu, Jun 16, 2016 at 11:50 PM, Grant Henke <
> > >> > > ghe...@cloudera.com>
> > >> > > >> > wrote:
> > >> > > >> > > >
> > >> > > >> > > > > I would like to initiate the voting process for the
> > "KIP-4
> > >> > > Create
> > >> > > >> > Topics
> > >> > > >> > > > > Schema changes". This is not a vote for all of KIP-4,
> but
> > >> > > >> > specifically
> > >> > > >> > > > for
> > >> > > >> > > > > the create topics changes. I have included the exact
> > changes
> > >> > > below
> > >> > > >> > for
> > >> > > >> > > > > clarity:
> > >> > > >> > > > > >
> > >> > > >> > > > > > Create Topics Request (KAFKA-2945
> > >> > > >> > > > > > )
> > >> > > >> > > > > >
> > >> > > >> > > > > > CreateTopics Request (Version: 0) =>
> > >> [create_topic_requests]
> > >> > > >> > timeout
> > >> > > >> > > > > >   create_topic_requests => topic num_partitions
> > >> > > >> replication_factor
> > >> > > >> > > > > [replica_assignment] [configs]
> > >> > > >> > > > > > topic => STRING
> > >> > > >> > > > > > num_partitions => INT32
> > >> > > >> > > > > > replication_factor => INT16
> > >> > > >> > > > > > replica_assignment => partition_id [replicas]
> > >> > > >> > > > > >   partition_id => INT32
> > >> > > >> > > > > >   replicas => INT32
> > >> > > >> > > > > > configs => config_key config_value
> > >> > > >> > > > > >   config_key => STRING
> > >> > > >> > > > > >   config_value => STRING
> > >> > > >> > > > > >   timeout => INT32
> > >> > > >> > > > > >
> > >> > > >> > > > > > CreateTopicsRequest is a batch request to initiate
> > topic
> > >> > > creation
> > >> > > >> > with
> > >> > > >> > > > > > either predefined or automatic replica assignment and
> > >> > > optionally
> > >> > > >> > topic
> > >> > > >> > > > > > configuration.
> > >> > > >> > > > > >
> > >> > > >> > > > > > Request semantics:
> > >> > > >> > > > > >
> > >> > > >> > > > > >1. Must be sent to the controller broker
> > >> > > >> > > > > >2. If there are multiple instructions for the same
> > >> topic in
> > >> > > >> one
> > >> > > >> > > > > >request an InvalidRequestException will be logged
> on
> > >> the
> > >> > > >> broker
> > >> > > >> > and
> > >> > > >> > > > > the
> > >> > > >> > > > > >client will be disconnected.
> > >> > > >> > > > > >   - This is because the list of topics is modeled
> > >> server
> > >> > > side
> > >> > > >> > as a
> > >> > > >> > > > > >   map with TopicName as the key
> > >> > > >> > > > > >3. The principal must be authorized to the

[jira] [Commented] (KAFKA-3872) OOM while running Kafka Streams integration tests

[ASF GitHub Bot (JIRA)]

[ 
https://issues.apache.org/jira/browse/KAFKA-3872?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel=15341178#comment-15341178
 ] 

ASF GitHub Bot commented on KAFKA-3872:
---

GitHub user enothereska opened a pull request:

https://github.com/apache/kafka/pull/1533

KAFKA-3872: Reduce log cleaner buffer size to 2 MB



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/enothereska/kafka 
KAFKA-3872-oom-integration-tests

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1533.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1533


commit 8d92325e7fbdfd65b64a9bcbee08ebabb861d2b5
Author: Eno Thereska 
Date:   2016-06-21T06:08:06Z

Reduce log cleaner buffer size to 2 MB




> OOM while running Kafka Streams integration tests
> -
>
> Key: KAFKA-3872
> URL: https://issues.apache.org/jira/browse/KAFKA-3872
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams, unit tests
>Reporter: Ismael Juma
>Assignee: Eno Thereska
>  Labels: transient-unit-test-failure
>
> Failures:
> org.apache.kafka.streams.integration.FanoutIntegrationTest.classMethod
> org.apache.kafka.streams.integration.WordCountIntegrationTest.classMethod
> Stracktrace:
> {code}
> java.lang.OutOfMemoryError: Java heap space
>   at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57)
>   at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
>   at kafka.log.SkimpyOffsetMap.(OffsetMap.scala:43)
>   at kafka.log.LogCleaner$CleanerThread.(LogCleaner.scala:193)
>   at kafka.log.LogCleaner$$anonfun$1.apply(LogCleaner.scala:83)
>   at kafka.log.LogCleaner$$anonfun$1.apply(LogCleaner.scala:83)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at scala.collection.immutable.Range.foreach(Range.scala:141)
>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
>   at scala.collection.AbstractTraversable.map(Traversable.scala:105)
>   at kafka.log.LogCleaner.(LogCleaner.scala:83)
>   at kafka.log.LogManager.(LogManager.scala:66)
>   at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:609)
>   at kafka.server.KafkaServer.startup(KafkaServer.scala:183)
>   at kafka.utils.TestUtils$.createServer(TestUtils.scala:132)
>   at kafka.utils.TestUtils.createServer(TestUtils.scala)
>   at 
> org.apache.kafka.streams.integration.utils.KafkaEmbedded.(KafkaEmbedded.java:79)
>   at 
> org.apache.kafka.streams.integration.utils.EmbeddedSingleNodeKafkaCluster.start(EmbeddedSingleNodeKafkaCluster.java:54)
>   at 
> {code}
> Two builds:
> https://builds.apache.org/job/kafka-trunk-jdk8/704/
> https://builds.apache.org/job/kafka-trunk-jdk8/705



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[GitHub] kafka pull request #1533: KAFKA-3872: Reduce log cleaner buffer size to 2 MB

[enothereska]

GitHub user enothereska opened a pull request:

https://github.com/apache/kafka/pull/1533

KAFKA-3872: Reduce log cleaner buffer size to 2 MB



You can merge this pull request into a Git repository by running:

$ git pull https://github.com/enothereska/kafka 
KAFKA-3872-oom-integration-tests

Alternatively you can review and apply these changes as the patch at:

https://github.com/apache/kafka/pull/1533.patch

To close this pull request, make a commit to your master/trunk branch
with (at least) the following in the commit message:

This closes #1533


commit 8d92325e7fbdfd65b64a9bcbee08ebabb861d2b5
Author: Eno Thereska 
Date:   2016-06-21T06:08:06Z

Reduce log cleaner buffer size to 2 MB




---
If your project is set up for it, you can reply to this email and have your
reply appear on GitHub as well. If your project does not have this feature
enabled and wishes so, or if the feature is enabled but not working, please
contact infrastructure at infrastruct...@apache.org or file a JIRA ticket
with INFRA.
---

[jira] [Work started] (KAFKA-3872) OOM while running Kafka Streams integration tests

[Eno Thereska (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Work on KAFKA-3872 started by Eno Thereska.
---
> OOM while running Kafka Streams integration tests
> -
>
> Key: KAFKA-3872
> URL: https://issues.apache.org/jira/browse/KAFKA-3872
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams, unit tests
>Reporter: Ismael Juma
>Assignee: Eno Thereska
>  Labels: transient-unit-test-failure
>
> Failures:
> org.apache.kafka.streams.integration.FanoutIntegrationTest.classMethod
> org.apache.kafka.streams.integration.WordCountIntegrationTest.classMethod
> Stracktrace:
> {code}
> java.lang.OutOfMemoryError: Java heap space
>   at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57)
>   at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
>   at kafka.log.SkimpyOffsetMap.(OffsetMap.scala:43)
>   at kafka.log.LogCleaner$CleanerThread.(LogCleaner.scala:193)
>   at kafka.log.LogCleaner$$anonfun$1.apply(LogCleaner.scala:83)
>   at kafka.log.LogCleaner$$anonfun$1.apply(LogCleaner.scala:83)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at scala.collection.immutable.Range.foreach(Range.scala:141)
>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
>   at scala.collection.AbstractTraversable.map(Traversable.scala:105)
>   at kafka.log.LogCleaner.(LogCleaner.scala:83)
>   at kafka.log.LogManager.(LogManager.scala:66)
>   at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:609)
>   at kafka.server.KafkaServer.startup(KafkaServer.scala:183)
>   at kafka.utils.TestUtils$.createServer(TestUtils.scala:132)
>   at kafka.utils.TestUtils.createServer(TestUtils.scala)
>   at 
> org.apache.kafka.streams.integration.utils.KafkaEmbedded.(KafkaEmbedded.java:79)
>   at 
> org.apache.kafka.streams.integration.utils.EmbeddedSingleNodeKafkaCluster.start(EmbeddedSingleNodeKafkaCluster.java:54)
>   at 
> {code}
> Two builds:
> https://builds.apache.org/job/kafka-trunk-jdk8/704/
> https://builds.apache.org/job/kafka-trunk-jdk8/705



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

[jira] [Assigned] (KAFKA-3872) OOM while running Kafka Streams integration tests

[Eno Thereska (JIRA)]

 [ 
https://issues.apache.org/jira/browse/KAFKA-3872?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
 ]

Eno Thereska reassigned KAFKA-3872:
---

Assignee: Eno Thereska  (was: Guozhang Wang)

> OOM while running Kafka Streams integration tests
> -
>
> Key: KAFKA-3872
> URL: https://issues.apache.org/jira/browse/KAFKA-3872
> Project: Kafka
>  Issue Type: Sub-task
>  Components: streams, unit tests
>Reporter: Ismael Juma
>Assignee: Eno Thereska
>  Labels: transient-unit-test-failure
>
> Failures:
> org.apache.kafka.streams.integration.FanoutIntegrationTest.classMethod
> org.apache.kafka.streams.integration.WordCountIntegrationTest.classMethod
> Stracktrace:
> {code}
> java.lang.OutOfMemoryError: Java heap space
>   at java.nio.HeapByteBuffer.(HeapByteBuffer.java:57)
>   at java.nio.ByteBuffer.allocate(ByteBuffer.java:335)
>   at kafka.log.SkimpyOffsetMap.(OffsetMap.scala:43)
>   at kafka.log.LogCleaner$CleanerThread.(LogCleaner.scala:193)
>   at kafka.log.LogCleaner$$anonfun$1.apply(LogCleaner.scala:83)
>   at kafka.log.LogCleaner$$anonfun$1.apply(LogCleaner.scala:83)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at 
> scala.collection.TraversableLike$$anonfun$map$1.apply(TraversableLike.scala:244)
>   at scala.collection.immutable.Range.foreach(Range.scala:141)
>   at scala.collection.TraversableLike$class.map(TraversableLike.scala:244)
>   at scala.collection.AbstractTraversable.map(Traversable.scala:105)
>   at kafka.log.LogCleaner.(LogCleaner.scala:83)
>   at kafka.log.LogManager.(LogManager.scala:66)
>   at kafka.server.KafkaServer.createLogManager(KafkaServer.scala:609)
>   at kafka.server.KafkaServer.startup(KafkaServer.scala:183)
>   at kafka.utils.TestUtils$.createServer(TestUtils.scala:132)
>   at kafka.utils.TestUtils.createServer(TestUtils.scala)
>   at 
> org.apache.kafka.streams.integration.utils.KafkaEmbedded.(KafkaEmbedded.java:79)
>   at 
> org.apache.kafka.streams.integration.utils.EmbeddedSingleNodeKafkaCluster.start(EmbeddedSingleNodeKafkaCluster.java:54)
>   at 
> {code}
> Two builds:
> https://builds.apache.org/job/kafka-trunk-jdk8/704/
> https://builds.apache.org/job/kafka-trunk-jdk8/705



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)